2024-09-02T09:48:13.487 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-02T09:48:13.507 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-02T09:48:13.609 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453 branch: wip-bharath3-testing-2024-09-01-1328 description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} email: skanta@redhat.com first_in_suite: false job_id: '7886453' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath3-testing-2024-09-01-1328 ceph: conf: client.2: debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd ms inject internal delays: 0.002 ms inject socket failures: 2500 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 3 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: min_in: 4 workunit: branch: wip-bharath3-testing-2024-09-01-1328 sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 2587 sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath3-testing-2024-09-01-1328 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_82daa6ef0815320144b25b212c6b8c25a1d59c07/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 targets: smithi151.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBP38StV7/mUabgpOQbcIWCtXizDU/U2HBz5qmEvzvbQvi2avoS0J5RXWAv1pylLOMOMuUEaCxeDL1eVcN7+ys4= smithi162.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH18nFgXSOGUlTKte/crjetHG9PSuEVdjh9pxq/T1UavEDXRrnqHGM7vdZgAVa8U9ga1baRkwte1NepB8q6r52A= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOJkSEF8p6yEXhMfBJZSfTvZuLHLKF2Hvp0cMdClklWm6UdNiv6H1qIivGMAsCz+nH25trm8IHhcbxWXCHei8qk= tasks: - install: branch: squid exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 - radosbench: clients: - client.2 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-02_08:22:44 tube: smithi user: skanta verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-02T09:48:13.610 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_82daa6ef0815320144b25b212c6b8c25a1d59c07/qa; will attempt to use it 2024-09-02T09:48:13.610 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_82daa6ef0815320144b25b212c6b8c25a1d59c07/qa/tasks 2024-09-02T09:48:13.610 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-02T09:48:13.612 INFO:teuthology.task.internal:Saving configuration 2024-09-02T09:48:13.627 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-02T09:48:13.634 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-02T09:48:13.666 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi151.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-02 09:43:03.669579', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBP38StV7/mUabgpOQbcIWCtXizDU/U2HBz5qmEvzvbQvi2avoS0J5RXWAv1pylLOMOMuUEaCxeDL1eVcN7+ys4='} 2024-09-02T09:48:13.699 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi162.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-02 09:43:03.671033', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH18nFgXSOGUlTKte/crjetHG9PSuEVdjh9pxq/T1UavEDXRrnqHGM7vdZgAVa8U9ga1baRkwte1NepB8q6r52A='} 2024-09-02T09:48:13.729 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-02 09:43:03.672141', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOJkSEF8p6yEXhMfBJZSfTvZuLHLKF2Hvp0cMdClklWm6UdNiv6H1qIivGMAsCz+nH25trm8IHhcbxWXCHei8qk='} 2024-09-02T09:48:13.729 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-02T09:48:13.739 INFO:teuthology.task.internal:roles: ubuntu@smithi151.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-02T09:48:13.739 INFO:teuthology.task.internal:roles: ubuntu@smithi162.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-09-02T09:48:13.739 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-09-02T09:48:13.739 INFO:teuthology.run_tasks:Running task console_log... 2024-09-02T09:48:13.880 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbdb6157be0>, signals=[15]) 2024-09-02T09:48:13.880 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-02T09:48:13.887 INFO:teuthology.task.internal:Opening connections... 2024-09-02T09:48:13.887 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi151.front.sepia.ceph.com 2024-09-02T09:48:13.888 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T09:48:13.972 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi162.front.sepia.ceph.com 2024-09-02T09:48:13.973 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T09:48:14.054 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2024-09-02T09:48:14.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T09:48:14.140 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-02T09:48:14.147 DEBUG:teuthology.orchestra.run.smithi151:> uname -m 2024-09-02T09:48:14.166 INFO:teuthology.orchestra.run.smithi151.stdout:x86_64 2024-09-02T09:48:14.166 DEBUG:teuthology.orchestra.run.smithi151:> cat /etc/os-release 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:NAME="CentOS Stream" 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION="9" 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:ID="centos" 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:ID_LIKE="rhel fedora" 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:VERSION_ID="9" 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:PLATFORM_ID="platform:el9" 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:ANSI_COLOR="0;31" 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:LOGO="fedora-logo-icon" 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-02T09:48:14.224 INFO:teuthology.orchestra.run.smithi151.stdout:HOME_URL="https://centos.org/" 2024-09-02T09:48:14.225 INFO:teuthology.orchestra.run.smithi151.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-02T09:48:14.225 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-02T09:48:14.225 INFO:teuthology.orchestra.run.smithi151.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-02T09:48:14.225 INFO:teuthology.lock.ops:Updating smithi151.front.sepia.ceph.com on lock server 2024-09-02T09:48:14.258 DEBUG:teuthology.orchestra.run.smithi162:> uname -m 2024-09-02T09:48:14.276 INFO:teuthology.orchestra.run.smithi162.stdout:x86_64 2024-09-02T09:48:14.277 DEBUG:teuthology.orchestra.run.smithi162:> cat /etc/os-release 2024-09-02T09:48:14.335 INFO:teuthology.orchestra.run.smithi162.stdout:NAME="CentOS Stream" 2024-09-02T09:48:14.335 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION="9" 2024-09-02T09:48:14.335 INFO:teuthology.orchestra.run.smithi162.stdout:ID="centos" 2024-09-02T09:48:14.335 INFO:teuthology.orchestra.run.smithi162.stdout:ID_LIKE="rhel fedora" 2024-09-02T09:48:14.335 INFO:teuthology.orchestra.run.smithi162.stdout:VERSION_ID="9" 2024-09-02T09:48:14.335 INFO:teuthology.orchestra.run.smithi162.stdout:PLATFORM_ID="platform:el9" 2024-09-02T09:48:14.335 INFO:teuthology.orchestra.run.smithi162.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-02T09:48:14.335 INFO:teuthology.orchestra.run.smithi162.stdout:ANSI_COLOR="0;31" 2024-09-02T09:48:14.335 INFO:teuthology.orchestra.run.smithi162.stdout:LOGO="fedora-logo-icon" 2024-09-02T09:48:14.335 INFO:teuthology.orchestra.run.smithi162.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-02T09:48:14.336 INFO:teuthology.orchestra.run.smithi162.stdout:HOME_URL="https://centos.org/" 2024-09-02T09:48:14.336 INFO:teuthology.orchestra.run.smithi162.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-02T09:48:14.336 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-02T09:48:14.336 INFO:teuthology.orchestra.run.smithi162.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-02T09:48:14.336 INFO:teuthology.lock.ops:Updating smithi162.front.sepia.ceph.com on lock server 2024-09-02T09:48:14.367 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2024-09-02T09:48:14.385 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2024-09-02T09:48:14.385 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2024-09-02T09:48:14.443 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2024-09-02T09:48:14.443 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="9" 2024-09-02T09:48:14.444 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2024-09-02T09:48:14.444 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2024-09-02T09:48:14.444 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="9" 2024-09-02T09:48:14.444 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el9" 2024-09-02T09:48:14.444 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-02T09:48:14.444 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2024-09-02T09:48:14.444 INFO:teuthology.orchestra.run.smithi187.stdout:LOGO="fedora-logo-icon" 2024-09-02T09:48:14.444 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-02T09:48:14.444 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2024-09-02T09:48:14.444 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-02T09:48:14.445 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-02T09:48:14.445 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-02T09:48:14.445 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2024-09-02T09:48:14.475 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-02T09:48:14.484 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-02T09:48:14.513 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-02T09:48:14.513 DEBUG:teuthology.orchestra.run.smithi151:> test '!' -e /home/ubuntu/cephtest 2024-09-02T09:48:14.516 DEBUG:teuthology.orchestra.run.smithi162:> test '!' -e /home/ubuntu/cephtest 2024-09-02T09:48:14.518 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2024-09-02T09:48:14.535 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-02T09:48:14.541 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-02T09:48:14.542 DEBUG:teuthology.orchestra.run.smithi151:> test -z $(ls -A /var/lib/ceph) 2024-09-02T09:48:14.573 DEBUG:teuthology.orchestra.run.smithi162:> test -z $(ls -A /var/lib/ceph) 2024-09-02T09:48:14.576 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2024-09-02T09:48:14.672 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-02T09:48:14.777 INFO:teuthology.run_tasks:Running task kernel... 2024-09-02T09:48:14.793 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-02T09:48:14.793 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-02T09:48:14.793 DEBUG:teuthology.orchestra.run.smithi151:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-02T09:48:14.794 DEBUG:teuthology.orchestra.run.smithi162:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-02T09:48:14.794 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-02T09:48:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T09:48:14.811 DEBUG:teuthology.orchestra.run.smithi162:> uname -r 2024-09-02T09:48:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T09:48:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T09:48:14.813 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-09-02T09:48:14.813 DEBUG:teuthology.orchestra.run.smithi151:> uname -r 2024-09-02T09:48:14.869 INFO:teuthology.orchestra.run.smithi162.stdout:5.14.0-503.el9.x86_64 2024-09-02T09:48:14.870 INFO:teuthology.task.kernel:Running kernel on smithi162: 5.14.0-503.el9.x86_64 2024-09-02T09:48:14.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum install -y kernel 2024-09-02T09:48:14.871 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-503.el9.x86_64 2024-09-02T09:48:14.871 INFO:teuthology.task.kernel:Running kernel on smithi187: 5.14.0-503.el9.x86_64 2024-09-02T09:48:14.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2024-09-02T09:48:14.872 INFO:teuthology.orchestra.run.smithi151.stdout:5.14.0-503.el9.x86_64 2024-09-02T09:48:14.872 INFO:teuthology.task.kernel:Running kernel on smithi151: 5.14.0-503.el9.x86_64 2024-09-02T09:48:14.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum install -y kernel 2024-09-02T09:48:17.291 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-02T09:48:18.678 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.2 MB 00:01 2024-09-02T09:48:18.685 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.2 MB 00:02 2024-09-02T09:48:21.858 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 20 MB 00:03 2024-09-02T09:48:27.228 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 20 MB 00:07 2024-09-02T09:48:28.224 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.6 MB 00:02 2024-09-02T09:48:29.916 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 18 kB 00:00 2024-09-02T09:48:31.471 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 16 MB/s | 23 MB 00:01 2024-09-02T09:48:33.544 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.6 MB 00:02 2024-09-02T09:48:35.003 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 18 kB 00:00 2024-09-02T09:48:35.757 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-02T09:48:36.892 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-02T09:48:36.985 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:17 2024-09-02T09:48:37.084 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:01 ago on Mon 02 Sep 2024 09:48:36 AM UTC. 2024-09-02T09:48:38.924 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-02T09:48:38.924 INFO:teuthology.orchestra.run.smithi162.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-02T09:48:39.008 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:48:39.009 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-02T09:48:39.009 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:48:39.119 DEBUG:teuthology.orchestra.run.smithi162:> echo no | sudo yum reinstall kernel || true 2024-09-02T09:48:39.659 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:03 ago on Mon 02 Sep 2024 09:48:36 AM UTC. 2024-09-02T09:48:39.794 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:48:39.806 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:48:39.806 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-02T09:48:39.806 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:48:39.806 INFO:teuthology.orchestra.run.smithi162.stdout:Reinstalling: 2024-09-02T09:48:39.807 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T09:48:39.807 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:48:39.807 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:48:39.807 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:48:39.807 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:48:39.807 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.9 M 2024-09-02T09:48:39.807 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 0 2024-09-02T09:48:39.807 INFO:teuthology.orchestra.run.smithi162.stderr:Operation aborted. 2024-09-02T09:48:39.864 INFO:teuthology.orchestra.run.smithi162.stdout:Is this ok [y/N]: 2024-09-02T09:48:39.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum reinstall -y kernel || true 2024-09-02T09:48:40.396 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:04 ago on Mon 02 Sep 2024 09:48:36 AM UTC. 2024-09-02T09:48:40.522 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:48:40.524 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:48:40.524 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-02T09:48:40.524 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:48:40.525 INFO:teuthology.orchestra.run.smithi162.stdout:Reinstalling: 2024-09-02T09:48:40.525 INFO:teuthology.orchestra.run.smithi162.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T09:48:40.525 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:48:40.525 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:48:40.525 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:48:40.525 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:48:40.525 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 1.9 M 2024-09-02T09:48:40.525 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 0 2024-09-02T09:48:40.525 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:48:40.753 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-503.el9.x86_64.rpm 9.0 MB/s | 1.9 MB 00:00 2024-09-02T09:48:40.753 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:48:40.753 INFO:teuthology.orchestra.run.smithi162.stdout:Total 8.2 MB/s | 1.9 MB 00:00 2024-09-02T09:48:40.781 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:48:40.999 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:48:40.999 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:48:41.029 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:48:41.029 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:48:41.180 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-02T09:48:41.211 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:48:41.305 INFO:teuthology.orchestra.run.smithi162.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T09:48:41.305 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T09:48:42.126 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T09:48:42.126 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T09:48:42.202 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 6.6 MB 00:01 2024-09-02T09:48:42.863 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T09:48:42.863 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:48:42.863 INFO:teuthology.orchestra.run.smithi162.stdout:Reinstalled: 2024-09-02T09:48:42.863 INFO:teuthology.orchestra.run.smithi162.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-02T09:48:42.863 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:48:42.863 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:48:43.070 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q kernel | sort -rV | head -n 1 2024-09-02T09:48:43.128 INFO:teuthology.orchestra.run.smithi162.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-02T09:48:43.129 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-02T09:48:43.129 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-02T09:48:43.130 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-02T09:48:43.130 DEBUG:teuthology.orchestra.run.smithi162:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-02T09:48:43.161 INFO:teuthology.orchestra.run.smithi162.stdout:ttyS1 2024-09-02T09:48:43.188 DEBUG:teuthology.parallel:result is None 2024-09-02T09:48:43.219 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-02T09:48:43.220 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-02T09:48:43.302 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:48:43.303 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-02T09:48:43.303 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:48:43.405 DEBUG:teuthology.orchestra.run.smithi187:> echo no | sudo yum reinstall kernel || true 2024-09-02T09:48:43.661 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 18 kB 00:00 2024-09-02T09:48:43.944 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:02 ago on Mon 02 Sep 2024 09:48:41 AM UTC. 2024-09-02T09:48:44.088 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:48:44.094 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:48:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-02T09:48:44.094 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:48:44.094 INFO:teuthology.orchestra.run.smithi187.stdout:Reinstalling: 2024-09-02T09:48:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T09:48:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:48:44.094 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:48:44.094 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:48:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:48:44.094 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 1.9 M 2024-09-02T09:48:44.095 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 0 2024-09-02T09:48:44.095 INFO:teuthology.orchestra.run.smithi187.stderr:Operation aborted. 2024-09-02T09:48:44.157 INFO:teuthology.orchestra.run.smithi187.stdout:Is this ok [y/N]: 2024-09-02T09:48:44.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum reinstall -y kernel || true 2024-09-02T09:48:44.694 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:03 ago on Mon 02 Sep 2024 09:48:41 AM UTC. 2024-09-02T09:48:44.732 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-02T09:48:44.821 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:48:44.822 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:48:44.822 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-02T09:48:44.822 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:48:44.822 INFO:teuthology.orchestra.run.smithi187.stdout:Reinstalling: 2024-09-02T09:48:44.822 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T09:48:44.822 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:48:44.823 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:48:44.823 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:48:44.823 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:48:44.823 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 1.9 M 2024-09-02T09:48:44.823 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 0 2024-09-02T09:48:44.823 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:48:45.248 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-503.el9.x86_64.rpm 5.7 MB/s | 1.9 MB 00:00 2024-09-02T09:48:45.248 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:48:45.248 INFO:teuthology.orchestra.run.smithi187.stdout:Total 4.4 MB/s | 1.9 MB 00:00 2024-09-02T09:48:45.277 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:48:45.506 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:48:45.506 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:48:45.537 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:48:45.537 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:48:45.727 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:48:45.825 INFO:teuthology.orchestra.run.smithi187.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T09:48:45.825 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T09:48:46.550 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T09:48:46.550 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T09:48:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T09:48:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:48:47.324 INFO:teuthology.orchestra.run.smithi187.stdout:Reinstalled: 2024-09-02T09:48:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-02T09:48:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:48:47.324 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:48:47.521 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-09-02T09:48:47.580 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-02T09:48:47.581 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-02T09:48:47.581 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-02T09:48:47.581 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-02T09:48:47.581 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-02T09:48:47.615 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2024-09-02T09:48:47.633 DEBUG:teuthology.parallel:result is None 2024-09-02T09:48:50.179 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-02T09:48:52.211 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-02T09:48:52.211 INFO:teuthology.orchestra.run.smithi151.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-02T09:48:52.289 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:48:52.290 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-02T09:48:52.290 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:48:52.379 DEBUG:teuthology.orchestra.run.smithi151:> echo no | sudo yum reinstall kernel || true 2024-09-02T09:48:52.920 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:02 ago on Mon 02 Sep 2024 09:48:50 AM UTC. 2024-09-02T09:48:53.059 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:48:53.070 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:48:53.070 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-09-02T09:48:53.070 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:48:53.070 INFO:teuthology.orchestra.run.smithi151.stdout:Reinstalling: 2024-09-02T09:48:53.070 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T09:48:53.070 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:48:53.070 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:48:53.070 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:48:53.070 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:48:53.070 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 1.9 M 2024-09-02T09:48:53.071 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 0 2024-09-02T09:48:53.071 INFO:teuthology.orchestra.run.smithi151.stderr:Operation aborted. 2024-09-02T09:48:53.149 INFO:teuthology.orchestra.run.smithi151.stdout:Is this ok [y/N]: 2024-09-02T09:48:53.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum reinstall -y kernel || true 2024-09-02T09:48:53.686 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:03 ago on Mon 02 Sep 2024 09:48:50 AM UTC. 2024-09-02T09:48:53.815 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout:Reinstalling: 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout: kernel x86_64 5.14.0-503.el9 baseos 1.9 M 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 1.9 M 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 0 2024-09-02T09:48:53.816 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:48:54.490 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-503.el9.x86_64.rpm 2.9 MB/s | 1.9 MB 00:00 2024-09-02T09:48:54.490 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:48:54.490 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.8 MB/s | 1.9 MB 00:00 2024-09-02T09:48:54.519 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:48:54.768 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:48:54.769 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:48:54.799 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:48:54.799 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:48:54.954 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:48:55.033 INFO:teuthology.orchestra.run.smithi151.stdout: Reinstalling : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T09:48:55.034 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T09:48:55.775 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T09:48:55.775 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 1/2 2024-09-02T09:48:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : kernel-5.14.0-503.el9.x86_64 2/2 2024-09-02T09:48:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:48:56.503 INFO:teuthology.orchestra.run.smithi151.stdout:Reinstalled: 2024-09-02T09:48:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: kernel-5.14.0-503.el9.x86_64 2024-09-02T09:48:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:48:56.503 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:48:56.712 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q kernel | sort -rV | head -n 1 2024-09-02T09:48:56.770 INFO:teuthology.orchestra.run.smithi151.stdout:kernel-5.14.0-503.el9.x86_64 2024-09-02T09:48:56.771 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-503.el9.x86_64 2024-09-02T09:48:56.771 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-09-02T09:48:56.771 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-02T09:48:56.771 DEBUG:teuthology.orchestra.run.smithi151:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-02T09:48:56.807 INFO:teuthology.orchestra.run.smithi151.stdout:ttyS1 2024-09-02T09:48:56.827 DEBUG:teuthology.parallel:result is None 2024-09-02T09:48:56.827 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-02T09:48:56.834 INFO:teuthology.task.internal:Creating test directory... 2024-09-02T09:48:56.834 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-02T09:48:56.870 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-02T09:48:56.872 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-02T09:48:56.890 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-02T09:48:56.896 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-02T09:48:56.924 INFO:teuthology.task.internal:Creating archive directory... 2024-09-02T09:48:56.924 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-02T09:48:56.927 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-02T09:48:56.931 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-02T09:48:56.999 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-02T09:48:57.004 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-02T09:48:57.005 DEBUG:teuthology.orchestra.run.smithi151:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-02T09:48:57.007 DEBUG:teuthology.orchestra.run.smithi162:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-02T09:48:57.009 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-02T09:48:57.046 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T09:48:57.059 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T09:48:57.061 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T09:48:57.071 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T09:48:57.071 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T09:48:57.083 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-02T09:48:57.084 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-02T09:48:57.090 INFO:teuthology.task.internal:Configuring sudo... 2024-09-02T09:48:57.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-02T09:48:57.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-02T09:48:57.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-02T09:48:57.129 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-02T09:48:57.137 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-02T09:48:57.137 DEBUG:teuthology.orchestra.run.smithi151:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-02T09:48:57.158 DEBUG:teuthology.orchestra.run.smithi162:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-02T09:48:57.164 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-02T09:48:57.186 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T09:48:57.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T09:48:57.301 DEBUG:teuthology.orchestra.run.smithi151:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T09:48:57.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T09:48:57.423 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:48:57.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-02T09:48:57.489 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T09:48:57.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T09:48:57.584 DEBUG:teuthology.orchestra.run.smithi162:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T09:48:57.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T09:48:57.711 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:48:57.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-02T09:48:57.778 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T09:48:57.832 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-02T09:48:57.880 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T09:48:57.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-02T09:48:58.008 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:48:58.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-02T09:48:58.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo service rsyslog restart 2024-09-02T09:48:58.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo service rsyslog restart 2024-09-02T09:48:58.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2024-09-02T09:48:58.114 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T09:48:58.116 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T09:48:58.156 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T09:48:58.633 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-02T09:48:58.640 INFO:teuthology.task.internal:Starting timer... 2024-09-02T09:48:58.641 INFO:teuthology.run_tasks:Running task pcp... 2024-09-02T09:48:58.678 INFO:teuthology.run_tasks:Running task selinux... 2024-09-02T09:48:58.709 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-02T09:48:58.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo service auditd rotate 2024-09-02T09:48:58.754 INFO:teuthology.orchestra.run.smithi151.stdout:Rotating logs: 2024-09-02T09:48:58.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo service auditd rotate 2024-09-02T09:48:58.817 INFO:teuthology.orchestra.run.smithi162.stdout:Rotating logs: 2024-09-02T09:48:58.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2024-09-02T09:48:58.869 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2024-09-02T09:48:58.871 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-02T09:48:58.871 DEBUG:teuthology.orchestra.run.smithi151:> /usr/sbin/getenforce 2024-09-02T09:48:58.894 INFO:teuthology.orchestra.run.smithi151.stdout:Permissive 2024-09-02T09:48:58.894 DEBUG:teuthology.orchestra.run.smithi162:> /usr/sbin/getenforce 2024-09-02T09:48:58.919 INFO:teuthology.orchestra.run.smithi162.stdout:Permissive 2024-09-02T09:48:58.919 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2024-09-02T09:48:58.938 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2024-09-02T09:48:58.938 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi151.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi162.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2024-09-02T09:48:58.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-02T09:48:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T09:48:58.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-02T09:48:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T09:48:58.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-02T09:48:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T09:48:59.017 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-02T09:48:59.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo /usr/sbin/setenforce permissive 2024-09-02T09:48:59.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo /usr/sbin/setenforce permissive 2024-09-02T09:48:59.091 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2024-09-02T09:48:59.127 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-02T09:48:59.136 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-02T09:48:59.147 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-02T09:48:59.148 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi151.front.sepia.ceph.com,smithi162.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-02T09:50:44.532 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi151.front.sepia.ceph.com'), Remote(name='ubuntu@smithi162.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2024-09-02T09:50:44.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-02T09:50:44.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi151.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T09:50:44.623 DEBUG:teuthology.orchestra.run.smithi151:> true 2024-09-02T09:50:44.712 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi151.front.sepia.ceph.com' 2024-09-02T09:50:44.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-02T09:50:44.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi162.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T09:50:44.796 DEBUG:teuthology.orchestra.run.smithi162:> true 2024-09-02T09:50:44.881 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi162.front.sepia.ceph.com' 2024-09-02T09:50:44.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-02T09:50:44.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-02T09:50:44.964 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-09-02T09:50:45.049 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-09-02T09:50:45.049 INFO:teuthology.run_tasks:Running task clock... 2024-09-02T09:50:45.059 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-02T09:50:45.059 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-02T09:50:45.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T09:50:45.062 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-02T09:50:45.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T09:50:45.065 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-02T09:50:45.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T09:50:45.090 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-02T09:50:45.093 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-02T09:50:45.105 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-02T09:50:45.107 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-02T09:50:45.119 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-02T09:50:45.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: ntpd: command not found 2024-09-02T09:50:45.131 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: ntpd: command not found 2024-09-02T09:50:45.134 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-02T09:50:45.140 INFO:teuthology.orchestra.run.smithi151.stdout:506 Cannot talk to daemon 2024-09-02T09:50:45.142 INFO:teuthology.orchestra.run.smithi162.stdout:506 Cannot talk to daemon 2024-09-02T09:50:45.155 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-02T09:50:45.156 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-02T09:50:45.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2024-09-02T09:50:45.169 INFO:teuthology.orchestra.run.smithi151.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-02T09:50:45.169 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2024-09-02T09:50:45.170 INFO:teuthology.orchestra.run.smithi162.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-02T09:50:45.183 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-02T09:50:45.197 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-02T09:50:45.225 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-09-02T09:50:45.229 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T09:50:45.229 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-09-02T09:50:45.229 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.229 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.229 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.229 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.234 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-09-02T09:50:45.238 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T09:50:45.238 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-09-02T09:50:45.238 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.238 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.238 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.238 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.261 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-09-02T09:50:45.265 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T09:50:45.265 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-09-02T09:50:45.265 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.265 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.265 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.265 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T09:50:45.266 INFO:teuthology.run_tasks:Running task install... 2024-09-02T09:50:45.274 DEBUG:teuthology.task.install:project ceph 2024-09-02T09:50:45.274 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07'}} 2024-09-02T09:50:45.274 DEBUG:teuthology.task.install:config {'branch': 'squid', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07'} 2024-09-02T09:50:45.274 INFO:teuthology.task.install:Using flavor: default 2024-09-02T09:50:45.280 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-02T09:50:45.280 INFO:teuthology.task.install:extra packages: [] 2024-09-02T09:50:45.280 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07', 'tag': None, 'wait_for_package': False} 2024-09-02T09:50:45.281 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-02T09:50:45.281 INFO:teuthology.packaging:ref: None 2024-09-02T09:50:45.281 INFO:teuthology.packaging:tag: None 2024-09-02T09:50:45.281 INFO:teuthology.packaging:branch: squid 2024-09-02T09:50:45.281 INFO:teuthology.packaging:sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:50:45.281 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-02T09:50:45.283 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07', 'tag': None, 'wait_for_package': False} 2024-09-02T09:50:45.283 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-02T09:50:45.283 INFO:teuthology.packaging:ref: None 2024-09-02T09:50:45.284 INFO:teuthology.packaging:tag: None 2024-09-02T09:50:45.284 INFO:teuthology.packaging:branch: squid 2024-09-02T09:50:45.284 INFO:teuthology.packaging:sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:50:45.284 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-02T09:50:45.285 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'squid', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07', 'tag': None, 'wait_for_package': False} 2024-09-02T09:50:45.285 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-02T09:50:45.285 INFO:teuthology.packaging:ref: None 2024-09-02T09:50:45.285 INFO:teuthology.packaging:tag: None 2024-09-02T09:50:45.285 INFO:teuthology.packaging:branch: squid 2024-09-02T09:50:45.285 INFO:teuthology.packaging:sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:50:45.285 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-02T09:50:45.513 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/ 2024-09-02T09:50:45.513 INFO:teuthology.task.install.rpm:Package version is 19.1.1-246.g7ad9cc7a 2024-09-02T09:50:45.530 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/ 2024-09-02T09:50:45.531 INFO:teuthology.task.install.rpm:Package version is 19.1.1-246.g7ad9cc7a 2024-09-02T09:50:45.551 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/ 2024-09-02T09:50:45.551 INFO:teuthology.task.install.rpm:Package version is 19.1.1-246.g7ad9cc7a 2024-09-02T09:50:45.650 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-02T09:50:45.651 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:50:45.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-02T09:50:45.656 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-02T09:50:45.656 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:50:45.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-02T09:50:45.665 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/7ad9cc7a6c6aa22edc5c410f5faa754e5f5a3ca1/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-02T09:50:45.665 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:50:45.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-02T09:50:45.677 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-02T09:50:45.677 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-02T09:50:45.677 INFO:teuthology.packaging:ref: None 2024-09-02T09:50:45.677 INFO:teuthology.packaging:tag: None 2024-09-02T09:50:45.677 INFO:teuthology.packaging:branch: squid 2024-09-02T09:50:45.677 INFO:teuthology.packaging:sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:50:45.678 DEBUG:teuthology.orchestra.run.smithi151:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-02T09:50:45.682 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-02T09:50:45.682 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-02T09:50:45.682 INFO:teuthology.packaging:ref: None 2024-09-02T09:50:45.682 INFO:teuthology.packaging:tag: None 2024-09-02T09:50:45.682 INFO:teuthology.packaging:branch: squid 2024-09-02T09:50:45.683 INFO:teuthology.packaging:sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:50:45.683 DEBUG:teuthology.orchestra.run.smithi162:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-02T09:50:45.691 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-02T09:50:45.691 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-02T09:50:45.691 INFO:teuthology.packaging:ref: None 2024-09-02T09:50:45.692 INFO:teuthology.packaging:tag: None 2024-09-02T09:50:45.692 INFO:teuthology.packaging:branch: squid 2024-09-02T09:50:45.692 INFO:teuthology.packaging:sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:50:45.692 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/squid/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-02T09:50:45.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-02T09:50:45.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-02T09:50:45.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-02T09:50:45.819 DEBUG:teuthology.orchestra.run.smithi151:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-02T09:50:45.823 DEBUG:teuthology.orchestra.run.smithi162:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-02T09:50:45.832 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-02T09:50:45.884 INFO:teuthology.orchestra.run.smithi151.stdout:check_obsoletes = 1 2024-09-02T09:50:45.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum clean all 2024-09-02T09:50:45.888 INFO:teuthology.orchestra.run.smithi162.stdout:check_obsoletes = 1 2024-09-02T09:50:45.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum clean all 2024-09-02T09:50:45.897 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2024-09-02T09:50:45.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2024-09-02T09:50:46.108 INFO:teuthology.orchestra.run.smithi151.stdout:45 files removed 2024-09-02T09:50:46.113 INFO:teuthology.orchestra.run.smithi162.stdout:45 files removed 2024-09-02T09:50:46.124 INFO:teuthology.orchestra.run.smithi187.stdout:45 files removed 2024-09-02T09:50:46.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-radosgw 2024-09-02T09:50:46.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-radosgw 2024-09-02T09:50:46.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw 2024-09-02T09:50:46.681 INFO:teuthology.orchestra.run.smithi151.stdout:ceph packages for x86_64 308 kB/s | 83 kB 00:00 2024-09-02T09:50:46.681 INFO:teuthology.orchestra.run.smithi162.stdout:ceph packages for x86_64 311 kB/s | 83 kB 00:00 2024-09-02T09:50:46.702 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 303 kB/s | 83 kB 00:00 2024-09-02T09:50:46.965 INFO:teuthology.orchestra.run.smithi151.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-02T09:50:46.966 INFO:teuthology.orchestra.run.smithi162.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-02T09:50:46.986 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-02T09:50:47.224 INFO:teuthology.orchestra.run.smithi151.stdout:ceph source packages 8.5 kB/s | 1.9 kB 00:00 2024-09-02T09:50:47.241 INFO:teuthology.orchestra.run.smithi162.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-02T09:50:47.262 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-02T09:50:47.882 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-02T09:50:47.899 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-02T09:50:48.895 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.2 MB 00:01 2024-09-02T09:50:49.975 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - AppStream 20 MB/s | 20 MB 00:01 2024-09-02T09:50:51.392 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - AppStream 8.5 MB/s | 20 MB 00:02 2024-09-02T09:50:55.518 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.6 MB 00:01 2024-09-02T09:50:55.884 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.6 MB 00:00 2024-09-02T09:50:57.202 INFO:teuthology.orchestra.run.smithi151.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 18 kB 00:00 2024-09-02T09:50:57.501 INFO:teuthology.orchestra.run.smithi162.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 18 kB 00:00 2024-09-02T09:50:58.114 INFO:teuthology.orchestra.run.smithi151.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-02T09:50:58.347 INFO:teuthology.orchestra.run.smithi162.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-02T09:51:03.636 INFO:teuthology.orchestra.run.smithi151.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-09-02T09:51:03.686 INFO:teuthology.orchestra.run.smithi162.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-02T09:51:05.965 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:51:05.968 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:05.968 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-02T09:51:05.968 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:05.968 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:51:05.969 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 11 M 2024-09-02T09:51:05.969 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrading: 2024-09-02T09:51:05.969 INFO:teuthology.orchestra.run.smithi151.stdout: librados2 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 3.4 M 2024-09-02T09:51:05.969 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 3.2 M 2024-09-02T09:51:05.970 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-02T09:51:05.970 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-02T09:51:05.970 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 5.4 M 2024-09-02T09:51:05.970 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 21 M 2024-09-02T09:51:05.970 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 25 k 2024-09-02T09:51:05.971 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-02T09:51:05.971 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-02T09:51:05.971 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-02T09:51:05.971 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 751 k 2024-09-02T09:51:05.971 INFO:teuthology.orchestra.run.smithi151.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-02T09:51:05.971 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-02T09:51:05.972 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-02T09:51:05.972 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 506 k 2024-09-02T09:51:05.972 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-02T09:51:05.972 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 5.3 M 2024-09-02T09:51:05.973 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-02T09:51:05.973 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-02T09:51:05.973 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-02T09:51:05.973 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 45 k 2024-09-02T09:51:05.974 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 136 k 2024-09-02T09:51:05.974 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 161 k 2024-09-02T09:51:05.974 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-02T09:51:05.974 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 320 k 2024-09-02T09:51:05.974 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 300 k 2024-09-02T09:51:05.975 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 99 k 2024-09-02T09:51:05.975 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-02T09:51:05.975 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-02T09:51:05.975 INFO:teuthology.orchestra.run.smithi151.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-02T09:51:05.975 INFO:teuthology.orchestra.run.smithi151.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-02T09:51:05.975 INFO:teuthology.orchestra.run.smithi151.stdout:Installing weak dependencies: 2024-09-02T09:51:05.976 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-02T09:51:05.976 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:05.976 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:51:05.976 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:05.976 INFO:teuthology.orchestra.run.smithi151.stdout:Install 30 Packages 2024-09-02T09:51:05.976 INFO:teuthology.orchestra.run.smithi151.stdout:Upgrade 2 Packages 2024-09-02T09:51:05.977 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:05.977 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 60 M 2024-09-02T09:51:05.978 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:51:06.029 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 11 M 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrading: 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout: librados2 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 3.4 M 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 3.2 M 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 5.4 M 2024-09-02T09:51:06.032 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 21 M 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 25 k 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 751 k 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 506 k 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 5.3 M 2024-09-02T09:51:06.033 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 45 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 136 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 161 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 320 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 300 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 99 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-02T09:51:06.034 INFO:teuthology.orchestra.run.smithi162.stdout:Installing weak dependencies: 2024-09-02T09:51:06.035 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-02T09:51:06.035 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:06.035 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:51:06.035 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:06.035 INFO:teuthology.orchestra.run.smithi162.stdout:Install 30 Packages 2024-09-02T09:51:06.035 INFO:teuthology.orchestra.run.smithi162.stdout:Upgrade 2 Packages 2024-09-02T09:51:06.035 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:06.036 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 60 M 2024-09-02T09:51:06.036 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:51:06.286 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 20 MB 00:16 2024-09-02T09:51:06.663 INFO:teuthology.orchestra.run.smithi151.stdout:(1/32): ceph-base-19.1.1-246.g7ad9cc7a.el9.x86_ 10 MB/s | 5.4 MB 00:00 2024-09-02T09:51:06.746 INFO:teuthology.orchestra.run.smithi151.stdout:(2/32): ceph-selinux-19.1.1-246.g7ad9cc7a.el9.x 303 kB/s | 25 kB 00:00 2024-09-02T09:51:06.855 INFO:teuthology.orchestra.run.smithi151.stdout:(3/32): libcephfs2-19.1.1-246.g7ad9cc7a.el9.x86 6.8 MB/s | 751 kB 00:00 2024-09-02T09:51:06.955 INFO:teuthology.orchestra.run.smithi151.stdout:(4/32): libradosstriper1-19.1.1-246.g7ad9cc7a.e 4.9 MB/s | 506 kB 00:00 2024-09-02T09:51:07.264 INFO:teuthology.orchestra.run.smithi151.stdout:(5/32): ceph-radosgw-19.1.1-246.g7ad9cc7a.el9.x 9.4 MB/s | 11 MB 00:01 2024-09-02T09:51:07.339 INFO:teuthology.orchestra.run.smithi151.stdout:(6/32): python3-ceph-argparse-19.1.1-246.g7ad9c 604 kB/s | 45 kB 00:00 2024-09-02T09:51:07.353 INFO:teuthology.orchestra.run.smithi162.stdout:(1/32): ceph-base-19.1.1-246.g7ad9cc7a.el9.x86_ 4.5 MB/s | 5.4 MB 00:01 2024-09-02T09:51:07.412 INFO:teuthology.orchestra.run.smithi162.stdout:(2/32): ceph-selinux-19.1.1-246.g7ad9cc7a.el9.x 430 kB/s | 25 kB 00:00 2024-09-02T09:51:07.416 INFO:teuthology.orchestra.run.smithi151.stdout:(7/32): librgw2-19.1.1-246.g7ad9cc7a.el9.x86_64 12 MB/s | 5.3 MB 00:00 2024-09-02T09:51:07.440 INFO:teuthology.orchestra.run.smithi151.stdout:(8/32): python3-ceph-common-19.1.1-246.g7ad9cc7 1.3 MB/s | 136 kB 00:00 2024-09-02T09:51:07.498 INFO:teuthology.orchestra.run.smithi151.stdout:(9/32): python3-cephfs-19.1.1-246.g7ad9cc7a.el9 1.9 MB/s | 161 kB 00:00 2024-09-02T09:51:07.532 INFO:teuthology.orchestra.run.smithi151.stdout:(10/32): python3-rados-19.1.1-246.g7ad9cc7a.el9 3.4 MB/s | 320 kB 00:00 2024-09-02T09:51:07.590 INFO:teuthology.orchestra.run.smithi151.stdout:(11/32): python3-rbd-19.1.1-246.g7ad9cc7a.el9.x 3.2 MB/s | 300 kB 00:00 2024-09-02T09:51:07.615 INFO:teuthology.orchestra.run.smithi151.stdout:(12/32): python3-rgw-19.1.1-246.g7ad9cc7a.el9.x 1.2 MB/s | 99 kB 00:00 2024-09-02T09:51:07.637 INFO:teuthology.orchestra.run.smithi162.stdout:(3/32): ceph-radosgw-19.1.1-246.g7ad9cc7a.el9.x 7.2 MB/s | 11 MB 00:01 2024-09-02T09:51:07.671 INFO:teuthology.orchestra.run.smithi162.stdout:(4/32): libcephfs2-19.1.1-246.g7ad9cc7a.el9.x86 2.8 MB/s | 751 kB 00:00 2024-09-02T09:51:07.696 INFO:teuthology.orchestra.run.smithi162.stdout:(5/32): libradosstriper1-19.1.1-246.g7ad9cc7a.e 8.5 MB/s | 506 kB 00:00 2024-09-02T09:51:07.724 INFO:teuthology.orchestra.run.smithi151.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 800 kB/s | 107 kB 00:00 2024-09-02T09:51:07.738 INFO:teuthology.orchestra.run.smithi162.stdout:(6/32): python3-ceph-argparse-19.1.1-246.g7ad9c 1.1 MB/s | 45 kB 00:00 2024-09-02T09:51:07.749 INFO:teuthology.orchestra.run.smithi151.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-09-02T09:51:07.775 INFO:teuthology.orchestra.run.smithi151.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 900 kB/s | 45 kB 00:00 2024-09-02T09:51:07.967 INFO:teuthology.orchestra.run.smithi151.stdout:(16/32): ceph-common-19.1.1-246.g7ad9cc7a.el9.x 12 MB/s | 21 MB 00:01 2024-09-02T09:51:07.980 INFO:teuthology.orchestra.run.smithi162.stdout:(7/32): ceph-common-19.1.1-246.g7ad9cc7a.el9.x8 12 MB/s | 21 MB 00:01 2024-09-02T09:51:07.992 INFO:teuthology.orchestra.run.smithi151.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.3 MB/s | 292 kB 00:00 2024-09-02T09:51:08.005 INFO:teuthology.orchestra.run.smithi162.stdout:(8/32): python3-ceph-common-19.1.1-246.g7ad9cc7 509 kB/s | 136 kB 00:00 2024-09-02T09:51:08.017 INFO:teuthology.orchestra.run.smithi151.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-09-02T09:51:08.031 INFO:teuthology.orchestra.run.smithi162.stdout:(9/32): python3-cephfs-19.1.1-246.g7ad9cc7a.el9 3.1 MB/s | 161 kB 00:00 2024-09-02T09:51:08.056 INFO:teuthology.orchestra.run.smithi162.stdout:(10/32): python3-rados-19.1.1-246.g7ad9cc7a.el9 6.2 MB/s | 320 kB 00:00 2024-09-02T09:51:08.059 INFO:teuthology.orchestra.run.smithi151.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-09-02T09:51:08.084 INFO:teuthology.orchestra.run.smithi151.stdout:(20/32): python3-prettytable-0.7.2-27.el9.noarc 360 kB/s | 42 kB 00:00 2024-09-02T09:51:08.089 INFO:teuthology.orchestra.run.smithi162.stdout:(11/32): python3-rbd-19.1.1-246.g7ad9cc7a.el9.x 5.0 MB/s | 300 kB 00:00 2024-09-02T09:51:08.110 INFO:teuthology.orchestra.run.smithi151.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 389 kB/s | 36 kB 00:00 2024-09-02T09:51:08.115 INFO:teuthology.orchestra.run.smithi162.stdout:(12/32): python3-rgw-19.1.1-246.g7ad9cc7a.el9.x 1.7 MB/s | 99 kB 00:00 2024-09-02T09:51:08.135 INFO:teuthology.orchestra.run.smithi151.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-09-02T09:51:08.193 INFO:teuthology.orchestra.run.smithi151.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-09-02T09:51:08.219 INFO:teuthology.orchestra.run.smithi151.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-02T09:51:08.232 INFO:teuthology.orchestra.run.smithi162.stdout:(13/32): librgw2-19.1.1-246.g7ad9cc7a.el9.x86_6 9.5 MB/s | 5.3 MB 00:00 2024-09-02T09:51:08.261 INFO:teuthology.orchestra.run.smithi151.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 232 kB/s | 26 kB 00:00 2024-09-02T09:51:08.286 INFO:teuthology.orchestra.run.smithi151.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-09-02T09:51:08.299 INFO:teuthology.orchestra.run.smithi162.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 510 kB/s | 107 kB 00:00 2024-09-02T09:51:08.320 INFO:teuthology.orchestra.run.smithi151.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-02T09:51:08.324 INFO:teuthology.orchestra.run.smithi162.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 767 kB/s | 160 kB 00:00 2024-09-02T09:51:08.349 INFO:teuthology.orchestra.run.smithi162.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-09-02T09:51:08.378 INFO:teuthology.orchestra.run.smithi151.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-02T09:51:08.416 INFO:teuthology.orchestra.run.smithi162.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 634 kB/s | 42 kB 00:00 2024-09-02T09:51:08.428 INFO:teuthology.orchestra.run.smithi151.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-02T09:51:08.441 INFO:teuthology.orchestra.run.smithi162.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 662 kB 00:00 2024-09-02T09:51:08.484 INFO:teuthology.orchestra.run.smithi162.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 861 kB/s | 36 kB 00:00 2024-09-02T09:51:08.495 INFO:teuthology.orchestra.run.smithi151.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-02T09:51:08.509 INFO:teuthology.orchestra.run.smithi162.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.3 MB/s | 221 kB 00:00 2024-09-02T09:51:08.534 INFO:teuthology.orchestra.run.smithi162.stdout:(21/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-09-02T09:51:08.559 INFO:teuthology.orchestra.run.smithi162.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 628 kB/s | 47 kB 00:00 2024-09-02T09:51:08.618 INFO:teuthology.orchestra.run.smithi162.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-09-02T09:51:08.629 INFO:teuthology.orchestra.run.smithi151.stdout:(31/32): librados2-19.1.1-246.g7ad9cc7a.el9.x86 14 MB/s | 3.4 MB 00:00 2024-09-02T09:51:08.643 INFO:teuthology.orchestra.run.smithi162.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-02T09:51:08.668 INFO:teuthology.orchestra.run.smithi162.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 276 kB/s | 26 kB 00:00 2024-09-02T09:51:08.694 INFO:teuthology.orchestra.run.smithi162.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-02T09:51:08.696 INFO:teuthology.orchestra.run.smithi151.stdout:(32/32): librbd1-19.1.1-246.g7ad9cc7a.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-02T09:51:08.701 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:08.701 INFO:teuthology.orchestra.run.smithi151.stdout:Total 22 MB/s | 60 MB 00:02 2024-09-02T09:51:08.727 INFO:teuthology.orchestra.run.smithi162.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-02T09:51:08.778 INFO:teuthology.orchestra.run.smithi162.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-02T09:51:08.803 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:51:08.828 INFO:teuthology.orchestra.run.smithi162.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-02T09:51:08.839 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:51:08.840 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:51:08.912 INFO:teuthology.orchestra.run.smithi162.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-02T09:51:09.087 INFO:teuthology.orchestra.run.smithi162.stdout:(31/32): librbd1-19.1.1-246.g7ad9cc7a.el9.x86_6 12 MB/s | 3.2 MB 00:00 2024-09-02T09:51:09.137 INFO:teuthology.orchestra.run.smithi162.stdout:(32/32): librados2-19.1.1-246.g7ad9cc7a.el9.x86 9.5 MB/s | 3.4 MB 00:00 2024-09-02T09:51:09.142 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:09.143 INFO:teuthology.orchestra.run.smithi162.stdout:Total 19 MB/s | 60 MB 00:03 2024-09-02T09:51:09.248 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:51:09.285 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:51:09.285 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:51:09.312 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:51:09.313 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:51:09.760 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:51:09.760 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:51:10.586 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.6 MB 00:00 2024-09-02T09:51:10.782 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:51:10.871 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-02T09:51:10.901 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-02T09:51:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-02T09:51:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T09:51:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T09:51:11.276 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T09:51:11.324 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:51:11.363 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T09:51:11.404 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-02T09:51:11.442 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-02T09:51:11.467 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-02T09:51:11.490 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T09:51:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T09:51:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-02T09:51:11.820 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T09:51:11.844 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 7/34 2024-09-02T09:51:11.911 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T09:51:11.936 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 7/34 2024-09-02T09:51:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 8/34 2024-09-02T09:51:12.054 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 18 kB 00:00 2024-09-02T09:51:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 9/34 2024-09-02T09:51:12.125 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 9/34 2024-09-02T09:51:12.154 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-02T09:51:12.197 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-argparse-2:19.1.1-246.g7ad9cc7a.el9.x 11/34 2024-09-02T09:51:12.242 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-02T09:51:12.267 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 12/34 2024-09-02T09:51:12.289 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 13/34 2024-09-02T09:51:12.344 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 7/34 2024-09-02T09:51:12.354 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 13/34 2024-09-02T09:51:12.388 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-02T09:51:12.436 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 7/34 2024-09-02T09:51:12.496 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-02T09:51:12.530 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-02T09:51:12.535 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 8/34 2024-09-02T09:51:12.570 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-02T09:51:12.571 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 9/34 2024-09-02T09:51:12.604 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-02T09:51:12.617 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 9/34 2024-09-02T09:51:12.633 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-02T09:51:12.662 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-02T09:51:12.665 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-02T09:51:12.706 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-argparse-2:19.1.1-246.g7ad9cc7a.el9.x 11/34 2024-09-02T09:51:12.777 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 12/34 2024-09-02T09:51:12.806 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 13/34 2024-09-02T09:51:12.854 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 13/34 2024-09-02T09:51:12.888 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-02T09:51:12.975 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-02T09:51:12.999 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-02T09:51:13.021 INFO:teuthology.orchestra.run.smithi151.stdout: Upgrading : librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 22/34 2024-09-02T09:51:13.030 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-02T09:51:13.071 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-02T09:51:13.096 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-02T09:51:13.109 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 22/34 2024-09-02T09:51:13.125 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-02T09:51:13.133 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-02T09:51:13.141 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 23/34 2024-09-02T09:51:13.158 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-02T09:51:13.176 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-02T09:51:13.475 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-02T09:51:13.521 INFO:teuthology.orchestra.run.smithi162.stdout: Upgrading : librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 22/34 2024-09-02T09:51:13.613 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 22/34 2024-09-02T09:51:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 23/34 2024-09-02T09:51:13.656 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-02T09:51:13.679 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-02T09:51:13.686 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 26/34 2024-09-02T09:51:13.752 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 26/34 2024-09-02T09:51:13.826 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 27/34 2024-09-02T09:51:13.908 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86 28/34 2024-09-02T09:51:14.164 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-02T09:51:14.203 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 26/34 2024-09-02T09:51:14.273 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 26/34 2024-09-02T09:51:14.342 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 27/34 2024-09-02T09:51:14.371 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86 28/34 2024-09-02T09:51:16.379 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 29/34 2024-09-02T09:51:16.446 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 29/34 2024-09-02T09:51:16.608 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 29/34 2024-09-02T09:51:16.651 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 29/34 2024-09-02T09:51:16.994 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 29/34 2024-09-02T09:51:17.033 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 30/34 2024-09-02T09:51:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 30/34 2024-09-02T09:51:17.125 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-02T09:51:17.125 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-02T09:51:17.125 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:17.160 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 31/34 2024-09-02T09:51:17.196 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 29/34 2024-09-02T09:51:17.229 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 30/34 2024-09-02T09:51:17.304 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 30/34 2024-09-02T09:51:17.304 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-02T09:51:17.304 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-02T09:51:17.304 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:17.339 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 31/34 2024-09-02T09:51:18.572 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-02T09:51:20.873 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:51:20.875 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:51:20.875 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-02T09:51:20.875 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:51:20.875 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:51:20.875 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 11 M 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 3.4 M 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 3.2 M 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 5.4 M 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 21 M 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 25 k 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-02T09:51:20.876 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 751 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 506 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 5.3 M 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 45 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 136 k 2024-09-02T09:51:20.877 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 161 k 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 320 k 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 300 k 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 99 k 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout:Installing weak dependencies: 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:51:20.878 INFO:teuthology.orchestra.run.smithi187.stdout:Install 30 Packages 2024-09-02T09:51:20.879 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 2 Packages 2024-09-02T09:51:20.879 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:20.879 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 60 M 2024-09-02T09:51:20.879 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:51:21.831 INFO:teuthology.orchestra.run.smithi187.stdout:(1/32): ceph-base-19.1.1-246.g7ad9cc7a.el9.x86_ 11 MB/s | 5.4 MB 00:00 2024-09-02T09:51:21.890 INFO:teuthology.orchestra.run.smithi187.stdout:(2/32): ceph-selinux-19.1.1-246.g7ad9cc7a.el9.x 431 kB/s | 25 kB 00:00 2024-09-02T09:51:21.974 INFO:teuthology.orchestra.run.smithi187.stdout:(3/32): libcephfs2-19.1.1-246.g7ad9cc7a.el9.x86 8.8 MB/s | 751 kB 00:00 2024-09-02T09:51:22.057 INFO:teuthology.orchestra.run.smithi187.stdout:(4/32): libradosstriper1-19.1.1-246.g7ad9cc7a.e 5.9 MB/s | 506 kB 00:00 2024-09-02T09:51:22.174 INFO:teuthology.orchestra.run.smithi187.stdout:(5/32): ceph-radosgw-19.1.1-246.g7ad9cc7a.el9.x 12 MB/s | 11 MB 00:00 2024-09-02T09:51:22.224 INFO:teuthology.orchestra.run.smithi187.stdout:(6/32): python3-ceph-argparse-19.1.1-246.g7ad9c 909 kB/s | 45 kB 00:00 2024-09-02T09:51:22.341 INFO:teuthology.orchestra.run.smithi187.stdout:(7/32): librgw2-19.1.1-246.g7ad9cc7a.el9.x86_64 19 MB/s | 5.3 MB 00:00 2024-09-02T09:51:22.367 INFO:teuthology.orchestra.run.smithi187.stdout:(8/32): python3-ceph-common-19.1.1-246.g7ad9cc7 956 kB/s | 136 kB 00:00 2024-09-02T09:51:22.392 INFO:teuthology.orchestra.run.smithi187.stdout:(9/32): python3-cephfs-19.1.1-246.g7ad9cc7a.el9 3.1 MB/s | 161 kB 00:00 2024-09-02T09:51:22.417 INFO:teuthology.orchestra.run.smithi187.stdout:(10/32): python3-rados-19.1.1-246.g7ad9cc7a.el9 6.2 MB/s | 320 kB 00:00 2024-09-02T09:51:22.443 INFO:teuthology.orchestra.run.smithi187.stdout:(11/32): python3-rbd-19.1.1-246.g7ad9cc7a.el9.x 5.8 MB/s | 300 kB 00:00 2024-09-02T09:51:22.468 INFO:teuthology.orchestra.run.smithi187.stdout:(12/32): python3-rgw-19.1.1-246.g7ad9cc7a.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-02T09:51:22.693 INFO:teuthology.orchestra.run.smithi187.stdout:(13/32): ceph-common-19.1.1-246.g7ad9cc7a.el9.x 15 MB/s | 21 MB 00:01 2024-09-02T09:51:22.735 INFO:teuthology.orchestra.run.smithi187.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 366 kB/s | 107 kB 00:00 2024-09-02T09:51:22.761 INFO:teuthology.orchestra.run.smithi187.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 547 kB/s | 160 kB 00:00 2024-09-02T09:51:22.786 INFO:teuthology.orchestra.run.smithi187.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 491 kB/s | 45 kB 00:00 2024-09-02T09:51:22.828 INFO:teuthology.orchestra.run.smithi187.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-09-02T09:51:22.853 INFO:teuthology.orchestra.run.smithi187.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 631 kB/s | 42 kB 00:00 2024-09-02T09:51:22.878 INFO:teuthology.orchestra.run.smithi187.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-02T09:51:22.912 INFO:teuthology.orchestra.run.smithi187.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-02T09:51:22.938 INFO:teuthology.orchestra.run.smithi187.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 428 kB/s | 36 kB 00:00 2024-09-02T09:51:22.963 INFO:teuthology.orchestra.run.smithi187.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-09-02T09:51:23.021 INFO:teuthology.orchestra.run.smithi187.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.6 MB/s | 309 kB 00:00 2024-09-02T09:51:23.046 INFO:teuthology.orchestra.run.smithi187.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-09-02T09:51:23.072 INFO:teuthology.orchestra.run.smithi187.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-09-02T09:51:23.097 INFO:teuthology.orchestra.run.smithi187.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-02T09:51:23.131 INFO:teuthology.orchestra.run.smithi187.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-02T09:51:23.164 INFO:teuthology.orchestra.run.smithi187.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-02T09:51:23.232 INFO:teuthology.orchestra.run.smithi187.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-02T09:51:23.323 INFO:teuthology.orchestra.run.smithi187.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-02T09:51:23.415 INFO:teuthology.orchestra.run.smithi187.stdout:(31/32): librados2-19.1.1-246.g7ad9cc7a.el9.x86 14 MB/s | 3.4 MB 00:00 2024-09-02T09:51:23.474 INFO:teuthology.orchestra.run.smithi187.stdout:(32/32): librbd1-19.1.1-246.g7ad9cc7a.el9.x86_6 13 MB/s | 3.2 MB 00:00 2024-09-02T09:51:23.479 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:23.479 INFO:teuthology.orchestra.run.smithi187.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-02T09:51:23.584 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:51:23.621 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:51:23.621 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:51:24.093 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:51:24.094 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:51:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:51:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-02T09:51:25.687 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-02T09:51:25.712 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-02T09:51:25.735 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T09:51:25.759 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-02T09:51:26.055 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T09:51:26.106 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-02T09:51:26.440 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-02T09:51:26.539 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 7/34 2024-09-02T09:51:26.631 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 7/34 2024-09-02T09:51:26.723 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 8/34 2024-09-02T09:51:26.749 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 9/34 2024-09-02T09:51:26.812 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 9/34 2024-09-02T09:51:26.840 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-02T09:51:26.884 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:19.1.1-246.g7ad9cc7a.el9.x 11/34 2024-09-02T09:51:26.954 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 12/34 2024-09-02T09:51:26.976 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 13/34 2024-09-02T09:51:27.040 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 13/34 2024-09-02T09:51:27.075 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-02T09:51:27.184 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-02T09:51:27.217 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-02T09:51:27.243 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 31/34 2024-09-02T09:51:27.243 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /sys 2024-09-02T09:51:27.243 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /proc 2024-09-02T09:51:27.243 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /mnt 2024-09-02T09:51:27.243 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /var/tmp 2024-09-02T09:51:27.243 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /home 2024-09-02T09:51:27.243 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /root 2024-09-02T09:51:27.243 INFO:teuthology.orchestra.run.smithi151.stdout:skipping the directory /tmp 2024-09-02T09:51:27.243 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:27.257 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-02T09:51:27.282 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-02T09:51:27.312 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-02T09:51:27.344 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-02T09:51:27.449 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 31/34 2024-09-02T09:51:27.449 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /sys 2024-09-02T09:51:27.449 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /proc 2024-09-02T09:51:27.449 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /mnt 2024-09-02T09:51:27.449 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /var/tmp 2024-09-02T09:51:27.450 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /home 2024-09-02T09:51:27.450 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /root 2024-09-02T09:51:27.450 INFO:teuthology.orchestra.run.smithi162.stdout:skipping the directory /tmp 2024-09-02T09:51:27.450 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:27.661 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-02T09:51:27.711 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 22/34 2024-09-02T09:51:27.801 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 22/34 2024-09-02T09:51:27.835 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 23/34 2024-09-02T09:51:27.871 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-02T09:51:28.218 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 32/34 2024-09-02T09:51:28.243 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 32/34 2024-09-02T09:51:28.243 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:28.243 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-02T09:51:28.244 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T09:51:28.244 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T09:51:28.244 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:28.245 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T09:51:28.329 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T09:51:28.330 INFO:teuthology.orchestra.run.smithi151.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T09:51:28.353 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-02T09:51:28.383 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 26/34 2024-09-02T09:51:28.454 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 26/34 2024-09-02T09:51:28.456 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 32/34 2024-09-02T09:51:28.482 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 32/34 2024-09-02T09:51:28.482 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:28.482 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-02T09:51:28.482 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T09:51:28.482 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T09:51:28.484 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:28.485 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T09:51:28.523 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 27/34 2024-09-02T09:51:28.552 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86 28/34 2024-09-02T09:51:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T09:51:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T09:51:29.683 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T09:51:29.683 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/34 2024-09-02T09:51:29.683 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/34 2024-09-02T09:51:29.683 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/34 2024-09-02T09:51:29.683 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/34 2024-09-02T09:51:29.683 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 5/34 2024-09-02T09:51:29.683 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 6/34 2024-09-02T09:51:29.683 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 7/34 2024-09-02T09:51:29.683 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-argparse-2:19.1.1-246.g7ad9cc7a.el9.x 8/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86 9/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 10/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 11/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 12/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 13/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-02T09:51:29.684 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-02T09:51:29.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-02T09:51:29.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-02T09:51:29.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-02T09:51:29.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-02T09:51:29.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-02T09:51:29.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-02T09:51:29.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-02T09:51:29.685 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-02T09:51:29.686 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-02T09:51:29.686 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-02T09:51:29.686 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 31/34 2024-09-02T09:51:29.686 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-02T09:51:29.686 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 33/34 2024-09-02T09:51:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T09:51:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/34 2024-09-02T09:51:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/34 2024-09-02T09:51:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/34 2024-09-02T09:51:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/34 2024-09-02T09:51:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 5/34 2024-09-02T09:51:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 6/34 2024-09-02T09:51:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 7/34 2024-09-02T09:51:29.896 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-argparse-2:19.1.1-246.g7ad9cc7a.el9.x 8/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86 9/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 10/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 11/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 12/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 13/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-02T09:51:29.897 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 31/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-02T09:51:29.898 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 33/34 2024-09-02T09:51:30.506 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout:Upgraded: 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout: librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout: librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-02T09:51:30.507 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-argparse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.508 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: re2-1:20211101-20.el9.x86_64 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:30.509 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:51:30.686 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout:Upgraded: 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout: librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout: librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.687 INFO:teuthology.orchestra.run.smithi162.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-02T09:51:30.688 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-argparse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: re2-1:20211101-20.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-02T09:51:30.689 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:30.690 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:51:30.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-test 2024-09-02T09:51:30.859 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 29/34 2024-09-02T09:51:30.899 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 29/34 2024-09-02T09:51:30.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-test 2024-09-02T09:51:31.387 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:28 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:51:31.445 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 29/34 2024-09-02T09:51:31.494 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 30/34 2024-09-02T09:51:31.524 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:51:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-02T09:51:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:51:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 49 M 2024-09-02T09:51:31.525 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-02T09:51:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 163 k 2024-09-02T09:51:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-02T09:51:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-02T09:51:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:51:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:Install 4 Packages 2024-09-02T09:51:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 49 M 2024-09-02T09:51:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 209 M 2024-09-02T09:51:31.526 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:51:31.529 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:28 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:51:31.590 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 30/34 2024-09-02T09:51:31.591 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-02T09:51:31.591 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-02T09:51:31.591 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:31.620 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 31/34 2024-09-02T09:51:31.670 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:51:31.671 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-02T09:51:31.671 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:31.671 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:51:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 49 M 2024-09-02T09:51:31.671 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-02T09:51:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 163 k 2024-09-02T09:51:31.671 INFO:teuthology.orchestra.run.smithi162.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-02T09:51:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-02T09:51:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:31.672 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:51:31.672 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:31.672 INFO:teuthology.orchestra.run.smithi162.stdout:Install 4 Packages 2024-09-02T09:51:31.672 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:31.672 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 49 M 2024-09-02T09:51:31.673 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 209 M 2024-09-02T09:51:31.673 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:51:31.868 INFO:teuthology.orchestra.run.smithi151.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-02T09:51:31.876 INFO:teuthology.orchestra.run.smithi162.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-02T09:51:31.894 INFO:teuthology.orchestra.run.smithi151.stdout:(2/4): libcephsqlite-19.1.1-246.g7ad9cc7a.el9.x 614 kB/s | 163 kB 00:00 2024-09-02T09:51:31.918 INFO:teuthology.orchestra.run.smithi162.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-02T09:51:31.936 INFO:teuthology.orchestra.run.smithi151.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 945 kB/s | 64 kB 00:00 2024-09-02T09:51:31.977 INFO:teuthology.orchestra.run.smithi162.stdout:(3/4): libcephsqlite-19.1.1-246.g7ad9cc7a.el9.x 640 kB/s | 163 kB 00:00 2024-09-02T09:51:33.493 INFO:teuthology.orchestra.run.smithi151.stdout:(4/4): ceph-test-19.1.1-246.g7ad9cc7a.el9.x86_6 26 MB/s | 49 MB 00:01 2024-09-02T09:51:33.495 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:33.495 INFO:teuthology.orchestra.run.smithi151.stdout:Total 25 MB/s | 49 MB 00:01 2024-09-02T09:51:33.522 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:51:33.542 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:51:33.542 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:51:33.914 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:51:33.914 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:51:34.049 INFO:teuthology.orchestra.run.smithi162.stdout:(4/4): ceph-test-19.1.1-246.g7ad9cc7a.el9.x86_6 21 MB/s | 49 MB 00:02 2024-09-02T09:51:34.051 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:34.052 INFO:teuthology.orchestra.run.smithi162.stdout:Total 21 MB/s | 49 MB 00:02 2024-09-02T09:51:34.080 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:51:34.100 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:51:34.100 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:51:34.444 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:51:34.471 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:51:34.471 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:51:34.518 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-02T09:51:34.559 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-02T09:51:34.595 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/4 2024-09-02T09:51:34.992 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:51:35.066 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-02T09:51:35.115 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-02T09:51:35.151 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/4 2024-09-02T09:51:39.188 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/4 2024-09-02T09:51:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/4 2024-09-02T09:51:39.727 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/4 2024-09-02T09:51:39.815 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/4 2024-09-02T09:51:41.570 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/4 2024-09-02T09:51:41.570 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/4 2024-09-02T09:51:41.570 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/4 2024-09-02T09:51:41.570 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-02T09:51:41.660 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 31/34 2024-09-02T09:51:41.660 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2024-09-02T09:51:41.660 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2024-09-02T09:51:41.660 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2024-09-02T09:51:41.660 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2024-09-02T09:51:41.660 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2024-09-02T09:51:41.660 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /root 2024-09-02T09:51:41.660 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2024-09-02T09:51:41.661 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:41.855 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-02T09:51:41.855 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:41.856 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:51:41.856 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:41.856 INFO:teuthology.orchestra.run.smithi151.stdout: libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:41.856 INFO:teuthology.orchestra.run.smithi151.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-02T09:51:41.856 INFO:teuthology.orchestra.run.smithi151.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-02T09:51:41.856 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:41.856 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:51:42.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph 2024-09-02T09:51:42.255 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/4 2024-09-02T09:51:42.255 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/4 2024-09-02T09:51:42.255 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/4 2024-09-02T09:51:42.255 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-02T09:51:42.563 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-02T09:51:42.563 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:42.563 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:51:42.563 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:42.563 INFO:teuthology.orchestra.run.smithi162.stdout: libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:42.563 INFO:teuthology.orchestra.run.smithi162.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-02T09:51:42.564 INFO:teuthology.orchestra.run.smithi162.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-02T09:51:42.564 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:42.564 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:51:42.611 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 32/34 2024-09-02T09:51:42.637 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 32/34 2024-09-02T09:51:42.637 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:42.637 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-02T09:51:42.637 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T09:51:42.638 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-02T09:51:42.638 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:42.639 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T09:51:42.712 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:39 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:51:42.721 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-02T09:51:42.721 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T09:51:42.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph 2024-09-02T09:51:42.857 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:51:42.860 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-02T09:51:42.860 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:42.860 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:51:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: ceph x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 6.6 k 2024-09-02T09:51:42.860 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-02T09:51:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 2.4 M 2024-09-02T09:51:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 1.5 M 2024-09-02T09:51:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core noarch 2:19.1.1-246.g7ad9cc7a.el9 ceph-noarch 249 k 2024-09-02T09:51:42.860 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 4.7 M 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 17 M 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-02T09:51:42.861 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-02T09:51:42.862 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout:Install 37 Packages 2024-09-02T09:51:42.863 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:42.864 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 30 M 2024-09-02T09:51:42.864 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 108 M 2024-09-02T09:51:42.864 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:51:43.363 INFO:teuthology.orchestra.run.smithi151.stdout:(1/37): ceph-19.1.1-246.g7ad9cc7a.el9.x86_64.rp 50 kB/s | 6.6 kB 00:00 2024-09-02T09:51:43.432 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:40 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:51:43.521 INFO:teuthology.orchestra.run.smithi151.stdout:(2/37): ceph-mgr-19.1.1-246.g7ad9cc7a.el9.x86_6 5.2 MB/s | 1.5 MB 00:00 2024-09-02T09:51:43.582 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:51:43.585 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:43.585 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-02T09:51:43.585 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:43.585 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:51:43.585 INFO:teuthology.orchestra.run.smithi162.stdout: ceph x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 6.6 k 2024-09-02T09:51:43.585 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-02T09:51:43.585 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 2.4 M 2024-09-02T09:51:43.585 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 1.5 M 2024-09-02T09:51:43.585 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core noarch 2:19.1.1-246.g7ad9cc7a.el9 ceph-noarch 249 k 2024-09-02T09:51:43.585 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 4.7 M 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 17 M 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-02T09:51:43.586 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-02T09:51:43.587 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-02T09:51:43.588 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-02T09:51:43.588 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-02T09:51:43.588 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-02T09:51:43.588 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-02T09:51:43.588 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-02T09:51:43.588 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:43.588 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:51:43.588 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:43.588 INFO:teuthology.orchestra.run.smithi162.stdout:Install 37 Packages 2024-09-02T09:51:43.588 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:43.590 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 30 M 2024-09-02T09:51:43.590 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 108 M 2024-09-02T09:51:43.590 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:51:43.614 INFO:teuthology.orchestra.run.smithi151.stdout:(3/37): ceph-mds-19.1.1-246.g7ad9cc7a.el9.x86_6 6.2 MB/s | 2.4 MB 00:00 2024-09-02T09:51:43.689 INFO:teuthology.orchestra.run.smithi151.stdout:(4/37): ceph-mgr-modules-core-19.1.1-246.g7ad9c 3.2 MB/s | 249 kB 00:00 2024-09-02T09:51:43.773 INFO:teuthology.orchestra.run.smithi151.stdout:(5/37): ceph-mon-19.1.1-246.g7ad9cc7a.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-02T09:51:43.948 INFO:teuthology.orchestra.run.smithi151.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 979 kB/s | 253 kB 00:00 2024-09-02T09:51:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T09:51:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/34 2024-09-02T09:51:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/34 2024-09-02T09:51:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/34 2024-09-02T09:51:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/34 2024-09-02T09:51:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 5/34 2024-09-02T09:51:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 6/34 2024-09-02T09:51:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 7/34 2024-09-02T09:51:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:19.1.1-246.g7ad9cc7a.el9.x 8/34 2024-09-02T09:51:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86 9/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 10/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 11/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 12/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 13/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-02T09:51:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-02T09:51:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-02T09:51:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-02T09:51:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-02T09:51:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-02T09:51:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-02T09:51:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-02T09:51:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-02T09:51:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-02T09:51:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-02T09:51:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 31/34 2024-09-02T09:51:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-02T09:51:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 33/34 2024-09-02T09:51:44.140 INFO:teuthology.orchestra.run.smithi151.stdout:(7/37): ceph-osd-19.1.1-246.g7ad9cc7a.el9.x86_6 27 MB/s | 17 MB 00:00 2024-09-02T09:51:44.165 INFO:teuthology.orchestra.run.smithi151.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 491 kB/s | 106 kB 00:00 2024-09-02T09:51:44.224 INFO:teuthology.orchestra.run.smithi151.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-02T09:51:44.258 INFO:teuthology.orchestra.run.smithi151.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.5 MB/s | 1.2 MB 00:00 2024-09-02T09:51:44.264 INFO:teuthology.orchestra.run.smithi162.stdout:(1/37): ceph-19.1.1-246.g7ad9cc7a.el9.x86_64.rp 53 kB/s | 6.6 kB 00:00 2024-09-02T09:51:44.283 INFO:teuthology.orchestra.run.smithi151.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-02T09:51:44.317 INFO:teuthology.orchestra.run.smithi151.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 766 kB/s | 135 kB 00:00 2024-09-02T09:51:44.456 INFO:teuthology.orchestra.run.smithi162.stdout:(2/37): ceph-mgr-19.1.1-246.g7ad9cc7a.el9.x86_6 4.8 MB/s | 1.5 MB 00:00 2024-09-02T09:51:44.523 INFO:teuthology.orchestra.run.smithi162.stdout:(3/37): ceph-mds-19.1.1-246.g7ad9cc7a.el9.x86_6 6.2 MB/s | 2.4 MB 00:00 2024-09-02T09:51:44.593 INFO:teuthology.orchestra.run.smithi151.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 126 kB/s | 35 kB 00:00 2024-09-02T09:51:44.616 INFO:teuthology.orchestra.run.smithi162.stdout:(4/37): ceph-mon-19.1.1-246.g7ad9cc7a.el9.x86_6 13 MB/s | 4.7 MB 00:00 2024-09-02T09:51:44.641 INFO:teuthology.orchestra.run.smithi162.stdout:(5/37): ceph-mgr-modules-core-19.1.1-246.g7ad9c 2.1 MB/s | 249 kB 00:00 2024-09-02T09:51:44.684 INFO:teuthology.orchestra.run.smithi151.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 442 kB/s | 188 kB 00:00 2024-09-02T09:51:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-02T09:51:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:44.763 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2024-09-02T09:51:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.763 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:51:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-02T09:51:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-02T09:51:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-02T09:51:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-02T09:51:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-02T09:51:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-02T09:51:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-02T09:51:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-02T09:51:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-02T09:51:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-02T09:51:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-02T09:51:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-02T09:51:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-02T09:51:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-02T09:51:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-02T09:51:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: re2-1:20211101-20.el9.x86_64 2024-09-02T09:51:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-02T09:51:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:44.768 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:51:44.776 INFO:teuthology.orchestra.run.smithi151.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 420 kB/s | 77 kB 00:00 2024-09-02T09:51:44.802 INFO:teuthology.orchestra.run.smithi151.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 332 kB/s | 172 kB 00:00 2024-09-02T09:51:44.827 INFO:teuthology.orchestra.run.smithi151.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 358 kB/s | 51 kB 00:00 2024-09-02T09:51:44.861 INFO:teuthology.orchestra.run.smithi151.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 379 kB/s | 22 kB 00:00 2024-09-02T09:51:44.886 INFO:teuthology.orchestra.run.smithi151.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 381 kB/s | 42 kB 00:00 2024-09-02T09:51:44.928 INFO:teuthology.orchestra.run.smithi151.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-09-02T09:51:44.953 INFO:teuthology.orchestra.run.smithi151.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 480 kB/s | 43 kB 00:00 2024-09-02T09:51:44.987 INFO:teuthology.orchestra.run.smithi151.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 318 kB/s | 11 kB 00:00 2024-09-02T09:51:45.013 INFO:teuthology.orchestra.run.smithi151.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 704 kB/s | 18 kB 00:00 2024-09-02T09:51:45.024 INFO:teuthology.orchestra.run.smithi162.stdout:(6/37): ceph-osd-19.1.1-246.g7ad9cc7a.el9.x86_6 30 MB/s | 17 MB 00:00 2024-09-02T09:51:45.038 INFO:teuthology.orchestra.run.smithi151.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-02T09:51:45.063 INFO:teuthology.orchestra.run.smithi151.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-02T09:51:45.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-test 2024-09-02T09:51:45.089 INFO:teuthology.orchestra.run.smithi151.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-02T09:51:45.114 INFO:teuthology.orchestra.run.smithi151.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-09-02T09:51:45.139 INFO:teuthology.orchestra.run.smithi151.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-02T09:51:45.165 INFO:teuthology.orchestra.run.smithi151.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-02T09:51:45.198 INFO:teuthology.orchestra.run.smithi151.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 939 kB/s | 79 kB 00:00 2024-09-02T09:51:45.224 INFO:teuthology.orchestra.run.smithi151.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-09-02T09:51:45.249 INFO:teuthology.orchestra.run.smithi151.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-02T09:51:45.274 INFO:teuthology.orchestra.run.smithi151.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-02T09:51:45.308 INFO:teuthology.orchestra.run.smithi151.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-02T09:51:45.333 INFO:teuthology.orchestra.run.smithi151.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-09-02T09:51:45.359 INFO:teuthology.orchestra.run.smithi151.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-09-02T09:51:45.366 INFO:teuthology.orchestra.run.smithi162.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 337 kB/s | 253 kB 00:00 2024-09-02T09:51:45.384 INFO:teuthology.orchestra.run.smithi151.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-02T09:51:45.390 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:45.391 INFO:teuthology.orchestra.run.smithi151.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-02T09:51:45.450 INFO:teuthology.orchestra.run.smithi162.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 251 kB/s | 106 kB 00:00 2024-09-02T09:51:45.475 INFO:teuthology.orchestra.run.smithi162.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.2 MB/s | 135 kB 00:00 2024-09-02T09:51:45.543 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:51:45.567 INFO:teuthology.orchestra.run.smithi162.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-09-02T09:51:45.584 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:51:45.584 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:51:45.593 INFO:teuthology.orchestra.run.smithi162.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.8 MB/s | 215 kB 00:00 2024-09-02T09:51:45.665 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:27 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:51:45.668 INFO:teuthology.orchestra.run.smithi162.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 573 kB/s | 125 kB 00:00 2024-09-02T09:51:45.760 INFO:teuthology.orchestra.run.smithi162.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-02T09:51:45.785 INFO:teuthology.orchestra.run.smithi162.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 866 kB/s | 188 kB 00:00 2024-09-02T09:51:45.798 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 49 M 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 163 k 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout:Install 4 Packages 2024-09-02T09:51:45.799 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:45.800 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 49 M 2024-09-02T09:51:45.800 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 209 M 2024-09-02T09:51:45.800 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:51:45.810 INFO:teuthology.orchestra.run.smithi162.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 245 kB/s | 35 kB 00:00 2024-09-02T09:51:45.836 INFO:teuthology.orchestra.run.smithi162.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.0 MB/s | 77 kB 00:00 2024-09-02T09:51:45.861 INFO:teuthology.orchestra.run.smithi162.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-02T09:51:45.884 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:51:45.884 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:51:45.894 INFO:teuthology.orchestra.run.smithi162.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 498 kB/s | 42 kB 00:00 2024-09-02T09:51:45.936 INFO:teuthology.orchestra.run.smithi162.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-02T09:51:45.962 INFO:teuthology.orchestra.run.smithi162.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 177 kB/s | 22 kB 00:00 2024-09-02T09:51:45.995 INFO:teuthology.orchestra.run.smithi162.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 441 kB/s | 43 kB 00:00 2024-09-02T09:51:46.016 INFO:teuthology.orchestra.run.smithi187.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-09-02T09:51:46.029 INFO:teuthology.orchestra.run.smithi162.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 338 kB/s | 11 kB 00:00 2024-09-02T09:51:46.054 INFO:teuthology.orchestra.run.smithi162.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-02T09:51:46.080 INFO:teuthology.orchestra.run.smithi162.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-02T09:51:46.083 INFO:teuthology.orchestra.run.smithi187.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 950 kB/s | 64 kB 00:00 2024-09-02T09:51:46.105 INFO:teuthology.orchestra.run.smithi162.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-02T09:51:46.130 INFO:teuthology.orchestra.run.smithi162.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-02T09:51:46.142 INFO:teuthology.orchestra.run.smithi187.stdout:(3/4): libcephsqlite-19.1.1-246.g7ad9cc7a.el9.x 556 kB/s | 163 kB 00:00 2024-09-02T09:51:46.190 INFO:teuthology.orchestra.run.smithi162.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 233 kB/s | 20 kB 00:00 2024-09-02T09:51:46.264 INFO:teuthology.orchestra.run.smithi162.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 347 kB/s | 46 kB 00:00 2024-09-02T09:51:46.290 INFO:teuthology.orchestra.run.smithi162.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 790 kB/s | 79 kB 00:00 2024-09-02T09:51:46.315 INFO:teuthology.orchestra.run.smithi162.stdout:(30/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-09-02T09:51:46.340 INFO:teuthology.orchestra.run.smithi162.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-02T09:51:46.366 INFO:teuthology.orchestra.run.smithi162.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-02T09:51:46.391 INFO:teuthology.orchestra.run.smithi162.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-02T09:51:46.416 INFO:teuthology.orchestra.run.smithi162.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-02T09:51:46.441 INFO:teuthology.orchestra.run.smithi162.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 797 kB/s | 20 kB 00:00 2024-09-02T09:51:46.467 INFO:teuthology.orchestra.run.smithi162.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-02T09:51:46.500 INFO:teuthology.orchestra.run.smithi162.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-02T09:51:46.503 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:46.504 INFO:teuthology.orchestra.run.smithi162.stdout:Total 10 MB/s | 30 MB 00:02 2024-09-02T09:51:46.658 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:51:46.699 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:51:46.699 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:51:46.997 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:51:46.997 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:51:47.262 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:51:47.329 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-02T09:51:47.370 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-02T09:51:47.409 INFO:teuthology.orchestra.run.smithi187.stdout:(4/4): ceph-test-19.1.1-246.g7ad9cc7a.el9.x86_6 31 MB/s | 49 MB 00:01 2024-09-02T09:51:47.412 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:47.412 INFO:teuthology.orchestra.run.smithi187.stdout:Total 31 MB/s | 49 MB 00:01 2024-09-02T09:51:47.415 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-02T09:51:47.440 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:51:47.460 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:51:47.461 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:51:47.488 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-02T09:51:47.557 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-02T09:51:47.589 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-02T09:51:47.683 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-02T09:51:47.761 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-02T09:51:47.810 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-02T09:51:47.847 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:51:47.848 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:51:47.853 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-02T09:51:47.883 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-02T09:51:47.918 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-02T09:51:47.960 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-02T09:51:48.001 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-02T09:51:48.044 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-02T09:51:48.079 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-02T09:51:48.128 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-02T09:51:48.160 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-02T09:51:48.258 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-02T09:51:48.383 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:51:48.406 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:51:48.414 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-02T09:51:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-02T09:51:48.480 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-02T09:51:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-02T09:51:48.494 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-02T09:51:48.520 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-02T09:51:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-02T09:51:48.548 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-02T09:51:48.548 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/4 2024-09-02T09:51:48.593 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-02T09:51:48.594 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-02T09:51:48.638 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-02T09:51:48.670 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-02T09:51:48.671 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-02T09:51:48.733 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-02T09:51:48.763 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-02T09:51:48.851 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-02T09:51:48.908 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-02T09:51:48.929 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-02T09:51:48.943 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-02T09:51:48.972 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-02T09:51:48.984 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-02T09:51:49.007 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-02T09:51:49.040 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-02T09:51:49.062 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-02T09:51:49.090 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-02T09:51:49.125 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-02T09:51:49.159 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-02T09:51:49.160 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-02T09:51:49.200 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-02T09:51:49.207 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-02T09:51:49.239 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-02T09:51:49.313 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-02T09:51:49.320 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-02T09:51:49.461 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-02T09:51:49.481 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-246.g7ad9cc7a.el9.n 32/37 2024-09-02T09:51:49.511 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 33/37 2024-09-02T09:51:49.537 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 33/37 2024-09-02T09:51:49.537 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:49.537 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-02T09:51:49.537 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T09:51:49.538 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T09:51:49.538 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:49.549 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-02T09:51:49.601 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-02T09:51:49.640 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-02T09:51:49.689 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-02T09:51:49.745 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-02T09:51:49.942 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-02T09:51:49.994 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-02T09:51:50.073 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-02T09:51:50.162 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-02T09:51:50.202 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-02T09:51:50.317 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-02T09:51:50.482 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-246.g7ad9cc7a.el9.n 32/37 2024-09-02T09:51:50.512 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 33/37 2024-09-02T09:51:50.540 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 33/37 2024-09-02T09:51:50.540 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:50.540 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-02T09:51:50.540 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T09:51:50.540 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T09:51:50.540 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:50.987 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 34/37 2024-09-02T09:51:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 34/37 2024-09-02T09:51:51.015 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:51.015 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-02T09:51:51.015 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T09:51:51.015 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T09:51:51.015 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:51.463 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 35/37 2024-09-02T09:51:51.487 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 35/37 2024-09-02T09:51:51.487 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:51.488 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-02T09:51:51.488 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T09:51:51.488 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T09:51:51.488 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:51.714 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 36/37 2024-09-02T09:51:51.738 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 36/37 2024-09-02T09:51:51.738 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:51.738 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-02T09:51:51.738 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T09:51:51.738 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T09:51:51.738 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:51.802 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 37/37 2024-09-02T09:51:51.972 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 34/37 2024-09-02T09:51:51.999 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 34/37 2024-09-02T09:51:51.999 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:51.999 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-02T09:51:51.999 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T09:51:51.999 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T09:51:51.999 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:52.439 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 35/37 2024-09-02T09:51:52.463 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 35/37 2024-09-02T09:51:52.463 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:52.463 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-02T09:51:52.463 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T09:51:52.464 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T09:51:52.464 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:52.698 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 36/37 2024-09-02T09:51:52.723 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 36/37 2024-09-02T09:51:52.723 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:52.723 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-02T09:51:52.723 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T09:51:52.723 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T09:51:52.723 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:52.785 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 37/37 2024-09-02T09:51:53.109 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/4 2024-09-02T09:51:53.194 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/4 2024-09-02T09:51:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 37/37 2024-09-02T09:51:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/37 2024-09-02T09:51:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/37 2024-09-02T09:51:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/37 2024-09-02T09:51:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/37 2024-09-02T09:51:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 5/37 2024-09-02T09:51:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-246.g7ad9cc7a.el9.n 6/37 2024-09-02T09:51:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-02T09:51:53.743 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-02T09:51:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-02T09:51:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-02T09:51:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-02T09:51:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-02T09:51:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-02T09:51:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-02T09:51:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-02T09:51:53.744 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-02T09:51:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-02T09:51:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-02T09:51:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-02T09:51:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-02T09:51:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-02T09:51:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-02T09:51:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-02T09:51:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-02T09:51:53.745 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-02T09:51:53.746 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-02T09:51:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-02T09:51:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-02T09:51:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-02T09:51:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-02T09:51:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-02T09:51:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-02T09:51:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-02T09:51:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-02T09:51:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-02T09:51:53.747 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-02T09:51:54.467 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 37/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 5/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-246.g7ad9cc7a.el9.n 6/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-02T09:51:54.468 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-02T09:51:54.469 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-02T09:51:54.471 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-02T09:51:54.696 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-02T09:51:54.696 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:54.696 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:51:54.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:54.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:54.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:54.696 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mgr-modules-core-2:19.1.1-246.g7ad9cc7a.el9.noarch 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: lua-5.4.4-4.el9.x86_64 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-02T09:51:54.697 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-ply-3.11-14.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-02T09:51:54.698 INFO:teuthology.orchestra.run.smithi151.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-02T09:51:54.699 INFO:teuthology.orchestra.run.smithi151.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-02T09:51:54.699 INFO:teuthology.orchestra.run.smithi151.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-02T09:51:54.699 INFO:teuthology.orchestra.run.smithi151.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-02T09:51:54.699 INFO:teuthology.orchestra.run.smithi151.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-02T09:51:54.699 INFO:teuthology.orchestra.run.smithi151.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-02T09:51:54.699 INFO:teuthology.orchestra.run.smithi151.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-02T09:51:54.699 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:54.699 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:51:55.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-base 2024-09-02T09:51:55.410 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-02T09:51:55.410 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mgr-modules-core-2:19.1.1-246.g7ad9cc7a.el9.noarch 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: lua-5.4.4-4.el9.x86_64 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-02T09:51:55.411 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-02T09:51:55.412 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-ply-3.11-14.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:55.413 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:51:55.593 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/4 2024-09-02T09:51:55.593 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/4 2024-09-02T09:51:55.593 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/4 2024-09-02T09:51:55.593 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-02T09:51:55.607 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:52 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:51:55.696 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:51:55.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-base 2024-09-02T09:51:55.737 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:51:55.738 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-02T09:51:55.738 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:51:55.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install cephadm 2024-09-02T09:51:55.916 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-02T09:51:55.916 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:55.917 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:51:55.917 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:55.917 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:51:55.917 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-02T09:51:55.917 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-02T09:51:55.917 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:55.917 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:51:56.155 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph 2024-09-02T09:51:56.286 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:53 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:51:56.364 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:53 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:51:56.376 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:51:56.417 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:51:56.419 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-02T09:51:56.419 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:51:56.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install cephadm 2024-09-02T09:51:56.495 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:51:56.496 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-02T09:51:56.496 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:56.496 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:51:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm noarch 2:19.1.1-246.g7ad9cc7a.el9 ceph-noarch 765 k 2024-09-02T09:51:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:56.496 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:51:56.496 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:56.496 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-02T09:51:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:56.497 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 765 k 2024-09-02T09:51:56.497 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 772 k 2024-09-02T09:51:56.497 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:51:56.730 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:38 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:51:56.751 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm-19.1.1-246.g7ad9cc7a.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-02T09:51:56.751 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:56.752 INFO:teuthology.orchestra.run.smithi151.stdout:Total 2.9 MB/s | 765 kB 00:00 2024-09-02T09:51:56.752 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:51:56.757 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:51:56.757 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:51:56.763 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:51:56.763 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:51:56.844 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:51:56.883 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:51:56.885 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:51:56.885 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-02T09:51:56.885 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 6.6 k 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 2.4 M 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 1.5 M 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:19.1.1-246.g7ad9cc7a.el9 ceph-noarch 249 k 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 4.7 M 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 17 M 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-02T09:51:56.886 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-02T09:51:56.887 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-02T09:51:56.888 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-02T09:51:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:56.889 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:51:56.889 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:51:56.889 INFO:teuthology.orchestra.run.smithi187.stdout:Install 37 Packages 2024-09-02T09:51:56.889 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:51:56.889 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 30 M 2024-09-02T09:51:56.889 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 108 M 2024-09-02T09:51:56.890 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:51:57.041 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:54 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:51:57.174 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:51:57.175 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:57.175 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-02T09:51:57.175 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:57.175 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:51:57.175 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm noarch 2:19.1.1-246.g7ad9cc7a.el9 ceph-noarch 765 k 2024-09-02T09:51:57.175 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:57.175 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:51:57.175 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:57.175 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-02T09:51:57.176 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:57.176 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 765 k 2024-09-02T09:51:57.176 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 772 k 2024-09-02T09:51:57.176 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:51:57.255 INFO:teuthology.orchestra.run.smithi187.stdout:(1/37): ceph-19.1.1-246.g7ad9cc7a.el9.x86_64.rp 51 kB/s | 6.6 kB 00:00 2024-09-02T09:51:57.304 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:51:57.379 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:51:57.447 INFO:teuthology.orchestra.run.smithi187.stdout:(2/37): ceph-mds-19.1.1-246.g7ad9cc7a.el9.x86_6 7.4 MB/s | 2.4 MB 00:00 2024-09-02T09:51:57.466 INFO:teuthology.orchestra.run.smithi162.stdout:cephadm-19.1.1-246.g7ad9cc7a.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-02T09:51:57.467 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:57.467 INFO:teuthology.orchestra.run.smithi162.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-02T09:51:57.467 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:51:57.472 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:51:57.472 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:51:57.478 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:51:57.478 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:51:57.489 INFO:teuthology.orchestra.run.smithi187.stdout:(3/37): ceph-mgr-19.1.1-246.g7ad9cc7a.el9.x86_6 4.2 MB/s | 1.5 MB 00:00 2024-09-02T09:51:57.550 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:51:57.581 INFO:teuthology.orchestra.run.smithi187.stdout:(4/37): ceph-mgr-modules-core-19.1.1-246.g7ad9c 2.7 MB/s | 249 kB 00:00 2024-09-02T09:51:57.726 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:51:57.756 INFO:teuthology.orchestra.run.smithi187.stdout:(5/37): ceph-mon-19.1.1-246.g7ad9cc7a.el9.x86_6 9.4 MB/s | 4.7 MB 00:00 2024-09-02T09:51:57.945 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:51:57.946 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:57.946 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:51:57.946 INFO:teuthology.orchestra.run.smithi151.stdout: cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 2024-09-02T09:51:57.946 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:57.946 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:51:57.972 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:51:58.006 INFO:teuthology.orchestra.run.smithi187.stdout:(6/37): ceph-osd-19.1.1-246.g7ad9cc7a.el9.x86_6 30 MB/s | 17 MB 00:00 2024-09-02T09:51:58.065 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:51:58.069 INFO:teuthology.orchestra.run.smithi187.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 518 kB/s | 253 kB 00:00 2024-09-02T09:51:58.099 INFO:teuthology.orchestra.run.smithi187.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-09-02T09:51:58.157 INFO:teuthology.orchestra.run.smithi187.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-02T09:51:58.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-immutable-object-cache 2024-09-02T09:51:58.216 INFO:teuthology.orchestra.run.smithi187.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.7 MB/s | 1.2 MB 00:00 2024-09-02T09:51:58.242 INFO:teuthology.orchestra.run.smithi187.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.5 MB/s | 215 kB 00:00 2024-09-02T09:51:58.267 INFO:teuthology.orchestra.run.smithi187.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 687 kB/s | 135 kB 00:00 2024-09-02T09:51:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:51:58.653 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:51:58.653 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:58.653 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:51:58.653 INFO:teuthology.orchestra.run.smithi162.stdout: cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 2024-09-02T09:51:58.653 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:58.654 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:51:58.742 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:55 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:51:58.842 INFO:teuthology.orchestra.run.smithi187.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 60 kB/s | 35 kB 00:00 2024-09-02T09:51:58.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-immutable-object-cache 2024-09-02T09:51:58.877 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:51:58.877 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:58.877 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-09-02T09:51:58.877 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:58.877 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:51:58.877 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 145 k 2024-09-02T09:51:58.878 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:58.878 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:51:58.878 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:51:58.878 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-02T09:51:58.878 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:58.878 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 145 k 2024-09-02T09:51:58.878 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 435 k 2024-09-02T09:51:58.878 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:51:58.950 INFO:teuthology.orchestra.run.smithi187.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 257 kB/s | 188 kB 00:00 2024-09-02T09:51:58.976 INFO:teuthology.orchestra.run.smithi187.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 235 kB/s | 172 kB 00:00 2024-09-02T09:51:59.094 INFO:teuthology.orchestra.run.smithi151.stdout:ceph-immutable-object-cache-19.1.1-246.g7ad9cc7 673 kB/s | 145 kB 00:00 2024-09-02T09:51:59.095 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:59.096 INFO:teuthology.orchestra.run.smithi151.stdout:Total 667 kB/s | 145 kB 00:00 2024-09-02T09:51:59.096 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:51:59.103 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:51:59.103 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:51:59.161 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:51:59.161 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:51:59.312 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:51:59.404 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.e 1/1 2024-09-02T09:51:59.424 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:56 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:51:59.430 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.e 1/1 2024-09-02T09:51:59.430 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:51:59.430 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-02T09:51:59.430 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:51:59.476 INFO:teuthology.orchestra.run.smithi187.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 122 kB/s | 77 kB 00:00 2024-09-02T09:51:59.551 INFO:teuthology.orchestra.run.smithi187.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 85 kB/s | 51 kB 00:00 2024-09-02T09:51:59.557 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:51:59.558 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-02T09:51:59.558 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:59.558 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:51:59.558 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 145 k 2024-09-02T09:51:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:59.559 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:51:59.559 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:51:59.559 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-02T09:51:59.559 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:51:59.559 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 145 k 2024-09-02T09:51:59.560 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 435 k 2024-09-02T09:51:59.560 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:51:59.593 INFO:teuthology.orchestra.run.smithi187.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-02T09:51:59.618 INFO:teuthology.orchestra.run.smithi187.stdout:(19/37): python3-toml-0.10.2-6.el9.noarch.rpm 65 kB/s | 42 kB 00:00 2024-09-02T09:51:59.644 INFO:teuthology.orchestra.run.smithi187.stdout:(20/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 882 kB/s | 43 kB 00:00 2024-09-02T09:51:59.685 INFO:teuthology.orchestra.run.smithi187.stdout:(21/37): python3-cherrypy-18.6.1-2.el9.noarch.r 10 MB/s | 358 kB 00:00 2024-09-02T09:51:59.711 INFO:teuthology.orchestra.run.smithi187.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 459 kB/s | 11 kB 00:00 2024-09-02T09:51:59.736 INFO:teuthology.orchestra.run.smithi187.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 713 kB/s | 18 kB 00:00 2024-09-02T09:51:59.761 INFO:teuthology.orchestra.run.smithi187.stdout:(24/37): lua-devel-5.4.4-4.el9.x86_64.rpm 78 kB/s | 22 kB 00:00 2024-09-02T09:51:59.769 INFO:teuthology.orchestra.run.smithi162.stdout:ceph-immutable-object-cache-19.1.1-246.g7ad9cc7 695 kB/s | 145 kB 00:00 2024-09-02T09:51:59.769 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:51:59.770 INFO:teuthology.orchestra.run.smithi162.stdout:Total 688 kB/s | 145 kB 00:00 2024-09-02T09:51:59.770 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:51:59.777 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:51:59.777 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:51:59.786 INFO:teuthology.orchestra.run.smithi187.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-02T09:51:59.820 INFO:teuthology.orchestra.run.smithi187.stdout:(26/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 852 kB/s | 172 kB 00:00 2024-09-02T09:51:59.834 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:51:59.835 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:51:59.845 INFO:teuthology.orchestra.run.smithi187.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 385 kB/s | 20 kB 00:00 2024-09-02T09:51:59.870 INFO:teuthology.orchestra.run.smithi187.stdout:(28/37): python3-jaraco-functools-3.5.0-2.el9.n 178 kB/s | 19 kB 00:00 2024-09-02T09:51:59.896 INFO:teuthology.orchestra.run.smithi187.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 628 kB/s | 46 kB 00:00 2024-09-02T09:51:59.921 INFO:teuthology.orchestra.run.smithi187.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-02T09:51:59.946 INFO:teuthology.orchestra.run.smithi187.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 329 kB/s | 16 kB 00:00 2024-09-02T09:51:59.971 INFO:teuthology.orchestra.run.smithi187.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.6 MB/s | 90 kB 00:00 2024-09-02T09:51:59.985 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:51:59.996 INFO:teuthology.orchestra.run.smithi187.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 718 kB/s | 36 kB 00:00 2024-09-02T09:52:00.022 INFO:teuthology.orchestra.run.smithi187.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-02T09:52:00.047 INFO:teuthology.orchestra.run.smithi187.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-02T09:52:00.072 INFO:teuthology.orchestra.run.smithi187.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-09-02T09:52:00.078 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.e 1/1 2024-09-02T09:52:00.103 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.e 1/1 2024-09-02T09:52:00.103 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:52:00.103 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-02T09:52:00.103 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:00.106 INFO:teuthology.orchestra.run.smithi187.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-02T09:52:00.112 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:00.112 INFO:teuthology.orchestra.run.smithi187.stdout:Total 9.4 MB/s | 30 MB 00:03 2024-09-02T09:52:00.270 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:52:00.310 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:52:00.310 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:52:00.354 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.e 1/1 2024-09-02T09:52:00.354 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:00.354 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:52:00.355 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:00.355 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:00.355 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:00.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-mgr 2024-09-02T09:52:00.608 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:52:00.608 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:52:01.057 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.e 1/1 2024-09-02T09:52:01.057 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:01.057 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:52:01.057 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:01.057 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:01.057 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:01.129 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:58 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:01.222 INFO:teuthology.orchestra.run.smithi151.stdout:Package ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:01.265 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:01.266 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-02T09:52:01.266 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:01.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-mgr 2024-09-02T09:52:01.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install ceph-fuse 2024-09-02T09:52:01.854 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:58 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:01.893 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:00:58 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:01.946 INFO:teuthology.orchestra.run.smithi162.stdout:Package ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:01.990 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:01.991 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-02T09:52:01.991 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:02.002 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:52:02.025 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:02.025 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:02.025 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repository Size 2024-09-02T09:52:02.025 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:02.026 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:52:02.026 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 896 k 2024-09-02T09:52:02.026 INFO:teuthology.orchestra.run.smithi151.stdout:Installing dependencies: 2024-09-02T09:52:02.026 INFO:teuthology.orchestra.run.smithi151.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-02T09:52:02.026 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:02.026 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:52:02.026 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:02.026 INFO:teuthology.orchestra.run.smithi151.stdout:Install 2 Packages 2024-09-02T09:52:02.026 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:02.027 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 976 k 2024-09-02T09:52:02.027 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 2.9 M 2024-09-02T09:52:02.027 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:52:02.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install ceph-fuse 2024-09-02T09:52:02.068 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-02T09:52:02.100 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-02T09:52:02.137 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-02T09:52:02.184 INFO:teuthology.orchestra.run.smithi151.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 584 kB/s | 80 kB 00:00 2024-09-02T09:52:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-02T09:52:02.260 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-02T09:52:02.293 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-02T09:52:02.326 INFO:teuthology.orchestra.run.smithi151.stdout:(2/2): ceph-fuse-19.1.1-246.g7ad9cc7a.el9.x86_6 3.1 MB/s | 896 kB 00:00 2024-09-02T09:52:02.326 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:02.327 INFO:teuthology.orchestra.run.smithi151.stdout:Total 3.2 MB/s | 976 kB 00:00 2024-09-02T09:52:02.338 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:52:02.360 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:52:02.385 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-02T09:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:52:02.433 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:52:02.464 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-02T09:52:02.504 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-02T09:52:02.539 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-02T09:52:02.569 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-02T09:52:02.603 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-02T09:52:02.614 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:00:59 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:02.638 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:52:02.645 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-02T09:52:02.686 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-02T09:52:02.722 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-02T09:52:02.745 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:02.746 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repository Size 2024-09-02T09:52:02.746 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:02.746 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:52:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 896 k 2024-09-02T09:52:02.746 INFO:teuthology.orchestra.run.smithi162.stdout:Installing dependencies: 2024-09-02T09:52:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-02T09:52:02.746 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:02.747 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:52:02.747 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:02.747 INFO:teuthology.orchestra.run.smithi162.stdout:Install 2 Packages 2024-09-02T09:52:02.747 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:02.747 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 976 k 2024-09-02T09:52:02.747 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 2.9 M 2024-09-02T09:52:02.747 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:52:02.764 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-02T09:52:02.778 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-02T09:52:02.812 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-02T09:52:02.844 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-02T09:52:02.847 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/2 2024-09-02T09:52:02.917 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-02T09:52:03.065 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-02T09:52:03.120 INFO:teuthology.orchestra.run.smithi162.stdout:(1/2): ceph-fuse-19.1.1-246.g7ad9cc7a.el9.x86_6 3.2 MB/s | 896 kB 00:00 2024-09-02T09:52:03.152 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-02T09:52:03.205 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-02T09:52:03.243 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-02T09:52:03.301 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-02T09:52:03.356 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-02T09:52:03.487 INFO:teuthology.orchestra.run.smithi162.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 125 kB/s | 80 kB 00:00 2024-09-02T09:52:03.488 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:03.488 INFO:teuthology.orchestra.run.smithi162.stdout:Total 1.3 MB/s | 976 kB 00:00 2024-09-02T09:52:03.501 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:52:03.525 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:52:03.525 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:52:03.553 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-02T09:52:03.563 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/2 2024-09-02T09:52:03.563 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/2 2024-09-02T09:52:03.598 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-02T09:52:03.600 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:52:03.600 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:52:03.676 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-02T09:52:03.780 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-02T09:52:03.785 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:52:03.820 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-02T09:52:03.820 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:03.820 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:52:03.820 INFO:teuthology.orchestra.run.smithi151.stdout: ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-02T09:52:03.820 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:03.820 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:03.821 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-02T09:52:03.930 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-02T09:52:03.939 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-02T09:52:04.008 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/2 2024-09-02T09:52:04.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados-devel 2024-09-02T09:52:04.102 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-246.g7ad9cc7a.el9.n 32/37 2024-09-02T09:52:04.131 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 33/37 2024-09-02T09:52:04.158 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 33/37 2024-09-02T09:52:04.158 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:52:04.158 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-02T09:52:04.158 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T09:52:04.158 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-02T09:52:04.159 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:04.631 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:01 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:04.705 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/2 2024-09-02T09:52:04.705 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/2 2024-09-02T09:52:04.769 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:04.770 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:04.770 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-09-02T09:52:04.770 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:04.770 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:52:04.770 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 127 k 2024-09-02T09:52:04.770 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:04.770 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:52:04.770 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:04.771 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-02T09:52:04.771 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:04.771 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 127 k 2024-09-02T09:52:04.771 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 456 k 2024-09-02T09:52:04.771 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:52:04.947 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-02T09:52:04.947 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:04.947 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:52:04.947 INFO:teuthology.orchestra.run.smithi162.stdout: ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-02T09:52:04.947 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:04.947 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:04.969 INFO:teuthology.orchestra.run.smithi151.stdout:librados-devel-19.1.1-246.g7ad9cc7a.el9.x86_64. 643 kB/s | 127 kB 00:00 2024-09-02T09:52:04.969 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:04.969 INFO:teuthology.orchestra.run.smithi151.stdout:Total 637 kB/s | 127 kB 00:00 2024-09-02T09:52:04.970 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:52:04.975 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:52:04.975 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:52:05.033 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:52:05.033 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:52:05.179 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:52:05.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados-devel 2024-09-02T09:52:05.289 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:05.591 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 34/37 2024-09-02T09:52:05.619 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 34/37 2024-09-02T09:52:05.619 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:52:05.619 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-02T09:52:05.619 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T09:52:05.619 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-02T09:52:05.619 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:05.687 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:05.764 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:02 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:05.899 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:05.900 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-02T09:52:05.900 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:05.900 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:52:05.900 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 127 k 2024-09-02T09:52:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:05.901 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:52:05.901 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:05.901 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-02T09:52:05.901 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:05.901 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 127 k 2024-09-02T09:52:05.901 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 456 k 2024-09-02T09:52:05.902 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:52:05.962 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:05.962 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:05.962 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:52:05.962 INFO:teuthology.orchestra.run.smithi151.stdout: librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:05.962 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:05.962 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:06.050 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 35/37 2024-09-02T09:52:06.074 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 35/37 2024-09-02T09:52:06.074 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:52:06.074 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-02T09:52:06.074 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T09:52:06.074 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-02T09:52:06.074 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:06.092 INFO:teuthology.orchestra.run.smithi162.stdout:librados-devel-19.1.1-246.g7ad9cc7a.el9.x86_64. 664 kB/s | 127 kB 00:00 2024-09-02T09:52:06.093 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:06.093 INFO:teuthology.orchestra.run.smithi162.stdout:Total 658 kB/s | 127 kB 00:00 2024-09-02T09:52:06.093 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:52:06.099 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:52:06.099 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:52:06.157 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:52:06.157 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:52:06.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs2 2024-09-02T09:52:06.300 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 36/37 2024-09-02T09:52:06.301 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:52:06.324 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 36/37 2024-09-02T09:52:06.324 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:52:06.324 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-02T09:52:06.324 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T09:52:06.324 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-02T09:52:06.324 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:06.387 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 37/37 2024-09-02T09:52:06.404 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:06.749 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:03 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:06.840 INFO:teuthology.orchestra.run.smithi151.stdout:Package libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:06.859 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:06.883 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:06.884 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-02T09:52:06.884 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:06.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install libcephfs-devel 2024-09-02T09:52:07.091 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:07.091 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:07.091 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:52:07.091 INFO:teuthology.orchestra.run.smithi162.stdout: librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:07.091 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:07.091 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:07.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs2 2024-09-02T09:52:07.511 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:04 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:07.641 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:07.642 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:07.642 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-09-02T09:52:07.642 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:07.642 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:52:07.642 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 32 k 2024-09-02T09:52:07.643 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:07.643 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:52:07.643 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:07.643 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-02T09:52:07.643 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:07.643 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 32 k 2024-09-02T09:52:07.643 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 142 k 2024-09-02T09:52:07.643 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:52:07.810 INFO:teuthology.orchestra.run.smithi151.stdout:libcephfs-devel-19.1.1-246.g7ad9cc7a.el9.x86_64 190 kB/s | 32 kB 00:00 2024-09-02T09:52:07.810 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:07.811 INFO:teuthology.orchestra.run.smithi151.stdout:Total 188 kB/s | 32 kB 00:00 2024-09-02T09:52:07.811 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:52:07.815 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:52:07.815 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:52:07.819 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:52:07.820 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:52:07.889 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:04 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:07.908 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:52:07.980 INFO:teuthology.orchestra.run.smithi162.stdout:Package libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:08.003 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:08.025 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:08.026 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-02T09:52:08.027 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:08.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install libcephfs-devel 2024-09-02T09:52:08.133 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 37/37 2024-09-02T09:52:08.133 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/37 2024-09-02T09:52:08.133 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/37 2024-09-02T09:52:08.133 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 3/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 4/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 5/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-246.g7ad9cc7a.el9.n 6/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-02T09:52:08.134 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-02T09:52:08.135 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-02T09:52:08.136 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-02T09:52:08.136 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-02T09:52:08.136 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-02T09:52:08.137 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-02T09:52:08.137 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-02T09:52:08.137 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-02T09:52:08.350 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:08.569 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:08.569 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:08.569 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:52:08.569 INFO:teuthology.orchestra.run.smithi151.stdout: libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:08.569 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:08.569 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:08.656 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:05 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:08.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librados2 2024-09-02T09:52:08.787 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:08.788 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:08.788 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-02T09:52:08.788 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:08.788 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:52:08.789 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 32 k 2024-09-02T09:52:08.789 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:08.789 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:52:08.789 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:08.789 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-02T09:52:08.789 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:08.789 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 32 k 2024-09-02T09:52:08.789 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 142 k 2024-09-02T09:52:08.789 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:52:08.938 INFO:teuthology.orchestra.run.smithi162.stdout:libcephfs-devel-19.1.1-246.g7ad9cc7a.el9.x86_64 213 kB/s | 32 kB 00:00 2024-09-02T09:52:08.939 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:08.940 INFO:teuthology.orchestra.run.smithi162.stdout:Total 211 kB/s | 32 kB 00:00 2024-09-02T09:52:08.940 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:52:08.943 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:52:08.943 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:52:08.947 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:52:08.947 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:19.1.1-246.g7ad9cc7a.el9.noarch 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: lua-5.4.4-4.el9.x86_64 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-02T09:52:09.019 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-02T09:52:09.020 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply-3.11-14.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-02T09:52:09.021 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-02T09:52:09.022 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-02T09:52:09.022 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-02T09:52:09.022 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:09.022 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:52:09.133 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:09.302 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:06 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:09.347 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-base 2024-09-02T09:52:09.391 INFO:teuthology.orchestra.run.smithi151.stdout:Package librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:09.433 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:09.434 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-02T09:52:09.434 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:09.471 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:09.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install librbd1 2024-09-02T09:52:09.696 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:09.696 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:09.696 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:52:09.697 INFO:teuthology.orchestra.run.smithi162.stdout: libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:09.697 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:09.697 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:09.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librados2 2024-09-02T09:52:09.909 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:51 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:09.999 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-base-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:10.040 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:10.041 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-02T09:52:10.041 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:10.057 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:07 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:10.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install cephadm 2024-09-02T09:52:10.146 INFO:teuthology.orchestra.run.smithi151.stdout:Package librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:10.189 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:10.190 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-02T09:52:10.190 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:10.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rados 2024-09-02T09:52:10.415 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:07 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:10.503 INFO:teuthology.orchestra.run.smithi162.stdout:Package librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:10.544 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:10.545 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-02T09:52:10.545 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:10.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install librbd1 2024-09-02T09:52:10.655 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:52 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:10.786 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:10.786 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:10.786 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:19.1.1-246.g7ad9cc7a.el9 ceph-noarch 765 k 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 765 k 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 772 k 2024-09-02T09:52:10.787 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:52:10.807 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:07 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:10.899 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:10.940 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:10.941 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-02T09:52:10.941 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:11.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rgw 2024-09-02T09:52:11.069 INFO:teuthology.orchestra.run.smithi187.stdout:cephadm-19.1.1-246.g7ad9cc7a.el9.noarch.rpm 2.7 MB/s | 765 kB 00:00 2024-09-02T09:52:11.070 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:11.070 INFO:teuthology.orchestra.run.smithi187.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-02T09:52:11.070 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:52:11.075 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:52:11.075 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:52:11.081 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:52:11.081 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:52:11.152 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:52:11.161 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:08 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:11.251 INFO:teuthology.orchestra.run.smithi162.stdout:Package librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:11.293 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:11.294 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-02T09:52:11.294 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:11.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rados 2024-09-02T09:52:11.550 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:08 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:11.638 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:52:11.640 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:11.683 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:11.684 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-02T09:52:11.684 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:11.722 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:52:11.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-cephfs 2024-09-02T09:52:11.922 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:08 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:12.011 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:12.046 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:52:12.052 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:12.053 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-02T09:52:12.054 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:12.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rgw 2024-09-02T09:52:12.269 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 1/1 2024-09-02T09:52:12.270 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:12.270 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:52:12.270 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:19.1.1-246.g7ad9cc7a.el9.noarch 2024-09-02T09:52:12.270 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:12.270 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:12.319 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:09 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:12.411 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:12.455 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:12.456 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-02T09:52:12.456 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:12.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-immutable-object-cache 2024-09-02T09:52:12.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install python3-rbd 2024-09-02T09:52:12.674 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:09 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:12.763 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:12.805 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:12.806 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-02T09:52:12.806 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:12.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-cephfs 2024-09-02T09:52:13.060 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:55 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:13.118 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:10 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:13.192 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:13.193 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:13.193 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-02T09:52:13.193 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:13.193 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:52:13.193 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 145 k 2024-09-02T09:52:13.193 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:13.193 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:52:13.194 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:13.194 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-02T09:52:13.194 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:13.194 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 145 k 2024-09-02T09:52:13.194 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 435 k 2024-09-02T09:52:13.194 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:52:13.210 INFO:teuthology.orchestra.run.smithi151.stdout:Package python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:13.255 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:13.256 INFO:teuthology.orchestra.run.smithi151.stdout:Nothing to do. 2024-09-02T09:52:13.256 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:13.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-fuse 2024-09-02T09:52:13.400 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-immutable-object-cache-19.1.1-246.g7ad9cc7 703 kB/s | 145 kB 00:00 2024-09-02T09:52:13.401 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:13.401 INFO:teuthology.orchestra.run.smithi187.stdout:Total 697 kB/s | 145 kB 00:00 2024-09-02T09:52:13.401 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:52:13.409 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:52:13.409 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:52:13.439 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:10 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:13.466 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:52:13.466 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:52:13.534 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:13.578 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:13.579 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-02T09:52:13.579 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:13.614 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:52:13.659 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install python3-rbd 2024-09-02T09:52:13.693 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.e 1/1 2024-09-02T09:52:13.718 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.e 1/1 2024-09-02T09:52:13.718 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:52:13.718 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-02T09:52:13.718 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:13.882 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:10 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:14.015 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:14.015 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:14.015 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-09-02T09:52:14.015 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:14.015 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:52:14.016 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 85 k 2024-09-02T09:52:14.016 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:14.016 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:52:14.016 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:14.016 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-02T09:52:14.016 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:14.016 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 85 k 2024-09-02T09:52:14.017 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 231 k 2024-09-02T09:52:14.017 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:52:14.207 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-fuse-19.1.1-246.g7ad9cc7a.el9.x86_64.rpm 450 kB/s | 85 kB 00:00 2024-09-02T09:52:14.207 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:14.208 INFO:teuthology.orchestra.run.smithi151.stdout:Total 447 kB/s | 85 kB 00:00 2024-09-02T09:52:14.208 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:52:14.214 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:52:14.214 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:52:14.232 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:11 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:14.272 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:52:14.272 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:52:14.322 INFO:teuthology.orchestra.run.smithi162.stdout:Package python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:14.365 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:14.366 INFO:teuthology.orchestra.run.smithi162.stdout:Nothing to do. 2024-09-02T09:52:14.366 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:14.413 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:52:14.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-fuse 2024-09-02T09:52:14.520 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:14.617 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.e 1/1 2024-09-02T09:52:14.617 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:14.617 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:52:14.617 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:14.617 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:14.617 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:14.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-mgr 2024-09-02T09:52:14.900 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:14.988 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:11 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:15.118 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:15.118 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 85 k 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 85 k 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 231 k 2024-09-02T09:52:15.119 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:52:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:15.157 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:52:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:15.157 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:15.309 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-fuse-19.1.1-246.g7ad9cc7a.el9.x86_64.rpm 452 kB/s | 85 kB 00:00 2024-09-02T09:52:15.309 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:15.309 INFO:teuthology.orchestra.run.smithi162.stdout:Total 449 kB/s | 85 kB 00:00 2024-09-02T09:52:15.310 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:52:15.316 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:52:15.317 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:52:15.373 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:52:15.373 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:52:15.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-mirror 2024-09-02T09:52:15.427 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:57 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:15.509 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:52:15.520 INFO:teuthology.orchestra.run.smithi187.stdout:Package ceph-mgr-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:15.562 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:15.563 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-02T09:52:15.563 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:15.610 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:15.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-fuse 2024-09-02T09:52:15.949 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:12 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:15.992 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:16.083 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:16.084 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: Package Arch Version Repo Size 2024-09-02T09:52:16.084 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:16.084 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:52:16.084 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 3.1 M 2024-09-02T09:52:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:16.085 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:52:16.085 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:16.085 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-02T09:52:16.085 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:16.085 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 3.1 M 2024-09-02T09:52:16.086 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 13 M 2024-09-02T09:52:16.086 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:52:16.216 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:58 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:16.226 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:16.226 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:16.227 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:52:16.227 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:16.227 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:16.227 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:16.347 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:16.347 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:16.347 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-09-02T09:52:16.347 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:16.347 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:52:16.347 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 896 k 2024-09-02T09:52:16.347 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-09-02T09:52:16.348 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-02T09:52:16.348 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:16.348 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:52:16.348 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:16.348 INFO:teuthology.orchestra.run.smithi187.stdout:Install 2 Packages 2024-09-02T09:52:16.348 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:16.348 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 976 k 2024-09-02T09:52:16.349 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 2.9 M 2024-09-02T09:52:16.349 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:52:16.422 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-mirror-19.1.1-246.g7ad9cc7a.el9.x86_64.rpm 9.2 MB/s | 3.1 MB 00:00 2024-09-02T09:52:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:Total 9.2 MB/s | 3.1 MB 00:00 2024-09-02T09:52:16.423 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:52:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:52:16.432 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:52:16.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-mirror 2024-09-02T09:52:16.487 INFO:teuthology.orchestra.run.smithi187.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 680 kB/s | 80 kB 00:00 2024-09-02T09:52:16.507 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:52:16.507 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:52:16.662 INFO:teuthology.orchestra.run.smithi187.stdout:(2/2): ceph-fuse-19.1.1-246.g7ad9cc7a.el9.x86_6 3.0 MB/s | 896 kB 00:00 2024-09-02T09:52:16.663 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:16.663 INFO:teuthology.orchestra.run.smithi187.stdout:Total 3.0 MB/s | 976 kB 00:00 2024-09-02T09:52:16.674 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:52:16.695 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:52:16.695 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:52:16.766 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:52:16.767 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:52:16.923 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:52:16.944 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:52:16.998 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:17.024 INFO:teuthology.orchestra.run.smithi151.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:52:17.024 INFO:teuthology.orchestra.run.smithi151.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-02T09:52:17.024 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T09:52:17.024 INFO:teuthology.orchestra.run.smithi151.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T09:52:17.024 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:17.046 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:14 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:17.089 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-02T09:52:17.149 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/2 2024-09-02T09:52:17.182 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:17.183 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:17.183 INFO:teuthology.orchestra.run.smithi162.stdout: Package Arch Version Repo Size 2024-09-02T09:52:17.183 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:17.183 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:52:17.183 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 3.1 M 2024-09-02T09:52:17.183 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:17.183 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:52:17.183 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:17.183 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-02T09:52:17.184 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:17.184 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 3.1 M 2024-09-02T09:52:17.184 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 13 M 2024-09-02T09:52:17.184 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:52:17.538 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-mirror-19.1.1-246.g7ad9cc7a.el9.x86_64.rpm 8.8 MB/s | 3.1 MB 00:00 2024-09-02T09:52:17.539 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:17.540 INFO:teuthology.orchestra.run.smithi162.stdout:Total 8.7 MB/s | 3.1 MB 00:00 2024-09-02T09:52:17.540 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:52:17.549 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:52:17.549 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:52:17.624 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:52:17.624 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:52:17.810 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2/2 2024-09-02T09:52:17.810 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/2 2024-09-02T09:52:18.045 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:52:18.060 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-02T09:52:18.061 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:18.061 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:52:18.061 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-02T09:52:18.061 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:18.061 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:18.084 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:18.084 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:18.085 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:52:18.085 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:18.085 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:18.085 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:18.105 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:18.131 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:18.131 INFO:teuthology.orchestra.run.smithi162.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:52:18.132 INFO:teuthology.orchestra.run.smithi162.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-02T09:52:18.132 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T09:52:18.132 INFO:teuthology.orchestra.run.smithi162.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T09:52:18.132 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:18.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo yum -y install rbd-nbd 2024-09-02T09:52:18.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados-devel 2024-09-02T09:52:18.870 INFO:teuthology.orchestra.run.smithi151.stdout:Last metadata expiration check: 0:01:15 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:18.925 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:00 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:19.006 INFO:teuthology.orchestra.run.smithi151.stdout:Dependencies resolved. 2024-09-02T09:52:19.007 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: Package Architecture Version Repository Size 2024-09-02T09:52:19.007 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:19.007 INFO:teuthology.orchestra.run.smithi151.stdout:Installing: 2024-09-02T09:52:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 171 k 2024-09-02T09:52:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:19.007 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction Summary 2024-09-02T09:52:19.007 INFO:teuthology.orchestra.run.smithi151.stdout:================================================================================ 2024-09-02T09:52:19.007 INFO:teuthology.orchestra.run.smithi151.stdout:Install 1 Package 2024-09-02T09:52:19.007 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:19.008 INFO:teuthology.orchestra.run.smithi151.stdout:Total download size: 171 k 2024-09-02T09:52:19.008 INFO:teuthology.orchestra.run.smithi151.stdout:Installed size: 494 k 2024-09-02T09:52:19.008 INFO:teuthology.orchestra.run.smithi151.stdout:Downloading Packages: 2024-09-02T09:52:19.062 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:19.063 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:19.063 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-02T09:52:19.064 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:19.064 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:52:19.064 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 127 k 2024-09-02T09:52:19.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:19.064 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:52:19.064 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:19.064 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-02T09:52:19.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:19.064 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 127 k 2024-09-02T09:52:19.065 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 456 k 2024-09-02T09:52:19.065 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:52:19.198 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:19.198 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:19.199 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:52:19.199 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:19.199 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:19.199 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:19.212 INFO:teuthology.orchestra.run.smithi151.stdout:rbd-nbd-19.1.1-246.g7ad9cc7a.el9.x86_64.rpm 836 kB/s | 171 kB 00:00 2024-09-02T09:52:19.213 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:19.213 INFO:teuthology.orchestra.run.smithi151.stdout:Total 830 kB/s | 171 kB 00:00 2024-09-02T09:52:19.213 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction check 2024-09-02T09:52:19.221 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction check succeeded. 2024-09-02T09:52:19.221 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction test 2024-09-02T09:52:19.258 INFO:teuthology.orchestra.run.smithi187.stdout:librados-devel-19.1.1-246.g7ad9cc7a.el9.x86_64. 652 kB/s | 127 kB 00:00 2024-09-02T09:52:19.259 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:19.259 INFO:teuthology.orchestra.run.smithi187.stdout:Total 646 kB/s | 127 kB 00:00 2024-09-02T09:52:19.259 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:52:19.265 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:52:19.265 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:52:19.278 INFO:teuthology.orchestra.run.smithi151.stdout:Transaction test succeeded. 2024-09-02T09:52:19.279 INFO:teuthology.orchestra.run.smithi151.stdout:Running transaction 2024-09-02T09:52:19.321 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:52:19.322 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:52:19.423 INFO:teuthology.orchestra.run.smithi151.stdout: Preparing : 1/1 2024-09-02T09:52:19.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo yum -y install rbd-nbd 2024-09-02T09:52:19.461 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:52:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: Installing : rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:19.552 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:19.907 INFO:teuthology.orchestra.run.smithi151.stdout: Running scriptlet: rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:19.959 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:20.006 INFO:teuthology.orchestra.run.smithi162.stdout:Last metadata expiration check: 0:01:17 ago on Mon 02 Sep 2024 09:51:03 AM UTC. 2024-09-02T09:52:20.140 INFO:teuthology.orchestra.run.smithi162.stdout:Dependencies resolved. 2024-09-02T09:52:20.141 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:20.141 INFO:teuthology.orchestra.run.smithi162.stdout: Package Architecture Version Repository Size 2024-09-02T09:52:20.141 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:20.141 INFO:teuthology.orchestra.run.smithi162.stdout:Installing: 2024-09-02T09:52:20.141 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 171 k 2024-09-02T09:52:20.141 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:20.141 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction Summary 2024-09-02T09:52:20.141 INFO:teuthology.orchestra.run.smithi162.stdout:================================================================================ 2024-09-02T09:52:20.141 INFO:teuthology.orchestra.run.smithi162.stdout:Install 1 Package 2024-09-02T09:52:20.142 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:20.142 INFO:teuthology.orchestra.run.smithi162.stdout:Total download size: 171 k 2024-09-02T09:52:20.142 INFO:teuthology.orchestra.run.smithi162.stdout:Installed size: 494 k 2024-09-02T09:52:20.142 INFO:teuthology.orchestra.run.smithi162.stdout:Downloading Packages: 2024-09-02T09:52:20.182 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:20.182 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:52:20.182 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:20.182 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:20.182 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:20.201 INFO:teuthology.orchestra.run.smithi151.stdout: Verifying : rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:20.202 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:20.202 INFO:teuthology.orchestra.run.smithi151.stdout:Installed: 2024-09-02T09:52:20.202 INFO:teuthology.orchestra.run.smithi151.stdout: rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:20.202 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:52:20.202 INFO:teuthology.orchestra.run.smithi151.stdout:Complete! 2024-09-02T09:52:20.344 INFO:teuthology.orchestra.run.smithi162.stdout:rbd-nbd-19.1.1-246.g7ad9cc7a.el9.x86_64.rpm 842 kB/s | 171 kB 00:00 2024-09-02T09:52:20.345 INFO:teuthology.orchestra.run.smithi162.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:20.345 INFO:teuthology.orchestra.run.smithi162.stdout:Total 837 kB/s | 171 kB 00:00 2024-09-02T09:52:20.345 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction check 2024-09-02T09:52:20.353 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction check succeeded. 2024-09-02T09:52:20.353 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction test 2024-09-02T09:52:20.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs2 2024-09-02T09:52:20.413 INFO:teuthology.orchestra.run.smithi162.stdout:Transaction test succeeded. 2024-09-02T09:52:20.413 INFO:teuthology.orchestra.run.smithi162.stdout:Running transaction 2024-09-02T09:52:20.451 DEBUG:teuthology.parallel:result is None 2024-09-02T09:52:20.551 INFO:teuthology.orchestra.run.smithi162.stdout: Preparing : 1/1 2024-09-02T09:52:20.655 INFO:teuthology.orchestra.run.smithi162.stdout: Installing : rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:20.971 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:02 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:21.048 INFO:teuthology.orchestra.run.smithi162.stdout: Running scriptlet: rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:21.062 INFO:teuthology.orchestra.run.smithi187.stdout:Package libcephfs2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:21.105 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:21.106 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-02T09:52:21.106 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:21.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install libcephfs-devel 2024-09-02T09:52:21.302 INFO:teuthology.orchestra.run.smithi162.stdout: Verifying : rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:21.302 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:21.302 INFO:teuthology.orchestra.run.smithi162.stdout:Installed: 2024-09-02T09:52:21.302 INFO:teuthology.orchestra.run.smithi162.stdout: rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:21.302 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:52:21.302 INFO:teuthology.orchestra.run.smithi162.stdout:Complete! 2024-09-02T09:52:21.541 DEBUG:teuthology.parallel:result is None 2024-09-02T09:52:21.739 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:03 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:21.872 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:21.873 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:21.873 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-02T09:52:21.873 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:21.873 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:52:21.873 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 32 k 2024-09-02T09:52:21.874 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:21.874 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:52:21.874 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:21.874 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-02T09:52:21.874 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:21.874 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 32 k 2024-09-02T09:52:21.874 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 142 k 2024-09-02T09:52:21.874 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:52:22.020 INFO:teuthology.orchestra.run.smithi187.stdout:libcephfs-devel-19.1.1-246.g7ad9cc7a.el9.x86_64 218 kB/s | 32 kB 00:00 2024-09-02T09:52:22.021 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:22.021 INFO:teuthology.orchestra.run.smithi187.stdout:Total 215 kB/s | 32 kB 00:00 2024-09-02T09:52:22.021 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:52:22.024 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:52:22.024 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:52:22.029 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:52:22.029 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:52:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:52:22.221 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:22.532 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:22.739 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:22.739 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:22.740 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:52:22.740 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:22.740 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:22.740 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:22.909 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librados2 2024-09-02T09:52:23.452 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:05 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:23.541 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:23.583 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:23.584 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-02T09:52:23.584 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:23.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install librbd1 2024-09-02T09:52:24.198 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:06 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:24.287 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:24.329 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:24.330 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-02T09:52:24.331 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:24.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rados 2024-09-02T09:52:24.944 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:06 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:25.033 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rados-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:25.074 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:25.075 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-02T09:52:25.075 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:25.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rgw 2024-09-02T09:52:25.688 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:07 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:25.778 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rgw-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:25.819 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:25.820 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-02T09:52:25.820 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:25.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-cephfs 2024-09-02T09:52:26.441 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:08 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:26.539 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-cephfs-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:26.583 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:26.584 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-02T09:52:26.584 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:26.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install python3-rbd 2024-09-02T09:52:27.229 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:09 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:27.321 INFO:teuthology.orchestra.run.smithi187.stdout:Package python3-rbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 is already installed. 2024-09-02T09:52:27.365 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:27.366 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-09-02T09:52:27.366 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:27.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-fuse 2024-09-02T09:52:27.984 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:09 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:28.113 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:28.114 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:28.114 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-02T09:52:28.114 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:28.114 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:52:28.114 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 85 k 2024-09-02T09:52:28.114 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:28.114 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:52:28.115 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:28.115 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-02T09:52:28.115 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:28.115 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 85 k 2024-09-02T09:52:28.115 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 231 k 2024-09-02T09:52:28.115 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:52:28.333 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-fuse-19.1.1-246.g7ad9cc7a.el9.x86_64.rpm 394 kB/s | 85 kB 00:00 2024-09-02T09:52:28.334 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:28.334 INFO:teuthology.orchestra.run.smithi187.stdout:Total 390 kB/s | 85 kB 00:00 2024-09-02T09:52:28.334 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:52:28.341 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:52:28.341 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:52:28.397 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:52:28.397 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:52:28.530 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:52:28.634 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:28.983 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:29.231 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:29.232 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:29.232 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:52:29.232 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:29.232 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:29.232 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:29.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-mirror 2024-09-02T09:52:30.029 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:12 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:30.163 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:30.164 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:30.164 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repo Size 2024-09-02T09:52:30.164 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:30.164 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:52:30.164 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 3.1 M 2024-09-02T09:52:30.165 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:30.165 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:52:30.165 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:30.165 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-02T09:52:30.165 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:30.165 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 3.1 M 2024-09-02T09:52:30.165 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 13 M 2024-09-02T09:52:30.165 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:52:30.476 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-mirror-19.1.1-246.g7ad9cc7a.el9.x86_64.rpm 10 MB/s | 3.1 MB 00:00 2024-09-02T09:52:30.477 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:30.477 INFO:teuthology.orchestra.run.smithi187.stdout:Total 9.9 MB/s | 3.1 MB 00:00 2024-09-02T09:52:30.477 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:52:30.486 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:52:30.486 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:52:30.561 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:52:30.561 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:52:30.978 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:52:31.035 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:31.060 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:31.061 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-02T09:52:31.061 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-02T09:52:31.061 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T09:52:31.061 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-02T09:52:31.061 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:32.114 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:32.114 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:32.114 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:52:32.114 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:32.114 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:32.114 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:32.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install rbd-nbd 2024-09-02T09:52:32.897 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:01:14 ago on Mon 02 Sep 2024 09:51:18 AM UTC. 2024-09-02T09:52:33.034 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-09-02T09:52:33.035 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:33.035 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-09-02T09:52:33.035 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:33.035 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-09-02T09:52:33.035 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:19.1.1-246.g7ad9cc7a.el9 ceph 171 k 2024-09-02T09:52:33.035 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:33.036 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-09-02T09:52:33.036 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-09-02T09:52:33.036 INFO:teuthology.orchestra.run.smithi187.stdout:Install 1 Package 2024-09-02T09:52:33.036 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:33.037 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 171 k 2024-09-02T09:52:33.037 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 494 k 2024-09-02T09:52:33.037 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-09-02T09:52:33.240 INFO:teuthology.orchestra.run.smithi187.stdout:rbd-nbd-19.1.1-246.g7ad9cc7a.el9.x86_64.rpm 837 kB/s | 171 kB 00:00 2024-09-02T09:52:33.240 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:52:33.241 INFO:teuthology.orchestra.run.smithi187.stdout:Total 830 kB/s | 171 kB 00:00 2024-09-02T09:52:33.241 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-09-02T09:52:33.248 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-09-02T09:52:33.248 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-09-02T09:52:33.306 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-09-02T09:52:33.306 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-09-02T09:52:33.446 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-09-02T09:52:33.533 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:33.924 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 1/1 2024-09-02T09:52:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:34.175 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-09-02T09:52:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:19.1.1-246.g7ad9cc7a.el9.x86_64 2024-09-02T09:52:34.175 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:52:34.175 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-09-02T09:52:34.393 DEBUG:teuthology.parallel:result is None 2024-09-02T09:52:34.393 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-02T09:52:34.394 INFO:teuthology.packaging:ref: None 2024-09-02T09:52:34.394 INFO:teuthology.packaging:tag: None 2024-09-02T09:52:34.394 INFO:teuthology.packaging:branch: squid 2024-09-02T09:52:34.394 INFO:teuthology.packaging:sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:52:34.394 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-02T09:52:34.627 DEBUG:teuthology.orchestra.run.smithi151:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-02T09:52:34.653 INFO:teuthology.orchestra.run.smithi151.stdout:19.1.1-246.g7ad9cc7a.el9 2024-09-02T09:52:34.654 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-246.g7ad9cc7a.el9 2024-09-02T09:52:34.654 INFO:teuthology.task.install:The correct ceph version 19.1.1-246.g7ad9cc7a is installed. 2024-09-02T09:52:34.656 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-02T09:52:34.656 INFO:teuthology.packaging:ref: None 2024-09-02T09:52:34.656 INFO:teuthology.packaging:tag: None 2024-09-02T09:52:34.656 INFO:teuthology.packaging:branch: squid 2024-09-02T09:52:34.656 INFO:teuthology.packaging:sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:52:34.656 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-02T09:52:34.905 DEBUG:teuthology.orchestra.run.smithi162:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-02T09:52:34.931 INFO:teuthology.orchestra.run.smithi162.stdout:19.1.1-246.g7ad9cc7a.el9 2024-09-02T09:52:34.931 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-246.g7ad9cc7a.el9 2024-09-02T09:52:34.932 INFO:teuthology.task.install:The correct ceph version 19.1.1-246.g7ad9cc7a is installed. 2024-09-02T09:52:34.933 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-02T09:52:34.933 INFO:teuthology.packaging:ref: None 2024-09-02T09:52:34.934 INFO:teuthology.packaging:tag: None 2024-09-02T09:52:34.934 INFO:teuthology.packaging:branch: squid 2024-09-02T09:52:34.934 INFO:teuthology.packaging:sha1: 82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:52:34.934 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-02T09:52:35.174 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-02T09:52:35.199 INFO:teuthology.orchestra.run.smithi187.stdout:19.1.1-246.g7ad9cc7a.el9 2024-09-02T09:52:35.199 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-246.g7ad9cc7a.el9 2024-09-02T09:52:35.199 INFO:teuthology.task.install:The correct ceph version 19.1.1-246.g7ad9cc7a is installed. 2024-09-02T09:52:35.201 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-02T09:52:35.202 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:52:35.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-02T09:52:35.232 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:52:35.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-02T09:52:35.262 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:52:35.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-02T09:52:35.292 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-02T09:52:35.293 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:52:35.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/daemon-helper 2024-09-02T09:52:35.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-02T09:52:35.390 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:52:35.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/daemon-helper 2024-09-02T09:52:35.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-02T09:52:35.490 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:52:35.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2024-09-02T09:52:35.520 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-02T09:52:35.587 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-02T09:52:35.588 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:52:35.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-02T09:52:35.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-02T09:52:35.686 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:52:35.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-02T09:52:35.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-02T09:52:35.784 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:52:35.784 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-02T09:52:35.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-02T09:52:35.879 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-02T09:52:35.880 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:52:35.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/usr/bin/stdin-killer 2024-09-02T09:52:35.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-02T09:52:35.978 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:52:35.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/usr/bin/stdin-killer 2024-09-02T09:52:36.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-02T09:52:36.073 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:52:36.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2024-09-02T09:52:36.102 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-02T09:52:36.172 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-02T09:52:36.264 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07'} 2024-09-02T09:52:36.264 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:52:36.265 INFO:tasks.cephadm:Cluster fsid is 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:52:36.265 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-02T09:52:36.265 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.151', 'mon.b': '172.21.15.162', 'mon.c': '172.21.15.187'} 2024-09-02T09:52:36.265 INFO:tasks.cephadm:First mon is mon.a on smithi151 2024-09-02T09:52:36.266 INFO:tasks.cephadm:First mgr is y 2024-09-02T09:52:36.266 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-02T09:52:36.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo hostname $(hostname -s) 2024-09-02T09:52:36.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo hostname $(hostname -s) 2024-09-02T09:52:36.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo hostname $(hostname -s) 2024-09-02T09:52:36.353 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-02T09:52:36.353 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:52:36.508 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/82683/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4642-g82daa6ef', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4642.g82daa6ef'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-bharath3-testing-2024-09-01-1328/82daa6ef0815320144b25b212c6b8c25a1d59c07/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-01 09:53:49.012596', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-bharath3-testing-2024-09-01-1328', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-bharath3-testing-2024-09-01-1328/82daa6ef0815320144b25b212c6b8c25a1d59c07/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/82681/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-4642-g82daa6ef', 'node_name': '172.21.2.5+braggi05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-4642.g82daa6ef'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-bharath3-testing-2024-09-01-1328/82daa6ef0815320144b25b212c6b8c25a1d59c07/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-01 09:49:21.824644', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-bharath3-testing-2024-09-01-1328', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-bharath3-testing-2024-09-01-1328/82daa6ef0815320144b25b212c6b8c25a1d59c07/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-02T09:52:36.666 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-bharath3-testing-2024-09-01-1328, sha1 82daa6ef0815320144b25b212c6b8c25a1d59c07 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:52:36.667 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-bharath3-testing-2024-09-01-1328/82daa6ef0815320144b25b212c6b8c25a1d59c07/centos/9/x86_64/flavors/default/cephadm 2024-09-02T09:52:36.668 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/wip-bharath3-testing-2024-09-01-1328/82daa6ef0815320144b25b212c6b8c25a1d59c07/centos/9/x86_64/flavors/default/cephadm 2024-09-02T09:52:36.668 DEBUG:teuthology.orchestra.run.smithi151:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-bharath3-testing-2024-09-01-1328/82daa6ef0815320144b25b212c6b8c25a1d59c07/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-02T09:52:37.278 INFO:teuthology.orchestra.run.smithi151.stdout:-rw-r--r--. 1 ubuntu ubuntu 802712 Sep 2 09:52 /home/ubuntu/cephtest/cephadm 2024-09-02T09:52:37.279 DEBUG:teuthology.orchestra.run.smithi162:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-bharath3-testing-2024-09-01-1328/82daa6ef0815320144b25b212c6b8c25a1d59c07/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-02T09:52:37.856 INFO:teuthology.orchestra.run.smithi162.stdout:-rw-r--r--. 1 ubuntu ubuntu 802712 Sep 2 09:52 /home/ubuntu/cephtest/cephadm 2024-09-02T09:52:37.857 DEBUG:teuthology.orchestra.run.smithi187:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-bharath3-testing-2024-09-01-1328/82daa6ef0815320144b25b212c6b8c25a1d59c07/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-02T09:52:38.459 INFO:teuthology.orchestra.run.smithi187.stdout:-rw-r--r--. 1 ubuntu ubuntu 802712 Sep 2 09:52 /home/ubuntu/cephtest/cephadm 2024-09-02T09:52:38.459 DEBUG:teuthology.orchestra.run.smithi151:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-02T09:52:38.479 DEBUG:teuthology.orchestra.run.smithi162:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-02T09:52:38.499 DEBUG:teuthology.orchestra.run.smithi187:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-02T09:52:38.531 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 on all hosts... 2024-09-02T09:52:38.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 pull 2024-09-02T09:52:38.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 pull 2024-09-02T09:52:38.542 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 pull 2024-09-02T09:52:40.085 INFO:teuthology.orchestra.run.smithi151.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07... 2024-09-02T09:52:40.185 INFO:teuthology.orchestra.run.smithi187.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07... 2024-09-02T09:52:40.190 INFO:teuthology.orchestra.run.smithi162.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07... 2024-09-02T09:53:12.793 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T09:53:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "ceph_version": "ceph version 19.3.0-4642-g82daa6ef (82daa6ef0815320144b25b212c6b8c25a1d59c07) squid (dev)", 2024-09-02T09:53:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "image_id": "27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0", 2024-09-02T09:53:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "repo_digests": [ 2024-09-02T09:53:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4" 2024-09-02T09:53:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T09:53:12.794 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T09:53:26.508 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T09:53:26.509 INFO:teuthology.orchestra.run.smithi162.stdout: "ceph_version": "ceph version 19.3.0-4642-g82daa6ef (82daa6ef0815320144b25b212c6b8c25a1d59c07) squid (dev)", 2024-09-02T09:53:26.509 INFO:teuthology.orchestra.run.smithi162.stdout: "image_id": "27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0", 2024-09-02T09:53:26.509 INFO:teuthology.orchestra.run.smithi162.stdout: "repo_digests": [ 2024-09-02T09:53:26.509 INFO:teuthology.orchestra.run.smithi162.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4" 2024-09-02T09:53:26.509 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T09:53:26.509 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T09:53:32.587 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T09:53:32.587 INFO:teuthology.orchestra.run.smithi187.stdout: "ceph_version": "ceph version 19.3.0-4642-g82daa6ef (82daa6ef0815320144b25b212c6b8c25a1d59c07) squid (dev)", 2024-09-02T09:53:32.587 INFO:teuthology.orchestra.run.smithi187.stdout: "image_id": "27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0", 2024-09-02T09:53:32.588 INFO:teuthology.orchestra.run.smithi187.stdout: "repo_digests": [ 2024-09-02T09:53:32.588 INFO:teuthology.orchestra.run.smithi187.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4" 2024-09-02T09:53:32.588 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T09:53:32.588 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T09:53:32.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo mkdir -p /etc/ceph 2024-09-02T09:53:32.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo mkdir -p /etc/ceph 2024-09-02T09:53:32.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph 2024-09-02T09:53:32.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 777 /etc/ceph 2024-09-02T09:53:32.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 777 /etc/ceph 2024-09-02T09:53:32.768 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /etc/ceph 2024-09-02T09:53:32.793 INFO:tasks.cephadm:Writing seed config... 2024-09-02T09:53:32.794 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-09-02T09:53:32.794 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-02T09:53:32.794 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-02T09:53:32.794 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-02T09:53:32.794 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-02T09:53:32.794 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [client.2] debug ms = 1 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [client.2] debug objecter = 20 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [client.2] debug rados = 20 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [global] ms inject delay max = 1 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [global] ms inject delay probability = 0.005 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [global] ms inject delay type = osd 2024-09-02T09:53:32.795 INFO:tasks.cephadm: override: [global] ms inject internal delays = 0.002 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 3 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [osd] osd backoff on degraded = True 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [osd] osd backoff on peering = True 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-02T09:53:32.796 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-02T09:53:32.797 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-02T09:53:32.798 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:53:32.798 DEBUG:teuthology.orchestra.run.smithi151:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-02T09:53:32.815 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1530c7be-6911-11ef-bcd4-c7b262605968 mon client directed command retry = 5 mon election default strategy = 3 ms inject delay max = 1 ms inject delay probability = 0.005 ms inject delay type = osd ms inject internal delays = 0.002 ms inject socket failures = 2500 osd_pool_default_min_size = 2 osd_pool_default_size = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd backoff on degraded = True osd backoff on peering = True osd blocked scrub grace period = 3600 osd debug reject backfill probability = 0.3 osd max backfills = 3 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.2] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-02T09:53:32.815 DEBUG:teuthology.orchestra.run.smithi151:mon.a> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.a.service 2024-09-02T09:53:32.858 DEBUG:teuthology.orchestra.run.smithi151:mgr.y> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mgr.y.service 2024-09-02T09:53:32.902 INFO:tasks.cephadm:Bootstrapping... 2024-09-02T09:53:32.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 -v bootstrap --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.151 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-02T09:53:33.106 INFO:teuthology.orchestra.run.smithi151.stdout:-------------------------------------------------------------------------------- 2024-09-02T09:53:33.106 INFO:teuthology.orchestra.run.smithi151.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07', '-v', 'bootstrap', '--fsid', '1530c7be-6911-11ef-bcd4-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.151', '--skip-admin-label'] 2024-09-02T09:53:33.106 INFO:teuthology.orchestra.run.smithi151.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-02T09:53:33.107 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying podman|docker is present... 2024-09-02T09:53:33.131 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 5.2.2 2024-09-02T09:53:33.131 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying lvm2 is present... 2024-09-02T09:53:33.131 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying time synchronization is in place... 2024-09-02T09:53:33.138 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-02T09:53:33.138 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-02T09:53:33.144 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-02T09:53:33.144 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-02T09:53:33.151 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout enabled 2024-09-02T09:53:33.157 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout active 2024-09-02T09:53:33.157 INFO:teuthology.orchestra.run.smithi151.stdout:Unit chronyd.service is enabled and running 2024-09-02T09:53:33.158 INFO:teuthology.orchestra.run.smithi151.stdout:Repeating the final host check... 2024-09-02T09:53:33.181 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 5.2.2 2024-09-02T09:53:33.181 INFO:teuthology.orchestra.run.smithi151.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-02T09:53:33.181 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl is present 2024-09-02T09:53:33.181 INFO:teuthology.orchestra.run.smithi151.stdout:lvcreate is present 2024-09-02T09:53:33.187 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-02T09:53:33.187 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-02T09:53:33.193 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-02T09:53:33.193 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-02T09:53:33.200 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout enabled 2024-09-02T09:53:33.206 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout active 2024-09-02T09:53:33.206 INFO:teuthology.orchestra.run.smithi151.stdout:Unit chronyd.service is enabled and running 2024-09-02T09:53:33.207 INFO:teuthology.orchestra.run.smithi151.stdout:Host looks OK 2024-09-02T09:53:33.207 INFO:teuthology.orchestra.run.smithi151.stdout:Cluster fsid: 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:53:33.207 INFO:teuthology.orchestra.run.smithi151.stdout:Acquiring lock 140482572364864 on /run/cephadm/1530c7be-6911-11ef-bcd4-c7b262605968.lock 2024-09-02T09:53:33.207 INFO:teuthology.orchestra.run.smithi151.stdout:Lock 140482572364864 acquired on /run/cephadm/1530c7be-6911-11ef-bcd4-c7b262605968.lock 2024-09-02T09:53:33.207 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying IP 172.21.15.151 port 3300 ... 2024-09-02T09:53:33.207 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying IP 172.21.15.151 port 6789 ... 2024-09-02T09:53:33.208 INFO:teuthology.orchestra.run.smithi151.stdout:Base mon IP(s) is [172.21.15.151:3300, 172.21.15.151:6789], mon addrv is [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-09-02T09:53:33.211 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.151 metric 100 2024-09-02T09:53:33.211 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.151 metric 100 2024-09-02T09:53:33.213 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-02T09:53:33.214 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-02T09:53:33.214 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-02T09:53:33.216 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-02T09:53:33.216 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-02T09:53:33.216 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-02T09:53:33.216 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-02T09:53:33.216 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6f63/64 scope link noprefixroute 2024-09-02T09:53:33.216 INFO:teuthology.orchestra.run.smithi151.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-02T09:53:33.217 INFO:teuthology.orchestra.run.smithi151.stdout:Mon IP `172.21.15.151` is in CIDR network `172.21.0.0/20` 2024-09-02T09:53:33.217 INFO:teuthology.orchestra.run.smithi151.stdout:Mon IP `172.21.15.151` is in CIDR network `172.21.0.0/20` 2024-09-02T09:53:33.217 INFO:teuthology.orchestra.run.smithi151.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-02T09:53:33.217 INFO:teuthology.orchestra.run.smithi151.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-02T09:53:33.218 INFO:teuthology.orchestra.run.smithi151.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07... 2024-09-02T09:53:33.659 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stdout 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 2024-09-02T09:53:33.659 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07... 2024-09-02T09:53:33.659 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Getting image source signatures 2024-09-02T09:53:33.660 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-02T09:53:33.660 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying blob sha256:f5c1a2ea38b758d3642954d35878eb76c43528d9d788bb3d049a8f82782f70e2 2024-09-02T09:53:33.660 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Copying config sha256:27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 2024-09-02T09:53:33.660 INFO:teuthology.orchestra.run.smithi151.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-02T09:53:34.639 INFO:teuthology.orchestra.run.smithi151.stdout:ceph: stdout ceph version 19.3.0-4642-g82daa6ef (82daa6ef0815320144b25b212c6b8c25a1d59c07) squid (dev) 2024-09-02T09:53:34.639 INFO:teuthology.orchestra.run.smithi151.stdout:Ceph version: ceph version 19.3.0-4642-g82daa6ef (82daa6ef0815320144b25b212c6b8c25a1d59c07) squid (dev) 2024-09-02T09:53:34.639 INFO:teuthology.orchestra.run.smithi151.stdout:Extracting ceph user uid/gid from container image... 2024-09-02T09:53:35.636 INFO:teuthology.orchestra.run.smithi151.stdout:stat: stdout 167 167 2024-09-02T09:53:35.636 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial keys... 2024-09-02T09:53:36.700 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQAfi9VmPPUnORAANg0/GFlDzk/COPFRb4SSYA== 2024-09-02T09:53:46.445 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQAji9VmQdM9KhAAo/WmVljVO1YE03t/Roy75g== 2024-09-02T09:53:49.786 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph-authtool: stdout AQAti9VmpczeCRAAZeR7DHdp10JU+lcECDpI7A== 2024-09-02T09:53:49.787 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial monmap... 2024-09-02T09:53:50.709 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-02T09:53:50.709 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-02T09:53:50.709 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:53:50.709 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-02T09:53:50.709 INFO:teuthology.orchestra.run.smithi151.stdout:monmaptool for a [v2:172.21.15.151:3300,v1:172.21.15.151:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-02T09:53:50.709 INFO:teuthology.orchestra.run.smithi151.stdout:setting min_mon_release = quincy 2024-09-02T09:53:50.710 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: set fsid to 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:53:50.710 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-02T09:53:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:53:50.710 INFO:teuthology.orchestra.run.smithi151.stdout:Creating mon... 2024-09-02T09:53:51.889 INFO:teuthology.orchestra.run.smithi151.stdout:create mon.a on 2024-09-02T09:53:52.101 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-02T09:53:52.239 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-02T09:53:52.420 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1530c7be-6911-11ef-bcd4-c7b262605968.target → /etc/systemd/system/ceph-1530c7be-6911-11ef-bcd4-c7b262605968.target. 2024-09-02T09:53:52.420 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1530c7be-6911-11ef-bcd4-c7b262605968.target → /etc/systemd/system/ceph-1530c7be-6911-11ef-bcd4-c7b262605968.target. 2024-09-02T09:53:52.644 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.a 2024-09-02T09:53:52.644 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.a.service: Unit ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.a.service not loaded. 2024-09-02T09:53:52.809 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1530c7be-6911-11ef-bcd4-c7b262605968.target.wants/ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.a.service → /etc/systemd/system/ceph-1530c7be-6911-11ef-bcd4-c7b262605968@.service. 2024-09-02T09:53:53.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:53 smithi151 podman[16249]: 2024-09-02 09:53:53.268993331 +0000 UTC m=+0.341318305 container start 6a62cca77c5b6fc4d7f0b59767a0ef561161677f5a5257542f08e82e815b4af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T09:53:53.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:53 smithi151 ceph-mon[16268]: mkfs 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:53:53.363 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-02T09:53:53.363 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-09-02T09:53:53.370 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-02T09:53:53.371 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-02T09:53:53.371 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-09-02T09:53:53.371 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to enable service . firewalld.service is not available 2024-09-02T09:53:53.371 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mon to start... 2024-09-02T09:53:53.371 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mon... 2024-09-02T09:53:53.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:53 smithi151 ceph-mon[16268]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-02T09:53:53.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:53 smithi151 bash[16249]: 6a62cca77c5b6fc4d7f0b59767a0ef561161677f5a5257542f08e82e815b4af1 2024-09-02T09:53:53.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:53 smithi151 systemd[1]: Started Ceph mon.a for 1530c7be-6911-11ef-bcd4-c7b262605968. 2024-09-02T09:53:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-02T09:53:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: monmap epoch 1 2024-09-02T09:53:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:53:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: last_changed 2024-09-02T09:53:50.141532+0000 2024-09-02T09:53:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: created 2024-09-02T09:53:50.141532+0000 2024-09-02T09:53:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: min_mon_release 19 (squid) 2024-09-02T09:53:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: election_strategy: 1 2024-09-02T09:53:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-02T09:53:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: fsmap 2024-09-02T09:53:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: osdmap e1: 0 total, 0 up, 0 in 2024-09-02T09:53:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: mgrmap e1: no daemons active 2024-09-02T09:53:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:54 smithi151 ceph-mon[16268]: from='client.? 172.21.15.151:0/1198090300' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-09-02T09:53:54.977 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout cluster: 2024-09-02T09:53:54.977 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout id: 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:53:54.977 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-02T09:53:54.977 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout services: 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.909445s) 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout data: 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout pgs: 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:mon is available 2024-09-02T09:53:54.978 INFO:teuthology.orchestra.run.smithi151.stdout:Assimilating anything we can from ceph.conf... 2024-09-02T09:53:55.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:55 smithi151 ceph-mon[16268]: from='client.? 172.21.15.151:0/4019373073' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-02T09:53:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:55 smithi151 ceph-mon[16268]: from='client.? 172.21.15.151:0/4019373073' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-02T09:53:56.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:53:56.018 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [global] 2024-09-02T09:53:56.018 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout fsid = 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:53:56.018 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-02T09:53:56.018 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-09-02T09:53:56.019 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-02T09:53:56.019 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-02T09:53:56.019 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-02T09:53:56.019 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-02T09:53:56.019 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:53:56.019 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-02T09:53:56.020 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-02T09:53:56.020 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:53:56.020 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [osd] 2024-09-02T09:53:56.020 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-02T09:53:56.020 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-02T09:53:56.021 INFO:teuthology.orchestra.run.smithi151.stdout:Generating new minimal ceph.conf... 2024-09-02T09:53:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:56 smithi151 ceph-mon[16268]: from='client.? 172.21.15.151:0/433126334' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:53:57.062 INFO:teuthology.orchestra.run.smithi151.stdout:Restarting the monitor... 2024-09-02T09:53:57.329 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:57 smithi151 systemd[1]: Stopping Ceph mon.a for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T09:53:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:57 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16264]: 2024-09-02T09:53:57.327+0000 7f6c89c32640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T09:53:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:57 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16264]: 2024-09-02T09:53:57.327+0000 7f6c89c32640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-02T09:53:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:57 smithi151 podman[16512]: 2024-09-02 09:53:57.53309158 +0000 UTC m=+0.312669586 container died 6a62cca77c5b6fc4d7f0b59767a0ef561161677f5a5257542f08e82e815b4af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS) 2024-09-02T09:53:58.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:57 smithi151 podman[16512]: 2024-09-02 09:53:57.811346597 +0000 UTC m=+0.590924651 container cleanup 6a62cca77c5b6fc4d7f0b59767a0ef561161677f5a5257542f08e82e815b4af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-02T09:53:58.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:57 smithi151 bash[16512]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a 2024-09-02T09:53:58.162 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:57 smithi151 podman[16527]: 2024-09-02 09:53:57.953550452 +0000 UTC m=+0.415714685 container remove 6a62cca77c5b6fc4d7f0b59767a0ef561161677f5a5257542f08e82e815b4af1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-02T09:53:58.520 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 systemd[1]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-02T09:53:58.520 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 systemd[1]: Stopped Ceph mon.a for 1530c7be-6911-11ef-bcd4-c7b262605968. 2024-09-02T09:53:58.520 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 systemd[1]: Starting Ceph mon.a for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T09:53:58.520 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 podman[16620]: 2024-09-02 09:53:58.301256882 +0000 UTC m=+0.015892642 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:53:58.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 podman[16620]: 2024-09-02 09:53:58.42142636 +0000 UTC m=+0.136062108 container create 51c3f66d7ab549ac4fd81bf0c90868257642b481b93cc6a839b50fd5ef4e297e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-09-02T09:53:58.695 INFO:teuthology.orchestra.run.smithi151.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-02T09:53:58.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 podman[16620]: 2024-09-02 09:53:58.615220887 +0000 UTC m=+0.329856633 container init 51c3f66d7ab549ac4fd81bf0c90868257642b481b93cc6a839b50fd5ef4e297e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a, GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-02T09:53:58.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 podman[16620]: 2024-09-02 09:53:58.618242691 +0000 UTC m=+0.332878440 container start 51c3f66d7ab549ac4fd81bf0c90868257642b481b93cc6a839b50fd5ef4e297e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T09:53:58.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: set uid:gid to 167:167 (ceph:ceph) 2024-09-02T09:53:58.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: ceph version 19.3.0-4642-g82daa6ef (82daa6ef0815320144b25b212c6b8c25a1d59c07) squid (dev), process ceph-mon, pid 2 2024-09-02T09:53:58.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: pidfile_write: ignore empty --pid-file 2024-09-02T09:53:58.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: load: jerasure load: lrc 2024-09-02T09:53:58.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: RocksDB version: 7.9.2 2024-09-02T09:53:58.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Git sha 0 2024-09-02T09:53:58.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Compile date 2024-09-01 08:27:02 2024-09-02T09:53:58.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: DB SUMMARY 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: DB Session ID: BE5IHQJOH9NZ36C5UYG5 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: CURRENT file: CURRENT 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: IDENTITY file: IDENTITY 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 102314 ; 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.error_if_exists: 0 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.create_if_missing: 0 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.paranoid_checks: 1 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-02T09:53:58.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.env: 0x5634f05f1160 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.fs: PosixFileSystem 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.info_log: 0x5634f1a6f860 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_file_opening_threads: 16 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.statistics: (nil) 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.use_fsync: 0 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_log_file_size: 0 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.keep_log_file_num: 1000 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.recycle_log_file_num: 0 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.allow_fallocate: 1 2024-09-02T09:53:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.allow_mmap_reads: 0 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.allow_mmap_writes: 0 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.use_direct_reads: 0 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.create_missing_column_families: 0 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.db_log_dir: 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.wal_dir: 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-02T09:53:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.advise_random_on_open: 1 2024-09-02T09:53:58.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.db_write_buffer_size: 0 2024-09-02T09:53:58.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.write_buffer_manager: 0x5634f1a72e60 2024-09-02T09:53:58.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-02T09:53:58.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-02T09:53:58.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-02T09:53:58.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.rate_limiter: (nil) 2024-09-02T09:53:58.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-02T09:53:58.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.wal_recovery_mode: 2 2024-09-02T09:53:58.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.enable_thread_tracking: 0 2024-09-02T09:53:58.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.enable_pipelined_write: 0 2024-09-02T09:53:58.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.unordered_write: 0 2024-09-02T09:53:58.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-02T09:53:58.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-02T09:53:58.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-02T09:53:58.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-02T09:53:58.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.row_cache: None 2024-09-02T09:53:58.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.wal_filter: None 2024-09-02T09:53:58.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-02T09:53:58.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.allow_ingest_behind: 0 2024-09-02T09:53:58.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.two_write_queues: 0 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.manual_wal_flush: 0 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.wal_compression: 0 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.atomic_flush: 0 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.log_readahead_size: 0 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.best_efforts_recovery: 0 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-02T09:53:58.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.allow_data_in_errors: 0 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.db_host_id: __hostname__ 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_background_jobs: 2 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_background_compactions: -1 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_subcompactions: 1 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_total_wal_size: 0 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-02T09:53:58.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_open_files: -1 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bytes_per_sync: 0 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_readahead_size: 0 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_background_flushes: -1 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Compression algorithms supported: 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: kZSTD supported: 0 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: kXpressCompression supported: 0 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: kBZip2Compression supported: 0 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: kLZ4Compression supported: 1 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: kZlibCompression supported: 1 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: kLZ4HCCompression supported: 1 2024-09-02T09:53:58.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: kSnappyCompression supported: 1 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.merge_operator: 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_filter: None 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_filter_factory: None 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.sst_partitioner_factory: None 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5634f1a6f4a0) 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout: cache_index_and_filter_blocks: 1 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-02T09:53:58.945 INFO:journalctl@ceph.mon.a.smithi151.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: pin_top_level_index_and_filter: 1 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_type: 0 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_block_index_type: 0 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_shortening: 1 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: checksum: 4 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: no_block_cache: 0 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache: 0x5634f1a63610 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_name: BinnedLRUCache 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_options: 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: capacity : 536870912 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: num_shard_bits : 4 2024-09-02T09:53:58.946 INFO:journalctl@ceph.mon.a.smithi151.stdout: strict_capacity_limit : 0 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: high_pri_pool_ratio: 0.000 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_cache_compressed: (nil) 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: persistent_cache: (nil) 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_size: 4096 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_size_deviation: 10 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_restart_interval: 16 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: index_block_restart_interval: 1 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: metadata_block_size: 4096 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: partition_filters: 0 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: use_delta_encoding: 1 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: filter_policy: bloomfilter 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: whole_key_filtering: 1 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: verify_compression: 0 2024-09-02T09:53:58.947 INFO:journalctl@ceph.mon.a.smithi151.stdout: read_amp_bytes_per_bit: 0 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout: format_version: 5 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout: enable_index_compression: 1 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout: block_align: 0 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout: max_auto_readahead_size: 262144 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout: prepopulate_block_cache: 0 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout: initial_auto_readahead_size: 8192 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout: num_file_reads_for_auto_readahead: 2 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.write_buffer_size: 33554432 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_write_buffer_number: 2 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compression: NoCompression 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bottommost_compression: Disabled 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.prefix_extractor: nullptr 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.num_levels: 7 2024-09-02T09:53:58.948 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-02T09:53:58.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compression_opts.level: 32767 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compression_opts.strategy: 0 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compression_opts.enabled: false 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.target_file_size_base: 67108864 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-02T09:53:58.950 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-02T09:53:58.951 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.arena_block_size: 1048576 2024-09-02T09:53:58.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-02T09:53:58.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-02T09:53:58.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.disable_auto_compactions: 0 2024-09-02T09:53:58.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-02T09:53:58.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-02T09:53:58.952 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.inplace_update_support: 0 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-02T09:53:58.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.bloom_locality: 0 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.max_successive_merges: 0 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.paranoid_file_checks: 0 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.force_consistency_checks: 1 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.report_bg_io_stats: 0 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.ttl: 2592000 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.enable_blob_files: false 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.min_blob_size: 0 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.blob_file_size: 268435456 2024-09-02T09:53:58.954 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.blob_file_starting_level: 0 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5b23cb27-0256-4b26-b786-b410c626100f 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725270838644724, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725270838645752, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 97561, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 95544, "index_size": 273, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13537, "raw_average_key_size": 50, "raw_value_size": 88202, "raw_average_value_size": 327, "num_data_blocks": 12, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1725270838, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5b23cb27-0256-4b26-b786-b410c626100f", "db_session_id": "BE5IHQJOH9NZ36C5UYG5", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-02T09:53:58.955 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725270838645848, "job": 1, "event": "recovery_finished"} 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5634f1a92e00 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: DB pointer 0x5634f1b92000 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** DB Stats ** 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-02T09:53:58.956 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: L0 2/0 97.13 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: Sum 2/0 97.13 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** Compaction Stats [default] ** 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 135.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-02T09:53:58.957 INFO:journalctl@ceph.mon.a.smithi151.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: Cumulative compaction: 0.00 GB write, 29.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: Interval compaction: 0.00 GB write, 29.26 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: Block cache BinnedLRUCache@0x5634f1a63610#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-02T09:53:58.958 INFO:journalctl@ceph.mon.a.smithi151.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: starting mon.a rank 0 at public addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] at bind addrs [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: mon.a@-1(???) e1 preinit fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: mon.a@-1(???).mds e1 new map 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: mon.a@-1(???).mds e1 print_map 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout: e1 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout: btime 2024-09-02T09:53:53:306254+0000 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout: legacy client fscid: -1 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout: 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout: No filesystems configured 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T09:53:58.959 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 bash[16620]: 51c3f66d7ab549ac4fd81bf0c90868257642b481b93cc6a839b50fd5ef4e297e 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 systemd[1]: Started Ceph mon.a for 1530c7be-6911-11ef-bcd4-c7b262605968. 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: monmap epoch 1 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: last_changed 2024-09-02T09:53:50.141532+0000 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: created 2024-09-02T09:53:50.141532+0000 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: min_mon_release 19 (squid) 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: election_strategy: 1 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: fsmap 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: osdmap e1: 0 total, 0 up, 0 in 2024-09-02T09:53:58.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:53:58 smithi151 ceph-mon[16639]: mgrmap e1: no daemons active 2024-09-02T09:53:59.774 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-02T09:53:59.796 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-02T09:53:59.796 INFO:teuthology.orchestra.run.smithi151.stdout:Creating mgr... 2024-09-02T09:53:59.796 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-02T09:53:59.797 INFO:teuthology.orchestra.run.smithi151.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-02T09:54:00.047 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mgr.y 2024-09-02T09:54:00.047 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mgr.y.service: Unit ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mgr.y.service not loaded. 2024-09-02T09:54:00.197 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1530c7be-6911-11ef-bcd4-c7b262605968.target.wants/ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-1530c7be-6911-11ef-bcd4-c7b262605968@.service. 2024-09-02T09:54:00.428 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2894244246' entity='client.admin' 2024-09-02T09:54:00.428 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:00 smithi151 systemd[1]: Starting Ceph mgr.y for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T09:54:00.688 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:00 smithi151 podman[16848]: 2024-09-02 09:54:00.329645106 +0000 UTC m=+0.015832348 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 2024-09-02T09:54:00.688 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:00 smithi151 podman[16848]: 2024-09-02 09:54:00.439744075 +0000 UTC m=+0.125931350 container create af37bedd602803253f07228280565f8fa40ee1b0781feac509fd2a9e874c976c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-02T09:54:00.688 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:00 smithi151 podman[16848]: 2024-09-02 09:54:00.641885167 +0000 UTC m=+0.328072414 container init af37bedd602803253f07228280565f8fa40ee1b0781feac509fd2a9e874c976c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T09:54:00.688 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:00 smithi151 podman[16848]: 2024-09-02 09:54:00.645065349 +0000 UTC m=+0.331252582 container start af37bedd602803253f07228280565f8fa40ee1b0781feac509fd2a9e874c976c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-02T09:54:00.731 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-02T09:54:00.731 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-09-02T09:54:00.738 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-02T09:54:00.738 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-02T09:54:00.738 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-09-02T09:54:00.738 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to enable service . firewalld.service is not available 2024-09-02T09:54:00.747 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-02T09:54:00.747 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-09-02T09:54:00.754 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-02T09:54:00.755 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-02T09:54:00.755 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-09-02T09:54:00.755 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-02T09:54:00.755 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr to start... 2024-09-02T09:54:00.755 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr... 2024-09-02T09:54:00.984 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:00 smithi151 bash[16848]: af37bedd602803253f07228280565f8fa40ee1b0781feac509fd2a9e874c976c 2024-09-02T09:54:00.984 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:00 smithi151 systemd[1]: Started Ceph mgr.y for 1530c7be-6911-11ef-bcd4-c7b262605968. 2024-09-02T09:54:01.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1790530555' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-02T09:54:01.870 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:54:01.870 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-02T09:54:01.870 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-02T09:54:01.871 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:01.872 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:01.876 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "btime": "2024-09-02T09:53:53:306254+0000", 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:01.877 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-02T09:54:01.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:01.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-09-02T09:53:53.308883+0000", 2024-09-02T09:54:01.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T09:54:01.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:01.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-02T09:54:01.878 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-02T09:54:01.878 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (1/15)... 2024-09-02T09:54:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:04 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1324569647' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-02T09:54:05.010 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:54:05.010 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-02T09:54:05.010 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T09:54:05.010 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-09-02T09:54:05.010 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-02T09:54:05.010 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-02T09:54:05.010 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-02T09:54:05.010 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:05.010 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-02T09:54:05.010 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:05.011 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-02T09:54:05.012 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-02T09:54:05.012 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-02T09:54:05.012 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-02T09:54:05.012 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-02T09:54:05.012 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-02T09:54:05.015 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:05.015 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-02T09:54:05.015 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-02T09:54:05.015 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-02T09:54:05.015 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-02T09:54:05.015 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "btime": "2024-09-02T09:53:53:306254+0000", 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-02T09:54:05.016 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-09-02T09:53:53.308883+0000", 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-02T09:54:05.017 INFO:teuthology.orchestra.run.smithi151.stdout:mgr not available, waiting (2/15)... 2024-09-02T09:54:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: Activating manager daemon y 2024-09-02T09:54:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: mgrmap e2: y(active, starting, since 0.00310487s) 2024-09-02T09:54:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: from='mgr.14100 172.21.15.151:0/1599914058' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-02T09:54:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: from='mgr.14100 172.21.15.151:0/1599914058' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-02T09:54:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: from='mgr.14100 172.21.15.151:0/1599914058' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-02T09:54:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: from='mgr.14100 172.21.15.151:0/1599914058' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:54:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: from='mgr.14100 172.21.15.151:0/1599914058' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-02T09:54:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: Manager daemon y is now available 2024-09-02T09:54:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: from='mgr.14100 172.21.15.151:0/1599914058' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-02T09:54:06.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: from='mgr.14100 172.21.15.151:0/1599914058' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-02T09:54:06.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: from='mgr.14100 172.21.15.151:0/1599914058' entity='mgr.y' 2024-09-02T09:54:06.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: from='mgr.14100 172.21.15.151:0/1599914058' entity='mgr.y' 2024-09-02T09:54:06.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:05 smithi151 ceph-mon[16639]: from='mgr.14100 172.21.15.151:0/1599914058' entity='mgr.y' 2024-09-02T09:54:08.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:07 smithi151 ceph-mon[16639]: mgrmap e3: y(active, since 1.00708s) 2024-09-02T09:54:08.398 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:54:08.398 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-02T09:54:08.398 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T09:54:08.398 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "health": { 2024-09-02T09:54:08.398 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-02T09:54:08.398 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-02T09:54:08.398 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-02T09:54:08.398 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 0 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "a" 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-02T09:54:08.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:08.400 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-02T09:54:08.400 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-02T09:54:08.400 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-02T09:54:08.400 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-02T09:54:08.400 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-02T09:54:08.400 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-02T09:54:08.403 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:08.403 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-02T09:54:08.403 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "btime": "2024-09-02T09:53:53:306254+0000", 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-02T09:54:08.404 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-02T09:54:08.405 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:08.405 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-02T09:54:08.405 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-02T09:54:08.405 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-02T09:54:08.405 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-02T09:54:08.405 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-02T09:54:08.405 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-02T09:54:08.405 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "restful" 2024-09-02T09:54:08.406 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ], 2024-09-02T09:54:08.406 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T09:54:08.406 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:08.406 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-02T09:54:08.406 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-02T09:54:08.406 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "modified": "2024-09-02T09:53:53.308883+0000", 2024-09-02T09:54:08.406 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-02T09:54:08.406 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout }, 2024-09-02T09:54:08.406 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-02T09:54:08.407 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-02T09:54:08.407 INFO:teuthology.orchestra.run.smithi151.stdout:mgr is available 2024-09-02T09:54:08.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:08 smithi151 ceph-mon[16639]: mgrmap e4: y(active, since 2s) 2024-09-02T09:54:08.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2790004440' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-09-02T09:54:09.645 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [global] 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout fsid = 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.151:3300,v1:172.21.15.151:6789] 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 2024-09-02T09:54:09.646 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout [osd] 2024-09-02T09:54:09.647 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-02T09:54:09.647 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-02T09:54:09.647 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling cephadm module... 2024-09-02T09:54:09.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:09 smithi151 ceph-mon[16639]: from='client.14115 172.21.15.151:0/2568286722' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-09-02T09:54:11.030 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4138271911' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-09-02T09:54:11.031 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:10 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: ignoring --setuser ceph since I am not root 2024-09-02T09:54:11.031 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:10 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: ignoring --setgroup ceph since I am not root 2024-09-02T09:54:11.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4138271911' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-02T09:54:11.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:11 smithi151 ceph-mon[16639]: mgrmap e5: y(active, since 5s) 2024-09-02T09:54:12.592 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-02T09:54:12.592 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-02T09:54:12.592 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-02T09:54:12.593 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-02T09:54:12.593 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-02T09:54:12.593 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-02T09:54:12.593 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for the mgr to restart... 2024-09-02T09:54:12.593 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr epoch 5... 2024-09-02T09:54:12.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/678372863' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-02T09:54:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:15 smithi151 ceph-mon[16639]: Active manager daemon y restarted 2024-09-02T09:54:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:15 smithi151 ceph-mon[16639]: Activating manager daemon y 2024-09-02T09:54:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:15 smithi151 ceph-mon[16639]: osdmap e2: 0 total, 0 up, 0 in 2024-09-02T09:54:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:15 smithi151 ceph-mon[16639]: mgrmap e6: y(active, starting, since 0.00348497s) 2024-09-02T09:54:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:15 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:54:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:15 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-02T09:54:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:15 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-02T09:54:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:15 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-02T09:54:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:15 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-02T09:54:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:15 smithi151 ceph-mon[16639]: Manager daemon y is now available 2024-09-02T09:54:17.237 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-02T09:54:17.238 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-02T09:54:17.238 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-02T09:54:17.238 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-02T09:54:17.238 INFO:teuthology.orchestra.run.smithi151.stdout:mgr epoch 5 is available 2024-09-02T09:54:17.238 INFO:teuthology.orchestra.run.smithi151.stdout:Setting orchestrator backend to cephadm... 2024-09-02T09:54:17.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: Found migration_current of "None". Setting to last migration. 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: mgrmap e7: y(active, since 1.00767s) 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='client.14123 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='client.14123 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-02T09:54:17.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:54:17.354 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: 2024-09-02T09:54:17.071+0000 7f10c5e03640 -1 log_channel(cephadm) log [ERR] : [02/Sep/2024:09:54:17] ENGINE Error in HTTPServer.serve 2024-09-02T09:54:17.354 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: Traceback (most recent call last): 2024-09-02T09:54:17.354 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-02T09:54:17.354 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: self._connections.run(self.expiration_interval) 2024-09-02T09:54:17.354 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-02T09:54:17.354 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: self._run(expiration_interval) 2024-09-02T09:54:17.354 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-02T09:54:17.354 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: new_conn = self._from_server_socket(self.server.socket) 2024-09-02T09:54:17.354 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-02T09:54:17.354 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-02T09:54:17.355 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-02T09:54:17.355 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: s = self.context.wrap_socket( 2024-09-02T09:54:17.355 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-02T09:54:17.355 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: return self.sslsocket_class._create( 2024-09-02T09:54:17.355 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-02T09:54:17.355 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: self.do_handshake() 2024-09-02T09:54:17.355 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-02T09:54:17.355 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: self._sslobj.do_handshake() 2024-09-02T09:54:17.355 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-02T09:54:17.355 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: 2024-09-02T09:54:18.432 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:18 smithi151 ceph-mon[16639]: [02/Sep/2024:09:54:16] ENGINE Bus STARTING 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:18 smithi151 ceph-mon[16639]: [02/Sep/2024:09:54:16] ENGINE Serving on http://172.21.15.151:8765 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:18 smithi151 ceph-mon[16639]: [02/Sep/2024:09:54:17] ENGINE Serving on https://172.21.15.151:7150 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:18 smithi151 ceph-mon[16639]: [02/Sep/2024:09:54:17] ENGINE Bus STARTED 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:18 smithi151 ceph-mon[16639]: [02/Sep/2024:09:54:17] ENGINE Error in HTTPServer.serve 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: Traceback (most recent call last): 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._run(expiration_interval) 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-02T09:54:18.433 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout: s = self.context.wrap_socket( 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout: return self.sslsocket_class._create( 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout: self.do_handshake() 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._sslobj.do_handshake() 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:18 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:18.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:18 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:54:19.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:19 smithi151 ceph-mon[16639]: from='client.14131 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:19.315 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:19 smithi151 ceph-mon[16639]: mgrmap e8: y(active, since 2s) 2024-09-02T09:54:19.568 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-02T09:54:19.568 INFO:teuthology.orchestra.run.smithi151.stdout:Generating ssh key... 2024-09-02T09:54:20.564 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-mon[16639]: from='client.14133 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: Generating public/private rsa key pair. 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: Your identification has been saved in /tmp/tmpx8jc31gw/key 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: Your public key has been saved in /tmp/tmpx8jc31gw/key.pub 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: The key fingerprint is: 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: SHA256:sJpy1byIasvjm+uYmUAmZk9O+gxsTVnOsAWkUPBih8k ceph-1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: The key's randomart image is: 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: +---[RSA 3072]----+ 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: |oooo | 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: |o.+ . | 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: |.E.o o. | 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: |... O = | 2024-09-02T09:54:20.565 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: |.= * oo S | 2024-09-02T09:54:20.566 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: |B O = . . | 2024-09-02T09:54:20.566 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: |.=.+= . . | 2024-09-02T09:54:20.566 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: |o**= | 2024-09-02T09:54:20.566 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: |=+XB | 2024-09-02T09:54:20.566 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: +----[SHA256]-----+ 2024-09-02T09:54:21.432 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:21 smithi151 ceph-mon[16639]: from='client.14135 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:21.432 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:21 smithi151 ceph-mon[16639]: Generating ssh key... 2024-09-02T09:54:21.432 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:21 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:21.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:21 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:22.031 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxwQNwpxhAXRIDj1z3XI6B2vaA94lB59/WwruLsgqf3CCudy/WT9mfHS7P8GPNQf1eICN7MmsZM8qYdEneIFXkw254G7kkBmcof0hThCKlyReI530+g+sm6jtb13MljQmk4/g8DPlUpw59GC+uq4xolG99OKhXpZQlnG5Krk9Am76U2RtdjHBzxt0XaQY+AyOsj6jEKBtETFnz9P7Cd9Q6q0eXXRtYd3h3jCoAFcVD6LSEhgcZ8hu6YL0ApKcfC6V0P3me2NDgqBxZzspJuO/U1bgwg0+OZalmkyFXpPA5VZpfCUT9jAIzSzrPxw7q99hiQDjV5TkRck1B5FSn3iyKQMAmnWhNataVpUuM2HAPfbI+YfbsuEEWi7lVczS2qobjvrGWY11F3dFxTKvCY+d//0R4XJcuwT/XoksZKXJCKHC+NosaGB9JWr9BHShp0G56c/8cZFX3IFq7MnODu8Nm5mzKNkEWkGf2nQgfu9rc/pK85J7BArLxsK4/KqqIKbc= ceph-1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:54:22.032 INFO:teuthology.orchestra.run.smithi151.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-02T09:54:22.032 INFO:teuthology.orchestra.run.smithi151.stdout:Adding key to root@localhost authorized_keys... 2024-09-02T09:54:22.032 INFO:teuthology.orchestra.run.smithi151.stdout:Adding host smithi151... 2024-09-02T09:54:22.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:22 smithi151 ceph-mon[16639]: from='client.14137 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:23.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:23 smithi151 ceph-mon[16639]: from='client.14139 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi151", "addr": "172.21.15.151", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:24.313 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:24 smithi151 ceph-mon[16639]: Deploying cephadm binary to smithi151 2024-09-02T09:54:26.399 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Added host 'smithi151' with addr '172.21.15.151' 2024-09-02T09:54:26.399 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying unmanaged mon service... 2024-09-02T09:54:27.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:26 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:27.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:26 smithi151 ceph-mon[16639]: Added host smithi151 2024-09-02T09:54:27.069 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:26 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:54:27.873 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-02T09:54:27.873 INFO:teuthology.orchestra.run.smithi151.stdout:Deploying unmanaged mgr service... 2024-09-02T09:54:28.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:28 smithi151 ceph-mon[16639]: Saving service mon spec with placement count:5 2024-09-02T09:54:28.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:28 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:28.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:28 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:54:29.147 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-02T09:54:29.732 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:29 smithi151 ceph-mon[16639]: from='client.14143 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:29.732 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:29 smithi151 ceph-mon[16639]: Saving service mgr spec with placement count:2 2024-09-02T09:54:29.732 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:29 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:31.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4162263087' entity='client.admin' 2024-09-02T09:54:31.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:30 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:32.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:31 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:32.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1358212165' entity='client.admin' 2024-09-02T09:54:32.802 INFO:teuthology.orchestra.run.smithi151.stdout:Enabling the dashboard module... 2024-09-02T09:54:34.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:33 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:34.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1092381520' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-09-02T09:54:34.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:33 smithi151 ceph-mon[16639]: from='mgr.14119 172.21.15.151:0/3205075793' entity='mgr.y' 2024-09-02T09:54:34.173 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:33 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: ignoring --setuser ceph since I am not root 2024-09-02T09:54:34.173 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:33 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: ignoring --setgroup ceph since I am not root 2024-09-02T09:54:35.143 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1092381520' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-02T09:54:35.143 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:34 smithi151 ceph-mon[16639]: mgrmap e9: y(active, since 18s) 2024-09-02T09:54:36.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4292694102' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-09-02T09:54:36.295 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-02T09:54:36.295 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-02T09:54:36.296 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-02T09:54:36.296 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-02T09:54:36.296 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-02T09:54:36.296 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-02T09:54:36.296 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for the mgr to restart... 2024-09-02T09:54:36.296 INFO:teuthology.orchestra.run.smithi151.stdout:Waiting for mgr epoch 9... 2024-09-02T09:54:39.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:39 smithi151 ceph-mon[16639]: Active manager daemon y restarted 2024-09-02T09:54:39.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:39 smithi151 ceph-mon[16639]: Activating manager daemon y 2024-09-02T09:54:39.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:39 smithi151 ceph-mon[16639]: osdmap e3: 0 total, 0 up, 0 in 2024-09-02T09:54:39.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:39 smithi151 ceph-mon[16639]: mgrmap e10: y(active, starting, since 0.00365403s) 2024-09-02T09:54:39.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:54:39.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-09-02T09:54:39.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-09-02T09:54:39.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-09-02T09:54:39.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-09-02T09:54:39.357 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:39 smithi151 ceph-mon[16639]: Manager daemon y is now available 2024-09-02T09:54:40.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:54:40.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-09-02T09:54:40.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-09-02T09:54:40.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-mon[16639]: mgrmap e11: y(active, since 1.00621s) 2024-09-02T09:54:41.046 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: 2024-09-02T09:54:40.975+0000 7fd74066e640 -1 log_channel(cephadm) log [ERR] : [02/Sep/2024:09:54:40] ENGINE Error in HTTPServer.serve 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: Traceback (most recent call last): 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: self._connections.run(self.expiration_interval) 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: self._run(expiration_interval) 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: new_conn = self._from_server_socket(self.server.socket) 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-02T09:54:41.047 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: s = self.context.wrap_socket( 2024-09-02T09:54:41.048 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-02T09:54:41.048 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: return self.sslsocket_class._create( 2024-09-02T09:54:41.048 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-02T09:54:41.048 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: self.do_handshake() 2024-09-02T09:54:41.048 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-02T09:54:41.048 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: self._sslobj.do_handshake() 2024-09-02T09:54:41.048 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-02T09:54:41.048 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 09:54:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: 2024-09-02T09:54:41.049 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout { 2024-09-02T09:54:41.049 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-02T09:54:41.049 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-02T09:54:41.049 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout } 2024-09-02T09:54:41.049 INFO:teuthology.orchestra.run.smithi151.stdout:mgr epoch 9 is available 2024-09-02T09:54:41.049 INFO:teuthology.orchestra.run.smithi151.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-09-02T09:54:41.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:41 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:41.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:41 smithi151 ceph-mon[16639]: [02/Sep/2024:09:54:40] ENGINE Bus STARTING 2024-09-02T09:54:41.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:41 smithi151 ceph-mon[16639]: [02/Sep/2024:09:54:40] ENGINE Serving on http://172.21.15.151:8765 2024-09-02T09:54:41.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:41 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:42 smithi151 ceph-mon[16639]: [02/Sep/2024:09:54:40] ENGINE Serving on https://172.21.15.151:7150 2024-09-02T09:54:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:42 smithi151 ceph-mon[16639]: [02/Sep/2024:09:54:40] ENGINE Bus STARTED 2024-09-02T09:54:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:42 smithi151 ceph-mon[16639]: [02/Sep/2024:09:54:40] ENGINE Error in HTTPServer.serve 2024-09-02T09:54:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout: Traceback (most recent call last): 2024-09-02T09:54:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-02T09:54:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._connections.run(self.expiration_interval) 2024-09-02T09:54:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-02T09:54:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._run(expiration_interval) 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: s = self.context.wrap_socket( 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: return self.sslsocket_class._create( 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: self.do_handshake() 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: self._sslobj.do_handshake() 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-02T09:54:42.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:42 smithi151 ceph-mon[16639]: mgrmap e12: y(active, since 2s) 2024-09-02T09:54:42.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:42 smithi151 ceph-mon[16639]: from='client.14163 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:43.043 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIE/DCCAuSgAwIBAgIUdnLSJtqroMzd3hbk0XImXNe47m8wDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDkwMjA5NTQ0MloXDTM0\nMDkwMzA5NTQ0MlowGDEWMBQGA1UEAwwNMTcyLjIxLjE1LjE1MTCCAiIwDQYJKoZI\nhvcNAQEBBQADggIPADCCAgoCggIBAKfgbRYEr7xEhdSeXHSg6S7pPvjyvg4OaCQh\nuWbcUCjbQSbpn051CDGnpE9jo6FMsE4Ph8/NJlNAB1pAEns4GIEW1ZfHKP82wKCx\nPdITTJzdKMSSbyelOP0/s+9ccUN1WTILAnl44nQ4Gmqm5VedLgR5CVOA+L3E42PS\nDDpIUfkplj5cR3r7Hmfr4wN+TFfdmbYrsyuKMtMr70syzu1AOy92DuqzIsHMqBNy\n+f/CjZoEaDMXYBYkq/7tBfM4BI/KggFWak6Q2bFjaH77tq6bKVZonm59fZ1ZW4zS\n9L1hZZhQDFyeli8TW9iSFFi/pO4gpomECIPoJBGBKOknyDxV+I0+GyaPUl/tu+vk\ncifd2jQnDyyoNGFu9+li1JXJul+2p8ds7Fsbi3yvi5tY8E59ltD+ODUX0P71vwpM\nttD+us1ptuB2DqFCpHPQes2A7rybIjoexfrVgevfsILRkbHqbg6ovu5BKMCFeCA/\ngCLCFU8oSYakAXfc6xkNX52TZrYqlXoctNas0uXvizGLq/R6+bRXr+HP2DKUz0EN\nNO6+CudZa2xSLCyeXvqLRhtFZC+ednIrhqi4IfrbDG6dhf4i+YuqeSOPz7skdLrc\nVspct3OZLJprg87ZoVhrAL3PQG/yTpq8VYnAiegl5hxd5+1PBeaD+txR6WCIDaLA\nNXQS+38XAgMBAAGjPzA9MC0GA1UdEQQmMCSCCXNtaXRoaTE1MYIRZGFzaGJvYXJk\nX3NlcnZlcnOHBKwVD5cwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQsFAAOCAgEA\nP2TScmvI25DNM6Ykgh985IBjryxjU4WqZVIz4Y4dyHoZ0s+MJ+9j3s+vv9kyYKDh\nIKbT0iKt6OiVf+HOvAtjL1ERggpQ+MVaM9Ji6Q/hDiI5zicTXtqwQcWOEn1hig0E\np/VZ9Pu20+niUm1BYsoIjUFkqWOJKa9Jl7+uho1hIAFag/iHllEcqOgGxXFe1mkS\n0GImMFBrATvhs1hnJrXdNcHsZrsfGt/CtJRIwqpl8GfkfGqYWTRkoMF5DMOSE/zy\nR7yDhfs3FyFbt0LQah18AvKh6VDPBw6Y+Q6VoSl21u/uQcTL87hTz8PifJHtb7cb\nOhdNfFu3SMRfr3WD/U2B40E1hggUTNSq7l1QPhA9HtOS4at2b6QIuiW94FYxkKMT\nGtNAn/W3DWwFf2OHJUvt1Dq4zJY1FEIi9EbXMsEsaX9btiHUP395w6PxQcBprvWQ\ntamLXvv5PsVteWeIjRGuiNaskA5g5wjrgl+oKbntjz4s7Z95LnHg381ilDJqLhYE\nRdPEDGui4hxWPt2n9WIN42cQOw9kTvl4owoS9WrDtLQeciM5ERZaF/s6LYGtMaHE\nyoqkujYooRdec861ONjaV0lGUPnBpDPQxlsJLZ6uFYesnB4vdYgB6ecgY3PffKEh\n8N2ZYOi9PBwQll6oOMwwq77r9PqTsXTFo3mmS+EeTwA=\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJJwIBAAKCAgEAp+BtFgSvvESF1J5cdKDpLuk++PK+Dg5oJCG5ZtxQKNtBJumf\nTnUIMaekT2OjoUywTg+Hz80mU0AHWkASezgYgRbVl8co/zbAoLE90hNMnN0oxJJv\nJ6U4/T+z71xxQ3VZMgsCeXjidDgaaqblV50uBHkJU4D4vcTjY9IMOkhR+SmWPlxH\nevseZ+vjA35MV92ZtiuzK4oy0yvvSzLO7UA7L3YO6rMiwcyoE3L5/8KNmgRoMxdg\nFiSr/u0F8zgEj8qCAVZqTpDZsWNofvu2rpspVmiebn19nVlbjNL0vWFlmFAMXJ6W\nLxNb2JIUWL+k7iCmiYQIg+gkEYEo6SfIPFX4jT4bJo9SX+276+RyJ93aNCcPLKg0\nYW736WLUlcm6X7anx2zsWxuLfK+Lm1jwTn2W0P44NRfQ/vW/Cky20P66zWm24HYO\noUKkc9B6zYDuvJsiOh7F+tWB69+wgtGRsepuDqi+7kEowIV4ID+AIsIVTyhJhqQB\nd9zrGQ1fnZNmtiqVehy01qzS5e+LMYur9Hr5tFev4c/YMpTPQQ007r4K51lrbFIs\nLJ5e+otGG0VkL552ciuGqLgh+tsMbp2F/iL5i6p5I4/PuyR0utxWyly3c5ksmmuD\nztmhWGsAvc9Ab/JOmrxVicCJ6CXmHF3n7U8F5oP63FHpYIgNosA1dBL7fxcCAwEA\nAQKCAgAQ5C3QYC1XKfqLSlnZfv1SHombCUBNRW7TaonvSxiiW8P3ndx3y6JvmJjE\nR4LF2TkIybBBHOO6eaqjuTxYJeyf7y5o+f2VMar6dYsCXGBs4iXWzY3In+vBiODa\neI3e8JZGc5bOpnampKR9JF4sIsu9YHkAlsKwbs3waR3uLqus1d3hAsOgdgGjtDeL\nI5tGi1dgMzpxFZPdQpeZFJDUjLjHzZRc/M7w+rN+AtIbxWAeZfYeueni9dKmhlkp\nv/JZjic8P40tFY8J4GDkcgBPcPDyWaQjUEDM4xpT65VQ3tVsQ2r9q2lHbaZj/qLJ\ny+f/m80V8MaIkIQx4gc979D1HTNdHOM+UFwm25saaxog9MWBH1RgYstcrXzZwRrZ\nOIHI39cceXtYPI/Y7fzDLb7m105jkYrDsiNb3MPUAlCOC/jGLjZ13mFDoetYNtTe\n7dHV0gm6iw6c2d93vaOIoMYL8kKf95m9LR9014dRfOiN1w+uA9m/FypK+wfgwPEN\nKdTIfUVK1WyD4VvMyZcwaw7vWvp1ow7z5ZElIt6DETLMPXfpKbmn3c1jOoqnvffC\nmRWpR+bc1WuLy+AntyEfovxPLJoXnch/56ZDeA/fSFPLOpdZzQ3vop3ZGhU/xjHq\nirI7t5440smVAZOJqlh2hjH2cI+SQlm+A4uWxCZMXdy0KfLbhQKCAQEA5d8hvlRy\nGQQmavGjp8Cz81lXUVCVbLIdGyuUL6ZSL9x2dnh/VAPwkaMgJTHT9x9TafjTntqj\n/jnM1nY3IRGKP54xxdS4Xwb3o+OybSBP/eWMywC8QTMY8IrO+9OFuvhYDeTQlPMm\nBRWuyEZUYRW5CM7ce/MLvITPDJD8gkR2Jue+nRmM0RlO+oODlGewx2Odhn7z8/cM\nWsLOw40+NRTjlzAK3BhcuqI7CzRlqve7FfunyMq6GnQb4nLEaX2u7iMT+VhtoxSx\nJGmHwK8XJkjRTlGNhX1UaYb2Rgucgmx3Kzr6HAWxw2Hoj1Gnng8p2/ey8WuVjJlX\n1zMBkwTjS2EE/QKCAQEAuvVZHwhF7YopLMtELZEVbzmJJpQ0vnmdvy1BHTaIE640\n8UHmGMf0JA3SXt2s45RO1ViPhqAoOLIOXxyhlX9n9+gjdxttZsvXDNLtWAn90hJ2\nFCDeTqyldNB6N6OhDDTdbL/G+aQ93aldP/4+R4YW43Z8svkfa7axFUR9QGU5rYPJ\nlshoUbfPpfTh/1dFcgXVlrVfbrW5rEH8FdvI7k6TcNrQPEsr2wuulPs3QYYR9GSv\nZK4NHSJiwupYsRtR0UFFmMGBPmL1uOvTDDa8/MI6kdU07btu1ta9wqv5MUKStqGg\n29lxrF89TPSB2IMkch3peha02lSzLqKqsdX/s6Q6owKCAQByObyUzXWdBLjqb4S5\nOUKya/S9zWNTDfb0p2UNgIJakhyzphDveL20QPEaSnZf2Esk36aYy2CuVZ55+2hN\noF+oWT22ZovvfJDV+bGSFHprWgPqZEUnyPJdpTxRWrybOgjXbppPSbvczvd5Qalw\nTNETlWdumEl+tigEwI8ZVFe54XKQsj+NMBxU08Ewn43ypoeCa9dhpaOX/jGsbqU4\nz1aArDjZQanie/cHDFRw7oUi2Vh4qHRt7MBmeE0tdmzUnM2yG8TvQrr7swzoLJk+\nRvTrDA9dA/hML+p0yBRPowJckwPxpMq+ogpkHVlDa5mLWryqxqEbqp9BqnDf04/2\nxxEpAoIBAHdYn+e7dEhK5jX+HCuqbjBuXisOeQ0OvbCIxwRLB1ro3ksUcjpnimZp\nnF2KzQ8KYbi3rAeYvZRTtfDpagzxJAGRYvJpw9DXVxSMoY6N8kXkGxSGVuOKOqTz\nYNGb1vbf2GnPkslSG1wg/bIYLtGhu4LA5KGrfm9YVHrrGbm95K/rbQigLlfcKuYa\nCRrjLyW2q75lUeJuh3R7Ot0JTJ8z8dMdG8khg4F1V6ErLP7bAyMsEvsGUMDEbp9W\n0f9dHUGjlTYKcOLOcWX2xOcLGmGP66ukmwEXoY3rz38yIQVdJiMH0pQFnxcWRR+X\nBjycmutuRNj3HcQyRaADn0ue/HhE4lkCggEAfcN21TR+Pv97WD+fyl4u8APUtvus\nIkrL3ImozlBD+smcQRyOp1n6LWhko2HGCjCBxEduRU1mRnY1MTy+s11m0eF3Ifp1\nJ9XgOxjUQRP70vEzWfTTr2xdPxYzksRE15Q4MphOAsgbsD6QtZ47t53R195QwCsg\nY9ydyR4MLqDq10iEanBIH75fthdH5y0H26GycONY4Du6XxappgW7uwZKbTvz0o/n\nlMCA4aCGl7A+ggnZ9hAU3m5TCegkR1LjlpMxpeFpQbX/80DBSYLzXWO2kuhn31Pi\nhCY+xH99r3ql7eA3ESOleJ68XlBJLlqIgfwcKy2Fa4eGqN/EGORbvb6l1A==\n-----END RSA PRIVATE KEY-----\n"} 2024-09-02T09:54:44.519 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-09-02T09:54:44.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:44 smithi151 ceph-mon[16639]: from='client.14165 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:44.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:44 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:45.792 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-09-02T09:54:45.792 INFO:teuthology.orchestra.run.smithi151.stdout:Creating initial admin user... 2024-09-02T09:54:46.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:46 smithi151 ceph-mon[16639]: from='client.14167 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:47.189 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$SP9jDcv2mkVsZ4IbOBfd5udZUOwgTEyjKjppDCzLedRrsrhTU7TQO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1725270886, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-02T09:54:47.189 INFO:teuthology.orchestra.run.smithi151.stdout:Fetching dashboard port number... 2024-09-02T09:54:47.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:47 smithi151 ceph-mon[16639]: from='client.14169 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:47.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:48.397 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stdout 8443 2024-09-02T09:54:48.408 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-02T09:54:48.408 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout disabled 2024-09-02T09:54:48.416 INFO:teuthology.orchestra.run.smithi151.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-02T09:54:48.416 INFO:teuthology.orchestra.run.smithi151.stdout:systemctl: stdout inactive 2024-09-02T09:54:48.416 INFO:teuthology.orchestra.run.smithi151.stdout:firewalld.service is not enabled 2024-09-02T09:54:48.416 INFO:teuthology.orchestra.run.smithi151.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-02T09:54:48.417 INFO:teuthology.orchestra.run.smithi151.stdout:Ceph Dashboard is now available at: 2024-09-02T09:54:48.417 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:54:48.417 INFO:teuthology.orchestra.run.smithi151.stdout: URL: https://smithi151:8443/ 2024-09-02T09:54:48.417 INFO:teuthology.orchestra.run.smithi151.stdout: User: admin 2024-09-02T09:54:48.417 INFO:teuthology.orchestra.run.smithi151.stdout: Password: dbk0p66dfg 2024-09-02T09:54:48.418 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:54:48.418 INFO:teuthology.orchestra.run.smithi151.stdout:Saving cluster configuration to /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config directory 2024-09-02T09:54:48.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1132573258' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-09-02T09:54:49.762 INFO:teuthology.orchestra.run.smithi151.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-02T09:54:49.762 INFO:teuthology.orchestra.run.smithi151.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-02T09:54:49.762 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:54:49.762 INFO:teuthology.orchestra.run.smithi151.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-02T09:54:49.762 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:54:49.763 INFO:teuthology.orchestra.run.smithi151.stdout:Or, if you are only running a single cluster on this host: 2024-09-02T09:54:49.763 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:54:49.763 INFO:teuthology.orchestra.run.smithi151.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-02T09:54:49.763 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:54:49.763 INFO:teuthology.orchestra.run.smithi151.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-02T09:54:49.763 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:54:49.764 INFO:teuthology.orchestra.run.smithi151.stdout: ceph telemetry on 2024-09-02T09:54:49.764 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:54:49.764 INFO:teuthology.orchestra.run.smithi151.stdout:For more information see: 2024-09-02T09:54:49.764 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:54:49.764 INFO:teuthology.orchestra.run.smithi151.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-02T09:54:49.764 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:54:49.765 INFO:teuthology.orchestra.run.smithi151.stdout:Bootstrap complete. 2024-09-02T09:54:49.803 INFO:tasks.cephadm:Fetching config... 2024-09-02T09:54:49.804 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:54:49.804 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-02T09:54:49.820 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-02T09:54:49.821 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:54:49.821 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-02T09:54:49.877 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-02T09:54:49.877 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:54:49.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-02T09:54:49.942 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-02T09:54:49.942 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:54:49.942 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-02T09:54:49.998 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-02T09:54:49.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxwQNwpxhAXRIDj1z3XI6B2vaA94lB59/WwruLsgqf3CCudy/WT9mfHS7P8GPNQf1eICN7MmsZM8qYdEneIFXkw254G7kkBmcof0hThCKlyReI530+g+sm6jtb13MljQmk4/g8DPlUpw59GC+uq4xolG99OKhXpZQlnG5Krk9Am76U2RtdjHBzxt0XaQY+AyOsj6jEKBtETFnz9P7Cd9Q6q0eXXRtYd3h3jCoAFcVD6LSEhgcZ8hu6YL0ApKcfC6V0P3me2NDgqBxZzspJuO/U1bgwg0+OZalmkyFXpPA5VZpfCUT9jAIzSzrPxw7q99hiQDjV5TkRck1B5FSn3iyKQMAmnWhNataVpUuM2HAPfbI+YfbsuEEWi7lVczS2qobjvrGWY11F3dFxTKvCY+d//0R4XJcuwT/XoksZKXJCKHC+NosaGB9JWr9BHShp0G56c/8cZFX3IFq7MnODu8Nm5mzKNkEWkGf2nQgfu9rc/pK85J7BArLxsK4/KqqIKbc= ceph-1530c7be-6911-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-02T09:54:50.073 INFO:teuthology.orchestra.run.smithi151.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxwQNwpxhAXRIDj1z3XI6B2vaA94lB59/WwruLsgqf3CCudy/WT9mfHS7P8GPNQf1eICN7MmsZM8qYdEneIFXkw254G7kkBmcof0hThCKlyReI530+g+sm6jtb13MljQmk4/g8DPlUpw59GC+uq4xolG99OKhXpZQlnG5Krk9Am76U2RtdjHBzxt0XaQY+AyOsj6jEKBtETFnz9P7Cd9Q6q0eXXRtYd3h3jCoAFcVD6LSEhgcZ8hu6YL0ApKcfC6V0P3me2NDgqBxZzspJuO/U1bgwg0+OZalmkyFXpPA5VZpfCUT9jAIzSzrPxw7q99hiQDjV5TkRck1B5FSn3iyKQMAmnWhNataVpUuM2HAPfbI+YfbsuEEWi7lVczS2qobjvrGWY11F3dFxTKvCY+d//0R4XJcuwT/XoksZKXJCKHC+NosaGB9JWr9BHShp0G56c/8cZFX3IFq7MnODu8Nm5mzKNkEWkGf2nQgfu9rc/pK85J7BArLxsK4/KqqIKbc= ceph-1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:54:50.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxwQNwpxhAXRIDj1z3XI6B2vaA94lB59/WwruLsgqf3CCudy/WT9mfHS7P8GPNQf1eICN7MmsZM8qYdEneIFXkw254G7kkBmcof0hThCKlyReI530+g+sm6jtb13MljQmk4/g8DPlUpw59GC+uq4xolG99OKhXpZQlnG5Krk9Am76U2RtdjHBzxt0XaQY+AyOsj6jEKBtETFnz9P7Cd9Q6q0eXXRtYd3h3jCoAFcVD6LSEhgcZ8hu6YL0ApKcfC6V0P3me2NDgqBxZzspJuO/U1bgwg0+OZalmkyFXpPA5VZpfCUT9jAIzSzrPxw7q99hiQDjV5TkRck1B5FSn3iyKQMAmnWhNataVpUuM2HAPfbI+YfbsuEEWi7lVczS2qobjvrGWY11F3dFxTKvCY+d//0R4XJcuwT/XoksZKXJCKHC+NosaGB9JWr9BHShp0G56c/8cZFX3IFq7MnODu8Nm5mzKNkEWkGf2nQgfu9rc/pK85J7BArLxsK4/KqqIKbc= ceph-1530c7be-6911-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-02T09:54:50.126 INFO:teuthology.orchestra.run.smithi162.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxwQNwpxhAXRIDj1z3XI6B2vaA94lB59/WwruLsgqf3CCudy/WT9mfHS7P8GPNQf1eICN7MmsZM8qYdEneIFXkw254G7kkBmcof0hThCKlyReI530+g+sm6jtb13MljQmk4/g8DPlUpw59GC+uq4xolG99OKhXpZQlnG5Krk9Am76U2RtdjHBzxt0XaQY+AyOsj6jEKBtETFnz9P7Cd9Q6q0eXXRtYd3h3jCoAFcVD6LSEhgcZ8hu6YL0ApKcfC6V0P3me2NDgqBxZzspJuO/U1bgwg0+OZalmkyFXpPA5VZpfCUT9jAIzSzrPxw7q99hiQDjV5TkRck1B5FSn3iyKQMAmnWhNataVpUuM2HAPfbI+YfbsuEEWi7lVczS2qobjvrGWY11F3dFxTKvCY+d//0R4XJcuwT/XoksZKXJCKHC+NosaGB9JWr9BHShp0G56c/8cZFX3IFq7MnODu8Nm5mzKNkEWkGf2nQgfu9rc/pK85J7BArLxsK4/KqqIKbc= ceph-1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:54:50.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxwQNwpxhAXRIDj1z3XI6B2vaA94lB59/WwruLsgqf3CCudy/WT9mfHS7P8GPNQf1eICN7MmsZM8qYdEneIFXkw254G7kkBmcof0hThCKlyReI530+g+sm6jtb13MljQmk4/g8DPlUpw59GC+uq4xolG99OKhXpZQlnG5Krk9Am76U2RtdjHBzxt0XaQY+AyOsj6jEKBtETFnz9P7Cd9Q6q0eXXRtYd3h3jCoAFcVD6LSEhgcZ8hu6YL0ApKcfC6V0P3me2NDgqBxZzspJuO/U1bgwg0+OZalmkyFXpPA5VZpfCUT9jAIzSzrPxw7q99hiQDjV5TkRck1B5FSn3iyKQMAmnWhNataVpUuM2HAPfbI+YfbsuEEWi7lVczS2qobjvrGWY11F3dFxTKvCY+d//0R4XJcuwT/XoksZKXJCKHC+NosaGB9JWr9BHShp0G56c/8cZFX3IFq7MnODu8Nm5mzKNkEWkGf2nQgfu9rc/pK85J7BArLxsK4/KqqIKbc= ceph-1530c7be-6911-11ef-bcd4-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-02T09:54:50.191 INFO:teuthology.orchestra.run.smithi187.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxwQNwpxhAXRIDj1z3XI6B2vaA94lB59/WwruLsgqf3CCudy/WT9mfHS7P8GPNQf1eICN7MmsZM8qYdEneIFXkw254G7kkBmcof0hThCKlyReI530+g+sm6jtb13MljQmk4/g8DPlUpw59GC+uq4xolG99OKhXpZQlnG5Krk9Am76U2RtdjHBzxt0XaQY+AyOsj6jEKBtETFnz9P7Cd9Q6q0eXXRtYd3h3jCoAFcVD6LSEhgcZ8hu6YL0ApKcfC6V0P3me2NDgqBxZzspJuO/U1bgwg0+OZalmkyFXpPA5VZpfCUT9jAIzSzrPxw7q99hiQDjV5TkRck1B5FSn3iyKQMAmnWhNataVpUuM2HAPfbI+YfbsuEEWi7lVczS2qobjvrGWY11F3dFxTKvCY+d//0R4XJcuwT/XoksZKXJCKHC+NosaGB9JWr9BHShp0G56c/8cZFX3IFq7MnODu8Nm5mzKNkEWkGf2nQgfu9rc/pK85J7BArLxsK4/KqqIKbc= ceph-1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:54:50.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-02T09:54:50.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/799108237' entity='client.admin' 2024-09-02T09:54:50.455 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:54:52.251 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-02T09:54:52.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-02T09:54:52.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:52 smithi151 ceph-mon[16639]: mgrmap e13: y(active, since 12s) 2024-09-02T09:54:52.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2876500829' entity='client.admin' 2024-09-02T09:54:52.775 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:54:54.146 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi162 2024-09-02T09:54:54.146 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:54:54.146 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.conf 2024-09-02T09:54:54.166 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:54:54.166 DEBUG:teuthology.orchestra.run.smithi162:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T09:54:54.224 INFO:tasks.cephadm:Adding host smithi162 to orchestrator... 2024-09-02T09:54:54.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch host add smithi162 2024-09-02T09:54:54.554 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:54:54.771 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:54 smithi151 ceph-mon[16639]: from='client.14177 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:54.771 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:54 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi151", "name": "osd_memory_target"} : dispatch 2024-09-02T09:54:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:54:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:54:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-02T09:54:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: Updating smithi151:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:54:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:54:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:54:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:54:55.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:56.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:56 smithi151 ceph-mon[16639]: Updating smithi151:/etc/ceph/ceph.client.admin.keyring 2024-09-02T09:54:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:56 smithi151 ceph-mon[16639]: Updating smithi151:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T09:54:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:56 smithi151 ceph-mon[16639]: from='client.14179 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi162", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:54:57.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:57 smithi151 ceph-mon[16639]: Deploying cephadm binary to smithi162 2024-09-02T09:54:58.567 INFO:teuthology.orchestra.run.smithi151.stdout:Added host 'smithi162' with addr '172.21.15.162' 2024-09-02T09:54:59.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch host ls --format=json 2024-09-02T09:54:59.418 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:54:59.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:59.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:59 smithi151 ceph-mon[16639]: Added host smithi162 2024-09-02T09:54:59.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:54:59.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:54:59.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:54:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:00.082 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:55:00.082 INFO:teuthology.orchestra.run.smithi151.stdout:[{"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}] 2024-09-02T09:55:00.663 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi187 2024-09-02T09:55:00.664 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:55:00.664 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.conf 2024-09-02T09:55:00.683 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:55:00.683 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T09:55:00.741 INFO:tasks.cephadm:Adding host smithi187 to orchestrator... 2024-09-02T09:55:00.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch host add smithi187 2024-09-02T09:55:00.984 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:55:01.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:00 smithi151 ceph-mon[16639]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:00 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:01 smithi151 ceph-mon[16639]: from='client.14181 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T09:55:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:02 smithi151 ceph-mon[16639]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:02 smithi151 ceph-mon[16639]: from='client.14183 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi187", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:55:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:02 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:02 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:02 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:02 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:02 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi162", "name": "osd_memory_target"} : dispatch 2024-09-02T09:55:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:02 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:02 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:04.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:03 smithi151 ceph-mon[16639]: Deploying cephadm binary to smithi187 2024-09-02T09:55:04.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:03 smithi151 ceph-mon[16639]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-02T09:55:04.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:03 smithi151 ceph-mon[16639]: Updating smithi162:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:04.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:04.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:04.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:04.818 INFO:teuthology.orchestra.run.smithi151.stdout:Added host 'smithi187' with addr '172.21.15.187' 2024-09-02T09:55:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:04 smithi151 ceph-mon[16639]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:04 smithi151 ceph-mon[16639]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-02T09:55:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:04 smithi151 ceph-mon[16639]: Updating smithi162:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T09:55:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:05.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch host ls --format=json 2024-09-02T09:55:05.745 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:55:06.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:05 smithi151 ceph-mon[16639]: Added host smithi187 2024-09-02T09:55:06.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:05 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:06.179 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:05 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:06.389 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:55:06.390 INFO:teuthology.orchestra.run.smithi151.stdout:[{"addr": "172.21.15.151", "hostname": "smithi151", "labels": [], "status": ""}, {"addr": "172.21.15.162", "hostname": "smithi162", "labels": [], "status": ""}, {"addr": "172.21.15.187", "hostname": "smithi187", "labels": [], "status": ""}] 2024-09-02T09:55:06.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:06 smithi151 ceph-mon[16639]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:06.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:06.949 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:06 smithi151 ceph-mon[16639]: from='client.14185 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T09:55:06.977 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-02T09:55:06.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd crush tunables default 2024-09-02T09:55:07.218 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:55:07.931 INFO:teuthology.orchestra.run.smithi151.stderr:adjusted tunables profile to default 2024-09-02T09:55:08.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:07 smithi151 ceph-mon[16639]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:08.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1306288317' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-02T09:55:08.550 INFO:tasks.cephadm:Adding mon.a on smithi151 2024-09-02T09:55:08.550 INFO:tasks.cephadm:Adding mon.b on smithi162 2024-09-02T09:55:08.550 INFO:tasks.cephadm:Adding mon.c on smithi187 2024-09-02T09:55:08.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch apply mon '3;smithi151:172.21.15.151=a;smithi162:172.21.15.162=b;smithi187:172.21.15.187=c' 2024-09-02T09:55:08.971 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-02T09:55:09.017 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-02T09:55:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1306288317' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-02T09:55:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:08 smithi151 ceph-mon[16639]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T09:55:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"} : dispatch 2024-09-02T09:55:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:08 smithi151 ceph-mon[16639]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-02T09:55:09.747 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mon update... 2024-09-02T09:55:10.286 DEBUG:teuthology.orchestra.run.smithi162:mon.b> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.b.service 2024-09-02T09:55:10.289 DEBUG:teuthology.orchestra.run.smithi187:mon.c> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.c.service 2024-09-02T09:55:10.292 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-02T09:55:10.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-09-02T09:55:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: Updating smithi187:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-09-02T09:55:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: Updating smithi187:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T09:55:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi151:172.21.15.151=a;smithi162:172.21.15.162=b;smithi187:172.21.15.187=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:55:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: Saving service mon spec with placement smithi151:172.21.15.151=a;smithi162:172.21.15.162=b;smithi187:172.21.15.187=c;count:3 2024-09-02T09:55:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:11.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:11.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:12.252 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:55:13.186 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:55:13.186 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":1,"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","modified":"2024-09-02T09:53:50.141532Z","created":"2024-09-02T09:53:50.141532Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-02T09:55:13.187 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 1 2024-09-02T09:55:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:13 smithi187 podman[20053]: 2024-09-02 09:55:13.973166415 +0000 UTC m=+0.118769245 container create b9415a1035d30d14c4ea0b57e4707050a6160536546f7cbeabe852acd81355af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-02T09:55:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/1986746237' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T09:55:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 podman[20053]: 2024-09-02 09:55:14.175664101 +0000 UTC m=+0.321266885 container init b9415a1035d30d14c4ea0b57e4707050a6160536546f7cbeabe852acd81355af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-c, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 podman[20053]: 2024-09-02 09:55:14.178993753 +0000 UTC m=+0.324596538 container start b9415a1035d30d14c4ea0b57e4707050a6160536546f7cbeabe852acd81355af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-c, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: set uid:gid to 167:167 (ceph:ceph) 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: ceph version 19.3.0-4642-g82daa6ef (82daa6ef0815320144b25b212c6b8c25a1d59c07) squid (dev), process ceph-mon, pid 2 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: pidfile_write: ignore empty --pid-file 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: load: jerasure load: lrc 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: RocksDB version: 7.9.2 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Git sha 0 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Compile date 2024-09-01 08:27:02 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: DB SUMMARY 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: DB Session ID: 709UWDL8WD3W3L6YLOZE 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: CURRENT file: CURRENT 2024-09-02T09:55:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: IDENTITY file: IDENTITY 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.error_if_exists: 0 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.create_if_missing: 0 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.paranoid_checks: 1 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.env: 0x55f90b4e4160 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.fs: PosixFileSystem 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.info_log: 0x55f90c0a3860 2024-09-02T09:55:14.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_file_opening_threads: 16 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.statistics: (nil) 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.use_fsync: 0 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_log_file_size: 0 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.keep_log_file_num: 1000 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.recycle_log_file_num: 0 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.allow_fallocate: 1 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.allow_mmap_reads: 0 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.allow_mmap_writes: 0 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.use_direct_reads: 0 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-02T09:55:14.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.create_missing_column_families: 0 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.db_log_dir: 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.wal_dir: 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.advise_random_on_open: 1 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.db_write_buffer_size: 0 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.write_buffer_manager: 0x55f90c0a6e60 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-02T09:55:14.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.rate_limiter: (nil) 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.wal_recovery_mode: 2 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.enable_thread_tracking: 0 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.enable_pipelined_write: 0 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.unordered_write: 0 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.row_cache: None 2024-09-02T09:55:14.511 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.wal_filter: None 2024-09-02T09:55:14.513 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-02T09:55:14.513 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.allow_ingest_behind: 0 2024-09-02T09:55:14.513 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.two_write_queues: 0 2024-09-02T09:55:14.513 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.manual_wal_flush: 0 2024-09-02T09:55:14.513 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.wal_compression: 0 2024-09-02T09:55:14.513 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.atomic_flush: 0 2024-09-02T09:55:14.513 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-02T09:55:14.513 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-02T09:55:14.513 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-02T09:55:14.513 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.log_readahead_size: 0 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.best_efforts_recovery: 0 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.allow_data_in_errors: 0 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.db_host_id: __hostname__ 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_background_jobs: 2 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_background_compactions: -1 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_subcompactions: 1 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-02T09:55:14.514 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_total_wal_size: 0 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_open_files: -1 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bytes_per_sync: 0 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_readahead_size: 0 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_background_flushes: -1 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Compression algorithms supported: 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: kZSTD supported: 0 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: kXpressCompression supported: 0 2024-09-02T09:55:14.515 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: kBZip2Compression supported: 0 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: kLZ4Compression supported: 1 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: kZlibCompression supported: 1 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: kLZ4HCCompression supported: 1 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: kSnappyCompression supported: 1 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-02T09:55:14.516 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.merge_operator: 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_filter: None 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_filter_factory: None 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.sst_partitioner_factory: None 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f90c0a34a0) 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout: cache_index_and_filter_blocks: 1 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout: pin_top_level_index_and_filter: 1 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout: index_type: 0 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout: data_block_index_type: 0 2024-09-02T09:55:14.517 INFO:journalctl@ceph.mon.c.smithi187.stdout: index_shortening: 1 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: checksum: 4 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: no_block_cache: 0 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache: 0x55f90c097610 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache_name: BinnedLRUCache 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache_options: 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: capacity : 536870912 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: num_shard_bits : 4 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: strict_capacity_limit : 0 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: high_pri_pool_ratio: 0.000 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_cache_compressed: (nil) 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: persistent_cache: (nil) 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_size: 4096 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_size_deviation: 10 2024-09-02T09:55:14.518 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_restart_interval: 16 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: index_block_restart_interval: 1 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: metadata_block_size: 4096 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: partition_filters: 0 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: use_delta_encoding: 1 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: filter_policy: bloomfilter 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: whole_key_filtering: 1 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: verify_compression: 0 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: read_amp_bytes_per_bit: 0 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: format_version: 5 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: enable_index_compression: 1 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: block_align: 0 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: max_auto_readahead_size: 262144 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: prepopulate_block_cache: 0 2024-09-02T09:55:14.519 INFO:journalctl@ceph.mon.c.smithi187.stdout: initial_auto_readahead_size: 8192 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout: num_file_reads_for_auto_readahead: 2 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.write_buffer_size: 33554432 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_write_buffer_number: 2 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compression: NoCompression 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bottommost_compression: Disabled 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.prefix_extractor: nullptr 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.num_levels: 7 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-02T09:55:14.520 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compression_opts.level: 32767 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compression_opts.strategy: 0 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-02T09:55:14.521 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compression_opts.enabled: false 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.target_file_size_base: 67108864 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-02T09:55:14.522 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.arena_block_size: 1048576 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.disable_auto_compactions: 0 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-02T09:55:14.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.inplace_update_support: 0 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-02T09:55:14.524 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.bloom_locality: 0 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.max_successive_merges: 0 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.paranoid_file_checks: 0 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.force_consistency_checks: 1 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.report_bg_io_stats: 0 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.ttl: 2592000 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.enable_blob_files: false 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.min_blob_size: 0 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.blob_file_size: 268435456 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-02T09:55:14.525 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.blob_file_starting_level: 0 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0f12a650-2232-4686-9189-2d399aec17d7 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725270914207884, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725270914208317, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1725270914, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0f12a650-2232-4686-9189-2d399aec17d7", "db_session_id": "709UWDL8WD3W3L6YLOZE", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725270914208411, "job": 1, "event": "recovery_finished"} 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-02T09:55:14.526 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f90c0c6e00 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: DB pointer 0x55f90c1c6000 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** DB Stats ** 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** Compaction Stats [default] ** 2024-09-02T09:55:14.527 INFO:journalctl@ceph.mon.c.smithi187.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** Compaction Stats [default] ** 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-02T09:55:14.528 INFO:journalctl@ceph.mon.c.smithi187.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: Block cache BinnedLRUCache@0x55f90c097610#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: using public_addr v2:172.21.15.187:0/0 -> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] 2024-09-02T09:55:14.529 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: starting mon.c rank -1 at public addrs [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] at bind addrs [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(???) e0 preinit fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).mds e1 new map 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout: e1 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout: btime 2024-09-02T09:53:53:306254+0000 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout: legacy client fscid: -1 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout: 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout: No filesystems configured 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-02T09:55:14.530 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Deploying cephadm binary to smithi187 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Updating smithi162:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:14.531 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:14.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Updating smithi162:/etc/ceph/ceph.client.admin.keyring 2024-09-02T09:55:14.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Updating smithi162:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T09:55:14.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:14.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Added host smithi187 2024-09-02T09:55:14.532 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='client.14185 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1306288317' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1306288317' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.533 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.534 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.534 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi187", "name": "osd_memory_target"} : dispatch 2024-09-02T09:55:14.534 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:14.534 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:14.534 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-02T09:55:14.534 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Updating smithi187:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:14.534 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Updating smithi187:/etc/ceph/ceph.client.admin.keyring 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Updating smithi187:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.client.admin.keyring 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='client.14189 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi151:172.21.15.151=a;smithi162:172.21.15.162=b;smithi187:172.21.15.187=c", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: Saving service mon spec with placement smithi151:172.21.15.151=a;smithi162:172.21.15.162=b;smithi187:172.21.15.187=c;count:3 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/1986746237' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 ceph-mon[20072]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 bash[20053]: b9415a1035d30d14c4ea0b57e4707050a6160536546f7cbeabe852acd81355af 2024-09-02T09:55:14.535 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:14 smithi187 systemd[1]: Started Ceph mon.c for 1530c7be-6911-11ef-bcd4-c7b262605968. 2024-09-02T09:55:14.821 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-02T09:55:14.821 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-09-02T09:55:15.071 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:55:19.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: Deploying daemon mon.c on smithi187 2024-09-02T09:55:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: mon.a calling monitor election 2024-09-02T09:55:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:19.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: mon.c calling monitor election 2024-09-02T09:55:19.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:19.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-02T09:55:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: monmap epoch 2 2024-09-02T09:55:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:55:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: last_changed 2024-09-02T09:55:14.237688+0000 2024-09-02T09:55:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: created 2024-09-02T09:53:50.141532+0000 2024-09-02T09:55:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: min_mon_release 19 (squid) 2024-09-02T09:55:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: election_strategy: 1 2024-09-02T09:55:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-02T09:55:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.c 2024-09-02T09:55:19.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: fsmap 2024-09-02T09:55:19.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T09:55:19.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: mgrmap e13: y(active, since 40s) 2024-09-02T09:55:19.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: overall HEALTH_OK 2024-09-02T09:55:19.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:19.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:19.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:19.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:19.692 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: Deploying daemon mon.c on smithi187 2024-09-02T09:55:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: mon.a calling monitor election 2024-09-02T09:55:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: mon.c calling monitor election 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: monmap epoch 2 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: last_changed 2024-09-02T09:55:14.237688+0000 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: created 2024-09-02T09:53:50.141532+0000 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: min_mon_release 19 (squid) 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: election_strategy: 1 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.c 2024-09-02T09:55:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: fsmap 2024-09-02T09:55:19.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T09:55:19.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: mgrmap e13: y(active, since 40s) 2024-09-02T09:55:19.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: overall HEALTH_OK 2024-09-02T09:55:19.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:19.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:19.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:19.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:19.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:20.164 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:55:20.164 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":2,"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","modified":"2024-09-02T09:55:14.237688Z","created":"2024-09-02T09:53:50.141532Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-02T09:55:20.164 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 2 2024-09-02T09:55:20.490 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:20 smithi187 ceph-mon[20072]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:20.491 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:20 smithi187 ceph-mon[20072]: Deploying daemon mon.b on smithi162 2024-09-02T09:55:20.491 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/3880849901' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T09:55:20.491 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:20 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:20.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:20 smithi151 ceph-mon[16639]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:20 smithi151 ceph-mon[16639]: Deploying daemon mon.b on smithi162 2024-09-02T09:55:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/3880849901' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T09:55:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:20 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:21.812 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-02T09:55:21.813 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-09-02T09:55:22.090 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:55:22.501 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:22 smithi187 ceph-mon[20072]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:22.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:22 smithi151 ceph-mon[16639]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:22.769 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:55:22.769 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":2,"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","modified":"2024-09-02T09:55:14.237688Z","created":"2024-09-02T09:53:50.141532Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-02T09:55:22.769 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 2 2024-09-02T09:55:23.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:22 smithi162 systemd[1]: Started Ceph mon.b for 1530c7be-6911-11ef-bcd4-c7b262605968. 2024-09-02T09:55:24.375 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-02T09:55:24.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph mon dump -f json 2024-09-02T09:55:24.628 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:55:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: mon.a calling monitor election 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: mon.c calling monitor election 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: mon.b calling monitor election 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-02T09:55:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: monmap epoch 3 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: last_changed 2024-09-02T09:55:22.933083+0000 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: created 2024-09-02T09:53:50.141532+0000 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: min_mon_release 19 (squid) 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: election_strategy: 1 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.c 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: 2: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: fsmap 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: mgrmap e13: y(active, since 48s) 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: overall HEALTH_OK 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.297 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:55:28.297 INFO:teuthology.orchestra.run.smithi187.stdout:{"epoch":3,"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","modified":"2024-09-02T09:55:22.933083Z","created":"2024-09-02T09:53:50.141532Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:3300","nonce":0},{"type":"v1","addr":"172.21.15.151:6789","nonce":0}]},"addr":"172.21.15.151:6789/0","public_addr":"172.21.15.151:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:3300","nonce":0},{"type":"v1","addr":"172.21.15.187:6789","nonce":0}]},"addr":"172.21.15.187:6789/0","public_addr":"172.21.15.187:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:3300","nonce":0},{"type":"v1","addr":"172.21.15.162:6789","nonce":0}]},"addr":"172.21.15.162:6789/0","public_addr":"172.21.15.162:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-02T09:55:28.298 INFO:teuthology.orchestra.run.smithi187.stderr:dumped monmap epoch 3 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: mon.a calling monitor election 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: mon.c calling monitor election 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: mon.b calling monitor election 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: monmap epoch 3 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: last_changed 2024-09-02T09:55:22.933083+0000 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: created 2024-09-02T09:53:50.141532+0000 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: min_mon_release 19 (squid) 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: election_strategy: 1 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.c 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: 2: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: fsmap 2024-09-02T09:55:28.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T09:55:28.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: mgrmap e13: y(active, since 48s) 2024-09-02T09:55:28.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: overall HEALTH_OK 2024-09-02T09:55:28.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: mon.a calling monitor election 2024-09-02T09:55:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: mon.c calling monitor election 2024-09-02T09:55:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: mon.b calling monitor election 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: monmap epoch 3 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: last_changed 2024-09-02T09:55:22.933083+0000 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: created 2024-09-02T09:53:50.141532+0000 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: min_mon_release 19 (squid) 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: election_strategy: 1 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: 0: [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] mon.a 2024-09-02T09:55:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: 1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.c 2024-09-02T09:55:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: 2: [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] mon.b 2024-09-02T09:55:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: fsmap 2024-09-02T09:55:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: osdmap e4: 0 total, 0 up, 0 in 2024-09-02T09:55:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: mgrmap e13: y(active, since 48s) 2024-09-02T09:55:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: overall HEALTH_OK 2024-09-02T09:55:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:28.899 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-02T09:55:28.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph config generate-minimal-conf 2024-09-02T09:55:29.160 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:55:29.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:29 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/3863782852' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T09:55:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:29 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:29 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:29 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:29 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:29 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/3863782852' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T09:55:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:29 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:29 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:29 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:29 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:29.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:29 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.187:0/3863782852' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T09:55:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:29 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:29 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:29 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:29 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:29.805 INFO:teuthology.orchestra.run.smithi151.stdout:# minimal ceph.conf for 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:55:29.805 INFO:teuthology.orchestra.run.smithi151.stdout:[global] 2024-09-02T09:55:29.805 INFO:teuthology.orchestra.run.smithi151.stdout: fsid = 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T09:55:29.805 INFO:teuthology.orchestra.run.smithi151.stdout: mon_host = [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] [v2:172.21.15.162:3300/0,v1:172.21.15.162:6789/0] [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] 2024-09-02T09:55:30.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:30 smithi151 ceph-mon[16639]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3100227015' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:30 smithi187 ceph-mon[20072]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3100227015' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:30.586 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-02T09:55:30.587 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:55:30.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.conf 2024-09-02T09:55:30.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:30 smithi162 ceph-mon[20279]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:30.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:30.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:30.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:30.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3100227015' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:30.614 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:55:30.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T09:55:30.682 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:55:30.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.conf 2024-09-02T09:55:30.710 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:55:30.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T09:55:30.776 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:55:30.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.conf 2024-09-02T09:55:30.804 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:55:30.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-02T09:55:30.869 INFO:tasks.cephadm:Adding mgr.y on smithi151 2024-09-02T09:55:30.870 INFO:tasks.cephadm:Adding mgr.x on smithi162 2024-09-02T09:55:30.870 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch apply mgr '2;smithi151=y;smithi162=x' 2024-09-02T09:55:31.144 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:55:31.585 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-02T09:55:31.585 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-02T09:55:31.585 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-02T09:55:31.585 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: Updating smithi187:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: Updating smithi162:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: Updating smithi151:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.586 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.587 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: Reconfiguring mon.a (unknown last config time)... 2024-09-02T09:55:31.587 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:31.587 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T09:55:31.587 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:31.587 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:31 smithi187 ceph-mon[20072]: Reconfiguring daemon mon.a on smithi151 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: Updating smithi187:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: Updating smithi162:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: Updating smithi151:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: Reconfiguring mon.a (unknown last config time)... 2024-09-02T09:55:31.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:31.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T09:55:31.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:31.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:31 smithi162 ceph-mon[20279]: Reconfiguring daemon mon.a on smithi151 2024-09-02T09:55:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: Updating smithi151:/etc/ceph/ceph.conf 2024-09-02T09:55:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: Updating smithi162:/etc/ceph/ceph.conf 2024-09-02T09:55:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: Updating smithi187:/etc/ceph/ceph.conf 2024-09-02T09:55:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: Updating smithi187:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: Updating smithi162:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: Updating smithi151:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/config/ceph.conf 2024-09-02T09:55:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: Reconfiguring mon.a (unknown last config time)... 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:31.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:31 smithi151 ceph-mon[16639]: Reconfiguring daemon mon.a on smithi151 2024-09-02T09:55:31.787 INFO:teuthology.orchestra.run.smithi187.stdout:Scheduled mgr update... 2024-09-02T09:55:32.386 DEBUG:teuthology.orchestra.run.smithi162:mgr.x> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mgr.x.service 2024-09-02T09:55:32.389 DEBUG:tasks.cephadm:set 0 configs 2024-09-02T09:55:32.389 INFO:tasks.cephadm:Deploying OSDs... 2024-09-02T09:55:32.389 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T09:55:32.389 DEBUG:teuthology.orchestra.run.smithi151:> dd if=/scratch_devs of=/dev/stdout 2024-09-02T09:55:32.405 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-02T09:55:32.405 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_1 2024-09-02T09:55:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-02T09:55:32.462 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:32.463 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-02T09:55:32.463 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:32.463 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:32.463 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-02 09:54:53.112275233 +0000 2024-09-02T09:55:32.463 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-02 09:50:24.117748568 +0000 2024-09-02T09:55:32.463 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-02 09:50:24.117748568 +0000 2024-09-02T09:55:32.463 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-09-02 09:50:24.117748568 +0000 2024-09-02T09:55:32.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-02T09:55:32.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:32 smithi151 ceph-mon[16639]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:32.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:32.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:32.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:32.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-02T09:55:32.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-02T09:55:32.526 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:32.528 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-02T09:55:32.528 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-02T09:55:32.528 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.00021003 s, 2.4 MB/s 2024-09-02T09:55:32.529 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-02T09:55:32.586 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_2 2024-09-02T09:55:32.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:32 smithi162 ceph-mon[20279]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:32.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:32.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-02T09:55:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-02T09:55:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:32.642 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-02T09:55:32.642 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:32.642 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-02T09:55:32.642 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:32.642 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:32.642 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-02 09:54:53.145274833 +0000 2024-09-02T09:55:32.642 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-02 09:50:24.443746610 +0000 2024-09-02T09:55:32.643 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-02 09:50:24.443746610 +0000 2024-09-02T09:55:32.643 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-09-02 09:50:24.443746610 +0000 2024-09-02T09:55:32.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-02T09:55:32.705 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-02T09:55:32.705 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-02T09:55:32.706 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000196059 s, 2.6 MB/s 2024-09-02T09:55:32.707 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-02T09:55:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:32 smithi187 ceph-mon[20072]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-02T09:55:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-02T09:55:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:32.763 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_3 2024-09-02T09:55:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-02T09:55:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:32.820 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-02T09:55:32.820 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:32.820 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:32.820 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-02 09:54:53.179274421 +0000 2024-09-02T09:55:32.820 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-02 09:50:24.761744698 +0000 2024-09-02T09:55:32.820 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-02 09:50:24.761744698 +0000 2024-09-02T09:55:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-09-02 09:50:24.761744698 +0000 2024-09-02T09:55:32.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-02T09:55:32.883 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-02T09:55:32.883 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-02T09:55:32.883 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000250915 s, 2.0 MB/s 2024-09-02T09:55:32.884 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-02T09:55:32.941 DEBUG:teuthology.orchestra.run.smithi151:> stat /dev/vg_nvme/lv_4 2024-09-02T09:55:32.996 INFO:teuthology.orchestra.run.smithi151.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-02T09:55:32.996 INFO:teuthology.orchestra.run.smithi151.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:32.996 INFO:teuthology.orchestra.run.smithi151.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-02T09:55:32.996 INFO:teuthology.orchestra.run.smithi151.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:32.996 INFO:teuthology.orchestra.run.smithi151.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:32.996 INFO:teuthology.orchestra.run.smithi151.stdout:Access: 2024-09-02 09:54:53.213274009 +0000 2024-09-02T09:55:32.996 INFO:teuthology.orchestra.run.smithi151.stdout:Modify: 2024-09-02 09:50:25.077742799 +0000 2024-09-02T09:55:32.996 INFO:teuthology.orchestra.run.smithi151.stdout:Change: 2024-09-02 09:50:25.077742799 +0000 2024-09-02T09:55:32.997 INFO:teuthology.orchestra.run.smithi151.stdout: Birth: 2024-09-02 09:50:25.077742799 +0000 2024-09-02T09:55:32.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-02T09:55:33.060 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records in 2024-09-02T09:55:33.061 INFO:teuthology.orchestra.run.smithi151.stderr:1+0 records out 2024-09-02T09:55:33.061 INFO:teuthology.orchestra.run.smithi151.stderr:512 bytes copied, 0.000173135 s, 3.0 MB/s 2024-09-02T09:55:33.062 DEBUG:teuthology.orchestra.run.smithi151:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-02T09:55:33.118 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T09:55:33.118 DEBUG:teuthology.orchestra.run.smithi162:> dd if=/scratch_devs of=/dev/stdout 2024-09-02T09:55:33.138 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-02T09:55:33.138 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_1 2024-09-02T09:55:33.196 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-02T09:55:33.196 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:33.197 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-02T09:55:33.197 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:33.197 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:33.197 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-02 09:55:22.378428621 +0000 2024-09-02T09:55:33.197 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-02 09:50:25.329468342 +0000 2024-09-02T09:55:33.197 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-02 09:50:25.329468342 +0000 2024-09-02T09:55:33.197 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-02 09:50:25.329468342 +0000 2024-09-02T09:55:33.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-02T09:55:33.265 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-02T09:55:33.265 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-02T09:55:33.265 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000290496 s, 1.8 MB/s 2024-09-02T09:55:33.267 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-02T09:55:33.325 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_2 2024-09-02T09:55:33.382 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-02T09:55:33.382 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:33.382 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-02T09:55:33.382 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:33.382 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:33.382 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-02 09:55:22.378428621 +0000 2024-09-02T09:55:33.382 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-02 09:50:25.656461562 +0000 2024-09-02T09:55:33.382 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-02 09:50:25.656461562 +0000 2024-09-02T09:55:33.382 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-02 09:50:25.656461562 +0000 2024-09-02T09:55:33.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-02T09:55:33.446 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-02T09:55:33.446 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-02T09:55:33.446 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000215415 s, 2.4 MB/s 2024-09-02T09:55:33.448 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-02T09:55:33.504 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_3 2024-09-02T09:55:33.560 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-02T09:55:33.560 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:33.560 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-02T09:55:33.560 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:33.560 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:33.560 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-02 09:55:22.379428601 +0000 2024-09-02T09:55:33.560 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-02 09:50:26.005454326 +0000 2024-09-02T09:55:33.561 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-02 09:50:26.005454326 +0000 2024-09-02T09:55:33.561 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-02 09:50:26.005454326 +0000 2024-09-02T09:55:33.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-02T09:55:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:33 smithi162 ceph-mon[20279]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi151=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:55:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:33 smithi162 ceph-mon[20279]: Saving service mgr spec with placement smithi151=y;smithi162=x;count:2 2024-09-02T09:55:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:33 smithi162 ceph-mon[20279]: Reconfiguring mgr.y (unknown last config time)... 2024-09-02T09:55:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:33 smithi162 ceph-mon[20279]: Reconfiguring daemon mgr.y on smithi151 2024-09-02T09:55:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T09:55:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:33.618 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-02T09:55:33.618 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-02T09:55:33.619 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000199613 s, 2.6 MB/s 2024-09-02T09:55:33.620 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-02T09:55:33.676 DEBUG:teuthology.orchestra.run.smithi162:> stat /dev/vg_nvme/lv_4 2024-09-02T09:55:33.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:33 smithi151 ceph-mon[16639]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi151=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:55:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:33 smithi151 ceph-mon[16639]: Saving service mgr spec with placement smithi151=y;smithi162=x;count:2 2024-09-02T09:55:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:33 smithi151 ceph-mon[16639]: Reconfiguring mgr.y (unknown last config time)... 2024-09-02T09:55:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:33 smithi151 ceph-mon[16639]: Reconfiguring daemon mgr.y on smithi151 2024-09-02T09:55:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T09:55:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-02T09:55:33.732 INFO:teuthology.orchestra.run.smithi162.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:33.732 INFO:teuthology.orchestra.run.smithi162.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-02T09:55:33.732 INFO:teuthology.orchestra.run.smithi162.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:33.732 INFO:teuthology.orchestra.run.smithi162.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:33.732 INFO:teuthology.orchestra.run.smithi162.stdout:Access: 2024-09-02 09:55:22.379428601 +0000 2024-09-02T09:55:33.733 INFO:teuthology.orchestra.run.smithi162.stdout:Modify: 2024-09-02 09:50:26.346447256 +0000 2024-09-02T09:55:33.733 INFO:teuthology.orchestra.run.smithi162.stdout:Change: 2024-09-02 09:50:26.346447256 +0000 2024-09-02T09:55:33.733 INFO:teuthology.orchestra.run.smithi162.stdout: Birth: 2024-09-02 09:50:26.346447256 +0000 2024-09-02T09:55:33.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-02T09:55:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:33 smithi187 ceph-mon[20072]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi151=y;smithi162=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:55:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:33 smithi187 ceph-mon[20072]: Saving service mgr spec with placement smithi151=y;smithi162=x;count:2 2024-09-02T09:55:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:33 smithi187 ceph-mon[20072]: Reconfiguring mgr.y (unknown last config time)... 2024-09-02T09:55:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:33 smithi187 ceph-mon[20072]: Reconfiguring daemon mgr.y on smithi151 2024-09-02T09:55:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:33.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:33.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T09:55:33.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:33.796 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records in 2024-09-02T09:55:33.796 INFO:teuthology.orchestra.run.smithi162.stderr:1+0 records out 2024-09-02T09:55:33.796 INFO:teuthology.orchestra.run.smithi162.stderr:512 bytes copied, 0.000237065 s, 2.2 MB/s 2024-09-02T09:55:33.797 DEBUG:teuthology.orchestra.run.smithi162:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-02T09:55:33.855 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T09:55:33.855 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2024-09-02T09:55:33.876 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-02T09:55:33.876 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2024-09-02T09:55:33.934 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-02T09:55:33.935 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:33.935 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-02T09:55:33.935 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:33.935 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:33.935 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-02 09:55:13.278271878 +0000 2024-09-02T09:55:33.935 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-02 09:50:24.744377991 +0000 2024-09-02T09:55:33.935 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-02 09:50:24.744377991 +0000 2024-09-02T09:55:33.935 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-02 09:50:24.744377991 +0000 2024-09-02T09:55:33.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-02T09:55:34.003 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-02T09:55:34.003 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-02T09:55:34.003 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000295618 s, 1.7 MB/s 2024-09-02T09:55:34.004 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-02T09:55:34.063 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2024-09-02T09:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-02T09:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-02T09:55:34.119 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-02 09:55:13.279271863 +0000 2024-09-02T09:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-02 09:50:25.069374532 +0000 2024-09-02T09:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-02 09:50:25.069374532 +0000 2024-09-02T09:55:34.120 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-02 09:50:25.069374532 +0000 2024-09-02T09:55:34.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-02T09:55:34.185 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-02T09:55:34.185 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-02T09:55:34.185 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000232409 s, 2.2 MB/s 2024-09-02T09:55:34.186 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-02T09:55:34.243 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2024-09-02T09:55:34.299 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-02T09:55:34.299 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:34.299 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-02T09:55:34.299 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:34.299 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:34.299 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-02 09:55:13.279271863 +0000 2024-09-02T09:55:34.299 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-02 09:50:25.377371255 +0000 2024-09-02T09:55:34.299 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-02 09:50:25.377371255 +0000 2024-09-02T09:55:34.299 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-02 09:50:25.377371255 +0000 2024-09-02T09:55:34.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-02T09:55:34.363 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-02T09:55:34.363 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-02T09:55:34.363 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000200049 s, 2.6 MB/s 2024-09-02T09:55:34.365 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-02T09:55:34.421 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2024-09-02T09:55:34.477 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-02T09:55:34.477 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-02T09:55:34.478 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-02T09:55:34.478 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-02T09:55:34.478 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-09-02T09:55:34.478 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-09-02 09:55:13.279271863 +0000 2024-09-02T09:55:34.478 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-09-02 09:50:25.690367924 +0000 2024-09-02T09:55:34.478 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-09-02 09:50:25.690367924 +0000 2024-09-02T09:55:34.478 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-09-02 09:50:25.690367924 +0000 2024-09-02T09:55:34.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-02T09:55:34.541 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-09-02T09:55:34.541 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-09-02T09:55:34.541 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000239237 s, 2.1 MB/s 2024-09-02T09:55:34.542 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-02T09:55:34.599 INFO:tasks.cephadm:Deploying osd.0 on smithi151 with /dev/vg_nvme/lv_4... 2024-09-02T09:55:34.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-02T09:55:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:34 smithi162 ceph-mon[20279]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:34 smithi162 ceph-mon[20279]: Reconfiguring mon.b (monmap changed)... 2024-09-02T09:55:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:34 smithi162 ceph-mon[20279]: Reconfiguring daemon mon.b on smithi162 2024-09-02T09:55:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:34 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:34 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:34 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:34 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T09:55:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:34 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:34.854 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:55:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:34 smithi151 ceph-mon[16639]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:34 smithi151 ceph-mon[16639]: Reconfiguring mon.b (monmap changed)... 2024-09-02T09:55:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:34 smithi151 ceph-mon[16639]: Reconfiguring daemon mon.b on smithi162 2024-09-02T09:55:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:34 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:34 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:34 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:34 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T09:55:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:34 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:34.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:34 smithi187 ceph-mon[20072]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:34.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:34 smithi187 ceph-mon[20072]: Reconfiguring mon.b (monmap changed)... 2024-09-02T09:55:34.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:34 smithi187 ceph-mon[20072]: Reconfiguring daemon mon.b on smithi162 2024-09-02T09:55:34.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:34 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:34.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:34 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:34.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:34 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-09-02T09:55:34.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:34 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-09-02T09:55:34.950 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:34 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:35 smithi151 ceph-mon[16639]: Reconfiguring mon.c (monmap changed)... 2024-09-02T09:55:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:35 smithi151 ceph-mon[16639]: Reconfiguring daemon mon.c on smithi187 2024-09-02T09:55:35.968 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:35 smithi187 ceph-mon[20072]: Reconfiguring mon.c (monmap changed)... 2024-09-02T09:55:35.968 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:35 smithi187 ceph-mon[20072]: Reconfiguring daemon mon.c on smithi187 2024-09-02T09:55:36.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:35 smithi162 ceph-mon[20279]: Reconfiguring mon.c (monmap changed)... 2024-09-02T09:55:36.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:35 smithi162 ceph-mon[20279]: Reconfiguring daemon mon.c on smithi187 2024-09-02T09:55:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-02T09:55:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-02T09:55:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-02T09:55:36.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:37.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:37.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-02T09:55:37.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-02T09:55:37.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-02T09:55:37.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:37.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:37.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:37.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:37.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:37.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:37.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:37.047 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:37.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-09-02T09:55:37.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-02T09:55:37.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-09-02T09:55:37.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:37.335 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:55:37.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_4 2024-09-02T09:55:37.594 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 09:55:37 smithi162 systemd[1]: Starting Ceph mgr.x for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T09:55:37.606 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:55:37.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:37 smithi151 ceph-mon[16639]: Deploying daemon mgr.x on smithi162 2024-09-02T09:55:37.956 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:37 smithi162 ceph-mon[20279]: Deploying daemon mgr.x on smithi162 2024-09-02T09:55:37.956 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 09:55:37 smithi162 podman[21630]: 2024-09-02 09:55:37.732333289 +0000 UTC m=+0.105907204 container create 4c2ca83c89da3997c37a50376989e34c9dd89a0352060f75947cbedcbafdfd33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-x, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T09:55:37.957 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 09:55:37 smithi162 podman[21630]: 2024-09-02 09:55:37.642363646 +0000 UTC m=+0.015937519 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T09:55:37.957 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 09:55:37 smithi162 podman[21630]: 2024-09-02 09:55:37.926217339 +0000 UTC m=+0.299791205 container init 4c2ca83c89da3997c37a50376989e34c9dd89a0352060f75947cbedcbafdfd33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-02T09:55:37.957 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 09:55:37 smithi162 podman[21630]: 2024-09-02 09:55:37.928948964 +0000 UTC m=+0.302522831 container start 4c2ca83c89da3997c37a50376989e34c9dd89a0352060f75947cbedcbafdfd33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-02T09:55:37.957 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 09:55:37 smithi162 ceph-mgr[21649]: -- 172.21.15.162:0/691977188 <== mon.2 v2:172.21.15.162:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e3f92a10e0 con 0x55e3f927f400 2024-09-02T09:55:38.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:37 smithi187 ceph-mon[20072]: Deploying daemon mgr.x on smithi162 2024-09-02T09:55:38.344 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 09:55:38 smithi162 bash[21630]: 4c2ca83c89da3997c37a50376989e34c9dd89a0352060f75947cbedcbafdfd33 2024-09-02T09:55:38.345 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 09:55:38 smithi162 systemd[1]: Started Ceph mgr.x for 1530c7be-6911-11ef-bcd4-c7b262605968. 2024-09-02T09:55:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:38 smithi151 ceph-mon[16639]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:38 smithi151 ceph-mon[16639]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:55:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:55:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:55:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:38 smithi187 ceph-mon[20072]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:38 smithi187 ceph-mon[20072]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:55:39.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:55:39.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:55:39.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:39.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:38 smithi162 ceph-mon[20279]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:39.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:39.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:39.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:39.021 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:39.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:39.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:38 smithi162 ceph-mon[20279]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:55:39.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:55:39.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:55:39.022 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:40.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:40.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:40.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:40.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:40.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:40.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:40.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/366049579' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29"} : dispatch 2024-09-02T09:55:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/366049579' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29"}]': finished 2024-09-02T09:55:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:41 smithi151 ceph-mon[16639]: osdmap e5: 1 total, 0 up, 1 in 2024-09-02T09:55:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:41 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/493914056' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:55:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/366049579' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29"} : dispatch 2024-09-02T09:55:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/366049579' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29"}]': finished 2024-09-02T09:55:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:41 smithi187 ceph-mon[20072]: osdmap e5: 1 total, 0 up, 1 in 2024-09-02T09:55:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:41 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/493914056' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:55:41.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/366049579' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29"} : dispatch 2024-09-02T09:55:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/366049579' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29"}]': finished 2024-09-02T09:55:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:41 smithi162 ceph-mon[20279]: osdmap e5: 1 total, 0 up, 1 in 2024-09-02T09:55:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:41 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/493914056' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:55:42.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:42 smithi162 ceph-mon[20279]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:42.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:42 smithi151 ceph-mon[16639]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:42 smithi187 ceph-mon[20072]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:43.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:43 smithi151 ceph-mon[16639]: Standby manager daemon x started 2024-09-02T09:55:43.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:43 smithi151 ceph-mon[16639]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-02T09:55:43.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:43 smithi151 ceph-mon[16639]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-02T09:55:43.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:43 smithi151 ceph-mon[16639]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-02T09:55:43.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:43 smithi151 ceph-mon[16639]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-02T09:55:43.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:43 smithi162 ceph-mon[20279]: Standby manager daemon x started 2024-09-02T09:55:43.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:43 smithi162 ceph-mon[20279]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-02T09:55:43.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:43 smithi162 ceph-mon[20279]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-02T09:55:43.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:43 smithi162 ceph-mon[20279]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-02T09:55:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:43 smithi162 ceph-mon[20279]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-02T09:55:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:43 smithi187 ceph-mon[20072]: Standby manager daemon x started 2024-09-02T09:55:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:43 smithi187 ceph-mon[20072]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-09-02T09:55:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:43 smithi187 ceph-mon[20072]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-09-02T09:55:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:43 smithi187 ceph-mon[20072]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-09-02T09:55:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:43 smithi187 ceph-mon[20072]: from='mgr.? 172.21.15.162:0/1099250871' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-09-02T09:55:44.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:44 smithi151 ceph-mon[16639]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:44.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:44 smithi151 ceph-mon[16639]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-02T09:55:44.810 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:44 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-02T09:55:44.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:44 smithi162 ceph-mon[20279]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:44 smithi162 ceph-mon[20279]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-02T09:55:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:44 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-02T09:55:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:44 smithi187 ceph-mon[20072]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:44 smithi187 ceph-mon[20072]: mgrmap e14: y(active, since 64s), standbys: x 2024-09-02T09:55:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:44 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-09-02T09:55:46.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:46 smithi151 ceph-mon[16639]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:46.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:46 smithi162 ceph-mon[20279]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:46 smithi187 ceph-mon[20072]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:47.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-02T09:55:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-02T09:55:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-09-02T09:55:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:48.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:48 smithi151 ceph-mon[16639]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:48.772 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:48 smithi151 ceph-mon[16639]: Deploying daemon osd.0 on smithi151 2024-09-02T09:55:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:48 smithi187 ceph-mon[20072]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:48 smithi187 ceph-mon[20072]: Deploying daemon osd.0 on smithi151 2024-09-02T09:55:49.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:48 smithi162 ceph-mon[20279]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:48 smithi162 ceph-mon[20279]: Deploying daemon osd.0 on smithi151 2024-09-02T09:55:50.897 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:50 smithi162 ceph-mon[20279]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:50 smithi151 ceph-mon[16639]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:50 smithi187 ceph-mon[20072]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:52.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:52 smithi151 ceph-mon[16639]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:52 smithi187 ceph-mon[20072]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:52 smithi162 ceph-mon[20279]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:53.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:53.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:54.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:54.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:54.761 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:54 smithi151 ceph-mon[16639]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:54 smithi187 ceph-mon[20072]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:55.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:54 smithi162 ceph-mon[20279]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:56.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:56 smithi151 ceph-mon[16639]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:56.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:56.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:56.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:56.628 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:56.628 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:56 smithi187 ceph-mon[20072]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:56.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:56 smithi162 ceph-mon[20279]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:55:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:55:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:56.920 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 0 on host 'smithi151' 2024-09-02T09:55:57.527 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:57.527 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:57.527 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:57.554 DEBUG:teuthology.orchestra.run.smithi151:osd.0> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.0.service 2024-09-02T09:55:57.557 INFO:tasks.cephadm:Deploying osd.1 on smithi151 with /dev/vg_nvme/lv_3... 2024-09-02T09:55:57.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-02T09:55:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:57.778 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:55:57 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0[30637]: 2024-09-02T09:55:57.774+0000 7f07e1d85640 -1 osd.0 0 log_to_monitors true 2024-09-02T09:55:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:55:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:55:58.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:58 smithi151 ceph-mon[16639]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:58 smithi151 ceph-mon[16639]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-02T09:55:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:58 smithi187 ceph-mon[20072]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:58 smithi187 ceph-mon[20072]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-02T09:55:58.792 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:55:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:58 smithi162 ceph-mon[20279]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:55:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:58 smithi162 ceph-mon[20279]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-09-02T09:55:59.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:59 smithi151 ceph-mon[16639]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-02T09:55:59.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:59 smithi151 ceph-mon[16639]: osdmap e6: 1 total, 0 up, 1 in 2024-09-02T09:55:59.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:59 smithi151 ceph-mon[16639]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:55:59.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:59.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:59 smithi151 ceph-mon[16639]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:55:59.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:59 smithi151 ceph-mon[16639]: osdmap e7: 1 total, 0 up, 1 in 2024-09-02T09:55:59.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:59.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:55:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:59.666 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:55:59 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0[30637]: 2024-09-02T09:55:59.355+0000 7f07ddd08640 -1 osd.0 0 waiting for initial osdmap 2024-09-02T09:55:59.666 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:55:59 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0[30637]: 2024-09-02T09:55:59.358+0000 7f07d9b2f640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:55:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:59 smithi187 ceph-mon[20072]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-02T09:55:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:59 smithi187 ceph-mon[20072]: osdmap e6: 1 total, 0 up, 1 in 2024-09-02T09:55:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:59 smithi187 ceph-mon[20072]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:55:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:59 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:59 smithi187 ceph-mon[20072]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:55:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:59 smithi187 ceph-mon[20072]: osdmap e7: 1 total, 0 up, 1 in 2024-09-02T09:55:59.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:59 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:59.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:55:59 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:59.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:59 smithi162 ceph-mon[20279]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-02T09:55:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:59 smithi162 ceph-mon[20279]: osdmap e6: 1 total, 0 up, 1 in 2024-09-02T09:55:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:59 smithi162 ceph-mon[20279]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:55:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:59 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:59 smithi162 ceph-mon[20279]: from='osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:55:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:59 smithi162 ceph-mon[20279]: osdmap e7: 1 total, 0 up, 1 in 2024-09-02T09:55:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:59 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:55:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:55:59 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:56:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:00 smithi151 ceph-mon[16639]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:56:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:00 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:56:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:00 smithi151 ceph-mon[16639]: osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772] boot 2024-09-02T09:56:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:00 smithi151 ceph-mon[16639]: osdmap e8: 1 total, 1 up, 1 in 2024-09-02T09:56:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:00 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:56:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:00 smithi187 ceph-mon[20072]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:56:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:00 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:56:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:00 smithi187 ceph-mon[20072]: osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772] boot 2024-09-02T09:56:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:00 smithi187 ceph-mon[20072]: osdmap e8: 1 total, 1 up, 1 in 2024-09-02T09:56:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:00 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:56:01.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:00 smithi162 ceph-mon[20279]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:56:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:00 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:56:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:00 smithi162 ceph-mon[20279]: osd.0 [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772] boot 2024-09-02T09:56:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:00 smithi162 ceph-mon[20279]: osdmap e8: 1 total, 1 up, 1 in 2024-09-02T09:56:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:00 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-09-02T09:56:01.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:01 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:56:01.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:01 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:56:01.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:01.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:01.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:01.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:01.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:01.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:01.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:02.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:01 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:56:02.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:01 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:56:02.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:02.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:02.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:02.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:02.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:02.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:02.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:02.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:01 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:56:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:01 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:56:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:02.463 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:56:02.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_3 2024-09-02T09:56:02.731 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:56:02.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:02 smithi151 ceph-mon[16639]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:56:02.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:02 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi151 2024-09-02T09:56:02.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:02 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi151 to 17088M 2024-09-02T09:56:02.839 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:02 smithi151 ceph-mon[16639]: osdmap e9: 1 total, 1 up, 1 in 2024-09-02T09:56:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:02 smithi187 ceph-mon[20072]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:56:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:02 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi151 2024-09-02T09:56:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:02 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi151 to 17088M 2024-09-02T09:56:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:02 smithi187 ceph-mon[20072]: osdmap e9: 1 total, 1 up, 1 in 2024-09-02T09:56:03.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:02 smithi162 ceph-mon[20279]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-02T09:56:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:02 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi151 2024-09-02T09:56:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:02 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi151 to 17088M 2024-09-02T09:56:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:02 smithi162 ceph-mon[20279]: osdmap e9: 1 total, 1 up, 1 in 2024-09-02T09:56:03.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:56:03.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:56:03.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:56:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:56:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:04.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:56:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:56:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:04.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:04 smithi151 ceph-mon[16639]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:04.822 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:04 smithi151 ceph-mon[16639]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:56:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:04 smithi187 ceph-mon[20072]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:04 smithi187 ceph-mon[20072]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:56:05.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:04 smithi162 ceph-mon[20279]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:05.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:04 smithi162 ceph-mon[20279]: from='client.14253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:56:05.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:05 smithi151 ceph-mon[16639]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:05 smithi187 ceph-mon[20072]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:05 smithi162 ceph-mon[20279]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4285400023' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1"} : dispatch 2024-09-02T09:56:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4285400023' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1"}]': finished 2024-09-02T09:56:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:06 smithi187 ceph-mon[20072]: osdmap e10: 2 total, 1 up, 2 in 2024-09-02T09:56:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2864167076' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:56:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4285400023' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1"} : dispatch 2024-09-02T09:56:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4285400023' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1"}]': finished 2024-09-02T09:56:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:06 smithi162 ceph-mon[20279]: osdmap e10: 2 total, 1 up, 2 in 2024-09-02T09:56:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2864167076' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:56:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4285400023' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1"} : dispatch 2024-09-02T09:56:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4285400023' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1"}]': finished 2024-09-02T09:56:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:06 smithi151 ceph-mon[16639]: osdmap e10: 2 total, 1 up, 2 in 2024-09-02T09:56:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2864167076' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:56:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:07 smithi187 ceph-mon[20072]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:07 smithi162 ceph-mon[20279]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:08.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:07 smithi151 ceph-mon[16639]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:10.272 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:10 smithi151 ceph-mon[16639]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:10.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:10 smithi162 ceph-mon[20279]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:10.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:10 smithi187 ceph-mon[20072]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:12.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:12 smithi151 ceph-mon[16639]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:12.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:12 smithi162 ceph-mon[20279]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:12 smithi187 ceph-mon[20072]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:14 smithi162 ceph-mon[20279]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:14 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-02T09:56:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:14 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:14 smithi162 ceph-mon[20279]: Deploying daemon osd.1 on smithi151 2024-09-02T09:56:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:14 smithi151 ceph-mon[16639]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:14 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-02T09:56:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:14 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:14 smithi151 ceph-mon[16639]: Deploying daemon osd.1 on smithi151 2024-09-02T09:56:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:14 smithi187 ceph-mon[20072]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-09-02T09:56:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:14 smithi187 ceph-mon[20072]: Deploying daemon osd.1 on smithi151 2024-09-02T09:56:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:16 smithi151 ceph-mon[16639]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:16.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:16 smithi162 ceph-mon[20279]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:16 smithi187 ceph-mon[20072]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:18.314 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:18 smithi151 ceph-mon[16639]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:18.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:18 smithi162 ceph-mon[20279]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:18 smithi187 ceph-mon[20072]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:19.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:19.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:19.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:20 smithi151 ceph-mon[16639]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:20 smithi187 ceph-mon[20072]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:21.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:20 smithi162 ceph-mon[20279]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:22.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:22 smithi151 ceph-mon[16639]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:22.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:22.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:22.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:22.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:22.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:22.497 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 1 on host 'smithi151' 2024-09-02T09:56:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:22 smithi187 ceph-mon[20072]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:22 smithi162 ceph-mon[20279]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:23.116 DEBUG:teuthology.orchestra.run.smithi151:osd.1> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.1.service 2024-09-02T09:56:23.220 INFO:tasks.cephadm:Deploying osd.2 on smithi151 with /dev/vg_nvme/lv_2... 2024-09-02T09:56:23.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-02T09:56:23.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:23 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:23.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:23 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:23.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:23 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:23 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:23 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:23 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:23 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:23 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:23 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:23.938 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:23 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1[36292]: 2024-09-02T09:56:23.854+0000 7f52bc4b2640 -1 osd.1 0 log_to_monitors true 2024-09-02T09:56:24.524 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:24 smithi151 ceph-mon[16639]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:24.524 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:24 smithi151 ceph-mon[16639]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-02T09:56:24.728 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:56:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:24 smithi187 ceph-mon[20072]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:24 smithi187 ceph-mon[20072]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-02T09:56:24.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:24 smithi162 ceph-mon[20279]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:24 smithi162 ceph-mon[20279]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-09-02T09:56:25.688 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:25 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1[36292]: 2024-09-02T09:56:25.502+0000 7f52b8435640 -1 osd.1 0 waiting for initial osdmap 2024-09-02T09:56:25.688 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:25 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1[36292]: 2024-09-02T09:56:25.506+0000 7f52b425c640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:56:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:25 smithi151 ceph-mon[16639]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-02T09:56:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:25 smithi151 ceph-mon[16639]: osdmap e11: 2 total, 1 up, 2 in 2024-09-02T09:56:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:25 smithi151 ceph-mon[16639]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:56:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:25 smithi151 ceph-mon[16639]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:56:25.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:25 smithi151 ceph-mon[16639]: osdmap e12: 2 total, 1 up, 2 in 2024-09-02T09:56:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:25 smithi162 ceph-mon[20279]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-02T09:56:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:25 smithi162 ceph-mon[20279]: osdmap e11: 2 total, 1 up, 2 in 2024-09-02T09:56:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:25 smithi162 ceph-mon[20279]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:56:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:25 smithi162 ceph-mon[20279]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:56:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:25 smithi162 ceph-mon[20279]: osdmap e12: 2 total, 1 up, 2 in 2024-09-02T09:56:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:25 smithi187 ceph-mon[20072]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-02T09:56:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:25 smithi187 ceph-mon[20072]: osdmap e11: 2 total, 1 up, 2 in 2024-09-02T09:56:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:25 smithi187 ceph-mon[20072]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:56:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:25 smithi187 ceph-mon[20072]: from='osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:56:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:25 smithi187 ceph-mon[20072]: osdmap e12: 2 total, 1 up, 2 in 2024-09-02T09:56:26.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:26.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:26.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:26.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:26.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:26.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:26.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:26.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:26.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:26.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:26.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:26.788 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:26.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:26.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:26.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:26.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:27.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:27.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:27.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:27.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:27.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:56:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:56:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:27 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi151 2024-09-02T09:56:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:27 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi151 to 8544M 2024-09-02T09:56:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:27 smithi162 ceph-mon[20279]: osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866] boot 2024-09-02T09:56:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:27 smithi162 ceph-mon[20279]: osdmap e13: 2 total, 2 up, 2 in 2024-09-02T09:56:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:27 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:27.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:56:27.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:56:27.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:27 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi151 2024-09-02T09:56:27.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:27 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi151 to 8544M 2024-09-02T09:56:27.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:27 smithi151 ceph-mon[16639]: osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866] boot 2024-09-02T09:56:27.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:27 smithi151 ceph-mon[16639]: osdmap e13: 2 total, 2 up, 2 in 2024-09-02T09:56:27.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:27 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:27.893 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:56:27.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_2 2024-09-02T09:56:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:56:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:56:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:27 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi151 2024-09-02T09:56:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:27 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi151 to 8544M 2024-09-02T09:56:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:27 smithi187 ceph-mon[20072]: osd.1 [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866] boot 2024-09-02T09:56:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:27 smithi187 ceph-mon[20072]: osdmap e13: 2 total, 2 up, 2 in 2024-09-02T09:56:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:27 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-09-02T09:56:28.165 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:56:28.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:28 smithi151 ceph-mon[16639]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:28.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:28 smithi151 ceph-mon[16639]: osdmap e14: 2 total, 2 up, 2 in 2024-09-02T09:56:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:28 smithi162 ceph-mon[20279]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:28 smithi162 ceph-mon[20279]: osdmap e14: 2 total, 2 up, 2 in 2024-09-02T09:56:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:28 smithi187 ceph-mon[20072]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-02T09:56:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:28 smithi187 ceph-mon[20072]: osdmap e14: 2 total, 2 up, 2 in 2024-09-02T09:56:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:29 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:56:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:29 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:56:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:29 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:29 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:56:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:29 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:56:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:29 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:30.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:29 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:56:30.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:29 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:56:30.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:29 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:30.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:30 smithi151 ceph-mon[16639]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:56:30.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:30 smithi151 ceph-mon[16639]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:30 smithi187 ceph-mon[20072]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:56:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:30 smithi187 ceph-mon[20072]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:31.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:30 smithi162 ceph-mon[20279]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:56:31.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:30 smithi162 ceph-mon[20279]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:31.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3055117586' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48"} : dispatch 2024-09-02T09:56:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3055117586' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48"}]': finished 2024-09-02T09:56:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:31 smithi151 ceph-mon[16639]: osdmap e15: 3 total, 2 up, 3 in 2024-09-02T09:56:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3055117586' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48"} : dispatch 2024-09-02T09:56:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3055117586' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48"}]': finished 2024-09-02T09:56:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:31 smithi187 ceph-mon[20072]: osdmap e15: 3 total, 2 up, 3 in 2024-09-02T09:56:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3055117586' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48"} : dispatch 2024-09-02T09:56:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3055117586' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48"}]': finished 2024-09-02T09:56:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:31 smithi162 ceph-mon[20279]: osdmap e15: 3 total, 2 up, 3 in 2024-09-02T09:56:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:32.882 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:32 smithi151 ceph-mon[16639]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:32.883 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1041587777' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:56:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:32 smithi187 ceph-mon[20072]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1041587777' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:56:33.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:32 smithi162 ceph-mon[20279]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:33.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1041587777' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:56:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:33 smithi187 ceph-mon[20072]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:34.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:33 smithi162 ceph-mon[20279]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:33 smithi151 ceph-mon[16639]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:36.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:36 smithi162 ceph-mon[20279]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:36.369 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:36 smithi151 ceph-mon[16639]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:36 smithi187 ceph-mon[20072]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:38.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:38 smithi151 ceph-mon[16639]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:38.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:38 smithi162 ceph-mon[20279]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:38 smithi187 ceph-mon[20072]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:39.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-02T09:56:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:39 smithi162 ceph-mon[20279]: Deploying daemon osd.2 on smithi151 2024-09-02T09:56:39.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-02T09:56:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:39 smithi151 ceph-mon[16639]: Deploying daemon osd.2 on smithi151 2024-09-02T09:56:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-09-02T09:56:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:39 smithi187 ceph-mon[20072]: Deploying daemon osd.2 on smithi151 2024-09-02T09:56:40.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:40 smithi151 ceph-mon[16639]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:40 smithi187 ceph-mon[20072]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:41.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:40 smithi162 ceph-mon[20279]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:42.929 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:42 smithi151 ceph-mon[16639]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:42 smithi187 ceph-mon[20072]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:43.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:42 smithi162 ceph-mon[20279]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:43.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:43 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:43 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:43 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:43 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:43 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:43 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:44.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:43 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:43 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:43 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:44 smithi151 ceph-mon[16639]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:44 smithi187 ceph-mon[20072]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:45.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:44 smithi162 ceph-mon[20279]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:46.830 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:46 smithi151 ceph-mon[16639]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:46.868 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 2 on host 'smithi151' 2024-09-02T09:56:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:46 smithi187 ceph-mon[20072]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:46 smithi162 ceph-mon[20279]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:47.586 DEBUG:teuthology.orchestra.run.smithi151:osd.2> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.2.service 2024-09-02T09:56:47.588 INFO:tasks.cephadm:Deploying osd.3 on smithi151 with /dev/vg_nvme/lv_1... 2024-09-02T09:56:47.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-02T09:56:47.883 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:56:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:47 smithi151 ceph-mon[16639]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.121 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:47 smithi187 ceph-mon[20072]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:47 smithi162 ceph-mon[20279]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:48.870 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:48 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2[42108]: 2024-09-02T09:56:48.529+0000 7fb384e95640 -1 osd.2 0 log_to_monitors true 2024-09-02T09:56:49.149 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:48 smithi151 ceph-mon[16639]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-02T09:56:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:48 smithi187 ceph-mon[20072]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-02T09:56:49.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:48 smithi162 ceph-mon[20279]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-09-02T09:56:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:49 smithi151 ceph-mon[16639]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-02T09:56:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:49 smithi151 ceph-mon[16639]: osdmap e16: 3 total, 2 up, 3 in 2024-09-02T09:56:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:49 smithi151 ceph-mon[16639]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:56:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:49 smithi151 ceph-mon[16639]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:49 smithi187 ceph-mon[20072]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-02T09:56:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:49 smithi187 ceph-mon[20072]: osdmap e16: 3 total, 2 up, 3 in 2024-09-02T09:56:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:49 smithi187 ceph-mon[20072]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:56:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:49 smithi187 ceph-mon[20072]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:49 smithi162 ceph-mon[20279]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-02T09:56:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:49 smithi162 ceph-mon[20279]: osdmap e16: 3 total, 2 up, 3 in 2024-09-02T09:56:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:49 smithi162 ceph-mon[20279]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:56:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:49 smithi162 ceph-mon[20279]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:50.577 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:50 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2[42108]: 2024-09-02T09:56:50.311+0000 7fb380e18640 -1 osd.2 0 waiting for initial osdmap 2024-09-02T09:56:50.577 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:50 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2[42108]: 2024-09-02T09:56:50.314+0000 7fb378c37640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:56:51.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:50 smithi187 ceph-mon[20072]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:56:51.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:50 smithi187 ceph-mon[20072]: osdmap e17: 3 total, 2 up, 3 in 2024-09-02T09:56:51.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:51.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:51.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:50 smithi187 ceph-mon[20072]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' 2024-09-02T09:56:51.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:51.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:50 smithi151 ceph-mon[16639]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:56:51.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:50 smithi151 ceph-mon[16639]: osdmap e17: 3 total, 2 up, 3 in 2024-09-02T09:56:51.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:51.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:51.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:50 smithi151 ceph-mon[16639]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' 2024-09-02T09:56:51.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:51.176 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T09:56:51.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi151:vg_nvme/lv_1 2024-09-02T09:56:51.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:50 smithi162 ceph-mon[20279]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:56:51.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:50 smithi162 ceph-mon[20279]: osdmap e17: 3 total, 2 up, 3 in 2024-09-02T09:56:51.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:51.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:51.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:50 smithi162 ceph-mon[20279]: from='osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340]' entity='osd.2' 2024-09-02T09:56:51.247 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:51.739 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T09:56:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:52 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:56:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:52 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:56:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:52 smithi151 ceph-mon[16639]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:52 smithi151 ceph-mon[16639]: osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340] boot 2024-09-02T09:56:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:52 smithi151 ceph-mon[16639]: osdmap e18: 3 total, 3 up, 3 in 2024-09-02T09:56:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:52 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:52.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:52 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:56:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:52 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:56:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:52 smithi162 ceph-mon[20279]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:52 smithi162 ceph-mon[20279]: osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340] boot 2024-09-02T09:56:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:52 smithi162 ceph-mon[20279]: osdmap e18: 3 total, 3 up, 3 in 2024-09-02T09:56:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:52 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:52 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:56:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:52 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:56:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:52 smithi187 ceph-mon[20072]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-02T09:56:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:52 smithi187 ceph-mon[20072]: osd.2 [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340] boot 2024-09-02T09:56:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:52 smithi187 ceph-mon[20072]: osdmap e18: 3 total, 3 up, 3 in 2024-09-02T09:56:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:52 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-09-02T09:56:53.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: osdmap e19: 3 total, 3 up, 3 in 2024-09-02T09:56:53.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:56:53.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:56:53.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:56:53.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:53.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:53.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:53.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:53.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.441 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:53.442 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:53.442 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.442 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:53.442 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:53.442 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:53.442 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.442 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-02T09:56:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: osdmap e19: 3 total, 3 up, 3 in 2024-09-02T09:56:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:56:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:56:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:56:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-02T09:56:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: osdmap e19: 3 total, 3 up, 3 in 2024-09-02T09:56:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi151:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:56:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:56:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:56:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:56:53.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:53.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-09-02T09:56:54.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:54 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi151 2024-09-02T09:56:54.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:54 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi151 to 5696M 2024-09-02T09:56:54.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:54 smithi151 ceph-mon[16639]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:56:54.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:54 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-02T09:56:54.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:54 smithi151 ceph-mon[16639]: osdmap e20: 3 total, 3 up, 3 in 2024-09-02T09:56:54.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:54 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-02T09:56:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:54 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi151 2024-09-02T09:56:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:54 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi151 to 5696M 2024-09-02T09:56:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:54 smithi187 ceph-mon[20072]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:56:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:54 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-02T09:56:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:54 smithi187 ceph-mon[20072]: osdmap e20: 3 total, 3 up, 3 in 2024-09-02T09:56:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:54 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-02T09:56:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:54 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi151 2024-09-02T09:56:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:54 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi151 to 5696M 2024-09-02T09:56:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:54 smithi162 ceph-mon[20279]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:56:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:54 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-02T09:56:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:54 smithi162 ceph-mon[20279]: osdmap e20: 3 total, 3 up, 3 in 2024-09-02T09:56:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:54 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-09-02T09:56:55.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46358]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T09:56:55.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46358]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:55.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46358]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:55.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46358]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:55.687 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46346]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T09:56:55.687 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46346]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46346]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46346]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46349]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46349]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46349]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46321]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46321]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46321]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46321]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:55.688 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46334]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46334]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46334]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46334]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46352]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46352]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46352]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46355]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46355]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46355]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:55.689 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46355]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:55.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46361]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T09:56:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46361]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46361]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:55 smithi151 sudo[46361]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:55 smithi187 sudo[22032]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T09:56:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:55 smithi187 sudo[22032]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:55 smithi187 sudo[22032]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:55 smithi187 sudo[22032]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:55 smithi187 sudo[22035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T09:56:56.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:55 smithi187 sudo[22035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:56.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:55 smithi187 sudo[22035]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:56.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:55 smithi187 sudo[22035]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:55 smithi162 sudo[22330]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:55 smithi162 sudo[22330]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:55 smithi162 sudo[22330]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 sudo[22330]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 sudo[22333]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 sudo[22333]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 sudo[22333]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 sudo[22333]: pam_unix(sudo:session): session closed for user root 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: osdmap e21: 3 total, 3 up, 3 in 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2738243729' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "524f32ed-95aa-4459-9585-72577db12107"} : dispatch 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2738243729' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "524f32ed-95aa-4459-9585-72577db12107"}]': finished 2024-09-02T09:56:56.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: osdmap e22: 4 total, 3 up, 4 in 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:56:56.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T09:56:56.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:56.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/348468277' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:56:56.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T09:56:56.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:56:56.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:56:56.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:56:56.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T09:56:56.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:56:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-02T09:56:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: osdmap e21: 3 total, 3 up, 3 in 2024-09-02T09:56:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2738243729' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "524f32ed-95aa-4459-9585-72577db12107"} : dispatch 2024-09-02T09:56:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2738243729' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "524f32ed-95aa-4459-9585-72577db12107"}]': finished 2024-09-02T09:56:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: osdmap e22: 4 total, 3 up, 4 in 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/348468277' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:56:56.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T09:56:56.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:56:56.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:56:56.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:56:56.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T09:56:56.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:56:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-02T09:56:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: osdmap e21: 3 total, 3 up, 3 in 2024-09-02T09:56:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2738243729' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "524f32ed-95aa-4459-9585-72577db12107"} : dispatch 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2738243729' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "524f32ed-95aa-4459-9585-72577db12107"}]': finished 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: osdmap e22: 4 total, 3 up, 4 in 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:56:56.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T09:56:56.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:56.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/348468277' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:56:56.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-02T09:56:56.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:56:56.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:56:56.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:56:56.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-02T09:56:56.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:56:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:57 smithi187 ceph-mon[20072]: osdmap e23: 4 total, 3 up, 4 in 2024-09-02T09:56:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:56:57.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:57 smithi162 ceph-mon[20279]: osdmap e23: 4 total, 3 up, 4 in 2024-09-02T09:56:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:56:57.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:57 smithi151 ceph-mon[16639]: osdmap e23: 4 total, 3 up, 4 in 2024-09-02T09:56:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:56:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:58 smithi187 ceph-mon[20072]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:56:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:56:58 smithi187 ceph-mon[20072]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-02T09:56:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:58 smithi162 ceph-mon[20279]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:56:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:56:58 smithi162 ceph-mon[20279]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-02T09:56:58.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:58 smithi151 ceph-mon[16639]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:56:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:56:58 smithi151 ceph-mon[16639]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-02T09:57:00.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:00 smithi151 ceph-mon[16639]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:01.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:00 smithi187 ceph-mon[20072]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:01.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:00 smithi162 ceph-mon[20279]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:02.843 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:02 smithi151 ceph-mon[16639]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:02.843 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:02 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-02T09:57:02.843 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:02 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:02 smithi187 ceph-mon[20072]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:02 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-02T09:57:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:02 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:03.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:02 smithi162 ceph-mon[20279]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:02 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-09-02T09:57:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:02 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:03.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:03 smithi151 ceph-mon[16639]: Deploying daemon osd.3 on smithi151 2024-09-02T09:57:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:03 smithi187 ceph-mon[20072]: Deploying daemon osd.3 on smithi151 2024-09-02T09:57:04.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:03 smithi162 ceph-mon[20279]: Deploying daemon osd.3 on smithi151 2024-09-02T09:57:04.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:04 smithi151 ceph-mon[16639]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:04 smithi187 ceph-mon[20072]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:05.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:04 smithi162 ceph-mon[20279]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:06 smithi187 ceph-mon[20072]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:07.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:06 smithi162 ceph-mon[20279]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:07.181 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:06 smithi151 ceph-mon[16639]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:08.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:08 smithi151 ceph-mon[16639]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:08.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:08.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:08.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:08 smithi187 ceph-mon[20072]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:08 smithi162 ceph-mon[20279]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:09.984 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:09 smithi151 ceph-mon[16639]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:09 smithi187 ceph-mon[20072]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:10.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:09 smithi162 ceph-mon[20279]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:11.572 INFO:teuthology.orchestra.run.smithi151.stdout:Created osd(s) 3 on host 'smithi151' 2024-09-02T09:57:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:11 smithi151 ceph-mon[16639]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:11 smithi187 ceph-mon[20072]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:12.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:12.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:12.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:12.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.282 DEBUG:teuthology.orchestra.run.smithi151:osd.3> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.3.service 2024-09-02T09:57:12.284 INFO:tasks.cephadm:Deploying osd.4 on smithi162 with /dev/vg_nvme/lv_4... 2024-09-02T09:57:12.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-02T09:57:12.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:11 smithi162 ceph-mon[20279]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:12.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:12.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:12.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:12.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.313 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:12.539 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T09:57:13.622 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 09:57:13 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3[47996]: 2024-09-02T09:57:13.281+0000 7fdb2bd4b640 -1 osd.3 0 log_to_monitors true 2024-09-02T09:57:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:14 smithi151 ceph-mon[16639]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:14 smithi151 ceph-mon[16639]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-02T09:57:14.209 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:14 smithi162 ceph-mon[20279]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:14.209 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:14 smithi162 ceph-mon[20279]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-02T09:57:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:14 smithi187 ceph-mon[20072]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:14 smithi187 ceph-mon[20072]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-09-02T09:57:14.940 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:57:14.967 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_4 2024-09-02T09:57:15.261 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T09:57:15.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:15 smithi162 ceph-mon[20279]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-02T09:57:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:15 smithi162 ceph-mon[20279]: osdmap e24: 4 total, 3 up, 4 in 2024-09-02T09:57:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:15 smithi162 ceph-mon[20279]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:57:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:15 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:15.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:15 smithi151 ceph-mon[16639]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-02T09:57:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:15 smithi151 ceph-mon[16639]: osdmap e24: 4 total, 3 up, 4 in 2024-09-02T09:57:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:15 smithi151 ceph-mon[16639]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:57:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:15 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:15 smithi187 ceph-mon[20072]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-02T09:57:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:15 smithi187 ceph-mon[20072]: osdmap e24: 4 total, 3 up, 4 in 2024-09-02T09:57:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:15 smithi187 ceph-mon[20072]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]} : dispatch 2024-09-02T09:57:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:15 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:15.941 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 09:57:15 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3[47996]: 2024-09-02T09:57:15.510+0000 7fdb284e2640 -1 osd.3 0 waiting for initial osdmap 2024-09-02T09:57:15.942 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 09:57:15 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3[47996]: 2024-09-02T09:57:15.514+0000 7fdb1faed640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:57:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:57:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: osdmap e25: 4 total, 3 up, 4 in 2024-09-02T09:57:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi151 2024-09-02T09:57:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi151 to 4272M 2024-09-02T09:57:16.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:16.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:16.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' 2024-09-02T09:57:16.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:57:16.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:57:16.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:16 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:57:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: osdmap e25: 4 total, 3 up, 4 in 2024-09-02T09:57:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi151 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi151 to 4272M 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' 2024-09-02T09:57:16.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:57:16.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:57:16.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi151", "root=default"]}]': finished 2024-09-02T09:57:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: osdmap e25: 4 total, 3 up, 4 in 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi151 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi151 to 4272M 2024-09-02T09:57:16.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:16.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:16.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:16.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696]' entity='osd.3' 2024-09-02T09:57:16.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:57:16.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:57:16.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:16 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:16.938 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 09:57:16 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3[47996]: 2024-09-02T09:57:16.505+0000 7fdb1b6e0640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:57:17.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:17 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:57:17.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:17 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:57:17.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:17 smithi162 ceph-mon[20279]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:57:17.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:17 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:17.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:17 smithi162 ceph-mon[20279]: osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696] boot 2024-09-02T09:57:17.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:17 smithi162 ceph-mon[20279]: osdmap e26: 4 total, 4 up, 4 in 2024-09-02T09:57:17.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:17 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:17 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:57:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:17 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:57:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:17 smithi151 ceph-mon[16639]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:57:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:17 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:17 smithi151 ceph-mon[16639]: osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696] boot 2024-09-02T09:57:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:17 smithi151 ceph-mon[16639]: osdmap e26: 4 total, 4 up, 4 in 2024-09-02T09:57:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:17 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:17 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:57:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:17 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:57:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:17 smithi187 ceph-mon[20072]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:57:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:17 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:17 smithi187 ceph-mon[20072]: osd.3 [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696] boot 2024-09-02T09:57:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:17 smithi187 ceph-mon[20072]: osdmap e26: 4 total, 4 up, 4 in 2024-09-02T09:57:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:17 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-09-02T09:57:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:18 smithi162 ceph-mon[20279]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:18 smithi162 ceph-mon[20279]: osdmap e27: 4 total, 4 up, 4 in 2024-09-02T09:57:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.162:0/2044346562' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4"} : dispatch 2024-09-02T09:57:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:18 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4"} : dispatch 2024-09-02T09:57:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:18 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4"}]': finished 2024-09-02T09:57:18.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:18 smithi162 ceph-mon[20279]: osdmap e28: 5 total, 4 up, 5 in 2024-09-02T09:57:18.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:18 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:18 smithi151 ceph-mon[16639]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:18 smithi151 ceph-mon[16639]: osdmap e27: 4 total, 4 up, 4 in 2024-09-02T09:57:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.162:0/2044346562' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4"} : dispatch 2024-09-02T09:57:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:18 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4"} : dispatch 2024-09-02T09:57:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:18 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4"}]': finished 2024-09-02T09:57:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:18 smithi151 ceph-mon[16639]: osdmap e28: 5 total, 4 up, 5 in 2024-09-02T09:57:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:18 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:19.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:18 smithi187 ceph-mon[20072]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-02T09:57:19.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:18 smithi187 ceph-mon[20072]: osdmap e27: 4 total, 4 up, 4 in 2024-09-02T09:57:19.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.162:0/2044346562' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4"} : dispatch 2024-09-02T09:57:19.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:18 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4"} : dispatch 2024-09-02T09:57:19.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:18 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4"}]': finished 2024-09-02T09:57:19.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:18 smithi187 ceph-mon[20072]: osdmap e28: 5 total, 4 up, 5 in 2024-09-02T09:57:19.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:18 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.162:0/268340337' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:57:19.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.162:0/268340337' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:57:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.162:0/268340337' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:57:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:20 smithi187 ceph-mon[20072]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:21.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:20 smithi162 ceph-mon[20279]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:21.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:20 smithi151 ceph-mon[16639]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:22.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:22 smithi162 ceph-mon[20279]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:22 smithi187 ceph-mon[20072]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:23.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:22 smithi151 ceph-mon[16639]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:24 smithi162 ceph-mon[20279]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:24 smithi187 ceph-mon[20072]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:25.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:24 smithi151 ceph-mon[16639]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:25.935 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:25 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:57:25.935 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:25 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:57:25.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:25 smithi162 ceph-mon[20279]: osdmap e29: 5 total, 4 up, 5 in 2024-09-02T09:57:25.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:25.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:25 smithi162 ceph-mon[20279]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:25.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-02T09:57:25.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:25.936 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:25 smithi162 ceph-mon[20279]: Deploying daemon osd.4 on smithi162 2024-09-02T09:57:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:25 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:57:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:25 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:57:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:25 smithi187 ceph-mon[20072]: osdmap e29: 5 total, 4 up, 5 in 2024-09-02T09:57:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:25 smithi187 ceph-mon[20072]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-02T09:57:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:25 smithi187 ceph-mon[20072]: Deploying daemon osd.4 on smithi162 2024-09-02T09:57:26.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:25 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:57:26.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:25 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:57:26.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:25 smithi151 ceph-mon[16639]: osdmap e29: 5 total, 4 up, 5 in 2024-09-02T09:57:26.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:26.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:25 smithi151 ceph-mon[16639]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:26.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-09-02T09:57:26.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:26.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:25 smithi151 ceph-mon[16639]: Deploying daemon osd.4 on smithi162 2024-09-02T09:57:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:28 smithi162 ceph-mon[20279]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:28 smithi151 ceph-mon[16639]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:28 smithi187 ceph-mon[20072]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:30 smithi162 ceph-mon[20279]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:30.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:30 smithi151 ceph-mon[16639]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:30 smithi187 ceph-mon[20072]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:31.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:32 smithi162 ceph-mon[20279]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:32.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:32 smithi151 ceph-mon[16639]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:32 smithi187 ceph-mon[20072]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:33.818 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 4 on host 'smithi162' 2024-09-02T09:57:34.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:33 smithi162 ceph-mon[20279]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:33 smithi151 ceph-mon[16639]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:33 smithi187 ceph-mon[20072]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:34.388 DEBUG:teuthology.orchestra.run.smithi162:osd.4> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.4.service 2024-09-02T09:57:34.390 INFO:tasks.cephadm:Deploying osd.5 on smithi162 with /dev/vg_nvme/lv_3... 2024-09-02T09:57:34.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-02T09:57:35.350 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T09:57:35.598 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 09:57:35 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T09:57:35.350+0000 7f5fe5d9f640 -1 osd.4 0 log_to_monitors true 2024-09-02T09:57:36.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:36 smithi162 ceph-mon[20279]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:36.316 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:36 smithi162 ceph-mon[20279]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-02T09:57:36.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:36 smithi151 ceph-mon[16639]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:36.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:36 smithi151 ceph-mon[16639]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-02T09:57:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:36 smithi187 ceph-mon[20072]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:36 smithi187 ceph-mon[20072]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-09-02T09:57:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:37 smithi162 ceph-mon[20279]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-02T09:57:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:37 smithi162 ceph-mon[20279]: osdmap e30: 5 total, 4 up, 5 in 2024-09-02T09:57:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:37 smithi162 ceph-mon[20279]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:57:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:37 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:37.345 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 09:57:37 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T09:57:37.074+0000 7f5fe1d22640 -1 osd.4 0 waiting for initial osdmap 2024-09-02T09:57:37.346 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 09:57:37 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T09:57:37.078+0000 7f5fddb49640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:57:37.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:37 smithi151 ceph-mon[16639]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-02T09:57:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:37 smithi151 ceph-mon[16639]: osdmap e30: 5 total, 4 up, 5 in 2024-09-02T09:57:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:37 smithi151 ceph-mon[16639]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:57:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:37 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:37 smithi187 ceph-mon[20072]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-02T09:57:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:37 smithi187 ceph-mon[20072]: osdmap e30: 5 total, 4 up, 5 in 2024-09-02T09:57:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:37 smithi187 ceph-mon[20072]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:57:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:37 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:38.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:38.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:57:38.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: osdmap e31: 5 total, 4 up, 5 in 2024-09-02T09:57:38.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:38.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:38.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.281 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:38.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:38.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:38.282 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: osdmap e31: 5 total, 4 up, 5 in 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:38.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: from='osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389]' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:57:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: osdmap e31: 5 total, 4 up, 5 in 2024-09-02T09:57:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:57:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:38.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:38.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:38.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:39.135 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:57:39.135 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:57:39.135 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:39 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi162 2024-09-02T09:57:39.135 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:39 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi162 to 17088M 2024-09-02T09:57:39.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:39 smithi162 ceph-mon[20279]: osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389] boot 2024-09-02T09:57:39.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:39 smithi162 ceph-mon[20279]: osdmap e32: 5 total, 5 up, 5 in 2024-09-02T09:57:39.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:39.136 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:39 smithi162 ceph-mon[20279]: osdmap e33: 5 total, 5 up, 5 in 2024-09-02T09:57:39.138 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:57:39.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_3 2024-09-02T09:57:39.409 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T09:57:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:57:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:57:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:39 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi162 2024-09-02T09:57:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:39 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi162 to 17088M 2024-09-02T09:57:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:39 smithi151 ceph-mon[16639]: osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389] boot 2024-09-02T09:57:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:39 smithi151 ceph-mon[16639]: osdmap e32: 5 total, 5 up, 5 in 2024-09-02T09:57:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:39 smithi151 ceph-mon[16639]: osdmap e33: 5 total, 5 up, 5 in 2024-09-02T09:57:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:57:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:57:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:39 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi162 2024-09-02T09:57:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:39 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi162 to 17088M 2024-09-02T09:57:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:39 smithi187 ceph-mon[20072]: osd.4 [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389] boot 2024-09-02T09:57:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:39 smithi187 ceph-mon[20072]: osdmap e32: 5 total, 5 up, 5 in 2024-09-02T09:57:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-09-02T09:57:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:39 smithi187 ceph-mon[20072]: osdmap e33: 5 total, 5 up, 5 in 2024-09-02T09:57:40.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:40 smithi162 ceph-mon[20279]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:40.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:40 smithi162 ceph-mon[20279]: osdmap e34: 5 total, 5 up, 5 in 2024-09-02T09:57:40.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:57:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:57:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:40.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:40 smithi151 ceph-mon[16639]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:40 smithi151 ceph-mon[16639]: osdmap e34: 5 total, 5 up, 5 in 2024-09-02T09:57:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:57:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:57:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:40 smithi187 ceph-mon[20072]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-02T09:57:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:40 smithi187 ceph-mon[20072]: osdmap e34: 5 total, 5 up, 5 in 2024-09-02T09:57:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:57:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:57:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:41.410 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:41 smithi162 ceph-mon[20279]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:57:41.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:41 smithi151 ceph-mon[16639]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:57:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:41 smithi187 ceph-mon[20072]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:57:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:42 smithi162 ceph-mon[20279]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:42.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:42 smithi151 ceph-mon[16639]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:42 smithi187 ceph-mon[20072]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.162:0/971557946' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a"} : dispatch 2024-09-02T09:57:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:43 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a"} : dispatch 2024-09-02T09:57:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:43 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a"}]': finished 2024-09-02T09:57:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:43 smithi162 ceph-mon[20279]: osdmap e35: 6 total, 5 up, 6 in 2024-09-02T09:57:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:43 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:57:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.162:0/248649251' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:57:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.162:0/971557946' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a"} : dispatch 2024-09-02T09:57:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:43 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a"} : dispatch 2024-09-02T09:57:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:43 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a"}]': finished 2024-09-02T09:57:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:43 smithi151 ceph-mon[16639]: osdmap e35: 6 total, 5 up, 6 in 2024-09-02T09:57:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:43 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:57:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.162:0/248649251' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:57:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.162:0/971557946' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a"} : dispatch 2024-09-02T09:57:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:43 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a"} : dispatch 2024-09-02T09:57:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:43 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a"}]': finished 2024-09-02T09:57:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:43 smithi187 ceph-mon[20072]: osdmap e35: 6 total, 5 up, 6 in 2024-09-02T09:57:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:43 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:57:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.162:0/248649251' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:57:44.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:44 smithi151 ceph-mon[16639]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-09-02T09:57:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:44 smithi187 ceph-mon[20072]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-09-02T09:57:44.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:44 smithi162 ceph-mon[20279]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-09-02T09:57:46.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:46 smithi162 ceph-mon[20279]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-02T09:57:46.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:46 smithi151 ceph-mon[16639]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-02T09:57:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:46 smithi187 ceph-mon[20072]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-02T09:57:48.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:48 smithi162 ceph-mon[20279]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:57:48.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:48 smithi151 ceph-mon[16639]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:57:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:48 smithi187 ceph-mon[20072]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:57:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:49 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:57:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:49 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:57:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:49 smithi162 ceph-mon[20279]: osdmap e36: 6 total, 5 up, 6 in 2024-09-02T09:57:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:57:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-02T09:57:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:49 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:57:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:49 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:57:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:49 smithi151 ceph-mon[16639]: osdmap e36: 6 total, 5 up, 6 in 2024-09-02T09:57:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:57:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-02T09:57:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:49 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:57:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:49 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:57:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:49 smithi187 ceph-mon[20072]: osdmap e36: 6 total, 5 up, 6 in 2024-09-02T09:57:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:57:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-09-02T09:57:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:50.670 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:50 smithi162 ceph-mon[20279]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:57:50.670 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:50 smithi162 ceph-mon[20279]: Deploying daemon osd.5 on smithi162 2024-09-02T09:57:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:50 smithi151 ceph-mon[16639]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:57:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:50 smithi151 ceph-mon[16639]: Deploying daemon osd.5 on smithi162 2024-09-02T09:57:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:50 smithi187 ceph-mon[20072]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:57:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:50 smithi187 ceph-mon[20072]: Deploying daemon osd.5 on smithi162 2024-09-02T09:57:52.665 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:52 smithi162 ceph-mon[20279]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-02T09:57:52.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:52 smithi151 ceph-mon[16639]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-02T09:57:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:52 smithi187 ceph-mon[20072]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-09-02T09:57:54.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:54 smithi151 ceph-mon[16639]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:54 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:54 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:54 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:54 smithi187 ceph-mon[20072]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:54 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:54 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:54 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:54.822 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:54 smithi162 ceph-mon[20279]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:54.823 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:54 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:54.823 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:54 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:54.823 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:54 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:56.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:56 smithi151 ceph-mon[16639]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:56 smithi187 ceph-mon[20072]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:56 smithi162 ceph-mon[20279]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:57.729 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 5 on host 'smithi162' 2024-09-02T09:57:58.081 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:58.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:58.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:57 smithi162 ceph-mon[20279]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:58.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:58.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.082 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:57 smithi151 ceph-mon[16639]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:57:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:57:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:57 smithi187 ceph-mon[20072]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:57:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:57:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:57:58.377 DEBUG:teuthology.orchestra.run.smithi162:osd.5> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.5.service 2024-09-02T09:57:58.379 INFO:tasks.cephadm:Deploying osd.6 on smithi162 with /dev/vg_nvme/lv_2... 2024-09-02T09:57:58.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-02T09:57:59.384 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T09:57:59.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:59 smithi162 ceph-mon[20279]: from='osd.5 [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T09:57:59.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:57:59 smithi162 ceph-mon[20279]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T09:57:59.792 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 09:57:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T09:57:59.645+0000 7ff35861e640 -1 osd.5 0 log_to_monitors true 2024-09-02T09:58:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:59 smithi187 ceph-mon[20072]: from='osd.5 [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T09:58:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:57:59 smithi187 ceph-mon[20072]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T09:58:00.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:59 smithi151 ceph-mon[16639]: from='osd.5 [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T09:58:00.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:57:59 smithi151 ceph-mon[16639]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-09-02T09:58:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:00 smithi162 ceph-mon[20279]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:58:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:00 smithi162 ceph-mon[20279]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-02T09:58:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:00 smithi162 ceph-mon[20279]: osdmap e37: 6 total, 5 up, 6 in 2024-09-02T09:58:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:00 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:00 smithi162 ceph-mon[20279]: from='osd.5 [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:00 smithi162 ceph-mon[20279]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:00 smithi187 ceph-mon[20072]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:58:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:00 smithi187 ceph-mon[20072]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-02T09:58:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:00 smithi187 ceph-mon[20072]: osdmap e37: 6 total, 5 up, 6 in 2024-09-02T09:58:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:00 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:00 smithi187 ceph-mon[20072]: from='osd.5 [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:00 smithi187 ceph-mon[20072]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:00 smithi151 ceph-mon[16639]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:58:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:00 smithi151 ceph-mon[16639]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-02T09:58:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:00 smithi151 ceph-mon[16639]: osdmap e37: 6 total, 5 up, 6 in 2024-09-02T09:58:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:00 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:00 smithi151 ceph-mon[16639]: from='osd.5 [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:00 smithi151 ceph-mon[16639]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:01.437 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T09:58:01.162+0000 7ff3545a1640 -1 osd.5 0 waiting for initial osdmap 2024-09-02T09:58:01.437 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T09:58:01.166+0000 7ff34c3c0640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:58:01.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:58:01.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: osdmap e38: 6 total, 5 up, 6 in 2024-09-02T09:58:01.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:01.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:01.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:58:01.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: from='osd.5 ' entity='osd.5' 2024-09-02T09:58:01.985 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:58:01.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:58:01.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: osd.5 [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981] boot 2024-09-02T09:58:01.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: osdmap e39: 6 total, 6 up, 6 in 2024-09-02T09:58:01.986 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:58:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: osdmap e38: 6 total, 5 up, 6 in 2024-09-02T09:58:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:58:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: from='osd.5 ' entity='osd.5' 2024-09-02T09:58:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:58:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:58:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: osd.5 [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981] boot 2024-09-02T09:58:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: osdmap e39: 6 total, 6 up, 6 in 2024-09-02T09:58:02.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:02.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:58:02.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: osdmap e38: 6 total, 5 up, 6 in 2024-09-02T09:58:02.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:02.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:02.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-02T09:58:02.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: from='osd.5 ' entity='osd.5' 2024-09-02T09:58:02.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:58:02.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:58:02.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: osd.5 [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981] boot 2024-09-02T09:58:02.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: osdmap e39: 6 total, 6 up, 6 in 2024-09-02T09:58:02.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-09-02T09:58:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:58:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:58:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi162 2024-09-02T09:58:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi162 to 8544M 2024-09-02T09:58:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:03.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:03.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:03 smithi162 ceph-mon[20279]: osdmap e40: 6 total, 6 up, 6 in 2024-09-02T09:58:03.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:58:03.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:58:03.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi162 2024-09-02T09:58:03.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:03.351 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:03.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi162 to 8544M 2024-09-02T09:58:03.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:03.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:03.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:03 smithi151 ceph-mon[16639]: osdmap e40: 6 total, 6 up, 6 in 2024-09-02T09:58:03.362 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:58:03.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_2 2024-09-02T09:58:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:58:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:58:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi162 2024-09-02T09:58:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:03.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:03.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi162 to 8544M 2024-09-02T09:58:03.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:03.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:03.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:03.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:03 smithi187 ceph-mon[20072]: osdmap e40: 6 total, 6 up, 6 in 2024-09-02T09:58:03.631 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T09:58:04.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:04 smithi151 ceph-mon[16639]: pgmap v132: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:04.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:04 smithi151 ceph-mon[16639]: osdmap e41: 6 total, 6 up, 6 in 2024-09-02T09:58:04.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:04 smithi162 ceph-mon[20279]: pgmap v132: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:04.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:04 smithi162 ceph-mon[20279]: osdmap e41: 6 total, 6 up, 6 in 2024-09-02T09:58:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:04 smithi187 ceph-mon[20072]: pgmap v132: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:04 smithi187 ceph-mon[20072]: osdmap e41: 6 total, 6 up, 6 in 2024-09-02T09:58:05.292 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:05 smithi162 ceph-mon[20279]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:58:05.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:05 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:58:05.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:05 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:58:05.293 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:05 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:05.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:05 smithi151 ceph-mon[16639]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:58:05.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:05 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:58:05.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:05 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:58:05.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:05 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:05.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:05 smithi187 ceph-mon[20072]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:58:05.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:05 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:58:05.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:05 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:58:05.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:05 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:06.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:06 smithi162 ceph-mon[20279]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:06.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:06 smithi151 ceph-mon[16639]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:06 smithi187 ceph-mon[20072]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:07.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.162:0/3777680745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308"} : dispatch 2024-09-02T09:58:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:07 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308"} : dispatch 2024-09-02T09:58:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:07 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308"}]': finished 2024-09-02T09:58:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:07 smithi162 ceph-mon[20279]: osdmap e42: 7 total, 6 up, 7 in 2024-09-02T09:58:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:07 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:07.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.162:0/3777680745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308"} : dispatch 2024-09-02T09:58:07.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:07 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308"} : dispatch 2024-09-02T09:58:07.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:07 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308"}]': finished 2024-09-02T09:58:07.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:07 smithi151 ceph-mon[16639]: osdmap e42: 7 total, 6 up, 7 in 2024-09-02T09:58:07.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:07 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.162:0/3777680745' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308"} : dispatch 2024-09-02T09:58:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:07 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308"} : dispatch 2024-09-02T09:58:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:07 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308"}]': finished 2024-09-02T09:58:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:07 smithi187 ceph-mon[20072]: osdmap e42: 7 total, 6 up, 7 in 2024-09-02T09:58:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:07 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:08 smithi162 ceph-mon[20279]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-02T09:58:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.162:0/2462607431' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:58:08.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:08 smithi151 ceph-mon[16639]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-02T09:58:08.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.162:0/2462607431' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:58:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:08 smithi187 ceph-mon[20072]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-09-02T09:58:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.162:0/2462607431' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:58:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:09 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T09:58:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:09 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T09:58:09.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:09 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T09:58:09.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:09 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T09:58:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:09 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T09:58:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:09 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T09:58:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:10 smithi162 ceph-mon[20279]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-02T09:58:10.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:10 smithi151 ceph-mon[16639]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-02T09:58:10.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:10 smithi187 ceph-mon[20072]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-02T09:58:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:12 smithi162 ceph-mon[20279]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:58:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:12 smithi151 ceph-mon[16639]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:58:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:12 smithi187 ceph-mon[20072]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:58:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:14 smithi151 ceph-mon[16639]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-02T09:58:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:14 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-02T09:58:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:14 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:14 smithi151 ceph-mon[16639]: Deploying daemon osd.6 on smithi162 2024-09-02T09:58:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:14 smithi162 ceph-mon[20279]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-02T09:58:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:14 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-02T09:58:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:14 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:14 smithi162 ceph-mon[20279]: Deploying daemon osd.6 on smithi162 2024-09-02T09:58:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:14 smithi187 ceph-mon[20072]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-02T09:58:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-09-02T09:58:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:14 smithi187 ceph-mon[20072]: Deploying daemon osd.6 on smithi162 2024-09-02T09:58:16.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:16 smithi162 ceph-mon[20279]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-02T09:58:16.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:16 smithi151 ceph-mon[16639]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-02T09:58:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:16 smithi187 ceph-mon[20072]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-02T09:58:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:18 smithi162 ceph-mon[20279]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-02T09:58:18.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:18 smithi151 ceph-mon[16639]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-02T09:58:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:18 smithi187 ceph-mon[20072]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-02T09:58:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:20.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:20.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:20.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:20.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:20 smithi187 ceph-mon[20072]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:21.039 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:20 smithi162 ceph-mon[20279]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:21.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:20 smithi151 ceph-mon[16639]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:22.478 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 6 on host 'smithi162' 2024-09-02T09:58:22.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:22 smithi162 ceph-mon[20279]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:22.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:22.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:22.758 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:22.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.759 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.809 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:22 smithi187 ceph-mon[20072]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:22.809 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.809 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.809 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:22.809 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:22.809 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.809 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:22.810 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.810 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:22 smithi151 ceph-mon[16639]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:23.087 DEBUG:teuthology.orchestra.run.smithi162:osd.6> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.6.service 2024-09-02T09:58:23.089 INFO:tasks.cephadm:Deploying osd.7 on smithi162 with /dev/vg_nvme/lv_1... 2024-09-02T09:58:23.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-02T09:58:23.935 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T09:58:24.340 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 09:58:24 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T09:58:24.245+0000 7f533f81e640 -1 osd.6 0 log_to_monitors true 2024-09-02T09:58:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:24 smithi187 ceph-mon[20072]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:24 smithi187 ceph-mon[20072]: from='osd.6 [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T09:58:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:24 smithi187 ceph-mon[20072]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T09:58:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:24 smithi162 ceph-mon[20279]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:25.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:24 smithi162 ceph-mon[20279]: from='osd.6 [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T09:58:25.060 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:24 smithi162 ceph-mon[20279]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T09:58:25.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:24 smithi151 ceph-mon[16639]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:25.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:24 smithi151 ceph-mon[16639]: from='osd.6 [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T09:58:25.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:24 smithi151 ceph-mon[16639]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-09-02T09:58:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:25 smithi187 ceph-mon[20072]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-02T09:58:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:25 smithi187 ceph-mon[20072]: osdmap e43: 7 total, 6 up, 7 in 2024-09-02T09:58:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:25 smithi187 ceph-mon[20072]: from='osd.6 [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:25 smithi187 ceph-mon[20072]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:25 smithi187 ceph-mon[20072]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:26.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:25 smithi162 ceph-mon[20279]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-02T09:58:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:25 smithi162 ceph-mon[20279]: osdmap e43: 7 total, 6 up, 7 in 2024-09-02T09:58:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:25 smithi162 ceph-mon[20279]: from='osd.6 [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:25 smithi162 ceph-mon[20279]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:25 smithi162 ceph-mon[20279]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:26.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:25 smithi151 ceph-mon[16639]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-02T09:58:26.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:25 smithi151 ceph-mon[16639]: osdmap e43: 7 total, 6 up, 7 in 2024-09-02T09:58:26.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:26.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:25 smithi151 ceph-mon[16639]: from='osd.6 [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:26.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:25 smithi151 ceph-mon[16639]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:26.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:25 smithi151 ceph-mon[16639]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-02T09:58:26.585 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 09:58:26 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T09:58:26.170+0000 7f533b7a1640 -1 osd.6 0 waiting for initial osdmap 2024-09-02T09:58:26.585 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 09:58:26 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T09:58:26.177+0000 7f53335c0640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:58:26.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:26 smithi162 ceph-mon[20279]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:58:26.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:26 smithi162 ceph-mon[20279]: osdmap e44: 7 total, 6 up, 7 in 2024-09-02T09:58:26.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:26.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:26 smithi162 ceph-mon[20279]: from='osd.6 ' entity='osd.6' 2024-09-02T09:58:26.848 INFO:teuthology.orchestra.run.smithi162.stdout: 2024-09-02T09:58:26.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi162:vg_nvme/lv_1 2024-09-02T09:58:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:26 smithi187 ceph-mon[20072]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:58:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:26 smithi187 ceph-mon[20072]: osdmap e44: 7 total, 6 up, 7 in 2024-09-02T09:58:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:26 smithi187 ceph-mon[20072]: from='osd.6 ' entity='osd.6' 2024-09-02T09:58:27.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:26 smithi151 ceph-mon[16639]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:58:27.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:26 smithi151 ceph-mon[16639]: osdmap e44: 7 total, 6 up, 7 in 2024-09-02T09:58:27.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:27.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:27.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:26 smithi151 ceph-mon[16639]: from='osd.6 ' entity='osd.6' 2024-09-02T09:58:27.440 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T09:58:27.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:58:27.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:58:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:27 smithi162 ceph-mon[20279]: osd.6 [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342] boot 2024-09-02T09:58:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:27 smithi162 ceph-mon[20279]: osdmap e45: 7 total, 7 up, 7 in 2024-09-02T09:58:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:27 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:27 smithi162 ceph-mon[20279]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:58:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:58:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:27 smithi187 ceph-mon[20072]: osd.6 [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342] boot 2024-09-02T09:58:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:27 smithi187 ceph-mon[20072]: osdmap e45: 7 total, 7 up, 7 in 2024-09-02T09:58:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:27 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:28.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:27 smithi187 ceph-mon[20072]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:58:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:58:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:27 smithi151 ceph-mon[16639]: osd.6 [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342] boot 2024-09-02T09:58:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:27 smithi151 ceph-mon[16639]: osdmap e45: 7 total, 7 up, 7 in 2024-09-02T09:58:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:27 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-09-02T09:58:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:27 smithi151 ceph-mon[16639]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:58:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:58:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: osdmap e46: 7 total, 7 up, 7 in 2024-09-02T09:58:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:58:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi162 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi162 to 5696M 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.042 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:58:29.042 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:58:29.042 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: osdmap e46: 7 total, 7 up, 7 in 2024-09-02T09:58:29.042 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:58:29.042 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:58:29.042 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:58:29.042 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:29.042 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi162 2024-09-02T09:58:29.042 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.042 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.043 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:29.043 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:29.043 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:29.043 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi162 to 5696M 2024-09-02T09:58:29.043 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.043 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:29.043 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:29.043 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:58:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:58:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: osdmap e46: 7 total, 7 up, 7 in 2024-09-02T09:58:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi162:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:58:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:58:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi162 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi162 to 5696M 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:30.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:29 smithi187 ceph-mon[20072]: osdmap e47: 7 total, 7 up, 7 in 2024-09-02T09:58:30.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:29 smithi187 ceph-mon[20072]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:30.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:29 smithi162 ceph-mon[20279]: osdmap e47: 7 total, 7 up, 7 in 2024-09-02T09:58:30.051 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:29 smithi162 ceph-mon[20279]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:30.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:29 smithi151 ceph-mon[16639]: osdmap e47: 7 total, 7 up, 7 in 2024-09-02T09:58:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:29 smithi151 ceph-mon[16639]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:31.083 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.162:0/22333384' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764"} : dispatch 2024-09-02T09:58:31.084 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:30 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764"} : dispatch 2024-09-02T09:58:31.084 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:30 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764"}]': finished 2024-09-02T09:58:31.084 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:30 smithi162 ceph-mon[20279]: osdmap e48: 8 total, 7 up, 8 in 2024-09-02T09:58:31.084 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:31.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.162:0/22333384' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764"} : dispatch 2024-09-02T09:58:31.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:30 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764"} : dispatch 2024-09-02T09:58:31.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:30 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764"}]': finished 2024-09-02T09:58:31.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:30 smithi151 ceph-mon[16639]: osdmap e48: 8 total, 7 up, 8 in 2024-09-02T09:58:31.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.162:0/22333384' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764"} : dispatch 2024-09-02T09:58:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:30 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764"} : dispatch 2024-09-02T09:58:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:30 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764"}]': finished 2024-09-02T09:58:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:30 smithi187 ceph-mon[20072]: osdmap e48: 8 total, 7 up, 8 in 2024-09-02T09:58:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:32.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:31 smithi151 ceph-mon[16639]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:32.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.162:0/2766759049' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:58:32.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:31 smithi162 ceph-mon[20279]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:32.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.162:0/2766759049' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:58:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:31 smithi187 ceph-mon[20072]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.162:0/2766759049' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:58:34.337 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:34 smithi162 ceph-mon[20279]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:34 smithi151 ceph-mon[16639]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:34 smithi187 ceph-mon[20072]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:36.241 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:36 smithi162 ceph-mon[20279]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:36.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:36 smithi151 ceph-mon[16639]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:36 smithi187 ceph-mon[20072]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:38 smithi162 ceph-mon[20279]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-02T09:58:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:38 smithi151 ceph-mon[16639]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-02T09:58:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:38 smithi187 ceph-mon[20072]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-09-02T09:58:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:39 smithi187 ceph-mon[20072]: Deploying daemon osd.7 on smithi162 2024-09-02T09:58:40.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:39 smithi162 ceph-mon[20279]: Deploying daemon osd.7 on smithi162 2024-09-02T09:58:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:39 smithi151 ceph-mon[16639]: Deploying daemon osd.7 on smithi162 2024-09-02T09:58:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:40 smithi162 ceph-mon[20279]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:40 smithi187 ceph-mon[20072]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:41.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:40 smithi151 ceph-mon[16639]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:41 smithi187 ceph-mon[20072]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:42.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:41 smithi162 ceph-mon[20279]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:42.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:41 smithi151 ceph-mon[16639]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:42 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:42 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:42 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:43.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:42 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:42 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:42 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:42 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:42 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:42 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:43 smithi187 ceph-mon[20072]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:44.311 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:43 smithi162 ceph-mon[20279]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:44.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:43 smithi151 ceph-mon[16639]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:46.305 INFO:teuthology.orchestra.run.smithi162.stdout:Created osd(s) 7 on host 'smithi162' 2024-09-02T09:58:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:46 smithi162 ceph-mon[20279]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:46.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:46.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:46 smithi151 ceph-mon[16639]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.960 DEBUG:teuthology.orchestra.run.smithi162:osd.7> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.7.service 2024-09-02T09:58:46.962 INFO:tasks.cephadm:Deploying osd.8 on smithi187 with /dev/vg_nvme/lv_4... 2024-09-02T09:58:46.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-02T09:58:46.990 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:46 smithi187 ceph-mon[20072]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:46.990 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.991 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.991 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:46.991 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:46.991 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.991 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:58:46.991 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:46.991 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:47.212 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:58:48.333 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 09:58:48 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T09:58:48.146+0000 7f09e11d0640 -1 osd.7 0 log_to_monitors true 2024-09-02T09:58:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:48 smithi162 ceph-mon[20279]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:48 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:58:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:48 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:58:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:48 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:58:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:48 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:58:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:48 smithi162 ceph-mon[20279]: osdmap e49: 8 total, 7 up, 8 in 2024-09-02T09:58:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:48 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:48 smithi162 ceph-mon[20279]: from='osd.7 [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T09:58:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:48 smithi162 ceph-mon[20279]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T09:58:48.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:48 smithi187 ceph-mon[20072]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:48.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:48 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:58:48.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:48 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:58:48.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:48 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:58:48.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:48 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:58:48.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:48 smithi187 ceph-mon[20072]: osdmap e49: 8 total, 7 up, 8 in 2024-09-02T09:58:48.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:48 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:48.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:48 smithi187 ceph-mon[20072]: from='osd.7 [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T09:58:48.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:48 smithi187 ceph-mon[20072]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T09:58:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:48 smithi151 ceph-mon[16639]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:48 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:58:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:48 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:58:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:48 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:58:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:48 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:58:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:48 smithi151 ceph-mon[16639]: osdmap e49: 8 total, 7 up, 8 in 2024-09-02T09:58:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:48 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:48 smithi151 ceph-mon[16639]: from='osd.7 [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T09:58:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:48 smithi151 ceph-mon[16639]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-09-02T09:58:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:49 smithi162 ceph-mon[20279]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-02T09:58:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:49 smithi162 ceph-mon[20279]: osdmap e50: 8 total, 7 up, 8 in 2024-09-02T09:58:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:49 smithi162 ceph-mon[20279]: from='osd.7 [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:49 smithi162 ceph-mon[20279]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:49.596 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 09:58:49 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T09:58:49.571+0000 7f09dd153640 -1 osd.7 0 waiting for initial osdmap 2024-09-02T09:58:49.831 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:49 smithi187 ceph-mon[20072]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-02T09:58:49.831 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:49 smithi187 ceph-mon[20072]: osdmap e50: 8 total, 7 up, 8 in 2024-09-02T09:58:49.831 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:49.831 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:49 smithi187 ceph-mon[20072]: from='osd.7 [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:49.831 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:49 smithi187 ceph-mon[20072]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:49.833 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:58:49.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi187:vg_nvme/lv_4 2024-09-02T09:58:49.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:49 smithi151 ceph-mon[16639]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-02T09:58:49.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:49 smithi151 ceph-mon[16639]: osdmap e50: 8 total, 7 up, 8 in 2024-09-02T09:58:49.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:49.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:49 smithi151 ceph-mon[16639]: from='osd.7 [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:49.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:49 smithi151 ceph-mon[16639]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]} : dispatch 2024-09-02T09:58:50.145 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:58:50.577 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T09:58:50.360+0000 7f09d8f7a640 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:58:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:58:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: osdmap e51: 8 total, 7 up, 8 in 2024-09-02T09:58:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:50.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:50.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:50.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:50.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:50.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:50.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:58:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: osdmap e51: 8 total, 7 up, 8 in 2024-09-02T09:58:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-02T09:58:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi162", "root=default"]}]': finished 2024-09-02T09:58:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: osdmap e51: 8 total, 7 up, 8 in 2024-09-02T09:58:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-09-02T09:58:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:58:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:58:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:58:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:58:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi162 2024-09-02T09:58:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi162 to 4272M 2024-09-02T09:58:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: osd.7 [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814] boot 2024-09-02T09:58:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: osdmap e52: 8 total, 8 up, 8 in 2024-09-02T09:58:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:58:51.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:58:51.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:51.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:51 smithi187 ceph-mon[20072]: osdmap e53: 8 total, 8 up, 8 in 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi162 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi162 to 4272M 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: osd.7 [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814] boot 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: osdmap e52: 8 total, 8 up, 8 in 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:51 smithi162 ceph-mon[20279]: osdmap e53: 8 total, 8 up, 8 in 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi162 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi162 to 4272M 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: osd.7 [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814] boot 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: osdmap e52: 8 total, 8 up, 8 in 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:58:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:51 smithi151 ceph-mon[16639]: osdmap e53: 8 total, 8 up, 8 in 2024-09-02T09:58:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:52 smithi151 ceph-mon[16639]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:58:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:52 smithi151 ceph-mon[16639]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:52 smithi187 ceph-mon[20072]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:58:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:52 smithi187 ceph-mon[20072]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:52 smithi162 ceph-mon[20279]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:58:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:52 smithi162 ceph-mon[20279]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:53 smithi151 ceph-mon[16639]: osdmap e54: 8 total, 8 up, 8 in 2024-09-02T09:58:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/3488702026' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634"} : dispatch 2024-09-02T09:58:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:53 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634"} : dispatch 2024-09-02T09:58:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:53 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634"}]': finished 2024-09-02T09:58:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:53 smithi151 ceph-mon[16639]: osdmap e55: 9 total, 8 up, 9 in 2024-09-02T09:58:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:58:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/3431891614' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:58:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:53 smithi187 ceph-mon[20072]: osdmap e54: 8 total, 8 up, 8 in 2024-09-02T09:58:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/3488702026' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634"} : dispatch 2024-09-02T09:58:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:53 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634"} : dispatch 2024-09-02T09:58:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:53 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634"}]': finished 2024-09-02T09:58:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:53 smithi187 ceph-mon[20072]: osdmap e55: 9 total, 8 up, 9 in 2024-09-02T09:58:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:58:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/3431891614' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:58:54.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:53 smithi162 ceph-mon[20279]: osdmap e54: 8 total, 8 up, 8 in 2024-09-02T09:58:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.187:0/3488702026' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634"} : dispatch 2024-09-02T09:58:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:53 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634"} : dispatch 2024-09-02T09:58:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:53 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634"}]': finished 2024-09-02T09:58:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:53 smithi162 ceph-mon[20279]: osdmap e55: 9 total, 8 up, 9 in 2024-09-02T09:58:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:58:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.187:0/3431891614' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:58:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:54 smithi151 ceph-mon[16639]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:54 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-02T09:58:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:54 smithi187 ceph-mon[20072]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:54 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-02T09:58:55.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:54 smithi162 ceph-mon[20279]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:54 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-02T09:58:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:56 smithi151 ceph-mon[16639]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:56 smithi187 ceph-mon[20072]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:57.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:56 smithi162 ceph-mon[20279]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:57.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:57 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-02T09:58:57.682 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:57 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T09:58:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:57 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-02T09:58:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:57 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T09:58:58.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:57 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-02T09:58:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:57 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T09:58:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:58:58 smithi151 ceph-mon[16639]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:58:58 smithi187 ceph-mon[20072]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:58:59.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:58:58 smithi162 ceph-mon[20279]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:00.957 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:00 smithi187 ceph-mon[20072]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-02T09:59:00.958 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:00 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-02T09:59:00.958 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:00 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:01.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:00 smithi162 ceph-mon[20279]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-02T09:59:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:00 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-02T09:59:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:00 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:00 smithi151 ceph-mon[16639]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-02T09:59:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:00 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-09-02T09:59:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:00 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:01.820 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:01 smithi187 ceph-mon[20072]: Deploying daemon osd.8 on smithi187 2024-09-02T09:59:02.089 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:01 smithi151 ceph-mon[16639]: Deploying daemon osd.8 on smithi187 2024-09-02T09:59:02.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:01 smithi162 ceph-mon[20279]: Deploying daemon osd.8 on smithi187 2024-09-02T09:59:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:02 smithi187 ceph-mon[20072]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-02T09:59:03.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:02 smithi162 ceph-mon[20279]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-02T09:59:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:02 smithi151 ceph-mon[16639]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-02T09:59:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:03 smithi187 ceph-mon[20072]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-02T09:59:04.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:03 smithi162 ceph-mon[20279]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-02T09:59:04.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:03 smithi151 ceph-mon[16639]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-02T09:59:06.190 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:59:06.191 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:59:06.191 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:06 smithi187 ceph-mon[20072]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-09-02T09:59:06.191 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:06 smithi187 ceph-mon[20072]: osdmap e56: 9 total, 8 up, 9 in 2024-09-02T09:59:06.191 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:06.191 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:06.191 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:06.191 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:59:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:59:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:06 smithi162 ceph-mon[20279]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-09-02T09:59:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:06 smithi162 ceph-mon[20279]: osdmap e56: 9 total, 8 up, 9 in 2024-09-02T09:59:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:59:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:59:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:06 smithi151 ceph-mon[16639]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-09-02T09:59:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:06 smithi151 ceph-mon[16639]: osdmap e56: 9 total, 8 up, 9 in 2024-09-02T09:59:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:08 smithi162 ceph-mon[20279]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:08.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:08 smithi151 ceph-mon[16639]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:08.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:08.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:08.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:08.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:08.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:08 smithi187 ceph-mon[20072]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:08.897 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 8 on host 'smithi187' 2024-09-02T09:59:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:09.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:09.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:09.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:09.577 DEBUG:teuthology.orchestra.run.smithi187:osd.8> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.8.service 2024-09-02T09:59:09.580 INFO:tasks.cephadm:Deploying osd.9 on smithi187 with /dev/vg_nvme/lv_3... 2024-09-02T09:59:09.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-02T09:59:10.185 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:10 smithi187 ceph-mon[20072]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:10.203 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:59:10.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:10 smithi162 ceph-mon[20279]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:10.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:10 smithi151 ceph-mon[16639]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:10.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 09:59:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T09:59:10.183+0000 7f4ba6be4640 -1 osd.8 0 log_to_monitors true 2024-09-02T09:59:11.169 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:11 smithi187 ceph-mon[20072]: from='osd.8 [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-02T09:59:11.170 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:11 smithi187 ceph-mon[20072]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-02T09:59:11.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:11 smithi162 ceph-mon[20279]: from='osd.8 [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-02T09:59:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:11 smithi162 ceph-mon[20279]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-02T09:59:11.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:11 smithi151 ceph-mon[16639]: from='osd.8 [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-02T09:59:11.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:11 smithi151 ceph-mon[16639]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-09-02T09:59:12.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:12 smithi162 ceph-mon[20279]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:12 smithi162 ceph-mon[20279]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-02T09:59:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:12 smithi162 ceph-mon[20279]: osdmap e57: 9 total, 8 up, 9 in 2024-09-02T09:59:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:12 smithi162 ceph-mon[20279]: from='osd.8 [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:12 smithi162 ceph-mon[20279]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:12.385 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:12 smithi187 ceph-mon[20072]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:12.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:12 smithi187 ceph-mon[20072]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-02T09:59:12.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:12 smithi187 ceph-mon[20072]: osdmap e57: 9 total, 8 up, 9 in 2024-09-02T09:59:12.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:12.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:12 smithi187 ceph-mon[20072]: from='osd.8 [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:12.386 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:12 smithi187 ceph-mon[20072]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:12.386 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 09:59:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T09:59:12.082+0000 7f4ba2b67640 -1 osd.8 0 waiting for initial osdmap 2024-09-02T09:59:12.386 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 09:59:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T09:59:12.090+0000 7f4b9e98e640 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:59:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:12 smithi151 ceph-mon[16639]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:12 smithi151 ceph-mon[16639]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-09-02T09:59:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:12 smithi151 ceph-mon[16639]: osdmap e57: 9 total, 8 up, 9 in 2024-09-02T09:59:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:12 smithi151 ceph-mon[16639]: from='osd.8 [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:12 smithi151 ceph-mon[16639]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T09:59:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: osdmap e58: 9 total, 8 up, 9 in 2024-09-02T09:59:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi187 2024-09-02T09:59:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi187 to 21184M 2024-09-02T09:59:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T09:59:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: osdmap e58: 9 total, 8 up, 9 in 2024-09-02T09:59:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi187 2024-09-02T09:59:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi187 to 21184M 2024-09-02T09:59:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:13.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:13.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: osdmap e58: 9 total, 8 up, 9 in 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi187 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi187 to 21184M 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:13.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:13.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:59:14.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi187:vg_nvme/lv_3 2024-09-02T09:59:14.323 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:59:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:14 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:59:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:14 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:59:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:14 smithi162 ceph-mon[20279]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:14 smithi162 ceph-mon[20279]: osd.8 [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013] boot 2024-09-02T09:59:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:14 smithi162 ceph-mon[20279]: osdmap e59: 9 total, 9 up, 9 in 2024-09-02T09:59:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:14 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:14 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:59:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:14 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:59:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:14 smithi151 ceph-mon[16639]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:14 smithi151 ceph-mon[16639]: osd.8 [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013] boot 2024-09-02T09:59:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:14 smithi151 ceph-mon[16639]: osdmap e59: 9 total, 9 up, 9 in 2024-09-02T09:59:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:14 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:14.443 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:14 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:59:14.443 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:14 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:59:14.443 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:14 smithi187 ceph-mon[20072]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-02T09:59:14.443 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:14 smithi187 ceph-mon[20072]: osd.8 [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013] boot 2024-09-02T09:59:14.443 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:14 smithi187 ceph-mon[20072]: osdmap e59: 9 total, 9 up, 9 in 2024-09-02T09:59:14.443 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-09-02T09:59:15.366 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:15 smithi187 ceph-mon[20072]: osdmap e60: 9 total, 9 up, 9 in 2024-09-02T09:59:15.366 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:15 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:59:15.366 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:15 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:59:15.366 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:15 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:15 smithi151 ceph-mon[16639]: osdmap e60: 9 total, 9 up, 9 in 2024-09-02T09:59:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:15 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:59:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:15 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:59:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:15 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:15 smithi162 ceph-mon[20279]: osdmap e60: 9 total, 9 up, 9 in 2024-09-02T09:59:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:15 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:59:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:15 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:59:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:15 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:16.361 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:16 smithi187 ceph-mon[20072]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:59:16.361 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:16 smithi187 ceph-mon[20072]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:16 smithi151 ceph-mon[16639]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:59:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:16 smithi151 ceph-mon[16639]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:16 smithi162 ceph-mon[20279]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:59:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:16 smithi162 ceph-mon[20279]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:18 smithi151 ceph-mon[16639]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-02T09:59:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/3748799469' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b"} : dispatch 2024-09-02T09:59:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:18 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b"} : dispatch 2024-09-02T09:59:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:18 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b"}]': finished 2024-09-02T09:59:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:18 smithi151 ceph-mon[16639]: osdmap e61: 10 total, 9 up, 10 in 2024-09-02T09:59:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:18 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:18 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:59:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:18 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:59:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/1350511807' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:59:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:18 smithi187 ceph-mon[20072]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-02T09:59:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/3748799469' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b"} : dispatch 2024-09-02T09:59:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:18 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b"} : dispatch 2024-09-02T09:59:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:18 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b"}]': finished 2024-09-02T09:59:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:18 smithi187 ceph-mon[20072]: osdmap e61: 10 total, 9 up, 10 in 2024-09-02T09:59:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:18 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:18 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:59:18.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:18 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:59:18.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/1350511807' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:59:18.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:18 smithi162 ceph-mon[20279]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-02T09:59:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.187:0/3748799469' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b"} : dispatch 2024-09-02T09:59:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:18 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b"} : dispatch 2024-09-02T09:59:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:18 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b"}]': finished 2024-09-02T09:59:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:18 smithi162 ceph-mon[20279]: osdmap e61: 10 total, 9 up, 10 in 2024-09-02T09:59:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:18 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:18 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:59:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:18 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:59:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.187:0/1350511807' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:59:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:19 smithi187 ceph-mon[20072]: osdmap e62: 10 total, 9 up, 10 in 2024-09-02T09:59:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:19.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:19 smithi162 ceph-mon[20279]: osdmap e62: 10 total, 9 up, 10 in 2024-09-02T09:59:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:19.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:19 smithi151 ceph-mon[16639]: osdmap e62: 10 total, 9 up, 10 in 2024-09-02T09:59:19.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:20 smithi187 ceph-mon[20072]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-02T09:59:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:20 smithi162 ceph-mon[20279]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-02T09:59:20.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:20 smithi151 ceph-mon[16639]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-09-02T09:59:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:22 smithi187 ceph-mon[20072]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-02T09:59:22.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:22 smithi162 ceph-mon[20279]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-02T09:59:22.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:22 smithi151 ceph-mon[16639]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-09-02T09:59:24.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:24 smithi187 ceph-mon[20072]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:59:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:24 smithi162 ceph-mon[20279]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:59:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:24 smithi151 ceph-mon[16639]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T09:59:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:25 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T09:59:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:25 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T09:59:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-02T09:59:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:25 smithi187 ceph-mon[20072]: Deploying daemon osd.9 on smithi187 2024-09-02T09:59:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:25 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:59:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:25 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:59:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:25 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T09:59:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:25 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T09:59:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-02T09:59:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:25 smithi162 ceph-mon[20279]: Deploying daemon osd.9 on smithi187 2024-09-02T09:59:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:25 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:59:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:25 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:59:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:25 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T09:59:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:25 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T09:59:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-09-02T09:59:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:25 smithi151 ceph-mon[16639]: Deploying daemon osd.9 on smithi187 2024-09-02T09:59:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:25 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:59:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:25 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:59:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:26 smithi187 ceph-mon[20072]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:26 smithi187 ceph-mon[20072]: osdmap e63: 10 total, 9 up, 10 in 2024-09-02T09:59:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:26 smithi162 ceph-mon[20279]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:26 smithi162 ceph-mon[20279]: osdmap e63: 10 total, 9 up, 10 in 2024-09-02T09:59:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:26 smithi151 ceph-mon[16639]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:26 smithi151 ceph-mon[16639]: osdmap e63: 10 total, 9 up, 10 in 2024-09-02T09:59:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:28.452 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:28 smithi187 ceph-mon[20072]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:28.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:28 smithi162 ceph-mon[20279]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:28.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:28 smithi151 ceph-mon[16639]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:30.428 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:30 smithi187 ceph-mon[20072]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:30.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:30.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:30.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:30.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:30 smithi162 ceph-mon[20279]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:30 smithi151 ceph-mon[16639]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:59:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:59:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:59:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:59:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:59:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:59:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:59:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:59:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:59:32.798 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 9 on host 'smithi187' 2024-09-02T09:59:33.145 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.145 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.145 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:33.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:33.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:33.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:33.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:33.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:33.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:33.466 DEBUG:teuthology.orchestra.run.smithi187:osd.9> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.9.service 2024-09-02T09:59:33.469 INFO:tasks.cephadm:Deploying osd.10 on smithi187 with /dev/vg_nvme/lv_2... 2024-09-02T09:59:33.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-02T09:59:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:33 smithi151 ceph-mon[16639]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:33 smithi187 ceph-mon[20072]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:34.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:33 smithi162 ceph-mon[20279]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:34.540 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:59:34.620 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 09:59:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T09:59:34.337+0000 7f0e18e37640 -1 osd.9 0 log_to_monitors true 2024-09-02T09:59:35.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:34 smithi187 ceph-mon[20072]: from='osd.9 [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-02T09:59:35.182 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:34 smithi187 ceph-mon[20072]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-02T09:59:35.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:34 smithi151 ceph-mon[16639]: from='osd.9 [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-02T09:59:35.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:34 smithi151 ceph-mon[16639]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-02T09:59:35.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:34 smithi162 ceph-mon[20279]: from='osd.9 [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-02T09:59:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:34 smithi162 ceph-mon[20279]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-09-02T09:59:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:35 smithi151 ceph-mon[16639]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-02T09:59:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:35 smithi151 ceph-mon[16639]: from='osd.9 [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:35 smithi151 ceph-mon[16639]: osdmap e64: 10 total, 9 up, 10 in 2024-09-02T09:59:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:35 smithi151 ceph-mon[16639]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:35 smithi151 ceph-mon[16639]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:35 smithi187 ceph-mon[20072]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-02T09:59:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:35 smithi187 ceph-mon[20072]: from='osd.9 [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:35 smithi187 ceph-mon[20072]: osdmap e64: 10 total, 9 up, 10 in 2024-09-02T09:59:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:35 smithi187 ceph-mon[20072]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:35 smithi187 ceph-mon[20072]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:35 smithi162 ceph-mon[20279]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-09-02T09:59:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:35 smithi162 ceph-mon[20279]: from='osd.9 [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:35 smithi162 ceph-mon[20279]: osdmap e64: 10 total, 9 up, 10 in 2024-09-02T09:59:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:35 smithi162 ceph-mon[20279]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:35 smithi162 ceph-mon[20279]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:36.731 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T09:59:36.367+0000 7f0e14dba640 -1 osd.9 0 waiting for initial osdmap 2024-09-02T09:59:36.732 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T09:59:36.377+0000 7f0e0cbd9640 -1 osd.9 65 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T09:59:37.003 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: osdmap e65: 10 total, 9 up, 10 in 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='osd.9 ' entity='osd.9' 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.004 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.005 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi187 to 10592M 2024-09-02T09:59:37.005 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:37.005 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:37.005 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:37.005 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:37.005 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:37.005 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:37.005 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:37.005 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:36 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: osdmap e65: 10 total, 9 up, 10 in 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='osd.9 ' entity='osd.9' 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:59:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi187 to 10592M 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:37.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:36 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T09:59:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: osdmap e65: 10 total, 9 up, 10 in 2024-09-02T09:59:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='osd.9 ' entity='osd.9' 2024-09-02T09:59:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T09:59:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi187 to 10592M 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:37.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:37.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:36 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:38.058 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T09:59:38.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi187:vg_nvme/lv_2 2024-09-02T09:59:38.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:37 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:59:38.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:37 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:59:38.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:37 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:38.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:37 smithi151 ceph-mon[16639]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:38.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:37 smithi151 ceph-mon[16639]: osd.9 [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727] boot 2024-09-02T09:59:38.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:37 smithi151 ceph-mon[16639]: osdmap e66: 10 total, 10 up, 10 in 2024-09-02T09:59:38.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:37 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:37 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:59:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:37 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:59:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:37 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:37 smithi187 ceph-mon[20072]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:37 smithi187 ceph-mon[20072]: osd.9 [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727] boot 2024-09-02T09:59:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:37 smithi187 ceph-mon[20072]: osdmap e66: 10 total, 10 up, 10 in 2024-09-02T09:59:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:37 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:38.325 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:59:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:37 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:59:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:37 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:59:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:37 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:37 smithi162 ceph-mon[20279]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-09-02T09:59:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:37 smithi162 ceph-mon[20279]: osd.9 [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727] boot 2024-09-02T09:59:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:37 smithi162 ceph-mon[20279]: osdmap e66: 10 total, 10 up, 10 in 2024-09-02T09:59:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:37 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-09-02T09:59:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:39 smithi151 ceph-mon[16639]: osdmap e67: 10 total, 10 up, 10 in 2024-09-02T09:59:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:59:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:59:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:39.706 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:39 smithi187 ceph-mon[20072]: osdmap e67: 10 total, 10 up, 10 in 2024-09-02T09:59:39.706 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:59:39.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:59:39.707 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:39 smithi162 ceph-mon[20279]: osdmap e67: 10 total, 10 up, 10 in 2024-09-02T09:59:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T09:59:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T09:59:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:40.576 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:40 smithi187 ceph-mon[20072]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:59:40.576 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:40 smithi187 ceph-mon[20072]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:40.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:40 smithi151 ceph-mon[16639]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:59:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:40 smithi151 ceph-mon[16639]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:40 smithi162 ceph-mon[20279]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T09:59:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:40 smithi162 ceph-mon[20279]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:41.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/461639519' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a"} : dispatch 2024-09-02T09:59:41.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:41 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a"} : dispatch 2024-09-02T09:59:41.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:41 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a"}]': finished 2024-09-02T09:59:41.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:41 smithi151 ceph-mon[16639]: osdmap e68: 11 total, 10 up, 11 in 2024-09-02T09:59:41.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:41 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:41.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/461639519' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a"} : dispatch 2024-09-02T09:59:41.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:41 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a"} : dispatch 2024-09-02T09:59:41.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:41 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a"}]': finished 2024-09-02T09:59:41.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:41 smithi187 ceph-mon[20072]: osdmap e68: 11 total, 10 up, 11 in 2024-09-02T09:59:41.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:41 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:41.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.187:0/461639519' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a"} : dispatch 2024-09-02T09:59:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:41 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a"} : dispatch 2024-09-02T09:59:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:41 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a"}]': finished 2024-09-02T09:59:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:41 smithi162 ceph-mon[20279]: osdmap e68: 11 total, 10 up, 11 in 2024-09-02T09:59:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:41 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:42.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:42 smithi151 ceph-mon[16639]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:42.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:42 smithi151 ceph-mon[16639]: from='client.24280 172.21.15.187:0/2105770763' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:59:42.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:42 smithi187 ceph-mon[20072]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:42.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:42 smithi187 ceph-mon[20072]: from='client.24280 172.21.15.187:0/2105770763' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:59:42.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:42 smithi162 ceph-mon[20279]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:42.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:42 smithi162 ceph-mon[20279]: from='client.24280 172.21.15.187:0/2105770763' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T09:59:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:44 smithi151 ceph-mon[16639]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:44 smithi187 ceph-mon[20072]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:44.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:44 smithi162 ceph-mon[20279]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:46.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:46 smithi151 ceph-mon[16639]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:46.727 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:46 smithi187 ceph-mon[20072]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:46.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:46 smithi162 ceph-mon[20279]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:48 smithi151 ceph-mon[16639]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:48 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-02T09:59:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:48 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:48.706 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:48 smithi187 ceph-mon[20072]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:48.706 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:48 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-02T09:59:48.706 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:48 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:48 smithi162 ceph-mon[20279]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:48 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-09-02T09:59:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:48 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:49.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:49 smithi151 ceph-mon[16639]: Deploying daemon osd.10 on smithi187 2024-09-02T09:59:49.720 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:49 smithi187 ceph-mon[20072]: Deploying daemon osd.10 on smithi187 2024-09-02T09:59:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:49 smithi162 ceph-mon[20279]: Deploying daemon osd.10 on smithi187 2024-09-02T09:59:50.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:50 smithi151 ceph-mon[16639]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:59:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:59:50.716 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:50 smithi187 ceph-mon[20072]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:50.716 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:59:50.717 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:59:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:50 smithi162 ceph-mon[20279]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:59:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:59:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:51 smithi151 ceph-mon[16639]: osdmap e69: 11 total, 10 up, 11 in 2024-09-02T09:59:51.700 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:51.700 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:51 smithi187 ceph-mon[20072]: osdmap e69: 11 total, 10 up, 11 in 2024-09-02T09:59:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:51 smithi162 ceph-mon[20279]: osdmap e69: 11 total, 10 up, 11 in 2024-09-02T09:59:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:52 smithi151 ceph-mon[16639]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:52 smithi162 ceph-mon[20279]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:52.754 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:52 smithi187 ceph-mon[20072]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:53.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:53 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:53 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:53 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:53 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:53 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:53 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:53.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:53 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:53 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:53 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:54.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:54 smithi151 ceph-mon[16639]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:54 smithi187 ceph-mon[20072]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:54.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:54 smithi162 ceph-mon[20279]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:55.722 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:55 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:59:55.722 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:55 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:59:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:55 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:59:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:55 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:59:55.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:55 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:59:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:55 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:59:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:56 smithi187 ceph-mon[20072]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:56 smithi187 ceph-mon[20072]: osdmap e70: 11 total, 10 up, 11 in 2024-09-02T09:59:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:56 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:56 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:56 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:56 smithi162 ceph-mon[20279]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:56 smithi162 ceph-mon[20279]: osdmap e70: 11 total, 10 up, 11 in 2024-09-02T09:59:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:56 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:56 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:56 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:56 smithi151 ceph-mon[16639]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:56 smithi151 ceph-mon[16639]: osdmap e70: 11 total, 10 up, 11 in 2024-09-02T09:59:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:56 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:56 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T09:59:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T09:59:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:56 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:57.108 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 10 on host 'smithi187' 2024-09-02T09:59:57.744 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:57 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:57.744 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:57 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:57.744 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:57 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:57.770 DEBUG:teuthology.orchestra.run.smithi187:osd.10> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.10.service 2024-09-02T09:59:57.772 INFO:tasks.cephadm:Deploying osd.11 on smithi187 with /dev/vg_nvme/lv_1... 2024-09-02T09:59:57.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-02T09:59:57.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:57 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:57.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:57 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:57 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:57.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:57 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T09:59:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:57 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:57 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T09:59:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:58 smithi187 ceph-mon[20072]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:58 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:59:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:58 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:59:58.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:58 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T09:59:58.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:58 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T09:59:58.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:58 smithi187 ceph-mon[20072]: from='osd.10 [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-02T09:59:58.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:58 smithi187 ceph-mon[20072]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-02T09:59:58.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 09:59:58 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T09:59:58.398+0000 7f7ed52b1640 -1 osd.10 0 log_to_monitors true 2024-09-02T09:59:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:58 smithi162 ceph-mon[20279]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:58 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:59:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:58 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:59:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:58 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T09:59:58.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:58 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T09:59:58.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:58 smithi162 ceph-mon[20279]: from='osd.10 [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-02T09:59:58.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:58 smithi162 ceph-mon[20279]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-02T09:59:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:58 smithi151 ceph-mon[16639]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T09:59:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:58 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:59:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:58 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:59:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:58 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T09:59:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:58 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T09:59:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:58 smithi151 ceph-mon[16639]: from='osd.10 [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-02T09:59:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:58 smithi151 ceph-mon[16639]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-09-02T09:59:58.998 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T09:59:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:59 smithi187 ceph-mon[20072]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-02T09:59:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:59 smithi187 ceph-mon[20072]: osdmap e71: 11 total, 10 up, 11 in 2024-09-02T09:59:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:59 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:59 smithi187 ceph-mon[20072]: from='osd.10 [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 09:59:59 smithi187 ceph-mon[20072]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:59 smithi162 ceph-mon[20279]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-02T09:59:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:59 smithi162 ceph-mon[20279]: osdmap e71: 11 total, 10 up, 11 in 2024-09-02T09:59:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:59 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:59 smithi162 ceph-mon[20279]: from='osd.10 [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 09:59:59 smithi162 ceph-mon[20279]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:59 smithi151 ceph-mon[16639]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-09-02T09:59:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:59 smithi151 ceph-mon[16639]: osdmap e71: 11 total, 10 up, 11 in 2024-09-02T09:59:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T09:59:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:59 smithi151 ceph-mon[16639]: from='osd.10 [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T09:59:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 09:59:59 smithi151 ceph-mon[16639]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T10:00:00.234 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 09:59:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T09:59:59.919+0000 7f7ed1234640 -1 osd.10 0 waiting for initial osdmap 2024-09-02T10:00:00.235 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 09:59:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T09:59:59.927+0000 7f7ec9053640 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T10:00:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:00 smithi187 ceph-mon[20072]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T10:00:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:00 smithi187 ceph-mon[20072]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T10:00:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:00 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:00 smithi187 ceph-mon[20072]: osdmap e72: 11 total, 10 up, 11 in 2024-09-02T10:00:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:00 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:00 smithi187 ceph-mon[20072]: from='osd.10 ' entity='osd.10' 2024-09-02T10:00:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:00 smithi187 ceph-mon[20072]: overall HEALTH_OK 2024-09-02T10:00:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:00 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:00.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:00 smithi162 ceph-mon[20279]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T10:00:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:00 smithi162 ceph-mon[20279]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T10:00:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:00 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:00 smithi162 ceph-mon[20279]: osdmap e72: 11 total, 10 up, 11 in 2024-09-02T10:00:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:00 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:00 smithi162 ceph-mon[20279]: from='osd.10 ' entity='osd.10' 2024-09-02T10:00:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:00 smithi162 ceph-mon[20279]: overall HEALTH_OK 2024-09-02T10:00:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:00 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:00.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:00 smithi151 ceph-mon[16639]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-09-02T10:00:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:00 smithi151 ceph-mon[16639]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T10:00:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:00 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:00 smithi151 ceph-mon[16639]: osdmap e72: 11 total, 10 up, 11 in 2024-09-02T10:00:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:00 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:00 smithi151 ceph-mon[16639]: from='osd.10 ' entity='osd.10' 2024-09-02T10:00:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:00 smithi151 ceph-mon[16639]: overall HEALTH_OK 2024-09-02T10:00:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:00 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:00:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:00:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: osd.10 [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597] boot 2024-09-02T10:00:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: osdmap e73: 11 total, 11 up, 11 in 2024-09-02T10:00:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi187 2024-09-02T10:00:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:00:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:00:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi187 to 7061M 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:02.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:02.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: osd.10 [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597] boot 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: osdmap e73: 11 total, 11 up, 11 in 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi187 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi187 to 7061M 2024-09-02T10:00:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:00:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:00:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: osd.10 [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597] boot 2024-09-02T10:00:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: osdmap e73: 11 total, 11 up, 11 in 2024-09-02T10:00:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi187 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi187 to 7061M 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:02.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:02.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:02.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:02.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:02.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:02.674 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-09-02T10:00:02.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph orch daemon add osd smithi187:vg_nvme/lv_1 2024-09-02T10:00:02.946 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:00:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:03 smithi151 ceph-mon[16639]: osdmap e74: 11 total, 11 up, 11 in 2024-09-02T10:00:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:03 smithi187 ceph-mon[20072]: osdmap e74: 11 total, 11 up, 11 in 2024-09-02T10:00:03.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:03 smithi162 ceph-mon[20279]: osdmap e74: 11 total, 11 up, 11 in 2024-09-02T10:00:04.442 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:04 smithi187 ceph-mon[20072]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:04.442 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:04 smithi187 ceph-mon[20072]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:00:04.442 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:04 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T10:00:04.442 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:04 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T10:00:04.442 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:04 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:04 smithi151 ceph-mon[16639]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:04 smithi151 ceph-mon[16639]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:00:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T10:00:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T10:00:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:04 smithi162 ceph-mon[20279]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:04 smithi162 ceph-mon[20279]: from='client.24295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi187:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:00:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:04 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-09-02T10:00:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:04 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-09-02T10:00:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:04 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:06 smithi187 ceph-mon[20072]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/120474035' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2"} : dispatch 2024-09-02T10:00:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:06 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2"} : dispatch 2024-09-02T10:00:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:06 smithi187 ceph-mon[20072]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2"}]': finished 2024-09-02T10:00:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:06 smithi187 ceph-mon[20072]: osdmap e75: 12 total, 11 up, 12 in 2024-09-02T10:00:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:06 smithi151 ceph-mon[16639]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/120474035' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2"} : dispatch 2024-09-02T10:00:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:06 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2"} : dispatch 2024-09-02T10:00:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:06 smithi151 ceph-mon[16639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2"}]': finished 2024-09-02T10:00:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:06 smithi151 ceph-mon[16639]: osdmap e75: 12 total, 11 up, 12 in 2024-09-02T10:00:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:06 smithi162 ceph-mon[20279]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.187:0/120474035' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2"} : dispatch 2024-09-02T10:00:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:06 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2"} : dispatch 2024-09-02T10:00:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:06 smithi162 ceph-mon[20279]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2"}]': finished 2024-09-02T10:00:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:06 smithi162 ceph-mon[20279]: osdmap e75: 12 total, 11 up, 12 in 2024-09-02T10:00:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/1595562709' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T10:00:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:07 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:00:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:07 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:00:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/1595562709' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T10:00:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:07 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:00:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:07 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:00:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.187:0/1595562709' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-09-02T10:00:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:07 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:00:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:07 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:00:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:08 smithi151 ceph-mon[16639]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:08 smithi151 ceph-mon[16639]: osdmap e76: 12 total, 11 up, 12 in 2024-09-02T10:00:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:08 smithi187 ceph-mon[20072]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:08 smithi187 ceph-mon[20072]: osdmap e76: 12 total, 11 up, 12 in 2024-09-02T10:00:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:08.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:08 smithi162 ceph-mon[20279]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:08 smithi162 ceph-mon[20279]: osdmap e76: 12 total, 11 up, 12 in 2024-09-02T10:00:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:10 smithi151 ceph-mon[16639]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:10 smithi187 ceph-mon[20072]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:10.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:10 smithi162 ceph-mon[20279]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:12.650 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:12 smithi187 ceph-mon[20072]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:12.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:12 smithi151 ceph-mon[16639]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:12.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:12 smithi162 ceph-mon[20279]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:14.667 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:14 smithi187 ceph-mon[20072]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:14.667 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-02T10:00:14.668 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:14.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:14 smithi151 ceph-mon[16639]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:14 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-02T10:00:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:14 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:14.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:14 smithi162 ceph-mon[20279]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:14.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:14 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-09-02T10:00:14.708 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:14 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:15.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:15 smithi187 ceph-mon[20072]: Deploying daemon osd.11 on smithi187 2024-09-02T10:00:15.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:15 smithi151 ceph-mon[16639]: Deploying daemon osd.11 on smithi187 2024-09-02T10:00:15.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:15 smithi162 ceph-mon[20279]: Deploying daemon osd.11 on smithi187 2024-09-02T10:00:16.618 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:16 smithi187 ceph-mon[20072]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:16.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:16 smithi151 ceph-mon[16639]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:16.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:16 smithi162 ceph-mon[20279]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:18 smithi187 ceph-mon[20072]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:18.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:18 smithi151 ceph-mon[16639]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:18.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:18 smithi162 ceph-mon[20279]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:19 smithi187 ceph-mon[20072]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:20.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:19 smithi162 ceph-mon[20279]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:20.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:20.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:20.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:20.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:19 smithi151 ceph-mon[16639]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:21 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:00:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:21 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:00:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:21 smithi162 ceph-mon[20279]: osdmap e77: 12 total, 11 up, 12 in 2024-09-02T10:00:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:21 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:21 smithi162 ceph-mon[20279]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:21 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:21 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:21 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:22.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:21 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:22.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:21 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:21 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:00:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:21 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:00:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:21 smithi151 ceph-mon[16639]: osdmap e77: 12 total, 11 up, 12 in 2024-09-02T10:00:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:21 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:21 smithi151 ceph-mon[16639]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:21 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:21 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:21 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:21 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:21 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:21 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:00:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:21 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:00:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:21 smithi187 ceph-mon[20072]: osdmap e77: 12 total, 11 up, 12 in 2024-09-02T10:00:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:21 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:21 smithi187 ceph-mon[20072]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:21 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:21 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:21 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:22.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:21 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:22.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:21 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.343 INFO:teuthology.orchestra.run.smithi187.stdout:Created osd(s) 11 on host 'smithi187' 2024-09-02T10:00:22.972 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:22.973 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.973 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:22.999 DEBUG:teuthology.orchestra.run.smithi187:osd.11> sudo journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.11.service 2024-09-02T10:00:23.001 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-09-02T10:00:23.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd stat -f json 2024-09-02T10:00:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:23.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:23.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:23.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:23.262 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:23.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:00:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:00:23.593+0000 7f0e88717640 -1 osd.11 0 log_to_monitors true 2024-09-02T10:00:23.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:23 smithi151 ceph-mon[16639]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:23 smithi151 ceph-mon[16639]: from='osd.11 [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-02T10:00:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:23 smithi151 ceph-mon[16639]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-02T10:00:23.949 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:00:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:23 smithi162 ceph-mon[20279]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:23 smithi162 ceph-mon[20279]: from='osd.11 [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-02T10:00:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:23 smithi162 ceph-mon[20279]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-02T10:00:24.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:23 smithi187 ceph-mon[20072]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:24.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:23 smithi187 ceph-mon[20072]: from='osd.11 [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-02T10:00:24.179 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:23 smithi187 ceph-mon[20072]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-09-02T10:00:24.690 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":78,"num_osds":12,"num_up_osds":11,"osd_up_since":1725271200,"num_in_osds":12,"osd_in_since":1725271205,"num_remapped_pgs":0} 2024-09-02T10:00:24.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:24 smithi151 ceph-mon[16639]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-02T10:00:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:24 smithi151 ceph-mon[16639]: from='osd.11 [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T10:00:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:24 smithi151 ceph-mon[16639]: osdmap e78: 12 total, 11 up, 12 in 2024-09-02T10:00:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:24 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:24 smithi151 ceph-mon[16639]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T10:00:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1787992877' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T10:00:25.093 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:24 smithi187 ceph-mon[20072]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-02T10:00:25.093 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:24 smithi187 ceph-mon[20072]: from='osd.11 [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T10:00:25.094 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:24 smithi187 ceph-mon[20072]: osdmap e78: 12 total, 11 up, 12 in 2024-09-02T10:00:25.094 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:24 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:25.094 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:24 smithi187 ceph-mon[20072]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T10:00:25.094 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1787992877' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T10:00:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:24 smithi162 ceph-mon[20279]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-09-02T10:00:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:24 smithi162 ceph-mon[20279]: from='osd.11 [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T10:00:25.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:24 smithi162 ceph-mon[20279]: osdmap e78: 12 total, 11 up, 12 in 2024-09-02T10:00:25.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:24 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:25.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:24 smithi162 ceph-mon[20279]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]} : dispatch 2024-09-02T10:00:25.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1787992877' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T10:00:25.402 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:00:25.258+0000 7f0e8469a640 -1 osd.11 0 waiting for initial osdmap 2024-09-02T10:00:25.402 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:00:25.265+0000 7f0e7c4b9640 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-02T10:00:25.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd stat -f json 2024-09-02T10:00:25.953 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T10:00:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: osdmap e79: 12 total, 11 up, 12 in 2024-09-02T10:00:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='osd.11 ' entity='osd.11' 2024-09-02T10:00:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T10:00:26.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: osdmap e79: 12 total, 11 up, 12 in 2024-09-02T10:00:26.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='osd.11 ' entity='osd.11' 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:26.075 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi187", "root=default"]}]': finished 2024-09-02T10:00:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: osdmap e79: 12 total, 11 up, 12 in 2024-09-02T10:00:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-09-02T10:00:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='osd.11 ' entity='osd.11' 2024-09-02T10:00:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-09-02T10:00:26.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:26.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:26.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:26.625 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:00:26.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:26 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:00:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:26 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:00:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:26 smithi151 ceph-mon[16639]: Detected new or changed devices on smithi187 2024-09-02T10:00:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:26 smithi151 ceph-mon[16639]: Adjusting osd_memory_target on smithi187 to 5296M 2024-09-02T10:00:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:26 smithi151 ceph-mon[16639]: osd.11 [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212] boot 2024-09-02T10:00:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:26 smithi151 ceph-mon[16639]: osdmap e80: 12 total, 12 up, 12 in 2024-09-02T10:00:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:26 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2997302927' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T10:00:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:26 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:00:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:26 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:00:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:26 smithi162 ceph-mon[20279]: Detected new or changed devices on smithi187 2024-09-02T10:00:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:26 smithi162 ceph-mon[20279]: Adjusting osd_memory_target on smithi187 to 5296M 2024-09-02T10:00:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:26 smithi162 ceph-mon[20279]: osd.11 [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212] boot 2024-09-02T10:00:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:26 smithi162 ceph-mon[20279]: osdmap e80: 12 total, 12 up, 12 in 2024-09-02T10:00:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:26 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2997302927' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T10:00:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:26 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:00:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:26 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:00:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:26 smithi187 ceph-mon[20072]: Detected new or changed devices on smithi187 2024-09-02T10:00:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:26 smithi187 ceph-mon[20072]: Adjusting osd_memory_target on smithi187 to 5296M 2024-09-02T10:00:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:26 smithi187 ceph-mon[20072]: osd.11 [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212] boot 2024-09-02T10:00:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:26 smithi187 ceph-mon[20072]: osdmap e80: 12 total, 12 up, 12 in 2024-09-02T10:00:27.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:26 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-09-02T10:00:27.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2997302927' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-09-02T10:00:27.301 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":12,"osd_in_since":1725271205,"num_remapped_pgs":1} 2024-09-02T10:00:27.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-09-02T10:00:27.549 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:27.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:00:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:00:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:27 smithi151 ceph-mon[16639]: osdmap e81: 12 total, 12 up, 12 in 2024-09-02T10:00:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:27 smithi151 ceph-mon[16639]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:00:28.227 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":82,"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","created":"2024-09-02T09:53:53.308187+0000","modified":"2024-09-02T10:00:27.840595+0000","last_up_change":"2024-09-02T10:00:25.816021+0000","last_in_change":"2024-09-02T10:00:05.936786+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-02T09:56:53.284814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e76499ad-5f7e-4788-aa85-0bad0d843e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6803","nonce":1059446772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6805","nonce":1059446772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6809","nonce":1059446772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6807","nonce":1059446772}]},"public_addr":"172.21.15.151:6803/1059446772","cluster_addr":"172.21.15.151:6805/1059446772","heartbeat_back_addr":"172.21.15.151:6809/1059446772","heartbeat_front_addr":"172.21.15.151:6807/1059446772","state":["exists","up"]},{"osd":1,"uuid":"8949ba9c-4a71-4327-a1dc-e7d25286d8f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6811","nonce":457777866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6813","nonce":457777866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6817","nonce":457777866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6815","nonce":457777866}]},"public_addr":"172.21.15.151:6811/457777866","cluster_addr":"172.21.15.151:6813/457777866","heartbeat_back_addr":"172.21.15.151:6817/457777866","heartbeat_front_addr":"172.21.15.151:6815/457777866","state":["exists","up"]},{"osd":2,"uuid":"2dead774-0f59-4e90-8503-9a3f0fb68b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6819","nonce":654044340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6821","nonce":654044340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6825","nonce":654044340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6823","nonce":654044340}]},"public_addr":"172.21.15.151:6819/654044340","cluster_addr":"172.21.15.151:6821/654044340","heartbeat_back_addr":"172.21.15.151:6825/654044340","heartbeat_front_addr":"172.21.15.151:6823/654044340","state":["exists","up"]},{"osd":3,"uuid":"524f32ed-95aa-4459-9585-72577db12107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6827","nonce":1912547696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6829","nonce":1912547696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6833","nonce":1912547696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6831","nonce":1912547696}]},"public_addr":"172.21.15.151:6827/1912547696","cluster_addr":"172.21.15.151:6829/1912547696","heartbeat_back_addr":"172.21.15.151:6833/1912547696","heartbeat_front_addr":"172.21.15.151:6831/1912547696","state":["exists","up"]},{"osd":4,"uuid":"2c43a79f-8ee6-4088-af19-6144fc8bdbe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6801","nonce":295415389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6803","nonce":295415389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6807","nonce":295415389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6805","nonce":295415389}]},"public_addr":"172.21.15.162:6801/295415389","cluster_addr":"172.21.15.162:6803/295415389","heartbeat_back_addr":"172.21.15.162:6807/295415389","heartbeat_front_addr":"172.21.15.162:6805/295415389","state":["exists","up"]},{"osd":5,"uuid":"71810aab-c54f-4048-9fe4-d1272302d53a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6809","nonce":3782691981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6811","nonce":3782691981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6815","nonce":3782691981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6813","nonce":3782691981}]},"public_addr":"172.21.15.162:6809/3782691981","cluster_addr":"172.21.15.162:6811/3782691981","heartbeat_back_addr":"172.21.15.162:6815/3782691981","heartbeat_front_addr":"172.21.15.162:6813/3782691981","state":["exists","up"]},{"osd":6,"uuid":"45f93bd7-8e30-43ee-b57a-c007aec6d308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6817","nonce":759599342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6819","nonce":759599342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6823","nonce":759599342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6821","nonce":759599342}]},"public_addr":"172.21.15.162:6817/759599342","cluster_addr":"172.21.15.162:6819/759599342","heartbeat_back_addr":"172.21.15.162:6823/759599342","heartbeat_front_addr":"172.21.15.162:6821/759599342","state":["exists","up"]},{"osd":7,"uuid":"e9ab0b99-0451-4ebd-8758-cbde6350c764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6825","nonce":2389923814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6827","nonce":2389923814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6831","nonce":2389923814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6829","nonce":2389923814}]},"public_addr":"172.21.15.162:6825/2389923814","cluster_addr":"172.21.15.162:6827/2389923814","heartbeat_back_addr":"172.21.15.162:6831/2389923814","heartbeat_front_addr":"172.21.15.162:6829/2389923814","state":["exists","up"]},{"osd":8,"uuid":"b0cf652b-4d1b-441a-85a8-fd5c16380634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6801","nonce":3014427013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6803","nonce":3014427013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6807","nonce":3014427013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6805","nonce":3014427013}]},"public_addr":"172.21.15.187:6801/3014427013","cluster_addr":"172.21.15.187:6803/3014427013","heartbeat_back_addr":"172.21.15.187:6807/3014427013","heartbeat_front_addr":"172.21.15.187:6805/3014427013","state":["exists","up"]},{"osd":9,"uuid":"1752a57c-aab9-4f0d-b2fd-629489170a0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6809","nonce":108453727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6811","nonce":108453727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6815","nonce":108453727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6813","nonce":108453727}]},"public_addr":"172.21.15.187:6809/108453727","cluster_addr":"172.21.15.187:6811/108453727","heartbeat_back_addr":"172.21.15.187:6815/108453727","heartbeat_front_addr":"172.21.15.187:6813/108453727","state":["exists","up"]},{"osd":10,"uuid":"6f48d49d-6636-4142-8cf1-04f371f6317a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6817","nonce":401612597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6819","nonce":401612597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6823","nonce":401612597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6821","nonce":401612597}]},"public_addr":"172.21.15.187:6817/401612597","cluster_addr":"172.21.15.187:6819/401612597","heartbeat_back_addr":"172.21.15.187:6823/401612597","heartbeat_front_addr":"172.21.15.187:6821/401612597","state":["exists","up"]},{"osd":11,"uuid":"f545e120-27c3-4fc6-883f-7c588d0eb8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6825","nonce":3639916212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6827","nonce":3639916212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6831","nonce":3639916212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6829","nonce":3639916212}]},"public_addr":"172.21.15.187:6825/3639916212","cluster_addr":"172.21.15.187:6827/3639916212","heartbeat_back_addr":"172.21.15.187:6831/3639916212","heartbeat_front_addr":"172.21.15.187:6829/3639916212","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:57.639013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:20.329120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:24.298463+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:57.947972+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:54.740043+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:26.755646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:49.640051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:06.880725+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:11.153654+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:35.307683+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:59.380057+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:24.608733+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/1566053508":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/1701887065":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/2339434463":"2024-09-03T09:54:15.686961+0000","172.21.15.151:6801/609062416":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/2251539148":"2024-09-03T09:54:15.686961+0000","172.21.15.151:0/394946850":"2024-09-03T09:54:38.970754+0000","172.21.15.151:6800/609062416":"2024-09-03T09:54:38.970754+0000","172.21.15.151:6801/3972807807":"2024-09-03T09:54:15.686961+0000","172.21.15.151:6800/3972807807":"2024-09-03T09:54:15.686961+0000","172.21.15.151:0/3380605706":"2024-09-03T09:54:15.686961+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-02T10:00:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:00:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:00:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:27 smithi187 ceph-mon[20072]: osdmap e81: 12 total, 12 up, 12 in 2024-09-02T10:00:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:27 smithi187 ceph-mon[20072]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:00:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:00:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:27 smithi162 ceph-mon[20279]: osdmap e81: 12 total, 12 up, 12 in 2024-09-02T10:00:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:27 smithi162 ceph-mon[20279]: pgmap v245: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:28.986 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: osdmap e82: 12 total, 12 up, 12 in 2024-09-02T10:00:28.986 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=1167488894,config=3313370793,health=861809967,kv=4075996571}) 2024-09-02T10:00:28.986 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1563309183,logm=4279762712}) 2024-09-02T10:00:28.986 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1867987804}) 2024-09-02T10:00:28.986 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137216853}) 2024-09-02T10:00:28.986 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2571411966}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=567155235}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1070286806}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2188705562}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368365097}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3597956973}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=825122348}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970054143}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4271342172}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3268156422}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3823360022}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3534642710}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=1318471370,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=3584152394}) 2024-09-02T10:00:28.987 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=7,mon_config_key=93} crc {mgrstat=4010152194,mon_config_key=1216732797}) 2024-09-02T10:00:28.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2971322121}) 2024-09-02T10:00:28.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=95,monmap=5} crc {mon_config_key=4196513136,monmap=3739207698}) 2024-09-02T10:00:28.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=12,osd_pg_creating=1,osdmap=87} crc {osd_metadata=1182194493,osd_pg_creating=803862196,osdmap=3443425083}) 2024-09-02T10:00:28.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=34} crc {osdmap=956697091}) 2024-09-02T10:00:28.988 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1938118382' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:00:29.014 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-02T10:00:29.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-02T10:00:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: osdmap e82: 12 total, 12 up, 12 in 2024-09-02T10:00:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=1167488894,config=3313370793,health=861809967,kv=4075996571}) 2024-09-02T10:00:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1563309183,logm=4279762712}) 2024-09-02T10:00:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1867987804}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137216853}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2571411966}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=567155235}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1070286806}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2188705562}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368365097}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3597956973}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=825122348}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970054143}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4271342172}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3268156422}) 2024-09-02T10:00:29.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3823360022}) 2024-09-02T10:00:29.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3534642710}) 2024-09-02T10:00:29.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=1318471370,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=3584152394}) 2024-09-02T10:00:29.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=7,mon_config_key=93} crc {mgrstat=4010152194,mon_config_key=1216732797}) 2024-09-02T10:00:29.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2971322121}) 2024-09-02T10:00:29.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=95,monmap=5} crc {mon_config_key=4196513136,monmap=3739207698}) 2024-09-02T10:00:29.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=12,osd_pg_creating=1,osdmap=87} crc {osd_metadata=1182194493,osd_pg_creating=803862196,osdmap=3443425083}) 2024-09-02T10:00:29.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=34} crc {osdmap=956697091}) 2024-09-02T10:00:29.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1938118382' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:00:29.259 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: osdmap e82: 12 total, 12 up, 12 in 2024-09-02T10:00:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {auth=22,config=2,health=12,kv=64} crc {auth=1167488894,config=3313370793,health=861809967,kv=4075996571}) 2024-09-02T10:00:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=1563309183,logm=4279762712}) 2024-09-02T10:00:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1867987804}) 2024-09-02T10:00:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137216853}) 2024-09-02T10:00:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2571411966}) 2024-09-02T10:00:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=567155235}) 2024-09-02T10:00:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1070286806}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2188705562}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=368365097}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3597956973}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=825122348}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3970054143}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4271342172}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3268156422}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3823360022}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3534642710}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=1318471370,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=3584152394}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=7,mon_config_key=93} crc {mgrstat=4010152194,mon_config_key=1216732797}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2971322121}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=95,monmap=5} crc {mon_config_key=4196513136,monmap=3739207698}) 2024-09-02T10:00:29.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=12,osd_pg_creating=1,osdmap=87} crc {osd_metadata=1182194493,osd_pg_creating=803862196,osdmap=3443425083}) 2024-09-02T10:00:29.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=34} crc {osdmap=956697091}) 2024-09-02T10:00:29.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1938118382' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:00:29.900 INFO:teuthology.orchestra.run.smithi151.stdout:pg_num: 1 2024-09-02T10:00:29.901 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:29 smithi151 ceph-mon[16639]: pgmap v247: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:29 smithi187 ceph-mon[20072]: pgmap v247: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:30.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:29 smithi162 ceph-mon[20279]: pgmap v247: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:30.614 INFO:tasks.cephadm:Setting up client nodes... 2024-09-02T10:00:30.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-02T10:00:30.860 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:31.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:30 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:00:31.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:30 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:00:31.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3562385846' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-02T10:00:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:30 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:00:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:30 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:00:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3562385846' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-02T10:00:31.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:30 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:00:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:30 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:00:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3562385846' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-09-02T10:00:31.561 INFO:teuthology.orchestra.run.smithi151.stdout:[client.0] 2024-09-02T10:00:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: key = AQC/jNVmA6VTIRAAUUucNB/VxtWHCGBV3u/KtQ== 2024-09-02T10:00:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:31 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:00:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:31 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:00:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:31 smithi151 ceph-mon[16639]: osdmap e83: 12 total, 12 up, 12 in 2024-09-02T10:00:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:31 smithi151 ceph-mon[16639]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-02T10:00:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4227215080' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4227215080' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T10:00:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:31 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:00:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:31 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:00:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:31 smithi187 ceph-mon[20072]: osdmap e83: 12 total, 12 up, 12 in 2024-09-02T10:00:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:31 smithi187 ceph-mon[20072]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-02T10:00:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4227215080' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4227215080' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T10:00:32.336 DEBUG:teuthology.orchestra.run.smithi151:> set -ex 2024-09-02T10:00:32.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-02T10:00:32.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-02T10:00:32.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:31 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:00:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:31 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:00:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:31 smithi162 ceph-mon[20279]: osdmap e83: 12 total, 12 up, 12 in 2024-09-02T10:00:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:31 smithi162 ceph-mon[20279]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s, 0 objects/s recovering 2024-09-02T10:00:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4227215080' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4227215080' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T10:00:32.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-02T10:00:32.635 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:00:33.346 INFO:teuthology.orchestra.run.smithi162.stdout:[client.1] 2024-09-02T10:00:33.346 INFO:teuthology.orchestra.run.smithi162.stdout: key = AQDBjNVmLJkuFBAAgwp5AR+MXU96BWgQNMqe/Q== 2024-09-02T10:00:34.068 DEBUG:teuthology.orchestra.run.smithi162:> set -ex 2024-09-02T10:00:34.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-02T10:00:34.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-02T10:00:34.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-02T10:00:34.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:34 smithi162 ceph-mon[20279]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-02T10:00:34.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.162:0/679697130' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:34.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:34 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:34.348 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:34 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T10:00:34.366 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:00:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:34 smithi151 ceph-mon[16639]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-02T10:00:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.162:0/679697130' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:34 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:34 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T10:00:34.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:34 smithi187 ceph-mon[20072]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-02T10:00:34.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.162:0/679697130' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:34.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:34 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:34.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:34 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T10:00:35.063 INFO:teuthology.orchestra.run.smithi187.stdout:[client.2] 2024-09-02T10:00:35.063 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQDDjNVmTkJ7AxAARJr1B/gD2Vg+wB3l7/Uc5A== 2024-09-02T10:00:35.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.187:0/2881941235' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:35.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:35 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:35.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:35 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T10:00:35.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.187:0/2881941235' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:35.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:35 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:35.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:35 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T10:00:35.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.187:0/2881941235' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:35.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:35 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-09-02T10:00:35.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:35 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-02T10:00:35.823 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-09-02T10:00:35.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-09-02T10:00:35.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-02T10:00:35.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph config log 1 --format=json 2024-09-02T10:00:36.120 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:36.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:36 smithi151 ceph-mon[16639]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T10:00:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:36 smithi187 ceph-mon[20072]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T10:00:36.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:36 smithi162 ceph-mon[20279]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-02T10:00:36.864 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:00:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2206171744' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-02T10:00:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2206171744' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-02T10:00:37.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2206171744' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-09-02T10:00:37.614 INFO:teuthology.orchestra.run.smithi151.stdout:[{"version":28,"timestamp":"2024-09-02T10:00:25.761514+0000","name":"","changes":[{"name":"osd/host:smithi187/osd_memory_target","previous_value":"7404379204","new_value":"5553284403"}]}] 2024-09-02T10:00:37.615 INFO:tasks.ceph_manager:config epoch is 28 2024-09-02T10:00:37.615 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-02T10:00:37.615 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-02T10:00:37.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph mgr dump --format=json 2024-09-02T10:00:37.864 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:38.285 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:38 smithi151 ceph-mon[16639]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-02T10:00:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:38 smithi187 ceph-mon[20072]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-02T10:00:38.537 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:00:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:38 smithi162 ceph-mon[20279]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-02T10:00:39.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/553387901' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-02T10:00:39.410 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":15,"flags":0,"active_gid":14151,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6800","nonce":1488421845},{"type":"v1","addr":"172.21.15.151:6801","nonce":1488421845}]},"active_addr":"172.21.15.151:6801/1488421845","active_change":"2024-09-02T09:54:38.970862+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":24110,"name":"x","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.151:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":3330033610}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":1942534576}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":410399136}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.151:0","nonce":3914610759}]}]} 2024-09-02T10:00:39.414 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-02T10:00:39.414 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-02T10:00:39.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-09-02T10:00:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/553387901' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-02T10:00:39.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/553387901' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-09-02T10:00:39.661 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:40.302 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:00:40.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:40 smithi151 ceph-mon[16639]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-02T10:00:40.303 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":83,"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","created":"2024-09-02T09:53:53.308187+0000","modified":"2024-09-02T10:00:30.852084+0000","last_up_change":"2024-09-02T10:00:25.816021+0000","last_in_change":"2024-09-02T10:00:05.936786+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-02T09:56:53.284814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e76499ad-5f7e-4788-aa85-0bad0d843e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6803","nonce":1059446772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6805","nonce":1059446772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6809","nonce":1059446772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6807","nonce":1059446772}]},"public_addr":"172.21.15.151:6803/1059446772","cluster_addr":"172.21.15.151:6805/1059446772","heartbeat_back_addr":"172.21.15.151:6809/1059446772","heartbeat_front_addr":"172.21.15.151:6807/1059446772","state":["exists","up"]},{"osd":1,"uuid":"8949ba9c-4a71-4327-a1dc-e7d25286d8f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6811","nonce":457777866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6813","nonce":457777866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6817","nonce":457777866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6815","nonce":457777866}]},"public_addr":"172.21.15.151:6811/457777866","cluster_addr":"172.21.15.151:6813/457777866","heartbeat_back_addr":"172.21.15.151:6817/457777866","heartbeat_front_addr":"172.21.15.151:6815/457777866","state":["exists","up"]},{"osd":2,"uuid":"2dead774-0f59-4e90-8503-9a3f0fb68b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6819","nonce":654044340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6821","nonce":654044340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6825","nonce":654044340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6823","nonce":654044340}]},"public_addr":"172.21.15.151:6819/654044340","cluster_addr":"172.21.15.151:6821/654044340","heartbeat_back_addr":"172.21.15.151:6825/654044340","heartbeat_front_addr":"172.21.15.151:6823/654044340","state":["exists","up"]},{"osd":3,"uuid":"524f32ed-95aa-4459-9585-72577db12107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6827","nonce":1912547696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6829","nonce":1912547696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6833","nonce":1912547696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6831","nonce":1912547696}]},"public_addr":"172.21.15.151:6827/1912547696","cluster_addr":"172.21.15.151:6829/1912547696","heartbeat_back_addr":"172.21.15.151:6833/1912547696","heartbeat_front_addr":"172.21.15.151:6831/1912547696","state":["exists","up"]},{"osd":4,"uuid":"2c43a79f-8ee6-4088-af19-6144fc8bdbe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6801","nonce":295415389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6803","nonce":295415389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6807","nonce":295415389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6805","nonce":295415389}]},"public_addr":"172.21.15.162:6801/295415389","cluster_addr":"172.21.15.162:6803/295415389","heartbeat_back_addr":"172.21.15.162:6807/295415389","heartbeat_front_addr":"172.21.15.162:6805/295415389","state":["exists","up"]},{"osd":5,"uuid":"71810aab-c54f-4048-9fe4-d1272302d53a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6809","nonce":3782691981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6811","nonce":3782691981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6815","nonce":3782691981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6813","nonce":3782691981}]},"public_addr":"172.21.15.162:6809/3782691981","cluster_addr":"172.21.15.162:6811/3782691981","heartbeat_back_addr":"172.21.15.162:6815/3782691981","heartbeat_front_addr":"172.21.15.162:6813/3782691981","state":["exists","up"]},{"osd":6,"uuid":"45f93bd7-8e30-43ee-b57a-c007aec6d308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6817","nonce":759599342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6819","nonce":759599342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6823","nonce":759599342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6821","nonce":759599342}]},"public_addr":"172.21.15.162:6817/759599342","cluster_addr":"172.21.15.162:6819/759599342","heartbeat_back_addr":"172.21.15.162:6823/759599342","heartbeat_front_addr":"172.21.15.162:6821/759599342","state":["exists","up"]},{"osd":7,"uuid":"e9ab0b99-0451-4ebd-8758-cbde6350c764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6825","nonce":2389923814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6827","nonce":2389923814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6831","nonce":2389923814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6829","nonce":2389923814}]},"public_addr":"172.21.15.162:6825/2389923814","cluster_addr":"172.21.15.162:6827/2389923814","heartbeat_back_addr":"172.21.15.162:6831/2389923814","heartbeat_front_addr":"172.21.15.162:6829/2389923814","state":["exists","up"]},{"osd":8,"uuid":"b0cf652b-4d1b-441a-85a8-fd5c16380634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6801","nonce":3014427013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6803","nonce":3014427013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6807","nonce":3014427013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6805","nonce":3014427013}]},"public_addr":"172.21.15.187:6801/3014427013","cluster_addr":"172.21.15.187:6803/3014427013","heartbeat_back_addr":"172.21.15.187:6807/3014427013","heartbeat_front_addr":"172.21.15.187:6805/3014427013","state":["exists","up"]},{"osd":9,"uuid":"1752a57c-aab9-4f0d-b2fd-629489170a0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6809","nonce":108453727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6811","nonce":108453727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6815","nonce":108453727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6813","nonce":108453727}]},"public_addr":"172.21.15.187:6809/108453727","cluster_addr":"172.21.15.187:6811/108453727","heartbeat_back_addr":"172.21.15.187:6815/108453727","heartbeat_front_addr":"172.21.15.187:6813/108453727","state":["exists","up"]},{"osd":10,"uuid":"6f48d49d-6636-4142-8cf1-04f371f6317a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6817","nonce":401612597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6819","nonce":401612597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6823","nonce":401612597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6821","nonce":401612597}]},"public_addr":"172.21.15.187:6817/401612597","cluster_addr":"172.21.15.187:6819/401612597","heartbeat_back_addr":"172.21.15.187:6823/401612597","heartbeat_front_addr":"172.21.15.187:6821/401612597","state":["exists","up"]},{"osd":11,"uuid":"f545e120-27c3-4fc6-883f-7c588d0eb8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6825","nonce":3639916212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6827","nonce":3639916212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6831","nonce":3639916212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6829","nonce":3639916212}]},"public_addr":"172.21.15.187:6825/3639916212","cluster_addr":"172.21.15.187:6827/3639916212","heartbeat_back_addr":"172.21.15.187:6831/3639916212","heartbeat_front_addr":"172.21.15.187:6829/3639916212","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:57.639013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:20.329120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:24.298463+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:57.947972+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:54.740043+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:26.755646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:49.640051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:06.880725+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:30.274940+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:35.307683+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:59.380057+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:24.608733+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/1566053508":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/1701887065":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/2339434463":"2024-09-03T09:54:15.686961+0000","172.21.15.151:6801/609062416":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/2251539148":"2024-09-03T09:54:15.686961+0000","172.21.15.151:0/394946850":"2024-09-03T09:54:38.970754+0000","172.21.15.151:6800/609062416":"2024-09-03T09:54:38.970754+0000","172.21.15.151:6801/3972807807":"2024-09-03T09:54:15.686961+0000","172.21.15.151:6800/3972807807":"2024-09-03T09:54:15.686961+0000","172.21.15.151:0/3380605706":"2024-09-03T09:54:15.686961+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-02T10:00:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:40 smithi187 ceph-mon[20072]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-02T10:00:40.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:40 smithi162 ceph-mon[20279]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-02T10:00:40.907 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-02T10:00:40.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-09-02T10:00:41.153 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:41.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1274229380' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:00:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1274229380' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:00:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1274229380' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:00:41.810 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:00:41.810 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":83,"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","created":"2024-09-02T09:53:53.308187+0000","modified":"2024-09-02T10:00:30.852084+0000","last_up_change":"2024-09-02T10:00:25.816021+0000","last_in_change":"2024-09-02T10:00:05.936786+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-02T09:56:53.284814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e76499ad-5f7e-4788-aa85-0bad0d843e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6803","nonce":1059446772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6805","nonce":1059446772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6809","nonce":1059446772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6807","nonce":1059446772}]},"public_addr":"172.21.15.151:6803/1059446772","cluster_addr":"172.21.15.151:6805/1059446772","heartbeat_back_addr":"172.21.15.151:6809/1059446772","heartbeat_front_addr":"172.21.15.151:6807/1059446772","state":["exists","up"]},{"osd":1,"uuid":"8949ba9c-4a71-4327-a1dc-e7d25286d8f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6811","nonce":457777866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6813","nonce":457777866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6817","nonce":457777866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6815","nonce":457777866}]},"public_addr":"172.21.15.151:6811/457777866","cluster_addr":"172.21.15.151:6813/457777866","heartbeat_back_addr":"172.21.15.151:6817/457777866","heartbeat_front_addr":"172.21.15.151:6815/457777866","state":["exists","up"]},{"osd":2,"uuid":"2dead774-0f59-4e90-8503-9a3f0fb68b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6819","nonce":654044340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6821","nonce":654044340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6825","nonce":654044340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6823","nonce":654044340}]},"public_addr":"172.21.15.151:6819/654044340","cluster_addr":"172.21.15.151:6821/654044340","heartbeat_back_addr":"172.21.15.151:6825/654044340","heartbeat_front_addr":"172.21.15.151:6823/654044340","state":["exists","up"]},{"osd":3,"uuid":"524f32ed-95aa-4459-9585-72577db12107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6827","nonce":1912547696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6829","nonce":1912547696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6833","nonce":1912547696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6831","nonce":1912547696}]},"public_addr":"172.21.15.151:6827/1912547696","cluster_addr":"172.21.15.151:6829/1912547696","heartbeat_back_addr":"172.21.15.151:6833/1912547696","heartbeat_front_addr":"172.21.15.151:6831/1912547696","state":["exists","up"]},{"osd":4,"uuid":"2c43a79f-8ee6-4088-af19-6144fc8bdbe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6801","nonce":295415389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6803","nonce":295415389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6807","nonce":295415389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6805","nonce":295415389}]},"public_addr":"172.21.15.162:6801/295415389","cluster_addr":"172.21.15.162:6803/295415389","heartbeat_back_addr":"172.21.15.162:6807/295415389","heartbeat_front_addr":"172.21.15.162:6805/295415389","state":["exists","up"]},{"osd":5,"uuid":"71810aab-c54f-4048-9fe4-d1272302d53a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6809","nonce":3782691981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6811","nonce":3782691981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6815","nonce":3782691981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6813","nonce":3782691981}]},"public_addr":"172.21.15.162:6809/3782691981","cluster_addr":"172.21.15.162:6811/3782691981","heartbeat_back_addr":"172.21.15.162:6815/3782691981","heartbeat_front_addr":"172.21.15.162:6813/3782691981","state":["exists","up"]},{"osd":6,"uuid":"45f93bd7-8e30-43ee-b57a-c007aec6d308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6817","nonce":759599342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6819","nonce":759599342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6823","nonce":759599342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6821","nonce":759599342}]},"public_addr":"172.21.15.162:6817/759599342","cluster_addr":"172.21.15.162:6819/759599342","heartbeat_back_addr":"172.21.15.162:6823/759599342","heartbeat_front_addr":"172.21.15.162:6821/759599342","state":["exists","up"]},{"osd":7,"uuid":"e9ab0b99-0451-4ebd-8758-cbde6350c764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6825","nonce":2389923814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6827","nonce":2389923814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6831","nonce":2389923814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6829","nonce":2389923814}]},"public_addr":"172.21.15.162:6825/2389923814","cluster_addr":"172.21.15.162:6827/2389923814","heartbeat_back_addr":"172.21.15.162:6831/2389923814","heartbeat_front_addr":"172.21.15.162:6829/2389923814","state":["exists","up"]},{"osd":8,"uuid":"b0cf652b-4d1b-441a-85a8-fd5c16380634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6801","nonce":3014427013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6803","nonce":3014427013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6807","nonce":3014427013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6805","nonce":3014427013}]},"public_addr":"172.21.15.187:6801/3014427013","cluster_addr":"172.21.15.187:6803/3014427013","heartbeat_back_addr":"172.21.15.187:6807/3014427013","heartbeat_front_addr":"172.21.15.187:6805/3014427013","state":["exists","up"]},{"osd":9,"uuid":"1752a57c-aab9-4f0d-b2fd-629489170a0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6809","nonce":108453727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6811","nonce":108453727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6815","nonce":108453727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6813","nonce":108453727}]},"public_addr":"172.21.15.187:6809/108453727","cluster_addr":"172.21.15.187:6811/108453727","heartbeat_back_addr":"172.21.15.187:6815/108453727","heartbeat_front_addr":"172.21.15.187:6813/108453727","state":["exists","up"]},{"osd":10,"uuid":"6f48d49d-6636-4142-8cf1-04f371f6317a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6817","nonce":401612597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6819","nonce":401612597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6823","nonce":401612597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6821","nonce":401612597}]},"public_addr":"172.21.15.187:6817/401612597","cluster_addr":"172.21.15.187:6819/401612597","heartbeat_back_addr":"172.21.15.187:6823/401612597","heartbeat_front_addr":"172.21.15.187:6821/401612597","state":["exists","up"]},{"osd":11,"uuid":"f545e120-27c3-4fc6-883f-7c588d0eb8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6825","nonce":3639916212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6827","nonce":3639916212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6831","nonce":3639916212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6829","nonce":3639916212}]},"public_addr":"172.21.15.187:6825/3639916212","cluster_addr":"172.21.15.187:6827/3639916212","heartbeat_back_addr":"172.21.15.187:6831/3639916212","heartbeat_front_addr":"172.21.15.187:6829/3639916212","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:57.639013+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:20.329120+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:24.298463+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:57.947972+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:54.740043+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:26.755646+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:49.640051+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:06.880725+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:30.274940+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:35.307683+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T09:59:59.380057+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:00:24.608733+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/1566053508":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/1701887065":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/2339434463":"2024-09-03T09:54:15.686961+0000","172.21.15.151:6801/609062416":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/2251539148":"2024-09-03T09:54:15.686961+0000","172.21.15.151:0/394946850":"2024-09-03T09:54:38.970754+0000","172.21.15.151:6800/609062416":"2024-09-03T09:54:38.970754+0000","172.21.15.151:6801/3972807807":"2024-09-03T09:54:15.686961+0000","172.21.15.151:6800/3972807807":"2024-09-03T09:54:15.686961+0000","172.21.15.151:0/3380605706":"2024-09-03T09:54:15.686961+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-02T10:00:42.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:42 smithi151 ceph-mon[16639]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:42.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2857272568' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:00:42.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-02T10:00:42.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-02T10:00:42.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-02T10:00:42.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-02T10:00:42.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-02T10:00:42.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-02T10:00:42.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-02T10:00:42.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-02T10:00:42.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-02T10:00:42.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-02T10:00:42.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-02T10:00:42.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-02T10:00:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:42 smithi187 ceph-mon[20072]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2857272568' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:00:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:42 smithi162 ceph-mon[20279]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:42.903 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2857272568' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:00:43.124 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:43.131 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:43.342 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:43.447 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:43.447 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:43.448 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:43.464 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:44.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:44 smithi151 ceph-mon[16639]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:44 smithi187 ceph-mon[20072]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:44.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:44 smithi162 ceph-mon[20279]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:45.464 INFO:teuthology.orchestra.run.smithi151.stdout:34359738429 2024-09-02T10:00:45.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-02T10:00:45.692 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:46.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:46 smithi151 ceph-mon[16639]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:46 smithi187 ceph-mon[20072]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:46.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:46 smithi162 ceph-mon[20279]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:46.728 INFO:teuthology.orchestra.run.smithi151.stdout:111669149742 2024-09-02T10:00:46.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-02T10:00:47.883 INFO:teuthology.orchestra.run.smithi151.stdout:167503724579 2024-09-02T10:00:47.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-02T10:00:48.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:48 smithi151 ceph-mon[16639]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:48 smithi187 ceph-mon[20072]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:48 smithi162 ceph-mon[20279]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:48.616 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:48.617 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:48.617 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:48.617 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:49.574 INFO:teuthology.orchestra.run.smithi151.stdout:343597383688 2024-09-02T10:00:49.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-02T10:00:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:00:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:00:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:50 smithi151 ceph-mon[16639]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:50 smithi151 ceph-mon[16639]: osdmap e84: 12 total, 12 up, 12 in 2024-09-02T10:00:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:00:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:00:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:50 smithi187 ceph-mon[20072]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:50 smithi187 ceph-mon[20072]: osdmap e84: 12 total, 12 up, 12 in 2024-09-02T10:00:50.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:00:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:00:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:50 smithi162 ceph-mon[20279]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:50 smithi162 ceph-mon[20279]: osdmap e84: 12 total, 12 up, 12 in 2024-09-02T10:00:51.035 INFO:teuthology.orchestra.run.smithi151.stdout:77309411378 2024-09-02T10:00:51.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-02T10:00:51.534 INFO:teuthology.orchestra.run.smithi151.stdout:193273528351 2024-09-02T10:00:51.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-02T10:00:51.559 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:00:51.559 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:00:51.559 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:00:51.559 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:00:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:00:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:00:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:51.712 INFO:teuthology.orchestra.run.smithi151.stdout:55834574904 2024-09-02T10:00:51.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-02T10:00:51.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:00:51.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:00:51.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:00:51.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:00:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:52 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:52 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:52 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:52.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:52 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:52 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:52 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:52 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:00:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:52 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:00:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:52 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:00:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:54 smithi187 ceph-mon[20072]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:54.629 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:54 smithi162 ceph-mon[20279]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:54.629 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:54 smithi151 ceph-mon[16639]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:54.838 INFO:teuthology.orchestra.run.smithi151.stdout:223338299419 2024-09-02T10:00:54.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-02T10:00:54.947 INFO:teuthology.orchestra.run.smithi151.stdout:283467841554 2024-09-02T10:00:54.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-02T10:00:55.407 INFO:teuthology.orchestra.run.smithi151.stdout:137438953513 2024-09-02T10:00:55.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-02T10:00:56.390 INFO:teuthology.orchestra.run.smithi151.stdout:313532612622 2024-09-02T10:00:56.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-02T10:00:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:56 smithi162 ceph-mon[20279]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:56.663 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:56 smithi151 ceph-mon[16639]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:56 smithi187 ceph-mon[20072]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:56.868 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:56.877 INFO:teuthology.orchestra.run.smithi151.stdout:253403070487 2024-09-02T10:00:56.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-02T10:00:56.896 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:56.897 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:56.897 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:56.902 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:56.932 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:56.935 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:00:58.020 INFO:teuthology.orchestra.run.smithi151.stdout:137438953514 2024-09-02T10:00:58.210 INFO:teuthology.orchestra.run.smithi151.stdout:283467841555 2024-09-02T10:00:58.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:58 smithi162 ceph-mon[20279]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4279478961' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-02T10:00:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1040091445' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-02T10:00:58.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:58 smithi151 ceph-mon[16639]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:58.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4279478961' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-02T10:00:58.598 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1040091445' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-02T10:00:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:58 smithi187 ceph-mon[20072]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:00:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4279478961' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-02T10:00:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1040091445' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-02T10:00:58.794 INFO:teuthology.orchestra.run.smithi151.stdout:111669149744 2024-09-02T10:00:59.139 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467841554 got 283467841555 for osd.9 2024-09-02T10:00:59.139 DEBUG:teuthology.parallel:result is None 2024-09-02T10:00:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:00:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4093226460' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-02T10:00:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:00:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4093226460' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-02T10:00:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:00:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4093226460' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-02T10:00:59.863 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149742 got 111669149744 for osd.3 2024-09-02T10:00:59.863 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:00.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:00 smithi151 ceph-mon[16639]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:00.551 INFO:teuthology.orchestra.run.smithi151.stdout:167503724582 2024-09-02T10:01:00.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:00 smithi162 ceph-mon[20279]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:00 smithi187 ceph-mon[20072]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:00.920 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953513 got 137438953514 for osd.4 2024-09-02T10:01:00.921 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2388859404' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-02T10:01:01.523 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:01.525 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:01.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2388859404' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-02T10:01:01.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2388859404' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-02T10:01:02.383 INFO:teuthology.orchestra.run.smithi151.stdout:343597383690 2024-09-02T10:01:02.518 INFO:teuthology.orchestra.run.smithi151.stdout:223338299421 2024-09-02T10:01:02.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:02 smithi162 ceph-mon[20279]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:02.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:02 smithi151 ceph-mon[16639]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:02 smithi187 ceph-mon[20072]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:03.329 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383688 got 343597383690 for osd.11 2024-09-02T10:01:03.329 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:03.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:03 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:03.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:03 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:03.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:03 smithi151 ceph-mon[16639]: osdmap e85: 12 total, 12 up, 12 in 2024-09-02T10:01:03.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2402057181' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-02T10:01:03.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3154515398' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-02T10:01:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:03 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:03 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:03 smithi187 ceph-mon[20072]: osdmap e85: 12 total, 12 up, 12 in 2024-09-02T10:01:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2402057181' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-02T10:01:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3154515398' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-02T10:01:03.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:03 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:03 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:03 smithi162 ceph-mon[20279]: osdmap e85: 12 total, 12 up, 12 in 2024-09-02T10:01:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2402057181' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-02T10:01:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3154515398' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-02T10:01:03.846 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724579 got 167503724582 for osd.5 2024-09-02T10:01:03.847 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:03.957 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:04.067 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:04.441 INFO:teuthology.orchestra.run.smithi151.stdout:193273528354 2024-09-02T10:01:04.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:04 smithi151 ceph-mon[16639]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:04 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3583607237' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-02T10:01:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:04 smithi187 ceph-mon[20072]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:04 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3583607237' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-02T10:01:04.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:04 smithi162 ceph-mon[20279]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:04 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3583607237' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-02T10:01:05.418 INFO:teuthology.orchestra.run.smithi151.stdout:55834574907 2024-09-02T10:01:05.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:05 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:05.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:05 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:05.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:05 smithi151 ceph-mon[16639]: osdmap e86: 12 total, 12 up, 12 in 2024-09-02T10:01:05.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1840408019' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-02T10:01:05.754 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528351 got 193273528354 for osd.6 2024-09-02T10:01:05.754 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:05 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:05 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:05 smithi187 ceph-mon[20072]: osdmap e86: 12 total, 12 up, 12 in 2024-09-02T10:01:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1840408019' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-02T10:01:05.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:05 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:05 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:05 smithi162 ceph-mon[20279]: osdmap e86: 12 total, 12 up, 12 in 2024-09-02T10:01:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1840408019' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-02T10:01:06.125 INFO:teuthology.orchestra.run.smithi151.stdout:313532612624 2024-09-02T10:01:06.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:06.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:06.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:01:06.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:01:06.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:01:06.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:01:06.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1716632133' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-02T10:01:06.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:01:06.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:01:06.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:01:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:01:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:01:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:01:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:01:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1716632133' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-02T10:01:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:01:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:01:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:01:06.757 INFO:teuthology.orchestra.run.smithi151.stdout:34359738433 2024-09-02T10:01:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:01:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:01:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:01:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:01:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1716632133' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-02T10:01:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:01:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:01:06.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:01:06.848 INFO:teuthology.orchestra.run.smithi151.stdout:77309411382 2024-09-02T10:01:07.643 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:07.656 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:07 smithi151 ceph-mon[16639]: osdmap e87: 12 total, 12 up, 12 in 2024-09-02T10:01:07.656 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3570619348' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-02T10:01:07.657 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1270085865' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-02T10:01:07.657 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411378 got 77309411382 for osd.2 2024-09-02T10:01:07.658 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:07 smithi187 ceph-mon[20072]: osdmap e87: 12 total, 12 up, 12 in 2024-09-02T10:01:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3570619348' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-02T10:01:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1270085865' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-02T10:01:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:07 smithi162 ceph-mon[20279]: osdmap e87: 12 total, 12 up, 12 in 2024-09-02T10:01:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3570619348' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-02T10:01:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1270085865' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-02T10:01:08.354 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299419 got 223338299421 for osd.7 2024-09-02T10:01:08.355 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:08.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:08 smithi151 ceph-mon[16639]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:08.730 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738429 got 34359738433 for osd.0 2024-09-02T10:01:08.731 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:08 smithi187 ceph-mon[20072]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:08.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:08 smithi162 ceph-mon[20279]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:09.228 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574904 got 55834574907 for osd.1 2024-09-02T10:01:09.228 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:09.666 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612622 got 313532612624 for osd.10 2024-09-02T10:01:09.666 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:10.039 INFO:teuthology.orchestra.run.smithi151.stdout:253403070490 2024-09-02T10:01:10.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:10 smithi151 ceph-mon[16639]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:10.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2190200587' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-02T10:01:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:10 smithi187 ceph-mon[20072]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2190200587' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-02T10:01:10.823 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070487 got 253403070490 for osd.8 2024-09-02T10:01:10.823 DEBUG:teuthology.parallel:result is None 2024-09-02T10:01:10.823 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-02T10:01:10.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:01:10.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:10 smithi162 ceph-mon[20279]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2190200587' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-02T10:01:11.071 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:01:11.744 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:01:12.619 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:12 smithi151 ceph-mon[16639]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:12.648 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-02T10:01:11.054299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337832,"kb_used_data":9168,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124603992,"statfs":{"total":1151940427776,"available":1151594487808,"internally_reserved":0,"allocated":9388032,"data_stored":5630146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":336115003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001423"},"pg_stats":[{"pgid":"1.0","version":"22'92","reported_seq":122,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-02T10:01:07.584275+0000","last_change":"2024-09-02T10:00:29.279739+0000","last_active":"2024-09-02T10:01:07.584275+0000","last_peered":"2024-09-02T10:01:07.584275+0000","last_clean":"2024-09-02T10:01:07.584275+0000","last_became_active":"2024-09-02T10:00:28.639420+0000","last_became_peered":"2024-09-02T10:00:28.639420+0000","last_unstale":"2024-09-02T10:01:07.584275+0000","last_undegraded":"2024-09-02T10:01:07.584275+0000","last_fullsized":"2024-09-02T10:01:07.584275+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:00:29.279658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:00:29.279658+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:01:47.551954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383692,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1175552,"data_stored":862168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612625,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":643072,"data_stored":333260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841558,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070491,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299423,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1187840,"data_stored":869565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528355,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28176,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716976,"statfs":{"total":95995035648,"available":95966183424,"internally_reserved":0,"allocated":1175552,"data_stored":862188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724584,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27740,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":643072,"data_stored":333260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411383,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31816,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713336,"statfs":{"total":95995035648,"available":95962456064,"internally_reserved":0,"allocated":643072,"data_stored":333260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738434,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":643072,"data_stored":333260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:01:12.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:01:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:12 smithi187 ceph-mon[20072]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:12.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:12 smithi162 ceph-mon[20279]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:12.894 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:13.557 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:01:13.557 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:01:13.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:13 smithi151 ceph-mon[16639]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:13 smithi187 ceph-mon[20072]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:13.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:13 smithi162 ceph-mon[20279]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:14.300 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-02T10:01:13.054688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337740,"kb_used_data":9240,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604084,"statfs":{"total":1151940427776,"available":1151594582016,"internally_reserved":0,"allocated":9461760,"data_stored":5674408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":336115003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.682934"},"pg_stats":[{"pgid":"1.0","version":"22'92","reported_seq":122,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-02T10:01:07.584275+0000","last_change":"2024-09-02T10:00:29.279739+0000","last_active":"2024-09-02T10:01:07.584275+0000","last_peered":"2024-09-02T10:01:07.584275+0000","last_clean":"2024-09-02T10:01:07.584275+0000","last_became_active":"2024-09-02T10:00:28.639420+0000","last_became_peered":"2024-09-02T10:00:28.639420+0000","last_unstale":"2024-09-02T10:01:07.584275+0000","last_undegraded":"2024-09-02T10:01:07.584275+0000","last_fullsized":"2024-09-02T10:01:07.584275+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:00:29.279658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:00:29.279658+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:01:47.551954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383693,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1187840,"data_stored":869545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070491,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299423,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1187840,"data_stored":869565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528356,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1187840,"data_stored":869565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724585,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31828,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713324,"statfs":{"total":95995035648,"available":95962443776,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:01:14.301 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-02T10:01:14.301 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-02T10:01:14.301 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-02T10:01:14.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph health --format=json 2024-09-02T10:01:14.549 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:14.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:14 smithi151 ceph-mon[16639]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:14.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:14 smithi151 ceph-mon[16639]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:14 smithi187 ceph-mon[20072]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:14 smithi187 ceph-mon[20072]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:14.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:14 smithi162 ceph-mon[20279]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:14 smithi162 ceph-mon[20279]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:15.221 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:01:15.221 INFO:teuthology.orchestra.run.smithi151.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-02T10:01:15.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:15 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:15.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:15 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:15.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3823115277' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-02T10:01:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:15 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:15 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3823115277' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-02T10:01:15.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:15 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:15 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3823115277' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-09-02T10:01:15.992 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-02T10:01:15.992 INFO:tasks.cephadm:Setup complete, yielding 2024-09-02T10:01:15.992 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-02T10:01:16.003 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-02T10:01:16.003 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-bharath3-testing-2024-09-01-1328'}, 'ceph': {'conf': {'client.2': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'osd', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-bharath3-testing-2024-09-01-1328', 'sha1': '82daa6ef0815320144b25b212c6b8c25a1d59c07'}} 2024-09-02T10:01:16.003 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2024-09-02T10:01:16.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-02T10:01:16.247 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:16.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:16 smithi151 ceph-mon[16639]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:16.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:16 smithi151 ceph-mon[16639]: osdmap e88: 12 total, 12 up, 12 in 2024-09-02T10:01:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:16 smithi187 ceph-mon[20072]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:16 smithi187 ceph-mon[20072]: osdmap e88: 12 total, 12 up, 12 in 2024-09-02T10:01:16.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:16 smithi162 ceph-mon[20279]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:16 smithi162 ceph-mon[20279]: osdmap e88: 12 total, 12 up, 12 in 2024-09-02T10:01:17.676 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-02T10:01:17.676 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-02T10:01:17.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:01:17.921 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3611592034' entity='client.admin' 2024-09-02T10:01:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:17 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:01:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:17 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:01:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:17 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:01:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:17 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:01:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:17 smithi151 ceph-mon[16639]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3611592034' entity='client.admin' 2024-09-02T10:01:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:17 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:01:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:17 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:01:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:17 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:01:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:17 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:01:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:17 smithi187 ceph-mon[20072]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:18.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3611592034' entity='client.admin' 2024-09-02T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:17 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:17 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:17 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:17 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:01:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:17 smithi162 ceph-mon[20279]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:18.602 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:01:18.602 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:01:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:18 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:18 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:18 smithi187 ceph-mon[20072]: osdmap e89: 12 total, 12 up, 12 in 2024-09-02T10:01:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:18 smithi187 ceph-mon[20072]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:19.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:18 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:18 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:18 smithi162 ceph-mon[20279]: osdmap e89: 12 total, 12 up, 12 in 2024-09-02T10:01:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:18 smithi162 ceph-mon[20279]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:19.410 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:18 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:19.410 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:18 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:19.410 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:18 smithi151 ceph-mon[16639]: osdmap e89: 12 total, 12 up, 12 in 2024-09-02T10:01:19.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:18 smithi151 ceph-mon[16639]: from='client.14664 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:19.445 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-09-02T10:01:17.939464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337752,"kb_used_data":9252,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604072,"statfs":{"total":1151940427776,"available":1151594569728,"internally_reserved":0,"allocated":9474048,"data_stored":5681785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":336115003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.885559"},"pg_stats":[{"pgid":"1.0","version":"22'92","reported_seq":122,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-02T10:01:07.584275+0000","last_change":"2024-09-02T10:00:29.279739+0000","last_active":"2024-09-02T10:01:07.584275+0000","last_peered":"2024-09-02T10:01:07.584275+0000","last_clean":"2024-09-02T10:01:07.584275+0000","last_became_active":"2024-09-02T10:00:28.639420+0000","last_became_peered":"2024-09-02T10:00:28.639420+0000","last_unstale":"2024-09-02T10:01:07.584275+0000","last_undegraded":"2024-09-02T10:01:07.584275+0000","last_fullsized":"2024-09-02T10:01:07.584275+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:00:29.279658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:00:29.279658+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:01:47.551954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1187840,"data_stored":869545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841559,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070492,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299424,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1187840,"data_stored":869565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1187840,"data_stored":869565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724586,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":667648,"data_stored":348014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31828,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713324,"statfs":{"total":95995035648,"available":95962443776,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:01:19.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:01:19.698 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:20.123 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:19 smithi151 ceph-mon[16639]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:19 smithi187 ceph-mon[20072]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:20.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:19 smithi162 ceph-mon[20279]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:20.388 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:01:20.388 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:01:21.072 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-02T10:01:19.055890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":3,"kb":1124941824,"kb_used":337788,"kb_used_data":9288,"kb_used_omap":18,"kb_used_meta":328237,"kb_avail":1124604036,"statfs":{"total":1151940427776,"available":1151594532864,"internally_reserved":0,"allocated":9510912,"data_stored":5703916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":336115003},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001590"},"pg_stats":[{"pgid":"1.0","version":"22'92","reported_seq":122,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-09-02T10:01:07.584275+0000","last_change":"2024-09-02T10:00:29.279739+0000","last_active":"2024-09-02T10:01:07.584275+0000","last_peered":"2024-09-02T10:01:07.584275+0000","last_clean":"2024-09-02T10:01:07.584275+0000","last_became_active":"2024-09-02T10:00:28.639420+0000","last_became_peered":"2024-09-02T10:00:28.639420+0000","last_unstale":"2024-09-02T10:01:07.584275+0000","last_undegraded":"2024-09-02T10:01:07.584275+0000","last_fullsized":"2024-09-02T10:01:07.584275+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:00:29.279658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:00:29.279658+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:01:47.551954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383694,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1187840,"data_stored":869545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841560,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":667648,"data_stored":348014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070492,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27672,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717480,"statfs":{"total":95995035648,"available":95966699520,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299424,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1187840,"data_stored":869565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28188,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716964,"statfs":{"total":95995035648,"available":95966171136,"internally_reserved":0,"allocated":1187840,"data_stored":869565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724586,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":667648,"data_stored":348014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953519,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717460,"statfs":{"total":95995035648,"available":95966679040,"internally_reserved":0,"allocated":679936,"data_stored":355391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":31828,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713324,"statfs":{"total":95995035648,"available":95962443776,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717484,"statfs":{"total":95995035648,"available":95966703616,"internally_reserved":0,"allocated":655360,"data_stored":340637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:01:21.073 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-02T10:01:21.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump 2024-09-02T10:01:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:20 smithi187 ceph-mon[20072]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:21.317 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:21.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:20 smithi162 ceph-mon[20279]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:21.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:20 smithi151 ceph-mon[16639]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:21.960 INFO:teuthology.orchestra.run.smithi151.stdout:epoch 89 2024-09-02T10:01:21.960 INFO:teuthology.orchestra.run.smithi151.stdout:fsid 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T10:01:21.961 INFO:teuthology.orchestra.run.smithi151.stdout:created 2024-09-02T09:53:53.308187+0000 2024-09-02T10:01:21.961 INFO:teuthology.orchestra.run.smithi151.stdout:modified 2024-09-02T10:01:17.934478+0000 2024-09-02T10:01:21.961 INFO:teuthology.orchestra.run.smithi151.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-02T10:01:21.961 INFO:teuthology.orchestra.run.smithi151.stdout:crush_version 26 2024-09-02T10:01:21.961 INFO:teuthology.orchestra.run.smithi151.stdout:full_ratio 0.95 2024-09-02T10:01:21.961 INFO:teuthology.orchestra.run.smithi151.stdout:backfillfull_ratio 0.9 2024-09-02T10:01:21.961 INFO:teuthology.orchestra.run.smithi151.stdout:nearfull_ratio 0.85 2024-09-02T10:01:21.961 INFO:teuthology.orchestra.run.smithi151.stdout:require_min_compat_client luminous 2024-09-02T10:01:21.961 INFO:teuthology.orchestra.run.smithi151.stdout:min_compat_client jewel 2024-09-02T10:01:21.962 INFO:teuthology.orchestra.run.smithi151.stdout:require_osd_release squid 2024-09-02T10:01:21.962 INFO:teuthology.orchestra.run.smithi151.stdout:stretch_mode_enabled false 2024-09-02T10:01:21.962 INFO:teuthology.orchestra.run.smithi151.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 23 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-09-02T10:01:21.962 INFO:teuthology.orchestra.run.smithi151.stdout:max_osd 12 2024-09-02T10:01:21.962 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 up in weight 1 up_from 8 up_thru 52 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772] [v2:172.21.15.151:6804/1059446772,v1:172.21.15.151:6805/1059446772] exists,up e76499ad-5f7e-4788-aa85-0bad0d843e29 2024-09-02T10:01:21.962 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 up in weight 1 up_from 13 up_thru 33 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866] [v2:172.21.15.151:6812/457777866,v1:172.21.15.151:6813/457777866] exists,up 8949ba9c-4a71-4327-a1dc-e7d25286d8f1 2024-09-02T10:01:21.962 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340] [v2:172.21.15.151:6820/654044340,v1:172.21.15.151:6821/654044340] exists,up 2dead774-0f59-4e90-8503-9a3f0fb68b48 2024-09-02T10:01:21.962 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696] [v2:172.21.15.151:6828/1912547696,v1:172.21.15.151:6829/1912547696] exists,up 524f32ed-95aa-4459-9585-72577db12107 2024-09-02T10:01:21.962 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389] [v2:172.21.15.162:6802/295415389,v1:172.21.15.162:6803/295415389] exists,up 2c43a79f-8ee6-4088-af19-6144fc8bdbe4 2024-09-02T10:01:21.962 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5 up in weight 1 up_from 39 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981] [v2:172.21.15.162:6810/3782691981,v1:172.21.15.162:6811/3782691981] exists,up 71810aab-c54f-4048-9fe4-d1272302d53a 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342] [v2:172.21.15.162:6818/759599342,v1:172.21.15.162:6819/759599342] exists,up 45f93bd7-8e30-43ee-b57a-c007aec6d308 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 up in weight 1 up_from 52 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814] [v2:172.21.15.162:6826/2389923814,v1:172.21.15.162:6827/2389923814] exists,up e9ab0b99-0451-4ebd-8758-cbde6350c764 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013] [v2:172.21.15.187:6802/3014427013,v1:172.21.15.187:6803/3014427013] exists,up b0cf652b-4d1b-441a-85a8-fd5c16380634 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727] [v2:172.21.15.187:6810/108453727,v1:172.21.15.187:6811/108453727] exists,up 1752a57c-aab9-4f0d-b2fd-629489170a0b 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10 up in weight 1 up_from 73 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597] [v2:172.21.15.187:6818/401612597,v1:172.21.15.187:6819/401612597] exists,up 6f48d49d-6636-4142-8cf1-04f371f6317a 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212] [v2:172.21.15.187:6826/3639916212,v1:172.21.15.187:6827/3639916212] exists,up f545e120-27c3-4fc6-883f-7c588d0eb8d2 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1566053508 expires 2024-09-03T09:54:38.970754+0000 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/1701887065 expires 2024-09-03T09:54:38.970754+0000 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/2339434463 expires 2024-09-03T09:54:15.686961+0000 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6801/609062416 expires 2024-09-03T09:54:38.970754+0000 2024-09-02T10:01:21.963 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/2251539148 expires 2024-09-03T09:54:15.686961+0000 2024-09-02T10:01:21.964 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/394946850 expires 2024-09-03T09:54:38.970754+0000 2024-09-02T10:01:21.964 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6800/609062416 expires 2024-09-03T09:54:38.970754+0000 2024-09-02T10:01:21.964 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6801/3972807807 expires 2024-09-03T09:54:15.686961+0000 2024-09-02T10:01:21.964 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:6800/3972807807 expires 2024-09-03T09:54:15.686961+0000 2024-09-02T10:01:21.964 INFO:teuthology.orchestra.run.smithi151.stdout:blocklist 172.21.15.151:0/3380605706 expires 2024-09-03T09:54:15.686961+0000 2024-09-02T10:01:22.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:21 smithi162 ceph-mon[20279]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:21 smithi187 ceph-mon[20072]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:22.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:21 smithi151 ceph-mon[16639]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:22.776 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 52 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6802/1059446772,v1:172.21.15.151:6803/1059446772] [v2:172.21.15.151:6804/1059446772,v1:172.21.15.151:6805/1059446772] exists,up e76499ad-5f7e-4788-aa85-0bad0d843e29', 'osd.1 up in weight 1 up_from 13 up_thru 33 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6810/457777866,v1:172.21.15.151:6811/457777866] [v2:172.21.15.151:6812/457777866,v1:172.21.15.151:6813/457777866] exists,up 8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6818/654044340,v1:172.21.15.151:6819/654044340] [v2:172.21.15.151:6820/654044340,v1:172.21.15.151:6821/654044340] exists,up 2dead774-0f59-4e90-8503-9a3f0fb68b48', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.151:6826/1912547696,v1:172.21.15.151:6827/1912547696] [v2:172.21.15.151:6828/1912547696,v1:172.21.15.151:6829/1912547696] exists,up 524f32ed-95aa-4459-9585-72577db12107', 'osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6800/295415389,v1:172.21.15.162:6801/295415389] [v2:172.21.15.162:6802/295415389,v1:172.21.15.162:6803/295415389] exists,up 2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'osd.5 up in weight 1 up_from 39 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6808/3782691981,v1:172.21.15.162:6809/3782691981] [v2:172.21.15.162:6810/3782691981,v1:172.21.15.162:6811/3782691981] exists,up 71810aab-c54f-4048-9fe4-d1272302d53a', 'osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6816/759599342,v1:172.21.15.162:6817/759599342] [v2:172.21.15.162:6818/759599342,v1:172.21.15.162:6819/759599342] exists,up 45f93bd7-8e30-43ee-b57a-c007aec6d308', 'osd.7 up in weight 1 up_from 52 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.162:6824/2389923814,v1:172.21.15.162:6825/2389923814] [v2:172.21.15.162:6826/2389923814,v1:172.21.15.162:6827/2389923814] exists,up e9ab0b99-0451-4ebd-8758-cbde6350c764', 'osd.8 up in weight 1 up_from 59 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6800/3014427013,v1:172.21.15.187:6801/3014427013] [v2:172.21.15.187:6802/3014427013,v1:172.21.15.187:6803/3014427013] exists,up b0cf652b-4d1b-441a-85a8-fd5c16380634', 'osd.9 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6808/108453727,v1:172.21.15.187:6809/108453727] [v2:172.21.15.187:6810/108453727,v1:172.21.15.187:6811/108453727] exists,up 1752a57c-aab9-4f0d-b2fd-629489170a0b', 'osd.10 up in weight 1 up_from 73 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6816/401612597,v1:172.21.15.187:6817/401612597] [v2:172.21.15.187:6818/401612597,v1:172.21.15.187:6819/401612597] exists,up 6f48d49d-6636-4142-8cf1-04f371f6317a', 'osd.11 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.187:6824/3639916212,v1:172.21.15.187:6825/3639916212] [v2:172.21.15.187:6826/3639916212,v1:172.21.15.187:6827/3639916212] exists,up f545e120-27c3-4fc6-883f-7c588d0eb8d2'] 2024-09-02T10:01:22.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon mon.a config show 2024-09-02T10:01:23.023 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:23.143 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1010537459' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-02T10:01:23.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1010537459' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-02T10:01:23.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1010537459' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-09-02T10:01:23.523 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:01:23.523 INFO:teuthology.orchestra.run.smithi151.stdout: "name": "mon.a", 2024-09-02T10:01:23.523 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster": "ceph", 2024-09-02T10:01:23.523 INFO:teuthology.orchestra.run.smithi151.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-02T10:01:23.523 INFO:teuthology.orchestra.run.smithi151.stdout: "admin_socket_mode": "", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_client_required": "cephx, none", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_cluster_required": "cephx", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_debug": "true", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_service_required": "cephx", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "auth_supported": "", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio": "true", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-02T10:01:23.524 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio_poll_ms": "250", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_aio_reap_max": "16", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_async_discard_threads": "0", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_block_size": "4096", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_aio": "false", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_debug_inflight_ios": "false", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_enable_discard": "false", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_flock_retry": "3", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_inject_crash": "0", 2024-09-02T10:01:23.525 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_ioring": "false", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_ioring_hipri": "false", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bdev_type": "", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_alloc_size": "1048576", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_allocator": "hybrid", 2024-09-02T10:01:23.526 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_buffered_io": "true", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_check_for_zeros": "false", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_compact_log_sync": "false", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_max_log_runway": "4194304", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_max_prefetch": "1048576", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_min_flush_size": "524288", 2024-09-02T10:01:23.527 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_min_log_runway": "1048576", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_replay_recovery": "false", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluefs_sync_write": "false", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_allocation_from_file": "true", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_allocator": "hybrid", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_async_db_compaction": "true", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-02T10:01:23.528 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bdev_label_multi": "true", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_create": "true", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_db_create": "false", 2024-09-02T10:01:23.529 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_db_path": "", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_db_size": "0", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_path": "", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_preallocate_file": "false", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_size": "107374182400", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_wal_create": "false", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_wal_path": "", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_block_wal_size": "100663296", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs": "true", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-02T10:01:23.530 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_autotune": "true", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_size": "0", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-02T10:01:23.531 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cache_type": "2q", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_clone_cow": "true", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_mode": "none", 2024-09-02T10:01:23.532 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_csum_type": "crc32c", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_freelist": "false", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-02T10:01:23.533 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_small_allocations": "0", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_default_buffered_read": "true", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_default_buffered_write": "false", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-02T10:01:23.534 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fail_eio": "false", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-02T10:01:23.535 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mount": "false", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_umount": "false", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_ignore_data_csum": "false", 2024-09-02T10:01:23.536 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_log_op_age": "5.000000", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_alloc_size": "0", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_blob_size": "0", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_max_deferred_txc": "32", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_min_alloc_size": "0", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-02T10:01:23.537 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_nid_prealloc": "1024", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_retry_disk_reads": "3", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_cf": "true", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-02T10:01:23.538 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_spdk_mem": "512", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_tracing": "false", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-02T10:01:23.539 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_write_v2": "false", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_write_v2_random": "false", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "bluestore_zero_block_detection": "false", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-02T10:01:23.540 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_cluster_require_signatures": "false", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_cluster_require_version": "2", 2024-09-02T10:01:23.541 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_require_signatures": "false", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_require_version": "2", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_service_require_signatures": "false", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_service_require_version": "2", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "cephx_sign_messages": "true", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "chdir": "", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "client_acl_type": "", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "client_asio_thread_count": "2", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "client_cache_mid": "0.750000", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "client_cache_size": "16384", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "client_caps_release_delay": "5", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "client_check_pool_perm": "true", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-02T10:01:23.542 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_force_sync_read": "false", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_getattr_caps": "false", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_inject_features": "", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_debug_inject_tick_delay": "0", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_die_on_failed_remount": "false", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_dirsize_rbytes": "true", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_force_lazyio": "false", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_fs": "", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_inject_release_failure": "false", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_max_inline_size": "4096", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mds_namespace": "", 2024-09-02T10:01:23.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_metadata": "", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mount_gid": "-1", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mount_timeout": "300", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mount_uid": "-1", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_mountpoint": "/", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_notify_timeout": "10", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc": "true", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_max_dirty": "104857600", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_max_objects": "1000", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_size": "209715200", 2024-09-02T10:01:23.544 INFO:teuthology.orchestra.run.smithi151.stdout: "client_oc_target_dirty": "8388608", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_permissions": "true", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_quota": "true", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_quota_df": "true", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_readahead_max_bytes": "0", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_readahead_max_periods": "4", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_readahead_min": "131072", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_reconnect_stale": "false", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_shutdown_timeout": "30", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_snapdir": ".snap", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_tick_interval": "1", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_trace": "", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_try_dentry_invalidate": "false", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_use_faked_inos": "false", 2024-09-02T10:01:23.545 INFO:teuthology.orchestra.run.smithi151.stdout: "client_use_random_mds": "false", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_graylog": "false", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_graylog_port": "12201", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_monitors": "default=true", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_syslog": "false", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "clog_to_syslog_level": "info", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "-", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_network": "", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_network_interface": "", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zlib_isal": "false", 2024-09-02T10:01:23.546 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zlib_level": "5", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zlib_winsize": "-15", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "compressor_zstd_level": "1", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_alien_op_num_threads": "6", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_osd_stat_interval": "0", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_seastar_cpu_cores": "", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crimson_seastar_num_threads": "0", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_location": "", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_location_hook": "", 2024-09-02T10:01:23.547 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_location_hook_timeout": "10", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "daemonize": "false", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "daos_pool": "tank", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_alienstore": "0/5", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_allow_any_pool_priority": "false", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_asok": "1/5", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_asok_assert_abort": "false", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_asserts_on_shutdown": "true", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_auth": "1/5", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_bdev": "1/3", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_bluefs": "1/5", 2024-09-02T10:01:23.548 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_bluestore": "1/5", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_buffer": "0/1", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_ceph_dedup": "0/5", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_ceph_exporter": "1/5", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_cephfs_mirror": "0/5", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_cephsqlite": "0/5", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_client": "0/5", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_compressor": "1/5", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_context": "0/1", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_crimson_interrupt": "0/5", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_crush": "1/1", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_crypto": "1/5", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_cyanstore": "0/5", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_deliberately_leak_memory": "false", 2024-09-02T10:01:23.549 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_disable_randomized_ping": "false", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_dpdk": "1/5", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_eventtrace": "1/5", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_filer": "0/1", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_filestore": "1/3", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_finisher": "1/1", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_fuse": "1/5", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_heartbeat_testing_span": "0", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_heartbeatmap": "1/5", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_javaclient": "1/5", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_journal": "1/3", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_journaler": "0/5", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_kstore": "1/5", 2024-09-02T10:01:23.550 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_lockdep": "0/1", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mclock": "1/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds": "1/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_balancer": "1/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_locker": "1/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_log": "1/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_log_expire": "1/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_migrator": "1/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mds_quiesce": "3/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_memstore": "1/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mgr": "2/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mgrc": "1/5", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_mon": "20/20", 2024-09-02T10:01:23.551 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_monc": "0/10", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_ms": "1/1", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_none": "0/5", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_objclass": "0/5", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_objectcacher": "0/5", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_objecter": "0/1", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_optracker": "0/5", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_osd": "1/5", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_paxos": "20/20", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_perfcounter": "1/5", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_prioritycache": "1/5", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rados": "0/5", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd": "0/5", 2024-09-02T10:01:23.552 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd_mirror": "0/5", 2024-09-02T10:01:23.553 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd_pwl": "0/5", 2024-09-02T10:01:23.553 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rbd_replay": "0/5", 2024-09-02T10:01:23.553 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_refs": "0/0", 2024-09-02T10:01:23.553 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_reserver": "1/1", 2024-09-02T10:01:23.553 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw": "1/5", 2024-09-02T10:01:23.553 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_access": "1/5", 2024-09-02T10:01:23.553 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_datacache": "1/5", 2024-09-02T10:01:23.553 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_dbstore": "1/5", 2024-09-02T10:01:23.553 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_flight": "1/5", 2024-09-02T10:01:23.554 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_notification": "1/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rgw_sync": "1/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_rocksdb": "4/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore": "0/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_backref": "0/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_cache": "0/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_cleaner": "0/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_device": "0/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_epm": "0/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_journal": "0/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_lba": "0/5", 2024-09-02T10:01:23.555 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_odata": "0/5", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_omap": "0/5", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_onode": "0/5", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_t": "0/5", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_seastore_tm": "0/5", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_striper": "0/1", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_test": "0/5", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_throttle": "1/1", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_timer": "0/1", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_tp": "0/5", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "debug_trace": "1/5", 2024-09-02T10:01:23.556 INFO:teuthology.orchestra.run.smithi151.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "device_failure_prediction_mode": "none", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_graylog": "false", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_journald": "false", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_stderr": "true", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "err_to_syslog": "false", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event_tracing": "false", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_addr": "0.0.0.0", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_cert_file": "", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_http_port": "9926", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_key_file": "", 2024-09-02T10:01:23.557 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_prio_limit": "5", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_sort_metrics": "true", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "exporter_stats_period": "5", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "fake_statfs_for_testing": "0", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "fatal_signal_handlers": "true", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "filer_max_purge_ops": "10", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "filer_max_truncate_ops": "128", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_apply_finisher_threads": "1", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_blackhole": "false", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_btrfs_clone_range": "true", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_btrfs_snap": "true", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_caller_concurrency": "10", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_collect_device_partition_information": "true", 2024-09-02T10:01:23.558 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_commit_timeout": "600.000000", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_debug_inject_read_err": "false", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_debug_omap_check": "false", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_debug_verify_split": "false", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_dump_file": "", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fadvise": "true", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fail_eio": "true", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fd_cache_shards": "16", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fd_cache_size": "128", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fiemap": "false", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fiemap_threshold": "4096", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-02T10:01:23.559 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_inject_stall": "0", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_journal_parallel": "false", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_journal_trailing": "false", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_journal_writeahead": "false", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_kill_at": "0", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs": "0", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-02T10:01:23.560 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size": "0", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_merge_threshold": "-10", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_odsync_write": "false", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_omap_backend": "rocksdb", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_omap_backend_path": "", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-02T10:01:23.561 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_op_thread_timeout": "60", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_op_threads": "2", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_punch_hole": "false", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_queue_max_ops": "50", 2024-09-02T10:01:23.562 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_seek_data_hole": "false", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_sloppy_crc": "false", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_splice": "false", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_split_multiple": "2", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_split_rand_factor": "20", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_update_to": "1000", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-02T10:01:23.563 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_enable": "true", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_xfs_extsize": "false", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "filestore_zfs_snap": "false", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "fio_dir": "/tmp/fio", 2024-09-02T10:01:23.564 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_allow_other": "true", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_atomic_o_trunc": "true", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_big_writes": "true", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_debug": "false", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_default_permissions": "false", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_disable_pagecache": "false", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_max_write": "0", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_multithreaded": "true", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_require_active_mds": "true", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_set_user_groups": "true", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_splice_move": "true", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_splice_read": "true", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_splice_write": "true", 2024-09-02T10:01:23.565 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "fuse_use_invalidate_cb": "true", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "gss_target_name": "ceph", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_file": "", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_inject_failure": "0", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_interval": "5", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "host": "smithi151", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-02T10:01:23.566 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-02T10:01:23.567 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-02T10:01:23.567 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-02T10:01:23.567 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-02T10:01:23.567 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-02T10:01:23.567 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-02T10:01:23.567 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-02T10:01:23.567 INFO:teuthology.orchestra.run.smithi151.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-02T10:01:23.567 INFO:teuthology.orchestra.run.smithi151.stdout: "inject_early_sigterm": "false", 2024-09-02T10:01:23.567 INFO:teuthology.orchestra.run.smithi151.stdout: "jaeger_agent_port": "6799", 2024-09-02T10:01:23.568 INFO:teuthology.orchestra.run.smithi151.stdout: "jaeger_tracing_enable": "false", 2024-09-02T10:01:23.568 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_aio": "true", 2024-09-02T10:01:23.568 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_align_min_size": "65536", 2024-09-02T10:01:23.568 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_block_align": "true", 2024-09-02T10:01:23.568 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_block_size": "4096", 2024-09-02T10:01:23.568 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_dio": "true", 2024-09-02T10:01:23.568 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_discard": "false", 2024-09-02T10:01:23.568 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_force_aio": "false", 2024-09-02T10:01:23.568 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_ignore_corruption": "false", 2024-09-02T10:01:23.568 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_max_write_bytes": "10485760", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_max_write_entries": "100", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_replay_from": "0", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_write_header_frequency": "0", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journal_zero_on_create": "false", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journaler_prefetch_periods": "10", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journaler_prezero_periods": "5", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "journaler_write_head_interval": "15", 2024-09-02T10:01:23.569 INFO:teuthology.orchestra.run.smithi151.stdout: "key": "", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "keyfile": "", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_backend": "rocksdb", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_default_stripe_size": "65536", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_fsck_on_mount": "false", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_max_bytes": "67108864", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_max_ops": "512", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_nid_prealloc": "1024", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_onode_map_size": "1024", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_sync_submit_transaction": "false", 2024-09-02T10:01:23.570 INFO:teuthology.orchestra.run.smithi151.stdout: "kstore_sync_transaction": "false", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "librados_thread_count": "2", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "lockdep": "false", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "lockdep_force_backtrace": "false", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "log_coarse_timestamps": "true", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "log_flush_on_exit": "false", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "log_graylog_host": "127.0.0.1", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "log_graylog_port": "12201", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "log_max_new": "1000", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "log_max_recent": "10000", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "log_stderr_prefix": "", 2024-09-02T10:01:23.571 INFO:teuthology.orchestra.run.smithi151.stdout: "log_stop_at_utilization": "0.970000", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_file": "true", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_graylog": "false", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_journald": "false", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_stderr": "false", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "log_to_syslog": "false", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "max_rotating_auth_attempts": "10", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_action_on_write_error": "1", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_alternate_name_max": "8192", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_asio_thread_count": "2", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_export_pin": "true", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_dirs": "true", 2024-09-02T10:01:23.572 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_interval": "5", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_interval": "10", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_max": "-1", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_max_until": "-1", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_merge_size": "50", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_midchunk": "0.300000", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_min_start": "0.200000", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_minchunk": "0.001000", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_mode": "0", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_need_max": "1.200000", 2024-09-02T10:01:23.573 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_need_min": "0.800000", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_overload_epochs": "2", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_bits": "3", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_size": "10000", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_target_decay": "10.000000", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_beacon_grace": "15.000000", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_beacon_interval": "4.000000", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-02T10:01:23.574 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_mid": "0.700000", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_delay": "0", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_release_free_interval": "10", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_reservation": "0.050000", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_trim_interval": "1", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cache_trim_threshold": "262144", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-02T10:01:23.575 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_client_prealloc_inos": "1000", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_connect_bootstrapping": "false", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_damage_table_max_entries": "10000", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_auth_pins": "false", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_frag": "false", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_scatterstat": "false", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_debug_subtrees": "false", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_decay_halflife": "5.000000", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_default_dir_hash": "2", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_defer_session_stale": "true", 2024-09-02T10:01:23.576 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_deny_all_reconnect": "false", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_keys_per_op": "16384", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_max_commit_size": "10", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_max_entries": "0", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dir_prefetch": "true", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_on_map": "false", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_early_reply": "true", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_enable_op_tracker": "true", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_enforce_unique_name": "true", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-02T10:01:23.577 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_random": "true", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_health_summarize_threshold": "10", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-02T10:01:23.578 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_health_dummy": "false", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_migrator_session_race": "false", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_join_fs": "", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_journal_format": "1", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_create_at": "0", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_dirfrag_at": "0", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_export_at": "0", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_import_at": "0", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_journal_at": "0", 2024-09-02T10:01:23.579 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_journal_expire_at": "0", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_journal_replay_at": "0", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_link_at": "0", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_mdstable_at": "0", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_openc_at": "0", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_rename_at": "0", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_kill_shutdown_at": "0", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_event_large_threshold": "524288", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_events_per_segment": "1024", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_max_events": "-1", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_max_segments": "128", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_pause": "false", 2024-09-02T10:01:23.580 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_segment_size": "0", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_trim_threshold": "128", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_log_warn_factor": "2.000000", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_caps_per_client": "1048576", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_completed_flushes": "100000", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_completed_requests": "100000", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_export_size": "20971520", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_file_recover": "32", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_purge_files": "64", 2024-09-02T10:01:23.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_purge_ops": "8192", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_max_snaps_per_dir": "100", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_metrics_update_interval": "2", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_min_caps_per_client": "100", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_min_caps_working_set": "10000", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_numa_node": "-1", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_complaint_time": "30.000000", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_duration": "600", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_size": "20", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_slow_op_size": "20", 2024-09-02T10:01:23.582 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_op_log_threshold": "5", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_ping_grace": "15", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_ping_interval": "5", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_max_caps": "30000", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_recall_warning_threshold": "262144", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_replay_interval": "1.000000", 2024-09-02T10:01:23.583 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_root_ino_gid": "0", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_root_ino_uid": "0", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-02T10:01:23.584 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_shutdown_check": "0", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_skip_ino": "0", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_snap_max_uid": "4294967294", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_snap_min_uid": "0", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_snap_rstat": "false", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_standby_replay_damaged": "false", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_symlink_recovery": "true", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_thrash_exports": "0", 2024-09-02T10:01:23.585 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_thrash_fragments": "0", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_tick_interval": "5.000000", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_valgrind_exit": "false", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_verify_backtrace": "1", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_verify_scatter": "false", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "mds_wipe_sessions": "false", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "mempool_debug": "false", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_device_bytes": "1073741824", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_page_set": "false", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "memstore_page_size": "65536", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_client_bytes": "134217728", 2024-09-02T10:01:23.586 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_client_messages": "512", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_disabled_modules": "", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_enable_op_tracker": "true", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_max_pg_creating": "1024", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_max_pg_num_change": "128", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mds_bytes": "134217728", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mds_messages": "128", 2024-09-02T10:01:23.587 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mon_bytes": "134217728", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_mon_messages": "128", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_duration": "600", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_size": "20", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_op_log_threshold": "5", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_osd_bytes": "536870912", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_osd_messages": "8192", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_pool": "true", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-02T10:01:23.588 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_standby_modules": "true", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_stats_period": "5", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_stats_threshold": "5", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_tick_period": "2", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_allow_pool_delete": "true", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_allow_pool_size_one": "false", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_auth_validate_all_caps": "true", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_bytes": "104857600", 2024-09-02T10:01:23.589 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_directed_command_retry": "5", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_hunt_parallel": "3", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_log_interval": "1.000000", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_ping_interval": "10.000000", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_client_target_rank": "-1", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-02T10:01:23.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_level": "debug", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_file": "true", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_journald": "true", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_compact_on_bootstrap": "false", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_compact_on_start": "false", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_compact_on_trim": "true", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-02T10:01:23.591 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_cpu_threads": "4", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_crush_min_required_version": "hammer", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_daemon_bytes": "419430400", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data_avail_crit": "5", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data_avail_warn": "5", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_data_size_warn": "16106127360", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_dump_json": "false", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_dump_transactions": "false", 2024-09-02T10:01:23.592 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_extra_checks": "false", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_require_reef": "false", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_no_require_squid": "false", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_down_mkfs_grace": "60", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_down_uptime_grace": "60", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_election_default_strategy": "3", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_election_timeout": "5.000000", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_elector_ping_divisor": "2", 2024-09-02T10:01:23.593 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_enable_op_tracker": "true", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_fake_pool_delete": "false", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_force_quorum_join": "false", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_globalid_prealloc": "10000", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_detail_to_clog": "true", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_log_update_period": "5", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_max_detail": "50", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_to_clog": "true", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_to_clog_interval": "600", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_host": "[v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0]", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_host_override": "", 2024-09-02T10:01:23.594 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_initial_members": "", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_lease": "5.000000", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_log_full_interval": "50", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_log_max": "10000", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_log_max_summary": "50", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-02T10:01:23.595 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_log_epochs": "500", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_osd": "10000", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_pg_per_osd": "10000", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_pool_pg_num": "65536", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mds_force_trim_to": "0", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mds_skip_sanity": "false", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_memory_autotune": "true", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_memory_target": "2147483648", 2024-09-02T10:01:23.596 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_beacon_grace": "30", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_digest_period": "5", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_inactive_grace": "60", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_min_osdmap_epochs": "50", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_complaint_time": "30", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_duration": "600", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_size": "20", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_slow_op_size": "20", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-02T10:01:23.597 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_op_log_threshold": "5", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_auto_mark_in": "false", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_cache_size": "500", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_down_out_interval": "600", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-02T10:01:23.598 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_force_trim_to": "0", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_min_down_reporters": "2", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-02T10:01:23.599 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_report_timeout": "900", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_stuck_threshold": "60", 2024-09-02T10:01:23.600 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_probe_timeout": "2.000000", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_max_change": "0.050000", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_max_osds": "4", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-02T10:01:23.601 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_interval": "300", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_max_keys": "100", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_scrub_timeout": "300", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_session_timeout": "300", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_smart_report_timeout": "5", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stat_smooth_intervals": "6", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_pool_min_size": "2", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_pool_size": "4", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_debug": "false", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-02T10:01:23.602 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_provider_kill_at": "0", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_requester_kill_at": "0", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_sync_timeout": "60.000000", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_target_pg_per_osd": "100", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_tick_interval": "5", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_timecheck_interval": "300.000000", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_use_min_delay_socket": "false", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_older_version_delay": "604800", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-02T10:01:23.603 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_misplaced": "false", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_older_version": "true", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_pool_no_app_grace": "300", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-02T10:01:23.604 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "monmap": "", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "motr_tracing_enabled": "false", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_op_threads": "3", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_cm": "false", 2024-09-02T10:01:23.605 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_device_name": "", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_dscp": "96", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_local_gid": "", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_port_num": "1", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_sl": "3", 2024-09-02T10:01:23.606 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_support_srq": "true", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_rdma_type": "ib", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_async_reap_threshold": "5", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_before_connect": "false", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_ipv4": "true", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_ipv6": "false", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_msgr1": "true", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_msgr2": "true", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_port_max": "7568", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_port_min": "6800", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_retry_count": "3", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_bind_retry_delay": "5", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_client": "false", 2024-09-02T10:01:23.607 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_mds": "false", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_mgr": "false", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_mon": "false", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_blackhole_osd": "false", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_client_mode": "crc secure", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_cluster_mode": "crc secure", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_cluster_type": "", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_compress_secure": "false", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_connection_idle_timeout": "900", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_connection_ready_timeout": "10", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_crc_data": "true", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_crc_header": "true", 2024-09-02T10:01:23.608 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_bad_msg": "false", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_bug": "true", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_old_message": "true", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_skipped_message": "false", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_coremask": "0xF", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_enable_tso": "true", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_hugepages": "", 2024-09-02T10:01:23.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_lro": "true", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_memory_channel": "4", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_pmd": "", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_port_id": "0", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_dump_on_send": "false", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_initial_backoff": "0.200000", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_delay_max": "1.000000", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_delay_probability": "0.005000", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_delay_type": "osd", 2024-09-02T10:01:23.610 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_internal_delays": "0.002000", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_network_congestion": "0", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_inject_socket_failures": "2500", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_learn_addr_from_peer": "true", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_max_accept_failures": "4", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_max_backoff": "15.000000", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_mon_client_mode": "secure crc", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_mon_service_mode": "secure crc", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_osd_compress_min_size": "1024", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_osd_compress_mode": "none", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-02T10:01:23.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_pq_min_cost": "65536", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_public_type": "", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_service_mode": "crc secure", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_listen_backlog": "512", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_nodelay": "true", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_tcp_rcvbuf": "0", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "ms_type": "async+posix", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "no_config_file": "false", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_completion_locks_per_session": "32", 2024-09-02T10:01:23.612 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_inflight_ops": "1024", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_tick_interval": "5.000000", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "objecter_timeout": "10.000000", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "objectstore_blackhole": "false", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "openssl_engine_opts": "", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_delay_time": "5.000000", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_hist_halflife": "1000", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_max_low_ops": "2", 2024-09-02T10:01:23.613 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_max_ops": "4", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_agent_slop": "0.020000", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_asio_thread_count": "2", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_async_recovery_min_cost": "100", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backfill_scan_max": "512", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backfill_scan_min": "64", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backoff_on_degraded": "false", 2024-09-02T10:01:23.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backoff_on_peering": "false", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_backoff_on_unfound": "true", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_beacon_report_interval": "300", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_duration": "30", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_max_block_size": "67108864", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_blkin_trace_all": "false", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_check_for_log_corruption": "false", 2024-09-02T10:01:23.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_class_update_on_start": "true", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_message_cap": "256", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_message_size_cap": "524288000", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_op_priority": "63", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_client_watch_timeout": "30", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_command_max_records": "256", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_compact_on_start": "false", 2024-09-02T10:01:23.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_update_on_start": "true", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_crush_update_weight_set": "true", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_feed_pullee": "-1", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-02T10:01:23.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_misdirected_ops": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_no_acting_change": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_no_purge_strays": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_op_order": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_shutdown": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_trim_objects": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-02T10:01:23.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_snaps": "false", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_interval_cv": "0.200000", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_keys": "1024", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_stride": "524288", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_default_notify_timeout": "30", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep": "0.000000", 2024-09-02T10:01:23.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-02T10:01:23.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-02T10:01:23.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-02T10:01:23.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_discard_disconnected_ops": "true", 2024-09-02T10:01:23.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_ec_partial_reads": "true", 2024-09-02T10:01:23.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_enable_op_tracker": "true", 2024-09-02T10:01:23.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-02T10:01:23.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-02T10:01:23.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-02T10:01:23.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_info": "true", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_shutdown": "true", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_function_tracing": "false", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_grace": "20", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_interval": "6", 2024-09-02T10:01:23.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_min_peers": "10", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_min_size": "2000", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_stale": "600", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_hit_set_max_size": "100000", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_hit_set_min_size": "1000", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-02T10:01:23.622 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_journal_size": "5120", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_kill_backfill_at": "0", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_cache_size": "50", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_dedup": "true", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_message_max": "40", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_map_share_max_epochs": "40", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_attr_name_len": "100", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_attr_size": "0", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_backfills": "1", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_markdown_count": "5", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_markdown_period": "600", 2024-09-02T10:01:23.623 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_object_name_len": "2048", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_object_namespace_len": "256", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_object_size": "134217728", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pg_blocked_by": "16", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pg_log_entries": "10000", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_pgls": "1024", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_push_cost": "8388608", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_push_objects": "10", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_scrubs": "3", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_trimming_pgs": "2", 2024-09-02T10:01:23.624 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_write_op_reply_len": "64", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_max_write_size": "90", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_profile": "balanced", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-02T10:01:23.625 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_base": "805306368", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_cache_min": "134217728", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-02T10:01:23.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_target": "4294967296", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_target_autotune": "false", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_min_pg_log_entries": "250", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_min_recovery_priority": "0", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_report_interval": "5", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_num_cache_shards": "32", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_num_op_tracker_shard": "32", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_numa_auto_affinity": "true", 2024-09-02T10:01:23.627 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_numa_node": "-1", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_numa_prefer_iface": "true", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objecter_finishers": "1", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore": "bluestore", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore_fuse": "false", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_objectstore_tracing": "false", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_complaint_time": "30.000000", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_duration": "600", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_size": "20", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_slow_op_size": "20", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-02T10:01:23.628 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_log_threshold": "5", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_shards": "0", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_shards_hdd": "5", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_shards_ssd": "8", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_pq_min_cost": "65536", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_queue_cut_off": "high", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_op_thread_timeout": "15", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_open_classes_on_start": "true", 2024-09-02T10:01:23.629 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_os_flags": "0", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_peering_op_priority": "255", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_delete_cost": "1048576", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_delete_priority": "5", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_log_trim_max": "10000", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_log_trim_min": "100", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-02T10:01:23.630 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_crimson": "false", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-02T10:01:23.631 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_flags": "0", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_min_size": "2", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_pg_num": "32", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_pgp_num": "0", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_read_ratio": "70", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_size": "3", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_default_type": "replicated", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_push_per_object_cost": "1000", 2024-09-02T10:01:23.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recover_clone_overlap": "true", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_cost": "20971520", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_active": "0", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_max_single_start": "1", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_op_priority": "3", 2024-09-02T10:01:23.633 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_priority": "5", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep": "0.000000", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_repair_during_recovery": "false", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_requested_scrub_priority": "120", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_auto_repair": "false", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-02T10:01:23.634 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_begin_hour": "0", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_begin_week_day": "0", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_chunk_max": "25", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_chunk_min": "5", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_cost": "52428800", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_during_recovery": "false", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_end_hour": "0", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_end_week_day": "0", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_event_cost": "4096", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_invalid_stats": "true", 2024-09-02T10:01:23.635 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_max_preemptions": "5", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_priority": "5", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_scrub_sleep": "0.000000", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_skip_data_digest": "false", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_smart_report_timeout": "5", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_cost": "1048576", 2024-09-02T10:01:23.636 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_priority": "5", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_target_transaction_size": "30", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-02T10:01:23.637 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_tracing": "false", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_use_stale_snap": "false", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "osdc_blkin_trace_all": "false", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_kill_at": "0", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_max_join_drift": "10", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_min": "500", 2024-09-02T10:01:23.638 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_min_wait": "0.050000", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_propose_interval": "1.000000", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_service_trim_max": "500", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_service_trim_min": "10", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_stash_full_interval": "25", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_trim_max": "500", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "paxos_trim_min": "250", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "perf": "true", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "pid_file": "", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "-", 2024-09-02T10:01:23.639 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrv": "", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "public_bind_addr": "-", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "public_network": "", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "public_network_interface": "", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "qat_compressor_busy_polling": "false", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "qat_compressor_enabled": "false", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "qat_compressor_session_max_number": "256", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "rados_mon_op_timeout": "0", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "rados_osd_op_timeout": "0", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "rados_tracing": "false", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_atime_update_interval": "60", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_balance_parent_reads": "false", 2024-09-02T10:01:23.640 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_balance_snap_reads": "false", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_blkin_trace_all": "false", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache": "true", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_policy": "writearound", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_size": "33554432", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_clone_copy_on_read": "false", 2024-09-02T10:01:23.641 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_compression_hint": "none", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_concurrent_management_ops": "10", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_clone_format": "auto", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_data_pool": "", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_features": "61", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_format": "2", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_map_options": "", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_order": "22", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_pool": "rbd", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_stripe_count": "0", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_default_stripe_unit": "0", 2024-09-02T10:01:23.642 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_enable_alloc_hint": "true", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_io_scheduler": "simple", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-02T10:01:23.643 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-02T10:01:23.644 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_order": "24", 2024-09-02T10:01:23.644 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_pool": "", 2024-09-02T10:01:23.644 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_journal_splay_width": "4", 2024-09-02T10:01:23.644 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_localize_parent_reads": "false", 2024-09-02T10:01:23.644 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_localize_snap_reads": "false", 2024-09-02T10:01:23.644 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-02T10:01:23.644 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-02T10:01:23.645 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-02T10:01:23.645 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-02T10:01:23.645 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-02T10:01:23.645 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-02T10:01:23.645 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-02T10:01:23.646 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-02T10:01:23.647 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_mtime_update_interval": "60", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_non_blocking_aio": "true", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_op_thread_timeout": "60", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_op_threads": "1", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_parent_cache_enabled": "false", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_plugins": "", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_bps_burst": "0", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-02T10:01:23.648 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_bps_limit": "0", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_exclude_ops": "0", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_iops_burst": "0", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_iops_limit": "0", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-02T10:01:23.649 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_read_from_replica_policy": "default", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_skip_partial_discard": "true", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_tracing": "false", 2024-09-02T10:01:23.650 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_validate_names": "true", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rbd_validate_pool": "true", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "restapi_base_url": "", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "restapi_log_level": "", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_acl_grants_max_num": "100", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_admin_entry": "admin", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_asio_assert_yielding": "false", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_backend_store": "rados", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_barbican_url": "", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_beast_enable_async": "true", 2024-09-02T10:01:23.651 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_counters_cache": "false", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cache_enabled": "true", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cache_expiry_interval": "900", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cache_lru_size": "25000", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_config_store": "rados", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_content_length_compat": "false", 2024-09-02T10:01:23.652 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_copy_obj_progress": "true", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cors_rules_max_num": "100", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_cross_domain_policy": "", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_addr": "", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_password": "", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_kmip_username": "", 2024-09-02T10:01:23.653 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_require_ssl": "true", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-02T10:01:23.654 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_addr": "", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_auth": "token", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_namespace": "", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_prefix": "", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_token_file": "", 2024-09-02T10:01:23.655 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-02T10:01:23.656 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_buffersize": "524288", 2024-09-02T10:01:23.656 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-02T10:01:23.656 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_low_speed_time": "300", 2024-09-02T10:01:23.656 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-02T10:01:23.656 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-02T10:01:23.656 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-02T10:01:23.656 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-02T10:01:23.656 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-02T10:01:23.656 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-02T10:01:23.656 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-02T10:01:23.657 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-02T10:01:23.657 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-02T10:01:23.657 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-02T10:01:23.657 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-02T10:01:23.657 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-02T10:01:23.657 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-02T10:01:23.657 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-02T10:01:23.657 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-02T10:01:23.657 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-02T10:01:23.657 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_changes_size": "1000", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_num_shards": "128", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_log_window": "30", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-02T10:01:23.658 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_disable_s3select": "false", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-02T10:01:23.659 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dns_name": "", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dns_s3website_name": "", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dynamic_resharding": "true", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_gc_threads": "true", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_lc_threads": "true", 2024-09-02T10:01:23.660 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_mdsearch": "true", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_ops_log": "false", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_quota_threads": "true", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_static_website": "false", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enable_usage_log": "false", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_enforce_swift_acls": "true", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_exit_timeout_secs": "120", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_expose_bucket": "false", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_extended_http_attrs": "", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_filter": "none", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_frontends": "beast port=7480", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-02T10:01:23.661 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_deferred": "50", 2024-09-02T10:01:23.662 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-02T10:01:23.662 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_objs": "32", 2024-09-02T10:01:23.662 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-02T10:01:23.662 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-02T10:01:23.662 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-02T10:01:23.662 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-02T10:01:23.664 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_gc_processor_period": "3600", 2024-09-02T10:01:23.664 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-02T10:01:23.664 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-02T10:01:23.664 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-02T10:01:23.664 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-02T10:01:23.664 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_init_timeout": "300", 2024-09-02T10:01:23.664 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-02T10:01:23.664 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-02T10:01:23.665 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_kafka_connection_idle": "300", 2024-09-02T10:01:23.665 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-02T10:01:23.665 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-02T10:01:23.665 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-02T10:01:23.665 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-02T10:01:23.665 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-02T10:01:23.665 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_domain": "", 2024-09-02T10:01:23.665 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_password": "", 2024-09-02T10:01:23.665 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_password_path": "", 2024-09-02T10:01:23.665 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_project": "", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_tenant": "", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_token": "", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_token_path": "", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_admin_user": "", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_domain": "", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_password": "", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_project": "", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_barbican_user": "", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-02T10:01:23.666 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_url": "", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_debug_interval": "-1", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_lock_max_time": "90", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_objs": "32", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_rules": "1000", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_worker": "3", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-02T10:01:23.667 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lc_thread_delay": "0", 2024-09-02T10:01:23.668 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-02T10:01:23.668 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_dnattr": "uid", 2024-09-02T10:01:23.668 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-02T10:01:23.668 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_searchfilter": "", 2024-09-02T10:01:23.668 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-02T10:01:23.668 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ldap_uri": "", 2024-09-02T10:01:23.668 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-02T10:01:23.668 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_http_headers": "", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_log_object_name_utc": "false", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_attr_name_len": "0", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_attr_size": "0", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-02T10:01:23.669 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_chunk_size": "4194304", 2024-09-02T10:01:23.670 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-02T10:01:23.670 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-02T10:01:23.670 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-02T10:01:23.670 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_listing_results": "1000", 2024-09-02T10:01:23.670 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_notify_retries": "10", 2024-09-02T10:01:23.670 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-02T10:01:23.670 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_put_param_size": "1048576", 2024-09-02T10:01:23.670 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_put_size": "5368709120", 2024-09-02T10:01:23.670 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_max_slo_entries": "1000", 2024-09-02T10:01:23.670 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_md_log_max_shards": "64", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_mp_lock_max_time": "600", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-02T10:01:23.671 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-02T10:01:23.672 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-02T10:01:23.672 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_max_gc": "300", 2024-09-02T10:01:23.672 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-02T10:01:23.672 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-02T10:01:23.672 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-02T10:01:23.672 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-02T10:01:23.672 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-02T10:01:23.672 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-02T10:01:23.672 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-02T10:01:23.672 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_num_async_rados_threads": "32", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_num_control_oids": "8", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_numa_node": "-1", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_objexp_chunk_size": "100", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_objexp_gc_interval": "600", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_op_thread_timeout": "600", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_op_tracing": "false", 2024-09-02T10:01:23.673 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_opa_token": "", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_opa_url": "", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_opa_verify_ssl": "true", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_rados": "false", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_ops_log_socket_path": "", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_push_interval": "2.000000", 2024-09-02T10:01:23.674 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-02T10:01:23.675 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-02T10:01:23.675 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-02T10:01:23.675 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-02T10:01:23.675 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_lanes": "3", 2024-09-02T10:01:23.675 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-02T10:01:23.675 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-02T10:01:23.675 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_cache_partitions": "3", 2024-09-02T10:01:23.675 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-02T10:01:23.675 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_print_continue": "true", 2024-09-02T10:01:23.675 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-02T10:01:23.676 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-02T10:01:23.676 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-02T10:01:23.676 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-02T10:01:23.676 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-02T10:01:23.676 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rados_tracing": "false", 2024-09-02T10:01:23.676 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_realm": "", 2024-09-02T10:01:23.676 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_realm_id": "", 2024-09-02T10:01:23.676 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-02T10:01:23.676 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_region": "", 2024-09-02T10:01:23.676 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_request_uri": "", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_batch_size": "64", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_max_aio": "128", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_num_logs": "16", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_reshard_thread_interval": "600", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_resolve_cname": "false", 2024-09-02T10:01:23.677 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_run_sync_thread": "true", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_scheduler_type": "throttler", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_script_uri": "", 2024-09-02T10:01:23.678 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_service_provider_name": "", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_client_id": "", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_client_secret": "", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_entry": "sts", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_key": "sts", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_min_session_duration": "900", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sts_token_introspection_url": "", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_account_in_url": "false", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_auth_entry": "auth", 2024-09-02T10:01:23.679 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_auth_url": "", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_custom_header": "", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_need_stats": "true", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_tenant_name": "", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_token_expiration": "86400", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_url": "", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_url_prefix": "swift", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_lease_period": "120", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-02T10:01:23.680 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_thread_pool_size": "512", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-02T10:01:23.681 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_comment": "", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_createby": "", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_encoding": "", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_flag": "false", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_origin": "", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_torrent_tracker": "", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_trust_forwarded_https": "false", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_max_shards": "32", 2024-09-02T10:01:23.682 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_usage_max_user_shards": "1", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_use_opa_authz": "false", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_counters_cache": "false", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_max_buckets": "1000", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_policies_max_num": "100", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_user_unique_email": "true", 2024-09-02T10:01:23.683 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_verify_ssl": "true", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zone": "", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zone_id": "", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zonegroup": "", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zonegroup_id": "", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rgwlc_auto_session_clear": "true", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_block_size": "4096", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-02T10:01:23.684 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_size": "536870912", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_index_type": "binary_search", 2024-09-02T10:01:23.685 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_partition_filters": "false", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_perf": "false", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "run_dir": "/var/run/ceph", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_block_create": "true", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_cache_lru_size": "67108864", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_default_max_object_size": "16777216", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-02T10:01:23.686 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_device_size": "53687091200", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_full_integrity_check": "false", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_batch_capacity": "16", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_main_device_type": "SSD", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "seastore_segment_size": "67108864", 2024-09-02T10:01:23.687 INFO:teuthology.orchestra.run.smithi151.stdout: "set_keepcaps": "false", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "setgroup": "ceph", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "setuser": "ceph", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "setuser_match_path": "", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "thp": "false", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "threadpool_default_timeout": "60", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "throttler_perf_counter": "true", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "tmp_dir": "/tmp", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "uadk_compressor_enabled": "false", 2024-09-02T10:01:23.688 INFO:teuthology.orchestra.run.smithi151.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-02T10:01:23.689 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:01:24.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:23 smithi187 ceph-mon[20072]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:24.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:23 smithi162 ceph-mon[20279]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:24.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:23 smithi151 ceph-mon[16639]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:24.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-02T10:01:24.924 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:25.244 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:24 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:25.244 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:24 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:24 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:24 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:24 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:24 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:25.457 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a: {} 2024-09-02T10:01:25.457 INFO:teuthology.orchestra.run.smithi151.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-02T10:01:25.471 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c: {} 2024-09-02T10:01:25.471 INFO:teuthology.orchestra.run.smithi151.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-02T10:01:25.485 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b: {} 2024-09-02T10:01:25.485 INFO:teuthology.orchestra.run.smithi151.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-02T10:01:26.224 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-02T10:01:26.234 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-02T10:01:26.244 INFO:tasks.radosbench:Beginning radosbench... 2024-09-02T10:01:26.245 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-02T10:01:26.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-02T10:01:26.245 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-02T10:01:26.246 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:01:26.246 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:01:26.246 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-02T10:01:26.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:26.246 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-02T10:01:26.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:01:26.247 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-02T10:01:26.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:01:26.247 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-02T10:01:26.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:01:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:25 smithi187 ceph-mon[20072]: osdmap e90: 12 total, 12 up, 12 in 2024-09-02T10:01:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:25 smithi187 ceph-mon[20072]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1974780742' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T10:01:26.274 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:25 smithi151 ceph-mon[16639]: osdmap e90: 12 total, 12 up, 12 in 2024-09-02T10:01:26.274 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:25 smithi151 ceph-mon[16639]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:26.274 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1974780742' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T10:01:26.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:26.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:26.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:26.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:25 smithi162 ceph-mon[20279]: osdmap e90: 12 total, 12 up, 12 in 2024-09-02T10:01:26.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:25 smithi162 ceph-mon[20279]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:26.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1974780742' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T10:01:26.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:26.403 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:26.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:26.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:26.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:26.530 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:26.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:26.552 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:26.552 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:26.552 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:26.553 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:26.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:26.656 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:26.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:26.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:26.783 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:26.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:26.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:26.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:26.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:26.913 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:27.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:27.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:27.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:27.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:27.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:27.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:27.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:27.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:27.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:27.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:27.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:27.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:27.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:27.539 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:27.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:27.540 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:27.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:27.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:27.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:27.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:27.791 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:27.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:27.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:27.918 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:27.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:27.977 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:01:27.977 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:01:27.998 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:01:27.998 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:01:28.018 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:01:28.018 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:01:28.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:28.038 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:01:28.038 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:01:28.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:28.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:28.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:01:28.059 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:01:28.079 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:01:28.080 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:01:28.101 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:01:28.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:01:28.117 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:01:28.124 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:01:28.124 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:01:28.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:28.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:01:28.148 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:01:28.169 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:01:28.169 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:01:28.171 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:28.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:28.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:28.191 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:01:28.191 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:01:28.216 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:01:28.216 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:01:28.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:28 smithi151 ceph-mon[16639]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:28.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3823855624' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:01:28.216 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1621457313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:28.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:28.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:28.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:28.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:28.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:28.423 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:28.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:28 smithi187 ceph-mon[20072]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3823855624' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:01:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1621457313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:28.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:28.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:28.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:28.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:28.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:28 smithi162 ceph-mon[20279]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3823855624' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:01:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1621457313' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:28.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:28.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:28.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:28.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:28.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:28.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:28.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:28.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:28.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:29.023 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:29.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:29.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:29.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:29.121 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_0' created 2024-09-02T10:01:29.121 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:29 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:29.121 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:29 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:29.121 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:29 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:29.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3823855624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:01:29.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:29 smithi151 ceph-mon[16639]: osdmap e91: 12 total, 12 up, 12 in 2024-09-02T10:01:29.122 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2110460793' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-02T10:01:29.124 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:01:29.124 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:01:29.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:29.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:29.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:29.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:29.301 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:29.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:29.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:29.431 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:29.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:29 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:29 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:29 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3823855624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:01:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:29 smithi187 ceph-mon[20072]: osdmap e91: 12 total, 12 up, 12 in 2024-09-02T10:01:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2110460793' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-02T10:01:29.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:29.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:29.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:29.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:29 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:29 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:29 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3823855624' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:01:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:29 smithi162 ceph-mon[20279]: osdmap e91: 12 total, 12 up, 12 in 2024-09-02T10:01:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2110460793' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} : dispatch 2024-09-02T10:01:29.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:29.683 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:29.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:29.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:29.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:29.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:29.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:29.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:29.935 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:29.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:30.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:30.070 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:30.070 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:30.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:30.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:30 smithi151 ceph-mon[16639]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2110460793' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-02T10:01:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:30 smithi151 ceph-mon[16639]: osdmap e92: 12 total, 12 up, 12 in 2024-09-02T10:01:30.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:30.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:30.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:30.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:30.322 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:30.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:30.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:30.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:30.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:30.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:30 smithi187 ceph-mon[20072]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2110460793' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-02T10:01:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:30 smithi187 ceph-mon[20072]: osdmap e92: 12 total, 12 up, 12 in 2024-09-02T10:01:30.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:30.574 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:30.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:30.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:30 smithi162 ceph-mon[20279]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:30.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2110460793' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-02T10:01:30.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:30 smithi162 ceph-mon[20279]: osdmap e92: 12 total, 12 up, 12 in 2024-09-02T10:01:30.675 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:30.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:30.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:30.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:01:30.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:30.830 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:30.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:30.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:01:30.932 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:30.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:30.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:31.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:31.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:31.085 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:31.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:31.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:31.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:31.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:31.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-02T10:01:31.283 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:31.311 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:31.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:31.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:31.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:31 smithi151 ceph-mon[16639]: osdmap e93: 12 total, 12 up, 12 in 2024-09-02T10:01:31.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:31.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:31.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:31 smithi187 ceph-mon[20072]: osdmap e93: 12 total, 12 up, 12 in 2024-09-02T10:01:31.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:31.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:31.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:31.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:31 smithi162 ceph-mon[20279]: osdmap e93: 12 total, 12 up, 12 in 2024-09-02T10:01:31.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:31.716 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:31.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:31.782 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:31.816 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:31.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:31.843 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:31.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:31.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:31.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:31.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:31.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:32.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:32.094 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:32.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:32.143 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:01:32.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:32.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:32.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:32.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:01:32.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:32.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:32 smithi151 ceph-mon[16639]: pgmap v289: 17 pgs: 3 active+clean, 14 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:32.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:32 smithi151 ceph-mon[16639]: osdmap e94: 12 total, 12 up, 12 in 2024-09-02T10:01:32.320 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2662430258' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:01:32.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:32.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:32.442 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:32.468 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:32.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:32.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:32 smithi187 ceph-mon[20072]: pgmap v289: 17 pgs: 3 active+clean, 14 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:32 smithi187 ceph-mon[20072]: osdmap e94: 12 total, 12 up, 12 in 2024-09-02T10:01:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2662430258' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:01:32.540 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:01:32.540 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:01:32.570 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:32.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:32 smithi162 ceph-mon[20279]: pgmap v289: 17 pgs: 3 active+clean, 14 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:32 smithi162 ceph-mon[20279]: osdmap e94: 12 total, 12 up, 12 in 2024-09-02T10:01:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2662430258' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:01:32.595 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:32.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:32.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:32.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:32.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:32.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:32.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:32.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:32.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:32.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:32.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:32.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:32.970 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:32.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:32.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:33.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:33.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:33.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:33.148 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-02T10:01:33.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:33.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:33.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:33.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:33.325 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2662430258' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:01:33.325 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:33 smithi151 ceph-mon[16639]: osdmap e95: 12 total, 12 up, 12 in 2024-09-02T10:01:33.325 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1628922373' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:01:33.343 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:33.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:33.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:33.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:33.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2662430258' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:01:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:33 smithi187 ceph-mon[20072]: osdmap e95: 12 total, 12 up, 12 in 2024-09-02T10:01:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1628922373' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:01:33.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:33.593 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:33.594 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2662430258' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:01:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:33 smithi162 ceph-mon[20279]: osdmap e95: 12 total, 12 up, 12 in 2024-09-02T10:01:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1628922373' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:01:33.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:33.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:33.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:33.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:33.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:33.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:33.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:33.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:33.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:33.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:33.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:33.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:34.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:34.091 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:34.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:34.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:34.218 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:34.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:34.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:34.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:34.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:34 smithi151 ceph-mon[16639]: pgmap v292: 17 pgs: 14 active+clean, 3 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:34.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1628922373' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:01:34.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:34 smithi151 ceph-mon[16639]: osdmap e96: 12 total, 12 up, 12 in 2024-09-02T10:01:34.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:34 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:34.346 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:34.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:34.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:34.410 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:34.424 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:34 smithi187 ceph-mon[20072]: pgmap v292: 17 pgs: 14 active+clean, 3 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:34.424 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1628922373' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:01:34.424 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:34 smithi187 ceph-mon[20072]: osdmap e96: 12 total, 12 up, 12 in 2024-09-02T10:01:34.424 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:34 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:34.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:34.473 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:34.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:34.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:34.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:34.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:34 smithi162 ceph-mon[20279]: pgmap v292: 17 pgs: 14 active+clean, 3 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:01:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1628922373' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:01:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:34 smithi162 ceph-mon[20279]: osdmap e96: 12 total, 12 up, 12 in 2024-09-02T10:01:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:34 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:34.598 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:34.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:34.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:34.727 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:34.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:34.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:34.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:01:34.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:34.854 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:34.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:34.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:34.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:34.982 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:34.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:35.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:35.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:35.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:35.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:35.241 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:35.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:35.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:35.350 DEBUG:teuthology.orchestra.run.smithi187:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-02T10:01:35.351 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-02T10:01:35.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:01:35.353 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-02T10:01:35.353 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-02T10:01:35.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:35.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:35.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:35.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:01:35.378 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:01:35.413 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:01:35.413 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:01:35.415 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:35.432 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:hints = 1 2024-09-02T10:01:35.432 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-02T10:01:35.432 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object prefix: benchmark_data_smithi187_45882 2024-09-02T10:01:35.432 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:01:35.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 0 0 0 0 0 0 - 0 2024-09-02T10:01:35.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:35.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:35.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:35.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:35.613 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:35.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:01:35.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:35.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:35.650 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:01:35.651 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:01:35.685 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:01:35.685 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:01:35.717 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:01:35.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:01:35.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:35.754 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:01:35.754 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:01:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:35.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:35.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:35.795 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:01:35.795 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:01:35.833 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:01:35.833 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:01:35.864 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:01:35.864 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:01:35.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:35.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:01:35.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:01:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:35.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:35.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:35.930 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:01:35.930 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:01:35.952 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:01:35.952 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:01:36.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:36.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:36.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:36.054 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:36.088 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:01:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:01:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:01:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:01:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.14157.0:118 1.0 e22/20 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 22'88, mlcod=22'83)", 2024-09-02T10:01:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T09:56:56.055536+0000", 2024-09-02T10:01:36.088 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 279.96997108199997, 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.079312797000000004, 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:56.055536+0000", 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:56.055536+0000", 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:56.055536+0000", 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:56.055541+0000", 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2270000000000001e-06 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:56.134423+0000", 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.078881092999999999 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:56.134429+0000", 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5390000000000004e-06 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:56.134488+0000", 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8854999999999998e-05 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:56.134503+0000", 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4636e-05 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:56.134818+0000", 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031517900000000001 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:56.134848+0000", 2024-09-02T10:01:36.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0592000000000001e-05 2024-09-02T10:01:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:432 2.1 e96/92 2:8394e448:::benchmark_data_smithi187_45882_object431:head v 96'34, mlcod=96'31)", 2024-09-02T10:01:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.558611+0000", 2024-09-02T10:01:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.46689515199999998, 2024-09-02T10:01:36.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011707801, 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.558611+0000", 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.558611+0000", 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.558612+0000", 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2200000000000003e-07 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.558658+0000", 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6215999999999999e-05 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.569641+0000", 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01098208 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.569648+0000", 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.661e-06 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.569725+0000", 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6699000000000005e-05 2024-09-02T10:01:36.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.569747+0000", 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2191000000000001e-05 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.570242+0000", 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049440400000000005 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.570319+0000", 2024-09-02T10:01:36.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7727999999999998e-05 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:472 2.1 e96/92 2:8466bae1:::benchmark_data_smithi187_45882_object471:head v 96'35, mlcod=96'32)", 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.569594+0000", 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.45591289099999999, 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014325853, 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.569594+0000", 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.569594+0000", 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.569595+0000", 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.006e-06 2024-09-02T10:01:36.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.569622+0000", 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6862000000000001e-05 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.582183+0000", 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012561104 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.582190+0000", 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9179999999999997e-06 2024-09-02T10:01:36.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.582246+0000", 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6060999999999997e-05 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.582282+0000", 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5957999999999997e-05 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.583868+0000", 2024-09-02T10:01:36.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00158672 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.583920+0000", 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1224000000000003e-05 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:602 2.5 e96/92 2:a9393917:::benchmark_data_smithi187_45882_object601:head v 96'37, mlcod=96'34)", 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.610957+0000", 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.41454910699999997, 2024-09-02T10:01:36.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025329132000000001, 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.610957+0000", 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.610957+0000", 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.610958+0000", 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.41e-07 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.610981+0000", 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3213999999999998e-05 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.635591+0000", 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024610150000000001 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.635599+0000", 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9289999999999998e-06 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.635719+0000", 2024-09-02T10:01:36.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120363 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.635742+0000", 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2419e-05 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.636241+0000", 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049863200000000005 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.636287+0000", 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6084000000000001e-05 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:667 2.1 e96/92 2:868f2647:::benchmark_data_smithi187_45882_object666:head v 96'47, mlcod=96'44)", 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.633561+0000", 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.391945337, 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011925406, 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.633561+0000", 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.633561+0000", 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.633562+0000", 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2900000000000002e-07 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.633588+0000", 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5669000000000001e-05 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.645001+0000", 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011413156000000001 2024-09-02T10:01:36.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.645009+0000", 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5939999999999996e-06 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.645073+0000", 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4193e-05 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.645093+0000", 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0526999999999998e-05 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.645440+0000", 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034710900000000001 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.645487+0000", 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6229000000000001e-05 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:736 2.1 e96/92 2:82cdab46:::benchmark_data_smithi187_45882_object735:head v 96'50, mlcod=96'47)", 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.659230+0000", 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.36627695399999999, 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018227710000000001, 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.659230+0000", 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.659230+0000", 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.659230+0000", 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4899999999999995e-07 2024-09-02T10:01:36.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.659255+0000", 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4745000000000001e-05 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.676878+0000", 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017622912000000001 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.676885+0000", 2024-09-02T10:01:36.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2599999999999999e-06 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.676934+0000", 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9082999999999997e-05 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.676953+0000", 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8766e-05 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.677433+0000", 2024-09-02T10:01:36.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00047997900000000001 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.677457+0000", 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4315999999999999e-05 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:775 2.1 e96/92 2:83e99205:::benchmark_data_smithi187_45882_object774:head v 96'51, mlcod=96'47)", 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.676830+0000", 2024-09-02T10:01:36.116 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.34867622999999998, 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024525133000000001, 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.676830+0000", 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.676830+0000", 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.676831+0000", 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2099999999999995e-07 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.676855+0000", 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3869000000000001e-05 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.700758+0000", 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023902730000000001 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.700766+0000", 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2859999999999999e-06 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.700831+0000", 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5109000000000003e-05 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.700851+0000", 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.001e-05 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.701309+0000", 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045811099999999998 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.701356+0000", 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6196999999999998e-05 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.120 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:820 2.5 e96/92 2:a1f21660:::benchmark_data_smithi187_45882_object819:head v 96'51, mlcod=96'47)", 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.696355+0000", 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.32915139500000001, 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.028385943, 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.696355+0000", 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.696355+0000", 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.696356+0000", 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7899999999999998e-07 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.696522+0000", 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016593900000000001 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.723788+0000", 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027266008000000001 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.723793+0000", 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4970000000000001e-06 2024-09-02T10:01:36.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.723935+0000", 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014180799999999999 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.723955+0000", 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0208999999999999e-05 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.724696+0000", 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00074093 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.724741+0000", 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4972999999999998e-05 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:888 2.5 e96/92 2:a1d5ca05:::benchmark_data_smithi187_45882_object887:head v 96'54, mlcod=96'51)", 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.733772+0000", 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.291734932, 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017888669999999999, 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.733772+0000", 2024-09-02T10:01:36.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.733772+0000", 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.733772+0000", 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5400000000000001e-07 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.733804+0000", 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2030000000000003e-05 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.751045+0000", 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017240848 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.751055+0000", 2024-09-02T10:01:36.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6600000000000007e-06 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.751120+0000", 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5151e-05 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.751142+0000", 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2184999999999999e-05 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.751606+0000", 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046380999999999999 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.751660+0000", 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4332000000000001e-05 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.130 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:889 2.1 e96/92 2:8a0fa524:::benchmark_data_smithi187_45882_object888:head v 96'58, mlcod=96'53)", 2024-09-02T10:01:36.131 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.734826+0000", 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.29068033500000001, 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023737693000000001, 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.734826+0000", 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.734826+0000", 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.734827+0000", 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4799999999999998e-07 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.734856+0000", 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9050000000000001e-05 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.757843+0000", 2024-09-02T10:01:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022987156000000002 2024-09-02T10:01:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.757850+0000", 2024-09-02T10:01:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.832e-06 2024-09-02T10:01:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.757945+0000", 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5433000000000006e-05 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.757963+0000", 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7853e-05 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.758515+0000", 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000552105 2024-09-02T10:01:36.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.758564+0000", 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8615999999999997e-05 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:906 2.5 e96/92 2:a802d548:::benchmark_data_smithi187_45882_object905:head v 96'55, mlcod=96'52)", 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.750976+0000", 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.27453046599999997, 2024-09-02T10:01:36.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012845565, 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.750976+0000", 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.750976+0000", 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.750978+0000", 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3710000000000001e-06 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.751019+0000", 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1746999999999999e-05 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.763270+0000", 2024-09-02T10:01:36.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012250756999999999 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.763276+0000", 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0560000000000001e-06 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.763361+0000", 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4988999999999996e-05 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.763381+0000", 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0041999999999999e-05 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.763788+0000", 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000406489 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.763822+0000", 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4113999999999999e-05 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.140 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:932 2.5 e96/92 2:af388f43:::benchmark_data_smithi187_45882_object931:head v 96'57, mlcod=96'54)", 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.764712+0000", 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.26079499, 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.03357102, 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.764712+0000", 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.764712+0000", 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.764713+0000", 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.099e-06 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.764745+0000", 2024-09-02T10:01:36.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1955000000000002e-05 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.797818+0000", 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.033073405 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.797824+0000", 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.297e-06 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.797910+0000", 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5829999999999996e-05 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.797929+0000", 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8678999999999999e-05 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.798256+0000", 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000327466 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.798283+0000", 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6288999999999999e-05 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:937 2.1 e96/92 2:8ac3bab2:::benchmark_data_smithi187_45882_object936:head v 96'62, mlcod=96'57)", 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.770977+0000", 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.254529167, 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023852398, 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.145 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.770977+0000", 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.770977+0000", 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.770978+0000", 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6300000000000004e-07 2024-09-02T10:01:36.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.771000+0000", 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1319e-05 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.794186+0000", 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023186140000000001 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.794193+0000", 2024-09-02T10:01:36.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2089999999999999e-06 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.794249+0000", 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6106000000000001e-05 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.794268+0000", 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.878e-05 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.794787+0000", 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051869800000000005 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.794830+0000", 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3282999999999999e-05 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:969 2.1 e96/92 2:857be463:::benchmark_data_smithi187_45882_object968:head v 96'63, mlcod=96'59)", 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.794137+0000", 2024-09-02T10:01:36.149 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.23136942499999999, 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017335869, 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.794137+0000", 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.794137+0000", 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.794139+0000", 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3820000000000001e-06 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.794162+0000", 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3112000000000002e-05 2024-09-02T10:01:36.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.810782+0000", 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016620697 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.810795+0000", 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2225999999999999e-05 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.810845+0000", 2024-09-02T10:01:36.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0824e-05 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.810866+0000", 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0557e-05 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.811424+0000", 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00055797500000000005 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.811473+0000", 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9095999999999999e-05 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:980 2.5 e96/92 2:a71edb35:::benchmark_data_smithi187_45882_object979:head v 96'59, mlcod=96'56)", 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.800352+0000", 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.22515487000000001, 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012297410999999999, 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.154 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.800352+0000", 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.800352+0000", 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.800352+0000", 2024-09-02T10:01:36.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3600000000000003e-07 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.800455+0000", 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102212 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.812130+0000", 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01167496 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.812137+0000", 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0879999999999997e-06 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.812220+0000", 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3163000000000003e-05 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.812241+0000", 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1089000000000001e-05 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.812613+0000", 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037240199999999998 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.812649+0000", 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5760999999999999e-05 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.158 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:992 2.1 e96/92 2:8d297835:::benchmark_data_smithi187_45882_object991:head v 96'64, mlcod=96'60)", 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.810720+0000", 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.214786596, 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015216161000000001, 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.810720+0000", 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.810720+0000", 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.810722+0000", 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5880000000000001e-06 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.810755+0000", 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3294999999999999e-05 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.824588+0000", 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013832664999999999 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.824596+0000", 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2330000000000006e-06 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.824658+0000", 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2287999999999997e-05 2024-09-02T10:01:36.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.824679+0000", 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1347000000000001e-05 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.825894+0000", 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001214288 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.825936+0000", 2024-09-02T10:01:36.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2456999999999997e-05 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:1004 2.5 e96/92 2:aced1ea8:::benchmark_data_smithi187_45882_object1003:head v 96'62, mlcod=96'59)", 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.818980+0000", 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.206526823, 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015936513999999999, 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.818980+0000", 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.818980+0000", 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.818981+0000", 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.319e-06 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.819069+0000", 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7834e-05 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.834312+0000", 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015243263999999999 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.834321+0000", 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6470000000000004e-06 2024-09-02T10:01:36.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.834388+0000", 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7181999999999995e-05 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.834414+0000", 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5771000000000001e-05 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.834867+0000", 2024-09-02T10:01:36.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045357299999999998 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.834916+0000", 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8924000000000001e-05 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:1042 2.5 e96/92 2:af95b5c0:::benchmark_data_smithi187_45882_object1041:head v 96'64, mlcod=96'61)", 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.840428+0000", 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.185079091, 2024-09-02T10:01:36.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01255441, 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.840428+0000", 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.840428+0000", 2024-09-02T10:01:36.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.840428+0000", 2024-09-02T10:01:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0699999999999996e-07 2024-09-02T10:01:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.840510+0000", 2024-09-02T10:01:36.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1847000000000004e-05 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.852457+0000", 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011947101 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.852464+0000", 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5019999999999998e-06 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.852527+0000", 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3456999999999999e-05 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.852556+0000", 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8399e-05 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.852939+0000", 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038383900000000002 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.852982+0000", 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2457999999999999e-05 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:1059 2.5 e96/92 2:a0543e42:::benchmark_data_smithi187_45882_object1058:head v 96'65, mlcod=96'62)", 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.852338+0000", 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.17316891700000001, 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017417922999999998, 2024-09-02T10:01:36.172 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.852338+0000", 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.852338+0000", 2024-09-02T10:01:36.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.852338+0000", 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4300000000000003e-07 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.852432+0000", 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3678999999999995e-05 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.869192+0000", 2024-09-02T10:01:36.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016760313999999998 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.869197+0000", 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.04e-06 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.869257+0000", 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.927e-05 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.869279+0000", 2024-09-02T10:01:36.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2501e-05 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.869675+0000", 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039544299999999999 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.869756+0000", 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1032999999999995e-05 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:1097 2.5 e96/92 2:aceedfd8:::benchmark_data_smithi187_45882_object1096:head v 96'67, mlcod=96'64)", 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.877259+0000", 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.14824769099999999, 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.094072450000000002, 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.877259+0000", 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.877259+0000", 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.877260+0000", 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.877292+0000", 2024-09-02T10:01:36.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2917999999999999e-05 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.970784+0000", 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.093491574999999993 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.970793+0000", 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4040000000000001e-06 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.970866+0000", 2024-09-02T10:01:36.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2457000000000001e-05 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.970888+0000", 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1827999999999998e-05 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.971288+0000", 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040000700000000002 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.971331+0000", 2024-09-02T10:01:36.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3599e-05 2024-09-02T10:01:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:36.181 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:36.181 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:01:36.181 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:36.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/457426001' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:36.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:01:36.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:01:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:01:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:01:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:01:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:01:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:36 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:01:36.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:36.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:36.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:36.336 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:36.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:36.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:36.432 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 1 16 1785 1769 110.556 110.562 0.00103162 0.00879135 2024-09-02T10:01:36.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:36.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:36.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:36.461 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/457426001' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:01:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:01:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:01:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:01:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:01:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:01:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:36 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:01:36.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:36.590 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:36.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:36.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/457426001' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:36.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:01:36.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:01:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:01:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:01:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:01:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:01:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:36 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:01:36.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:36.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:36.719 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:36.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:36.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:36.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:36.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:36.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:36.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:36.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:37.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:37.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:37.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:37.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:37 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:37.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:37 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:37.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:37.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:37.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:37.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:37.361 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:37.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:37.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:37.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:01:37.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 2 16 3973 3957 123.648 136.75 0.00169833 0.00806912 2024-09-02T10:01:37.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:37.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:37.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:37.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:37.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:37 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:37 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:37.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:37 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:37.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:37 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:37.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:37.618 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:37.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:37.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:37.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:37.747 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:37.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:37.748 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:37.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:37.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:37.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:37.880 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:37.921 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:37.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:37.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:37.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:37.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:38.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:38.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:38.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:38.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:38.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:38.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:38.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:38.372 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:38.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:38.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 3 16 5452 5436 113.241 92.4375 0.0833653 0.00861933 2024-09-02T10:01:38.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:38 smithi151 ceph-mon[16639]: pgmap v295: 17 pgs: 17 active+clean; 77 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 202 op/s 2024-09-02T10:01:38.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:38 smithi151 ceph-mon[16639]: osdmap e97: 12 total, 12 up, 12 in 2024-09-02T10:01:38.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:38.496 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:38.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:38.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:38 smithi187 ceph-mon[20072]: pgmap v295: 17 pgs: 17 active+clean; 77 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 202 op/s 2024-09-02T10:01:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:38 smithi187 ceph-mon[20072]: osdmap e97: 12 total, 12 up, 12 in 2024-09-02T10:01:38.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:38 smithi162 ceph-mon[20279]: pgmap v295: 17 pgs: 17 active+clean; 77 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 202 op/s 2024-09-02T10:01:38.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:38 smithi162 ceph-mon[20279]: osdmap e97: 12 total, 12 up, 12 in 2024-09-02T10:01:38.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:38.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:38.621 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:38.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:38.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:38.745 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:38.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:38.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:38.825 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:01:38.826 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:01:38.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:38.873 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:38.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:38.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:38.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:38.998 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:38.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:39.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:39.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:39.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:39.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:39.256 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:39.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:39.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:39.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:39.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:39.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 4 16 7686 7670 119.834 139.625 0.00123227 0.00829253 2024-09-02T10:01:39.472 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:39.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:39.512 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:39.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:39.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:39.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:39.636 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:39.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:39.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:39.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:39.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:39.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:39.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:39.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:39.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:01:39.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:40.014 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:40.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:40.045 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-02T10:01:37.158367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":80216608,"num_objects":1217,"num_object_clones":0,"num_object_copies":3651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1217,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":1338,"num_write_kb":79910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1307,"ondisk_log_size":1307,"up":51,"acting":51,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":49,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":475136,"kb_used_data":130004,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124466688,"statfs":{"total":1151940427776,"available":1151453888512,"internally_reserved":0,"allocated":133124096,"data_stored":128868745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":353154363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":79626240,"num_objects":1215,"num_object_clones":0,"num_object_copies":3645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1215,"num_write_kb":77760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.020340"},"pg_stats":[{"pgid":"2.f","version":"0'0","reported_seq":19,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:33.151492+0000","last_change":"2024-09-02T10:01:30.183309+0000","last_active":"2024-09-02T10:01:33.151492+0000","last_peered":"2024-09-02T10:01:33.151492+0000","last_clean":"2024-09-02T10:01:33.151492+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:01:33.151492+0000","last_undegraded":"2024-09-02T10:01:33.151492+0000","last_fullsized":"2024-09-02T10:01:33.151492+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:32.591873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"96'169","reported_seq":187,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:36.906731+0000","last_change":"2024-09-02T10:01:30.561460+0000","last_active":"2024-09-02T10:01:36.906731+0000","last_peered":"2024-09-02T10:01:36.906731+0000","last_clean":"2024-09-02T10:01:36.906731+0000","last_became_active":"2024-09-02T10:01:30.560995+0000","last_became_peered":"2024-09-02T10:01:30.560995+0000","last_unstale":"2024-09-02T10:01:36.906731+0000","last_undegraded":"2024-09-02T10:01:36.906731+0000","last_fullsized":"2024-09-02T10:01:36.906731+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:55.406791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":169,"num_object_clones":0,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":10816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-02T10:01:32.409561+0000","last_change":"2024-09-02T10:01:30.560154+0000","last_active":"2024-09-02T10:01:32.409561+0000","last_peered":"2024-09-02T10:01:32.409561+0000","last_clean":"2024-09-02T10:01:32.409561+0000","last_became_active":"2024-09-02T10:01:30.559704+0000","last_became_peered":"2024-09-02T10:01:30.559704+0000","last_unstale":"2024-09-02T10:01:32.409561+0000","last_undegraded":"2024-09-02T10:01:32.409561+0000","last_fullsized":"2024-09-02T10:01:32.409561+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:34.021980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"96'165","reported_seq":183,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:36.791378+0000","last_change":"2024-09-02T10:01:30.139707+0000","last_active":"2024-09-02T10:01:36.791378+0000","last_peered":"2024-09-02T10:01:36.791378+0000","last_clean":"2024-09-02T10:01:36.791378+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:01:36.791378+0000","last_undegraded":"2024-09-02T10:01:36.791378+0000","last_fullsized":"2024-09-02T10:01:36.791378+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:38.110684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10813440,"num_objects":165,"num_object_clones":0,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":10560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":140,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:33.490282+0000","last_change":"2024-09-02T10:00:29.279739+0000","last_active":"2024-09-02T10:01:33.490282+0000","last_peered":"2024-09-02T10:01:33.490282+0000","last_clean":"2024-09-02T10:01:33.490282+0000","last_became_active":"2024-09-02T10:00:28.639420+0000","last_became_peered":"2024-09-02T10:00:28.639420+0000","last_unstale":"2024-09-02T10:01:33.490282+0000","last_undegraded":"2024-09-02T10:01:33.490282+0000","last_fullsized":"2024-09-02T10:01:33.490282+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:00:29.279658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:00:29.279658+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:01:47.551954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"96'155","reported_seq":173,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:36.708681+0000","last_change":"2024-09-02T10:01:30.141172+0000","last_active":"2024-09-02T10:01:36.708681+0000","last_peered":"2024-09-02T10:01:36.708681+0000","last_clean":"2024-09-02T10:01:36.708681+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:01:36.708681+0000","last_undegraded":"2024-09-02T10:01:36.708681+0000","last_fullsized":"2024-09-02T10:01:36.708681+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:32.087935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10158080,"num_objects":155,"num_object_clones":0,"num_object_copies":465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":155,"num_write_kb":9920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"96'31","reported_seq":50,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:35.602427+0000","last_change":"2024-09-02T10:01:30.560296+0000","last_active":"2024-09-02T10:01:35.602427+0000","last_peered":"2024-09-02T10:01:35.602427+0000","last_clean":"2024-09-02T10:01:35.602427+0000","last_became_active":"2024-09-02T10:01:30.559903+0000","last_became_peered":"2024-09-02T10:01:30.559903+0000","last_unstale":"2024-09-02T10:01:35.602427+0000","last_undegraded":"2024-09-02T10:01:35.602427+0000","last_fullsized":"2024-09-02T10:01:35.602427+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:50.499241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":1984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"96'171","reported_seq":190,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:36.901602+0000","last_change":"2024-09-02T10:01:30.560301+0000","last_active":"2024-09-02T10:01:36.901602+0000","last_peered":"2024-09-02T10:01:36.901602+0000","last_clean":"2024-09-02T10:01:36.901602+0000","last_became_active":"2024-09-02T10:01:30.559997+0000","last_became_peered":"2024-09-02T10:01:30.559997+0000","last_unstale":"2024-09-02T10:01:36.901602+0000","last_undegraded":"2024-09-02T10:01:36.901602+0000","last_fullsized":"2024-09-02T10:01:36.901602+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:34.011670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"96'141","reported_seq":159,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:36.704524+0000","last_change":"2024-09-02T10:01:30.140642+0000","last_active":"2024-09-02T10:01:36.704524+0000","last_peered":"2024-09-02T10:01:36.704524+0000","last_clean":"2024-09-02T10:01:36.704524+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:01:36.704524+0000","last_undegraded":"2024-09-02T10:01:36.704524+0000","last_fullsized":"2024-09-02T10:01:36.704524+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:54.429959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9240576,"num_objects":141,"num_object_clones":0,"num_object_copies":423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":141,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-02T10:01:32.409519+0000","last_change":"2024-09-02T10:01:30.560172+0000","last_active":"2024-09-02T10:01:32.409519+0000","last_peered":"2024-09-02T10:01:32.409519+0000","last_clean":"2024-09-02T10:01:32.409519+0000","last_became_active":"2024-09-02T10:01:30.559685+0000","last_became_peered":"2024-09-02T10:01:30.559685+0000","last_unstale":"2024-09-02T10:01:32.409519+0000","last_undegraded":"2024-09-02T10:01:32.409519+0000","last_fullsized":"2024-09-02T10:01:32.409519+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:39.931059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:33.151482+0000","last_change":"2024-09-02T10:01:30.183497+0000","last_active":"2024-09-02T10:01:33.151482+0000","last_peered":"2024-09-02T10:01:33.151482+0000","last_clean":"2024-09-02T10:01:33.151482+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:01:33.151482+0000","last_undegraded":"2024-09-02T10:01:33.151482+0000","last_fullsized":"2024-09-02T10:01:33.151482+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:40.071109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-02T10:01:32.409311+0000","last_change":"2024-09-02T10:01:30.560649+0000","last_active":"2024-09-02T10:01:32.409311+0000","last_peered":"2024-09-02T10:01:32.409311+0000","last_clean":"2024-09-02T10:01:32.409311+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:01:32.409311+0000","last_undegraded":"2024-09-02T10:01:32.409311+0000","last_fullsized":"2024-09-02T10:01:32.409311+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:49.970361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"96'192","reported_seq":210,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:36.914945+0000","last_change":"2024-09-02T10:01:30.560449+0000","last_active":"2024-09-02T10:01:36.914945+0000","last_peered":"2024-09-02T10:01:36.914945+0000","last_clean":"2024-09-02T10:01:36.914945+0000","last_became_active":"2024-09-02T10:01:30.559962+0000","last_became_peered":"2024-09-02T10:01:30.559962+0000","last_unstale":"2024-09-02T10:01:36.914945+0000","last_undegraded":"2024-09-02T10:01:36.914945+0000","last_fullsized":"2024-09-02T10:01:36.914945+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:55.111495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"96'27","reported_seq":46,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:35.587315+0000","last_change":"2024-09-02T10:01:30.141564+0000","last_active":"2024-09-02T10:01:35.587315+0000","last_peered":"2024-09-02T10:01:35.587315+0000","last_clean":"2024-09-02T10:01:35.587315+0000","last_became_active":"2024-09-02T10:01:30.141263+0000","last_became_peered":"2024-09-02T10:01:30.141263+0000","last_unstale":"2024-09-02T10:01:35.587315+0000","last_undegraded":"2024-09-02T10:01:35.587315+0000","last_fullsized":"2024-09-02T10:01:35.587315+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:56.004849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-02T10:01:32.409305+0000","last_change":"2024-09-02T10:01:30.560354+0000","last_active":"2024-09-02T10:01:32.409305+0000","last_peered":"2024-09-02T10:01:32.409305+0000","last_clean":"2024-09-02T10:01:32.409305+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:01:32.409305+0000","last_undegraded":"2024-09-02T10:01:32.409305+0000","last_fullsized":"2024-09-02T10:01:32.409305+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:42.037207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.a","version":"96'164","reported_seq":182,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:36.814006+0000","last_change":"2024-09-02T10:01:30.560175+0000","last_active":"2024-09-02T10:01:36.814006+0000","last_peered":"2024-09-02T10:01:36.814006+0000","last_clean":"2024-09-02T10:01:36.814006+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:01:36.814006+0000","last_undegraded":"2024-09-02T10:01:36.814006+0000","last_fullsized":"2024-09-02T10:01:36.814006+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:38.707848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10747904,"num_objects":164,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":164,"num_write_kb":10496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":19,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-09-02T10:01:33.151431+0000","last_change":"2024-09-02T10:01:30.140619+0000","last_active":"2024-09-02T10:01:33.151431+0000","last_peered":"2024-09-02T10:01:33.151431+0000","last_clean":"2024-09-02T10:01:33.151431+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:01:33.151431+0000","last_undegraded":"2024-09-02T10:01:33.151431+0000","last_fullsized":"2024-09-02T10:01:33.151431+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:39.195715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":79626240,"num_objects":1215,"num_object_clones":0,"num_object_copies":3645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1215,"num_write_kb":77760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":190185472,"data_stored":190185472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1215,"ondisk_log_size":1215,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":9}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383698,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61764,"kb_used_data":30580,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683388,"statfs":{"total":95995035648,"available":95931789312,"internally_reserved":0,"allocated":31313920,"data_stored":30954052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612631,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46716,"kb_used_data":15532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93698436,"statfs":{"total":95995035648,"available":95947198464,"internally_reserved":0,"allocated":15904768,"data_stored":15548472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841563,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27752,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":741376,"data_stored":393945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070496,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27780,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717372,"statfs":{"total":95995035648,"available":95966588928,"internally_reserved":0,"allocated":765952,"data_stored":409656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299428,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28304,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716848,"statfs":{"total":95995035648,"available":95966052352,"internally_reserved":0,"allocated":1298432,"data_stored":938584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528361,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44676,"kb_used_data":13492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93700476,"statfs":{"total":95995035648,"available":95949287424,"internally_reserved":0,"allocated":13815808,"data_stored":13455960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724590,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40444,"kb_used_data":9260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93704708,"statfs":{"total":95995035648,"available":95953620992,"internally_reserved":0,"allocated":9482240,"data_stored":9125944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953523,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":765952,"data_stored":409656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27752,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717400,"statfs":{"total":95995035648,"available":95966617600,"internally_reserved":0,"allocated":741376,"data_stored":393945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62460,"kb_used_data":31276,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93682692,"statfs":{"total":95995035648,"available":95931076608,"internally_reserved":0,"allocated":32026624,"data_stored":31670328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27748,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":729088,"data_stored":385939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":51964,"kb_used_data":24940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93693188,"statfs":{"total":95995035648,"available":95941824512,"internally_reserved":0,"allocated":25538560,"data_stored":25182264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36044800,"data_stored":36044800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45678592,"data_stored":45678592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19464192,"data_stored":19464192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23265280,"data_stored":23265280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":11010048,"data_stored":11010048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":23003136,"data_stored":23003136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":31719424,"data_stored":31719424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:01:40.046 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-02T10:01:40.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-02T10:01:40.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:40.137 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:40.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:40.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:40.165 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:01:40.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:40.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:40.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:40.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:40.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:40.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:40.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:40.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:40.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 5 16 8896 8880 110.991 75.625 0.00298129 0.00899111 2024-09-02T10:01:40.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:40 smithi151 ceph-mon[16639]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:40.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:40 smithi151 ceph-mon[16639]: pgmap v297: 17 pgs: 17 active+clean; 144 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 332 op/s 2024-09-02T10:01:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/110220618' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:01:40.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:40.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:40 smithi187 ceph-mon[20072]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:40 smithi187 ceph-mon[20072]: pgmap v297: 17 pgs: 17 active+clean; 144 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 332 op/s 2024-09-02T10:01:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/110220618' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:01:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:40.521 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:40.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:40.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:40 smithi162 ceph-mon[20279]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:40.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:40 smithi162 ceph-mon[20279]: pgmap v297: 17 pgs: 17 active+clean; 144 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 21 MiB/s wr, 332 op/s 2024-09-02T10:01:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/110220618' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:01:40.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:40.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:40.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:40.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:40.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:40.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:40.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:40.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:40.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:40.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:40.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:41.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:41.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:41.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:41.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:41.143 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:41.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:41.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:41.273 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:41.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:41.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:41.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:41.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:41.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:41.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 6 16 10159 10143 105.648 78.9375 0.00246301 0.00945623 2024-09-02T10:01:41.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:41 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:41.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/110220618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:01:41.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:41 smithi151 ceph-mon[16639]: osdmap e98: 12 total, 12 up, 12 in 2024-09-02T10:01:41.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:41 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/110220618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:01:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:41 smithi187 ceph-mon[20072]: osdmap e98: 12 total, 12 up, 12 in 2024-09-02T10:01:41.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:41.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:41 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/110220618' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:01:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:41 smithi162 ceph-mon[20279]: osdmap e98: 12 total, 12 up, 12 in 2024-09-02T10:01:41.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:41.654 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:41.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:41.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:41.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:41.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:41.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:41.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:41.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:41.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:41.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:41.969 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:01:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:01:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:10683 2.9 2:977c1210:::benchmark_data_smithi187_45882_object10682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:01:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:41.958226+0000", 2024-09-02T10:01:41.969 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0094182759999999997, 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0094603910000000003, 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 10683 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:41.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.958226+0000", 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.958226+0000", 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.958227+0000", 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4499999999999996e-07 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.958289+0000", 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1894000000000001e-05 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.958300+0000", 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0757000000000001e-05 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.958305+0000", 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1630000000000004e-06 2024-09-02T10:01:41.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.958358+0000", 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3387999999999998e-05 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.958478+0000", 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011985900000000001 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.958619+0000", 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000140743 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.959352+0000", 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073344800000000004 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.959527+0000", 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000174413 2024-09-02T10:01:41.974 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:10697 2.6 2:65e82eda:::benchmark_data_smithi187_45882_object10696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:41.966360+0000", 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0012851659999999999, 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014260640000000001, 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:01:41.975 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 10697 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.966360+0000", 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.966360+0000", 2024-09-02T10:01:41.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.966361+0000", 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4959999999999999e-06 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.966404+0000", 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3059000000000002e-05 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.966418+0000", 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.438e-05 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.966425+0000", 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9439999999999999e-06 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.966473+0000", 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7676999999999998e-05 2024-09-02T10:01:41.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.966556+0000", 2024-09-02T10:01:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3057000000000001e-05 2024-09-02T10:01:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.966643+0000", 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7101000000000005e-05 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:10699 2.6 2:6c4e7560:::benchmark_data_smithi187_45882_object10698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:41.967007+0000", 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00063794999999999998, 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00084141299999999995, 2024-09-02T10:01:41.981 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 10699 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967007+0000", 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967007+0000", 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967008+0000", 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.001e-06 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967091+0000", 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3610999999999996e-05 2024-09-02T10:01:41.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967105+0000", 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3577999999999999e-05 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967113+0000", 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0039999999999999e-06 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:41.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967204+0000", 2024-09-02T10:01:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1415999999999993e-05 2024-09-02T10:01:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967313+0000", 2024-09-02T10:01:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108311 2024-09-02T10:01:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:01:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967456+0000", 2024-09-02T10:01:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000143256 2024-09-02T10:01:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:10697 2.6 e98/92)", 2024-09-02T10:01:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:41.967391+0000", 2024-09-02T10:01:41.987 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00025396799999999998, 2024-09-02T10:01:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051150800000000004, 2024-09-02T10:01:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:01:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967391+0000", 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967391+0000", 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967391+0000", 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6700000000000003e-07 2024-09-02T10:01:41.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967395+0000", 2024-09-02T10:01:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0879999999999997e-06 2024-09-02T10:01:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967430+0000", 2024-09-02T10:01:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4115000000000001e-05 2024-09-02T10:01:41.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967436+0000", 2024-09-02T10:01:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0399999999999998e-06 2024-09-02T10:01:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:41.991 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:10697 2.6 e98/92)", 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:41.967548+0000", 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.7104999999999999e-05, 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038833599999999998, 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:41.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967548+0000", 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967548+0000", 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967548+0000", 2024-09-02T10:01:41.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4600000000000005e-07 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967552+0000", 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.534e-06 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967580+0000", 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8138000000000001e-05 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.967586+0000", 2024-09-02T10:01:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0839999999999997e-06 2024-09-02T10:01:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:10606 2.4 2:26a0e741:::benchmark_data_smithi187_45882_object10605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:01:41.995 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:41.906656+0000", 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.060988422, 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.061322094000000001, 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 10606 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.906656+0000", 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.906656+0000", 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.906658+0000", 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1100000000000001e-06 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.906703+0000", 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4205000000000001e-05 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.906725+0000", 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2847999999999999e-05 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.906732+0000", 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.906792+0000", 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9762999999999997e-05 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.906952+0000", 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000160426 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:41.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.907120+0000", 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000167753 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.907833+0000", 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00071311200000000001 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.917087+0000", 2024-09-02T10:01:42.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0092536849999999993 2024-09-02T10:01:42.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:42.001 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:42.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:42.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:42.001 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:01:42.001 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 6 2024-09-02T10:01:42.001 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:01:42.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:42.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:42.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:42.136 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:42.163 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:42.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:42.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:42.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:42.288 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:42.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:42.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:42.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:42.412 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:42.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:42.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 7 16 11359 11343 101.269 75 0.00172063 0.00980619 2024-09-02T10:01:42.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:42 smithi151 ceph-mon[16639]: pgmap v299: 17 pgs: 17 active+clean; 294 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 592 op/s 2024-09-02T10:01:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:42 smithi187 ceph-mon[20072]: pgmap v299: 17 pgs: 17 active+clean; 294 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 592 op/s 2024-09-02T10:01:42.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:42.537 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:42.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:42.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:42 smithi162 ceph-mon[20279]: pgmap v299: 17 pgs: 17 active+clean; 294 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 592 op/s 2024-09-02T10:01:42.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:42.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:42.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:42.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:42.789 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:42.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:42.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:42.913 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:42.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:42.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:01:43.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:43.027 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:43.040 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:43.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:43.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:43.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:43.172 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:43.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:43.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:43.299 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:43.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:43.401 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:43.424 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:43.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:43.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:43.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 8 16 12749 12733 99.4685 86.875 0.0129419 0.0100309 2024-09-02T10:01:43.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:43.552 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:43.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:43.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:43.676 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:43.678 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:43.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:01:43.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:43.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:43.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:43.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:43.924 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:43.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:44.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:44.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:44.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:44.146 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:01:44.146 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:01:44.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:44.174 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:44.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:44.178 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:01:44.179 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:01:44.202 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:44.202 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:01:44.203 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:01:44.224 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:01:44.225 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:01:44.246 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:01:44.246 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:01:44.267 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:01:44.267 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:01:44.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:44.288 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:01:44.289 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:01:44.289 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:44 smithi151 ceph-mon[16639]: pgmap v300: 17 pgs: 17 active+clean; 623 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-02T10:01:44.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3458742229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:44.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:44.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:44.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:01:44.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:01:44.331 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:01:44.331 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:01:44.359 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:01:44.360 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:01:44.386 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:01:44.386 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:01:44.400 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:44.408 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:01:44.409 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:01:44.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:44.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:44.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:44.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 9 16 13753 13737 95.3881 62.75 0.00721162 0.010471 2024-09-02T10:01:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:44 smithi187 ceph-mon[20072]: pgmap v300: 17 pgs: 17 active+clean; 623 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-02T10:01:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3458742229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:44.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:44.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:44.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:44.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:44.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:44 smithi162 ceph-mon[20279]: pgmap v300: 17 pgs: 17 active+clean; 623 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-02T10:01:44.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3458742229' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:44.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:44.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:44.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:44.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:44.806 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:44.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:44.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:44.932 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:44.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:45.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:45.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:45.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:45.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:45.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:45.176 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pg_num to 11 2024-09-02T10:01:45.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:45.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:45.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:45.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:45.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:45.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:45.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 10 16 15103 15087 94.2861 84.375 0.00326493 0.010594 2024-09-02T10:01:45.434 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:45.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:45.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3646851954' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-02T10:01:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3646851954' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-02T10:01:45.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:45.561 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:45.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:45.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3646851954' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"} : dispatch 2024-09-02T10:01:45.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:45.685 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:45.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:45.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:45.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:45.809 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:45.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:45.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:45.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:45.935 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:45.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:46.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:46.064 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:46.065 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:46.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:46.194 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:46.195 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:46.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:46.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:46.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:46.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:46 smithi151 ceph-mon[16639]: pgmap v301: 17 pgs: 17 active+clean; 699 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-02T10:01:46.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3646851954' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-02T10:01:46.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:46 smithi151 ceph-mon[16639]: osdmap e99: 12 total, 12 up, 12 in 2024-09-02T10:01:46.422 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:46.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 11 16 16372 16356 92.9243 79.3125 0.18426 0.0105928 2024-09-02T10:01:46.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:46.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:46.444 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:46 smithi187 ceph-mon[20072]: pgmap v301: 17 pgs: 17 active+clean; 699 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-02T10:01:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3646851954' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-02T10:01:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:46 smithi187 ceph-mon[20072]: osdmap e99: 12 total, 12 up, 12 in 2024-09-02T10:01:46.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:46.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:46.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:46.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:46 smithi162 ceph-mon[20279]: pgmap v301: 17 pgs: 17 active+clean; 699 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.24k op/s 2024-09-02T10:01:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3646851954' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-02T10:01:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:46 smithi162 ceph-mon[20279]: osdmap e99: 12 total, 12 up, 12 in 2024-09-02T10:01:46.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:46.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:46.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:46.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:46.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:46.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:46.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:46.829 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:01:46.829 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:01:46.927 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:46.954 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:46.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:47.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:47.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:47.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:47.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:47.210 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:47.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:47.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:01:47.311 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:47.332 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:47.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:47.333 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:47.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 12 16 17764 17748 92.43 87 0.0862001 0.0107719 2024-09-02T10:01:47.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:47.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:47.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:47 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:47.457 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:47.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:47 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:47.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:47.580 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:47.581 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:47.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:47.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:47 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:47.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:47.705 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:47.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:47.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:01:47.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:47.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:47.829 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:47.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:47.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:47.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:47.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:48.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:48.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:48.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:48.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:48.189 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:01:48.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:48.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:48.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:48.335 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:48.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:48.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:48.433 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 13 16 19066 19050 91.579 81.375 0.00156573 0.01091 2024-09-02T10:01:48.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:48 smithi151 ceph-mon[16639]: pgmap v303: 17 pgs: 17 active+clean; 928 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-02T10:01:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:48 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-02T10:01:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:48 smithi151 ceph-mon[16639]: osdmap e100: 12 total, 12 up, 12 in 2024-09-02T10:01:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1293515205' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:01:48.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:48.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:48 smithi187 ceph-mon[20072]: pgmap v303: 17 pgs: 17 active+clean; 928 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-02T10:01:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:48 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-02T10:01:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:48 smithi187 ceph-mon[20072]: osdmap e100: 12 total, 12 up, 12 in 2024-09-02T10:01:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1293515205' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:01:48.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:48.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:48.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:48 smithi162 ceph-mon[20279]: pgmap v303: 17 pgs: 17 active+clean; 928 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-02T10:01:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:48 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-02T10:01:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:48 smithi162 ceph-mon[20279]: osdmap e100: 12 total, 12 up, 12 in 2024-09-02T10:01:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1293515205' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:01:48.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:48.714 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:48.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:48.816 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:48.839 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:48.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:48.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:48.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:48.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:49.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:49.094 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:49.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:49.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:49.220 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:49.221 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:49.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:49.352 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:49.367 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:49.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:49.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:49.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:49.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 14 16 20430 20414 91.1263 85.25 0.0249031 0.0109117 2024-09-02T10:01:49.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:49 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:01:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:49 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:01:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1293515205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:01:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:49 smithi151 ceph-mon[16639]: osdmap e101: 12 total, 12 up, 12 in 2024-09-02T10:01:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:49 smithi151 ceph-mon[16639]: osdmap e102: 12 total, 12 up, 12 in 2024-09-02T10:01:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:49 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:01:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:49 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:01:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:49 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:49.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:49.502 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:49.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:49 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:01:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:49 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:01:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1293515205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:01:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:49 smithi187 ceph-mon[20072]: osdmap e101: 12 total, 12 up, 12 in 2024-09-02T10:01:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:49 smithi187 ceph-mon[20072]: osdmap e102: 12 total, 12 up, 12 in 2024-09-02T10:01:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:49 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:01:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:49 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:01:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:49 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:49 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:01:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:49 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:01:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1293515205' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:01:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:49 smithi162 ceph-mon[20279]: osdmap e101: 12 total, 12 up, 12 in 2024-09-02T10:01:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:49 smithi162 ceph-mon[20279]: osdmap e102: 12 total, 12 up, 12 in 2024-09-02T10:01:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:49 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:01:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:49 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:01:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:49 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"} : dispatch 2024-09-02T10:01:49.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:49.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:49.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:49.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:49.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:49.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:49.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:49.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:49.880 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:49.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:49.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:50.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:50.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:50.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:50.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:50.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:50.182 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:01:50.183 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:01:50.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:50.264 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:50.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:50.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:50 smithi151 ceph-mon[16639]: pgmap v307: 27 pgs: 10 unknown, 17 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:01:50.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:50.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:50.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:50 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-02T10:01:50.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:50 smithi151 ceph-mon[16639]: osdmap e103: 12 total, 12 up, 12 in 2024-09-02T10:01:50.365 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:50.388 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:01:50.388 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:01:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:50.389 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:50.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:50.410 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:01:50.410 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:01:50.431 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:01:50.431 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:01:50.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 15 16 21438 21422 89.2509 63 0.00916281 0.0111863 2024-09-02T10:01:50.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:01:50.458 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:01:50.482 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:01:50.483 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:01:50.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:50.504 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:01:50.504 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:01:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:50 smithi187 ceph-mon[20072]: pgmap v307: 27 pgs: 10 unknown, 17 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:01:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:50.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:50.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:50 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-02T10:01:50.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:50 smithi187 ceph-mon[20072]: osdmap e103: 12 total, 12 up, 12 in 2024-09-02T10:01:50.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:50.513 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:50.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:50.528 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:01:50.529 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:01:50.549 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:01:50.549 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:01:50.571 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:01:50.571 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:01:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:50 smithi162 ceph-mon[20279]: pgmap v307: 27 pgs: 10 unknown, 17 active+clean; 1.0 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:01:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:50 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-02T10:01:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:50 smithi162 ceph-mon[20279]: osdmap e103: 12 total, 12 up, 12 in 2024-09-02T10:01:50.597 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:01:50.598 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:01:50.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:50.624 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:01:50.624 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:01:50.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:50.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:50.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:01:50.648 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:01:50.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:50.769 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:50.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:50.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:50.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:50.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:50.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:50.997 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:51.025 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:51.026 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:51.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:51.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:51.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:51.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:01:51.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:51.275 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:51.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:51.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:01:51.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:51.399 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:51.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:51.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 16 16 22527 22511 87.9263 68.0625 0.0117965 0.011358 2024-09-02T10:01:51.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:51.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:51.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/965080082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:51.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:51 smithi151 ceph-mon[16639]: osdmap e104: 12 total, 12 up, 12 in 2024-09-02T10:01:51.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:51 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:01:51.502 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:51.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/965080082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:51 smithi187 ceph-mon[20072]: osdmap e104: 12 total, 12 up, 12 in 2024-09-02T10:01:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:51 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:01:51.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:51.526 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:51.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:51.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:51.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/965080082' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:51 smithi162 ceph-mon[20279]: osdmap e104: 12 total, 12 up, 12 in 2024-09-02T10:01:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:51 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:01:51.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:51.649 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:51.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:51.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:51.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:51.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:51.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:51.879 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:51.908 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:51.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:52.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:52.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:52.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:52.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:52.174 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:52.175 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:52.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:52.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:52.183 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:52.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:52.302 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:52.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:52.306 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: pgmap v310: 27 pgs: 2 peering, 8 unknown, 17 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 879 op/s 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:01:52.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:52 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:01:52.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:52.428 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:52.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:52.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 17 16 24199 24183 88.9007 104.5 0.0358563 0.0112409 2024-09-02T10:01:52.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:01:52.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:01:52.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:01:52.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:01:52.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:01:52.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:01:52.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: pgmap v310: 27 pgs: 2 peering, 8 unknown, 17 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 879 op/s 2024-09-02T10:01:52.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:01:52.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:01:52.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:01:52.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:01:52.497 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:52 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:01:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:01:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:01:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:01:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:01:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:01:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:01:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: pgmap v310: 27 pgs: 2 peering, 8 unknown, 17 active+clean; 1.1 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 879 op/s 2024-09-02T10:01:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:01:52.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:01:52.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:01:52.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:01:52.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:52 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:01:52.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:52.553 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:52.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:52.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:52.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:52.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:52.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:52.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:52.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:52.810 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:52.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:52.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:52.934 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:52.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:52.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:53.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:53.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:53.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:53.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:53.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:53.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:53.262 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:01:53.263 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:01:53.263 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-02T10:01:53.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:01:53.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:53.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:53.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:53.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:53.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 18 16 25462 25446 88.3469 78.9375 0.0145509 0.0113099 2024-09-02T10:01:53.456 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:53.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:53.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:53 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:01:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:53 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:01:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:53 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:01:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:53 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:01:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:53 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:01:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:53 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:01:53.517 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:53 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:01:53.517 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:53 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:01:53.517 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:53 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:01:53.517 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:53 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:01:53.517 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:53 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:01:53.517 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:53 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:01:53.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:53.559 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:01:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:01:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:25653 2.e 2:74761604:::benchmark_data_smithi187_45882_object25652:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-02T10:01:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:53.555068+0000", 2024-09-02T10:01:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0014299479999999999, 2024-09-02T10:01:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014607070000000001, 2024-09-02T10:01:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 25653 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.555068+0000", 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.555068+0000", 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.555070+0000", 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5850000000000001e-06 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.555111+0000", 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1010999999999997e-05 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.555130+0000", 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9258000000000001e-05 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.555139+0000", 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.183e-06 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.555206+0000", 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7018000000000001e-05 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.555364+0000", 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015813000000000001 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.555519+0000", 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015492699999999999 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:25654 2.e 2:7a53e2e9:::benchmark_data_smithi187_45882_object25653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:53.556015+0000", 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.000482795, 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00058961599999999995, 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 25654 2024-09-02T10:01:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556015+0000", 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556015+0000", 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556016+0000", 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3319999999999999e-06 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556051+0000", 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4082999999999999e-05 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556068+0000", 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7187e-05 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556075+0000", 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3769999999999998e-06 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556574+0000", 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049921099999999995 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:25656 2.1 2:8c90c31a:::benchmark_data_smithi187_45882_object25655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:53.556448+0000", 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.9447999999999998e-05, 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000214842, 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 25656 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:01:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556448+0000", 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556448+0000", 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556450+0000", 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1689999999999999e-06 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556523+0000", 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3763000000000005e-05 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556536+0000", 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.307e-05 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556576+0000", 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9416999999999999e-05 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:01:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:01:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:01:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:53.556628+0000", 2024-09-02T10:01:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1947999999999998e-05 2024-09-02T10:01:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:01:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:01:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:01:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 3 2024-09-02T10:01:53.571 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:01:53.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:53 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:01:53.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:53 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:01:53.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:53 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:01:53.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:53 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:01:53.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:53 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:01:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:53 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:01:53.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:53.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:53.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:53.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:53.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:53.750 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:01:53.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:53.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:53.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:53.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:53.995 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:53.996 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:54.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:01:54.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:54.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:54.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:54.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:54.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:54.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:54.345 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:54.370 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:54.371 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:54.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 19 16 26842 26826 88.2362 86.25 0.00231175 0.0113141 2024-09-02T10:01:54.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:54 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:01:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:54 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:01:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:54 smithi151 ceph-mon[16639]: pgmap v311: 27 pgs: 2 peering, 25 active+clean; 1.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.26k op/s 2024-09-02T10:01:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1213822397' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:01:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:54 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:01:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:54 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:54 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:01:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:54 smithi151 ceph-mon[16639]: osdmap e105: 12 total, 12 up, 12 in 2024-09-02T10:01:54.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:54.493 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:54.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:54 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:01:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:54 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:01:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:54 smithi187 ceph-mon[20072]: pgmap v311: 27 pgs: 2 peering, 25 active+clean; 1.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.26k op/s 2024-09-02T10:01:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1213822397' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:01:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:54 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:01:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:54 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:54 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:01:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:54 smithi187 ceph-mon[20072]: osdmap e105: 12 total, 12 up, 12 in 2024-09-02T10:01:54.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:54 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:01:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:54 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:01:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:54 smithi162 ceph-mon[20279]: pgmap v311: 27 pgs: 2 peering, 25 active+clean; 1.4 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.26k op/s 2024-09-02T10:01:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1213822397' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:01:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:54 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:01:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:54 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:01:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:54 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:01:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:54 smithi162 ceph-mon[20279]: osdmap e105: 12 total, 12 up, 12 in 2024-09-02T10:01:54.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:54.620 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:54.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:54.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:54.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:54.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:54.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:54.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:54.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:54.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:54.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:54.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:54.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:55.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:55.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:01:55.109 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:55.117 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:55.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:55.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:55.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:55.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:55.246 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:55.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:55.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:55.372 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:55.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:55.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:55.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:01:55.434419+0000 min lat: 0.00101783 max lat: 0.3086 avg lat: 0.0114314 2024-09-02T10:01:55.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:01:55.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 20 16 27975 27959 87.3648 70.8125 0.00249558 0.0114314 2024-09-02T10:01:55.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:55.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:55.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:55.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:55.626 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:55.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:55.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:01:55.749 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:55.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:01:55.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:55.881 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:01:55.881 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:01:55.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:55.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:55.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:55.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:56.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:56.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:56.111 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:56.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:56.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:56.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:56.264 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:56.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:56.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:56.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:56.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:56.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:56.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 21 16 29059 29043 86.4304 67.75 0.0139281 0.0115633 2024-09-02T10:01:56.475 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:56 smithi151 ceph-mon[16639]: pgmap v313: 27 pgs: 2 peering, 25 active+clean; 1.5 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-02T10:01:56.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:56.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:01:56.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:01:56.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:56 smithi187 ceph-mon[20072]: pgmap v313: 27 pgs: 2 peering, 25 active+clean; 1.5 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-02T10:01:56.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:56.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:56.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:56.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:01:56.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:01:56.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:01:56.558 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:01:56.586 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:01:56.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:01:56.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:56 smithi162 ceph-mon[20279]: pgmap v313: 27 pgs: 2 peering, 25 active+clean; 1.5 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-02T10:01:56.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:01:56.614 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:01:56.614 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:01:56.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:01:56.642 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:01:56.642 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:01:56.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:56.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:01:56.675 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:01:56.675 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:01:56.702 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:01:56.703 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:01:56.741 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:01:56.741 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:01:56.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:56.777 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:01:56.777 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:01:56.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:56.801 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:56.811 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:01:56.811 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:01:56.841 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:01:56.841 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:01:56.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:56.926 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:56.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:56.990 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-02T10:01:55.063402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1564756152,"num_objects":23867,"num_object_clones":0,"num_object_copies":71601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23867,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":23909,"num_write_kb":1528078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":384674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24876,"ondisk_log_size":24876,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":87,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":4896028,"kb_used_data":4480164,"kb_used_omap":18,"kb_used_meta":415597,"kb_avail":1120045796,"statfs":{"total":1151940427776,"available":1146926895104,"internally_reserved":0,"allocated":4587687936,"data_stored":4582977476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":425571643},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":488130200,"num_objects":7446,"num_object_clones":0,"num_object_copies":22338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7446,"num_whiteouts":0,"num_read":-48,"num_read_kb":-40,"num_write":7367,"num_write_kb":475112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-1202110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001732"},"pg_stats":[{"pgid":"2.f","version":"105'1635","reported_seq":1672,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-02T10:01:54.041475+0000","last_change":"2024-09-02T10:01:30.183309+0000","last_active":"2024-09-02T10:01:54.041475+0000","last_peered":"2024-09-02T10:01:54.041475+0000","last_clean":"2024-09-02T10:01:54.041475+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:01:54.041475+0000","last_undegraded":"2024-09-02T10:01:54.041475+0000","last_fullsized":"2024-09-02T10:01:54.041475+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1635,"log_dups_size":0,"ondisk_log_size":1635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:32.591873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107151360,"num_objects":1635,"num_object_clones":0,"num_object_copies":4905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1635,"num_write_kb":104640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"104'1474","reported_seq":1508,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:51.905070+0000","last_change":"2024-09-02T10:01:30.561460+0000","last_active":"2024-09-02T10:01:51.905070+0000","last_peered":"2024-09-02T10:01:51.905070+0000","last_clean":"2024-09-02T10:01:51.905070+0000","last_became_active":"2024-09-02T10:01:30.560995+0000","last_became_peered":"2024-09-02T10:01:30.560995+0000","last_unstale":"2024-09-02T10:01:51.905070+0000","last_undegraded":"2024-09-02T10:01:51.905070+0000","last_fullsized":"2024-09-02T10:01:51.905070+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1474,"log_dups_size":0,"ondisk_log_size":1474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:55.406791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96600064,"num_objects":1474,"num_object_clones":0,"num_object_copies":4422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1474,"num_write_kb":94336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"104'1507","reported_seq":1540,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:52.853943+0000","last_change":"2024-09-02T10:01:30.560154+0000","last_active":"2024-09-02T10:01:52.853943+0000","last_peered":"2024-09-02T10:01:52.853943+0000","last_clean":"2024-09-02T10:01:52.853943+0000","last_became_active":"2024-09-02T10:01:30.559704+0000","last_became_peered":"2024-09-02T10:01:30.559704+0000","last_unstale":"2024-09-02T10:01:52.853943+0000","last_undegraded":"2024-09-02T10:01:52.853943+0000","last_fullsized":"2024-09-02T10:01:52.853943+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:46.256601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98762752,"num_objects":1507,"num_object_clones":0,"num_object_copies":4521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1507,"num_write_kb":96448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"104'1454","reported_seq":1488,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:51.811685+0000","last_change":"2024-09-02T10:01:30.139707+0000","last_active":"2024-09-02T10:01:51.811685+0000","last_peered":"2024-09-02T10:01:51.811685+0000","last_clean":"2024-09-02T10:01:51.811685+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:01:51.811685+0000","last_undegraded":"2024-09-02T10:01:51.811685+0000","last_fullsized":"2024-09-02T10:01:51.811685+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1454,"log_dups_size":0,"ondisk_log_size":1454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:38.110684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95289344,"num_objects":1454,"num_object_clones":0,"num_object_copies":4362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1454,"num_write_kb":93056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":172,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-02T10:01:49.753826+0000","last_change":"2024-09-02T10:01:48.974659+0000","last_active":"2024-09-02T10:01:49.753826+0000","last_peered":"2024-09-02T10:01:49.753826+0000","last_clean":"2024-09-02T10:01:49.753826+0000","last_became_active":"2024-09-02T10:01:48.206613+0000","last_became_peered":"2024-09-02T10:01:48.206613+0000","last_unstale":"2024-09-02T10:01:49.753826+0000","last_undegraded":"2024-09-02T10:01:49.753826+0000","last_fullsized":"2024-09-02T10:01:49.753826+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:48.974614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:48.974614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:13.084748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"104'1435","reported_seq":1469,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:51.725512+0000","last_change":"2024-09-02T10:01:30.141172+0000","last_active":"2024-09-02T10:01:51.725512+0000","last_peered":"2024-09-02T10:01:51.725512+0000","last_clean":"2024-09-02T10:01:51.725512+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:01:51.725512+0000","last_undegraded":"2024-09-02T10:01:51.725512+0000","last_fullsized":"2024-09-02T10:01:51.725512+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1435,"log_dups_size":0,"ondisk_log_size":1435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:32.087935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94044160,"num_objects":1435,"num_object_clones":0,"num_object_copies":4305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1435,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1435,"num_write_kb":91840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:50.950219+0000","last_change":"2024-09-02T10:01:50.950127+0000","last_active":"2024-09-02T10:01:50.950219+0000","last_peered":"2024-09-02T10:01:50.950219+0000","last_clean":"2024-09-02T10:01:50.950219+0000","last_became_active":"2024-09-02T10:01:50.764818+0000","last_became_peered":"2024-09-02T10:01:50.764818+0000","last_unstale":"2024-09-02T10:01:50.950219+0000","last_undegraded":"2024-09-02T10:01:50.950219+0000","last_fullsized":"2024-09-02T10:01:50.950219+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:50.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:50.950013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:59.037659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"103'1339","reported_seq":1372,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-02T10:01:50.602890+0000","last_change":"2024-09-02T10:01:30.560296+0000","last_active":"2024-09-02T10:01:50.602890+0000","last_peered":"2024-09-02T10:01:50.602890+0000","last_clean":"2024-09-02T10:01:50.602890+0000","last_became_active":"2024-09-02T10:01:30.559903+0000","last_became_peered":"2024-09-02T10:01:30.559903+0000","last_unstale":"2024-09-02T10:01:50.602890+0000","last_undegraded":"2024-09-02T10:01:50.602890+0000","last_fullsized":"2024-09-02T10:01:50.602890+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1339,"log_dups_size":0,"ondisk_log_size":1339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:50.499241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87752704,"num_objects":1339,"num_object_clones":0,"num_object_copies":4017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1339,"num_write_kb":85696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":162,"reported_epoch":103,"state":"peering","last_fresh":"2024-09-02T10:01:49.754932+0000","last_change":"2024-09-02T10:01:49.754932+0000","last_active":"2024-09-02T10:01:49.754932+0000","last_peered":"2024-09-02T10:01:48.747095+0000","last_clean":"2024-09-02T10:01:48.747095+0000","last_became_active":"2024-09-02T10:01:48.216460+0000","last_became_peered":"2024-09-02T10:01:48.216460+0000","last_unstale":"2024-09-02T10:01:49.754932+0000","last_undegraded":"2024-09-02T10:01:49.754932+0000","last_fullsized":"2024-09-02T10:01:49.754932+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:00:29.279658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:00:29.279658+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":12,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"104'1480","reported_seq":1514,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:51.896278+0000","last_change":"2024-09-02T10:01:30.560301+0000","last_active":"2024-09-02T10:01:51.896278+0000","last_peered":"2024-09-02T10:01:51.896278+0000","last_clean":"2024-09-02T10:01:51.896278+0000","last_became_active":"2024-09-02T10:01:30.559997+0000","last_became_peered":"2024-09-02T10:01:30.559997+0000","last_unstale":"2024-09-02T10:01:51.896278+0000","last_undegraded":"2024-09-02T10:01:51.896278+0000","last_fullsized":"2024-09-02T10:01:51.896278+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1480,"log_dups_size":0,"ondisk_log_size":1480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:34.011670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96993280,"num_objects":1480,"num_object_clones":0,"num_object_copies":4440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1480,"num_write_kb":94720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:50.898143+0000","last_change":"2024-09-02T10:01:50.898052+0000","last_active":"2024-09-02T10:01:50.898143+0000","last_peered":"2024-09-02T10:01:50.898143+0000","last_clean":"2024-09-02T10:01:50.898143+0000","last_became_active":"2024-09-02T10:01:50.765441+0000","last_became_peered":"2024-09-02T10:01:50.765441+0000","last_unstale":"2024-09-02T10:01:50.898143+0000","last_undegraded":"2024-09-02T10:01:50.898143+0000","last_fullsized":"2024-09-02T10:01:50.898143+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:50.897996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:50.897996+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:20.788049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"104'1516","reported_seq":1550,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:51.720197+0000","last_change":"2024-09-02T10:01:30.140642+0000","last_active":"2024-09-02T10:01:51.720197+0000","last_peered":"2024-09-02T10:01:51.720197+0000","last_clean":"2024-09-02T10:01:51.720197+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:01:51.720197+0000","last_undegraded":"2024-09-02T10:01:51.720197+0000","last_fullsized":"2024-09-02T10:01:51.720197+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1516,"log_dups_size":0,"ondisk_log_size":1516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:54.429959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99352576,"num_objects":1516,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1516,"num_write_kb":97024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":44,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:51.173172+0000","last_change":"2024-09-02T10:01:51.173117+0000","last_active":"2024-09-02T10:01:51.173172+0000","last_peered":"2024-09-02T10:01:51.173172+0000","last_clean":"2024-09-02T10:01:51.173172+0000","last_became_active":"2024-09-02T10:01:50.778902+0000","last_became_peered":"2024-09-02T10:01:50.778902+0000","last_unstale":"2024-09-02T10:01:51.173172+0000","last_undegraded":"2024-09-02T10:01:51.173172+0000","last_fullsized":"2024-09-02T10:01:51.173172+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.173075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.173075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:04.633166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"104'1516","reported_seq":1550,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:52.509717+0000","last_change":"2024-09-02T10:01:30.560172+0000","last_active":"2024-09-02T10:01:52.509717+0000","last_peered":"2024-09-02T10:01:52.509717+0000","last_clean":"2024-09-02T10:01:52.509717+0000","last_became_active":"2024-09-02T10:01:30.559685+0000","last_became_peered":"2024-09-02T10:01:30.559685+0000","last_unstale":"2024-09-02T10:01:52.509717+0000","last_undegraded":"2024-09-02T10:01:52.509717+0000","last_fullsized":"2024-09-02T10:01:52.509717+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1516,"log_dups_size":0,"ondisk_log_size":1516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:57.437147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99352576,"num_objects":1516,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1516,"num_write_kb":97024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:50.766561+0000","last_change":"2024-09-02T10:01:50.765918+0000","last_active":"2024-09-02T10:01:50.766561+0000","last_peered":"2024-09-02T10:01:50.766561+0000","last_clean":"2024-09-02T10:01:50.766561+0000","last_became_active":"2024-09-02T10:01:50.765475+0000","last_became_peered":"2024-09-02T10:01:50.765475+0000","last_unstale":"2024-09-02T10:01:50.766561+0000","last_undegraded":"2024-09-02T10:01:50.766561+0000","last_fullsized":"2024-09-02T10:01:50.766561+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:00:29.279658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:00:29.279658+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"105'1612","reported_seq":1647,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-02T10:01:54.090324+0000","last_change":"2024-09-02T10:01:30.183497+0000","last_active":"2024-09-02T10:01:54.090324+0000","last_peered":"2024-09-02T10:01:54.090324+0000","last_clean":"2024-09-02T10:01:54.090324+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:01:54.090324+0000","last_undegraded":"2024-09-02T10:01:54.090324+0000","last_fullsized":"2024-09-02T10:01:54.090324+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1612,"log_dups_size":0,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:40.071109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105644032,"num_objects":1612,"num_object_clones":0,"num_object_copies":4836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1612,"num_write_kb":103168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:50.775435+0000","last_change":"2024-09-02T10:01:50.775022+0000","last_active":"2024-09-02T10:01:50.775435+0000","last_peered":"2024-09-02T10:01:50.775435+0000","last_clean":"2024-09-02T10:01:50.775435+0000","last_became_active":"2024-09-02T10:01:50.774645+0000","last_became_peered":"2024-09-02T10:01:50.774645+0000","last_unstale":"2024-09-02T10:01:50.775435+0000","last_undegraded":"2024-09-02T10:01:50.775435+0000","last_fullsized":"2024-09-02T10:01:50.775435+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:00:29.279658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:00:29.279658+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":48084,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"104'1469","reported_seq":1502,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:52.509451+0000","last_change":"2024-09-02T10:01:30.560649+0000","last_active":"2024-09-02T10:01:52.509451+0000","last_peered":"2024-09-02T10:01:52.509451+0000","last_clean":"2024-09-02T10:01:52.509451+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:01:52.509451+0000","last_undegraded":"2024-09-02T10:01:52.509451+0000","last_fullsized":"2024-09-02T10:01:52.509451+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1469,"log_dups_size":0,"ondisk_log_size":1469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:34.163733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96272384,"num_objects":1469,"num_object_clones":0,"num_object_copies":4407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1469,"num_write_kb":94016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":112,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:51.746834+0000","last_change":"2024-09-02T10:01:51.746769+0000","last_active":"2024-09-02T10:01:51.746834+0000","last_peered":"2024-09-02T10:01:51.746834+0000","last_clean":"2024-09-02T10:01:51.746834+0000","last_became_active":"2024-09-02T10:01:50.768028+0000","last_became_peered":"2024-09-02T10:01:50.768028+0000","last_unstale":"2024-09-02T10:01:51.746834+0000","last_undegraded":"2024-09-02T10:01:51.746834+0000","last_fullsized":"2024-09-02T10:01:51.746834+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.746691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.746691+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:00.251729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"104'1501","reported_seq":1536,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:51.925086+0000","last_change":"2024-09-02T10:01:30.560449+0000","last_active":"2024-09-02T10:01:51.925086+0000","last_peered":"2024-09-02T10:01:51.925086+0000","last_clean":"2024-09-02T10:01:51.925086+0000","last_became_active":"2024-09-02T10:01:30.559962+0000","last_became_peered":"2024-09-02T10:01:30.559962+0000","last_unstale":"2024-09-02T10:01:51.925086+0000","last_undegraded":"2024-09-02T10:01:51.925086+0000","last_fullsized":"2024-09-02T10:01:51.925086+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1501,"log_dups_size":0,"ondisk_log_size":1501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:55.111495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98369536,"num_objects":1501,"num_object_clones":0,"num_object_copies":4503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1501,"num_write_kb":96064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"103'1355","reported_seq":1388,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-02T10:01:50.592535+0000","last_change":"2024-09-02T10:01:30.141564+0000","last_active":"2024-09-02T10:01:50.592535+0000","last_peered":"2024-09-02T10:01:50.592535+0000","last_clean":"2024-09-02T10:01:50.592535+0000","last_became_active":"2024-09-02T10:01:30.141263+0000","last_became_peered":"2024-09-02T10:01:30.141263+0000","last_unstale":"2024-09-02T10:01:50.592535+0000","last_undegraded":"2024-09-02T10:01:50.592535+0000","last_fullsized":"2024-09-02T10:01:50.592535+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:56.004849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88801280,"num_objects":1355,"num_object_clones":0,"num_object_copies":4065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1355,"num_write_kb":86720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":20,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:51.614443+0000","last_change":"2024-09-02T10:01:51.614379+0000","last_active":"2024-09-02T10:01:51.614443+0000","last_peered":"2024-09-02T10:01:51.614443+0000","last_clean":"2024-09-02T10:01:51.614443+0000","last_became_active":"2024-09-02T10:01:50.781305+0000","last_became_peered":"2024-09-02T10:01:50.781305+0000","last_unstale":"2024-09-02T10:01:51.614443+0000","last_undegraded":"2024-09-02T10:01:51.614443+0000","last_fullsized":"2024-09-02T10:01:51.614443+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.614334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.614334+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:04.348678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"104'1531","reported_seq":1564,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:52.532499+0000","last_change":"2024-09-02T10:01:30.560354+0000","last_active":"2024-09-02T10:01:52.532499+0000","last_peered":"2024-09-02T10:01:52.532499+0000","last_clean":"2024-09-02T10:01:52.532499+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:01:52.532499+0000","last_undegraded":"2024-09-02T10:01:52.532499+0000","last_fullsized":"2024-09-02T10:01:52.532499+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:48.623221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100335616,"num_objects":1531,"num_object_clones":0,"num_object_copies":4593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1531,"num_write_kb":97984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":162,"reported_epoch":103,"state":"peering","last_fresh":"2024-09-02T10:01:49.754324+0000","last_change":"2024-09-02T10:01:49.754324+0000","last_active":"2024-09-02T10:01:49.754324+0000","last_peered":"2024-09-02T10:01:48.747032+0000","last_clean":"2024-09-02T10:01:48.747032+0000","last_became_active":"2024-09-02T10:01:48.217576+0000","last_became_peered":"2024-09-02T10:01:48.217576+0000","last_unstale":"2024-09-02T10:01:49.754324+0000","last_undegraded":"2024-09-02T10:01:49.754324+0000","last_fullsized":"2024-09-02T10:01:49.754324+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:00:29.279658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:00:29.279658+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6,"num_write":11,"num_write_kb":195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":144253,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"104'1417","reported_seq":1451,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:51.818651+0000","last_change":"2024-09-02T10:01:30.560175+0000","last_active":"2024-09-02T10:01:51.818651+0000","last_peered":"2024-09-02T10:01:51.818651+0000","last_clean":"2024-09-02T10:01:51.818651+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:01:51.818651+0000","last_undegraded":"2024-09-02T10:01:51.818651+0000","last_fullsized":"2024-09-02T10:01:51.818651+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1417,"log_dups_size":0,"ondisk_log_size":1417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:38.707848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92864512,"num_objects":1417,"num_object_clones":0,"num_object_copies":4251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1417,"num_write_kb":90688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":27,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-02T10:01:50.947480+0000","last_change":"2024-09-02T10:01:50.947414+0000","last_active":"2024-09-02T10:01:50.947480+0000","last_peered":"2024-09-02T10:01:50.947480+0000","last_clean":"2024-09-02T10:01:50.947480+0000","last_became_active":"2024-09-02T10:01:50.763116+0000","last_became_peered":"2024-09-02T10:01:50.763116+0000","last_unstale":"2024-09-02T10:01:50.947480+0000","last_undegraded":"2024-09-02T10:01:50.947480+0000","last_fullsized":"2024-09-02T10:01:50.947480+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:50.947356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:50.947356+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:05.422847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"105'1623","reported_seq":1658,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-09-02T10:01:54.025967+0000","last_change":"2024-09-02T10:01:30.140619+0000","last_active":"2024-09-02T10:01:54.025967+0000","last_peered":"2024-09-02T10:01:54.025967+0000","last_clean":"2024-09-02T10:01:54.025967+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:01:54.025967+0000","last_undegraded":"2024-09-02T10:01:54.025967+0000","last_fullsized":"2024-09-02T10:01:54.025967+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":1623,"log_dups_size":0,"ondisk_log_size":1623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:39.195715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106364928,"num_objects":1623,"num_object_clones":0,"num_object_copies":4869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1623,"num_write_kb":103872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1563951104,"num_objects":23864,"num_object_clones":0,"num_object_copies":71592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23864,"num_write_kb":1527296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4699193344,"data_stored":4699193344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23864,"ondisk_log_size":23864,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":805048,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":24,"num_write":45,"num_write_kb":782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":384674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383701,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302752,"kb_used_data":271564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93442400,"statfs":{"total":95995035648,"available":95685017600,"internally_reserved":0,"allocated":278081536,"data_stored":277689796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612634,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298408,"kb_used_data":267212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93446744,"statfs":{"total":95995035648,"available":95689465856,"internally_reserved":0,"allocated":273625088,"data_stored":273233356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841567,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":617628,"kb_used_data":586444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93127524,"statfs":{"total":95995035648,"available":95362584576,"internally_reserved":0,"allocated":600518656,"data_stored":600126924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":430940,"kb_used_data":395596,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93314212,"statfs":{"total":95995035648,"available":95553753088,"internally_reserved":0,"allocated":405090304,"data_stored":404698572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299431,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451552,"kb_used_data":416200,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93293600,"statfs":{"total":95995035648,"available":95532646400,"internally_reserved":0,"allocated":426188800,"data_stored":425798056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528364,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":227744,"kb_used_data":188236,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93517408,"statfs":{"total":95995035648,"available":95761825792,"internally_reserved":0,"allocated":192753664,"data_stored":192361932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724593,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":222184,"kb_used_data":182676,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93522968,"statfs":{"total":95995035648,"available":95767519232,"internally_reserved":0,"allocated":187060224,"data_stored":186664940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953527,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":323740,"kb_used_data":296716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93421412,"statfs":{"total":95995035648,"available":95663525888,"internally_reserved":0,"allocated":303837184,"data_stored":303445452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":626596,"kb_used_data":595412,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93118556,"statfs":{"total":95995035648,"available":95353401344,"internally_reserved":0,"allocated":609701888,"data_stored":609306604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":402224,"kb_used_data":366868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93342928,"statfs":{"total":95995035648,"available":95583158272,"internally_reserved":0,"allocated":375672832,"data_stored":375277548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":505436,"kb_used_data":465932,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93239716,"statfs":{"total":95995035648,"available":95477469184,"internally_reserved":0,"allocated":477114368,"data_stored":476722636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":486824,"kb_used_data":447308,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93258328,"statfs":{"total":95995035648,"available":95496527872,"internally_reserved":0,"allocated":458043392,"data_stored":457651660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":461766656,"data_stored":461766656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":502136832,"data_stored":502136832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":386465792,"data_stored":386465792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":610926592,"data_stored":610926592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":318373888,"data_stored":318373888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":193462272,"data_stored":193462272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":198443008,"data_stored":198443008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":438304768,"data_stored":438304768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":425656320,"data_stored":425656320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":605093888,"data_stored":605093888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":279314432,"data_stored":279314432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":279248896,"data_stored":279248896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:01:56.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-02T10:01:57.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:57.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:57.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:57.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:57.177 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:57.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:57.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:57.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:57.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:57.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:57.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:01:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:57.427 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:57.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:01:57.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 22 16 30374 30358 86.237 82.1875 0.0433438 0.0115451 2024-09-02T10:01:57.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:57 smithi151 ceph-mon[16639]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:57.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1758031153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:57.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:57 smithi187 ceph-mon[20072]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:57.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1758031153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:57.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:57.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:57.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:57.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:57 smithi162 ceph-mon[20279]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:01:57.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1758031153' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:01:57.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:57.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:57.677 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:57.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:57.736 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:57.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:57.787 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:57.790 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:01:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:57.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:57.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:57.915 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:01:57.945 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:57.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:01:58.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:58.070 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:58.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:58.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:01:58.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:58.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:58.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:01:58.289 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:58 smithi151 ceph-mon[16639]: pgmap v314: 27 pgs: 27 active+clean; 1.7 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.36k op/s 2024-09-02T10:01:58.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:58.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:58.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:58.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:58.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 23 15 31665 31650 85.9982 80.75 0.00611212 0.0116198 2024-09-02T10:01:58.449 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:58.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:58.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:01:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:58 smithi187 ceph-mon[20072]: pgmap v314: 27 pgs: 27 active+clean; 1.7 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.36k op/s 2024-09-02T10:01:58.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:58.567 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:01:58.567 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:01:58.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:58.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:58.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:58.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:58 smithi162 ceph-mon[20279]: pgmap v314: 27 pgs: 27 active+clean; 1.7 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.36k op/s 2024-09-02T10:01:58.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:58.708 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:58.710 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:58.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:58.836 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:58.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:58.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:58.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:58.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:58.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:59.072 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:59.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:59.097 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:59.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:01:59.223 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:59.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:01:59.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:01:59.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:59.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:59.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:01:59.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 24 16 33315 33299 86.7088 103.062 0.0090119 0.0115207 2024-09-02T10:01:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:59 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:01:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:59 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:01:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:59 smithi151 ceph-mon[16639]: osdmap e106: 12 total, 12 up, 12 in 2024-09-02T10:01:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:59 smithi151 ceph-mon[16639]: osdmap e107: 12 total, 12 up, 12 in 2024-09-02T10:01:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1983158506' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-02T10:01:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:01:59 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-02T10:01:59.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:01:59.474 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:59.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:59.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:01:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:59 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:01:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:59 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:01:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:59 smithi187 ceph-mon[20072]: osdmap e106: 12 total, 12 up, 12 in 2024-09-02T10:01:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:59 smithi187 ceph-mon[20072]: osdmap e107: 12 total, 12 up, 12 in 2024-09-02T10:01:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1983158506' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-02T10:01:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:01:59 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-02T10:01:59.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:01:59.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:01:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:59 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:01:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:59 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:01:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:59 smithi162 ceph-mon[20279]: osdmap e106: 12 total, 12 up, 12 in 2024-09-02T10:01:59.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:59 smithi162 ceph-mon[20279]: osdmap e107: 12 total, 12 up, 12 in 2024-09-02T10:01:59.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1983158506' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-02T10:01:59.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:01:59 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"} : dispatch 2024-09-02T10:01:59.598 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:59.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:01:59.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:01:59.700 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:01:59.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:01:59.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:59.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:01:59.755 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pgp_num to 11 2024-09-02T10:01:59.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:01:59.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:59.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:01:59.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:01:59.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:01:59.981 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:01:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:01:59.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:00.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:00.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:00.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:00.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:00.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:00.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:00.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:00.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:00.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:00.434 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 25 16 34582 34566 86.4079 79.1875 0.00447675 0.0115674 2024-09-02T10:02:00.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:00 smithi151 ceph-mon[16639]: pgmap v317: 27 pgs: 27 active+clean; 1.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-02T10:02:00.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:00 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-02T10:02:00.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:00 smithi151 ceph-mon[16639]: osdmap e108: 12 total, 12 up, 12 in 2024-09-02T10:02:00.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:00.493 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:00.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:00.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:00 smithi187 ceph-mon[20072]: pgmap v317: 27 pgs: 27 active+clean; 1.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-02T10:02:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:00 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-02T10:02:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:00 smithi187 ceph-mon[20072]: osdmap e108: 12 total, 12 up, 12 in 2024-09-02T10:02:00.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:00 smithi162 ceph-mon[20279]: pgmap v317: 27 pgs: 27 active+clean; 1.8 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-02T10:02:00.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:00 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-02T10:02:00.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:00 smithi162 ceph-mon[20279]: osdmap e108: 12 total, 12 up, 12 in 2024-09-02T10:02:00.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:00.617 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:00.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:00.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:00.741 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:00.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:00.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:00.810 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:00.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:00.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:00.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:00.966 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:00.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:00.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:00.990 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:01.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:01.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:01.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:01.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:01.225 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:02:01.245 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:01.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:01.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:01.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:01.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:01.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:01.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 26 16 35444 35428 85.1564 53.875 0.00942781 0.0117379 2024-09-02T10:02:01.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:01.496 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2258443662' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2258443662' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:01.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:01.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:01.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2258443662' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:01.613 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:02:01.613 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:02:01.614 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:02:01.614 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:02:01.614 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:02:01.614 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:02:01.614 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:02:01.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:02:01.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:02:01.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:02:01.615 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:02:01.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:02:01.615 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:01.621 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:01.627 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:02:01.627 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:02:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:01.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:01.644 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:01.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:02:01.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:02:01.692 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:02:01.692 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:02:01.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:02:01.719 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:02:01.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:01.746 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:02:01.746 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:02:01.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:01.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:01.773 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:02:01.773 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:02:01.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:01.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:01.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:01.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:02.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:02.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:02.024 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:02.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:02.126 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:02.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:02.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:02.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:02.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:02.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:02.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:02.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:02.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:02.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:02.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:02.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:02.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 27 16 36721 36705 84.9583 79.8125 0.00153623 0.0117607 2024-09-02T10:02:02.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:02 smithi187 ceph-mon[20072]: pgmap v319: 27 pgs: 27 active+clean; 1.9 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:02:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:02 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2258443662' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:02 smithi187 ceph-mon[20072]: osdmap e109: 12 total, 12 up, 12 in 2024-09-02T10:02:02.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:02 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:02:02.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:02 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:02:02.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2833187272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:02.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:02.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:02.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:02.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:02 smithi162 ceph-mon[20279]: pgmap v319: 27 pgs: 27 active+clean; 1.9 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:02:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:02 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2258443662' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:02 smithi162 ceph-mon[20279]: osdmap e109: 12 total, 12 up, 12 in 2024-09-02T10:02:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:02 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:02:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:02 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:02:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2833187272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:02.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:02.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:02.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:02.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:02.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:02 smithi151 ceph-mon[16639]: pgmap v319: 27 pgs: 27 active+clean; 1.9 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:02:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:02 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2258443662' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:02 smithi151 ceph-mon[16639]: osdmap e109: 12 total, 12 up, 12 in 2024-09-02T10:02:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:02 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:02:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:02 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:02:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2833187272' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:02.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:02.777 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:02.778 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:02.879 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:02.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:02.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:03.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:03.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:03.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:03.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:03.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:03.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:03.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:03.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:03.283 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:03.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:03.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:03.410 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:03.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:03.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 28 16 37897 37881 84.5488 73.5 0.00580981 0.0118197 2024-09-02T10:02:03.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:02:03.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:03 smithi151 ceph-mon[16639]: osdmap e110: 12 total, 12 up, 12 in 2024-09-02T10:02:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:02:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:02:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:02:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:02:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:02:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:02:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:02:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:03 smithi187 ceph-mon[20072]: osdmap e110: 12 total, 12 up, 12 in 2024-09-02T10:02:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:02:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:02:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:02:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:02:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:02:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:02:03.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:03.537 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:03.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:03.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:02:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:03 smithi162 ceph-mon[20279]: osdmap e110: 12 total, 12 up, 12 in 2024-09-02T10:02:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:02:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:02:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:02:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:02:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:02:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:02:03.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:03.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:03.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:03.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:03.793 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:03.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:03.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:03.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:03.918 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:03.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:04.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:04.042 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:04.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:04.059 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:02:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:02:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:02:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:02:04.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:1142 2.7 e96/92 2:e2cb33eb:::benchmark_data_smithi187_45882_object1141:head v 96'87, mlcod=96'83)", 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.907033+0000", 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.149964404999999, 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18660511599999999, 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.907033+0000", 2024-09-02T10:02:04.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.907033+0000", 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.907034+0000", 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0189999999999999e-06 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.907076+0000", 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1548000000000003e-05 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093236+0000", 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.186160574 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093243+0000", 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8589999999999999e-06 2024-09-02T10:02:04.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093298+0000", 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5068999999999997e-05 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093319+0000", 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1234999999999999e-05 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093610+0000", 2024-09-02T10:02:04.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000291184 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093638+0000", 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7628e-05 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:7877 2.1 2:80876db5:::benchmark_data_smithi187_45882_object7876:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:39.673042+0000", 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.383955313000001, 2024-09-02T10:02:04.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.124192889, 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7877 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.673042+0000", 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.673042+0000", 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.673043+0000", 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.251e-06 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.673076+0000", 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3000000000000003e-05 2024-09-02T10:02:04.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.673089+0000", 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3004e-05 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.673096+0000", 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0319999999999998e-06 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.673152+0000", 2024-09-02T10:02:04.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5293000000000001e-05 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.673240+0000", 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8652000000000004e-05 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.673329+0000", 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8466000000000003e-05 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.673861+0000", 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053191899999999995 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.674264+0000", 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040376000000000002 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.797157+0000", 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12289219 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.797192+0000", 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5295e-05 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.797235+0000", 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3027e-05 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.070 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:7887 2.1 2:81374d58:::benchmark_data_smithi187_45882_object7886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:39.686303+0000", 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.370694298, 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.121079271, 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7887 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.686303+0000", 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.686303+0000", 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.686304+0000", 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-02T10:02:04.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.686358+0000", 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3837e-05 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.686369+0000", 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1644000000000001e-05 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.686376+0000", 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8739999999999997e-06 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.686419+0000", 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2815000000000003e-05 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.686496+0000", 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6594000000000006e-05 2024-09-02T10:02:04.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.686579+0000", 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3814999999999996e-05 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.687092+0000", 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000512394 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.687386+0000", 2024-09-02T10:02:04.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029374499999999999 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.807327+0000", 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.119941354 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.807358+0000", 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1269999999999997e-05 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.807382+0000", 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4227999999999999e-05 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:8372 2.e 2:77370da9:::benchmark_data_smithi187_45882_object8371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.033332+0000", 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.023665165000001, 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.181158239, 2024-09-02T10:02:04.077 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033332+0000", 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033332+0000", 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033334+0000", 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.274e-06 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033372+0000", 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8554999999999999e-05 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033384+0000", 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2204999999999999e-05 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033392+0000", 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8809999999999996e-06 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033454+0000", 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1784000000000003e-05 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033610+0000", 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015655399999999999 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033773+0000", 2024-09-02T10:02:04.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016284099999999999 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.034409+0000", 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063585900000000001 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035230+0000", 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00082086500000000005 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214426+0000", 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17919570700000001 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214457+0000", 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1108000000000001e-05 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214490+0000", 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3606000000000003e-05 2024-09-02T10:02:04.083 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:8372 2.e e97/92 ondisk, result = 0)", 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.034947+0000", 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.022050487000001, 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17954930999999999, 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:02:04.084 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.034947+0000", 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.034947+0000", 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.034948+0000", 2024-09-02T10:02:04.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0270000000000001e-06 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.034953+0000", 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3059999999999997e-06 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214376+0000", 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17942266400000001 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214381+0000", 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6970000000000001e-06 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214418+0000", 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7444000000000003e-05 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214423+0000", 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5079999999999999e-06 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214496+0000", 2024-09-02T10:02:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3664000000000005e-05 2024-09-02T10:02:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:8373 2.1 2:8eb82fd1:::benchmark_data_smithi187_45882_object8372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:02:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.035325+0000", 2024-09-02T10:02:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 24.021672354, 2024-09-02T10:02:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.164965424, 2024-09-02T10:02:04.089 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035325+0000", 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035325+0000", 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035327+0000", 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.48e-06 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035387+0000", 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0078999999999999e-05 2024-09-02T10:02:04.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035401+0000", 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4331000000000001e-05 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035410+0000", 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7430000000000008e-06 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035461+0000", 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0932e-05 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035562+0000", 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101665 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035658+0000", 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5521999999999998e-05 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.036305+0000", 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00064723299999999999 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.036691+0000", 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038610199999999998 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.200216+0000", 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16352460099999999 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.200251+0000", 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5323e-05 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.200290+0000", 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9412999999999997e-05 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.095 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:9147 2.3 e98/92 2:c3cba1c2:::benchmark_data_smithi187_45882_object9146:head v 98'562, mlcod=98'558)", 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.606830+0000", 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.450167775000001, 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18017456100000001, 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.606830+0000", 2024-09-02T10:02:04.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.606830+0000", 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.606830+0000", 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5199999999999996e-07 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.606854+0000", 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3634000000000001e-05 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.786442+0000", 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17958851300000001 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.786448+0000", 2024-09-02T10:02:04.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.786498+0000", 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9654000000000003e-05 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.786519+0000", 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1489e-05 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.786972+0000", 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045280299999999999 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.787004+0000", 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2336999999999999e-05 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.100 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:9155 2.e 2:7586a36f:::benchmark_data_smithi187_45882_object9154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.615656+0000", 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.441341530999999, 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.161630628, 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9155 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615656+0000", 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615656+0000", 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615657+0000", 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8800000000000002e-07 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615719+0000", 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2508999999999994e-05 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615728+0000", 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7099999999999996e-06 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615733+0000", 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6689999999999997e-06 2024-09-02T10:02:04.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615781+0000", 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8442999999999997e-05 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615892+0000", 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011149700000000001 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.616031+0000", 2024-09-02T10:02:04.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000138861 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.616610+0000", 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057823100000000001 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.616808+0000", 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019807700000000001 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777215+0000", 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.160407311 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777255+0000", 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9620999999999999e-05 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777287+0000", 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1911e-05 2024-09-02T10:02:04.106 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:9155 2.e e98/92 ondisk, result = 0)", 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.616793+0000", 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.440204218000002, 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16050207699999999, 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.616793+0000", 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.616793+0000", 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.616794+0000", 2024-09-02T10:02:04.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0900000000000001e-07 2024-09-02T10:02:04.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.616797+0000", 2024-09-02T10:02:04.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5410000000000001e-06 2024-09-02T10:02:04.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777152+0000", 2024-09-02T10:02:04.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.160354788 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777160+0000", 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4590000000000001e-06 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777206+0000", 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6128000000000003e-05 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777211+0000", 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3789999999999997e-06 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777295+0000", 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4073000000000006e-05 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.111 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:9161 2.7 e98/92 2:e5b7fd38:::benchmark_data_smithi187_45882_object9160:head v 98'620, mlcod=98'617)", 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.620172+0000", 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 23.436825531, 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15707931, 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.620172+0000", 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.620172+0000", 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.620173+0000", 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0900000000000001e-07 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.620266+0000", 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3268000000000002e-05 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.776511+0000", 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.156245034 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.776519+0000", 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5779999999999999e-06 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.776599+0000", 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9995999999999998e-05 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.776619+0000", 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9113000000000001e-05 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777221+0000", 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060215900000000001 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777251+0000", 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0453000000000001e-05 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:10409 2.1 2:852777bc:::benchmark_data_smithi187_45882_object10408:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:41.637421+0000", 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.419575948999999, 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.122601081, 2024-09-02T10:02:04.116 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 10409 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.637421+0000", 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.637421+0000", 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.637422+0000", 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5899999999999995e-07 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.637476+0000", 2024-09-02T10:02:04.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3576999999999999e-05 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.637486+0000", 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0380000000000001e-05 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.637492+0000", 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.322e-06 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.637528+0000", 2024-09-02T10:02:04.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.659e-05 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.637610+0000", 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1329999999999996e-05 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.637692+0000", 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2082000000000005e-05 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:02:04.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.638232+0000", 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00054000499999999998 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.638433+0000", 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000201437 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.759925+0000", 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.121491777 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.759980+0000", 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5349000000000001e-05 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.760023+0000", 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2373000000000002e-05 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:10409 2.1 e98/92 ondisk, result = 0)", 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:41.638381+0000", 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.418616817, 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12164939600000001, 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.638381+0000", 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.638381+0000", 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.638381+0000", 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4499999999999997e-07 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.638384+0000", 2024-09-02T10:02:04.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.281e-06 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.759858+0000", 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.121473976 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.759865+0000", 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.438e-06 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.759911+0000", 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6612e-05 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.759919+0000", 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5739999999999998e-06 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:41.760030+0000", 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011097 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.126 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:11788 2.7 e98/92 2:ee0a4980:::benchmark_data_smithi187_45882_object11787:head v 98'788, mlcod=98'785)", 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:42.750110+0000", 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 21.306887815, 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13464451899999999, 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.750110+0000", 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.750110+0000", 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.750110+0000", 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5700000000000002e-07 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.750195+0000", 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4665000000000003e-05 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.884153+0000", 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13395847299999999 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.884160+0000", 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1109999999999996e-06 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.884200+0000", 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0524e-05 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.884216+0000", 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5430999999999998e-05 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.884728+0000", 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051205800000000002 2024-09-02T10:02:04.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.884754+0000", 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.65e-05 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:13541 2.1 2:80f77a31:::benchmark_data_smithi187_45882_object13540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:44.030249+0000", 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 20.026748787999999, 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13230950399999999, 2024-09-02T10:02:04.131 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 13541 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030249+0000", 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030249+0000", 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030250+0000", 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1990000000000001e-06 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030297+0000", 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6900999999999997e-05 2024-09-02T10:02:04.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030309+0000", 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2282e-05 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030315+0000", 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.088e-06 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030360+0000", 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4787999999999999e-05 2024-09-02T10:02:04.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030457+0000", 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7176999999999995e-05 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030602+0000", 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014452399999999999 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.031389+0000", 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00078788800000000004 2024-09-02T10:02:04.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.031654+0000", 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00026434000000000001 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.162457+0000", 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13080345800000001 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.162513+0000", 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5238999999999997e-05 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.162558+0000", 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.562e-05 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:27112 2.1 2:82fe9840:::benchmark_data_smithi187_45882_object27111:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:54.682194+0000", 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 9.3748036040000002, 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12247353599999999, 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 27112 2024-09-02T10:02:04.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.682194+0000", 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.682194+0000", 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.682195+0000", 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6799999999999999e-07 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.682249+0000", 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4549000000000002e-05 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.682261+0000", 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2359e-05 2024-09-02T10:02:04.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.682269+0000", 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4019999999999996e-06 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.682338+0000", 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8700000000000003e-05 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.682454+0000", 2024-09-02T10:02:04.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116915 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.682594+0000", 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000139441 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.684094+0000", 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014997649999999999 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.687169+0000", 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030755829999999998 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.804594+0000", 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.11742509900000001 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.804634+0000", 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9816e-05 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.804667+0000", 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3139000000000002e-05 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:28222 2.3 e105/92 2:ce2e2400:::benchmark_data_smithi187_45882_object28221:head v 105'1793, mlcod=105'1789)", 2024-09-02T10:02:04.144 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:55.623915+0000", 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.4330820489999994, 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15956630499999999, 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:55.623915+0000", 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:55.623915+0000", 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:55.623916+0000", 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0800000000000003e-07 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:55.623961+0000", 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4610999999999997e-05 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:55.782532+0000", 2024-09-02T10:02:04.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:04.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15857083599999999 2024-09-02T10:02:04.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:55.782536+0000", 2024-09-02T10:02:04.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3880000000000002e-06 2024-09-02T10:02:04.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:55.782586+0000", 2024-09-02T10:02:04.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9598000000000002e-05 2024-09-02T10:02:04.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:55.782600+0000", 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.448e-05 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:55.783434+0000", 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00083393599999999997 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:55.783482+0000", 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7547999999999999e-05 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.150 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:36445 2.1 2:8b5b62e6:::benchmark_data_smithi187_45882_object36444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:02.093891+0000", 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.963106738, 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21621346299999999, 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:04.151 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 36445 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093891+0000", 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093891+0000", 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093892+0000", 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1400000000000001e-06 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093951+0000", 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9660000000000001e-05 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093967+0000", 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6018000000000001e-05 2024-09-02T10:02:04.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093975+0000", 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.841e-06 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.094034+0000", 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8899e-05 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.094180+0000", 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014549699999999999 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.094278+0000", 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8599999999999998e-05 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.095564+0000", 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012853669999999999 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.102929+0000", 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073655550000000002 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.309532+0000", 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.206602962 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.310053+0000", 2024-09-02T10:02:04.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052091099999999999 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.310104+0000", 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1013000000000001e-05 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:36445 2.1 e109/92 ondisk, result = 0)", 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:02.095796+0000", 2024-09-02T10:02:04.157 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.9612014609999999, 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21432100700000001, 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.095796+0000", 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.095796+0000", 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.095797+0000", 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7899999999999997e-07 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.095801+0000", 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6430000000000003e-06 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.308322+0000", 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21252061899999999 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.308328+0000", 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3489999999999997e-06 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.309518+0000", 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001190008 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.309526+0000", 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7950000000000008e-06 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.310117+0000", 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00059081400000000005 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:37176 2.7 e110/92 2:e895c9fb:::benchmark_data_smithi187_45882_object37175:head v 110'2343, mlcod=110'2339)", 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:02.872757+0000", 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.1842408630000001, 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.145116575, 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.872757+0000", 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.872757+0000", 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.872758+0000", 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1400000000000001e-06 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.872786+0000", 2024-09-02T10:02:04.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.866e-05 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.016870+0000", 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.144083458 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.016874+0000", 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6630000000000001e-06 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.016920+0000", 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5516999999999998e-05 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.016934+0000", 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3716999999999999e-05 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.017846+0000", 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00091248100000000003 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.017873+0000", 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6939000000000001e-05 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:37178 2.1 2:8a9d5e65:::benchmark_data_smithi187_45882_object37177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:02.876493+0000", 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.180504598, 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.124424192, 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:02:04.166 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 37178 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876493+0000", 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876493+0000", 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876494+0000", 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4160000000000001e-06 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876539+0000", 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4357000000000002e-05 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876550+0000", 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1036000000000001e-05 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876557+0000", 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5900000000000002e-06 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876605+0000", 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7555000000000001e-05 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876698+0000", 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2740999999999999e-05 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876792+0000", 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4629000000000005e-05 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.877841+0000", 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010490479999999999 2024-09-02T10:02:04.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.878171+0000", 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032957800000000002 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.000847+0000", 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.122676728 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:02:04.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.000883+0000", 2024-09-02T10:02:04.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5209999999999997e-05 2024-09-02T10:02:04.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:04.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:04.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:02:04.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.000917+0000", 2024-09-02T10:02:04.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4304000000000002e-05 2024-09-02T10:02:04.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.172 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.173 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:04.173 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:04.173 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:02:04.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:04.174 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:04.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:04.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:04.304 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:04.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:04.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:04.429 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:04.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:04.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 29 16 39260 39244 84.5706 85.1875 0.00207058 0.0118136 2024-09-02T10:02:04.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:04 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:02:04.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:04 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:02:04.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:04 smithi151 ceph-mon[16639]: pgmap v322: 27 pgs: 27 active+clean; 2.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-02T10:02:04.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:04 smithi151 ceph-mon[16639]: osdmap e111: 12 total, 12 up, 12 in 2024-09-02T10:02:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:04 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:02:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:04 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:02:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:04 smithi187 ceph-mon[20072]: pgmap v322: 27 pgs: 27 active+clean; 2.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-02T10:02:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:04 smithi187 ceph-mon[20072]: osdmap e111: 12 total, 12 up, 12 in 2024-09-02T10:02:04.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:04.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:04.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:04.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:04 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:02:04.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:04 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:02:04.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:04 smithi162 ceph-mon[20279]: pgmap v322: 27 pgs: 27 active+clean; 2.2 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-02T10:02:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:04 smithi162 ceph-mon[20279]: osdmap e111: 12 total, 12 up, 12 in 2024-09-02T10:02:04.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:04.677 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:04.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:04.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:04.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:04.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:04.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:04.909 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:04.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:04.937 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:05.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:05.062 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:05.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:05.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:05.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:02:05.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:05.189 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:05.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:05.315 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:05.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:05.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:05.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 30 16 40546 40530 84.4305 80.375 0.00728227 0.0118382 2024-09-02T10:02:05.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:05.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:05.445 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:05.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:02:05.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:05.553 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:05.562 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:02:05.562 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:02:05.562 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-02T10:02:05.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:02:05.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:05.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:05.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:05.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:05.699 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:05.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:05.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:05.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:02:05.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:05.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:05.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:05.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:05.976 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:05.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:06.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:06.100 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:02:06.100 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:02:06.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:06.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:06.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:06.127 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:06.129 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:06.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:02:06.133 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:02:06.167 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:02:06.167 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:02:06.173 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:06.189 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:02:06.189 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:02:06.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:06.211 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:02:06.211 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:02:06.224 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:06.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:06.231 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:02:06.231 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:02:06.252 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:02:06.252 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:02:06.273 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:02:06.273 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:02:06.295 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:02:06.296 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:02:06.318 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:02:06.319 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:02:06.319 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:06 smithi151 ceph-mon[16639]: pgmap v324: 27 pgs: 27 active+clean; 2.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-02T10:02:06.319 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1063494108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:06.325 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:06.340 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:02:06.340 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:02:06.346 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:06.347 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:06.364 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:02:06.364 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:02:06.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 31 16 41246 41230 83.1182 43.75 0.0058911 0.0120246 2024-09-02T10:02:06.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:06.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:06.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:06 smithi187 ceph-mon[20072]: pgmap v324: 27 pgs: 27 active+clean; 2.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-02T10:02:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1063494108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:06.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:06.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:06 smithi162 ceph-mon[20279]: pgmap v324: 27 pgs: 27 active+clean; 2.2 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-02T10:02:06.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1063494108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:06.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:06.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:06.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:06.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:06.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:06.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:06.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:06.857 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:06.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:06.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:06.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:06.983 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:06.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:07.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:07.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:07.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:07.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:07.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:07.237 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:07.238 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:07.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:07.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:07.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:07.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 32 16 42723 42707 83.4053 92.3125 0.00217863 0.0119811 2024-09-02T10:02:07.447 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:07.447 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:07.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:07.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:07.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:07.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:07.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:07.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:07.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:07.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:07.735 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:07.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:07.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:07.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:07.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:07.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:07.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:07.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:07.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:08.084 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:08.109 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:08.109 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:08.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:08.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:08.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:08.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:08.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:08.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:08.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:08.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 33 16 43544 43528 82.4326 51.3125 0.0192385 0.0120948 2024-09-02T10:02:08.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:08.493 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:08.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:08 smithi187 ceph-mon[20072]: pgmap v325: 27 pgs: 27 active+clean; 2.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-02T10:02:08.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:08 smithi162 ceph-mon[20279]: pgmap v325: 27 pgs: 27 active+clean; 2.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-02T10:02:08.596 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:08.622 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:08.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:08.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:08 smithi151 ceph-mon[16639]: pgmap v325: 27 pgs: 27 active+clean; 2.4 GiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-02T10:02:08.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:08.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:08.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:08.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:08.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:08.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:08.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:08.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:09.005 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:09.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:09.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:09.128 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:09.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:09.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:09.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:09.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:09.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:09.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:09.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:09.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:09.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:09.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 34 16 44784 44768 82.2873 77.5 0.0161675 0.0121461 2024-09-02T10:02:09.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:09.512 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:09.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:09.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:09.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:09.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:09.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:09.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:09.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:09.768 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:09.769 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:09.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:09.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:09.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:09.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:10.025 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:10.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:10.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:09 smithi162 ceph-mon[20279]: pgmap v326: 27 pgs: 27 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-02T10:02:10.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:10.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:10.154 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:10.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:10.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:09 smithi151 ceph-mon[16639]: pgmap v326: 27 pgs: 27 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-02T10:02:10.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:09 smithi187 ceph-mon[20072]: pgmap v326: 27 pgs: 27 active+clean; 2.5 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-02T10:02:10.283 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:10.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:10.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:10.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:10.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:10.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 35 16 46455 46439 82.9199 104.438 0.00256256 0.0120512 2024-09-02T10:02:10.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:10.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:10.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:10.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:10.660 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:10.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:10.661 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:10.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:10.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:10.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:10.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:10.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:02:10.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:10.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:10.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:11.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:11.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:11.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:11.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:11.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:11.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:11.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:11.313 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:11.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:11.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:11.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 36 16 47584 47568 82.5765 70.5625 0.00418595 0.0121043 2024-09-02T10:02:11.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:11.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:11.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:11.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:11.567 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:11.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:11.668 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:11.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:11.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:11.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:11.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:11.818 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:11.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:11.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:11.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:11.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:11.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:12.017 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:02:12.018 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:02:12.018 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 111, 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:02:03.725575+0000", 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:00:05.936786+0000", 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5806080, 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "nodeep-scrub", 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "noscrub", 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:02:12.019 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 2, 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:02:12.020 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "jewel", 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:02:12.021 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 11, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 11, 2024-09-02T10:02:12.022 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 11, 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 11, 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 11, 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "108", 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:02:12.023 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:02:12.024 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "100", 2024-09-02T10:02:12.024 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:02:12.024 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:02:12.024 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:02:12.024 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:02:12.024 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:02:12.024 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:02:12.024 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:02:12.024 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:02:12.025 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:02:12.026 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 3.2699999809265137, 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 3.2699999809265137, 2024-09-02T10:02:12.027 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 3.2699999809265137, 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 3.2699999809265137, 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 2, 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_0", 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:01:28.387897+0000", 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:02:12.028 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-09-02T10:02:12.029 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "96", 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:02:12.030 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:02:12.031 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:02:12.032 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.033 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.25, 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.25, 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.25, 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.25, 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.034 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:02:12.035 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 52, 2024-09-02T10:02:12.036 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.036 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.036 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.036 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.036 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.036 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:02:12.036 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:12.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.037 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.037 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:02:12.037 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:12.037 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.037 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.037 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.037 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.038 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.038 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:02:12.038 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:02:12.039 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:12.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.040 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.040 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:02:12.040 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:12.040 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.040 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.040 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:12.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:02:12.042 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.044 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.045 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.046 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.046 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.046 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:02:12.046 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:12.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.046 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.046 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:02:12.046 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:02:12.047 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 92, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.049 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.051 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.052 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.053 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.054 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 0, 2024-09-02T10:02:12.055 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:12.056 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:12.057 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:02:12.058 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:02:12.059 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 0, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.061 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.063 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:12.064 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:12.065 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.066 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.067 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.068 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.069 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.070 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.071 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.072 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.073 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.073 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.073 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.073 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.073 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:02:12.073 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:12.073 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.073 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.073 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.077 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.077 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.077 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.077 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:02:12.077 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:12.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.077 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:12.078 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:02:12.078 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:12.078 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.078 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:02:12.079 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.082 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.083 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.084 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:02:12.085 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 92, 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.086 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.089 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:02:12.090 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.091 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.092 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.092 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.092 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.092 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:02:12.092 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:12.092 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:12.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:12.094 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:02:12.095 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.096 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.098 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:02:12.098 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:12.098 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:02:12.098 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.098 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.099 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.100 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:02:12.103 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:12.104 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.105 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:12.109 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.110 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:14.548239+0000", 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:24.147373+0000", 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.111 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:02:01.278605+0000", 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.112 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:27.616817+0000", 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.113 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:03.548446+0000", 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:49.271030+0000", 2024-09-02T10:02:12.114 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:05.682930+0000", 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.115 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:36.161551+0000", 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.116 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:57.585331+0000", 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.117 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:02:02.732527+0000", 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:17.522791+0000", 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:49.719022+0000", 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [], 2024-09-02T10:02:12.119 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [], 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:02:12.120 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:12.122 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:02:12.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:12.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:12.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:12.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:12.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:12.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:12.398 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:12.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:12.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:12.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 37 16 48834 48818 82.456 78.125 0.0316191 0.0121162 2024-09-02T10:02:12.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:12 smithi151 ceph-mon[16639]: pgmap v327: 27 pgs: 27 active+clean; 2.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:02:12.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2224083667' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:02:12.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:12.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:12 smithi187 ceph-mon[20072]: pgmap v327: 27 pgs: 27 active+clean; 2.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:02:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2224083667' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:02:12.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:12.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:12.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:12 smithi162 ceph-mon[20279]: pgmap v327: 27 pgs: 27 active+clean; 2.6 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:02:12.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2224083667' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:02:12.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:12.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:12.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:12.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:12.780 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:12.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:12.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:12.905 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:12.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:12.936 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:12.936 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:13.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:13.031 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:13.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:13.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:13.137 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:02:13.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:13.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:13.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:02:13.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:13.289 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:13.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:13.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:13.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:13.416 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:13.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:13.435 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 38 16 50134 50118 82.4241 81.25 0.0140007 0.0121256 2024-09-02T10:02:13.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4049843109' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:13.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:13 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4049843109' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:13 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:13.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:13.541 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:13.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:13.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4049843109' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:13.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:13 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:13.615 INFO:tasks.thrashosds.thrasher:j is {'epoch': 111, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:02:03.725575+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:00:05.936786+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '100', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2699999809265137, 'score_stable': 3.2699999809265137, 'optimal_score': 1, 'raw_score_acting': 3.2699999809265137, 'raw_score_stable': 3.2699999809265137, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-02T10:01:28.387897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:14.548239+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:24.147373+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:02:01.278605+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:27.616817+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:03.548446+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:49.271030+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:05.682930+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:36.161551+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:57.585331+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:02:02.732527+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:17.522791+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:49.719022+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:02:13.615 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-02T10:02:13.643 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:13.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:13.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:13.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:13.795 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:13.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:13.897 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:13.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:13.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:14.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:14.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:14.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:14.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:14.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:14.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:14.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:14.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:14.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:14.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:14.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:14.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 39 16 51406 51390 82.3488 79.5 0.00296461 0.0121376 2024-09-02T10:02:14.437 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:14.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:14 smithi151 ceph-mon[16639]: pgmap v328: 27 pgs: 27 active+clean; 2.9 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-02T10:02:14.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:14 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:14 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:02:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:14 smithi151 ceph-mon[16639]: osdmap e112: 12 total, 12 up, 12 in 2024-09-02T10:02:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:14 smithi151 ceph-mon[16639]: osdmap e113: 12 total, 12 up, 12 in 2024-09-02T10:02:14.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:14.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:14 smithi187 ceph-mon[20072]: pgmap v328: 27 pgs: 27 active+clean; 2.9 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-02T10:02:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:14 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:14 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:02:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:14 smithi187 ceph-mon[20072]: osdmap e112: 12 total, 12 up, 12 in 2024-09-02T10:02:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:14 smithi187 ceph-mon[20072]: osdmap e113: 12 total, 12 up, 12 in 2024-09-02T10:02:14.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:14.564 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:14.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:14.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:02:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:14 smithi162 ceph-mon[20279]: pgmap v328: 27 pgs: 27 active+clean; 2.9 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-02T10:02:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:14 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:14 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:02:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:14 smithi162 ceph-mon[20279]: osdmap e112: 12 total, 12 up, 12 in 2024-09-02T10:02:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:14 smithi162 ceph-mon[20279]: osdmap e113: 12 total, 12 up, 12 in 2024-09-02T10:02:14.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:14.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:14.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:14.790 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:14.813 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:14.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:14.915 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:14.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:14.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:15.038 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:15.061 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:15.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:15.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:15.184 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:15.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:15.285 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:15.288 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:15.307 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:15.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:15.322 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:15.409 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:15.431 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:15.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:15.432 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:15.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:02:15.436163+0000 min lat: 0.00101783 max lat: 0.633228 avg lat: 0.0122051 2024-09-02T10:02:15.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:02:15.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 40 16 52368 52352 81.7931 60.125 0.0102119 0.0122051 2024-09-02T10:02:15.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:15.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:15.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:15.556 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:15.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:15.681 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:15.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:15.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:15.810 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:15.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:15.912 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:15.940 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:15.941 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:16.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:16.068 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:16.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:16.087 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:16.087 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:16.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:16.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:16.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:16.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:16.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:16.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:16.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:16.362 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:02:16.362 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:02:16.363 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:16 smithi151 ceph-mon[16639]: pgmap v331: 27 pgs: 27 active+clean; 3.0 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-02T10:02:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:02:16.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:02:16.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:02:16.435 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:02:16.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 41 16 53820 53804 82.0114 90.75 0.00477853 0.0121881 2024-09-02T10:02:16.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:16.467 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:02:16.467 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:02:16.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:16.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:16.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:16.496 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:02:16.496 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:02:16.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:16 smithi187 ceph-mon[20072]: pgmap v331: 27 pgs: 27 active+clean; 3.0 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-02T10:02:16.525 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:02:16.526 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:02:16.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:02:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:02:16.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:16.588 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:02:16.589 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:02:16.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:16 smithi162 ceph-mon[20279]: pgmap v331: 27 pgs: 27 active+clean; 3.0 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-02T10:02:16.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:16.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:16.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:16.623 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:02:16.623 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:02:16.654 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:02:16.654 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:02:16.685 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:02:16.685 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:02:16.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:16.713 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:02:16.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:02:16.729 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:16.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:16.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:16.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:16.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:16.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:16.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:16.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:16.984 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:16.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:17.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:17.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:17.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:17.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:17.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:17.237 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:17.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:17.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:02:17.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:17.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:02:17.361 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:17.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:17.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 42 16 55016 55000 81.8383 74.75 0.00922433 0.012213 2024-09-02T10:02:17.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2709994123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:17.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:17.486 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:17.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2709994123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:17.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:17.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2709994123' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:17.607 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:17.610 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:17.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:17.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:17.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:17.733 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:17.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:17.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:17.860 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:17.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:17.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:17.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:17.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:18.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:18.111 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:18.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:18.213 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:18.234 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:02:18.234 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:02:18.234 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:02:18.234 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:02:18.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:18.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:18.293 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:18.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:18.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:18.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:18.427 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:18 smithi151 ceph-mon[16639]: pgmap v332: 27 pgs: 27 active+clean; 3.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-02T10:02:18.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 43 16 55929 55913 81.2621 57.0625 0.00507844 0.0123009 2024-09-02T10:02:18.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:18.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:18.502 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:18 smithi187 ceph-mon[20072]: pgmap v332: 27 pgs: 27 active+clean; 3.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-02T10:02:18.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:18 smithi162 ceph-mon[20279]: pgmap v332: 27 pgs: 27 active+clean; 3.2 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.35k op/s 2024-09-02T10:02:18.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:18.616 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:02:18.616 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:02:18.616 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-02T10:02:18.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:02:18.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:18.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:18.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:18.768 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:18.769 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:18.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:18.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:18.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:18.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:18.987 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:19.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:19.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:02:19.030 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:19.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:19.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:19.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:19.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:19.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:19.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:19.295 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:19.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:19.346 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:19.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:19.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:19.427 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:19.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 44 16 57329 57313 81.4037 87.5 0.00270796 0.0122784 2024-09-02T10:02:19.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:19.553 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:19.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:19.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:19.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:19.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:19.763 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-02T10:02:19.761+0000 7fbc9d530640 0 --2- 172.21.15.151:0/1926314544 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc98101a40 0x7fbc981979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-02T10:02:19.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:19.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:19.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:19.862 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:02:19.862 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:02:19.862 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:02:19.862 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:02:19.862 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:02:19.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:19.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:19.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:20.024 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:02:20.024 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:02:20.024 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 113, 2024-09-02T10:02:20.024 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:02:20.024 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:02:20.024 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:02:13.749307+0000", 2024-09-02T10:02:20.024 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:02:20.024 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:00:05.936786+0000", 2024-09-02T10:02:20.024 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:02:20.024 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5804032, 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "nodeep-scrub", 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:02:20.025 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 2, 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "jewel", 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:02:20.026 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 11, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 11, 2024-09-02T10:02:20.027 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 11, 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 11, 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 11, 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "108", 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:02:20.028 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "100", 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:02:20.029 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:02:20.030 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:02:20.030 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:02:20.030 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:02:20.030 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:02:20.030 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:02:20.030 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:02:20.030 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:02:20.030 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:02:20.030 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:02:20.030 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:02:20.031 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 3.2699999809265137, 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 3.2699999809265137, 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 3.2699999809265137, 2024-09-02T10:02:20.032 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 3.2699999809265137, 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 2, 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_0", 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:01:28.387897+0000", 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:02:20.033 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-09-02T10:02:20.034 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "96", 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:02:20.035 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:02:20.036 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:02:20.037 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:02:20.038 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.25, 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.25, 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.25, 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.25, 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.039 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 52, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.042 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:20.043 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:02:20.043 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:20.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.043 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.043 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:02:20.043 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:20.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.043 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:02:20.044 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:20.047 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:20.048 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:02:20.049 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:02:20.050 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.051 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 92, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.053 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.054 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.055 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.056 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:02:20.058 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 0, 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.059 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.060 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.061 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.062 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:02:20.063 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.064 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 0, 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.065 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.066 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.067 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.068 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.070 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.071 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.072 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.073 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:02:20.074 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.075 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:20.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.079 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.080 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.081 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.082 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.083 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:20.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:02:20.085 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:02:20.086 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.087 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.087 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.087 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.087 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:02:20.087 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:02:20.087 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.087 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 92, 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.088 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.089 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:02:20.092 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.093 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:02:20.094 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.095 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.096 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:02:20.098 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:02:20.099 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:02:20.099 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:02:20.099 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.099 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.099 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.099 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.099 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:02:20.100 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:02:20.100 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.100 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.100 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:02:20.101 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.102 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.103 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:02:20.105 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:02:20.106 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 103, 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:20.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:02:20.109 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:20.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:02:20.111 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:14.548239+0000", 2024-09-02T10:02:20.112 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:24.147373+0000", 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:02:20.113 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:02:01.278605+0000", 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.114 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:27.616817+0000", 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:03.548446+0000", 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.115 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:49.271030+0000", 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.116 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:05.682930+0000", 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:36.161551+0000", 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:57.585331+0000", 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.118 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:02:02.732527+0000", 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.119 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:17.522791+0000", 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:01:49.719022+0000", 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.120 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [], 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [], 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:02:20.121 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:02:20.122 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:02:20.123 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:02:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:20.125 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:20.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:20.144 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:02:20.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:02:20.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:20.251 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:20.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:20.289 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:20 smithi151 ceph-mon[16639]: pgmap v333: 27 pgs: 27 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:02:20.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/745817601' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:20.290 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1926314544' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:02:20.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:20.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:20.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:20.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 45 16 58526 58510 81.2571 74.8125 0.00671298 0.0123012 2024-09-02T10:02:20.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:20.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:20 smithi162 ceph-mon[20279]: pgmap v333: 27 pgs: 27 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:02:20.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/745817601' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:20.481 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1926314544' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:02:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:20 smithi187 ceph-mon[20072]: pgmap v333: 27 pgs: 27 active+clean; 3.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:02:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/745817601' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1926314544' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:02:20.511 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:20.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:20.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:02:20.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:20.639 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:20.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:20.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:20.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:20.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:20.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:20.865 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:20.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:20.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:20.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:20.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:21.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:21.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:21.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:21.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:21.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:21.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:21.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:21.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:21.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:21.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:21.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:21 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:02:21.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:21 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:02:21.392 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/745817601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:21.393 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:21 smithi162 ceph-mon[20279]: osdmap e114: 12 total, 12 up, 12 in 2024-09-02T10:02:21.394 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:02:21.394 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:02:21.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:02:21.394 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:02:21.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.394 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:1453 2.f e96/92 2:f209e40b:::benchmark_data_smithi187_45882_object1452:head v 96'96, mlcod=96'93)", 2024-09-02T10:02:21.395 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:36.199651+0000", 2024-09-02T10:02:21.395 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 45.190624038000003, 2024-09-02T10:02:21.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064765608000000002, 2024-09-02T10:02:21.396 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.396 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.396 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.396 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.199651+0000", 2024-09-02T10:02:21.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.199651+0000", 2024-09-02T10:02:21.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.199652+0000", 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.99e-07 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.199692+0000", 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0222999999999998e-05 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.264001+0000", 2024-09-02T10:02:21.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064309058000000002 2024-09-02T10:02:21.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.264006+0000", 2024-09-02T10:02:21.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7740000000000004e-06 2024-09-02T10:02:21.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.264052+0000", 2024-09-02T10:02:21.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5889000000000001e-05 2024-09-02T10:02:21.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.264071+0000", 2024-09-02T10:02:21.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9145000000000001e-05 2024-09-02T10:02:21.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.264384+0000", 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000313725 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.264416+0000", 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2094999999999997e-05 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.401 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.402 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.402 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:7827 2.d e97/92 2:bcda9261:::benchmark_data_smithi187_45882_object7826:head v 97'481, mlcod=97'478)", 2024-09-02T10:02:21.402 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.566890+0000", 2024-09-02T10:02:21.402 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 41.823385309000003, 2024-09-02T10:02:21.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11124561400000001, 2024-09-02T10:02:21.402 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.402 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.404 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.404 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.566890+0000", 2024-09-02T10:02:21.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.566890+0000", 2024-09-02T10:02:21.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.566890+0000", 2024-09-02T10:02:21.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:02:21.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.566956+0000", 2024-09-02T10:02:21.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5946999999999997e-05 2024-09-02T10:02:21.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.677581+0000", 2024-09-02T10:02:21.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.110625051 2024-09-02T10:02:21.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.677586+0000", 2024-09-02T10:02:21.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7500000000000003e-06 2024-09-02T10:02:21.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.677628+0000", 2024-09-02T10:02:21.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2539e-05 2024-09-02T10:02:21.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.677642+0000", 2024-09-02T10:02:21.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.36e-05 2024-09-02T10:02:21.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.678097+0000", 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045541000000000001 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.678135+0000", 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7787999999999998e-05 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:7880 2.d e97/92 2:b3511cc8:::benchmark_data_smithi187_45882_object7879:head v 97'482, mlcod=97'479)", 2024-09-02T10:02:21.411 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.677506+0000", 2024-09-02T10:02:21.412 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 41.712768427, 2024-09-02T10:02:21.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083550625000000003, 2024-09-02T10:02:21.412 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.412 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.412 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.412 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.677506+0000", 2024-09-02T10:02:21.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.677506+0000", 2024-09-02T10:02:21.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.677507+0000", 2024-09-02T10:02:21.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8500000000000001e-07 2024-09-02T10:02:21.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.677567+0000", 2024-09-02T10:02:21.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9799000000000001e-05 2024-09-02T10:02:21.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.760541+0000", 2024-09-02T10:02:21.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082973747 2024-09-02T10:02:21.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.760546+0000", 2024-09-02T10:02:21.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1769999999999997e-06 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.760591+0000", 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4948999999999999e-05 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.760604+0000", 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2723e-05 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.761014+0000", 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041035099999999999 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.761057+0000", 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3194e-05 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:8318 2.d e97/92 2:bbd40d80:::benchmark_data_smithi187_45882_object8317:head v 97'502, mlcod=97'499)", 2024-09-02T10:02:21.417 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.955988+0000", 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 41.434286647999997, 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.069387544999999995, 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.955988+0000", 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.955988+0000", 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.955989+0000", 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5099999999999999e-07 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.956049+0000", 2024-09-02T10:02:21.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0049000000000001e-05 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.024830+0000", 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.068780505000000006 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.024836+0000", 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4699999999999999e-06 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.024882+0000", 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6278e-05 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.024899+0000", 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6813999999999999e-05 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.025331+0000", 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00043164499999999998 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.025376+0000", 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5033000000000001e-05 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:8359 2.d e97/92 2:b38a8551:::benchmark_data_smithi187_45882_object8358:head v 97'504, mlcod=97'501)", 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:40.026181+0000", 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 41.364093756999999, 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.118770081, 2024-09-02T10:02:21.422 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.026181+0000", 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.026181+0000", 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.026182+0000", 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7800000000000001e-07 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.026275+0000", 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2788000000000007e-05 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.144327+0000", 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11805257199999999 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.144349+0000", 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2172999999999999e-05 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.144396+0000", 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6298999999999998e-05 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.144417+0000", 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1781999999999999e-05 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.144917+0000", 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049998999999999996 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.144951+0000", 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3899000000000002e-05 2024-09-02T10:02:21.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:8383 2.d e97/92 2:b50fbda7:::benchmark_data_smithi187_45882_object8382:head v 97'505, mlcod=97'502)", 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:40.144289+0000", 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 41.245986406999997, 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.086084335999999997, 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.144289+0000", 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.144289+0000", 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.144289+0000", 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5899999999999995e-07 2024-09-02T10:02:21.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.144313+0000", 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3368000000000001e-05 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.229720+0000", 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085407003999999995 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.229725+0000", 2024-09-02T10:02:21.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1240000000000004e-06 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.229828+0000", 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102855 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.229851+0000", 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3824000000000001e-05 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.230335+0000", 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00048340500000000001 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.230373+0000", 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7997000000000002e-05 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:10387 2.5 e98/92 2:a033a37c:::benchmark_data_smithi187_45882_object10386:head v 98'649, mlcod=98'647)", 2024-09-02T10:02:21.431 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:41.532828+0000", 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.857447393000001, 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.116819113, 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:41.532828+0000", 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:21.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:41.532828+0000", 2024-09-02T10:02:21.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:41.532828+0000", 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3e-07 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:41.532926+0000", 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7835000000000001e-05 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:41.648990+0000", 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11606391200000001 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:41.648995+0000", 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2190000000000003e-06 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:41.649050+0000", 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4413000000000002e-05 2024-09-02T10:02:21.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:41.649071+0000", 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1569999999999998e-05 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:41.649617+0000", 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00054601599999999997 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:41.649647+0000", 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9518000000000001e-05 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "replica_scrub(pg: 1.2,from:0'0,to:0'0,epoch:104/103,start:1:40000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:50.941506+0000", 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.448769148, 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.3952948, 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:02:21.438 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941506+0000", 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941506+0000", 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941510+0000", 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7570000000000002e-06 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941517+0000", 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0249999999999997e-06 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336499+0000", 2024-09-02T10:02:21.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.39498246199999998 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336515+0000", 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.562e-05 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336678+0000", 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016350699999999999 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336718+0000", 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0278999999999998e-05 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336801+0000", 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2150000000000005e-05 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:26805 2.d e105/92 2:b55b17e2:::benchmark_data_smithi187_45882_object26804:head v 105'1630, mlcod=105'1626)", 2024-09-02T10:02:21.442 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:54.377703+0000", 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.012571693000002, 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082770935000000004, 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.377703+0000", 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.377703+0000", 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.377704+0000", 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.16e-07 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.377743+0000", 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9152000000000001e-05 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.459521+0000", 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.081778285000000006 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.459527+0000", 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.04e-06 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.459565+0000", 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8832999999999999e-05 2024-09-02T10:02:21.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.459582+0000", 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6609e-05 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.460447+0000", 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00086546099999999999 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.460474+0000", 2024-09-02T10:02:21.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6738999999999999e-05 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:27094 2.d e105/92 2:b197dc2d:::benchmark_data_smithi187_45882_object27093:head v 105'1648, mlcod=105'1644)", 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:54.667587+0000", 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.722687684, 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.113776259, 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.667587+0000", 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.667587+0000", 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.667588+0000", 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7700000000000004e-07 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.667638+0000", 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0528999999999997e-05 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.780499+0000", 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11286025500000001 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.780505+0000", 2024-09-02T10:02:21.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4450000000000001e-06 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.780553+0000", 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8273999999999999e-05 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.780568+0000", 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4228e-05 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.781336+0000", 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076805600000000003 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.781363+0000", 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7795000000000001e-05 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:27140 2.d e105/92 2:b65e1068:::benchmark_data_smithi187_45882_object27139:head v 105'1649, mlcod=105'1645)", 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:54.780398+0000", 2024-09-02T10:02:21.451 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.609876599, 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.121175281, 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.780398+0000", 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.780398+0000", 2024-09-02T10:02:21.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.780400+0000", 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.677e-06 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.780449+0000", 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9159000000000001e-05 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.900666+0000", 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.120216526 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.900674+0000", 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5629999999999992e-06 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.900719+0000", 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5087999999999999e-05 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.900731+0000", 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2078e-05 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.901542+0000", 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000810705 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:54.901574+0000", 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1485000000000001e-05 2024-09-02T10:02:21.455 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:27708 2.5 e105/92 2:ac5cc291:::benchmark_data_smithi187_45882_object27707:head v 105'1688, mlcod=105'1685)", 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:55.202325+0000", 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.187949967000002, 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.074340895000000004, 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.456 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.202325+0000", 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.202325+0000", 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.202325+0000", 2024-09-02T10:02:21.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.44e-07 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.202425+0000", 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9102000000000001e-05 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.275653+0000", 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.073228371 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.275659+0000", 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7590000000000003e-06 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.275704+0000", 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5439999999999999e-05 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.275719+0000", 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4982e-05 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.276623+0000", 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00090362699999999995 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.276666+0000", 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3069999999999999e-05 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:38709 2.5 e111/92 2:a7eaa9f1:::benchmark_data_smithi187_45882_object38708:head v 111'2402, mlcod=111'2400)", 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:03.965975+0000", 2024-09-02T10:02:21.460 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.424300159000001, 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082400096000000006, 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.965975+0000", 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.965975+0000", 2024-09-02T10:02:21.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.965977+0000", 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.776e-06 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.966029+0000", 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2238999999999999e-05 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.047186+0000", 2024-09-02T10:02:21.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.081157433000000001 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.047209+0000", 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2413999999999999e-05 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.047257+0000", 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8170999999999997e-05 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.047271+0000", 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.376e-05 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.048340+0000", 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010690350000000001 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.048375+0000", 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5268000000000002e-05 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.464 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:40690 2.5 e111/92 2:a18f9fdc:::benchmark_data_smithi187_45882_object40689:head v 111'2526, mlcod=111'2523)", 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:05.603546+0000", 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.786729416, 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11754614400000001, 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.603546+0000", 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.603546+0000", 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.603546+0000", 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6400000000000002e-07 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.603568+0000", 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2330000000000001e-05 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.719735+0000", 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11616620699999999 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.719740+0000", 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5940000000000001e-06 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.719798+0000", 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7544999999999999e-05 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.719824+0000", 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6239000000000001e-05 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.721047+0000", 2024-09-02T10:02:21.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001223065 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.721092+0000", 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4499999999999997e-05 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:40750 2.d e111/92 2:bbc92d25:::benchmark_data_smithi187_45882_object40749:head v 111'2490, mlcod=111'2486)", 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:05.735031+0000", 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.655244292999999, 2024-09-02T10:02:21.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.157739516, 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.735031+0000", 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.735031+0000", 2024-09-02T10:02:21.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.735055+0000", 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3975e-05 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.735075+0000", 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0886e-05 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.891552+0000", 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15647692899999999 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.891557+0000", 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4519999999999999e-06 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.891622+0000", 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5080999999999996e-05 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.891644+0000", 2024-09-02T10:02:21.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2110999999999999e-05 2024-09-02T10:02:21.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.892726+0000", 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001082279 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.892770+0000", 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3803000000000001e-05 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:40752 2.5 e111/92 2:a1795e5d:::benchmark_data_smithi187_45882_object40751:head v 111'2527, mlcod=111'2525)", 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:05.738230+0000", 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.652044423, 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10539425500000001, 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.738230+0000", 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.738230+0000", 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.738232+0000", 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.19e-06 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.738285+0000", 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3523000000000002e-05 2024-09-02T10:02:21.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.842398+0000", 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.104113253 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.842404+0000", 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1440000000000002e-06 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.842448+0000", 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4644999999999997e-05 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.842462+0000", 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3327e-05 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.843585+0000", 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001123708 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.843625+0000", 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9465000000000002e-05 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:54949 2.5 e113/92 2:afa8899f:::benchmark_data_smithi187_45882_object54948:head v 113'3419, mlcod=113'3416)", 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.246304+0000", 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.1439709730000001, 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13970089399999999, 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.479 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.246304+0000", 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.246304+0000", 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.246310+0000", 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.443e-06 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.246358+0000", 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7611999999999997e-05 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.384500+0000", 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.138142343 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.384503+0000", 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8600000000000001e-06 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.384559+0000", 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5475e-05 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.384581+0000", 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1971e-05 2024-09-02T10:02:21.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.385966+0000", 2024-09-02T10:02:21.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013849279999999999 2024-09-02T10:02:21.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.386005+0000", 2024-09-02T10:02:21.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9261999999999998e-05 2024-09-02T10:02:21.483 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:55086 2.f e113/92 2:f2dac436:::benchmark_data_smithi187_45882_object55085:head v 113'3428, mlcod=113'3425)", 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.519264+0000", 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.8710109839999998, 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24656193800000001, 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.519264+0000", 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.519264+0000", 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.519265+0000", 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1100000000000005e-07 2024-09-02T10:02:21.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.519312+0000", 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7227e-05 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.764322+0000", 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.245010432 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.764329+0000", 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4520000000000002e-06 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.764405+0000", 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6408000000000004e-05 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.764421+0000", 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5994999999999998e-05 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.765794+0000", 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013725709999999999 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.765826+0000", 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2042000000000003e-05 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:58859 2.5 e114/92 2:a6f0ee06:::benchmark_data_smithi187_45882_object58858:head v 114'3649, mlcod=114'3646)", 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:20.654103+0000", 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.73617217400000001, 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.070919101999999998, 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.488 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:20.654103+0000", 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:20.654103+0000", 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:20.654103+0000", 2024-09-02T10:02:21.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0200000000000001e-07 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:20.654232+0000", 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012881500000000001 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:20.723424+0000", 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.069191340000000004 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:20.723431+0000", 2024-09-02T10:02:21.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2869999999999999e-06 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:20.723486+0000", 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4880000000000003e-05 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:20.723509+0000", 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2881999999999999e-05 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:20.724988+0000", 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014795959999999999 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:20.725022+0000", 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3599999999999997e-05 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:59319 2.d e114/92 2:bc9d580d:::benchmark_data_smithi187_45882_object59318:head v 114'3631, mlcod=114'3628)", 2024-09-02T10:02:21.492 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:21.122934+0000", 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.267340941, 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.061264691000000003, 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:21.122934+0000", 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:21.122934+0000", 2024-09-02T10:02:21.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:21.122934+0000", 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7999999999999996e-07 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:21.123014+0000", 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9939999999999997e-05 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:21.182635+0000", 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059620290999999999 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:21.182640+0000", 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7389999999999996e-06 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:21.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:21.182684+0000", 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3494000000000001e-05 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:21.182697+0000", 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2806000000000001e-05 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:21.184171+0000", 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001474389 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:21.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:21.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:21.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:21.184199+0000", 2024-09-02T10:02:21.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7552e-05 2024-09-02T10:02:21.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.497 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.497 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:21.497 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:21.497 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:02:21.497 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:21.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:21.498 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:21 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:02:21.498 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:21 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:02:21.498 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/745817601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:21.498 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:21 smithi151 ceph-mon[16639]: osdmap e114: 12 total, 12 up, 12 in 2024-09-02T10:02:21.498 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 46 16 59639 59623 81.0027 69.5625 0.0332659 0.0123378 2024-09-02T10:02:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:21 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:02:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:21 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:02:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/745817601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:21 smithi187 ceph-mon[20072]: osdmap e114: 12 total, 12 up, 12 in 2024-09-02T10:02:21.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:21.624 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:21.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:21.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:21.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:21.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:21.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:21.879 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:21.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:21.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:22.009 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:22.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:22.111 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:22.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:22.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:22.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:22.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:02:22.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:22.267 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:22.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:22.268 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:22.330 INFO:tasks.thrashosds.thrasher:j is {'epoch': 113, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:02:13.749307+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:00:05.936786+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '100', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.2699999809265137, 'score_stable': 3.2699999809265137, 'optimal_score': 1, 'raw_score_acting': 3.2699999809265137, 'raw_score_stable': 3.2699999809265137, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-02T10:01:28.387897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 52, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 92, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:14.548239+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:24.147373+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:02:01.278605+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:27.616817+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:03.548446+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:49.271030+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:05.682930+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:36.161551+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:57.585331+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:02:02.732527+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:17.522791+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:01:49.719022+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:02:22.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:02:22.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:22.396 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:22.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:22.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 47 16 60888 60872 80.94 78.0625 0.0058408 0.0123477 2024-09-02T10:02:22.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:22 smithi151 ceph-mon[16639]: pgmap v335: 27 pgs: 27 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-02T10:02:22.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:22.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:22 smithi187 ceph-mon[20072]: pgmap v335: 27 pgs: 27 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-02T10:02:22.520 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:22.524 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:22.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:22.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:22 smithi162 ceph-mon[20279]: pgmap v335: 27 pgs: 27 active+clean; 3.4 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-02T10:02:22.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:22.650 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:22.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:22.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:22.777 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:22.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:22.863 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:22.877 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:22.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:22.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:22.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:22.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:23.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:23.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:23.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:23.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:23.164 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:23.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:23.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:23.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:23.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:23.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:23.362 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:02:23.363 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:02:23.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.363 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:61609 2.3 2:c78626b8:::benchmark_data_smithi187_45882_object61608:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-02T10:02:23.363 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:23.104777+0000", 2024-09-02T10:02:23.363 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.039172029999999997, 2024-09-02T10:02:23.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039205007, 2024-09-02T10:02:23.363 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:23.363 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:23.363 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:02:23.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 61609 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.104777+0000", 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.104777+0000", 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.104778+0000", 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.64e-07 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.104819+0000", 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1424000000000002e-05 2024-09-02T10:02:23.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.104834+0000", 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5187e-05 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.104843+0000", 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1179999999999992e-06 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.105056+0000", 2024-09-02T10:02:23.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000213177 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.105134+0000", 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7842000000000005e-05 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.105251+0000", 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117328 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:23.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.106802+0000", 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015507089999999999 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.106989+0000", 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000187319 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:61642 2.2 2:4ceec3d1:::benchmark_data_smithi187_45882_object61641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:23.135747+0000", 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0082019280000000007, 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0083115559999999995, 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 61642 2024-09-02T10:02:23.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.370 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:23.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:23.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.135747+0000", 2024-09-02T10:02:23.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:23.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:23.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.135747+0000", 2024-09-02T10:02:23.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:23.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:23.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.135749+0000", 2024-09-02T10:02:23.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8169999999999999e-06 2024-09-02T10:02:23.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:23.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.135803+0000", 2024-09-02T10:02:23.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4486e-05 2024-09-02T10:02:23.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:23.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.135820+0000", 2024-09-02T10:02:23.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6956000000000002e-05 2024-09-02T10:02:23.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:23.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.135829+0000", 2024-09-02T10:02:23.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6920000000000008e-06 2024-09-02T10:02:23.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:23.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.135881+0000", 2024-09-02T10:02:23.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1700999999999998e-05 2024-09-02T10:02:23.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:23.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.135979+0000", 2024-09-02T10:02:23.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7870999999999999e-05 2024-09-02T10:02:23.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:02:23.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.136078+0000", 2024-09-02T10:02:23.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9252000000000004e-05 2024-09-02T10:02:23.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:23.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.138265+0000", 2024-09-02T10:02:23.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021868590000000002 2024-09-02T10:02:23.375 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:23.376 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:23.376 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:23.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.376 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:61651 2.3 2:c9271000:::benchmark_data_smithi187_45882_object61650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-02T10:02:23.376 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:23.142372+0000", 2024-09-02T10:02:23.376 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.001576762, 2024-09-02T10:02:23.376 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017497560000000001, 2024-09-02T10:02:23.377 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:23.377 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:23.377 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:02:23.377 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:23.377 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:23.377 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:23.377 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 61651 2024-09-02T10:02:23.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.378 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:23.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:23.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.142372+0000", 2024-09-02T10:02:23.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:23.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:23.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.142372+0000", 2024-09-02T10:02:23.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:23.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:23.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.142374+0000", 2024-09-02T10:02:23.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9479999999999998e-06 2024-09-02T10:02:23.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:23.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.142415+0000", 2024-09-02T10:02:23.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1301999999999999e-05 2024-09-02T10:02:23.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:23.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.142435+0000", 2024-09-02T10:02:23.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9833000000000001e-05 2024-09-02T10:02:23.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:23.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.142445+0000", 2024-09-02T10:02:23.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5780000000000005e-06 2024-09-02T10:02:23.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:23.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.142512+0000", 2024-09-02T10:02:23.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7581000000000002e-05 2024-09-02T10:02:23.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:23.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.142615+0000", 2024-09-02T10:02:23.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103002 2024-09-02T10:02:23.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:23.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:23.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:02:23.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:23.142749+0000", 2024-09-02T10:02:23.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013316900000000001 2024-09-02T10:02:23.383 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:23.383 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:23.383 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:23.383 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:23.383 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:02:23.383 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 3 2024-09-02T10:02:23.383 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:02:23.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:23.422 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:23.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:23.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:23.436 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 48 16 62049 62033 80.7654 72.5625 0.0440819 0.0123727 2024-09-02T10:02:23.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:23.549 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:23.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:23.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:23.671 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:02:23.671 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:02:23.677 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:23.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:23.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:02:23.700 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:02:23.723 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:02:23.724 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:02:23.749 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:02:23.750 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:02:23.778 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:23.785 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:02:23.786 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:02:23.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:23.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:23.824 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:02:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:02:23.858 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:02:23.858 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:02:23.888 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:02:23.889 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:02:23.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:23.924 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:02:23.924 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:02:23.930 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:23.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:23.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:23.955 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:02:23.955 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:02:23.988 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:02:23.989 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:02:24.023 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:02:24.023 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:02:24.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:24.056 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:24.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:24.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:24.185 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:24.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:24.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:24.290 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:02:24.291 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:02:24.311 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:24.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:24.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:24 smithi162 ceph-mon[20279]: pgmap v336: 27 pgs: 27 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-02T10:02:24.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2539753462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:24.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:24 smithi162 ceph-mon[20279]: osdmap e115: 12 total, 12 up, 12 in 2024-09-02T10:02:24.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:24 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:02:24.413 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:24.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 49 16 63219 63203 80.6094 73.125 0.017969 0.0124003 2024-09-02T10:02:24.441 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:24.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:24.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:02:24.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:24 smithi187 ceph-mon[20072]: pgmap v336: 27 pgs: 27 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-02T10:02:24.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2539753462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:24.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:24 smithi187 ceph-mon[20072]: osdmap e115: 12 total, 12 up, 12 in 2024-09-02T10:02:24.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:24 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:02:24.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:24.568 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:24.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:24.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:24 smithi151 ceph-mon[16639]: pgmap v336: 27 pgs: 27 active+clean; 3.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-02T10:02:24.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2539753462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:24.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:24 smithi151 ceph-mon[16639]: osdmap e115: 12 total, 12 up, 12 in 2024-09-02T10:02:24.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:24 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:02:24.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:24.696 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:24.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:24.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:24.833 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:02:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:24.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:24.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:24.851 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:24.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:24.972 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:24.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:25.002 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-02T10:02:23.754526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3903783456,"num_objects":59560,"num_object_clones":0,"num_object_copies":178680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59558,"num_write_kb":3811712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60570,"ondisk_log_size":60570,"up":81,"acting":81,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":11872692,"kb_used_data":11348580,"kb_used_omap":18,"kb_used_meta":523885,"kb_avail":1113069132,"statfs":{"total":1151940427776,"available":1139782791168,"internally_reserved":0,"allocated":11620945920,"data_stored":11615702980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":536458555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,4,3,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":716636160,"num_objects":10935,"num_object_clones":0,"num_object_copies":32805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10935,"num_write_kb":699840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.687089"},"pg_stats":[{"pgid":"2.f","version":"113'3535","reported_seq":3588,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-02T10:02:19.098156+0000","last_change":"2024-09-02T10:01:30.183309+0000","last_active":"2024-09-02T10:02:19.098156+0000","last_peered":"2024-09-02T10:02:19.098156+0000","last_clean":"2024-09-02T10:02:19.098156+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:02:19.098156+0000","last_undegraded":"2024-09-02T10:02:19.098156+0000","last_fullsized":"2024-09-02T10:02:19.098156+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3535,"log_dups_size":0,"ondisk_log_size":3535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:32.591873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231669760,"num_objects":3535,"num_object_clones":0,"num_object_copies":10605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3535,"num_write_kb":226240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"114'3762","reported_seq":3817,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:21.889672+0000","last_change":"2024-09-02T10:01:30.561460+0000","last_active":"2024-09-02T10:02:21.889672+0000","last_peered":"2024-09-02T10:02:21.889672+0000","last_clean":"2024-09-02T10:02:21.889672+0000","last_became_active":"2024-09-02T10:01:30.560995+0000","last_became_peered":"2024-09-02T10:01:30.560995+0000","last_unstale":"2024-09-02T10:02:21.889672+0000","last_undegraded":"2024-09-02T10:02:21.889672+0000","last_fullsized":"2024-09-02T10:02:21.889672+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3762,"log_dups_size":0,"ondisk_log_size":3762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:55.406791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246546432,"num_objects":3762,"num_object_clones":0,"num_object_copies":11286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3762,"num_write_kb":240768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"114'3770","reported_seq":3823,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:22.927610+0000","last_change":"2024-09-02T10:01:30.560154+0000","last_active":"2024-09-02T10:02:22.927610+0000","last_peered":"2024-09-02T10:02:22.927610+0000","last_clean":"2024-09-02T10:02:22.927610+0000","last_became_active":"2024-09-02T10:01:30.559704+0000","last_became_peered":"2024-09-02T10:01:30.559704+0000","last_unstale":"2024-09-02T10:02:22.927610+0000","last_undegraded":"2024-09-02T10:02:22.927610+0000","last_fullsized":"2024-09-02T10:02:22.927610+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3770,"log_dups_size":0,"ondisk_log_size":3770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:46.256601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247070720,"num_objects":3770,"num_object_clones":0,"num_object_copies":11310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3770,"num_write_kb":241280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"114'3781","reported_seq":3836,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:21.779805+0000","last_change":"2024-09-02T10:01:30.139707+0000","last_active":"2024-09-02T10:02:21.779805+0000","last_peered":"2024-09-02T10:02:21.779805+0000","last_clean":"2024-09-02T10:02:21.779805+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:02:21.779805+0000","last_undegraded":"2024-09-02T10:02:21.779805+0000","last_fullsized":"2024-09-02T10:02:21.779805+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:38.110684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247791616,"num_objects":3781,"num_object_clones":0,"num_object_copies":11343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3781,"num_write_kb":241984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":195,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.148176+0000","last_change":"2024-09-02T10:01:48.974659+0000","last_active":"2024-09-02T10:02:20.148176+0000","last_peered":"2024-09-02T10:02:20.148176+0000","last_clean":"2024-09-02T10:02:20.148176+0000","last_became_active":"2024-09-02T10:01:48.206613+0000","last_became_peered":"2024-09-02T10:01:48.206613+0000","last_unstale":"2024-09-02T10:02:20.148176+0000","last_undegraded":"2024-09-02T10:02:20.148176+0000","last_fullsized":"2024-09-02T10:02:20.148176+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:48.974614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:48.974614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:06.550945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"114'3815","reported_seq":3869,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:21.679894+0000","last_change":"2024-09-02T10:01:30.141172+0000","last_active":"2024-09-02T10:02:21.679894+0000","last_peered":"2024-09-02T10:02:21.679894+0000","last_clean":"2024-09-02T10:02:21.679894+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:02:21.679894+0000","last_undegraded":"2024-09-02T10:02:21.679894+0000","last_fullsized":"2024-09-02T10:02:21.679894+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:32.087935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250019840,"num_objects":3815,"num_object_clones":0,"num_object_copies":11445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3815,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3815,"num_write_kb":244160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":41,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.146916+0000","last_change":"2024-09-02T10:01:50.950127+0000","last_active":"2024-09-02T10:02:20.146916+0000","last_peered":"2024-09-02T10:02:20.146916+0000","last_clean":"2024-09-02T10:02:20.146916+0000","last_became_active":"2024-09-02T10:01:50.764818+0000","last_became_peered":"2024-09-02T10:01:50.764818+0000","last_unstale":"2024-09-02T10:02:20.146916+0000","last_undegraded":"2024-09-02T10:02:20.146916+0000","last_fullsized":"2024-09-02T10:02:20.146916+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:50.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:50.950013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:11.939457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"114'3665","reported_seq":3719,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.569884+0000","last_change":"2024-09-02T10:01:30.560296+0000","last_active":"2024-09-02T10:02:20.569884+0000","last_peered":"2024-09-02T10:02:20.569884+0000","last_clean":"2024-09-02T10:02:20.569884+0000","last_became_active":"2024-09-02T10:01:30.559903+0000","last_became_peered":"2024-09-02T10:01:30.559903+0000","last_unstale":"2024-09-02T10:02:20.569884+0000","last_undegraded":"2024-09-02T10:02:20.569884+0000","last_fullsized":"2024-09-02T10:02:20.569884+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3665,"log_dups_size":0,"ondisk_log_size":3665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:50.499241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240189440,"num_objects":3665,"num_object_clones":0,"num_object_copies":10995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3665,"num_write_kb":234560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,9],"acting":[7,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":201,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.148047+0000","last_change":"2024-09-02T10:01:51.337316+0000","last_active":"2024-09-02T10:02:20.148047+0000","last_peered":"2024-09-02T10:02:20.148047+0000","last_clean":"2024-09-02T10:02:20.148047+0000","last_became_active":"2024-09-02T10:01:50.766826+0000","last_became_peered":"2024-09-02T10:01:50.766826+0000","last_unstale":"2024-09-02T10:02:20.148047+0000","last_undegraded":"2024-09-02T10:02:20.148047+0000","last_fullsized":"2024-09-02T10:02:20.148047+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.337264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.337264+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:08.499993+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"114'3776","reported_seq":3830,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:21.914143+0000","last_change":"2024-09-02T10:01:30.560301+0000","last_active":"2024-09-02T10:02:21.914143+0000","last_peered":"2024-09-02T10:02:21.914143+0000","last_clean":"2024-09-02T10:02:21.914143+0000","last_became_active":"2024-09-02T10:01:30.559997+0000","last_became_peered":"2024-09-02T10:01:30.559997+0000","last_unstale":"2024-09-02T10:02:21.914143+0000","last_undegraded":"2024-09-02T10:02:21.914143+0000","last_fullsized":"2024-09-02T10:02:21.914143+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:34.011670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247463936,"num_objects":3776,"num_object_clones":0,"num_object_copies":11328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3776,"num_write_kb":241664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":41,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.155319+0000","last_change":"2024-09-02T10:01:50.898052+0000","last_active":"2024-09-02T10:02:20.155319+0000","last_peered":"2024-09-02T10:02:20.155319+0000","last_clean":"2024-09-02T10:02:20.155319+0000","last_became_active":"2024-09-02T10:01:50.765441+0000","last_became_peered":"2024-09-02T10:01:50.765441+0000","last_unstale":"2024-09-02T10:02:20.155319+0000","last_undegraded":"2024-09-02T10:02:20.155319+0000","last_fullsized":"2024-09-02T10:02:20.155319+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:50.897996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:50.897996+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:00.393021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"114'3839","reported_seq":3893,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:21.722120+0000","last_change":"2024-09-02T10:01:30.140642+0000","last_active":"2024-09-02T10:02:21.722120+0000","last_peered":"2024-09-02T10:02:21.722120+0000","last_clean":"2024-09-02T10:02:21.722120+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:02:21.722120+0000","last_undegraded":"2024-09-02T10:02:21.722120+0000","last_fullsized":"2024-09-02T10:02:21.722120+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3839,"log_dups_size":0,"ondisk_log_size":3839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:54.429959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251592704,"num_objects":3839,"num_object_clones":0,"num_object_copies":11517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3839,"num_write_kb":245696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":65,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.158258+0000","last_change":"2024-09-02T10:01:51.173117+0000","last_active":"2024-09-02T10:02:20.158258+0000","last_peered":"2024-09-02T10:02:20.158258+0000","last_clean":"2024-09-02T10:02:20.158258+0000","last_became_active":"2024-09-02T10:01:50.778902+0000","last_became_peered":"2024-09-02T10:01:50.778902+0000","last_unstale":"2024-09-02T10:02:20.158258+0000","last_undegraded":"2024-09-02T10:02:20.158258+0000","last_fullsized":"2024-09-02T10:02:20.158258+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.173075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.173075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:52.401881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"114'3767","reported_seq":3821,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:22.519657+0000","last_change":"2024-09-02T10:01:30.560172+0000","last_active":"2024-09-02T10:02:22.519657+0000","last_peered":"2024-09-02T10:02:22.519657+0000","last_clean":"2024-09-02T10:02:22.519657+0000","last_became_active":"2024-09-02T10:01:30.559685+0000","last_became_peered":"2024-09-02T10:01:30.559685+0000","last_unstale":"2024-09-02T10:02:22.519657+0000","last_undegraded":"2024-09-02T10:02:22.519657+0000","last_fullsized":"2024-09-02T10:02:22.519657+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3767,"log_dups_size":0,"ondisk_log_size":3767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:57.437147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246874112,"num_objects":3767,"num_object_clones":0,"num_object_copies":11301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3767,"num_write_kb":241088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":41,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.155335+0000","last_change":"2024-09-02T10:01:51.898421+0000","last_active":"2024-09-02T10:02:20.155335+0000","last_peered":"2024-09-02T10:02:20.155335+0000","last_clean":"2024-09-02T10:02:20.155335+0000","last_became_active":"2024-09-02T10:01:50.765475+0000","last_became_peered":"2024-09-02T10:01:50.765475+0000","last_unstale":"2024-09-02T10:02:20.155335+0000","last_undegraded":"2024-09-02T10:02:20.155335+0000","last_fullsized":"2024-09-02T10:02:20.155335+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.898350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.898350+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:53.457481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"113'3515","reported_seq":3568,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-02T10:02:19.089603+0000","last_change":"2024-09-02T10:01:30.183497+0000","last_active":"2024-09-02T10:02:19.089603+0000","last_peered":"2024-09-02T10:02:19.089603+0000","last_clean":"2024-09-02T10:02:19.089603+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:02:19.089603+0000","last_undegraded":"2024-09-02T10:02:19.089603+0000","last_fullsized":"2024-09-02T10:02:19.089603+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3515,"log_dups_size":0,"ondisk_log_size":3515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:40.071109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230359040,"num_objects":3515,"num_object_clones":0,"num_object_copies":10545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3515,"num_write_kb":224960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":41,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.155333+0000","last_change":"2024-09-02T10:01:52.926252+0000","last_active":"2024-09-02T10:02:20.155333+0000","last_peered":"2024-09-02T10:02:20.155333+0000","last_clean":"2024-09-02T10:02:20.155333+0000","last_became_active":"2024-09-02T10:01:50.774645+0000","last_became_peered":"2024-09-02T10:01:50.774645+0000","last_unstale":"2024-09-02T10:02:20.155333+0000","last_undegraded":"2024-09-02T10:02:20.155333+0000","last_fullsized":"2024-09-02T10:02:20.155333+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:52.926205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:52.926205+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:11.347737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"114'3752","reported_seq":3806,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:22.510881+0000","last_change":"2024-09-02T10:01:30.560649+0000","last_active":"2024-09-02T10:02:22.510881+0000","last_peered":"2024-09-02T10:02:22.510881+0000","last_clean":"2024-09-02T10:02:22.510881+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:02:22.510881+0000","last_undegraded":"2024-09-02T10:02:22.510881+0000","last_fullsized":"2024-09-02T10:02:22.510881+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3752,"log_dups_size":0,"ondisk_log_size":3752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:34.163733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":245891072,"num_objects":3752,"num_object_clones":0,"num_object_copies":11256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3752,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3752,"num_write_kb":240128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":133,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.150396+0000","last_change":"2024-09-02T10:01:51.746769+0000","last_active":"2024-09-02T10:02:20.150396+0000","last_peered":"2024-09-02T10:02:20.150396+0000","last_clean":"2024-09-02T10:02:20.150396+0000","last_became_active":"2024-09-02T10:01:50.768028+0000","last_became_peered":"2024-09-02T10:01:50.768028+0000","last_unstale":"2024-09-02T10:02:20.150396+0000","last_undegraded":"2024-09-02T10:02:20.150396+0000","last_fullsized":"2024-09-02T10:02:20.150396+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.746691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.746691+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:06.196416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"114'3819","reported_seq":3874,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:21.936361+0000","last_change":"2024-09-02T10:01:30.560449+0000","last_active":"2024-09-02T10:02:21.936361+0000","last_peered":"2024-09-02T10:02:21.936361+0000","last_clean":"2024-09-02T10:02:21.936361+0000","last_became_active":"2024-09-02T10:01:30.559962+0000","last_became_peered":"2024-09-02T10:01:30.559962+0000","last_unstale":"2024-09-02T10:02:21.936361+0000","last_undegraded":"2024-09-02T10:02:21.936361+0000","last_fullsized":"2024-09-02T10:02:21.936361+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:55.111495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250281984,"num_objects":3819,"num_object_clones":0,"num_object_copies":11457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3819,"num_write_kb":244416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"114'3664","reported_seq":3718,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.596524+0000","last_change":"2024-09-02T10:01:30.141564+0000","last_active":"2024-09-02T10:02:20.596524+0000","last_peered":"2024-09-02T10:02:20.596524+0000","last_clean":"2024-09-02T10:02:20.596524+0000","last_became_active":"2024-09-02T10:01:30.141263+0000","last_became_peered":"2024-09-02T10:01:30.141263+0000","last_unstale":"2024-09-02T10:02:20.596524+0000","last_undegraded":"2024-09-02T10:02:20.596524+0000","last_fullsized":"2024-09-02T10:02:20.596524+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3664,"log_dups_size":0,"ondisk_log_size":3664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:56.004849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240123904,"num_objects":3664,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3664,"num_write_kb":234496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":41,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.149400+0000","last_change":"2024-09-02T10:01:51.614379+0000","last_active":"2024-09-02T10:02:20.149400+0000","last_peered":"2024-09-02T10:02:20.149400+0000","last_clean":"2024-09-02T10:02:20.149400+0000","last_became_active":"2024-09-02T10:01:50.781305+0000","last_became_peered":"2024-09-02T10:01:50.781305+0000","last_unstale":"2024-09-02T10:02:20.149400+0000","last_undegraded":"2024-09-02T10:02:20.149400+0000","last_fullsized":"2024-09-02T10:02:20.149400+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.614334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.614334+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:04.278629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"114'3852","reported_seq":3905,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:22.502127+0000","last_change":"2024-09-02T10:01:30.560354+0000","last_active":"2024-09-02T10:02:22.502127+0000","last_peered":"2024-09-02T10:02:22.502127+0000","last_clean":"2024-09-02T10:02:22.502127+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:02:22.502127+0000","last_undegraded":"2024-09-02T10:02:22.502127+0000","last_fullsized":"2024-09-02T10:02:22.502127+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3852,"log_dups_size":0,"ondisk_log_size":3852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:48.623221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252444672,"num_objects":3852,"num_object_clones":0,"num_object_copies":11556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3852,"num_write_kb":246528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":201,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.148045+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:02:20.148045+0000","last_peered":"2024-09-02T10:02:20.148045+0000","last_clean":"2024-09-02T10:02:20.148045+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:02:20.148045+0000","last_undegraded":"2024-09-02T10:02:20.148045+0000","last_fullsized":"2024-09-02T10:02:20.148045+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:20.037124+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"114'3728","reported_seq":3783,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:21.789870+0000","last_change":"2024-09-02T10:01:30.560175+0000","last_active":"2024-09-02T10:02:21.789870+0000","last_peered":"2024-09-02T10:02:21.789870+0000","last_clean":"2024-09-02T10:02:21.789870+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:02:21.789870+0000","last_undegraded":"2024-09-02T10:02:21.789870+0000","last_fullsized":"2024-09-02T10:02:21.789870+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3728,"log_dups_size":0,"ondisk_log_size":3728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:38.707848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244318208,"num_objects":3728,"num_object_clones":0,"num_object_copies":11184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3728,"num_write_kb":238592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":48,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-09-02T10:02:20.145643+0000","last_change":"2024-09-02T10:01:50.947414+0000","last_active":"2024-09-02T10:02:20.145643+0000","last_peered":"2024-09-02T10:02:20.145643+0000","last_clean":"2024-09-02T10:02:20.145643+0000","last_became_active":"2024-09-02T10:01:50.763116+0000","last_became_peered":"2024-09-02T10:01:50.763116+0000","last_unstale":"2024-09-02T10:02:20.145643+0000","last_undegraded":"2024-09-02T10:02:20.145643+0000","last_fullsized":"2024-09-02T10:02:20.145643+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:50.947356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:50.947356+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:20.448518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"113'3518","reported_seq":3570,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-09-02T10:02:19.081115+0000","last_change":"2024-09-02T10:01:30.140619+0000","last_active":"2024-09-02T10:02:19.081115+0000","last_peered":"2024-09-02T10:02:19.081115+0000","last_clean":"2024-09-02T10:02:19.081115+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:02:19.081115+0000","last_undegraded":"2024-09-02T10:02:19.081115+0000","last_fullsized":"2024-09-02T10:02:19.081115+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":3518,"log_dups_size":0,"ondisk_log_size":3518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:39.195715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230555648,"num_objects":3518,"num_object_clones":0,"num_object_copies":10554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3518,"num_write_kb":225152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":3903193088,"num_objects":59558,"num_object_clones":0,"num_object_copies":178674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59558,"num_write_kb":3811712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11765153792,"data_stored":11765153792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59558,"ondisk_log_size":59558,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383707,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":746136,"kb_used_data":706628,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":92999016,"statfs":{"total":95995035648,"available":95230992384,"internally_reserved":0,"allocated":723587072,"data_stored":723149892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612640,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":752664,"kb_used_data":713156,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":92992488,"statfs":{"total":95995035648,"available":95224307712,"internally_reserved":0,"allocated":730271744,"data_stored":729834572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":40434112},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841573,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506772,"kb_used_data":1453380,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92238380,"statfs":{"total":95995035648,"available":94452101120,"internally_reserved":0,"allocated":1488261120,"data_stored":1487823948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":915080,"kb_used_data":883896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92830072,"statfs":{"total":95995035648,"available":95057993728,"internally_reserved":0,"allocated":905109504,"data_stored":904676928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299437,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1217480,"kb_used_data":1164088,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92527672,"statfs":{"total":95995035648,"available":94748336128,"internally_reserved":0,"allocated":1192026112,"data_stored":1191593536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528370,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":513044,"kb_used_data":486020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93232108,"statfs":{"total":95995035648,"available":95469678592,"internally_reserved":0,"allocated":497684480,"data_stored":497247308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724599,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509916,"kb_used_data":482892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93235236,"statfs":{"total":95995035648,"available":95472881664,"internally_reserved":0,"allocated":494481408,"data_stored":494040684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953532,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696584,"kb_used_data":661240,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93048568,"statfs":{"total":95995035648,"available":95281733632,"internally_reserved":0,"allocated":677109760,"data_stored":676677184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501468,"kb_used_data":1448076,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92243684,"statfs":{"total":95995035648,"available":94457532416,"internally_reserved":0,"allocated":1482829824,"data_stored":1482389100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1015840,"kb_used_data":958284,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92729312,"statfs":{"total":95995035648,"available":94954815488,"internally_reserved":0,"allocated":981282816,"data_stored":980842092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1260884,"kb_used_data":1207492,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92484268,"statfs":{"total":95995035648,"available":94703890432,"internally_reserved":0,"allocated":1236471808,"data_stored":1236034636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1236824,"kb_used_data":1183428,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92508328,"statfs":{"total":95995035648,"available":94728527872,"internally_reserved":0,"allocated":1211830272,"data_stored":1211393100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1217724416,"data_stored":1217724416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1251934208,"data_stored":1251934208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":994508800,"data_stored":994508800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1507328000,"data_stored":1507328000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":689569792,"data_stored":689569792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":501612544,"data_stored":501612544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":502857728,"data_stored":502857728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1205927936,"data_stored":1205927936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":922025984,"data_stored":922025984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1501429760,"data_stored":1501429760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":735051776,"data_stored":735051776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":735182848,"data_stored":735182848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:02:25.003 INFO:tasks.thrashosds.thrasher:Setting 2.0 to [10, 6, 2, 8, 5, 1] 2024-09-02T10:02:25.004 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.0', '10', '6', '2', '8', '5', '1'] 2024-09-02T10:02:25.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 2.0 10 6 2 8 5 1 2024-09-02T10:02:25.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:25.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:25.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:25.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:25.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:25.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:25.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:25 smithi162 ceph-mon[20279]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:02:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:25 smithi162 ceph-mon[20279]: osdmap e116: 12 total, 12 up, 12 in 2024-09-02T10:02:25.367 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:25.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:25.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 50 16 64605 64589 80.7296 86.625 0.00531931 0.0123796 2024-09-02T10:02:25.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:25.492 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:25 smithi151 ceph-mon[16639]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:02:25.493 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:25 smithi151 ceph-mon[16639]: osdmap e116: 12 total, 12 up, 12 in 2024-09-02T10:02:25.493 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:25.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:25.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:25.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:25 smithi187 ceph-mon[20072]: from='client.14850 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:02:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:25 smithi187 ceph-mon[20072]: osdmap e116: 12 total, 12 up, 12 in 2024-09-02T10:02:25.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:25.620 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:25.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:25.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:25.623 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:25.653 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:25.722 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:25.750 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:25.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:25.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:25.850 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:25.850 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:25.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:25.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:25.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:25.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:25.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:26.007 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:26.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:26.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:26.133 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:26.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:26.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:26.259 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:26.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:26.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:26.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:26.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:26 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:02:26.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:26 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:02:26.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:26 smithi151 ceph-mon[16639]: pgmap v339: 27 pgs: 27 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-02T10:02:26.385 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:26.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:26.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:26.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 51 16 65929 65913 80.7691 82.75 0.0024528 0.0123713 2024-09-02T10:02:26.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:26 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:02:26.600 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:26 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:02:26.600 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:26 smithi187 ceph-mon[20072]: pgmap v339: 27 pgs: 27 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-02T10:02:26.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:26 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:02:26.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:26 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:02:26.601 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:26 smithi162 ceph-mon[20279]: pgmap v339: 27 pgs: 27 active+clean; 3.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-02T10:02:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:26.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:26.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:26.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:02:26.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:26.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:26.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:26.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:26.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:26.866 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:26.956 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:26.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:26.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:26.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:26.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:27.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:27.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:27.124 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:27.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:27.232 INFO:teuthology.orchestra.run.smithi151.stderr:set 2.0 pg_upmap_items mapping to [10->6,2->8,5->1] 2024-09-02T10:02:27.248 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:27.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:27.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:02:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:02:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:27 smithi162 ceph-mon[20279]: osdmap e117: 12 total, 12 up, 12 in 2024-09-02T10:02:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1001172544' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [10, 6, 2, 8, 5, 1]} : dispatch 2024-09-02T10:02:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:27 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [10, 6, 2, 8, 5, 1]} : dispatch 2024-09-02T10:02:27.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:27.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:27.372 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:27.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 52 16 67013 66997 80.5186 67.75 0.0174239 0.0123995 2024-09-02T10:02:27.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:27.476 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:27.476 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:27.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:27.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:02:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:02:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:27 smithi187 ceph-mon[20072]: osdmap e117: 12 total, 12 up, 12 in 2024-09-02T10:02:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1001172544' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [10, 6, 2, 8, 5, 1]} : dispatch 2024-09-02T10:02:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:27 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [10, 6, 2, 8, 5, 1]} : dispatch 2024-09-02T10:02:27.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:02:27.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:02:27.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:02:27.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:27 smithi151 ceph-mon[16639]: osdmap e117: 12 total, 12 up, 12 in 2024-09-02T10:02:27.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1001172544' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [10, 6, 2, 8, 5, 1]} : dispatch 2024-09-02T10:02:27.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:27 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [10, 6, 2, 8, 5, 1]} : dispatch 2024-09-02T10:02:27.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:27.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:27.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:27.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:27.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:27.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:27.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:27.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:27.872 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:27.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:27.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:27.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:27.980 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:27.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:27.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:27.995 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:28.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:28.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:28.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:28.219 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:28.230 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:02:28.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:28.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:28.242 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:28.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:02:28.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:28.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:28.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:28.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:28.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 53 16 67250 67234 79.2788 14.8125 0.00281625 0.0124398 2024-09-02T10:02:28.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:28 smithi151 ceph-mon[16639]: pgmap v341: 27 pgs: 27 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-02T10:02:28.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:28 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [10, 6, 2, 8, 5, 1]}]': finished 2024-09-02T10:02:28.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:28 smithi151 ceph-mon[16639]: osdmap e118: 12 total, 12 up, 12 in 2024-09-02T10:02:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2630482960' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:02:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:28 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:02:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:28 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:02:28.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:28.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:28.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:28.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:28.505 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:28.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:28 smithi187 ceph-mon[20072]: pgmap v341: 27 pgs: 27 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-02T10:02:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:28 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [10, 6, 2, 8, 5, 1]}]': finished 2024-09-02T10:02:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:28 smithi187 ceph-mon[20072]: osdmap e118: 12 total, 12 up, 12 in 2024-09-02T10:02:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2630482960' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:02:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:28 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:02:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:28 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:02:28.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:28.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:28 smithi162 ceph-mon[20279]: pgmap v341: 27 pgs: 27 active+clean; 3.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 90 MiB/s wr, 1.43k op/s 2024-09-02T10:02:28.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:28 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.0", "id": [10, 6, 2, 8, 5, 1]}]': finished 2024-09-02T10:02:28.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:28 smithi162 ceph-mon[20279]: osdmap e118: 12 total, 12 up, 12 in 2024-09-02T10:02:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2630482960' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:02:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:28 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:02:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:28 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:02:28.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:28.612 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:28.658 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:02:28.658 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:02:28.679 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:02:28.679 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:02:28.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:02:28.701 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:02:28.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:28.723 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:02:28.723 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:02:28.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:28.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:28.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:28.755 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:02:28.755 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:02:28.777 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:02:28.777 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:02:28.797 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:02:28.797 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:02:28.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:28.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:28.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:28.949 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:02:28.949 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:02:28.949 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:02:28.949 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:02:28.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:28.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:28.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:29.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:29.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:29.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:29.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:29.237 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:29.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:29.239 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:29.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:29.362 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:29.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:29.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 54 16 67273 67257 77.8372 1.4375 0.00272514 0.0124442 2024-09-02T10:02:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:29 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2630482960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:02:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:29 smithi151 ceph-mon[16639]: osdmap e119: 12 total, 12 up, 12 in 2024-09-02T10:02:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/196566569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:29.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:29.490 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:29.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:29 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2630482960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:02:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:29 smithi187 ceph-mon[20072]: osdmap e119: 12 total, 12 up, 12 in 2024-09-02T10:02:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/196566569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:29.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:29.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:29 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2630482960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:02:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:29 smithi162 ceph-mon[20279]: osdmap e119: 12 total, 12 up, 12 in 2024-09-02T10:02:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/196566569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:29.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:02:29.622 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:02:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:29.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:29.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:29.655 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:02:29.656 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:02:29.690 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:02:29.690 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:02:29.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:02:29.724 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:02:29.724 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:02:29.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:29.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:02:29.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:02:29.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:29.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:29.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:29.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:29.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:29.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:29.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:29.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:30.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:30.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:30.018 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:30.018 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:30.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:30.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:30.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:30.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:30.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:30.276 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:30.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:30.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:30.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:30.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:30.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:30 smithi162 ceph-mon[20279]: pgmap v344: 27 pgs: 27 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:02:30.428 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:30 smithi162 ceph-mon[20279]: osdmap e120: 12 total, 12 up, 12 in 2024-09-02T10:02:30.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 55 16 68372 68356 77.6707 68.6875 0.0504669 0.0128672 2024-09-02T10:02:30.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:30 smithi151 ceph-mon[16639]: pgmap v344: 27 pgs: 27 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:02:30.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:30 smithi151 ceph-mon[16639]: osdmap e120: 12 total, 12 up, 12 in 2024-09-02T10:02:30.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:30 smithi187 ceph-mon[20072]: pgmap v344: 27 pgs: 27 active+clean; 4.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:02:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:30 smithi187 ceph-mon[20072]: osdmap e120: 12 total, 12 up, 12 in 2024-09-02T10:02:30.526 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:30.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:30.548 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:02:30.548 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:02:30.548 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:02:30.548 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:02:30.548 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:02:30.628 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:30.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:30.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:30.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:30.779 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:30.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:30.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:30.909 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:30.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:31.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:31.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:31.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:31.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:31.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:02:31.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:31.158 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:31.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:31.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:31.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:31.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:31.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:31.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:31.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:31.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:31.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:31.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:31.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 56 16 69380 69364 77.4086 63 0.00586859 0.0129126 2024-09-02T10:02:31.461 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:31.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:31.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:31.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:31.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:31.676 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:31.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:31.677 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:31.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:31.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:31.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:31.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:02:31.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:31.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:31.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:32.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:32.067 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:32.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:32.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:32.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:32.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:32.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:32.200 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:32.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:32.320 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:32.321 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:32.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:32.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 57 16 70403 70387 77.1722 63.9375 0.00302395 0.0129513 2024-09-02T10:02:32.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:32.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:32.490 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:32 smithi162 ceph-mon[20279]: pgmap v346: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s; 4139/201003 objects misplaced (2.059%); 858 KiB/s, 13 objects/s recovering 2024-09-02T10:02:32.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:02:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:32 smithi187 ceph-mon[20072]: pgmap v346: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s; 4139/201003 objects misplaced (2.059%); 858 KiB/s, 13 objects/s recovering 2024-09-02T10:02:32.520 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:32 smithi151 ceph-mon[16639]: pgmap v346: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 71 MiB/s wr, 1.14k op/s; 4139/201003 objects misplaced (2.059%); 858 KiB/s, 13 objects/s recovering 2024-09-02T10:02:32.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:32.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:32.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:32.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:32.707 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:32.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:32.742 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:02:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:02:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:02:32.742 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:1129 2.7 2:eb0d96c7:::benchmark_data_smithi187_45882_object1128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:35.893627+0000", 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.845103315999999, 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20029433599999999, 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1129 2024-09-02T10:02:32.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893627+0000", 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893627+0000", 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893628+0000", 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1200000000000001e-07 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893695+0000", 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7444e-05 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893707+0000", 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2024999999999999e-05 2024-09-02T10:02:32.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893713+0000", 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3650000000000003e-06 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893761+0000", 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8760000000000001e-05 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893846+0000", 2024-09-02T10:02:32.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4368000000000003e-05 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893930+0000", 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4179999999999997e-05 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.894256+0000", 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032631199999999999 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.908716+0000", 2024-09-02T10:02:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014459929999999999 2024-09-02T10:02:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093866+0000", 2024-09-02T10:02:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18514956099999999 2024-09-02T10:02:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093896+0000", 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0238000000000001e-05 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093921+0000", 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5241000000000001e-05 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:1142 2.7 2:e2cb33eb:::benchmark_data_smithi187_45882_object1141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:35.906559+0000", 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.832171060999997, 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20112611999999999, 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1142 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906559+0000", 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906559+0000", 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906560+0000", 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2780000000000001e-06 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906583+0000", 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2288000000000001e-05 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906594+0000", 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1235e-05 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906599+0000", 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.006e-06 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906640+0000", 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0808e-05 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906719+0000", 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9271999999999996e-05 2024-09-02T10:02:32.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906808+0000", 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9093999999999998e-05 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907153+0000", 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034475 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093837+0000", 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18668417100000001 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.107626+0000", 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013788761 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.107659+0000", 2024-09-02T10:02:32.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3714000000000003e-05 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.107685+0000", 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5743000000000001e-05 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:1129 2.7 e96/92 ondisk, result = 0)", 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:35.907472+0000", 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.831257811, 2024-09-02T10:02:32.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18645468200000001, 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907472+0000", 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907472+0000", 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907473+0000", 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8400000000000004e-07 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907476+0000", 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1769999999999998e-06 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093797+0000", 2024-09-02T10:02:32.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.186320872 2024-09-02T10:02:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093803+0000", 2024-09-02T10:02:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9880000000000001e-06 2024-09-02T10:02:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093857+0000", 2024-09-02T10:02:32.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4190000000000001e-05 2024-09-02T10:02:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093862+0000", 2024-09-02T10:02:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5290000000000002e-06 2024-09-02T10:02:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093927+0000", 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5341999999999999e-05 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:8369 2.7 2:e9505c5a:::benchmark_data_smithi187_45882_object8368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:40.031876+0000", 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.706854317000001, 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10717694899999999, 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.761 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 8369 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.031876+0000", 2024-09-02T10:02:32.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.031876+0000", 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.031877+0000", 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2600000000000001e-07 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.031933+0000", 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6394000000000003e-05 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.031943+0000", 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0091e-05 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.031948+0000", 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9660000000000004e-06 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.031986+0000", 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7273e-05 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.032056+0000", 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0425000000000004e-05 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.032136+0000", 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0055999999999994e-05 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.032558+0000", 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042177799999999999 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.032893+0000", 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033532800000000003 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.138994+0000", 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.106100954 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.139027+0000", 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3169e-05 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.139053+0000", 2024-09-02T10:02:32.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5588999999999998e-05 2024-09-02T10:02:32.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.768 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:8376 2.2 e97/92 2:42cf9cd7:::benchmark_data_smithi187_45882_object8375:head v 97'548, mlcod=97'545)", 2024-09-02T10:02:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:40.037608+0000", 2024-09-02T10:02:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.701122415999997, 2024-09-02T10:02:32.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18157076599999999, 2024-09-02T10:02:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.037608+0000", 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.037608+0000", 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.037608+0000", 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0999999999999998e-07 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.037636+0000", 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7625999999999999e-05 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.218489+0000", 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18085346899999999 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.218495+0000", 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6840000000000001e-06 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.218555+0000", 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9864000000000001e-05 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.218575+0000", 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0304000000000001e-05 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.219120+0000", 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00054517499999999996 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.219179+0000", 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8034e-05 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.774 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:9161 2.7 2:e5b7fd38:::benchmark_data_smithi187_45882_object9160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:40.619894+0000", 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 52.118835955000002, 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15764898999999999, 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.776 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 9161 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.619894+0000", 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.619894+0000", 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.619895+0000", 2024-09-02T10:02:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4600000000000003e-07 2024-09-02T10:02:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.619940+0000", 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5118999999999999e-05 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.619951+0000", 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0543999999999999e-05 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.619956+0000", 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9830000000000004e-06 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.619994+0000", 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8012000000000001e-05 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.620067+0000", 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2806999999999996e-05 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.620148+0000", 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1315000000000003e-05 2024-09-02T10:02:32.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.620650+0000", 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050230299999999995 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.621108+0000", 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045783899999999998 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.777484+0000", 2024-09-02T10:02:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.1563764 2024-09-02T10:02:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.777519+0000", 2024-09-02T10:02:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4202999999999998e-05 2024-09-02T10:02:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.777543+0000", 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4519e-05 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:11303 2.7 2:eeb591e6:::benchmark_data_smithi187_45882_object11302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:42.313985+0000", 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.424745299000001, 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.117688217, 2024-09-02T10:02:32.783 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 11303 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.313985+0000", 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.313985+0000", 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.313986+0000", 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2600000000000001e-07 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.314006+0000", 2024-09-02T10:02:32.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0571e-05 2024-09-02T10:02:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.314017+0000", 2024-09-02T10:02:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0815000000000001e-05 2024-09-02T10:02:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.314022+0000", 2024-09-02T10:02:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2700000000000004e-06 2024-09-02T10:02:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.314097+0000", 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4144e-05 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.314171+0000", 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4177000000000005e-05 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.314248+0000", 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6948999999999998e-05 2024-09-02T10:02:32.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.314657+0000", 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040954099999999997 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.315099+0000", 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00044177300000000001 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.431609+0000", 2024-09-02T10:02:32.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11651033700000001 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.431643+0000", 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3708000000000003e-05 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.431673+0000", 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0105999999999999e-05 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:11788 2.7 2:ee0a4980:::benchmark_data_smithi187_45882_object11787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:42.749811+0000", 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.988919068000001, 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13543021099999999, 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 11788 2024-09-02T10:02:32.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.749811+0000", 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.749811+0000", 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.749812+0000", 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2799999999999995e-07 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.749835+0000", 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2955999999999998e-05 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.749846+0000", 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1088e-05 2024-09-02T10:02:32.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.749870+0000", 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3986e-05 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.749918+0000", 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7694000000000001e-05 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.749995+0000", 2024-09-02T10:02:32.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7583000000000006e-05 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.750074+0000", 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8959000000000001e-05 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.750545+0000", 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00047059100000000002 2024-09-02T10:02:32.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.884924+0000", 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13437923600000001 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.885152+0000", 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022833099999999999 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.885209+0000", 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6325e-05 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.885241+0000", 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2634e-05 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:11809 2.2 e98/92 2:4494e80c:::benchmark_data_smithi187_45882_object11808:head v 98'776, mlcod=98'772)", 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:42.778399+0000", 2024-09-02T10:02:32.797 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.960331132, 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.112277814, 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.778399+0000", 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.778399+0000", 2024-09-02T10:02:32.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.778400+0000", 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0219999999999999e-06 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.778425+0000", 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4816000000000001e-05 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.890103+0000", 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.111678256 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.890108+0000", 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.454e-06 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.890176+0000", 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8283999999999999e-05 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.890189+0000", 2024-09-02T10:02:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2998e-05 2024-09-02T10:02:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.890646+0000", 2024-09-02T10:02:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045710400000000001 2024-09-02T10:02:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.890677+0000", 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0880000000000002e-05 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:11812 2.2 e98/92 2:463cda4e:::benchmark_data_smithi187_45882_object11811:head v 98'777, mlcod=98'773)", 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:42.782801+0000", 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.955929156000003, 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.112367616, 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.802 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.782801+0000", 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.782801+0000", 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.782802+0000", 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3e-07 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.782823+0000", 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1097000000000001e-05 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.894512+0000", 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.1116888 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.894517+0000", 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9089999999999997e-06 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.894571+0000", 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3599e-05 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.894590+0000", 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8919e-05 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.895141+0000", 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00055061800000000001 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.895169+0000", 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8044000000000001e-05 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:11871 2.7 2:e7fd85b3:::benchmark_data_smithi187_45882_object11870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:42.930713+0000", 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.808017386000003, 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.107269367, 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 11871 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.930713+0000", 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.930713+0000", 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.930714+0000", 2024-09-02T10:02:32.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.333e-06 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.930747+0000", 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2500999999999999e-05 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.930763+0000", 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6395000000000001e-05 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.930768+0000", 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3770000000000004e-06 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.930816+0000", 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7123000000000003e-05 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.930893+0000", 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7877e-05 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.930973+0000", 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0008999999999999e-05 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.931486+0000", 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051270600000000003 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:42.996127+0000", 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.064641293000000002 2024-09-02T10:02:32.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:43.037917+0000", 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.041789740999999998 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:43.037950+0000", 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3163000000000001e-05 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:43.037982+0000", 2024-09-02T10:02:32.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1848999999999999e-05 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:28223 2.7 2:e2bc6ec9:::benchmark_data_smithi187_45882_object28222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:55.624055+0000", 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 37.114674934, 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.110618595, 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.814 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 28223 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.624055+0000", 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.624055+0000", 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.624056+0000", 2024-09-02T10:02:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2449999999999999e-06 2024-09-02T10:02:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.624097+0000", 2024-09-02T10:02:32.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0831000000000003e-05 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.624108+0000", 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0382e-05 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.624115+0000", 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3799999999999996e-06 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.624168+0000", 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2703e-05 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.624279+0000", 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110963 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.624404+0000", 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125494 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.625413+0000", 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00100873 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.625734+0000", 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032147499999999999 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.734611+0000", 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10887652 2024-09-02T10:02:32.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.734646+0000", 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5169000000000002e-05 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.734674+0000", 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7702999999999999e-05 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:28223 2.7 e105/92 ondisk, result = 0)", 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:55.625562+0000", 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 37.113168406, 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.109118681, 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.625562+0000", 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.625562+0000", 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.625563+0000", 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5600000000000005e-07 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.625566+0000", 2024-09-02T10:02:32.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4570000000000001e-06 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.734558+0000", 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10899173099999999 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.734565+0000", 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.435e-06 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.734602+0000", 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6708000000000002e-05 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.734607+0000", 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8359999999999999e-06 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:55.734680+0000", 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3757999999999994e-05 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.824 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:36452 2.7 2:ec15f213:::benchmark_data_smithi187_45882_object36451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:02.097683+0000", 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.641046813999999, 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16571392700000001, 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 36452 2024-09-02T10:02:32.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.097683+0000", 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.097683+0000", 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.097685+0000", 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.271e-06 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.097763+0000", 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7900999999999998e-05 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.097773+0000", 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0596e-05 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.097779+0000", 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5149999999999997e-06 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.097842+0000", 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3207000000000007e-05 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.097973+0000", 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013108999999999999 2024-09-02T10:02:32.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.098116+0000", 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014284500000000001 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.099305+0000", 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00118873 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.099861+0000", 2024-09-02T10:02:32.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00055656900000000003 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.263334+0000", 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16347300300000001 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.263370+0000", 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5478000000000002e-05 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.263397+0000", 2024-09-02T10:02:32.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7722e-05 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:36452 2.7 e109/92 ondisk, result = 0)", 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:02.099368+0000", 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 30.639362421000001, 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.164035923, 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.099368+0000", 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.099368+0000", 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.099368+0000", 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1099999999999995e-07 2024-09-02T10:02:32.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.099371+0000", 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0680000000000001e-06 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.263264+0000", 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.163892909 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.263269+0000", 2024-09-02T10:02:32.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7010000000000004e-06 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.263318+0000", 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9234000000000003e-05 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.263326+0000", 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9200000000000004e-06 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.263404+0000", 2024-09-02T10:02:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7479999999999997e-05 2024-09-02T10:02:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.835 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:37151 2.7 2:ed8bcd27:::benchmark_data_smithi187_45882_object37150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-02T10:02:32.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:32.837 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:02.819510+0000", 2024-09-02T10:02:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.919220484, 2024-09-02T10:02:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14499812000000001, 2024-09-02T10:02:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.837 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 37151 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.819510+0000", 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.819510+0000", 2024-09-02T10:02:32.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.819511+0000", 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.714e-06 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.819553+0000", 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1675000000000003e-05 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.819572+0000", 2024-09-02T10:02:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8482999999999999e-05 2024-09-02T10:02:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.819580+0000", 2024-09-02T10:02:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2360000000000004e-06 2024-09-02T10:02:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.819634+0000", 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3749000000000003e-05 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.819715+0000", 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1619999999999994e-05 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.819811+0000", 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5383000000000005e-05 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.821041+0000", 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012304729999999999 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.937126+0000", 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.116085233 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.964403+0000", 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027276775999999999 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.964462+0000", 2024-09-02T10:02:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9364000000000003e-05 2024-09-02T10:02:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.964508+0000", 2024-09-02T10:02:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5414000000000003e-05 2024-09-02T10:02:32.843 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:37176 2.7 2:e895c9fb:::benchmark_data_smithi187_45882_object37175:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:02.872103+0000", 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.866626767, 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14611950600000001, 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.844 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 37176 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.872103+0000", 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.872103+0000", 2024-09-02T10:02:32.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.872110+0000", 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0670000000000003e-06 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.872156+0000", 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5121999999999999e-05 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.872185+0000", 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9332999999999999e-05 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.872199+0000", 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3953e-05 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.872341+0000", 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014260100000000001 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.872497+0000", 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015571500000000001 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.872650+0000", 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015308699999999999 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.873715+0000", 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010649940000000001 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.018034+0000", 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14431878000000001 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.018124+0000", 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9948e-05 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.018179+0000", 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5208999999999999e-05 2024-09-02T10:02:32.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.018223+0000", 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3696999999999999e-05 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:38681 2.2 e111/92 2:44fed43f:::benchmark_data_smithi187_45882_object38680:head v 111'2519, mlcod=111'2514)", 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:03.912144+0000", 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.826586177999999, 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12585829800000001, 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.850 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.912144+0000", 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.912144+0000", 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.912145+0000", 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3300000000000001e-07 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.912168+0000", 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2904999999999998e-05 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.036835+0000", 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12466687999999999 2024-09-02T10:02:32.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.036841+0000", 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.849e-06 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.036897+0000", 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5519999999999997e-05 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.036918+0000", 2024-09-02T10:02:32.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1429000000000001e-05 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.037970+0000", 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00105145 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.038002+0000", 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2431999999999997e-05 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:38701 2.2 e111/92 2:40aa588f:::benchmark_data_smithi187_45882_object38700:head v 111'2520, mlcod=111'2515)", 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:03.949354+0000", 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.789375967000002, 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16566154699999999, 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.949354+0000", 2024-09-02T10:02:32.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.949354+0000", 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.949355+0000", 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5499999999999997e-07 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.949374+0000", 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9165e-05 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.113653+0000", 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16427919299999999 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.113659+0000", 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7599999999999999e-06 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.113716+0000", 2024-09-02T10:02:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6715000000000002e-05 2024-09-02T10:02:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.113735+0000", 2024-09-02T10:02:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8856e-05 2024-09-02T10:02:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.114985+0000", 2024-09-02T10:02:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001250442 2024-09-02T10:02:32.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.115016+0000", 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0660999999999997e-05 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:40743 2.7 2:e5b99752:::benchmark_data_smithi187_45882_object40742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:05.728239+0000", 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.010491177999999, 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12207173, 2024-09-02T10:02:32.861 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 40743 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.728239+0000", 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.728239+0000", 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.728240+0000", 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1999999999999998e-07 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.728290+0000", 2024-09-02T10:02:32.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0287999999999997e-05 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.728301+0000", 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0533999999999999e-05 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.728308+0000", 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8700000000000003e-06 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:32.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.728349+0000", 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1576999999999999e-05 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.728422+0000", 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2977000000000003e-05 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.728511+0000", 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8480000000000007e-05 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.729412+0000", 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00090128699999999996 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.729953+0000", 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00054111800000000005 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.850252+0000", 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.120298949 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.866 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.850285+0000", 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2703999999999997e-05 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:05.850311+0000", 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6026000000000001e-05 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:32.867 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:02:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:32.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:32.869 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:32.947 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:02:32.947 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:02:32.948 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-02T10:02:32.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:02:32.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:32.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:32.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:32.995 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:33.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:33.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:33.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:33.119 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:33.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:33.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:33.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:33.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:33.246 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:02:33.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:33.370 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:33.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:33.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:02:33.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:02:33.413 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:02:33.413 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:02:33.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 58 16 71302 71286 76.8103 56.1875 0.0106244 0.0130142 2024-09-02T10:02:33.445 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:02:33.445 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:02:33.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:33.477 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:02:33.478 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:02:33.505 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:02:33.505 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:02:33.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1680200138' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:33.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:33.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:33.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:33.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1680200138' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:33.531 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:02:33.531 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:02:33.559 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:02:33.560 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:02:33.589 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:02:33.590 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:02:33.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1680200138' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:33.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:33.624 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:02:33.625 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:02:33.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:33.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:33.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:02:33.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:02:33.690 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:02:33.691 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:02:33.720 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:02:33.721 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:02:33.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:02:33.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:33.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:33.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:33.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:33.903 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:33.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:34.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:34.036 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:34.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:34.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:34.043 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:34.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:34.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:34.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:34.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:34.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:34.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:34.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:34.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:34.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:34 smithi162 ceph-mon[20279]: pgmap v347: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 853 op/s; 4139/208122 objects misplaced (1.989%); 693 KiB/s, 10 objects/s recovering 2024-09-02T10:02:34.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:34 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:34.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1680200138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:34.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:34 smithi162 ceph-mon[20279]: osdmap e121: 12 total, 12 up, 12 in 2024-09-02T10:02:34.309 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3052872868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:34.393 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:34.421 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:34.422 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:34.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 59 16 72217 72201 76.4777 57.1875 0.00389785 0.0130707 2024-09-02T10:02:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:34 smithi187 ceph-mon[20072]: pgmap v347: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 853 op/s; 4139/208122 objects misplaced (1.989%); 693 KiB/s, 10 objects/s recovering 2024-09-02T10:02:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:34 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1680200138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:34 smithi187 ceph-mon[20072]: osdmap e121: 12 total, 12 up, 12 in 2024-09-02T10:02:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3052872868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:34.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:34.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:34.554 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:34.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:34.585 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:34.585 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:34.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:34.680 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:34.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:34.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:34 smithi151 ceph-mon[16639]: pgmap v347: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 853 op/s; 4139/208122 objects misplaced (1.989%); 693 KiB/s, 10 objects/s recovering 2024-09-02T10:02:34.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:34 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:34.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1680200138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:34.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:34 smithi151 ceph-mon[16639]: osdmap e121: 12 total, 12 up, 12 in 2024-09-02T10:02:34.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3052872868' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:34.781 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:34.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:34.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:34.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:34.895 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:34.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:34.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:34.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:35.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:35.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:35.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:35.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:35.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:35.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:35.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:35.317 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:35.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:35.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:35.438 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:02:35.437820+0000 min lat: 0.00101783 max lat: 2.21831 avg lat: 0.0131416 2024-09-02T10:02:35.438 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:02:35.438 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 60 16 73037 73021 76.0572 51.25 0.0107287 0.0131416 2024-09-02T10:02:35.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:35.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:35.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:35.581 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:35.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:35.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:35.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:35.704 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:35.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:35.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:35.727 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:02:35.728 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:02:35.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:35.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:35.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:35.935 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:35.957 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:35.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:35.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:36.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:02:36.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:36.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:36.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:36.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:02:36.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:36.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:36.229 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:36.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:36.349 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:36.361 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:36.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:36.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:36 smithi151 ceph-mon[16639]: pgmap v349: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 662 op/s; 4139/210765 objects misplaced (1.964%); 608 KiB/s, 9 objects/s recovering 2024-09-02T10:02:36.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:36 smithi151 ceph-mon[16639]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:02:36.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 61 16 74028 74012 75.8256 61.9375 0.0052197 0.0131813 2024-09-02T10:02:36.446 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-02T10:02:35.071534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4604690976,"num_objects":70255,"num_object_clones":0,"num_object_copies":210765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4139,"num_objects_unfound":0,"num_objects_dirty":70255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70253,"num_write_kb":4496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71265,"ondisk_log_size":71265,"up":81,"acting":80,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":81,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":13885892,"kb_used_data":13395060,"kb_used_omap":18,"kb_used_meta":490605,"kb_avail":1111055932,"statfs":{"total":1151940427776,"available":1137721274368,"internally_reserved":0,"allocated":13716541440,"data_stored":13710954290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":502379835},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,1,0,6,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":297140224,"num_objects":4534,"num_object_clones":0,"num_object_copies":13602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4139,"num_objects_unfound":0,"num_objects_dirty":4534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4534,"num_write_kb":290176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.840508"},"pg_stats":[{"pgid":"2.f","version":"121'4499","reported_seq":4568,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-02T10:02:34.093465+0000","last_change":"2024-09-02T10:01:30.183309+0000","last_active":"2024-09-02T10:02:34.093465+0000","last_peered":"2024-09-02T10:02:34.093465+0000","last_clean":"2024-09-02T10:02:34.093465+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:02:34.093465+0000","last_undegraded":"2024-09-02T10:02:34.093465+0000","last_fullsized":"2024-09-02T10:02:34.093465+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4499,"log_dups_size":0,"ondisk_log_size":4499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294846464,"num_objects":4499,"num_object_clones":0,"num_object_copies":13497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4499,"num_write_kb":287936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"120'4383","reported_seq":4450,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:31.885424+0000","last_change":"2024-09-02T10:01:30.561460+0000","last_active":"2024-09-02T10:02:31.885424+0000","last_peered":"2024-09-02T10:02:31.885424+0000","last_clean":"2024-09-02T10:02:31.885424+0000","last_became_active":"2024-09-02T10:01:30.560995+0000","last_became_peered":"2024-09-02T10:01:30.560995+0000","last_unstale":"2024-09-02T10:02:31.885424+0000","last_undegraded":"2024-09-02T10:02:31.885424+0000","last_fullsized":"2024-09-02T10:02:31.885424+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4383,"log_dups_size":0,"ondisk_log_size":4383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:55.406791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287244288,"num_objects":4383,"num_object_clones":0,"num_object_copies":13149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4383,"num_write_kb":280512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"120'4344","reported_seq":4410,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:32.862626+0000","last_change":"2024-09-02T10:01:30.560154+0000","last_active":"2024-09-02T10:02:32.862626+0000","last_peered":"2024-09-02T10:02:32.862626+0000","last_clean":"2024-09-02T10:02:32.862626+0000","last_became_active":"2024-09-02T10:01:30.559704+0000","last_became_peered":"2024-09-02T10:01:30.559704+0000","last_unstale":"2024-09-02T10:02:32.862626+0000","last_undegraded":"2024-09-02T10:02:32.862626+0000","last_fullsized":"2024-09-02T10:02:32.862626+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4344,"log_dups_size":0,"ondisk_log_size":4344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:46.256601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284688384,"num_objects":4344,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4344,"num_write_kb":278016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.c","version":"120'4405","reported_seq":4471,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:31.778937+0000","last_change":"2024-09-02T10:01:30.139707+0000","last_active":"2024-09-02T10:02:31.778937+0000","last_peered":"2024-09-02T10:02:31.778937+0000","last_clean":"2024-09-02T10:02:31.778937+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:02:31.778937+0000","last_undegraded":"2024-09-02T10:02:31.778937+0000","last_fullsized":"2024-09-02T10:02:31.778937+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:38.110684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288686080,"num_objects":4405,"num_object_clones":0,"num_object_copies":13215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4405,"num_write_kb":281920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":207,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.236673+0000","last_change":"2024-09-02T10:01:48.974659+0000","last_active":"2024-09-02T10:02:29.236673+0000","last_peered":"2024-09-02T10:02:29.236673+0000","last_clean":"2024-09-02T10:02:29.236673+0000","last_became_active":"2024-09-02T10:01:48.206613+0000","last_became_peered":"2024-09-02T10:01:48.206613+0000","last_unstale":"2024-09-02T10:02:29.236673+0000","last_undegraded":"2024-09-02T10:02:29.236673+0000","last_fullsized":"2024-09-02T10:02:29.236673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:48.974614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:48.974614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:06.550945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"120'4417","reported_seq":4483,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:31.747072+0000","last_change":"2024-09-02T10:01:30.141172+0000","last_active":"2024-09-02T10:02:31.747072+0000","last_peered":"2024-09-02T10:02:31.747072+0000","last_clean":"2024-09-02T10:02:31.747072+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:02:31.747072+0000","last_undegraded":"2024-09-02T10:02:31.747072+0000","last_fullsized":"2024-09-02T10:02:31.747072+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4417,"log_dups_size":0,"ondisk_log_size":4417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:32.087935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289472512,"num_objects":4417,"num_object_clones":0,"num_object_copies":13251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4417,"num_write_kb":282688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":53,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.365179+0000","last_change":"2024-09-02T10:01:50.950127+0000","last_active":"2024-09-02T10:02:29.365179+0000","last_peered":"2024-09-02T10:02:29.365179+0000","last_clean":"2024-09-02T10:02:29.365179+0000","last_became_active":"2024-09-02T10:01:50.764818+0000","last_became_peered":"2024-09-02T10:01:50.764818+0000","last_unstale":"2024-09-02T10:02:29.365179+0000","last_undegraded":"2024-09-02T10:02:29.365179+0000","last_fullsized":"2024-09-02T10:02:29.365179+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:50.950013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:50.950013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:11.939457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"120'4273","reported_seq":4473,"reported_epoch":120,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:02:30.609431+0000","last_change":"2024-09-02T10:02:29.461447+0000","last_active":"2024-09-02T10:02:30.609431+0000","last_peered":"2024-09-02T10:02:30.609431+0000","last_clean":"2024-09-02T10:02:27.223209+0000","last_became_active":"2024-09-02T10:02:29.439334+0000","last_became_peered":"2024-09-02T10:02:29.439334+0000","last_unstale":"2024-09-02T10:02:30.609431+0000","last_undegraded":"2024-09-02T10:02:30.609431+0000","last_fullsized":"2024-09-02T10:02:29.236626+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280035328,"num_objects":4273,"num_object_clones":0,"num_object_copies":12819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4139,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4273,"num_write_kb":273472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,9],"avail_no_missing":["7","9","10"],"object_location_counts":[{"shards":"7,9","objects":134},{"shards":"7,9,10","objects":4139}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":213,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.236647+0000","last_change":"2024-09-02T10:01:51.337316+0000","last_active":"2024-09-02T10:02:29.236647+0000","last_peered":"2024-09-02T10:02:29.236647+0000","last_clean":"2024-09-02T10:02:29.236647+0000","last_became_active":"2024-09-02T10:01:50.766826+0000","last_became_peered":"2024-09-02T10:01:50.766826+0000","last_unstale":"2024-09-02T10:02:29.236647+0000","last_undegraded":"2024-09-02T10:02:29.236647+0000","last_fullsized":"2024-09-02T10:02:29.236647+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.337264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.337264+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:08.499993+0000","scrub_duration":395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"120'4400","reported_seq":4466,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:31.912424+0000","last_change":"2024-09-02T10:01:30.560301+0000","last_active":"2024-09-02T10:02:31.912424+0000","last_peered":"2024-09-02T10:02:31.912424+0000","last_clean":"2024-09-02T10:02:31.912424+0000","last_became_active":"2024-09-02T10:01:30.559997+0000","last_became_peered":"2024-09-02T10:01:30.559997+0000","last_unstale":"2024-09-02T10:02:31.912424+0000","last_undegraded":"2024-09-02T10:02:31.912424+0000","last_fullsized":"2024-09-02T10:02:31.912424+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4400,"log_dups_size":0,"ondisk_log_size":4400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:34.011670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288358400,"num_objects":4400,"num_object_clones":0,"num_object_copies":13200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":281600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":53,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.252105+0000","last_change":"2024-09-02T10:01:50.898052+0000","last_active":"2024-09-02T10:02:29.252105+0000","last_peered":"2024-09-02T10:02:29.252105+0000","last_clean":"2024-09-02T10:02:29.252105+0000","last_became_active":"2024-09-02T10:01:50.765441+0000","last_became_peered":"2024-09-02T10:01:50.765441+0000","last_unstale":"2024-09-02T10:02:29.252105+0000","last_undegraded":"2024-09-02T10:02:29.252105+0000","last_fullsized":"2024-09-02T10:02:29.252105+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:50.897996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:50.897996+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:00.393021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"120'4482","reported_seq":4546,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:31.682234+0000","last_change":"2024-09-02T10:01:30.140642+0000","last_active":"2024-09-02T10:02:31.682234+0000","last_peered":"2024-09-02T10:02:31.682234+0000","last_clean":"2024-09-02T10:02:31.682234+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:02:31.682234+0000","last_undegraded":"2024-09-02T10:02:31.682234+0000","last_fullsized":"2024-09-02T10:02:31.682234+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:54.429959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293732352,"num_objects":4482,"num_object_clones":0,"num_object_copies":13446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":286848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":77,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.365772+0000","last_change":"2024-09-02T10:01:51.173117+0000","last_active":"2024-09-02T10:02:29.365772+0000","last_peered":"2024-09-02T10:02:29.365772+0000","last_clean":"2024-09-02T10:02:29.365772+0000","last_became_active":"2024-09-02T10:01:50.778902+0000","last_became_peered":"2024-09-02T10:01:50.778902+0000","last_unstale":"2024-09-02T10:02:29.365772+0000","last_undegraded":"2024-09-02T10:02:29.365772+0000","last_fullsized":"2024-09-02T10:02:29.365772+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.173075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.173075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:52.401881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"120'4327","reported_seq":4393,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:32.487982+0000","last_change":"2024-09-02T10:01:30.560172+0000","last_active":"2024-09-02T10:02:32.487982+0000","last_peered":"2024-09-02T10:02:32.487982+0000","last_clean":"2024-09-02T10:02:32.487982+0000","last_became_active":"2024-09-02T10:01:30.559685+0000","last_became_peered":"2024-09-02T10:01:30.559685+0000","last_unstale":"2024-09-02T10:02:32.487982+0000","last_undegraded":"2024-09-02T10:02:32.487982+0000","last_fullsized":"2024-09-02T10:02:32.487982+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4327,"log_dups_size":0,"ondisk_log_size":4327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:57.437147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283574272,"num_objects":4327,"num_object_clones":0,"num_object_copies":12981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4327,"num_write_kb":276928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,10],"acting":[1,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":53,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.252135+0000","last_change":"2024-09-02T10:01:51.898421+0000","last_active":"2024-09-02T10:02:29.252135+0000","last_peered":"2024-09-02T10:02:29.252135+0000","last_clean":"2024-09-02T10:02:29.252135+0000","last_became_active":"2024-09-02T10:01:50.765475+0000","last_became_peered":"2024-09-02T10:01:50.765475+0000","last_unstale":"2024-09-02T10:02:29.252135+0000","last_undegraded":"2024-09-02T10:02:29.252135+0000","last_fullsized":"2024-09-02T10:02:29.252135+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.898350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.898350+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:53.457481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.5","version":"121'4452","reported_seq":4519,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-02T10:02:34.085626+0000","last_change":"2024-09-02T10:01:30.183497+0000","last_active":"2024-09-02T10:02:34.085626+0000","last_peered":"2024-09-02T10:02:34.085626+0000","last_clean":"2024-09-02T10:02:34.085626+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:02:34.085626+0000","last_undegraded":"2024-09-02T10:02:34.085626+0000","last_fullsized":"2024-09-02T10:02:34.085626+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:40.071109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291766272,"num_objects":4452,"num_object_clones":0,"num_object_copies":13356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4452,"num_write_kb":284928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":53,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.252118+0000","last_change":"2024-09-02T10:01:52.926252+0000","last_active":"2024-09-02T10:02:29.252118+0000","last_peered":"2024-09-02T10:02:29.252118+0000","last_clean":"2024-09-02T10:02:29.252118+0000","last_became_active":"2024-09-02T10:01:50.774645+0000","last_became_peered":"2024-09-02T10:01:50.774645+0000","last_unstale":"2024-09-02T10:02:29.252118+0000","last_undegraded":"2024-09-02T10:02:29.252118+0000","last_fullsized":"2024-09-02T10:02:29.252118+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:52.926205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:52.926205+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:11.347737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.6","version":"120'4393","reported_seq":4459,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:32.502987+0000","last_change":"2024-09-02T10:01:30.560649+0000","last_active":"2024-09-02T10:02:32.502987+0000","last_peered":"2024-09-02T10:02:32.502987+0000","last_clean":"2024-09-02T10:02:32.502987+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:02:32.502987+0000","last_undegraded":"2024-09-02T10:02:32.502987+0000","last_fullsized":"2024-09-02T10:02:32.502987+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:34.163733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287899648,"num_objects":4393,"num_object_clones":0,"num_object_copies":13179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":281152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":145,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.365392+0000","last_change":"2024-09-02T10:01:51.746769+0000","last_active":"2024-09-02T10:02:29.365392+0000","last_peered":"2024-09-02T10:02:29.365392+0000","last_clean":"2024-09-02T10:02:29.365392+0000","last_became_active":"2024-09-02T10:01:50.768028+0000","last_became_peered":"2024-09-02T10:01:50.768028+0000","last_unstale":"2024-09-02T10:02:29.365392+0000","last_undegraded":"2024-09-02T10:02:29.365392+0000","last_fullsized":"2024-09-02T10:02:29.365392+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.746691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.746691+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:06.196416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"120'4393","reported_seq":4460,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:31.923643+0000","last_change":"2024-09-02T10:01:30.560449+0000","last_active":"2024-09-02T10:02:31.923643+0000","last_peered":"2024-09-02T10:02:31.923643+0000","last_clean":"2024-09-02T10:02:31.923643+0000","last_became_active":"2024-09-02T10:01:30.559962+0000","last_became_peered":"2024-09-02T10:01:30.559962+0000","last_unstale":"2024-09-02T10:02:31.923643+0000","last_undegraded":"2024-09-02T10:02:31.923643+0000","last_fullsized":"2024-09-02T10:02:31.923643+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:55.111495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287899648,"num_objects":4393,"num_object_clones":0,"num_object_copies":13179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":281152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,2],"acting":[6,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"120'4283","reported_seq":4350,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:30.593612+0000","last_change":"2024-09-02T10:01:30.141564+0000","last_active":"2024-09-02T10:02:30.593612+0000","last_peered":"2024-09-02T10:02:30.593612+0000","last_clean":"2024-09-02T10:02:30.593612+0000","last_became_active":"2024-09-02T10:01:30.141263+0000","last_became_peered":"2024-09-02T10:01:30.141263+0000","last_unstale":"2024-09-02T10:02:30.593612+0000","last_undegraded":"2024-09-02T10:02:30.593612+0000","last_fullsized":"2024-09-02T10:02:30.593612+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4283,"log_dups_size":0,"ondisk_log_size":4283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:56.004849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280690688,"num_objects":4283,"num_object_clones":0,"num_object_copies":12849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4283,"num_write_kb":274112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":53,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.252816+0000","last_change":"2024-09-02T10:01:51.614379+0000","last_active":"2024-09-02T10:02:29.252816+0000","last_peered":"2024-09-02T10:02:29.252816+0000","last_clean":"2024-09-02T10:02:29.252816+0000","last_became_active":"2024-09-02T10:01:50.781305+0000","last_became_peered":"2024-09-02T10:01:50.781305+0000","last_unstale":"2024-09-02T10:02:29.252816+0000","last_undegraded":"2024-09-02T10:02:29.252816+0000","last_fullsized":"2024-09-02T10:02:29.252816+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.614334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.614334+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:04.278629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"120'4428","reported_seq":4493,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:32.537417+0000","last_change":"2024-09-02T10:01:30.560354+0000","last_active":"2024-09-02T10:02:32.537417+0000","last_peered":"2024-09-02T10:02:32.537417+0000","last_clean":"2024-09-02T10:02:32.537417+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:02:32.537417+0000","last_undegraded":"2024-09-02T10:02:32.537417+0000","last_fullsized":"2024-09-02T10:02:32.537417+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:48.623221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290193408,"num_objects":4428,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4428,"num_write_kb":283392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":213,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.236627+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:02:29.236627+0000","last_peered":"2024-09-02T10:02:29.236627+0000","last_clean":"2024-09-02T10:02:29.236627+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:02:29.236627+0000","last_undegraded":"2024-09-02T10:02:29.236627+0000","last_fullsized":"2024-09-02T10:02:29.236627+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:20.037124+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"120'4325","reported_seq":4392,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:31.752315+0000","last_change":"2024-09-02T10:01:30.560175+0000","last_active":"2024-09-02T10:02:31.752315+0000","last_peered":"2024-09-02T10:02:31.752315+0000","last_clean":"2024-09-02T10:02:31.752315+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:02:31.752315+0000","last_undegraded":"2024-09-02T10:02:31.752315+0000","last_fullsized":"2024-09-02T10:02:31.752315+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4325,"log_dups_size":0,"ondisk_log_size":4325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:38.707848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283443200,"num_objects":4325,"num_object_clones":0,"num_object_copies":12975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4325,"num_write_kb":276800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":60,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-02T10:02:29.234939+0000","last_change":"2024-09-02T10:01:50.947414+0000","last_active":"2024-09-02T10:02:29.234939+0000","last_peered":"2024-09-02T10:02:29.234939+0000","last_clean":"2024-09-02T10:02:29.234939+0000","last_became_active":"2024-09-02T10:01:50.763116+0000","last_became_peered":"2024-09-02T10:01:50.763116+0000","last_unstale":"2024-09-02T10:02:29.234939+0000","last_undegraded":"2024-09-02T10:02:29.234939+0000","last_fullsized":"2024-09-02T10:02:29.234939+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:50.947356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:50.947356+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:20.448518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"121'4449","reported_seq":4518,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-02T10:02:34.098590+0000","last_change":"2024-09-02T10:01:30.140619+0000","last_active":"2024-09-02T10:02:34.098590+0000","last_peered":"2024-09-02T10:02:34.098590+0000","last_clean":"2024-09-02T10:02:34.098590+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:02:34.098590+0000","last_undegraded":"2024-09-02T10:02:34.098590+0000","last_fullsized":"2024-09-02T10:02:34.098590+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":4449,"log_dups_size":0,"ondisk_log_size":4449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:02:39.195715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291569664,"num_objects":4449,"num_object_clones":0,"num_object_copies":13347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4449,"num_write_kb":284736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":4604100608,"num_objects":70253,"num_object_clones":0,"num_object_copies":210759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4139,"num_objects_unfound":0,"num_objects_dirty":70253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70253,"num_write_kb":4496192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":4259840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13835436032,"data_stored":13835436032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70253,"ondisk_log_size":70253,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1012,"ondisk_log_size":1012,"up":33,"acting":33,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383709,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858208,"kb_used_data":827020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92886944,"statfs":{"total":95995035648,"available":95116230656,"internally_reserved":0,"allocated":846868480,"data_stored":846404143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612642,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850140,"kb_used_data":818956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92895012,"statfs":{"total":95995035648,"available":95124492288,"internally_reserved":0,"allocated":838610944,"data_stored":838146615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841575,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1736988,"kb_used_data":1683596,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92008164,"statfs":{"total":95995035648,"available":94216359936,"internally_reserved":0,"allocated":1724002304,"data_stored":1723537975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":54655424},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070508,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1163292,"kb_used_data":1132108,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92581860,"statfs":{"total":95995035648,"available":94803824640,"internally_reserved":0,"allocated":1159278592,"data_stored":1158814263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,2,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299439,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1412452,"kb_used_data":1363212,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92332700,"statfs":{"total":95995035648,"available":94548684800,"internally_reserved":0,"allocated":1395929088,"data_stored":1395464759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528372,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":605148,"kb_used_data":573964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93140004,"statfs":{"total":95995035648,"available":95375364096,"internally_reserved":0,"allocated":587739136,"data_stored":587274807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724601,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":593252,"kb_used_data":562068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93151900,"statfs":{"total":95995035648,"available":95387545600,"internally_reserved":0,"allocated":575557632,"data_stored":575089751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953535,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":887212,"kb_used_data":856024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92857940,"statfs":{"total":95995035648,"available":95086530560,"internally_reserved":0,"allocated":876568576,"data_stored":876099709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1737508,"kb_used_data":1684116,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92007644,"statfs":{"total":95995035648,"available":94215827456,"internally_reserved":0,"allocated":1724534784,"data_stored":1724066903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1160868,"kb_used_data":1111636,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92584284,"statfs":{"total":95995035648,"available":94806306816,"internally_reserved":0,"allocated":1138315264,"data_stored":1137847383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1455004,"kb_used_data":1405772,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92290148,"statfs":{"total":95995035648,"available":94505111552,"internally_reserved":0,"allocated":1439510528,"data_stored":1439046199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1425820,"kb_used_data":1376588,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92319332,"statfs":{"total":95995035648,"available":94534995968,"internally_reserved":0,"allocated":1409626112,"data_stored":1409161783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1417740288,"data_stored":1417740288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1449721856,"data_stored":1449721856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1153892352,"data_stored":1153892352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1742340096,"data_stored":1742340096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":876019712,"data_stored":876019712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":583270400,"data_stored":583270400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":601620480,"data_stored":601620480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1403584512,"data_stored":1403584512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1172176896,"data_stored":1172176896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1737228288,"data_stored":1737228288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":841613312,"data_stored":841613312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":856227840,"data_stored":856227840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:02:36.448 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-02T10:02:36.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pg_num 21 2024-09-02T10:02:36.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:36.499 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:36 smithi187 ceph-mon[20072]: pgmap v349: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 662 op/s; 4139/210765 objects misplaced (1.964%); 608 KiB/s, 9 objects/s recovering 2024-09-02T10:02:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:36 smithi187 ceph-mon[20072]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:02:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:36.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:36 smithi162 ceph-mon[20279]: pgmap v349: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 662 op/s; 4139/210765 objects misplaced (1.964%); 608 KiB/s, 9 objects/s recovering 2024-09-02T10:02:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:36 smithi162 ceph-mon[20279]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:02:36.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:36.657 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:36.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:36.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:36.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:36.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:36.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:36.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:02:36.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:36.911 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:36.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:37.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:37.040 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:37.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:37.041 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:37.120 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:37.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:37.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:37.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:37.302 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:37.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:37.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:37.404 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:37.430 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:37.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:37.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 62 16 75179 75163 75.7628 71.9375 0.0166941 0.0131937 2024-09-02T10:02:37.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:37.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:37.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:37.663 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:37.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:37.692 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:37.724 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:37.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:37.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:37.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:37.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:37.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:37.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:37.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:38.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:38.079 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:38.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:38.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:38.208 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:38.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:38.209 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:38.256 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:02:38.256 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pg_num to 21 2024-09-02T10:02:38.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:38.338 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:38.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:38.356 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:02:38.357 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:02:38.388 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:02:38.389 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:02:38.423 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:02:38.423 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:02:38.437 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 63 16 76355 76339 75.7268 73.5 0.00454859 0.0132004 2024-09-02T10:02:38.440 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:38.454 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:02:38.454 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:02:38.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:38 smithi151 ceph-mon[16639]: pgmap v350: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 895 op/s; 3826/218652 objects misplaced (1.750%); 3.0 MiB/s, 47 objects/s recovering 2024-09-02T10:02:38.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3872555911' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:38.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:38 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:38.454 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3464442280' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-02T10:02:38.455 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:38 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-02T10:02:38.467 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:38.468 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:38.484 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:02:38.485 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:02:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:38 smithi187 ceph-mon[20072]: pgmap v350: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 895 op/s; 3826/218652 objects misplaced (1.750%); 3.0 MiB/s, 47 objects/s recovering 2024-09-02T10:02:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3872555911' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:38 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3464442280' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-02T10:02:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:38 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-02T10:02:38.518 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:02:38.518 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:02:38.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:02:38.547 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:02:38.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:38.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:02:38.584 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:02:38.593 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:38.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:38 smithi162 ceph-mon[20279]: pgmap v350: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 895 op/s; 3826/218652 objects misplaced (1.750%); 3.0 MiB/s, 47 objects/s recovering 2024-09-02T10:02:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3872555911' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:38 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:02:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3464442280' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-02T10:02:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:38 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"} : dispatch 2024-09-02T10:02:38.614 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:02:38.614 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:02:38.649 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:02:38.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:02:38.685 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:02:38.685 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:02:38.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:38.720 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:02:38.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:02:38.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:38.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:38.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:38.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:38.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:38.937 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:02:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:02:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:02:38.937 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:02:38.937 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:02:38.967 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:38.996 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:38.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:39.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:39.123 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:39.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:39.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:39.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:39.251 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:39.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:39.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:39.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:39.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:39 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:39.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:39 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:02:39.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:39 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-02T10:02:39.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:39 smithi151 ceph-mon[16639]: osdmap e122: 12 total, 12 up, 12 in 2024-09-02T10:02:39.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3124645916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:39.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:39 smithi151 ceph-mon[16639]: osdmap e123: 12 total, 12 up, 12 in 2024-09-02T10:02:39.366 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-02T10:02:39.384 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:39.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:39.438 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 64 16 77589 77573 75.7485 77.125 0.00297191 0.0131954 2024-09-02T10:02:39.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:39 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:39 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:02:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:39 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-02T10:02:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:39 smithi187 ceph-mon[20072]: osdmap e122: 12 total, 12 up, 12 in 2024-09-02T10:02:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3124645916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:39 smithi187 ceph-mon[20072]: osdmap e123: 12 total, 12 up, 12 in 2024-09-02T10:02:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-02T10:02:39.516 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:39.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:39.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:39 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:39 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:02:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:39 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-02T10:02:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:39 smithi162 ceph-mon[20279]: osdmap e122: 12 total, 12 up, 12 in 2024-09-02T10:02:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3124645916' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:39 smithi162 ceph-mon[20279]: osdmap e123: 12 total, 12 up, 12 in 2024-09-02T10:02:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"} : dispatch 2024-09-02T10:02:39.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:39.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:39.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:39.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:39.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:39.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:39.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:02:39.876 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:39.939 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:39.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:39.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:40.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:40.068 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:40.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:40.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:40.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:40.192 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:02:40.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:40.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:40.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:40.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:40.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:40.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:40.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:40 smithi151 ceph-mon[16639]: 2.3 scrub starts 2024-09-02T10:02:40.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:40 smithi151 ceph-mon[16639]: pgmap v353: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 782 op/s; 3826/222204 objects misplaced (1.722%); 3.3 MiB/s, 52 objects/s recovering 2024-09-02T10:02:40.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-02T10:02:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:40 smithi151 ceph-mon[16639]: osdmap e124: 12 total, 12 up, 12 in 2024-09-02T10:02:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:40 smithi151 ceph-mon[16639]: 2.b scrub starts 2024-09-02T10:02:40.438 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 65 15 78679 78664 75.6321 68.1875 0.0145648 0.0132133 2024-09-02T10:02:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:40.453 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:40.454 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:40 smithi187 ceph-mon[20072]: 2.3 scrub starts 2024-09-02T10:02:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:40 smithi187 ceph-mon[20072]: pgmap v353: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 782 op/s; 3826/222204 objects misplaced (1.722%); 3.3 MiB/s, 52 objects/s recovering 2024-09-02T10:02:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-02T10:02:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:40 smithi187 ceph-mon[20072]: osdmap e124: 12 total, 12 up, 12 in 2024-09-02T10:02:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:40 smithi187 ceph-mon[20072]: 2.b scrub starts 2024-09-02T10:02:40.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:40.578 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:40.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:40.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:40 smithi162 ceph-mon[20279]: 2.3 scrub starts 2024-09-02T10:02:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:40 smithi162 ceph-mon[20279]: pgmap v353: 27 pgs: 1 active+recovering+undersized+remapped, 26 active+clean; 4.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 782 op/s; 3826/222204 objects misplaced (1.722%); 3.3 MiB/s, 52 objects/s recovering 2024-09-02T10:02:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-02T10:02:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:40 smithi162 ceph-mon[20279]: osdmap e124: 12 total, 12 up, 12 in 2024-09-02T10:02:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:40 smithi162 ceph-mon[20279]: 2.b scrub starts 2024-09-02T10:02:40.680 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:40.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:40.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:40.723 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:37163 2.0 2:019f9432:::benchmark_data_smithi187_45882_object37162:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:02.833080+0000", 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 37.887465933999998, 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.194089859, 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.724 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 37163 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.833080+0000", 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.833080+0000", 2024-09-02T10:02:40.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.833081+0000", 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4449999999999999e-06 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.833137+0000", 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6087e-05 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.833153+0000", 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5228000000000001e-05 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.833160+0000", 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5800000000000003e-06 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.833219+0000", 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9200000000000002e-05 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.833342+0000", 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122977 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.833493+0000", 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015021500000000001 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.835110+0000", 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001617453 2024-09-02T10:02:40.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:02.835122+0000", 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2052000000000001e-05 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.027103+0000", 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.191980606 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.027135+0000", 2024-09-02T10:02:40.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1953999999999999e-05 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.027170+0000", 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5061999999999997e-05 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.730 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:55055 2.8 2:1127c854:::benchmark_data_smithi187_45882_object55054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.471137+0000", 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 23.249408341999999, 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28183072799999997, 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55055 2024-09-02T10:02:40.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471137+0000", 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471137+0000", 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471138+0000", 2024-09-02T10:02:40.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0070000000000001e-06 2024-09-02T10:02:40.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471192+0000", 2024-09-02T10:02:40.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3325000000000002e-05 2024-09-02T10:02:40.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471202+0000", 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0468999999999999e-05 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471207+0000", 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8720000000000001e-06 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471242+0000", 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4702000000000001e-05 2024-09-02T10:02:40.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471322+0000", 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0093999999999996e-05 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471411+0000", 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9015000000000001e-05 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.472901+0000", 2024-09-02T10:02:40.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014899150000000001 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473274+0000", 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037354899999999998 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752887+0000", 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27961233699999999 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752928+0000", 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1560999999999997e-05 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752968+0000", 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9882000000000002e-05 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:55055 2.8 e113/92 ondisk, result = 0)", 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.473073+0000", 2024-09-02T10:02:40.737 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 23.247472768000002, 2024-09-02T10:02:40.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27990168399999998, 2024-09-02T10:02:40.738 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.738 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.738 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:02:40.738 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473073+0000", 2024-09-02T10:02:40.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473073+0000", 2024-09-02T10:02:40.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473074+0000", 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7100000000000002e-07 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473076+0000", 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4549999999999998e-06 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752722+0000", 2024-09-02T10:02:40.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27964593100000001 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752743+0000", 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1087e-05 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752872+0000", 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012929900000000001 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752881+0000", 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9339999999999995e-06 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752975+0000", 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3406999999999995e-05 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:55067 2.8 2:1ce73561:::benchmark_data_smithi187_45882_object55066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.485554+0000", 2024-09-02T10:02:40.742 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 23.234992087999998, 2024-09-02T10:02:40.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27014719300000001, 2024-09-02T10:02:40.743 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.743 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.743 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.743 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.743 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55067 2024-09-02T10:02:40.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.744 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485554+0000", 2024-09-02T10:02:40.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485554+0000", 2024-09-02T10:02:40.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485556+0000", 2024-09-02T10:02:40.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.885e-06 2024-09-02T10:02:40.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485587+0000", 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1435e-05 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485605+0000", 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7668000000000001e-05 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485615+0000", 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0302999999999999e-05 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485678+0000", 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2840000000000001e-05 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485807+0000", 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128947 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485908+0000", 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101378 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487422+0000", 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001514236 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487766+0000", 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034318 2024-09-02T10:02:40.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755648+0000", 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26788223500000002 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755679+0000", 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0703000000000001e-05 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755701+0000", 2024-09-02T10:02:40.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2382999999999999e-05 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:55067 2.8 e113/92 ondisk, result = 0)", 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.487535+0000", 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 23.233011112, 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26817209600000003, 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487535+0000", 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487535+0000", 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487536+0000", 2024-09-02T10:02:40.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1379999999999999e-06 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487541+0000", 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3959999999999997e-06 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755584+0000", 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26804281099999999 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755590+0000", 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4399999999999996e-06 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755640+0000", 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0973999999999997e-05 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755645+0000", 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2289999999999997e-06 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755707+0000", 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2107999999999996e-05 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:66926 2.0 2:00d49c5e:::benchmark_data_smithi187_45882_object66925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.250823+0000", 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469722853, 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.192431139, 2024-09-02T10:02:40.755 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 66926 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.250823+0000", 2024-09-02T10:02:40.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.250823+0000", 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.250825+0000", 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.646e-06 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.250910+0000", 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5117999999999994e-05 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.250924+0000", 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4428e-05 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.250936+0000", 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1936999999999999e-05 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251314+0000", 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037749499999999999 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251321+0000", 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.61e-06 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.443181+0000", 2024-09-02T10:02:40.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19185957200000001 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.443254+0000", 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3332999999999997e-05 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:66994 2.0 2:048746f0:::benchmark_data_smithi187_45882_object66993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.250960+0000", 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469585917, 2024-09-02T10:02:40.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.201492058, 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 66994 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.250960+0000", 2024-09-02T10:02:40.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.250960+0000", 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.250960+0000", 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7100000000000002e-07 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.250996+0000", 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5479999999999999e-05 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251004+0000", 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9549999999999992e-06 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251010+0000", 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0190000000000003e-06 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251352+0000", 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034173800000000001 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251356+0000", 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7210000000000002e-06 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.444298+0000", 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19294157000000001 2024-09-02T10:02:40.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.444385+0000", 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6992999999999999e-05 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.444496+0000", 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110849 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.450549+0000", 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060533310000000003 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.450648+0000", 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9171000000000003e-05 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.450657+0000", 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1710000000000008e-06 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.450695+0000", 2024-09-02T10:02:40.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7431999999999997e-05 2024-09-02T10:02:40.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.452452+0000", 2024-09-02T10:02:40.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001757057 2024-09-02T10:02:40.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67047 2.0 2:0990e74e:::benchmark_data_smithi187_45882_object67046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251022+0000", 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469524141999999, 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.202691016, 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67047 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251022+0000", 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251022+0000", 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251022+0000", 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2799999999999996e-07 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251053+0000", 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0562000000000003e-05 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251060+0000", 2024-09-02T10:02:40.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1169999999999998e-06 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251065+0000", 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.276e-06 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251382+0000", 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000316621 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251386+0000", 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7290000000000002e-06 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.446665+0000", 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.195279854 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.446784+0000", 2024-09-02T10:02:40.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000118713 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.446896+0000", 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111698 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.450602+0000", 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037061339999999998 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.452462+0000", 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001859664 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.452573+0000", 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011189299999999999 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.452616+0000", 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2051000000000001e-05 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.453713+0000", 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010971760000000001 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67053 2.0 2:0fe58c27:::benchmark_data_smithi187_45882_object67052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251073+0000", 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469472465000001, 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20891902600000001, 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.775 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67053 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251073+0000", 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251073+0000", 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251074+0000", 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7800000000000002e-07 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251103+0000", 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9258e-05 2024-09-02T10:02:40.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251110+0000", 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.861e-06 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251115+0000", 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8890000000000001e-06 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251406+0000", 2024-09-02T10:02:40.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000291635 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251410+0000", 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6320000000000001e-06 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.448636+0000", 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.197226022 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.448733+0000", 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7357999999999999e-05 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.448882+0000", 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014857999999999999 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.452532+0000", 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036495479999999999 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.453800+0000", 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001268253 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.458088+0000", 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042879820000000001 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.458126+0000", 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7664999999999999e-05 2024-09-02T10:02:40.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.459992+0000", 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018668650000000001 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67069 2.0 2:0a5d0e66:::benchmark_data_smithi187_45882_object67068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251122+0000", 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469423425, 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.211244143, 2024-09-02T10:02:40.782 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67069 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251122+0000", 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251122+0000", 2024-09-02T10:02:40.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251123+0000", 2024-09-02T10:02:40.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2e-07 2024-09-02T10:02:40.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251164+0000", 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1284e-05 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251172+0000", 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5870000000000004e-06 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251177+0000", 2024-09-02T10:02:40.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5899999999999998e-06 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251430+0000", 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025309799999999999 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251434+0000", 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.642e-06 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.453911+0000", 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.202476725 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.454033+0000", 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012238799999999999 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.454147+0000", 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113263 2024-09-02T10:02:40.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.460011+0000", 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0058641689999999998 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.460073+0000", 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1927999999999994e-05 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.461405+0000", 2024-09-02T10:02:40.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001332281 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.461446+0000", 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1477000000000003e-05 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.462367+0000", 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000920191 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.790 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67087 2.0 2:0a4419d0:::benchmark_data_smithi187_45882_object67086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251187+0000", 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469358348, 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21261585899999999, 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.791 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67087 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251187+0000", 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251187+0000", 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251188+0000", 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9999999999999998e-07 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251228+0000", 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9629999999999998e-05 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251236+0000", 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4069999999999993e-06 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251241+0000", 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1490000000000002e-06 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251463+0000", 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022229899999999999 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251471+0000", 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.277e-06 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.456060+0000", 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20458938600000001 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.456143+0000", 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3127999999999994e-05 2024-09-02T10:02:40.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.456254+0000", 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111177 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.461323+0000", 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00506808 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.461364+0000", 2024-09-02T10:02:40.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1140000000000003e-05 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.462435+0000", 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010713540000000001 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.462505+0000", 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0226000000000002e-05 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.463803+0000", 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001298106 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67089 2.0 2:0b7aa344:::benchmark_data_smithi187_45882_object67088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251251+0000", 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469294719000001, 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22925858900000001, 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.798 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67089 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251251+0000", 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251251+0000", 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251252+0000", 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7100000000000002e-07 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251288+0000", 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5930000000000003e-05 2024-09-02T10:02:40.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251299+0000", 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1482e-05 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251304+0000", 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2639999999999999e-06 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251492+0000", 2024-09-02T10:02:40.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000188075 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251496+0000", 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5619999999999999e-06 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.463852+0000", 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21235569500000001 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.463958+0000", 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010686100000000001 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.464070+0000", 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011157199999999999 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.478144+0000", 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014074081 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.478195+0000", 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0862999999999997e-05 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.478262+0000", 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7374999999999998e-05 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.478303+0000", 2024-09-02T10:02:40.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0572999999999999e-05 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.480510+0000", 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022067850000000002 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67098 2.0 2:0e6c4d4d:::benchmark_data_smithi187_45882_object67097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251312+0000", 2024-09-02T10:02:40.805 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469233442, 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23061732600000001, 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67098 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251312+0000", 2024-09-02T10:02:40.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251312+0000", 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251313+0000", 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7899999999999999e-07 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251344+0000", 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1381000000000003e-05 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251352+0000", 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6990000000000003e-06 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251357+0000", 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4729999999999999e-06 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251524+0000", 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000166427 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251530+0000", 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9610000000000002e-06 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.465987+0000", 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21445724699999999 2024-09-02T10:02:40.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.466134+0000", 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014704 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.466308+0000", 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000174026 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.478251+0000", 2024-09-02T10:02:40.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011943014 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.480525+0000", 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022738630000000001 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.480569+0000", 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4128999999999997e-05 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.480613+0000", 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4240999999999999e-05 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.481930+0000", 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013163459999999999 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67110 2.0 2:0d8b2a81:::benchmark_data_smithi187_45882_object67109:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.812 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251382+0000", 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469164044999999, 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23470874999999999, 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67110 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251382+0000", 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251382+0000", 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251382+0000", 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5699999999999998e-07 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251421+0000", 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8498000000000003e-05 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251428+0000", 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5850000000000002e-06 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251434+0000", 2024-09-02T10:02:40.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5720000000000002e-06 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251552+0000", 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011834 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251556+0000", 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7680000000000002e-06 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.475504+0000", 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22394810200000001 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.475642+0000", 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137949 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.475820+0000", 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017796399999999999 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.484236+0000", 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0084156140000000001 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.484285+0000", 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9793000000000003e-05 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.484346+0000", 2024-09-02T10:02:40.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0859000000000001e-05 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.484420+0000", 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3800999999999994e-05 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.486091+0000", 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016704479999999999 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67143 2.0 2:0bf8a217:::benchmark_data_smithi187_45882_object67142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251459+0000", 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469086520999999, 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23922826599999999, 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67143 2024-09-02T10:02:40.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251459+0000", 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251459+0000", 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251460+0000", 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7800000000000002e-07 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251499+0000", 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9702e-05 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251507+0000", 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8390000000000007e-06 2024-09-02T10:02:40.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251513+0000", 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7490000000000004e-06 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251577+0000", 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3571000000000005e-05 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251584+0000", 2024-09-02T10:02:40.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9890000000000003e-06 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.481980+0000", 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23039633800000001 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.482159+0000", 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017931300000000001 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.482352+0000", 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000192998 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.488298+0000", 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059454870000000002 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.488313+0000", 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5460000000000001e-05 2024-09-02T10:02:40.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.488341+0000", 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8042e-05 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.488378+0000", 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6293e-05 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.490688+0000", 2024-09-02T10:02:40.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023100070000000002 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67151 2.0 2:0d008404:::benchmark_data_smithi187_45882_object67150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251522+0000", 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.469024126000001, 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.248181598, 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.827 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67151 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251522+0000", 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251522+0000", 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251522+0000", 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8999999999999997e-07 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251552+0000", 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9374999999999999e-05 2024-09-02T10:02:40.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251558+0000", 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9249999999999998e-06 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251564+0000", 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1540000000000001e-06 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251617+0000", 2024-09-02T10:02:40.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3603999999999997e-05 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251621+0000", 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.664e-06 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.486140+0000", 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23451887099999999 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.486258+0000", 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117853 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.486374+0000", 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116797 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.493396+0000", 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070217049999999996 2024-09-02T10:02:40.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.497901+0000", 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045047589999999997 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.497985+0000", 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3819000000000005e-05 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.498019+0000", 2024-09-02T10:02:40.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4258000000000003e-05 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.499703+0000", 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016843240000000001 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67227 2.0 2:0efa4299:::benchmark_data_smithi187_45882_object67226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251573+0000", 2024-09-02T10:02:40.834 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.468972792000001, 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24924685399999999, 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67227 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251573+0000", 2024-09-02T10:02:40.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251573+0000", 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251573+0000", 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5400000000000002e-07 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251601+0000", 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7942000000000001e-05 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251609+0000", 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.153e-06 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251614+0000", 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3800000000000002e-06 2024-09-02T10:02:40.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251667+0000", 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3044999999999998e-05 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251671+0000", 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7799999999999998e-06 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.490742+0000", 2024-09-02T10:02:40.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23907071699999999 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.490898+0000", 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015618399999999999 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.491071+0000", 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017328299999999999 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.497939+0000", 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.006867789 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.497972+0000", 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3513000000000003e-05 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.499715+0000", 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001742646 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.499753+0000", 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7728000000000002e-05 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.500820+0000", 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010672399999999999 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67228 2.0 2:051987d6:::benchmark_data_smithi187_45882_object67227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251623+0000", 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.468923262000001, 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25360533099999999, 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67228 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251623+0000", 2024-09-02T10:02:40.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251623+0000", 2024-09-02T10:02:40.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251623+0000", 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.63e-07 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251651+0000", 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7744000000000001e-05 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251658+0000", 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1679999999999998e-06 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251663+0000", 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0830000000000003e-06 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251692+0000", 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9198999999999999e-05 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251696+0000", 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4860000000000002e-06 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.493431+0000", 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24173510300000001 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.493591+0000", 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160218 2024-09-02T10:02:40.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.493778+0000", 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186584 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.500830+0000", 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070525040000000002 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.503265+0000", 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024344700000000002 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.503332+0000", 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7432000000000001e-05 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.503381+0000", 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8523000000000003e-05 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505228+0000", 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018473540000000001 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67245 2.0 2:0a07d6c2:::benchmark_data_smithi187_45882_object67244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.468873756000001, 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25464746599999999, 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.849 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67245 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1699999999999999e-07 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251698+0000", 2024-09-02T10:02:40.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5471999999999999e-05 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251705+0000", 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.81e-06 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251710+0000", 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.169e-06 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251741+0000", 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0579e-05 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251744+0000", 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4489999999999999e-06 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.495787+0000", 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.244043118 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.495917+0000", 2024-09-02T10:02:40.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130099 2024-09-02T10:02:40.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.496029+0000", 2024-09-02T10:02:40.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011203499999999999 2024-09-02T10:02:40.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505241+0000", 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092117080000000007 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505301+0000", 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0078000000000003e-05 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505348+0000", 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6650999999999998e-05 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505388+0000", 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9974e-05 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.506320+0000", 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00093190700000000005 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67250 2.0 2:003e299c:::benchmark_data_smithi187_45882_object67249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 11.468825008, 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26035086499999999, 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:40.859 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67250 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.51e-07 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251748+0000", 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6327000000000001e-05 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251766+0000", 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8689e-05 2024-09-02T10:02:40.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251772+0000", 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6409999999999999e-06 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251811+0000", 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9113000000000003e-05 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251818+0000", 2024-09-02T10:02:40.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2339999999999997e-06 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.500859+0000", 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.249040962 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.500962+0000", 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103015 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:02:40.863 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.501075+0000", 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113211 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.506329+0000", 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052531840000000002 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.506377+0000", 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7914999999999998e-05 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.864 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.510244+0000", 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038674730000000002 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.510281+0000", 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6504000000000002e-05 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.512072+0000", 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001791146 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.865 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.866 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.866 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:02:40.866 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:02:40.866 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:02:40.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:40.891 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:40.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:40.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:41.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:41.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:41.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:41.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:41.143 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:41.143 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:41.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:41.269 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:41.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:41.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:41.397 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:41.398 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:41.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:02:41.438 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 66 16 79404 79388 75.1717 45.25 0.0264645 0.0132964 2024-09-02T10:02:41.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:41.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:41 smithi187 ceph-mon[20072]: 2.3 scrub ok 2024-09-02T10:02:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:41 smithi187 ceph-mon[20072]: osdmap e125: 12 total, 12 up, 12 in 2024-09-02T10:02:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:41 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:02:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:41 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:02:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:41 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:02:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:41 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:02:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:41 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-02T10:02:41.524 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:41.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:41.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:41 smithi162 ceph-mon[20279]: 2.3 scrub ok 2024-09-02T10:02:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:41 smithi162 ceph-mon[20279]: osdmap e125: 12 total, 12 up, 12 in 2024-09-02T10:02:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:41 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:02:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:41 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:02:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:41 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:02:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:41 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:02:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:41 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-02T10:02:41.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:41.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:41.655 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:41.656 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:41.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:41 smithi151 ceph-mon[16639]: 2.3 scrub ok 2024-09-02T10:02:41.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:41 smithi151 ceph-mon[16639]: osdmap e125: 12 total, 12 up, 12 in 2024-09-02T10:02:41.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:41 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:02:41.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:41 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:02:41.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:41 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:02:41.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:41 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:02:41.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:41 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"} : dispatch 2024-09-02T10:02:41.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:41.781 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:41.787 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:41.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:41.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:41.923 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:41.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:41.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:02:42.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:42.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:42.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:42.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:42.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:02:42.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:42.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:42.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:42.298 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:42.311 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:42.340 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:42.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:42.341 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:02:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:02:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:02:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: pgmap v356: 37 pgs: 1 active+recovering+undersized+remapped, 2 peering, 10 unknown, 24 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 717 op/s; 3593/227268 objects misplaced (1.581%); 3.6 MiB/s, 58 objects/s recovering 2024-09-02T10:02:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:02:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:02:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:02:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: 2.b scrub ok 2024-09-02T10:02:42.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-02T10:02:42.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: osdmap e126: 12 total, 12 up, 12 in 2024-09-02T10:02:42.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:02:42.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:42 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:02:42.439 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 67 16 80767 80751 75.3211 85.1875 0.00928677 0.0132712 2024-09-02T10:02:42.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:42.468 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:42.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:02:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:02:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:02:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: pgmap v356: 37 pgs: 1 active+recovering+undersized+remapped, 2 peering, 10 unknown, 24 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 717 op/s; 3593/227268 objects misplaced (1.581%); 3.6 MiB/s, 58 objects/s recovering 2024-09-02T10:02:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:02:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:02:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:02:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: 2.b scrub ok 2024-09-02T10:02:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-02T10:02:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: osdmap e126: 12 total, 12 up, 12 in 2024-09-02T10:02:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:02:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:42 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:02:42.570 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: pgmap v356: 37 pgs: 1 active+recovering+undersized+remapped, 2 peering, 10 unknown, 24 active+clean; 4.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 717 op/s; 3593/227268 objects misplaced (1.581%); 3.6 MiB/s, 58 objects/s recovering 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: 2.b scrub ok 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: osdmap e126: 12 total, 12 up, 12 in 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:02:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:42 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:02:42.596 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:42.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:42.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:42.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:42.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:42.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:42.856 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:42.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:42.865 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:42.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:42.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:42.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:43.047 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:02:43.047 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:02:43.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.047 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:81340 2.5 2:a54953d9:::benchmark_data_smithi187_45882_object81339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-02T10:02:43.047 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:42.919077+0000", 2024-09-02T10:02:43.047 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0038132919999999998, 2024-09-02T10:02:43.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0038473729999999999, 2024-09-02T10:02:43.047 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:43.047 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 81340 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.919077+0000", 2024-09-02T10:02:43.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.919077+0000", 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.919078+0000", 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.77e-06 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.919104+0000", 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6103000000000001e-05 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.919121+0000", 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6098000000000001e-05 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.919129+0000", 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2539999999999992e-06 2024-09-02T10:02:43.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:43.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.919190+0000", 2024-09-02T10:02:43.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0785999999999997e-05 2024-09-02T10:02:43.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:43.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.919302+0000", 2024-09-02T10:02:43.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011199100000000001 2024-09-02T10:02:43.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.919402+0000", 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010017399999999999 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.921508+0000", 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021060139999999998 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:43.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:81343 2.5 2:afe7000c:::benchmark_data_smithi187_45882_object81342:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:42.922220+0000", 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00066954899999999997, 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00074784199999999997, 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 81343 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.053 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.922220+0000", 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.922220+0000", 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.922221+0000", 2024-09-02T10:02:43.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1310000000000001e-06 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.922245+0000", 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.368e-05 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.922259+0000", 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3681e-05 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.922267+0000", 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8229999999999995e-06 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.922310+0000", 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3838000000000003e-05 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.922393+0000", 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2227999999999999e-05 2024-09-02T10:02:43.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:42.922475+0000", 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2130000000000001e-05 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 2 2024-09-02T10:02:43.057 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:02:43.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:43.113 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:43.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:43.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:43.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:43.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:43.306 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:43 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:02:43.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:43 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:02:43.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:43 smithi151 ceph-mon[16639]: osdmap e127: 12 total, 12 up, 12 in 2024-09-02T10:02:43.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:43 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:02:43.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:43 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:02:43.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:43 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:02:43.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:43 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:02:43.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1646996413' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:43.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:43 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:02:43.307 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:43 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:02:43.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:43.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:43.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:43.438 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 68 16 81996 81980 75.343 76.8125 0.0128807 0.0132671 2024-09-02T10:02:43.468 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:02:43.469 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:02:43.479 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:43.501 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:02:43.501 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:02:43.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:43.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:43 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:02:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:43 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:02:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:43 smithi187 ceph-mon[20072]: osdmap e127: 12 total, 12 up, 12 in 2024-09-02T10:02:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:43 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:02:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:43 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:02:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:43 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:02:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:43 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:02:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1646996413' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:43 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:02:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:43 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:02:43.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:02:43.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:02:43.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:02:43.561 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:02:43.591 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:02:43.592 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:02:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:43 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:02:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:43 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:02:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:43 smithi162 ceph-mon[20279]: osdmap e127: 12 total, 12 up, 12 in 2024-09-02T10:02:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:43 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:02:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:43 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:02:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:43 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:02:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:43 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:02:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1646996413' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:43 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:02:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:43 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:02:43.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:43.622 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:02:43.622 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:02:43.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:43.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:43.654 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:02:43.654 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:02:43.683 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:02:43.683 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:02:43.712 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:02:43.712 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:02:43.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:43.739 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:02:43.739 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:02:43.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:43.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:43.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:43.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:02:43.767 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:02:43.768 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:02:43.781 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:02:43.793 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:02:43.793 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:02:43.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:43.882 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:43.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:43.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:44.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:44.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:44.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:44.113 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:44.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:44.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:44.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:44.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:44.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:44.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:44.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:44.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:44.438 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 69 15 83068 83053 75.2229 67.0625 0.0181951 0.0132886 2024-09-02T10:02:44.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: pgmap v359: 37 pgs: 1 active+recovering+undersized+remapped, 6 peering, 3 unknown, 27 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 86 MiB/s wr, 1.37k op/s; 3593/238635 objects misplaced (1.506%); 3.7 MiB/s, 58 objects/s recovering 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3131897175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:44.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:02:44.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:02:44.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1646996413' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:44.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: osdmap e128: 12 total, 12 up, 12 in 2024-09-02T10:02:44.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:44 smithi187 ceph-mon[20072]: 2.f scrub starts 2024-09-02T10:02:44.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:44.528 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:02:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:02:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:02:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:02:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:02:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: pgmap v359: 37 pgs: 1 active+recovering+undersized+remapped, 6 peering, 3 unknown, 27 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 86 MiB/s wr, 1.37k op/s; 3593/238635 objects misplaced (1.506%); 3.7 MiB/s, 58 objects/s recovering 2024-09-02T10:02:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:02:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:02:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:02:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:02:44.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:02:44.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:02:44.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:02:44.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3131897175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:44.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:02:44.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:02:44.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1646996413' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:44.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: osdmap e128: 12 total, 12 up, 12 in 2024-09-02T10:02:44.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:44 smithi162 ceph-mon[20279]: 2.f scrub starts 2024-09-02T10:02:44.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:44.655 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:44.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:44.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:44.664 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:44.664 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:02:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:02:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:02:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:02:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:02:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: pgmap v359: 37 pgs: 1 active+recovering+undersized+remapped, 6 peering, 3 unknown, 27 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 86 MiB/s wr, 1.37k op/s; 3593/238635 objects misplaced (1.506%); 3.7 MiB/s, 58 objects/s recovering 2024-09-02T10:02:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:02:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:02:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:02:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:02:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:02:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:02:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:02:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3131897175' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:02:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:02:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1646996413' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: osdmap e128: 12 total, 12 up, 12 in 2024-09-02T10:02:44.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:44 smithi151 ceph-mon[16639]: 2.f scrub starts 2024-09-02T10:02:44.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:44.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:44.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:44.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:44.910 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:44.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:45.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:45.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:45.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:45.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:45.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:45.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:45.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:45.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:45.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:45.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:45.400 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:45.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:45.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:45.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:45.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:45 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:02:45.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:45 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:02:45.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:45 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:02:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:45 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:02:45.438 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 70 16 84181 84165 75.141 69.5 0.00780199 0.0132994 2024-09-02T10:02:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:45 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:02:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:45 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:02:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:45 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:02:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:45 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:02:45.526 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:45.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:45.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:45.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:45 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:02:45.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:45 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:02:45.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:45 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:02:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:45 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:02:45.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:02:45.644 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:02:45.644 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:02:45.644 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:02:45.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd out 10 2024-09-02T10:02:45.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:45.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:45.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:45.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:45.836 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:45.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:45.882 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:45.904 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:45.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:45.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:45.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:45.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:46.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:46.093 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:46.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:46.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:46.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:46.221 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:46.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:46.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:46.300 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:46 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:02:46.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:46 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:02:46.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:46 smithi151 ceph-mon[16639]: pgmap v361: 37 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 6 peering, 2 unknown, 27 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 951 B/s rd, 70 MiB/s wr, 1.13k op/s; 3593/241824 objects misplaced (1.486%); 96 KiB/s, 0 objects/s recovering 2024-09-02T10:02:46.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:46 smithi151 ceph-mon[16639]: 2.c scrub starts 2024-09-02T10:02:46.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:46 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:02:46.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:46 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:02:46.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:46 smithi151 ceph-mon[16639]: 2.f scrub ok 2024-09-02T10:02:46.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:46.355 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:46.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:46.438 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 71 16 84785 84769 74.6144 37.75 0.0118483 0.0133967 2024-09-02T10:02:46.457 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:46.462 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:46.462 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:46.481 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:46.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:46.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:46.584 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:46.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:46 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:02:46.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:46 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:02:46.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:46 smithi162 ceph-mon[20279]: pgmap v361: 37 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 6 peering, 2 unknown, 27 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 951 B/s rd, 70 MiB/s wr, 1.13k op/s; 3593/241824 objects misplaced (1.486%); 96 KiB/s, 0 objects/s recovering 2024-09-02T10:02:46.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:46 smithi162 ceph-mon[20279]: 2.c scrub starts 2024-09-02T10:02:46.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:46 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:02:46.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:46 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:02:46.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:46 smithi162 ceph-mon[20279]: 2.f scrub ok 2024-09-02T10:02:46.607 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:46.609 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:46.710 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:46.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:46.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:46 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:02:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:46 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:02:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:46 smithi187 ceph-mon[20072]: pgmap v361: 37 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 6 peering, 2 unknown, 27 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 951 B/s rd, 70 MiB/s wr, 1.13k op/s; 3593/241824 objects misplaced (1.486%); 96 KiB/s, 0 objects/s recovering 2024-09-02T10:02:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:46 smithi187 ceph-mon[20072]: 2.c scrub starts 2024-09-02T10:02:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:46 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:02:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:46 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:02:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:46 smithi187 ceph-mon[20072]: 2.f scrub ok 2024-09-02T10:02:46.836 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:46.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:02:46.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:46.860 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:46.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:46.986 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:46.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:47.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:47.112 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:47.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:47.163 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:46 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:02:46.724+0000 7fe3ffed4640 -1 mon.a@0(leader).osd e129 definitely_dead 0 2024-09-02T10:02:47.198 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:02:47.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:47.264 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:47.265 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:47.287 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.10. 2024-09-02T10:02:47.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:47.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:47.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:47.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:47.439 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 72 16 85529 85513 74.2238 46.5 0.14482 0.013443 2024-09-02T10:02:47.460 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:47.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:47.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:02:47.512 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:47.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:47.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:47.592 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:47 smithi187 ceph-mon[20072]: osdmap e129: 12 total, 12 up, 12 in 2024-09-02T10:02:47.592 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:47 smithi187 ceph-mon[20072]: 2.c scrub ok 2024-09-02T10:02:47.592 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3993227840' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-02T10:02:47.592 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:47 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-02T10:02:47.592 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:47 smithi187 ceph-mon[20072]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-02T10:02:47.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:47 smithi162 ceph-mon[20279]: osdmap e129: 12 total, 12 up, 12 in 2024-09-02T10:02:47.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:47 smithi162 ceph-mon[20279]: 2.c scrub ok 2024-09-02T10:02:47.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3993227840' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-02T10:02:47.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:47 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-02T10:02:47.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:47 smithi162 ceph-mon[20279]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-02T10:02:47.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:47.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:47.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:47.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:47.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:47 smithi151 ceph-mon[16639]: osdmap e129: 12 total, 12 up, 12 in 2024-09-02T10:02:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:47 smithi151 ceph-mon[16639]: 2.c scrub ok 2024-09-02T10:02:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3993227840' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-02T10:02:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:47 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-02T10:02:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:47 smithi151 ceph-mon[16639]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-02T10:02:47.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:47.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:47.761 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:47.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:47.886 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:47.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:47.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:48.009 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:48.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:48.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:48.078 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:48.079 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:48.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:48.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:48.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:48.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:48.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:48.255 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:48.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:48.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:48.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:48.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:48.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:48.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:48.439 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 73 16 85552 85536 73.226 1.4375 0.00372836 0.0134912 2024-09-02T10:02:48.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:48.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:48.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:48.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:48 smithi162 ceph-mon[20279]: 2.1 scrub starts 2024-09-02T10:02:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:48 smithi162 ceph-mon[20279]: pgmap v363: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 peering, 34 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 386 B/s rd, 84 MiB/s wr, 1.34k op/s; 3292/250587 objects misplaced (1.314%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:02:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:48 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-02T10:02:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:48 smithi162 ceph-mon[20279]: osdmap e130: 12 total, 12 up, 11 in 2024-09-02T10:02:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:48 smithi162 ceph-mon[20279]: 2.a scrub starts 2024-09-02T10:02:48.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:48.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:48.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:48.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:48.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:48 smithi151 ceph-mon[16639]: 2.1 scrub starts 2024-09-02T10:02:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:48 smithi151 ceph-mon[16639]: pgmap v363: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 peering, 34 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 386 B/s rd, 84 MiB/s wr, 1.34k op/s; 3292/250587 objects misplaced (1.314%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:02:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:48 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-02T10:02:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:48 smithi151 ceph-mon[16639]: osdmap e130: 12 total, 12 up, 11 in 2024-09-02T10:02:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:48 smithi151 ceph-mon[16639]: 2.a scrub starts 2024-09-02T10:02:48.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:48.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:48 smithi187 ceph-mon[20072]: 2.1 scrub starts 2024-09-02T10:02:48.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:48 smithi187 ceph-mon[20072]: pgmap v363: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 1 peering, 34 active+clean; 5.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 386 B/s rd, 84 MiB/s wr, 1.34k op/s; 3292/250587 objects misplaced (1.314%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:02:48.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:48 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-02T10:02:48.748 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:48 smithi187 ceph-mon[20072]: osdmap e130: 12 total, 12 up, 11 in 2024-09-02T10:02:48.749 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:48 smithi187 ceph-mon[20072]: 2.a scrub starts 2024-09-02T10:02:48.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:48.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:48.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:02:48.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:48.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:48.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:48.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:49.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:49.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:49.080 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:49.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:49.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:49.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:49.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:49.252 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:49.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:49.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:49.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:49.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:49.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:49.439 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 74 16 85573 85557 72.2542 1.3125 0.00390965 0.0135892 2024-09-02T10:02:49.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:49.486 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:49 smithi187 ceph-mon[20072]: 2.d scrub starts 2024-09-02T10:02:49.486 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:49 smithi187 ceph-mon[20072]: 2.1 scrub ok 2024-09-02T10:02:49.486 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:49 smithi187 ceph-mon[20072]: osdmap e131: 12 total, 12 up, 11 in 2024-09-02T10:02:49.486 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:49 smithi187 ceph-mon[20072]: 2.6 scrub starts 2024-09-02T10:02:49.486 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3326160829' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:02:49.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:49 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:49.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3326160829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:02:49.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:49 smithi187 ceph-mon[20072]: osdmap e132: 12 total, 12 up, 11 in 2024-09-02T10:02:49.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:49.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:49.504 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:49.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:49 smithi162 ceph-mon[20279]: 2.d scrub starts 2024-09-02T10:02:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:49 smithi162 ceph-mon[20279]: 2.1 scrub ok 2024-09-02T10:02:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:49 smithi162 ceph-mon[20279]: osdmap e131: 12 total, 12 up, 11 in 2024-09-02T10:02:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:49 smithi162 ceph-mon[20279]: 2.6 scrub starts 2024-09-02T10:02:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3326160829' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:02:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:49 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3326160829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:02:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:49 smithi162 ceph-mon[20279]: osdmap e132: 12 total, 12 up, 11 in 2024-09-02T10:02:49.595 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:49.595 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:49.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:49.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:49.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:49.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:49 smithi151 ceph-mon[16639]: 2.d scrub starts 2024-09-02T10:02:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:49 smithi151 ceph-mon[16639]: 2.1 scrub ok 2024-09-02T10:02:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:49 smithi151 ceph-mon[16639]: osdmap e131: 12 total, 12 up, 11 in 2024-09-02T10:02:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:49 smithi151 ceph-mon[16639]: 2.6 scrub starts 2024-09-02T10:02:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3326160829' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:02:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:49 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3326160829' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:02:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:49 smithi151 ceph-mon[16639]: osdmap e132: 12 total, 12 up, 11 in 2024-09-02T10:02:49.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:49.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:49.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:49.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:49.873 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:49.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:49.925 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:02:49.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:49.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:49.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:50.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:50.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:50.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:50.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:50.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:02:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:50.245 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:50.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:50.346 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:50.368 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:50.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:50.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 75 16 85747 85731 71.4358 10.875 0.0291911 0.013994 2024-09-02T10:02:50.470 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:50.500 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:50.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:50 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:02:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:50 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:02:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:50 smithi187 ceph-mon[20072]: 2.d scrub ok 2024-09-02T10:02:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:50 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:02:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:50 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:02:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:50 smithi187 ceph-mon[20072]: pgmap v366: 37 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 4 peering, 30 active+clean; 5.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 387 B/s rd, 60 MiB/s wr, 962 op/s; 3292/253905 objects misplaced (1.297%); 3.5 MiB/s, 56 objects/s recovering 2024-09-02T10:02:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:50 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-02T10:02:50.527 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:50.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:50 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:02:50.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:50 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:02:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:50 smithi162 ceph-mon[20279]: 2.d scrub ok 2024-09-02T10:02:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:50 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:02:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:50 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:02:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:50 smithi162 ceph-mon[20279]: pgmap v366: 37 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 4 peering, 30 active+clean; 5.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 387 B/s rd, 60 MiB/s wr, 962 op/s; 3292/253905 objects misplaced (1.297%); 3.5 MiB/s, 56 objects/s recovering 2024-09-02T10:02:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:50 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-02T10:02:50.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:50.627 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:50.628 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:50 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:02:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:50 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:02:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:50 smithi151 ceph-mon[16639]: 2.d scrub ok 2024-09-02T10:02:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:50 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:02:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:50 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:02:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:50 smithi151 ceph-mon[16639]: pgmap v366: 37 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 4 peering, 30 active+clean; 5.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 387 B/s rd, 60 MiB/s wr, 962 op/s; 3292/253905 objects misplaced (1.297%); 3.5 MiB/s, 56 objects/s recovering 2024-09-02T10:02:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:50 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-02T10:02:50.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:50.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:50.750 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:50.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:50.874 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:50.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:50.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:51.004 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:51.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:51.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:51.131 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:51.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:51.233 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:51.259 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:51.260 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:51.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:51.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:51 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:02:51.391 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:02:51.391 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:02:51.391 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:51 smithi151 ceph-mon[16639]: osdmap e133: 12 total, 12 up, 11 in 2024-09-02T10:02:51.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:51.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:51.439 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 76 16 86531 86515 71.1406 49 0.0138075 0.0140517 2024-09-02T10:02:51.459 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:02:51.459 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:02:51.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:51.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:51.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:51.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:51 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:02:51.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:02:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:02:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:51 smithi162 ceph-mon[20279]: osdmap e133: 12 total, 12 up, 11 in 2024-09-02T10:02:51.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:51.659 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:51.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:51 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:02:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:02:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:02:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:51 smithi187 ceph-mon[20072]: osdmap e133: 12 total, 12 up, 11 in 2024-09-02T10:02:51.760 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:51.785 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:51.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:51.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:51.909 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:51.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:51.933 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:52.010 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:52.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:52.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:52.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:52.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:02:52.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:52.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:52.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:52.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:52.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:52.286 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:52.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:52.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:52.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:52.395 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:52.416 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:52.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:52.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:52.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 77 16 87210 87194 70.7678 42.4375 0.00365625 0.014125 2024-09-02T10:02:52.491 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:52 smithi151 ceph-mon[16639]: pgmap v369: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 peering, 31 active+clean; 5.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1023 B/s rd, 29 MiB/s wr, 467 op/s; 5167/256173 objects misplaced (2.017%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:02:52.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:52 smithi187 ceph-mon[20072]: pgmap v369: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 peering, 31 active+clean; 5.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1023 B/s rd, 29 MiB/s wr, 467 op/s; 5167/256173 objects misplaced (2.017%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:02:52.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:52.548 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:52.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:52.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:02:52.558 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:02:52.593 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:02:52.593 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:02:52.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:52 smithi162 ceph-mon[20279]: pgmap v369: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 4 peering, 31 active+clean; 5.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 1023 B/s rd, 29 MiB/s wr, 467 op/s; 5167/256173 objects misplaced (2.017%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:02:52.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:02:52.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:02:52.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:52.655 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:02:52.655 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:02:52.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:52.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:52.688 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:02:52.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:02:52.720 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:02:52.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:02:52.748 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:02:52.748 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:02:52.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:52.783 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:02:52.783 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:02:52.807 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:52.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:52.819 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:02:52.819 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:02:52.852 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:02:52.852 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:02:52.884 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:02:52.884 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:02:52.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:52.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:02:52.914 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:02:52.929 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:02:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:02:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:02:52.929 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:02:52.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:52.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:53.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:53.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:53.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:53.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:53.173 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:53.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:53.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:53.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:53.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:53.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:53.331 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:53.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:53.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 78 16 88143 88127 70.608 58.3125 0.0182302 0.0141562 2024-09-02T10:02:53.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:53.458 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:53.501 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1793554841' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:53.559 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:53.584 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:53.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:53.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1793554841' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:53.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1793554841' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:53.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:53.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:53.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:53.842 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:53.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:53.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:02:53.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:02:53.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:53.968 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:53.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:53.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:54.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:54.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:54.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:54.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:54.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:54.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:54.236 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:54.293 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:54.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:54.361 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:54.362 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:54.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 79 16 88982 88966 70.378 52.4375 0.0525822 0.0142029 2024-09-02T10:02:54.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:54.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:54.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:54.586 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:54 smithi151 ceph-mon[16639]: pgmap v370: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 34 MiB/s wr, 542 op/s; 15543/260010 objects misplaced (5.978%); 5.8 MiB/s, 93 objects/s recovering 2024-09-02T10:02:54.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:54 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-02T10:02:54.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:54 smithi162 ceph-mon[20279]: pgmap v370: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 34 MiB/s wr, 542 op/s; 15543/260010 objects misplaced (5.978%); 5.8 MiB/s, 93 objects/s recovering 2024-09-02T10:02:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:54 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-02T10:02:54.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:54.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:54.672 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:54 smithi187 ceph-mon[20072]: pgmap v370: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 34 MiB/s wr, 542 op/s; 15543/260010 objects misplaced (5.978%); 5.8 MiB/s, 93 objects/s recovering 2024-09-02T10:02:54.672 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:54 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-02T10:02:54.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:54.743 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:54.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:54.803 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:02:54.803 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:02:54.803 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:02:54.803 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:02:54.803 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:02:54.845 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:54.873 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:54.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:54.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:55.000 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:55.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:55.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:55.127 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:55.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:55.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:55.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:55.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:55.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:55.315 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:02:55.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:55.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:55.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:55.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:02:55.440192+0000 min lat: 0.00101783 max lat: 3.08878 avg lat: 0.0142746 2024-09-02T10:02:55.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:02:55.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 80 16 89657 89641 70.0256 42.1875 0.00507837 0.0142746 2024-09-02T10:02:55.493 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:55 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/804657491' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:55.493 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:55 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:55.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:55.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:02:55.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:55.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:55.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:55 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/804657491' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:55.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:55 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:55.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:55.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:02:55.664 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:55.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:55.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:55 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/804657491' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:55.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:55 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:02:55.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:55.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:55.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:55.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:55.828 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:55.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:55.935 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:55.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:56.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:56.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:56.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:56.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:56.198 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:56.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:56.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:56.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:56.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:56.361 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:02:56.361 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:02:56.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:02:56.362 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:02:56.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.362 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1131 2.0 e96/92 2:021aab7d:::benchmark_data_smithi187_45882_object1130:head v 96'61, mlcod=96'58)", 2024-09-02T10:02:56.362 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:02:56.362 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 80.459567707999994, 2024-09-02T10:02:56.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077611448, 2024-09-02T10:02:56.363 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.363 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.363 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.363 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:02:56.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:02:56.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898452+0000", 2024-09-02T10:02:56.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8899999999999998e-07 2024-09-02T10:02:56.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898480+0000", 2024-09-02T10:02:56.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.817e-05 2024-09-02T10:02:56.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975643+0000", 2024-09-02T10:02:56.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077163063000000004 2024-09-02T10:02:56.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975651+0000", 2024-09-02T10:02:56.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.028e-06 2024-09-02T10:02:56.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975697+0000", 2024-09-02T10:02:56.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5544999999999998e-05 2024-09-02T10:02:56.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975710+0000", 2024-09-02T10:02:56.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2928000000000001e-05 2024-09-02T10:02:56.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.976007+0000", 2024-09-02T10:02:56.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029711499999999998 2024-09-02T10:02:56.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.976063+0000", 2024-09-02T10:02:56.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5810000000000003e-05 2024-09-02T10:02:56.372 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.372 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.372 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.372 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1142 2.7 e96/92 2:e2cb33eb:::benchmark_data_smithi187_45882_object1141:head v 96'87, mlcod=96'83)", 2024-09-02T10:02:56.373 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:02:56.373 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 80.451065360000001, 2024-09-02T10:02:56.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18675391099999999, 2024-09-02T10:02:56.373 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.373 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.374 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.374 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:02:56.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:02:56.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906955+0000", 2024-09-02T10:02:56.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.181e-06 2024-09-02T10:02:56.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906991+0000", 2024-09-02T10:02:56.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6523999999999998e-05 2024-09-02T10:02:56.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093330+0000", 2024-09-02T10:02:56.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18633878100000001 2024-09-02T10:02:56.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093335+0000", 2024-09-02T10:02:56.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5909999999999998e-06 2024-09-02T10:02:56.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093381+0000", 2024-09-02T10:02:56.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6925999999999998e-05 2024-09-02T10:02:56.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093396+0000", 2024-09-02T10:02:56.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4216000000000001e-05 2024-09-02T10:02:56.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.381 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093682+0000", 2024-09-02T10:02:56.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028659900000000001 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093707+0000", 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5092999999999998e-05 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.382 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1442 2.0 e96/92 2:04aa890d:::benchmark_data_smithi187_45882_object1441:head v 96'80, mlcod=96'77)", 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 80.168650005999993, 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087334971999999997, 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8100000000000002e-07 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189467+0000", 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7173e-05 2024-09-02T10:02:56.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276310+0000", 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.086843528000000003 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276315+0000", 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8670000000000002e-06 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276365+0000", 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9459999999999997e-05 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276382+0000", 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7045999999999999e-05 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276666+0000", 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028404500000000002 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276704+0000", 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8172e-05 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11303 2.7 e98/92 2:eeb591e6:::benchmark_data_smithi187_45882_object11302:head v 98'755, mlcod=98'751)", 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 74.043671836000001, 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.117126889, 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.387 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314348+0000", 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8599999999999998e-07 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314428+0000", 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0412000000000002e-05 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430860+0000", 2024-09-02T10:02:56.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.116432137 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430866+0000", 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.083e-06 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430918+0000", 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.215e-05 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430932+0000", 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3591000000000001e-05 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.431445+0000", 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051315900000000001 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.431474+0000", 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8671e-05 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.391 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.392 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11334 2.4 e98/92 2:22fc355e:::benchmark_data_smithi187_45882_object11333:head v 98'688, mlcod=98'684)", 2024-09-02T10:02:56.392 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:02:56.392 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 74.00435143, 2024-09-02T10:02:56.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.072501649000000001, 2024-09-02T10:02:56.392 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.392 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.392 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.393 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:02:56.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353668+0000", 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5899999999999995e-07 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353720+0000", 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1539999999999998e-05 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425505+0000", 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.071785287000000003 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425512+0000", 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2980000000000003e-06 2024-09-02T10:02:56.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425576+0000", 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3614000000000005e-05 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425598+0000", 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2076e-05 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426142+0000", 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00054422300000000004 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426169+0000", 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6852e-05 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.397 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11352 2.4 e98/92 2:2477406b:::benchmark_data_smithi187_45882_object11351:head v 98'690, mlcod=98'686)", 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 73.931341343, 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11620186, 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426679+0000", 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2580000000000001e-06 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426733+0000", 2024-09-02T10:02:56.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4398999999999999e-05 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542090+0000", 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.115357134 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542105+0000", 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4596999999999999e-05 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542170+0000", 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5376999999999994e-05 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542194+0000", 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3611000000000001e-05 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542825+0000", 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00063158400000000003 2024-09-02T10:02:56.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542879+0000", 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3900000000000002e-05 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11358 2.7 e98/92 2:e5a3134d:::benchmark_data_smithi187_45882_object11357:head v 98'756, mlcod=98'753)", 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 73.927229328999999, 2024-09-02T10:02:56.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078694929999999996, 2024-09-02T10:02:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:02:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:02:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430790+0000", 2024-09-02T10:02:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7499999999999999e-07 2024-09-02T10:02:56.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430846+0000", 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5544999999999998e-05 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508940+0000", 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078093869999999996 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508946+0000", 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7120000000000003e-06 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508990+0000", 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4095999999999999e-05 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509005+0000", 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5000999999999999e-05 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509456+0000", 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00045036799999999998 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509484+0000", 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8563000000000001e-05 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11782 2.4 e98/92 2:2417a9e9:::benchmark_data_smithi187_45882_object11781:head v 98'718, mlcod=98'714)", 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 73.615026337000003, 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12480622099999999, 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742993+0000", 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2399999999999997e-07 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.743050+0000", 2024-09-02T10:02:56.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6356e-05 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867097+0000", 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.124047 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867103+0000", 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0399999999999998e-06 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867172+0000", 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8996000000000001e-05 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867187+0000", 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5218999999999999e-05 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867764+0000", 2024-09-02T10:02:56.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00057719100000000001 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867799+0000", 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4694999999999999e-05 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11788 2.7 e98/92 2:ee0a4980:::benchmark_data_smithi187_45882_object11787:head v 98'788, mlcod=98'785)", 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:02:56.413 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 73.607742565999999, 2024-09-02T10:02:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.134742468, 2024-09-02T10:02:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:02:56.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750278+0000", 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5430000000000001e-06 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750323+0000", 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4706000000000002e-05 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884234+0000", 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13391187800000001 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884240+0000", 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4190000000000001e-06 2024-09-02T10:02:56.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884280+0000", 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0012000000000002e-05 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884316+0000", 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6318e-05 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884977+0000", 2024-09-02T10:02:56.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00066127600000000003 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.885019+0000", 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1316000000000002e-05 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11794 2.4 e98/92 2:22bc5fbb:::benchmark_data_smithi187_45882_object11793:head v 98'719, mlcod=98'715)", 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 73.600669726999996, 2024-09-02T10:02:56.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.125805317, 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757350+0000", 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5300000000000003e-07 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757408+0000", 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8291000000000001e-05 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882534+0000", 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.125126182 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882540+0000", 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0689999999999998e-06 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882583+0000", 2024-09-02T10:02:56.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2962e-05 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882599+0000", 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5718000000000001e-05 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.883121+0000", 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052199199999999999 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.883154+0000", 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3349999999999997e-05 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11897 2.4 e98/92 2:2a9822cc:::benchmark_data_smithi187_45882_object11896:head v 98'727, mlcod=98'723)", 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 73.398989516, 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088729108000000001, 2024-09-02T10:02:56.423 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959030+0000", 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0200000000000001e-07 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959076+0000", 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5896000000000002e-05 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047130+0000", 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088054190000000004 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047135+0000", 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2379999999999997e-06 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047218+0000", 2024-09-02T10:02:56.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2892000000000005e-05 2024-09-02T10:02:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047233+0000", 2024-09-02T10:02:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5126999999999999e-05 2024-09-02T10:02:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047729+0000", 2024-09-02T10:02:56.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000495847 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047758+0000", 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9116e-05 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:13212 2.0 e98/92 2:00f0b824:::benchmark_data_smithi187_45882_object13211:head v 98'832, mlcod=98'829)", 2024-09-02T10:02:56.428 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.586304573999996, 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.075308911000000006, 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:02:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771715+0000", 2024-09-02T10:02:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1599999999999999e-06 2024-09-02T10:02:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771775+0000", 2024-09-02T10:02:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9979999999999998e-05 2024-09-02T10:02:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846360+0000", 2024-09-02T10:02:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.074584792999999996 2024-09-02T10:02:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846367+0000", 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1489999999999997e-06 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846446+0000", 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8652999999999994e-05 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846461+0000", 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5254e-05 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846996+0000", 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053431400000000003 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.847023+0000", 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7608000000000001e-05 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.433 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:13565 2.0 e98/92 2:0460cc71:::benchmark_data_smithi187_45882_object13564:head v 98'854, mlcod=98'850)", 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.296792100000005, 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29179487700000001, 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9400000000000005e-07 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061284+0000", 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6251000000000001e-05 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352274+0000", 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29099082599999998 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352280+0000", 2024-09-02T10:02:56.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1950000000000002e-06 2024-09-02T10:02:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352346+0000", 2024-09-02T10:02:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5837e-05 2024-09-02T10:02:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352360+0000", 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.433e-05 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352980+0000", 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00062060200000000005 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.353022+0000", 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1142e-05 2024-09-02T10:02:56.438 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:22449 2.0 e104/92 2:0733b693:::benchmark_data_smithi187_45882_object22448:head v 104'1397, mlcod=104'1393)", 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 65.077308994000006, 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12946363599999999, 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280711+0000", 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1800000000000005e-07 2024-09-02T10:02:56.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280779+0000", 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8644000000000002e-05 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409376+0000", 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12859704799999999 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409382+0000", 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5589999999999996e-06 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409441+0000", 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8746000000000003e-05 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409466+0000", 2024-09-02T10:02:56.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5493000000000001e-05 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.410146+0000", 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00067948499999999996 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.410173+0000", 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7943e-05 2024-09-02T10:02:56.443 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:22536 2.0 e104/92 2:01400da2:::benchmark_data_smithi187_45882_object22535:head v 104'1399, mlcod=104'1395)", 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 64.916420223000003, 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077533989999999997, 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:02:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:02:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:02:56.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.47e-07 2024-09-02T10:02:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441696+0000", 2024-09-02T10:02:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6059999999999998e-05 2024-09-02T10:02:56.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518264+0000", 2024-09-02T10:02:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.076568862000000001 2024-09-02T10:02:56.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518272+0000", 2024-09-02T10:02:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5789999999999998e-06 2024-09-02T10:02:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518333+0000", 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1159000000000002e-05 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518358+0000", 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5182e-05 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.519104+0000", 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00074525100000000005 2024-09-02T10:02:56.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.519133+0000", 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9050000000000001e-05 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:36974 2.7 e110/92 2:ea266fd4:::benchmark_data_smithi187_45882_object36973:head v 110'2332, mlcod=110'2328)", 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 53.758069472000003, 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080151230000000004, 2024-09-02T10:02:56.451 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599953+0000", 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6090000000000001e-06 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599999+0000", 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5561e-05 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679063+0000", 2024-09-02T10:02:56.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.079064375000000006 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679068+0000", 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3619999999999997e-06 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679116+0000", 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8084999999999998e-05 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679132+0000", 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5264000000000002e-05 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.680070+0000", 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00093846200000000004 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.680101+0000", 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0511999999999999e-05 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.455 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:37371 2.0 e110/92 2:00cc6826:::benchmark_data_smithi187_45882_object37370:head v 110'2365, mlcod=110'2361)", 2024-09-02T10:02:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:02:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 53.229690992999998, 2024-09-02T10:02:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083380836999999999, 2024-09-02T10:02:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.456 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:02:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128329+0000", 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.207e-06 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128397+0000", 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7869999999999999e-05 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210301+0000", 2024-09-02T10:02:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081903702999999994 2024-09-02T10:02:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210307+0000", 2024-09-02T10:02:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.737e-06 2024-09-02T10:02:56.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210368+0000", 2024-09-02T10:02:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0795000000000003e-05 2024-09-02T10:02:56.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210383+0000", 2024-09-02T10:02:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4538999999999999e-05 2024-09-02T10:02:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.211539+0000", 2024-09-02T10:02:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011564310000000001 2024-09-02T10:02:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.211709+0000", 2024-09-02T10:02:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000169555 2024-09-02T10:02:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:67961 2.7 e120/92 2:ed776389:::benchmark_data_smithi187_45882_object67960:head v 120'4269, mlcod=120'4266)", 2024-09-02T10:02:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:02:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 26.449313534000002, 2024-09-02T10:02:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.095923729999999999, 2024-09-02T10:02:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.464 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:02:56.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:02:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908706+0000", 2024-09-02T10:02:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1399999999999997e-07 2024-09-02T10:02:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908770+0000", 2024-09-02T10:02:56.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4410000000000002e-05 2024-09-02T10:02:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002909+0000", 2024-09-02T10:02:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.094138870999999999 2024-09-02T10:02:56.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002918+0000", 2024-09-02T10:02:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3489999999999992e-06 2024-09-02T10:02:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002977+0000", 2024-09-02T10:02:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9299999999999998e-05 2024-09-02T10:02:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002993+0000", 2024-09-02T10:02:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6424000000000001e-05 2024-09-02T10:02:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.004595+0000", 2024-09-02T10:02:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001601725 2024-09-02T10:02:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.004629+0000", 2024-09-02T10:02:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4037e-05 2024-09-02T10:02:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:78685 2.4 e124/92 2:26d5ebbe:::benchmark_data_smithi187_45882_object78684:head v 124'4867, mlcod=124'4864)", 2024-09-02T10:02:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:02:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.909524337000001, 2024-09-02T10:02:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.135791625, 2024-09-02T10:02:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.474 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:02:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:02:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448496+0000", 2024-09-02T10:02:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.257e-06 2024-09-02T10:02:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448603+0000", 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106925 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.581994+0000", 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13339121900000001 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582001+0000", 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7950000000000002e-06 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582057+0000", 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6724000000000001e-05 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582074+0000", 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7037999999999998e-05 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.584242+0000", 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021673809999999999 2024-09-02T10:02:56.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.584286+0000", 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4286000000000003e-05 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:79093 2.4 e125/92 2:2f2be3a1:::benchmark_data_smithi187_45882_object79092:head v 125'4892, mlcod=125'4888)", 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.405091367000001, 2024-09-02T10:02:56.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16453667699999999, 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952928+0000", 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952991+0000", 2024-09-02T10:02:56.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3367999999999994e-05 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115366+0000", 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16237467799999999 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115374+0000", 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6240000000000002e-06 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115428+0000", 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3885999999999999e-05 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115442+0000", 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4438e-05 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.117418+0000", 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019764600000000002 2024-09-02T10:02:56.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:02:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.117464+0000", 2024-09-02T10:02:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5694e-05 2024-09-02T10:02:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:56.485 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:56.485 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:02:56.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:56.486 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:02:56.486 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:02:56.486 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-02T10:02:56.486 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-02T10:02:56.486 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-02T10:02:56.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-02T10:02:56.487 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 81 16 90426 90410 69.7543 48.0625 0.00683897 0.0143303 2024-09-02T10:02:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:56 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:02:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:56 smithi187 ceph-mon[20072]: pgmap v371: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 24 MiB/s wr, 382 op/s; 15543/261723 objects misplaced (5.939%); 5.0 MiB/s, 79 objects/s recovering 2024-09-02T10:02:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:56 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:56 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:56 smithi187 ceph-mon[20072]: osdmap e134: 12 total, 12 up, 11 in 2024-09-02T10:02:56.513 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:56.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:56.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:56 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:02:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:56 smithi162 ceph-mon[20279]: pgmap v371: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 24 MiB/s wr, 382 op/s; 15543/261723 objects misplaced (5.939%); 5.0 MiB/s, 79 objects/s recovering 2024-09-02T10:02:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:56 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:56 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:56 smithi162 ceph-mon[20279]: osdmap e134: 12 total, 12 up, 11 in 2024-09-02T10:02:56.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:02:56.640 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:56.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:02:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:56 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:02:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:56 smithi151 ceph-mon[16639]: pgmap v371: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.3 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 24 MiB/s wr, 382 op/s; 15543/261723 objects misplaced (5.939%); 5.0 MiB/s, 79 objects/s recovering 2024-09-02T10:02:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:56 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:02:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:56 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:02:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:56 smithi151 ceph-mon[16639]: osdmap e134: 12 total, 12 up, 11 in 2024-09-02T10:02:56.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:56.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:56.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:56.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:56.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:56.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:56.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:56.911 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:57.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:57.038 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:57.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:57.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:57.165 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:57.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:57.267 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:57.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:02:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:57.293 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:57.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:57.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:02:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:57.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:57.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:02:57.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 82 16 91234 91218 69.5195 50.5 0.00622074 0.0143783 2024-09-02T10:02:57.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:02:57.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:57.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:02:57.599 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:57.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:02:57.685 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:57.685 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:02:57.702 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:57.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:57.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:02:57.805 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:57.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:57.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:57.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:57.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:57.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:58.062 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:58.087 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:58.089 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:58.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:58.220 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:58.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:58.242 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:91977 2.c 2:310d123d:::benchmark_data_smithi187_45882_object91976:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:58.170007+0000", 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.039256995000000003, 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.039287256, 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:02:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 91977 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.170007+0000", 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.170007+0000", 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.170009+0000", 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.052e-06 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.170033+0000", 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3972e-05 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.170048+0000", 2024-09-02T10:02:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4686e-05 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.170057+0000", 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8389999999999996e-06 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.170126+0000", 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9147999999999996e-05 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.170265+0000", 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013886200000000001 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.170430+0000", 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016534300000000001 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.172882+0000", 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024524439999999998 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.173272+0000", 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039006400000000002 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:92012 2.a 2:506b08fb:::benchmark_data_smithi187_45882_object92011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:58.203789+0000", 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0054749960000000002, 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0055824050000000004, 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 92012 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.203789+0000", 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.203789+0000", 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.203791+0000", 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2879999999999999e-06 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.203818+0000", 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.711e-05 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.203847+0000", 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8351e-05 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.203858+0000", 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1121000000000001e-05 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.203926+0000", 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8466999999999993e-05 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204063+0000", 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013642 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204248+0000", 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000185677 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:02:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.207210+0000", 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029612929999999998 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:92013 2.c 2:3acd35d5:::benchmark_data_smithi187_45882_object92012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:58.204579+0000", 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0046849960000000003, 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004858322, 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:02:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 92013 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204579+0000", 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204579+0000", 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204580+0000", 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.238e-06 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204602+0000", 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1639000000000001e-05 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204615+0000", 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3181e-05 2024-09-02T10:02:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204624+0000", 2024-09-02T10:02:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.721e-06 2024-09-02T10:02:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204693+0000", 2024-09-02T10:02:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9046000000000003e-05 2024-09-02T10:02:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204816+0000", 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012301200000000001 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.204984+0000", 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000168584 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.208047+0000", 2024-09-02T10:02:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030629810000000002 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.208629+0000", 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058160600000000003 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:92015 2.b e134/92 2:de431381:::benchmark_data_smithi187_45882_object92014:head v 134'5713, mlcod=134'5711)", 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:58.207590+0000", 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.001673742, 2024-09-02T10:02:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019130379999999999, 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.207590+0000", 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.207590+0000", 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.207591+0000", 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8599999999999998e-07 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.207616+0000", 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5452e-05 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.207652+0000", 2024-09-02T10:02:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5259999999999998e-05 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.207659+0000", 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8220000000000007e-06 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.207718+0000", 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8377e-05 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:58.207737+0000", 2024-09-02T10:02:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9301e-05 2024-09-02T10:02:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:02:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:02:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:02:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 4 2024-09-02T10:02:58.262 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:02:58.322 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:58.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:58.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:58.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:58.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:58 smithi151 ceph-mon[16639]: pgmap v373: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 34 MiB/s wr, 541 op/s; 15088/268278 objects misplaced (5.624%); 8.0 MiB/s, 127 objects/s recovering 2024-09-02T10:02:58.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 83 16 92205 92189 69.413 60.6875 0.00395747 0.0144004 2024-09-02T10:02:58.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:02:58.471 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:58.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:02:58.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:02:58.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:58 smithi162 ceph-mon[20279]: pgmap v373: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 34 MiB/s wr, 541 op/s; 15088/268278 objects misplaced (5.624%); 8.0 MiB/s, 127 objects/s recovering 2024-09-02T10:02:58.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:58.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:02:58.667 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:02:58.668 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:02:58.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:58 smithi187 ceph-mon[20072]: pgmap v373: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 34 MiB/s wr, 541 op/s; 15088/268278 objects misplaced (5.624%); 8.0 MiB/s, 127 objects/s recovering 2024-09-02T10:02:58.703 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:02:58.704 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:02:58.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:58.720 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:02:58.720 INFO:teuthology.orchestra.run.smithi151.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-02T10:02:58.720 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:02:58.733 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:02:58.733 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:02:58.737 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:58.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:58.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:02:58.765 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:02:58.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:02:58.797 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:02:58.832 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:02:58.833 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:02:58.839 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:02:58.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:02:58.862 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:02:58.864 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:58.865 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:02:58.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:02:58.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:02:58.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:02:58.926 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:02:58.952 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:02:58.953 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:02:58.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:02:58.982 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:02:58.982 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:02:58.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:02:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:58.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:02:59.014 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:02:59.014 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:02:59.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:02:59.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:02:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:59.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:59.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:02:59.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:02:59.252 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:59.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:02:59.334 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:02:59.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:59.380 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:59.381 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:59.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 84 16 93076 93060 69.2347 54.4375 0.00466439 0.0144371 2024-09-02T10:02:59.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:59.509 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:59.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:59.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:02:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3830541578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:02:59 smithi162 ceph-mon[20279]: osdmap e135: 12 total, 12 up, 11 in 2024-09-02T10:02:59.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3830541578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:59.618 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:02:59 smithi151 ceph-mon[16639]: osdmap e135: 12 total, 12 up, 11 in 2024-09-02T10:02:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:59.636 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:59.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:59.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:02:59.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3830541578' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:02:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:02:59 smithi187 ceph-mon[20072]: osdmap e135: 12 total, 12 up, 11 in 2024-09-02T10:02:59.760 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:02:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:59.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:02:59.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:02:59.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:02:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:02:59.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:02:59.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:00.019 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:00.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:00.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:00.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:00.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:00.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:00.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:00.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:00.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:00.389 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:00.404 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:00.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:00.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:00.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 85 16 93795 93779 68.9488 44.9375 0.122958 0.0144839 2024-09-02T10:03:00.478 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:00 smithi187 ceph-mon[20072]: pgmap v375: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 39 MiB/s wr, 630 op/s; 14909/271320 objects misplaced (5.495%); 6.9 MiB/s, 111 objects/s recovering 2024-09-02T10:03:00.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:00.536 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:00.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:00.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:00 smithi162 ceph-mon[20279]: pgmap v375: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 39 MiB/s wr, 630 op/s; 14909/271320 objects misplaced (5.495%); 6.9 MiB/s, 111 objects/s recovering 2024-09-02T10:03:00.609 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:03:00.610 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:03:00.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:00.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:00 smithi151 ceph-mon[16639]: pgmap v375: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 39 MiB/s wr, 630 op/s; 14909/271320 objects misplaced (5.495%); 6.9 MiB/s, 111 objects/s recovering 2024-09-02T10:03:00.664 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:00.665 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:00.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:00.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:00.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:00.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:00.925 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:00.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:00.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:01.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:01.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:01.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:01.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:01.186 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:01.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:01.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:01.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:01.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:01.316 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:03:01.317 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:03:01.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:01.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 86 16 94677 94661 68.788 55.125 0.0151646 0.0145322 2024-09-02T10:03:01.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:01.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3215045110' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:03:01.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:01.576 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:01.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:01.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:01.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3215045110' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:03:01.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:01.681 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:03:01.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3215045110' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:03:01.701 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:01.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:01.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:01.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:01.828 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:01.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:01.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:01.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:01.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:02.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:02.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:02.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:02.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:02.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:02.225 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:02.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:02.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:02.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:02.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:02.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:03:02.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:02 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:03:02.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:02 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:03:02.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:02 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:03:02.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:02 smithi151 ceph-mon[16639]: pgmap v376: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 39 MiB/s wr, 628 op/s; 14682/275088 objects misplaced (5.337%); 6.7 MiB/s, 107 objects/s recovering 2024-09-02T10:03:02.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:02 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:02.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3215045110' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:03:02.419 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:02 smithi151 ceph-mon[16639]: osdmap e136: 12 total, 12 up, 11 in 2024-09-02T10:03:02.421 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:03:02.421 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:03:02.421 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:02 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:03:02.422 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:02 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:03:02.422 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:02 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:03:02.422 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:02 smithi187 ceph-mon[20072]: pgmap v376: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 39 MiB/s wr, 628 op/s; 14682/275088 objects misplaced (5.337%); 6.7 MiB/s, 107 objects/s recovering 2024-09-02T10:03:02.422 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:02 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:02.422 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3215045110' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:03:02.422 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:02 smithi187 ceph-mon[20072]: osdmap e136: 12 total, 12 up, 11 in 2024-09-02T10:03:02.440 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 87 16 95576 95560 68.6431 56.1875 0.00996877 0.0145626 2024-09-02T10:03:02.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:02.480 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:02.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:02.582 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:02.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:02 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:03:02.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:02 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:03:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:02 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:03:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:02 smithi162 ceph-mon[20279]: pgmap v376: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.6 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 39 MiB/s wr, 628 op/s; 14682/275088 objects misplaced (5.337%); 6.7 MiB/s, 107 objects/s recovering 2024-09-02T10:03:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:02 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3215045110' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:03:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:02 smithi162 ceph-mon[20279]: osdmap e136: 12 total, 12 up, 11 in 2024-09-02T10:03:02.606 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:02.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:02.607 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:02.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:02.742 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:02.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:02.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:02.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:02.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:02.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:02.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:03.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:03.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:03.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:03.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:03.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:03.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:03.262 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:03.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:03.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:03:03.364 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:03.390 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:03.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:03.391 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:03.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 88 16 96288 96272 68.3687 44.5 0.0106703 0.0146203 2024-09-02T10:03:03.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:03.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:03.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:03.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:03 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:03:03.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:03.656 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:03.657 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:03.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:03 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:03:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:03 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:03:03.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:03.781 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:03.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:03.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:03.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:03.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:03.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:04.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:04.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:04.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:04.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:04.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:04.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:03:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:04.171 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:04.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:04.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:04.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:04.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:04.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:04.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 89 16 97132 97116 68.1931 52.75 0.0288943 0.0146593 2024-09-02T10:03:04.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:04.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:04.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:04 smithi187 ceph-mon[20072]: pgmap v378: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 60 MiB/s wr, 957 op/s; 14313/284019 objects misplaced (5.039%); 9.9 MiB/s, 158 objects/s recovering 2024-09-02T10:03:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:04 smithi187 ceph-mon[20072]: 2.2 scrub starts 2024-09-02T10:03:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:04 smithi187 ceph-mon[20072]: osdmap e137: 12 total, 12 up, 11 in 2024-09-02T10:03:04.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:04.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:04 smithi162 ceph-mon[20279]: pgmap v378: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 60 MiB/s wr, 957 op/s; 14313/284019 objects misplaced (5.039%); 9.9 MiB/s, 158 objects/s recovering 2024-09-02T10:03:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:04 smithi162 ceph-mon[20279]: 2.2 scrub starts 2024-09-02T10:03:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:04 smithi162 ceph-mon[20279]: osdmap e137: 12 total, 12 up, 11 in 2024-09-02T10:03:04.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:04.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:04.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:04 smithi151 ceph-mon[16639]: pgmap v378: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 60 MiB/s wr, 957 op/s; 14313/284019 objects misplaced (5.039%); 9.9 MiB/s, 158 objects/s recovering 2024-09-02T10:03:04.641 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:04 smithi151 ceph-mon[16639]: 2.2 scrub starts 2024-09-02T10:03:04.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:04 smithi151 ceph-mon[16639]: osdmap e137: 12 total, 12 up, 11 in 2024-09-02T10:03:04.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:04.721 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:04.730 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:03:04.730 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:03:04.743 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:04.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:04.774 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:03:04.774 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:03:04.778 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:04.786 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:03:04.786 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:03:04.786 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-02T10:03:04.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:03:04.810 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:03:04.810 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:03:04.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:04.847 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:03:04.847 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:03:04.873 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:04.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:04.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:04.878 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:03:04.879 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:03:04.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:03:04.914 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:03:04.945 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:03:04.945 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:03:04.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:04.978 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:03:04.978 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:03:05.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:05.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:05.018 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:03:05.056 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:03:05.057 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:03:05.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:05.126 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:03:05.127 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:03:05.132 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:05.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:05.156 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:03:05.157 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:03:05.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:05.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:05.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:05.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:05.384 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:05.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:05.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:05.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:05 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:03:05.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:05 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:03:05.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:05 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:03:05.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1239776805' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:05.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 90 5 97748 97743 67.8708 39.1875 0.0505631 0.014729 2024-09-02T10:03:05.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:05.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:05.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:05.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:05 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:03:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:05 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:03:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:05 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:03:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1239776805' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:05.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:05.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:05.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:05.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:05.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:05.755 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:05 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:03:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:05 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:03:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:05 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:03:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1239776805' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:05.757 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:05.771 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:03:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:03:05.771 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:03:05.771 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:03:05.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:05.879 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:05.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:05.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:05.981 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:06.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:06.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:06.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:06.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:06.127 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:06.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:06.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:06.251 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:06.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:06.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:06.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:06.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:06.376 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:06.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:06.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total time run: 90.7548 2024-09-02T10:03:06.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total writes made: 97748 2024-09-02T10:03:06.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Write size: 65536 2024-09-02T10:03:06.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object size: 65536 2024-09-02T10:03:06.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Bandwidth (MB/sec): 67.316 2024-09-02T10:03:06.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Bandwidth: 24.0477 2024-09-02T10:03:06.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max bandwidth (MB/sec): 139.625 2024-09-02T10:03:06.441 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min bandwidth (MB/sec): 1.3125 2024-09-02T10:03:06.442 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average IOPS: 1077 2024-09-02T10:03:06.442 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev IOPS: 384.764 2024-09-02T10:03:06.442 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max IOPS: 2234 2024-09-02T10:03:06.442 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min IOPS: 21 2024-09-02T10:03:06.442 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average Latency(s): 0.0147496 2024-09-02T10:03:06.442 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Latency(s): 0.0447565 2024-09-02T10:03:06.442 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max latency(s): 3.08878 2024-09-02T10:03:06.442 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min latency(s): 0.00101783 2024-09-02T10:03:06.445 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Cleaning up (deleting benchmark objects) 2024-09-02T10:03:06.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:06.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:06.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:06.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:06.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:06.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:06.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:06.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:06.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:06.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:06.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:06.895 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:06.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:06.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:06 smithi151 ceph-mon[16639]: pgmap v380: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 49 MiB/s wr, 790 op/s; 14313/286308 objects misplaced (4.999%); 5.9 MiB/s, 94 objects/s recovering 2024-09-02T10:03:06.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:06 smithi151 ceph-mon[16639]: 2.2 scrub ok 2024-09-02T10:03:06.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:06 smithi151 ceph-mon[16639]: 2.5 scrub starts 2024-09-02T10:03:06.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:07.005 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:06 smithi187 ceph-mon[20072]: pgmap v380: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 49 MiB/s wr, 790 op/s; 14313/286308 objects misplaced (4.999%); 5.9 MiB/s, 94 objects/s recovering 2024-09-02T10:03:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:06 smithi187 ceph-mon[20072]: 2.2 scrub ok 2024-09-02T10:03:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:06 smithi187 ceph-mon[20072]: 2.5 scrub starts 2024-09-02T10:03:07.022 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:07.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:07.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:07.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:06 smithi162 ceph-mon[20279]: pgmap v380: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.8 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 49 MiB/s wr, 790 op/s; 14313/286308 objects misplaced (4.999%); 5.9 MiB/s, 94 objects/s recovering 2024-09-02T10:03:07.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:06 smithi162 ceph-mon[20279]: 2.2 scrub ok 2024-09-02T10:03:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:06 smithi162 ceph-mon[20279]: 2.5 scrub starts 2024-09-02T10:03:07.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:07.147 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:07.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:07.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:07.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:07.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:07.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:07.380 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:07.405 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:07.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:07.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:07.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:07.533 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:07.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:07.631 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:03:07.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:07.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:03:07.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:07.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:07.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:07.810 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:07.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:07.883 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:07 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:03:07.883 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:07 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:03:07.883 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4167487240' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:07.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:07.937 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:07.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:07 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:03:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:07 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:03:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4167487240' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:08.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:08.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:08.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:08.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:08.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:07 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:03:08.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:07 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:03:08.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4167487240' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:08.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:08.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:08.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:08.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:08.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:08.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:08.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:08.422 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:08.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:08.446 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:08.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:08.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:08.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:08.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:08.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:08.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:08.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:08.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:08.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:08.712 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:08.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:03:08.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:08.828 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:08.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:08.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:08 smithi151 ceph-mon[16639]: pgmap v381: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 52 MiB/s wr, 853 op/s; 13859/290661 objects misplaced (4.768%); 8.2 MiB/s, 131 objects/s recovering 2024-09-02T10:03:08.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:08 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:03:08.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:08 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:03:08.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4167487240' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:03:08.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:08 smithi151 ceph-mon[16639]: osdmap e138: 12 total, 12 up, 11 in 2024-09-02T10:03:08.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:08.957 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:08.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:08 smithi187 ceph-mon[20072]: pgmap v381: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 52 MiB/s wr, 853 op/s; 13859/290661 objects misplaced (4.768%); 8.2 MiB/s, 131 objects/s recovering 2024-09-02T10:03:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:08 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:03:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:08 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:03:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4167487240' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:03:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:08 smithi187 ceph-mon[20072]: osdmap e138: 12 total, 12 up, 11 in 2024-09-02T10:03:09.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:09.081 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:09.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:09.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:08 smithi162 ceph-mon[20279]: pgmap v381: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 52 MiB/s wr, 853 op/s; 13859/290661 objects misplaced (4.768%); 8.2 MiB/s, 131 objects/s recovering 2024-09-02T10:03:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:08 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:03:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:08 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:03:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4167487240' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:03:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:08 smithi162 ceph-mon[20279]: osdmap e138: 12 total, 12 up, 11 in 2024-09-02T10:03:09.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:09.205 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:09.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:09.307 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:03:09.308 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:03:09.309 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:09.331 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:09.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:09.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:09.459 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:09.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:09.529 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:03:09.529 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:03:09.529 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:03:09.529 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:03:09.529 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:03:09.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:09.586 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:09.687 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:09.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:09.713 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:09.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:09.839 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:09.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:09.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:09.966 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:09.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:10.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:10.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:09 smithi151 ceph-mon[16639]: 2.5 scrub ok 2024-09-02T10:03:10.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:09 smithi151 ceph-mon[16639]: osdmap e139: 12 total, 12 up, 11 in 2024-09-02T10:03:10.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:09 smithi151 ceph-mon[16639]: pgmap v384: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 30 MiB/s wr, 556 op/s; 13645/291516 objects misplaced (4.681%); 7.0 MiB/s, 111 objects/s recovering 2024-09-02T10:03:10.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:09 smithi151 ceph-mon[16639]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:03:10.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:09 smithi162 ceph-mon[20279]: 2.5 scrub ok 2024-09-02T10:03:10.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:09 smithi162 ceph-mon[20279]: osdmap e139: 12 total, 12 up, 11 in 2024-09-02T10:03:10.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:09 smithi162 ceph-mon[20279]: pgmap v384: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 30 MiB/s wr, 556 op/s; 13645/291516 objects misplaced (4.681%); 7.0 MiB/s, 111 objects/s recovering 2024-09-02T10:03:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:09 smithi162 ceph-mon[20279]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:03:10.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:10.095 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:10.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:10.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:10.219 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:10.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:09 smithi187 ceph-mon[20072]: 2.5 scrub ok 2024-09-02T10:03:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:09 smithi187 ceph-mon[20072]: osdmap e139: 12 total, 12 up, 11 in 2024-09-02T10:03:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:09 smithi187 ceph-mon[20072]: pgmap v384: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 30 MiB/s wr, 556 op/s; 13645/291516 objects misplaced (4.681%); 7.0 MiB/s, 111 objects/s recovering 2024-09-02T10:03:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:09 smithi187 ceph-mon[20072]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:03:10.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:10.343 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:10.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:10.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:10.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:03:10.445 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-02T10:03:09.080523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6368657976,"num_objects":97172,"num_object_clones":0,"num_object_copies":291516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13645,"num_objects_unfound":0,"num_objects_dirty":97172,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98018,"num_write_kb":6245953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3351,"num_bytes_recovered":219611136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99950,"ondisk_log_size":99950,"up":111,"acting":108,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":18594348,"kb_used_data":17950056,"kb_used_omap":17,"kb_used_meta":644078,"kb_avail":1012602324,"statfs":{"total":1055945392128,"available":1036904779776,"internally_reserved":0,"allocated":18380857344,"data_stored":18374903970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":659536763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":163682117,"num_objects":2499,"num_object_clones":0,"num_object_copies":7497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-668,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":3342,"num_write_kb":186912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":668,"num_bytes_recovered":43754006,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002243"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.772995+0000","last_change":"2024-09-02T10:02:43.131354+0000","last_active":"2024-09-02T10:03:03.772995+0000","last_peered":"2024-09-02T10:03:03.772995+0000","last_clean":"2024-09-02T10:03:03.772995+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:03:03.772995+0000","last_undegraded":"2024-09-02T10:03:03.772995+0000","last_fullsized":"2024-09-02T10:03:03.772995+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.131299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.131299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:10.176059+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":117,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.765373+0000","last_change":"2024-09-02T10:02:48.288012+0000","last_active":"2024-09-02T10:03:03.765373+0000","last_peered":"2024-09-02T10:03:03.765373+0000","last_clean":"2024-09-02T10:03:03.765373+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:03:03.765373+0000","last_undegraded":"2024-09-02T10:03:03.765373+0000","last_fullsized":"2024-09-02T10:03:03.765373+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:42.887531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:42.887531+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:11.502014+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.769151+0000","last_change":"2024-09-02T10:02:43.732505+0000","last_active":"2024-09-02T10:03:03.769151+0000","last_peered":"2024-09-02T10:03:03.769151+0000","last_clean":"2024-09-02T10:03:03.769151+0000","last_became_active":"2024-09-02T10:02:42.784852+0000","last_became_peered":"2024-09-02T10:02:42.784852+0000","last_unstale":"2024-09-02T10:03:03.769151+0000","last_undegraded":"2024-09-02T10:03:03.769151+0000","last_fullsized":"2024-09-02T10:03:03.769151+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.732449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.732449+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:47.435965+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":42,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.772336+0000","last_change":"2024-09-02T10:02:42.836988+0000","last_active":"2024-09-02T10:03:03.772336+0000","last_peered":"2024-09-02T10:03:03.772336+0000","last_clean":"2024-09-02T10:03:03.772336+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:03:03.772336+0000","last_undegraded":"2024-09-02T10:03:03.772336+0000","last_fullsized":"2024-09-02T10:03:03.772336+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:42.836911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:42.836911+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:56.258322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":47,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.764732+0000","last_change":"2024-09-02T10:02:48.305104+0000","last_active":"2024-09-02T10:03:03.764732+0000","last_peered":"2024-09-02T10:03:03.764732+0000","last_clean":"2024-09-02T10:03:03.764732+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:03:03.764732+0000","last_undegraded":"2024-09-02T10:03:03.764732+0000","last_fullsized":"2024-09-02T10:03:03.764732+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:44.736816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:44.736816+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:02.856494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"137'6028","reported_seq":6186,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:04.079983+0000","last_change":"2024-09-02T10:02:41.214667+0000","last_active":"2024-09-02T10:03:04.079983+0000","last_peered":"2024-09-02T10:03:04.079983+0000","last_clean":"2024-09-02T10:03:04.079983+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:03:04.079983+0000","last_undegraded":"2024-09-02T10:03:04.079983+0000","last_fullsized":"2024-09-02T10:03:04.079983+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'4912","last_scrub_stamp":"2024-09-02T10:02:41.214588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:41.214588+0000","objects_scrubbed":4898,"log_size":6028,"log_dups_size":0,"ondisk_log_size":6028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:03.619688+0000","scrub_duration":1065,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395051008,"num_objects":6028,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6028,"num_write_kb":385792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":97,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.764785+0000","last_change":"2024-09-02T10:01:50.947414+0000","last_active":"2024-09-02T10:03:03.764785+0000","last_peered":"2024-09-02T10:03:03.764785+0000","last_clean":"2024-09-02T10:03:03.764785+0000","last_became_active":"2024-09-02T10:01:50.763116+0000","last_became_peered":"2024-09-02T10:01:50.763116+0000","last_unstale":"2024-09-02T10:03:03.764785+0000","last_undegraded":"2024-09-02T10:03:03.764785+0000","last_fullsized":"2024-09-02T10:03:03.764785+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:50.947356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:50.947356+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:17.828734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"137'6148","reported_seq":6295,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:06.831968+0000","last_change":"2024-09-02T10:02:49.450573+0000","last_active":"2024-09-02T10:03:06.831968+0000","last_peered":"2024-09-02T10:03:06.831968+0000","last_clean":"2024-09-02T10:03:06.831968+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:03:06.831968+0000","last_undegraded":"2024-09-02T10:03:06.831968+0000","last_fullsized":"2024-09-02T10:03:06.831968+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":4249,"log_size":6148,"log_dups_size":0,"ondisk_log_size":6148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399114240,"num_objects":6090,"num_object_clones":0,"num_object_copies":18270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6148,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":250,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.767778+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:03:03.767778+0000","last_peered":"2024-09-02T10:03:03.767778+0000","last_clean":"2024-09-02T10:03:03.767778+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:03:03.767778+0000","last_undegraded":"2024-09-02T10:03:03.767778+0000","last_fullsized":"2024-09-02T10:03:03.767778+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:12.384522+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"137'6209","reported_seq":6309,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:07.556175+0000","last_change":"2024-09-02T10:01:30.560354+0000","last_active":"2024-09-02T10:03:07.556175+0000","last_peered":"2024-09-02T10:03:07.556175+0000","last_clean":"2024-09-02T10:03:07.556175+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:03:07.556175+0000","last_undegraded":"2024-09-02T10:03:07.556175+0000","last_fullsized":"2024-09-02T10:03:07.556175+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":6209,"log_dups_size":0,"ondisk_log_size":6209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399572992,"num_objects":6097,"num_object_clones":0,"num_object_copies":18291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6209,"num_write_kb":393792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":92,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-02T10:03:07.634307+0000","last_change":"2024-09-02T10:01:51.614379+0000","last_active":"2024-09-02T10:03:07.634307+0000","last_peered":"2024-09-02T10:03:07.634307+0000","last_clean":"2024-09-02T10:03:07.634307+0000","last_became_active":"2024-09-02T10:01:50.781305+0000","last_became_peered":"2024-09-02T10:01:50.781305+0000","last_unstale":"2024-09-02T10:03:07.634307+0000","last_undegraded":"2024-09-02T10:03:07.634307+0000","last_fullsized":"2024-09-02T10:03:07.634307+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.614334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.614334+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"137'6110","reported_seq":6211,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:05.436544+0000","last_change":"2024-09-02T10:01:30.141564+0000","last_active":"2024-09-02T10:03:05.436544+0000","last_peered":"2024-09-02T10:03:05.436544+0000","last_clean":"2024-09-02T10:03:05.436544+0000","last_became_active":"2024-09-02T10:01:30.141263+0000","last_became_peered":"2024-09-02T10:01:30.141263+0000","last_unstale":"2024-09-02T10:03:05.436544+0000","last_undegraded":"2024-09-02T10:03:05.436544+0000","last_fullsized":"2024-09-02T10:03:05.436544+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":6110,"log_dups_size":0,"ondisk_log_size":6110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400424960,"num_objects":6110,"num_object_clones":0,"num_object_copies":18330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6110,"num_write_kb":391040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.767154+0000","last_change":"2024-09-02T10:02:43.245406+0000","last_active":"2024-09-02T10:03:03.767154+0000","last_peered":"2024-09-02T10:03:03.767154+0000","last_clean":"2024-09-02T10:03:03.767154+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:03:03.767154+0000","last_undegraded":"2024-09-02T10:03:03.767154+0000","last_fullsized":"2024-09-02T10:03:03.767154+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.245354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.245354+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:43.645905+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"137'6099","reported_seq":7570,"reported_epoch":137,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:03:06.942169+0000","last_change":"2024-09-02T10:02:50.428989+0000","last_active":"2024-09-02T10:03:06.942169+0000","last_peered":"2024-09-02T10:03:06.942169+0000","last_clean":"2024-09-02T10:02:47.284306+0000","last_became_active":"2024-09-02T10:02:50.338274+0000","last_became_peered":"2024-09-02T10:02:50.338274+0000","last_unstale":"2024-09-02T10:03:06.942169+0000","last_undegraded":"2024-09-02T10:03:06.942169+0000","last_fullsized":"2024-09-02T10:02:49.284588+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":6099,"log_dups_size":0,"ondisk_log_size":6099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395640832,"num_objects":6037,"num_object_clones":0,"num_object_copies":18111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4649,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6099,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":44826624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2],"avail_no_missing":["6","2","10"],"object_location_counts":[{"shards":"2,6","objects":1388},{"shards":"2,6,10","objects":4649}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":196,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.767154+0000","last_change":"2024-09-02T10:02:41.206594+0000","last_active":"2024-09-02T10:03:03.767154+0000","last_peered":"2024-09-02T10:03:03.767154+0000","last_clean":"2024-09-02T10:03:03.767154+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:03:03.767154+0000","last_undegraded":"2024-09-02T10:03:03.767154+0000","last_fullsized":"2024-09-02T10:03:03.767154+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:41.206524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:41.206524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:43.244570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"137'6156","reported_seq":6284,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:07.574359+0000","last_change":"2024-09-02T10:02:49.443887+0000","last_active":"2024-09-02T10:03:07.574359+0000","last_peered":"2024-09-02T10:03:07.574359+0000","last_clean":"2024-09-02T10:03:07.574359+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:03:07.574359+0000","last_undegraded":"2024-09-02T10:03:07.574359+0000","last_fullsized":"2024-09-02T10:03:07.574359+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":2499,"log_size":6156,"log_dups_size":0,"ondisk_log_size":6156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397017088,"num_objects":6058,"num_object_clones":0,"num_object_copies":18174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6156,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":34,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.772885+0000","last_change":"2024-09-02T10:02:49.000327+0000","last_active":"2024-09-02T10:03:03.772885+0000","last_peered":"2024-09-02T10:03:03.772885+0000","last_clean":"2024-09-02T10:03:03.772885+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:03:03.772885+0000","last_undegraded":"2024-09-02T10:03:03.772885+0000","last_fullsized":"2024-09-02T10:03:03.772885+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:49.000247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:49.000247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:05.328170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"137'6018","reported_seq":6119,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:04.105610+0000","last_change":"2024-09-02T10:01:30.183497+0000","last_active":"2024-09-02T10:03:04.105610+0000","last_peered":"2024-09-02T10:03:04.105610+0000","last_clean":"2024-09-02T10:03:04.105610+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:03:04.105610+0000","last_undegraded":"2024-09-02T10:03:04.105610+0000","last_fullsized":"2024-09-02T10:03:04.105610+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":6018,"log_dups_size":0,"ondisk_log_size":6018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394395648,"num_objects":6018,"num_object_clones":0,"num_object_copies":18054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6018,"num_write_kb":385152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":45,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-02T10:03:07.634186+0000","last_change":"2024-09-02T10:03:06.643365+0000","last_active":"2024-09-02T10:03:07.634186+0000","last_peered":"2024-09-02T10:03:07.634186+0000","last_clean":"2024-09-02T10:03:07.634186+0000","last_became_active":"2024-09-02T10:02:48.295440+0000","last_became_peered":"2024-09-02T10:02:48.295440+0000","last_unstale":"2024-09-02T10:03:07.634186+0000","last_undegraded":"2024-09-02T10:03:07.634186+0000","last_fullsized":"2024-09-02T10:03:07.634186+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:06.643296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:06.643296+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:29.959714+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"137'6125","reported_seq":7715,"reported_epoch":137,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:03:07.569778+0000","last_change":"2024-09-02T10:02:49.464255+0000","last_active":"2024-09-02T10:03:07.569778+0000","last_peered":"2024-09-02T10:03:07.569778+0000","last_clean":"2024-09-02T10:02:47.284846+0000","last_became_active":"2024-09-02T10:02:49.412682+0000","last_became_peered":"2024-09-02T10:02:49.412682+0000","last_unstale":"2024-09-02T10:03:07.569778+0000","last_undegraded":"2024-09-02T10:03:07.569778+0000","last_fullsized":"2024-09-02T10:02:49.287036+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":6125,"log_dups_size":0,"ondisk_log_size":6125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393936896,"num_objects":6011,"num_object_clones":0,"num_object_copies":18033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4555,"num_objects_unfound":0,"num_objects_dirty":6011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6125,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":743,"num_bytes_recovered":48693248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0],"avail_no_missing":["1","0","10"],"object_location_counts":[{"shards":"0,1","objects":1456},{"shards":"0,1,10","objects":4555}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":135,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.784250+0000","last_change":"2024-09-02T10:02:48.286345+0000","last_active":"2024-09-02T10:03:03.784250+0000","last_peered":"2024-09-02T10:03:03.784250+0000","last_clean":"2024-09-02T10:03:03.784250+0000","last_became_active":"2024-09-02T10:02:48.285983+0000","last_became_peered":"2024-09-02T10:02:48.285983+0000","last_unstale":"2024-09-02T10:03:03.784250+0000","last_undegraded":"2024-09-02T10:03:03.784250+0000","last_fullsized":"2024-09-02T10:03:03.784250+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:41.125723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:41.125723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:10.902134+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"137'6293","reported_seq":6462,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:06.738014+0000","last_change":"2024-09-02T10:03:05.402657+0000","last_active":"2024-09-02T10:03:06.738014+0000","last_peered":"2024-09-02T10:03:06.738014+0000","last_clean":"2024-09-02T10:03:06.738014+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:03:06.738014+0000","last_undegraded":"2024-09-02T10:03:06.738014+0000","last_fullsized":"2024-09-02T10:03:06.738014+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'6270","last_scrub_stamp":"2024-09-02T10:03:05.402533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:03:05.402533+0000","objects_scrubbed":6239,"log_size":6293,"log_dups_size":0,"ondisk_log_size":6293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:32.189270+0000","scrub_duration":1831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409534464,"num_objects":6249,"num_object_clones":0,"num_object_copies":18747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6293,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":36,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-02T10:03:07.634200+0000","last_change":"2024-09-02T10:02:48.497765+0000","last_active":"2024-09-02T10:03:07.634200+0000","last_peered":"2024-09-02T10:03:07.634200+0000","last_clean":"2024-09-02T10:03:07.634200+0000","last_became_active":"2024-09-02T10:02:48.288446+0000","last_became_peered":"2024-09-02T10:02:48.288446+0000","last_unstale":"2024-09-02T10:03:07.634200+0000","last_undegraded":"2024-09-02T10:03:07.634200+0000","last_fullsized":"2024-09-02T10:03:07.634200+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:48.497719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.497719+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:13.437417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"137'6162","reported_seq":6324,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:06.887389+0000","last_change":"2024-09-02T10:02:48.223675+0000","last_active":"2024-09-02T10:03:06.887389+0000","last_peered":"2024-09-02T10:03:06.887389+0000","last_clean":"2024-09-02T10:03:06.887389+0000","last_became_active":"2024-09-02T10:01:30.559997+0000","last_became_peered":"2024-09-02T10:01:30.559997+0000","last_unstale":"2024-09-02T10:03:06.887389+0000","last_undegraded":"2024-09-02T10:03:06.887389+0000","last_fullsized":"2024-09-02T10:03:06.887389+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'5361","last_scrub_stamp":"2024-09-02T10:02:48.223591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.223591+0000","objects_scrubbed":5355,"log_size":6162,"log_dups_size":0,"ondisk_log_size":6162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:00.041296+0000","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399376384,"num_objects":6094,"num_object_clones":0,"num_object_copies":18282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6162,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":264,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.767772+0000","last_change":"2024-09-02T10:02:41.054329+0000","last_active":"2024-09-02T10:03:03.767772+0000","last_peered":"2024-09-02T10:03:03.767772+0000","last_clean":"2024-09-02T10:03:03.767772+0000","last_became_active":"2024-09-02T10:02:40.782398+0000","last_became_peered":"2024-09-02T10:02:40.782398+0000","last_unstale":"2024-09-02T10:03:03.767772+0000","last_undegraded":"2024-09-02T10:03:03.767772+0000","last_fullsized":"2024-09-02T10:03:03.767772+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:41.054260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:41.054260+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:57.333212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"137'6037","reported_seq":9994,"reported_epoch":137,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:03:05.633950+0000","last_change":"2024-09-02T10:02:48.453927+0000","last_active":"2024-09-02T10:03:05.633950+0000","last_peered":"2024-09-02T10:03:05.633950+0000","last_clean":"2024-09-02T10:02:27.223209+0000","last_became_active":"2024-09-02T10:02:48.419929+0000","last_became_peered":"2024-09-02T10:02:48.419929+0000","last_unstale":"2024-09-02T10:03:05.633950+0000","last_undegraded":"2024-09-02T10:03:05.633950+0000","last_fullsized":"2024-09-02T10:02:48.284397+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":6037,"log_dups_size":0,"ondisk_log_size":6037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395640832,"num_objects":6037,"num_object_clones":0,"num_object_copies":18111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4441,"num_objects_unfound":0,"num_objects_dirty":6037,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6037,"num_write_kb":386368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1924,"num_bytes_recovered":126091264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9],"avail_no_missing":["7","9","10"],"object_location_counts":[{"shards":"6,7,9","objects":1112},{"shards":"6,7,9,10","objects":174},{"shards":"7,9","objects":1596},{"shards":"7,9,10","objects":3155}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":104,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.765469+0000","last_change":"2024-09-02T10:02:40.877537+0000","last_active":"2024-09-02T10:03:03.765469+0000","last_peered":"2024-09-02T10:03:03.765469+0000","last_clean":"2024-09-02T10:03:03.765469+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:03:03.765469+0000","last_undegraded":"2024-09-02T10:03:03.765469+0000","last_fullsized":"2024-09-02T10:03:03.765469+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:40.877492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:40.877492+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:49.183890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"137'6197","reported_seq":6353,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:06.724054+0000","last_change":"2024-09-02T10:02:39.902318+0000","last_active":"2024-09-02T10:03:06.724054+0000","last_peered":"2024-09-02T10:03:06.724054+0000","last_clean":"2024-09-02T10:03:06.724054+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:03:06.724054+0000","last_undegraded":"2024-09-02T10:03:06.724054+0000","last_fullsized":"2024-09-02T10:03:06.724054+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'4936","last_scrub_stamp":"2024-09-02T10:02:39.902123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:39.902123+0000","objects_scrubbed":4899,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:01.836567+0000","scrub_duration":1193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402718720,"num_objects":6145,"num_object_clones":0,"num_object_copies":18435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6197,"num_write_kb":394944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":259,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.770458+0000","last_change":"2024-09-02T10:02:42.117622+0000","last_active":"2024-09-02T10:03:03.770458+0000","last_peered":"2024-09-02T10:03:03.770458+0000","last_clean":"2024-09-02T10:03:03.770458+0000","last_became_active":"2024-09-02T10:02:40.806179+0000","last_became_peered":"2024-09-02T10:02:40.806179+0000","last_unstale":"2024-09-02T10:03:03.770458+0000","last_undegraded":"2024-09-02T10:03:03.770458+0000","last_fullsized":"2024-09-02T10:03:03.770458+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:42.117564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:42.117564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:02.110972+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":134,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.784201+0000","last_change":"2024-09-02T10:02:43.126713+0000","last_active":"2024-09-02T10:03:03.784201+0000","last_peered":"2024-09-02T10:03:03.784201+0000","last_clean":"2024-09-02T10:03:03.784201+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:03:03.784201+0000","last_undegraded":"2024-09-02T10:03:03.784201+0000","last_fullsized":"2024-09-02T10:03:03.784201+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.126668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.126668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:50.432379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"137'6160","reported_seq":6322,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:06.775734+0000","last_change":"2024-09-02T10:02:46.577683+0000","last_active":"2024-09-02T10:03:06.775734+0000","last_peered":"2024-09-02T10:03:06.775734+0000","last_clean":"2024-09-02T10:03:06.775734+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:03:06.775734+0000","last_undegraded":"2024-09-02T10:03:06.775734+0000","last_fullsized":"2024-09-02T10:03:06.775734+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'5355","last_scrub_stamp":"2024-09-02T10:02:46.577625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:46.577625+0000","objects_scrubbed":5316,"log_size":6160,"log_dups_size":0,"ondisk_log_size":6160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:09.392542+0000","scrub_duration":1343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400949248,"num_objects":6118,"num_object_clones":0,"num_object_copies":18354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6160,"num_write_kb":392896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.773453+0000","last_change":"2024-09-02T10:02:43.195354+0000","last_active":"2024-09-02T10:03:03.773453+0000","last_peered":"2024-09-02T10:03:03.773453+0000","last_clean":"2024-09-02T10:03:03.773453+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:03:03.773453+0000","last_undegraded":"2024-09-02T10:03:03.773453+0000","last_fullsized":"2024-09-02T10:03:03.773453+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.195281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.195281+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:56.115958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"138'6010","reported_seq":6170,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-02T10:03:07.835800+0000","last_change":"2024-09-02T10:02:48.587952+0000","last_active":"2024-09-02T10:03:07.835800+0000","last_peered":"2024-09-02T10:03:07.835800+0000","last_clean":"2024-09-02T10:03:07.835800+0000","last_became_active":"2024-09-02T10:01:30.559704+0000","last_became_peered":"2024-09-02T10:01:30.559704+0000","last_unstale":"2024-09-02T10:03:07.835800+0000","last_undegraded":"2024-09-02T10:03:07.835800+0000","last_fullsized":"2024-09-02T10:03:07.835800+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'5182","last_scrub_stamp":"2024-09-02T10:02:48.587872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.587872+0000","objects_scrubbed":5181,"log_size":6010,"log_dups_size":0,"ondisk_log_size":6010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:03.968763+0000","scrub_duration":1077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384630808,"num_objects":5870,"num_object_clones":0,"num_object_copies":17610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5870,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6010,"num_write_kb":380097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[9,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":40,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:03.767865+0000","last_change":"2024-09-02T10:02:43.100824+0000","last_active":"2024-09-02T10:03:03.767865+0000","last_peered":"2024-09-02T10:03:03.767865+0000","last_clean":"2024-09-02T10:03:03.767865+0000","last_became_active":"2024-09-02T10:02:42.791266+0000","last_became_peered":"2024-09-02T10:02:42.791266+0000","last_unstale":"2024-09-02T10:03:03.767865+0000","last_undegraded":"2024-09-02T10:03:03.767865+0000","last_fullsized":"2024-09-02T10:03:03.767865+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.100746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.100746+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:01.608678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"137'6177","reported_seq":6277,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:06.925234+0000","last_change":"2024-09-02T10:01:30.561460+0000","last_active":"2024-09-02T10:03:06.925234+0000","last_peered":"2024-09-02T10:03:06.925234+0000","last_clean":"2024-09-02T10:03:06.925234+0000","last_became_active":"2024-09-02T10:01:30.560995+0000","last_became_peered":"2024-09-02T10:01:30.560995+0000","last_unstale":"2024-09-02T10:03:06.925234+0000","last_undegraded":"2024-09-02T10:03:06.925234+0000","last_fullsized":"2024-09-02T10:03:06.925234+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":401014784,"num_objects":6119,"num_object_clones":0,"num_object_copies":18357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"137'6089","reported_seq":6251,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-02T10:03:04.102478+0000","last_change":"2024-09-02T10:02:45.316231+0000","last_active":"2024-09-02T10:03:04.102478+0000","last_peered":"2024-09-02T10:03:04.102478+0000","last_clean":"2024-09-02T10:03:04.102478+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:03:04.102478+0000","last_undegraded":"2024-09-02T10:03:04.102478+0000","last_fullsized":"2024-09-02T10:03:04.102478+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'5298","last_scrub_stamp":"2024-09-02T10:02:45.316085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:45.316085+0000","objects_scrubbed":5264,"log_size":6089,"log_dups_size":0,"ondisk_log_size":6089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:04:11.881759+0000","scrub_duration":1234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399048704,"num_objects":6089,"num_object_clones":0,"num_object_copies":18267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6089,"num_write_kb":389696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":49,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-02T10:03:07.634796+0000","last_change":"2024-09-02T10:02:43.402931+0000","last_active":"2024-09-02T10:03:07.634796+0000","last_peered":"2024-09-02T10:03:07.634796+0000","last_clean":"2024-09-02T10:03:07.634796+0000","last_became_active":"2024-09-02T10:02:42.785794+0000","last_became_peered":"2024-09-02T10:02:42.785794+0000","last_unstale":"2024-09-02T10:03:07.634796+0000","last_undegraded":"2024-09-02T10:03:07.634796+0000","last_fullsized":"2024-09-02T10:03:07.634796+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.402852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.402852+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:03:58.497757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":6368067608,"num_objects":97170,"num_object_clones":0,"num_object_copies":291510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13645,"num_objects_unfound":0,"num_objects_dirty":97170,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":98018,"num_write_kb":6245953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3351,"num_bytes_recovered":219611136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19311239168,"data_stored":19311231024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":98018,"ondisk_log_size":98018,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383716,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1323432,"kb_used_data":1278360,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92421720,"statfs":{"total":95995035648,"available":94639841280,"internally_reserved":0,"allocated":1309040640,"data_stored":1308501126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":46135748},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612649,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841582,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2376636,"kb_used_data":2313512,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":91368516,"statfs":{"total":95995035648,"available":93561360384,"internally_reserved":0,"allocated":2369036288,"data_stored":2368488201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1605036,"kb_used_data":1547480,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92140116,"statfs":{"total":95995035648,"available":94351478784,"internally_reserved":0,"allocated":1584619520,"data_stored":1584080014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2023212,"kb_used_data":1951768,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91721940,"statfs":{"total":95995035648,"available":93923266560,"internally_reserved":0,"allocated":1998610432,"data_stored":1998070926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528379,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":987116,"kb_used_data":929560,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92758036,"statfs":{"total":95995035648,"available":94984228864,"internally_reserved":0,"allocated":951869440,"data_stored":951329934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828272,"kb_used_data":797088,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92916880,"statfs":{"total":95995035648,"available":95146885120,"internally_reserved":0,"allocated":816218112,"data_stored":815675054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953541,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1193884,"kb_used_data":1144652,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92551268,"statfs":{"total":95995035648,"available":94772498432,"internally_reserved":0,"allocated":1172123648,"data_stored":1171588651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149772,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2480244,"kb_used_data":2412964,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91264908,"statfs":{"total":95995035648,"available":93455265792,"internally_reserved":0,"allocated":2470875136,"data_stored":2470328006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1729460,"kb_used_data":1662176,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92015692,"statfs":{"total":95995035648,"available":94224068608,"internally_reserved":0,"allocated":1702068224,"data_stored":1701525166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2025256,"kb_used_data":1957976,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91719896,"statfs":{"total":95995035648,"available":93921173504,"internally_reserved":0,"allocated":2004967424,"data_stored":2004427918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2021800,"kb_used_data":1954520,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91723352,"statfs":{"total":95995035648,"available":93924712448,"internally_reserved":0,"allocated":2001428480,"data_stored":2000888974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1999831040,"data_stored":1999831040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1996619776,"data_stored":1996619776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1693515776,"data_stored":1693515776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2456555520,"data_stored":2456551448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1179779072,"data_stored":1179779072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":812253184,"data_stored":812253184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":946143232,"data_stored":946143232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2002124800,"data_stored":2002124800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1585512448,"data_stored":1585512448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2363166720,"data_stored":2363162648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":972226560,"data_stored":972226560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1303511040,"data_stored":1303511040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:03:10.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 21 2024-09-02T10:03:10.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:10.470 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:10.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:10.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:10.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:10.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:10.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:10.763 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:10.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:10.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:10.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:10.892 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:10.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:10.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:11.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:11.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:11.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:11.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:10 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:03:11.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:10 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:03:11.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:10 smithi151 ceph-mon[16639]: osdmap e140: 12 total, 12 up, 11 in 2024-09-02T10:03:11.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:10 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:03:11.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:10 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:03:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:10 smithi162 ceph-mon[20279]: osdmap e140: 12 total, 12 up, 11 in 2024-09-02T10:03:11.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:03:11.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:11.131 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:11.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:11.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:10 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:03:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:10 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:03:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:10 smithi187 ceph-mon[20072]: osdmap e140: 12 total, 12 up, 11 in 2024-09-02T10:03:11.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:11.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:11.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:11.314 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:11.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:11.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:11.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:11.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:11.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:11.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:11.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:11.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:11.737 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:11.738 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:11.786 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:11.794 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:11.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:11.851 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:03:11.851 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:03:11.856 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:11.864 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:11.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:11.898 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:03:11.898 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:03:11.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:11 smithi151 ceph-mon[16639]: pgmap v386: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 24 MiB/s wr, 583 op/s; 13405/289668 objects misplaced (4.628%); 9.4 MiB/s, 151 objects/s recovering 2024-09-02T10:03:11.936 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:03:11.936 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:03:11.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:11.975 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:03:11.976 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:03:11.990 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:11.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:11.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:12.017 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:03:12.020 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:03:12.052 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:03:12.052 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:03:12.091 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:03:12.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:03:12.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:12.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:11 smithi162 ceph-mon[20279]: pgmap v386: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 24 MiB/s wr, 583 op/s; 13405/289668 objects misplaced (4.628%); 9.4 MiB/s, 151 objects/s recovering 2024-09-02T10:03:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:12.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:12.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:11 smithi187 ceph-mon[20072]: pgmap v386: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.9 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 24 MiB/s wr, 583 op/s; 13405/289668 objects misplaced (4.628%); 9.4 MiB/s, 151 objects/s recovering 2024-09-02T10:03:12.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:12.284 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:12.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:12.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:12.412 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:12.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:12.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:12.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:12.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:12.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:12.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:12.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:12.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:12.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:12.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:12.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:12.933 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:03:12.933 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:03:12.945 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:12.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:12.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:12.997 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:03:12.997 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:03:13.030 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:03:13.030 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:03:13.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:13.062 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:03:13.062 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:03:13.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:13.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:13.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:13.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3363084718' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:13.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:12 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:03:13.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:12 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:03:13.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3363084718' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:13.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:12 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:03:13.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:12 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:03:13.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:03:13.099 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:03:13.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:13.189 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:03:13.189 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:03:13.198 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:13.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3363084718' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:12 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:03:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:12 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:03:13.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:13.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:13.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:13.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:13.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:13.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:13.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:13.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:13.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:13.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:13.710 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:13.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:13.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:13.829 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:03:13.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:13.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:13.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:13.974 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:13.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:14.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:14.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:13 smithi162 ceph-mon[20279]: osdmap e141: 12 total, 12 up, 11 in 2024-09-02T10:03:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1660122898' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-02T10:03:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:13 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-02T10:03:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3186313171' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:03:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:13 smithi162 ceph-mon[20279]: pgmap v388: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.9 MiB/s wr, 731 op/s; 12990/280974 objects misplaced (4.623%); 7.5 MiB/s, 120 objects/s recovering 2024-09-02T10:03:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:13 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:03:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:13 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:03:14.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:14.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:14.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:13 smithi151 ceph-mon[16639]: osdmap e141: 12 total, 12 up, 11 in 2024-09-02T10:03:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1660122898' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-02T10:03:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:13 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-02T10:03:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3186313171' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:03:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:13 smithi151 ceph-mon[16639]: pgmap v388: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.9 MiB/s wr, 731 op/s; 12990/280974 objects misplaced (4.623%); 7.5 MiB/s, 120 objects/s recovering 2024-09-02T10:03:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:13 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:03:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:13 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:03:14.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:14.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:14.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:14.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:13 smithi187 ceph-mon[20072]: osdmap e141: 12 total, 12 up, 11 in 2024-09-02T10:03:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1660122898' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-02T10:03:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:13 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"} : dispatch 2024-09-02T10:03:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3186313171' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:03:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:13 smithi187 ceph-mon[20072]: pgmap v388: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 2.9 MiB/s wr, 731 op/s; 12990/280974 objects misplaced (4.623%); 7.5 MiB/s, 120 objects/s recovering 2024-09-02T10:03:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:13 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:03:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:13 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:03:14.259 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pgp_num to 21 2024-09-02T10:03:14.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:14.361 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:14.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:14.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:14.489 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:14.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:14.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:14.614 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:14.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:14.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:14.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:14.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:14.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:14.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:14.872 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:14.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:14.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:14.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:14.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:14.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:15.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:14 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:15.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:14 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-02T10:03:15.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3186313171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:03:15.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:14 smithi151 ceph-mon[16639]: osdmap e142: 12 total, 12 up, 11 in 2024-09-02T10:03:15.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:14 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:15.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:14 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-02T10:03:15.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3186313171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:03:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:14 smithi162 ceph-mon[20279]: osdmap e142: 12 total, 12 up, 11 in 2024-09-02T10:03:15.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:15.124 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:15.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:15.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:15.249 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:15.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:15.255 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:14 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:14 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-02T10:03:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3186313171' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:03:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:14 smithi187 ceph-mon[20072]: osdmap e142: 12 total, 12 up, 11 in 2024-09-02T10:03:15.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:15.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:15.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:15.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:15.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:15.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:15.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:15.622 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:15.623 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:15.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:15.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:15.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:15.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:15.871 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:15.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:15.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:15.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:15.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:15.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:16.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:15 smithi162 ceph-mon[20279]: pgmap v390: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 1.6 MiB/s wr, 775 op/s; 12990/278478 objects misplaced (4.665%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:03:16.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:16.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:16.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:16.124 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:16.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:15 smithi151 ceph-mon[16639]: pgmap v390: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 1.6 MiB/s wr, 775 op/s; 12990/278478 objects misplaced (4.665%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:03:16.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:16.248 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:16.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:15 smithi187 ceph-mon[20072]: pgmap v390: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.7 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 1.6 MiB/s wr, 775 op/s; 12990/278478 objects misplaced (4.665%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:03:16.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:16.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:16.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:16.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:16.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:16.501 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:16.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:16.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:16.629 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:16.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:16.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:16.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:16.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:16.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:16.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:16.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:16.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:16.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:17.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:03:17.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:17.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:17.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:17.161 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:17.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:17.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:17.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:17.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:17.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:03:17.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:17.428 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:17.429 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:17.439 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:17.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:17.554 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:17.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:17.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:17.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:17.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:17.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:17.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:17.817 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:17.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:17.919 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:17.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:03:17.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:17.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:17.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:18.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:18.066 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:18.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:18.101 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:18.101 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:03:18.101 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:03:18.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:18.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:18.195 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:18.195 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:18.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:18.331 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:18.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:18.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:18.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:18.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:18 smithi151 ceph-mon[16639]: pgmap v391: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 897 op/s; 12568/271188 objects misplaced (4.634%); 9.2 MiB/s, 147 objects/s recovering 2024-09-02T10:03:18.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:18.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:18 smithi187 ceph-mon[20072]: pgmap v391: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 897 op/s; 12568/271188 objects misplaced (4.634%); 9.2 MiB/s, 147 objects/s recovering 2024-09-02T10:03:18.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:18.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:18 smithi162 ceph-mon[20279]: pgmap v391: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 897 op/s; 12568/271188 objects misplaced (4.634%); 9.2 MiB/s, 147 objects/s recovering 2024-09-02T10:03:18.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:18.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:18.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:18.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:18.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:18.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:18.836 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:18.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:18.861 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:18.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:18.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:18.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:18.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:19.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:03:19.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:19.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:19.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:19.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:19.278 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:19.279 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:19.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:19.404 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:19.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:19.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:19.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:19.530 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:19.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:19.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:19.658 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:19.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:19.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:19.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:19.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:19.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:19.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:19.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:19.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:19.931 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:19.970 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:03:19.971 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:03:20.002 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:03:20.003 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:03:20.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:20.033 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:03:20.033 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:03:20.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:19 smithi151 ceph-mon[16639]: osdmap e143: 12 total, 12 up, 11 in 2024-09-02T10:03:20.033 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:19 smithi151 ceph-mon[16639]: pgmap v393: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 12398/268188 objects misplaced (4.623%); 5.9 MiB/s, 94 objects/s recovering 2024-09-02T10:03:20.041 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:20.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:20.064 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:03:20.064 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:03:20.090 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:03:20.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:03:20.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:19 smithi162 ceph-mon[20279]: osdmap e143: 12 total, 12 up, 11 in 2024-09-02T10:03:20.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:19 smithi162 ceph-mon[20279]: pgmap v393: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 12398/268188 objects misplaced (4.623%); 5.9 MiB/s, 94 objects/s recovering 2024-09-02T10:03:20.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:03:20.121 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:03:20.133 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:03:20.134 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:03:20.134 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-02T10:03:20.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 110 3.0 2024-09-02T10:03:20.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:20.146 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:03:20.146 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:03:20.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:20.165 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:20.178 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:03:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:03:20.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:03:20.209 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:03:20.246 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:03:20.246 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:03:20.255 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:19 smithi187 ceph-mon[20072]: osdmap e143: 12 total, 12 up, 11 in 2024-09-02T10:03:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:19 smithi187 ceph-mon[20072]: pgmap v393: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.5 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 720 op/s; 12398/268188 objects misplaced (4.623%); 5.9 MiB/s, 94 objects/s recovering 2024-09-02T10:03:20.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:20.277 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:03:20.277 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:03:20.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:20.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:20.315 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:03:20.315 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:03:20.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:20.411 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:20.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:20.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:20.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:20.537 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:20.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:20.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:20.663 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:20.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:20.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:20.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:20.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:20.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:20.930 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:20.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:20.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:21.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:21.055 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:21.056 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:21.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3986192793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:21.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:21.180 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:21.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:21.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:21.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3986192793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3986192793' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:21.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:21.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:21.308 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:21.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:21.426 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:21.432 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:21.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:21.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:21.442 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:21.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:21.561 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:21.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:21.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:21.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:21.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:21.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:21.794 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:03:21.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:21.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:21.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:21.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1312773061' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:21.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:21 smithi151 ceph-mon[16639]: pgmap v394: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 12115/264192 objects misplaced (4.586%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:03:21.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:21.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:21.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:21.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:22.058 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:03:22.058 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:03:22.058 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:03:22.058 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:03:22.058 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:03:22.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:22.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1312773061' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:22.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:21 smithi162 ceph-mon[20279]: pgmap v394: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 12115/264192 objects misplaced (4.586%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:03:22.099 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:22.100 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:22.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1312773061' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:22.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:21 smithi187 ceph-mon[20072]: pgmap v394: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.4 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 699 op/s; 12115/264192 objects misplaced (4.586%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:03:22.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:22.232 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:22.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:22.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:22.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:22.370 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:22.371 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:22.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:22.496 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:22.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:22.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:22.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:22.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:22.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:22.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:22.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:22.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:22.894 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:22.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:22.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:22 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1312773061' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:03:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:22 smithi187 ceph-mon[20072]: osdmap e144: 12 total, 12 up, 11 in 2024-09-02T10:03:23.022 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:23.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:23.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:22 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1312773061' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:03:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:22 smithi162 ceph-mon[20279]: osdmap e144: 12 total, 12 up, 11 in 2024-09-02T10:03:23.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:23.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:22 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:23.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1312773061' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:03:23.145 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:22 smithi151 ceph-mon[16639]: osdmap e144: 12 total, 12 up, 11 in 2024-09-02T10:03:23.152 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:23.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:23.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:23.279 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:23.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:23.280 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:23.381 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:23.412 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:23.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:23.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:03:23.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:23.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:03:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:23.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:23.543 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:23.644 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:23.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:23.672 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:23.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:23.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:23.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:23.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:23.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:23 smithi162 ceph-mon[20279]: pgmap v396: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 971 op/s; 11744/255159 objects misplaced (4.603%); 9.7 MiB/s, 155 objects/s recovering 2024-09-02T10:03:23.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:23 smithi162 ceph-mon[20279]: osdmap e145: 12 total, 12 up, 11 in 2024-09-02T10:03:23.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:23.921 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:23.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:23 smithi187 ceph-mon[20072]: pgmap v396: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 971 op/s; 11744/255159 objects misplaced (4.603%); 9.7 MiB/s, 155 objects/s recovering 2024-09-02T10:03:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:23 smithi187 ceph-mon[20072]: osdmap e145: 12 total, 12 up, 11 in 2024-09-02T10:03:24.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:24.033 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:24.044 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:24.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:24.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:24.068 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:24.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:24.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:24.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:24.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:23 smithi151 ceph-mon[16639]: pgmap v396: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.2 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 971 op/s; 11744/255159 objects misplaced (4.603%); 9.7 MiB/s, 155 objects/s recovering 2024-09-02T10:03:24.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:23 smithi151 ceph-mon[16639]: osdmap e145: 12 total, 12 up, 11 in 2024-09-02T10:03:24.272 INFO:teuthology.orchestra.run.smithi151.stdout:moved 8 / 111 (7.20721%) 2024-09-02T10:03:24.272 INFO:teuthology.orchestra.run.smithi151.stdout:avg 10.0909 2024-09-02T10:03:24.272 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 1.50481 -> 1.23983 (expected baseline 3.02879) 2024-09-02T10:03:24.272 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.5 with 8 -> 9 pgs (0.792793 -> 0.891892 * mean) 2024-09-02T10:03:24.272 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 13 -> 11 pgs (1.28829 -> 1.09009 * mean) 2024-09-02T10:03:24.272 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:03:24.272 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:03:24.272 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-09-02T10:03:24.273 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:03:24.273 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0159 2024-09-02T10:03:24.273 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0175 2024-09-02T10:03:24.273 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9 weight 1.0000 -> 0.7213 2024-09-02T10:03:24.273 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 1.0000 -> 0.6917 2024-09-02T10:03:24.273 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 weight 1.0000 -> 0.8319 2024-09-02T10:03:24.273 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 1.0000 -> 0.8402 2024-09-02T10:03:24.273 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:03:24.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:24.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:24.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:24.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:24.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:24.438 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 10:03:24 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: 2024-09-02T10:03:24.270+0000 7fd7798ae640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-02T10:03:24.440 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:24.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:24.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:24.565 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:24.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:24.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:24.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:24.690 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:24.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:24.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:24.817 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:24.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:24.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:24.946 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:24.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:25.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:03:25.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:25.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:24 smithi151 ceph-mon[16639]: from='client.24691 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:25.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:24 smithi162 ceph-mon[20279]: from='client.24691 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:25.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:25.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:25.163 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-02T10:03:25.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-02T10:03:25.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:25.238 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:25.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:25.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:24 smithi187 ceph-mon[20072]: from='client.24691 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:25.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:25.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:25.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:25.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:25.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:25.510 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:25.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:25.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:25.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:25.747 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:25.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:25.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:25.772 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:25.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:25.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:25.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:26.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:26.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:26.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:26.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:25 smithi162 ceph-mon[20279]: pgmap v398: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 809 op/s; 11744/252852 objects misplaced (4.645%); 6.5 MiB/s, 103 objects/s recovering 2024-09-02T10:03:26.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:25 smithi151 ceph-mon[16639]: pgmap v398: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 809 op/s; 11744/252852 objects misplaced (4.645%); 6.5 MiB/s, 103 objects/s recovering 2024-09-02T10:03:26.138 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:26.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:26.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:26.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:25 smithi187 ceph-mon[20072]: pgmap v398: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.1 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 809 op/s; 11744/252852 objects misplaced (4.645%); 6.5 MiB/s, 103 objects/s recovering 2024-09-02T10:03:26.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:26.290 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:26.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:26.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:26.419 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:26.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:26.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:26.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:26.545 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:26.546 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:26.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:26.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:26.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:26.671 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:26.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:26.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:26.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:26.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:26.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:26.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:26.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:27.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:27.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:27.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:27.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:27.194 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:27.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:27.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:27.306 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:03:27.306 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:03:27.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:03:27.306 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:03:27.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.306 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.14157.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-02T10:03:27.306 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T09:56:55.078494+0000", 2024-09-02T10:03:27.306 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 391.18657996799999, 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37154881400000001, 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.14157", 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.151:0/1942534576", 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 1 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.078494+0000", 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.078494+0000", 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.078495+0000", 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.294e-06 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.078508+0000", 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3256e-05 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.078518+0000", 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2709999999999998e-06 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.078526+0000", 2024-09-02T10:03:27.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8809999999999996e-06 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.078570+0000", 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3979e-05 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.078576+0000", 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7379999999999997e-06 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.447448+0000", 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36887160699999999 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.447587+0000", 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013918599999999999 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,2", 2024-09-02T10:03:27.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.447732+0000", 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145069 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.449910+0000", 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002177544 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.449944+0000", 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4721000000000002e-05 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.449986+0000", 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2001e-05 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.450018+0000", 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1557999999999998e-05 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T09:56:55.450043+0000", 2024-09-02T10:03:27.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4709e-05 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:7841 2.2 e97/92 2:45ea9ad1:::benchmark_data_smithi187_45882_object7840:head v 97'525, mlcod=97'521)", 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:39.581770+0000", 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 106.683304319, 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21803672299999999, 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.581770+0000", 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.581770+0000", 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.581770+0000", 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6300000000000005e-07 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.581819+0000", 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8532999999999998e-05 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799209+0000", 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21738985399999999 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799213+0000", 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2830000000000003e-06 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799266+0000", 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2677999999999999e-05 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799283+0000", 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7192000000000001e-05 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799767+0000", 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000484537 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799806+0000", 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8983000000000003e-05 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:9139 2.4 2:2450fae4:::benchmark_data_smithi187_45882_object9138:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.593813+0000", 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 105.671261259, 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18925758400000001, 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.318 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9139 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.593813+0000", 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.593813+0000", 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.593813+0000", 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.593863+0000", 2024-09-02T10:03:27.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9240000000000003e-05 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.593874+0000", 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1669e-05 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.593880+0000", 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5879999999999997e-06 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.593919+0000", 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8758999999999999e-05 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.593999+0000", 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9878999999999999e-05 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.594082+0000", 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3362999999999995e-05 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.594554+0000", 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00047226100000000002 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.595110+0000", 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00055612900000000004 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.783007+0000", 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.187897172 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.783041+0000", 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3905000000000001e-05 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.783070+0000", 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8761000000000001e-05 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.324 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:9139 2.4 e98/92 ondisk, result = 0)", 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.594677+0000", 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 105.67039709399999, 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18839860999999999, 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.594677+0000", 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.594677+0000", 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.594677+0000", 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1500000000000004e-07 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.594681+0000", 2024-09-02T10:03:27.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3009999999999998e-06 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.782958+0000", 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18827759999999999 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.782963+0000", 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.267e-06 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.782999+0000", 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6075999999999998e-05 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.783004+0000", 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7300000000000002e-06 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.783075+0000", 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1020999999999996e-05 2024-09-02T10:03:27.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:9160 2.6 2:662c4571:::benchmark_data_smithi187_45882_object9159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.618528+0000", 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 105.646545393, 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18547920600000001, 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.329 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9160 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618528+0000", 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618528+0000", 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618529+0000", 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.92e-07 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618599+0000", 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0054000000000004e-05 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618610+0000", 2024-09-02T10:03:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0784999999999999e-05 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618615+0000", 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7129999999999996e-06 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618645+0000", 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0524000000000001e-05 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618725+0000", 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0030000000000005e-05 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618852+0000", 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012663799999999999 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.619289+0000", 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00043685600000000001 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.623485+0000", 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041958489999999998 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.803950+0000", 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18046525899999999 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.803984+0000", 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3804999999999999e-05 2024-09-02T10:03:27.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.804008+0000", 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3901e-05 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:11302 2.c e98/92 2:30c239d5:::benchmark_data_smithi187_45882_object11301:head v 98'706, mlcod=98'703)", 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:42.312851+0000", 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 103.95222293899999, 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21628996, 2024-09-02T10:03:27.335 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.312851+0000", 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.312851+0000", 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.312852+0000", 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9400000000000004e-07 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.312909+0000", 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6777999999999997e-05 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.528456+0000", 2024-09-02T10:03:27.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21554783999999999 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.528469+0000", 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2279999999999999e-05 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.528572+0000", 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010367899999999999 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.528587+0000", 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4841000000000001e-05 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.529109+0000", 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052206500000000005 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.529141+0000", 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1582999999999999e-05 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.339 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:21412 2.9 e103/92 ondisk, result = 0)", 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:50.215949+0000", 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 96.049124328999994, 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19593877900000001, 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:50.215949+0000", 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:50.215949+0000", 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:50.215950+0000", 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000002e-07 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:50.215953+0000", 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7650000000000002e-06 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:50.411742+0000", 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19578912700000001 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:50.411748+0000", 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4590000000000003e-06 2024-09-02T10:03:27.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:50.411797+0000", 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8819999999999997e-05 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:50.411805+0000", 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3440000000000001e-06 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:50.411888+0000", 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2867000000000005e-05 2024-09-02T10:03:27.343 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:27115 2.4 2:211dbe49:::benchmark_data_smithi187_45882_object27114:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:54.683246+0000", 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 91.581827704000005, 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.192378888, 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.344 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 27115 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.683246+0000", 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.683246+0000", 2024-09-02T10:03:27.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.683248+0000", 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5379999999999999e-06 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.683283+0000", 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5342999999999997e-05 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.683297+0000", 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4378e-05 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.683304+0000", 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9720000000000003e-06 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.683820+0000", 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000515789 2024-09-02T10:03:27.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.683939+0000", 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119252 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.684241+0000", 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030161800000000002 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.685903+0000", 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016616809999999999 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.686114+0000", 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000211795 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.875555+0000", 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.189440104 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.875592+0000", 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7741000000000003e-05 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.875625+0000", 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2676999999999999e-05 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.350 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:40720 2.4 2:2da6b0e4:::benchmark_data_smithi187_45882_object40719:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:05.659791+0000", 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.605283315999998, 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20840622, 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 40720 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.659791+0000", 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.659791+0000", 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.659792+0000", 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.33e-06 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.659837+0000", 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4938999999999998e-05 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.659848+0000", 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1243999999999999e-05 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.659856+0000", 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7090000000000002e-06 2024-09-02T10:03:27.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.659911+0000", 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4864000000000002e-05 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.659999+0000", 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8899000000000004e-05 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:03:27.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.660093+0000", 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3278999999999999e-05 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.661687+0000", 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001594217 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.661836+0000", 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148603 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.868113+0000", 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.206277551 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.868150+0000", 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6525e-05 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.868197+0000", 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.706e-05 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.356 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:40720 2.4 e111/92 ondisk, result = 0)", 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:05.661290+0000", 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.603783781999994, 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20691416200000001, 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.661290+0000", 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.661290+0000", 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.661291+0000", 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3799999999999996e-07 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.661294+0000", 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.852e-06 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.868045+0000", 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20675172 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.868052+0000", 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7340000000000002e-06 2024-09-02T10:03:27.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.868099+0000", 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6536000000000003e-05 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.868108+0000", 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0010000000000007e-06 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.868204+0000", 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6481e-05 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:40751 2.6 2:6519cd7e:::benchmark_data_smithi187_45882_object40750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:05.736122+0000", 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 80.528951383999996, 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.189574402, 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.361 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 40751 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.736122+0000", 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.736122+0000", 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.736123+0000", 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1200000000000002e-07 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.736184+0000", 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0390000000000003e-05 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.736196+0000", 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687000000000001e-05 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.736204+0000", 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9039999999999992e-06 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.736258+0000", 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3538000000000001e-05 2024-09-02T10:03:27.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.736376+0000", 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118062 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.736465+0000", 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9446999999999999e-05 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.737437+0000", 2024-09-02T10:03:27.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097162699999999997 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.827581+0000", 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.090144554000000002 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.925631+0000", 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.098049387000000002 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.925665+0000", 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4666999999999999e-05 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.925697+0000", 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1427000000000003e-05 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54905 2.2 e113/92 2:470b2221:::benchmark_data_smithi187_45882_object54904:head v 113'3507, mlcod=113'3504)", 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.133347+0000", 2024-09-02T10:03:27.367 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 69.131726740000005, 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26424163000000001, 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.133347+0000", 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.133347+0000", 2024-09-02T10:03:27.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.133348+0000", 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5400000000000001e-07 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.133405+0000", 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7853000000000003e-05 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.395793+0000", 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26238781799999999 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.395800+0000", 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2639999999999997e-06 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.395862+0000", 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2212999999999995e-05 2024-09-02T10:03:27.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.395880+0000", 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8236e-05 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.397552+0000", 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001671715 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.397589+0000", 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6977000000000002e-05 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:54924 2.6 2:6f9b4ae3:::benchmark_data_smithi187_45882_object54923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.150918+0000", 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 69.114155902999997, 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22757286900000001, 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.372 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 54924 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150918+0000", 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150918+0000", 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150920+0000", 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1160000000000002e-06 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150949+0000", 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9124000000000001e-05 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150964+0000", 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5111e-05 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150971+0000", 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9349999999999997e-06 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.151021+0000", 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0238999999999998e-05 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.151120+0000", 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8702999999999993e-05 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.151213+0000", 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.302e-05 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.152690+0000", 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001476761 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.153002+0000", 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031222999999999998 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.378379+0000", 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22537722900000001 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.378438+0000", 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9086999999999999e-05 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.378491+0000", 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2314000000000001e-05 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:55061 2.6 2:670fe1b2:::benchmark_data_smithi187_45882_object55060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.478779+0000", 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 68.786295315000004, 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33186097799999997, 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55061 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.379 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478779+0000", 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478779+0000", 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478780+0000", 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7039999999999999e-06 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478820+0000", 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0166000000000001e-05 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478837+0000", 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.615e-05 2024-09-02T10:03:27.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478843+0000", 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478902+0000", 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8517000000000002e-05 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.382 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.479044+0000", 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014225300000000001 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.479203+0000", 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015919400000000001 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.480839+0000", 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016358060000000001 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.482347+0000", 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015076200000000001 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.810532+0000", 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32818559200000003 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.810591+0000", 2024-09-02T10:03:27.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8059000000000001e-05 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.810639+0000", 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8965000000000003e-05 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:55087 2.9 2:9c3a08ac:::benchmark_data_smithi187_45882_object55086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.521103+0000", 2024-09-02T10:03:27.385 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 68.743970441000002, 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.239958381, 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55087 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521103+0000", 2024-09-02T10:03:27.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521103+0000", 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521104+0000", 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6200000000000006e-07 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521126+0000", 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1237e-05 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521137+0000", 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1243000000000001e-05 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521142+0000", 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.057e-06 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521179+0000", 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6770999999999997e-05 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521263+0000", 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4365999999999998e-05 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521352+0000", 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9184000000000005e-05 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.522733+0000", 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013806129999999999 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.523181+0000", 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044799100000000001 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.760954+0000", 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23777305800000001 2024-09-02T10:03:27.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.761014+0000", 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9953e-05 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.761062+0000", 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7945999999999998e-05 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:59888 2.9 2:941a6e5c:::benchmark_data_smithi187_45882_object59887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:21.653029+0000", 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.612045136999996, 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21294406900000001, 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 59888 2024-09-02T10:03:27.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.653029+0000", 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.653029+0000", 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.653030+0000", 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1850000000000001e-06 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.653073+0000", 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.282e-05 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.653087+0000", 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4034e-05 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.653093+0000", 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1e-06 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.653150+0000", 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6929000000000003e-05 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.653277+0000", 2024-09-02T10:03:27.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012680999999999999 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.653424+0000", 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014701300000000001 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.655162+0000", 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017381090000000001 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.659326+0000", 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041647940000000003 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.865895+0000", 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.206568056 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.865928+0000", 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3615999999999998e-05 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.865973+0000", 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4603e-05 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:59925 2.9 2:9903c958:::benchmark_data_smithi187_45882_object59924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:21.686970+0000", 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.578103553000005, 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17917035100000001, 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.398 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 59925 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.686970+0000", 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.686970+0000", 2024-09-02T10:03:27.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.686971+0000", 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1999999999999999e-06 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.687050+0000", 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8034999999999994e-05 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.687062+0000", 2024-09-02T10:03:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2649e-05 2024-09-02T10:03:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.687068+0000", 2024-09-02T10:03:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.806e-06 2024-09-02T10:03:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.687123+0000", 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5172e-05 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.687232+0000", 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108686 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.687321+0000", 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8778999999999998e-05 2024-09-02T10:03:27.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.689264+0000", 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019435609999999999 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.692397+0000", 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031328630000000001 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.866016+0000", 2024-09-02T10:03:27.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17361922399999999 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.866069+0000", 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2802e-05 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.866141+0000", 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1574000000000003e-05 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.404 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:59931 2.9 2:97e421be:::benchmark_data_smithi187_45882_object59930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:21.690043+0000", 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.575030341000001, 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17781038900000001, 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.405 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 59931 2024-09-02T10:03:27.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.406 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.690043+0000", 2024-09-02T10:03:27.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.690043+0000", 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.690044+0000", 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8599999999999997e-07 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.408 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.690106+0000", 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1557999999999996e-05 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.690116+0000", 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.995e-06 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.690121+0000", 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8509999999999998e-06 2024-09-02T10:03:27.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.690157+0000", 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6584000000000001e-05 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.690247+0000", 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9591000000000003e-05 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.690359+0000", 2024-09-02T10:03:27.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011254100000000001 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.692216+0000", 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018560670000000001 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.704800+0000", 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012584062999999999 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.867752+0000", 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16295267199999999 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.867801+0000", 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8440999999999999e-05 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.867854+0000", 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3140000000000003e-05 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.412 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:67911 2.4 2:20b40a62:::benchmark_data_smithi187_45882_object67910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:29.867114+0000", 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.397959348000001, 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.235349739, 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.413 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 67911 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867114+0000", 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867114+0000", 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867116+0000", 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.491e-06 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867159+0000", 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2629999999999997e-05 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867171+0000", 2024-09-02T10:03:27.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2342e-05 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867179+0000", 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7649999999999993e-06 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867242+0000", 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3178999999999999e-05 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867410+0000", 2024-09-02T10:03:27.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000167823 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867560+0000", 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015035499999999999 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.870186+0000", 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002626354 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.870907+0000", 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000720679 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.102373+0000", 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.231466269 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.102417+0000", 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3509e-05 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.102464+0000", 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7342999999999997e-05 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:79093 2.4 2:2f2be3a1:::benchmark_data_smithi187_45882_object79092:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:40.951793+0000", 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.313280716999998, 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19596131899999999, 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:27.419 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 79093 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.951793+0000", 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.951793+0000", 2024-09-02T10:03:27.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.951794+0000", 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2500000000000004e-07 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.951841+0000", 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6925000000000002e-05 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.951851+0000", 2024-09-02T10:03:27.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0312000000000001e-05 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.951858+0000", 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2779999999999996e-06 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.952540+0000", 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068135600000000004 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.952627+0000", 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6753000000000001e-05 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.952719+0000", 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2622999999999997e-05 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.954237+0000", 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015175939999999999 2024-09-02T10:03:27.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.954756+0000", 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051936699999999996 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:41.147666+0000", 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19291018600000001 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:41.147696+0000", 2024-09-02T10:03:27.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0125e-05 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:41.147754+0000", 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7975e-05 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:27.425 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:03:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:27.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:27.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:27.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:27.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:27.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:27.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:27.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:27.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:27.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:27.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:27.825 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:27.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:27.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:27.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:27.956 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:27.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:28.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:28.084 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:28.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:28.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:03:28.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:03:28.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:27 smithi162 ceph-mon[20279]: osdmap e146: 12 total, 12 up, 11 in 2024-09-02T10:03:28.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:27 smithi162 ceph-mon[20279]: pgmap v400: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 11242/245676 objects misplaced (4.576%); 9.1 MiB/s, 145 objects/s recovering 2024-09-02T10:03:28.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:28.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:03:28.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:03:28.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:27 smithi151 ceph-mon[16639]: osdmap e146: 12 total, 12 up, 11 in 2024-09-02T10:03:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:27 smithi151 ceph-mon[16639]: pgmap v400: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 11242/245676 objects misplaced (4.576%); 9.1 MiB/s, 145 objects/s recovering 2024-09-02T10:03:28.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:28.210 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:03:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:03:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:27 smithi187 ceph-mon[20072]: osdmap e146: 12 total, 12 up, 11 in 2024-09-02T10:03:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:27 smithi187 ceph-mon[20072]: pgmap v400: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 5.0 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 11242/245676 objects misplaced (4.576%); 9.1 MiB/s, 145 objects/s recovering 2024-09-02T10:03:28.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:28.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:28.336 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:28.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:28.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:28.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:28.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:28.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:28.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:28.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:28.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:28.710 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:28.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:28.711 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:28.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:28.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:28.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:28.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:28.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:28.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:29.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:29.105 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:29.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:29.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:29.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:29.234 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:29.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:29.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:29.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:29.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:29.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:29.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:29.493 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:29.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:29.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:29.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:29.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:29.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:29.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:29.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:29.748 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:29.749 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:29.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:29.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:29.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:29.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:30.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:30.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:30.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:30.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:29 smithi162 ceph-mon[20279]: osdmap e147: 12 total, 12 up, 11 in 2024-09-02T10:03:30.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:29 smithi162 ceph-mon[20279]: pgmap v402: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 690 op/s; 11081/242730 objects misplaced (4.565%); 6.9 MiB/s, 110 objects/s recovering 2024-09-02T10:03:30.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:30.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:30.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:29 smithi151 ceph-mon[16639]: osdmap e147: 12 total, 12 up, 11 in 2024-09-02T10:03:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:29 smithi151 ceph-mon[16639]: pgmap v402: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 690 op/s; 11081/242730 objects misplaced (4.565%); 6.9 MiB/s, 110 objects/s recovering 2024-09-02T10:03:30.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:29 smithi187 ceph-mon[20072]: osdmap e147: 12 total, 12 up, 11 in 2024-09-02T10:03:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:29 smithi187 ceph-mon[20072]: pgmap v402: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.9 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 690 op/s; 11081/242730 objects misplaced (4.565%); 6.9 MiB/s, 110 objects/s recovering 2024-09-02T10:03:30.266 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:30.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:30.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:30.394 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:30.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:30.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:30.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:30.520 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:30.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:30.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:30.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:30.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:30.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:30.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:30.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:30.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:30.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:30.924 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:30.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:30.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:30 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:03:30.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:30 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:03:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:30 smithi151 ceph-mon[16639]: osdmap e148: 12 total, 12 up, 11 in 2024-09-02T10:03:31.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:31.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:31.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:30 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:03:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:30 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:03:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:30 smithi162 ceph-mon[20279]: osdmap e148: 12 total, 12 up, 11 in 2024-09-02T10:03:31.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:31.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:31.180 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:31.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:30 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:03:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:30 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:03:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:30 smithi187 ceph-mon[20072]: osdmap e148: 12 total, 12 up, 11 in 2024-09-02T10:03:31.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:31.305 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:31.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:31.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:31.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:03:31.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:31.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:31.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:31.542 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:31.565 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:31.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:31.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:31.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:31.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:31.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:31.814 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:31.837 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:31.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:31.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:31.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:31 smithi151 ceph-mon[16639]: pgmap v404: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 756 op/s; 10753/239229 objects misplaced (4.495%); 10 MiB/s, 165 objects/s recovering 2024-09-02T10:03:31.929 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:03:31.930 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:03:31.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:31.960 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:03:31.960 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:03:31.963 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:31.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:31.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:03:31.988 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:03:32.024 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:03:32.024 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:03:32.059 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:03:32.059 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:03:32.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:32.069 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:32.070 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:32.070 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:32.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:32.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:32.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:31 smithi162 ceph-mon[20279]: pgmap v404: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 756 op/s; 10753/239229 objects misplaced (4.495%); 10 MiB/s, 165 objects/s recovering 2024-09-02T10:03:32.096 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:03:32.096 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:03:32.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:03:32.133 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:03:32.160 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:03:32.161 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:03:32.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:32.191 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:03:32.192 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:03:32.215 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:32.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:32.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:03:32.222 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:03:32.252 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:03:32.253 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:03:32.255 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:31 smithi187 ceph-mon[20072]: pgmap v404: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 756 op/s; 10753/239229 objects misplaced (4.495%); 10 MiB/s, 165 objects/s recovering 2024-09-02T10:03:32.287 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:03:32.287 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:03:32.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:32.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:32.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:32.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:32.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:32.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:32.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:32.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:32.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:32.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:32.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:32.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:32.722 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:32.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:32.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:32.850 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:32.851 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:32.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:32.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:32.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:33.066 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2914865832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:33.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:33.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2914865832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:33.103 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:33.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:33.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2914865832' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:33.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:33.232 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:33.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:33.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:33.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:33.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:33.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:33.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:33.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:33.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:33.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:33.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:33.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:33.718 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 111 (0%) 2024-09-02T10:03:33.718 INFO:teuthology.orchestra.run.smithi151.stdout:avg 10.0909 2024-09-02T10:03:33.718 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 1.50481 -> 1.50481 (expected baseline 3.02879) 2024-09-02T10:03:33.718 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.5 with 8 -> 8 pgs (0.792793 -> 0.792793 * mean) 2024-09-02T10:03:33.718 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 13 -> 13 pgs (1.28829 -> 1.28829 * mean) 2024-09-02T10:03:33.718 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:03:33.719 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-09-02T10:03:33.719 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-09-02T10:03:33.719 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:03:33.719 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0146 2024-09-02T10:03:33.719 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.1462 2024-09-02T10:03:33.719 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:03:33.766 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:33.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:33.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:33.880 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:03:33.881 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:03:33.881 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:33 smithi151 ceph-mon[16639]: pgmap v405: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 994 op/s; 10280/230952 objects misplaced (4.451%); 12 MiB/s, 191 objects/s recovering 2024-09-02T10:03:33.881 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:33 smithi151 ceph-mon[16639]: from='client.24752 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:33.881 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:33 smithi151 ceph-mon[16639]: osdmap e149: 12 total, 12 up, 11 in 2024-09-02T10:03:33.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:33.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:33.915 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:33 smithi187 ceph-mon[20072]: pgmap v405: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 994 op/s; 10280/230952 objects misplaced (4.451%); 12 MiB/s, 191 objects/s recovering 2024-09-02T10:03:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:33 smithi187 ceph-mon[20072]: from='client.24752 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:33 smithi187 ceph-mon[20072]: osdmap e149: 12 total, 12 up, 11 in 2024-09-02T10:03:34.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:34.041 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:34.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:34.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:34.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:33 smithi162 ceph-mon[20279]: pgmap v405: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 994 op/s; 10280/230952 objects misplaced (4.451%); 12 MiB/s, 191 objects/s recovering 2024-09-02T10:03:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:33 smithi162 ceph-mon[20279]: from='client.24752 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:33 smithi162 ceph-mon[20279]: osdmap e149: 12 total, 12 up, 11 in 2024-09-02T10:03:34.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:34.169 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:34.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:34.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:34.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:34.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:34.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:34.424 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:34.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:34.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:34.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:34.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:34.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:34.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:34.682 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:34.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:34.784 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:34.808 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:34.809 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:34.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:34.937 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:34.938 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:35.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:35.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:35.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:35.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:35.204 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:35.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:35.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:35.341 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:35.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:35.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:35.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:35.468 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:35.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:35.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:35.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:35.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:35.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:35.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:03:35.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:35.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:35.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:35.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:35.874 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:35.876 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:35.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:36.000 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:36.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:36.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:36.128 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:36.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:36.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:36.150 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:03:36.196 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-02T10:03:36.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-09-02T10:03:36.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:36.256 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:36.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:36.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:36.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:36.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:36.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:36.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:36 smithi151 ceph-mon[16639]: pgmap v407: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 10280/228765 objects misplaced (4.494%); 7.9 MiB/s, 126 objects/s recovering 2024-09-02T10:03:36.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2726206222' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:03:36.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:36 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:03:36.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:36 smithi187 ceph-mon[20072]: pgmap v407: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 10280/228765 objects misplaced (4.494%); 7.9 MiB/s, 126 objects/s recovering 2024-09-02T10:03:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2726206222' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:03:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:36 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:03:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:36.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:36.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:36.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:36 smithi162 ceph-mon[20279]: pgmap v407: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 10280/228765 objects misplaced (4.494%); 7.9 MiB/s, 126 objects/s recovering 2024-09-02T10:03:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2726206222' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:03:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:36 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:03:36.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:36.651 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:36.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:36.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:36.778 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:36.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:36.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:03:36.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:36.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:36.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:37.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:37.040 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:37.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:37.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:37.067 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:37.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:37.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:37.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:37.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:37.302 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:37.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:37.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:37.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:37 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:37.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:37 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:03:37.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:37 smithi151 ceph-mon[16639]: osdmap e150: 12 total, 12 up, 11 in 2024-09-02T10:03:37.428 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:37.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:37.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:37 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:37 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:03:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:37 smithi187 ceph-mon[20072]: osdmap e150: 12 total, 12 up, 11 in 2024-09-02T10:03:37.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:37.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:37.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:37.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:37 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:37.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:37 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:03:37.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:37 smithi162 ceph-mon[20279]: osdmap e150: 12 total, 12 up, 11 in 2024-09-02T10:03:37.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:37.692 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:37.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:37.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:37.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:37.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:37.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:37.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:37.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:37.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:38.055 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:38.079 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:38.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:38.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:38.172 INFO:teuthology.orchestra.run.smithi151.stdout:moved 9 / 111 (8.10811%) 2024-09-02T10:03:38.172 INFO:teuthology.orchestra.run.smithi151.stdout:avg 10.0909 2024-09-02T10:03:38.172 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 1.50481 -> 1.50481 (expected baseline 3.02879) 2024-09-02T10:03:38.172 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.5 with 8 -> 9 pgs (0.792793 -> 0.891892 * mean) 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 13 -> 11 pgs (1.28829 -> 1.09009 * mean) 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0141 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0155 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9 weight 1.0000 -> 0.7031 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 1.0000 -> 0.6639 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 weight 1.0000 -> 0.8176 2024-09-02T10:03:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 weight 1.0000 -> 0.8534 2024-09-02T10:03:38.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:38.283 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:38.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:38.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:38.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:38.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:38.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:38 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:03:38.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:38 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:03:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:38 smithi151 ceph-mon[16639]: pgmap v409: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 857 op/s; 9719/221964 objects misplaced (4.379%); 12 MiB/s, 186 objects/s recovering 2024-09-02T10:03:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:38 smithi151 ceph-mon[16639]: osdmap e151: 12 total, 12 up, 11 in 2024-09-02T10:03:38.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'53583','3':'43508','7':'55926','9':'46078'}"} : dispatch 2024-09-02T10:03:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:38 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:03:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:38 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:03:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:38 smithi187 ceph-mon[20072]: pgmap v409: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 857 op/s; 9719/221964 objects misplaced (4.379%); 12 MiB/s, 186 objects/s recovering 2024-09-02T10:03:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:38 smithi187 ceph-mon[20072]: osdmap e151: 12 total, 12 up, 11 in 2024-09-02T10:03:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'53583','3':'43508','7':'55926','9':'46078'}"} : dispatch 2024-09-02T10:03:38.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:38.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:38.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:38.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:38 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:03:38.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:38 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:03:38.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:38 smithi162 ceph-mon[20279]: pgmap v409: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 857 op/s; 9719/221964 objects misplaced (4.379%); 12 MiB/s, 186 objects/s recovering 2024-09-02T10:03:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:38 smithi162 ceph-mon[20279]: osdmap e151: 12 total, 12 up, 11 in 2024-09-02T10:03:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'53583','3':'43508','7':'55926','9':'46078'}"} : dispatch 2024-09-02T10:03:38.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:38.656 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:38.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:38.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:38.783 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:38.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:38.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:38.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:03:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:38.908 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:39.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:39.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:39.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:39.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:39.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:39.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:39.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:39.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:39.205 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:03:39.205 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:03:39.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:39.280 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:39.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:39.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:39.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:39.404 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:39.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:39.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:03:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:03:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:39 smithi151 ceph-mon[16639]: from='client.24712 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'53583','3':'43508','7':'55926','9':'46078'}"}]': finished 2024-09-02T10:03:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:39 smithi151 ceph-mon[16639]: osdmap e152: 12 total, 12 up, 11 in 2024-09-02T10:03:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:39 smithi151 ceph-mon[16639]: osdmap e153: 12 total, 12 up, 11 in 2024-09-02T10:03:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:03:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:03:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:39 smithi187 ceph-mon[20072]: from='client.24712 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'53583','3':'43508','7':'55926','9':'46078'}"}]': finished 2024-09-02T10:03:39.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:39 smithi187 ceph-mon[20072]: osdmap e152: 12 total, 12 up, 11 in 2024-09-02T10:03:39.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:39 smithi187 ceph-mon[20072]: osdmap e153: 12 total, 12 up, 11 in 2024-09-02T10:03:39.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:39.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:39.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:39.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:39.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:03:39.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:03:39.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:39 smithi162 ceph-mon[20279]: from='client.24712 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'53583','3':'43508','7':'55926','9':'46078'}"}]': finished 2024-09-02T10:03:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:39 smithi162 ceph-mon[20279]: osdmap e152: 12 total, 12 up, 11 in 2024-09-02T10:03:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:39 smithi162 ceph-mon[20279]: osdmap e153: 12 total, 12 up, 11 in 2024-09-02T10:03:39.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:39.652 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:39.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:39.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:39.776 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:39.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:39.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:39.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:39.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:39.901 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:39.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:03:40.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:40.025 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:40.026 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:40.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:40.134 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:40.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:40.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:40.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:40.277 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:40.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:40.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:40.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:40.405 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:40.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:40.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:40.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:40 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:03:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:40 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:03:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:40 smithi151 ceph-mon[16639]: pgmap v413: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 820 op/s; 9535/218913 objects misplaced (4.356%); 12 MiB/s, 186 objects/s recovering 2024-09-02T10:03:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:40 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:03:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:40 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:03:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:40 smithi151 ceph-mon[16639]: osdmap e154: 12 total, 12 up, 11 in 2024-09-02T10:03:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:40 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:03:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:40 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:03:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:40 smithi187 ceph-mon[20072]: pgmap v413: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 820 op/s; 9535/218913 objects misplaced (4.356%); 12 MiB/s, 186 objects/s recovering 2024-09-02T10:03:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:40 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:03:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:40 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:03:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:40 smithi187 ceph-mon[20072]: osdmap e154: 12 total, 12 up, 11 in 2024-09-02T10:03:40.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:40.532 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:40.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:40.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:40.589 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-02T10:03:40.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-09-02T10:03:40.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:40 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:03:40.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:40 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:03:40.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:40 smithi162 ceph-mon[20279]: pgmap v413: 37 pgs: 3 active+recovering+undersized+remapped, 34 active+clean; 4.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 820 op/s; 9535/218913 objects misplaced (4.356%); 12 MiB/s, 186 objects/s recovering 2024-09-02T10:03:40.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:40 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:03:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:40 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:03:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:40 smithi162 ceph-mon[20279]: osdmap e154: 12 total, 12 up, 11 in 2024-09-02T10:03:40.609 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:40.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:40.661 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:40.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:40.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:40.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:40.791 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:40.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:40.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:40.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:40.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:40.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:41.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:41.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:41.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:41.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:41.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:41.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:41.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:41.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:41.334 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:41.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:41.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:03:41.431 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:03:41.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:41 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:03:41.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:41 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:03:41.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:41 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:03:41.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:41 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:03:41.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:41.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:41.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:41.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:41.461 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:03:41.462 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:03:41.490 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:03:41.491 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:03:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:41 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:03:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:41 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:03:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:41 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:03:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:41 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:03:41.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:03:41.519 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:03:41.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:03:41.558 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:03:41.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:41.590 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:41.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:41.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:41.594 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:03:41.594 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:03:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:41 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:03:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:41 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:03:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:41 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:03:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:41 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:03:41.629 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:03:41.630 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:03:41.667 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:03:41.668 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:03:41.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:41.712 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:41.714 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:41.715 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:41.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:03:41.716 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:03:41.737 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:41.761 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:03:41.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:03:41.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:03:41.797 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:03:41.816 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:41.839 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:03:41.839 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:03:41.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:41.840 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:41.840 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:41.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:41.964 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:41.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:42.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:42.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:42.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:42.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:42.195 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:03:42.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:42.227 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:42.228 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:42.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:42.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:42.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:42.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:42.408 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:42 smithi151 ceph-mon[16639]: pgmap v415: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 4.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 767 B/s rd, 24 KiB/s wr, 497 op/s; 7/216021 objects degraded (0.003%); 13640/216021 objects misplaced (6.314%); 8.8 MiB/s, 140 objects/s recovering 2024-09-02T10:03:42.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:42 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 7/216021 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:03:42.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3702189773' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:42.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2530889788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:42.409 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:42 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:42.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:42.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:42.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:42 smithi187 ceph-mon[20072]: pgmap v415: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 4.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 767 B/s rd, 24 KiB/s wr, 497 op/s; 7/216021 objects degraded (0.003%); 13640/216021 objects misplaced (6.314%); 8.8 MiB/s, 140 objects/s recovering 2024-09-02T10:03:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:42 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 7/216021 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:03:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3702189773' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2530889788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:42 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:42.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:42.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:42 smithi162 ceph-mon[20279]: pgmap v415: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 4.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 767 B/s rd, 24 KiB/s wr, 497 op/s; 7/216021 objects degraded (0.003%); 13640/216021 objects misplaced (6.314%); 8.8 MiB/s, 140 objects/s recovering 2024-09-02T10:03:42.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:42 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 7/216021 objects degraded (0.003%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:03:42.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3702189773' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2530889788' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:42 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:03:42.616 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:42.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:42.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:42.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:42.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:42.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:42.869 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:42.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:42.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:43.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:43.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:43.059 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:03:43.059 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:1142 2.7 e96/92 2:e2cb33eb:::benchmark_data_smithi187_45882_object1141:head v 96'87, mlcod=96'83)", 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:35.907033+0000", 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 126.499579602, 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18660511599999999, 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.907033+0000", 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.907033+0000", 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.907034+0000", 2024-09-02T10:03:43.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0189999999999999e-06 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:35.907076+0000", 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1548000000000003e-05 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093236+0000", 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.186160574 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093243+0000", 2024-09-02T10:03:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8589999999999999e-06 2024-09-02T10:03:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093298+0000", 2024-09-02T10:03:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5068999999999997e-05 2024-09-02T10:03:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093319+0000", 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1234999999999999e-05 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093610+0000", 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000291184 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:36.093638+0000", 2024-09-02T10:03:43.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7628e-05 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:8372 2.e 2:77370da9:::benchmark_data_smithi187_45882_object8371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.033332+0000", 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 122.373280362, 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.181158239, 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.065 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8372 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033332+0000", 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.066 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033332+0000", 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033334+0000", 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.274e-06 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033372+0000", 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8554999999999999e-05 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033384+0000", 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2204999999999999e-05 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033392+0000", 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8809999999999996e-06 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033454+0000", 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1784000000000003e-05 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033610+0000", 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015655399999999999 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.033773+0000", 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016284099999999999 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.034409+0000", 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063585900000000001 2024-09-02T10:03:43.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035230+0000", 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00082086500000000005 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214426+0000", 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17919570700000001 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214457+0000", 2024-09-02T10:03:43.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1108000000000001e-05 2024-09-02T10:03:43.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214490+0000", 2024-09-02T10:03:43.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3606000000000003e-05 2024-09-02T10:03:43.071 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:8372 2.e e97/92 ondisk, result = 0)", 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.034947+0000", 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 122.37166568400001, 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17954930999999999, 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.072 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.034947+0000", 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.034947+0000", 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.034948+0000", 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0270000000000001e-06 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.034953+0000", 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3059999999999997e-06 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214376+0000", 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17942266400000001 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214381+0000", 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6970000000000001e-06 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214418+0000", 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7444000000000003e-05 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214423+0000", 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5079999999999999e-06 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.214496+0000", 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3664000000000005e-05 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:8373 2.1 2:8eb82fd1:::benchmark_data_smithi187_45882_object8372:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.035325+0000", 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 122.37128755099999, 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.164965424, 2024-09-02T10:03:43.076 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 8373 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035325+0000", 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035325+0000", 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035327+0000", 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.48e-06 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035387+0000", 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0078999999999999e-05 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035401+0000", 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4331000000000001e-05 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035410+0000", 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7430000000000008e-06 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035461+0000", 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0932e-05 2024-09-02T10:03:43.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035562+0000", 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101665 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035658+0000", 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5521999999999998e-05 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.036305+0000", 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00064723299999999999 2024-09-02T10:03:43.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.036691+0000", 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038610199999999998 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.200216+0000", 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16352460099999999 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.200251+0000", 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5323e-05 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.200290+0000", 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9412999999999997e-05 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:9147 2.3 e98/92 2:c3cba1c2:::benchmark_data_smithi187_45882_object9146:head v 98'562, mlcod=98'558)", 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.606830+0000", 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 121.799782972, 2024-09-02T10:03:43.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18017456100000001, 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.606830+0000", 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.606830+0000", 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.606830+0000", 2024-09-02T10:03:43.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5199999999999996e-07 2024-09-02T10:03:43.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.606854+0000", 2024-09-02T10:03:43.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3634000000000001e-05 2024-09-02T10:03:43.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.786442+0000", 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17958851300000001 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.786448+0000", 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.786498+0000", 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9654000000000003e-05 2024-09-02T10:03:43.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.786519+0000", 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1489e-05 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.786972+0000", 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045280299999999999 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.787004+0000", 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2336999999999999e-05 2024-09-02T10:03:43.086 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:9155 2.e 2:7586a36f:::benchmark_data_smithi187_45882_object9154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.615656+0000", 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 121.790956728, 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.161630628, 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:43.087 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 9155 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615656+0000", 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615656+0000", 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615657+0000", 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8800000000000002e-07 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615719+0000", 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2508999999999994e-05 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615728+0000", 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7099999999999996e-06 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615733+0000", 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6689999999999997e-06 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615781+0000", 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8442999999999997e-05 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.615892+0000", 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011149700000000001 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.616031+0000", 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000138861 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.616610+0000", 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057823100000000001 2024-09-02T10:03:43.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.616808+0000", 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019807700000000001 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777215+0000", 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.160407311 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777255+0000", 2024-09-02T10:03:43.092 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9620999999999999e-05 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.777287+0000", 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1911e-05 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:36445 2.1 2:8b5b62e6:::benchmark_data_smithi187_45882_object36444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:02.093891+0000", 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 100.312721935, 2024-09-02T10:03:43.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21621346299999999, 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 36445 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093891+0000", 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093891+0000", 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093892+0000", 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1400000000000001e-06 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093951+0000", 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9660000000000001e-05 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093967+0000", 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6018000000000001e-05 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.093975+0000", 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.841e-06 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.094034+0000", 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8899e-05 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.094180+0000", 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014549699999999999 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.094278+0000", 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8599999999999998e-05 2024-09-02T10:03:43.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.095564+0000", 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012853669999999999 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.102929+0000", 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0073655550000000002 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.309532+0000", 2024-09-02T10:03:43.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.206602962 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.310053+0000", 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052091099999999999 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.310104+0000", 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1013000000000001e-05 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:36445 2.1 e109/92 ondisk, result = 0)", 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:02.095796+0000", 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 100.31081665799999, 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21432100700000001, 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.095796+0000", 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.095796+0000", 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.095797+0000", 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7899999999999997e-07 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.095801+0000", 2024-09-02T10:03:43.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6430000000000003e-06 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.308322+0000", 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21252061899999999 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.308328+0000", 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3489999999999997e-06 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.309518+0000", 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001190008 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.309526+0000", 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7950000000000008e-06 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.310117+0000", 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00059081400000000005 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.103 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:38667 2.e 2:7d8529bb:::benchmark_data_smithi187_45882_object38666:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:03.897567+0000", 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 98.509045138999994, 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21087376599999999, 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:43.104 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 38667 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.897567+0000", 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.897567+0000", 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.897568+0000", 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.009e-06 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.897624+0000", 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5680999999999998e-05 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.897635+0000", 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0395e-05 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.897640+0000", 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2959999999999998e-06 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.897679+0000", 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9465000000000002e-05 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.897809+0000", 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012959000000000001 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.897932+0000", 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012332900000000001 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.899163+0000", 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012308130000000001 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.899643+0000", 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00048018099999999999 2024-09-02T10:03:43.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:04.108319+0000", 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.208675795 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:04.108371+0000", 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2169999999999997e-05 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:04.108441+0000", 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0042000000000005e-05 2024-09-02T10:03:43.109 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:38667 2.e e111/92 ondisk, result = 0)", 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:03.899344+0000", 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 98.507268432999993, 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.209128658, 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.899344+0000", 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.899344+0000", 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.899345+0000", 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1399999999999997e-07 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.899349+0000", 2024-09-02T10:03:43.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8659999999999999e-06 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:04.107778+0000", 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20842907899999999 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:04.107784+0000", 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3990000000000001e-06 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:04.108305+0000", 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052106500000000003 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:04.108313+0000", 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8499999999999994e-06 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:04.108473+0000", 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015978499999999999 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.113 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54934 2.3 e113/92 2:c516e2ca:::benchmark_data_smithi187_45882_object54933:head v 113'3500, mlcod=113'3496)", 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.164932+0000", 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 85.241680595999995, 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21055763, 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.164932+0000", 2024-09-02T10:03:43.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.164932+0000", 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.164933+0000", 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3e-07 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.164990+0000", 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7528000000000002e-05 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.373836+0000", 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.208846167 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.373845+0000", 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6999999999999997e-06 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.373903+0000", 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8158000000000001e-05 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.373923+0000", 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9596999999999999e-05 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.375452+0000", 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015294989999999999 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.375490+0000", 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7251e-05 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54937 2.3 e113/92 2:c9a686f1:::benchmark_data_smithi187_45882_object54936:head v 113'3501, mlcod=113'3497)", 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.171718+0000", 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 85.234894529000002, 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23192291400000001, 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.118 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.171718+0000", 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.171718+0000", 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.171719+0000", 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5600000000000005e-07 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.171741+0000", 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2254000000000001e-05 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.401888+0000", 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23014675900000001 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.401896+0000", 2024-09-02T10:03:43.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4870000000000002e-06 2024-09-02T10:03:43.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.401956+0000", 2024-09-02T10:03:43.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0183999999999998e-05 2024-09-02T10:03:43.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.401977+0000", 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0281000000000001e-05 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.403589+0000", 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016121620000000001 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.403641+0000", 2024-09-02T10:03:43.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2130999999999999e-05 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:55030 2.3 e113/92 2:c75f1b46:::benchmark_data_smithi187_45882_object55029:head v 113'3508, mlcod=113'3504)", 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.447258+0000", 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 84.959354133000005, 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.203635239, 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.447258+0000", 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.447258+0000", 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.447260+0000", 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1200000000000001e-06 2024-09-02T10:03:43.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.447320+0000", 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0192999999999998e-05 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.649301+0000", 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.201981625 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.649307+0000", 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0360000000000003e-06 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.649381+0000", 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3782999999999995e-05 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.649398+0000", 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7088999999999999e-05 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.650857+0000", 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001459042 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.650894+0000", 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6350999999999998e-05 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.128 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:55038 2.3 e113/92 2:c94b451a:::benchmark_data_smithi187_45882_object55037:head v 113'3509, mlcod=113'3505)", 2024-09-02T10:03:43.128 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.454012+0000", 2024-09-02T10:03:43.128 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 84.952600214, 2024-09-02T10:03:43.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.331582871, 2024-09-02T10:03:43.128 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.128 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.128 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.128 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.454012+0000", 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.454012+0000", 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.454013+0000", 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3600000000000003e-07 2024-09-02T10:03:43.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.454065+0000", 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1953000000000002e-05 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.784076+0000", 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.330010939 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.784081+0000", 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5179999999999998e-06 2024-09-02T10:03:43.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.784123+0000", 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2416000000000002e-05 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.784136+0000", 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3198e-05 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.785553+0000", 2024-09-02T10:03:43.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014169740000000001 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.785595+0000", 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2137e-05 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:55064 2.1 2:81a0c3de:::benchmark_data_smithi187_45882_object55063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.482783+0000", 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 84.923829871999999, 2024-09-02T10:03:43.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31606567899999999, 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55064 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.482783+0000", 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.482783+0000", 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.482784+0000", 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0100000000000004e-07 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.482839+0000", 2024-09-02T10:03:43.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5569000000000003e-05 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.482849+0000", 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7990000000000005e-06 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.482854+0000", 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7099999999999998e-06 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.482900+0000", 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6785e-05 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.482989+0000", 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8240999999999998e-05 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.483074+0000", 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5782e-05 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.484851+0000", 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001776148 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.485743+0000", 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00089271900000000004 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.798743+0000", 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31299996299999999 2024-09-02T10:03:43.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.798801+0000", 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7493e-05 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.798848+0000", 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7669e-05 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.138 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:55064 2.1 e113/92 ondisk, result = 0)", 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.485173+0000", 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 84.921439699000004, 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31368485699999998, 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.485173+0000", 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.485173+0000", 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.485174+0000", 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1799999999999995e-07 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.485177+0000", 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5590000000000001e-06 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.798666+0000", 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31348927999999998 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.798671+0000", 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8409999999999999e-06 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.798727+0000", 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5491000000000001e-05 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.798736+0000", 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0999999999999993e-06 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.798858+0000", 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121968 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:55079 2.1 2:8bad8344:::benchmark_data_smithi187_45882_object55078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:03:43.142 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.502421+0000", 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 84.904191284999996, 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.323267943, 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55079 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502421+0000", 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502421+0000", 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502423+0000", 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.395e-06 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502451+0000", 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8428e-05 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502463+0000", 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2182e-05 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502469+0000", 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2079999999999999e-06 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502514+0000", 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5736999999999999e-05 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502608+0000", 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3659999999999994e-05 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502696+0000", 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8391000000000001e-05 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.504184+0000", 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001487904 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.664304+0000", 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16011998399999999 2024-09-02T10:03:43.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.825582+0000", 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16127815500000001 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.825641+0000", 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8647999999999998e-05 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.825689+0000", 2024-09-02T10:03:43.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8251000000000003e-05 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:55079 2.1 e113/92 ondisk, result = 0)", 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.504406+0000", 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 84.902206504999995, 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32128970099999998, 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.504406+0000", 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.504406+0000", 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.504407+0000", 2024-09-02T10:03:43.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3300000000000001e-07 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.504411+0000", 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.245e-06 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.825518+0000", 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32110676900000001 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.825526+0000", 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5469999999999997e-06 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.825570+0000", 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3404e-05 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.825577+0000", 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2849999999999997e-06 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.825696+0000", 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118718 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:97747 2.1 2:8268f1a3:::benchmark_data_smithi187_45882_object97746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:05.429537+0000", 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.977075536000001, 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75786974200000001, 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:03:43.153 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 97747 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429537+0000", 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429537+0000", 2024-09-02T10:03:43.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429538+0000", 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0690000000000001e-06 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429603+0000", 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4875000000000005e-05 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429614+0000", 2024-09-02T10:03:43.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0604999999999999e-05 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429620+0000", 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7070000000000003e-06 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.432050+0000", 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002429381 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.432155+0000", 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010553899999999999 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.432282+0000", 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012681899999999999 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.435226+0000", 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029439179999999998 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.436254+0000", 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010281400000000001 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187337+0000", 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75108334499999996 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187376+0000", 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8974000000000003e-05 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187407+0000", 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0369999999999999e-05 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:97747 2.1 e137/92 ondisk, result = 0)", 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:05.435888+0000", 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.970724646999997, 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75152487700000004, 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:03:43.159 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.435888+0000", 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.435888+0000", 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.435889+0000", 2024-09-02T10:03:43.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8999999999999996e-07 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.435893+0000", 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3279999999999999e-06 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187266+0000", 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75137313999999999 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:43.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187271+0000", 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3709999999999999e-06 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187326+0000", 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4203999999999998e-05 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187333+0000", 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8800000000000002e-06 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:43.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:03:43.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187413+0000", 2024-09-02T10:03:43.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0264000000000003e-05 2024-09-02T10:03:43.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.163 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:43.163 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:43.163 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:03:43.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:43.187 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:43.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:43.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:43.314 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:43.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:43.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:43.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:43 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:03:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:43 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:03:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:43 smithi151 ceph-mon[16639]: osdmap e155: 12 total, 12 up, 11 in 2024-09-02T10:03:43.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:43.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:43 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:03:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:43 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:03:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:43 smithi187 ceph-mon[20072]: osdmap e155: 12 total, 12 up, 11 in 2024-09-02T10:03:43.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:43.575 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:43.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:43 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:03:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:43 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:03:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:43 smithi162 ceph-mon[20279]: osdmap e155: 12 total, 12 up, 11 in 2024-09-02T10:03:43.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:43.703 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:43.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:43.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:43.832 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:43.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:43.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:03:43.933 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:43.958 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:43.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:44.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:44.086 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:44.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:44.188 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:44.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:44.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:44.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:44.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:44.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:44.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:44.464 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:44.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:44.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:44 smithi187 ceph-mon[20072]: pgmap v417: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 32 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 624 B/s rd, 20 KiB/s wr, 510 op/s; 7/211416 objects degraded (0.003%); 21862/211416 objects misplaced (10.341%); 13 MiB/s, 206 objects/s recovering 2024-09-02T10:03:44.519 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:44 smithi151 ceph-mon[16639]: pgmap v417: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 32 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 624 B/s rd, 20 KiB/s wr, 510 op/s; 7/211416 objects degraded (0.003%); 21862/211416 objects misplaced (10.341%); 13 MiB/s, 206 objects/s recovering 2024-09-02T10:03:44.565 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:44.589 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:44.590 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:44.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:44 smithi162 ceph-mon[20279]: pgmap v417: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 32 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 624 B/s rd, 20 KiB/s wr, 510 op/s; 7/211416 objects degraded (0.003%); 21862/211416 objects misplaced (10.341%); 13 MiB/s, 206 objects/s recovering 2024-09-02T10:03:44.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:44.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:44.716 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:44.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:44.849 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:44.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:44.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:44.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:44.978 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:44.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:45.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:45.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:45.106 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:45.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:45.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:45.215 INFO:teuthology.orchestra.run.smithi151.stdout:moved 7 / 111 (6.30631%) 2024-09-02T10:03:45.215 INFO:teuthology.orchestra.run.smithi151.stdout:avg 10.0909 2024-09-02T10:03:45.215 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 1.50481 -> 2.1513 (expected baseline 3.02879) 2024-09-02T10:03:45.215 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.1 with 8 -> 8 pgs (0.792793 -> 0.792793 * mean) 2024-09-02T10:03:45.215 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.11 with 13 -> 13 pgs (1.28829 -> 1.28829 * mean) 2024-09-02T10:03:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:03:45.216 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:03:45.216 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-09-02T10:03:45.216 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:03:45.216 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0135 2024-09-02T10:03:45.216 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0148 2024-09-02T10:03:45.216 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9 weight 0.7031 -> 0.5056 2024-09-02T10:03:45.216 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 0.6639 -> 0.4450 2024-09-02T10:03:45.216 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7 weight 0.8534 -> 0.7163 2024-09-02T10:03:45.216 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 1.0000 -> 0.8461 2024-09-02T10:03:45.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:45.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:45.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:45.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'55451','3':'29165','7':'46946','9':'33133'}"} : dispatch 2024-09-02T10:03:45.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:45.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:45.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:45.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'55451','3':'29165','7':'46946','9':'33133'}"} : dispatch 2024-09-02T10:03:45.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:45.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'55451','3':'29165','7':'46946','9':'33133'}"} : dispatch 2024-09-02T10:03:45.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:45.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:45.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:45.735 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:45.736 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:45.837 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:45.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:45.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:45.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:45.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:45.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:46.002 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-02T10:03:46.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-02T10:03:46.051 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:46.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:46.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:46.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:46.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:46.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:46.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:46.239 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:46.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:46.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:46.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:46.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:03:46.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:46.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:46.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:46.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:46 smithi151 ceph-mon[16639]: pgmap v418: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 32 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 473 op/s; 7/209949 objects degraded (0.003%); 21862/209949 objects misplaced (10.413%); 10 MiB/s, 160 objects/s recovering 2024-09-02T10:03:46.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:46 smithi151 ceph-mon[16639]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:46.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'55451','3':'29165','7':'46946','9':'33133'}"}]': finished 2024-09-02T10:03:46.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:46 smithi151 ceph-mon[16639]: osdmap e156: 12 total, 12 up, 11 in 2024-09-02T10:03:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:46 smithi187 ceph-mon[20072]: pgmap v418: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 32 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 473 op/s; 7/209949 objects degraded (0.003%); 21862/209949 objects misplaced (10.413%); 10 MiB/s, 160 objects/s recovering 2024-09-02T10:03:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:46 smithi187 ceph-mon[20072]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'55451','3':'29165','7':'46946','9':'33133'}"}]': finished 2024-09-02T10:03:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:46 smithi187 ceph-mon[20072]: osdmap e156: 12 total, 12 up, 11 in 2024-09-02T10:03:46.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:46 smithi162 ceph-mon[20279]: pgmap v418: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 32 active+clean; 4.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 473 op/s; 7/209949 objects degraded (0.003%); 21862/209949 objects misplaced (10.413%); 10 MiB/s, 160 objects/s recovering 2024-09-02T10:03:46.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:46 smithi162 ceph-mon[20279]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'55451','3':'29165','7':'46946','9':'33133'}"}]': finished 2024-09-02T10:03:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:46 smithi162 ceph-mon[20279]: osdmap e156: 12 total, 12 up, 11 in 2024-09-02T10:03:46.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:46.626 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:46.626 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:03:46.626 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(2.7 157/131 [PushOp(2:eafff48b:::benchmark_data_smithi187_45882_object43986:head, version: 111'2771, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:eafff48b:::benchmark_data_smithi187_45882_object43986:head@111'2771, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:46.624554+0000", 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00052498400000000004, 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057859899999999995, 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.624554+0000", 2024-09-02T10:03:46.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.624554+0000", 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.624556+0000", 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4500000000000001e-06 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:46.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.624611+0000", 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5823999999999999e-05 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.624667+0000", 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5871000000000001e-05 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.624688+0000", 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0636e-05 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:46.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.624803+0000", 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114999 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.624820+0000", 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7020999999999998e-05 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:03:46.630 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 1 2024-09-02T10:03:46.631 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:03:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:46.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:46.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:46.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:46.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:46.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:46.880 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:46.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:46.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:47.003 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:47.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:47.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:47.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:47.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:47.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:47.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:47.253 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:47.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:47.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:03:47.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:03:47.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:47.383 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:47.396 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:47.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:47.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:47 smithi187 ceph-mon[20072]: osdmap e157: 12 total, 12 up, 11 in 2024-09-02T10:03:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:47 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:03:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:47 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:03:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:47 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:03:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:47 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:03:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:47 smithi187 ceph-mon[20072]: osdmap e158: 12 total, 12 up, 11 in 2024-09-02T10:03:47.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:47 smithi151 ceph-mon[16639]: osdmap e157: 12 total, 12 up, 11 in 2024-09-02T10:03:47.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:47 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:03:47.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:47 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:03:47.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:47 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:03:47.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:47 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:03:47.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:47 smithi151 ceph-mon[16639]: osdmap e158: 12 total, 12 up, 11 in 2024-09-02T10:03:47.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:47.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:47 smithi162 ceph-mon[20279]: osdmap e157: 12 total, 12 up, 11 in 2024-09-02T10:03:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:47 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:03:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:47 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:03:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:47 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:03:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:47 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:03:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:47 smithi162 ceph-mon[20279]: osdmap e158: 12 total, 12 up, 11 in 2024-09-02T10:03:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:47.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:47.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:47.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:47.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:47.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:47.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:47.891 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:47.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:47.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:47.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:48.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:48.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:48.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:48.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:48.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:48.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:48.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:48.292 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:48.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:48.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:48.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:48 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:03:48.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:48 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:03:48.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:48 smithi151 ceph-mon[16639]: pgmap v421: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 3 active+recovering+undersized+remapped, 31 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 577 op/s; 51/205608 objects degraded (0.025%); 18409/205608 objects misplaced (8.953%); 17 MiB/s, 269 objects/s recovering 2024-09-02T10:03:48.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:48 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 51/205608 objects degraded (0.025%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:03:48.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:48.420 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:48.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:48.442 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:03:48.443 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:03:48.496 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:03:48.496 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:03:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:48 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:03:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:48 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:03:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:48 smithi187 ceph-mon[20072]: pgmap v421: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 3 active+recovering+undersized+remapped, 31 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 577 op/s; 51/205608 objects degraded (0.025%); 18409/205608 objects misplaced (8.953%); 17 MiB/s, 269 objects/s recovering 2024-09-02T10:03:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:48 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 51/205608 objects degraded (0.025%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:03:48.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:48.552 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:03:48.552 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:03:48.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:48.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:48 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:03:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:48 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:03:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:48 smithi162 ceph-mon[20279]: pgmap v421: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 3 active+recovering+undersized+remapped, 31 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 577 op/s; 51/205608 objects degraded (0.025%); 18409/205608 objects misplaced (8.953%); 17 MiB/s, 269 objects/s recovering 2024-09-02T10:03:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:48 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 51/205608 objects degraded (0.025%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:03:48.596 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:03:48.596 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:03:48.628 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 111 (0%) 2024-09-02T10:03:48.629 INFO:teuthology.orchestra.run.smithi151.stdout:avg 10.0909 2024-09-02T10:03:48.629 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.1513 -> 2.1513 (expected baseline 3.02879) 2024-09-02T10:03:48.629 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.9 with 6 -> 6 pgs (0.594595 -> 0.594595 * mean) 2024-09-02T10:03:48.629 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.11 with 13 -> 13 pgs (1.28829 -> 1.28829 * mean) 2024-09-02T10:03:48.629 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:03:48.629 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-09-02T10:03:48.629 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-09-02T10:03:48.629 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:03:48.629 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0132 2024-09-02T10:03:48.629 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.1324 2024-09-02T10:03:48.630 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:03:48.634 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:03:48.634 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:03:48.669 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:03:48.670 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:03:48.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:48.708 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:03:48.708 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:03:48.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:48.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:48.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:48.742 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:03:48.742 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:03:48.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:03:48.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:03:48.812 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:03:48.812 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:03:48.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:48.848 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:03:48.848 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:03:48.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:48.852 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:48.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:48.886 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:03:48.886 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:03:48.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:48.980 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:48.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:48.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:49.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:49.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:49.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:49.213 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:49.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:49.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:49.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:49.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:49.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:49.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:49.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:49.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:49.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4130640865' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:49 smithi187 ceph-mon[20072]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:49 smithi187 ceph-mon[20072]: osdmap e159: 12 total, 12 up, 11 in 2024-09-02T10:03:49.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:49.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4130640865' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:49.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:49 smithi162 ceph-mon[20279]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:49.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:49 smithi162 ceph-mon[20279]: osdmap e159: 12 total, 12 up, 11 in 2024-09-02T10:03:49.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:49.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:49.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:49.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4130640865' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:03:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:49 smithi151 ceph-mon[16639]: from='client.15024 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:03:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:49 smithi151 ceph-mon[16639]: osdmap e159: 12 total, 12 up, 11 in 2024-09-02T10:03:49.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:49.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:49.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:49.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:49.890 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:49.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:49.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:50.014 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:50.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:50.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:50.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:50.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:50.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:50.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:50.281 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:50.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:50.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:50.383 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:50.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:50.405 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:03:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:03:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:50 smithi187 ceph-mon[20072]: pgmap v424: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 3 active+recovering+undersized+remapped, 31 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 511 B/s rd, 16 KiB/s wr, 451 op/s; 51/204549 objects degraded (0.025%); 18083/204549 objects misplaced (8.840%); 21 MiB/s, 327 objects/s recovering 2024-09-02T10:03:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:50 smithi187 ceph-mon[20072]: osdmap e160: 12 total, 12 up, 11 in 2024-09-02T10:03:50.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:50.531 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:50.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:50.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:03:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:03:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:50 smithi162 ceph-mon[20279]: pgmap v424: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 3 active+recovering+undersized+remapped, 31 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 511 B/s rd, 16 KiB/s wr, 451 op/s; 51/204549 objects degraded (0.025%); 18083/204549 objects misplaced (8.840%); 21 MiB/s, 327 objects/s recovering 2024-09-02T10:03:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:50 smithi162 ceph-mon[20279]: osdmap e160: 12 total, 12 up, 11 in 2024-09-02T10:03:50.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:50.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:50.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:50.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:50.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:03:50.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:03:50.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:50 smithi151 ceph-mon[16639]: pgmap v424: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 3 active+recovering+undersized+remapped, 31 active+clean; 4.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 511 B/s rd, 16 KiB/s wr, 451 op/s; 51/204549 objects degraded (0.025%); 18083/204549 objects misplaced (8.840%); 21 MiB/s, 327 objects/s recovering 2024-09-02T10:03:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:50 smithi151 ceph-mon[16639]: osdmap e160: 12 total, 12 up, 11 in 2024-09-02T10:03:50.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:50.800 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:50.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:50.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:50.930 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:50.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:51.031 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:51.054 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:51.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:51.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:51.182 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:51.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:51.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:51.308 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:51.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:51.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:51.433 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:51.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:51.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:51.559 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:51.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:51.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:51.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:51.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:51.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:51.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:51.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:51.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:51.938 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:51.962 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:51.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:52.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:52.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:52.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:52.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:52.229 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:52.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:52.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:52.359 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:52.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:52.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:52.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:52.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:52.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:52.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:52 smithi187 ceph-mon[20072]: pgmap v426: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 30 active+clean; 4.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 420 B/s rd, 14 KiB/s wr, 426 op/s; 191/202428 objects degraded (0.094%); 28207/202428 objects misplaced (13.934%); 15 MiB/s, 236 objects/s recovering 2024-09-02T10:03:52.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:52.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:52 smithi162 ceph-mon[20279]: pgmap v426: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 30 active+clean; 4.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 420 B/s rd, 14 KiB/s wr, 426 op/s; 191/202428 objects degraded (0.094%); 28207/202428 objects misplaced (13.934%); 15 MiB/s, 236 objects/s recovering 2024-09-02T10:03:52.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:52.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:52 smithi151 ceph-mon[16639]: pgmap v426: 37 pgs: 3 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 3 active+recovering+undersized+remapped, 30 active+clean; 4.1 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 420 B/s rd, 14 KiB/s wr, 426 op/s; 191/202428 objects degraded (0.094%); 28207/202428 objects misplaced (13.934%); 15 MiB/s, 236 objects/s recovering 2024-09-02T10:03:52.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:52.751 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:52.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:52.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:52.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:52.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:52.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:53.011 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:53.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:53.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:53.139 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:53.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:53.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:53.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:53.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:53.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:53.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:53.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:53.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:53.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:53.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:53.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:53.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:53.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:53 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 228/197739 objects degraded (0.115%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:03:53.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:53.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:53.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:53.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:53.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:53 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 228/197739 objects degraded (0.115%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:03:53.754 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:53 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 228/197739 objects degraded (0.115%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:03:53.777 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:53.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:53.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:53.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:53.907 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:53.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:53.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:54.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:54.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:54.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:54.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:54.166 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:54.171 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:54.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:54.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:54.190 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:03:54.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:54.301 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:54.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:54.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:54.432 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:54.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:54.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:54 smithi151 ceph-mon[16639]: pgmap v427: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 436 op/s; 228/197739 objects degraded (0.115%); 31602/197739 objects misplaced (15.982%); 14 MiB/s, 218 objects/s recovering 2024-09-02T10:03:54.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:54 smithi151 ceph-mon[16639]: osdmap e161: 12 total, 12 up, 11 in 2024-09-02T10:03:54.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:54.567 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:54.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:54 smithi162 ceph-mon[20279]: pgmap v427: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 436 op/s; 228/197739 objects degraded (0.115%); 31602/197739 objects misplaced (15.982%); 14 MiB/s, 218 objects/s recovering 2024-09-02T10:03:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:54 smithi162 ceph-mon[20279]: osdmap e161: 12 total, 12 up, 11 in 2024-09-02T10:03:54.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:54.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:54.694 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:54 smithi187 ceph-mon[20072]: pgmap v427: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 436 op/s; 228/197739 objects degraded (0.115%); 31602/197739 objects misplaced (15.982%); 14 MiB/s, 218 objects/s recovering 2024-09-02T10:03:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:54 smithi187 ceph-mon[20072]: osdmap e161: 12 total, 12 up, 11 in 2024-09-02T10:03:54.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:54.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:54.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:54.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:54.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:54.954 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:54.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:55.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:55.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:55.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:55.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:55.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:55.213 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:55.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:55.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:55.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:55.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:55.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:55.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:55.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:55.589 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:55.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:55.590 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:55.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:55.719 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:55.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:55.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:55.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:55.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:55.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:03:55.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:55.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:03:56.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:56.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:56.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:56.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:56.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:56.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:56.242 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:56.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:56.373 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:56.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:56.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:56.499 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:56.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:56.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:56.551 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:56 smithi151 ceph-mon[16639]: pgmap v429: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 459 op/s; 243/195834 objects degraded (0.124%); 31417/195834 objects misplaced (16.043%); 11 MiB/s, 182 objects/s recovering 2024-09-02T10:03:56.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:56 smithi162 ceph-mon[20279]: pgmap v429: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 459 op/s; 243/195834 objects degraded (0.124%); 31417/195834 objects misplaced (16.043%); 11 MiB/s, 182 objects/s recovering 2024-09-02T10:03:56.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:56.625 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:56.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:56.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:56.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:56.754 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:56.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:56.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:56 smithi187 ceph-mon[20072]: pgmap v429: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 4.0 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 459 op/s; 243/195834 objects degraded (0.124%); 31417/195834 objects misplaced (16.043%); 11 MiB/s, 182 objects/s recovering 2024-09-02T10:03:56.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:03:56.882 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:56.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:03:56.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:57.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:57.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:57.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:57.095 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:03:57.096 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:03:57.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:57.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:57.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:57.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:57.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:57.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:57.306 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:03:57.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:57.412 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:57.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:57.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:03:57.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:57.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:03:57.558 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2042436913' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:03:57.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2042436913' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:03:57.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:57.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:57.663 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2042436913' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:03:57.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:57.786 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:57.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:57.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:57.915 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:57.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:57.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:58.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:58.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:58.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:58.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:03:58.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:03:58.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:03:58.179 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:58.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:03:58.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:03:58.304 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:58.305 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:03:58.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:58.433 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:58.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:58 smithi151 ceph-mon[16639]: pgmap v430: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 485 op/s; 396/192879 objects degraded (0.205%); 30532/192879 objects misplaced (15.830%); 14 MiB/s, 220 objects/s recovering 2024-09-02T10:03:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:58 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2042436913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:03:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:58 smithi151 ceph-mon[16639]: osdmap e162: 12 total, 12 up, 11 in 2024-09-02T10:03:58.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:58.565 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:58.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:58.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:58 smithi162 ceph-mon[20279]: pgmap v430: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 485 op/s; 396/192879 objects degraded (0.205%); 30532/192879 objects misplaced (15.830%); 14 MiB/s, 220 objects/s recovering 2024-09-02T10:03:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:58 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2042436913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:03:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:58 smithi162 ceph-mon[20279]: osdmap e162: 12 total, 12 up, 11 in 2024-09-02T10:03:58.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:58.695 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:58.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:58 smithi187 ceph-mon[20072]: pgmap v430: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 485 op/s; 396/192879 objects degraded (0.205%); 30532/192879 objects misplaced (15.830%); 14 MiB/s, 220 objects/s recovering 2024-09-02T10:03:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:58 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:03:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2042436913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:03:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:58 smithi187 ceph-mon[20072]: osdmap e162: 12 total, 12 up, 11 in 2024-09-02T10:03:58.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:58.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:58.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:03:58.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:03:58.959 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:58.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:58.960 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:03:59.061 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:03:59.088 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:59.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:59.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:03:59.141 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:03:59.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-09-02T10:03:59.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:03:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:59.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:59.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:03:59.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:59.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:59.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 396/192879 objects degraded (0.205%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:03:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:03:59 smithi151 ceph-mon[16639]: osdmap e163: 12 total, 12 up, 11 in 2024-09-02T10:03:59.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:03:59.502 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:03:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:59.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:03:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 396/192879 objects degraded (0.205%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:03:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:03:59 smithi162 ceph-mon[20279]: osdmap e163: 12 total, 12 up, 11 in 2024-09-02T10:03:59.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:03:59.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:03:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:59.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:03:59.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:03:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 396/192879 objects degraded (0.205%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:03:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:03:59 smithi187 ceph-mon[20072]: osdmap e163: 12 total, 12 up, 11 in 2024-09-02T10:03:59.757 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:59.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:03:59.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:03:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:03:59.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:03:59.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:00.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:00.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:00.031 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:00.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:00.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:00.157 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:00.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:00.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:00.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:00.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:00.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:00.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:00.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:00.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:00.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:00.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:00.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:00 smithi162 ceph-mon[20279]: pgmap v433: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 397 op/s; 404/190584 objects degraded (0.212%); 29795/190584 objects misplaced (15.634%); 14 MiB/s, 222 objects/s recovering 2024-09-02T10:04:00.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:00.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:00 smithi151 ceph-mon[16639]: pgmap v433: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 397 op/s; 404/190584 objects degraded (0.212%); 29795/190584 objects misplaced (15.634%); 14 MiB/s, 222 objects/s recovering 2024-09-02T10:04:00.713 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:00.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:00 smithi187 ceph-mon[20072]: pgmap v433: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.9 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 397 op/s; 404/190584 objects degraded (0.212%); 29795/190584 objects misplaced (15.634%); 14 MiB/s, 222 objects/s recovering 2024-09-02T10:04:00.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:00.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:00.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:00.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:00.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:01.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:01.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:01.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:01.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:01.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:01.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:01.235 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:01.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:01.361 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:01.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:01.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:01.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:01.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:01.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:01.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:01.571 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:01.572 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:01.589 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:01.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:01.612 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:01.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:01.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:01.741 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:01.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:01.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:01.872 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:01.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:01.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:01.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:02.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:02.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:02.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:02.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:02.148 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:02.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:02.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:02.276 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:02.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:02.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:02.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:02.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:02.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:02.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:02.531 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:02.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:02.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:02 smithi162 ceph-mon[20279]: pgmap v434: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 452 op/s; 565/187797 objects degraded (0.301%); 28984/187797 objects misplaced (15.434%); 16 MiB/s, 254 objects/s recovering 2024-09-02T10:04:02.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:02.660 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:02.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:02.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:02 smithi151 ceph-mon[16639]: pgmap v434: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 452 op/s; 565/187797 objects degraded (0.301%); 28984/187797 objects misplaced (15.434%); 16 MiB/s, 254 objects/s recovering 2024-09-02T10:04:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:02 smithi187 ceph-mon[20072]: pgmap v434: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.8 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 452 op/s; 565/187797 objects degraded (0.301%); 28984/187797 objects misplaced (15.434%); 16 MiB/s, 254 objects/s recovering 2024-09-02T10:04:02.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:02.785 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:02.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:02.786 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:02.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:02.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:02.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:02.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:03.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:03.052 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:03.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:03.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:03.181 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:03.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:03.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:03.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:03.314 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:03.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:03.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:03.441 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:03.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:03.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:03.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:04:03.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:03.567 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:03.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:03.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:03.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:03.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:03.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:03.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:03.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:03.942 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:03.964 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:03.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:03.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:04.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:04.092 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:04.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:04.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:04.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:04.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:04.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:04.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:04.350 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:04.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:04.452 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:04.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:04.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:04.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:04.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:04 smithi162 ceph-mon[20279]: pgmap v435: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 542 op/s; 564/182811 objects degraded (0.309%); 28042/182811 objects misplaced (15.339%); 19 MiB/s, 308 objects/s recovering 2024-09-02T10:04:04.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 564/182811 objects degraded (0.309%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:04.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:04.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:04.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:04 smithi151 ceph-mon[16639]: pgmap v435: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 542 op/s; 564/182811 objects degraded (0.309%); 28042/182811 objects misplaced (15.339%); 19 MiB/s, 308 objects/s recovering 2024-09-02T10:04:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 564/182811 objects degraded (0.309%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:04.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:04.732 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:04.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:04.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:04 smithi187 ceph-mon[20072]: pgmap v435: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 542 op/s; 564/182811 objects degraded (0.309%); 28042/182811 objects misplaced (15.339%); 19 MiB/s, 308 objects/s recovering 2024-09-02T10:04:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 564/182811 objects degraded (0.309%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:04.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:04.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:04.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:04.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:05.025 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:05.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:05.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:05.152 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:05.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:05.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:05.279 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:05.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:05.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:05.381 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:05.408 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:05.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:05.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:05.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:05.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:05.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:05 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:04:05.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:05.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:05.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:05.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:05 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:04:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:05 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:04:05.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:05.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:05.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:05.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:05.942 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:05.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:06.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:06.068 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:06.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:06.070 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:06.171 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:06.195 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:06.196 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:06.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:06.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:06.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:06.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:06.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:06.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:06.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:06.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:06.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:06 smithi162 ceph-mon[20279]: pgmap v436: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 564/180882 objects degraded (0.312%); 27840/180882 objects misplaced (15.391%); 16 MiB/s, 257 objects/s recovering 2024-09-02T10:04:06.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:06 smithi162 ceph-mon[20279]: osdmap e164: 12 total, 12 up, 11 in 2024-09-02T10:04:06.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:06.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:06.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:06 smithi151 ceph-mon[16639]: pgmap v436: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 564/180882 objects degraded (0.312%); 27840/180882 objects misplaced (15.391%); 16 MiB/s, 257 objects/s recovering 2024-09-02T10:04:06.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:06.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:06.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:06 smithi151 ceph-mon[16639]: osdmap e164: 12 total, 12 up, 11 in 2024-09-02T10:04:06.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:06.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:06.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:06 smithi187 ceph-mon[20072]: pgmap v436: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 564/180882 objects degraded (0.312%); 27840/180882 objects misplaced (15.391%); 16 MiB/s, 257 objects/s recovering 2024-09-02T10:04:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:06 smithi187 ceph-mon[20072]: osdmap e164: 12 total, 12 up, 11 in 2024-09-02T10:04:06.837 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:06.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:06.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:06.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:06.992 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:06.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:07.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:07.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:07.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:07.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:07.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:07.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:07.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:07.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:07.373 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:07.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:07.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:07.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:07.502 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:07.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:07.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:07.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4165834146' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:07.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:07.634 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:07.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:07.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4165834146' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:07.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4165834146' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:07.769 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:07.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:07.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:07.894 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:07.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:07.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:07.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:08.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:08.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:08.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:08.157 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:08.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:08.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:08.282 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:08.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:08.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:08.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:08.407 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:08.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:08.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:08.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:08.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:08.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:08 smithi162 ceph-mon[20279]: pgmap v438: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 526 op/s; 697/177459 objects degraded (0.393%); 26963/177459 objects misplaced (15.194%); 16 MiB/s, 254 objects/s recovering 2024-09-02T10:04:08.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:08.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:08.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:08.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:08.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:08 smithi151 ceph-mon[16639]: pgmap v438: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 526 op/s; 697/177459 objects degraded (0.393%); 26963/177459 objects misplaced (15.194%); 16 MiB/s, 254 objects/s recovering 2024-09-02T10:04:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:08 smithi187 ceph-mon[20072]: pgmap v438: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 526 op/s; 697/177459 objects degraded (0.393%); 26963/177459 objects misplaced (15.194%); 16 MiB/s, 254 objects/s recovering 2024-09-02T10:04:08.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:08.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:08.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:08.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:08.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:08.970 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:08.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:09.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:09.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:09.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:09.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:09.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:09.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:09.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:09.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:09.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:09.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:09.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:09.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:09.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:09.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:09.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 697/177459 objects degraded (0.393%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:09.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:09 smithi162 ceph-mon[20279]: osdmap e165: 12 total, 12 up, 11 in 2024-09-02T10:04:09.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:09.659 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:09.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:09.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 697/177459 objects degraded (0.393%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:09.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:09 smithi151 ceph-mon[16639]: osdmap e165: 12 total, 12 up, 11 in 2024-09-02T10:04:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 697/177459 objects degraded (0.393%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:09 smithi187 ceph-mon[20072]: osdmap e165: 12 total, 12 up, 11 in 2024-09-02T10:04:09.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:09.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:09.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:09.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:09.922 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:09.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:10.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:10.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:10.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:10.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:10.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:10.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:10.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:10.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:10.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:10.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:10.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:10.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:10.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:10.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:10.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:10.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:10.571 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:10.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:10.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:10 smithi162 ceph-mon[20279]: pgmap v440: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 524 op/s; 697/175215 objects degraded (0.398%); 26211/175215 objects misplaced (14.959%); 17 MiB/s, 271 objects/s recovering 2024-09-02T10:04:10.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:04:10.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:04:10.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:10.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:10.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:10 smithi151 ceph-mon[16639]: pgmap v440: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 524 op/s; 697/175215 objects degraded (0.398%); 26211/175215 objects misplaced (14.959%); 17 MiB/s, 271 objects/s recovering 2024-09-02T10:04:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:04:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:04:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:10.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:10.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:10.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:10 smithi187 ceph-mon[20072]: pgmap v440: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 524 op/s; 697/175215 objects degraded (0.398%); 26211/175215 objects misplaced (14.959%); 17 MiB/s, 271 objects/s recovering 2024-09-02T10:04:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:04:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:04:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:10.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:10.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:10.826 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:10.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:10.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:10.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:10.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:11.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:11.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:11.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:11.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:04:11.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:04:11.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:04:11.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:04:11.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:04:11.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:04:11.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:04:11.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:04:11.156 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:04:11.156 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:04:11.156 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:04:11.156 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:04:11.156 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:04:11.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:11.234 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:11.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:11.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:11.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:11.364 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:11.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:11.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:11.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:11.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:11.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:11.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:11.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:11.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:11.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:11.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:11.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:11.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:11.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:11.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:11.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:11.994 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:11.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:11.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:12.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:12.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:12.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:12.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:12.245 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:12.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:12.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:12.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:12.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:12.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:12.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:12.503 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:12.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:12.529 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:12 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:12.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:12 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:12.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:12 smithi151 ceph-mon[16639]: pgmap v441: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 438 op/s; 870/172278 objects degraded (0.505%); 25407/172278 objects misplaced (14.748%); 15 MiB/s, 237 objects/s recovering 2024-09-02T10:04:12.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:12 smithi151 ceph-mon[16639]: osdmap e166: 12 total, 12 up, 11 in 2024-09-02T10:04:12.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:12.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:12.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:12.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:12 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:12 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:12 smithi187 ceph-mon[20072]: pgmap v441: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 438 op/s; 870/172278 objects degraded (0.505%); 25407/172278 objects misplaced (14.748%); 15 MiB/s, 237 objects/s recovering 2024-09-02T10:04:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:12 smithi187 ceph-mon[20072]: osdmap e166: 12 total, 12 up, 11 in 2024-09-02T10:04:12.764 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:04:12.764 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:04:12.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:04:12.764 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:04:12.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.764 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:9163 2.9 e98/92 2:91551e63:::benchmark_data_smithi187_45882_object9162:head v 98'571, mlcod=98'569)", 2024-09-02T10:04:12.764 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.622884+0000", 2024-09-02T10:04:12.764 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 152.089161499, 2024-09-02T10:04:12.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16134657399999999, 2024-09-02T10:04:12.764 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.622884+0000", 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.622884+0000", 2024-09-02T10:04:12.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.622884+0000", 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2900000000000003e-07 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.622903+0000", 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8499999999999999e-05 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.783619+0000", 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.160716573 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.783626+0000", 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1349999999999997e-06 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.783675+0000", 2024-09-02T10:04:12.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9777000000000001e-05 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.783696+0000", 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0492e-05 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.784199+0000", 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00050298499999999996 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.784230+0000", 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1483000000000003e-05 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:13541 2.1 e98/92 2:80f77a31:::benchmark_data_smithi187_45882_object13540:head v 98'890, mlcod=98'888)", 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:44.030729+0000", 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.68131600199999, 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.131562753, 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030729+0000", 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030729+0000", 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030731+0000", 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.341e-06 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.030757+0000", 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6631e-05 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.161433+0000", 2024-09-02T10:04:12.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13067601100000001 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.161444+0000", 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0509999999999999e-05 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.161506+0000", 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2144000000000007e-05 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.161528+0000", 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2215999999999999e-05 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.162241+0000", 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00071297199999999998 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:44.162292+0000", 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0927999999999998e-05 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.773 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:24661 2.d e104/92 2:b7cc3c46:::benchmark_data_smithi187_45882_object24660:head v 104'1504, mlcod=104'1502)", 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:52.676309+0000", 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 140.03573591899999, 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15761450699999999, 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:52.676309+0000", 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:52.676309+0000", 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:52.676311+0000", 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2160000000000001e-06 2024-09-02T10:04:12.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:52.676394+0000", 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3082000000000002e-05 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:52.833115+0000", 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.156721903 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:52.833122+0000", 2024-09-02T10:04:12.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.421e-06 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:52.833167+0000", 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5300999999999999e-05 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:52.833180+0000", 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3236e-05 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:52.833894+0000", 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00071346699999999996 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:52.833924+0000", 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9881000000000001e-05 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:37178 2.1 e110/92 2:8a9d5e65:::benchmark_data_smithi187_45882_object37177:head v 110'2322, mlcod=110'2320)", 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:02.876876+0000", 2024-09-02T10:04:12.778 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 129.83516883999999, 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.123823473, 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876876+0000", 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876876+0000", 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876877+0000", 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.876897+0000", 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.018e-05 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.999472+0000", 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.122574999 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.999480+0000", 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3900000000000004e-06 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.999536+0000", 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6709000000000002e-05 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.999553+0000", 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6748999999999998e-05 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.000666+0000", 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011126689999999999 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:03.000700+0000", 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3933999999999997e-05 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:40750 2.d e111/92 2:bbc92d25:::benchmark_data_smithi187_45882_object40749:head v 111'2490, mlcod=111'2486)", 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:05.734877+0000", 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 126.977167869, 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15769079999999999, 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.734877+0000", 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.734877+0000", 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.734878+0000", 2024-09-02T10:04:12.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9299999999999997e-07 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.734977+0000", 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8468999999999995e-05 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.891422+0000", 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15644538699999999 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.891427+0000", 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3530000000000002e-06 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.891477+0000", 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9985000000000003e-05 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.891500+0000", 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2911000000000001e-05 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.892538+0000", 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010375569999999999 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.892568+0000", 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0445000000000001e-05 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54904 2.6 e113/92 2:67a80d93:::benchmark_data_smithi187_45882_object54903:head v 113'3377, mlcod=113'3374)", 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.130975+0000", 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 115.58107044899999, 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.124364958, 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130975+0000", 2024-09-02T10:04:12.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130975+0000", 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130976+0000", 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.048e-06 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130999+0000", 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3119999999999999e-05 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.253795+0000", 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12279625 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.253802+0000", 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0520000000000004e-06 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.253859+0000", 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6335999999999997e-05 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.253877+0000", 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8660000000000001e-05 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.255312+0000", 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001434774 2024-09-02T10:04:12.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.255340+0000", 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7718000000000001e-05 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54922 2.a e113/92 2:5831fc15:::benchmark_data_smithi187_45882_object54921:head v 113'3408, mlcod=113'3405)", 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.148929+0000", 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 115.56311593700001, 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.239248567, 2024-09-02T10:04:12.792 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.148929+0000", 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.148929+0000", 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.148930+0000", 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2299999999999999e-07 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.148973+0000", 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3205999999999999e-05 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.386786+0000", 2024-09-02T10:04:12.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23781280199999999 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.386793+0000", 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6540000000000002e-06 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.386838+0000", 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4875e-05 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.386850+0000", 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2554999999999999e-05 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.388125+0000", 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012751629999999999 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.388178+0000", 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2389000000000003e-05 2024-09-02T10:04:12.796 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54924 2.6 e113/92 2:6f9b4ae3:::benchmark_data_smithi187_45882_object54923:head v 113'3378, mlcod=113'3374)", 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.151288+0000", 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 115.560756883, 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22697457500000001, 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.151288+0000", 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.151288+0000", 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.151289+0000", 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0400000000000005e-07 2024-09-02T10:04:12.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.151307+0000", 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8362000000000001e-05 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.376689+0000", 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.225381937 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.376697+0000", 2024-09-02T10:04:12.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3100000000000003e-06 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.376761+0000", 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4154999999999998e-05 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.376775+0000", 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3959e-05 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.378233+0000", 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014579580000000001 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.378263+0000", 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0090000000000002e-05 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54952 2.6 e113/92 2:6373c65b:::benchmark_data_smithi187_45882_object54951:head v 113'3379, mlcod=113'3375)", 2024-09-02T10:04:12.801 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.253759+0000", 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 115.458285862, 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.126782591, 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.253759+0000", 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.253759+0000", 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.253760+0000", 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8800000000000001e-07 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.253780+0000", 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9361999999999999e-05 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.379362+0000", 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12558259899999999 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.379367+0000", 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6750000000000001e-06 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.379409+0000", 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2586999999999997e-05 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.379423+0000", 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3286e-05 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.380512+0000", 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010897389999999999 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.380542+0000", 2024-09-02T10:04:12.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9454999999999998e-05 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54953 2.e e113/92 2:75418cd7:::benchmark_data_smithi187_45882_object54952:head v 113'3471, mlcod=113'3469)", 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.256038+0000", 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 115.456006739, 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.127361797, 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.256038+0000", 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.256038+0000", 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.256039+0000", 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-02T10:04:12.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.256062+0000", 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2365999999999999e-05 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.381521+0000", 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.125459766 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.381527+0000", 2024-09-02T10:04:12.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6230000000000002e-06 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.381577+0000", 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9836000000000002e-05 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.381597+0000", 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0041e-05 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.383357+0000", 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017603919999999999 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.383400+0000", 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3013000000000003e-05 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:55073 2.a e113/92 2:5320fa11:::benchmark_data_smithi187_45882_object55072:head v 113'3418, mlcod=113'3415)", 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.493334+0000", 2024-09-02T10:04:12.810 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 115.218711193, 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16270283999999999, 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.493334+0000", 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.493334+0000", 2024-09-02T10:04:12.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.493335+0000", 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4899999999999995e-07 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.493389+0000", 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3888999999999999e-05 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.654560+0000", 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16117157500000001 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.654566+0000", 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2060000000000004e-06 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.654613+0000", 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6684000000000002e-05 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.654628+0000", 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5041e-05 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.655993+0000", 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013652600000000001 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.656037+0000", 2024-09-02T10:04:12.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3535999999999998e-05 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:55079 2.1 e113/92 2:8bad8344:::benchmark_data_smithi187_45882_object55078:head v 113'3460, mlcod=113'3457)", 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.502809+0000", 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 115.209236354, 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16137133400000001, 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502809+0000", 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502809+0000", 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502810+0000", 2024-09-02T10:04:12.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5000000000000002e-07 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.502832+0000", 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2269999999999999e-05 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.662625+0000", 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15979274600000001 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.662630+0000", 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9289999999999997e-06 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.662682+0000", 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2225000000000002e-05 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.662709+0000", 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6784e-05 2024-09-02T10:04:12.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.664147+0000", 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001438876 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.664180+0000", 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2753999999999998e-05 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:55087 2.9 e113/92 2:9c3a08ac:::benchmark_data_smithi187_45882_object55086:head v 113'3460, mlcod=113'3458)", 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.521418+0000", 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 115.190627127, 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23942591599999999, 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521418+0000", 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521418+0000", 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521419+0000", 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0440000000000001e-06 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521444+0000", 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5020000000000001e-05 2024-09-02T10:04:12.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.759031+0000", 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23758688 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.759038+0000", 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5599999999999999e-06 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.759094+0000", 2024-09-02T10:04:12.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6855999999999999e-05 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.759110+0000", 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5590000000000002e-05 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.760813+0000", 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017026610000000001 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.760844+0000", 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1304999999999999e-05 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:60015 2.d e114/92 2:b3a970a5:::benchmark_data_smithi187_45882_object60014:head v 114'3676, mlcod=114'3673)", 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:21.757710+0000", 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 110.954334992, 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12250931399999999, 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.757710+0000", 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.757710+0000", 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.757711+0000", 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6700000000000002e-07 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.757791+0000", 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0323999999999999e-05 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.875740+0000", 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.117948314 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.875746+0000", 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4749999999999998e-06 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.875819+0000", 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2982e-05 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.875838+0000", 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8950999999999999e-05 2024-09-02T10:04:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.880178+0000", 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043396149999999998 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:21.880220+0000", 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1786000000000003e-05 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:98339 2.a e137/92 2:583e6512:::benchmark_data_smithi187_45882_object588:head v 137'6165, mlcod=137'6162)", 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:07.125927+0000", 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 65.586118216000003, 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.155708289, 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:07.125927+0000", 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:07.125927+0000", 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:07.125928+0000", 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1999999999999998e-07 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:07.125934+0000", 2024-09-02T10:04:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.443e-06 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:07.278673+0000", 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.152738342 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:07.278681+0000", 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4959999999999996e-06 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:07.278742+0000", 2024-09-02T10:04:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0563000000000003e-05 2024-09-02T10:04:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:07.278802+0000", 2024-09-02T10:04:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0831000000000001e-05 2024-09-02T10:04:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:07.281597+0000", 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027941379999999998 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:07.281635+0000", 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8655999999999997e-05 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:116179 2.6 e146/92 2:61ccb785:::benchmark_data_smithi187_45882_object18428:head v 146'7225, mlcod=146'7223)", 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:28.480769+0000", 2024-09-02T10:04:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 44.231276547999997, 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13199582000000001, 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:28.480769+0000", 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:28.480769+0000", 2024-09-02T10:04:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:28.480769+0000", 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:28.480775+0000", 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4469999999999997e-06 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:28.609199+0000", 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12842387899999999 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:28.609205+0000", 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8809999999999998e-06 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:28.609267+0000", 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1247000000000005e-05 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:28.609286+0000", 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9823999999999999e-05 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:28.612740+0000", 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034536329999999998 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:28.612764+0000", 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.438e-05 2024-09-02T10:04:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:126969 2.1 e155/92 2:8fa8ba5c:::benchmark_data_smithi187_45882_object29218:head v 155'7965, mlcod=155'7962)", 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:44.478380+0000", 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.233664952000002, 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14886056, 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.478380+0000", 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.478380+0000", 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.478381+0000", 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3300000000000001e-07 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.478387+0000", 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5149999999999997e-06 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.623112+0000", 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.144725676 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.623120+0000", 2024-09-02T10:04:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8679999999999999e-06 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.623182+0000", 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2397999999999994e-05 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.623202+0000", 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9690999999999998e-05 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.627160+0000", 2024-09-02T10:04:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039583450000000003 2024-09-02T10:04:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.627241+0000", 2024-09-02T10:04:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0333999999999994e-05 2024-09-02T10:04:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:138527 2.d e165/157 2:bbdd4c82:::benchmark_data_smithi187_45882_object40776:head v 165'8432, mlcod=111'2621)", 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:09.017634+0000", 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.6944112200000001, 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17373502499999999, 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:09.017634+0000", 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:09.017634+0000", 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:09.017635+0000", 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2610000000000001e-06 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:09.017643+0000", 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4560000000000003e-06 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:09.187106+0000", 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16946349899999999 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:09.187113+0000", 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6000000000000003e-06 2024-09-02T10:04:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:09.187166+0000", 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2914999999999997e-05 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:09.187190+0000", 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4008000000000001e-05 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:09.191325+0000", 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041351649999999997 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:09.191369+0000", 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4121e-05 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(2.7 165/131 [PushOp(2:e4762eaf:::benchmark_data_smithi187_45882_object67216:head, version: 118'4226, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:e4762eaf:::benchmark_data_smithi187_45882_object67216:head@118'4226, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:04:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:10.766134+0000", 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.945911317, 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20040254399999999, 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.766134+0000", 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.766134+0000", 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.766135+0000", 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3289999999999999e-06 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.766176+0000", 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0654000000000001e-05 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.961997+0000", 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19582065500000001 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.962015+0000", 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8457999999999998e-05 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.962230+0000", 2024-09-02T10:04:12.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021529900000000001 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.962243+0000", 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2369e-05 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.966536+0000", 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042937799999999996 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.851 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:139550 2.9 e165/92 2:9beffd54:::benchmark_data_smithi187_45882_object41799:head v 165'8761, mlcod=165'8759)", 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:10.808477+0000", 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.903567714, 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.136469864, 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:04:12.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.808477+0000", 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.808477+0000", 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.808478+0000", 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8899999999999999e-07 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.808484+0000", 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3709999999999999e-06 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.940345+0000", 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13186102599999999 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.940353+0000", 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4579999999999993e-06 2024-09-02T10:04:12.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.940454+0000", 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100974 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.940474+0000", 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0285e-05 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.944898+0000", 2024-09-02T10:04:12.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044236079999999999 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:10.944947+0000", 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9453000000000002e-05 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:04:12.856 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:04:12.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:12.858 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:12.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:12.858 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:12 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:12.858 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:12 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:12.858 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:12 smithi162 ceph-mon[20279]: pgmap v441: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 438 op/s; 870/172278 objects degraded (0.505%); 25407/172278 objects misplaced (14.748%); 15 MiB/s, 237 objects/s recovering 2024-09-02T10:04:12.858 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:12 smithi162 ceph-mon[20279]: osdmap e166: 12 total, 12 up, 11 in 2024-09-02T10:04:12.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:12.985 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:12.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:13.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:13.109 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:13.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:13.117 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:13.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:13.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:13.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:13.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:13.360 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:13.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:13.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:13.366 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-09-02T10:04:13.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:13.491 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:13.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:13.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:13.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:13.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:13.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:13 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:13.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:13 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:13.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:13 smithi151 ceph-mon[16639]: osdmap e167: 12 total, 12 up, 11 in 2024-09-02T10:04:13.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2173218165' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:04:13.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:13.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:04:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:13 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:13 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:13 smithi187 ceph-mon[20072]: osdmap e167: 12 total, 12 up, 11 in 2024-09-02T10:04:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2173218165' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:04:13.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:13.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:13.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:13.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:13 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:13.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:13 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:13.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:13 smithi162 ceph-mon[20279]: osdmap e167: 12 total, 12 up, 11 in 2024-09-02T10:04:13.785 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2173218165' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:04:13.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:13.897 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:13.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:13.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:13.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:14.030 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:14.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:14.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:14.103 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:14.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:14.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:14.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:14.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:14.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:14.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:14.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:14.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:14.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:14.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:14.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:14 smithi162 ceph-mon[20279]: pgmap v444: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 870/167208 objects degraded (0.520%); 24312/167208 objects misplaced (14.540%); 21 MiB/s, 334 objects/s recovering 2024-09-02T10:04:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2173218165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:04:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:14 smithi162 ceph-mon[20279]: osdmap e168: 12 total, 12 up, 11 in 2024-09-02T10:04:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 870/167208 objects degraded (0.520%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:14 smithi162 ceph-mon[20279]: osdmap e169: 12 total, 12 up, 11 in 2024-09-02T10:04:14.597 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:14.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:14 smithi151 ceph-mon[16639]: pgmap v444: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 870/167208 objects degraded (0.520%); 24312/167208 objects misplaced (14.540%); 21 MiB/s, 334 objects/s recovering 2024-09-02T10:04:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2173218165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:04:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:14 smithi151 ceph-mon[16639]: osdmap e168: 12 total, 12 up, 11 in 2024-09-02T10:04:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 870/167208 objects degraded (0.520%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:14 smithi151 ceph-mon[16639]: osdmap e169: 12 total, 12 up, 11 in 2024-09-02T10:04:14.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:14.722 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:14.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:14 smithi187 ceph-mon[20072]: pgmap v444: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 569 op/s; 870/167208 objects degraded (0.520%); 24312/167208 objects misplaced (14.540%); 21 MiB/s, 334 objects/s recovering 2024-09-02T10:04:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2173218165' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:04:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:14 smithi187 ceph-mon[20072]: osdmap e168: 12 total, 12 up, 11 in 2024-09-02T10:04:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 870/167208 objects degraded (0.520%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:14 smithi187 ceph-mon[20072]: osdmap e169: 12 total, 12 up, 11 in 2024-09-02T10:04:14.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:14.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:14.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:14.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:14.991 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:14.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:15.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-09-02T10:04:15.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:15.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:15.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:15.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:15.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:04:15.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:15.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:15.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:15.379 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:04:15.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:15.416 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:15.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:15.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:15.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:15.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:15.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3895174518' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:15.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:15 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:15.643 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:15.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:15.666 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:15.667 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3895174518' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:15 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:15.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:15.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:15.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:15.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:15.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3895174518' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:15 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:15.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:15.944 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:15.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:15.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:16.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:16.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:16.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:16.183 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:16.209 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:16.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:16.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:16.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:16.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:16.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:16.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:16.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:16.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:16.503 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:16.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:16.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:16.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:16.634 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:16.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:16 smithi151 ceph-mon[16639]: pgmap v447: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 870/165315 objects degraded (0.526%); 24121/165315 objects misplaced (14.591%); 17 MiB/s, 278 objects/s recovering 2024-09-02T10:04:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:16 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:16 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:04:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:16 smithi151 ceph-mon[16639]: osdmap e170: 12 total, 12 up, 11 in 2024-09-02T10:04:16.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:16.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:16 smithi162 ceph-mon[20279]: pgmap v447: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 870/165315 objects degraded (0.526%); 24121/165315 objects misplaced (14.591%); 17 MiB/s, 278 objects/s recovering 2024-09-02T10:04:16.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:16 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:16.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:16 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:04:16.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:16 smithi162 ceph-mon[20279]: osdmap e170: 12 total, 12 up, 11 in 2024-09-02T10:04:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:16 smithi187 ceph-mon[20072]: pgmap v447: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 580 op/s; 870/165315 objects degraded (0.526%); 24121/165315 objects misplaced (14.591%); 17 MiB/s, 278 objects/s recovering 2024-09-02T10:04:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:16 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:16 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:04:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:16 smithi187 ceph-mon[20072]: osdmap e170: 12 total, 12 up, 11 in 2024-09-02T10:04:16.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:16.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:16.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:16.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:16.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:16.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:17.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:17.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:17.039 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:17.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:17.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:17.145 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:04:17.145 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:04:17.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.145 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:142826 2.d 2:b5d63b6c:::benchmark_data_smithi187_45882_object45075:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-02T10:04:17.145 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:17.101781+0000", 2024-09-02T10:04:17.145 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.038642685000000003, 2024-09-02T10:04:17.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.038682844000000001, 2024-09-02T10:04:17.145 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:17.145 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:17.145 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 142826 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.101781+0000", 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.101781+0000", 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.101783+0000", 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.778e-06 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:17.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.101795+0000", 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1691999999999999e-05 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.101812+0000", 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7592999999999999e-05 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.101820+0000", 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9189999999999999e-06 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.109232+0000", 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074120990000000001 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.109563+0000", 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033039099999999997 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.109958+0000", 2024-09-02T10:04:17.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00039490499999999997 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.120085+0000", 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010127343 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.120152+0000", 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7065999999999998e-05 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:17.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:142843 2.d 2:ba114ea3:::benchmark_data_smithi187_45882_object45092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:17.131182+0000", 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00924233, 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0093637860000000007, 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 142843 2024-09-02T10:04:17.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.131182+0000", 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.131182+0000", 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.131183+0000", 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.685e-06 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.131190+0000", 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3389999999999998e-06 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.131202+0000", 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2308e-05 2024-09-02T10:04:17.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.131210+0000", 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7810000000000005e-06 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.137912+0000", 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0067023330000000004 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:17.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.138109+0000", 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000196617 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:17.138258+0000", 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014940799999999999 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-02T10:04:17.155 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:04:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:17.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:17.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:17.291 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:17.327 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:17.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:17.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:17.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:17.455 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:17.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:17.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:17.585 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:17.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:17.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:17.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:17.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:17.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:17.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:17.844 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:17.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:17.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:17.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:17.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:18.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:18.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:18.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:18.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:04:18.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:04:18.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:18.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:18.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:18.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:18.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:18.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:18 smithi151 ceph-mon[16639]: pgmap v449: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 477 op/s; 1016/162000 objects degraded (0.627%); 23088/162000 objects misplaced (14.252%); 16 MiB/s, 256 objects/s recovering 2024-09-02T10:04:18.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:18.509 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:18.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:18.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:18.535 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:18.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:18.634 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:18.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:18.664 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:18 smithi162 ceph-mon[20279]: pgmap v449: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 477 op/s; 1016/162000 objects degraded (0.627%); 23088/162000 objects misplaced (14.252%); 16 MiB/s, 256 objects/s recovering 2024-09-02T10:04:18.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:18 smithi187 ceph-mon[20072]: pgmap v449: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 477 op/s; 1016/162000 objects degraded (0.627%); 23088/162000 objects misplaced (14.252%); 16 MiB/s, 256 objects/s recovering 2024-09-02T10:04:18.765 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:18.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:18.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:18.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:18.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:18.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:19.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:19.024 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:19.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:19.098 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:04:19.099 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:04:19.099 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:04:19.099 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:04:19.099 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:04:19.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:19.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:19.153 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:19.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:19.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:19.286 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:19.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:19.392 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:19.423 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:19.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:19.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:19.551 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:19.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:19.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:19.682 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:19.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:19.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1016/162000 objects degraded (0.627%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:19.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:19 smithi151 ceph-mon[16639]: osdmap e171: 12 total, 12 up, 11 in 2024-09-02T10:04:19.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1016/162000 objects degraded (0.627%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:19.692 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:19 smithi162 ceph-mon[20279]: osdmap e171: 12 total, 12 up, 11 in 2024-09-02T10:04:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1016/162000 objects degraded (0.627%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:19 smithi187 ceph-mon[20072]: osdmap e171: 12 total, 12 up, 11 in 2024-09-02T10:04:19.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:19.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:04:19.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:19.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:19.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:19.964 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:19.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:20.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:20.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:20.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:20.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:20.146 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:20.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:20.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:20.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:20.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:20.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:20.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:20.361 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:20.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:20.420 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-02T10:04:20.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:20.493 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:20.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:20.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:20 smithi162 ceph-mon[20279]: pgmap v451: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 1033/159459 objects degraded (0.648%); 22396/159459 objects misplaced (14.045%); 16 MiB/s, 258 objects/s recovering 2024-09-02T10:04:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:20 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:20 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/367487137' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:04:20.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:20.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:20.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:20 smithi151 ceph-mon[16639]: pgmap v451: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 1033/159459 objects degraded (0.648%); 22396/159459 objects misplaced (14.045%); 16 MiB/s, 258 objects/s recovering 2024-09-02T10:04:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:20 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:20 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/367487137' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:04:20.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:20.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:20.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:20.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:20 smithi187 ceph-mon[20072]: pgmap v451: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 450 op/s; 1033/159459 objects degraded (0.648%); 22396/159459 objects misplaced (14.045%); 16 MiB/s, 258 objects/s recovering 2024-09-02T10:04:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:20 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:20 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/367487137' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:04:20.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:20.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:20.880 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:20.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:21.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:21.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:21.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 2 1 2024-09-02T10:04:21.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:21.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:21.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:21.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:21.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:21.286 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:21.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:21.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:21.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:21.411 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:21.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:21.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:21.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:21.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:21.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:21.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/367487137' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:04:21.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:21 smithi162 ceph-mon[20279]: osdmap e172: 12 total, 12 up, 11 in 2024-09-02T10:04:21.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:21 smithi162 ceph-mon[20279]: osdmap e173: 12 total, 12 up, 11 in 2024-09-02T10:04:21.639 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/367487137' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:04:21.639 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:21 smithi151 ceph-mon[16639]: osdmap e172: 12 total, 12 up, 11 in 2024-09-02T10:04:21.639 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:21 smithi151 ceph-mon[16639]: osdmap e173: 12 total, 12 up, 11 in 2024-09-02T10:04:21.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:21.686 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:04:21.686 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:04:21.702 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:21.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:21.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:21.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:04:21.731 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:04:21.738 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/367487137' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:04:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:21 smithi187 ceph-mon[20072]: osdmap e172: 12 total, 12 up, 11 in 2024-09-02T10:04:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:21 smithi187 ceph-mon[20072]: osdmap e173: 12 total, 12 up, 11 in 2024-09-02T10:04:21.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:04:21.775 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:04:21.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:21.817 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:04:21.817 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:04:21.818 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:21.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:21.830 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:21.858 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:04:21.859 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:04:21.892 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:04:21.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:04:21.929 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:04:21.930 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:04:21.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:21.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:21.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:21.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:22.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:22.098 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:22.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:22.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:22.227 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:22.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:22.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:22.353 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:22.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:22.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:22 smithi151 ceph-mon[16639]: pgmap v453: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 494 op/s; 1184/156420 objects degraded (0.757%); 21574/156420 objects misplaced (13.792%); 19 MiB/s, 304 objects/s recovering 2024-09-02T10:04:22.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/433343414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:22.454 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:22.478 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:22.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:22.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:22.604 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:22.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:22.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:22.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:22.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:22.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:22.755 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:22 smithi187 ceph-mon[20072]: pgmap v453: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 494 op/s; 1184/156420 objects degraded (0.757%); 21574/156420 objects misplaced (13.792%); 19 MiB/s, 304 objects/s recovering 2024-09-02T10:04:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/433343414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:22.778 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:04:22.778 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:04:22.815 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:04:22.816 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:04:22.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:22 smithi162 ceph-mon[20279]: pgmap v453: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 494 op/s; 1184/156420 objects degraded (0.757%); 21574/156420 objects misplaced (13.792%); 19 MiB/s, 304 objects/s recovering 2024-09-02T10:04:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/433343414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:22.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:22.850 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:04:22.851 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:04:22.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:22.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:22.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:22.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:04:22.885 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:04:22.922 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:04:22.922 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:04:22.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:23.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:23.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:23.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:23.153 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:23.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:23.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:23.280 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:23.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:23.373 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:04:23.373 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:04:23.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:23.409 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:23.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:23.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:23.419 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:04:23.426 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-09-02T10:04:23.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:23.539 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:23.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:23.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:23.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:23.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:23.668 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2940945040' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:04:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:23 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:04:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:23 smithi151 ceph-mon[16639]: from='client.15069 172.21.15.151:0/414632751' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:04:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2940945040' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:04:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:23 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:04:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:23 smithi187 ceph-mon[20072]: from='client.15069 172.21.15.151:0/414632751' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:04:23.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:23.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:23.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:23.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:23.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2940945040' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:04:23.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:23 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:04:23.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:23 smithi162 ceph-mon[20279]: from='client.15069 172.21.15.151:0/414632751' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:04:23.907 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:23.936 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:23.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:23.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:24.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:24.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:24.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:24.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:24.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:24.195 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:24.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:24.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:24.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:24.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:24.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:24.458 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:24.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:24.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:24.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:24 smithi151 ceph-mon[16639]: pgmap v455: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 576 op/s; 1212/151623 objects degraded (0.799%); 20353/151623 objects misplaced (13.423%); 22 MiB/s, 347 objects/s recovering 2024-09-02T10:04:24.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:24 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:24.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:24 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:04:24.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:24 smithi151 ceph-mon[16639]: from='client.15069 172.21.15.151:0/414632751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:04:24.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:24 smithi151 ceph-mon[16639]: osdmap e174: 12 total, 12 up, 11 in 2024-09-02T10:04:24.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1212/151623 objects degraded (0.799%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:24.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:24 smithi151 ceph-mon[16639]: osdmap e175: 12 total, 12 up, 11 in 2024-09-02T10:04:24.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:24.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:24.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:24.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:24.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:24.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:24 smithi187 ceph-mon[20072]: pgmap v455: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 576 op/s; 1212/151623 objects degraded (0.799%); 20353/151623 objects misplaced (13.423%); 22 MiB/s, 347 objects/s recovering 2024-09-02T10:04:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:24 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:24 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:04:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:24 smithi187 ceph-mon[20072]: from='client.15069 172.21.15.151:0/414632751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:04:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:24 smithi187 ceph-mon[20072]: osdmap e174: 12 total, 12 up, 11 in 2024-09-02T10:04:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1212/151623 objects degraded (0.799%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:24 smithi187 ceph-mon[20072]: osdmap e175: 12 total, 12 up, 11 in 2024-09-02T10:04:24.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:24.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:24 smithi162 ceph-mon[20279]: pgmap v455: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 576 op/s; 1212/151623 objects degraded (0.799%); 20353/151623 objects misplaced (13.423%); 22 MiB/s, 347 objects/s recovering 2024-09-02T10:04:24.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:24 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:24 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:04:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:24 smithi162 ceph-mon[20279]: from='client.15069 172.21.15.151:0/414632751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:04:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:24 smithi162 ceph-mon[20279]: osdmap e174: 12 total, 12 up, 11 in 2024-09-02T10:04:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1212/151623 objects degraded (0.799%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:24 smithi162 ceph-mon[20279]: osdmap e175: 12 total, 12 up, 11 in 2024-09-02T10:04:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:24.855 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:24.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:24.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:24.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:24.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:25.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:25.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:25.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:25.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:25.248 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:25.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:25.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:25.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:25.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:25.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:25.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:25.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:25.500 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:25.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:25.627 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:25.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:25.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:25.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:25.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:25.752 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:25.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:25.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:25.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:25.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:25.978 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:26.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:26.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:26.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:26.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:26.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:26.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:26.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:26.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:26.249 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:26.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:26.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:26.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:26.375 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:26.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:26.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:26.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:26.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:26.634 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:26.635 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:26.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:04:26.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:26 smithi187 ceph-mon[20072]: pgmap v458: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 1212/149541 objects degraded (0.810%); 20146/149541 objects misplaced (13.472%); 21 MiB/s, 332 objects/s recovering 2024-09-02T10:04:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:26 smithi187 ceph-mon[20072]: osdmap e176: 12 total, 12 up, 11 in 2024-09-02T10:04:26.762 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:26.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:26.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:26 smithi162 ceph-mon[20279]: pgmap v458: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 1212/149541 objects degraded (0.810%); 20146/149541 objects misplaced (13.472%); 21 MiB/s, 332 objects/s recovering 2024-09-02T10:04:26.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:26 smithi162 ceph-mon[20279]: osdmap e176: 12 total, 12 up, 11 in 2024-09-02T10:04:26.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:26.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:26 smithi151 ceph-mon[16639]: pgmap v458: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 30 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 1212/149541 objects degraded (0.810%); 20146/149541 objects misplaced (13.472%); 21 MiB/s, 332 objects/s recovering 2024-09-02T10:04:26.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:26 smithi151 ceph-mon[16639]: osdmap e176: 12 total, 12 up, 11 in 2024-09-02T10:04:26.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:26.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:26.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:27.024 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:27.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:27.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:27.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:27.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:27.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:27.287 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:27.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:27.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:27.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:27.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:27.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:27.414 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:27.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:27.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:27.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:27.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:27.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:27.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:27.671 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:27.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:27 smithi187 ceph-mon[20072]: osdmap e177: 12 total, 12 up, 11 in 2024-09-02T10:04:27.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:27.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:27.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:27.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:27.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:04:27.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:27 smithi162 ceph-mon[20279]: osdmap e177: 12 total, 12 up, 11 in 2024-09-02T10:04:27.868 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:27 smithi151 ceph-mon[16639]: osdmap e177: 12 total, 12 up, 11 in 2024-09-02T10:04:27.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:27.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:27.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:28.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:28.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:28.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:28.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:28.188 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:28.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:28.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:28.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:28.317 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:28.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:28.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:28.442 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:28.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:28.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:28.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:28.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:28.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:28.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:28.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:28.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:28 smithi187 ceph-mon[20072]: pgmap v461: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 373 op/s; 1349/147138 objects degraded (0.917%); 19347/147138 objects misplaced (13.149%); 11 MiB/s, 178 objects/s recovering 2024-09-02T10:04:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:28 smithi187 ceph-mon[20072]: 2.7 scrub starts 2024-09-02T10:04:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:28 smithi187 ceph-mon[20072]: 2.7 scrub ok 2024-09-02T10:04:28.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:28.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:28.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:28 smithi162 ceph-mon[20279]: pgmap v461: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 373 op/s; 1349/147138 objects degraded (0.917%); 19347/147138 objects misplaced (13.149%); 11 MiB/s, 178 objects/s recovering 2024-09-02T10:04:28.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:28 smithi162 ceph-mon[20279]: 2.7 scrub starts 2024-09-02T10:04:28.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:28 smithi162 ceph-mon[20279]: 2.7 scrub ok 2024-09-02T10:04:28.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:28.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:28.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:28 smithi151 ceph-mon[16639]: pgmap v461: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 373 op/s; 1349/147138 objects degraded (0.917%); 19347/147138 objects misplaced (13.149%); 11 MiB/s, 178 objects/s recovering 2024-09-02T10:04:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:28 smithi151 ceph-mon[16639]: 2.7 scrub starts 2024-09-02T10:04:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:28 smithi151 ceph-mon[16639]: 2.7 scrub ok 2024-09-02T10:04:28.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:28.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:28.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:28.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:29.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:29.107 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:29.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:29.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:29.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:29.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:29.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:29.347 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:29.372 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:29.373 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:29.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:29.499 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:29.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:29.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:29.627 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:29.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:29.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:29 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:04:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:29 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:04:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:29 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:29 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:29 smithi187 ceph-mon[20072]: osdmap e178: 12 total, 12 up, 11 in 2024-09-02T10:04:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:29 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1349/147138 objects degraded (0.917%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:04:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:29 smithi187 ceph-mon[20072]: osdmap e179: 12 total, 12 up, 11 in 2024-09-02T10:04:29.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:29.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:29.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:29.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:29 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:04:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:29 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:04:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:29 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:29 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:29 smithi162 ceph-mon[20279]: osdmap e178: 12 total, 12 up, 11 in 2024-09-02T10:04:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:29 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1349/147138 objects degraded (0.917%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:04:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:29 smithi162 ceph-mon[20279]: osdmap e179: 12 total, 12 up, 11 in 2024-09-02T10:04:29.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:29.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:29.886 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:29.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:29 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:04:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:29 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:04:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:29 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:29 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:29 smithi151 ceph-mon[16639]: osdmap e178: 12 total, 12 up, 11 in 2024-09-02T10:04:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:29 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1349/147138 objects degraded (0.917%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:04:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:29 smithi151 ceph-mon[16639]: osdmap e179: 12 total, 12 up, 11 in 2024-09-02T10:04:29.987 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:30.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:30.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:30.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:30.137 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:30.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:30.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:30.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:04:30.307 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:30.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:30.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:30.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:30.438 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:30.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:30.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-09-02T10:04:30.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:30.579 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:30.588 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:30.602 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:30.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:30.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:30 smithi151 ceph-mon[16639]: pgmap v464: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 469 op/s; 1349/143904 objects degraded (0.937%); 18508/143904 objects misplaced (12.861%); 19 MiB/s, 300 objects/s recovering 2024-09-02T10:04:30.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:30 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:04:30.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:30 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:04:30.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:30.730 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:30 smithi162 ceph-mon[20279]: pgmap v464: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 469 op/s; 1349/143904 objects degraded (0.937%); 18508/143904 objects misplaced (12.861%); 19 MiB/s, 300 objects/s recovering 2024-09-02T10:04:30.730 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:30 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:04:30.730 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:30 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:04:30.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:30.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:30.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:30 smithi187 ceph-mon[20072]: pgmap v464: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 469 op/s; 1349/143904 objects degraded (0.937%); 18508/143904 objects misplaced (12.861%); 19 MiB/s, 300 objects/s recovering 2024-09-02T10:04:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:30 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:04:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:30 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:04:30.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:30.883 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:30.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:30.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:30.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:31.009 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:31.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:31.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:31.139 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:31.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:31.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:31.267 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:31.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:31.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:31.400 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:31.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:31.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:31.502 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:31.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:31.528 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:31.589 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:04:31.589 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:04:31.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:31.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:31.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:31.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:31.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:31.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:31.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:31.931 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:31.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:32.032 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:32.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:32.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:32.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:32.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:04:32.190 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:04:32.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:32.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:32.261 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:04:32.262 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:04:32.283 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:32.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:32.350 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:32.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:32.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:32.476 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:32.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:32.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:32.480 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:04:32.534 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:04:32.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:04:32.575 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:04:32.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:04:32.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:32.603 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:32.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:32.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:32.616 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:04:32.617 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:04:32.661 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:04:32.662 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:04:32.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:04:32.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:32 smithi151 ceph-mon[16639]: pgmap v465: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 487 op/s; 1349/141255 objects degraded (0.955%); 17821/141255 objects misplaced (12.616%); 19 MiB/s, 312 objects/s recovering 2024-09-02T10:04:32.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3320036266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:32.700 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2030086747' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:32.701 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:04:32.701 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:04:32.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:32.736 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:04:32.736 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:04:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:32 smithi187 ceph-mon[20072]: pgmap v465: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 487 op/s; 1349/141255 objects degraded (0.955%); 17821/141255 objects misplaced (12.616%); 19 MiB/s, 312 objects/s recovering 2024-09-02T10:04:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3320036266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2030086747' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:32.756 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:32.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:32.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:32.770 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:04:32.770 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:04:32.806 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:04:32.806 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:04:32.841 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:04:32.842 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:04:32.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:32 smithi162 ceph-mon[20279]: pgmap v465: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.9 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 487 op/s; 1349/141255 objects degraded (0.955%); 17821/141255 objects misplaced (12.616%); 19 MiB/s, 312 objects/s recovering 2024-09-02T10:04:32.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3320036266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:32.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2030086747' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:32.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:32.877 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:04:32.877 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:04:32.883 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:32.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:32.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:33.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:33.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:33.025 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:33.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:33.143 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:33.144 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:33.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:33.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:33.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:33.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:33.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:33.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:33.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:33.548 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:33 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:04:33.548 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2030086747' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:04:33.548 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:33 smithi151 ceph-mon[16639]: osdmap e180: 12 total, 12 up, 11 in 2024-09-02T10:04:33.569 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:04:33.569 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:04:33.569 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:04:33.569 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:04:33.569 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:04:33.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:33.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:33.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:33.680 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:33.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:33.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:33.706 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:33 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:04:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2030086747' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:04:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:33 smithi187 ceph-mon[20072]: osdmap e180: 12 total, 12 up, 11 in 2024-09-02T10:04:33.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:33.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:33.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:33.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:33 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:04:33.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2030086747' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:04:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:33 smithi162 ceph-mon[20279]: osdmap e180: 12 total, 12 up, 11 in 2024-09-02T10:04:33.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:33.962 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:33.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:33.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:34.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:34.088 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:34.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:34.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:34.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:34.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:34.215 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:34.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:04:34.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:34.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:34.346 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:34.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:34.446 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:34.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:34.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:34.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:34.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:34.595 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:34.602 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:34.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:34.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:34.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:34.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:34 smithi187 ceph-mon[20072]: pgmap v467: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.8 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 619 op/s; 1349/135978 objects degraded (0.992%); 17041/135978 objects misplaced (12.532%); 19 MiB/s, 311 objects/s recovering 2024-09-02T10:04:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:34 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1349/135978 objects degraded (0.992%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:04:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:34 smithi187 ceph-mon[20072]: osdmap e181: 12 total, 12 up, 11 in 2024-09-02T10:04:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/495341057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:04:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:34 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:04:34.804 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-02T10:04:34.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:34 smithi151 ceph-mon[16639]: pgmap v467: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.8 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 619 op/s; 1349/135978 objects degraded (0.992%); 17041/135978 objects misplaced (12.532%); 19 MiB/s, 311 objects/s recovering 2024-09-02T10:04:34.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:34 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1349/135978 objects degraded (0.992%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:04:34.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:34 smithi151 ceph-mon[16639]: osdmap e181: 12 total, 12 up, 11 in 2024-09-02T10:04:34.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/495341057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:04:34.804 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:34 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:04:34.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:34 smithi162 ceph-mon[20279]: pgmap v467: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.8 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 619 op/s; 1349/135978 objects degraded (0.992%); 17041/135978 objects misplaced (12.532%); 19 MiB/s, 311 objects/s recovering 2024-09-02T10:04:34.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:34 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1349/135978 objects degraded (0.992%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:04:34.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:34 smithi162 ceph-mon[20279]: osdmap e181: 12 total, 12 up, 11 in 2024-09-02T10:04:34.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/495341057' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:04:34.815 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:34 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:04:34.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:34.857 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:34.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:34.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:34.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:34.981 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:34.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:35.083 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:35.107 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:35.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:35.174 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:04:35.174 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:04:35.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:7836 2.2 2:422fedf8:::benchmark_data_smithi187_45882_object7835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.578259+0000", 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 175.59404290099999, 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22181919, 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.175 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7836 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578259+0000", 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578259+0000", 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578259+0000", 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6400000000000001e-07 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578319+0000", 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0126e-05 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578330+0000", 2024-09-02T10:04:35.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0740999999999999e-05 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578335+0000", 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9259999999999999e-06 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578412+0000", 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6732999999999999e-05 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578483+0000", 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1316000000000006e-05 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578564+0000", 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0821000000000004e-05 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578946+0000", 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038204899999999997 2024-09-02T10:04:35.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582380+0000", 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034338699999999999 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.800002+0000", 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21762209299999999 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.800053+0000", 2024-09-02T10:04:35.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1387000000000001e-05 2024-09-02T10:04:35.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.800078+0000", 2024-09-02T10:04:35.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4363999999999999e-05 2024-09-02T10:04:35.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.181 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.181 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:7841 2.2 2:45ea9ad1:::benchmark_data_smithi187_45882_object7840:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.581408+0000", 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 175.59089374800001, 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22151944800000001, 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7841 2024-09-02T10:04:35.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581408+0000", 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581408+0000", 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581408+0000", 2024-09-02T10:04:35.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3099999999999997e-07 2024-09-02T10:04:35.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581444+0000", 2024-09-02T10:04:35.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5735000000000003e-05 2024-09-02T10:04:35.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581456+0000", 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1603999999999999e-05 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581463+0000", 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9750000000000001e-06 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581508+0000", 2024-09-02T10:04:35.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5061000000000001e-05 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581596+0000", 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8367000000000003e-05 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581677+0000", 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0819e-05 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582132+0000", 2024-09-02T10:04:35.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045448199999999998 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799923+0000", 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21779146199999999 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.802865+0000", 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002941777 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.802896+0000", 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1526999999999998e-05 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.802927+0000", 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0908000000000003e-05 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:7841 2.2 e97/92 ondisk, result = 0)", 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.582308+0000", 2024-09-02T10:04:35.188 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 175.58999365700001, 2024-09-02T10:04:35.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21762236700000001, 2024-09-02T10:04:35.189 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.189 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.189 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:04:35.189 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582308+0000", 2024-09-02T10:04:35.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582308+0000", 2024-09-02T10:04:35.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582308+0000", 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7899999999999998e-07 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582311+0000", 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0709999999999999e-06 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799863+0000", 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.217551945 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799868+0000", 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6099999999999999e-06 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799910+0000", 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.172e-05 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799918+0000", 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1310000000000006e-06 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799930+0000", 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2311e-05 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.193 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:7836 2.2 e97/92 ondisk, result = 0)", 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.582436+0000", 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 175.589865753, 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21764771099999999, 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582436+0000", 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582436+0000", 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582436+0000", 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2300000000000001e-07 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582440+0000", 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8519999999999997e-06 2024-09-02T10:04:35.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799942+0000", 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21750224000000001 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799949+0000", 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2750000000000001e-06 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799990+0000", 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1003e-05 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799997+0000", 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2479999999999999e-06 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.800083+0000", 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6570000000000006e-05 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.197 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:8321 2.3 2:c249931d:::benchmark_data_smithi187_45882_object8320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 175.21426653, 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25867863899999999, 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 8321 2024-09-02T10:04:35.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8100000000000003e-07 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958104+0000", 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8448000000000006e-05 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958111+0000", 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0469999999999996e-06 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958115+0000", 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9689999999999996e-06 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958155+0000", 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0398999999999997e-05 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958225+0000", 2024-09-02T10:04:35.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9578000000000004e-05 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958316+0000", 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1249000000000007e-05 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958733+0000", 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041638500000000002 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.959078+0000", 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000345855 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216656+0000", 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25757725999999997 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216689+0000", 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3368999999999998e-05 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216714+0000", 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4598999999999999e-05 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:8321 2.3 e97/92 ondisk, result = 0)", 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 175.21340667600001, 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25782361500000001, 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.204 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1600000000000001e-07 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958898+0000", 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.066e-06 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216621+0000", 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25772234500000002 2024-09-02T10:04:35.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216625+0000", 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2989999999999998e-06 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216650+0000", 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4939e-05 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216653+0000", 2024-09-02T10:04:35.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1559999999999999e-06 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216718+0000", 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5294000000000002e-05 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:22420 2.2 2:4520d8a5:::benchmark_data_smithi187_45882_object22419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:51.202853+0000", 2024-09-02T10:04:35.208 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 163.96944804899999, 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21069840400000001, 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 22420 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202853+0000", 2024-09-02T10:04:35.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202853+0000", 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202854+0000", 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5899999999999995e-07 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202879+0000", 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4481000000000002e-05 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202890+0000", 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1116e-05 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202895+0000", 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5670000000000002e-06 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202947+0000", 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1085999999999998e-05 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.203066+0000", 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119224 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.203222+0000", 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015642 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.203968+0000", 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000745661 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.204233+0000", 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026555900000000001 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.413490+0000", 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20925611899999999 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.413523+0000", 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3667000000000001e-05 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.413552+0000", 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8645000000000001e-05 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.214 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:38681 2.2 2:44fed43f:::benchmark_data_smithi187_45882_object38680:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:03.911565+0000", 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 151.26073645700001, 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203667875, 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 38681 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.911565+0000", 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.911565+0000", 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.911567+0000", 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3039999999999999e-06 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.216 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.911618+0000", 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0198999999999998e-05 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.911649+0000", 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1788999999999997e-05 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.911664+0000", 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4513e-05 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.911804+0000", 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014040999999999999 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.911964+0000", 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000159853 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.912051+0000", 2024-09-02T10:04:35.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7057000000000004e-05 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.913262+0000", 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012104290000000001 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:03.913462+0000", 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000200497 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.115137+0000", 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.201674874 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.115184+0000", 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6894000000000002e-05 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:04.115233+0000", 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9055999999999999e-05 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.220 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:54905 2.2 2:470b2221:::benchmark_data_smithi187_45882_object54904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.133024+0000", 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 138.03927745999999, 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26483498300000002, 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.221 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54905 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133024+0000", 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133024+0000", 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133026+0000", 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.517e-06 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133067+0000", 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1332999999999999e-05 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133081+0000", 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4272e-05 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133088+0000", 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3600000000000001e-06 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133138+0000", 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0037999999999998e-05 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133240+0000", 2024-09-02T10:04:35.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102081 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133332+0000", 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2132999999999994e-05 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.134852+0000", 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001519963 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.135073+0000", 2024-09-02T10:04:35.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022145800000000001 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.397790+0000", 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26271683099999998 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.397827+0000", 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7119000000000002e-05 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.397859+0000", 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1878000000000002e-05 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:54934 2.3 2:c516e2ca:::benchmark_data_smithi187_45882_object54933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.163207+0000", 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 138.00909401600001, 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21258112400000001, 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.227 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54934 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163207+0000", 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163207+0000", 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163208+0000", 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9100000000000003e-07 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163270+0000", 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1891999999999996e-05 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163278+0000", 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7600000000000002e-06 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163282+0000", 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6820000000000002e-06 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.164696+0000", 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001413231 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.164791+0000", 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5189e-05 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.164911+0000", 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119924 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.165973+0000", 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010621020000000001 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:35.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.173396+0000", 2024-09-02T10:04:35.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074228159999999996 2024-09-02T10:04:35.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.375677+0000", 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20228159500000001 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.375737+0000", 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0074000000000001e-05 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.375789+0000", 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1168000000000002e-05 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:54937 2.3 2:c9a686f1:::benchmark_data_smithi187_45882_object54936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 138.000882932, 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23343879200000001, 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.235 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54937 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1500000000000003e-07 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171466+0000", 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6523000000000002e-05 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171477+0000", 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0784e-05 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171482+0000", 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2079999999999999e-06 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171521+0000", 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8736999999999999e-05 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171596+0000", 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5042000000000004e-05 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171675+0000", 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9369999999999994e-05 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.173027+0000", 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013518409999999999 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.273746+0000", 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.100718883 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.404789+0000", 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13104327700000001 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.404822+0000", 2024-09-02T10:04:35.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2811000000000001e-05 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.404857+0000", 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5401000000000002e-05 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:55030 2.3 2:c75f1b46:::benchmark_data_smithi187_45882_object55029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.446913+0000", 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 137.72538819499999, 2024-09-02T10:04:35.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20451122799999999, 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55030 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.446913+0000", 2024-09-02T10:04:35.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.446913+0000", 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.446914+0000", 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5499999999999997e-07 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.447023+0000", 2024-09-02T10:04:35.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108615 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.447034+0000", 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0749e-05 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.447039+0000", 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2719999999999997e-06 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.447079+0000", 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9807e-05 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.447159+0000", 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0896999999999995e-05 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.447242+0000", 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2638000000000004e-05 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.448658+0000", 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001415445 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.448954+0000", 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000296304 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.651343+0000", 2024-09-02T10:04:35.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20238925599999999 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.651388+0000", 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4736e-05 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.651425+0000", 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6653999999999998e-05 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:55038 2.3 2:c94b451a:::benchmark_data_smithi187_45882_object55037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.453569+0000", 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 137.71873203300001, 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33233287500000003, 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55038 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.248 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453569+0000", 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453569+0000", 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453571+0000", 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.702e-06 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453616+0000", 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5033000000000001e-05 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453635+0000", 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8595000000000001e-05 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453644+0000", 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0100000000000001e-06 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453705+0000", 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0816000000000002e-05 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453835+0000", 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013086799999999999 2024-09-02T10:04:35.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453981+0000", 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014516900000000001 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.455854+0000", 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001873537 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.456078+0000", 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000224083 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.785794+0000", 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32971594999999998 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.785853+0000", 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9083999999999999e-05 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.785902+0000", 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9027999999999999e-05 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67958 2.2 2:4a50163d:::benchmark_data_smithi187_45882_object67957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.903059+0000", 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 125.26924285699999, 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20627833600000001, 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.254 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67958 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.903059+0000", 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.903059+0000", 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.903059+0000", 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2900000000000003e-07 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.903110+0000", 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0408000000000003e-05 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.903121+0000", 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1535000000000001e-05 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.903127+0000", 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8130000000000001e-06 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.903179+0000", 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1714e-05 2024-09-02T10:04:35.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.903276+0000", 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6810000000000003e-05 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.903358+0000", 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2008000000000005e-05 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.905328+0000", 2024-09-02T10:04:35.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019701020000000001 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.905943+0000", 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00061497600000000004 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:30.109232+0000", 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203289308 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:30.109289+0000", 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7009999999999998e-05 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:30.109337+0000", 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7923000000000002e-05 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:67958 2.2 e120/92 ondisk, result = 0)", 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.905677+0000", 2024-09-02T10:04:35.260 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 125.266624506, 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20366662499999999, 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.905677+0000", 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.905677+0000", 2024-09-02T10:04:35.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.905678+0000", 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2699999999999999e-07 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.905682+0000", 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9230000000000004e-06 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:30.109162+0000", 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20348076100000001 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:30.109168+0000", 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.127e-06 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:30.109218+0000", 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0012000000000002e-05 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:30.109226+0000", 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.038e-06 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:30.109344+0000", 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117137 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.264 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:79038 2.3 2:c3e1c878:::benchmark_data_smithi187_45882_object79037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:40.868535+0000", 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.303766924, 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21948013999999999, 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 79038 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868535+0000", 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868535+0000", 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868536+0000", 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.514e-06 2024-09-02T10:04:35.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868561+0000", 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5130000000000002e-05 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868580+0000", 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8865e-05 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868589+0000", 2024-09-02T10:04:35.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3370000000000006e-06 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868656+0000", 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6608999999999999e-05 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868840+0000", 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018391300000000001 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:04:35.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.869015+0000", 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000175093 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.871861+0000", 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00284636 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.875704+0000", 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038424940000000001 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.087928+0000", 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21222416399999999 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.087981+0000", 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3270999999999999e-05 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.088015+0000", 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3389999999999997e-05 2024-09-02T10:04:35.270 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:79041 2.3 2:ce3f4927:::benchmark_data_smithi187_45882_object79040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:40.872042+0000", 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.30025968699999, 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21882871000000001, 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.271 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 79041 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872042+0000", 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872042+0000", 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872046+0000", 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2780000000000004e-06 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872089+0000", 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2681e-05 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872102+0000", 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2977e-05 2024-09-02T10:04:35.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872108+0000", 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7919999999999998e-06 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872159+0000", 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1678000000000002e-05 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872302+0000", 2024-09-02T10:04:35.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014248200000000001 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872462+0000", 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016049400000000001 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.875431+0000", 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002969143 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.875828+0000", 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00039666099999999998 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.090754+0000", 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.214925543 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.090820+0000", 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6431000000000001e-05 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.090871+0000", 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0550000000000002e-05 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:97742 2.2 2:492b4420:::benchmark_data_smithi187_45882_object97741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:05.413372+0000", 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 89.758929322, 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26451705399999997, 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.277 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 97742 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413372+0000", 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413372+0000", 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413373+0000", 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0200000000000001e-07 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413450+0000", 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6959000000000006e-05 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413460+0000", 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.842e-06 2024-09-02T10:04:35.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413465+0000", 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8749999999999999e-06 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413516+0000", 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1252000000000003e-05 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413625+0000", 2024-09-02T10:04:35.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010952500000000001 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413745+0000", 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011921000000000001 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.416627+0000", 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028824359999999999 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.427846+0000", 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011219084000000001 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677815+0000", 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.249968879 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677858+0000", 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3084000000000003e-05 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677889+0000", 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1106000000000003e-05 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:138044 2.2 2:4365713b:::benchmark_data_smithi187_45882_object40293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:08.061901+0000", 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 27.110400722000001, 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204381225, 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.283 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 138044 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.061901+0000", 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.061901+0000", 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.061902+0000", 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6390000000000001e-06 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.061908+0000", 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9909999999999999e-06 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.061922+0000", 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3329000000000001e-05 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.061930+0000", 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1720000000000007e-06 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.061988+0000", 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8572000000000001e-05 2024-09-02T10:04:35.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.062123+0000", 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000134764 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.062259+0000", 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013569400000000001 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.066467+0000", 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042082580000000003 2024-09-02T10:04:35.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.066806+0000", 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033890400000000001 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.266177+0000", 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19937089199999999 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.266236+0000", 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8829000000000002e-05 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:08.266282+0000", 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6180999999999997e-05 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:141512 2.3 2:c7378ca5:::benchmark_data_smithi187_45882_object43761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:14.693802+0000", 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.478499097, 2024-09-02T10:04:35.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37122683899999998, 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141512 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693802+0000", 2024-09-02T10:04:35.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693802+0000", 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693803+0000", 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2500000000000005e-07 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:35.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693807+0000", 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.681e-06 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693815+0000", 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9140000000000008e-06 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693821+0000", 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3090000000000001e-06 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693879+0000", 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7862000000000003e-05 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693996+0000", 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116896 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:04:35.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694103+0000", 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010771400000000001 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.698406+0000", 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043025820000000001 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.703236+0000", 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048299390000000001 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.064923+0000", 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36168711999999997 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.064970+0000", 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7067000000000002e-05 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.065029+0000", 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9030000000000002e-05 2024-09-02T10:04:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:35.296 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:04:35.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:35.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:35.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:35.442 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:35.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:35.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:35.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-09-02T10:04:35.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:35.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:35.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:35.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:35.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:35.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:35.772 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:35.794 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:35.816 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:35.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:35.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:35 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:04:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:35 smithi162 ceph-mon[20279]: osdmap e182: 12 total, 12 up, 11 in 2024-09-02T10:04:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:35 smithi162 ceph-mon[20279]: pgmap v470: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.7 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 1349/133752 objects degraded (1.009%); 16859/133752 objects misplaced (12.605%); 14 MiB/s, 226 objects/s recovering 2024-09-02T10:04:35.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:35.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:35.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:36.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:36.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:04:36.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:35 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:04:36.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:35 smithi151 ceph-mon[16639]: osdmap e182: 12 total, 12 up, 11 in 2024-09-02T10:04:36.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:35 smithi151 ceph-mon[16639]: pgmap v470: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.7 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 1349/133752 objects degraded (1.009%); 16859/133752 objects misplaced (12.605%); 14 MiB/s, 226 objects/s recovering 2024-09-02T10:04:36.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:36.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:36.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:04:36.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:36.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:36.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:35 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:04:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:35 smithi187 ceph-mon[20072]: osdmap e182: 12 total, 12 up, 11 in 2024-09-02T10:04:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:35 smithi187 ceph-mon[20072]: pgmap v470: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.7 GiB data, 9.8 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 563 op/s; 1349/133752 objects degraded (1.009%); 16859/133752 objects misplaced (12.605%); 14 MiB/s, 226 objects/s recovering 2024-09-02T10:04:36.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:36.334 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:36.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:36.357 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:36.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:36.441 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:36.461 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:36.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:36.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:36.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:04:36.585 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:36.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:36.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:36.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:36.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:36.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:36.806 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-09-02T10:04:36.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:36.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:36 smithi151 ceph-mon[16639]: osdmap e183: 12 total, 12 up, 11 in 2024-09-02T10:04:36.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:36 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:04:36.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:36 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:04:36.823 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2412615157' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:04:36.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:36.833 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:36.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:36.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:36.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:36.998 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:04:36.998 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:04:37.026 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:04:37.027 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:04:37.048 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:04:37.048 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:04:37.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:37.082 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:04:37.082 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:04:37.089 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:37.091 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:37.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:36 smithi162 ceph-mon[20279]: osdmap e183: 12 total, 12 up, 11 in 2024-09-02T10:04:37.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:36 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:04:37.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:36 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:04:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2412615157' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:04:37.110 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:04:37.111 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:04:37.142 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:04:37.142 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:04:37.176 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:04:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:04:37.192 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:37.211 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:04:37.211 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:04:37.215 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:37.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:37.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:04:37.250 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:04:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:36 smithi187 ceph-mon[20072]: osdmap e183: 12 total, 12 up, 11 in 2024-09-02T10:04:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:36 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:04:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:36 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:04:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2412615157' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:04:37.287 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:04:37.287 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:04:37.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:37.325 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:04:37.325 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:04:37.343 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:37.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:37.359 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:04:37.359 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:04:37.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:37.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:37.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:37.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:37.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:37.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:37.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:37.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-09-02T10:04:37.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:37.748 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:37.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:37.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:37.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:37.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:37.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:37 smithi151 ceph-mon[16639]: 2.b scrub starts 2024-09-02T10:04:37.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:37 smithi151 ceph-mon[16639]: 2.b scrub ok 2024-09-02T10:04:37.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2412615157' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:04:37.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:37 smithi151 ceph-mon[16639]: osdmap e184: 12 total, 12 up, 11 in 2024-09-02T10:04:37.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4062468854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:37.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:37 smithi151 ceph-mon[16639]: pgmap v473: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.7 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 399 op/s; 1349/131181 objects degraded (1.028%); 16348/131181 objects misplaced (12.462%); 8.5 MiB/s, 135 objects/s recovering 2024-09-02T10:04:37.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:37 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:04:37.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:37 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:04:37.949 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:37.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:37.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:37.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:37.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:37 smithi162 ceph-mon[20279]: 2.b scrub starts 2024-09-02T10:04:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:37 smithi162 ceph-mon[20279]: 2.b scrub ok 2024-09-02T10:04:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2412615157' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:04:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:37 smithi162 ceph-mon[20279]: osdmap e184: 12 total, 12 up, 11 in 2024-09-02T10:04:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4062468854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:37 smithi162 ceph-mon[20279]: pgmap v473: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.7 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 399 op/s; 1349/131181 objects degraded (1.028%); 16348/131181 objects misplaced (12.462%); 8.5 MiB/s, 135 objects/s recovering 2024-09-02T10:04:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:37 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:04:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:37 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:04:38.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:38.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:38.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:38.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:38.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:38.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:38.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:37 smithi187 ceph-mon[20072]: 2.b scrub starts 2024-09-02T10:04:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:37 smithi187 ceph-mon[20072]: 2.b scrub ok 2024-09-02T10:04:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2412615157' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:04:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:37 smithi187 ceph-mon[20072]: osdmap e184: 12 total, 12 up, 11 in 2024-09-02T10:04:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4062468854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:37 smithi187 ceph-mon[20072]: pgmap v473: 37 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.7 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 399 op/s; 1349/131181 objects degraded (1.028%); 16348/131181 objects misplaced (12.462%); 8.5 MiB/s, 135 objects/s recovering 2024-09-02T10:04:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:37 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:04:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:37 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:04:38.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:38.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:38.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:38.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:38.505 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:38.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:38.608 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:38.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:38.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:38.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:38.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:38.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:38.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:38.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:38.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:38.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:38.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:38.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:38 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:04:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:38 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:04:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:38 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1349/131181 objects degraded (1.028%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:04:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:38 smithi151 ceph-mon[16639]: osdmap e185: 12 total, 12 up, 11 in 2024-09-02T10:04:39.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:39.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:39.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:38 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:04:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:38 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:04:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:38 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1349/131181 objects degraded (1.028%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:04:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:38 smithi162 ceph-mon[20279]: osdmap e185: 12 total, 12 up, 11 in 2024-09-02T10:04:39.131 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:39.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:39.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:39.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:39.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:38 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:04:39.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:38 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:04:39.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:38 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1349/131181 objects degraded (1.028%), 4 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:04:39.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:38 smithi187 ceph-mon[20072]: osdmap e185: 12 total, 12 up, 11 in 2024-09-02T10:04:39.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:39.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:39.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:39.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:39.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:39.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:39.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:39.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:39.571 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:39.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:39.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:39.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:39.703 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:39.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:39.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:39.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:39.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:39.834 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:39.871 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:04:39.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:40.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:40.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:40.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:40.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:40.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:39 smithi162 ceph-mon[20279]: 2.f scrub starts 2024-09-02T10:04:40.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:39 smithi162 ceph-mon[20279]: 2.f scrub ok 2024-09-02T10:04:40.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:39 smithi162 ceph-mon[20279]: pgmap v475: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.6 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 330 op/s; 7405/129483 objects degraded (5.719%); 7613/129483 objects misplaced (5.880%); 13 MiB/s, 240 objects/s recovering 2024-09-02T10:04:40.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:40.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:40.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2971630377' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:04:40.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:39 smithi162 ceph-mon[20279]: 2.3 scrub starts 2024-09-02T10:04:40.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:40.133 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:40.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:39 smithi151 ceph-mon[16639]: 2.f scrub starts 2024-09-02T10:04:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:39 smithi151 ceph-mon[16639]: 2.f scrub ok 2024-09-02T10:04:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:39 smithi151 ceph-mon[16639]: pgmap v475: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.6 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 330 op/s; 7405/129483 objects degraded (5.719%); 7613/129483 objects misplaced (5.880%); 13 MiB/s, 240 objects/s recovering 2024-09-02T10:04:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2971630377' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:04:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:39 smithi151 ceph-mon[16639]: 2.3 scrub starts 2024-09-02T10:04:40.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:39 smithi187 ceph-mon[20072]: 2.f scrub starts 2024-09-02T10:04:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:39 smithi187 ceph-mon[20072]: 2.f scrub ok 2024-09-02T10:04:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:39 smithi187 ceph-mon[20072]: pgmap v475: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.6 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 330 op/s; 7405/129483 objects degraded (5.719%); 7613/129483 objects misplaced (5.880%); 13 MiB/s, 240 objects/s recovering 2024-09-02T10:04:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:40.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:40.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2971630377' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:04:40.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:39 smithi187 ceph-mon[20072]: 2.3 scrub starts 2024-09-02T10:04:40.259 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:40.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:40.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:40.386 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:40.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:40.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:04:40.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:40.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:40.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:40.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:40.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:40.645 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:40.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:40.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:40.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:40.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:40.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:40.826 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:40.873 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-02T10:04:40.874 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:40 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:40.874 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2971630377' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:04:40.875 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:40 smithi151 ceph-mon[16639]: osdmap e186: 12 total, 12 up, 11 in 2024-09-02T10:04:40.875 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1915753817' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:04:40.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:40.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:40.914 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:41.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:41.043 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:41.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:41.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:40 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2971630377' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:04:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:40 smithi162 ceph-mon[20279]: osdmap e186: 12 total, 12 up, 11 in 2024-09-02T10:04:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1915753817' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:04:41.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:41.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:41.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:40 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2971630377' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:04:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:40 smithi187 ceph-mon[20072]: osdmap e186: 12 total, 12 up, 11 in 2024-09-02T10:04:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1915753817' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:04:41.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:41.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:41.296 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:41.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:41.419 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:41.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:41.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:41.475 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:04:41.476 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:04:41.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:41.544 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:41.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:41.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:41.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:41.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:41.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:41.806 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:41.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:41.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:41.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:41.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:42.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:42.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-09-02T10:04:42.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:42.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:42.173 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:42.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1915753817' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:04:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:41 smithi151 ceph-mon[16639]: osdmap e187: 12 total, 12 up, 11 in 2024-09-02T10:04:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:41 smithi151 ceph-mon[16639]: pgmap v478: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.6 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 449 op/s; 7405/127107 objects degraded (5.826%); 6814/127107 objects misplaced (5.361%); 17 MiB/s, 304 objects/s recovering 2024-09-02T10:04:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3540263218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:42.198 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:42.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:42.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:42.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1915753817' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:04:42.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:41 smithi187 ceph-mon[20072]: osdmap e187: 12 total, 12 up, 11 in 2024-09-02T10:04:42.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:41 smithi187 ceph-mon[20072]: pgmap v478: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.6 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 449 op/s; 7405/127107 objects degraded (5.826%); 6814/127107 objects misplaced (5.361%); 17 MiB/s, 304 objects/s recovering 2024-09-02T10:04:42.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3540263218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:42.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:42.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:42.325 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:42.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1915753817' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:04:42.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:41 smithi162 ceph-mon[20279]: osdmap e187: 12 total, 12 up, 11 in 2024-09-02T10:04:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:41 smithi162 ceph-mon[20279]: pgmap v478: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 31 active+clean; 2.6 GiB data, 9.4 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 449 op/s; 7405/127107 objects degraded (5.826%); 6814/127107 objects misplaced (5.361%); 17 MiB/s, 304 objects/s recovering 2024-09-02T10:04:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3540263218' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:42.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:42.432 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:04:42.433 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:04:42.450 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:42.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:42.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:42.455 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:42.470 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:04:42.471 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:04:42.507 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:04:42.507 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:04:42.545 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:04:42.545 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:04:42.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:42.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:04:42.585 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:04:42.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:42.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:42.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:42.616 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:04:42.616 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:04:42.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:04:42.643 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:04:42.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:04:42.679 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:04:42.679 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:04:42.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:42.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:04:42.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:04:42.720 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:42.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:42.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:42.757 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:04:42.758 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:04:42.800 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:04:42.800 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:04:42.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:42.839 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:04:42.839 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:04:42.847 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:42.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:42.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:42.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:42.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:42.973 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:43.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:43.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:43.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:43.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:43.221 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:43.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:43.323 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:43.344 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:43.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:43.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:43.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:43.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:43.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:43.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:43.596 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:43.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:43.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:43.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:43.720 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:43.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:43.801 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-02T10:04:43.806 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:43.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:43.848 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:43.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:43.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:43.980 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:43.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:43.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:44.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:44.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:44.110 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:44.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:44.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:43 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:44.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:43 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:44.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:43 smithi151 ceph-mon[16639]: osdmap e188: 12 total, 12 up, 11 in 2024-09-02T10:04:44.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:43 smithi151 ceph-mon[16639]: pgmap v480: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 2.5 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 7405/121191 objects degraded (6.110%); 6501/121191 objects misplaced (5.364%); 14 MiB/s, 265 objects/s recovering 2024-09-02T10:04:44.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/381396978' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:04:44.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:43 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:04:44.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:43 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 7405/127107 objects degraded (5.826%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:44.168 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:43 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:04:44.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:43 smithi151 ceph-mon[16639]: osdmap e189: 12 total, 12 up, 11 in 2024-09-02T10:04:44.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:44.239 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:44.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:44.240 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:43 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:43 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:43 smithi187 ceph-mon[20072]: osdmap e188: 12 total, 12 up, 11 in 2024-09-02T10:04:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:43 smithi187 ceph-mon[20072]: pgmap v480: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 2.5 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 7405/121191 objects degraded (6.110%); 6501/121191 objects misplaced (5.364%); 14 MiB/s, 265 objects/s recovering 2024-09-02T10:04:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/381396978' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:04:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:43 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:04:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:43 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 7405/127107 objects degraded (5.826%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:43 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:04:44.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:43 smithi187 ceph-mon[20072]: osdmap e189: 12 total, 12 up, 11 in 2024-09-02T10:04:44.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:43 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:43 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:43 smithi162 ceph-mon[20279]: osdmap e188: 12 total, 12 up, 11 in 2024-09-02T10:04:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:43 smithi162 ceph-mon[20279]: pgmap v480: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 2.5 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 641 op/s; 7405/121191 objects degraded (6.110%); 6501/121191 objects misplaced (5.364%); 14 MiB/s, 265 objects/s recovering 2024-09-02T10:04:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/381396978' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:04:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:43 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:04:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:43 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 7405/127107 objects degraded (5.826%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:43 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:04:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:43 smithi162 ceph-mon[20279]: osdmap e189: 12 total, 12 up, 11 in 2024-09-02T10:04:44.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:44.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:44.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:44.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:44.512 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:44.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:44.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:44.639 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:44.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:44.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:44.764 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:44.765 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:44.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-09-02T10:04:44.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:44.890 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:44.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:44.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:44 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:44 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:44.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:45.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:45.018 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:45.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:45.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:45.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:45.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:45.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:44 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:44 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:45.269 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:45.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:45.276 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:44 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:45.276 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:44 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:45.319 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:45.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:45.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:45.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:45.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:45.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:45.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:45.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:45.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:45.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:45.691 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:45.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:04:45.793 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:45.816 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:45.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:45.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:45.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:45.931 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:04:45.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:45.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:45.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:46.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:46.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:46.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:46.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: osdmap e190: 12 total, 12 up, 11 in 2024-09-02T10:04:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:04:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:04:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:04:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:04:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: pgmap v483: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 2.4 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 7405/119229 objects degraded (6.211%); 6218/119229 objects misplaced (5.215%); 8.8 MiB/s, 187 objects/s recovering 2024-09-02T10:04:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:46.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1665745769' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:46.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:46.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:46.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:46.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:04:46.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:04:46.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:46.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:46.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:46.218 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: osdmap e190: 12 total, 12 up, 11 in 2024-09-02T10:04:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:04:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: pgmap v483: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 2.4 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 7405/119229 objects degraded (6.211%); 6218/119229 objects misplaced (5.215%); 8.8 MiB/s, 187 objects/s recovering 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1665745769' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:04:46.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:46.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: osdmap e190: 12 total, 12 up, 11 in 2024-09-02T10:04:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:04:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:04:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:04:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:04:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: pgmap v483: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 2.4 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 7405/119229 objects degraded (6.211%); 6218/119229 objects misplaced (5.215%); 8.8 MiB/s, 187 objects/s recovering 2024-09-02T10:04:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1665745769' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:46.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:04:46.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:46.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:46.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:04:46.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:04:46.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:04:46.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:46.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:46.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:46.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:46.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:46.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:46.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:46.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:46.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:46.719 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:46.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:46.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:46.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:46.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:46.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:46.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:46.930 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-02T10:04:46.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:47.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:47.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:47.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:47.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:47.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:47.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:47.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:46 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:46 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:04:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:46 smithi151 ceph-mon[16639]: osdmap e191: 12 total, 12 up, 11 in 2024-09-02T10:04:47.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/51877362' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:04:47.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:46 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:04:47.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:46 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:46 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:04:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:46 smithi187 ceph-mon[20072]: osdmap e191: 12 total, 12 up, 11 in 2024-09-02T10:04:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/51877362' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:04:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:46 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:04:47.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:47.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:47.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:46 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:47.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:46 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:04:47.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:46 smithi162 ceph-mon[20279]: osdmap e191: 12 total, 12 up, 11 in 2024-09-02T10:04:47.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/51877362' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:04:47.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:46 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:04:47.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:47.387 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:47.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:47.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:47.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:47.516 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:47.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:47.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:47.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:47.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:47.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:47.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:47.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:47.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:47.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:47.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:47.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:47.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:48.014 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:48.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:48.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:48.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:48.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-09-02T10:04:48.141 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:48.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:48.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:47 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:04:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:47 smithi187 ceph-mon[20072]: osdmap e192: 12 total, 12 up, 11 in 2024-09-02T10:04:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:47 smithi187 ceph-mon[20072]: pgmap v486: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 2.4 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 474 op/s; 7405/115491 objects degraded (6.412%); 5694/115491 objects misplaced (4.930%); 10 MiB/s, 166 objects/s recovering 2024-09-02T10:04:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1930373515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:47 smithi187 ceph-mon[20072]: osdmap e193: 12 total, 12 up, 11 in 2024-09-02T10:04:48.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:48.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:48.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:48.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:04:48.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:04:48.323 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:47 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:04:48.323 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:47 smithi151 ceph-mon[16639]: osdmap e192: 12 total, 12 up, 11 in 2024-09-02T10:04:48.323 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:47 smithi151 ceph-mon[16639]: pgmap v486: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 2.4 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 474 op/s; 7405/115491 objects degraded (6.412%); 5694/115491 objects misplaced (4.930%); 10 MiB/s, 166 objects/s recovering 2024-09-02T10:04:48.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1930373515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:48.324 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:47 smithi151 ceph-mon[16639]: osdmap e193: 12 total, 12 up, 11 in 2024-09-02T10:04:48.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:47 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:04:48.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:47 smithi162 ceph-mon[20279]: osdmap e192: 12 total, 12 up, 11 in 2024-09-02T10:04:48.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:47 smithi162 ceph-mon[20279]: pgmap v486: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 31 active+clean; 2.4 GiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 474 op/s; 7405/115491 objects degraded (6.412%); 5694/115491 objects misplaced (4.930%); 10 MiB/s, 166 objects/s recovering 2024-09-02T10:04:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1930373515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:47 smithi162 ceph-mon[20279]: osdmap e193: 12 total, 12 up, 11 in 2024-09-02T10:04:48.359 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:04:48.359 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:04:48.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:48.399 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:04:48.399 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:04:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:48.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:48.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:48.440 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:04:48.440 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:04:48.480 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:04:48.480 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:04:48.487 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:48.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:04:48.516 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:04:48.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:48.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:48.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:48.554 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:04:48.554 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:04:48.587 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:04:48.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:04:48.622 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:04:48.623 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:04:48.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:48.655 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:04:48.655 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:04:48.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:04:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:48.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:48.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:48.691 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:04:48.691 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:04:48.729 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:04:48.729 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:04:48.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:48.818 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:48.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:48.919 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:48.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:48.943 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:49.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:49.071 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:49.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:49.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:04:49.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:48 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:04:49.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:48 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 7405/115491 objects degraded (6.412%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:49.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:49.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:49.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:48 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:04:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:48 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 7405/115491 objects degraded (6.412%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:49.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:49.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:49.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:49.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:48 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:04:49.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:48 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 7405/115491 objects degraded (6.412%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:49.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:49.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:49.468 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:49.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:49.491 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:49.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:49.594 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:49.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:49.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:49.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:49.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:49.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:49.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:49.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:49.858 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:49.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:49.981 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:49.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:50.041 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:50.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:50.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:50.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:50.182 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:49 smithi151 ceph-mon[16639]: pgmap v488: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.3 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 477 op/s; 1260/113232 objects degraded (1.113%); 5694/113232 objects misplaced (5.029%); 5.7 MiB/s, 183 objects/s recovering 2024-09-02T10:04:50.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:49 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:04:50.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:50.254 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:49 smithi162 ceph-mon[20279]: pgmap v488: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.3 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 477 op/s; 1260/113232 objects degraded (1.113%); 5694/113232 objects misplaced (5.029%); 5.7 MiB/s, 183 objects/s recovering 2024-09-02T10:04:50.254 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:50 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:04:50.255 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:50.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:49 smithi187 ceph-mon[20072]: pgmap v488: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.3 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 477 op/s; 1260/113232 objects degraded (1.113%); 5694/113232 objects misplaced (5.029%); 5.7 MiB/s, 183 objects/s recovering 2024-09-02T10:04:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:49 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:04:50.356 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:50.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:50.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:50.480 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:50.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:50.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:50.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:50.536 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:04:50.536 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:04:50.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:50.645 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:50.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:50.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:50.778 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:50.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:50.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:50.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:50.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:50.965 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-09-02T10:04:51.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:51.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:51.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:51.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:51.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:51.189 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:51.191 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:51.191 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1411105786' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:04:51.191 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:50 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:04:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1411105786' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:04:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:50 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:04:51.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:51.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:51.315 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:51.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1411105786' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:04:51.380 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:50 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:04:51.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:04:51.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:51.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:51.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:51.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:51.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:51.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:51.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:04:51.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:51.720 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:51.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:51.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-09-02T10:04:51.770 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:51.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:51.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:51.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:51.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:51.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:51.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:51.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:51.986 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:04:52.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:52.105 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:52.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:52.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:52.169 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:51 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:04:52.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:51 smithi162 ceph-mon[20279]: osdmap e194: 12 total, 12 up, 11 in 2024-09-02T10:04:52.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1210847996' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:04:52.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:51 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:04:52.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:51 smithi162 ceph-mon[20279]: pgmap v490: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.2 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 986 B/s rd, 38 KiB/s wr, 509 op/s; 1260/110241 objects degraded (1.143%); 4807/110241 objects misplaced (4.360%); 8.6 MiB/s, 211 objects/s recovering 2024-09-02T10:04:52.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:52.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:52.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:51 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:04:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:51 smithi187 ceph-mon[20072]: osdmap e194: 12 total, 12 up, 11 in 2024-09-02T10:04:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1210847996' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:04:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:51 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:04:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:51 smithi187 ceph-mon[20072]: pgmap v490: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.2 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 986 B/s rd, 38 KiB/s wr, 509 op/s; 1260/110241 objects degraded (1.143%); 4807/110241 objects misplaced (4.360%); 8.6 MiB/s, 211 objects/s recovering 2024-09-02T10:04:52.312 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:04:52.312 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:04:52.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:52.371 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:52.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:52.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:51 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:04:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:51 smithi151 ceph-mon[16639]: osdmap e194: 12 total, 12 up, 11 in 2024-09-02T10:04:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1210847996' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:04:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:51 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:04:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:51 smithi151 ceph-mon[16639]: pgmap v490: 37 pgs: 1 remapped+peering, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.2 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 986 B/s rd, 38 KiB/s wr, 509 op/s; 1260/110241 objects degraded (1.143%); 4807/110241 objects misplaced (4.360%); 8.6 MiB/s, 211 objects/s recovering 2024-09-02T10:04:52.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:52.501 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:52.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:52.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:52.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:52.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:52.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:52.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:52.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:52.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:52.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:52.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:52.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:52.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:52.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:04:53.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:53.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:52 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:53.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:52 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:04:53.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:52 smithi151 ceph-mon[16639]: osdmap e195: 12 total, 12 up, 11 in 2024-09-02T10:04:53.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:53.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:53.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:53.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:53.207 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:53.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:53.226 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:53.234 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:52 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:52 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:04:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:52 smithi187 ceph-mon[20072]: osdmap e195: 12 total, 12 up, 11 in 2024-09-02T10:04:53.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:53.318 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:53.341 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:53.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:53.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:52 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:04:53.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:52 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:04:53.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:52 smithi162 ceph-mon[20279]: osdmap e195: 12 total, 12 up, 11 in 2024-09-02T10:04:53.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:53.464 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:53.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:53.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:53.590 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:53.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:53.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:53.724 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:53.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:53.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:53.856 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:53.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:53.881 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:04:53.881 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:04:53.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:53.985 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:53.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:53.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:54.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:54.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:54.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:54.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:54.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:54.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:54.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:53 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:53 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:53 smithi187 ceph-mon[20072]: osdmap e196: 12 total, 12 up, 11 in 2024-09-02T10:04:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:53 smithi187 ceph-mon[20072]: pgmap v493: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 987 B/s rd, 38 KiB/s wr, 532 op/s; 9151/104997 objects degraded (8.715%); 4807/104997 objects misplaced (4.578%); 8.6 MiB/s, 188 objects/s recovering 2024-09-02T10:04:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:53 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1260/110241 objects degraded (1.143%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:53 smithi187 ceph-mon[20072]: osdmap e197: 12 total, 12 up, 11 in 2024-09-02T10:04:54.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:53 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:53 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:53 smithi162 ceph-mon[20279]: osdmap e196: 12 total, 12 up, 11 in 2024-09-02T10:04:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:53 smithi162 ceph-mon[20279]: pgmap v493: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 987 B/s rd, 38 KiB/s wr, 532 op/s; 9151/104997 objects degraded (8.715%); 4807/104997 objects misplaced (4.578%); 8.6 MiB/s, 188 objects/s recovering 2024-09-02T10:04:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:53 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1260/110241 objects degraded (1.143%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:53 smithi162 ceph-mon[20279]: osdmap e197: 12 total, 12 up, 11 in 2024-09-02T10:04:54.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:54.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:54.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:54.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:53 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:53 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:53 smithi151 ceph-mon[16639]: osdmap e196: 12 total, 12 up, 11 in 2024-09-02T10:04:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:53 smithi151 ceph-mon[16639]: pgmap v493: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.1 GiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 987 B/s rd, 38 KiB/s wr, 532 op/s; 9151/104997 objects degraded (8.715%); 4807/104997 objects misplaced (4.578%); 8.6 MiB/s, 188 objects/s recovering 2024-09-02T10:04:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:53 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1260/110241 objects degraded (1.143%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:53 smithi151 ceph-mon[16639]: osdmap e197: 12 total, 12 up, 11 in 2024-09-02T10:04:54.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:54.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:54.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:54.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:54.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:04:54.636 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:54.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:54.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:54.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:54.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:54.760 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:54.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:54.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:54.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:54.962 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:54.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:55.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:55.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:55.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:55.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:04:55.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:55.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:55.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:55.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:55.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:55 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:04:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:55 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:04:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:55 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:04:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:55 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:04:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:55 smithi151 ceph-mon[16639]: 2.3 scrub starts 2024-09-02T10:04:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:55 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:04:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:55 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:04:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:55 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:04:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:55 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:04:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:55 smithi187 ceph-mon[20072]: 2.3 scrub starts 2024-09-02T10:04:55.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:55.280 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:55.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:55.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:55 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:04:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:55 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:04:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:55 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:04:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:55 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:04:55.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:55 smithi162 ceph-mon[20279]: 2.3 scrub starts 2024-09-02T10:04:55.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:55.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:55.406 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:55.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:55.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:55.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:55.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:55.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:55.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:55.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:55.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:55.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:55.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:04:55.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:04:55.922 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:04:55.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:55.968 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:04:55.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:04:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:55.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:55.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:56.009 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:04:56.009 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:04:56.013 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-09-02T10:04:56.055 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:04:56.055 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:04:56.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:04:56.085 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:04:56.085 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:04:56.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:56.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:56.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:04:56.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:56 smithi151 ceph-mon[16639]: 2.9 scrub starts 2024-09-02T10:04:56.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:56 smithi151 ceph-mon[16639]: 2.3 scrub ok 2024-09-02T10:04:56.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:56 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:04:56.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:56 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:04:56.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:56 smithi151 ceph-mon[16639]: pgmap v495: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 9151/102687 objects degraded (8.912%); 4548/102687 objects misplaced (4.429%); 3.9 MiB/s, 125 objects/s recovering 2024-09-02T10:04:56.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/370031229' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:04:56.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1347922293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:56.113 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:04:56.114 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:04:56.138 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:04:56.138 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:04:56.165 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:04:56.165 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:04:56.189 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:04:56.189 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:04:56.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:56.217 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:04:56.217 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:04:56.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:56.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:56.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:04:56.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:04:56.276 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:04:56.276 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:04:56.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:56.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:56 smithi162 ceph-mon[20279]: 2.9 scrub starts 2024-09-02T10:04:56.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:56 smithi162 ceph-mon[20279]: 2.3 scrub ok 2024-09-02T10:04:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:56 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:04:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:56 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:04:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:56 smithi162 ceph-mon[20279]: pgmap v495: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 9151/102687 objects degraded (8.912%); 4548/102687 objects misplaced (4.429%); 3.9 MiB/s, 125 objects/s recovering 2024-09-02T10:04:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/370031229' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:04:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1347922293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:56.354 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:56.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:56.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:56.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:56.479 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:56.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:56 smithi187 ceph-mon[20072]: 2.9 scrub starts 2024-09-02T10:04:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:56 smithi187 ceph-mon[20072]: 2.3 scrub ok 2024-09-02T10:04:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:56 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:04:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:56 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:04:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:56 smithi187 ceph-mon[20072]: pgmap v495: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.1 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 606 op/s; 9151/102687 objects degraded (8.912%); 4548/102687 objects misplaced (4.429%); 3.9 MiB/s, 125 objects/s recovering 2024-09-02T10:04:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/370031229' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:04:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1347922293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:04:56.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:56.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:56.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:56.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:56.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:56.730 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:56.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:56.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:56.858 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:56.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:56.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-09-02T10:04:56.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:57.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:57.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:57.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:57.146 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:57.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:57.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:57.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:57 smithi151 ceph-mon[16639]: 2.9 scrub ok 2024-09-02T10:04:57.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:57 smithi151 ceph-mon[16639]: 2.6 scrub starts 2024-09-02T10:04:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/370031229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:04:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:57 smithi151 ceph-mon[16639]: osdmap e198: 12 total, 12 up, 11 in 2024-09-02T10:04:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:57 smithi151 ceph-mon[16639]: 2.a scrub starts 2024-09-02T10:04:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:57 smithi151 ceph-mon[16639]: 2.a scrub ok 2024-09-02T10:04:57.205 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:04:57.205 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:04:57.205 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:04:57.206 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:04:57.206 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:04:57.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:57.270 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:57.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:57.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:57 smithi162 ceph-mon[20279]: 2.9 scrub ok 2024-09-02T10:04:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:57 smithi162 ceph-mon[20279]: 2.6 scrub starts 2024-09-02T10:04:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/370031229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:04:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:57 smithi162 ceph-mon[20279]: osdmap e198: 12 total, 12 up, 11 in 2024-09-02T10:04:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:57 smithi162 ceph-mon[20279]: 2.a scrub starts 2024-09-02T10:04:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:57 smithi162 ceph-mon[20279]: 2.a scrub ok 2024-09-02T10:04:57.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:57.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:57.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:57.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:57 smithi187 ceph-mon[20072]: 2.9 scrub ok 2024-09-02T10:04:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:57 smithi187 ceph-mon[20072]: 2.6 scrub starts 2024-09-02T10:04:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/370031229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:04:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:57 smithi187 ceph-mon[20072]: osdmap e198: 12 total, 12 up, 11 in 2024-09-02T10:04:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:57 smithi187 ceph-mon[20072]: 2.a scrub starts 2024-09-02T10:04:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:57 smithi187 ceph-mon[20072]: 2.a scrub ok 2024-09-02T10:04:57.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:57.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:57.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:57.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:57.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:57.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:57.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:04:57.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:57.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:04:57.883 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:04:57.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:04:57.921 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:57.922 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:04:58.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:58.051 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:58.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:58.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:58.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:58 smithi162 ceph-mon[20279]: 2.6 scrub ok 2024-09-02T10:04:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:58 smithi162 ceph-mon[20279]: osdmap e199: 12 total, 12 up, 11 in 2024-09-02T10:04:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:58 smithi162 ceph-mon[20279]: pgmap v498: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.0 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 461 op/s; 9151/99249 objects degraded (9.220%); 3980/99249 objects misplaced (4.010%); 10 MiB/s, 159 objects/s recovering 2024-09-02T10:04:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:58 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:04:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:58 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:04:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:58 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:04:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:58 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:04:58.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:58.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:58.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:58.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:58.205 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:04:58.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:58.305 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:58.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:58.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:58.431 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:58.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:58.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:58 smithi151 ceph-mon[16639]: 2.6 scrub ok 2024-09-02T10:04:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:58 smithi151 ceph-mon[16639]: osdmap e199: 12 total, 12 up, 11 in 2024-09-02T10:04:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:58 smithi151 ceph-mon[16639]: pgmap v498: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.0 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 461 op/s; 9151/99249 objects degraded (9.220%); 3980/99249 objects misplaced (4.010%); 10 MiB/s, 159 objects/s recovering 2024-09-02T10:04:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:58 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:04:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:58 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:04:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:58 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:04:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:58 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:04:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:58 smithi187 ceph-mon[20072]: 2.6 scrub ok 2024-09-02T10:04:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:58 smithi187 ceph-mon[20072]: osdmap e199: 12 total, 12 up, 11 in 2024-09-02T10:04:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:58 smithi187 ceph-mon[20072]: pgmap v498: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.0 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 461 op/s; 9151/99249 objects degraded (9.220%); 3980/99249 objects misplaced (4.010%); 10 MiB/s, 159 objects/s recovering 2024-09-02T10:04:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:58 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:04:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:58 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:04:58.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:58 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:04:58.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:58 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:04:58.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:04:58.557 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:58.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:04:58.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:58.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:58.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:58.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:58.737 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:04:58.737 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:04:58.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:04:58.737 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:04:58.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.737 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:55055 2.8 2:1127c854:::benchmark_data_smithi187_45882_object55054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:04:58.737 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.471137+0000", 2024-09-02T10:04:58.737 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 161.26353401700001, 2024-09-02T10:04:58.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28183072799999997, 2024-09-02T10:04:58.737 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55055 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471137+0000", 2024-09-02T10:04:58.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471137+0000", 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471138+0000", 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0070000000000001e-06 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471192+0000", 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3325000000000002e-05 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471202+0000", 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0468999999999999e-05 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471207+0000", 2024-09-02T10:04:58.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8720000000000001e-06 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471242+0000", 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4702000000000001e-05 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471322+0000", 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0093999999999996e-05 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471411+0000", 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9015000000000001e-05 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.472901+0000", 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014899150000000001 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473274+0000", 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037354899999999998 2024-09-02T10:04:58.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752887+0000", 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27961233699999999 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752928+0000", 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1560999999999997e-05 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752968+0000", 2024-09-02T10:04:58.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9882000000000002e-05 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:55055 2.8 e113/92 ondisk, result = 0)", 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.473073+0000", 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 161.261598443, 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27990168399999998, 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:04:58.744 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473073+0000", 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473073+0000", 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473074+0000", 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7100000000000002e-07 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473076+0000", 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4549999999999998e-06 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752722+0000", 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27964593100000001 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752743+0000", 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1087e-05 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752872+0000", 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012929900000000001 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752881+0000", 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9339999999999995e-06 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752975+0000", 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3406999999999995e-05 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:55067 2.8 2:1ce73561:::benchmark_data_smithi187_45882_object55066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.485554+0000", 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 161.24911776299999, 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27014719300000001, 2024-09-02T10:04:58.748 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55067 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485554+0000", 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485554+0000", 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485556+0000", 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.885e-06 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485587+0000", 2024-09-02T10:04:58.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1435e-05 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485605+0000", 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7668000000000001e-05 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485615+0000", 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0302999999999999e-05 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485678+0000", 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2840000000000001e-05 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485807+0000", 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128947 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485908+0000", 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101378 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487422+0000", 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001514236 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487766+0000", 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034318 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755648+0000", 2024-09-02T10:04:58.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26788223500000002 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755679+0000", 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0703000000000001e-05 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755701+0000", 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2382999999999999e-05 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:55067 2.8 e113/92 ondisk, result = 0)", 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.487535+0000", 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 161.24713678699999, 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26817209600000003, 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487535+0000", 2024-09-02T10:04:58.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487535+0000", 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487536+0000", 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1379999999999999e-06 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487541+0000", 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3959999999999997e-06 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755584+0000", 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26804281099999999 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755590+0000", 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4399999999999996e-06 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755640+0000", 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0973999999999997e-05 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755645+0000", 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2289999999999997e-06 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755707+0000", 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2107999999999996e-05 2024-09-02T10:04:58.758 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67151 2.0 2:0d008404:::benchmark_data_smithi187_45882_object67150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251522+0000", 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.483149801, 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.248181598, 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.759 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67151 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251522+0000", 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251522+0000", 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251522+0000", 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8999999999999997e-07 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251552+0000", 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9374999999999999e-05 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251558+0000", 2024-09-02T10:04:58.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9249999999999998e-06 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251564+0000", 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1540000000000001e-06 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251617+0000", 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3603999999999997e-05 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251621+0000", 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.664e-06 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.486140+0000", 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23451887099999999 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.486258+0000", 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117853 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.486374+0000", 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116797 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.493396+0000", 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070217049999999996 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.497901+0000", 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045047589999999997 2024-09-02T10:04:58.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.497985+0000", 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3819000000000005e-05 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.498019+0000", 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4258000000000003e-05 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.499703+0000", 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016843240000000001 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67227 2.0 2:0efa4299:::benchmark_data_smithi187_45882_object67226:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251573+0000", 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.48309846699999, 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24924685399999999, 2024-09-02T10:04:58.767 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67227 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251573+0000", 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251573+0000", 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251573+0000", 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5400000000000002e-07 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251601+0000", 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7942000000000001e-05 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251609+0000", 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.153e-06 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251614+0000", 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3800000000000002e-06 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251667+0000", 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3044999999999998e-05 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251671+0000", 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7799999999999998e-06 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.490742+0000", 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23907071699999999 2024-09-02T10:04:58.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.490898+0000", 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015618399999999999 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.491071+0000", 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017328299999999999 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.497939+0000", 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.006867789 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.497972+0000", 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3513000000000003e-05 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.499715+0000", 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001742646 2024-09-02T10:04:58.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.499753+0000", 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7728000000000002e-05 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.500820+0000", 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010672399999999999 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67228 2.0 2:051987d6:::benchmark_data_smithi187_45882_object67227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251623+0000", 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.48304893700001, 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25360533099999999, 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.775 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67228 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251623+0000", 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251623+0000", 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251623+0000", 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.63e-07 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251651+0000", 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7744000000000001e-05 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251658+0000", 2024-09-02T10:04:58.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1679999999999998e-06 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251663+0000", 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0830000000000003e-06 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251692+0000", 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9198999999999999e-05 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251696+0000", 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4860000000000002e-06 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.493431+0000", 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24173510300000001 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.493591+0000", 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160218 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.493778+0000", 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000186584 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.500830+0000", 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070525040000000002 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.503265+0000", 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024344700000000002 2024-09-02T10:04:58.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.503332+0000", 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7432000000000001e-05 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.503381+0000", 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8523000000000003e-05 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505228+0000", 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018473540000000001 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67245 2.0 2:0a07d6c2:::benchmark_data_smithi187_45882_object67244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.482999431, 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25464746599999999, 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.782 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67245 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1699999999999999e-07 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251698+0000", 2024-09-02T10:04:58.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5471999999999999e-05 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251705+0000", 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.81e-06 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251710+0000", 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.169e-06 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251741+0000", 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0579e-05 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251744+0000", 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4489999999999999e-06 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.495787+0000", 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.244043118 2024-09-02T10:04:58.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.495917+0000", 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130099 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.496029+0000", 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011203499999999999 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:58.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505241+0000", 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092117080000000007 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505301+0000", 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0078000000000003e-05 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505348+0000", 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6650999999999998e-05 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505388+0000", 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9974e-05 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.506320+0000", 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00093190700000000005 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67250 2.0 2:003e299c:::benchmark_data_smithi187_45882_object67249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 149.48295068300001, 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26035086499999999, 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67250 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.790 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.51e-07 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251748+0000", 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6327000000000001e-05 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251766+0000", 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8689e-05 2024-09-02T10:04:58.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251772+0000", 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6409999999999999e-06 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251811+0000", 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9113000000000003e-05 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251818+0000", 2024-09-02T10:04:58.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2339999999999997e-06 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.500859+0000", 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.249040962 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.500962+0000", 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103015 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.501075+0000", 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113211 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.506329+0000", 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052531840000000002 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.506377+0000", 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7914999999999998e-05 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.510244+0000", 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038674730000000002 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.510281+0000", 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6504000000000002e-05 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.512072+0000", 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001791146 2024-09-02T10:04:58.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127423 2.0 2:06eddedb:::benchmark_data_smithi187_45882_object29672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.199335+0000", 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.535336032999993, 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0271604599999999, 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127423 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199335+0000", 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199335+0000", 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199337+0000", 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.564e-06 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199343+0000", 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7760000000000003e-06 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199354+0000", 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1321000000000001e-05 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199362+0000", 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6040000000000004e-06 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199422+0000", 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0145000000000001e-05 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199562+0000", 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013979699999999999 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199712+0000", 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015045799999999999 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.209645+0000", 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0099324600000000006 2024-09-02T10:04:58.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.217410+0000", 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077656410000000002 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232716+0000", 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015306024 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232726+0000", 2024-09-02T10:04:58.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3230000000000004e-06 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226479+0000", 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99375334900000001 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226496+0000", 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6997999999999999e-05 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127440 2.0 2.9f94cd30 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.515041869000001, 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0069557170000001, 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127440 2024-09-02T10:04:58.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.63e-07 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219633+0000", 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.818e-06 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219639+0000", 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8000000000000004e-06 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219645+0000", 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3980000000000004e-06 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232802+0000", 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013156805000000001 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232812+0000", 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0394000000000001e-05 2024-09-02T10:04:58.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226575+0000", 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99376249500000002 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226585+0000", 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0543999999999999e-05 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127423 2.0 2.db7bb760 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.275687+0000", 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.458984212999994, 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95098331899999999, 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127423 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275687+0000", 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275687+0000", 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275689+0000", 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5829999999999999e-06 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275696+0000", 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1359999999999999e-06 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275707+0000", 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1024e-05 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275715+0000", 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9750000000000007e-06 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279013+0000", 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003297688 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279020+0000", 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0790000000000003e-06 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226644+0000", 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94762439200000004 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226671+0000", 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6441999999999999e-05 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.814 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127440 2.0 2.9f94cd30 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:04:58.814 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:04:58.814 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.458940647000006, 2024-09-02T10:04:58.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95100121400000004, 2024-09-02T10:04:58.814 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.814 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.814 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:04:58.814 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.814 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.814 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127440 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:04:58.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6499999999999999e-07 2024-09-02T10:04:58.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275734+0000", 2024-09-02T10:04:58.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2299999999999998e-06 2024-09-02T10:04:58.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275740+0000", 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8970000000000002e-06 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275745+0000", 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.814e-06 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279036+0000", 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003290837 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279041+0000", 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4769999999999997e-06 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226722+0000", 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94768102099999996 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226732+0000", 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0473e-05 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127464 2.0 2.e26bb170 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.277404+0000", 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.457267916999996, 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94936489599999996, 2024-09-02T10:04:58.819 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127464 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277404+0000", 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277404+0000", 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277405+0000", 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4799999999999997e-07 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277410+0000", 2024-09-02T10:04:58.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0529999999999997e-06 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277419+0000", 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6420000000000002e-06 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277424+0000", 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.831e-06 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279299+0000", 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018751639999999999 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279303+0000", 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.143e-06 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226761+0000", 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94745774000000005 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226768+0000", 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3749999999999997e-06 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127473 2.0 2.e700b5b0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.298011+0000", 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.436660832000001, 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.92878800299999997, 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.824 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127473 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298011+0000", 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298011+0000", 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298012+0000", 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.412e-06 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298018+0000", 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9190000000000004e-06 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298030+0000", 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2262000000000001e-05 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298035+0000", 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8969999999999999e-06 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298101+0000", 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5538999999999997e-05 2024-09-02T10:04:58.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298112+0000", 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1017e-05 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226792+0000", 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.92868043 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226799+0000", 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5270000000000004e-06 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127507 2.0 2.51c134f0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.510346+0000", 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.224325688999997, 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.71648088300000001, 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.829 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127507 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510346+0000", 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510346+0000", 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510347+0000", 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2869999999999999e-06 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510353+0000", 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9290000000000003e-06 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510376+0000", 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3251000000000002e-05 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510384+0000", 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5129999999999999e-06 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510437+0000", 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3245000000000003e-05 2024-09-02T10:04:58.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510446+0000", 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9910000000000008e-06 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226821+0000", 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.716374493 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226827+0000", 2024-09-02T10:04:58.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1739999999999997e-06 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.8 157/153 [PushReplyOp(2:10f8d703:::benchmark_data_smithi187_45882_object31118:head)])", 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:46.348475+0000", 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 72.386196210999998, 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0031679010000001, 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:04:58.834 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348475+0000", 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348475+0000", 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348476+0000", 2024-09-02T10:04:58.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2900000000000002e-07 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348484+0000", 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.695e-06 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.265961+0000", 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91747688900000002 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.265975+0000", 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3981e-05 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.347454+0000", 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.081478748000000004 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.351643+0000", 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041896590000000001 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.837 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:141512 2.3 e169/92 2:c7378ca5:::benchmark_data_smithi187_45882_object43761:head v 169'8968, mlcod=169'8965)", 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:14.694189+0000", 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.040482713000003, 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.365733013, 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694189+0000", 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694189+0000", 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694190+0000", 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1209999999999999e-06 2024-09-02T10:04:58.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:04:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694197+0000", 2024-09-02T10:04:58.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5700000000000004e-06 2024-09-02T10:04:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056209+0000", 2024-09-02T10:04:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36201126900000002 2024-09-02T10:04:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056216+0000", 2024-09-02T10:04:58.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5390000000000002e-06 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056294+0000", 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7521000000000005e-05 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056309+0000", 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4691e-05 2024-09-02T10:04:58.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.059878+0000", 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003569582 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.059922+0000", 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3720000000000002e-05 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.844 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:156192 2.8 2:133c662a:::benchmark_data_smithi187_45882_object58441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:43.142777+0000", 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.591894317, 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.60071848299999997, 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:04:58.845 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 156192 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142777+0000", 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142777+0000", 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142779+0000", 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.655e-06 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142784+0000", 2024-09-02T10:04:58.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5350000000000004e-06 2024-09-02T10:04:58.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142798+0000", 2024-09-02T10:04:58.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3525e-05 2024-09-02T10:04:58.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142805+0000", 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4050000000000003e-06 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.147079+0000", 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042733989999999998 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.147202+0000", 2024-09-02T10:04:58.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012349799999999999 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,11", 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.147292+0000", 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9788000000000002e-05 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151073+0000", 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037812150000000001 2024-09-02T10:04:58.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.153573+0000", 2024-09-02T10:04:58.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025002380000000001 2024-09-02T10:04:58.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:04:58.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742725+0000", 2024-09-02T10:04:58.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.58915150900000002 2024-09-02T10:04:58.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:04:58.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742781+0000", 2024-09-02T10:04:58.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6039000000000003e-05 2024-09-02T10:04:58.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.743496+0000", 2024-09-02T10:04:58.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00071467700000000004 2024-09-02T10:04:58.852 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:156192 2.8 e188/153 ondisk, result = 0)", 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:43.151803+0000", 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.582868914000001, 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.59169890800000002, 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:04:58.853 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:04:58.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:04:58.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151803+0000", 2024-09-02T10:04:58.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:04:58.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151803+0000", 2024-09-02T10:04:58.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:04:58.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151804+0000", 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5960000000000001e-06 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151817+0000", 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2957e-05 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.740670+0000", 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.58885322699999998 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.740675+0000", 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7310000000000001e-06 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.856 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742713+0000", 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020379510000000001 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742719+0000", 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2979999999999997e-06 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.743501+0000", 2024-09-02T10:04:58.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00078214799999999998 2024-09-02T10:04:58.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.858 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.858 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:04:58.858 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:04:58.858 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:04:58.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:58.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:58.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:04:58.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:59.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:59.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:59.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:04:59.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:59.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:59.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:04:59.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:59 smithi151 ceph-mon[16639]: osdmap e200: 12 total, 12 up, 11 in 2024-09-02T10:04:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:59 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:04:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:59 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:04:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:59 smithi151 ceph-mon[16639]: 2.2 scrub starts 2024-09-02T10:04:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 9151/99249 objects degraded (9.220%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:04:59 smithi151 ceph-mon[16639]: osdmap e201: 12 total, 12 up, 11 in 2024-09-02T10:04:59.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:04:59.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:59.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:59.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:04:59.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:59 smithi162 ceph-mon[20279]: osdmap e200: 12 total, 12 up, 11 in 2024-09-02T10:04:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:59 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:04:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:59 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:04:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:59 smithi162 ceph-mon[20279]: 2.2 scrub starts 2024-09-02T10:04:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 9151/99249 objects degraded (9.220%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:04:59 smithi162 ceph-mon[20279]: osdmap e201: 12 total, 12 up, 11 in 2024-09-02T10:04:59.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:04:59.388 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:59.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:04:59.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:04:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:59 smithi187 ceph-mon[20072]: osdmap e200: 12 total, 12 up, 11 in 2024-09-02T10:04:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:59 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:04:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:59 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:04:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:59 smithi187 ceph-mon[20072]: 2.2 scrub starts 2024-09-02T10:04:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 9151/99249 objects degraded (9.220%), 4 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:04:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:04:59 smithi187 ceph-mon[20072]: osdmap e201: 12 total, 12 up, 11 in 2024-09-02T10:04:59.513 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:59.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:04:59.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:04:59.638 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:04:59.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:59.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:04:59.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:04:59.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:04:59.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:04:59.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:59.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:04:59.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:04:59.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:04:59.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:04:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:04:59.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:04:59.992 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:00.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:00.059 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:05:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:00.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:00.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:00.090 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:00.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:00 smithi151 ceph-mon[16639]: 2.2 scrub ok 2024-09-02T10:05:00.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:00 smithi151 ceph-mon[16639]: pgmap v501: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 1260/97086 objects degraded (1.298%); 3980/97086 objects misplaced (4.099%); 6.3 MiB/s, 250 objects/s recovering 2024-09-02T10:05:00.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1019141929' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:00.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:00.192 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:00.192 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:00.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:00.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:00.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:00.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:00.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:00 smithi162 ceph-mon[20279]: 2.2 scrub ok 2024-09-02T10:05:00.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:00 smithi162 ceph-mon[20279]: pgmap v501: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 1260/97086 objects degraded (1.298%); 3980/97086 objects misplaced (4.099%); 6.3 MiB/s, 250 objects/s recovering 2024-09-02T10:05:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1019141929' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:00.352 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:00 smithi187 ceph-mon[20072]: 2.2 scrub ok 2024-09-02T10:05:00.352 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:00 smithi187 ceph-mon[20072]: pgmap v501: 37 pgs: 1 active+recovering+undersized+remapped, 1 active+undersized+remapped, 3 active+recovering+undersized+degraded+remapped, 32 active+clean; 2.0 GiB data, 7.9 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 465 op/s; 1260/97086 objects degraded (1.298%); 3980/97086 objects misplaced (4.099%); 6.3 MiB/s, 250 objects/s recovering 2024-09-02T10:05:00.352 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1019141929' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:00.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:00.440 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:00.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:00.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:00.564 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:00.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:00.614 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:05:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:05:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165288 2.5 2:acb75efc:::benchmark_data_smithi187_45882_object67537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.475324+0000", 2024-09-02T10:05:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.13749254299999999, 2024-09-02T10:05:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.137526763, 2024-09-02T10:05:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165288 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.475324+0000", 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.475324+0000", 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.475325+0000", 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6700000000000001e-06 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.475331+0000", 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5650000000000001e-06 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.475345+0000", 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4105e-05 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.475352+0000", 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0520000000000004e-06 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.475406+0000", 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4565999999999999e-05 2024-09-02T10:05:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.475543+0000", 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013646400000000001 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.475673+0000", 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000130202 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.479535+0000", 2024-09-02T10:05:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0038620809999999998 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.484458+0000", 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0049232390000000003 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165290 2.5 2:a6925835:::benchmark_data_smithi187_45882_object67539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.480174+0000", 2024-09-02T10:05:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.13264171599999999, 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.132724281, 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165290 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.480174+0000", 2024-09-02T10:05:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.480174+0000", 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.480176+0000", 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3939999999999999e-06 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.480181+0000", 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.699e-06 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.480194+0000", 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2480000000000001e-05 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.480202+0000", 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4870000000000002e-06 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.480249+0000", 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6088000000000003e-05 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.480335+0000", 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6735000000000002e-05 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.480429+0000", 2024-09-02T10:05:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3524999999999996e-05 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.484529+0000", 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041005089999999996 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.484700+0000", 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017087 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165300 2.f 2:f6f5f760:::benchmark_data_smithi187_45882_object67549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.540285+0000", 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.072531417000000001, 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.072653370999999994, 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165300 2024-09-02T10:05:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.540285+0000", 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.540285+0000", 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.540287+0000", 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9369999999999998e-06 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.540295+0000", 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2039999999999997e-06 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.540309+0000", 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4304000000000001e-05 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.540317+0000", 2024-09-02T10:05:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4649999999999997e-06 2024-09-02T10:05:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.540376+0000", 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9256999999999999e-05 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.540509+0000", 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013326399999999999 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:05:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.540645+0000", 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013616 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.544413+0000", 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037678630000000002 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.544623+0000", 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000209404 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165303 2.f 2:f4ed2529:::benchmark_data_smithi187_45882_object67552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.556780+0000", 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.056035965, 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.056196272999999998, 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165303 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.556780+0000", 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.556780+0000", 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.556782+0000", 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.513e-06 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.556789+0000", 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0659999999999998e-06 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.556806+0000", 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7014999999999999e-05 2024-09-02T10:05:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.556813+0000", 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8229999999999997e-06 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.556889+0000", 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6370000000000002e-05 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.557041+0000", 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000152039 2024-09-02T10:05:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.557185+0000", 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014430199999999999 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.561950+0000", 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047652170000000004 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.562015+0000", 2024-09-02T10:05:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4702000000000005e-05 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165242 2.5 2:ac5dab90:::benchmark_data_smithi187_45882_object67491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.402390+0000", 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.21042651200000001, 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21062415400000001, 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165242 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.402390+0000", 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.402390+0000", 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.402391+0000", 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7290000000000001e-06 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.402398+0000", 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9999999999999999e-06 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.402416+0000", 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7227999999999998e-05 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.402423+0000", 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8620000000000003e-06 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.402488+0000", 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4962999999999994e-05 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.402623+0000", 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013460699999999999 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.402705+0000", 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1876e-05 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.407067+0000", 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043623239999999999 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.409262+0000", 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021951470000000002 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165246 2.f 2:f40b145a:::benchmark_data_smithi187_45882_object67495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.404073+0000", 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.208742909, 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.208977144, 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165246 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.404073+0000", 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.404073+0000", 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.404074+0000", 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.54e-07 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.404079+0000", 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6689999999999997e-06 2024-09-02T10:05:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.404090+0000", 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1531e-05 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.404096+0000", 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3000000000000001e-06 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.404152+0000", 2024-09-02T10:05:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6014999999999998e-05 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.404274+0000", 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122352 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.404391+0000", 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117067 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.408070+0000", 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036792330000000001 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.409046+0000", 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00097625800000000003 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165270 2.f 2:f43d1519:::benchmark_data_smithi187_45882_object67519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.445398+0000", 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.16741772999999999, 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16769065, 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165270 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.445398+0000", 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.445398+0000", 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.445401+0000", 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.199e-06 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.445408+0000", 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5100000000000001e-06 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.445428+0000", 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9479000000000001e-05 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.445437+0000", 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3579999999999992e-06 2024-09-02T10:05:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.445491+0000", 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3718999999999998e-05 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.445574+0000", 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2960999999999994e-05 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.445644+0000", 2024-09-02T10:05:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0617000000000005e-05 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449515+0000", 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0038712659999999999 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449609+0000", 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3160999999999998e-05 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165272 2.5 2:a01e39d1:::benchmark_data_smithi187_45882_object67521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.449335+0000", 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.163480657, 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.163789569, 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165272 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449335+0000", 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449335+0000", 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449337+0000", 2024-09-02T10:05:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.968e-06 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449345+0000", 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6369999999999999e-06 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449359+0000", 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4276e-05 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449368+0000", 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0329999999999997e-06 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449434+0000", 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5862000000000001e-05 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449566+0000", 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013199800000000001 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.449689+0000", 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122581 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.457542+0000", 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0078531269999999997 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.461049+0000", 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035066979999999999 2024-09-02T10:05:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165273 2.5 2:a940d75b:::benchmark_data_smithi187_45882_object67522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.450520+0000", 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.162295679, 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16264183300000001, 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165273 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.450520+0000", 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.450520+0000", 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.450521+0000", 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5899999999999996e-07 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.450524+0000", 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1930000000000002e-06 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.450533+0000", 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8789999999999992e-06 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.450538+0000", 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7230000000000004e-06 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.453964+0000", 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034260470000000002 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.454046+0000", 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2587000000000001e-05 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.454120+0000", 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4010000000000005e-05 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.461016+0000", 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0068957580000000001 2024-09-02T10:05:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465212+0000", 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041958159999999998 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165276 2.5 2:a1b41cb1:::benchmark_data_smithi187_45882_object67525:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.452845+0000", 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.15997061500000001, 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16035406599999999, 2024-09-02T10:05:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165276 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.452845+0000", 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.452845+0000", 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.452847+0000", 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1200000000000001e-06 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.452851+0000", 2024-09-02T10:05:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.814e-06 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.452862+0000", 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0922e-05 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.452868+0000", 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4990000000000002e-06 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.457567+0000", 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046986900000000002 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.457649+0000", 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2502000000000004e-05 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.457726+0000", 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6730999999999994e-05 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.461170+0000", 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034447359999999999 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465277+0000", 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041062329999999999 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:165281 2.f 2:fadd1c60:::benchmark_data_smithi187_45882_object67530:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e202)", 2024-09-02T10:05:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:00.465007+0000", 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.147809047, 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.148233739, 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 165281 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465007+0000", 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465007+0000", 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465008+0000", 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3799999999999996e-07 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465013+0000", 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3539999999999999e-06 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465023+0000", 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0037000000000001e-05 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465030+0000", 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.832e-06 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465080+0000", 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9982999999999999e-05 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465164+0000", 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3597999999999995e-05 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.465286+0000", 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012256699999999999 2024-09-02T10:05:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.469038+0000", 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037516149999999998 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:00.470021+0000", 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00098278100000000006 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:05:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 11 2024-09-02T10:05:00.674 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:05:00.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:00.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:00.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:00.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:00.826 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:00.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:00.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:00.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:00.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:00.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:01.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:01.097 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:01.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:01.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:01.222 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:01.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:01 smithi187 ceph-mon[20072]: 2.4 scrub starts 2024-09-02T10:05:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:01 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:05:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1019141929' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:01 smithi187 ceph-mon[20072]: osdmap e202: 12 total, 12 up, 11 in 2024-09-02T10:05:01.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:01.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:01 smithi162 ceph-mon[20279]: 2.4 scrub starts 2024-09-02T10:05:01.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:01 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:05:01.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1019141929' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:01.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:01 smithi162 ceph-mon[20279]: osdmap e202: 12 total, 12 up, 11 in 2024-09-02T10:05:01.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:01.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:01 smithi151 ceph-mon[16639]: 2.4 scrub starts 2024-09-02T10:05:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:01 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:05:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1019141929' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:01 smithi151 ceph-mon[16639]: osdmap e202: 12 total, 12 up, 11 in 2024-09-02T10:05:01.452 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:01.483 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:01.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:01.484 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:01.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:05:01.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:01.607 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:01.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:01.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:01.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:01.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:01.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:01.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:01.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:01.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:01.924 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:01.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:02.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:02.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:02.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:02.131 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:02.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:02.132 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:02.233 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:02.269 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:02.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:02 smithi187 ceph-mon[20072]: 2.4 scrub ok 2024-09-02T10:05:02.306 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:02 smithi187 ceph-mon[20072]: pgmap v503: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 32 active+clean; 1.9 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 749 B/s rd, 24 KiB/s wr, 414 op/s; 2229/94602 objects degraded (2.356%); 137/94602 objects misplaced (0.145%); 4.2 MiB/s, 273 objects/s recovering 2024-09-02T10:05:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:02 smithi162 ceph-mon[20279]: 2.4 scrub ok 2024-09-02T10:05:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:02 smithi162 ceph-mon[20279]: pgmap v503: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 32 active+clean; 1.9 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 749 B/s rd, 24 KiB/s wr, 414 op/s; 2229/94602 objects degraded (2.356%); 137/94602 objects misplaced (0.145%); 4.2 MiB/s, 273 objects/s recovering 2024-09-02T10:05:02.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:02.398 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:02.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:02.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:02.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:02 smithi151 ceph-mon[16639]: 2.4 scrub ok 2024-09-02T10:05:02.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:02 smithi151 ceph-mon[16639]: pgmap v503: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 active+undersized+remapped, 32 active+clean; 1.9 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 749 B/s rd, 24 KiB/s wr, 414 op/s; 2229/94602 objects degraded (2.356%); 137/94602 objects misplaced (0.145%); 4.2 MiB/s, 273 objects/s recovering 2024-09-02T10:05:02.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:02.505 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:05:02.506 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:05:02.506 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:05:02.506 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:05:02.506 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:05:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:02.528 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:02.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:02.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:02.660 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:02.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:02.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:02.794 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:02.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:02.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:02.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:02.924 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:03.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:03.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:03.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:03.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:03.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:03.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:03.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:03.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:05:03.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:03.344 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:03.345 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:03.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:03.467 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:03.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:03.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:03.603 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:03.604 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:03.613 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:03.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:03.728 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:03.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:03.830 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:03.852 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:03.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:03.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:03.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:03.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:04.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:04.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:04.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:04.210 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:05:04.210 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:05:04.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:05:04.210 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:05:04.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:36428 2.b 2:da93149a:::benchmark_data_smithi187_45882_object36427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:02.079845+0000", 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 182.12821801199999, 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22416934999999999, 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36428 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.211 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079845+0000", 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079845+0000", 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079846+0000", 2024-09-02T10:05:04.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3900000000000004e-07 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079867+0000", 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1290000000000001e-05 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079878+0000", 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0677e-05 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079887+0000", 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1109999999999997e-06 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.081179+0000", 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001292444 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.081311+0000", 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000131855 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.081411+0000", 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9522e-05 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.082483+0000", 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010728630000000001 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.143788+0000", 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.061304751999999997 2024-09-02T10:05:04.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.303899+0000", 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16011112 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.303951+0000", 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1687999999999997e-05 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.304014+0000", 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3288999999999997e-05 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:37153 2.f 2:f8e1d5aa:::benchmark_data_smithi187_45882_object37152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:02.820299+0000", 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 181.387764299, 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.214813331, 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.217 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 37153 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.820299+0000", 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.820299+0000", 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.820300+0000", 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1659999999999999e-06 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.820319+0000", 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9160999999999998e-05 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.820332+0000", 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2574e-05 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.820337+0000", 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4569999999999996e-06 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.820381+0000", 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4323999999999998e-05 2024-09-02T10:05:04.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.820477+0000", 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5212999999999998e-05 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.820565+0000", 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8599999999999999e-05 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.821970+0000", 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001405143 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.822087+0000", 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116661 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.035008+0000", 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.212920994 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.035069+0000", 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0690999999999999e-05 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.035112+0000", 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3346999999999997e-05 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:54932 2.b 2:d9ab64ae:::benchmark_data_smithi187_45882_object54931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:05:04.223 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.160766+0000", 2024-09-02T10:05:04.224 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 167.04729667500001, 2024-09-02T10:05:04.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26274712500000003, 2024-09-02T10:05:04.224 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.224 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.224 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.224 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 54932 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160766+0000", 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160766+0000", 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160767+0000", 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2e-07 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160785+0000", 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8139e-05 2024-09-02T10:05:04.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160792+0000", 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5109999999999997e-06 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160797+0000", 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2030000000000002e-06 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160835+0000", 2024-09-02T10:05:04.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8591999999999999e-05 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160936+0000", 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100828 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.161018+0000", 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1940999999999994e-05 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162378+0000", 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001359968 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162459+0000", 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0600999999999996e-05 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423449+0000", 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26099011300000002 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423482+0000", 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.379e-05 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423513+0000", 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0919e-05 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:54932 2.b e113/92 ondisk, result = 0)", 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 167.04517043999999, 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26062987399999998, 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4199999999999995e-07 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162896+0000", 2024-09-02T10:05:04.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.23e-06 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423394+0000", 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26049795999999997 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423399+0000", 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4360000000000004e-06 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.233 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423440+0000", 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1412999999999998e-05 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423445+0000", 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1499999999999998e-06 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423522+0000", 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7043000000000001e-05 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.234 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:55061 2.6 e113/92 2:670fe1b2:::benchmark_data_smithi187_45882_object55060:head v 113'3388, mlcod=113'3385)", 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.479402+0000", 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 166.72866121000001, 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33103881099999999, 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479402+0000", 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479402+0000", 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479403+0000", 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3600000000000002e-07 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479437+0000", 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4811999999999998e-05 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809044+0000", 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32960664000000001 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809050+0000", 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7679999999999997e-06 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809098+0000", 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7859999999999999e-05 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809112+0000", 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4579000000000001e-05 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.810407+0000", 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012949649999999999 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.810441+0000", 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3250999999999997e-05 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55063 2.b 2:d2962897:::benchmark_data_smithi187_45882_object55062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.481328+0000", 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 166.72673483200001, 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.292836235, 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.239 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55063 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481328+0000", 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481328+0000", 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481329+0000", 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0500000000000003e-07 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481351+0000", 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2506000000000001e-05 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481361+0000", 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0105999999999999e-05 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481366+0000", 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8119999999999998e-06 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481408+0000", 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1689e-05 2024-09-02T10:05:04.242 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481498+0000", 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0192000000000007e-05 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481585+0000", 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7214999999999998e-05 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.482940+0000", 2024-09-02T10:05:04.243 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013545709999999999 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483445+0000", 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050479699999999997 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774099+0000", 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29065394700000002 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774135+0000", 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6665999999999998e-05 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774164+0000", 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9028999999999999e-05 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:55063 2.b e113/92 ondisk, result = 0)", 2024-09-02T10:05:04.245 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.483081+0000", 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 166.72498185699999, 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29108941100000002, 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483081+0000", 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483081+0000", 2024-09-02T10:05:04.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483082+0000", 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6700000000000003e-07 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483085+0000", 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3009999999999998e-06 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.247 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774042+0000", 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29095665500000001 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774047+0000", 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8629999999999996e-06 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774087+0000", 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9206999999999999e-05 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.248 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774093+0000", 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6710000000000002e-06 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774170+0000", 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7146999999999998e-05 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.249 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55077 2.b 2:d13d7ad6:::benchmark_data_smithi187_45882_object55076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.497835+0000", 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 166.710227637, 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29033786299999997, 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55077 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.250 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497835+0000", 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497835+0000", 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497836+0000", 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.06e-07 2024-09-02T10:05:04.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497866+0000", 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9967999999999998e-05 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497875+0000", 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9090000000000006e-06 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497882+0000", 2024-09-02T10:05:04.252 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1409999999999999e-06 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497929+0000", 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6683e-05 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498008+0000", 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9110000000000007e-05 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.253 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498101+0000", 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3379000000000002e-05 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.499478+0000", 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001376637 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.500299+0000", 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00082144399999999995 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.254 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.788094+0000", 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28779512299999999 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.788131+0000", 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6924000000000001e-05 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.788173+0000", 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1838999999999997e-05 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.255 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55083 2.5 2:a3eaecaa:::benchmark_data_smithi187_45882_object55082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.511889+0000", 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 166.69617349500001, 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.254154883, 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.256 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55083 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511889+0000", 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511889+0000", 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.257 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511890+0000", 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.046e-06 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511910+0000", 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9162999999999999e-05 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.258 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511921+0000", 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0995000000000001e-05 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511926+0000", 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2650000000000004e-06 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511967+0000", 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1461999999999997e-05 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.259 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.512047+0000", 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9907999999999995e-05 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.512138+0000", 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0909999999999995e-05 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.513495+0000", 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001356307 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.260 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.513828+0000", 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00033362600000000001 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765934+0000", 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25210628000000002 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765996+0000", 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1614999999999999e-05 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.261 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.766044+0000", 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8306000000000001e-05 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55086 2.f 2:f2dac436:::benchmark_data_smithi187_45882_object55085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.518890+0000", 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 166.68917306, 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25559369300000001, 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.262 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55086 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518890+0000", 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.263 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518890+0000", 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518891+0000", 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8199999999999998e-07 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518909+0000", 2024-09-02T10:05:04.264 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8649000000000001e-05 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518920+0000", 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0560000000000001e-05 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518925+0000", 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0830000000000003e-06 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.265 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518974+0000", 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8979e-05 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.519056+0000", 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1796000000000001e-05 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.519137+0000", 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1422999999999996e-05 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.266 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.267 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:04.268 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.268 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520575+0000", 2024-09-02T10:05:04.268 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014377159999999999 2024-09-02T10:05:04.268 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.268 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.268 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.268 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520789+0000", 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00021378200000000001 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774432+0000", 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25364359800000003 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774462+0000", 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9416000000000001e-05 2024-09-02T10:05:04.269 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774484+0000", 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1809000000000001e-05 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:55083 2.5 e113/92 ondisk, result = 0)", 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.520803+0000", 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 166.68725980799999, 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24524790699999999, 2024-09-02T10:05:04.270 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520803+0000", 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520803+0000", 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.271 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520804+0000", 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1200000000000003e-07 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520807+0000", 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.523e-06 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765868+0000", 2024-09-02T10:05:04.272 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.245060377 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765872+0000", 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6949999999999999e-06 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765920+0000", 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7593000000000003e-05 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.273 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765929+0000", 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6260000000000001e-06 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.766051+0000", 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122481 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:67930 2.f 2:f0dce748:::benchmark_data_smithi187_45882_object67929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-02T10:05:04.274 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:29.880675+0000", 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 154.32738788699999, 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21552980599999999, 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 67930 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.275 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.880675+0000", 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.880675+0000", 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.880676+0000", 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7900000000000007e-07 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.880694+0000", 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8428e-05 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.880704+0000", 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9350000000000006e-06 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.880709+0000", 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7060000000000003e-06 2024-09-02T10:05:04.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.880751+0000", 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2024000000000003e-05 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.880849+0000", 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7643e-05 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.880935+0000", 2024-09-02T10:05:04.278 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6416999999999996e-05 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.883074+0000", 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021385279999999998 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.886310+0000", 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032364759999999998 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.279 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.096092+0000", 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20978184899999999 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.096142+0000", 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9882000000000001e-05 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.096205+0000", 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2939000000000002e-05 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.280 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:88937 2.b 2:df41f0dd:::benchmark_data_smithi187_45882_object88936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:54.396394+0000", 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 129.81166938300001, 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21625391299999999, 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.281 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 88937 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.396394+0000", 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.396394+0000", 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.396396+0000", 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.199e-06 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.396421+0000", 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5199999999999999e-05 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.396441+0000", 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9593999999999999e-05 2024-09-02T10:05:04.283 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.396450+0000", 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9389999999999992e-06 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.396528+0000", 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7435999999999995e-05 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.284 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.396661+0000", 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000132977 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.396797+0000", 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013648299999999999 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.400010+0000", 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032121659999999998 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.400561+0000", 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00055166299999999998 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.612580+0000", 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.212018702 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.612617+0000", 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6647000000000003e-05 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.286 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:54.612647+0000", 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0907000000000001e-05 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:97743 2.5 2:a58175a7:::benchmark_data_smithi187_45882_object97742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:05.424082+0000", 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 118.78398087799999, 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.60300915200000005, 2024-09-02T10:05:04.287 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 97743 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424082+0000", 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424082+0000", 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424084+0000", 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6160000000000001e-06 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424109+0000", 2024-09-02T10:05:04.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5466999999999999e-05 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424122+0000", 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2765999999999999e-05 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424129+0000", 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2599999999999999e-06 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424197+0000", 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7782999999999998e-05 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424308+0000", 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000110645 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424453+0000", 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014499099999999999 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427671+0000", 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032188590000000001 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.555569+0000", 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.127897921 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027018+0000", 2024-09-02T10:05:04.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.47144856699999999 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027061+0000", 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3092e-05 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027091+0000", 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0185e-05 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.293 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:97743 2.5 e137/92 ondisk, result = 0)", 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:05.427175+0000", 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 118.780887486, 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.59992140800000004, 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427175+0000", 2024-09-02T10:05:04.294 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427175+0000", 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427177+0000", 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.511e-06 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427183+0000", 2024-09-02T10:05:04.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4629999999999998e-06 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.026965+0000", 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.59978128600000002 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.026969+0000", 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1810000000000003e-06 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027007+0000", 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8557999999999999e-05 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027013+0000", 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7649999999999999e-06 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027097+0000", 2024-09-02T10:05:04.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3644000000000001e-05 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:116089 2.b 2:d15ff9ee:::benchmark_data_smithi187_45882_object18338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:28.305342+0000", 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 95.902720736999996, 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21823024899999999, 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.298 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 116089 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.305342+0000", 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.305342+0000", 2024-09-02T10:05:04.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.305344+0000", 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3230000000000001e-06 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.305350+0000", 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0299999999999999e-06 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.305363+0000", 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3655999999999999e-05 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.305369+0000", 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.02e-06 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.305435+0000", 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5623999999999994e-05 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.305544+0000", 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108693 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.305647+0000", 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103552 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.308334+0000", 2024-09-02T10:05:04.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026865890000000001 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.308479+0000", 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000145406 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.523476+0000", 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.214996463 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.523532+0000", 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6561999999999998e-05 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:28.523572+0000", 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0330999999999997e-05 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:117060 2.f 2:f36bc4f3:::benchmark_data_smithi187_45882_object19309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-02T10:05:04.304 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:29.622070+0000", 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 94.585993196000004, 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.266536263, 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 117060 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622070+0000", 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622070+0000", 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622071+0000", 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.001e-06 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622075+0000", 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1509999999999997e-06 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622084+0000", 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8580000000000006e-06 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622088+0000", 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5970000000000002e-06 2024-09-02T10:05:04.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622136+0000", 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7803999999999998e-05 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622256+0000", 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119537 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:05:04.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622368+0000", 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011192900000000001 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.625537+0000", 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003168911 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.637552+0000", 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012015965999999999 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.888531+0000", 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25097857000000001 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.888578+0000", 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7426000000000003e-05 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.888606+0000", 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7512999999999999e-05 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.310 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:163327 2.f 2:f9a47073:::benchmark_data_smithi187_45882_object65576:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:56.455190+0000", 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 7.7528730829999999, 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.422262621, 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.311 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 163327 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455190+0000", 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455190+0000", 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455192+0000", 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.889e-06 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455200+0000", 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7710000000000006e-06 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455216+0000", 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6943999999999999e-05 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455224+0000", 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1550000000000001e-06 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455280+0000", 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6443999999999997e-05 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455368+0000", 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8307999999999995e-05 2024-09-02T10:05:04.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455446+0000", 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7698e-05 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.459097+0000", 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036506360000000001 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.459525+0000", 2024-09-02T10:05:04.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00042830199999999998 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877381+0000", 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41785591900000002 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877420+0000", 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8832999999999999e-05 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877452+0000", 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2722000000000003e-05 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:164051 2.b 2:dd30ecaf:::benchmark_data_smithi187_45882_object66300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:58.383444+0000", 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.8246185690000001, 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26956966300000001, 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:04.317 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 164051 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383444+0000", 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383444+0000", 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383446+0000", 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.347e-06 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383452+0000", 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.088e-06 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383469+0000", 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7153999999999999e-05 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383478+0000", 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9109999999999999e-06 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383537+0000", 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8687999999999998e-05 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383632+0000", 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5619999999999996e-05 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383712+0000", 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9435000000000001e-05 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387376+0000", 2024-09-02T10:05:04.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003664488 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387533+0000", 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015662400000000001 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652930+0000", 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26539721100000002 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:04.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652964+0000", 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4489999999999997e-05 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.653014+0000", 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9607000000000001e-05 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:164051 2.b e200/92 ondisk, result = 0)", 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.820396283, 2024-09-02T10:05:04.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26535247699999998, 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:04.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:05:04.325 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:04 smithi187 ceph-mon[20072]: pgmap v504: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.8 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 2229/88737 objects degraded (2.512%); 137/88737 objects misplaced (0.154%); 2.9 MiB/s, 216 objects/s recovering 2024-09-02T10:05:04.325 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2229/88737 objects degraded (2.512%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:04.325 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:04 smithi187 ceph-mon[20072]: osdmap e203: 12 total, 12 up, 11 in 2024-09-02T10:05:04.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:04.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:04.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1799999999999995e-07 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387670+0000", 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8219999999999998e-06 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652865+0000", 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26519509200000002 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652872+0000", 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3509999999999996e-06 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652921+0000", 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8124999999999997e-05 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652926+0000", 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.682e-06 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:04.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:04.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:04.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.653019+0000", 2024-09-02T10:05:04.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2886999999999993e-05 2024-09-02T10:05:04.328 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.328 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.328 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.328 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:04.328 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:04.328 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:05:04.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:04.360 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:04.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:04 smithi151 ceph-mon[16639]: pgmap v504: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.8 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 2229/88737 objects degraded (2.512%); 137/88737 objects misplaced (0.154%); 2.9 MiB/s, 216 objects/s recovering 2024-09-02T10:05:04.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2229/88737 objects degraded (2.512%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:04.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:04 smithi151 ceph-mon[16639]: osdmap e203: 12 total, 12 up, 11 in 2024-09-02T10:05:04.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:04.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:04.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:04.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:04 smithi162 ceph-mon[20279]: pgmap v504: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.8 GiB data, 7.5 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 2229/88737 objects degraded (2.512%); 137/88737 objects misplaced (0.154%); 2.9 MiB/s, 216 objects/s recovering 2024-09-02T10:05:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2229/88737 objects degraded (2.512%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:04 smithi162 ceph-mon[20279]: osdmap e203: 12 total, 12 up, 11 in 2024-09-02T10:05:04.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:04.636 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:04.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:04.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:04.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:04.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:04.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:04.888 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:04.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:04.991 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:05.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:05.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:05.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:05.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:05.139 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:05.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:05.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:05.265 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:05.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:05.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:05.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:05.382 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:05:05.389 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:05.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:05.397 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:05.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:05 smithi151 ceph-mon[16639]: 2.5 scrub starts 2024-09-02T10:05:05.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:05.565 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:05 smithi187 ceph-mon[20072]: 2.5 scrub starts 2024-09-02T10:05:05.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:05 smithi162 ceph-mon[20279]: 2.5 scrub starts 2024-09-02T10:05:05.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:05.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:05.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:05.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:05.727 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:05.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:05.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:05.866 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:05.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:05.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:05.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:05.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:06.024 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:06.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:06.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:05:06.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:05:06.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:05:06.114 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:05:06.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:06.148 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:06.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:06.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:06.152 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:05:06.152 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:05:06.185 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-09-02T10:05:06.194 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:05:06.194 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:05:06.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:05:06.234 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:05:06.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:06.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:05:06.271 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:05:06.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:06.277 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:06.279 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:05:06.279 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:05:06.279 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:06.279 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:169471 2.e e204/198)", 2024-09-02T10:05:06.279 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:06.275362+0000", 2024-09-02T10:05:06.279 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.002607687, 2024-09-02T10:05:06.279 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026246289999999999, 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:06.275362+0000", 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:06.275362+0000", 2024-09-02T10:05:06.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:06.275363+0000", 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3099999999999997e-07 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:06.275369+0000", 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4999999999999996e-06 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:06.281 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:06.275386+0000", 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6563999999999999e-05 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:06.275393+0000", 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7220000000000002e-06 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:05:06.282 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:05:06.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:06.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:06.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:06 smithi151 ceph-mon[16639]: 2.5 scrub ok 2024-09-02T10:05:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:06 smithi151 ceph-mon[16639]: pgmap v506: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 1.7 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 2229/85968 objects degraded (2.593%); 137/85968 objects misplaced (0.159%); 2.7 MiB/s, 111 objects/s recovering 2024-09-02T10:05:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3630144979' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:05:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:06 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:05:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:06 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:05:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:06 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:05:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3218697847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:06.504 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:05:06.504 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:05:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:06 smithi187 ceph-mon[20072]: 2.5 scrub ok 2024-09-02T10:05:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:06 smithi187 ceph-mon[20072]: pgmap v506: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 1.7 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 2229/85968 objects degraded (2.593%); 137/85968 objects misplaced (0.159%); 2.7 MiB/s, 111 objects/s recovering 2024-09-02T10:05:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3630144979' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:05:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:06 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:05:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:06 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:05:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:06 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:05:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3218697847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:06.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:06.529 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:06.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:06.541 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:05:06.541 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:05:06.575 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:05:06.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:05:06.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:06 smithi162 ceph-mon[20279]: 2.5 scrub ok 2024-09-02T10:05:06.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:06 smithi162 ceph-mon[20279]: pgmap v506: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 1.7 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 2229/85968 objects degraded (2.593%); 137/85968 objects misplaced (0.159%); 2.7 MiB/s, 111 objects/s recovering 2024-09-02T10:05:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3630144979' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:05:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:06 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:05:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:06 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:05:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:06 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:05:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3218697847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:06.617 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:05:06.617 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:05:06.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:06.654 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:05:06.654 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:05:06.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:06.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:06.685 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:05:06.686 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:05:06.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:06.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:06.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:06.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:06.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:06.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:06.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:06.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:05:07.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:07.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:07.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:07.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:07.182 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:07.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:07.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:07.194 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:07 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:05:07.194 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:07 smithi187 ceph-mon[20072]: osdmap e204: 12 total, 12 up, 11 in 2024-09-02T10:05:07.195 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:07 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:05:07.195 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:07 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:05:07.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:07.286 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:07.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:07.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:07.374 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-02T10:05:07.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:05:07.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:05:07.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:07.444 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:07 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:05:07.444 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:07 smithi151 ceph-mon[16639]: osdmap e204: 12 total, 12 up, 11 in 2024-09-02T10:05:07.444 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:07 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:05:07.444 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:07 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:05:07.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:07.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:07.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:07.585 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:07.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:07.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:07.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:07 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:05:07.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:07 smithi162 ceph-mon[20279]: osdmap e204: 12 total, 12 up, 11 in 2024-09-02T10:05:07.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:07 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:05:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:07 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:05:07.687 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:07.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:07.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:07.713 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:07.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:07.837 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:05:07.837 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:05:07.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:07.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:07.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:07.930 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:07.933 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:07.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:08.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:08.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:08.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:08.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:08.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:08.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:08.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:08.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:08.257 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:08.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:08.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:08 smithi151 ceph-mon[16639]: pgmap v508: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 1.7 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 2229/81339 objects degraded (2.740%); 3.3 MiB/s, 171 objects/s recovering 2024-09-02T10:05:08.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:08.385 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:08.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:08.482 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:08 smithi187 ceph-mon[20072]: pgmap v508: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 1.7 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 2229/81339 objects degraded (2.740%); 3.3 MiB/s, 171 objects/s recovering 2024-09-02T10:05:08.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:08.509 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:08.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:08.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:05:08.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:08 smithi162 ceph-mon[20279]: pgmap v508: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 1.7 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 666 op/s; 2229/81339 objects degraded (2.740%); 3.3 MiB/s, 171 objects/s recovering 2024-09-02T10:05:08.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:08.642 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:08.643 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:08.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:08.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:08.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:08.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:08.860 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:08.876 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:08.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:08.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:09.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:09.030 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:09.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:09.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:09.158 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:09.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2229/81339 objects degraded (2.740%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:09 smithi187 ceph-mon[20072]: osdmap e205: 12 total, 12 up, 11 in 2024-09-02T10:05:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3899993664' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:05:09.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:09.283 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:09.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:09.302 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:05:09.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2229/81339 objects degraded (2.740%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:09.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:09 smithi151 ceph-mon[16639]: osdmap e205: 12 total, 12 up, 11 in 2024-09-02T10:05:09.302 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3899993664' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:05:09.303 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:05:09.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:09.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:09.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:09.424 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:05:09.424 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:05:09.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:09.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:09.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:09.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2229/81339 objects degraded (2.740%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:09 smithi162 ceph-mon[20279]: osdmap e205: 12 total, 12 up, 11 in 2024-09-02T10:05:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3899993664' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:05:09.648 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:09.677 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:09.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:09.678 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:09.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:09.806 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:09.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:09.821 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:05:09.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:05:09.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:09.932 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:09.934 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:10.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:10.058 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:10.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:10.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:10.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:10.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:10.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:10.260 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-02T10:05:09.114840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1705050680,"num_objects":26011,"num_object_clones":0,"num_object_copies":78033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26011,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169489,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28982,"num_bytes_recovered":1725703232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162621,"ondisk_log_size":162621,"up":111,"acting":107,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6962248,"kb_used_data":5922172,"kb_used_omap":17,"kb_used_meta":1039726,"kb_avail":1024234424,"statfs":{"total":1055945392128,"available":1048816050176,"internally_reserved":0,"allocated":6064304128,"data_stored":6055121517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1064680315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,4,2,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-233832448,"num_objects":-3568,"num_object_clones":0,"num_object_copies":-10704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-137,"num_objects_unfound":0,"num_objects_dirty":-3568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3568,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":777,"num_bytes_recovered":8978432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001436"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":176,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.187379+0000","last_change":"2024-09-02T10:02:43.131354+0000","last_active":"2024-09-02T10:05:06.187379+0000","last_peered":"2024-09-02T10:05:06.187379+0000","last_clean":"2024-09-02T10:05:06.187379+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:05:06.187379+0000","last_undegraded":"2024-09-02T10:05:06.187379+0000","last_fullsized":"2024-09-02T10:05:06.187379+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.131299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.131299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":261,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.197991+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:05:06.197991+0000","last_peered":"2024-09-02T10:05:06.197991+0000","last_clean":"2024-09-02T10:05:06.197991+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:05:06.197991+0000","last_undegraded":"2024-09-02T10:05:06.197991+0000","last_fullsized":"2024-09-02T10:05:06.197991+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.233390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":198,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.192041+0000","last_change":"2024-09-02T10:04:37.422274+0000","last_active":"2024-09-02T10:05:06.192041+0000","last_peered":"2024-09-02T10:05:06.192041+0000","last_clean":"2024-09-02T10:05:06.192041+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:05:06.192041+0000","last_undegraded":"2024-09-02T10:05:06.192041+0000","last_fullsized":"2024-09-02T10:05:06.192041+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:37.422128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:37.422128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:02.404595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":178,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.189765+0000","last_change":"2024-09-02T10:02:42.836988+0000","last_active":"2024-09-02T10:05:06.189765+0000","last_peered":"2024-09-02T10:05:06.189765+0000","last_clean":"2024-09-02T10:05:06.189765+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:05:06.189765+0000","last_undegraded":"2024-09-02T10:05:06.189765+0000","last_fullsized":"2024-09-02T10:05:06.189765+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:42.836911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:42.836911+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":191,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.192024+0000","last_change":"2024-09-02T10:04:38.471750+0000","last_active":"2024-09-02T10:05:06.192024+0000","last_peered":"2024-09-02T10:05:06.192024+0000","last_clean":"2024-09-02T10:05:06.192024+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:05:06.192024+0000","last_undegraded":"2024-09-02T10:05:06.192024+0000","last_fullsized":"2024-09-02T10:05:06.192024+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:38.471681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.471681+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:49.847941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"203'10440","reported_seq":10766,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-02T10:05:04.153480+0000","last_change":"2024-09-02T10:04:36.789912+0000","last_active":"2024-09-02T10:05:04.153480+0000","last_peered":"2024-09-02T10:05:04.153480+0000","last_clean":"2024-09-02T10:05:04.153480+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:05:04.153480+0000","last_undegraded":"2024-09-02T10:05:04.153480+0000","last_fullsized":"2024-09-02T10:05:04.153480+0000","mapping_epoch":92,"log_start":"97'400","ondisk_log_start":"97'400","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'9455","last_scrub_stamp":"2024-09-02T10:04:36.789821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:36.789821+0000","objects_scrubbed":2703,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:41.193203+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112590848,"num_objects":1718,"num_object_clones":0,"num_object_copies":5154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10440,"num_write_kb":389056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":263,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.191949+0000","last_change":"2024-09-02T10:04:36.384243+0000","last_active":"2024-09-02T10:05:06.191949+0000","last_peered":"2024-09-02T10:05:06.191949+0000","last_clean":"2024-09-02T10:05:06.191949+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:05:06.191949+0000","last_undegraded":"2024-09-02T10:05:06.191949+0000","last_fullsized":"2024-09-02T10:05:06.191949+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:36.384163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:36.384163+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:37.175901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"204'10592","reported_seq":10901,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.871589+0000","last_change":"2024-09-02T10:04:56.617908+0000","last_active":"2024-09-02T10:05:06.871589+0000","last_peered":"2024-09-02T10:05:06.871589+0000","last_clean":"2024-09-02T10:05:06.871589+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:05:06.871589+0000","last_undegraded":"2024-09-02T10:05:06.871589+0000","last_fullsized":"2024-09-02T10:05:06.871589+0000","mapping_epoch":92,"log_start":"97'500","ondisk_log_start":"97'500","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10186","last_scrub_stamp":"2024-09-02T10:04:56.617746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.617746+0000","objects_scrubbed":2052,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:15.697367+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":4938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10592,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":384,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-02T10:05:03.808146+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:05:03.808146+0000","last_peered":"2024-09-02T10:05:03.808146+0000","last_clean":"2024-09-02T10:05:03.808146+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:05:03.808146+0000","last_undegraded":"2024-09-02T10:05:03.808146+0000","last_fullsized":"2024-09-02T10:05:03.808146+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"204'10716","reported_seq":10980,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:07.832906+0000","last_change":"2024-09-02T10:04:55.460032+0000","last_active":"2024-09-02T10:05:07.832906+0000","last_peered":"2024-09-02T10:05:07.832906+0000","last_clean":"2024-09-02T10:05:07.832906+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:05:07.832906+0000","last_undegraded":"2024-09-02T10:05:07.832906+0000","last_fullsized":"2024-09-02T10:05:07.832906+0000","mapping_epoch":92,"log_start":"98'700","ondisk_log_start":"98'700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10251","last_scrub_stamp":"2024-09-02T10:04:55.459770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.459770+0000","objects_scrubbed":2066,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.035919+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104202240,"num_objects":1590,"num_object_clones":0,"num_object_copies":4770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10716,"num_write_kb":393792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":30,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.192741+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:05:06.192741+0000","last_peered":"2024-09-02T10:05:06.192741+0000","last_clean":"2024-09-02T10:05:06.192741+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:05:06.192741+0000","last_undegraded":"2024-09-02T10:05:06.192741+0000","last_fullsized":"2024-09-02T10:05:06.192741+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"203'10567","reported_seq":18205,"reported_epoch":203,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:05.820436+0000","last_change":"2024-09-02T10:04:57.234268+0000","last_active":"2024-09-02T10:05:05.820436+0000","last_peered":"2024-09-02T10:05:05.820436+0000","last_clean":"2024-09-02T10:03:38.126822+0000","last_became_active":"2024-09-02T10:04:57.195493+0000","last_became_peered":"2024-09-02T10:04:57.195493+0000","last_unstale":"2024-09-02T10:05:05.820436+0000","last_undegraded":"2024-09-02T10:05:05.820436+0000","last_fullsized":"2024-09-02T10:04:57.070642+0000","mapping_epoch":198,"log_start":"97'500","ondisk_log_start":"97'500","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108331008,"num_objects":1653,"num_object_clones":0,"num_object_copies":4959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10567,"num_write_kb":391040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3698,"num_bytes_recovered":208863232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1653}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":176,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.189234+0000","last_change":"2024-09-02T10:02:43.245406+0000","last_active":"2024-09-02T10:05:06.189234+0000","last_peered":"2024-09-02T10:05:06.189234+0000","last_clean":"2024-09-02T10:05:06.189234+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:05:06.189234+0000","last_undegraded":"2024-09-02T10:05:06.189234+0000","last_fullsized":"2024-09-02T10:05:06.189234+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.245354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.245354+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"204'10608","reported_seq":21557,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:07.008079+0000","last_change":"2024-09-02T10:04:27.932509+0000","last_active":"2024-09-02T10:05:07.008079+0000","last_peered":"2024-09-02T10:05:07.008079+0000","last_clean":"2024-09-02T10:05:07.008079+0000","last_became_active":"2024-09-02T10:04:26.448185+0000","last_became_peered":"2024-09-02T10:04:26.448185+0000","last_unstale":"2024-09-02T10:05:07.008079+0000","last_undegraded":"2024-09-02T10:05:07.008079+0000","last_fullsized":"2024-09-02T10:05:07.008079+0000","mapping_epoch":176,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9259","last_scrub_stamp":"2024-09-02T10:04:27.932362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:27.932362+0000","objects_scrubbed":2896,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:56.556476+0000","scrub_duration":606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100139008,"num_objects":1528,"num_object_clones":0,"num_object_copies":4584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10608,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":349503488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":332,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.189037+0000","last_change":"2024-09-02T10:02:41.206594+0000","last_active":"2024-09-02T10:05:06.189037+0000","last_peered":"2024-09-02T10:05:06.189037+0000","last_clean":"2024-09-02T10:05:06.189037+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:05:06.189037+0000","last_undegraded":"2024-09-02T10:05:06.189037+0000","last_fullsized":"2024-09-02T10:05:06.189037+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:41.206524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:41.206524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"204'10632","reported_seq":10922,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:07.826110+0000","last_change":"2024-09-02T10:04:56.300674+0000","last_active":"2024-09-02T10:05:07.826110+0000","last_peered":"2024-09-02T10:05:07.826110+0000","last_clean":"2024-09-02T10:05:07.826110+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:05:07.826110+0000","last_undegraded":"2024-09-02T10:05:07.826110+0000","last_fullsized":"2024-09-02T10:05:07.826110+0000","mapping_epoch":92,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10171","last_scrub_stamp":"2024-09-02T10:04:56.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.300400+0000","objects_scrubbed":2044,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:07.471560+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103677952,"num_objects":1582,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":170,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.186826+0000","last_change":"2024-09-02T10:02:49.000327+0000","last_active":"2024-09-02T10:05:06.186826+0000","last_peered":"2024-09-02T10:05:06.186826+0000","last_clean":"2024-09-02T10:05:06.186826+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:05:06.186826+0000","last_undegraded":"2024-09-02T10:05:06.186826+0000","last_fullsized":"2024-09-02T10:05:06.186826+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:49.000247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:49.000247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"203'10412","reported_seq":10730,"reported_epoch":203,"state":"active+clean+scrubbing","last_fresh":"2024-09-02T10:05:04.190490+0000","last_change":"2024-09-02T10:05:03.504508+0000","last_active":"2024-09-02T10:05:04.190490+0000","last_peered":"2024-09-02T10:05:04.190490+0000","last_clean":"2024-09-02T10:05:04.190490+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:05:04.190490+0000","last_undegraded":"2024-09-02T10:05:04.190490+0000","last_fullsized":"2024-09-02T10:05:04.190490+0000","mapping_epoch":92,"log_start":"97'400","ondisk_log_start":"97'400","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'6154","last_scrub_stamp":"2024-09-02T10:03:08.252878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:03:08.252878+0000","objects_scrubbed":1049,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"scrubbing for 0s","scrub_duration":2225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":1728,"num_object_clones":0,"num_object_copies":5184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10412,"num_write_kb":388480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":30,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.192852+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:05:06.192852+0000","last_peered":"2024-09-02T10:05:06.192852+0000","last_clean":"2024-09-02T10:05:06.192852+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:05:06.192852+0000","last_undegraded":"2024-09-02T10:05:06.192852+0000","last_fullsized":"2024-09-02T10:05:06.192852+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.611307+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"204'10544","reported_seq":21412,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:07.815166+0000","last_change":"2024-09-02T10:05:00.573995+0000","last_active":"2024-09-02T10:05:07.815166+0000","last_peered":"2024-09-02T10:05:07.815166+0000","last_clean":"2024-09-02T10:05:07.815166+0000","last_became_active":"2024-09-02T10:04:43.806359+0000","last_became_peered":"2024-09-02T10:04:43.806359+0000","last_unstale":"2024-09-02T10:05:07.815166+0000","last_undegraded":"2024-09-02T10:05:07.815166+0000","last_fullsized":"2024-09-02T10:05:07.815166+0000","mapping_epoch":188,"log_start":"98'500","ondisk_log_start":"98'500","created":92,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'10224","last_scrub_stamp":"2024-09-02T10:05:00.573420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:00.573420+0000","objects_scrubbed":1920,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.148546+0000","scrub_duration":803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104333312,"num_objects":1592,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10544,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5298,"num_bytes_recovered":347209728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":293,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.193411+0000","last_change":"2024-09-02T10:04:35.810947+0000","last_active":"2024-09-02T10:05:06.193411+0000","last_peered":"2024-09-02T10:05:06.193411+0000","last_clean":"2024-09-02T10:05:06.193411+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:05:06.193411+0000","last_undegraded":"2024-09-02T10:05:06.193411+0000","last_fullsized":"2024-09-02T10:05:06.193411+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:28.355594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:28.355594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:51.765081+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"204'10888","reported_seq":11222,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.792287+0000","last_change":"2024-09-02T10:04:58.724992+0000","last_active":"2024-09-02T10:05:06.792287+0000","last_peered":"2024-09-02T10:05:06.792287+0000","last_clean":"2024-09-02T10:05:06.792287+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:05:06.792287+0000","last_undegraded":"2024-09-02T10:05:06.792287+0000","last_fullsized":"2024-09-02T10:05:06.792287+0000","mapping_epoch":92,"log_start":"98'800","ondisk_log_start":"98'800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'10526","last_scrub_stamp":"2024-09-02T10:04:58.724845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.724845+0000","objects_scrubbed":2020,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:18.034443+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108396544,"num_objects":1654,"num_object_clones":0,"num_object_copies":4962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10888,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":22,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.192764+0000","last_change":"2024-09-02T10:04:57.021532+0000","last_active":"2024-09-02T10:05:06.192764+0000","last_peered":"2024-09-02T10:05:06.192764+0000","last_clean":"2024-09-02T10:05:06.192764+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:05:06.192764+0000","last_undegraded":"2024-09-02T10:05:06.192764+0000","last_fullsized":"2024-09-02T10:05:06.192764+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:38.873802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:38.873802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"204'10685","reported_seq":17115,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:07.350007+0000","last_change":"2024-09-02T10:04:58.839358+0000","last_active":"2024-09-02T10:05:07.350007+0000","last_peered":"2024-09-02T10:05:07.350007+0000","last_clean":"2024-09-02T10:05:07.350007+0000","last_became_active":"2024-09-02T10:04:58.808736+0000","last_became_peered":"2024-09-02T10:04:58.808736+0000","last_unstale":"2024-09-02T10:05:07.350007+0000","last_undegraded":"2024-09-02T10:05:07.350007+0000","last_fullsized":"2024-09-02T10:05:07.350007+0000","mapping_epoch":200,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'5361","last_scrub_stamp":"2024-09-02T10:02:48.223591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.223591+0000","objects_scrubbed":5355,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102957056,"num_objects":1571,"num_object_clones":0,"num_object_copies":4713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10685,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3059,"num_bytes_recovered":97517568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":32,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-02T10:05:03.808216+0000","last_change":"2024-09-02T10:04:47.929535+0000","last_active":"2024-09-02T10:05:03.808216+0000","last_peered":"2024-09-02T10:05:03.808216+0000","last_clean":"2024-09-02T10:05:03.808216+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:05:03.808216+0000","last_undegraded":"2024-09-02T10:05:03.808216+0000","last_fullsized":"2024-09-02T10:05:03.808216+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:39.894365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:39.894365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"203'10518","reported_seq":25688,"reported_epoch":203,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:05.825975+0000","last_change":"2024-09-02T10:04:57.232835+0000","last_active":"2024-09-02T10:05:05.825975+0000","last_peered":"2024-09-02T10:05:05.825975+0000","last_clean":"2024-09-02T10:02:27.223209+0000","last_became_active":"2024-09-02T10:04:57.144949+0000","last_became_peered":"2024-09-02T10:04:57.144949+0000","last_unstale":"2024-09-02T10:05:05.825975+0000","last_undegraded":"2024-09-02T10:05:05.825975+0000","last_fullsized":"2024-09-02T10:04:57.017574+0000","mapping_epoch":198,"log_start":"97'500","ondisk_log_start":"97'500","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101974016,"num_objects":1556,"num_object_clones":0,"num_object_copies":4668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1556,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10518,"num_write_kb":386368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6516,"num_bytes_recovered":417005568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,11],"avail_no_missing":["7","10","11"],"object_location_counts":[{"shards":"7,11","objects":1556}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":248,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.815241+0000","last_change":"2024-09-02T10:05:06.815241+0000","last_active":"2024-09-02T10:05:06.815241+0000","last_peered":"2024-09-02T10:05:06.815241+0000","last_clean":"2024-09-02T10:05:06.815241+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:05:06.815241+0000","last_undegraded":"2024-09-02T10:05:06.815241+0000","last_fullsized":"2024-09-02T10:05:06.815241+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:06.815191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:06.815191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:14.504648+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"204'10726","reported_seq":11065,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.708837+0000","last_change":"2024-09-02T10:04:54.927090+0000","last_active":"2024-09-02T10:05:06.708837+0000","last_peered":"2024-09-02T10:05:06.708837+0000","last_clean":"2024-09-02T10:05:06.708837+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:05:06.708837+0000","last_undegraded":"2024-09-02T10:05:06.708837+0000","last_fullsized":"2024-09-02T10:05:06.708837+0000","mapping_epoch":92,"log_start":"98'700","ondisk_log_start":"98'700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10296","last_scrub_stamp":"2024-09-02T10:04:54.926873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.926873+0000","objects_scrubbed":2056,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.100584+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105906176,"num_objects":1616,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10726,"num_write_kb":394944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":43,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-02T10:05:03.813422+0000","last_change":"2024-09-02T10:04:55.111033+0000","last_active":"2024-09-02T10:05:03.813422+0000","last_peered":"2024-09-02T10:05:03.813422+0000","last_clean":"2024-09-02T10:05:03.813422+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:05:03.813422+0000","last_undegraded":"2024-09-02T10:05:03.813422+0000","last_fullsized":"2024-09-02T10:05:03.813422+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:55.110979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.110979+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:01.238104+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":278,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.193356+0000","last_change":"2024-09-02T10:04:30.278703+0000","last_active":"2024-09-02T10:05:06.193356+0000","last_peered":"2024-09-02T10:05:06.193356+0000","last_clean":"2024-09-02T10:05:06.193356+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:05:06.193356+0000","last_undegraded":"2024-09-02T10:05:06.193356+0000","last_fullsized":"2024-09-02T10:05:06.193356+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:30.278632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:30.278632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:48.025871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"204'10672","reported_seq":10969,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.866803+0000","last_change":"2024-09-02T10:02:46.577683+0000","last_active":"2024-09-02T10:05:06.866803+0000","last_peered":"2024-09-02T10:05:06.866803+0000","last_clean":"2024-09-02T10:05:06.866803+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:05:06.866803+0000","last_undegraded":"2024-09-02T10:05:06.866803+0000","last_fullsized":"2024-09-02T10:05:06.866803+0000","mapping_epoch":92,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'5355","last_scrub_stamp":"2024-09-02T10:02:46.577625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:46.577625+0000","objects_scrubbed":5316,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105250816,"num_objects":1606,"num_object_clones":0,"num_object_copies":4818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10672,"num_write_kb":392896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":182,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-02T10:05:03.807865+0000","last_change":"2024-09-02T10:04:57.391785+0000","last_active":"2024-09-02T10:05:03.807865+0000","last_peered":"2024-09-02T10:05:03.807865+0000","last_clean":"2024-09-02T10:05:03.807865+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:05:03.807865+0000","last_undegraded":"2024-09-02T10:05:03.807865+0000","last_fullsized":"2024-09-02T10:05:03.807865+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.391723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.391723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:12.774478+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"203'10242","reported_seq":16357,"reported_epoch":203,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:05:04.134521+0000","last_change":"2024-09-02T10:04:57.390580+0000","last_active":"2024-09-02T10:05:04.134521+0000","last_peered":"2024-09-02T10:05:04.134521+0000","last_clean":"2024-09-02T10:03:45.188955+0000","last_became_active":"2024-09-02T10:04:57.363859+0000","last_became_peered":"2024-09-02T10:04:57.363859+0000","last_unstale":"2024-09-02T10:05:04.134521+0000","last_undegraded":"2024-09-02T10:04:57.027533+0000","last_fullsized":"2024-09-02T10:04:57.018785+0000","mapping_epoch":198,"log_start":"97'200","ondisk_log_start":"97'200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'5182","last_scrub_stamp":"2024-09-02T10:02:48.587872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.587872+0000","objects_scrubbed":5181,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107282456,"num_objects":1638,"num_object_clones":0,"num_object_copies":4914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1638,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10242,"num_write_kb":380097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":190513152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1638}],"blocked_by":[],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":43,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-02T10:05:03.808477+0000","last_change":"2024-09-02T10:04:54.128208+0000","last_active":"2024-09-02T10:05:03.808477+0000","last_peered":"2024-09-02T10:05:03.808477+0000","last_clean":"2024-09-02T10:05:03.808477+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:05:03.808477+0000","last_undegraded":"2024-09-02T10:05:03.808477+0000","last_fullsized":"2024-09-02T10:05:03.808477+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:54.128142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.128142+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.955526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"204'10715","reported_seq":17466,"reported_epoch":204,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:07.464027+0000","last_change":"2024-09-02T10:05:07.464027+0000","last_active":"2024-09-02T10:05:07.464027+0000","last_peered":"2024-09-02T10:05:07.464027+0000","last_clean":"2024-09-02T10:03:38.171966+0000","last_became_active":"2024-09-02T10:04:57.182024+0000","last_became_peered":"2024-09-02T10:04:57.182024+0000","last_unstale":"2024-09-02T10:05:07.464027+0000","last_undegraded":"2024-09-02T10:05:07.464027+0000","last_fullsized":"2024-09-02T10:04:57.016771+0000","mapping_epoch":198,"log_start":"98'700","ondisk_log_start":"98'700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103612416,"num_objects":1581,"num_object_clones":0,"num_object_copies":4743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10715,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2167,"num_bytes_recovered":114360320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2,3","objects":1581}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"203'10532","reported_seq":10861,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-09-02T10:05:04.201332+0000","last_change":"2024-09-02T10:04:38.925752+0000","last_active":"2024-09-02T10:05:04.201332+0000","last_peered":"2024-09-02T10:05:04.201332+0000","last_clean":"2024-09-02T10:05:04.201332+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:05:04.201332+0000","last_undegraded":"2024-09-02T10:05:04.201332+0000","last_fullsized":"2024-09-02T10:05:04.201332+0000","mapping_epoch":92,"log_start":"97'500","ondisk_log_start":"97'500","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9618","last_scrub_stamp":"2024-09-02T10:04:38.925565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.925565+0000","objects_scrubbed":2671,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.842942+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688000,"num_objects":1750,"num_object_clones":0,"num_object_copies":5250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10532,"num_write_kb":393024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":58,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.197430+0000","last_change":"2024-09-02T10:04:35.814407+0000","last_active":"2024-09-02T10:05:06.197430+0000","last_peered":"2024-09-02T10:05:06.197430+0000","last_clean":"2024-09-02T10:05:06.197430+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:05:06.197430+0000","last_undegraded":"2024-09-02T10:05:06.197430+0000","last_fullsized":"2024-09-02T10:05:06.197430+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:46.975807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:46.975807+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1704460312,"num_objects":26009,"num_object_clones":0,"num_object_copies":78027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26009,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169489,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28979,"num_bytes_recovered":1724973056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6225932288,"data_stored":6225920072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160689,"ondisk_log_size":160689,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383740,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595408,"kb_used_data":465660,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93149744,"statfs":{"total":95995035648,"available":95385337856,"internally_reserved":0,"allocated":476835840,"data_stored":476001095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":132839876},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841606,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240848,"kb_used_data":1129152,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92504304,"statfs":{"total":95995035648,"available":94724407296,"internally_reserved":0,"allocated":1156251648,"data_stored":1155412839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070539,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":516020,"kb_used_data":452900,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93229132,"statfs":{"total":95995035648,"available":95466631168,"internally_reserved":0,"allocated":463769600,"data_stored":462943815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299470,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":635264,"kb_used_data":536048,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93109888,"statfs":{"total":95995035648,"available":95344525312,"internally_reserved":0,"allocated":548913152,"data_stored":548082887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528403,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":409744,"kb_used_data":346620,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93335408,"statfs":{"total":95995035648,"available":95575457792,"internally_reserved":0,"allocated":354938880,"data_stored":354104143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724632,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":281944,"kb_used_data":214660,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93463208,"statfs":{"total":95995035648,"available":95706324992,"internally_reserved":0,"allocated":219811840,"data_stored":218973551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953565,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408312,"kb_used_data":336872,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93336840,"statfs":{"total":95995035648,"available":95576924160,"internally_reserved":0,"allocated":344956928,"data_stored":344127071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":973532,"kb_used_data":843784,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92771620,"statfs":{"total":95995035648,"available":94998138880,"internally_reserved":0,"allocated":864034816,"data_stored":863192455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":132839880},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":693208,"kb_used_data":557764,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93051944,"statfs":{"total":95995035648,"available":95285190656,"internally_reserved":0,"allocated":571150336,"data_stored":570312047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":588496,"kb_used_data":512892,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93156656,"statfs":{"total":95995035648,"available":95392415744,"internally_reserved":0,"allocated":525201408,"data_stored":524366671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":619472,"kb_used_data":525820,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93125680,"statfs":{"total":95995035648,"available":95360696320,"internally_reserved":0,"allocated":538439680,"data_stored":537604943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532742144,"data_stored":532742144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520749056,"data_stored":520749056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":561184768,"data_stored":561184768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":845746176,"data_stored":845742104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":335220736,"data_stored":335216664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":214368256,"data_stored":214368256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":349437952,"data_stored":349437952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":535822336,"data_stored":535822336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":454098944,"data_stored":454098944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1132072960,"data_stored":1132068888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":275513344,"data_stored":275513344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":468975616,"data_stored":468975616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:05:10.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:05:10.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:10.313 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:10.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:10.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:10.439 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:10.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:10.492 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:10 smithi187 ceph-mon[20072]: pgmap v510: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 1.6 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 594 op/s; 2229/78033 objects degraded (2.856%); 1.4 MiB/s, 129 objects/s recovering 2024-09-02T10:05:10.492 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:10 smithi187 ceph-mon[20072]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:10.492 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:10 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:10.492 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3899993664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:05:10.493 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:10 smithi187 ceph-mon[20072]: osdmap e206: 12 total, 12 up, 11 in 2024-09-02T10:05:10.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:05:10.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:10.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:10.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:10.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:10 smithi162 ceph-mon[20279]: pgmap v510: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 1.6 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 594 op/s; 2229/78033 objects degraded (2.856%); 1.4 MiB/s, 129 objects/s recovering 2024-09-02T10:05:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:10 smithi162 ceph-mon[20279]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:10 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3899993664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:05:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:10 smithi162 ceph-mon[20279]: osdmap e206: 12 total, 12 up, 11 in 2024-09-02T10:05:10.668 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:10.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:10 smithi151 ceph-mon[16639]: pgmap v510: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 3 active+recovering+undersized+remapped, 32 active+clean; 1.6 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 594 op/s; 2229/78033 objects degraded (2.856%); 1.4 MiB/s, 129 objects/s recovering 2024-09-02T10:05:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:10 smithi151 ceph-mon[16639]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:10 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3899993664' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:05:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:10 smithi151 ceph-mon[16639]: osdmap e206: 12 total, 12 up, 11 in 2024-09-02T10:05:10.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:10.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:10.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:10.791 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:10.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:10.822 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:10.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:10.845 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:10.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:10.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:10.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:11.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:11.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:11.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:11.184 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:11.207 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:11.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:11.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:11.310 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:11.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:11.338 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:11.417 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:05:11.418 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:05:11.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:11.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:11.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:11.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:11.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:11.597 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:05:11.597 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:05:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:11.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:11.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:11.636 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:05:11.636 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:05:11.672 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:05:11.672 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:05:11.701 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:11.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:11.705 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:05:11.706 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:05:11.732 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:11.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:11.742 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:05:11.742 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:05:11.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:05:11.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:05:11.813 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:05:11.813 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:05:11.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:11.847 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:05:11.847 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:05:11.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:11.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:11.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:05:11.876 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:05:11.911 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:05:11.911 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:05:11.944 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:05:11.945 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:05:11.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:11.976 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:05:11.977 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:05:11.984 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:11.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:11.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:12.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:12.110 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:12.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:12.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:12.235 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:12.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:12.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:12.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:12.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:12.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:12.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:12.439 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:12 smithi187 ceph-mon[20072]: pgmap v512: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 653 op/s; 2229/74196 objects degraded (3.004%); 1.4 MiB/s, 226 objects/s recovering 2024-09-02T10:05:12.439 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2175279142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:12.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:12.501 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:12.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:12.581 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:12 smithi151 ceph-mon[16639]: pgmap v512: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 653 op/s; 2229/74196 objects degraded (3.004%); 1.4 MiB/s, 226 objects/s recovering 2024-09-02T10:05:12.582 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2175279142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:12.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:12 smithi162 ceph-mon[20279]: pgmap v512: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.5 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 653 op/s; 2229/74196 objects degraded (3.004%); 1.4 MiB/s, 226 objects/s recovering 2024-09-02T10:05:12.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2175279142' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:12.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:12.625 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:12.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:12.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:12.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:12.760 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:12.786 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:05:12.795 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:05:12.795 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:05:12.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:05:12.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:12.904 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:12.905 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:13.006 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:13.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:13.030 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:13.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:13.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:13.142 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:13.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:13.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:13.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:13.286 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:13.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:13.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:13.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:13.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:13.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:13.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:13.548 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:13.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:13.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:13.651 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:13.674 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-09-02T10:05:11.115253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1621230136,"num_objects":24732,"num_object_clones":0,"num_object_copies":74196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24732,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170768,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29564,"num_bytes_recovered":1725703232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162700,"ondisk_log_size":162700,"up":111,"acting":107,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6811676,"kb_used_data":5759120,"kb_used_omap":17,"kb_used_meta":1052206,"kb_avail":1024384996,"statfs":{"total":1055945392128,"available":1048970235904,"internally_reserved":0,"allocated":5897338880,"data_stored":5888124933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1077459835},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-257163264,"num_objects":-3924,"num_object_clones":0,"num_object_copies":-11772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-137,"num_objects_unfound":0,"num_objects_dirty":-3924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3924,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1359,"num_bytes_recovered":8978432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001321"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":176,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.187379+0000","last_change":"2024-09-02T10:02:43.131354+0000","last_active":"2024-09-02T10:05:06.187379+0000","last_peered":"2024-09-02T10:05:06.187379+0000","last_clean":"2024-09-02T10:05:06.187379+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:05:06.187379+0000","last_undegraded":"2024-09-02T10:05:06.187379+0000","last_fullsized":"2024-09-02T10:05:06.187379+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.131299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.131299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":261,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.197991+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:05:06.197991+0000","last_peered":"2024-09-02T10:05:06.197991+0000","last_clean":"2024-09-02T10:05:06.197991+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:05:06.197991+0000","last_undegraded":"2024-09-02T10:05:06.197991+0000","last_fullsized":"2024-09-02T10:05:06.197991+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.233390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":198,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.192041+0000","last_change":"2024-09-02T10:04:37.422274+0000","last_active":"2024-09-02T10:05:06.192041+0000","last_peered":"2024-09-02T10:05:06.192041+0000","last_clean":"2024-09-02T10:05:06.192041+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:05:06.192041+0000","last_undegraded":"2024-09-02T10:05:06.192041+0000","last_fullsized":"2024-09-02T10:05:06.192041+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:37.422128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:37.422128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:02.404595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":178,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.189765+0000","last_change":"2024-09-02T10:02:42.836988+0000","last_active":"2024-09-02T10:05:06.189765+0000","last_peered":"2024-09-02T10:05:06.189765+0000","last_clean":"2024-09-02T10:05:06.189765+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:05:06.189765+0000","last_undegraded":"2024-09-02T10:05:06.189765+0000","last_fullsized":"2024-09-02T10:05:06.189765+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:42.836911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:42.836911+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":191,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.192024+0000","last_change":"2024-09-02T10:04:38.471750+0000","last_active":"2024-09-02T10:05:06.192024+0000","last_peered":"2024-09-02T10:05:06.192024+0000","last_clean":"2024-09-02T10:05:06.192024+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:05:06.192024+0000","last_undegraded":"2024-09-02T10:05:06.192024+0000","last_fullsized":"2024-09-02T10:05:06.192024+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:38.471681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.471681+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:49.847941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"205'10632","reported_seq":10962,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.146903+0000","last_change":"2024-09-02T10:04:36.789912+0000","last_active":"2024-09-02T10:05:09.146903+0000","last_peered":"2024-09-02T10:05:09.146903+0000","last_clean":"2024-09-02T10:05:09.146903+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:05:09.146903+0000","last_undegraded":"2024-09-02T10:05:09.146903+0000","last_fullsized":"2024-09-02T10:05:09.146903+0000","mapping_epoch":92,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'9455","last_scrub_stamp":"2024-09-02T10:04:36.789821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:36.789821+0000","objects_scrubbed":2703,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:41.193203+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100007936,"num_objects":1526,"num_object_clones":0,"num_object_copies":4578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":389056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":263,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.191949+0000","last_change":"2024-09-02T10:04:36.384243+0000","last_active":"2024-09-02T10:05:06.191949+0000","last_peered":"2024-09-02T10:05:06.191949+0000","last_clean":"2024-09-02T10:05:06.191949+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:05:06.191949+0000","last_undegraded":"2024-09-02T10:05:06.191949+0000","last_fullsized":"2024-09-02T10:05:06.191949+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:36.384163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:36.384163+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:37.175901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"204'10592","reported_seq":10901,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.871589+0000","last_change":"2024-09-02T10:04:56.617908+0000","last_active":"2024-09-02T10:05:06.871589+0000","last_peered":"2024-09-02T10:05:06.871589+0000","last_clean":"2024-09-02T10:05:06.871589+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:05:06.871589+0000","last_undegraded":"2024-09-02T10:05:06.871589+0000","last_fullsized":"2024-09-02T10:05:06.871589+0000","mapping_epoch":92,"log_start":"97'500","ondisk_log_start":"97'500","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10186","last_scrub_stamp":"2024-09-02T10:04:56.617746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.617746+0000","objects_scrubbed":2052,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:15.697367+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107872256,"num_objects":1646,"num_object_clones":0,"num_object_copies":4938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10592,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":390,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.824289+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:05:09.824289+0000","last_peered":"2024-09-02T10:05:09.824289+0000","last_clean":"2024-09-02T10:05:09.824289+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:05:09.824289+0000","last_undegraded":"2024-09-02T10:05:09.824289+0000","last_fullsized":"2024-09-02T10:05:09.824289+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"204'10716","reported_seq":10980,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:07.832906+0000","last_change":"2024-09-02T10:04:55.460032+0000","last_active":"2024-09-02T10:05:07.832906+0000","last_peered":"2024-09-02T10:05:07.832906+0000","last_clean":"2024-09-02T10:05:07.832906+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:05:07.832906+0000","last_undegraded":"2024-09-02T10:05:07.832906+0000","last_fullsized":"2024-09-02T10:05:07.832906+0000","mapping_epoch":92,"log_start":"98'700","ondisk_log_start":"98'700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10251","last_scrub_stamp":"2024-09-02T10:04:55.459770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.459770+0000","objects_scrubbed":2066,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.035919+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104202240,"num_objects":1590,"num_object_clones":0,"num_object_copies":4770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10716,"num_write_kb":393792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":30,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.192741+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:05:06.192741+0000","last_peered":"2024-09-02T10:05:06.192741+0000","last_clean":"2024-09-02T10:05:06.192741+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:05:06.192741+0000","last_undegraded":"2024-09-02T10:05:06.192741+0000","last_fullsized":"2024-09-02T10:05:06.192741+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"206'10787","reported_seq":19026,"reported_epoch":206,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:10.830905+0000","last_change":"2024-09-02T10:04:57.234268+0000","last_active":"2024-09-02T10:05:10.830905+0000","last_peered":"2024-09-02T10:05:10.830905+0000","last_clean":"2024-09-02T10:03:38.126822+0000","last_became_active":"2024-09-02T10:04:57.195493+0000","last_became_peered":"2024-09-02T10:04:57.195493+0000","last_unstale":"2024-09-02T10:05:10.830905+0000","last_undegraded":"2024-09-02T10:05:10.830905+0000","last_fullsized":"2024-09-02T10:04:57.070642+0000","mapping_epoch":198,"log_start":"98'700","ondisk_log_start":"98'700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93913088,"num_objects":1433,"num_object_clones":0,"num_object_copies":4299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10787,"num_write_kb":391040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3995,"num_bytes_recovered":208863232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1433}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":176,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.189234+0000","last_change":"2024-09-02T10:02:43.245406+0000","last_active":"2024-09-02T10:05:06.189234+0000","last_peered":"2024-09-02T10:05:06.189234+0000","last_clean":"2024-09-02T10:05:06.189234+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:05:06.189234+0000","last_undegraded":"2024-09-02T10:05:06.189234+0000","last_fullsized":"2024-09-02T10:05:06.189234+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.245354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.245354+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"204'10608","reported_seq":21557,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:07.008079+0000","last_change":"2024-09-02T10:04:27.932509+0000","last_active":"2024-09-02T10:05:07.008079+0000","last_peered":"2024-09-02T10:05:07.008079+0000","last_clean":"2024-09-02T10:05:07.008079+0000","last_became_active":"2024-09-02T10:04:26.448185+0000","last_became_peered":"2024-09-02T10:04:26.448185+0000","last_unstale":"2024-09-02T10:05:07.008079+0000","last_undegraded":"2024-09-02T10:05:07.008079+0000","last_fullsized":"2024-09-02T10:05:07.008079+0000","mapping_epoch":176,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9259","last_scrub_stamp":"2024-09-02T10:04:27.932362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:27.932362+0000","objects_scrubbed":2896,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:56.556476+0000","scrub_duration":606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100139008,"num_objects":1528,"num_object_clones":0,"num_object_copies":4584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10608,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":349503488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":332,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.189037+0000","last_change":"2024-09-02T10:02:41.206594+0000","last_active":"2024-09-02T10:05:06.189037+0000","last_peered":"2024-09-02T10:05:06.189037+0000","last_clean":"2024-09-02T10:05:06.189037+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:05:06.189037+0000","last_undegraded":"2024-09-02T10:05:06.189037+0000","last_fullsized":"2024-09-02T10:05:06.189037+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:41.206524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:41.206524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"204'10632","reported_seq":10922,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:07.826110+0000","last_change":"2024-09-02T10:04:56.300674+0000","last_active":"2024-09-02T10:05:07.826110+0000","last_peered":"2024-09-02T10:05:07.826110+0000","last_clean":"2024-09-02T10:05:07.826110+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:05:07.826110+0000","last_undegraded":"2024-09-02T10:05:07.826110+0000","last_fullsized":"2024-09-02T10:05:07.826110+0000","mapping_epoch":92,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10171","last_scrub_stamp":"2024-09-02T10:04:56.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.300400+0000","objects_scrubbed":2044,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:07.471560+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103677952,"num_objects":1582,"num_object_clones":0,"num_object_copies":4746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10632,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":170,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.186826+0000","last_change":"2024-09-02T10:02:49.000327+0000","last_active":"2024-09-02T10:05:06.186826+0000","last_peered":"2024-09-02T10:05:06.186826+0000","last_clean":"2024-09-02T10:05:06.186826+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:05:06.186826+0000","last_undegraded":"2024-09-02T10:05:06.186826+0000","last_fullsized":"2024-09-02T10:05:06.186826+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:49.000247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:49.000247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"205'10622","reported_seq":10955,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.162047+0000","last_change":"2024-09-02T10:05:04.366916+0000","last_active":"2024-09-02T10:05:09.162047+0000","last_peered":"2024-09-02T10:05:09.162047+0000","last_clean":"2024-09-02T10:05:09.162047+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:05:09.162047+0000","last_undegraded":"2024-09-02T10:05:09.162047+0000","last_fullsized":"2024-09-02T10:05:09.162047+0000","mapping_epoch":92,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'10418","last_scrub_stamp":"2024-09-02T10:05:04.366613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:04.366613+0000","objects_scrubbed":1736,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:11.371378+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99483648,"num_objects":1518,"num_object_clones":0,"num_object_copies":4554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10622,"num_write_kb":388480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":30,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.192852+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:05:06.192852+0000","last_peered":"2024-09-02T10:05:06.192852+0000","last_clean":"2024-09-02T10:05:06.192852+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:05:06.192852+0000","last_undegraded":"2024-09-02T10:05:06.192852+0000","last_fullsized":"2024-09-02T10:05:06.192852+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.611307+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"204'10544","reported_seq":21412,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:07.815166+0000","last_change":"2024-09-02T10:05:00.573995+0000","last_active":"2024-09-02T10:05:07.815166+0000","last_peered":"2024-09-02T10:05:07.815166+0000","last_clean":"2024-09-02T10:05:07.815166+0000","last_became_active":"2024-09-02T10:04:43.806359+0000","last_became_peered":"2024-09-02T10:04:43.806359+0000","last_unstale":"2024-09-02T10:05:07.815166+0000","last_undegraded":"2024-09-02T10:05:07.815166+0000","last_fullsized":"2024-09-02T10:05:07.815166+0000","mapping_epoch":188,"log_start":"98'500","ondisk_log_start":"98'500","created":92,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'10224","last_scrub_stamp":"2024-09-02T10:05:00.573420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:00.573420+0000","objects_scrubbed":1920,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.148546+0000","scrub_duration":803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104333312,"num_objects":1592,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10544,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5298,"num_bytes_recovered":347209728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":293,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.193411+0000","last_change":"2024-09-02T10:04:35.810947+0000","last_active":"2024-09-02T10:05:06.193411+0000","last_peered":"2024-09-02T10:05:06.193411+0000","last_clean":"2024-09-02T10:05:06.193411+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:05:06.193411+0000","last_undegraded":"2024-09-02T10:05:06.193411+0000","last_fullsized":"2024-09-02T10:05:06.193411+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:28.355594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:28.355594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:51.765081+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"204'10888","reported_seq":11222,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.792287+0000","last_change":"2024-09-02T10:04:58.724992+0000","last_active":"2024-09-02T10:05:06.792287+0000","last_peered":"2024-09-02T10:05:06.792287+0000","last_clean":"2024-09-02T10:05:06.792287+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:05:06.792287+0000","last_undegraded":"2024-09-02T10:05:06.792287+0000","last_fullsized":"2024-09-02T10:05:06.792287+0000","mapping_epoch":92,"log_start":"98'800","ondisk_log_start":"98'800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'10526","last_scrub_stamp":"2024-09-02T10:04:58.724845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.724845+0000","objects_scrubbed":2020,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:18.034443+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108396544,"num_objects":1654,"num_object_clones":0,"num_object_copies":4962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10888,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":22,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.192764+0000","last_change":"2024-09-02T10:04:57.021532+0000","last_active":"2024-09-02T10:05:06.192764+0000","last_peered":"2024-09-02T10:05:06.192764+0000","last_clean":"2024-09-02T10:05:06.192764+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:05:06.192764+0000","last_undegraded":"2024-09-02T10:05:06.192764+0000","last_fullsized":"2024-09-02T10:05:06.192764+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:38.873802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:38.873802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"204'10685","reported_seq":17115,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:07.350007+0000","last_change":"2024-09-02T10:04:58.839358+0000","last_active":"2024-09-02T10:05:07.350007+0000","last_peered":"2024-09-02T10:05:07.350007+0000","last_clean":"2024-09-02T10:05:07.350007+0000","last_became_active":"2024-09-02T10:04:58.808736+0000","last_became_peered":"2024-09-02T10:04:58.808736+0000","last_unstale":"2024-09-02T10:05:07.350007+0000","last_undegraded":"2024-09-02T10:05:07.350007+0000","last_fullsized":"2024-09-02T10:05:07.350007+0000","mapping_epoch":200,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'5361","last_scrub_stamp":"2024-09-02T10:02:48.223591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.223591+0000","objects_scrubbed":5355,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102957056,"num_objects":1571,"num_object_clones":0,"num_object_copies":4713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10685,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3059,"num_bytes_recovered":97517568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":38,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.824463+0000","last_change":"2024-09-02T10:04:47.929535+0000","last_active":"2024-09-02T10:05:09.824463+0000","last_peered":"2024-09-02T10:05:09.824463+0000","last_clean":"2024-09-02T10:05:09.824463+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:05:09.824463+0000","last_undegraded":"2024-09-02T10:05:09.824463+0000","last_fullsized":"2024-09-02T10:05:09.824463+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:39.894365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:39.894365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"206'10731","reported_seq":26480,"reported_epoch":206,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:10.830726+0000","last_change":"2024-09-02T10:04:57.232835+0000","last_active":"2024-09-02T10:05:10.830726+0000","last_peered":"2024-09-02T10:05:10.830726+0000","last_clean":"2024-09-02T10:02:27.223209+0000","last_became_active":"2024-09-02T10:04:57.144949+0000","last_became_peered":"2024-09-02T10:04:57.144949+0000","last_unstale":"2024-09-02T10:05:10.830726+0000","last_undegraded":"2024-09-02T10:05:10.830726+0000","last_fullsized":"2024-09-02T10:04:57.017574+0000","mapping_epoch":198,"log_start":"98'700","ondisk_log_start":"98'700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88014848,"num_objects":1343,"num_object_clones":0,"num_object_copies":4029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10731,"num_write_kb":386368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6801,"num_bytes_recovered":417005568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,11],"avail_no_missing":["7","10","11"],"object_location_counts":[{"shards":"7,11","objects":1343}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":248,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.815241+0000","last_change":"2024-09-02T10:05:06.815241+0000","last_active":"2024-09-02T10:05:06.815241+0000","last_peered":"2024-09-02T10:05:06.815241+0000","last_clean":"2024-09-02T10:05:06.815241+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:05:06.815241+0000","last_undegraded":"2024-09-02T10:05:06.815241+0000","last_fullsized":"2024-09-02T10:05:06.815241+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:06.815191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:06.815191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:14.504648+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"204'10726","reported_seq":11065,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.708837+0000","last_change":"2024-09-02T10:04:54.927090+0000","last_active":"2024-09-02T10:05:06.708837+0000","last_peered":"2024-09-02T10:05:06.708837+0000","last_clean":"2024-09-02T10:05:06.708837+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:05:06.708837+0000","last_undegraded":"2024-09-02T10:05:06.708837+0000","last_fullsized":"2024-09-02T10:05:06.708837+0000","mapping_epoch":92,"log_start":"98'700","ondisk_log_start":"98'700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10296","last_scrub_stamp":"2024-09-02T10:04:54.926873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.926873+0000","objects_scrubbed":2056,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.100584+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105906176,"num_objects":1616,"num_object_clones":0,"num_object_copies":4848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10726,"num_write_kb":394944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":49,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.824167+0000","last_change":"2024-09-02T10:04:55.111033+0000","last_active":"2024-09-02T10:05:09.824167+0000","last_peered":"2024-09-02T10:05:09.824167+0000","last_clean":"2024-09-02T10:05:09.824167+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:05:09.824167+0000","last_undegraded":"2024-09-02T10:05:09.824167+0000","last_fullsized":"2024-09-02T10:05:09.824167+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:55.110979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.110979+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:01.238104+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":278,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.193356+0000","last_change":"2024-09-02T10:04:30.278703+0000","last_active":"2024-09-02T10:05:06.193356+0000","last_peered":"2024-09-02T10:05:06.193356+0000","last_clean":"2024-09-02T10:05:06.193356+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:05:06.193356+0000","last_undegraded":"2024-09-02T10:05:06.193356+0000","last_fullsized":"2024-09-02T10:05:06.193356+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:30.278632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:30.278632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:48.025871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"204'10672","reported_seq":10969,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.866803+0000","last_change":"2024-09-02T10:02:46.577683+0000","last_active":"2024-09-02T10:05:06.866803+0000","last_peered":"2024-09-02T10:05:06.866803+0000","last_clean":"2024-09-02T10:05:06.866803+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:05:06.866803+0000","last_undegraded":"2024-09-02T10:05:06.866803+0000","last_fullsized":"2024-09-02T10:05:06.866803+0000","mapping_epoch":92,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'5355","last_scrub_stamp":"2024-09-02T10:02:46.577625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:46.577625+0000","objects_scrubbed":5316,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105250816,"num_objects":1606,"num_object_clones":0,"num_object_copies":4818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1606,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10672,"num_write_kb":392896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":186,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-02T10:05:08.813905+0000","last_change":"2024-09-02T10:04:57.391785+0000","last_active":"2024-09-02T10:05:08.813905+0000","last_peered":"2024-09-02T10:05:08.813905+0000","last_clean":"2024-09-02T10:05:08.813905+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:05:08.813905+0000","last_undegraded":"2024-09-02T10:05:08.813905+0000","last_fullsized":"2024-09-02T10:05:08.813905+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.391723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.391723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:12.774478+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"205'10455","reported_seq":16575,"reported_epoch":205,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:05:09.170471+0000","last_change":"2024-09-02T10:04:57.390580+0000","last_active":"2024-09-02T10:05:09.170471+0000","last_peered":"2024-09-02T10:05:09.170471+0000","last_clean":"2024-09-02T10:03:45.188955+0000","last_became_active":"2024-09-02T10:04:57.363859+0000","last_became_peered":"2024-09-02T10:04:57.363859+0000","last_unstale":"2024-09-02T10:05:09.170471+0000","last_undegraded":"2024-09-02T10:04:57.027533+0000","last_fullsized":"2024-09-02T10:04:57.018785+0000","mapping_epoch":198,"log_start":"97'400","ondisk_log_start":"97'400","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'5182","last_scrub_stamp":"2024-09-02T10:02:48.587872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.587872+0000","objects_scrubbed":5181,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93323288,"num_objects":1425,"num_object_clones":0,"num_object_copies":4275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1425,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10455,"num_write_kb":380097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":190513152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1425}],"blocked_by":[],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":49,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.824657+0000","last_change":"2024-09-02T10:04:54.128208+0000","last_active":"2024-09-02T10:05:09.824657+0000","last_peered":"2024-09-02T10:05:09.824657+0000","last_clean":"2024-09-02T10:05:09.824657+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:05:09.824657+0000","last_undegraded":"2024-09-02T10:05:09.824657+0000","last_fullsized":"2024-09-02T10:05:09.824657+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:54.128142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.128142+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.955526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"204'10715","reported_seq":17466,"reported_epoch":204,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:07.464027+0000","last_change":"2024-09-02T10:05:07.464027+0000","last_active":"2024-09-02T10:05:07.464027+0000","last_peered":"2024-09-02T10:05:07.464027+0000","last_clean":"2024-09-02T10:03:38.171966+0000","last_became_active":"2024-09-02T10:04:57.182024+0000","last_became_peered":"2024-09-02T10:04:57.182024+0000","last_unstale":"2024-09-02T10:05:07.464027+0000","last_undegraded":"2024-09-02T10:05:07.464027+0000","last_fullsized":"2024-09-02T10:04:57.016771+0000","mapping_epoch":198,"log_start":"98'700","ondisk_log_start":"98'700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103612416,"num_objects":1581,"num_object_clones":0,"num_object_copies":4743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1581,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10715,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2167,"num_bytes_recovered":114360320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2,3","objects":1581}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"205'10763","reported_seq":11095,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.194092+0000","last_change":"2024-09-02T10:04:38.925752+0000","last_active":"2024-09-02T10:05:09.194092+0000","last_peered":"2024-09-02T10:05:09.194092+0000","last_clean":"2024-09-02T10:05:09.194092+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:05:09.194092+0000","last_undegraded":"2024-09-02T10:05:09.194092+0000","last_fullsized":"2024-09-02T10:05:09.194092+0000","mapping_epoch":92,"log_start":"98'700","ondisk_log_start":"98'700","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9618","last_scrub_stamp":"2024-09-02T10:04:38.925565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.925565+0000","objects_scrubbed":2671,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.842942+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99549184,"num_objects":1519,"num_object_clones":0,"num_object_copies":4557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10763,"num_write_kb":393024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":58,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-02T10:05:06.197430+0000","last_change":"2024-09-02T10:04:35.814407+0000","last_active":"2024-09-02T10:05:06.197430+0000","last_peered":"2024-09-02T10:05:06.197430+0000","last_clean":"2024-09-02T10:05:06.197430+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:05:06.197430+0000","last_undegraded":"2024-09-02T10:05:06.197430+0000","last_fullsized":"2024-09-02T10:05:06.197430+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:46.975807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:46.975807+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1620639768,"num_objects":24730,"num_object_clones":0,"num_object_copies":74190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24730,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":170768,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29561,"num_bytes_recovered":1724973056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6062419968,"data_stored":6062407752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160768,"ondisk_log_size":160768,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383740,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595408,"kb_used_data":465660,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93149744,"statfs":{"total":95995035648,"available":95385337856,"internally_reserved":0,"allocated":476835840,"data_stored":476001095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":132839876},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841606,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1240848,"kb_used_data":1129152,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92504304,"statfs":{"total":95995035648,"available":94724407296,"internally_reserved":0,"allocated":1156251648,"data_stored":1155412839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":114358720},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070540,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":467724,"kb_used_data":400444,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93277428,"statfs":{"total":95995035648,"available":95516086272,"internally_reserved":0,"allocated":410054656,"data_stored":409219919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299471,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":571620,"kb_used_data":468244,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93173532,"statfs":{"total":95995035648,"available":95409696768,"internally_reserved":0,"allocated":479481856,"data_stored":478638159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528403,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":409744,"kb_used_data":346620,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93335408,"statfs":{"total":95995035648,"available":95575457792,"internally_reserved":0,"allocated":354938880,"data_stored":354104143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724632,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":281944,"kb_used_data":214660,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93463208,"statfs":{"total":95995035648,"available":95706324992,"internally_reserved":0,"allocated":219811840,"data_stored":218973551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953566,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":369680,"kb_used_data":294080,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93375472,"statfs":{"total":95995035648,"available":95616483328,"internally_reserved":0,"allocated":301137920,"data_stored":300299111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149796,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":973532,"kb_used_data":843784,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92771620,"statfs":{"total":95995035648,"available":94998138880,"internally_reserved":0,"allocated":864034816,"data_stored":863192455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":132839880},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":693208,"kb_used_data":557764,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93051944,"statfs":{"total":95995035648,"available":95285190656,"internally_reserved":0,"allocated":571150336,"data_stored":570312047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":588496,"kb_used_data":512892,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93156656,"statfs":{"total":95995035648,"available":95392415744,"internally_reserved":0,"allocated":525201408,"data_stored":524366671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":619472,"kb_used_data":525820,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93125680,"statfs":{"total":95995035648,"available":95360696320,"internally_reserved":0,"allocated":538439680,"data_stored":537604943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532742144,"data_stored":532742144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":520749056,"data_stored":520749056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":561184768,"data_stored":561184768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":845746176,"data_stored":845742104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":292425728,"data_stored":292421656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":214368256,"data_stored":214368256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":349437952,"data_stored":349437952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":469303296,"data_stored":469303296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":399900672,"data_stored":399900672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1132072960,"data_stored":1132068888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":275513344,"data_stored":275513344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":468975616,"data_stored":468975616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:05:13.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:05:13.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:13.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:13.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:13.716 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:05:13.716 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:05:13.716 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:05:13.716 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:05:13.716 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:05:13.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:13.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:13.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:13.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:13.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:13.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:13.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:14.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:14.077 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:14.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:14.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:14.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:14.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:14.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:14 smithi151 ceph-mon[16639]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:14.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:14 smithi151 ceph-mon[16639]: pgmap v513: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 807 op/s; 2229/67788 objects degraded (3.288%); 0 B/s, 148 objects/s recovering 2024-09-02T10:05:14.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2229/67788 objects degraded (3.288%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:05:14.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:14 smithi151 ceph-mon[16639]: osdmap e207: 12 total, 12 up, 11 in 2024-09-02T10:05:14.280 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:14.316 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:14.341 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:14.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:14.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:14.465 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:14 smithi187 ceph-mon[20072]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:14.466 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:14 smithi187 ceph-mon[20072]: pgmap v513: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 807 op/s; 2229/67788 objects degraded (3.288%); 0 B/s, 148 objects/s recovering 2024-09-02T10:05:14.466 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2229/67788 objects degraded (3.288%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:05:14.466 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:14 smithi187 ceph-mon[20072]: osdmap e207: 12 total, 12 up, 11 in 2024-09-02T10:05:14.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:05:14.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:14.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:14.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:14 smithi162 ceph-mon[20279]: from='client.15192 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:14 smithi162 ceph-mon[20279]: pgmap v513: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 1.4 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 807 op/s; 2229/67788 objects degraded (3.288%); 0 B/s, 148 objects/s recovering 2024-09-02T10:05:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2229/67788 objects degraded (3.288%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:05:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:14 smithi162 ceph-mon[20279]: osdmap e207: 12 total, 12 up, 11 in 2024-09-02T10:05:14.599 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:14.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:14.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:14.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:14.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:14.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:14.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:14.806 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:14.820 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:05:14.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:14.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:14.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:14.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:05:14.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:14.997 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:14.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:15.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:15.123 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:15.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:15.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:15.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:15.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:15.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:15.254 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:05:15.254 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2229/67788 objects degraded (3.288%), 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":587,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":208,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":11,"osd_in_since":1725271367,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":33},{"state_name":"active+recovering+undersized+remapped","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":37,"num_pools":2,"num_objects":22596,"data_bytes":1481245240,"bytes_used":6502088704,"bytes_avail":1049443303424,"bytes_total":1055945392128,"degraded_objects":2229,"degraded_total":67788,"degraded_ratio":0.032881926004602588,"recovering_objects_per_sec":148,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1032,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":807},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":21,"modified":"2024-09-02T10:05:11.115338+0000","services":{}},"progress_events":{}} 2024-09-02T10:05:15.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:15.382 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:15.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:15.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:15 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:05:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:15 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:05:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3956557166' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:15 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3956557166' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:15 smithi187 ceph-mon[20072]: osdmap e208: 12 total, 12 up, 11 in 2024-09-02T10:05:15.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:15.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:15.560 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:15 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:05:15.560 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:15 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:05:15.560 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3956557166' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:15.560 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:15 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:15.560 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3956557166' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:15.560 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:15 smithi151 ceph-mon[16639]: osdmap e208: 12 total, 12 up, 11 in 2024-09-02T10:05:15.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:15 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:05:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:15 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:05:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3956557166' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:15 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3956557166' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:15 smithi162 ceph-mon[20279]: osdmap e208: 12 total, 12 up, 11 in 2024-09-02T10:05:15.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:15.617 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:05:15.617 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:05:15.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1131 2.0 e96/92 2:021aab7d:::benchmark_data_smithi187_45882_object1130:head v 96'61, mlcod=96'58)", 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 219.46142104800001, 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077611448, 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898452+0000", 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8899999999999998e-07 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898480+0000", 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.817e-05 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975643+0000", 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077163063000000004 2024-09-02T10:05:15.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975651+0000", 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.028e-06 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975697+0000", 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5544999999999998e-05 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975710+0000", 2024-09-02T10:05:15.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2928000000000001e-05 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.976007+0000", 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029711499999999998 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.976063+0000", 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5810000000000003e-05 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.622 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.623 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.623 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1142 2.7 e96/92 2:e2cb33eb:::benchmark_data_smithi187_45882_object1141:head v 96'87, mlcod=96'83)", 2024-09-02T10:05:15.623 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:05:15.623 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 219.4529187, 2024-09-02T10:05:15.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18675391099999999, 2024-09-02T10:05:15.623 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.623 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.623 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906955+0000", 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.181e-06 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906991+0000", 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6523999999999998e-05 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093330+0000", 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18633878100000001 2024-09-02T10:05:15.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093335+0000", 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5909999999999998e-06 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093381+0000", 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6925999999999998e-05 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093396+0000", 2024-09-02T10:05:15.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4216000000000001e-05 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093682+0000", 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028659900000000001 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093707+0000", 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5092999999999998e-05 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.627 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.628 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1442 2.0 e96/92 2:04aa890d:::benchmark_data_smithi187_45882_object1441:head v 96'80, mlcod=96'77)", 2024-09-02T10:05:15.628 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:05:15.628 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 219.170503346, 2024-09-02T10:05:15.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087334971999999997, 2024-09-02T10:05:15.628 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.628 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.628 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.628 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8100000000000002e-07 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189467+0000", 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7173e-05 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276310+0000", 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.086843528000000003 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276315+0000", 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8670000000000002e-06 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276365+0000", 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9459999999999997e-05 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276382+0000", 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7045999999999999e-05 2024-09-02T10:05:15.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276666+0000", 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028404500000000002 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276704+0000", 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8172e-05 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11303 2.7 e98/92 2:eeb591e6:::benchmark_data_smithi187_45882_object11302:head v 98'755, mlcod=98'751)", 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 213.04552517600001, 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.117126889, 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:05:15.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314348+0000", 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8599999999999998e-07 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314428+0000", 2024-09-02T10:05:15.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0412000000000002e-05 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430860+0000", 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.116432137 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430866+0000", 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.083e-06 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430918+0000", 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.215e-05 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430932+0000", 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3591000000000001e-05 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.431445+0000", 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051315900000000001 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.431474+0000", 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8671e-05 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11334 2.4 e98/92 2:22fc355e:::benchmark_data_smithi187_45882_object11333:head v 98'688, mlcod=98'684)", 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 213.00620477000001, 2024-09-02T10:05:15.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.072501649000000001, 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:05:15.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353668+0000", 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5899999999999995e-07 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353720+0000", 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1539999999999998e-05 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425505+0000", 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.071785287000000003 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425512+0000", 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2980000000000003e-06 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425576+0000", 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3614000000000005e-05 2024-09-02T10:05:15.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425598+0000", 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2076e-05 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426142+0000", 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00054422300000000004 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426169+0000", 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6852e-05 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11352 2.4 e98/92 2:2477406b:::benchmark_data_smithi187_45882_object11351:head v 98'690, mlcod=98'686)", 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 212.93319468300001, 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11620186, 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.642 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426679+0000", 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2580000000000001e-06 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426733+0000", 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4398999999999999e-05 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542090+0000", 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.115357134 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542105+0000", 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4596999999999999e-05 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542170+0000", 2024-09-02T10:05:15.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5376999999999994e-05 2024-09-02T10:05:15.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542194+0000", 2024-09-02T10:05:15.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3611000000000001e-05 2024-09-02T10:05:15.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542825+0000", 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00063158400000000003 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542879+0000", 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3900000000000002e-05 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11358 2.7 e98/92 2:e5a3134d:::benchmark_data_smithi187_45882_object11357:head v 98'756, mlcod=98'753)", 2024-09-02T10:05:15.647 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 212.929082669, 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078694929999999996, 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430790+0000", 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7499999999999999e-07 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430846+0000", 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5544999999999998e-05 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508940+0000", 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078093869999999996 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508946+0000", 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7120000000000003e-06 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508990+0000", 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4095999999999999e-05 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509005+0000", 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5000999999999999e-05 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509456+0000", 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00045036799999999998 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509484+0000", 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8563000000000001e-05 2024-09-02T10:05:15.651 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11782 2.4 e98/92 2:2417a9e9:::benchmark_data_smithi187_45882_object11781:head v 98'718, mlcod=98'714)", 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 212.61687967699999, 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12480622099999999, 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742993+0000", 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2399999999999997e-07 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.743050+0000", 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6356e-05 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867097+0000", 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.124047 2024-09-02T10:05:15.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867103+0000", 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0399999999999998e-06 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867172+0000", 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8996000000000001e-05 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867187+0000", 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5218999999999999e-05 2024-09-02T10:05:15.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867764+0000", 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00057719100000000001 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867799+0000", 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4694999999999999e-05 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.656 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11788 2.7 e98/92 2:ee0a4980:::benchmark_data_smithi187_45882_object11787:head v 98'788, mlcod=98'785)", 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 212.60959590600001, 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.134742468, 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:05:15.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750278+0000", 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5430000000000001e-06 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750323+0000", 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4706000000000002e-05 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884234+0000", 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13391187800000001 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884240+0000", 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4190000000000001e-06 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884280+0000", 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0012000000000002e-05 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884316+0000", 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6318e-05 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884977+0000", 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00066127600000000003 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.885019+0000", 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1316000000000002e-05 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11794 2.4 e98/92 2:22bc5fbb:::benchmark_data_smithi187_45882_object11793:head v 98'719, mlcod=98'715)", 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 212.60252306699999, 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.125805317, 2024-09-02T10:05:15.661 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:05:15.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757350+0000", 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5300000000000003e-07 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757408+0000", 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8291000000000001e-05 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882534+0000", 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.125126182 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882540+0000", 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0689999999999998e-06 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882583+0000", 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2962e-05 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882599+0000", 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5718000000000001e-05 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.883121+0000", 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052199199999999999 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.883154+0000", 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3349999999999997e-05 2024-09-02T10:05:15.665 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11897 2.4 e98/92 2:2a9822cc:::benchmark_data_smithi187_45882_object11896:head v 98'727, mlcod=98'723)", 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 212.400842856, 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088729108000000001, 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959030+0000", 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0200000000000001e-07 2024-09-02T10:05:15.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959076+0000", 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5896000000000002e-05 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047130+0000", 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088054190000000004 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047135+0000", 2024-09-02T10:05:15.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2379999999999997e-06 2024-09-02T10:05:15.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047218+0000", 2024-09-02T10:05:15.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2892000000000005e-05 2024-09-02T10:05:15.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047233+0000", 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5126999999999999e-05 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047729+0000", 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000495847 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047758+0000", 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9116e-05 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:13212 2.0 e98/92 2:00f0b824:::benchmark_data_smithi187_45882_object13211:head v 98'832, mlcod=98'829)", 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 211.58815791399999, 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.075308911000000006, 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.671 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771715+0000", 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1599999999999999e-06 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771775+0000", 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9979999999999998e-05 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846360+0000", 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.074584792999999996 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846367+0000", 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1489999999999997e-06 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846446+0000", 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8652999999999994e-05 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846461+0000", 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5254e-05 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846996+0000", 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053431400000000003 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.847023+0000", 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7608000000000001e-05 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.675 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:13565 2.0 e98/92 2:0460cc71:::benchmark_data_smithi187_45882_object13564:head v 98'854, mlcod=98'850)", 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 211.29864544, 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29179487700000001, 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9400000000000005e-07 2024-09-02T10:05:15.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061284+0000", 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6251000000000001e-05 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352274+0000", 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29099082599999998 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352280+0000", 2024-09-02T10:05:15.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1950000000000002e-06 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352346+0000", 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5837e-05 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352360+0000", 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.433e-05 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.679 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352980+0000", 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00062060200000000005 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.353022+0000", 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1142e-05 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:22449 2.0 e104/92 2:0733b693:::benchmark_data_smithi187_45882_object22448:head v 104'1397, mlcod=104'1393)", 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:05:15.680 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 204.07916233399999, 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12946363599999999, 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:05:15.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280711+0000", 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1800000000000005e-07 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280779+0000", 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8644000000000002e-05 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409376+0000", 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12859704799999999 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409382+0000", 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5589999999999996e-06 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409441+0000", 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8746000000000003e-05 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409466+0000", 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5493000000000001e-05 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.410146+0000", 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00067948499999999996 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.410173+0000", 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7943e-05 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.684 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:22536 2.0 e104/92 2:01400da2:::benchmark_data_smithi187_45882_object22535:head v 104'1399, mlcod=104'1395)", 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 203.91827356300001, 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077533989999999997, 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:05:15.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.47e-07 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441696+0000", 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6059999999999998e-05 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518264+0000", 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.076568862000000001 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518272+0000", 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5789999999999998e-06 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518333+0000", 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1159000000000002e-05 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518358+0000", 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5182e-05 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.519104+0000", 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00074525100000000005 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.519133+0000", 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9050000000000001e-05 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:36974 2.7 e110/92 2:ea266fd4:::benchmark_data_smithi187_45882_object36973:head v 110'2332, mlcod=110'2328)", 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 192.75992281200001, 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080151230000000004, 2024-09-02T10:05:15.689 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599953+0000", 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6090000000000001e-06 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599999+0000", 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5561e-05 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679063+0000", 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.079064375000000006 2024-09-02T10:05:15.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679068+0000", 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3619999999999997e-06 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679116+0000", 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8084999999999998e-05 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679132+0000", 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5264000000000002e-05 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.680070+0000", 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00093846200000000004 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.680101+0000", 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0511999999999999e-05 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.693 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:37371 2.0 e110/92 2:00cc6826:::benchmark_data_smithi187_45882_object37370:head v 110'2365, mlcod=110'2361)", 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 192.23154433299999, 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083380836999999999, 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:05:15.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128329+0000", 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.207e-06 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128397+0000", 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7869999999999999e-05 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210301+0000", 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081903702999999994 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210307+0000", 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.737e-06 2024-09-02T10:05:15.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210368+0000", 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0795000000000003e-05 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210383+0000", 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4538999999999999e-05 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.211539+0000", 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011564310000000001 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.211709+0000", 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000169555 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:67961 2.7 e120/92 2:ed776389:::benchmark_data_smithi187_45882_object67960:head v 120'4269, mlcod=120'4266)", 2024-09-02T10:05:15.698 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 165.45116687399999, 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.095923729999999999, 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908706+0000", 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1399999999999997e-07 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908770+0000", 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4410000000000002e-05 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002909+0000", 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.094138870999999999 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002918+0000", 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3489999999999992e-06 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002977+0000", 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9299999999999998e-05 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002993+0000", 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6424000000000001e-05 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.004595+0000", 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001601725 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.004629+0000", 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4037e-05 2024-09-02T10:05:15.702 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:78685 2.4 e124/92 2:26d5ebbe:::benchmark_data_smithi187_45882_object78684:head v 124'4867, mlcod=124'4864)", 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 154.91137767699999, 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.135791625, 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448496+0000", 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.257e-06 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448603+0000", 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106925 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.581994+0000", 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13339121900000001 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582001+0000", 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7950000000000002e-06 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582057+0000", 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6724000000000001e-05 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582074+0000", 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7037999999999998e-05 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.584242+0000", 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021673809999999999 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.584286+0000", 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4286000000000003e-05 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:79093 2.4 e125/92 2:2f2be3a1:::benchmark_data_smithi187_45882_object79092:head v 125'4892, mlcod=125'4888)", 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 154.40694470700001, 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16453667699999999, 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:15.707 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952928+0000", 2024-09-02T10:05:15.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952991+0000", 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3367999999999994e-05 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115366+0000", 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16237467799999999 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115374+0000", 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6240000000000002e-06 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115428+0000", 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3885999999999999e-05 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115442+0000", 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4438e-05 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.117418+0000", 2024-09-02T10:05:15.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019764600000000002 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.117464+0000", 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5694e-05 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:05:15.711 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:05:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:15.712 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:15.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:15.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:15.837 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:15.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:15.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:15.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:15.962 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:15.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:15.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:16.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:16.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:16.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:16.194 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:16.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:16.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:16.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:16.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:16.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:16.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:16.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:16.464 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:16 smithi151 ceph-mon[16639]: pgmap v516: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 1.3 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 2229/65208 objects degraded (3.418%); 0 B/s, 134 objects/s recovering 2024-09-02T10:05:16.464 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/948028457' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:05:16.464 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:16 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:05:16.465 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:16 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:05:16.474 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:16 smithi187 ceph-mon[20072]: pgmap v516: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 1.3 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 2229/65208 objects degraded (3.418%); 0 B/s, 134 objects/s recovering 2024-09-02T10:05:16.474 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/948028457' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:05:16.474 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:16 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:05:16.474 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:16 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:05:16.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:16.491 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:16.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:16.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:16 smithi162 ceph-mon[20279]: pgmap v516: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 1.3 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 711 op/s; 2229/65208 objects degraded (3.418%); 0 B/s, 134 objects/s recovering 2024-09-02T10:05:16.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/948028457' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:05:16.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:16 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:05:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:16 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:05:16.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:16.618 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:16.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:16.697 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:05:16.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:05:16.713 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:16.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:16.742 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:16.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:16.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:16.866 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:16.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:16.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:05:16.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:17.013 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:17.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:17.014 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:17.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:17.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:17.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:17.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:17.279 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:17.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:17.280 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:17.284 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:17.346 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:17.364 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:05:17.365 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:05:17.381 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:17.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:17.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:17.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:17.407 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:05:17.407 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:05:17.441 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:05:17.442 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:05:17.467 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:05:17.468 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:05:17.488 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:05:17.489 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:05:17.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:17.509 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:05:17.509 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:05:17.528 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:17.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:17.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:17.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:05:17.533 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:05:17.563 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:05:17.563 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:05:17.603 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:05:17.603 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:05:17.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:05:17.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:17.642 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:05:17.642 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:05:17.684 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:05:17.684 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:05:17.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:17.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:17.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:17.749 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:05:17.749 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:05:17.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:17.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:17.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:17.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:17.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:17.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:17 smithi187 ceph-mon[20072]: osdmap e209: 12 total, 12 up, 11 in 2024-09-02T10:05:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:17 smithi187 ceph-mon[20072]: pgmap v518: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 1.2 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 748 op/s; 2229/60711 objects degraded (3.671%); 0 B/s, 141 objects/s recovering 2024-09-02T10:05:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4001093019' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:18.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:18.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:17 smithi162 ceph-mon[20279]: osdmap e209: 12 total, 12 up, 11 in 2024-09-02T10:05:18.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:17 smithi162 ceph-mon[20279]: pgmap v518: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 1.2 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 748 op/s; 2229/60711 objects degraded (3.671%); 0 B/s, 141 objects/s recovering 2024-09-02T10:05:18.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4001093019' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:18.103 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:18.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:18.171 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:17 smithi151 ceph-mon[16639]: osdmap e209: 12 total, 12 up, 11 in 2024-09-02T10:05:18.171 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:17 smithi151 ceph-mon[16639]: pgmap v518: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 1.2 GiB data, 5.6 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 748 op/s; 2229/60711 objects degraded (3.671%); 0 B/s, 141 objects/s recovering 2024-09-02T10:05:18.171 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4001093019' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:18.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:18.229 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:18.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:18.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:18.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:18.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:18.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:18.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:18.497 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:18.498 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:18.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:18.626 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:05:18.626 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:05:18.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:18.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:18.697 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:05:18.698 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:05:18.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:18.763 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:18.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:18.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:18.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:18.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:18.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:19.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:18 smithi151 ceph-mon[16639]: from='client.24965 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:19.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:18 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2229/60711 objects degraded (3.671%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:05:19.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:19.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:18 smithi162 ceph-mon[20279]: from='client.24965 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:18 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2229/60711 objects degraded (3.671%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:05:19.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:19.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:19.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:19.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:19.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:18 smithi187 ceph-mon[20072]: from='client.24965 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:18 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2229/60711 objects degraded (3.671%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:05:19.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:19.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:19.382 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:19.407 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:19.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:19.408 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:19.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:05:19.509 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:19.557 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:19.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:19.655 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-09-02T10:05:17.119833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1326645816,"num_objects":20237,"num_object_clones":0,"num_object_copies":60711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20237,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175263,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30411,"num_bytes_recovered":1725703232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162495,"ondisk_log_size":162495,"up":111,"acting":107,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":5918716,"kb_used_data":4869048,"kb_used_omap":17,"kb_used_meta":1049454,"kb_avail":1025277956,"statfs":{"total":1055945392128,"available":1049884626944,"internally_reserved":0,"allocated":4985905152,"data_stored":4976557055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1074641787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,1,2,0,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-294584320,"num_objects":-4495,"num_object_clones":0,"num_object_copies":-13485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4495,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":847,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.004579"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":180,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.823461+0000","last_change":"2024-09-02T10:02:43.131354+0000","last_active":"2024-09-02T10:05:09.823461+0000","last_peered":"2024-09-02T10:05:09.823461+0000","last_clean":"2024-09-02T10:05:09.823461+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:05:09.823461+0000","last_undegraded":"2024-09-02T10:05:09.823461+0000","last_fullsized":"2024-09-02T10:05:09.823461+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.131299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.131299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":265,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.832495+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:05:09.832495+0000","last_peered":"2024-09-02T10:05:09.832495+0000","last_clean":"2024-09-02T10:05:09.832495+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:05:09.832495+0000","last_undegraded":"2024-09-02T10:05:09.832495+0000","last_fullsized":"2024-09-02T10:05:09.832495+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.233390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":208,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.826667+0000","last_change":"2024-09-02T10:04:37.422274+0000","last_active":"2024-09-02T10:05:16.826667+0000","last_peered":"2024-09-02T10:05:16.826667+0000","last_clean":"2024-09-02T10:05:16.826667+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:05:16.826667+0000","last_undegraded":"2024-09-02T10:05:16.826667+0000","last_fullsized":"2024-09-02T10:05:16.826667+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:37.422128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:37.422128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:02.404595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":186,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-02T10:05:14.824576+0000","last_change":"2024-09-02T10:02:42.836988+0000","last_active":"2024-09-02T10:05:14.824576+0000","last_peered":"2024-09-02T10:05:14.824576+0000","last_clean":"2024-09-02T10:05:14.824576+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:05:14.824576+0000","last_undegraded":"2024-09-02T10:05:14.824576+0000","last_fullsized":"2024-09-02T10:05:14.824576+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:42.836911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:42.836911+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":201,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.826609+0000","last_change":"2024-09-02T10:04:38.471750+0000","last_active":"2024-09-02T10:05:16.826609+0000","last_peered":"2024-09-02T10:05:16.826609+0000","last_clean":"2024-09-02T10:05:16.826609+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:05:16.826609+0000","last_undegraded":"2024-09-02T10:05:16.826609+0000","last_fullsized":"2024-09-02T10:05:16.826609+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:38.471681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.471681+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:49.847941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"207'10855","reported_seq":11189,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-02T10:05:14.169760+0000","last_change":"2024-09-02T10:04:36.789912+0000","last_active":"2024-09-02T10:05:14.169760+0000","last_peered":"2024-09-02T10:05:14.169760+0000","last_clean":"2024-09-02T10:05:14.169760+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:05:14.169760+0000","last_undegraded":"2024-09-02T10:05:14.169760+0000","last_fullsized":"2024-09-02T10:05:14.169760+0000","mapping_epoch":92,"log_start":"98'800","ondisk_log_start":"98'800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'9455","last_scrub_stamp":"2024-09-02T10:04:36.789821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:36.789821+0000","objects_scrubbed":2703,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:41.193203+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85393408,"num_objects":1303,"num_object_clones":0,"num_object_copies":3909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10855,"num_write_kb":389056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":273,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.826503+0000","last_change":"2024-09-02T10:04:36.384243+0000","last_active":"2024-09-02T10:05:16.826503+0000","last_peered":"2024-09-02T10:05:16.826503+0000","last_clean":"2024-09-02T10:05:16.826503+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:05:16.826503+0000","last_undegraded":"2024-09-02T10:05:16.826503+0000","last_fullsized":"2024-09-02T10:05:16.826503+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:36.384163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:36.384163+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:37.175901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"209'11044","reported_seq":11363,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.878866+0000","last_change":"2024-09-02T10:04:56.617908+0000","last_active":"2024-09-02T10:05:16.878866+0000","last_peered":"2024-09-02T10:05:16.878866+0000","last_clean":"2024-09-02T10:05:16.878866+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:05:16.878866+0000","last_undegraded":"2024-09-02T10:05:16.878866+0000","last_fullsized":"2024-09-02T10:05:16.878866+0000","mapping_epoch":92,"log_start":"99'1000","ondisk_log_start":"99'1000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10186","last_scrub_stamp":"2024-09-02T10:04:56.617746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.617746+0000","objects_scrubbed":2052,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:15.697367+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78249984,"num_objects":1194,"num_object_clones":0,"num_object_copies":3582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11044,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":394,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-02T10:05:14.821785+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:05:14.821785+0000","last_peered":"2024-09-02T10:05:14.821785+0000","last_clean":"2024-09-02T10:05:14.821785+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:05:14.821785+0000","last_undegraded":"2024-09-02T10:05:14.821785+0000","last_fullsized":"2024-09-02T10:05:14.821785+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"206'10921","reported_seq":11189,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:12.845080+0000","last_change":"2024-09-02T10:04:55.460032+0000","last_active":"2024-09-02T10:05:12.845080+0000","last_peered":"2024-09-02T10:05:12.845080+0000","last_clean":"2024-09-02T10:05:12.845080+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:05:12.845080+0000","last_undegraded":"2024-09-02T10:05:12.845080+0000","last_fullsized":"2024-09-02T10:05:12.845080+0000","mapping_epoch":92,"log_start":"98'900","ondisk_log_start":"98'900","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10251","last_scrub_stamp":"2024-09-02T10:04:55.459770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.459770+0000","objects_scrubbed":2066,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.035919+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90767360,"num_objects":1385,"num_object_clones":0,"num_object_copies":4155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10921,"num_write_kb":393792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":34,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.827001+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:05:09.827001+0000","last_peered":"2024-09-02T10:05:09.827001+0000","last_clean":"2024-09-02T10:05:09.827001+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:05:09.827001+0000","last_undegraded":"2024-09-02T10:05:09.827001+0000","last_fullsized":"2024-09-02T10:05:09.827001+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"208'11004","reported_seq":19886,"reported_epoch":208,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:15.836031+0000","last_change":"2024-09-02T10:04:57.234268+0000","last_active":"2024-09-02T10:05:15.836031+0000","last_peered":"2024-09-02T10:05:15.836031+0000","last_clean":"2024-09-02T10:03:38.126822+0000","last_became_active":"2024-09-02T10:04:57.195493+0000","last_became_peered":"2024-09-02T10:04:57.195493+0000","last_unstale":"2024-09-02T10:05:15.836031+0000","last_undegraded":"2024-09-02T10:05:15.836031+0000","last_fullsized":"2024-09-02T10:04:57.070642+0000","mapping_epoch":198,"log_start":"99'1000","ondisk_log_start":"99'1000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":1216,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11004,"num_write_kb":391040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4315,"num_bytes_recovered":208863232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1216}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":186,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.832748+0000","last_change":"2024-09-02T10:02:43.245406+0000","last_active":"2024-09-02T10:05:16.832748+0000","last_peered":"2024-09-02T10:05:16.832748+0000","last_clean":"2024-09-02T10:05:16.832748+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:05:16.832748+0000","last_undegraded":"2024-09-02T10:05:16.832748+0000","last_fullsized":"2024-09-02T10:05:16.832748+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.245354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.245354+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"209'11014","reported_seq":21972,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.957928+0000","last_change":"2024-09-02T10:04:27.932509+0000","last_active":"2024-09-02T10:05:16.957928+0000","last_peered":"2024-09-02T10:05:16.957928+0000","last_clean":"2024-09-02T10:05:16.957928+0000","last_became_active":"2024-09-02T10:04:26.448185+0000","last_became_peered":"2024-09-02T10:04:26.448185+0000","last_unstale":"2024-09-02T10:05:16.957928+0000","last_undegraded":"2024-09-02T10:05:16.957928+0000","last_fullsized":"2024-09-02T10:05:16.957928+0000","mapping_epoch":176,"log_start":"99'1000","ondisk_log_start":"99'1000","created":92,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9259","last_scrub_stamp":"2024-09-02T10:04:27.932362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:27.932362+0000","objects_scrubbed":2896,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:56.556476+0000","scrub_duration":606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73531392,"num_objects":1122,"num_object_clones":0,"num_object_copies":3366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11014,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":349503488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":342,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.832677+0000","last_change":"2024-09-02T10:02:41.206594+0000","last_active":"2024-09-02T10:05:16.832677+0000","last_peered":"2024-09-02T10:05:16.832677+0000","last_clean":"2024-09-02T10:05:16.832677+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:05:16.832677+0000","last_undegraded":"2024-09-02T10:05:16.832677+0000","last_fullsized":"2024-09-02T10:05:16.832677+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:41.206524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:41.206524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"206'10832","reported_seq":11126,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:12.858003+0000","last_change":"2024-09-02T10:04:56.300674+0000","last_active":"2024-09-02T10:05:12.858003+0000","last_peered":"2024-09-02T10:05:12.858003+0000","last_clean":"2024-09-02T10:05:12.858003+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:05:12.858003+0000","last_undegraded":"2024-09-02T10:05:12.858003+0000","last_fullsized":"2024-09-02T10:05:12.858003+0000","mapping_epoch":92,"log_start":"98'800","ondisk_log_start":"98'800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10171","last_scrub_stamp":"2024-09-02T10:04:56.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.300400+0000","objects_scrubbed":2044,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:07.471560+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90570752,"num_objects":1382,"num_object_clones":0,"num_object_copies":4146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10832,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":174,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.823726+0000","last_change":"2024-09-02T10:02:49.000327+0000","last_active":"2024-09-02T10:05:09.823726+0000","last_peered":"2024-09-02T10:05:09.823726+0000","last_clean":"2024-09-02T10:05:09.823726+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:05:09.823726+0000","last_undegraded":"2024-09-02T10:05:09.823726+0000","last_fullsized":"2024-09-02T10:05:09.823726+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:49.000247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:49.000247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"207'10826","reported_seq":11162,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-02T10:05:14.032454+0000","last_change":"2024-09-02T10:05:04.366916+0000","last_active":"2024-09-02T10:05:14.032454+0000","last_peered":"2024-09-02T10:05:14.032454+0000","last_clean":"2024-09-02T10:05:14.032454+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:05:14.032454+0000","last_undegraded":"2024-09-02T10:05:14.032454+0000","last_fullsized":"2024-09-02T10:05:14.032454+0000","mapping_epoch":92,"log_start":"98'800","ondisk_log_start":"98'800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'10418","last_scrub_stamp":"2024-09-02T10:05:04.366613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:04.366613+0000","objects_scrubbed":1736,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:11.371378+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86114304,"num_objects":1314,"num_object_clones":0,"num_object_copies":3942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10826,"num_write_kb":388480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":34,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.827331+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:05:09.827331+0000","last_peered":"2024-09-02T10:05:09.827331+0000","last_clean":"2024-09-02T10:05:09.827331+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:05:09.827331+0000","last_undegraded":"2024-09-02T10:05:09.827331+0000","last_fullsized":"2024-09-02T10:05:09.827331+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.611307+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"206'10778","reported_seq":21651,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:12.841275+0000","last_change":"2024-09-02T10:05:00.573995+0000","last_active":"2024-09-02T10:05:12.841275+0000","last_peered":"2024-09-02T10:05:12.841275+0000","last_clean":"2024-09-02T10:05:12.841275+0000","last_became_active":"2024-09-02T10:04:43.806359+0000","last_became_peered":"2024-09-02T10:04:43.806359+0000","last_unstale":"2024-09-02T10:05:12.841275+0000","last_undegraded":"2024-09-02T10:05:12.841275+0000","last_fullsized":"2024-09-02T10:05:12.841275+0000","mapping_epoch":188,"log_start":"98'700","ondisk_log_start":"98'700","created":92,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'10224","last_scrub_stamp":"2024-09-02T10:05:00.573420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:00.573420+0000","objects_scrubbed":1920,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.148546+0000","scrub_duration":803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88997888,"num_objects":1358,"num_object_clones":0,"num_object_copies":4074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10778,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5298,"num_bytes_recovered":347209728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":303,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.843050+0000","last_change":"2024-09-02T10:04:35.810947+0000","last_active":"2024-09-02T10:05:16.843050+0000","last_peered":"2024-09-02T10:05:16.843050+0000","last_clean":"2024-09-02T10:05:16.843050+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:05:16.843050+0000","last_undegraded":"2024-09-02T10:05:16.843050+0000","last_fullsized":"2024-09-02T10:05:16.843050+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:28.355594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:28.355594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:51.765081+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"208'11302","reported_seq":11646,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.826517+0000","last_change":"2024-09-02T10:04:58.724992+0000","last_active":"2024-09-02T10:05:16.826517+0000","last_peered":"2024-09-02T10:05:16.826517+0000","last_clean":"2024-09-02T10:05:16.826517+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:05:16.826517+0000","last_undegraded":"2024-09-02T10:05:16.826517+0000","last_fullsized":"2024-09-02T10:05:16.826517+0000","mapping_epoch":92,"log_start":"101'1300","ondisk_log_start":"101'1300","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'10526","last_scrub_stamp":"2024-09-02T10:04:58.724845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.724845+0000","objects_scrubbed":2020,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:18.034443+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81264640,"num_objects":1240,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11302,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":26,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.827004+0000","last_change":"2024-09-02T10:04:57.021532+0000","last_active":"2024-09-02T10:05:09.827004+0000","last_peered":"2024-09-02T10:05:09.827004+0000","last_clean":"2024-09-02T10:05:09.827004+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:05:09.827004+0000","last_undegraded":"2024-09-02T10:05:09.827004+0000","last_fullsized":"2024-09-02T10:05:09.827004+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:38.873802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:38.873802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"206'10920","reported_seq":17354,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:12.466918+0000","last_change":"2024-09-02T10:04:58.839358+0000","last_active":"2024-09-02T10:05:12.466918+0000","last_peered":"2024-09-02T10:05:12.466918+0000","last_clean":"2024-09-02T10:05:12.466918+0000","last_became_active":"2024-09-02T10:04:58.808736+0000","last_became_peered":"2024-09-02T10:04:58.808736+0000","last_unstale":"2024-09-02T10:05:12.466918+0000","last_undegraded":"2024-09-02T10:05:12.466918+0000","last_fullsized":"2024-09-02T10:05:12.466918+0000","mapping_epoch":200,"log_start":"98'900","ondisk_log_start":"98'900","created":92,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'5361","last_scrub_stamp":"2024-09-02T10:02:48.223591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.223591+0000","objects_scrubbed":5355,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87556096,"num_objects":1336,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10920,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3059,"num_bytes_recovered":97517568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":42,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-02T10:05:14.821875+0000","last_change":"2024-09-02T10:04:47.929535+0000","last_active":"2024-09-02T10:05:14.821875+0000","last_peered":"2024-09-02T10:05:14.821875+0000","last_clean":"2024-09-02T10:05:14.821875+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:05:14.821875+0000","last_undegraded":"2024-09-02T10:05:14.821875+0000","last_fullsized":"2024-09-02T10:05:14.821875+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:39.894365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:39.894365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"208'10916","reported_seq":27275,"reported_epoch":208,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:15.835861+0000","last_change":"2024-09-02T10:04:57.232835+0000","last_active":"2024-09-02T10:05:15.835861+0000","last_peered":"2024-09-02T10:05:15.835861+0000","last_clean":"2024-09-02T10:02:27.223209+0000","last_became_active":"2024-09-02T10:04:57.144949+0000","last_became_peered":"2024-09-02T10:04:57.144949+0000","last_unstale":"2024-09-02T10:05:15.835861+0000","last_undegraded":"2024-09-02T10:05:15.835861+0000","last_fullsized":"2024-09-02T10:04:57.017574+0000","mapping_epoch":198,"log_start":"98'900","ondisk_log_start":"98'900","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75890688,"num_objects":1158,"num_object_clones":0,"num_object_copies":3474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":386368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7104,"num_bytes_recovered":417005568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,11],"avail_no_missing":["7","10","11"],"object_location_counts":[{"shards":"7,11","objects":1158}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":252,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.832593+0000","last_change":"2024-09-02T10:05:06.815241+0000","last_active":"2024-09-02T10:05:09.832593+0000","last_peered":"2024-09-02T10:05:09.832593+0000","last_clean":"2024-09-02T10:05:09.832593+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:05:09.832593+0000","last_undegraded":"2024-09-02T10:05:09.832593+0000","last_fullsized":"2024-09-02T10:05:09.832593+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:06.815191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:06.815191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:14.504648+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"208'11156","reported_seq":11506,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.826539+0000","last_change":"2024-09-02T10:04:54.927090+0000","last_active":"2024-09-02T10:05:16.826539+0000","last_peered":"2024-09-02T10:05:16.826539+0000","last_clean":"2024-09-02T10:05:16.826539+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:05:16.826539+0000","last_undegraded":"2024-09-02T10:05:16.826539+0000","last_fullsized":"2024-09-02T10:05:16.826539+0000","mapping_epoch":92,"log_start":"99'1100","ondisk_log_start":"99'1100","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10296","last_scrub_stamp":"2024-09-02T10:04:54.926873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.926873+0000","objects_scrubbed":2056,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.100584+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77725696,"num_objects":1186,"num_object_clones":0,"num_object_copies":3558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11156,"num_write_kb":394944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":53,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-02T10:05:14.822614+0000","last_change":"2024-09-02T10:04:55.111033+0000","last_active":"2024-09-02T10:05:14.822614+0000","last_peered":"2024-09-02T10:05:14.822614+0000","last_clean":"2024-09-02T10:05:14.822614+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:05:14.822614+0000","last_undegraded":"2024-09-02T10:05:14.822614+0000","last_fullsized":"2024-09-02T10:05:14.822614+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:55.110979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.110979+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:01.238104+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":288,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.842915+0000","last_change":"2024-09-02T10:04:30.278703+0000","last_active":"2024-09-02T10:05:16.842915+0000","last_peered":"2024-09-02T10:05:16.842915+0000","last_clean":"2024-09-02T10:05:16.842915+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:05:16.842915+0000","last_undegraded":"2024-09-02T10:05:16.842915+0000","last_fullsized":"2024-09-02T10:05:16.842915+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:30.278632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:30.278632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:48.025871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"209'11119","reported_seq":11426,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.868667+0000","last_change":"2024-09-02T10:02:46.577683+0000","last_active":"2024-09-02T10:05:16.868667+0000","last_peered":"2024-09-02T10:05:16.868667+0000","last_clean":"2024-09-02T10:05:16.868667+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:05:16.868667+0000","last_undegraded":"2024-09-02T10:05:16.868667+0000","last_fullsized":"2024-09-02T10:05:16.868667+0000","mapping_epoch":92,"log_start":"99'1100","ondisk_log_start":"99'1100","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'5355","last_scrub_stamp":"2024-09-02T10:02:46.577625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:46.577625+0000","objects_scrubbed":5316,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75956224,"num_objects":1159,"num_object_clones":0,"num_object_copies":3477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11119,"num_write_kb":392896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":190,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-02T10:05:13.813847+0000","last_change":"2024-09-02T10:04:57.391785+0000","last_active":"2024-09-02T10:05:13.813847+0000","last_peered":"2024-09-02T10:05:13.813847+0000","last_clean":"2024-09-02T10:05:13.813847+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:05:13.813847+0000","last_undegraded":"2024-09-02T10:05:13.813847+0000","last_fullsized":"2024-09-02T10:05:13.813847+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.391723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.391723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:12.774478+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"207'10653","reported_seq":16775,"reported_epoch":207,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:05:14.183313+0000","last_change":"2024-09-02T10:04:57.390580+0000","last_active":"2024-09-02T10:05:14.183313+0000","last_peered":"2024-09-02T10:05:14.183313+0000","last_clean":"2024-09-02T10:03:45.188955+0000","last_became_active":"2024-09-02T10:04:57.363859+0000","last_became_peered":"2024-09-02T10:04:57.363859+0000","last_unstale":"2024-09-02T10:05:14.183313+0000","last_undegraded":"2024-09-02T10:04:57.027533+0000","last_fullsized":"2024-09-02T10:04:57.018785+0000","mapping_epoch":198,"log_start":"98'600","ondisk_log_start":"98'600","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'5182","last_scrub_stamp":"2024-09-02T10:02:48.587872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.587872+0000","objects_scrubbed":5181,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80347160,"num_objects":1227,"num_object_clones":0,"num_object_copies":3681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1227,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10653,"num_write_kb":380097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":190513152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1227}],"blocked_by":[],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":53,"reported_epoch":208,"state":"active+clean","last_fresh":"2024-09-02T10:05:14.822105+0000","last_change":"2024-09-02T10:04:54.128208+0000","last_active":"2024-09-02T10:05:14.822105+0000","last_peered":"2024-09-02T10:05:14.822105+0000","last_clean":"2024-09-02T10:05:14.822105+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:05:14.822105+0000","last_undegraded":"2024-09-02T10:05:14.822105+0000","last_fullsized":"2024-09-02T10:05:14.822105+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:54.128142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.128142+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.955526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"206'10925","reported_seq":18351,"reported_epoch":206,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:05:12.474318+0000","last_change":"2024-09-02T10:05:12.474648+0000","last_active":"2024-09-02T10:05:12.474318+0000","last_peered":"2024-09-02T10:05:12.474318+0000","last_clean":"2024-09-02T10:03:38.171966+0000","last_became_active":"2024-09-02T10:04:57.182024+0000","last_became_peered":"2024-09-02T10:04:57.182024+0000","last_unstale":"2024-09-02T10:05:12.474318+0000","last_undegraded":"2024-09-02T10:05:12.474318+0000","last_fullsized":"2024-09-02T10:04:57.016771+0000","mapping_epoch":198,"log_start":"98'900","ondisk_log_start":"98'900","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89849856,"num_objects":1371,"num_object_clones":0,"num_object_copies":4113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1371,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10925,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2391,"num_bytes_recovered":114360320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2,3","objects":1371}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"207'10998","reported_seq":11335,"reported_epoch":207,"state":"active+clean","last_fresh":"2024-09-02T10:05:14.181319+0000","last_change":"2024-09-02T10:04:38.925752+0000","last_active":"2024-09-02T10:05:14.181319+0000","last_peered":"2024-09-02T10:05:14.181319+0000","last_clean":"2024-09-02T10:05:14.181319+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:05:14.181319+0000","last_undegraded":"2024-09-02T10:05:14.181319+0000","last_fullsized":"2024-09-02T10:05:14.181319+0000","mapping_epoch":92,"log_start":"98'900","ondisk_log_start":"98'900","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9618","last_scrub_stamp":"2024-09-02T10:04:38.925565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.925565+0000","objects_scrubbed":2671,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.842942+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84148224,"num_objects":1284,"num_object_clones":0,"num_object_copies":3852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10998,"num_write_kb":393024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":62,"reported_epoch":206,"state":"active+clean","last_fresh":"2024-09-02T10:05:09.828322+0000","last_change":"2024-09-02T10:04:35.814407+0000","last_active":"2024-09-02T10:05:09.828322+0000","last_peered":"2024-09-02T10:05:09.828322+0000","last_clean":"2024-09-02T10:05:09.828322+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:05:09.828322+0000","last_undegraded":"2024-09-02T10:05:09.828322+0000","last_fullsized":"2024-09-02T10:05:09.828322+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:46.975807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:46.975807+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":1326055448,"num_objects":20235,"num_object_clones":0,"num_object_copies":60705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20235,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175263,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30408,"num_bytes_recovered":1724973056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5127221248,"data_stored":5127209032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160563,"ondisk_log_size":160563,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383742,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":477824,"kb_used_data":356396,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93267328,"statfs":{"total":95995035648,"available":95505743872,"internally_reserved":0,"allocated":364949504,"data_stored":364096889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":124320196},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841607,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1099368,"kb_used_data":991832,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92645784,"statfs":{"total":95995035648,"available":94869282816,"internally_reserved":0,"allocated":1015635968,"data_stored":1014788199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070541,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":410148,"kb_used_data":342868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93335004,"statfs":{"total":95995035648,"available":95575044096,"internally_reserved":0,"allocated":351096832,"data_stored":350253135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299472,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":507460,"kb_used_data":399916,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93237692,"statfs":{"total":95995035648,"available":95475396608,"internally_reserved":0,"allocated":409513984,"data_stored":408661377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528405,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":362300,"kb_used_data":295020,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93382852,"statfs":{"total":95995035648,"available":95624040448,"internally_reserved":0,"allocated":302100480,"data_stored":301247873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724634,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":232264,"kb_used_data":160820,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93512888,"statfs":{"total":95995035648,"available":95757197312,"internally_reserved":0,"allocated":164679680,"data_stored":163823521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953567,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314536,"kb_used_data":251416,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93430616,"statfs":{"total":95995035648,"available":95672950784,"internally_reserved":0,"allocated":257449984,"data_stored":256602215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149797,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":875376,"kb_used_data":735904,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92869776,"statfs":{"total":95995035648,"available":95098650624,"internally_reserved":0,"allocated":753565696,"data_stored":752714375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":142801352},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":628780,"kb_used_data":489308,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93116372,"statfs":{"total":95995035648,"available":95351164928,"internally_reserved":0,"allocated":501051392,"data_stored":500204143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":526692,"kb_used_data":446932,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93218460,"statfs":{"total":95995035648,"available":95455703040,"internally_reserved":0,"allocated":457658368,"data_stored":456814671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738484,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":483968,"kb_used_data":398636,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93261184,"statfs":{"total":95995035648,"available":95499452416,"internally_reserved":0,"allocated":408203264,"data_stored":407350657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":389283840,"data_stored":389283840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":451346432,"data_stored":451346432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":489357312,"data_stored":489357312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":731451392,"data_stored":731447320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":250482688,"data_stored":250478616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":159055872,"data_stored":159055872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":295632896,"data_stored":295632896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":398721024,"data_stored":398721024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":343080960,"data_stored":343080960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":989597696,"data_stored":989593624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":275513344,"data_stored":275513344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":353697792,"data_stored":353697792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:05:19.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:19.684 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:19.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:19.686 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:19.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:19.809 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:19.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:19.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:19.824 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:19.856 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:19 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:05:19.856 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:19 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:05:19.856 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:19 smithi151 ceph-mon[16639]: pgmap v519: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.2 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 2229/57432 objects degraded (3.881%); 0 B/s, 131 objects/s recovering 2024-09-02T10:05:19.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:19.940 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:19.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:19.945 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:20.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:20.069 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:20.071 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:20.123 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:19 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:05:20.123 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:19 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:05:20.123 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:19 smithi187 ceph-mon[20072]: pgmap v519: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.2 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 2229/57432 objects degraded (3.881%); 0 B/s, 131 objects/s recovering 2024-09-02T10:05:20.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:20.198 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:20.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:20.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:20.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:20.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:20.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:19 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:05:20.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:19 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:05:20.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:19 smithi162 ceph-mon[20279]: pgmap v519: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.2 GiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 2229/57432 objects degraded (3.881%); 0 B/s, 131 objects/s recovering 2024-09-02T10:05:20.389 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:05:20.389 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:05:20.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:20.462 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:20.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:20.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:05:20.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:20.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:20.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:20.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:20.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:20.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:20.863 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:05:20.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:20.914 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:20.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:20.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:21.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:21.038 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:21.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:21.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:21.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:20 smithi151 ceph-mon[16639]: osdmap e210: 12 total, 12 up, 11 in 2024-09-02T10:05:21.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2035533774' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:05:21.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:20 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:05:21.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:21.163 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:21.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:21.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:20 smithi187 ceph-mon[20072]: osdmap e210: 12 total, 12 up, 11 in 2024-09-02T10:05:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2035533774' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:05:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:20 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:05:21.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:21.296 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:21.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:21.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:05:21.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:20 smithi162 ceph-mon[20279]: osdmap e210: 12 total, 12 up, 11 in 2024-09-02T10:05:21.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2035533774' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:05:21.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:20 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:05:21.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:21.421 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:21.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:21.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:21.547 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:21.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:21.575 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:21.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:21.674 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:21.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:21.689 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:05:21.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:21.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:21.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:21.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:21.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:21.937 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:21.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:22.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:22.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:22.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:22.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:22.098 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:21 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:22.099 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:21 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:05:22.099 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:21 smithi187 ceph-mon[20072]: osdmap e211: 12 total, 12 up, 11 in 2024-09-02T10:05:22.099 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:21 smithi187 ceph-mon[20072]: pgmap v522: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.1 GiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 2229/53262 objects degraded (4.185%); 0 B/s, 226 objects/s recovering 2024-09-02T10:05:22.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:21 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:22.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:21 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:05:22.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:21 smithi151 ceph-mon[16639]: osdmap e211: 12 total, 12 up, 11 in 2024-09-02T10:05:22.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:21 smithi151 ceph-mon[16639]: pgmap v522: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.1 GiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 2229/53262 objects degraded (4.185%); 0 B/s, 226 objects/s recovering 2024-09-02T10:05:22.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:22.198 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:22.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:22.225 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:05:22.225 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:05:22.239 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:05:22.239 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:05:22.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:21 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:22.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:21 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:05:22.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:21 smithi162 ceph-mon[20279]: osdmap e211: 12 total, 12 up, 11 in 2024-09-02T10:05:22.245 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:21 smithi162 ceph-mon[20279]: pgmap v522: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 1.1 GiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 663 op/s; 2229/53262 objects degraded (4.185%); 0 B/s, 226 objects/s recovering 2024-09-02T10:05:22.261 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:05:22.261 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:05:22.300 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:05:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:05:22.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:22.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:22.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:22.346 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:05:22.346 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:05:22.382 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:05:22.382 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:05:22.418 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:05:22.418 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:05:22.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:22.456 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:05:22.456 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:05:22.468 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:22.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:22.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:22.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:05:22.493 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:05:22.536 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:05:22.536 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:05:22.570 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:22.573 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:05:22.573 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:05:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:22.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:22.602 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:22.608 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:05:22.609 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:05:22.644 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:05:22.644 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:05:22.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:05:22.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:22.731 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:22.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:22.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:22.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:22.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:22.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:22.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:05:23.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:23.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:23.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:23.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3914038053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:23.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3914038053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:23.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:23.133 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:23.133 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:23.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:23.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3914038053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:23.259 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:23.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:23.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:23.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:23.382 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:23.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:23.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:23.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:23.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:23.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:23.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:05:23.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:23.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:23.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:23.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:23.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:23.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:23.895 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:23.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:23.935 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:23.935 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:23 smithi151 ceph-mon[16639]: pgmap v523: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 954 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 965 op/s; 2229/45771 objects degraded (4.870%); 0 B/s, 141 objects/s recovering 2024-09-02T10:05:23.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:23 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2229/53262 objects degraded (4.185%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:23.955 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:24.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:24.062 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:24.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:24.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:23 smithi162 ceph-mon[20279]: pgmap v523: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 954 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 965 op/s; 2229/45771 objects degraded (4.870%); 0 B/s, 141 objects/s recovering 2024-09-02T10:05:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:23 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2229/53262 objects degraded (4.185%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:24.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:24.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:24.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:24.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:24.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:23 smithi187 ceph-mon[20072]: pgmap v523: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 954 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 965 op/s; 2229/45771 objects degraded (4.870%); 0 B/s, 141 objects/s recovering 2024-09-02T10:05:24.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:23 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2229/53262 objects degraded (4.185%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:24.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:24.333 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:24.334 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:24.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:24.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:24.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:24.466 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:24.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:24.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:24.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:24.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:24.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:24.724 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:24.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:24.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:24.850 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:24.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:24.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:24.985 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:24.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:24.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:25.094 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:05:25.094 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:05:25.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:25.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:25.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:183203 2.1 e211/200 2:8734cc95:::benchmark_data_smithi187_45882_object85452:head v 211'11481, mlcod=211'11478)", 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:25.135201+0000", 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0022498309999999999, 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0022833150000000002, 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:25.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:25.135201+0000", 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:25.135201+0000", 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:25.135202+0000", 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4499999999999995e-07 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:25.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:25.135207+0000", 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.344e-06 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:25.135233+0000", 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5704e-05 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:25.135241+0000", 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0930000000000003e-06 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:25.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:25.135294+0000", 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3402000000000001e-05 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:25.135314+0000", 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9630999999999999e-05 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:25.142 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:05:25.143 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 1 2024-09-02T10:05:25.143 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:05:25.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:25.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:25.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:25.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:25.371 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:25.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:25.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:25.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:25.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:25.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:25.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:25.630 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:25.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:25.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:25.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:25.756 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:25.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:25.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:05:25.857 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:25.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:25.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:25.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:25.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:26.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:26.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:26.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:26.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:26.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:26.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:26.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:26.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:26.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:26.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:26.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:26.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:26 smithi151 ceph-mon[16639]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:26.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:26 smithi151 ceph-mon[16639]: pgmap v524: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 893 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 2229/42822 objects degraded (5.205%); 0 B/s, 110 objects/s recovering 2024-09-02T10:05:26.441 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:26.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:26 smithi187 ceph-mon[20072]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:26 smithi187 ceph-mon[20072]: pgmap v524: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 893 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 2229/42822 objects degraded (5.205%); 0 B/s, 110 objects/s recovering 2024-09-02T10:05:26.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:26.565 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:26.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:26.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:26.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:26 smithi162 ceph-mon[20279]: from='client.24980 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:26.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:26 smithi162 ceph-mon[20279]: pgmap v524: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 893 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 745 op/s; 2229/42822 objects degraded (5.205%); 0 B/s, 110 objects/s recovering 2024-09-02T10:05:26.667 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:26.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:26.696 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:26.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:26.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:26.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:26.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:26.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:26.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:26.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:27.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:27.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:27.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:27.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:27.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:27.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:27.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:27.333 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:27.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:27.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:27.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:27.462 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:27.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:27.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:27.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:27.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:27.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:27.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:27.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:27.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:27.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:27.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:27.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:27.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:27.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:27.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:28.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:28.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:28.098 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:28.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:28.221 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:28.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:28.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:28.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:28.348 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:28.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: pgmap v525: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 784 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 826 op/s; 2229/37596 objects degraded (5.929%); 0 B/s, 153 objects/s recovering 2024-09-02T10:05:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3887479965,config=2353167246,health=3861344057,kv=3348256268,logm=4066114734}) 2024-09-02T10:05:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1819132657}) 2024-09-02T10:05:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2370513843}) 2024-09-02T10:05:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729609204}) 2024-09-02T10:05:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414289009}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469340023}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1858776497}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950828255}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=378749059}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2635961548}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=148350910}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1922414307}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2275754555}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2300585704}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3699842388}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877029489}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=643826711}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1444579838}) 2024-09-02T10:05:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1584606628}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3321531220}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=296104702}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3675530362}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=769279126}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940623835}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893190636}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667560892}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1163105567}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=54,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=4} crc {logm=233934599,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=2052410099,mon_config_key=574391484}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3584746276}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=418915031}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=3992167019,monmap=3739207698,osd_metadata=2278182237}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=2292831711,osd_pg_creating=876283072,osdmap=2546433854}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3856175749}) 2024-09-02T10:05:28.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=14} crc {osdmap=4230367490}) 2024-09-02T10:05:28.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:28.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:28.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:28.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: pgmap v525: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 784 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 826 op/s; 2229/37596 objects degraded (5.929%); 0 B/s, 153 objects/s recovering 2024-09-02T10:05:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3887479965,config=2353167246,health=3861344057,kv=3348256268,logm=4066114734}) 2024-09-02T10:05:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1819132657}) 2024-09-02T10:05:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2370513843}) 2024-09-02T10:05:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729609204}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414289009}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469340023}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1858776497}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950828255}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=378749059}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2635961548}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=148350910}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1922414307}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2275754555}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2300585704}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3699842388}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877029489}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=643826711}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1444579838}) 2024-09-02T10:05:28.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1584606628}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3321531220}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=296104702}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3675530362}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=769279126}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940623835}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893190636}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667560892}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1163105567}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=54,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=4} crc {logm=233934599,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=2052410099,mon_config_key=574391484}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3584746276}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=418915031}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=3992167019,monmap=3739207698,osd_metadata=2278182237}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=2292831711,osd_pg_creating=876283072,osdmap=2546433854}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3856175749}) 2024-09-02T10:05:28.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=14} crc {osdmap=4230367490}) 2024-09-02T10:05:28.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: pgmap v525: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 784 MiB data, 4.2 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 826 op/s; 2229/37596 objects degraded (5.929%); 0 B/s, 153 objects/s recovering 2024-09-02T10:05:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=53,logm=7} crc {auth=3887479965,config=2353167246,health=3861344057,kv=3348256268,logm=4066114734}) 2024-09-02T10:05:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1819132657}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2370513843}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3729609204}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2414289009}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1469340023}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1858776497}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2950828255}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=378749059}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2635961548}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=148350910}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1922414307}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2275754555}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2300585704}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3699842388}) 2024-09-02T10:05:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3877029489}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=643826711}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1444579838}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1584606628}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3321531220}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=296104702}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3675530362}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=769279126}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3940623835}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893190636}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=667560892}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1163105567}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=54,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=18,mon_config_key=4} crc {logm=233934599,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=2052410099,mon_config_key=574391484}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3584746276}) 2024-09-02T10:05:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=418915031}) 2024-09-02T10:05:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=87,monmap=5,osd_metadata=8} crc {mon_config_key=3992167019,monmap=3739207698,osd_metadata=2278182237}) 2024-09-02T10:05:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=4,osd_pg_creating=1,osdmap=95} crc {osd_metadata=2292831711,osd_pg_creating=876283072,osdmap=2546433854}) 2024-09-02T10:05:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3856175749}) 2024-09-02T10:05:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=14} crc {osdmap=4230367490}) 2024-09-02T10:05:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:28.598 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:28.598 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:28.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:28.725 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:28.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:28.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:28.854 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:28.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:28.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:28.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:28.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:29.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:29.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:29.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:29.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:29.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:29.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:29.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:29.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:29.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:29.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:29.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:29 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2229/37596 objects degraded (5.929%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:29.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:29.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:29.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:29.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:29.505 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:29 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2229/37596 objects degraded (5.929%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:29 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2229/37596 objects degraded (5.929%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:29.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:29.624 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:29.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:29.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:29.752 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:29.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:29.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:29.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:29.876 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:29.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:29.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:29.978 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:30.008 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:30.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:30.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:30.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:30.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:30.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:30.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:30.270 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:30.271 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:30.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:30.396 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:30.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:30.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:30 smithi151 ceph-mon[16639]: pgmap v526: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 709 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 2229/34026 objects degraded (6.551%); 0 B/s, 156 objects/s recovering 2024-09-02T10:05:30.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:30 smithi187 ceph-mon[20072]: pgmap v526: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 709 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 2229/34026 objects degraded (6.551%); 0 B/s, 156 objects/s recovering 2024-09-02T10:05:30.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:30.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:30.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:30.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:30 smithi162 ceph-mon[20279]: pgmap v526: 37 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 709 MiB data, 3.8 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 742 op/s; 2229/34026 objects degraded (6.551%); 0 B/s, 156 objects/s recovering 2024-09-02T10:05:30.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:30.654 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:30.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:30.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:30.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:30.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:30.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:30.888 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:30.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:30.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:31.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:31.046 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:31.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:31.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:31.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:31.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:31.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:31.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:31.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:31.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:31.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:31.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:31.441 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:31.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:31.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:31.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:31.566 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:31.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:31.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:31.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:31.700 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-09-02T10:05:23.121291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1000276536,"num_objects":15257,"num_object_clones":0,"num_object_copies":45771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15257,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180243,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31299,"num_bytes_recovered":1725703232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162875,"ondisk_log_size":162875,"up":111,"acting":107,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":4914604,"kb_used_data":3792948,"kb_used_omap":17,"kb_used_meta":1120238,"kb_avail":1026282068,"statfs":{"total":1055945392128,"available":1050912837632,"internally_reserved":0,"allocated":3883978752,"data_stored":3875863856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1147124603},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,3,2,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-398262272,"num_objects":-6077,"num_object_clones":0,"num_object_copies":-18231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6077,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":888,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.296302"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":190,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.864116+0000","last_change":"2024-09-02T10:02:43.131354+0000","last_active":"2024-09-02T10:05:20.864116+0000","last_peered":"2024-09-02T10:05:20.864116+0000","last_clean":"2024-09-02T10:05:20.864116+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:05:20.864116+0000","last_undegraded":"2024-09-02T10:05:20.864116+0000","last_fullsized":"2024-09-02T10:05:20.864116+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.131299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.131299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":275,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.864755+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:05:20.864755+0000","last_peered":"2024-09-02T10:05:20.864755+0000","last_clean":"2024-09-02T10:05:20.864755+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:05:20.864755+0000","last_undegraded":"2024-09-02T10:05:20.864755+0000","last_fullsized":"2024-09-02T10:05:20.864755+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.233390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":212,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.859809+0000","last_change":"2024-09-02T10:04:37.422274+0000","last_active":"2024-09-02T10:05:20.859809+0000","last_peered":"2024-09-02T10:05:20.859809+0000","last_clean":"2024-09-02T10:05:20.859809+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:05:20.859809+0000","last_undegraded":"2024-09-02T10:05:20.859809+0000","last_fullsized":"2024-09-02T10:05:20.859809+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:37.422128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:37.422128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:02.404595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":192,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.863030+0000","last_change":"2024-09-02T10:02:42.836988+0000","last_active":"2024-09-02T10:05:20.863030+0000","last_peered":"2024-09-02T10:05:20.863030+0000","last_clean":"2024-09-02T10:05:20.863030+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:05:20.863030+0000","last_undegraded":"2024-09-02T10:05:20.863030+0000","last_fullsized":"2024-09-02T10:05:20.863030+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:42.836911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:42.836911+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":205,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.859617+0000","last_change":"2024-09-02T10:04:38.471750+0000","last_active":"2024-09-02T10:05:20.859617+0000","last_peered":"2024-09-02T10:05:20.859617+0000","last_clean":"2024-09-02T10:05:20.859617+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:05:20.859617+0000","last_undegraded":"2024-09-02T10:05:20.859617+0000","last_fullsized":"2024-09-02T10:05:20.859617+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:38.471681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.471681+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:49.847941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"209'11084","reported_seq":11422,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:19.205641+0000","last_change":"2024-09-02T10:04:36.789912+0000","last_active":"2024-09-02T10:05:19.205641+0000","last_peered":"2024-09-02T10:05:19.205641+0000","last_clean":"2024-09-02T10:05:19.205641+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:05:19.205641+0000","last_undegraded":"2024-09-02T10:05:19.205641+0000","last_fullsized":"2024-09-02T10:05:19.205641+0000","mapping_epoch":92,"log_start":"99'1000","ondisk_log_start":"99'1000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'9455","last_scrub_stamp":"2024-09-02T10:04:36.789821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:36.789821+0000","objects_scrubbed":2703,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:41.193203+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70385664,"num_objects":1074,"num_object_clones":0,"num_object_copies":3222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11084,"num_write_kb":389056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":277,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.859801+0000","last_change":"2024-09-02T10:04:36.384243+0000","last_active":"2024-09-02T10:05:20.859801+0000","last_peered":"2024-09-02T10:05:20.859801+0000","last_clean":"2024-09-02T10:05:20.859801+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:05:20.859801+0000","last_undegraded":"2024-09-02T10:05:20.859801+0000","last_fullsized":"2024-09-02T10:05:20.859801+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:36.384163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:36.384163+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:37.175901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"211'11299","reported_seq":11623,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:21.908976+0000","last_change":"2024-09-02T10:04:56.617908+0000","last_active":"2024-09-02T10:05:21.908976+0000","last_peered":"2024-09-02T10:05:21.908976+0000","last_clean":"2024-09-02T10:05:21.908976+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:05:21.908976+0000","last_undegraded":"2024-09-02T10:05:21.908976+0000","last_fullsized":"2024-09-02T10:05:21.908976+0000","mapping_epoch":92,"log_start":"102'1200","ondisk_log_start":"102'1200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10186","last_scrub_stamp":"2024-09-02T10:04:56.617746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.617746+0000","objects_scrubbed":2052,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:15.697367+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61538304,"num_objects":939,"num_object_clones":0,"num_object_copies":2817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11299,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":398,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-02T10:05:19.853814+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:05:19.853814+0000","last_peered":"2024-09-02T10:05:19.853814+0000","last_clean":"2024-09-02T10:05:19.853814+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:05:19.853814+0000","last_undegraded":"2024-09-02T10:05:19.853814+0000","last_fullsized":"2024-09-02T10:05:19.853814+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"211'11396","reported_seq":11674,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:22.805910+0000","last_change":"2024-09-02T10:04:55.460032+0000","last_active":"2024-09-02T10:05:22.805910+0000","last_peered":"2024-09-02T10:05:22.805910+0000","last_clean":"2024-09-02T10:05:22.805910+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:05:22.805910+0000","last_undegraded":"2024-09-02T10:05:22.805910+0000","last_fullsized":"2024-09-02T10:05:22.805910+0000","mapping_epoch":92,"log_start":"103'1300","ondisk_log_start":"103'1300","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10251","last_scrub_stamp":"2024-09-02T10:04:55.459770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.459770+0000","objects_scrubbed":2066,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.035919+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59637760,"num_objects":910,"num_object_clones":0,"num_object_copies":2730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11396,"num_write_kb":393792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":44,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.861418+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:05:20.861418+0000","last_peered":"2024-09-02T10:05:20.861418+0000","last_clean":"2024-09-02T10:05:20.861418+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:05:20.861418+0000","last_undegraded":"2024-09-02T10:05:20.861418+0000","last_fullsized":"2024-09-02T10:05:20.861418+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"210'11235","reported_seq":20707,"reported_epoch":210,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:20.832801+0000","last_change":"2024-09-02T10:04:57.234268+0000","last_active":"2024-09-02T10:05:20.832801+0000","last_peered":"2024-09-02T10:05:20.832801+0000","last_clean":"2024-09-02T10:03:38.126822+0000","last_became_active":"2024-09-02T10:04:57.195493+0000","last_became_peered":"2024-09-02T10:04:57.195493+0000","last_unstale":"2024-09-02T10:05:20.832801+0000","last_undegraded":"2024-09-02T10:05:20.832801+0000","last_fullsized":"2024-09-02T10:04:57.070642+0000","mapping_epoch":198,"log_start":"101'1200","ondisk_log_start":"101'1200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64552960,"num_objects":985,"num_object_clones":0,"num_object_copies":2955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11235,"num_write_kb":391040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4608,"num_bytes_recovered":208863232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":985}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":190,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.864150+0000","last_change":"2024-09-02T10:02:43.245406+0000","last_active":"2024-09-02T10:05:20.864150+0000","last_peered":"2024-09-02T10:05:20.864150+0000","last_clean":"2024-09-02T10:05:20.864150+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:05:20.864150+0000","last_undegraded":"2024-09-02T10:05:20.864150+0000","last_fullsized":"2024-09-02T10:05:20.864150+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.245354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.245354+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"211'11249","reported_seq":22211,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:22.009752+0000","last_change":"2024-09-02T10:04:27.932509+0000","last_active":"2024-09-02T10:05:22.009752+0000","last_peered":"2024-09-02T10:05:22.009752+0000","last_clean":"2024-09-02T10:05:22.009752+0000","last_became_active":"2024-09-02T10:04:26.448185+0000","last_became_peered":"2024-09-02T10:04:26.448185+0000","last_unstale":"2024-09-02T10:05:22.009752+0000","last_undegraded":"2024-09-02T10:05:22.009752+0000","last_fullsized":"2024-09-02T10:05:22.009752+0000","mapping_epoch":176,"log_start":"100'1200","ondisk_log_start":"100'1200","created":92,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9259","last_scrub_stamp":"2024-09-02T10:04:27.932362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:27.932362+0000","objects_scrubbed":2896,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:56.556476+0000","scrub_duration":606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58130432,"num_objects":887,"num_object_clones":0,"num_object_copies":2661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11249,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":349503488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":346,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.864187+0000","last_change":"2024-09-02T10:02:41.206594+0000","last_active":"2024-09-02T10:05:20.864187+0000","last_peered":"2024-09-02T10:05:20.864187+0000","last_clean":"2024-09-02T10:05:20.864187+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:05:20.864187+0000","last_undegraded":"2024-09-02T10:05:20.864187+0000","last_fullsized":"2024-09-02T10:05:20.864187+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:41.206524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:41.206524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"211'11301","reported_seq":11606,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:22.812346+0000","last_change":"2024-09-02T10:04:56.300674+0000","last_active":"2024-09-02T10:05:22.812346+0000","last_peered":"2024-09-02T10:05:22.812346+0000","last_clean":"2024-09-02T10:05:22.812346+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:05:22.812346+0000","last_undegraded":"2024-09-02T10:05:22.812346+0000","last_fullsized":"2024-09-02T10:05:22.812346+0000","mapping_epoch":92,"log_start":"103'1300","ondisk_log_start":"103'1300","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10171","last_scrub_stamp":"2024-09-02T10:04:56.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.300400+0000","objects_scrubbed":2044,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:07.471560+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59834368,"num_objects":913,"num_object_clones":0,"num_object_copies":2739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11301,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":184,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.864311+0000","last_change":"2024-09-02T10:02:49.000327+0000","last_active":"2024-09-02T10:05:20.864311+0000","last_peered":"2024-09-02T10:05:20.864311+0000","last_clean":"2024-09-02T10:05:20.864311+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:05:20.864311+0000","last_undegraded":"2024-09-02T10:05:20.864311+0000","last_fullsized":"2024-09-02T10:05:20.864311+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:49.000247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:49.000247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"209'11067","reported_seq":11408,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:19.037107+0000","last_change":"2024-09-02T10:05:04.366916+0000","last_active":"2024-09-02T10:05:19.037107+0000","last_peered":"2024-09-02T10:05:19.037107+0000","last_clean":"2024-09-02T10:05:19.037107+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:05:19.037107+0000","last_undegraded":"2024-09-02T10:05:19.037107+0000","last_fullsized":"2024-09-02T10:05:19.037107+0000","mapping_epoch":92,"log_start":"99'1000","ondisk_log_start":"99'1000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'10418","last_scrub_stamp":"2024-09-02T10:05:04.366613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:04.366613+0000","objects_scrubbed":1736,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:11.371378+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70320128,"num_objects":1073,"num_object_clones":0,"num_object_copies":3219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11067,"num_write_kb":388480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":44,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.861438+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:05:20.861438+0000","last_peered":"2024-09-02T10:05:20.861438+0000","last_clean":"2024-09-02T10:05:20.861438+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:05:20.861438+0000","last_undegraded":"2024-09-02T10:05:20.861438+0000","last_fullsized":"2024-09-02T10:05:20.861438+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.611307+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"211'11248","reported_seq":22131,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:22.876588+0000","last_change":"2024-09-02T10:05:00.573995+0000","last_active":"2024-09-02T10:05:22.876588+0000","last_peered":"2024-09-02T10:05:22.876588+0000","last_clean":"2024-09-02T10:05:22.876588+0000","last_became_active":"2024-09-02T10:04:43.806359+0000","last_became_peered":"2024-09-02T10:04:43.806359+0000","last_unstale":"2024-09-02T10:05:22.876588+0000","last_undegraded":"2024-09-02T10:05:22.876588+0000","last_fullsized":"2024-09-02T10:05:22.876588+0000","mapping_epoch":188,"log_start":"101'1200","ondisk_log_start":"101'1200","created":92,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'10224","last_scrub_stamp":"2024-09-02T10:05:00.573420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:00.573420+0000","objects_scrubbed":1920,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.148546+0000","scrub_duration":803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58195968,"num_objects":888,"num_object_clones":0,"num_object_copies":2664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11248,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5298,"num_bytes_recovered":347209728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":307,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.861820+0000","last_change":"2024-09-02T10:04:35.810947+0000","last_active":"2024-09-02T10:05:20.861820+0000","last_peered":"2024-09-02T10:05:20.861820+0000","last_clean":"2024-09-02T10:05:20.861820+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:05:20.861820+0000","last_undegraded":"2024-09-02T10:05:20.861820+0000","last_fullsized":"2024-09-02T10:05:20.861820+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:28.355594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:28.355594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:51.765081+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"211'11565","reported_seq":11911,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:21.787082+0000","last_change":"2024-09-02T10:04:58.724992+0000","last_active":"2024-09-02T10:05:21.787082+0000","last_peered":"2024-09-02T10:05:21.787082+0000","last_clean":"2024-09-02T10:05:21.787082+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:05:21.787082+0000","last_undegraded":"2024-09-02T10:05:21.787082+0000","last_fullsized":"2024-09-02T10:05:21.787082+0000","mapping_epoch":92,"log_start":"104'1500","ondisk_log_start":"104'1500","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'10526","last_scrub_stamp":"2024-09-02T10:04:58.724845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.724845+0000","objects_scrubbed":2020,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:18.034443+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64028672,"num_objects":977,"num_object_clones":0,"num_object_copies":2931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11565,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":36,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.861230+0000","last_change":"2024-09-02T10:04:57.021532+0000","last_active":"2024-09-02T10:05:20.861230+0000","last_peered":"2024-09-02T10:05:20.861230+0000","last_clean":"2024-09-02T10:05:20.861230+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:05:20.861230+0000","last_undegraded":"2024-09-02T10:05:20.861230+0000","last_fullsized":"2024-09-02T10:05:20.861230+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:38.873802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:38.873802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"211'11376","reported_seq":17820,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:22.472222+0000","last_change":"2024-09-02T10:04:58.839358+0000","last_active":"2024-09-02T10:05:22.472222+0000","last_peered":"2024-09-02T10:05:22.472222+0000","last_clean":"2024-09-02T10:05:22.472222+0000","last_became_active":"2024-09-02T10:04:58.808736+0000","last_became_peered":"2024-09-02T10:04:58.808736+0000","last_unstale":"2024-09-02T10:05:22.472222+0000","last_undegraded":"2024-09-02T10:05:22.472222+0000","last_fullsized":"2024-09-02T10:05:22.472222+0000","mapping_epoch":200,"log_start":"102'1300","ondisk_log_start":"102'1300","created":92,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'5361","last_scrub_stamp":"2024-09-02T10:02:48.223591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.223591+0000","objects_scrubbed":5355,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57671680,"num_objects":880,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3059,"num_bytes_recovered":97517568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":46,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-02T10:05:19.853901+0000","last_change":"2024-09-02T10:04:47.929535+0000","last_active":"2024-09-02T10:05:19.853901+0000","last_peered":"2024-09-02T10:05:19.853901+0000","last_clean":"2024-09-02T10:05:19.853901+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:05:19.853901+0000","last_undegraded":"2024-09-02T10:05:19.853901+0000","last_fullsized":"2024-09-02T10:05:19.853901+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:39.894365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:39.894365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"210'11159","reported_seq":28075,"reported_epoch":210,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:20.841126+0000","last_change":"2024-09-02T10:04:57.232835+0000","last_active":"2024-09-02T10:05:20.841126+0000","last_peered":"2024-09-02T10:05:20.841126+0000","last_clean":"2024-09-02T10:02:27.223209+0000","last_became_active":"2024-09-02T10:04:57.144949+0000","last_became_peered":"2024-09-02T10:04:57.144949+0000","last_unstale":"2024-09-02T10:05:20.841126+0000","last_undegraded":"2024-09-02T10:05:20.841126+0000","last_fullsized":"2024-09-02T10:04:57.017574+0000","mapping_epoch":198,"log_start":"100'1100","ondisk_log_start":"100'1100","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59965440,"num_objects":915,"num_object_clones":0,"num_object_copies":2745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11159,"num_write_kb":386368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7381,"num_bytes_recovered":417005568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,11],"avail_no_missing":["7","10","11"],"object_location_counts":[{"shards":"7,11","objects":915}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":262,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.864849+0000","last_change":"2024-09-02T10:05:06.815241+0000","last_active":"2024-09-02T10:05:20.864849+0000","last_peered":"2024-09-02T10:05:20.864849+0000","last_clean":"2024-09-02T10:05:20.864849+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:05:20.864849+0000","last_undegraded":"2024-09-02T10:05:20.864849+0000","last_fullsized":"2024-09-02T10:05:20.864849+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:06.815191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:06.815191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:14.504648+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"211'11376","reported_seq":11730,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:21.782775+0000","last_change":"2024-09-02T10:04:54.927090+0000","last_active":"2024-09-02T10:05:21.782775+0000","last_peered":"2024-09-02T10:05:21.782775+0000","last_clean":"2024-09-02T10:05:21.782775+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:05:21.782775+0000","last_undegraded":"2024-09-02T10:05:21.782775+0000","last_fullsized":"2024-09-02T10:05:21.782775+0000","mapping_epoch":92,"log_start":"103'1300","ondisk_log_start":"103'1300","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10296","last_scrub_stamp":"2024-09-02T10:04:54.926873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.926873+0000","objects_scrubbed":2056,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.100584+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63307776,"num_objects":966,"num_object_clones":0,"num_object_copies":2898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11376,"num_write_kb":394944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":57,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-02T10:05:19.853982+0000","last_change":"2024-09-02T10:04:55.111033+0000","last_active":"2024-09-02T10:05:19.853982+0000","last_peered":"2024-09-02T10:05:19.853982+0000","last_clean":"2024-09-02T10:05:19.853982+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:05:19.853982+0000","last_undegraded":"2024-09-02T10:05:19.853982+0000","last_fullsized":"2024-09-02T10:05:19.853982+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:55.110979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.110979+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:01.238104+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":292,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:20.861776+0000","last_change":"2024-09-02T10:04:30.278703+0000","last_active":"2024-09-02T10:05:20.861776+0000","last_peered":"2024-09-02T10:05:20.861776+0000","last_clean":"2024-09-02T10:05:20.861776+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:05:20.861776+0000","last_undegraded":"2024-09-02T10:05:20.861776+0000","last_fullsized":"2024-09-02T10:05:20.861776+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:30.278632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:30.278632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:48.025871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"211'11360","reported_seq":11671,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-09-02T10:05:21.913339+0000","last_change":"2024-09-02T10:02:46.577683+0000","last_active":"2024-09-02T10:05:21.913339+0000","last_peered":"2024-09-02T10:05:21.913339+0000","last_clean":"2024-09-02T10:05:21.913339+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:05:21.913339+0000","last_undegraded":"2024-09-02T10:05:21.913339+0000","last_fullsized":"2024-09-02T10:05:21.913339+0000","mapping_epoch":92,"log_start":"102'1300","ondisk_log_start":"102'1300","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'5355","last_scrub_stamp":"2024-09-02T10:02:46.577625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:46.577625+0000","objects_scrubbed":5316,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60162048,"num_objects":918,"num_object_clones":0,"num_object_copies":2754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11360,"num_write_kb":392896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":194,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.833550+0000","last_change":"2024-09-02T10:04:57.391785+0000","last_active":"2024-09-02T10:05:16.833550+0000","last_peered":"2024-09-02T10:05:16.833550+0000","last_clean":"2024-09-02T10:05:16.833550+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:05:16.833550+0000","last_undegraded":"2024-09-02T10:05:16.833550+0000","last_fullsized":"2024-09-02T10:05:16.833550+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.391723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.391723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:12.774478+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"209'10859","reported_seq":16987,"reported_epoch":209,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:05:19.143661+0000","last_change":"2024-09-02T10:04:57.390580+0000","last_active":"2024-09-02T10:05:19.143661+0000","last_peered":"2024-09-02T10:05:19.143661+0000","last_clean":"2024-09-02T10:03:45.188955+0000","last_became_active":"2024-09-02T10:04:57.363859+0000","last_became_peered":"2024-09-02T10:04:57.363859+0000","last_unstale":"2024-09-02T10:05:19.143661+0000","last_undegraded":"2024-09-02T10:04:57.027533+0000","last_fullsized":"2024-09-02T10:04:57.018785+0000","mapping_epoch":198,"log_start":"98'800","ondisk_log_start":"98'800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'5182","last_scrub_stamp":"2024-09-02T10:02:48.587872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.587872+0000","objects_scrubbed":5181,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846744,"num_objects":1021,"num_object_clones":0,"num_object_copies":3063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10859,"num_write_kb":380097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":190513152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1021}],"blocked_by":[],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":57,"reported_epoch":210,"state":"active+clean","last_fresh":"2024-09-02T10:05:19.854143+0000","last_change":"2024-09-02T10:04:54.128208+0000","last_active":"2024-09-02T10:05:19.854143+0000","last_peered":"2024-09-02T10:05:19.854143+0000","last_clean":"2024-09-02T10:05:19.854143+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:05:19.854143+0000","last_undegraded":"2024-09-02T10:05:19.854143+0000","last_fullsized":"2024-09-02T10:05:19.854143+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:54.128142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.128142+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.955526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"211'11431","reported_seq":19820,"reported_epoch":211,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:22.477542+0000","last_change":"2024-09-02T10:05:22.477240+0000","last_active":"2024-09-02T10:05:22.477542+0000","last_peered":"2024-09-02T10:05:22.477542+0000","last_clean":"2024-09-02T10:03:38.171966+0000","last_became_active":"2024-09-02T10:04:57.182024+0000","last_became_peered":"2024-09-02T10:04:57.182024+0000","last_unstale":"2024-09-02T10:05:22.477542+0000","last_undegraded":"2024-09-02T10:05:22.477542+0000","last_fullsized":"2024-09-02T10:04:57.016771+0000","mapping_epoch":198,"log_start":"104'1400","ondisk_log_start":"104'1400","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56688640,"num_objects":865,"num_object_clones":0,"num_object_copies":2595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":865,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11431,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2709,"num_bytes_recovered":114360320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2,3","objects":865}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"209'11238","reported_seq":11579,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:19.209780+0000","last_change":"2024-09-02T10:04:38.925752+0000","last_active":"2024-09-02T10:05:19.209780+0000","last_peered":"2024-09-02T10:05:19.209780+0000","last_clean":"2024-09-02T10:05:19.209780+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:05:19.209780+0000","last_undegraded":"2024-09-02T10:05:19.209780+0000","last_fullsized":"2024-09-02T10:05:19.209780+0000","mapping_epoch":92,"log_start":"102'1200","ondisk_log_start":"102'1200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9618","last_scrub_stamp":"2024-09-02T10:04:38.925565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.925565+0000","objects_scrubbed":2671,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.842942+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68419584,"num_objects":1044,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11238,"num_write_kb":393024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":68,"reported_epoch":209,"state":"active+clean","last_fresh":"2024-09-02T10:05:16.832130+0000","last_change":"2024-09-02T10:04:35.814407+0000","last_active":"2024-09-02T10:05:16.832130+0000","last_peered":"2024-09-02T10:05:16.832130+0000","last_clean":"2024-09-02T10:05:16.832130+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:05:16.832130+0000","last_undegraded":"2024-09-02T10:05:16.832130+0000","last_fullsized":"2024-09-02T10:05:16.832130+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:46.975807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:46.975807+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":999686168,"num_objects":15255,"num_object_clones":0,"num_object_copies":45765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15255,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":180243,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31296,"num_bytes_recovered":1724973056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4042731520,"data_stored":4042719304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160943,"ondisk_log_size":160943,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383743,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":419392,"kb_used_data":293624,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93325760,"statfs":{"total":95995035648,"available":95565578240,"internally_reserved":0,"allocated":300670976,"data_stored":299992037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":128580036},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841609,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":818636,"kb_used_data":719420,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":92926516,"statfs":{"total":95995035648,"available":95156752384,"internally_reserved":0,"allocated":736686080,"data_stored":736003077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":101579200},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070542,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":374536,"kb_used_data":285048,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93370616,"statfs":{"total":95995035648,"available":95611510784,"internally_reserved":0,"allocated":291889152,"data_stored":291032115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299473,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":436504,"kb_used_data":324640,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93308648,"statfs":{"total":95995035648,"available":95548055552,"internally_reserved":0,"allocated":332431360,"data_stored":331662972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528406,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":330568,"kb_used_data":263288,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93414584,"statfs":{"total":95995035648,"available":95656534016,"internally_reserved":0,"allocated":269606912,"data_stored":268927981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724635,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":201288,"kb_used_data":129664,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93543864,"statfs":{"total":95995035648,"available":95788916736,"internally_reserved":0,"allocated":132775936,"data_stored":132093453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953568,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":271244,"kb_used_data":208124,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93473908,"statfs":{"total":95995035648,"available":95717281792,"internally_reserved":0,"allocated":213118976,"data_stored":212257867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149798,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776668,"kb_used_data":628868,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92968484,"statfs":{"total":95995035648,"available":95199727616,"internally_reserved":0,"allocated":643960832,"data_stored":643096171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":151321032},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411434,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":488456,"kb_used_data":340416,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93256696,"statfs":{"total":95995035648,"available":95494856704,"internally_reserved":0,"allocated":348585984,"data_stored":347903501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574959,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401280,"kb_used_data":293560,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93343872,"statfs":{"total":95995035648,"available":95584124928,"internally_reserved":0,"allocated":300605440,"data_stored":299926509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738485,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":396032,"kb_used_data":306296,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93349120,"statfs":{"total":95995035648,"available":95589498880,"internally_reserved":0,"allocated":313647104,"data_stored":312968173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":310902784,"data_stored":310902784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294780928,"data_stored":294780928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":337969152,"data_stored":337969152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":622661632,"data_stored":622657560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":205656064,"data_stored":205651992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":127336448,"data_stored":127336448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262864896,"data_stored":262864896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":320798720,"data_stored":320798720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":280625152,"data_stored":280625152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":713428992,"data_stored":713424920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":275513344,"data_stored":275513344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":290193408,"data_stored":290193408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:05:31.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:05:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:31.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:31.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:31.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:31.841 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:31.842 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:31.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:05:31.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:31.966 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:31.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:31.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:32.068 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:32.093 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:32.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:32.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:32.216 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:32.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:32.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:32.244 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:32.319 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:32.343 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:32.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:32.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:32.364 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:32 smithi151 ceph-mon[16639]: pgmap v527: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 624 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 2229/29925 objects degraded (7.449%); 0 B/s, 155 objects/s recovering 2024-09-02T10:05:32.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:32.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:32.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:32 smithi187 ceph-mon[20072]: pgmap v527: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 624 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 2229/29925 objects degraded (7.449%); 0 B/s, 155 objects/s recovering 2024-09-02T10:05:32.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:32.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:32 smithi162 ceph-mon[20279]: pgmap v527: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 33 active+clean; 624 MiB data, 3.6 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 757 op/s; 2229/29925 objects degraded (7.449%); 0 B/s, 155 objects/s recovering 2024-09-02T10:05:32.599 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:32.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:32.600 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:32.701 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:32.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:32.728 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:32.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:32.859 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:32.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:32.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:32.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:32.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:33.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:33.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:33.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:33.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:33.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:33.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:33.206 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:05:33.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:33.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:33.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:33.357 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:05:33.358 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2229/23055 objects degraded (9.668%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":605,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":212,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":11,"osd_in_since":1725271367,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":33},{"state_name":"active+recovering+undersized+remapped","count":3},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":37,"num_pools":2,"num_objects":7685,"data_bytes":504037944,"bytes_used":3413045248,"bytes_avail":1052532346880,"bytes_total":1055945392128,"degraded_objects":2229,"degraded_total":23055,"degraded_ratio":0.096681847755367603,"recovering_objects_per_sec":146,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1758,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":838},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":21,"modified":"2024-09-02T10:05:11.115338+0000","services":{}},"progress_events":{}} 2024-09-02T10:05:33.358 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2585239763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:33.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:33.407 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:33.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2585239763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:33.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:33.537 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:33.538 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:33.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2585239763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:33.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:33.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:33.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:33.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:05:33.756 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:05:33.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:33.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:05:33.796 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:05:33.809 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:33.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:33.838 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:05:33.838 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:05:33.877 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:05:33.877 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:05:33.909 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:05:33.909 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:05:33.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:33.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:33.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:33.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:33.951 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:05:33.952 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:05:33.985 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:05:33.985 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:05:34.013 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:05:34.014 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:05:34.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:34.052 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:05:34.053 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:05:34.072 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:34.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:34.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:34.088 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:05:34.088 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:05:34.119 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:05:34.119 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:05:34.152 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:05:34.153 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:05:34.174 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:34.202 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:34.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:34.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:34.304 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:34.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:34.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:34.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:34.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:34.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:34.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:34 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:05:34.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:34 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:05:34.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:34 smithi151 ceph-mon[16639]: pgmap v528: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 481 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 838 op/s; 2229/23055 objects degraded (9.668%); 0 B/s, 146 objects/s recovering 2024-09-02T10:05:34.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:34 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:05:34.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2585239763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:34.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:34 smithi151 ceph-mon[16639]: osdmap e212: 12 total, 12 up, 11 in 2024-09-02T10:05:34.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2418701767' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:05:34.495 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3275897294' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:34.495 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:34 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2229/23055 objects degraded (9.668%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:34.495 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:34 smithi151 ceph-mon[16639]: osdmap e213: 12 total, 12 up, 11 in 2024-09-02T10:05:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:34 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:05:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:34 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:05:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:34 smithi187 ceph-mon[20072]: pgmap v528: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 481 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 838 op/s; 2229/23055 objects degraded (9.668%); 0 B/s, 146 objects/s recovering 2024-09-02T10:05:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:34 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:05:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2585239763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:34 smithi187 ceph-mon[20072]: osdmap e212: 12 total, 12 up, 11 in 2024-09-02T10:05:34.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2418701767' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:05:34.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3275897294' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:34.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:34 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2229/23055 objects degraded (9.668%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:34.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:34 smithi187 ceph-mon[20072]: osdmap e213: 12 total, 12 up, 11 in 2024-09-02T10:05:34.541 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:05:34.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:05:34.559 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:34.581 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:34.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:34.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:34 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:05:34.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:34 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:05:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:34 smithi162 ceph-mon[20279]: pgmap v528: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 481 MiB data, 3.2 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 838 op/s; 2229/23055 objects degraded (9.668%); 0 B/s, 146 objects/s recovering 2024-09-02T10:05:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:34 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:05:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2585239763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:34 smithi162 ceph-mon[20279]: osdmap e212: 12 total, 12 up, 11 in 2024-09-02T10:05:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2418701767' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:05:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3275897294' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:34 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2229/23055 objects degraded (9.668%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:34 smithi162 ceph-mon[20279]: osdmap e213: 12 total, 12 up, 11 in 2024-09-02T10:05:34.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:34.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:34.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:34.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:34.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:34.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:34.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:34.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:34.990 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:05:34.991 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:05:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:35.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:35.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:35.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:35.145 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:35.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:35.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:35.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:35.271 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:35.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:35.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:35.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:35.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:35.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:35.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:35.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:35.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:35.665 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:35.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:35.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:35.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:35.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:35.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:35.922 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:35.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:36.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:36.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:36.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:36.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:36.181 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:36.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:36.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:36.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:36.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:36.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:36.467 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:36.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:36 smithi187 ceph-mon[20072]: pgmap v531: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 421 MiB data, 2.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 725 op/s; 2229/20178 objects degraded (11.047%); 0 B/s, 137 objects/s recovering 2024-09-02T10:05:36.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:36.590 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:36 smithi151 ceph-mon[16639]: pgmap v531: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 421 MiB data, 2.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 725 op/s; 2229/20178 objects degraded (11.047%); 0 B/s, 137 objects/s recovering 2024-09-02T10:05:36.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:36 smithi162 ceph-mon[20279]: pgmap v531: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 421 MiB data, 2.9 GiB used, 980 GiB / 983 GiB avail; 0 B/s wr, 725 op/s; 2229/20178 objects degraded (11.047%); 0 B/s, 137 objects/s recovering 2024-09-02T10:05:36.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:36.595 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:36.596 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:36.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:36.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:36.726 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:36.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:36.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:36.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:36.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:36.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:36.987 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:36.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:37.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:37.115 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:37.116 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:37.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:37.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:37.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:37.244 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:37.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:37.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:37.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:37.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:37.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:05:37.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:37.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:37.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:37.548 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:37.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:05:37.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:37.637 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:37.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:37.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:37.760 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:37.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:37.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:37.883 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:37.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:37.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:38.008 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:38.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:38.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:38.111 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:38.130 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:38.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:38.131 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:38.135 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:38.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:38.239 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:38 smithi151 ceph-mon[16639]: pgmap v532: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 319 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 780 op/s; 2229/15294 objects degraded (14.574%); 0 B/s, 198 objects/s recovering 2024-09-02T10:05:38.239 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:38 smithi151 ceph-mon[16639]: osdmap e214: 12 total, 12 up, 11 in 2024-09-02T10:05:38.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:38.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:38.283 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:05:38.284 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:05:38.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:38.386 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:38.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:38.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:38 smithi187 ceph-mon[20072]: pgmap v532: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 319 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 780 op/s; 2229/15294 objects degraded (14.574%); 0 B/s, 198 objects/s recovering 2024-09-02T10:05:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:38 smithi187 ceph-mon[20072]: osdmap e214: 12 total, 12 up, 11 in 2024-09-02T10:05:38.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:38.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:38.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:38.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:38 smithi162 ceph-mon[20279]: pgmap v532: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 319 MiB data, 2.7 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 780 op/s; 2229/15294 objects degraded (14.574%); 0 B/s, 198 objects/s recovering 2024-09-02T10:05:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:38 smithi162 ceph-mon[20279]: osdmap e214: 12 total, 12 up, 11 in 2024-09-02T10:05:38.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:38.649 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:38.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:38.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:38.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:38.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:38.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:38.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:05:38.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:38.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:38.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:39.036 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:39.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:39.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:39.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:39.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:39.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:39.193 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:39.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:39.246 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":534,"stamp":"2024-09-02T10:05:38.220768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":260178488,"num_objects":3964,"num_object_clones":0,"num_object_copies":11892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3964,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191536,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33847,"num_bytes_recovered":1725703232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162568,"ondisk_log_size":162568,"up":111,"acting":107,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":2510628,"kb_used_data":1392448,"kb_used_omap":17,"kb_used_meta":1117486,"kb_avail":1028686044,"statfs":{"total":1055945392128,"available":1053374509056,"internally_reserved":0,"allocated":1425866752,"data_stored":1419670937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1144306555},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,2,4,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-243859456,"num_objects":-3721,"num_object_clones":0,"num_object_copies":-11163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3721,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":940,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.097004"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":196,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.223578+0000","last_change":"2024-09-02T10:02:43.131354+0000","last_active":"2024-09-02T10:05:37.223578+0000","last_peered":"2024-09-02T10:05:37.223578+0000","last_clean":"2024-09-02T10:05:37.223578+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:05:37.223578+0000","last_undegraded":"2024-09-02T10:05:37.223578+0000","last_fullsized":"2024-09-02T10:05:37.223578+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.131299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.131299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":281,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.219609+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:05:37.219609+0000","last_peered":"2024-09-02T10:05:37.219609+0000","last_clean":"2024-09-02T10:05:37.219609+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:05:37.219609+0000","last_undegraded":"2024-09-02T10:05:37.219609+0000","last_fullsized":"2024-09-02T10:05:37.219609+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.233390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":216,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.827890+0000","last_change":"2024-09-02T10:04:37.422274+0000","last_active":"2024-09-02T10:05:33.827890+0000","last_peered":"2024-09-02T10:05:33.827890+0000","last_clean":"2024-09-02T10:05:33.827890+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:05:33.827890+0000","last_undegraded":"2024-09-02T10:05:33.827890+0000","last_fullsized":"2024-09-02T10:05:33.827890+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:37.422128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:37.422128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:02.404595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":196,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.828307+0000","last_change":"2024-09-02T10:02:42.836988+0000","last_active":"2024-09-02T10:05:33.828307+0000","last_peered":"2024-09-02T10:05:33.828307+0000","last_clean":"2024-09-02T10:05:33.828307+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:05:33.828307+0000","last_undegraded":"2024-09-02T10:05:33.828307+0000","last_fullsized":"2024-09-02T10:05:33.828307+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:42.836911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:42.836911+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":209,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.827742+0000","last_change":"2024-09-02T10:04:38.471750+0000","last_active":"2024-09-02T10:05:33.827742+0000","last_peered":"2024-09-02T10:05:33.827742+0000","last_clean":"2024-09-02T10:05:33.827742+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:05:33.827742+0000","last_undegraded":"2024-09-02T10:05:33.827742+0000","last_fullsized":"2024-09-02T10:05:33.827742+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:38.471681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.471681+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:49.847941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"213'11810","reported_seq":12156,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:34.202551+0000","last_change":"2024-09-02T10:04:36.789912+0000","last_active":"2024-09-02T10:05:34.202551+0000","last_peered":"2024-09-02T10:05:34.202551+0000","last_clean":"2024-09-02T10:05:34.202551+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:05:34.202551+0000","last_undegraded":"2024-09-02T10:05:34.202551+0000","last_fullsized":"2024-09-02T10:05:34.202551+0000","mapping_epoch":92,"log_start":"105'1800","ondisk_log_start":"105'1800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'9455","last_scrub_stamp":"2024-09-02T10:04:36.789821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:36.789821+0000","objects_scrubbed":2703,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:41.193203+0000","scrub_duration":271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22806528,"num_objects":348,"num_object_clones":0,"num_object_copies":1044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11810,"num_write_kb":389056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":281,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.827811+0000","last_change":"2024-09-02T10:04:36.384243+0000","last_active":"2024-09-02T10:05:33.827811+0000","last_peered":"2024-09-02T10:05:33.827811+0000","last_clean":"2024-09-02T10:05:33.827811+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:05:33.827811+0000","last_undegraded":"2024-09-02T10:05:33.827811+0000","last_fullsized":"2024-09-02T10:05:33.827811+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:36.384163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:36.384163+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:37.175901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"213'12000","reported_seq":12328,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:36.888265+0000","last_change":"2024-09-02T10:04:56.617908+0000","last_active":"2024-09-02T10:05:36.888265+0000","last_peered":"2024-09-02T10:05:36.888265+0000","last_clean":"2024-09-02T10:05:36.888265+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:05:36.888265+0000","last_undegraded":"2024-09-02T10:05:36.888265+0000","last_fullsized":"2024-09-02T10:05:36.888265+0000","mapping_epoch":92,"log_start":"105'1900","ondisk_log_start":"105'1900","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10186","last_scrub_stamp":"2024-09-02T10:04:56.617746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.617746+0000","objects_scrubbed":2052,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:15.697367+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15597568,"num_objects":238,"num_object_clones":0,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12000,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":404,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.823960+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:05:33.823960+0000","last_peered":"2024-09-02T10:05:33.823960+0000","last_clean":"2024-09-02T10:05:33.823960+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:05:33.823960+0000","last_undegraded":"2024-09-02T10:05:33.823960+0000","last_fullsized":"2024-09-02T10:05:33.823960+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"214'12134","reported_seq":12418,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.734930+0000","last_change":"2024-09-02T10:04:55.460032+0000","last_active":"2024-09-02T10:05:37.734930+0000","last_peered":"2024-09-02T10:05:37.734930+0000","last_clean":"2024-09-02T10:05:37.734930+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:05:37.734930+0000","last_undegraded":"2024-09-02T10:05:37.734930+0000","last_fullsized":"2024-09-02T10:05:37.734930+0000","mapping_epoch":92,"log_start":"107'2100","ondisk_log_start":"107'2100","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10251","last_scrub_stamp":"2024-09-02T10:04:55.459770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.459770+0000","objects_scrubbed":2066,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.035919+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11272192,"num_objects":172,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12134,"num_write_kb":393792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":50,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.219015+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:05:37.219015+0000","last_peered":"2024-09-02T10:05:37.219015+0000","last_clean":"2024-09-02T10:05:37.219015+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:05:37.219015+0000","last_undegraded":"2024-09-02T10:05:37.219015+0000","last_fullsized":"2024-09-02T10:05:37.219015+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"213'11975","reported_seq":23440,"reported_epoch":213,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:35.853806+0000","last_change":"2024-09-02T10:04:57.234268+0000","last_active":"2024-09-02T10:05:35.853806+0000","last_peered":"2024-09-02T10:05:35.853806+0000","last_clean":"2024-09-02T10:03:38.126822+0000","last_became_active":"2024-09-02T10:04:57.195493+0000","last_became_peered":"2024-09-02T10:04:57.195493+0000","last_unstale":"2024-09-02T10:05:35.853806+0000","last_undegraded":"2024-09-02T10:05:35.853806+0000","last_fullsized":"2024-09-02T10:04:57.070642+0000","mapping_epoch":198,"log_start":"105'1900","ondisk_log_start":"105'1900","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16056320,"num_objects":245,"num_object_clones":0,"num_object_copies":735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11975,"num_write_kb":391040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5602,"num_bytes_recovered":208863232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":245}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":194,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.834590+0000","last_change":"2024-09-02T10:02:43.245406+0000","last_active":"2024-09-02T10:05:33.834590+0000","last_peered":"2024-09-02T10:05:33.834590+0000","last_clean":"2024-09-02T10:05:33.834590+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:05:33.834590+0000","last_undegraded":"2024-09-02T10:05:33.834590+0000","last_fullsized":"2024-09-02T10:05:33.834590+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.245354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.245354+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"213'11923","reported_seq":22889,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:36.972013+0000","last_change":"2024-09-02T10:04:27.932509+0000","last_active":"2024-09-02T10:05:36.972013+0000","last_peered":"2024-09-02T10:05:36.972013+0000","last_clean":"2024-09-02T10:05:36.972013+0000","last_became_active":"2024-09-02T10:04:26.448185+0000","last_became_peered":"2024-09-02T10:04:26.448185+0000","last_unstale":"2024-09-02T10:05:36.972013+0000","last_undegraded":"2024-09-02T10:05:36.972013+0000","last_fullsized":"2024-09-02T10:05:36.972013+0000","mapping_epoch":176,"log_start":"105'1900","ondisk_log_start":"105'1900","created":92,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9259","last_scrub_stamp":"2024-09-02T10:04:27.932362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:27.932362+0000","objects_scrubbed":2896,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:56.556476+0000","scrub_duration":606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13959168,"num_objects":213,"num_object_clones":0,"num_object_copies":639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11923,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":349503488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":350,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.834281+0000","last_change":"2024-09-02T10:02:41.206594+0000","last_active":"2024-09-02T10:05:33.834281+0000","last_peered":"2024-09-02T10:05:33.834281+0000","last_clean":"2024-09-02T10:05:33.834281+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:05:33.834281+0000","last_undegraded":"2024-09-02T10:05:33.834281+0000","last_fullsized":"2024-09-02T10:05:33.834281+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:41.206524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:41.206524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"214'12021","reported_seq":12332,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.603821+0000","last_change":"2024-09-02T10:04:56.300674+0000","last_active":"2024-09-02T10:05:37.603821+0000","last_peered":"2024-09-02T10:05:37.603821+0000","last_clean":"2024-09-02T10:05:37.603821+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:05:37.603821+0000","last_undegraded":"2024-09-02T10:05:37.603821+0000","last_fullsized":"2024-09-02T10:05:37.603821+0000","mapping_epoch":92,"log_start":"107'2000","ondisk_log_start":"107'2000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10171","last_scrub_stamp":"2024-09-02T10:04:56.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.300400+0000","objects_scrubbed":2044,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:07.471560+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12648448,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12021,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":190,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.220510+0000","last_change":"2024-09-02T10:02:49.000327+0000","last_active":"2024-09-02T10:05:37.220510+0000","last_peered":"2024-09-02T10:05:37.220510+0000","last_clean":"2024-09-02T10:05:37.220510+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:05:37.220510+0000","last_undegraded":"2024-09-02T10:05:37.220510+0000","last_fullsized":"2024-09-02T10:05:37.220510+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:49.000247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:49.000247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"213'11805","reported_seq":12153,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:34.154557+0000","last_change":"2024-09-02T10:05:04.366916+0000","last_active":"2024-09-02T10:05:34.154557+0000","last_peered":"2024-09-02T10:05:34.154557+0000","last_clean":"2024-09-02T10:05:34.154557+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:05:34.154557+0000","last_undegraded":"2024-09-02T10:05:34.154557+0000","last_fullsized":"2024-09-02T10:05:34.154557+0000","mapping_epoch":92,"log_start":"105'1800","ondisk_log_start":"105'1800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'10418","last_scrub_stamp":"2024-09-02T10:05:04.366613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:04.366613+0000","objects_scrubbed":1736,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:11.371378+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21954560,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11805,"num_write_kb":388480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":50,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.219163+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:05:37.219163+0000","last_peered":"2024-09-02T10:05:37.219163+0000","last_clean":"2024-09-02T10:05:37.219163+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:05:37.219163+0000","last_undegraded":"2024-09-02T10:05:37.219163+0000","last_fullsized":"2024-09-02T10:05:37.219163+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.611307+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"214'11928","reported_seq":22817,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.637115+0000","last_change":"2024-09-02T10:05:00.573995+0000","last_active":"2024-09-02T10:05:37.637115+0000","last_peered":"2024-09-02T10:05:37.637115+0000","last_clean":"2024-09-02T10:05:37.637115+0000","last_became_active":"2024-09-02T10:04:43.806359+0000","last_became_peered":"2024-09-02T10:04:43.806359+0000","last_unstale":"2024-09-02T10:05:37.637115+0000","last_undegraded":"2024-09-02T10:05:37.637115+0000","last_fullsized":"2024-09-02T10:05:37.637115+0000","mapping_epoch":188,"log_start":"105'1900","ondisk_log_start":"105'1900","created":92,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'10224","last_scrub_stamp":"2024-09-02T10:05:00.573420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:00.573420+0000","objects_scrubbed":1920,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.148546+0000","scrub_duration":803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13631488,"num_objects":208,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11928,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5298,"num_bytes_recovered":347209728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":311,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.829142+0000","last_change":"2024-09-02T10:04:35.810947+0000","last_active":"2024-09-02T10:05:33.829142+0000","last_peered":"2024-09-02T10:05:33.829142+0000","last_clean":"2024-09-02T10:05:33.829142+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:05:33.829142+0000","last_undegraded":"2024-09-02T10:05:33.829142+0000","last_fullsized":"2024-09-02T10:05:33.829142+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:28.355594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:28.355594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:51.765081+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"213'12323","reported_seq":12675,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:36.782910+0000","last_change":"2024-09-02T10:04:58.724992+0000","last_active":"2024-09-02T10:05:36.782910+0000","last_peered":"2024-09-02T10:05:36.782910+0000","last_clean":"2024-09-02T10:05:36.782910+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:05:36.782910+0000","last_undegraded":"2024-09-02T10:05:36.782910+0000","last_fullsized":"2024-09-02T10:05:36.782910+0000","mapping_epoch":92,"log_start":"109'2300","ondisk_log_start":"109'2300","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'10526","last_scrub_stamp":"2024-09-02T10:04:58.724845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.724845+0000","objects_scrubbed":2020,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:18.034443+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14352384,"num_objects":219,"num_object_clones":0,"num_object_copies":657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12323,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":42,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.218921+0000","last_change":"2024-09-02T10:04:57.021532+0000","last_active":"2024-09-02T10:05:37.218921+0000","last_peered":"2024-09-02T10:05:37.218921+0000","last_clean":"2024-09-02T10:05:37.218921+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:05:37.218921+0000","last_undegraded":"2024-09-02T10:05:37.218921+0000","last_fullsized":"2024-09-02T10:05:37.218921+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:38.873802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:38.873802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"214'12061","reported_seq":18511,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.447464+0000","last_change":"2024-09-02T10:04:58.839358+0000","last_active":"2024-09-02T10:05:37.447464+0000","last_peered":"2024-09-02T10:05:37.447464+0000","last_clean":"2024-09-02T10:05:37.447464+0000","last_became_active":"2024-09-02T10:04:58.808736+0000","last_became_peered":"2024-09-02T10:04:58.808736+0000","last_unstale":"2024-09-02T10:05:37.447464+0000","last_undegraded":"2024-09-02T10:05:37.447464+0000","last_fullsized":"2024-09-02T10:05:37.447464+0000","mapping_epoch":200,"log_start":"106'2000","ondisk_log_start":"106'2000","created":92,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'5361","last_scrub_stamp":"2024-09-02T10:02:48.223591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.223591+0000","objects_scrubbed":5355,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12061,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3059,"num_bytes_recovered":97517568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":52,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.823997+0000","last_change":"2024-09-02T10:04:47.929535+0000","last_active":"2024-09-02T10:05:33.823997+0000","last_peered":"2024-09-02T10:05:33.823997+0000","last_clean":"2024-09-02T10:05:33.823997+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:05:33.823997+0000","last_undegraded":"2024-09-02T10:05:33.823997+0000","last_fullsized":"2024-09-02T10:05:33.823997+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:39.894365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:39.894365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"213'11833","reported_seq":30618,"reported_epoch":213,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:35.853955+0000","last_change":"2024-09-02T10:04:57.232835+0000","last_active":"2024-09-02T10:05:35.853955+0000","last_peered":"2024-09-02T10:05:35.853955+0000","last_clean":"2024-09-02T10:02:27.223209+0000","last_became_active":"2024-09-02T10:04:57.144949+0000","last_became_peered":"2024-09-02T10:04:57.144949+0000","last_unstale":"2024-09-02T10:05:35.853955+0000","last_undegraded":"2024-09-02T10:05:35.853955+0000","last_fullsized":"2024-09-02T10:04:57.017574+0000","mapping_epoch":198,"log_start":"105'1800","ondisk_log_start":"105'1800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15794176,"num_objects":241,"num_object_clones":0,"num_object_copies":723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11833,"num_write_kb":386368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8313,"num_bytes_recovered":417005568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,11],"avail_no_missing":["7","10","11"],"object_location_counts":[{"shards":"7,11","objects":241}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":268,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-02T10:05:37.219736+0000","last_change":"2024-09-02T10:05:06.815241+0000","last_active":"2024-09-02T10:05:37.219736+0000","last_peered":"2024-09-02T10:05:37.219736+0000","last_clean":"2024-09-02T10:05:37.219736+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:05:37.219736+0000","last_undegraded":"2024-09-02T10:05:37.219736+0000","last_fullsized":"2024-09-02T10:05:37.219736+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:06.815191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:06.815191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:14.504648+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"213'12103","reported_seq":12461,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:36.823575+0000","last_change":"2024-09-02T10:04:54.927090+0000","last_active":"2024-09-02T10:05:36.823575+0000","last_peered":"2024-09-02T10:05:36.823575+0000","last_clean":"2024-09-02T10:05:36.823575+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:05:36.823575+0000","last_undegraded":"2024-09-02T10:05:36.823575+0000","last_fullsized":"2024-09-02T10:05:36.823575+0000","mapping_epoch":92,"log_start":"107'2100","ondisk_log_start":"107'2100","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10296","last_scrub_stamp":"2024-09-02T10:04:54.926873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.926873+0000","objects_scrubbed":2056,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.100584+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15663104,"num_objects":239,"num_object_clones":0,"num_object_copies":717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12103,"num_write_kb":394944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":63,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.823930+0000","last_change":"2024-09-02T10:04:55.111033+0000","last_active":"2024-09-02T10:05:33.823930+0000","last_peered":"2024-09-02T10:05:33.823930+0000","last_clean":"2024-09-02T10:05:33.823930+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:05:33.823930+0000","last_undegraded":"2024-09-02T10:05:33.823930+0000","last_fullsized":"2024-09-02T10:05:33.823930+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:55.110979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.110979+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:01.238104+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":296,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.829043+0000","last_change":"2024-09-02T10:04:30.278703+0000","last_active":"2024-09-02T10:05:33.829043+0000","last_peered":"2024-09-02T10:05:33.829043+0000","last_clean":"2024-09-02T10:05:33.829043+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:05:33.829043+0000","last_undegraded":"2024-09-02T10:05:33.829043+0000","last_fullsized":"2024-09-02T10:05:33.829043+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:30.278632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:30.278632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:48.025871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"213'12046","reported_seq":12361,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:36.844525+0000","last_change":"2024-09-02T10:02:46.577683+0000","last_active":"2024-09-02T10:05:36.844525+0000","last_peered":"2024-09-02T10:05:36.844525+0000","last_clean":"2024-09-02T10:05:36.844525+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:05:36.844525+0000","last_undegraded":"2024-09-02T10:05:36.844525+0000","last_fullsized":"2024-09-02T10:05:36.844525+0000","mapping_epoch":92,"log_start":"105'2000","ondisk_log_start":"105'2000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'5355","last_scrub_stamp":"2024-09-02T10:02:46.577625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:46.577625+0000","objects_scrubbed":5316,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":232,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12046,"num_write_kb":392896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":202,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.827626+0000","last_change":"2024-09-02T10:04:57.391785+0000","last_active":"2024-09-02T10:05:33.827626+0000","last_peered":"2024-09-02T10:05:33.827626+0000","last_clean":"2024-09-02T10:05:33.827626+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:05:33.827626+0000","last_undegraded":"2024-09-02T10:05:33.827626+0000","last_fullsized":"2024-09-02T10:05:33.827626+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.391723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.391723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:12.774478+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"213'11523","reported_seq":17659,"reported_epoch":213,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:05:34.148427+0000","last_change":"2024-09-02T10:04:57.390580+0000","last_active":"2024-09-02T10:05:34.148427+0000","last_peered":"2024-09-02T10:05:34.148427+0000","last_clean":"2024-09-02T10:03:45.188955+0000","last_became_active":"2024-09-02T10:04:57.363859+0000","last_became_peered":"2024-09-02T10:04:57.363859+0000","last_unstale":"2024-09-02T10:05:34.148427+0000","last_undegraded":"2024-09-02T10:04:57.027533+0000","last_fullsized":"2024-09-02T10:04:57.018785+0000","mapping_epoch":198,"log_start":"104'1500","ondisk_log_start":"104'1500","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'5182","last_scrub_stamp":"2024-09-02T10:02:48.587872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.587872+0000","objects_scrubbed":5181,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23330840,"num_objects":357,"num_object_clones":0,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11523,"num_write_kb":380097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2908,"num_bytes_recovered":190513152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":357}],"blocked_by":[],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":63,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.824069+0000","last_change":"2024-09-02T10:04:54.128208+0000","last_active":"2024-09-02T10:05:33.824069+0000","last_peered":"2024-09-02T10:05:33.824069+0000","last_clean":"2024-09-02T10:05:33.824069+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:05:33.824069+0000","last_undegraded":"2024-09-02T10:05:33.824069+0000","last_fullsized":"2024-09-02T10:05:33.824069+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:54.128142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.128142+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.955526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"214'12096","reported_seq":22359,"reported_epoch":214,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:37.520330+0000","last_change":"2024-09-02T10:05:37.520330+0000","last_active":"2024-09-02T10:05:37.520330+0000","last_peered":"2024-09-02T10:05:37.520330+0000","last_clean":"2024-09-02T10:03:38.171966+0000","last_became_active":"2024-09-02T10:04:57.182024+0000","last_became_peered":"2024-09-02T10:04:57.182024+0000","last_unstale":"2024-09-02T10:05:37.520330+0000","last_undegraded":"2024-09-02T10:05:37.520330+0000","last_fullsized":"2024-09-02T10:04:57.016771+0000","mapping_epoch":198,"log_start":"106'2000","ondisk_log_start":"106'2000","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13107200,"num_objects":200,"num_object_clones":0,"num_object_copies":600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":200,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12096,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3331,"num_bytes_recovered":114360320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2,3","objects":200}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"213'11955","reported_seq":12304,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-02T10:05:34.196826+0000","last_change":"2024-09-02T10:04:38.925752+0000","last_active":"2024-09-02T10:05:34.196826+0000","last_peered":"2024-09-02T10:05:34.196826+0000","last_clean":"2024-09-02T10:05:34.196826+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:05:34.196826+0000","last_undegraded":"2024-09-02T10:05:34.196826+0000","last_fullsized":"2024-09-02T10:05:34.196826+0000","mapping_epoch":92,"log_start":"105'1900","ondisk_log_start":"105'1900","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9618","last_scrub_stamp":"2024-09-02T10:04:38.925565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.925565+0000","objects_scrubbed":2671,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.842942+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21430272,"num_objects":327,"num_object_clones":0,"num_object_copies":981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11955,"num_write_kb":393024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":74,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-09-02T10:05:33.210061+0000","last_change":"2024-09-02T10:04:35.814407+0000","last_active":"2024-09-02T10:05:33.210061+0000","last_peered":"2024-09-02T10:05:33.210061+0000","last_clean":"2024-09-02T10:05:33.210061+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:05:33.210061+0000","last_undegraded":"2024-09-02T10:05:33.210061+0000","last_fullsized":"2024-09-02T10:05:33.210061+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:46.975807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:46.975807+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":259588120,"num_objects":3962,"num_object_clones":0,"num_object_copies":11886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2229,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3962,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":191536,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33844,"num_bytes_recovered":1724973056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1610297344,"data_stored":1610285128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160636,"ondisk_log_size":160636,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383746,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":235908,"kb_used_data":114484,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93509244,"statfs":{"total":95995035648,"available":95753465856,"internally_reserved":0,"allocated":117231616,"data_stored":116687827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":124320196},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841612,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412232,"kb_used_data":290808,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93332920,"statfs":{"total":95995035648,"available":95572910080,"internally_reserved":0,"allocated":297787392,"data_stored":297239539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":124320192},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070545,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":184088,"kb_used_data":98576,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93561064,"statfs":{"total":95995035648,"available":95806529536,"internally_reserved":0,"allocated":100941824,"data_stored":100335755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299476,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205060,"kb_used_data":97524,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93540092,"statfs":{"total":95995035648,"available":95785054208,"internally_reserved":0,"allocated":99864576,"data_stored":99320795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528409,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":245764,"kb_used_data":170164,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93499388,"statfs":{"total":95995035648,"available":95743373312,"internally_reserved":0,"allocated":174247936,"data_stored":173704155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724638,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":98380,"kb_used_data":35260,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93646772,"statfs":{"total":95995035648,"available":95894294528,"internally_reserved":0,"allocated":36106240,"data_stored":35558907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953571,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":142948,"kb_used_data":71508,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93602204,"statfs":{"total":95995035648,"available":95848656896,"internally_reserved":0,"allocated":73224192,"data_stored":72614051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149801,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":390628,"kb_used_data":255132,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93354524,"statfs":{"total":95995035648,"available":95595032576,"internally_reserved":0,"allocated":261255168,"data_stored":260641475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":138541512},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":261004,"kb_used_data":117244,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93484148,"statfs":{"total":95995035648,"available":95727767552,"internally_reserved":0,"allocated":120057856,"data_stored":119510523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":165140,"kb_used_data":61760,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93580012,"statfs":{"total":95995035648,"available":95825932288,"internally_reserved":0,"allocated":63242240,"data_stored":62693979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738488,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169476,"kb_used_data":79988,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93575676,"statfs":{"total":95995035648,"available":95821492224,"internally_reserved":0,"allocated":81907712,"data_stored":81363931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79691776,"data_stored":79691776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61734912,"data_stored":61734912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":112132096,"data_stored":112132096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257822720,"data_stored":257818648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":66785280,"data_stored":66781208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":30081024,"data_stored":30081024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":169213952,"data_stored":169213952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":85327872,"data_stored":85327872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":90439680,"data_stored":90439680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":271323136,"data_stored":271319064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":275513344,"data_stored":275513344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":110231552,"data_stored":110231552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:05:39.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:39.318 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:39.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:39.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:39.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:39.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:39.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:39.485 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:39 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:05:39.485 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:39 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:05:39.485 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:39 smithi151 ceph-mon[16639]: osdmap e215: 12 total, 12 up, 11 in 2024-09-02T10:05:39.485 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:39 smithi151 ceph-mon[16639]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:39.485 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:39 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:05:39.485 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:39 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:05:39.485 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2229/15294 objects degraded (14.574%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:39 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:05:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:39 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:05:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:39 smithi187 ceph-mon[20072]: osdmap e215: 12 total, 12 up, 11 in 2024-09-02T10:05:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:39 smithi187 ceph-mon[20072]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:39 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:05:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:39 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:05:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2229/15294 objects degraded (14.574%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:39.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:39.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:39.579 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:39.579 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:39 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:05:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:39 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:05:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:39 smithi162 ceph-mon[20279]: osdmap e215: 12 total, 12 up, 11 in 2024-09-02T10:05:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:39 smithi162 ceph-mon[20279]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:39 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:05:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:39 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:05:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2229/15294 objects degraded (14.574%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:05:39.679 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:39.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:39.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:39.795 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:05:39.795 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:05:39.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:39.829 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:05:39.829 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:05:39.831 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:39.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:39.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:39.860 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:05:39.861 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:05:39.895 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:05:39.895 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:05:39.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:05:39.928 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:05:39.932 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:39.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:39.959 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:05:39.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:39.959 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:05:39.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:40.001 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:05:40.001 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:05:40.038 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:05:40.038 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:05:40.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:40.086 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:05:40.086 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:05:40.104 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:5391 2.5 e97/92 2:a41f69a4:::benchmark_data_smithi187_45882_object5390:head v 97'325, mlcod=97'323)", 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:38.216030+0000", 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 241.883482582, 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13279522399999999, 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:38.216030+0000", 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:38.216030+0000", 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:38.216030+0000", 2024-09-02T10:05:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.68e-07 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:38.216130+0000", 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9691999999999994e-05 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:38.348350+0000", 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.132219748 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:38.348356+0000", 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4019999999999999e-06 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:38.348402+0000", 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5865999999999998e-05 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:38.348416+0000", 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3657e-05 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:38.348783+0000", 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000367656 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:38.348825+0000", 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1634999999999997e-05 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:8373 2.1 e97/92 2:8eb82fd1:::benchmark_data_smithi187_45882_object8372:head v 97'536, mlcod=97'533)", 2024-09-02T10:05:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.035752+0000", 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 240.063760479, 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16431514799999999, 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035752+0000", 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035752+0000", 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035753+0000", 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1999999999999998e-07 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035789+0000", 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6365999999999997e-05 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.199174+0000", 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.163385155 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.199186+0000", 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1626e-05 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.199261+0000", 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4898e-05 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.199284+0000", 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2804999999999999e-05 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.199999+0000", 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00071496199999999998 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.200067+0000", 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8416000000000004e-05 2024-09-02T10:05:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:9159 2.8 e98/92 2:1c146a3c:::benchmark_data_smithi187_45882_object9158:head v 98'582, mlcod=98'580)", 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.618746+0000", 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 239.48076679100001, 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.163030018, 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618746+0000", 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618746+0000", 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618746+0000", 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3499999999999996e-07 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618850+0000", 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010351100000000001 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781192+0000", 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16234200400000001 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781196+0000", 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5290000000000002e-06 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781247+0000", 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1226e-05 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781276+0000", 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9014999999999999e-05 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781733+0000", 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045623300000000001 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781776+0000", 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2864999999999997e-05 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:22417 2.8 e104/92 2:10aa9598:::benchmark_data_smithi187_45882_object22416:head v 104'1402, mlcod=104'1398)", 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:51.200691+0000", 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 228.898821527, 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13786187599999999, 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:51.200691+0000", 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:51.200691+0000", 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:51.200691+0000", 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2699999999999999e-07 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:51.200774+0000", 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3067999999999998e-05 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:51.337495+0000", 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13672111100000001 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:51.337505+0000", 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6560000000000004e-06 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:51.337586+0000", 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0383000000000006e-05 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:51.337612+0000", 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6115999999999999e-05 2024-09-02T10:05:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:51.338506+0000", 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00089438400000000002 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:51.338553+0000", 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6631000000000001e-05 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:27115 2.4 e105/92 2:211dbe49:::benchmark_data_smithi187_45882_object27114:head v 105'1674, mlcod=105'1671)", 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:54.684341+0000", 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 225.415170905, 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19103002599999999, 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.684341+0000", 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.684341+0000", 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.684342+0000", 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.684366+0000", 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3853999999999999e-05 2024-09-02T10:05:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.874351+0000", 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18998515999999999 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.874360+0000", 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7460000000000006e-06 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.874419+0000", 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8624e-05 2024-09-02T10:05:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.874439+0000", 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0698000000000001e-05 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.875320+0000", 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00088058199999999998 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.875371+0000", 2024-09-02T10:05:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1518999999999999e-05 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:37137 2.8 e110/92 2:19842f3c:::benchmark_data_smithi187_45882_object37136:head v 110'2294, mlcod=110'2291)", 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:02.805961+0000", 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.29355169499999, 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.160898445, 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.805961+0000", 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.805961+0000", 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.805962+0000", 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0100000000000003e-07 2024-09-02T10:05:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.806067+0000", 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105635 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.965743+0000", 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.159676075 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.965749+0000", 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-02T10:05:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.965794+0000", 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5435999999999997e-05 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.965808+0000", 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3380999999999999e-05 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.966824+0000", 2024-09-02T10:05:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001015988 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.966859+0000", 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5485000000000003e-05 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:40737 2.c e111/92 2:307dfd04:::benchmark_data_smithi187_45882_object40736:head v 111'2615, mlcod=111'2612)", 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:05.722004+0000", 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 214.37750875200001, 2024-09-02T10:05:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.168674661, 2024-09-02T10:05:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.722004+0000", 2024-09-02T10:05:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:40.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:40.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.722004+0000", 2024-09-02T10:05:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.722005+0000", 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.547e-06 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.722112+0000", 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107323 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.889099+0000", 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16698626999999999 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.889108+0000", 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0469999999999999e-06 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.889176+0000", 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7788999999999998e-05 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.889198+0000", 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1970000000000001e-05 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.890635+0000", 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001437186 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.890678+0000", 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3529000000000003e-05 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54903 2.c e113/92 2:3cdee963:::benchmark_data_smithi187_45882_object54902:head v 113'3490, mlcod=113'3487)", 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.130749+0000", 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 202.96876347200001, 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.263935692, 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130749+0000", 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130749+0000", 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130750+0000", 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1699999999999997e-07 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130858+0000", 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108033 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.392762+0000", 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.261904201 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.392768+0000", 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6810000000000001e-06 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.392838+0000", 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9361000000000002e-05 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.392859+0000", 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1509999999999999e-05 2024-09-02T10:05:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.394641+0000", 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001781691 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.394685+0000", 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3498000000000003e-05 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:60685 2.c e114/92 2:332694f8:::benchmark_data_smithi187_45882_object60684:head v 114'3818, mlcod=114'3815)", 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:22.282941+0000", 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 197.816571772, 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.140047478, 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:22.282941+0000", 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:22.282941+0000", 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:22.282942+0000", 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.141e-06 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:22.282976+0000", 2024-09-02T10:05:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4084000000000001e-05 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:22.421199+0000", 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13822309999999999 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:22.421206+0000", 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1559999999999998e-06 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:22.421263+0000", 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6709000000000002e-05 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:22.421279+0000", 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6554000000000001e-05 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:22.422938+0000", 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016586560000000001 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:22.422988+0000", 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0077999999999997e-05 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:67911 2.4 e120/92 2:20b40a62:::benchmark_data_smithi187_45882_object67910:head v 120'4182, mlcod=120'4179)", 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:29.867694+0000", 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 190.23181853, 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23454814500000001, 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867694+0000", 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867694+0000", 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867695+0000", 2024-09-02T10:05:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2669999999999999e-06 2024-09-02T10:05:40.149 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:05:40.149 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:05:40.149 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867729+0000", 2024-09-02T10:05:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3627000000000002e-05 2024-09-02T10:05:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.099825+0000", 2024-09-02T10:05:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23209649800000001 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.099832+0000", 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.099878+0000", 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6208999999999998e-05 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.099899+0000", 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0531e-05 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.102216+0000", 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023169240000000002 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.102242+0000", 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6282999999999999e-05 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:78685 2.4 e124/92 2:26d5ebbe:::benchmark_data_smithi187_45882_object78684:head v 124'4867, mlcod=124'4864)", 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:40.448379+0000", 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 179.65113331699999, 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13553587, 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.448379+0000", 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.448379+0000", 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.448380+0000", 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1999999999999999e-07 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.448400+0000", 2024-09-02T10:05:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0126e-05 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.581862+0000", 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13346169299999999 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.581868+0000", 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7390000000000002e-06 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.581954+0000", 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6217000000000005e-05 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.581974+0000", 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.995e-05 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.583884+0000", 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001909712 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:40.583915+0000", 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0713000000000002e-05 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:89010 2.1 e133/92 2:8aab5c95:::benchmark_data_smithi187_45882_object89009:head v 133'5587, mlcod=133'5582)", 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:54.476969+0000", 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 165.62254310700001, 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14618494600000001, 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:54.476969+0000", 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:54.476969+0000", 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:54.476970+0000", 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7499999999999999e-07 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:54.476995+0000", 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5091000000000001e-05 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:54.620094+0000", 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.143098802 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:54.620100+0000", 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2020000000000001e-06 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:54.620146+0000", 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5550999999999998e-05 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:54.620164+0000", 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8046e-05 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:54.623106+0000", 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029427440000000002 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:54.623154+0000", 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7735000000000003e-05 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:127291 2.8 e155/153 2:135f62a8:::benchmark_data_smithi187_45882_object29540:head v 155'7963, mlcod=105'1854)", 2024-09-02T10:05:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:44.998279+0000", 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 115.101232825, 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15613745700000001, 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.998279+0000", 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.998279+0000", 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.998281+0000", 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.296e-06 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.998287+0000", 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2990000000000002e-06 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.150388+0000", 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15210099099999999 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.150398+0000", 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0151e-05 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.150477+0000", 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8641999999999997e-05 2024-09-02T10:05:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.150504+0000", 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7169999999999999e-05 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.154342+0000", 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003837549 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.154417+0000", 2024-09-02T10:05:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5358999999999995e-05 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:127816 2.c e158/92 2:39e867c5:::benchmark_data_smithi187_45882_object30065:head v 158'8077, mlcod=158'8074)", 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:47.962337+0000", 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 112.137175361, 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.136642244, 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:47.962337+0000", 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:47.962337+0000", 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:47.962339+0000", 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6020000000000001e-06 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:47.962349+0000", 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0368e-05 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:48.095105+0000", 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.132756504 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:48.095114+0000", 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3580000000000003e-06 2024-09-02T10:05:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:48.095181+0000", 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7267999999999994e-05 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:48.095201+0000", 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9701e-05 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:48.098886+0000", 2024-09-02T10:05:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036856810000000001 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:48.098979+0000", 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2762000000000004e-05 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:141612 2.4 e169/131 2:218ceefe:::benchmark_data_smithi187_45882_object43861:head v 169'8782, mlcod=113'3590)", 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:14.899954+0000", 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 85.199558683999996, 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17027746399999999, 2024-09-02T10:05:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.899954+0000", 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.899954+0000", 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.899955+0000", 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3990000000000001e-06 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.899965+0000", 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6069999999999997e-06 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.065161+0000", 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16519602799999999 2024-09-02T10:05:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.065170+0000", 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5380000000000008e-06 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.065278+0000", 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107441 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.065304+0000", 2024-09-02T10:05:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6769000000000001e-05 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.070039+0000", 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047348110000000002 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.070231+0000", 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019187100000000001 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:141636 2.4 e169/131 2:2765e408:::benchmark_data_smithi187_45882_object43885:head v 169'8783, mlcod=113'3590)", 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:14.954262+0000", 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 85.145250117000003, 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13056775900000001, 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.954262+0000", 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.954262+0000", 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.954263+0000", 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3039999999999999e-06 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.954273+0000", 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2809999999999997e-06 2024-09-02T10:05:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.072473+0000", 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.11819982499999999 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.072480+0000", 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9720000000000003e-06 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.072560+0000", 2024-09-02T10:05:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0725999999999999e-05 2024-09-02T10:05:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.072590+0000", 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9842999999999999e-05 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.081636+0000", 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0090455780000000003 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.084830+0000", 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031942300000000002 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:152189 2.8 e181/153 2:1833c2b6:::benchmark_data_smithi187_45882_object54438:head v 181'9498, mlcod=114'3827)", 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:34.787034+0000", 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 65.312478447000004, 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.170816568, 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.787034+0000", 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.787034+0000", 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.787035+0000", 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.09e-07 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.787041+0000", 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7819999999999996e-06 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.954068+0000", 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16702689800000001 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.954084+0000", 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.592e-05 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.954292+0000", 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020735799999999999 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.954363+0000", 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1605000000000003e-05 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.957819+0000", 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034561090000000002 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.957850+0000", 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1087000000000002e-05 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:167526 2.c e202/92 2:3d2435c3:::benchmark_data_smithi187_45882_object69775:head v 202'10544, mlcod=202'10541)", 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:03.431252+0000", 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.668260552, 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.138485052, 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.431252+0000", 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.431252+0000", 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.431252+0000", 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6700000000000003e-07 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.431257+0000", 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8550000000000001e-06 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.560059+0000", 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12880180399999999 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.560065+0000", 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.144e-06 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.560111+0000", 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6017999999999999e-05 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.560141+0000", 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0117999999999999e-05 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.569626+0000", 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0094848680000000005 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.569737+0000", 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110578 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:167761 2.c e202/92 2:3f32c72e:::benchmark_data_smithi187_45882_object70010:head v 202'10555, mlcod=202'10552)", 2024-09-02T10:05:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:03.800156+0000", 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.299356072999998, 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.148199367, 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.800156+0000", 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.800156+0000", 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.800157+0000", 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3900000000000003e-07 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.800165+0000", 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4529999999999996e-06 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.942683+0000", 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.142518123 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.942689+0000", 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.942820+0000", 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013107200000000001 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.942844+0000", 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3723e-05 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.948275+0000", 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054306700000000003 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.948356+0000", 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0754999999999995e-05 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:184461 2.c e211/92 2:344b268c:::benchmark_data_smithi187_45882_object86710:head v 211'11601, mlcod=211'11597)", 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:26.676548+0000", 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.422964559, 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13361698899999999, 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:26.676548+0000", 2024-09-02T10:05:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:26.676548+0000", 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:26.676549+0000", 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0270000000000001e-06 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:05:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:26.676555+0000", 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3740000000000003e-06 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:26.804460+0000", 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12790460400000001 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:26.804468+0000", 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1920000000000005e-06 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:05:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:26.804603+0000", 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000135248 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:26.804627+0000", 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.374e-05 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:26.810061+0000", 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054341160000000001 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:05:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:05:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:26.810165+0000", 2024-09-02T10:05:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103688 2024-09-02T10:05:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:05:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:05:40.197 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:05:40.197 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:05:40.198 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:05:40.205 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:05:40.205 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:05:40.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:40.260 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:40 smithi151 ceph-mon[16639]: pgmap v535: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 248 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 628 op/s; 2229/11892 objects degraded (18.744%); 0 B/s, 158 objects/s recovering 2024-09-02T10:05:40.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:40.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:40.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:40.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:40.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:40.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:40 smithi187 ceph-mon[20072]: pgmap v535: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 248 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 628 op/s; 2229/11892 objects degraded (18.744%); 0 B/s, 158 objects/s recovering 2024-09-02T10:05:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3842046171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:40.518 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:40.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:40.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:40 smithi162 ceph-mon[20279]: pgmap v535: 37 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 33 active+clean; 248 MiB data, 2.3 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 628 op/s; 2229/11892 objects degraded (18.744%); 0 B/s, 158 objects/s recovering 2024-09-02T10:05:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3842046171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:40.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:40.642 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:40.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:40.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3842046171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:40.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:40.766 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:40.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:40.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:40.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:40.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:40.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:40.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:41.015 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:41.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:41.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:41.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:41.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:41.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:41.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:41.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:41.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:41.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:41.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:41.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:41.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:41.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:41 smithi187 ceph-mon[20072]: osdmap e216: 12 total, 12 up, 11 in 2024-09-02T10:05:41.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:41.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:41.586 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:41 smithi151 ceph-mon[16639]: osdmap e216: 12 total, 12 up, 11 in 2024-09-02T10:05:41.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:41 smithi162 ceph-mon[20279]: osdmap e216: 12 total, 12 up, 11 in 2024-09-02T10:05:41.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:41.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:41.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:41.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:41.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:41.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:41.788 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:41.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:41.912 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:41.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:41.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:41.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:05:42.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:42.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:42.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:42.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:42.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:42.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:42.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:42.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:05:42.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:42.308 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:42.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:42.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:42.433 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:42.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:42.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:42 smithi151 ceph-mon[16639]: pgmap v537: 37 pgs: 1 peering, 2 active+recovering+undersized+remapped, 34 active+clean; 177 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 0 B/s, 270 objects/s recovering 2024-09-02T10:05:42.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:42 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2229/11892 objects degraded (18.744%), 1 pg degraded) 2024-09-02T10:05:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:42 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:05:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:42 smithi151 ceph-mon[16639]: osdmap e217: 12 total, 12 up, 11 in 2024-09-02T10:05:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:42 smithi151 ceph-mon[16639]: 2.b scrub starts 2024-09-02T10:05:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:42 smithi187 ceph-mon[20072]: pgmap v537: 37 pgs: 1 peering, 2 active+recovering+undersized+remapped, 34 active+clean; 177 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 0 B/s, 270 objects/s recovering 2024-09-02T10:05:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:42 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2229/11892 objects degraded (18.744%), 1 pg degraded) 2024-09-02T10:05:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:42 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:05:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:42 smithi187 ceph-mon[20072]: osdmap e217: 12 total, 12 up, 11 in 2024-09-02T10:05:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:42 smithi187 ceph-mon[20072]: 2.b scrub starts 2024-09-02T10:05:42.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:42.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:42.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:42.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:42 smithi162 ceph-mon[20279]: pgmap v537: 37 pgs: 1 peering, 2 active+recovering+undersized+remapped, 34 active+clean; 177 MiB data, 2.2 GiB used, 981 GiB / 983 GiB avail; 0 B/s wr, 650 op/s; 0 B/s, 270 objects/s recovering 2024-09-02T10:05:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:42 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2229/11892 objects degraded (18.744%), 1 pg degraded) 2024-09-02T10:05:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:42 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:05:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:42 smithi162 ceph-mon[20279]: osdmap e217: 12 total, 12 up, 11 in 2024-09-02T10:05:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:42 smithi162 ceph-mon[20279]: 2.b scrub starts 2024-09-02T10:05:42.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:42.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:42.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:42.805 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:42.827 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:42.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:42.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:42.929 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:42.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:42.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:42.958 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:43.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:43.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:43.081 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:43.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:43.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:43.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:43.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:43.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:43.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:43.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:43.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:43.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:43.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:43.452 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:43.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:43.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:43 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:05:43.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:43 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:05:43.484 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:43 smithi151 ceph-mon[16639]: 2.b scrub ok 2024-09-02T10:05:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:43 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:05:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:43 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:05:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:43 smithi187 ceph-mon[20072]: 2.b scrub ok 2024-09-02T10:05:43.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:43.576 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:43.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:43 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:05:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:43 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:05:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:43 smithi162 ceph-mon[20279]: 2.b scrub ok 2024-09-02T10:05:43.677 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:43.694 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Removed 97748 objects 2024-09-02T10:05:43.694 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Clean up completed and total clean up time :157.248 2024-09-02T10:05:43.699 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:43.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:43.700 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:43.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:43.824 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:43.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:43.836 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:43.858 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:43.926 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:43.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:43.948 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:44.049 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:44.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:44.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:44.072 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:44.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:44.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:44.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:44.267 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:05:44.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:44.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:44.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:44.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:44 smithi151 ceph-mon[16639]: pgmap v539: 37 pgs: 1 peering, 2 active+recovering+undersized+remapped, 34 active+clean; 64 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 0 B/s, 210 objects/s recovering 2024-09-02T10:05:44.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/595150228' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:05:44.372 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:44 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:05:44.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:44.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:44.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:44 smithi187 ceph-mon[20072]: pgmap v539: 37 pgs: 1 peering, 2 active+recovering+undersized+remapped, 34 active+clean; 64 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 0 B/s, 210 objects/s recovering 2024-09-02T10:05:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/595150228' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:05:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:44 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:05:44.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:44.565 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:44.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:44.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:44 smithi162 ceph-mon[20279]: pgmap v539: 37 pgs: 1 peering, 2 active+recovering+undersized+remapped, 34 active+clean; 64 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 689 op/s; 0 B/s, 210 objects/s recovering 2024-09-02T10:05:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/595150228' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:05:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:44 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:05:44.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:44.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:44.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:44.787 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:05:44.787 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:05:44.787 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:05:44.787 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:05:44.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:44.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:44.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:44.920 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:44.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:44.942 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:44.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:45.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:45.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:45.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:45.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:45.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:45.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:45.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:45.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:45.312 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:45.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:45.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:45.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:45.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:45.436 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:45.524 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-02T10:05:45.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-02T10:05:45.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:45.547 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:45 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:45.547 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:45 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:05:45.547 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:45 smithi151 ceph-mon[16639]: osdmap e218: 12 total, 12 up, 11 in 2024-09-02T10:05:45.547 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:45 smithi151 ceph-mon[16639]: 2.c scrub starts 2024-09-02T10:05:45.547 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:45 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:05:45.547 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:45 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:05:45.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:45.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:45.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:45.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:05:45.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:45 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:45 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:05:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:45 smithi162 ceph-mon[20279]: osdmap e218: 12 total, 12 up, 11 in 2024-09-02T10:05:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:45 smithi162 ceph-mon[20279]: 2.c scrub starts 2024-09-02T10:05:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:45 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:05:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:45 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:05:45.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:45.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:45.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:45.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:45 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:45 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:05:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:45 smithi187 ceph-mon[20072]: osdmap e218: 12 total, 12 up, 11 in 2024-09-02T10:05:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:45 smithi187 ceph-mon[20072]: 2.c scrub starts 2024-09-02T10:05:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:45 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:05:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:45 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:05:45.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:45.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:45.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:45.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:45.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:05:45.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:45.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:45.945 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:45.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:45.967 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:05:45.967 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:05:46.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:46.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:46.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:46.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:46.192 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:46.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:46.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:46.316 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:46.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:46.419 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:46.440 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:46.441 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:46.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:46.565 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:46.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:46.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:46 smithi162 ceph-mon[20279]: pgmap v541: 37 pgs: 1 peering, 2 active+recovering+undersized+remapped, 34 active+clean; 30 MiB data, 1.5 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 0 B/s, 249 objects/s recovering 2024-09-02T10:05:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:46 smithi162 ceph-mon[20279]: osdmap e219: 12 total, 12 up, 11 in 2024-09-02T10:05:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:05:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:46 smithi151 ceph-mon[16639]: pgmap v541: 37 pgs: 1 peering, 2 active+recovering+undersized+remapped, 34 active+clean; 30 MiB data, 1.5 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 0 B/s, 249 objects/s recovering 2024-09-02T10:05:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:46 smithi151 ceph-mon[16639]: osdmap e219: 12 total, 12 up, 11 in 2024-09-02T10:05:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:05:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:46.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:46 smithi187 ceph-mon[20072]: pgmap v541: 37 pgs: 1 peering, 2 active+recovering+undersized+remapped, 34 active+clean; 30 MiB data, 1.5 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 0 B/s, 249 objects/s recovering 2024-09-02T10:05:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:46 smithi187 ceph-mon[20072]: osdmap e219: 12 total, 12 up, 11 in 2024-09-02T10:05:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:05:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:46.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:46.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:46.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:46.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:46.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:46.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:47.031 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":542,"stamp":"2024-09-02T10:05:45.270142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":30933536,"num_objects":465,"num_object_clones":0,"num_object_copies":1395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":465,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195035,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35342,"num_bytes_recovered":1725703232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":162967,"ondisk_log_size":162967,"up":111,"acting":109,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1615496,"kb_used_data":460348,"kb_used_omap":17,"kb_used_meta":1154926,"kb_avail":1029581176,"statfs":{"total":1055945392128,"available":1054291124224,"internally_reserved":0,"allocated":471396352,"data_stored":465215832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1182645115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,3,4,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-176160792,"num_objects":-2689,"num_object_clones":0,"num_object_copies":-8067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2689,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1175,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.031268"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":202,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.245544+0000","last_change":"2024-09-02T10:02:43.131354+0000","last_active":"2024-09-02T10:05:41.245544+0000","last_peered":"2024-09-02T10:05:41.245544+0000","last_clean":"2024-09-02T10:05:41.245544+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:05:41.245544+0000","last_undegraded":"2024-09-02T10:05:41.245544+0000","last_fullsized":"2024-09-02T10:05:41.245544+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.131299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.131299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":287,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.292558+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:05:41.292558+0000","last_peered":"2024-09-02T10:05:41.292558+0000","last_clean":"2024-09-02T10:05:41.292558+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:05:41.292558+0000","last_undegraded":"2024-09-02T10:05:41.292558+0000","last_fullsized":"2024-09-02T10:05:41.292558+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.233390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":224,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.244563+0000","last_change":"2024-09-02T10:04:37.422274+0000","last_active":"2024-09-02T10:05:41.244563+0000","last_peered":"2024-09-02T10:05:41.244563+0000","last_clean":"2024-09-02T10:05:41.244563+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:05:41.244563+0000","last_undegraded":"2024-09-02T10:05:41.244563+0000","last_fullsized":"2024-09-02T10:05:41.244563+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:37.422128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:37.422128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:02.404595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":212,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.244331+0000","last_change":"2024-09-02T10:05:38.438762+0000","last_active":"2024-09-02T10:05:41.244331+0000","last_peered":"2024-09-02T10:05:41.244331+0000","last_clean":"2024-09-02T10:05:41.244331+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:05:41.244331+0000","last_undegraded":"2024-09-02T10:05:41.244331+0000","last_fullsized":"2024-09-02T10:05:41.244331+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.438672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.438672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:03.647240+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":217,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.244553+0000","last_change":"2024-09-02T10:04:38.471750+0000","last_active":"2024-09-02T10:05:41.244553+0000","last_peered":"2024-09-02T10:05:41.244553+0000","last_clean":"2024-09-02T10:05:41.244553+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:05:41.244553+0000","last_undegraded":"2024-09-02T10:05:41.244553+0000","last_fullsized":"2024-09-02T10:05:41.244553+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:38.471681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.471681+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:49.847941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.b","version":"217'12158","reported_seq":12523,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:42.988117+0000","last_change":"2024-09-02T10:05:42.563382+0000","last_active":"2024-09-02T10:05:42.988117+0000","last_peered":"2024-09-02T10:05:42.988117+0000","last_clean":"2024-09-02T10:05:42.988117+0000","last_became_active":"2024-09-02T10:01:30.140303+0000","last_became_peered":"2024-09-02T10:01:30.140303+0000","last_unstale":"2024-09-02T10:05:42.988117+0000","last_undegraded":"2024-09-02T10:05:42.988117+0000","last_fullsized":"2024-09-02T10:05:42.988117+0000","mapping_epoch":92,"log_start":"108'2100","ondisk_log_start":"108'2100","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'12130","last_scrub_stamp":"2024-09-02T10:05:42.563300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:42.563300+0000","objects_scrubbed":32,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:08.046792+0000","scrub_duration":870,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12158,"num_write_kb":389056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":297,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.244555+0000","last_change":"2024-09-02T10:05:38.077751+0000","last_active":"2024-09-02T10:05:41.244555+0000","last_peered":"2024-09-02T10:05:41.244555+0000","last_clean":"2024-09-02T10:05:41.244555+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:05:41.244555+0000","last_undegraded":"2024-09-02T10:05:41.244555+0000","last_fullsized":"2024-09-02T10:05:41.244555+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.077684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.077684+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:57.694686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.a","version":"217'12180","reported_seq":12516,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.901526+0000","last_change":"2024-09-02T10:04:56.617908+0000","last_active":"2024-09-02T10:05:41.901526+0000","last_peered":"2024-09-02T10:05:41.901526+0000","last_clean":"2024-09-02T10:05:41.901526+0000","last_became_active":"2024-09-02T10:01:30.559821+0000","last_became_peered":"2024-09-02T10:01:30.559821+0000","last_unstale":"2024-09-02T10:05:41.901526+0000","last_undegraded":"2024-09-02T10:05:41.901526+0000","last_fullsized":"2024-09-02T10:05:41.901526+0000","mapping_epoch":92,"log_start":"108'2100","ondisk_log_start":"108'2100","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10186","last_scrub_stamp":"2024-09-02T10:04:56.617746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.617746+0000","objects_scrubbed":2052,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:15.697367+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12180,"num_write_kb":391616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,9],"acting":[11,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":410,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-02T10:05:40.240037+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:05:40.240037+0000","last_peered":"2024-09-02T10:05:40.240037+0000","last_clean":"2024-09-02T10:05:40.240037+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:05:40.240037+0000","last_undegraded":"2024-09-02T10:05:40.240037+0000","last_fullsized":"2024-09-02T10:05:40.240037+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.9","version":"217'12298","reported_seq":12588,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:42.872482+0000","last_change":"2024-09-02T10:04:55.460032+0000","last_active":"2024-09-02T10:05:42.872482+0000","last_peered":"2024-09-02T10:05:42.872482+0000","last_clean":"2024-09-02T10:05:42.872482+0000","last_became_active":"2024-09-02T10:01:30.560040+0000","last_became_peered":"2024-09-02T10:01:30.560040+0000","last_unstale":"2024-09-02T10:05:42.872482+0000","last_undegraded":"2024-09-02T10:05:42.872482+0000","last_fullsized":"2024-09-02T10:05:42.872482+0000","mapping_epoch":92,"log_start":"109'2200","ondisk_log_start":"109'2200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10251","last_scrub_stamp":"2024-09-02T10:04:55.459770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.459770+0000","objects_scrubbed":2066,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.035919+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524288,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12298,"num_write_kb":393792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":56,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.248212+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:05:41.248212+0000","last_peered":"2024-09-02T10:05:41.248212+0000","last_clean":"2024-09-02T10:05:41.248212+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:05:41.248212+0000","last_undegraded":"2024-09-02T10:05:41.248212+0000","last_fullsized":"2024-09-02T10:05:41.248212+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.8","version":"216'12154","reported_seq":23739,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-02T10:05:40.797829+0000","last_change":"2024-09-02T10:05:38.279924+0000","last_active":"2024-09-02T10:05:40.797829+0000","last_peered":"2024-09-02T10:05:40.797829+0000","last_clean":"2024-09-02T10:05:40.797829+0000","last_became_active":"2024-09-02T10:05:38.234025+0000","last_became_peered":"2024-09-02T10:05:38.234025+0000","last_unstale":"2024-09-02T10:05:40.797829+0000","last_undegraded":"2024-09-02T10:05:40.797829+0000","last_fullsized":"2024-09-02T10:05:40.797829+0000","mapping_epoch":214,"log_start":"108'2100","ondisk_log_start":"108'2100","created":92,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4325376,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12154,"num_write_kb":391040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5654,"num_bytes_recovered":208863232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,0],"acting":[7,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":202,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.248374+0000","last_change":"2024-09-02T10:02:43.245406+0000","last_active":"2024-09-02T10:05:41.248374+0000","last_peered":"2024-09-02T10:05:41.248374+0000","last_clean":"2024-09-02T10:05:41.248374+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:05:41.248374+0000","last_undegraded":"2024-09-02T10:05:41.248374+0000","last_fullsized":"2024-09-02T10:05:41.248374+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.245354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.245354+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"217'12092","reported_seq":23065,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.990521+0000","last_change":"2024-09-02T10:04:27.932509+0000","last_active":"2024-09-02T10:05:41.990521+0000","last_peered":"2024-09-02T10:05:41.990521+0000","last_clean":"2024-09-02T10:05:41.990521+0000","last_became_active":"2024-09-02T10:04:26.448185+0000","last_became_peered":"2024-09-02T10:04:26.448185+0000","last_unstale":"2024-09-02T10:05:41.990521+0000","last_undegraded":"2024-09-02T10:05:41.990521+0000","last_fullsized":"2024-09-02T10:05:41.990521+0000","mapping_epoch":176,"log_start":"106'2000","ondisk_log_start":"106'2000","created":92,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"177'9259","last_scrub_stamp":"2024-09-02T10:04:27.932362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:27.932362+0000","objects_scrubbed":2896,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:56.556476+0000","scrub_duration":606,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12092,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5333,"num_bytes_recovered":349503488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":358,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.248556+0000","last_change":"2024-09-02T10:02:41.206594+0000","last_active":"2024-09-02T10:05:41.248556+0000","last_peered":"2024-09-02T10:05:41.248556+0000","last_clean":"2024-09-02T10:05:41.248556+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:05:41.248556+0000","last_undegraded":"2024-09-02T10:05:41.248556+0000","last_fullsized":"2024-09-02T10:05:41.248556+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:41.206524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:41.206524+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.6","version":"217'12205","reported_seq":12522,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:42.872776+0000","last_change":"2024-09-02T10:04:56.300674+0000","last_active":"2024-09-02T10:05:42.872776+0000","last_peered":"2024-09-02T10:05:42.872776+0000","last_clean":"2024-09-02T10:05:42.872776+0000","last_became_active":"2024-09-02T10:01:30.560374+0000","last_became_peered":"2024-09-02T10:01:30.560374+0000","last_unstale":"2024-09-02T10:05:42.872776+0000","last_undegraded":"2024-09-02T10:05:42.872776+0000","last_fullsized":"2024-09-02T10:05:42.872776+0000","mapping_epoch":92,"log_start":"109'2200","ondisk_log_start":"109'2200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'10171","last_scrub_stamp":"2024-09-02T10:04:56.300400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:56.300400+0000","objects_scrubbed":2044,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:07.471560+0000","scrub_duration":454,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12205,"num_write_kb":390848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,3],"acting":[1,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":196,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.245640+0000","last_change":"2024-09-02T10:02:49.000327+0000","last_active":"2024-09-02T10:05:41.245640+0000","last_peered":"2024-09-02T10:05:41.245640+0000","last_clean":"2024-09-02T10:05:41.245640+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:05:41.245640+0000","last_undegraded":"2024-09-02T10:05:41.245640+0000","last_fullsized":"2024-09-02T10:05:41.245640+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:49.000247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:49.000247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"217'12140","reported_seq":12497,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:43.035609+0000","last_change":"2024-09-02T10:05:04.366916+0000","last_active":"2024-09-02T10:05:43.035609+0000","last_peered":"2024-09-02T10:05:43.035609+0000","last_clean":"2024-09-02T10:05:43.035609+0000","last_became_active":"2024-09-02T10:01:30.182935+0000","last_became_peered":"2024-09-02T10:01:30.182935+0000","last_unstale":"2024-09-02T10:05:43.035609+0000","last_undegraded":"2024-09-02T10:05:43.035609+0000","last_fullsized":"2024-09-02T10:05:43.035609+0000","mapping_epoch":92,"log_start":"108'2100","ondisk_log_start":"108'2100","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'10418","last_scrub_stamp":"2024-09-02T10:05:04.366613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:04.366613+0000","objects_scrubbed":1736,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:11.371378+0000","scrub_duration":861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12140,"num_write_kb":388480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":56,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.248407+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:05:41.248407+0000","last_peered":"2024-09-02T10:05:41.248407+0000","last_clean":"2024-09-02T10:05:41.248407+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:05:41.248407+0000","last_undegraded":"2024-09-02T10:05:41.248407+0000","last_fullsized":"2024-09-02T10:05:41.248407+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.611307+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"217'12128","reported_seq":23024,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:42.895354+0000","last_change":"2024-09-02T10:05:00.573995+0000","last_active":"2024-09-02T10:05:42.895354+0000","last_peered":"2024-09-02T10:05:42.895354+0000","last_clean":"2024-09-02T10:05:42.895354+0000","last_became_active":"2024-09-02T10:04:43.806359+0000","last_became_peered":"2024-09-02T10:04:43.806359+0000","last_unstale":"2024-09-02T10:05:42.895354+0000","last_undegraded":"2024-09-02T10:05:42.895354+0000","last_fullsized":"2024-09-02T10:05:42.895354+0000","mapping_epoch":188,"log_start":"108'2100","ondisk_log_start":"108'2100","created":92,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"202'10224","last_scrub_stamp":"2024-09-02T10:05:00.573420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:05:00.573420+0000","objects_scrubbed":1920,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:20.148546+0000","scrub_duration":803,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524288,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12128,"num_write_kb":388352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5298,"num_bytes_recovered":347209728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":319,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.332612+0000","last_change":"2024-09-02T10:04:35.810947+0000","last_active":"2024-09-02T10:05:41.332612+0000","last_peered":"2024-09-02T10:05:41.332612+0000","last_clean":"2024-09-02T10:05:41.332612+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:05:41.332612+0000","last_undegraded":"2024-09-02T10:05:41.332612+0000","last_fullsized":"2024-09-02T10:05:41.332612+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:28.355594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:28.355594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:51.765081+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"217'12495","reported_seq":12856,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.789037+0000","last_change":"2024-09-02T10:04:58.724992+0000","last_active":"2024-09-02T10:05:41.789037+0000","last_peered":"2024-09-02T10:05:41.789037+0000","last_clean":"2024-09-02T10:05:41.789037+0000","last_became_active":"2024-09-02T10:01:30.140204+0000","last_became_peered":"2024-09-02T10:01:30.140204+0000","last_unstale":"2024-09-02T10:05:41.789037+0000","last_undegraded":"2024-09-02T10:05:41.789037+0000","last_fullsized":"2024-09-02T10:05:41.789037+0000","mapping_epoch":92,"log_start":"110'2400","ondisk_log_start":"110'2400","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"200'10526","last_scrub_stamp":"2024-09-02T10:04:58.724845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.724845+0000","objects_scrubbed":2020,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:18.034443+0000","scrub_duration":326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12495,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":48,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.248553+0000","last_change":"2024-09-02T10:04:57.021532+0000","last_active":"2024-09-02T10:05:41.248553+0000","last_peered":"2024-09-02T10:05:41.248553+0000","last_clean":"2024-09-02T10:05:41.248553+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:05:41.248553+0000","last_undegraded":"2024-09-02T10:05:41.248553+0000","last_fullsized":"2024-09-02T10:05:41.248553+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:38.873802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:38.873802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"217'12236","reported_seq":18692,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:42.486664+0000","last_change":"2024-09-02T10:04:58.839358+0000","last_active":"2024-09-02T10:05:42.486664+0000","last_peered":"2024-09-02T10:05:42.486664+0000","last_clean":"2024-09-02T10:05:42.486664+0000","last_became_active":"2024-09-02T10:04:58.808736+0000","last_became_peered":"2024-09-02T10:04:58.808736+0000","last_unstale":"2024-09-02T10:05:42.486664+0000","last_undegraded":"2024-09-02T10:05:42.486664+0000","last_fullsized":"2024-09-02T10:05:42.486664+0000","mapping_epoch":200,"log_start":"109'2200","ondisk_log_start":"109'2200","created":92,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'5361","last_scrub_stamp":"2024-09-02T10:02:48.223591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.223591+0000","objects_scrubbed":5355,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1246,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12236,"num_write_kb":392192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3059,"num_bytes_recovered":97517568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":58,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-02T10:05:40.240112+0000","last_change":"2024-09-02T10:04:47.929535+0000","last_active":"2024-09-02T10:05:40.240112+0000","last_peered":"2024-09-02T10:05:40.240112+0000","last_clean":"2024-09-02T10:05:40.240112+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:05:40.240112+0000","last_undegraded":"2024-09-02T10:05:40.240112+0000","last_fullsized":"2024-09-02T10:05:40.240112+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:39.894365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:39.894365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"215'11985","reported_seq":31395,"reported_epoch":215,"state":"peering","last_fresh":"2024-09-02T10:05:40.239369+0000","last_change":"2024-09-02T10:05:40.249793+0000","last_active":"2024-09-02T10:05:40.249793+0000","last_peered":"2024-09-02T10:05:40.239369+0000","last_clean":"2024-09-02T10:02:27.223209+0000","last_became_active":"2024-09-02T10:04:57.144949+0000","last_became_peered":"2024-09-02T10:04:57.144949+0000","last_unstale":"2024-09-02T10:05:40.239369+0000","last_undegraded":"2024-09-02T10:05:40.239369+0000","last_fullsized":"2024-09-02T10:04:57.017574+0000","mapping_epoch":216,"log_start":"105'1900","ondisk_log_start":"105'1900","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5832704,"num_objects":89,"num_object_clones":0,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11985,"num_write_kb":386368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8622,"num_bytes_recovered":417005568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,11],"acting":[7,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,9,10,11],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":274,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.292663+0000","last_change":"2024-09-02T10:05:06.815241+0000","last_active":"2024-09-02T10:05:41.292663+0000","last_peered":"2024-09-02T10:05:41.292663+0000","last_clean":"2024-09-02T10:05:41.292663+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:05:41.292663+0000","last_undegraded":"2024-09-02T10:05:41.292663+0000","last_fullsized":"2024-09-02T10:05:41.292663+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:06.815191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:06.815191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:14.504648+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"217'12285","reported_seq":12651,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.829499+0000","last_change":"2024-09-02T10:04:54.927090+0000","last_active":"2024-09-02T10:05:41.829499+0000","last_peered":"2024-09-02T10:05:41.829499+0000","last_clean":"2024-09-02T10:05:41.829499+0000","last_became_active":"2024-09-02T10:01:30.140667+0000","last_became_peered":"2024-09-02T10:01:30.140667+0000","last_unstale":"2024-09-02T10:05:41.829499+0000","last_undegraded":"2024-09-02T10:05:41.829499+0000","last_fullsized":"2024-09-02T10:05:41.829499+0000","mapping_epoch":92,"log_start":"108'2200","ondisk_log_start":"108'2200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'10296","last_scrub_stamp":"2024-09-02T10:04:54.926873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.926873+0000","objects_scrubbed":2056,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.100584+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12285,"num_write_kb":394944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":69,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-02T10:05:40.240019+0000","last_change":"2024-09-02T10:04:55.111033+0000","last_active":"2024-09-02T10:05:40.240019+0000","last_peered":"2024-09-02T10:05:40.240019+0000","last_clean":"2024-09-02T10:05:40.240019+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:05:40.240019+0000","last_undegraded":"2024-09-02T10:05:40.240019+0000","last_fullsized":"2024-09-02T10:05:40.240019+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:55.110979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.110979+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:01.238104+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":304,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.332600+0000","last_change":"2024-09-02T10:04:30.278703+0000","last_active":"2024-09-02T10:05:41.332600+0000","last_peered":"2024-09-02T10:05:41.332600+0000","last_clean":"2024-09-02T10:05:41.332600+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:05:41.332600+0000","last_undegraded":"2024-09-02T10:05:41.332600+0000","last_fullsized":"2024-09-02T10:05:41.332600+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:30.278632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:30.278632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:48.025871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"217'12244","reported_seq":12567,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.873069+0000","last_change":"2024-09-02T10:02:46.577683+0000","last_active":"2024-09-02T10:05:41.873069+0000","last_peered":"2024-09-02T10:05:41.873069+0000","last_clean":"2024-09-02T10:05:41.873069+0000","last_became_active":"2024-09-02T10:01:30.139363+0000","last_became_peered":"2024-09-02T10:01:30.139363+0000","last_unstale":"2024-09-02T10:05:41.873069+0000","last_undegraded":"2024-09-02T10:05:41.873069+0000","last_fullsized":"2024-09-02T10:05:41.873069+0000","mapping_epoch":92,"log_start":"108'2200","ondisk_log_start":"108'2200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'5355","last_scrub_stamp":"2024-09-02T10:02:46.577625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:46.577625+0000","objects_scrubbed":5316,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2228224,"num_objects":34,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12244,"num_write_kb":392896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,1],"acting":[11,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":210,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.246681+0000","last_change":"2024-09-02T10:04:57.391785+0000","last_active":"2024-09-02T10:05:41.246681+0000","last_peered":"2024-09-02T10:05:41.246681+0000","last_clean":"2024-09-02T10:05:41.246681+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:05:41.246681+0000","last_undegraded":"2024-09-02T10:05:41.246681+0000","last_fullsized":"2024-09-02T10:05:41.246681+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.391723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.391723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:12.774478+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"217'11880","reported_seq":19639,"reported_epoch":217,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:43.958192+0000","last_change":"2024-09-02T10:05:36.423753+0000","last_active":"2024-09-02T10:05:43.958192+0000","last_peered":"2024-09-02T10:05:43.958192+0000","last_clean":"2024-09-02T10:03:45.188955+0000","last_became_active":"2024-09-02T10:04:57.363859+0000","last_became_peered":"2024-09-02T10:04:57.363859+0000","last_unstale":"2024-09-02T10:05:43.958192+0000","last_undegraded":"2024-09-02T10:05:43.958192+0000","last_fullsized":"2024-09-02T10:04:57.018785+0000","mapping_epoch":198,"log_start":"105'1800","ondisk_log_start":"105'1800","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'5182","last_scrub_stamp":"2024-09-02T10:02:48.587872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:02:48.587872+0000","objects_scrubbed":5181,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11880,"num_write_kb":380097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3715,"num_bytes_recovered":190513152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":69,"reported_epoch":216,"state":"active+clean","last_fresh":"2024-09-02T10:05:40.240350+0000","last_change":"2024-09-02T10:04:54.128208+0000","last_active":"2024-09-02T10:05:40.240350+0000","last_peered":"2024-09-02T10:05:40.240350+0000","last_clean":"2024-09-02T10:05:40.240350+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:05:40.240350+0000","last_undegraded":"2024-09-02T10:05:40.240350+0000","last_fullsized":"2024-09-02T10:05:40.240350+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:54.128142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.128142+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.955526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"217'12273","reported_seq":23356,"reported_epoch":217,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:05:42.519838+0000","last_change":"2024-09-02T10:05:40.980217+0000","last_active":"2024-09-02T10:05:42.519838+0000","last_peered":"2024-09-02T10:05:42.519838+0000","last_clean":"2024-09-02T10:03:38.171966+0000","last_became_active":"2024-09-02T10:04:57.182024+0000","last_became_peered":"2024-09-02T10:04:57.182024+0000","last_unstale":"2024-09-02T10:05:42.519838+0000","last_undegraded":"2024-09-02T10:05:42.519838+0000","last_fullsized":"2024-09-02T10:04:57.016771+0000","mapping_epoch":198,"log_start":"108'2200","ondisk_log_start":"108'2200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:01:29.117290+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12273,"num_write_kb":393472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3658,"num_bytes_recovered":114360320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":23}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"217'12282","reported_seq":12639,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:43.026465+0000","last_change":"2024-09-02T10:04:38.925752+0000","last_active":"2024-09-02T10:05:43.026465+0000","last_peered":"2024-09-02T10:05:43.026465+0000","last_clean":"2024-09-02T10:05:43.026465+0000","last_became_active":"2024-09-02T10:01:30.182988+0000","last_became_peered":"2024-09-02T10:01:30.182988+0000","last_unstale":"2024-09-02T10:05:43.026465+0000","last_undegraded":"2024-09-02T10:05:43.026465+0000","last_fullsized":"2024-09-02T10:05:43.026465+0000","mapping_epoch":92,"log_start":"109'2200","ondisk_log_start":"109'2200","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'9618","last_scrub_stamp":"2024-09-02T10:04:38.925565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:01:29.117290+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.925565+0000","objects_scrubbed":2671,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.842942+0000","scrub_duration":401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12282,"num_write_kb":393024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,4],"acting":[8,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":84,"reported_epoch":217,"state":"active+clean","last_fresh":"2024-09-02T10:05:41.297926+0000","last_change":"2024-09-02T10:04:35.814407+0000","last_active":"2024-09-02T10:05:41.297926+0000","last_peered":"2024-09-02T10:05:41.297926+0000","last_clean":"2024-09-02T10:05:41.297926+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:05:41.297926+0000","last_undegraded":"2024-09-02T10:05:41.297926+0000","last_fullsized":"2024-09-02T10:05:41.297926+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:46.975807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:46.975807+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":16,"stat_sum":{"num_bytes":30343168,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":463,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":195035,"num_write_kb":6255873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35339,"num_bytes_recovered":1724973056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":517349376,"data_stored":517341232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161035,"ondisk_log_size":161035,"up":48,"acting":46,"num_store_stats":12},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383747,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":142708,"kb_used_data":21284,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93602444,"statfs":{"total":95995035648,"available":95848902656,"internally_reserved":0,"allocated":21794816,"data_stored":21233007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":124320196},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612680,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841613,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":250880,"kb_used_data":133608,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93494272,"statfs":{"total":95995035648,"available":95738134528,"internally_reserved":0,"allocated":136814592,"data_stored":136248719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":120060352},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070547,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95160,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93649992,"statfs":{"total":95995035648,"available":95897591808,"internally_reserved":0,"allocated":1544192,"data_stored":982391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299477,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144928,"kb_used_data":33228,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93600224,"statfs":{"total":95995035648,"available":95846629376,"internally_reserved":0,"allocated":34025472,"data_stored":33472710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528410,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":225964,"kb_used_data":150360,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93519188,"statfs":{"total":95995035648,"available":95763648512,"internally_reserved":0,"allocated":153968640,"data_stored":153411369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724639,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74548,"kb_used_data":11424,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93670604,"statfs":{"total":95995035648,"available":95918698496,"internally_reserved":0,"allocated":11698176,"data_stored":11137353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953573,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81268,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93663884,"statfs":{"total":95995035648,"available":95911817216,"internally_reserved":0,"allocated":1544192,"data_stored":982391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149803,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":165440,"kb_used_data":17648,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93579712,"statfs":{"total":95995035648,"available":95825625088,"internally_reserved":0,"allocated":18071552,"data_stored":17502127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":151321032},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411438,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":207548,"kb_used_data":59756,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93537604,"statfs":{"total":95995035648,"available":95782506496,"internally_reserved":0,"allocated":61190144,"data_stored":60624791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112884,"kb_used_data":5348,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93632268,"statfs":{"total":95995035648,"available":95879442432,"internally_reserved":0,"allocated":5476352,"data_stored":4914551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738489,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114168,"kb_used_data":24676,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93630984,"statfs":{"total":95995035648,"available":95878127616,"internally_reserved":0,"allocated":25268224,"data_stored":24706423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":22806528,"data_stored":22806528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2555904,"data_stored":2555904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":54657024,"data_stored":54657024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":6881280,"data_stored":6881280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5570560,"data_stored":5570560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":25952256,"data_stored":25952256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":135073792,"data_stored":135069720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":251985920,"data_stored":251985920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":11862016,"data_stored":11862016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:05:47.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:05:47.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:47.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:47.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:47.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:47.079 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:47.087 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:47.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:47.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:47.183 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:47.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:47.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:47.306 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:47.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:47.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:47.430 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:47.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:47.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:47.554 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:47.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:47.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:47.594 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:47 smithi151 ceph-mon[16639]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:47.594 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:47 smithi151 ceph-mon[16639]: osdmap e220: 12 total, 12 up, 11 in 2024-09-02T10:05:47.594 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:05:47.594 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:05:47.594 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:47 smithi162 ceph-mon[20279]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:47 smithi162 ceph-mon[20279]: osdmap e220: 12 total, 12 up, 11 in 2024-09-02T10:05:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:05:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:05:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:47.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:47.679 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:47.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:47.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:47 smithi187 ceph-mon[20072]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:47.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:47 smithi187 ceph-mon[20072]: osdmap e220: 12 total, 12 up, 11 in 2024-09-02T10:05:47.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:05:47.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:05:47.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:05:47.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:47.801 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:47.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:47.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:47.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:47.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:48.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:48.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:48.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:48.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:48.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:48.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:48.298 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:48.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:48.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:05:48.358 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:48.386 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:48.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:48.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:48.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:48.545 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:48.546 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:48.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:48.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:48.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:48.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:48.677 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:05:48.677 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:05:48.697 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:05:48.697 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:05:48.718 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:05:48.718 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:05:48.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:05:48.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:05:48.758 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:05:48.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:05:48.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:48.780 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:05:48.781 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:05:48.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:48.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:48.806 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:05:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:05:48.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:48 smithi151 ceph-mon[16639]: pgmap v544: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 4.8 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 0 B/s, 138 objects/s recovering 2024-09-02T10:05:48.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2294754876' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:48.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:48 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:05:48.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:48 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:05:48.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:05:48.828 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:05:48.840 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_0' removed 2024-09-02T10:05:48.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:48 smithi162 ceph-mon[20279]: pgmap v544: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 4.8 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 0 B/s, 138 objects/s recovering 2024-09-02T10:05:48.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2294754876' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:48.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:48 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:05:48.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:48 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:05:48.874 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:05:48.875 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:05:48.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:48.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:05:48.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:05:48.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:48.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:48.926 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:05:48.927 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:05:48.948 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:05:48.949 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:05:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:48 smithi187 ceph-mon[20072]: pgmap v544: 37 pgs: 2 active+recovering+undersized+remapped, 35 active+clean; 4.8 MiB data, 1.3 GiB used, 982 GiB / 983 GiB avail; 0 B/s wr, 467 op/s; 0 B/s, 138 objects/s recovering 2024-09-02T10:05:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2294754876' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:48 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:05:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:48 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:05:49.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:49.047 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:05:49.047 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:05:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:49.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:49.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:49.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:49.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:49.174 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:49.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:49.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:49.298 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:49.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:49.400 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:49.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:49.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:49.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:49.551 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:49.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:49.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:49.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:49.676 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:49.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:49.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:49.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:49.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:49 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:05:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:49 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:05:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:49 smithi162 ceph-mon[20279]: osdmap e221: 12 total, 12 up, 11 in 2024-09-02T10:05:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/763030242' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:05:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:49 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:05:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:49 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:05:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:49 smithi162 ceph-mon[20279]: osdmap e222: 12 total, 12 up, 11 in 2024-09-02T10:05:49.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:49.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:49 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:05:49.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:49 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:05:49.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:49 smithi151 ceph-mon[16639]: osdmap e221: 12 total, 12 up, 11 in 2024-09-02T10:05:49.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/763030242' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:05:49.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:49 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:05:49.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:49 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:05:49.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:49 smithi151 ceph-mon[16639]: osdmap e222: 12 total, 12 up, 11 in 2024-09-02T10:05:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:49.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:49.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:49 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:05:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:49 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:05:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:49 smithi187 ceph-mon[20072]: osdmap e221: 12 total, 12 up, 11 in 2024-09-02T10:05:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/763030242' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:05:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:49 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:05:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:49 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:05:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:49 smithi187 ceph-mon[20072]: osdmap e222: 12 total, 12 up, 11 in 2024-09-02T10:05:50.025 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:50.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:50.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:50.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:50.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:50.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:50.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:50.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:50.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:50.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:50.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:50.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:50.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:50.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:50.553 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:50.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:50.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:50.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:50.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:50.682 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:50.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:50.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:05:50.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:50.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:50 smithi151 ceph-mon[16639]: pgmap v547: 21 pgs: 21 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:50.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:50.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:50 smithi162 ceph-mon[20279]: pgmap v547: 21 pgs: 21 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:50.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:50.933 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:50.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:50 smithi187 ceph-mon[20072]: pgmap v547: 21 pgs: 21 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:51.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:51.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:51.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:51.146 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:05:51.146 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":623,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":222,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":11,"osd_in_since":1725271367,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":21}],"num_pgs":21,"num_pools":1,"num_objects":2,"data_bytes":590368,"bytes_used":1301590016,"bytes_avail":1054643802112,"bytes_total":1055945392128},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":21,"modified":"2024-09-02T10:05:11.115338+0000","services":{}},"progress_events":{}} 2024-09-02T10:05:51.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:51.182 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:51.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:51.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:51.310 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:51.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:51.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:51.438 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:51.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:51.511 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-02T10:05:51.512 INFO:tasks.radosbench:Beginning radosbench... 2024-09-02T10:05:51.513 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-02T10:05:51.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-02T10:05:51.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:51.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:51.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:51.679 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:51.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:51.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:51.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:05:51.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:51.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1342153909' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:05:51.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:51.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:51.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1342153909' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:05:51.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:51.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:51.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1342153909' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:05:52.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:52.084 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:52.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:52.127 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:52.129 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-02T10:05:52.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:05:52.178 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:52.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:52.210 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:52.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:52.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:52.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:52.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:52.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:52.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:52.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:52.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:52.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:52.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:52.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:52.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:52.663 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:52.663 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:52 smithi151 ceph-mon[16639]: pgmap v548: 21 pgs: 21 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:52 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:05:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:52 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:05:52.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:52.704 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:52.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:52.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:52.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:52.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:52.841 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:52.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:52.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:52 smithi162 ceph-mon[20279]: pgmap v548: 21 pgs: 21 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:52.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:52 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:05:52.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:52 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:05:52.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:52 smithi187 ceph-mon[20072]: pgmap v548: 21 pgs: 21 active+clean; 577 KiB data, 1.2 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:52.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:52 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:05:52.908 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:52 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:05:52.942 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:52.968 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:52.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:52.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:53.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:53.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:53.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:53.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:53.222 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:53.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:53.224 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:53.295 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:05:53.295 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:05:53.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:53.353 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:53.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:53.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:53.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:53.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:53.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:53.566 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:05:53.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:53.603 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:53.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:53.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:53.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:53.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:53.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:53.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:53.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:53.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:53 smithi162 ceph-mon[20279]: osdmap e223: 12 total, 12 up, 11 in 2024-09-02T10:05:53.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1461224956' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:53.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:53.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:53.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:53 smithi151 ceph-mon[16639]: osdmap e223: 12 total, 12 up, 11 in 2024-09-02T10:05:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1461224956' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:53.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:53.979 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:53.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:53.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:53 smithi187 ceph-mon[20072]: osdmap e223: 12 total, 12 up, 11 in 2024-09-02T10:05:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1461224956' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:05:54.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:05:54.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:54.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:54.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:54.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:54.233 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:54.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:54.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:54.360 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:54.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:54.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:54.488 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:54.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:54.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:54.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:54.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:54.654 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:54 smithi151 ceph-mon[16639]: pgmap v550: 21 pgs: 21 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:54.655 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:54 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:54.655 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1461224956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:54.655 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:54 smithi151 ceph-mon[16639]: osdmap e224: 12 total, 12 up, 11 in 2024-09-02T10:05:54.655 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:54 smithi151 ceph-mon[16639]: osdmap e225: 12 total, 12 up, 11 in 2024-09-02T10:05:54.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:54.739 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:54.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:54.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:05:54.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:54 smithi162 ceph-mon[20279]: pgmap v550: 21 pgs: 21 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:54.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:54 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1461224956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:54 smithi162 ceph-mon[20279]: osdmap e224: 12 total, 12 up, 11 in 2024-09-02T10:05:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:54 smithi162 ceph-mon[20279]: osdmap e225: 12 total, 12 up, 11 in 2024-09-02T10:05:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:54.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:54.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:05:54.929 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:05:54.930 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:05:54.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:54.990 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:54.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:54.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:54 smithi187 ceph-mon[20072]: pgmap v550: 21 pgs: 21 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:54 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:05:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1461224956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:05:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:54 smithi187 ceph-mon[20072]: osdmap e224: 12 total, 12 up, 11 in 2024-09-02T10:05:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:54 smithi187 ceph-mon[20072]: osdmap e225: 12 total, 12 up, 11 in 2024-09-02T10:05:55.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:55.117 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:55.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:55.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:55.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:55.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:55.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:55.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:55.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:55.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:55.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:55.497 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:55.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:55.572 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_1' created 2024-09-02T10:05:55.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:55.623 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:55.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:55.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:55.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:55.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:55.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:55 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3337171857' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-02T10:05:55.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:55.874 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:55.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:55 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3337171857' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-02T10:05:55.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:05:56.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:56.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:05:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:55 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3337171857' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16} : dispatch 2024-09-02T10:05:56.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:56.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:56.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:56.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:56.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:56.255 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:56.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:56.357 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:56.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-02T10:05:56.384 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:56.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:56.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:56.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:56.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:56.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:56.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:56.622 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:56 smithi151 ceph-mon[16639]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:56.622 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:56 smithi151 ceph-mon[16639]: pgmap v553: 21 pgs: 21 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:56.622 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3337171857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-02T10:05:56.622 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:56 smithi151 ceph-mon[16639]: osdmap e226: 12 total, 12 up, 11 in 2024-09-02T10:05:56.633 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:56.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:56.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:56.757 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:56.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:56 smithi162 ceph-mon[20279]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:56 smithi162 ceph-mon[20279]: pgmap v553: 21 pgs: 21 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3337171857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-02T10:05:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:56 smithi162 ceph-mon[20279]: osdmap e226: 12 total, 12 up, 11 in 2024-09-02T10:05:56.849 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:05:56.849 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:05:56.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:56.885 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:56.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:56.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:56.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:56 smithi187 ceph-mon[20072]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:05:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:56 smithi187 ceph-mon[20072]: pgmap v553: 21 pgs: 21 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3337171857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-02T10:05:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:56 smithi187 ceph-mon[20072]: osdmap e226: 12 total, 12 up, 11 in 2024-09-02T10:05:57.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:57.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:57.071 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:05:57.072 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:05:57.092 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:05:57.092 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:05:57.112 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:05:57.112 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:05:57.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:57.135 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:05:57.135 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:05:57.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:57.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:57.142 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:57.157 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:05:57.157 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:05:57.178 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:05:57.178 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:05:57.200 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:05:57.200 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:05:57.221 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:05:57.222 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:05:57.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:05:57.243 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:05:57.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:05:57.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:05:57.265 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:05:57.269 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:57.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:57.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:05:57.283 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":552,"stamp":"2024-09-02T10:05:53.834262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1174028,"kb_used_data":18820,"kb_used_omap":17,"kb_used_meta":1154926,"kb_avail":1030022644,"statfs":{"total":1055945392128,"available":1054743187456,"internally_reserved":0,"allocated":19271680,"data_stored":12804666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1182645115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":74,"apply_latency_ms":74,"commit_latency_ns":74000000,"apply_latency_ns":74000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.004846"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":212,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.054555+0000","last_change":"2024-09-02T10:02:43.131354+0000","last_active":"2024-09-02T10:05:49.054555+0000","last_peered":"2024-09-02T10:05:49.054555+0000","last_clean":"2024-09-02T10:05:49.054555+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:05:49.054555+0000","last_undegraded":"2024-09-02T10:05:49.054555+0000","last_fullsized":"2024-09-02T10:05:49.054555+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.131299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.131299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":297,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.610679+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:05:49.610679+0000","last_peered":"2024-09-02T10:05:49.610679+0000","last_clean":"2024-09-02T10:05:49.610679+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:05:49.610679+0000","last_undegraded":"2024-09-02T10:05:49.610679+0000","last_fullsized":"2024-09-02T10:05:49.610679+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.233390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":234,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:48.915050+0000","last_change":"2024-09-02T10:04:37.422274+0000","last_active":"2024-09-02T10:05:48.915050+0000","last_peered":"2024-09-02T10:05:48.915050+0000","last_clean":"2024-09-02T10:05:48.915050+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:05:48.915050+0000","last_undegraded":"2024-09-02T10:05:48.915050+0000","last_fullsized":"2024-09-02T10:05:48.915050+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:37.422128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:37.422128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:02.404595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":222,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:48.830171+0000","last_change":"2024-09-02T10:05:38.438762+0000","last_active":"2024-09-02T10:05:48.830171+0000","last_peered":"2024-09-02T10:05:48.830171+0000","last_clean":"2024-09-02T10:05:48.830171+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:05:48.830171+0000","last_undegraded":"2024-09-02T10:05:48.830171+0000","last_fullsized":"2024-09-02T10:05:48.830171+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.438672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.438672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:03.647240+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":227,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:48.915026+0000","last_change":"2024-09-02T10:04:38.471750+0000","last_active":"2024-09-02T10:05:48.915026+0000","last_peered":"2024-09-02T10:05:48.915026+0000","last_clean":"2024-09-02T10:05:48.915026+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:05:48.915026+0000","last_undegraded":"2024-09-02T10:05:48.915026+0000","last_fullsized":"2024-09-02T10:05:48.915026+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:38.471681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.471681+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:49.847941+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":307,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:48.915026+0000","last_change":"2024-09-02T10:05:38.077751+0000","last_active":"2024-09-02T10:05:48.915026+0000","last_peered":"2024-09-02T10:05:48.915026+0000","last_clean":"2024-09-02T10:05:48.915026+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:05:48.915026+0000","last_undegraded":"2024-09-02T10:05:48.915026+0000","last_fullsized":"2024-09-02T10:05:48.915026+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.077684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.077684+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:57.694686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":422,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.055280+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:05:49.055280+0000","last_peered":"2024-09-02T10:05:49.055280+0000","last_clean":"2024-09-02T10:05:49.055280+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:05:49.055280+0000","last_undegraded":"2024-09-02T10:05:49.055280+0000","last_fullsized":"2024-09-02T10:05:49.055280+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":68,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-02T10:05:53.015987+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:05:53.015987+0000","last_peered":"2024-09-02T10:05:53.015987+0000","last_clean":"2024-09-02T10:05:53.015987+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:05:53.015987+0000","last_undegraded":"2024-09-02T10:05:53.015987+0000","last_fullsized":"2024-09-02T10:05:53.015987+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":212,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.054330+0000","last_change":"2024-09-02T10:02:43.245406+0000","last_active":"2024-09-02T10:05:49.054330+0000","last_peered":"2024-09-02T10:05:49.054330+0000","last_clean":"2024-09-02T10:05:49.054330+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:05:49.054330+0000","last_undegraded":"2024-09-02T10:05:49.054330+0000","last_fullsized":"2024-09-02T10:05:49.054330+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.245354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.245354+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:52.376675+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":376,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.054199+0000","last_change":"2024-09-02T10:05:42.420814+0000","last_active":"2024-09-02T10:05:49.054199+0000","last_peered":"2024-09-02T10:05:49.054199+0000","last_clean":"2024-09-02T10:05:49.054199+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:05:49.054199+0000","last_undegraded":"2024-09-02T10:05:49.054199+0000","last_fullsized":"2024-09-02T10:05:49.054199+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:42.420722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:42.420722+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:00.199426+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":206,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.054626+0000","last_change":"2024-09-02T10:02:49.000327+0000","last_active":"2024-09-02T10:05:49.054626+0000","last_peered":"2024-09-02T10:05:49.054626+0000","last_clean":"2024-09-02T10:05:49.054626+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:05:49.054626+0000","last_undegraded":"2024-09-02T10:05:49.054626+0000","last_fullsized":"2024-09-02T10:05:49.054626+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:49.000247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:49.000247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":68,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-02T10:05:53.016081+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:05:53.016081+0000","last_peered":"2024-09-02T10:05:53.016081+0000","last_clean":"2024-09-02T10:05:53.016081+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:05:53.016081+0000","last_undegraded":"2024-09-02T10:05:53.016081+0000","last_fullsized":"2024-09-02T10:05:53.016081+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.611307+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":329,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.054812+0000","last_change":"2024-09-02T10:04:35.810947+0000","last_active":"2024-09-02T10:05:49.054812+0000","last_peered":"2024-09-02T10:05:49.054812+0000","last_clean":"2024-09-02T10:05:49.054812+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:05:49.054812+0000","last_undegraded":"2024-09-02T10:05:49.054812+0000","last_fullsized":"2024-09-02T10:05:49.054812+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:28.355594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:28.355594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:51.765081+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":60,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-02T10:05:53.015987+0000","last_change":"2024-09-02T10:04:57.021532+0000","last_active":"2024-09-02T10:05:53.015987+0000","last_peered":"2024-09-02T10:05:53.015987+0000","last_clean":"2024-09-02T10:05:53.015987+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:05:53.015987+0000","last_undegraded":"2024-09-02T10:05:53.015987+0000","last_fullsized":"2024-09-02T10:05:53.015987+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:38.873802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:38.873802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:56.001320+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":70,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.054368+0000","last_change":"2024-09-02T10:04:47.929535+0000","last_active":"2024-09-02T10:05:49.054368+0000","last_peered":"2024-09-02T10:05:49.054368+0000","last_clean":"2024-09-02T10:05:49.054368+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:05:49.054368+0000","last_undegraded":"2024-09-02T10:05:49.054368+0000","last_fullsized":"2024-09-02T10:05:49.054368+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:39.894365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:39.894365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":284,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.610698+0000","last_change":"2024-09-02T10:05:06.815241+0000","last_active":"2024-09-02T10:05:49.610698+0000","last_peered":"2024-09-02T10:05:49.610698+0000","last_clean":"2024-09-02T10:05:49.610698+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:05:49.610698+0000","last_undegraded":"2024-09-02T10:05:49.610698+0000","last_fullsized":"2024-09-02T10:05:49.610698+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:06.815191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:06.815191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:14.504648+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":81,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.054222+0000","last_change":"2024-09-02T10:04:55.111033+0000","last_active":"2024-09-02T10:05:49.054222+0000","last_peered":"2024-09-02T10:05:49.054222+0000","last_clean":"2024-09-02T10:05:49.054222+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:05:49.054222+0000","last_undegraded":"2024-09-02T10:05:49.054222+0000","last_fullsized":"2024-09-02T10:05:49.054222+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:55.110979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.110979+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:01.238104+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":314,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.054704+0000","last_change":"2024-09-02T10:04:30.278703+0000","last_active":"2024-09-02T10:05:49.054704+0000","last_peered":"2024-09-02T10:05:49.054704+0000","last_clean":"2024-09-02T10:05:49.054704+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:05:49.054704+0000","last_undegraded":"2024-09-02T10:05:49.054704+0000","last_fullsized":"2024-09-02T10:05:49.054704+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:30.278632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:30.278632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":220,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.054035+0000","last_change":"2024-09-02T10:04:57.391785+0000","last_active":"2024-09-02T10:05:49.054035+0000","last_peered":"2024-09-02T10:05:49.054035+0000","last_clean":"2024-09-02T10:05:49.054035+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:05:49.054035+0000","last_undegraded":"2024-09-02T10:05:49.054035+0000","last_fullsized":"2024-09-02T10:05:49.054035+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.391723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.391723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:12.774478+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":81,"reported_epoch":222,"state":"active+clean","last_fresh":"2024-09-02T10:05:49.054482+0000","last_change":"2024-09-02T10:04:54.128208+0000","last_active":"2024-09-02T10:05:49.054482+0000","last_peered":"2024-09-02T10:05:49.054482+0000","last_clean":"2024-09-02T10:05:49.054482+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:05:49.054482+0000","last_undegraded":"2024-09-02T10:05:49.054482+0000","last_fullsized":"2024-09-02T10:05:49.054482+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:54.128142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.128142+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.955526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":96,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-02T10:05:52.737795+0000","last_change":"2024-09-02T10:04:35.814407+0000","last_active":"2024-09-02T10:05:52.737795+0000","last_peered":"2024-09-02T10:05:52.737795+0000","last_clean":"2024-09-02T10:05:52.737795+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:05:52.737795+0000","last_undegraded":"2024-09-02T10:05:52.737795+0000","last_fullsized":"2024-09-02T10:05:52.737795+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:46.975807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:46.975807+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:57.443479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383749,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127164,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93617988,"statfs":{"total":95995035648,"available":95864819712,"internally_reserved":0,"allocated":1609728,"data_stored":1021204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":128580036},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841615,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118848,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93626304,"statfs":{"total":95995035648,"available":95873335296,"internally_reserved":0,"allocated":1609728,"data_stored":1021212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":120060352},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070548,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95208,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93649944,"statfs":{"total":95995035648,"available":95897542656,"internally_reserved":0,"allocated":1593344,"data_stored":1013491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299479,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100800,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93644352,"statfs":{"total":95995035648,"available":95891816448,"internally_reserved":0,"allocated":1609728,"data_stored":1021212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528412,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77500,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93667652,"statfs":{"total":95995035648,"available":95915675648,"internally_reserved":0,"allocated":1937408,"data_stored":1348892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724641,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64900,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93680252,"statfs":{"total":95995035648,"available":95928578048,"internally_reserved":0,"allocated":1814528,"data_stored":1222460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953574,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81320,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93663832,"statfs":{"total":95995035648,"available":95911763968,"internally_reserved":0,"allocated":1593344,"data_stored":1013491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149805,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":154052,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":151934,"kb_avail":93591100,"statfs":{"total":95995035648,"available":95837286400,"internally_reserved":0,"allocated":2142208,"data_stored":1550140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":155580872},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411440,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":149572,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93595580,"statfs":{"total":95995035648,"available":95841873920,"internally_reserved":0,"allocated":1814528,"data_stored":1222460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109116,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636036,"statfs":{"total":95995035648,"available":95883300864,"internally_reserved":0,"allocated":1609728,"data_stored":1021212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95548,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93649604,"statfs":{"total":95995035648,"available":95897194496,"internally_reserved":0,"allocated":1937408,"data_stored":1348892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:05:57.288 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:05:57.289 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:05:57.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:57.397 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:57.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:57.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:57.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:57.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:57.546 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:57.558 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:57.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:57.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:57.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:57.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:05:57.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:05:57.773 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:57.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:05:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:57 smithi162 ceph-mon[20279]: osdmap e227: 12 total, 12 up, 11 in 2024-09-02T10:05:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/23474231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:57.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:57.900 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:57.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:57.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:57 smithi151 ceph-mon[16639]: osdmap e227: 12 total, 12 up, 11 in 2024-09-02T10:05:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/23474231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:58.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:57 smithi187 ceph-mon[20072]: osdmap e227: 12 total, 12 up, 11 in 2024-09-02T10:05:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/23474231' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:05:58.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:58.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:58.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:58.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:58.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:58.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:58.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:58.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:05:58.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:58.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:05:58.382 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:58.407 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:58.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:58.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:58.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:58.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:58.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:58.659 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:58.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:58.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:58.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:58.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:58.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:58.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:58.814 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:58 smithi151 ceph-mon[16639]: pgmap v556: 37 pgs: 4 creating+peering, 6 unknown, 27 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:58.814 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:58 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:05:58.814 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:58 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:05:58.814 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:58 smithi151 ceph-mon[16639]: osdmap e228: 12 total, 12 up, 11 in 2024-09-02T10:05:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:58 smithi162 ceph-mon[20279]: pgmap v556: 37 pgs: 4 creating+peering, 6 unknown, 27 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:58 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:05:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:58 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:05:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:58 smithi162 ceph-mon[20279]: osdmap e228: 12 total, 12 up, 11 in 2024-09-02T10:05:58.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:58.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:58.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:58.939 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:05:58.939 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:05:58.939 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:05:58.939 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:05:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:58 smithi187 ceph-mon[20072]: pgmap v556: 37 pgs: 4 creating+peering, 6 unknown, 27 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:58 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:05:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:58 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:05:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:58 smithi187 ceph-mon[20072]: osdmap e228: 12 total, 12 up, 11 in 2024-09-02T10:05:59.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:05:59.022 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-02T10:05:59.022+0000 7fc64259c640 0 --2- 172.21.15.151:0/988803105 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fc644107470 0x7fc644199070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-02T10:05:59.033 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:59.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:59.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:05:59.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:05:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:59.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:59.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:05:59.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:05:59.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:59.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:05:59.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:05:59.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:59.416 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:05:59.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:05:59.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:59.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:59.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:59.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:59.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:05:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:59.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:59.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:05:59.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:05:59.722 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:05:59.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:05:59.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:05:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:59.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:05:59.837 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-02T10:05:59.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:05:59.913 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:05:59.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:05:59.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:05:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:59 smithi151 ceph-mon[16639]: osdmap e229: 12 total, 12 up, 11 in 2024-09-02T10:05:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:59 smithi151 ceph-mon[16639]: pgmap v559: 37 pgs: 4 creating+peering, 2 unknown, 31 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:05:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:05:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/988803105' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:06:00.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:00.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:06:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:00.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:00.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:59 smithi162 ceph-mon[20279]: osdmap e229: 12 total, 12 up, 11 in 2024-09-02T10:06:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:59 smithi162 ceph-mon[20279]: pgmap v559: 37 pgs: 4 creating+peering, 2 unknown, 31 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:06:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:05:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/988803105' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:06:00.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:00.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:00.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:59 smithi187 ceph-mon[20072]: osdmap e229: 12 total, 12 up, 11 in 2024-09-02T10:06:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:59 smithi187 ceph-mon[20072]: pgmap v559: 37 pgs: 4 creating+peering, 2 unknown, 31 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:06:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:05:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/988803105' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:06:00.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:00.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:06:00.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:00.293 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:00.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:00.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:00.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:00.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:00.517 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:00.543 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:00.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:00.590 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:00.590 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:00.629 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:00.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:00.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:00.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:00.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:00.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:00.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:00.797 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:00.839 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:06:00.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:00.924 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:00.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:00.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:01.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:01.052 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:01.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:01.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/988803105' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:06:01.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:00 smithi151 ceph-mon[16639]: osdmap e230: 12 total, 12 up, 11 in 2024-09-02T10:06:01.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3290753918' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:06:01.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/988803105' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:06:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:00 smithi162 ceph-mon[20279]: osdmap e230: 12 total, 12 up, 11 in 2024-09-02T10:06:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3290753918' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:06:01.154 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:01.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:01.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:01.201 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/988803105' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:06:01.202 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:00 smithi187 ceph-mon[20072]: osdmap e230: 12 total, 12 up, 11 in 2024-09-02T10:06:01.202 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3290753918' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:06:01.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:01.309 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:01.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:01.411 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:01.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:01.438 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:01.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:01.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:01.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:01.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:01.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:01.690 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:01.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:01.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:01.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:01.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:01.888 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:01.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:02.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:02.015 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:02.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:02.117 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3290753918' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: osdmap e231: 12 total, 12 up, 11 in 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: pgmap v562: 37 pgs: 2 creating+peering, 35 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:06:02.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:01 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:06:02.142 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:02.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:02.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:02.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3290753918' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:06:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: osdmap e231: 12 total, 12 up, 11 in 2024-09-02T10:06:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:06:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:06:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: pgmap v562: 37 pgs: 2 creating+peering, 35 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:06:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:06:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:06:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:06:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:06:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:06:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:01 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:06:02.270 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:02.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:02.294 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:06:02.295 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3290753918' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: osdmap e231: 12 total, 12 up, 11 in 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: pgmap v562: 37 pgs: 2 creating+peering, 35 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:06:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:06:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:01 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:06:02.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:02.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:02.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:02.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:02.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:02.521 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:02.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:02.522 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:02.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:02.650 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:02.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:02.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:02.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:02.777 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:02.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:02.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:02.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:02.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:02.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:02.953 DEBUG:teuthology.orchestra.run.smithi187:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-02T10:06:02.958 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-02T10:06:02.958 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-02T10:06:03.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:03.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:hints = 1 2024-09-02T10:06:03.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-02T10:06:03.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object prefix: benchmark_data_smithi187_59989 2024-09-02T10:06:03.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:06:03.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 0 0 0 0 0 0 - 0 2024-09-02T10:06:03.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:03.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:03.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:03.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:02 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:06:03.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:02 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:06:03.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:02 smithi151 ceph-mon[16639]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:03.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:02 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:06:03.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:02 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:06:03.148 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:03.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:03.160 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":562,"stamp":"2024-09-02T10:06:01.129898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":80,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":1174756,"kb_used_data":19536,"kb_used_omap":17,"kb_used_meta":1154926,"kb_avail":1030021916,"statfs":{"total":1055945392128,"available":1054742441984,"internally_reserved":0,"allocated":20004864,"data_stored":13252849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1182645115},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000704"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":221,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.689062+0000","last_change":"2024-09-02T10:02:43.131354+0000","last_active":"2024-09-02T10:05:56.689062+0000","last_peered":"2024-09-02T10:05:56.689062+0000","last_clean":"2024-09-02T10:05:56.689062+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:05:56.689062+0000","last_undegraded":"2024-09-02T10:05:56.689062+0000","last_fullsized":"2024-09-02T10:05:56.689062+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.131299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.131299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.072280+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":309,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:57.586992+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:05:57.586992+0000","last_peered":"2024-09-02T10:05:57.586992+0000","last_clean":"2024-09-02T10:05:57.586992+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:05:57.586992+0000","last_undegraded":"2024-09-02T10:05:57.586992+0000","last_fullsized":"2024-09-02T10:05:57.586992+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.233390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":244,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.577162+0000","last_change":"2024-09-02T10:04:37.422274+0000","last_active":"2024-09-02T10:05:56.577162+0000","last_peered":"2024-09-02T10:05:56.577162+0000","last_clean":"2024-09-02T10:05:56.577162+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:05:56.577162+0000","last_undegraded":"2024-09-02T10:05:56.577162+0000","last_fullsized":"2024-09-02T10:05:56.577162+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:37.422128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:37.422128+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:02.404595+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":230,"reported_epoch":226,"state":"active+clean","last_fresh":"2024-09-02T10:05:55.574942+0000","last_change":"2024-09-02T10:05:38.438762+0000","last_active":"2024-09-02T10:05:55.574942+0000","last_peered":"2024-09-02T10:05:55.574942+0000","last_clean":"2024-09-02T10:05:55.574942+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:05:55.574942+0000","last_undegraded":"2024-09-02T10:05:55.574942+0000","last_fullsized":"2024-09-02T10:05:55.574942+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.438672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.438672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:03.647240+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":237,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.576824+0000","last_change":"2024-09-02T10:04:38.471750+0000","last_active":"2024-09-02T10:05:56.576824+0000","last_peered":"2024-09-02T10:05:56.576824+0000","last_clean":"2024-09-02T10:05:56.576824+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:05:56.576824+0000","last_undegraded":"2024-09-02T10:05:56.576824+0000","last_fullsized":"2024-09-02T10:05:56.576824+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:38.471681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:38.471681+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.171498+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.584580+0000","last_change":"2024-09-02T10:05:56.584135+0000","last_active":"2024-09-02T10:05:56.584580+0000","last_peered":"2024-09-02T10:05:56.584580+0000","last_clean":"2024-09-02T10:05:56.584580+0000","last_became_active":"2024-09-02T10:05:56.583663+0000","last_became_peered":"2024-09-02T10:05:56.583663+0000","last_unstale":"2024-09-02T10:05:56.584580+0000","last_undegraded":"2024-09-02T10:05:56.584580+0000","last_fullsized":"2024-09-02T10:05:56.584580+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:21.868607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":317,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.576971+0000","last_change":"2024-09-02T10:05:38.077751+0000","last_active":"2024-09-02T10:05:56.576971+0000","last_peered":"2024-09-02T10:05:56.576971+0000","last_clean":"2024-09-02T10:05:56.576971+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:05:56.576971+0000","last_undegraded":"2024-09-02T10:05:56.576971+0000","last_fullsized":"2024-09-02T10:05:56.576971+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.077684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.077684+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:57.694686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:58.020226+0000","last_change":"2024-09-02T10:05:56.583572+0000","last_active":"2024-09-02T10:05:58.020226+0000","last_peered":"2024-09-02T10:05:58.020226+0000","last_clean":"2024-09-02T10:05:58.020226+0000","last_became_active":"2024-09-02T10:05:56.583278+0000","last_became_peered":"2024-09-02T10:05:56.583278+0000","last_unstale":"2024-09-02T10:05:58.020226+0000","last_undegraded":"2024-09-02T10:05:58.020226+0000","last_fullsized":"2024-09-02T10:05:58.020226+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:55.966086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":438,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.020294+0000","last_change":"2024-09-02T10:01:51.982209+0000","last_active":"2024-09-02T10:06:00.020294+0000","last_peered":"2024-09-02T10:06:00.020294+0000","last_clean":"2024-09-02T10:06:00.020294+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:06:00.020294+0000","last_undegraded":"2024-09-02T10:06:00.020294+0000","last_fullsized":"2024-09-02T10:06:00.020294+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:01:51.982162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:01:51.982162+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:05.019040+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.583925+0000","last_change":"2024-09-02T10:05:56.583542+0000","last_active":"2024-09-02T10:05:56.583925+0000","last_peered":"2024-09-02T10:05:56.583925+0000","last_clean":"2024-09-02T10:05:56.583925+0000","last_became_active":"2024-09-02T10:05:56.583249+0000","last_became_peered":"2024-09-02T10:05:56.583249+0000","last_unstale":"2024-09-02T10:05:56.583925+0000","last_undegraded":"2024-09-02T10:05:56.583925+0000","last_fullsized":"2024-09-02T10:05:56.583925+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:02.205162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":78,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:58.020330+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:05:58.020330+0000","last_peered":"2024-09-02T10:05:58.020330+0000","last_clean":"2024-09-02T10:05:58.020330+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:05:58.020330+0000","last_undegraded":"2024-09-02T10:05:58.020330+0000","last_fullsized":"2024-09-02T10:05:58.020330+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":16,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-02T10:05:58.836352+0000","last_change":"2024-09-02T10:05:56.584669+0000","last_active":"2024-09-02T10:05:58.836352+0000","last_peered":"2024-09-02T10:05:58.836352+0000","last_clean":"2024-09-02T10:05:58.836352+0000","last_became_active":"2024-09-02T10:05:56.584313+0000","last_became_peered":"2024-09-02T10:05:56.584313+0000","last_unstale":"2024-09-02T10:05:58.836352+0000","last_undegraded":"2024-09-02T10:05:58.836352+0000","last_fullsized":"2024-09-02T10:05:58.836352+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:09.326147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":222,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.580116+0000","last_change":"2024-09-02T10:02:43.245406+0000","last_active":"2024-09-02T10:05:56.580116+0000","last_peered":"2024-09-02T10:05:56.580116+0000","last_clean":"2024-09-02T10:05:56.580116+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:05:56.580116+0000","last_undegraded":"2024-09-02T10:05:56.580116+0000","last_fullsized":"2024-09-02T10:05:56.580116+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:43.245354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:43.245354+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:57.428331+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2024-09-02T10:05:55.576468+0000","last_change":"2024-09-02T10:05:55.576468+0000","last_active":"2024-09-02T10:05:55.567155+0000","last_peered":"2024-09-02T10:05:55.567155+0000","last_clean":"2024-09-02T10:05:55.567155+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:05:55.576468+0000","last_undegraded":"2024-09-02T10:05:55.576468+0000","last_fullsized":"2024-09-02T10:05:55.576468+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":386,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.580193+0000","last_change":"2024-09-02T10:05:42.420814+0000","last_active":"2024-09-02T10:05:56.580193+0000","last_peered":"2024-09-02T10:05:56.580193+0000","last_clean":"2024-09-02T10:05:56.580193+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:05:56.580193+0000","last_undegraded":"2024-09-02T10:05:56.580193+0000","last_fullsized":"2024-09-02T10:05:56.580193+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:42.420722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:42.420722+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:00.199426+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:58.020462+0000","last_change":"2024-09-02T10:05:56.585087+0000","last_active":"2024-09-02T10:05:58.020462+0000","last_peered":"2024-09-02T10:05:58.020462+0000","last_clean":"2024-09-02T10:05:58.020462+0000","last_became_active":"2024-09-02T10:05:56.584888+0000","last_became_peered":"2024-09-02T10:05:56.584888+0000","last_unstale":"2024-09-02T10:05:58.020462+0000","last_undegraded":"2024-09-02T10:05:58.020462+0000","last_fullsized":"2024-09-02T10:05:58.020462+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:04.472294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":215,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.689330+0000","last_change":"2024-09-02T10:02:49.000327+0000","last_active":"2024-09-02T10:05:56.689330+0000","last_peered":"2024-09-02T10:05:56.689330+0000","last_clean":"2024-09-02T10:05:56.689330+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:05:56.689330+0000","last_undegraded":"2024-09-02T10:05:56.689330+0000","last_fullsized":"2024-09-02T10:05:56.689330+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:02:49.000247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:02:49.000247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:57.087115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:57.587110+0000","last_change":"2024-09-02T10:05:56.581266+0000","last_active":"2024-09-02T10:05:57.587110+0000","last_peered":"2024-09-02T10:05:57.587110+0000","last_clean":"2024-09-02T10:05:57.587110+0000","last_became_active":"2024-09-02T10:05:56.580910+0000","last_became_peered":"2024-09-02T10:05:56.580910+0000","last_unstale":"2024-09-02T10:05:57.587110+0000","last_undegraded":"2024-09-02T10:05:57.587110+0000","last_fullsized":"2024-09-02T10:05:57.587110+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:23.072076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":78,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:58.020317+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:05:58.020317+0000","last_peered":"2024-09-02T10:05:58.020317+0000","last_clean":"2024-09-02T10:05:58.020317+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:05:58.020317+0000","last_undegraded":"2024-09-02T10:05:58.020317+0000","last_fullsized":"2024-09-02T10:05:58.020317+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.611307+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.690806+0000","last_change":"2024-09-02T10:05:56.690497+0000","last_active":"2024-09-02T10:05:56.690806+0000","last_peered":"2024-09-02T10:05:56.690806+0000","last_clean":"2024-09-02T10:05:56.690806+0000","last_became_active":"2024-09-02T10:05:56.690229+0000","last_became_peered":"2024-09-02T10:05:56.690229+0000","last_unstale":"2024-09-02T10:05:56.690806+0000","last_undegraded":"2024-09-02T10:05:56.690806+0000","last_fullsized":"2024-09-02T10:05:56.690806+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:00.397515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":339,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.581221+0000","last_change":"2024-09-02T10:04:35.810947+0000","last_active":"2024-09-02T10:05:56.581221+0000","last_peered":"2024-09-02T10:05:56.581221+0000","last_clean":"2024-09-02T10:05:56.581221+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:05:56.581221+0000","last_undegraded":"2024-09-02T10:05:56.581221+0000","last_fullsized":"2024-09-02T10:05:56.581221+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:28.355594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:28.355594+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:57.579509+0000","last_change":"2024-09-02T10:05:56.582278+0000","last_active":"2024-09-02T10:05:57.579509+0000","last_peered":"2024-09-02T10:05:57.579509+0000","last_clean":"2024-09-02T10:05:57.579509+0000","last_became_active":"2024-09-02T10:05:56.581969+0000","last_became_peered":"2024-09-02T10:05:56.581969+0000","last_unstale":"2024-09-02T10:05:57.579509+0000","last_undegraded":"2024-09-02T10:05:57.579509+0000","last_fullsized":"2024-09-02T10:05:57.579509+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:24.349478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":70,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:58.020224+0000","last_change":"2024-09-02T10:04:57.021532+0000","last_active":"2024-09-02T10:05:58.020224+0000","last_peered":"2024-09-02T10:05:58.020224+0000","last_clean":"2024-09-02T10:05:58.020224+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:05:58.020224+0000","last_undegraded":"2024-09-02T10:05:58.020224+0000","last_fullsized":"2024-09-02T10:05:58.020224+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:38.873802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:38.873802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:01.874231+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.877213+0000","last_change":"2024-09-02T10:05:56.876782+0000","last_active":"2024-09-02T10:05:56.877213+0000","last_peered":"2024-09-02T10:05:56.877213+0000","last_clean":"2024-09-02T10:05:56.877213+0000","last_became_active":"2024-09-02T10:05:56.876397+0000","last_became_peered":"2024-09-02T10:05:56.876397+0000","last_unstale":"2024-09-02T10:05:56.877213+0000","last_undegraded":"2024-09-02T10:05:56.877213+0000","last_fullsized":"2024-09-02T10:05:56.877213+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:07.285755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":86,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.020081+0000","last_change":"2024-09-02T10:04:47.929535+0000","last_active":"2024-09-02T10:06:00.020081+0000","last_peered":"2024-09-02T10:06:00.020081+0000","last_clean":"2024-09-02T10:06:00.020081+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:06:00.020081+0000","last_undegraded":"2024-09-02T10:06:00.020081+0000","last_fullsized":"2024-09-02T10:06:00.020081+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:39.894365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:39.894365+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1,"reported_epoch":226,"state":"creating+peering","last_fresh":"2024-09-02T10:05:55.576478+0000","last_change":"2024-09-02T10:05:55.576478+0000","last_active":"2024-09-02T10:05:55.567155+0000","last_peered":"2024-09-02T10:05:55.567155+0000","last_clean":"2024-09-02T10:05:55.567155+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:05:55.576478+0000","last_undegraded":"2024-09-02T10:05:55.576478+0000","last_fullsized":"2024-09-02T10:05:55.576478+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":296,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:57.586979+0000","last_change":"2024-09-02T10:05:06.815241+0000","last_active":"2024-09-02T10:05:57.586979+0000","last_peered":"2024-09-02T10:05:57.586979+0000","last_clean":"2024-09-02T10:05:57.586979+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:05:57.586979+0000","last_undegraded":"2024-09-02T10:05:57.586979+0000","last_fullsized":"2024-09-02T10:05:57.586979+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:06.815191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:06.815191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:14.504648+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:58.020382+0000","last_change":"2024-09-02T10:05:56.876935+0000","last_active":"2024-09-02T10:05:58.020382+0000","last_peered":"2024-09-02T10:05:58.020382+0000","last_clean":"2024-09-02T10:05:58.020382+0000","last_became_active":"2024-09-02T10:05:56.876573+0000","last_became_peered":"2024-09-02T10:05:56.876573+0000","last_unstale":"2024-09-02T10:05:58.020382+0000","last_undegraded":"2024-09-02T10:05:58.020382+0000","last_fullsized":"2024-09-02T10:05:58.020382+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:06.299382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":97,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.020121+0000","last_change":"2024-09-02T10:04:55.111033+0000","last_active":"2024-09-02T10:06:00.020121+0000","last_peered":"2024-09-02T10:06:00.020121+0000","last_clean":"2024-09-02T10:06:00.020121+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:06:00.020121+0000","last_undegraded":"2024-09-02T10:06:00.020121+0000","last_fullsized":"2024-09-02T10:06:00.020121+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:55.110979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:55.110979+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:01.238104+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":19,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.020363+0000","last_change":"2024-09-02T10:05:56.877006+0000","last_active":"2024-09-02T10:06:00.020363+0000","last_peered":"2024-09-02T10:06:00.020363+0000","last_clean":"2024-09-02T10:06:00.020363+0000","last_became_active":"2024-09-02T10:05:56.876540+0000","last_became_peered":"2024-09-02T10:05:56.876540+0000","last_unstale":"2024-09-02T10:06:00.020363+0000","last_undegraded":"2024-09-02T10:06:00.020363+0000","last_fullsized":"2024-09-02T10:06:00.020363+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:02.154423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":324,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.581105+0000","last_change":"2024-09-02T10:04:30.278703+0000","last_active":"2024-09-02T10:05:56.581105+0000","last_peered":"2024-09-02T10:05:56.581105+0000","last_clean":"2024-09-02T10:05:56.581105+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:05:56.581105+0000","last_undegraded":"2024-09-02T10:05:56.581105+0000","last_fullsized":"2024-09-02T10:05:56.581105+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:30.278632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:30.278632+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:05:58.171946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.586788+0000","last_change":"2024-09-02T10:05:56.586356+0000","last_active":"2024-09-02T10:05:56.586788+0000","last_peered":"2024-09-02T10:05:56.586788+0000","last_clean":"2024-09-02T10:05:56.586788+0000","last_became_active":"2024-09-02T10:05:56.585883+0000","last_became_peered":"2024-09-02T10:05:56.585883+0000","last_unstale":"2024-09-02T10:05:56.586788+0000","last_undegraded":"2024-09-02T10:05:56.586788+0000","last_fullsized":"2024-09-02T10:05:56.586788+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:06.342689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":232,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:57.579772+0000","last_change":"2024-09-02T10:04:57.391785+0000","last_active":"2024-09-02T10:05:57.579772+0000","last_peered":"2024-09-02T10:05:57.579772+0000","last_clean":"2024-09-02T10:05:57.579772+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:05:57.579772+0000","last_undegraded":"2024-09-02T10:05:57.579772+0000","last_fullsized":"2024-09-02T10:05:57.579772+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.391723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.391723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:12.774478+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.020433+0000","last_change":"2024-09-02T10:05:56.584811+0000","last_active":"2024-09-02T10:06:00.020433+0000","last_peered":"2024-09-02T10:06:00.020433+0000","last_clean":"2024-09-02T10:06:00.020433+0000","last_became_active":"2024-09-02T10:05:56.584450+0000","last_became_peered":"2024-09-02T10:05:56.584450+0000","last_unstale":"2024-09-02T10:06:00.020433+0000","last_undegraded":"2024-09-02T10:06:00.020433+0000","last_fullsized":"2024-09-02T10:06:00.020433+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:04.195054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":97,"reported_epoch":230,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.020081+0000","last_change":"2024-09-02T10:04:54.128208+0000","last_active":"2024-09-02T10:06:00.020081+0000","last_peered":"2024-09-02T10:06:00.020081+0000","last_clean":"2024-09-02T10:06:00.020081+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:06:00.020081+0000","last_undegraded":"2024-09-02T10:06:00.020081+0000","last_fullsized":"2024-09-02T10:06:00.020081+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:54.128142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:54.128142+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-02T10:05:56.586526+0000","last_change":"2024-09-02T10:05:56.586192+0000","last_active":"2024-09-02T10:05:56.586526+0000","last_peered":"2024-09-02T10:05:56.586526+0000","last_clean":"2024-09-02T10:05:56.586526+0000","last_became_active":"2024-09-02T10:05:56.585870+0000","last_became_peered":"2024-09-02T10:05:56.585870+0000","last_unstale":"2024-09-02T10:05:56.586526+0000","last_undegraded":"2024-09-02T10:05:56.586526+0000","last_fullsized":"2024-09-02T10:05:56.586526+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:58.713527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":105,"reported_epoch":228,"state":"active+clean","last_fresh":"2024-09-02T10:05:58.020105+0000","last_change":"2024-09-02T10:04:35.814407+0000","last_active":"2024-09-02T10:05:58.020105+0000","last_peered":"2024-09-02T10:05:58.020105+0000","last_clean":"2024-09-02T10:05:58.020105+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:05:58.020105+0000","last_undegraded":"2024-09-02T10:05:58.020105+0000","last_fullsized":"2024-09-02T10:05:58.020105+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:03:46.975807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:03:46.975807+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383750,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127216,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93617936,"statfs":{"total":95995035648,"available":95864766464,"internally_reserved":0,"allocated":1658880,"data_stored":1051286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":128580036},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841616,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118904,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93626248,"statfs":{"total":95995035648,"available":95873277952,"internally_reserved":0,"allocated":1671168,"data_stored":1059045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":120060352},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070550,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95316,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93649836,"statfs":{"total":95995035648,"available":95897432064,"internally_reserved":0,"allocated":1695744,"data_stored":1074780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299481,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100876,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93644276,"statfs":{"total":95995035648,"available":95891738624,"internally_reserved":0,"allocated":1695744,"data_stored":1074780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528413,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77544,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93667608,"statfs":{"total":95995035648,"available":95915630592,"internally_reserved":0,"allocated":1986560,"data_stored":1378974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724642,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64952,"kb_used_data":1820,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93680200,"statfs":{"total":95995035648,"available":95928524800,"internally_reserved":0,"allocated":1863680,"data_stored":1252542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953576,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81416,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93663736,"statfs":{"total":95995035648,"available":95911665664,"internally_reserved":0,"allocated":1683456,"data_stored":1067101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":154112,"kb_used_data":2152,"kb_used_omap":1,"kb_used_meta":151934,"kb_avail":93591040,"statfs":{"total":95995035648,"available":95837224960,"internally_reserved":0,"allocated":2203648,"data_stored":1587973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":155580872},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":149636,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93595516,"statfs":{"total":95995035648,"available":95841808384,"internally_reserved":0,"allocated":1875968,"data_stored":1260293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":109184,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93635968,"statfs":{"total":95995035648,"available":95883231232,"internally_reserved":0,"allocated":1683456,"data_stored":1067101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738492,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95600,"kb_used_data":1940,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93649552,"statfs":{"total":95995035648,"available":95897141248,"internally_reserved":0,"allocated":1986560,"data_stored":1378974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:06:03.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:06:03.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:02 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:06:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:02 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:06:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:02 smithi187 ceph-mon[20072]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:02 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:06:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:02 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:06:03.284 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:03.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:03.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:02 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:06:03.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:02 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:06:03.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:02 smithi162 ceph-mon[20279]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:03.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:02 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:06:03.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:02 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:06:03.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:03.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:03.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:03.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:03.445 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:06:03.445 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:06:03.445 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:06:03.445 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:06:03.445 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:06:03.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:03.546 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:03.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:03.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:03.647 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:03.669 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:03.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:03.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:03.805 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:03.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:03.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:03.807 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-02T10:06:03.806+0000 7f7b09853640 0 --2- 172.21.15.151:0/2068057612 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f7b04108ed0 0x7f7b0410f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-02T10:06:03.855 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:06:03.855 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:06:03.890 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:06:03.891 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:06:03.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:03 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:06:03.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:03 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:06:03.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:03 smithi151 ceph-mon[16639]: pgmap v563: 37 pgs: 37 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:06:03.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:03 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:06:03.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:03 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:06:03.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2027039376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:03.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:03.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:06:03.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:06:03.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:03.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:03.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:03.956 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:06:03.956 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:06:03.984 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:06:03.985 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:06:04.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:06:04.015 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:06:04.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 1 16 2196 2180 136.242 136.25 0.00220423 0.00729981 2024-09-02T10:06:04.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:04.045 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:06:04.046 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:06:04.065 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:04.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:04.071 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:06:04.071 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:06:04.096 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:06:04.096 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:06:04.122 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:06:04.122 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:06:04.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:06:04.148 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:06:04.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:04.177 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:06:04.177 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:06:04.193 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:04.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:03 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:06:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:03 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:06:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:03 smithi187 ceph-mon[20072]: pgmap v563: 37 pgs: 37 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:06:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:03 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:06:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:03 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:06:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2027039376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:04.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:04.321 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:04.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:04.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:03 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:06:04.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:03 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:06:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:03 smithi162 ceph-mon[20279]: pgmap v563: 37 pgs: 37 active+clean; 577 KiB data, 1.1 GiB used, 982 GiB / 983 GiB avail 2024-09-02T10:06:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:03 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:06:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:03 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:06:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2027039376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:04.423 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:04.445 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:04.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:04.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:06:04.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:04.574 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:04.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:04.676 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:04.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:04.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:04.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:04.823 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:04.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:04.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:04.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:04.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:05.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 2 16 3799 3783 118.211 100.188 0.00122006 0.00835652 2024-09-02T10:06:05.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:05.073 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:05.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:05.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:05.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:05.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:04 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:06:05.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:04 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:06:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:04 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:06:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:04 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:06:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:04 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:06:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:04 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:06:05.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:05.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:04 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:06:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:04 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:06:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:04 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:06:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:04 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:06:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:04 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:06:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:04 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:06:05.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:05.326 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:05.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:05.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:04 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:06:05.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:04 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:06:05.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:04 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:06:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:04 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:06:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:04 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:06:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:04 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:06:05.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:05.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:05.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:05.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:05.578 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:05.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:05.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:05.694 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:05.705 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:05.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:05.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:05.832 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:05.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:05.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:05.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:05 smithi151 ceph-mon[16639]: pgmap v564: 37 pgs: 37 active+clean; 26 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 4.0 MiB/s wr, 64 op/s 2024-09-02T10:06:05.961 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:05.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:05.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:06.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 3 16 5507 5491 114.388 106.75 0.0173635 0.00868617 2024-09-02T10:06:06.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:06.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:06.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:06.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:06.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:06.214 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:06.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:06.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:05 smithi187 ceph-mon[20072]: pgmap v564: 37 pgs: 37 active+clean; 26 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 4.0 MiB/s wr, 64 op/s 2024-09-02T10:06:06.307 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:06.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:06.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:05 smithi162 ceph-mon[20279]: pgmap v564: 37 pgs: 37 active+clean; 26 MiB data, 1.2 GiB used, 982 GiB / 983 GiB avail; 4.0 MiB/s wr, 64 op/s 2024-09-02T10:06:06.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:06.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:06.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:06.471 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:06.472 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:06.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:06.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:06.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:06.609 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:06:06.609 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":638,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":231,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":11,"osd_in_since":1725271367,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":37}],"num_pgs":37,"num_pools":2,"num_objects":407,"data_bytes":27132448,"bytes_used":1236692992,"bytes_avail":1054708699136,"bytes_total":1055945392128,"write_bytes_sec":4214177,"read_op_per_sec":0,"write_op_per_sec":64},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":21,"modified":"2024-09-02T10:05:11.115338+0000","services":{}},"progress_events":{}} 2024-09-02T10:06:06.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:06.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:06.722 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:06.822 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:06.822 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:06.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:06.843 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:06.844 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:06.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:06:06.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:06.968 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:06.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:07.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 4 16 6767 6751 105.477 78.75 0.124576 0.00943724 2024-09-02T10:06:07.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:07.093 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:07.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:06 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:06:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:06 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:06:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:06 smithi151 ceph-mon[16639]: from='client.15354 172.21.15.151:0/3955454079' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:06:07.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:07.222 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:07.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:07.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:06 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:06:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:06 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:06:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:06 smithi187 ceph-mon[20072]: from='client.15354 172.21.15.151:0/3955454079' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:06:07.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:07.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:06 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:06:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:06 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:06:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:06 smithi162 ceph-mon[20279]: from='client.15354 172.21.15.151:0/3955454079' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:06:07.351 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:07.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:07.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:07.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:07.476 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:07.477 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:07.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:07.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:07.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:07.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:07.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:06:07.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:07.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:07.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:07.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:07.884 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:07.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:07.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:07 smithi151 ceph-mon[16639]: pgmap v565: 37 pgs: 37 active+clean; 274 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 34 MiB/s wr, 546 op/s 2024-09-02T10:06:07.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:08.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:08.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:08.011 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:08.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 5 16 8663 8647 108.08 118.5 0.00288874 0.00923723 2024-09-02T10:06:08.075 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-09-02T10:06:08.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:06:08.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:08.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:08.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:08.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:07 smithi187 ceph-mon[20072]: pgmap v565: 37 pgs: 37 active+clean; 274 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 34 MiB/s wr, 546 op/s 2024-09-02T10:06:08.269 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:08.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:08.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:07 smithi162 ceph-mon[20279]: pgmap v565: 37 pgs: 37 active+clean; 274 MiB data, 1.7 GiB used, 982 GiB / 983 GiB avail; 34 MiB/s wr, 546 op/s 2024-09-02T10:06:08.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:08.393 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:08.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:08.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:08.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:08.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:08.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:08.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:08.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:08.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:08.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:08.679 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:08.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:06:08.706 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:08.754 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:08.781 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:08.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:08.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:08.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:08.907 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:08.907 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:09.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:09.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 6 16 10046 10030 104.472 86.4375 0.00750859 0.0095403 2024-09-02T10:06:09.031 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:09.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:09.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:09.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:09.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:09.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:09.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:09.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:09.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:09.403 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:09.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:09.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:09.526 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:09.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:09.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:09.658 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:09.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:09.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:09.783 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:09.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:09.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:09.881 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:06:09.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:09.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:09 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:06:09.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:09 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:06:09.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:09 smithi151 ceph-mon[16639]: osdmap e232: 12 total, 12 up, 11 in 2024-09-02T10:06:09.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:09 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:06:09.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:09 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:06:09.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:09 smithi151 ceph-mon[16639]: pgmap v567: 37 pgs: 37 active+clean; 405 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 49 MiB/s wr, 780 op/s 2024-09-02T10:06:09.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/866134809' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:09.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:06:09.914 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:06:09.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:09.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:10.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:10.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 7 15 11293 11278 100.689 78 0.00628972 0.00992093 2024-09-02T10:06:10.046 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:10.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:10.143 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:06:10.143 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:06:10.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:10.173 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:06:10.173 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:06:10.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:10.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:10.201 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:06:10.201 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:06:10.226 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:06:10.227 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:06:10.252 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:06:10.252 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:06:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:09 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:06:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:09 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:06:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:09 smithi187 ceph-mon[20072]: osdmap e232: 12 total, 12 up, 11 in 2024-09-02T10:06:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:09 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:06:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:09 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:06:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:09 smithi187 ceph-mon[20072]: pgmap v567: 37 pgs: 37 active+clean; 405 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 49 MiB/s wr, 780 op/s 2024-09-02T10:06:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/866134809' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:10.274 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:06:10.275 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:06:10.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:10.300 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:06:10.301 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:06:10.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:10.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:10.326 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:06:10.327 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:06:10.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:09 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:06:10.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:09 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:06:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:09 smithi162 ceph-mon[20279]: osdmap e232: 12 total, 12 up, 11 in 2024-09-02T10:06:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:09 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:06:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:09 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:06:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:09 smithi162 ceph-mon[20279]: pgmap v567: 37 pgs: 37 active+clean; 405 MiB data, 2.1 GiB used, 981 GiB / 983 GiB avail; 49 MiB/s wr, 780 op/s 2024-09-02T10:06:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/866134809' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:10.351 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:06:10.351 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:06:10.375 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:06:10.375 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:06:10.396 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:06:10.396 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:06:10.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:10.426 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:10.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:10.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:10.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:10.550 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:10.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:10.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:10.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:10.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:10.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:10.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:10.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:10.814 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:10.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:10.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:10 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:06:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:10 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:06:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2571453980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/866134809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:10 smithi151 ceph-mon[16639]: osdmap e233: 12 total, 12 up, 11 in 2024-09-02T10:06:10.940 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:10.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:11.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 8 16 12888 12872 100.555 99.625 0.00157339 0.00993032 2024-09-02T10:06:11.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:11.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:11.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:11.168 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:11.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:11.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:10 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:06:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:10 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:06:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2571453980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/866134809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:10 smithi187 ceph-mon[20072]: osdmap e233: 12 total, 12 up, 11 in 2024-09-02T10:06:11.291 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:11.317 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:11.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:11.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:10 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:06:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:10 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:06:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2571453980' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/866134809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:10 smithi162 ceph-mon[20279]: osdmap e233: 12 total, 12 up, 11 in 2024-09-02T10:06:11.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:11.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:11.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:11.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:11.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:11.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:11.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:11.575 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:11.575 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:11.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:11.694 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:11.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:11.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:11.820 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:11.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:11.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:11.945 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:11.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:11.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:12.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 9 16 14052 14036 97.4648 72.75 0.0399484 0.0101706 2024-09-02T10:06:12.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:12.070 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:12.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:12.171 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:12.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:11 smithi151 ceph-mon[16639]: pgmap v569: 37 pgs: 37 active+clean; 524 MiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-02T10:06:12.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:12.198 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:11 smithi187 ceph-mon[20072]: pgmap v569: 37 pgs: 37 active+clean; 524 MiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-02T10:06:12.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:12.322 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:12.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:12.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:11 smithi162 ceph-mon[20279]: pgmap v569: 37 pgs: 37 active+clean; 524 MiB data, 2.5 GiB used, 981 GiB / 983 GiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-02T10:06:12.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:12.447 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:12.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:12.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:12.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:12.574 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:12.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:12.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:12.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:12.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:06:12.702 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:12.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:12.769 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:06:12.769 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:06:12.779 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:06:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:06:12.779 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:06:12.779 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:06:12.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:12.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:12.826 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:12.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:12.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:12.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:13.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 10 16 15329 15313 95.6989 79.8125 0.00691952 0.010441 2024-09-02T10:06:13.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:13.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:13.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:13.184 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:13.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:12 smithi151 ceph-mon[16639]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:13.207 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:13.208 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:12 smithi187 ceph-mon[20072]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:13.309 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:13.331 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:13.332 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:13.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:12 smithi162 ceph-mon[20279]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:13.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:13.457 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:13.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:13.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:06:13.559 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:13.581 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:13.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:13.683 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:13.704 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:13.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:13.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:06:13.806 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:13.829 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:13.831 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:13.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:13.851 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:13.932 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:13.956 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:13.957 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:14.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 11 16 16740 16724 95.0155 88.1875 0.004036 0.0105135 2024-09-02T10:06:14.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:14.084 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:14.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:14.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:14.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:13 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:06:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:13 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:06:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:13 smithi151 ceph-mon[16639]: pgmap v570: 37 pgs: 37 active+clean; 776 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-02T10:06:14.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:14.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:13 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:06:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:13 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:06:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:13 smithi187 ceph-mon[20072]: pgmap v570: 37 pgs: 37 active+clean; 776 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-02T10:06:14.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:14.341 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:14.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:14.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:13 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:06:14.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:13 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:06:14.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:13 smithi162 ceph-mon[20279]: pgmap v570: 37 pgs: 37 active+clean; 776 MiB data, 3.4 GiB used, 980 GiB / 983 GiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-02T10:06:14.408 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":569,"stamp":"2024-09-02T10:06:11.131768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":549388832,"num_objects":8376,"num_object_clones":0,"num_object_copies":25128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8374,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10306,"ondisk_log_size":10306,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":2649640,"kb_used_data":1481964,"kb_used_omap":17,"kb_used_meta":1167470,"kb_avail":1028547032,"statfs":{"total":1055945392128,"available":1053232160768,"internally_reserved":0,"allocated":1517531136,"data_stored":1510578808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1195490171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":548798464,"num_objects":8374,"num_object_clones":0,"num_object_copies":25122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8374,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001514"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":237,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:01.163588+0000","last_change":"2024-09-02T10:06:01.163588+0000","last_active":"2024-09-02T10:06:01.163588+0000","last_peered":"2024-09-02T10:06:01.163588+0000","last_clean":"2024-09-02T10:06:01.163588+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:06:01.163588+0000","last_undegraded":"2024-09-02T10:06:01.163588+0000","last_fullsized":"2024-09-02T10:06:01.163588+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.163525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.163525+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:12.385214+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":315,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.840378+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:06:00.840378+0000","last_peered":"2024-09-02T10:06:00.840378+0000","last_clean":"2024-09-02T10:06:00.840378+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:06:00.840378+0000","last_undegraded":"2024-09-02T10:06:00.840378+0000","last_fullsized":"2024-09-02T10:06:00.840378+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.233390+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":259,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:03.254398+0000","last_change":"2024-09-02T10:06:03.254398+0000","last_active":"2024-09-02T10:06:03.254398+0000","last_peered":"2024-09-02T10:06:03.254398+0000","last_clean":"2024-09-02T10:06:03.254398+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:06:03.254398+0000","last_undegraded":"2024-09-02T10:06:03.254398+0000","last_fullsized":"2024-09-02T10:06:03.254398+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.254327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.254327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:16.949688+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":239,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.988406+0000","last_change":"2024-09-02T10:05:38.438762+0000","last_active":"2024-09-02T10:06:00.988406+0000","last_peered":"2024-09-02T10:06:00.988406+0000","last_clean":"2024-09-02T10:06:00.988406+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:06:00.988406+0000","last_undegraded":"2024-09-02T10:06:00.988406+0000","last_fullsized":"2024-09-02T10:06:00.988406+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.438672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.438672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:03.647240+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":252,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:01.152342+0000","last_change":"2024-09-02T10:06:01.152342+0000","last_active":"2024-09-02T10:06:01.152342+0000","last_peered":"2024-09-02T10:06:01.152342+0000","last_clean":"2024-09-02T10:06:01.152342+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:06:01.152342+0000","last_undegraded":"2024-09-02T10:06:01.152342+0000","last_fullsized":"2024-09-02T10:06:01.152342+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.152278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.152278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:01.602916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"231'426","reported_seq":446,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:06.890867+0000","last_change":"2024-09-02T10:05:56.584135+0000","last_active":"2024-09-02T10:06:06.890867+0000","last_peered":"2024-09-02T10:06:06.890867+0000","last_clean":"2024-09-02T10:06:06.890867+0000","last_became_active":"2024-09-02T10:05:56.583663+0000","last_became_peered":"2024-09-02T10:05:56.583663+0000","last_unstale":"2024-09-02T10:06:06.890867+0000","last_undegraded":"2024-09-02T10:06:06.890867+0000","last_fullsized":"2024-09-02T10:06:06.890867+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:15.291683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27918336,"num_objects":426,"num_object_clones":0,"num_object_copies":1278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":426,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":426,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":324,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.989035+0000","last_change":"2024-09-02T10:05:38.077751+0000","last_active":"2024-09-02T10:06:00.989035+0000","last_peered":"2024-09-02T10:06:00.989035+0000","last_clean":"2024-09-02T10:06:00.989035+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:06:00.989035+0000","last_undegraded":"2024-09-02T10:06:00.989035+0000","last_fullsized":"2024-09-02T10:06:00.989035+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.077684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.077684+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:57.694686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"231'553","reported_seq":571,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:08.313300+0000","last_change":"2024-09-02T10:05:56.583572+0000","last_active":"2024-09-02T10:06:08.313300+0000","last_peered":"2024-09-02T10:06:08.313300+0000","last_clean":"2024-09-02T10:06:08.313300+0000","last_became_active":"2024-09-02T10:05:56.583278+0000","last_became_peered":"2024-09-02T10:05:56.583278+0000","last_unstale":"2024-09-02T10:06:08.313300+0000","last_undegraded":"2024-09-02T10:06:08.313300+0000","last_fullsized":"2024-09-02T10:06:08.313300+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:11.499666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36241408,"num_objects":553,"num_object_clones":0,"num_object_copies":1659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":553,"num_write_kb":35392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":453,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-02T10:06:09.884841+0000","last_change":"2024-09-02T10:06:08.953931+0000","last_active":"2024-09-02T10:06:09.884841+0000","last_peered":"2024-09-02T10:06:09.884841+0000","last_clean":"2024-09-02T10:06:09.884841+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:06:09.884841+0000","last_undegraded":"2024-09-02T10:06:09.884841+0000","last_fullsized":"2024-09-02T10:06:09.884841+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:08.953856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:08.953856+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:10.024416+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"231'393","reported_seq":411,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:06.888933+0000","last_change":"2024-09-02T10:05:56.583542+0000","last_active":"2024-09-02T10:06:06.888933+0000","last_peered":"2024-09-02T10:06:06.888933+0000","last_clean":"2024-09-02T10:06:06.888933+0000","last_became_active":"2024-09-02T10:05:56.583249+0000","last_became_peered":"2024-09-02T10:05:56.583249+0000","last_unstale":"2024-09-02T10:06:06.888933+0000","last_undegraded":"2024-09-02T10:06:06.888933+0000","last_fullsized":"2024-09-02T10:06:06.888933+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:01.823622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25755648,"num_objects":393,"num_object_clones":0,"num_object_copies":1179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":393,"num_write_kb":25152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":82,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.988986+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:06:00.988986+0000","last_peered":"2024-09-02T10:06:00.988986+0000","last_clean":"2024-09-02T10:06:00.988986+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:06:00.988986+0000","last_undegraded":"2024-09-02T10:06:00.988986+0000","last_fullsized":"2024-09-02T10:06:00.988986+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"232'650","reported_seq":672,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-02T10:06:09.165543+0000","last_change":"2024-09-02T10:05:56.584669+0000","last_active":"2024-09-02T10:06:09.165543+0000","last_peered":"2024-09-02T10:06:09.165543+0000","last_clean":"2024-09-02T10:06:09.165543+0000","last_became_active":"2024-09-02T10:05:56.584313+0000","last_became_peered":"2024-09-02T10:05:56.584313+0000","last_unstale":"2024-09-02T10:06:09.165543+0000","last_undegraded":"2024-09-02T10:06:09.165543+0000","last_fullsized":"2024-09-02T10:06:09.165543+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:08.498885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42598400,"num_objects":650,"num_object_clones":0,"num_object_copies":1950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":650,"num_write_kb":41600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":237,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:02.338906+0000","last_change":"2024-09-02T10:06:02.338906+0000","last_active":"2024-09-02T10:06:02.338906+0000","last_peered":"2024-09-02T10:06:02.338906+0000","last_clean":"2024-09-02T10:06:02.338906+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:06:02.338906+0000","last_undegraded":"2024-09-02T10:06:02.338906+0000","last_fullsized":"2024-09-02T10:06:02.338906+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:02.338299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:02.338299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:23.238276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"231'402","reported_seq":421,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:06.499481+0000","last_change":"2024-09-02T10:05:56.579293+0000","last_active":"2024-09-02T10:06:06.499481+0000","last_peered":"2024-09-02T10:06:06.499481+0000","last_clean":"2024-09-02T10:06:06.499481+0000","last_became_active":"2024-09-02T10:05:56.578973+0000","last_became_peered":"2024-09-02T10:05:56.578973+0000","last_unstale":"2024-09-02T10:06:06.499481+0000","last_undegraded":"2024-09-02T10:06:06.499481+0000","last_fullsized":"2024-09-02T10:06:06.499481+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:17.084065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26345472,"num_objects":402,"num_object_clones":0,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":402,"num_write_kb":25728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":394,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.988274+0000","last_change":"2024-09-02T10:05:42.420814+0000","last_active":"2024-09-02T10:06:00.988274+0000","last_peered":"2024-09-02T10:06:00.988274+0000","last_clean":"2024-09-02T10:06:00.988274+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:06:00.988274+0000","last_undegraded":"2024-09-02T10:06:00.988274+0000","last_fullsized":"2024-09-02T10:06:00.988274+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:42.420722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:42.420722+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:00.199426+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"231'539","reported_seq":557,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:08.165973+0000","last_change":"2024-09-02T10:05:56.585087+0000","last_active":"2024-09-02T10:06:08.165973+0000","last_peered":"2024-09-02T10:06:08.165973+0000","last_clean":"2024-09-02T10:06:08.165973+0000","last_became_active":"2024-09-02T10:05:56.584888+0000","last_became_peered":"2024-09-02T10:05:56.584888+0000","last_unstale":"2024-09-02T10:06:08.165973+0000","last_undegraded":"2024-09-02T10:06:08.165973+0000","last_fullsized":"2024-09-02T10:06:08.165973+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.136803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35323904,"num_objects":539,"num_object_clones":0,"num_object_copies":1617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":539,"num_write_kb":34496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":231,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:02.167573+0000","last_change":"2024-09-02T10:06:02.167573+0000","last_active":"2024-09-02T10:06:02.167573+0000","last_peered":"2024-09-02T10:06:02.167573+0000","last_clean":"2024-09-02T10:06:02.167573+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:06:02.167573+0000","last_undegraded":"2024-09-02T10:06:02.167573+0000","last_fullsized":"2024-09-02T10:06:02.167573+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:02.167523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:02.167523+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.321956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"231'544","reported_seq":564,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:07.891835+0000","last_change":"2024-09-02T10:05:56.581266+0000","last_active":"2024-09-02T10:06:07.891835+0000","last_peered":"2024-09-02T10:06:07.891835+0000","last_clean":"2024-09-02T10:06:07.891835+0000","last_became_active":"2024-09-02T10:05:56.580910+0000","last_became_peered":"2024-09-02T10:05:56.580910+0000","last_unstale":"2024-09-02T10:06:07.891835+0000","last_undegraded":"2024-09-02T10:06:07.891835+0000","last_fullsized":"2024-09-02T10:06:07.891835+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":544,"log_dups_size":0,"ondisk_log_size":544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:05.300847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35651584,"num_objects":544,"num_object_clones":0,"num_object_copies":1632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":544,"num_write_kb":34816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":82,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.989047+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:06:00.989047+0000","last_peered":"2024-09-02T10:06:00.989047+0000","last_clean":"2024-09-02T10:06:00.989047+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:06:00.989047+0000","last_undegraded":"2024-09-02T10:06:00.989047+0000","last_fullsized":"2024-09-02T10:06:00.989047+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:16.611307+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"231'413","reported_seq":432,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:06.742174+0000","last_change":"2024-09-02T10:05:56.690497+0000","last_active":"2024-09-02T10:06:06.742174+0000","last_peered":"2024-09-02T10:06:06.742174+0000","last_clean":"2024-09-02T10:06:06.742174+0000","last_became_active":"2024-09-02T10:05:56.690229+0000","last_became_peered":"2024-09-02T10:05:56.690229+0000","last_unstale":"2024-09-02T10:06:06.742174+0000","last_undegraded":"2024-09-02T10:06:06.742174+0000","last_fullsized":"2024-09-02T10:06:06.742174+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:05.936975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27066368,"num_objects":413,"num_object_clones":0,"num_object_copies":1239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":413,"num_write_kb":26432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":354,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:05.307800+0000","last_change":"2024-09-02T10:06:05.307800+0000","last_active":"2024-09-02T10:06:05.307800+0000","last_peered":"2024-09-02T10:06:05.307800+0000","last_clean":"2024-09-02T10:06:05.307800+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:06:05.307800+0000","last_undegraded":"2024-09-02T10:06:05.307800+0000","last_fullsized":"2024-09-02T10:06:05.307800+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:05.307738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:05.307738+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:30.265023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"232'658","reported_seq":680,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-02T10:06:09.184589+0000","last_change":"2024-09-02T10:05:56.582278+0000","last_active":"2024-09-02T10:06:09.184589+0000","last_peered":"2024-09-02T10:06:09.184589+0000","last_clean":"2024-09-02T10:06:09.184589+0000","last_became_active":"2024-09-02T10:05:56.581969+0000","last_became_peered":"2024-09-02T10:05:56.581969+0000","last_unstale":"2024-09-02T10:06:09.184589+0000","last_undegraded":"2024-09-02T10:06:09.184589+0000","last_fullsized":"2024-09-02T10:06:09.184589+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":658,"log_dups_size":0,"ondisk_log_size":658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.236882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43122688,"num_objects":658,"num_object_clones":0,"num_object_copies":1974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":658,"num_write_kb":42112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":82,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:03.857986+0000","last_change":"2024-09-02T10:06:03.857986+0000","last_active":"2024-09-02T10:06:03.857986+0000","last_peered":"2024-09-02T10:06:03.857986+0000","last_clean":"2024-09-02T10:06:03.857986+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:06:03.857986+0000","last_undegraded":"2024-09-02T10:06:03.857986+0000","last_fullsized":"2024-09-02T10:06:03.857986+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.857913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.857913+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:27.151711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"231'392","reported_seq":412,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:06.898420+0000","last_change":"2024-09-02T10:05:56.876782+0000","last_active":"2024-09-02T10:06:06.898420+0000","last_peered":"2024-09-02T10:06:06.898420+0000","last_clean":"2024-09-02T10:06:06.898420+0000","last_became_active":"2024-09-02T10:05:56.876397+0000","last_became_peered":"2024-09-02T10:05:56.876397+0000","last_unstale":"2024-09-02T10:06:06.898420+0000","last_undegraded":"2024-09-02T10:06:06.898420+0000","last_fullsized":"2024-09-02T10:06:06.898420+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:10.344558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":392,"num_object_clones":0,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":392,"num_write_kb":25088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":100,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-02T10:06:09.884768+0000","last_change":"2024-09-02T10:06:01.039876+0000","last_active":"2024-09-02T10:06:09.884768+0000","last_peered":"2024-09-02T10:06:09.884768+0000","last_clean":"2024-09-02T10:06:09.884768+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:06:09.884768+0000","last_undegraded":"2024-09-02T10:06:09.884768+0000","last_fullsized":"2024-09-02T10:06:09.884768+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.039815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.039815+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:24.958614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"231'389","reported_seq":407,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:06.483527+0000","last_change":"2024-09-02T10:05:56.581467+0000","last_active":"2024-09-02T10:06:06.483527+0000","last_peered":"2024-09-02T10:06:06.483527+0000","last_clean":"2024-09-02T10:06:06.483527+0000","last_became_active":"2024-09-02T10:05:56.581229+0000","last_became_peered":"2024-09-02T10:05:56.581229+0000","last_unstale":"2024-09-02T10:06:06.483527+0000","last_undegraded":"2024-09-02T10:06:06.483527+0000","last_fullsized":"2024-09-02T10:06:06.483527+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:03.921090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":389,"num_write_kb":24896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":303,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:00.840512+0000","last_change":"2024-09-02T10:05:06.815241+0000","last_active":"2024-09-02T10:06:00.840512+0000","last_peered":"2024-09-02T10:06:00.840512+0000","last_clean":"2024-09-02T10:06:00.840512+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:06:00.840512+0000","last_undegraded":"2024-09-02T10:06:00.840512+0000","last_fullsized":"2024-09-02T10:06:00.840512+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:06.815191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:06.815191+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:14.504648+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"231'554","reported_seq":575,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:08.181663+0000","last_change":"2024-09-02T10:05:56.876935+0000","last_active":"2024-09-02T10:06:08.181663+0000","last_peered":"2024-09-02T10:06:08.181663+0000","last_clean":"2024-09-02T10:06:08.181663+0000","last_became_active":"2024-09-02T10:05:56.876573+0000","last_became_peered":"2024-09-02T10:05:56.876573+0000","last_unstale":"2024-09-02T10:06:08.181663+0000","last_undegraded":"2024-09-02T10:06:08.181663+0000","last_fullsized":"2024-09-02T10:06:08.181663+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:24.250156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36306944,"num_objects":554,"num_object_clones":0,"num_object_copies":1662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":554,"num_write_kb":35456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":111,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-02T10:06:09.884763+0000","last_change":"2024-09-02T10:06:04.062386+0000","last_active":"2024-09-02T10:06:09.884763+0000","last_peered":"2024-09-02T10:06:09.884763+0000","last_clean":"2024-09-02T10:06:09.884763+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:06:09.884763+0000","last_undegraded":"2024-09-02T10:06:09.884763+0000","last_fullsized":"2024-09-02T10:06:09.884763+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:04.062325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:04.062325+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:34.030289+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"233'818","reported_seq":843,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-02T10:06:10.847307+0000","last_change":"2024-09-02T10:05:56.877006+0000","last_active":"2024-09-02T10:06:10.847307+0000","last_peered":"2024-09-02T10:06:10.847307+0000","last_clean":"2024-09-02T10:06:10.847307+0000","last_became_active":"2024-09-02T10:05:56.876540+0000","last_became_peered":"2024-09-02T10:05:56.876540+0000","last_unstale":"2024-09-02T10:06:10.847307+0000","last_undegraded":"2024-09-02T10:06:10.847307+0000","last_fullsized":"2024-09-02T10:06:10.847307+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:02.154423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53608448,"num_objects":818,"num_object_clones":0,"num_object_copies":2454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":818,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":818,"num_write_kb":52352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":339,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:04.265827+0000","last_change":"2024-09-02T10:06:04.265827+0000","last_active":"2024-09-02T10:06:04.265827+0000","last_peered":"2024-09-02T10:06:04.265827+0000","last_clean":"2024-09-02T10:06:04.265827+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:06:04.265827+0000","last_undegraded":"2024-09-02T10:06:04.265827+0000","last_fullsized":"2024-09-02T10:06:04.265827+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:04.265776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:04.265776+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.085143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"231'428","reported_seq":446,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:06.815524+0000","last_change":"2024-09-02T10:05:56.586356+0000","last_active":"2024-09-02T10:06:06.815524+0000","last_peered":"2024-09-02T10:06:06.815524+0000","last_clean":"2024-09-02T10:06:06.815524+0000","last_became_active":"2024-09-02T10:05:56.585883+0000","last_became_peered":"2024-09-02T10:05:56.585883+0000","last_unstale":"2024-09-02T10:06:06.815524+0000","last_undegraded":"2024-09-02T10:06:06.815524+0000","last_fullsized":"2024-09-02T10:06:06.815524+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:06.966563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28049408,"num_objects":428,"num_object_clones":0,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":428,"num_write_kb":27392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":240,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-02T10:06:08.880604+0000","last_change":"2024-09-02T10:04:57.391785+0000","last_active":"2024-09-02T10:06:08.880604+0000","last_peered":"2024-09-02T10:06:08.880604+0000","last_clean":"2024-09-02T10:06:08.880604+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:06:08.880604+0000","last_undegraded":"2024-09-02T10:06:08.880604+0000","last_fullsized":"2024-09-02T10:06:08.880604+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.391723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.391723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:12.774478+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"233'792","reported_seq":816,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-02T10:06:10.841121+0000","last_change":"2024-09-02T10:05:56.584811+0000","last_active":"2024-09-02T10:06:10.841121+0000","last_peered":"2024-09-02T10:06:10.841121+0000","last_clean":"2024-09-02T10:06:10.841121+0000","last_became_active":"2024-09-02T10:05:56.584450+0000","last_became_peered":"2024-09-02T10:05:56.584450+0000","last_unstale":"2024-09-02T10:06:10.841121+0000","last_undegraded":"2024-09-02T10:06:10.841121+0000","last_fullsized":"2024-09-02T10:06:10.841121+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":792,"log_dups_size":0,"ondisk_log_size":792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:04.195054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51904512,"num_objects":792,"num_object_clones":0,"num_object_copies":2376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":50688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":111,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-02T10:06:09.884906+0000","last_change":"2024-09-02T10:06:03.083413+0000","last_active":"2024-09-02T10:06:09.884906+0000","last_peered":"2024-09-02T10:06:09.884906+0000","last_clean":"2024-09-02T10:06:09.884906+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:06:09.884906+0000","last_undegraded":"2024-09-02T10:06:09.884906+0000","last_fullsized":"2024-09-02T10:06:09.884906+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.083338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.083338+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:20.993372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"231'423","reported_seq":442,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:06.884559+0000","last_change":"2024-09-02T10:05:56.586192+0000","last_active":"2024-09-02T10:06:06.884559+0000","last_peered":"2024-09-02T10:06:06.884559+0000","last_clean":"2024-09-02T10:06:06.884559+0000","last_became_active":"2024-09-02T10:05:56.585870+0000","last_became_peered":"2024-09-02T10:05:56.585870+0000","last_unstale":"2024-09-02T10:06:06.884559+0000","last_undegraded":"2024-09-02T10:06:06.884559+0000","last_fullsized":"2024-09-02T10:06:06.884559+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:59.656901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":423,"num_object_clones":0,"num_object_copies":1269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":423,"num_write_kb":27072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":119,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-09-02T10:06:01.391977+0000","last_change":"2024-09-02T10:06:01.391977+0000","last_active":"2024-09-02T10:06:01.391977+0000","last_peered":"2024-09-02T10:06:01.391977+0000","last_clean":"2024-09-02T10:06:01.391977+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:06:01.391977+0000","last_undegraded":"2024-09-02T10:06:01.391977+0000","last_fullsized":"2024-09-02T10:06:01.391977+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.391925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.391925+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:04.473676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":548798464,"num_objects":8374,"num_object_clones":0,"num_object_copies":25122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8374,"num_write_kb":535936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1644822528,"data_stored":1644822528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8374,"ondisk_log_size":8374,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383752,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321952,"kb_used_data":186640,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93423200,"statfs":{"total":95995035648,"available":95665356800,"internally_reserved":0,"allocated":191119360,"data_stored":190489536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":138541508},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841618,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":221984,"kb_used_data":100560,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93523168,"statfs":{"total":95995035648,"available":95767724032,"internally_reserved":0,"allocated":102973440,"data_stored":102343624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":124320192},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070552,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163888,"kb_used_data":82716,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93581264,"statfs":{"total":95995035648,"available":95827214336,"internally_reserved":0,"allocated":84701184,"data_stored":84066822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299483,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312960,"kb_used_data":183208,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93432192,"statfs":{"total":95995035648,"available":95674564608,"internally_reserved":0,"allocated":187604992,"data_stored":186966034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":132839882},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528415,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":187424,"kb_used_data":120144,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93557728,"statfs":{"total":95995035648,"available":95803113472,"internally_reserved":0,"allocated":123027456,"data_stored":122397640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724644,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169256,"kb_used_data":97816,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93575896,"statfs":{"total":95995035648,"available":95821717504,"internally_reserved":0,"allocated":100163584,"data_stored":99530216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953578,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":349792,"kb_used_data":270032,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93395360,"statfs":{"total":95995035648,"available":95636848640,"internally_reserved":0,"allocated":276512768,"data_stored":275882952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81656261},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":314920,"kb_used_data":153240,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93430232,"statfs":{"total":95995035648,"available":95672557568,"internally_reserved":0,"allocated":156917760,"data_stored":156284392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":165542344},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411443,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":164200,"kb_used_data":28888,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93580952,"statfs":{"total":95995035648,"available":95826894848,"internally_reserved":0,"allocated":29581312,"data_stored":28947944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":233312,"kb_used_data":134096,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93511840,"statfs":{"total":95995035648,"available":95756124160,"internally_reserved":0,"allocated":137314304,"data_stored":136684488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":209952,"kb_used_data":124624,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93535200,"statfs":{"total":95995035648,"available":95780044800,"internally_reserved":0,"allocated":127614976,"data_stored":126985160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":130416640,"data_stored":130416640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139001856,"data_stored":139001856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31195136,"data_stored":31195136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":179765248,"data_stored":179765248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":293797888,"data_stored":293797888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":113180672,"data_stored":113180672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":138084352,"data_stored":138084352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":209321984,"data_stored":209321984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":84869120,"data_stored":84869120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":105971712,"data_stored":105971712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":219217920,"data_stored":219217920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:06:14.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:14.471 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:14.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:14.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:14.598 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:14.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:14.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:14.722 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:14.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:14.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:14.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:14.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:14.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:14.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:14.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:14.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:15.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 12 16 18121 18105 94.2897 86.3125 0.00197528 0.0105982 2024-09-02T10:06:15.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:15.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:15.095 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:15.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:14 smithi151 ceph-mon[16639]: osdmap e234: 12 total, 12 up, 11 in 2024-09-02T10:06:15.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:15.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:15.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:14 smithi187 ceph-mon[20072]: osdmap e234: 12 total, 12 up, 11 in 2024-09-02T10:06:15.319 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:15.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:14 smithi162 ceph-mon[20279]: osdmap e234: 12 total, 12 up, 11 in 2024-09-02T10:06:15.347 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:15.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:15.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:15.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:15.472 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:15.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:15.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:15.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:15.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:15.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:15.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:15.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:15.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:15.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:15.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:15.950 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:15.966 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:06:15.966 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:06:15.966 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:06:15.966 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:06:15.966 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:06:15.966 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:06:15.966 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:06:15.967 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:06:15.967 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:06:15.967 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:06:15.967 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:06:15.967 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:06:15.967 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:06:15.967 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:06:15.967 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:06:15.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:06:15.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:06:15.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:06:15.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:06:15.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:06:15.968 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:06:15.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:06:15.974 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:15.975 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:16.003 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:06:16.004 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:06:16.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 13 16 19335 19319 92.8727 75.875 0.0113658 0.0107558 2024-09-02T10:06:16.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:16.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:16.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:16.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:15 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:06:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:15 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:06:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:15 smithi151 ceph-mon[16639]: pgmap v572: 37 pgs: 37 active+clean; 910 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-02T10:06:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4121447438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:16.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:16.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:16.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:16.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:15 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:06:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:15 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:06:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:15 smithi187 ceph-mon[20072]: pgmap v572: 37 pgs: 37 active+clean; 910 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-02T10:06:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4121447438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:16.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:16.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:15 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:06:16.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:15 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:06:16.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:15 smithi162 ceph-mon[20279]: pgmap v572: 37 pgs: 37 active+clean; 910 MiB data, 3.7 GiB used, 980 GiB / 983 GiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-02T10:06:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4121447438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:16.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:16.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:16.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:16.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:16.483 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:16.484 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:16.584 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:16.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:16.612 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:16.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:16.741 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:16.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:16.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:16.867 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:16.868 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:16.909 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:06:16.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:16 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:06:16.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:16 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:06:16.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3032917269' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:06:16.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:16.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:16.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:16.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:17.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 14 16 20979 20963 93.5777 102.75 0.0156161 0.0106767 2024-09-02T10:06:17.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:17.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:17.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:17.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:17.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:17.253 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:17.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:16 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:06:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:16 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:06:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3032917269' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:06:17.341 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:17.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:16 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:06:17.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:16 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:06:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3032917269' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:06:17.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:17.380 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:17.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:17.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:06:17.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:17.510 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:17.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:17.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:17.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:17.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:17.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:17.737 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:17.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:17.762 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:17.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:17.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:17.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:17.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:18.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:18.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:18.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 15 16 22497 22481 93.6637 94.875 0.00228244 0.010662 2024-09-02T10:06:18.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:18.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:18.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:18.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:17 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3032917269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:06:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:17 smithi151 ceph-mon[16639]: osdmap e235: 12 total, 12 up, 11 in 2024-09-02T10:06:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:17 smithi151 ceph-mon[16639]: pgmap v574: 37 pgs: 37 active+clean; 1.1 GiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-02T10:06:18.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:17 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3032917269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:06:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:17 smithi187 ceph-mon[20072]: osdmap e235: 12 total, 12 up, 11 in 2024-09-02T10:06:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:17 smithi187 ceph-mon[20072]: pgmap v574: 37 pgs: 37 active+clean; 1.1 GiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-02T10:06:18.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:18.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:18.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:18.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:17 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3032917269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:06:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:17 smithi162 ceph-mon[20279]: osdmap e235: 12 total, 12 up, 11 in 2024-09-02T10:06:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:17 smithi162 ceph-mon[20279]: pgmap v574: 37 pgs: 37 active+clean; 1.1 GiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-02T10:06:18.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:18.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:18.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:18.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:18.514 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:18.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:18.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:18.636 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:18.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:18.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:18.760 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:18.761 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:18.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:18.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:18.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:18.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:19.008 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:19.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:19.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 16 16 23411 23395 91.3798 57.125 0.00366014 0.0109133 2024-09-02T10:06:19.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:19.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:06:19.134 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:19.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:19.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:19.244 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:19.244 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:19.259 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:19.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:19.361 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:19.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:19.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:19.487 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:19.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:19.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:19.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:19.636 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:19.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:19.637 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:19.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:19.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:19.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:19.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:19.885 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:19.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:19.987 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:20.013 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:20.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:20.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 17 16 24592 24576 90.346 73.8125 0.00475463 0.01106 2024-09-02T10:06:20.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:06:20.115 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:20.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:20.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:20.141 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:20.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:20.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:20.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:20.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:20.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:20.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:20.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:20 smithi151 ceph-mon[16639]: pgmap v575: 37 pgs: 37 active+clean; 1.3 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-09-02T10:06:20.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:20 smithi187 ceph-mon[20072]: pgmap v575: 37 pgs: 37 active+clean; 1.3 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-09-02T10:06:20.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:20.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:20.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:20.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:20 smithi162 ceph-mon[20279]: pgmap v575: 37 pgs: 37 active+clean; 1.3 GiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 95 MiB/s wr, 1.51k op/s 2024-09-02T10:06:20.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:20.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:20.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:20.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:20.732 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:20.738 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:20.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:20.774 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:20.778 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:20.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:20.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:20.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:20.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:21.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:21.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 18 16 26302 26286 91.2639 106.875 0.0044948 0.0109502 2024-09-02T10:06:21.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:21.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:21.138 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:21.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:21.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:21.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:21.294 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:21.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:21.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:21.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:21.421 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:21.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:21.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:21.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:21.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:21.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:21.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:21.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:21.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:21.813 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:21.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:21.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:21.941 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:21.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:21.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:22.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 19 16 27856 27840 91.572 97.125 0.0124452 0.0109145 2024-09-02T10:06:22.043 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:22.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:22.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:22.068 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:22.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:22.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:22.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:22.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:22.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:22.393 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:06:22.393 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:06:22.395 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:22.395 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:22.413 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:22.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:22 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:06:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:22 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:06:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:22 smithi187 ceph-mon[20072]: pgmap v576: 37 pgs: 37 active+clean; 1.4 GiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:06:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:22 smithi187 ceph-mon[20072]: osdmap e236: 12 total, 12 up, 11 in 2024-09-02T10:06:22.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:22.546 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:22.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:22.547 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:22.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:22 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:06:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:22 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:06:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:22 smithi162 ceph-mon[20279]: pgmap v576: 37 pgs: 37 active+clean; 1.4 GiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:06:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:22 smithi162 ceph-mon[20279]: osdmap e236: 12 total, 12 up, 11 in 2024-09-02T10:06:22.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:22.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:22.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:22.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:22 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:06:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:22 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:06:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:22 smithi151 ceph-mon[16639]: pgmap v576: 37 pgs: 37 active+clean; 1.4 GiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:06:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:22 smithi151 ceph-mon[16639]: osdmap e236: 12 total, 12 up, 11 in 2024-09-02T10:06:22.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:06:22.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:22.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:22.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:22.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:22.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:22.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:23.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:06:23.024929+0000 min lat: 0.0010322 max lat: 0.259301 avg lat: 0.0109981 2024-09-02T10:06:23.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:06:23.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 20 16 29038 29022 90.6869 73.875 0.00226343 0.0109981 2024-09-02T10:06:23.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:23.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:23.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:23.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:23.168 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":577,"stamp":"2024-09-02T10:06:21.192307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1455227424,"num_objects":22198,"num_object_clones":0,"num_object_copies":66594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22196,"num_write_kb":1420544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24128,"ondisk_log_size":24128,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":5328232,"kb_used_data":4125864,"kb_used_omap":17,"kb_used_meta":1202158,"kb_avail":1025868440,"statfs":{"total":1055945392128,"available":1050489282560,"internally_reserved":0,"allocated":4224884736,"data_stored":4217763356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1231010683},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,2,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":558825472,"num_objects":8527,"num_object_clones":0,"num_object_copies":25581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8527,"num_write_kb":545728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.295221"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":245,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.912206+0000","last_change":"2024-09-02T10:06:01.163588+0000","last_active":"2024-09-02T10:06:16.912206+0000","last_peered":"2024-09-02T10:06:16.912206+0000","last_clean":"2024-09-02T10:06:16.912206+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:06:16.912206+0000","last_undegraded":"2024-09-02T10:06:16.912206+0000","last_fullsized":"2024-09-02T10:06:16.912206+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.163525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.163525+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:12.385214+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":323,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.914645+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:06:16.914645+0000","last_peered":"2024-09-02T10:06:16.914645+0000","last_clean":"2024-09-02T10:06:16.914645+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:06:16.914645+0000","last_undegraded":"2024-09-02T10:06:16.914645+0000","last_fullsized":"2024-09-02T10:06:16.914645+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":265,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-02T10:06:13.902546+0000","last_change":"2024-09-02T10:06:03.254398+0000","last_active":"2024-09-02T10:06:13.902546+0000","last_peered":"2024-09-02T10:06:13.902546+0000","last_clean":"2024-09-02T10:06:13.902546+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:06:13.902546+0000","last_undegraded":"2024-09-02T10:06:13.902546+0000","last_fullsized":"2024-09-02T10:06:13.902546+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.254327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.254327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:16.949688+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":245,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-02T10:06:13.903877+0000","last_change":"2024-09-02T10:05:38.438762+0000","last_active":"2024-09-02T10:06:13.903877+0000","last_peered":"2024-09-02T10:06:13.903877+0000","last_clean":"2024-09-02T10:06:13.903877+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:06:13.903877+0000","last_undegraded":"2024-09-02T10:06:13.903877+0000","last_fullsized":"2024-09-02T10:06:13.903877+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.438672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.438672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:03.647240+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":258,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-02T10:06:13.902508+0000","last_change":"2024-09-02T10:06:01.152342+0000","last_active":"2024-09-02T10:06:13.902508+0000","last_peered":"2024-09-02T10:06:13.902508+0000","last_clean":"2024-09-02T10:06:13.902508+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:06:13.902508+0000","last_undegraded":"2024-09-02T10:06:13.902508+0000","last_fullsized":"2024-09-02T10:06:13.902508+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.152278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.152278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:01.602916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"235'1339","reported_seq":1367,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.989406+0000","last_change":"2024-09-02T10:05:56.584135+0000","last_active":"2024-09-02T10:06:16.989406+0000","last_peered":"2024-09-02T10:06:16.989406+0000","last_clean":"2024-09-02T10:06:16.989406+0000","last_became_active":"2024-09-02T10:05:56.583663+0000","last_became_peered":"2024-09-02T10:05:56.583663+0000","last_unstale":"2024-09-02T10:06:16.989406+0000","last_undegraded":"2024-09-02T10:06:16.989406+0000","last_fullsized":"2024-09-02T10:06:16.989406+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1339,"log_dups_size":0,"ondisk_log_size":1339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:15.291683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87752704,"num_objects":1339,"num_object_clones":0,"num_object_copies":4017,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1339,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1339,"num_write_kb":85696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":331,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-02T10:06:13.902545+0000","last_change":"2024-09-02T10:05:38.077751+0000","last_active":"2024-09-02T10:06:13.902545+0000","last_peered":"2024-09-02T10:06:13.902545+0000","last_clean":"2024-09-02T10:06:13.902545+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:06:13.902545+0000","last_undegraded":"2024-09-02T10:06:13.902545+0000","last_fullsized":"2024-09-02T10:06:13.902545+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.077684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.077684+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:57.694686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"235'1436","reported_seq":1464,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:18.308114+0000","last_change":"2024-09-02T10:05:56.583572+0000","last_active":"2024-09-02T10:06:18.308114+0000","last_peered":"2024-09-02T10:06:18.308114+0000","last_clean":"2024-09-02T10:06:18.308114+0000","last_became_active":"2024-09-02T10:05:56.583278+0000","last_became_peered":"2024-09-02T10:05:56.583278+0000","last_unstale":"2024-09-02T10:06:18.308114+0000","last_undegraded":"2024-09-02T10:06:18.308114+0000","last_fullsized":"2024-09-02T10:06:18.308114+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1436,"log_dups_size":0,"ondisk_log_size":1436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:11.499666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94109696,"num_objects":1436,"num_object_clones":0,"num_object_copies":4308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1436,"num_write_kb":91904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":457,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.913681+0000","last_change":"2024-09-02T10:06:08.953931+0000","last_active":"2024-09-02T10:06:16.913681+0000","last_peered":"2024-09-02T10:06:16.913681+0000","last_clean":"2024-09-02T10:06:16.913681+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:06:16.913681+0000","last_undegraded":"2024-09-02T10:06:16.913681+0000","last_fullsized":"2024-09-02T10:06:16.913681+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:08.953856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:08.953856+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:10.024416+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"234'1250","reported_seq":1278,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.911451+0000","last_change":"2024-09-02T10:05:56.583542+0000","last_active":"2024-09-02T10:06:16.911451+0000","last_peered":"2024-09-02T10:06:16.911451+0000","last_clean":"2024-09-02T10:06:16.911451+0000","last_became_active":"2024-09-02T10:05:56.583249+0000","last_became_peered":"2024-09-02T10:05:56.583249+0000","last_unstale":"2024-09-02T10:06:16.911451+0000","last_undegraded":"2024-09-02T10:06:16.911451+0000","last_fullsized":"2024-09-02T10:06:16.911451+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:01.823622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1250,"num_write_kb":80000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":90,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.928034+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:06:16.928034+0000","last_peered":"2024-09-02T10:06:16.928034+0000","last_clean":"2024-09-02T10:06:16.928034+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:06:16.928034+0000","last_undegraded":"2024-09-02T10:06:16.928034+0000","last_fullsized":"2024-09-02T10:06:16.928034+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"235'1498","reported_seq":1526,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:19.222094+0000","last_change":"2024-09-02T10:05:56.584669+0000","last_active":"2024-09-02T10:06:19.222094+0000","last_peered":"2024-09-02T10:06:19.222094+0000","last_clean":"2024-09-02T10:06:19.222094+0000","last_became_active":"2024-09-02T10:05:56.584313+0000","last_became_peered":"2024-09-02T10:05:56.584313+0000","last_unstale":"2024-09-02T10:06:19.222094+0000","last_undegraded":"2024-09-02T10:06:19.222094+0000","last_fullsized":"2024-09-02T10:06:19.222094+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:08.498885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98172928,"num_objects":1498,"num_object_clones":0,"num_object_copies":4494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1498,"num_write_kb":95872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":245,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.911593+0000","last_change":"2024-09-02T10:06:02.338906+0000","last_active":"2024-09-02T10:06:16.911593+0000","last_peered":"2024-09-02T10:06:16.911593+0000","last_clean":"2024-09-02T10:06:16.911593+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:06:16.911593+0000","last_undegraded":"2024-09-02T10:06:16.911593+0000","last_fullsized":"2024-09-02T10:06:16.911593+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:02.338299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:02.338299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:23.238276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"234'1277","reported_seq":1302,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.471429+0000","last_change":"2024-09-02T10:05:56.579293+0000","last_active":"2024-09-02T10:06:16.471429+0000","last_peered":"2024-09-02T10:06:16.471429+0000","last_clean":"2024-09-02T10:06:16.471429+0000","last_became_active":"2024-09-02T10:05:56.578973+0000","last_became_peered":"2024-09-02T10:05:56.578973+0000","last_unstale":"2024-09-02T10:06:16.471429+0000","last_undegraded":"2024-09-02T10:06:16.471429+0000","last_fullsized":"2024-09-02T10:06:16.471429+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1277,"log_dups_size":0,"ondisk_log_size":1277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:17.084065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83689472,"num_objects":1277,"num_object_clones":0,"num_object_copies":3831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1277,"num_write_kb":81728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":402,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.911629+0000","last_change":"2024-09-02T10:05:42.420814+0000","last_active":"2024-09-02T10:06:16.911629+0000","last_peered":"2024-09-02T10:06:16.911629+0000","last_clean":"2024-09-02T10:06:16.911629+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:06:16.911629+0000","last_undegraded":"2024-09-02T10:06:16.911629+0000","last_fullsized":"2024-09-02T10:06:16.911629+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:42.420722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:42.420722+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:00.199426+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"235'1398","reported_seq":1426,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:18.321218+0000","last_change":"2024-09-02T10:05:56.585087+0000","last_active":"2024-09-02T10:06:18.321218+0000","last_peered":"2024-09-02T10:06:18.321218+0000","last_clean":"2024-09-02T10:06:18.321218+0000","last_became_active":"2024-09-02T10:05:56.584888+0000","last_became_peered":"2024-09-02T10:05:56.584888+0000","last_unstale":"2024-09-02T10:06:18.321218+0000","last_undegraded":"2024-09-02T10:06:18.321218+0000","last_fullsized":"2024-09-02T10:06:18.321218+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1398,"log_dups_size":0,"ondisk_log_size":1398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.136803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91619328,"num_objects":1398,"num_object_clones":0,"num_object_copies":4194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1398,"num_write_kb":89472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":239,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.912321+0000","last_change":"2024-09-02T10:06:02.167573+0000","last_active":"2024-09-02T10:06:16.912321+0000","last_peered":"2024-09-02T10:06:16.912321+0000","last_clean":"2024-09-02T10:06:16.912321+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:06:16.912321+0000","last_undegraded":"2024-09-02T10:06:16.912321+0000","last_fullsized":"2024-09-02T10:06:16.912321+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:02.167523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:02.167523+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.321956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"235'1437","reported_seq":1466,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:17.879420+0000","last_change":"2024-09-02T10:05:56.581266+0000","last_active":"2024-09-02T10:06:17.879420+0000","last_peered":"2024-09-02T10:06:17.879420+0000","last_clean":"2024-09-02T10:06:17.879420+0000","last_became_active":"2024-09-02T10:05:56.580910+0000","last_became_peered":"2024-09-02T10:05:56.580910+0000","last_unstale":"2024-09-02T10:06:17.879420+0000","last_undegraded":"2024-09-02T10:06:17.879420+0000","last_fullsized":"2024-09-02T10:06:17.879420+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1437,"log_dups_size":0,"ondisk_log_size":1437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:05.300847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94175232,"num_objects":1437,"num_object_clones":0,"num_object_copies":4311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1437,"num_write_kb":91968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":90,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.928087+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:06:16.928087+0000","last_peered":"2024-09-02T10:06:16.928087+0000","last_clean":"2024-09-02T10:06:16.928087+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:06:16.928087+0000","last_undegraded":"2024-09-02T10:06:16.928087+0000","last_fullsized":"2024-09-02T10:06:16.928087+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"234'1274","reported_seq":1299,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.795557+0000","last_change":"2024-09-02T10:05:56.690497+0000","last_active":"2024-09-02T10:06:16.795557+0000","last_peered":"2024-09-02T10:06:16.795557+0000","last_clean":"2024-09-02T10:06:16.795557+0000","last_became_active":"2024-09-02T10:05:56.690229+0000","last_became_peered":"2024-09-02T10:05:56.690229+0000","last_unstale":"2024-09-02T10:06:16.795557+0000","last_undegraded":"2024-09-02T10:06:16.795557+0000","last_fullsized":"2024-09-02T10:06:16.795557+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:05.936975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1274,"num_write_kb":81536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":363,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.913368+0000","last_change":"2024-09-02T10:06:05.307800+0000","last_active":"2024-09-02T10:06:16.913368+0000","last_peered":"2024-09-02T10:06:16.913368+0000","last_clean":"2024-09-02T10:06:16.913368+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:06:16.913368+0000","last_undegraded":"2024-09-02T10:06:16.913368+0000","last_fullsized":"2024-09-02T10:06:16.913368+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:05.307738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:05.307738+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:30.265023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"235'1506","reported_seq":1535,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:19.172791+0000","last_change":"2024-09-02T10:05:56.582278+0000","last_active":"2024-09-02T10:06:19.172791+0000","last_peered":"2024-09-02T10:06:19.172791+0000","last_clean":"2024-09-02T10:06:19.172791+0000","last_became_active":"2024-09-02T10:05:56.581969+0000","last_became_peered":"2024-09-02T10:05:56.581969+0000","last_unstale":"2024-09-02T10:06:19.172791+0000","last_undegraded":"2024-09-02T10:06:19.172791+0000","last_fullsized":"2024-09-02T10:06:19.172791+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1506,"log_dups_size":0,"ondisk_log_size":1506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.236882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98697216,"num_objects":1506,"num_object_clones":0,"num_object_copies":4518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1506,"num_write_kb":96384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":90,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.928002+0000","last_change":"2024-09-02T10:06:03.857986+0000","last_active":"2024-09-02T10:06:16.928002+0000","last_peered":"2024-09-02T10:06:16.928002+0000","last_clean":"2024-09-02T10:06:16.928002+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:06:16.928002+0000","last_undegraded":"2024-09-02T10:06:16.928002+0000","last_fullsized":"2024-09-02T10:06:16.928002+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.857913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.857913+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:27.151711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"234'1261","reported_seq":1289,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.911442+0000","last_change":"2024-09-02T10:05:56.876782+0000","last_active":"2024-09-02T10:06:16.911442+0000","last_peered":"2024-09-02T10:06:16.911442+0000","last_clean":"2024-09-02T10:06:16.911442+0000","last_became_active":"2024-09-02T10:05:56.876397+0000","last_became_peered":"2024-09-02T10:05:56.876397+0000","last_unstale":"2024-09-02T10:06:16.911442+0000","last_undegraded":"2024-09-02T10:06:16.911442+0000","last_fullsized":"2024-09-02T10:06:16.911442+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1261,"log_dups_size":0,"ondisk_log_size":1261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:10.344558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82640896,"num_objects":1261,"num_object_clones":0,"num_object_copies":3783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1261,"num_write_kb":80704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":104,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.913715+0000","last_change":"2024-09-02T10:06:01.039876+0000","last_active":"2024-09-02T10:06:16.913715+0000","last_peered":"2024-09-02T10:06:16.913715+0000","last_clean":"2024-09-02T10:06:16.913715+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:06:16.913715+0000","last_undegraded":"2024-09-02T10:06:16.913715+0000","last_fullsized":"2024-09-02T10:06:16.913715+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.039815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.039815+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:24.958614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"234'1209","reported_seq":1235,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.475628+0000","last_change":"2024-09-02T10:05:56.581467+0000","last_active":"2024-09-02T10:06:16.475628+0000","last_peered":"2024-09-02T10:06:16.475628+0000","last_clean":"2024-09-02T10:06:16.475628+0000","last_became_active":"2024-09-02T10:05:56.581229+0000","last_became_peered":"2024-09-02T10:05:56.581229+0000","last_unstale":"2024-09-02T10:06:16.475628+0000","last_undegraded":"2024-09-02T10:06:16.475628+0000","last_fullsized":"2024-09-02T10:06:16.475628+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1209,"log_dups_size":0,"ondisk_log_size":1209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:03.921090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79233024,"num_objects":1209,"num_object_clones":0,"num_object_copies":3627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1209,"num_write_kb":77376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":319,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.914657+0000","last_change":"2024-09-02T10:06:15.608866+0000","last_active":"2024-09-02T10:06:16.914657+0000","last_peered":"2024-09-02T10:06:16.914657+0000","last_clean":"2024-09-02T10:06:16.914657+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:06:16.914657+0000","last_undegraded":"2024-09-02T10:06:16.914657+0000","last_fullsized":"2024-09-02T10:06:16.914657+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:15.608802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:15.608802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:17.634785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"235'1382","reported_seq":1411,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:18.299690+0000","last_change":"2024-09-02T10:05:56.876935+0000","last_active":"2024-09-02T10:06:18.299690+0000","last_peered":"2024-09-02T10:06:18.299690+0000","last_clean":"2024-09-02T10:06:18.299690+0000","last_became_active":"2024-09-02T10:05:56.876573+0000","last_became_peered":"2024-09-02T10:05:56.876573+0000","last_unstale":"2024-09-02T10:06:18.299690+0000","last_undegraded":"2024-09-02T10:06:18.299690+0000","last_fullsized":"2024-09-02T10:06:18.299690+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1382,"log_dups_size":0,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:24.250156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90570752,"num_objects":1382,"num_object_clones":0,"num_object_copies":4146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1382,"num_write_kb":88448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":115,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.913599+0000","last_change":"2024-09-02T10:06:04.062386+0000","last_active":"2024-09-02T10:06:16.913599+0000","last_peered":"2024-09-02T10:06:16.913599+0000","last_clean":"2024-09-02T10:06:16.913599+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:06:16.913599+0000","last_undegraded":"2024-09-02T10:06:16.913599+0000","last_fullsized":"2024-09-02T10:06:16.913599+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:04.062325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:04.062325+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:34.030289+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"235'1699","reported_seq":1726,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:20.845316+0000","last_change":"2024-09-02T10:05:56.877006+0000","last_active":"2024-09-02T10:06:20.845316+0000","last_peered":"2024-09-02T10:06:20.845316+0000","last_clean":"2024-09-02T10:06:20.845316+0000","last_became_active":"2024-09-02T10:05:56.876540+0000","last_became_peered":"2024-09-02T10:05:56.876540+0000","last_unstale":"2024-09-02T10:06:20.845316+0000","last_undegraded":"2024-09-02T10:06:20.845316+0000","last_fullsized":"2024-09-02T10:06:20.845316+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1699,"log_dups_size":0,"ondisk_log_size":1699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:02.154423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111345664,"num_objects":1699,"num_object_clones":0,"num_object_copies":5097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1699,"num_write_kb":108736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":347,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.913330+0000","last_change":"2024-09-02T10:06:04.265827+0000","last_active":"2024-09-02T10:06:16.913330+0000","last_peered":"2024-09-02T10:06:16.913330+0000","last_clean":"2024-09-02T10:06:16.913330+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:06:16.913330+0000","last_undegraded":"2024-09-02T10:06:16.913330+0000","last_fullsized":"2024-09-02T10:06:16.913330+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:04.265776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:04.265776+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.085143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"234'1307","reported_seq":1332,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.792264+0000","last_change":"2024-09-02T10:05:56.586356+0000","last_active":"2024-09-02T10:06:16.792264+0000","last_peered":"2024-09-02T10:06:16.792264+0000","last_clean":"2024-09-02T10:06:16.792264+0000","last_became_active":"2024-09-02T10:05:56.585883+0000","last_became_peered":"2024-09-02T10:05:56.585883+0000","last_unstale":"2024-09-02T10:06:16.792264+0000","last_undegraded":"2024-09-02T10:06:16.792264+0000","last_fullsized":"2024-09-02T10:06:16.792264+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:06.966563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85655552,"num_objects":1307,"num_object_clones":0,"num_object_copies":3921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1307,"num_write_kb":83648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":254,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.914774+0000","last_change":"2024-09-02T10:06:14.804152+0000","last_active":"2024-09-02T10:06:16.914774+0000","last_peered":"2024-09-02T10:06:16.914774+0000","last_clean":"2024-09-02T10:06:16.914774+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:06:16.914774+0000","last_undegraded":"2024-09-02T10:06:16.914774+0000","last_fullsized":"2024-09-02T10:06:16.914774+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:14.804108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:14.804108+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:35.214952+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"235'1613","reported_seq":1642,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:20.845873+0000","last_change":"2024-09-02T10:05:56.584811+0000","last_active":"2024-09-02T10:06:20.845873+0000","last_peered":"2024-09-02T10:06:20.845873+0000","last_clean":"2024-09-02T10:06:20.845873+0000","last_became_active":"2024-09-02T10:05:56.584450+0000","last_became_peered":"2024-09-02T10:05:56.584450+0000","last_unstale":"2024-09-02T10:06:20.845873+0000","last_undegraded":"2024-09-02T10:06:20.845873+0000","last_fullsized":"2024-09-02T10:06:20.845873+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1613,"log_dups_size":0,"ondisk_log_size":1613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:04.195054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":105709568,"num_objects":1613,"num_object_clones":0,"num_object_copies":4839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1613,"num_write_kb":103232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":115,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.913843+0000","last_change":"2024-09-02T10:06:03.083413+0000","last_active":"2024-09-02T10:06:16.913843+0000","last_peered":"2024-09-02T10:06:16.913843+0000","last_clean":"2024-09-02T10:06:16.913843+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:06:16.913843+0000","last_undegraded":"2024-09-02T10:06:16.913843+0000","last_fullsized":"2024-09-02T10:06:16.913843+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.083338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.083338+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:20.993372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"234'1310","reported_seq":1338,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.911567+0000","last_change":"2024-09-02T10:05:56.586192+0000","last_active":"2024-09-02T10:06:16.911567+0000","last_peered":"2024-09-02T10:06:16.911567+0000","last_clean":"2024-09-02T10:06:16.911567+0000","last_became_active":"2024-09-02T10:05:56.585870+0000","last_became_peered":"2024-09-02T10:05:56.585870+0000","last_unstale":"2024-09-02T10:06:16.911567+0000","last_undegraded":"2024-09-02T10:06:16.911567+0000","last_fullsized":"2024-09-02T10:06:16.911567+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1310,"log_dups_size":0,"ondisk_log_size":1310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:59.656901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85852160,"num_objects":1310,"num_object_clones":0,"num_object_copies":3930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1310,"num_write_kb":83840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":127,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-02T10:06:16.920729+0000","last_change":"2024-09-02T10:06:01.391977+0000","last_active":"2024-09-02T10:06:16.920729+0000","last_peered":"2024-09-02T10:06:16.920729+0000","last_clean":"2024-09-02T10:06:16.920729+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:06:16.920729+0000","last_undegraded":"2024-09-02T10:06:16.920729+0000","last_fullsized":"2024-09-02T10:06:16.920729+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.391925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.391925+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:04.473676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1454637056,"num_objects":22196,"num_object_clones":0,"num_object_copies":66588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22196,"num_write_kb":1420544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4357160960,"data_stored":4357160960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22196,"ondisk_log_size":22196,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383754,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":779268,"kb_used_data":631476,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92965884,"statfs":{"total":95995035648,"available":95197065216,"internally_reserved":0,"allocated":646631424,"data_stored":645987912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":151321028},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612687,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841620,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":385876,"kb_used_data":268608,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93359276,"statfs":{"total":95995035648,"available":95599898624,"internally_reserved":0,"allocated":275054592,"data_stored":274406492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":120060352},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070554,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":281296,"kb_used_data":191808,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93463856,"statfs":{"total":95995035648,"available":95706988544,"internally_reserved":0,"allocated":196411392,"data_stored":195763292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299485,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":520016,"kb_used_data":394432,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93225136,"statfs":{"total":95995035648,"available":95462539264,"internally_reserved":0,"allocated":403898368,"data_stored":403250268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528417,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":463876,"kb_used_data":396596,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93281276,"statfs":{"total":95995035648,"available":95520026624,"internally_reserved":0,"allocated":406114304,"data_stored":405470800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724646,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":386316,"kb_used_data":319036,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93358836,"statfs":{"total":95995035648,"available":95599448064,"internally_reserved":0,"allocated":326692864,"data_stored":326045808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953580,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":719248,"kb_used_data":651968,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93025904,"statfs":{"total":95995035648,"available":95258525696,"internally_reserved":0,"allocated":667615232,"data_stored":666967132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":591324,"kb_used_data":429640,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93153828,"statfs":{"total":95995035648,"available":95389519872,"internally_reserved":0,"allocated":439951360,"data_stored":439299708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":165542344},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411445,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":242332,"kb_used_data":84808,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93502820,"statfs":{"total":95995035648,"available":95746887680,"internally_reserved":0,"allocated":86843392,"data_stored":86191740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":472724,"kb_used_data":356864,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93272428,"statfs":{"total":95995035648,"available":95510966272,"internally_reserved":0,"allocated":365428736,"data_stored":364780636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":118618565},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":485956,"kb_used_data":400628,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93259196,"statfs":{"total":95995035648,"available":95497416704,"internally_reserved":0,"allocated":410243072,"data_stored":409599568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":413597696,"data_stored":413597696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":369754112,"data_stored":369754112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":87425024,"data_stored":87425024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":463208448,"data_stored":463208448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":675151872,"data_stored":675151872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":343605248,"data_stored":343605248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":429981696,"data_stored":429981696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":430112768,"data_stored":430112768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":194838528,"data_stored":194838528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":274726912,"data_stored":274726912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":674758656,"data_stored":674758656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:06:23.170 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-02T10:06:23.171 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:06:23.171 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-02T10:06:23.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:06:23.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:23.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:23.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:23.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:23.310 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:23.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:23.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:23.435 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:23.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:23.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:23 smithi151 ceph-mon[16639]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:23.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:23 smithi187 ceph-mon[20072]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:23.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:23.559 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:23.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:23.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:23 smithi162 ceph-mon[20279]: from='client.25091 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:23.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:23.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:23.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:23.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:23.810 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:23.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:23.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:23.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:23.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:23.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:23.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:24.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 21 16 30413 30397 90.4604 85.9375 0.00215829 0.0110468 2024-09-02T10:06:24.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:24.073 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:24.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:24.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:06:24.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:24.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:24.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:24.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:24.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:24.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:24.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:24.437 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:24.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:24 smithi151 ceph-mon[16639]: pgmap v578: 37 pgs: 37 active+clean; 1.6 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-02T10:06:24.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/641851486' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:24.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:24 smithi151 ceph-mon[16639]: osdmap e237: 12 total, 12 up, 11 in 2024-09-02T10:06:24.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:24.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:24.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:24.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:06:24.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:06:24.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:06:24.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:06:24.493 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:06:24.493 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:06:24.493 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:06:24.493 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:06:24.494 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:06:24.494 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:06:24.494 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:06:24.494 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:06:24.494 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:06:24.494 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:06:24.494 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:06:24.494 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:06:24.494 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:06:24.495 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:06:24.495 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:06:24.495 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:06:24.495 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:06:24.495 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:06:24.495 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:06:24.495 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:06:24.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:24 smithi187 ceph-mon[20072]: pgmap v578: 37 pgs: 37 active+clean; 1.6 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-02T10:06:24.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/641851486' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:24.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:24 smithi187 ceph-mon[20072]: osdmap e237: 12 total, 12 up, 11 in 2024-09-02T10:06:24.508 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:24.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:24.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:24 smithi162 ceph-mon[20279]: pgmap v578: 37 pgs: 37 active+clean; 1.6 GiB data, 5.9 GiB used, 978 GiB / 983 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-09-02T10:06:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/641851486' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:24 smithi162 ceph-mon[20279]: osdmap e237: 12 total, 12 up, 11 in 2024-09-02T10:06:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:24.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:24.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:24.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:24.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:24.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:24.743 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:24.781 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:24.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:24.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:24.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:24.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:24.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:24.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:24.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:25.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 22 16 32436 32420 92.0953 126.438 0.0127068 0.0108501 2024-09-02T10:06:25.062 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:06:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:06:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:32444 3.3 3:c281c5d5:::benchmark_data_smithi187_59989_object32443:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:06:25.032184+0000", 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.028389615, 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.028412617000000001, 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 32444 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.032184+0000", 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.032184+0000", 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.032186+0000", 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2950000000000001e-06 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.032227+0000", 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1904999999999999e-05 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.032238+0000", 2024-09-02T10:06:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0954e-05 2024-09-02T10:06:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.032247+0000", 2024-09-02T10:06:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.259e-06 2024-09-02T10:06:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.032302+0000", 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5253000000000001e-05 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.032432+0000", 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012977 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:06:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.032533+0000", 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101191 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.033652+0000", 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001119613 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:25.052158+0000", 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018505972999999998 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:06:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-02T10:06:25.071 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:06:25.084 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:25.109 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:25.110 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:25.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:25.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:25.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:25.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:25.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:25.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:25.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:25.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:25.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:25.487 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:25.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:25.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:25.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:25.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:25.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:25.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:25.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:25.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:25.872 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:25.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:25.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:25.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:25.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:25.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:26.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 23 16 33628 33612 91.33 74.5 0.00326702 0.0109414 2024-09-02T10:06:26.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:26.120 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:06:26.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:26.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:26.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:26.201 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:06:26.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:26.258 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:26.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:26.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:26 smithi162 ceph-mon[20279]: pgmap v580: 37 pgs: 37 active+clean; 1.7 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-02T10:06:26.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3913802418' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:26.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:26.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:26.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:26.476 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:26.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:26.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:26 smithi187 ceph-mon[20072]: pgmap v580: 37 pgs: 37 active+clean; 1.7 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-02T10:06:26.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3913802418' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:26.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:26.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:26.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:26.633 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:06:26.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:26 smithi151 ceph-mon[16639]: pgmap v580: 37 pgs: 37 active+clean; 1.7 GiB data, 6.3 GiB used, 977 GiB / 983 GiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-02T10:06:26.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3913802418' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:26.635 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:06:26.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:26.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:26.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:26.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:26.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:26.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:26.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:26.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:26.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:26.991 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:26.992 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:06:26.993 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:06:26.993 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:06:26.993 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:06:26.993 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:06:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:27.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:27.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:27.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 24 16 34795 34779 90.5633 72.9375 0.00860308 0.0110364 2024-09-02T10:06:27.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:27.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:27.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:27.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:27.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:27.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:27.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:27.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:27.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:27.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:27.491 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:27 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:27.491 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3913802418' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:27.491 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:27 smithi151 ceph-mon[16639]: osdmap e238: 12 total, 12 up, 11 in 2024-09-02T10:06:27.491 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:27 smithi151 ceph-mon[16639]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:27.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:27 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3913802418' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:27 smithi187 ceph-mon[20072]: osdmap e238: 12 total, 12 up, 11 in 2024-09-02T10:06:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:27 smithi187 ceph-mon[20072]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:27.531 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:27.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:27.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:06:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:27 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3913802418' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:27 smithi162 ceph-mon[20279]: osdmap e238: 12 total, 12 up, 11 in 2024-09-02T10:06:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:27 smithi162 ceph-mon[20279]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:06:27.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:27.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:06:27.659 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:27.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:27.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:27.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:27.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:27.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:27.908 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:27.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:27.934 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:28.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:28.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 25 16 36192 36176 90.433 87.3125 0.0418593 0.0110466 2024-09-02T10:06:28.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:28.033 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:28.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:28.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:28.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:28.160 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:28.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:28.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:28.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:28.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:28 smithi162 ceph-mon[20279]: pgmap v582: 37 pgs: 37 active+clean; 2.0 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-02T10:06:28.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:28.433 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:28.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:28.481 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:28 smithi151 ceph-mon[16639]: pgmap v582: 37 pgs: 37 active+clean; 2.0 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-02T10:06:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:28 smithi187 ceph-mon[20072]: pgmap v582: 37 pgs: 37 active+clean; 2.0 GiB data, 7.0 GiB used, 976 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s 2024-09-02T10:06:28.523 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":581,"stamp":"2024-09-02T10:06:26.201574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1906836000,"num_objects":29089,"num_object_clones":0,"num_object_copies":87267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29087,"num_write_kb":1861568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31019,"ondisk_log_size":31019,"up":111,"acting":111,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":111,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":6696176,"kb_used_data":5466024,"kb_used_omap":17,"kb_used_meta":1229934,"kb_avail":1024500496,"statfs":{"total":1055945392128,"available":1049088507904,"internally_reserved":0,"allocated":5597208576,"data_stored":5590014160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1259453307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":562888704,"num_objects":8589,"num_object_clones":0,"num_object_copies":25767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8589,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8589,"num_write_kb":549696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.068023"},"pg_stats":[{"pgid":"1.14","version":"22'92","reported_seq":247,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.193917+0000","last_change":"2024-09-02T10:06:01.163588+0000","last_active":"2024-09-02T10:06:21.193917+0000","last_peered":"2024-09-02T10:06:21.193917+0000","last_clean":"2024-09-02T10:06:21.193917+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:06:21.193917+0000","last_undegraded":"2024-09-02T10:06:21.193917+0000","last_fullsized":"2024-09-02T10:06:21.193917+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.163525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.163525+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:12.385214+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":325,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.197216+0000","last_change":"2024-09-02T10:05:05.778602+0000","last_active":"2024-09-02T10:06:21.197216+0000","last_peered":"2024-09-02T10:06:21.197216+0000","last_clean":"2024-09-02T10:06:21.197216+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:06:21.197216+0000","last_undegraded":"2024-09-02T10:06:21.197216+0000","last_fullsized":"2024-09-02T10:06:21.197216+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:05.778537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:05.778537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:24.669900+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":269,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.195071+0000","last_change":"2024-09-02T10:06:03.254398+0000","last_active":"2024-09-02T10:06:21.195071+0000","last_peered":"2024-09-02T10:06:21.195071+0000","last_clean":"2024-09-02T10:06:21.195071+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:06:21.195071+0000","last_undegraded":"2024-09-02T10:06:21.195071+0000","last_fullsized":"2024-09-02T10:06:21.195071+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.254327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.254327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:16.949688+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":249,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.196157+0000","last_change":"2024-09-02T10:05:38.438762+0000","last_active":"2024-09-02T10:06:21.196157+0000","last_peered":"2024-09-02T10:06:21.196157+0000","last_clean":"2024-09-02T10:06:21.196157+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:06:21.196157+0000","last_undegraded":"2024-09-02T10:06:21.196157+0000","last_fullsized":"2024-09-02T10:06:21.196157+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.438672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.438672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:03.647240+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":262,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.195044+0000","last_change":"2024-09-02T10:06:01.152342+0000","last_active":"2024-09-02T10:06:21.195044+0000","last_peered":"2024-09-02T10:06:21.195044+0000","last_clean":"2024-09-02T10:06:21.195044+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:06:21.195044+0000","last_undegraded":"2024-09-02T10:06:21.195044+0000","last_fullsized":"2024-09-02T10:06:21.195044+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.152278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.152278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:01.602916+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"236'1764","reported_seq":1794,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:22.022957+0000","last_change":"2024-09-02T10:05:56.584135+0000","last_active":"2024-09-02T10:06:22.022957+0000","last_peered":"2024-09-02T10:06:22.022957+0000","last_clean":"2024-09-02T10:06:22.022957+0000","last_became_active":"2024-09-02T10:05:56.583663+0000","last_became_peered":"2024-09-02T10:05:56.583663+0000","last_unstale":"2024-09-02T10:06:22.022957+0000","last_undegraded":"2024-09-02T10:06:22.022957+0000","last_fullsized":"2024-09-02T10:06:22.022957+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1764,"log_dups_size":0,"ondisk_log_size":1764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:15.291683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115605504,"num_objects":1764,"num_object_clones":0,"num_object_copies":5292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1764,"num_write_kb":112896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":335,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.195044+0000","last_change":"2024-09-02T10:05:38.077751+0000","last_active":"2024-09-02T10:06:21.195044+0000","last_peered":"2024-09-02T10:06:21.195044+0000","last_clean":"2024-09-02T10:06:21.195044+0000","last_became_active":"2024-09-02T10:04:35.811251+0000","last_became_peered":"2024-09-02T10:04:35.811251+0000","last_unstale":"2024-09-02T10:06:21.195044+0000","last_undegraded":"2024-09-02T10:06:21.195044+0000","last_fullsized":"2024-09-02T10:06:21.195044+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:38.077684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:38.077684+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:57.694686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"236'1847","reported_seq":1878,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:23.313649+0000","last_change":"2024-09-02T10:05:56.583572+0000","last_active":"2024-09-02T10:06:23.313649+0000","last_peered":"2024-09-02T10:06:23.313649+0000","last_clean":"2024-09-02T10:06:23.313649+0000","last_became_active":"2024-09-02T10:05:56.583278+0000","last_became_peered":"2024-09-02T10:05:56.583278+0000","last_unstale":"2024-09-02T10:06:23.313649+0000","last_undegraded":"2024-09-02T10:06:23.313649+0000","last_fullsized":"2024-09-02T10:06:23.313649+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1847,"log_dups_size":0,"ondisk_log_size":1847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:11.499666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121044992,"num_objects":1847,"num_object_clones":0,"num_object_copies":5541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1847,"num_write_kb":118208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":461,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-02T10:06:23.840870+0000","last_change":"2024-09-02T10:06:08.953931+0000","last_active":"2024-09-02T10:06:23.840870+0000","last_peered":"2024-09-02T10:06:23.840870+0000","last_clean":"2024-09-02T10:06:23.840870+0000","last_became_active":"2024-09-02T10:01:50.768786+0000","last_became_peered":"2024-09-02T10:01:50.768786+0000","last_unstale":"2024-09-02T10:06:23.840870+0000","last_undegraded":"2024-09-02T10:06:23.840870+0000","last_fullsized":"2024-09-02T10:06:23.840870+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:08.953856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:08.953856+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:10.024416+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"236'1692","reported_seq":1722,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.909429+0000","last_change":"2024-09-02T10:05:56.583542+0000","last_active":"2024-09-02T10:06:21.909429+0000","last_peered":"2024-09-02T10:06:21.909429+0000","last_clean":"2024-09-02T10:06:21.909429+0000","last_became_active":"2024-09-02T10:05:56.583249+0000","last_became_peered":"2024-09-02T10:05:56.583249+0000","last_unstale":"2024-09-02T10:06:21.909429+0000","last_undegraded":"2024-09-02T10:06:21.909429+0000","last_fullsized":"2024-09-02T10:06:21.909429+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1692,"log_dups_size":0,"ondisk_log_size":1692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:01.823622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110886912,"num_objects":1692,"num_object_clones":0,"num_object_copies":5076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1692,"num_write_kb":108288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":92,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.196794+0000","last_change":"2024-09-02T10:04:57.193877+0000","last_active":"2024-09-02T10:06:21.196794+0000","last_peered":"2024-09-02T10:06:21.196794+0000","last_clean":"2024-09-02T10:06:21.196794+0000","last_became_active":"2024-09-02T10:04:57.020528+0000","last_became_peered":"2024-09-02T10:04:57.020528+0000","last_unstale":"2024-09-02T10:06:21.196794+0000","last_undegraded":"2024-09-02T10:06:21.196794+0000","last_fullsized":"2024-09-02T10:06:21.196794+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:57.193822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:57.193822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:26.554027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"237'1933","reported_seq":1965,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-02T10:06:24.209715+0000","last_change":"2024-09-02T10:05:56.584669+0000","last_active":"2024-09-02T10:06:24.209715+0000","last_peered":"2024-09-02T10:06:24.209715+0000","last_clean":"2024-09-02T10:06:24.209715+0000","last_became_active":"2024-09-02T10:05:56.584313+0000","last_became_peered":"2024-09-02T10:05:56.584313+0000","last_unstale":"2024-09-02T10:06:24.209715+0000","last_undegraded":"2024-09-02T10:06:24.209715+0000","last_fullsized":"2024-09-02T10:06:24.209715+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1933,"log_dups_size":0,"ondisk_log_size":1933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:08.498885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126681088,"num_objects":1933,"num_object_clones":0,"num_object_copies":5799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1933,"num_write_kb":123712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":247,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.195599+0000","last_change":"2024-09-02T10:06:02.338906+0000","last_active":"2024-09-02T10:06:21.195599+0000","last_peered":"2024-09-02T10:06:21.195599+0000","last_clean":"2024-09-02T10:06:21.195599+0000","last_became_active":"2024-09-02T10:02:42.786740+0000","last_became_peered":"2024-09-02T10:02:42.786740+0000","last_unstale":"2024-09-02T10:06:21.195599+0000","last_undegraded":"2024-09-02T10:06:21.195599+0000","last_fullsized":"2024-09-02T10:06:21.195599+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:02.338299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:02.338299+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:23.238276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"236'1689","reported_seq":1719,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.501925+0000","last_change":"2024-09-02T10:05:56.579293+0000","last_active":"2024-09-02T10:06:21.501925+0000","last_peered":"2024-09-02T10:06:21.501925+0000","last_clean":"2024-09-02T10:06:21.501925+0000","last_became_active":"2024-09-02T10:05:56.578973+0000","last_became_peered":"2024-09-02T10:05:56.578973+0000","last_unstale":"2024-09-02T10:06:21.501925+0000","last_undegraded":"2024-09-02T10:06:21.501925+0000","last_fullsized":"2024-09-02T10:06:21.501925+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1689,"log_dups_size":0,"ondisk_log_size":1689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:17.084065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110690304,"num_objects":1689,"num_object_clones":0,"num_object_copies":5067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1689,"num_write_kb":108096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":404,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.195589+0000","last_change":"2024-09-02T10:05:42.420814+0000","last_active":"2024-09-02T10:06:21.195589+0000","last_peered":"2024-09-02T10:06:21.195589+0000","last_clean":"2024-09-02T10:06:21.195589+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:06:21.195589+0000","last_undegraded":"2024-09-02T10:06:21.195589+0000","last_fullsized":"2024-09-02T10:06:21.195589+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:05:42.420722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:05:42.420722+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:00.199426+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"236'1821","reported_seq":1850,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:23.292274+0000","last_change":"2024-09-02T10:05:56.585087+0000","last_active":"2024-09-02T10:06:23.292274+0000","last_peered":"2024-09-02T10:06:23.292274+0000","last_clean":"2024-09-02T10:06:23.292274+0000","last_became_active":"2024-09-02T10:05:56.584888+0000","last_became_peered":"2024-09-02T10:05:56.584888+0000","last_unstale":"2024-09-02T10:06:23.292274+0000","last_undegraded":"2024-09-02T10:06:23.292274+0000","last_fullsized":"2024-09-02T10:06:23.292274+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1821,"log_dups_size":0,"ondisk_log_size":1821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.136803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":119341056,"num_objects":1821,"num_object_clones":0,"num_object_copies":5463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1821,"num_write_kb":116544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":241,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.193994+0000","last_change":"2024-09-02T10:06:02.167573+0000","last_active":"2024-09-02T10:06:21.193994+0000","last_peered":"2024-09-02T10:06:21.193994+0000","last_clean":"2024-09-02T10:06:21.193994+0000","last_became_active":"2024-09-02T10:02:48.287986+0000","last_became_peered":"2024-09-02T10:02:48.287986+0000","last_unstale":"2024-09-02T10:06:21.193994+0000","last_undegraded":"2024-09-02T10:06:21.193994+0000","last_fullsized":"2024-09-02T10:06:21.193994+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:02.167523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:02.167523+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.321956+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"236'1890","reported_seq":1921,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:22.760877+0000","last_change":"2024-09-02T10:05:56.581266+0000","last_active":"2024-09-02T10:06:22.760877+0000","last_peered":"2024-09-02T10:06:22.760877+0000","last_clean":"2024-09-02T10:06:22.760877+0000","last_became_active":"2024-09-02T10:05:56.580910+0000","last_became_peered":"2024-09-02T10:05:56.580910+0000","last_unstale":"2024-09-02T10:06:22.760877+0000","last_undegraded":"2024-09-02T10:06:22.760877+0000","last_fullsized":"2024-09-02T10:06:22.760877+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1890,"log_dups_size":0,"ondisk_log_size":1890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:05.300847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123863040,"num_objects":1890,"num_object_clones":0,"num_object_copies":5670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1890,"num_write_kb":120960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":92,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.196866+0000","last_change":"2024-09-02T10:04:58.284511+0000","last_active":"2024-09-02T10:06:21.196866+0000","last_peered":"2024-09-02T10:06:21.196866+0000","last_clean":"2024-09-02T10:06:21.196866+0000","last_became_active":"2024-09-02T10:04:57.020607+0000","last_became_peered":"2024-09-02T10:04:57.020607+0000","last_unstale":"2024-09-02T10:06:21.196866+0000","last_undegraded":"2024-09-02T10:06:21.196866+0000","last_fullsized":"2024-09-02T10:06:21.196866+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:04:58.284452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:04:58.284452+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:23.953648+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"236'1695","reported_seq":1725,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.831477+0000","last_change":"2024-09-02T10:05:56.690497+0000","last_active":"2024-09-02T10:06:21.831477+0000","last_peered":"2024-09-02T10:06:21.831477+0000","last_clean":"2024-09-02T10:06:21.831477+0000","last_became_active":"2024-09-02T10:05:56.690229+0000","last_became_peered":"2024-09-02T10:05:56.690229+0000","last_unstale":"2024-09-02T10:06:21.831477+0000","last_undegraded":"2024-09-02T10:06:21.831477+0000","last_fullsized":"2024-09-02T10:06:21.831477+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1695,"log_dups_size":0,"ondisk_log_size":1695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:05.936975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111083520,"num_objects":1695,"num_object_clones":0,"num_object_copies":5085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1695,"num_write_kb":108480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":365,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.197454+0000","last_change":"2024-09-02T10:06:05.307800+0000","last_active":"2024-09-02T10:06:21.197454+0000","last_peered":"2024-09-02T10:06:21.197454+0000","last_clean":"2024-09-02T10:06:21.197454+0000","last_became_active":"2024-09-02T10:04:35.810587+0000","last_became_peered":"2024-09-02T10:04:35.810587+0000","last_unstale":"2024-09-02T10:06:21.197454+0000","last_undegraded":"2024-09-02T10:06:21.197454+0000","last_fullsized":"2024-09-02T10:06:21.197454+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:05.307738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:05.307738+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:30.265023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"237'1945","reported_seq":1978,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-02T10:06:24.197612+0000","last_change":"2024-09-02T10:05:56.582278+0000","last_active":"2024-09-02T10:06:24.197612+0000","last_peered":"2024-09-02T10:06:24.197612+0000","last_clean":"2024-09-02T10:06:24.197612+0000","last_became_active":"2024-09-02T10:05:56.581969+0000","last_became_peered":"2024-09-02T10:05:56.581969+0000","last_unstale":"2024-09-02T10:06:24.197612+0000","last_undegraded":"2024-09-02T10:06:24.197612+0000","last_fullsized":"2024-09-02T10:06:24.197612+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1945,"log_dups_size":0,"ondisk_log_size":1945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.236882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127467520,"num_objects":1945,"num_object_clones":0,"num_object_copies":5835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1945,"num_write_kb":124480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":92,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.196767+0000","last_change":"2024-09-02T10:06:03.857986+0000","last_active":"2024-09-02T10:06:21.196767+0000","last_peered":"2024-09-02T10:06:21.196767+0000","last_clean":"2024-09-02T10:06:21.196767+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:06:21.196767+0000","last_undegraded":"2024-09-02T10:06:21.196767+0000","last_fullsized":"2024-09-02T10:06:21.196767+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.857913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.857913+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:27.151711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"236'1698","reported_seq":1727,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.918790+0000","last_change":"2024-09-02T10:05:56.876782+0000","last_active":"2024-09-02T10:06:21.918790+0000","last_peered":"2024-09-02T10:06:21.918790+0000","last_clean":"2024-09-02T10:06:21.918790+0000","last_became_active":"2024-09-02T10:05:56.876397+0000","last_became_peered":"2024-09-02T10:05:56.876397+0000","last_unstale":"2024-09-02T10:06:21.918790+0000","last_undegraded":"2024-09-02T10:06:21.918790+0000","last_fullsized":"2024-09-02T10:06:21.918790+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1698,"log_dups_size":0,"ondisk_log_size":1698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:10.344558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111280128,"num_objects":1698,"num_object_clones":0,"num_object_copies":5094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1698,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1698,"num_write_kb":108672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":108,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-02T10:06:23.840846+0000","last_change":"2024-09-02T10:06:01.039876+0000","last_active":"2024-09-02T10:06:23.840846+0000","last_peered":"2024-09-02T10:06:23.840846+0000","last_clean":"2024-09-02T10:06:23.840846+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:06:23.840846+0000","last_undegraded":"2024-09-02T10:06:23.840846+0000","last_fullsized":"2024-09-02T10:06:23.840846+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.039815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.039815+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:24.958614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"236'1627","reported_seq":1657,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.428691+0000","last_change":"2024-09-02T10:05:56.581467+0000","last_active":"2024-09-02T10:06:21.428691+0000","last_peered":"2024-09-02T10:06:21.428691+0000","last_clean":"2024-09-02T10:06:21.428691+0000","last_became_active":"2024-09-02T10:05:56.581229+0000","last_became_peered":"2024-09-02T10:05:56.581229+0000","last_unstale":"2024-09-02T10:06:21.428691+0000","last_undegraded":"2024-09-02T10:06:21.428691+0000","last_fullsized":"2024-09-02T10:06:21.428691+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1627,"log_dups_size":0,"ondisk_log_size":1627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:03.921090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106627072,"num_objects":1627,"num_object_clones":0,"num_object_copies":4881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1627,"num_write_kb":104128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":321,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.197322+0000","last_change":"2024-09-02T10:06:15.608866+0000","last_active":"2024-09-02T10:06:21.197322+0000","last_peered":"2024-09-02T10:06:21.197322+0000","last_clean":"2024-09-02T10:06:21.197322+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:06:21.197322+0000","last_undegraded":"2024-09-02T10:06:21.197322+0000","last_fullsized":"2024-09-02T10:06:21.197322+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:15.608802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:15.608802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:17.634785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"236'1792","reported_seq":1822,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:23.321877+0000","last_change":"2024-09-02T10:05:56.876935+0000","last_active":"2024-09-02T10:06:23.321877+0000","last_peered":"2024-09-02T10:06:23.321877+0000","last_clean":"2024-09-02T10:06:23.321877+0000","last_became_active":"2024-09-02T10:05:56.876573+0000","last_became_peered":"2024-09-02T10:05:56.876573+0000","last_unstale":"2024-09-02T10:06:23.321877+0000","last_undegraded":"2024-09-02T10:06:23.321877+0000","last_fullsized":"2024-09-02T10:06:23.321877+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1792,"log_dups_size":0,"ondisk_log_size":1792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:24.250156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":1792,"num_object_clones":0,"num_object_copies":5376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1792,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":119,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-02T10:06:23.840790+0000","last_change":"2024-09-02T10:06:04.062386+0000","last_active":"2024-09-02T10:06:23.840790+0000","last_peered":"2024-09-02T10:06:23.840790+0000","last_clean":"2024-09-02T10:06:23.840790+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:06:23.840790+0000","last_undegraded":"2024-09-02T10:06:23.840790+0000","last_fullsized":"2024-09-02T10:06:23.840790+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:04.062325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:04.062325+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:34.030289+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"237'2145","reported_seq":2178,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-02T10:06:25.836525+0000","last_change":"2024-09-02T10:05:56.877006+0000","last_active":"2024-09-02T10:06:25.836525+0000","last_peered":"2024-09-02T10:06:25.836525+0000","last_clean":"2024-09-02T10:06:25.836525+0000","last_became_active":"2024-09-02T10:05:56.876540+0000","last_became_peered":"2024-09-02T10:05:56.876540+0000","last_unstale":"2024-09-02T10:06:25.836525+0000","last_undegraded":"2024-09-02T10:06:25.836525+0000","last_fullsized":"2024-09-02T10:06:25.836525+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":2145,"log_dups_size":0,"ondisk_log_size":2145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:02.154423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140574720,"num_objects":2145,"num_object_clones":0,"num_object_copies":6435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2145,"num_write_kb":137280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":349,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.197348+0000","last_change":"2024-09-02T10:06:04.265827+0000","last_active":"2024-09-02T10:06:21.197348+0000","last_peered":"2024-09-02T10:06:21.197348+0000","last_clean":"2024-09-02T10:06:21.197348+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:06:21.197348+0000","last_undegraded":"2024-09-02T10:06:21.197348+0000","last_fullsized":"2024-09-02T10:06:21.197348+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:04.265776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:04.265776+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:19.085143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"236'1758","reported_seq":1787,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.837882+0000","last_change":"2024-09-02T10:05:56.586356+0000","last_active":"2024-09-02T10:06:21.837882+0000","last_peered":"2024-09-02T10:06:21.837882+0000","last_clean":"2024-09-02T10:06:21.837882+0000","last_became_active":"2024-09-02T10:05:56.585883+0000","last_became_peered":"2024-09-02T10:05:56.585883+0000","last_unstale":"2024-09-02T10:06:21.837882+0000","last_undegraded":"2024-09-02T10:06:21.837882+0000","last_fullsized":"2024-09-02T10:06:21.837882+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1758,"log_dups_size":0,"ondisk_log_size":1758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:06.966563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115212288,"num_objects":1758,"num_object_clones":0,"num_object_copies":5274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1758,"num_write_kb":112512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":258,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-02T10:06:23.842726+0000","last_change":"2024-09-02T10:06:14.804152+0000","last_active":"2024-09-02T10:06:23.842726+0000","last_peered":"2024-09-02T10:06:23.842726+0000","last_clean":"2024-09-02T10:06:23.842726+0000","last_became_active":"2024-09-02T10:02:42.791251+0000","last_became_peered":"2024-09-02T10:02:42.791251+0000","last_unstale":"2024-09-02T10:06:23.842726+0000","last_undegraded":"2024-09-02T10:06:23.842726+0000","last_fullsized":"2024-09-02T10:06:23.842726+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:14.804108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:14.804108+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:35.214952+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"237'2067","reported_seq":2100,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-02T10:06:25.849734+0000","last_change":"2024-09-02T10:05:56.584811+0000","last_active":"2024-09-02T10:06:25.849734+0000","last_peered":"2024-09-02T10:06:25.849734+0000","last_clean":"2024-09-02T10:06:25.849734+0000","last_became_active":"2024-09-02T10:05:56.584450+0000","last_became_peered":"2024-09-02T10:05:56.584450+0000","last_unstale":"2024-09-02T10:06:25.849734+0000","last_undegraded":"2024-09-02T10:06:25.849734+0000","last_fullsized":"2024-09-02T10:06:25.849734+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:04.195054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135462912,"num_objects":2067,"num_object_clones":0,"num_object_copies":6201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2067,"num_write_kb":132288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":119,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-02T10:06:23.840971+0000","last_change":"2024-09-02T10:06:03.083413+0000","last_active":"2024-09-02T10:06:23.840971+0000","last_peered":"2024-09-02T10:06:23.840971+0000","last_clean":"2024-09-02T10:06:23.840971+0000","last_became_active":"2024-09-02T10:04:47.929075+0000","last_became_peered":"2024-09-02T10:04:47.929075+0000","last_unstale":"2024-09-02T10:06:23.840971+0000","last_undegraded":"2024-09-02T10:06:23.840971+0000","last_fullsized":"2024-09-02T10:06:23.840971+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.083338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.083338+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:20.993372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"236'1724","reported_seq":1750,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.921039+0000","last_change":"2024-09-02T10:05:56.586192+0000","last_active":"2024-09-02T10:06:21.921039+0000","last_peered":"2024-09-02T10:06:21.921039+0000","last_clean":"2024-09-02T10:06:21.921039+0000","last_became_active":"2024-09-02T10:05:56.585870+0000","last_became_peered":"2024-09-02T10:05:56.585870+0000","last_unstale":"2024-09-02T10:06:21.921039+0000","last_undegraded":"2024-09-02T10:06:21.921039+0000","last_fullsized":"2024-09-02T10:06:21.921039+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":1724,"log_dups_size":0,"ondisk_log_size":1724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:06:59.656901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112984064,"num_objects":1724,"num_object_clones":0,"num_object_copies":5172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1724,"num_write_kb":110336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":129,"reported_epoch":236,"state":"active+clean","last_fresh":"2024-09-02T10:06:21.194477+0000","last_change":"2024-09-02T10:06:01.391977+0000","last_active":"2024-09-02T10:06:21.194477+0000","last_peered":"2024-09-02T10:06:21.194477+0000","last_clean":"2024-09-02T10:06:21.194477+0000","last_became_active":"2024-09-02T10:04:35.810119+0000","last_became_peered":"2024-09-02T10:04:35.810119+0000","last_unstale":"2024-09-02T10:06:21.194477+0000","last_undegraded":"2024-09-02T10:06:21.194477+0000","last_fullsized":"2024-09-02T10:06:21.194477+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.391925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.391925+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:04.473676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1906245632,"num_objects":29087,"num_object_clones":0,"num_object_copies":87261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29087,"num_write_kb":1861568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5715656704,"data_stored":5715656704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29087,"ondisk_log_size":29087,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1932,"ondisk_log_size":1932,"up":63,"acting":63,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383755,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988768,"kb_used_data":845132,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92756384,"statfs":{"total":95995035648,"available":94982537216,"internally_reserved":0,"allocated":865415168,"data_stored":864762514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":147061188},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841621,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":469852,"kb_used_data":348428,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93275300,"statfs":{"total":95995035648,"available":95513907200,"internally_reserved":0,"allocated":356790272,"data_stored":356137626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":124320192},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070555,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":337900,"kb_used_data":248408,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93407252,"statfs":{"total":95995035648,"available":95649026048,"internally_reserved":0,"allocated":254369792,"data_stored":253712550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299486,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":656872,"kb_used_data":539608,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":93088280,"statfs":{"total":95995035648,"available":95322398720,"internally_reserved":0,"allocated":552558592,"data_stored":551901350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":120060362},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528418,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":611744,"kb_used_data":536140,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93133408,"statfs":{"total":95995035648,"available":95368609792,"internally_reserved":0,"allocated":549007360,"data_stored":548354714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724647,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":508760,"kb_used_data":433160,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93236392,"statfs":{"total":95995035648,"available":95474065408,"internally_reserved":0,"allocated":443555840,"data_stored":442904188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953581,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916380,"kb_used_data":835212,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":92828772,"statfs":{"total":95995035648,"available":95056662528,"internally_reserved":0,"allocated":855257088,"data_stored":854604442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":721124,"kb_used_data":567764,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93024028,"statfs":{"total":95995035648,"available":95256604672,"internally_reserved":0,"allocated":581390336,"data_stored":580734138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":157022664},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":269736,"kb_used_data":112212,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93475416,"statfs":{"total":95995035648,"available":95718825984,"internally_reserved":0,"allocated":114905088,"data_stored":114248890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569376,"kb_used_data":465996,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93175776,"statfs":{"total":95995035648,"available":95411994624,"internally_reserved":0,"allocated":477179904,"data_stored":476527258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738497,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":645664,"kb_used_data":533964,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93099488,"statfs":{"total":95995035648,"available":95333875712,"internally_reserved":0,"allocated":546779136,"data_stored":546126490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":550895616,"data_stored":550895616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":476512256,"data_stored":476512256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":114819072,"data_stored":114819072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":599785472,"data_stored":599785472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":877789184,"data_stored":877789184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":462684160,"data_stored":462684160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":571539456,"data_stored":571539456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":550895616,"data_stored":550895616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":252575744,"data_stored":252575744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":355139584,"data_stored":355139584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":903020544,"data_stored":903020544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:06:28.526 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-02T10:06:28.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pg_num 31 2024-09-02T10:06:28.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "replica_scrub(pg: 1.2,from:0'0,to:0'0,epoch:104/103,start:1:40000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:50.941506+0000", 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 277.59307286400002, 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.3952948, 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.538 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941506+0000", 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941506+0000", 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941510+0000", 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7570000000000002e-06 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941517+0000", 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0249999999999997e-06 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336499+0000", 2024-09-02T10:06:28.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.39498246199999998 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336515+0000", 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.562e-05 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336678+0000", 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016350699999999999 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336718+0000", 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0278999999999998e-05 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336801+0000", 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2150000000000005e-05 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:55086 2.f e113/92 2:f2dac436:::benchmark_data_smithi187_45882_object55085:head v 113'3428, mlcod=113'3425)", 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.519264+0000", 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 251.0153147, 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24656193800000001, 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.519264+0000", 2024-09-02T10:06:28.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.519264+0000", 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.519265+0000", 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1100000000000005e-07 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.519312+0000", 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7227e-05 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.764322+0000", 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.245010432 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.764329+0000", 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4520000000000002e-06 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.764405+0000", 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6408000000000004e-05 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.764421+0000", 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5994999999999998e-05 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.765794+0000", 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013725709999999999 2024-09-02T10:06:28.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.765826+0000", 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2042000000000003e-05 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127455 2.d 2.f151987d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.256858+0000", 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 163.27772057600001, 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96215398699999999, 2024-09-02T10:06:28.547 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127455 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256858+0000", 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256858+0000", 2024-09-02T10:06:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256860+0000", 2024-09-02T10:06:28.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5039999999999999e-06 2024-09-02T10:06:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256866+0000", 2024-09-02T10:06:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0959999999999997e-06 2024-09-02T10:06:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.552 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256875+0000", 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1940000000000004e-06 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256884+0000", 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3030000000000005e-06 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256938+0000", 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4259999999999999e-05 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256949+0000", 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0624e-05 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.218985+0000", 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96203561699999995 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219012+0000", 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7389000000000002e-05 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.554 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127536 2.d 2.4160433d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:06:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.711015+0000", 2024-09-02T10:06:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.82356339899999, 2024-09-02T10:06:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.50804419700000003, 2024-09-02T10:06:28.555 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127536 2024-09-02T10:06:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711015+0000", 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711015+0000", 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711017+0000", 2024-09-02T10:06:28.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9400000000000001e-06 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711026+0000", 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3729999999999993e-06 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711044+0000", 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8017e-05 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711068+0000", 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4510000000000001e-05 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711155+0000", 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7280999999999993e-05 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711167+0000", 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1497e-05 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219052+0000", 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.50788502000000002 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219059+0000", 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.559e-06 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127544 2.d 2.969bbf1d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:06:28.560 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.788037+0000", 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 162.74654131599999, 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.43104598700000002, 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127544 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788037+0000", 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788037+0000", 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788039+0000", 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0760000000000001e-06 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788049+0000", 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4150000000000005e-06 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788070+0000", 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0809999999999999e-05 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788079+0000", 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8320000000000001e-06 2024-09-02T10:06:28.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788139+0000", 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9336000000000002e-05 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788147+0000", 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7940000000000003e-06 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219079+0000", 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.43093289000000001 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219083+0000", 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8340000000000001e-06 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127455 2.d 2:be198a8f:::benchmark_data_smithi187_45882_object29704:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:47.249583+0000", 2024-09-02T10:06:28.565 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 161.28499518699999, 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30158227999999998, 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127455 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249583+0000", 2024-09-02T10:06:28.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249583+0000", 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249586+0000", 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3070000000000001e-06 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249595+0000", 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3219999999999999e-06 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249611+0000", 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5908000000000001e-05 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249625+0000", 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3954999999999999e-05 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250314+0000", 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00068951199999999998 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250328+0000", 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3358e-05 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.516250+0000", 2024-09-02T10:06:28.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26592224800000003 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.518815+0000", 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025647310000000002 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.518985+0000", 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017049599999999999 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531298+0000", 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012313098 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531372+0000", 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3818000000000004e-05 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.549895+0000", 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018523234999999999 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.549957+0000", 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1729000000000005e-05 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.551166+0000", 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001208563 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127536 2.d 2:bcc20682:::benchmark_data_smithi187_45882_object29785:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:47.249649+0000", 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 161.284929589, 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30254320299999998, 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127536 2024-09-02T10:06:28.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249649+0000", 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249649+0000", 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249650+0000", 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.37e-07 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249654+0000", 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9929999999999997e-06 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249662+0000", 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6139999999999992e-06 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249668+0000", 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8479999999999997e-06 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250366+0000", 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069775700000000002 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250375+0000", 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.285e-06 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.523344+0000", 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27296902099999998 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.523517+0000", 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017299000000000001 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.523683+0000", 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016615299999999999 2024-09-02T10:06:28.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531437+0000", 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007753551 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531462+0000", 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4754000000000001e-05 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.551213+0000", 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019751898 2024-09-02T10:06:28.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.551247+0000", 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3800000000000002e-05 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.552192+0000", 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00094500200000000002 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127544 2.d 2:b8fdd969:::benchmark_data_smithi187_45882_object29793:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:47.249677+0000", 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 161.28490133599999, 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30345529100000002, 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127544 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.580 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249677+0000", 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249677+0000", 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249678+0000", 2024-09-02T10:06:28.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5499999999999998e-07 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249680+0000", 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.17e-06 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249688+0000", 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6000000000000001e-06 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249693+0000", 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0989999999999998e-06 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250429+0000", 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073661199999999999 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250440+0000", 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0710000000000001e-05 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.527337+0000", 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27689745300000002 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.527498+0000", 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160342 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.527654+0000", 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000156021 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.535483+0000", 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078291300000000001 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.537825+0000", 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023424259999999999 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.552232+0000", 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014406672000000001 2024-09-02T10:06:28.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.552266+0000", 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4202000000000002e-05 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.553133+0000", 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00086639900000000001 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.586 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127547 2.d 2:bf1255db:::benchmark_data_smithi187_45882_object29796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:47.249699+0000", 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 161.284879413, 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.34997614900000001, 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127547 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249699+0000", 2024-09-02T10:06:28.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249699+0000", 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249700+0000", 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3000000000000001e-07 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249702+0000", 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.933e-06 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249709+0000", 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3309999999999998e-06 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249714+0000", 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0880000000000002e-06 2024-09-02T10:06:28.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250467+0000", 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075314800000000003 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250476+0000", 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6170000000000007e-06 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531604+0000", 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28112852799999999 2024-09-02T10:06:28.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531742+0000", 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137353 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531881+0000", 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013981700000000001 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.535629+0000", 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037478059999999998 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.538014+0000", 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002384976 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.598001+0000", 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059986314999999998 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.598060+0000", 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9583999999999997e-05 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.599675+0000", 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001615223 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.d 183/157 [PushReplyOp(2:b0969168:::benchmark_data_smithi187_45882_object60943:head)])", 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:36.457663+0000", 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 112.076915704, 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36433320800000002, 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457663+0000", 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457663+0000", 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457664+0000", 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.13e-06 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457670+0000", 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2910000000000004e-06 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.820382+0000", 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.362711703 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.820397+0000", 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5423000000000001e-05 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.820576+0000", 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017897 2024-09-02T10:06:28.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.821996+0000", 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014196910000000001 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163245 2.d 2:b53557ce:::benchmark_data_smithi187_45882_object65494:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027769+0000", 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 91.506809876999995, 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36269144800000003, 2024-09-02T10:06:28.597 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.598 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.598 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:28.598 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:28.599 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.599 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163245 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027769+0000", 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027769+0000", 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027771+0000", 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7020000000000001e-06 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027781+0000", 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6700000000000006e-06 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027798+0000", 2024-09-02T10:06:28.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6416999999999999e-05 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027810+0000", 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2568000000000001e-05 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028091+0000", 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028051500000000001 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028103+0000", 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2768999999999999e-05 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390352+0000", 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36224896000000001 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390460+0000", 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107847 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.603 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163286 2.d 2:b26ad3af:::benchmark_data_smithi187_45882_object65535:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027840+0000", 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 91.506738530999996, 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38364109499999999, 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163286 2024-09-02T10:06:28.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027840+0000", 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027840+0000", 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027841+0000", 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3600000000000003e-07 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027844+0000", 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4170000000000001e-06 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027856+0000", 2024-09-02T10:06:28.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1659e-05 2024-09-02T10:06:28.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027862+0000", 2024-09-02T10:06:28.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5089999999999999e-06 2024-09-02T10:06:28.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028144+0000", 2024-09-02T10:06:28.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028157300000000002 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028154+0000", 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7010000000000008e-06 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390697+0000", 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36254281199999999 2024-09-02T10:06:28.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390784+0000", 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7717000000000001e-05 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390867+0000", 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2432e-05 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.401652+0000", 2024-09-02T10:06:28.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010785041 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.401689+0000", 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7502000000000001e-05 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411205+0000", 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0095160409999999994 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411244+0000", 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8296000000000001e-05 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411481+0000", 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023765900000000001 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.611 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163296 2.d 2:b1ae2ae4:::benchmark_data_smithi187_45882_object65545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 91.506707999, 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38880961400000003, 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163296 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4200000000000002e-07 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027883+0000", 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2315e-05 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027926+0000", 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2339000000000002e-05 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027946+0000", 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0801999999999998e-05 2024-09-02T10:06:28.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028193+0000", 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024658900000000002 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028198+0000", 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6009999999999997e-06 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.394381+0000", 2024-09-02T10:06:28.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36618306299999998 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.394457+0000", 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6509999999999998e-05 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.394532+0000", 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4348e-05 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402216+0000", 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076842819999999997 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405720+0000", 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035040480000000001 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416431+0000", 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010711178999999999 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416463+0000", 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2008999999999998e-05 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416680+0000", 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021718700000000001 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163297 2.d 2:b94d3aa0:::benchmark_data_smithi187_45882_object65546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027961+0000", 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 91.506617348999995, 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38891082999999999, 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163297 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027961+0000", 2024-09-02T10:06:28.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027961+0000", 2024-09-02T10:06:28.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027962+0000", 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8800000000000002e-07 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027967+0000", 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6299999999999997e-06 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027975+0000", 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7819999999999999e-06 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027981+0000", 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8309999999999997e-06 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028219+0000", 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000237774 2024-09-02T10:06:28.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028223+0000", 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5980000000000001e-06 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.397985+0000", 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36976200399999998 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.398112+0000", 2024-09-02T10:06:28.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127565 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.398213+0000", 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100368 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405733+0000", 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0075208200000000001 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411164+0000", 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054306110000000001 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416727+0000", 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055628049999999997 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416766+0000", 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9609e-05 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416872+0000", 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105645 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163303 2.d 2:b483d011:::benchmark_data_smithi187_45882_object65552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 91.506587796999995, 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.39586157599999999, 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.626 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163303 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7800000000000002e-07 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027995+0000", 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3500000000000001e-06 2024-09-02T10:06:28.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028001+0000", 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2859999999999996e-06 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028006+0000", 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6430000000000003e-06 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028242+0000", 2024-09-02T10:06:28.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023680199999999999 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028246+0000", 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2260000000000001e-06 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402251+0000", 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.374004893 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402324+0000", 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3100000000000001e-05 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402391+0000", 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7261999999999994e-05 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411493+0000", 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091022120000000001 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416394+0000", 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049004870000000002 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.423496+0000", 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071020859999999996 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.423554+0000", 2024-09-02T10:06:28.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8511000000000003e-05 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.423852+0000", 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029824000000000002 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163305 2.d 2:bdea317b:::benchmark_data_smithi187_45882_object65554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:06:28.633 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 91.506554924, 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40774826600000003, 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163305 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:06:28.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6199999999999998e-07 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028027+0000", 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9299999999999999e-06 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028034+0000", 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5039999999999999e-06 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028038+0000", 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6739999999999996e-06 2024-09-02T10:06:28.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028260+0000", 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022132900000000001 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028263+0000", 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.163e-06 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405770+0000", 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.377506856 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405877+0000", 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010699799999999999 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405954+0000", 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7732999999999996e-05 2024-09-02T10:06:28.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411501+0000", 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005546212 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.425267+0000", 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013766327 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435398+0000", 2024-09-02T10:06:28.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010131506 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435457+0000", 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8248000000000002e-05 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435772+0000", 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031532400000000002 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.640 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163342 2.d 2:b4bdf525:::benchmark_data_smithi187_45882_object65591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.028055+0000", 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 91.506523384999994, 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40808169100000002, 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.641 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163342 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028055+0000", 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028055+0000", 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028056+0000", 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7999999999999996e-07 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028059+0000", 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1109999999999999e-06 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028065+0000", 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6189999999999996e-06 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028071+0000", 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4410000000000001e-06 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028292+0000", 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221406 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028296+0000", 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3840000000000001e-06 2024-09-02T10:06:28.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.408774+0000", 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38047832599999998 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.408848+0000", 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4216999999999998e-05 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.408915+0000", 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7244999999999997e-05 2024-09-02T10:06:28.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416361+0000", 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074451420000000001 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.425326+0000", 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089654829999999998 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435842+0000", 2024-09-02T10:06:28.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010515531999999999 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435934+0000", 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2621000000000006e-05 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.436137+0000", 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020268400000000001 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:168004 2.d 2:b1d0904c:::benchmark_data_smithi187_45882_object70253:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:04.134944+0000", 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 84.399634410000004, 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23814383, 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168004 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.134944+0000", 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.134944+0000", 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.134945+0000", 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.035e-06 2024-09-02T10:06:28.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.134951+0000", 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6080000000000003e-06 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.134961+0000", 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0526999999999999e-05 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.134968+0000", 2024-09-02T10:06:28.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8109999999999997e-06 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.135033+0000", 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4350000000000006e-05 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.135146+0000", 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011369700000000001 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:06:28.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.135238+0000", 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2087000000000001e-05 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.138796+0000", 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035580389999999998 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.372684+0000", 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23388784800000001 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.372725+0000", 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0593999999999998e-05 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.372792+0000", 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7009999999999997e-05 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:04.373088+0000", 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029622399999999999 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.653 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(2.d e219,198 2:b2e165ef:::benchmark_data_smithi187_45882_object46440:head,171'8780)", 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:45.488577+0000", 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.046001615000002, 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57263216900000002, 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488577+0000", 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488577+0000", 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488578+0000", 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000004e-07 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488584+0000", 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2720000000000003e-06 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.060736+0000", 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57215209199999995 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.060751+0000", 2024-09-02T10:06:28.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5387e-05 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.060841+0000", 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9415999999999999e-05 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.061209+0000", 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036841100000000002 2024-09-02T10:06:28.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(2.d e219,198 2:b1206fd5:::benchmark_data_smithi187_45882_object46449:head,171'8781)", 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:45.492423+0000", 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 43.04215542, 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57615407699999999, 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:28.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492423+0000", 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492423+0000", 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492430+0000", 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6959999999999999e-06 2024-09-02T10:06:28.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492440+0000", 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9299999999999998e-06 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068235+0000", 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57579494799999997 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:28.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068243+0000", 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4330000000000004e-06 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068306+0000", 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2854000000000005e-05 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068577+0000", 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027121600000000001 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.661 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.662 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:28.662 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:28.662 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:06:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:28.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:28.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:28.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:28.789 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:28.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:28.790 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:28.892 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:28.917 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:28.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:28.918 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:29.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:29.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 26 16 37427 37411 89.9234 77.1875 0.0189607 0.0111128 2024-09-02T10:06:29.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:29.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:29.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:29.143 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:06:29.144 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:06:29.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:29.155 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:29.173 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:06:29.174 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:06:29.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:29.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:29.204 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:06:29.204 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:06:29.229 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:06:29.230 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:06:29.258 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:06:29.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:06:29.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:29.284 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:06:29.284 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:06:29.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:29.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:29.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:06:29.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:06:29.343 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:06:29.343 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:06:29.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:06:29.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:06:29.395 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:06:29.395 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:06:29.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:29.420 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:06:29.421 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:06:29.432 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:29.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:29.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:06:29.448 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:06:29.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:06:29.534 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:29.559 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:29.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:29.560 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:29.661 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:29.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:29.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:29.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:29.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:29.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:29.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:29.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:29 smithi162 ceph-mon[20279]: osdmap e239: 12 total, 12 up, 11 in 2024-09-02T10:06:29.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4243094648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:29.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:29 smithi162 ceph-mon[20279]: pgmap v584: 37 pgs: 37 active+clean; 2.1 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-02T10:06:29.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:06:29.862 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:29.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:29.942 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:29.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:30.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 27 16 38997 38981 90.2269 98.125 0.00243151 0.0110748 2024-09-02T10:06:30.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:30.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:30.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:30.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:29 smithi151 ceph-mon[16639]: osdmap e239: 12 total, 12 up, 11 in 2024-09-02T10:06:30.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4243094648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:30.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:29 smithi151 ceph-mon[16639]: pgmap v584: 37 pgs: 37 active+clean; 2.1 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-02T10:06:30.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:30.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:30.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:30.240 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:29 smithi187 ceph-mon[20072]: osdmap e239: 12 total, 12 up, 11 in 2024-09-02T10:06:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4243094648' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:29 smithi187 ceph-mon[20072]: pgmap v584: 37 pgs: 37 active+clean; 2.1 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-02T10:06:30.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:30.330 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:30.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:30.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:30.398 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:30.398 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:30.432 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:30.460 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:30.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:30.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:30.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:30.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:30.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:30.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:30.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:30.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:30.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:30.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:30.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:30.854 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pg_num to 31 2024-09-02T10:06:30.951 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:30.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:30.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:30.978 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:31.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 28 16 40479 40463 90.312 92.625 0.00818333 0.0110597 2024-09-02T10:06:31.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3062293191' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-02T10:06:31.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:31.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:31.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:31.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:31.232 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:31.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3062293191' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-02T10:06:31.310 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3062293191' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"} : dispatch 2024-09-02T10:06:31.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:31.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:06:31.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:31.358 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:31.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:31.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:31.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:31.483 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:31.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:31.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:31.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:31.633 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:31.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:31.729 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:31.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:31.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:31.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:31.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:31.863 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:06:31.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:32.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:32.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:32.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:32.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 29 16 41498 41482 89.3938 63.6875 0.00403934 0.0111772 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3062293191' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: osdmap e240: 12 total, 12 up, 11 in 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:06:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/122649974' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:06:32.110 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3062293191' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-02T10:06:32.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: osdmap e240: 12 total, 12 up, 11 in 2024-09-02T10:06:32.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-02T10:06:32.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:06:32.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:06:32.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:06:32.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:06:32.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:06:32.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:06:32.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:06:32.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/122649974' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:06:32.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:32.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:32.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:32.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3062293191' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "31"}]': finished 2024-09-02T10:06:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: osdmap e240: 12 total, 12 up, 11 in 2024-09-02T10:06:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"} : dispatch 2024-09-02T10:06:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:06:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:06:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:06:32.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:06:32.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:06:32.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:06:32.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:06:32.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/122649974' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:06:32.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:32.267 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:32.323 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:32.324 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:32.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:32.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:32.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:32.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:32.521 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:32.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:32.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:32.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:32.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:32.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:32.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:32.781 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:32.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:32.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:06:32.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:32.907 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:32.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:33.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:33.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 30 16 43068 43052 89.6846 98.125 0.00733083 0.0111421 2024-09-02T10:06:33.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:32 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:33.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-02T10:06:33.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/122649974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:06:33.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:32 smithi162 ceph-mon[20279]: osdmap e241: 12 total, 12 up, 11 in 2024-09-02T10:06:33.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:32 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:06:33.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:32 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:06:33.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:33.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:33.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:06:33.118 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:33.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:33.162 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:33.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:32 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-02T10:06:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/122649974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:06:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:32 smithi151 ceph-mon[16639]: osdmap e241: 12 total, 12 up, 11 in 2024-09-02T10:06:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:32 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:06:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:32 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:06:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:32 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-02T10:06:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/122649974' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:06:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:32 smithi187 ceph-mon[20072]: osdmap e241: 12 total, 12 up, 11 in 2024-09-02T10:06:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:32 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:06:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:32 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:06:33.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:33.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:33.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:33.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:33.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:33.405 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:33.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:33.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:33.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:33.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:33.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:33.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:33.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:33.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:33.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:33.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:33.710 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:06:33.710 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:06:33.733 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:06:33.733 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:06:33.757 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:06:33.758 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:06:33.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:33.783 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:06:33.783 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:06:33.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:33.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:33.810 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:06:33.810 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:06:33.837 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:06:33.838 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:06:33.869 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:06:33.869 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:06:33.896 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:06:33.897 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:06:33.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:33.921 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:06:33.922 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:06:33.928 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:33.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:33.945 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:06:33.945 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:06:33.964 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:06:33.964 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:06:33.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:06:33.964 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:06:33.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:7836 2.2 2:422fedf8:::benchmark_data_smithi187_45882_object7835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.578259+0000", 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 294.38390749199999, 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22181919, 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7836 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:33.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578259+0000", 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578259+0000", 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578259+0000", 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6400000000000001e-07 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578319+0000", 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0126e-05 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578330+0000", 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0740999999999999e-05 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578335+0000", 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9259999999999999e-06 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578412+0000", 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6732999999999999e-05 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578483+0000", 2024-09-02T10:06:33.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1316000000000006e-05 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578564+0000", 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0821000000000004e-05 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.578946+0000", 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038204899999999997 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582380+0000", 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034338699999999999 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.800002+0000", 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21762209299999999 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.800053+0000", 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1387000000000001e-05 2024-09-02T10:06:33.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.800078+0000", 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4363999999999999e-05 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:7841 2.2 2:45ea9ad1:::benchmark_data_smithi187_45882_object7840:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.581408+0000", 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 294.38075833900001, 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22151944800000001, 2024-09-02T10:06:33.971 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7841 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581408+0000", 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581408+0000", 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581408+0000", 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3099999999999997e-07 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581444+0000", 2024-09-02T10:06:33.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5735000000000003e-05 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581456+0000", 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1603999999999999e-05 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581463+0000", 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9750000000000001e-06 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:33.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581508+0000", 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5061000000000001e-05 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581596+0000", 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8367000000000003e-05 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.581677+0000", 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0819e-05 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582132+0000", 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045448199999999998 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799923+0000", 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21779146199999999 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.802865+0000", 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002941777 2024-09-02T10:06:33.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.802896+0000", 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1526999999999998e-05 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.802927+0000", 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0908000000000003e-05 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:7841 2.2 e97/92 ondisk, result = 0)", 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.582308+0000", 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 294.37985824800001, 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21762236700000001, 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:33.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582308+0000", 2024-09-02T10:06:33.979 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:06:33.980 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:06:33.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:33.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582308+0000", 2024-09-02T10:06:33.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582308+0000", 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7899999999999998e-07 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582311+0000", 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0709999999999999e-06 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799863+0000", 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.217551945 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799868+0000", 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6099999999999999e-06 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799910+0000", 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.172e-05 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799918+0000", 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1310000000000006e-06 2024-09-02T10:06:33.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799930+0000", 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2311e-05 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:7836 2.2 e97/92 ondisk, result = 0)", 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.582436+0000", 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 294.379730344, 2024-09-02T10:06:33.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21764771099999999, 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582436+0000", 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582436+0000", 2024-09-02T10:06:33.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582436+0000", 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2300000000000001e-07 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.582440+0000", 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8519999999999997e-06 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:33.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799942+0000", 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21750224000000001 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799949+0000", 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2750000000000001e-06 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799990+0000", 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1003e-05 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.799997+0000", 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2479999999999999e-06 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.800083+0000", 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6570000000000006e-05 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:8321 2.3 2:c249931d:::benchmark_data_smithi187_45882_object8320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:06:33.987 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 294.004131121, 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25867863899999999, 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 8321 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:33.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8100000000000003e-07 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958104+0000", 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8448000000000006e-05 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958111+0000", 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0469999999999996e-06 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958115+0000", 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9689999999999996e-06 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958155+0000", 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0398999999999997e-05 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958225+0000", 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9578000000000004e-05 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958316+0000", 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1249000000000007e-05 2024-09-02T10:06:33.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958733+0000", 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041638500000000002 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.959078+0000", 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000345855 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216656+0000", 2024-09-02T10:06:33.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25757725999999997 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216689+0000", 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3368999999999998e-05 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216714+0000", 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4598999999999999e-05 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:8321 2.3 e97/92 ondisk, result = 0)", 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 294.003271267, 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25782361500000001, 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:06:33.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1600000000000001e-07 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:33.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958898+0000", 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.066e-06 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216621+0000", 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25772234500000002 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216625+0000", 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2989999999999998e-06 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216650+0000", 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4939e-05 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216653+0000", 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1559999999999999e-06 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216718+0000", 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5294000000000002e-05 2024-09-02T10:06:33.997 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:22420 2.2 2:4520d8a5:::benchmark_data_smithi187_45882_object22419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:51.202853+0000", 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 282.75931264000002, 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21069840400000001, 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:33.998 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 22420 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202853+0000", 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202853+0000", 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:33.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202854+0000", 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5899999999999995e-07 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202879+0000", 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4481000000000002e-05 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202890+0000", 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1116e-05 2024-09-02T10:06:34.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202895+0000", 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5670000000000002e-06 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.202947+0000", 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1085999999999998e-05 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.203066+0000", 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119224 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.203222+0000", 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015642 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.203968+0000", 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000745661 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.204233+0000", 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026555900000000001 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.413490+0000", 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20925611899999999 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.413523+0000", 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3667000000000001e-05 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.413552+0000", 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8645000000000001e-05 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:54905 2.2 2:470b2221:::benchmark_data_smithi187_45882_object54904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.133024+0000", 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 256.82914205100002, 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26483498300000002, 2024-09-02T10:06:34.004 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54905 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133024+0000", 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133024+0000", 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133026+0000", 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.517e-06 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133067+0000", 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1332999999999999e-05 2024-09-02T10:06:34.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133081+0000", 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4272e-05 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133088+0000", 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3600000000000001e-06 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133138+0000", 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0037999999999998e-05 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133240+0000", 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102081 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133332+0000", 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2132999999999994e-05 2024-09-02T10:06:34.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.134852+0000", 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001519963 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.135073+0000", 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022145800000000001 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.397790+0000", 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26271683099999998 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.397827+0000", 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7119000000000002e-05 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.397859+0000", 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1878000000000002e-05 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:54934 2.3 2:c516e2ca:::benchmark_data_smithi187_45882_object54933:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.163207+0000", 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 256.79895860699997, 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21258112400000001, 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54934 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.011 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163207+0000", 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163207+0000", 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163208+0000", 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9100000000000003e-07 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163270+0000", 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1891999999999996e-05 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163278+0000", 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7600000000000002e-06 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.163282+0000", 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6820000000000002e-06 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.164696+0000", 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001413231 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.164791+0000", 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5189e-05 2024-09-02T10:06:34.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.164911+0000", 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119924 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.165973+0000", 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010621020000000001 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.173396+0000", 2024-09-02T10:06:34.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074228159999999996 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.375677+0000", 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20228159500000001 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.375737+0000", 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0074000000000001e-05 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.375789+0000", 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1168000000000002e-05 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:54937 2.3 2:c9a686f1:::benchmark_data_smithi187_45882_object54936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 256.79074752299999, 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23343879200000001, 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.017 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54937 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1500000000000003e-07 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171466+0000", 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6523000000000002e-05 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171477+0000", 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0784e-05 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171482+0000", 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2079999999999999e-06 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171521+0000", 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8736999999999999e-05 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171596+0000", 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5042000000000004e-05 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171675+0000", 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9369999999999994e-05 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.173027+0000", 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013518409999999999 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.022 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:06:34.023 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:06:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.273746+0000", 2024-09-02T10:06:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.100718883 2024-09-02T10:06:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.404789+0000", 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13104327700000001 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.404822+0000", 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2811000000000001e-05 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.404857+0000", 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5401000000000002e-05 2024-09-02T10:06:34.024 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:55038 2.3 2:c94b451a:::benchmark_data_smithi187_45882_object55037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.453569+0000", 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 256.50859662400001, 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33233287500000003, 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.025 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55038 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453569+0000", 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453569+0000", 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453571+0000", 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.702e-06 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453616+0000", 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5033000000000001e-05 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453635+0000", 2024-09-02T10:06:34.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8595000000000001e-05 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453644+0000", 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0100000000000001e-06 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453705+0000", 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0816000000000002e-05 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453835+0000", 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013086799999999999 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453981+0000", 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014516900000000001 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.455854+0000", 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001873537 2024-09-02T10:06:34.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.456078+0000", 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000224083 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.785794+0000", 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32971594999999998 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.785853+0000", 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9083999999999999e-05 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.785902+0000", 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9027999999999999e-05 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.031 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:79038 2.3 2:c3e1c878:::benchmark_data_smithi187_45882_object79037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:40.868535+0000", 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 233.093631515, 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21948013999999999, 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 79038 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868535+0000", 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868535+0000", 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868536+0000", 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.514e-06 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868561+0000", 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5130000000000002e-05 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868580+0000", 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8865e-05 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868589+0000", 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3370000000000006e-06 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868656+0000", 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6608999999999999e-05 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.868840+0000", 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018391300000000001 2024-09-02T10:06:34.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.869015+0000", 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000175093 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.871861+0000", 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00284636 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.875704+0000", 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038424940000000001 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.087928+0000", 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21222416399999999 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.087981+0000", 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3270999999999999e-05 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.088015+0000", 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3389999999999997e-05 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:79041 2.3 2:ce3f4927:::benchmark_data_smithi187_45882_object79040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:40.872042+0000", 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 233.09012427799999, 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21882871000000001, 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.038 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 79041 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872042+0000", 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872042+0000", 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872046+0000", 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2780000000000004e-06 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872089+0000", 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2681e-05 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872102+0000", 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2977e-05 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872108+0000", 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7919999999999998e-06 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872159+0000", 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1678000000000002e-05 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872302+0000", 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014248200000000001 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.872462+0000", 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016049400000000001 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.875431+0000", 2024-09-02T10:06:34.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002969143 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:40.875828+0000", 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00039666099999999998 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.090754+0000", 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.214925543 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.090820+0000", 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6431000000000001e-05 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:41.090871+0000", 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0550000000000002e-05 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:97742 2.2 2:492b4420:::benchmark_data_smithi187_45882_object97741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:05.413372+0000", 2024-09-02T10:06:34.044 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 208.548793913, 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26451705399999997, 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 97742 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413372+0000", 2024-09-02T10:06:34.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413372+0000", 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413373+0000", 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0200000000000001e-07 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413450+0000", 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6959000000000006e-05 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413460+0000", 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.842e-06 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413465+0000", 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8749999999999999e-06 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413516+0000", 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1252000000000003e-05 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413625+0000", 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010952500000000001 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413745+0000", 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011921000000000001 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.416627+0000", 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028824359999999999 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.427846+0000", 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011219084000000001 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677815+0000", 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.249968879 2024-09-02T10:06:34.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677858+0000", 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3084000000000003e-05 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677889+0000", 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1106000000000003e-05 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:141512 2.3 2:c7378ca5:::benchmark_data_smithi187_45882_object43761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:14.693802+0000", 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 139.26836368799999, 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37122683899999998, 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141512 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693802+0000", 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693802+0000", 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693803+0000", 2024-09-02T10:06:34.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2500000000000005e-07 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693807+0000", 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.681e-06 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693815+0000", 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9140000000000008e-06 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693821+0000", 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3090000000000001e-06 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693879+0000", 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7862000000000003e-05 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693996+0000", 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116896 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694103+0000", 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010771400000000001 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.698406+0000", 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043025820000000001 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.703236+0000", 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048299390000000001 2024-09-02T10:06:34.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.064923+0000", 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36168711999999997 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.064970+0000", 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7067000000000002e-05 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.065029+0000", 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9030000000000002e-05 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:156215 2.2 2:47721b3f:::benchmark_data_smithi187_45882_object58464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:43.294501+0000", 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 110.667665051, 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27888434200000001, 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.057 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 156215 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294501+0000", 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294501+0000", 2024-09-02T10:06:34.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294503+0000", 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.525e-06 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294509+0000", 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.905e-06 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294527+0000", 2024-09-02T10:06:34.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7744000000000002e-05 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294534+0000", 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0160000000000003e-06 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294588+0000", 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3582000000000003e-05 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294669+0000", 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1450000000000001e-05 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294743+0000", 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3330000000000004e-05 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.298500+0000", 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037571219999999999 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.298571+0000", 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1128000000000001e-05 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573304+0000", 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27473348800000003 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573343+0000", 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8288999999999999e-05 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573385+0000", 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2762999999999997e-05 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:156215 2.2 e188/92 ondisk, result = 0)", 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:43.299839+0000", 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 110.662326663, 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27355232099999999, 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.063 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.299839+0000", 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.299839+0000", 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.299840+0000", 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3799999999999997e-07 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.299845+0000", 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8999999999999997e-06 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573233+0000", 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27338810800000002 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573241+0000", 2024-09-02T10:06:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7950000000000008e-06 2024-09-02T10:06:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573293+0000", 2024-09-02T10:06:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2540999999999997e-05 2024-09-02T10:06:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573301+0000", 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3900000000000004e-06 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573392+0000", 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1049000000000002e-05 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.067 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:195489 2.3 2:c2cfeaf0:::benchmark_data_smithi187_45882_object97738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:43.024659+0000", 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 50.937507508000003, 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298585832, 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 195489 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024659+0000", 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024659+0000", 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024660+0000", 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000001e-07 2024-09-02T10:06:34.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024664+0000", 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5120000000000002e-06 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024676+0000", 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2391e-05 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024683+0000", 2024-09-02T10:06:34.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2480000000000001e-06 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.028947+0000", 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042639560000000002 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029054+0000", 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107088 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:06:34.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029120+0000", 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5988999999999994e-05 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.033127+0000", 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040077639999999996 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.038683+0000", 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055557139999999998 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.323168+0000", 2024-09-02T10:06:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.284485031 2024-09-02T10:06:34.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.323211+0000", 2024-09-02T10:06:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3260999999999998e-05 2024-09-02T10:06:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.323244+0000", 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2928000000000001e-05 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:12911 3.5 3:a2419a0b:::benchmark_data_smithi187_59989_object12910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:06:11.043329+0000", 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.918837492000002, 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22769582599999999, 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.077 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 12911 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043329+0000", 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043329+0000", 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043330+0000", 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0380000000000001e-06 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043373+0000", 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3662000000000003e-05 2024-09-02T10:06:34.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043384+0000", 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0804e-05 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043391+0000", 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8630000000000002e-06 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043432+0000", 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0735999999999998e-05 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043511+0000", 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9149999999999999e-05 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043594+0000", 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2744000000000006e-05 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.044220+0000", 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00062675399999999998 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.176189+0000", 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.131968694 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.270931+0000", 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094741994999999996 2024-09-02T10:06:34.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.270981+0000", 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0244000000000002e-05 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.271024+0000", 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3142000000000001e-05 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:17982 3.5 3:aeaaefe5:::benchmark_data_smithi187_59989_object17981:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:06:14.771734+0000", 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 19.190431696000001, 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20946442700000001, 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 17982 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.771734+0000", 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.771734+0000", 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.771735+0000", 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7700000000000004e-07 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.771795+0000", 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9849000000000002e-05 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.771806+0000", 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.131e-05 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.771811+0000", 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1889999999999998e-06 2024-09-02T10:06:34.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.771851+0000", 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9860999999999997e-05 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.771934+0000", 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2383e-05 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.772016+0000", 2024-09-02T10:06:34.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2812999999999995e-05 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.772865+0000", 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00084833700000000001 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.773175+0000", 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031034699999999999 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.981102+0000", 2024-09-02T10:06:34.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20792670999999999 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.981152+0000", 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0337999999999998e-05 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:14.981199+0000", 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6613000000000002e-05 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.089 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:34.090 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:34.090 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:06:34.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:34.091 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 31 15 44351 44336 89.38 80.25 0.0164056 0.0111789 2024-09-02T10:06:34.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: osdmap e242: 12 total, 12 up, 11 in 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: pgmap v589: 47 pgs: 3 peering, 8 unknown, 36 active+clean; 2.4 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:06:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:06:34.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:06:34.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:06:34.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1952607618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:34.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-02T10:06:34.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:33 smithi162 ceph-mon[20279]: osdmap e243: 12 total, 12 up, 11 in 2024-09-02T10:06:34.113 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:34.114 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: osdmap e242: 12 total, 12 up, 11 in 2024-09-02T10:06:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:06:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:06:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:06:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:06:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:06:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:06:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:06:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:06:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: pgmap v589: 47 pgs: 3 peering, 8 unknown, 36 active+clean; 2.4 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-02T10:06:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-02T10:06:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:06:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:06:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:06:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:06:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1952607618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-02T10:06:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:33 smithi151 ceph-mon[16639]: osdmap e243: 12 total, 12 up, 11 in 2024-09-02T10:06:34.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:34.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:34.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: osdmap e242: 12 total, 12 up, 11 in 2024-09-02T10:06:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:06:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:06:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:06:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:06:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: pgmap v589: 47 pgs: 3 peering, 8 unknown, 36 active+clean; 2.4 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"} : dispatch 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1952607618' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-02T10:06:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:33 smithi187 ceph-mon[20072]: osdmap e243: 12 total, 12 up, 11 in 2024-09-02T10:06:34.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:34.364 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:34.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:34.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:34.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:34.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:34.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:34.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:34.617 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:34.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:34.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:34.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:34.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:34.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:34.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:34.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:34.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:06:34.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:06:34.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:34.999 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:35.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:35.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 32 16 45529 45513 88.8855 73.5625 0.00242719 0.0112413 2024-09-02T10:06:35.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:35.129 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:35.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:34 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:06:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:34 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:06:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:34 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:06:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:34 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:06:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:34 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:06:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:34 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:06:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:34 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:06:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:34 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:06:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:34 smithi151 ceph-mon[16639]: osdmap e244: 12 total, 12 up, 11 in 2024-09-02T10:06:35.231 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:34 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:06:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:34 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:06:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:34 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:06:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:34 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:06:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:34 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:06:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:34 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:06:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:34 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:06:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:34 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:06:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:34 smithi187 ceph-mon[20072]: osdmap e244: 12 total, 12 up, 11 in 2024-09-02T10:06:35.257 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:35.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:35.259 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:35.286 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:35.337 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:34 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:06:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:34 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:06:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:34 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:06:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:34 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:06:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:34 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:06:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:34 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:06:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:34 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:06:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:34 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:06:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:34 smithi162 ceph-mon[20279]: osdmap e244: 12 total, 12 up, 11 in 2024-09-02T10:06:35.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:35.390 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:35.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:35.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:35.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:35.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:35.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:35.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:35.639 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:35.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:35.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:35.765 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:35.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:35.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:35.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:35.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:35.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:36.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 33 16 46815 46799 88.6274 80.375 0.0134404 0.0112781 2024-09-02T10:06:36.028 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:36.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:36.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:06:36.065 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: pgmap v592: 47 pgs: 7 peering, 5 unknown, 35 active+clean; 2.6 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 767 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:06:36.066 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:35 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:46867 3.a 3:53f219c4:::benchmark_data_smithi187_59989_object46866:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:06:36.060880+0000", 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0031862079999999998, 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032602769999999998, 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:36.067 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 46867 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.060880+0000", 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:36.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.060880+0000", 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.060891+0000", 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1467e-05 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.060924+0000", 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2943e-05 2024-09-02T10:06:36.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.060944+0000", 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9731000000000001e-05 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.060952+0000", 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4200000000000007e-06 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:36.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.061038+0000", 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5708e-05 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.061184+0000", 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014588399999999999 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.061322+0000", 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000138615 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.063294+0000", 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019719989999999999 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.063457+0000", 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000162442 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:46849 3.c e244/226 3:3b4e5173:::benchmark_data_smithi187_59989_object46848:head v 244'2938, mlcod=244'2935)", 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:06:36.052065+0000", 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.012000937999999999, 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012163396, 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.052065+0000", 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:36.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.052065+0000", 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.052066+0000", 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3200000000000004e-07 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.052101+0000", 2024-09-02T10:06:36.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5274999999999997e-05 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.063004+0000", 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010903123000000001 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.063011+0000", 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5710000000000003e-06 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.063070+0000", 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9367999999999998e-05 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.063090+0000", 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0146999999999999e-05 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-02T10:06:36.076 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:06:36.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:36.158 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:36.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:06:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:06:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:06:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:06:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:06:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:06:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:06:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:06:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:06:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:06:36.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: pgmap v592: 47 pgs: 7 peering, 5 unknown, 35 active+clean; 2.6 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 767 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-09-02T10:06:36.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:06:36.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:06:36.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:06:36.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:35 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:06:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:06:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:06:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:06:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:06:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:06:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:06:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:06:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:06:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:06:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:06:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: pgmap v592: 47 pgs: 7 peering, 5 unknown, 35 active+clean; 2.6 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 767 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-09-02T10:06:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:06:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:06:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:06:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:35 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:06:36.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:36.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:36.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:36.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:36.414 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:36.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:36.516 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:36.537 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:36.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:36.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:36.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:36.660 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:36.660 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:36.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:06:36.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:36.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:36.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:36.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:36.913 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:36.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:36.921 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:06:37.015 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:37.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 34 16 47929 47913 88.0684 69.625 0.00434219 0.0113461 2024-09-02T10:06:37.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:37.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:37.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:37.056 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:37.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:06:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:06:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:06:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:06:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:06:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:06:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-02T10:06:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:06:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:06:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2200241517' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:36 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:37.117 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:06:37.117 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:06:37.117 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-02T10:06:37.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-02T10:06:37.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:37.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:37.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:37.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:37.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:06:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:06:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:06:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:06:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:06:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:06:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-02T10:06:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:06:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:06:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2200241517' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:37.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:36 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:06:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:06:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:06:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:06:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:06:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:06:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-09-02T10:06:37.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:06:37.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:06:37.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2200241517' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:37.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:36 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:37.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:06:37.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:37.311 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:37.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:37.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:37.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:37.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:37.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:37.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:37.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:37.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:37.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:37.699 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:06:37.699 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:06:37.700 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:06:37.700 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:06:37.700 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:06:37.701 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:37.702 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:37.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:37.827 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:37.833 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:37.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:37.869 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:37.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:37.965 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:37.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:37.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:37 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:06:37.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:37 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:06:37.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:37 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:06:37.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:37 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:37.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:37 smithi151 ceph-mon[16639]: osdmap e245: 12 total, 12 up, 11 in 2024-09-02T10:06:37.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:37 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:06:37.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:37 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:06:37.998 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:37 smithi151 ceph-mon[16639]: pgmap v594: 47 pgs: 5 peering, 1 unknown, 41 active+clean; 2.8 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 719 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-02T10:06:38.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 35 15 49270 49255 87.9484 83.875 0.00553958 0.0113646 2024-09-02T10:06:38.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:38.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:38.097 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:38.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:38.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:38.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:38.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:37 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:06:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:37 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:06:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:37 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:06:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:37 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:37 smithi187 ceph-mon[20072]: osdmap e245: 12 total, 12 up, 11 in 2024-09-02T10:06:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:37 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:06:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:37 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:06:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:37 smithi187 ceph-mon[20072]: pgmap v594: 47 pgs: 5 peering, 1 unknown, 41 active+clean; 2.8 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 719 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-02T10:06:38.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:38.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:37 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:06:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:37 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:06:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:37 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:06:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:37 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:37 smithi162 ceph-mon[20279]: osdmap e245: 12 total, 12 up, 11 in 2024-09-02T10:06:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:37 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:06:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:37 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:06:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:37 smithi162 ceph-mon[20279]: pgmap v594: 47 pgs: 5 peering, 1 unknown, 41 active+clean; 2.8 GiB data, 9.6 GiB used, 974 GiB / 983 GiB avail; 719 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-02T10:06:38.354 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:38.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:38.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:06:38.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:38.492 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:38.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:38.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:38.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:38.618 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:38.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:38.619 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:38.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:38.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:38.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:38.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:38.751 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:38.761 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:06:38.761 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:06:38.761 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:06:38.761 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:06:38.761 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:06:38.761 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:06:38.761 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:06:38.761 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:06:38.762 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:06:38.762 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:06:38.762 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:06:38.762 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:06:38.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:38.869 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:38.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:38.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:38.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:39.000 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:39.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:39.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 36 16 50628 50612 87.8611 84.8125 0.00323315 0.0113756 2024-09-02T10:06:39.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:39.127 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:39.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:39.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2846234060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:39.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:39.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2846234060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:39.258 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:39.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:39.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2846234060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:39.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:39.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:39.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:39.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:39.517 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:39.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:06:39.618 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:06:39.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-02T10:06:39.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:39.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:39.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:39.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:39.713 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:06:39.713 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:06:39.713 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:06:39.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:06:39.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:06:39.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:06:39.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:06:39.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:06:39.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:06:39.715 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:06:39.715 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:06:39.715 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:06:39.715 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:06:39.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:06:39.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:06:39.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:06:39.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:06:39.716 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:06:39.716 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:06:39.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:06:39.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:06:39.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:06:39.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:06:39.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:06:39.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:39.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:39.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:39.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:06:39.863 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:39.863 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:39.876 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:39.903 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:39.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:39.904 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:40.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:40.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 37 16 52248 52232 88.2228 101.25 0.00269596 0.0113272 2024-09-02T10:06:40.032 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:40.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:40.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:06:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:06:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:39 smithi162 ceph-mon[20279]: pgmap v595: 47 pgs: 4 peering, 43 active+clean; 2.9 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 511 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-02T10:06:40.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:40.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:40.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:40.160 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:40.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:06:40.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:06:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:39 smithi151 ceph-mon[16639]: pgmap v595: 47 pgs: 4 peering, 43 active+clean; 2.9 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 511 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-02T10:06:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:06:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:06:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:39 smithi187 ceph-mon[20072]: pgmap v595: 47 pgs: 4 peering, 43 active+clean; 2.9 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 511 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-02T10:06:40.260 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:40.283 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:40.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:40.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:40.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:40.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:40.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:40.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:40.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:40.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:40.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:06:40.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:40.654 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:40.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:40.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:40.777 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:40.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:40.778 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:40.816 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:40.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:40.899 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:40.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:41.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:41.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:41.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 38 16 52986 52970 87.1149 46.125 0.00438596 0.0114732 2024-09-02T10:06:41.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:41.027 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:41.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:41.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:41.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:40 smithi151 ceph-mon[16639]: osdmap e246: 12 total, 12 up, 11 in 2024-09-02T10:06:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:40 smithi187 ceph-mon[20072]: osdmap e246: 12 total, 12 up, 11 in 2024-09-02T10:06:41.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:41.282 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:41.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:41.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:40 smithi162 ceph-mon[20279]: osdmap e246: 12 total, 12 up, 11 in 2024-09-02T10:06:41.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:41.410 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:41.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:41.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:41.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:41.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:41.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:41.666 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:41.667 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:41.722 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:41.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:41.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:41.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:41.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:41.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:41.917 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:41.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:41.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:42.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:42.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 39 16 54217 54201 86.8538 76.9375 0.00541135 0.0115079 2024-09-02T10:06:42.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:42.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:42.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:42.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:42.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:42.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:42.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:41 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:06:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:41 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:06:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:41 smithi151 ceph-mon[16639]: pgmap v597: 47 pgs: 47 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-02T10:06:42.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:41 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:06:42.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:41 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:06:42.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:41 smithi187 ceph-mon[20072]: pgmap v597: 47 pgs: 47 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-02T10:06:42.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:42.305 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:42.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:42.307 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:42.344 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:06:42.344 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:06:42.344 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:06:42.344 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:06:42.344 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:06:42.344 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:06:42.345 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:06:42.345 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:06:42.345 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:06:42.345 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:06:42.345 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:06:42.345 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:06:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:41 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:06:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:41 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:06:42.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:41 smithi162 ceph-mon[20279]: pgmap v597: 47 pgs: 47 active+clean; 3.0 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-09-02T10:06:42.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:42.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:42.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:42.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:42.556 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:42.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:42.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:42.692 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:42.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:42.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:42.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:42.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:42.870 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:42.871 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:42.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:42.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:42.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:43.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:06:43.026558+0000 min lat: 0.0010322 max lat: 0.2764 avg lat: 0.0115608 2024-09-02T10:06:43.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:06:43.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 40 16 55344 55328 86.4432 70.4375 0.0122381 0.0115608 2024-09-02T10:06:43.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:42 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:06:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:42 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:06:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:42 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-09-02T10:06:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:42 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:06:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:42 smithi162 ceph-mon[20279]: osdmap e247: 12 total, 12 up, 11 in 2024-09-02T10:06:43.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:43.117 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:43.118 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:43.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:42 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:06:43.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:42 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:06:43.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:42 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-09-02T10:06:43.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:42 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:06:43.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:42 smithi151 ceph-mon[16639]: osdmap e247: 12 total, 12 up, 11 in 2024-09-02T10:06:43.219 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:43.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:43.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:43.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:42 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:06:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:42 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:06:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:42 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-09-02T10:06:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:42 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:06:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:42 smithi187 ceph-mon[20072]: osdmap e247: 12 total, 12 up, 11 in 2024-09-02T10:06:43.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:43.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:43.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:43.479 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:43.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:43.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:43.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:06:43.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:43.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:43.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:43.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:06:43.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:43.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:43.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:43.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:43.870 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:43.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:43.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:43.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:43.936 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:06:43.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:43 smithi151 ceph-mon[16639]: pgmap v599: 47 pgs: 47 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-02T10:06:43.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1243929249' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:06:43.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:06:43.995 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:06:43.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-02T10:06:44.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:44.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:44.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:44.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 41 16 56629 56613 86.2935 80.3125 0.00444849 0.011582 2024-09-02T10:06:44.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:44.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:44.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:44.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:43 smithi187 ceph-mon[20072]: pgmap v599: 47 pgs: 47 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-02T10:06:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1243929249' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:06:44.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:44.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:44.289 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:43 smithi162 ceph-mon[20279]: pgmap v599: 47 pgs: 47 active+clean; 3.2 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-02T10:06:44.289 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1243929249' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:06:44.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:44.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:44.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:44.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:44.427 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:44.427 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:44.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:44.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:44.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:44.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:44.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:44.654 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:44.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:44.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:44.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:44.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:44.779 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:44.797 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:44.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:44.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:44.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:45.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:45.024 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:45.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:45.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:45.026 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 42 16 57637 57621 85.7386 63 0.0159655 0.0116403 2024-09-02T10:06:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:44 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1243929249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:06:45.071 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:44 smithi151 ceph-mon[16639]: osdmap e248: 12 total, 12 up, 11 in 2024-09-02T10:06:45.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:06:45.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:45.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:45.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:44 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1243929249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:06:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:44 smithi187 ceph-mon[20072]: osdmap e248: 12 total, 12 up, 11 in 2024-09-02T10:06:45.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:45.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:45.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:45.339 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:45.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:44 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:45.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1243929249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:06:45.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:44 smithi162 ceph-mon[20279]: osdmap e248: 12 total, 12 up, 11 in 2024-09-02T10:06:45.372 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:06:45.372 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:06:45.401 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:06:45.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:06:45.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:45.429 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:06:45.429 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:06:45.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:45.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:45.456 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:06:45.456 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:06:45.466 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:06:45.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:45.559 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:45.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:45.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:45.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:45.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:45.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:45.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:45.725 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:06:45.725 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:06:45.747 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:06:45.747 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:06:45.770 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:06:45.770 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:06:45.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:45.793 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:06:45.794 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:06:45.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:45.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:45.816 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:06:45.816 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:06:45.844 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:06:45.844 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:06:45.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:06:45.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:06:45.900 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:06:45.900 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:06:45.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:45.914 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:06:45.914 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:06:45.915 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:06:45.915 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:06:45.915 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:06:45.915 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:06:45.915 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:06:45.915 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:06:45.915 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:06:45.915 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:06:45.915 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:06:45.915 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:06:45.937 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:45.938 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:46.006 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:06:46.006 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:06:46.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:06:46.006 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:06:46.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:55055 2.8 2:1127c854:::benchmark_data_smithi187_45882_object55054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.471137+0000", 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 268.53191484000001, 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28183072799999997, 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55055 2024-09-02T10:06:46.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.008 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471137+0000", 2024-09-02T10:06:46.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471137+0000", 2024-09-02T10:06:46.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471138+0000", 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0070000000000001e-06 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471192+0000", 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3325000000000002e-05 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471202+0000", 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0468999999999999e-05 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471207+0000", 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8720000000000001e-06 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471242+0000", 2024-09-02T10:06:46.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4702000000000001e-05 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471322+0000", 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0093999999999996e-05 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.471411+0000", 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9015000000000001e-05 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:46.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.472901+0000", 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014899150000000001 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473274+0000", 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037354899999999998 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752887+0000", 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27961233699999999 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752928+0000", 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1560999999999997e-05 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752968+0000", 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9882000000000002e-05 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:55055 2.8 e113/92 ondisk, result = 0)", 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.473073+0000", 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 268.529979266, 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27990168399999998, 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473073+0000", 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473073+0000", 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473074+0000", 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7100000000000002e-07 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.473076+0000", 2024-09-02T10:06:46.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4549999999999998e-06 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752722+0000", 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27964593100000001 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752743+0000", 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1087e-05 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752872+0000", 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012929900000000001 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752881+0000", 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9339999999999995e-06 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.752975+0000", 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3406999999999995e-05 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.017 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:55067 2.8 2:1ce73561:::benchmark_data_smithi187_45882_object55066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.485554+0000", 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 268.51749858599999, 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27014719300000001, 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55067 2024-09-02T10:06:46.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485554+0000", 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485554+0000", 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485556+0000", 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.885e-06 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485587+0000", 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1435e-05 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485605+0000", 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7668000000000001e-05 2024-09-02T10:06:46.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485615+0000", 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0302999999999999e-05 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485678+0000", 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2840000000000001e-05 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485807+0000", 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128947 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.485908+0000", 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101378 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487422+0000", 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001514236 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487766+0000", 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034318 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755648+0000", 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26788223500000002 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755679+0000", 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0703000000000001e-05 2024-09-02T10:06:46.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755701+0000", 2024-09-02T10:06:46.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2382999999999999e-05 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:55067 2.8 e113/92 ondisk, result = 0)", 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.487535+0000", 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 268.51551761000002, 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26817209600000003, 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:06:46.025 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487535+0000", 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487535+0000", 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487536+0000", 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1379999999999999e-06 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.487541+0000", 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3959999999999997e-06 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755584+0000", 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26804281099999999 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755590+0000", 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4399999999999996e-06 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755640+0000", 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0973999999999997e-05 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755645+0000", 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2289999999999997e-06 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.755707+0000", 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2107999999999996e-05 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67245 2.0 2:0a07d6c2:::benchmark_data_smithi187_45882_object67244:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 256.75138025400003, 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25464746599999999, 2024-09-02T10:06:46.029 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67245 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251672+0000", 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1699999999999999e-07 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251698+0000", 2024-09-02T10:06:46.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5471999999999999e-05 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251705+0000", 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.81e-06 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251710+0000", 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.169e-06 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251741+0000", 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0579e-05 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251744+0000", 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4489999999999999e-06 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.495787+0000", 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.244043118 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.495917+0000", 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000130099 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.496029+0000", 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011203499999999999 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505241+0000", 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0092117080000000007 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505301+0000", 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0078000000000003e-05 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505348+0000", 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6650999999999998e-05 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.505388+0000", 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9974e-05 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.036 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 43 16 58660 58644 85.2316 63.9375 0.0503799 0.011726 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.506320+0000", 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00093190700000000005 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:67250 2.0 2:003e299c:::benchmark_data_smithi187_45882_object67249:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e119)", 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 256.75133150599999, 2024-09-02T10:06:46.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26035086499999999, 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 67250 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251721+0000", 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.51e-07 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251748+0000", 2024-09-02T10:06:46.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6327000000000001e-05 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251766+0000", 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8689e-05 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251772+0000", 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6409999999999999e-06 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251811+0000", 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9113000000000003e-05 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.251818+0000", 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2339999999999997e-06 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.500859+0000", 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.249040962 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.500962+0000", 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103015 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.501075+0000", 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113211 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.506329+0000", 2024-09-02T10:06:46.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052531840000000002 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.506377+0000", 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7914999999999998e-05 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.510244+0000", 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038674730000000002 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:46.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.510281+0000", 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6504000000000002e-05 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:29.512072+0000", 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001791146 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127423 2.0 2:06eddedb:::benchmark_data_smithi187_45882_object29672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-02T10:06:46.044 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.199335+0000", 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.80371685599999, 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0271604599999999, 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127423 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199335+0000", 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199335+0000", 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199337+0000", 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.564e-06 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199343+0000", 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7760000000000003e-06 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199354+0000", 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1321000000000001e-05 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199362+0000", 2024-09-02T10:06:46.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6040000000000004e-06 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199422+0000", 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0145000000000001e-05 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199562+0000", 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013979699999999999 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199712+0000", 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015045799999999999 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.209645+0000", 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0099324600000000006 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.217410+0000", 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077656410000000002 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232716+0000", 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015306024 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232726+0000", 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3230000000000004e-06 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226479+0000", 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99375334900000001 2024-09-02T10:06:46.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226496+0000", 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6997999999999999e-05 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127440 2.0 2.9f94cd30 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.78342269199999, 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0069557170000001, 2024-09-02T10:06:46.051 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127440 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.63e-07 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219633+0000", 2024-09-02T10:06:46.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.818e-06 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219639+0000", 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8000000000000004e-06 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219645+0000", 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3980000000000004e-06 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232802+0000", 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013156805000000001 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232812+0000", 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0394000000000001e-05 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226575+0000", 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99376249500000002 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226585+0000", 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0543999999999999e-05 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127423 2.0 2.db7bb760 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.275687+0000", 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.72736503600001, 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95098331899999999, 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.056 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127423 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275687+0000", 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275687+0000", 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275689+0000", 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5829999999999999e-06 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275696+0000", 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1359999999999999e-06 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275707+0000", 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1024e-05 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275715+0000", 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9750000000000007e-06 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279013+0000", 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003297688 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:46.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279020+0000", 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0790000000000003e-06 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226644+0000", 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94762439200000004 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226671+0000", 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6441999999999999e-05 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127440 2.0 2.9f94cd30 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.72732146999999, 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95100121400000004, 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127440 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6499999999999999e-07 2024-09-02T10:06:46.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275734+0000", 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2299999999999998e-06 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275740+0000", 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8970000000000002e-06 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275745+0000", 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.814e-06 2024-09-02T10:06:46.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279036+0000", 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003290837 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279041+0000", 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4769999999999997e-06 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226722+0000", 2024-09-02T10:06:46.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94768102099999996 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226732+0000", 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0473e-05 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127464 2.0 2.e26bb170 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.277404+0000", 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.72564874, 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94936489599999996, 2024-09-02T10:06:46.065 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127464 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277404+0000", 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277404+0000", 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277405+0000", 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4799999999999997e-07 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277410+0000", 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0529999999999997e-06 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277419+0000", 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6420000000000002e-06 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277424+0000", 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.831e-06 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279299+0000", 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018751639999999999 2024-09-02T10:06:46.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279303+0000", 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.143e-06 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226761+0000", 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94745774000000005 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226768+0000", 2024-09-02T10:06:46.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3749999999999997e-06 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127473 2.0 2.e700b5b0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.298011+0000", 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.705041655, 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.92878800299999997, 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.070 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127473 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298011+0000", 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298011+0000", 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298012+0000", 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.412e-06 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298018+0000", 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9190000000000004e-06 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298030+0000", 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2262000000000001e-05 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298035+0000", 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8969999999999999e-06 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298101+0000", 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5538999999999997e-05 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298112+0000", 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1017e-05 2024-09-02T10:06:46.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226792+0000", 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.92868043 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226799+0000", 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5270000000000004e-06 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127507 2.0 2.51c134f0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.510346+0000", 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.49270651200001, 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.71648088300000001, 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127507 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510346+0000", 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510346+0000", 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510347+0000", 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2869999999999999e-06 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510353+0000", 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9290000000000003e-06 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510376+0000", 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3251000000000002e-05 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510384+0000", 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5129999999999999e-06 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510437+0000", 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3245000000000003e-05 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510446+0000", 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9910000000000008e-06 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226821+0000", 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.716374493 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226827+0000", 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1739999999999997e-06 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.8 157/153 [PushReplyOp(2:10f8d703:::benchmark_data_smithi187_45882_object31118:head)])", 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:46.348475+0000", 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 179.654577034, 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0031679010000001, 2024-09-02T10:06:46.079 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348475+0000", 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348475+0000", 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348476+0000", 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2900000000000002e-07 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348484+0000", 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.695e-06 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.265961+0000", 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91747688900000002 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.265975+0000", 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3981e-05 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.347454+0000", 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.081478748000000004 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.351643+0000", 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041896590000000001 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:141512 2.3 e169/92 2:c7378ca5:::benchmark_data_smithi187_45882_object43761:head v 169'8968, mlcod=169'8965)", 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:14.694189+0000", 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 151.30886353599999, 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.365733013, 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694189+0000", 2024-09-02T10:06:46.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694189+0000", 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694190+0000", 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1209999999999999e-06 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694197+0000", 2024-09-02T10:06:46.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5700000000000004e-06 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056209+0000", 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36201126900000002 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056216+0000", 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5390000000000002e-06 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056294+0000", 2024-09-02T10:06:46.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7521000000000005e-05 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056309+0000", 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4691e-05 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.059878+0000", 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003569582 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.059922+0000", 2024-09-02T10:06:46.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3720000000000002e-05 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:156192 2.8 2:133c662a:::benchmark_data_smithi187_45882_object58441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:43.142777+0000", 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 122.86027514, 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.60071848299999997, 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 156192 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142777+0000", 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142777+0000", 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142779+0000", 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.655e-06 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142784+0000", 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5350000000000004e-06 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142798+0000", 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3525e-05 2024-09-02T10:06:46.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142805+0000", 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4050000000000003e-06 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.147079+0000", 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042733989999999998 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.147202+0000", 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012349799999999999 2024-09-02T10:06:46.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,11", 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.147292+0000", 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9788000000000002e-05 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151073+0000", 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037812150000000001 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.153573+0000", 2024-09-02T10:06:46.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025002380000000001 2024-09-02T10:06:46.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:46.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742725+0000", 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.58915150900000002 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742781+0000", 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6039000000000003e-05 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.743496+0000", 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00071467700000000004 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:156192 2.8 e188/153 ondisk, result = 0)", 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:43.151803+0000", 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 122.851249737, 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.59169890800000002, 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.094 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151803+0000", 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151803+0000", 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151804+0000", 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5960000000000001e-06 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151817+0000", 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2957e-05 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.740670+0000", 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.58885322699999998 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.740675+0000", 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7310000000000001e-06 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742713+0000", 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020379510000000001 2024-09-02T10:06:46.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742719+0000", 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2979999999999997e-06 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.743501+0000", 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00078214799999999998 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:195494 2.0 2:019f944c:::benchmark_data_smithi187_45882_object97743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:43.029456+0000", 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 62.973596479999998, 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.66465173200000005, 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 195494 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.099 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029456+0000", 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029456+0000", 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029457+0000", 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.609e-06 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029464+0000", 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1240000000000002e-06 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029479+0000", 2024-09-02T10:06:46.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5832999999999999e-05 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029487+0000", 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6149999999999999e-06 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029544+0000", 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6965000000000001e-05 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029647+0000", 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103219 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029750+0000", 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102916 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034083+0000", 2024-09-02T10:06:46.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043323980000000003 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034644+0000", 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056196000000000002 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694038+0000", 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65939311300000003 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694078+0000", 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0043999999999998e-05 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694108+0000", 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9935999999999999e-05 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:195489 2.3 e217/92 2:c2cfeaf0:::benchmark_data_smithi187_45882_object97738:head v 217'12342, mlcod=217'12336)", 2024-09-02T10:06:46.105 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:43.029872+0000", 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 62.973180550000002, 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29312492699999998, 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029872+0000", 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029872+0000", 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029872+0000", 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9200000000000002e-07 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029876+0000", 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7699999999999999e-06 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.318539+0000", 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28866344799999999 2024-09-02T10:06:46.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.318554+0000", 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4922999999999999e-05 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.318666+0000", 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111931 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.318709+0000", 2024-09-02T10:06:46.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2512999999999998e-05 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.322959+0000", 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042500250000000002 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.322997+0000", 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8025000000000003e-05 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:195494 2.0 e217/216 ondisk, result = 0)", 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:43.034227+0000", 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 62.968825369999998, 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65988618399999999, 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034227+0000", 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:06:46.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034227+0000", 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034228+0000", 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1459999999999999e-06 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034232+0000", 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9149999999999998e-06 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:06:46.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.693957+0000", 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65972478499999998 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.693968+0000", 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1198e-05 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694026+0000", 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7663e-05 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694033+0000", 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9750000000000001e-06 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694113+0000", 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0501999999999996e-05 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:06:46.113 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:06:46.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:46.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:46.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:46.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:45 smithi151 ceph-mon[16639]: pgmap v601: 47 pgs: 47 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-02T10:06:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3436935625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:46.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:45 smithi187 ceph-mon[20072]: pgmap v601: 47 pgs: 47 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-02T10:06:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3436935625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:46.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:46.264 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:46.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:45 smithi162 ceph-mon[20279]: pgmap v601: 47 pgs: 47 active+clean; 3.3 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-02T10:06:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3436935625' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:46.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:46.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:46.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:46.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:46.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:46.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:46.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:46.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:46.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:46.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:46.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:46.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:46.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:46.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:06:46.876 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:46.907 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:46.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:46.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:46.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:06:47.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:47.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 44 16 60011 59995 85.2134 84.4375 0.0151215 0.0117288 2024-09-02T10:06:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:47.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:47.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:47.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:47.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:47.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:47.276 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:06:47.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:47.301 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:47.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:47.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:47.404 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:47.431 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:47.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:47.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:47.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:47.559 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:47.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:47.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:47.661 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:47.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:47.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:06:47.791 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:06:47.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-02T10:06:47.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:47.792 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:47.818 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:47.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:47.824 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:06:47.824 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:06:47.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:47.824 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:60938 3.f e248/226)", 2024-09-02T10:06:47.824 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:06:47.822222+0000", 2024-09-02T10:06:47.824 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00012279399999999999, 2024-09-02T10:06:47.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014362500000000001, 2024-09-02T10:06:47.824 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:47.822222+0000", 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:47.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:47.822222+0000", 2024-09-02T10:06:47.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:47.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:47.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:47.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:47.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:47.822223+0000", 2024-09-02T10:06:47.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5099999999999998e-07 2024-09-02T10:06:47.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:47.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:47.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:47.822304+0000", 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1223999999999993e-05 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:47.822335+0000", 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0253999999999999e-05 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:47.822342+0000", 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9179999999999997e-06 2024-09-02T10:06:47.827 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:47.828 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:47.828 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:47.828 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:47.828 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:06:47.828 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:06:47.828 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:06:47.920 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:47.948 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:47.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:48.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 45 16 61199 61183 84.9696 74.25 0.0138367 0.0117614 2024-09-02T10:06:48.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:48.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:48.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:48.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:48.201 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:48.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:48.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:48.332 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:48.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:48.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:48.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:48 smithi151 ceph-mon[16639]: pgmap v602: 47 pgs: 47 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-02T10:06:48.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:48.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:48 smithi187 ceph-mon[20072]: pgmap v602: 47 pgs: 47 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-02T10:06:48.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:48.586 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:48.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:48.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:06:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:48 smithi162 ceph-mon[20279]: pgmap v602: 47 pgs: 47 active+clean; 3.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 81 MiB/s wr, 1.30k op/s 2024-09-02T10:06:48.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:48.710 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:48.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:48.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:48.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:48.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:48.846 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:06:48.851 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:48.879 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:06:48.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:48.968 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:48.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:48.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:49.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 46 16 62345 62329 84.6794 71.625 0.0224095 0.011804 2024-09-02T10:06:49.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:49.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:49.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:49.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:49.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:49.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:49.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/45841844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/45841844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:49 smithi187 ceph-mon[20072]: osdmap e249: 12 total, 12 up, 11 in 2024-09-02T10:06:49.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:06:49.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:49.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:49.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:49.416 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:06:49.416 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:06:49.416 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:06:49.416 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:06:49.416 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:06:49.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:49.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:49.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:49.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:49.593 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:49.594 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:49.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/45841844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:49.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/45841844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:49.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:49 smithi162 ceph-mon[20279]: osdmap e249: 12 total, 12 up, 11 in 2024-09-02T10:06:49.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/45841844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:06:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/45841844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:06:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:49 smithi151 ceph-mon[16639]: osdmap e249: 12 total, 12 up, 11 in 2024-09-02T10:06:49.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:49.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:49.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:49.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:49.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:49.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:49.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:49.968 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:49.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:49.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:50.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 47 16 63274 63258 84.1129 58.0625 0.0922645 0.0118734 2024-09-02T10:06:50.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:50.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:06:50.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:50.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:50.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:50.232 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:50.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:50.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:50.290 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:06:50.290 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:06:50.290 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:06:50.290 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:06:50.290 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:06:50.290 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:06:50.291 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:06:50.291 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:06:50.291 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:06:50.291 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:06:50.291 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:06:50.291 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:06:50.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:50.354 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:50.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:50.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:50.396 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:06:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:50 smithi151 ceph-mon[16639]: pgmap v604: 47 pgs: 47 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-02T10:06:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:06:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:06:50.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:50.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:50.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:50 smithi187 ceph-mon[20072]: pgmap v604: 47 pgs: 47 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-02T10:06:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:06:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:06:50.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:50.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:50 smithi162 ceph-mon[20279]: pgmap v604: 47 pgs: 47 active+clean; 3.6 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-02T10:06:50.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:06:50.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:06:50.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:50.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:50.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:50.709 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:50.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:50.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:50.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:50.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:50.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:50.940 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:36428 2.b 2:da93149a:::benchmark_data_smithi187_45882_object36427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:02.079845+0000", 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 288.85843695900002, 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22416934999999999, 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:50.941 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36428 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079845+0000", 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079845+0000", 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079846+0000", 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3900000000000004e-07 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079867+0000", 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1290000000000001e-05 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079878+0000", 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0677e-05 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079887+0000", 2024-09-02T10:06:50.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1109999999999997e-06 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.081179+0000", 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001292444 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.081311+0000", 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000131855 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.081411+0000", 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9522e-05 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.082483+0000", 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010728630000000001 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.143788+0000", 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.061304751999999997 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.303899+0000", 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16011112 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.303951+0000", 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1687999999999997e-05 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:50.947 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.304014+0000", 2024-09-02T10:06:50.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3288999999999997e-05 2024-09-02T10:06:50.948 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.948 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:50.948 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.948 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:54932 2.b 2:d9ab64ae:::benchmark_data_smithi187_45882_object54931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.160766+0000", 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 273.77751562200001, 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26274712500000003, 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 54932 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160766+0000", 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160766+0000", 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160767+0000", 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2e-07 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160785+0000", 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8139e-05 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160792+0000", 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5109999999999997e-06 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160797+0000", 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2030000000000002e-06 2024-09-02T10:06:50.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160835+0000", 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8591999999999999e-05 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160936+0000", 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100828 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.161018+0000", 2024-09-02T10:06:50.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1940999999999994e-05 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162378+0000", 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001359968 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162459+0000", 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0600999999999996e-05 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423449+0000", 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26099011300000002 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423482+0000", 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.379e-05 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423513+0000", 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0919e-05 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.954 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:54932 2.b e113/92 ondisk, result = 0)", 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 273.77538938700002, 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26062987399999998, 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:50.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:06:50.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4199999999999995e-07 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162896+0000", 2024-09-02T10:06:50.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.23e-06 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423394+0000", 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26049795999999997 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423399+0000", 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4360000000000004e-06 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:50.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423440+0000", 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1412999999999998e-05 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423445+0000", 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1499999999999998e-06 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423522+0000", 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7043000000000001e-05 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:50.959 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:55061 2.6 e113/92 2:670fe1b2:::benchmark_data_smithi187_45882_object55060:head v 113'3388, mlcod=113'3385)", 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.479402+0000", 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 273.45888015700001, 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33103881099999999, 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:50.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479402+0000", 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479402+0000", 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479403+0000", 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3600000000000002e-07 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479437+0000", 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4811999999999998e-05 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809044+0000", 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32960664000000001 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809050+0000", 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7679999999999997e-06 2024-09-02T10:06:50.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809098+0000", 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7859999999999999e-05 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809112+0000", 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4579000000000001e-05 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.810407+0000", 2024-09-02T10:06:50.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012949649999999999 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.810441+0000", 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3250999999999997e-05 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55063 2.b 2:d2962897:::benchmark_data_smithi187_45882_object55062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.481328+0000", 2024-09-02T10:06:50.964 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 273.456953779, 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.292836235, 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55063 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:50.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481328+0000", 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481328+0000", 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481329+0000", 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0500000000000003e-07 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481351+0000", 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2506000000000001e-05 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481361+0000", 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0105999999999999e-05 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481366+0000", 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8119999999999998e-06 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481408+0000", 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1689e-05 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481498+0000", 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0192000000000007e-05 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481585+0000", 2024-09-02T10:06:50.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7214999999999998e-05 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.482940+0000", 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013545709999999999 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483445+0000", 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050479699999999997 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774099+0000", 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29065394700000002 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774135+0000", 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6665999999999998e-05 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774164+0000", 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9028999999999999e-05 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:50.970 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:55063 2.b e113/92 ondisk, result = 0)", 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.483081+0000", 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 273.45520080400001, 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29108941100000002, 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:50.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483081+0000", 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483081+0000", 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483082+0000", 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6700000000000003e-07 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483085+0000", 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3009999999999998e-06 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774042+0000", 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29095665500000001 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774047+0000", 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8629999999999996e-06 2024-09-02T10:06:50.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774087+0000", 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9206999999999999e-05 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774093+0000", 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6710000000000002e-06 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:50.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774170+0000", 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7146999999999998e-05 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55077 2.b 2:d13d7ad6:::benchmark_data_smithi187_45882_object55076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.497835+0000", 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 273.44044658399997, 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29033786299999997, 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:50.975 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55077 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497835+0000", 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497835+0000", 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497836+0000", 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.06e-07 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497866+0000", 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9967999999999998e-05 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497875+0000", 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9090000000000006e-06 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497882+0000", 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1409999999999999e-06 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497929+0000", 2024-09-02T10:06:50.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6683e-05 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498008+0000", 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9110000000000007e-05 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498101+0000", 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3379000000000002e-05 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.499478+0000", 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001376637 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.500299+0000", 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00082144399999999995 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.788094+0000", 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28779512299999999 2024-09-02T10:06:50.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.788131+0000", 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6924000000000001e-05 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.788173+0000", 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1838999999999997e-05 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55083 2.5 2:a3eaecaa:::benchmark_data_smithi187_45882_object55082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.511889+0000", 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 273.42639244200001, 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.254154883, 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55083 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.982 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511889+0000", 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511889+0000", 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511890+0000", 2024-09-02T10:06:50.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.046e-06 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511910+0000", 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9162999999999999e-05 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511921+0000", 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0995000000000001e-05 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511926+0000", 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2650000000000004e-06 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511967+0000", 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1461999999999997e-05 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.512047+0000", 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9907999999999995e-05 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.985 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.512138+0000", 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0909999999999995e-05 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.513495+0000", 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001356307 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.513828+0000", 2024-09-02T10:06:50.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00033362600000000001 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765934+0000", 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25210628000000002 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765996+0000", 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1614999999999999e-05 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.766044+0000", 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8306000000000001e-05 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55086 2.f 2:f2dac436:::benchmark_data_smithi187_45882_object55085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.518890+0000", 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 273.419392007, 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25559369300000001, 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:50.988 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55086 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518890+0000", 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518890+0000", 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518891+0000", 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8199999999999998e-07 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518909+0000", 2024-09-02T10:06:50.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8649000000000001e-05 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518920+0000", 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0560000000000001e-05 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518925+0000", 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0830000000000003e-06 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:50.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518974+0000", 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8979e-05 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.519056+0000", 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1796000000000001e-05 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.519137+0000", 2024-09-02T10:06:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1422999999999996e-05 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520575+0000", 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014377159999999999 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520789+0000", 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00021378200000000001 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774432+0000", 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25364359800000003 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774462+0000", 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9416000000000001e-05 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774484+0000", 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1809000000000001e-05 2024-09-02T10:06:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:55083 2.5 e113/92 ondisk, result = 0)", 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.520803+0000", 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 273.41747875499999, 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24524790699999999, 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520803+0000", 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520803+0000", 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520804+0000", 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1200000000000003e-07 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520807+0000", 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.523e-06 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765868+0000", 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.245060377 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765872+0000", 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6949999999999999e-06 2024-09-02T10:06:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765920+0000", 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7593000000000003e-05 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765929+0000", 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6260000000000001e-06 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.766051+0000", 2024-09-02T10:06:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122481 2024-09-02T10:06:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:97743 2.5 2:a58175a7:::benchmark_data_smithi187_45882_object97742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-02T10:06:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:05.424082+0000", 2024-09-02T10:06:50.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 225.51419982499999, 2024-09-02T10:06:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.60300915200000005, 2024-09-02T10:06:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 97743 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424082+0000", 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424082+0000", 2024-09-02T10:06:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424084+0000", 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6160000000000001e-06 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424109+0000", 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5466999999999999e-05 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424122+0000", 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2765999999999999e-05 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424129+0000", 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2599999999999999e-06 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424197+0000", 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7782999999999998e-05 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424308+0000", 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000110645 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424453+0000", 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014499099999999999 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427671+0000", 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032188590000000001 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.555569+0000", 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.127897921 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027018+0000", 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.47144856699999999 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027061+0000", 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3092e-05 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027091+0000", 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0185e-05 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:97743 2.5 e137/92 ondisk, result = 0)", 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:05.427175+0000", 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 225.51110643300001, 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.59992140800000004, 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427175+0000", 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427175+0000", 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427177+0000", 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.511e-06 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427183+0000", 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4629999999999998e-06 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.026965+0000", 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.59978128600000002 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.026969+0000", 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1810000000000003e-06 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027007+0000", 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8557999999999999e-05 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027013+0000", 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7649999999999999e-06 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027097+0000", 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3644000000000001e-05 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:117060 2.f 2:f36bc4f3:::benchmark_data_smithi187_45882_object19309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-02T10:06:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:29.622070+0000", 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 201.316212143, 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.266536263, 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 117060 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622070+0000", 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622070+0000", 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622071+0000", 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.001e-06 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622075+0000", 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1509999999999997e-06 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622084+0000", 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8580000000000006e-06 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622088+0000", 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5970000000000002e-06 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622136+0000", 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7803999999999998e-05 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622256+0000", 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119537 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:06:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622368+0000", 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011192900000000001 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.625537+0000", 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003168911 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.637552+0000", 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012015965999999999 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.888531+0000", 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25097857000000001 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.888578+0000", 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7426000000000003e-05 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.888606+0000", 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7512999999999999e-05 2024-09-02T10:06:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:163327 2.f 2:f9a47073:::benchmark_data_smithi187_45882_object65576:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:56.455190+0000", 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 114.48309202999999, 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.422262621, 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 163327 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455190+0000", 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455190+0000", 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455192+0000", 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.889e-06 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455200+0000", 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7710000000000006e-06 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455216+0000", 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6943999999999999e-05 2024-09-02T10:06:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455224+0000", 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1550000000000001e-06 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455280+0000", 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6443999999999997e-05 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455368+0000", 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8307999999999995e-05 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455446+0000", 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7698e-05 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.459097+0000", 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036506360000000001 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.459525+0000", 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00042830199999999998 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877381+0000", 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41785591900000002 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877420+0000", 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8832999999999999e-05 2024-09-02T10:06:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877452+0000", 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2722000000000003e-05 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:164051 2.b 2:dd30ecaf:::benchmark_data_smithi187_45882_object66300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:58.383444+0000", 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 112.55483751600001, 2024-09-02T10:06:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26956966300000001, 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 164051 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383444+0000", 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383444+0000", 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383446+0000", 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.347e-06 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383452+0000", 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.088e-06 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383469+0000", 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7153999999999999e-05 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383478+0000", 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9109999999999999e-06 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383537+0000", 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8687999999999998e-05 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383632+0000", 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5619999999999996e-05 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383712+0000", 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9435000000000001e-05 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387376+0000", 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003664488 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387533+0000", 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015662400000000001 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652930+0000", 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26539721100000002 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652964+0000", 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4489999999999997e-05 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.653014+0000", 2024-09-02T10:06:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9607000000000001e-05 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:164051 2.b e200/92 ondisk, result = 0)", 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 112.55061523000001, 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26535247699999998, 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1799999999999995e-07 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387670+0000", 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8219999999999998e-06 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652865+0000", 2024-09-02T10:06:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26519509200000002 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652872+0000", 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3509999999999996e-06 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652921+0000", 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8124999999999997e-05 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652926+0000", 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.682e-06 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.653019+0000", 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2886999999999993e-05 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "rep_scrubmap(2.b e92 from shard 7)", 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:42.047060+0000", 2024-09-02T10:06:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 68.891221819999998, 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.229439527, 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.047060+0000", 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.047060+0000", 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.047061+0000", 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7899999999999997e-07 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.047075+0000", 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4046000000000001e-05 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.271919+0000", 2024-09-02T10:06:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22484417700000001 2024-09-02T10:06:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.271925+0000", 2024-09-02T10:06:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0190000000000003e-06 2024-09-02T10:06:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.276438+0000", 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045127470000000001 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.276447+0000", 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2539999999999998e-06 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.276500+0000", 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2404999999999997e-05 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:195477 2.f 2:f32fe7b9:::benchmark_data_smithi187_45882_object97726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:43.014428+0000", 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 67.923853604000001, 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43313299799999999, 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:06:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 195477 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014428+0000", 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014428+0000", 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014430+0000", 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.55e-06 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014459+0000", 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9539e-05 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014470+0000", 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1002000000000001e-05 2024-09-02T10:06:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014477+0000", 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0720000000000002e-06 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.019153+0000", 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046751120000000004 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.019267+0000", 2024-09-02T10:06:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011444500000000001 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.019338+0000", 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1087999999999994e-05 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.024391+0000", 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005052506 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.025167+0000", 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00077600499999999997 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447446+0000", 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42227959799999998 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447512+0000", 2024-09-02T10:06:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5617000000000006e-05 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447561+0000", 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9463999999999999e-05 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:195477 2.f e217/92 ondisk, result = 0)", 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:43.026172+0000", 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 67.912110287999994, 2024-09-02T10:06:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42139491699999998, 2024-09-02T10:06:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:06:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.026172+0000", 2024-09-02T10:06:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:51.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.026172+0000", 2024-09-02T10:06:51.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.026172+0000", 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8999999999999996e-07 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.026175+0000", 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2610000000000002e-06 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447363+0000", 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42118752500000001 2024-09-02T10:06:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447370+0000", 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.126e-06 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447428+0000", 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7776999999999999e-05 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447438+0000", 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.039e-05 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447567+0000", 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000128248 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24976.0:12905 3.9 3:91cb3b60:::benchmark_data_smithi187_59989_object12904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:06:11.038738+0000", 2024-09-02T10:06:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 39.899543700000002, 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22354797100000001, 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24976", 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 12905 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.038738+0000", 2024-09-02T10:06:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.038738+0000", 2024-09-02T10:06:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.038739+0000", 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.296e-06 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.038760+0000", 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0400000000000001e-05 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.038773+0000", 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.287e-05 2024-09-02T10:06:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.038779+0000", 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.64e-06 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.038824+0000", 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4399e-05 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.038912+0000", 2024-09-02T10:06:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8578999999999993e-05 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.038999+0000", 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.619e-05 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.039658+0000", 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00065941000000000003 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.043281+0000", 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036229830000000002 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.262174+0000", 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21889354 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.262234+0000", 2024-09-02T10:06:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9954999999999998e-05 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.262286+0000", 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1709000000000002e-05 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:51.058 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:06:51.059 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 48 16 64327 64311 83.7316 65.8125 0.0037751 0.0119339 2024-09-02T10:06:51.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:51.060 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:51.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:51.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:51.196 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:51.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:51.298 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:51.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:51.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:51.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:51.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:51.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:51.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:51.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:51.580 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:51.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:51.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:51.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:51.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:51.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:51.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:51.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:51.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:51.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:51.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:06:51.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:51.964 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:51.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:51.966 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:51 smithi187 ceph-mon[20072]: osdmap e250: 12 total, 12 up, 11 in 2024-09-02T10:06:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:06:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:06:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:06:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:06:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:51 smithi187 ceph-mon[20072]: pgmap v606: 47 pgs: 47 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-02T10:06:52.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:06:52.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:06:52.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:06:52.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 49 16 65590 65574 83.6336 78.9375 0.00853621 0.0119515 2024-09-02T10:06:52.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:52.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:52.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:52.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:51 smithi151 ceph-mon[16639]: osdmap e250: 12 total, 12 up, 11 in 2024-09-02T10:06:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:06:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:06:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:06:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:06:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:51 smithi151 ceph-mon[16639]: pgmap v606: 47 pgs: 47 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-02T10:06:52.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:06:52.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:06:52.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:06:52.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:52.215 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:06:52.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:52.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:52.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:06:52.236 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:06:52.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-02T10:06:52.268 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:52.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:52.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:51 smithi162 ceph-mon[20279]: osdmap e250: 12 total, 12 up, 11 in 2024-09-02T10:06:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:06:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:06:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:06:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:06:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:51 smithi162 ceph-mon[20279]: pgmap v606: 47 pgs: 47 active+clean; 3.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-02T10:06:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:06:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:06:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:06:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:52.361 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:52.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:52.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:52.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:52.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:52.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:52.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:52.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:52.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:52.753 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:52.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:66561 3.d e250/226 3:b498a663:::benchmark_data_smithi187_59989_object66560:head v 250'4157, mlcod=250'4154)", 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:06:52.761136+0000", 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.025072085000000001, 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.025096408000000001, 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:52.761136+0000", 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:52.761136+0000", 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:52.761137+0000", 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3009999999999999e-06 2024-09-02T10:06:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:06:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:52.761198+0000", 2024-09-02T10:06:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1062000000000006e-05 2024-09-02T10:06:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:06:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:52.785731+0000", 2024-09-02T10:06:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.024533368999999999 2024-09-02T10:06:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:52.785741+0000", 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8840000000000006e-06 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:52.785796+0000", 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4438999999999998e-05 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:06:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:52.785813+0000", 2024-09-02T10:06:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7243000000000001e-05 2024-09-02T10:06:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:06:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:06:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:06:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:06:52.792 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:06:52.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:52.886 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:52.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:52.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:53.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:53.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:53.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:53.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 50 16 66881 66865 83.5745 80.6875 0.00329117 0.0119581 2024-09-02T10:06:53.056 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:06:53.057 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:06:53.085 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:06:53.087 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:06:53.113 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:06:53.113 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:06:53.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:53.144 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:06:53.144 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:06:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:53.154 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:53.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:53.157 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:53.177 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:06:53.177 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:06:53.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1482482776' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:53.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:06:53.210 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:06:53.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:06:53.242 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:06:53.242 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:06:53.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1482482776' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:53.272 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:06:53.273 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:06:53.286 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:53.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:53.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:06:53.302 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:06:53.333 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:06:53.333 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:06:53.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1482482776' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:53.361 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:06:53.361 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:06:53.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:53.391 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:06:53.391 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:06:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:53.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:53.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:53.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:06:53.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:53.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:53.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:53.574 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:53.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:53.701 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:53.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:53.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:53.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:53.814 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:06:53.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:53.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:53.906 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:06:53.907 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:06:53.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:53.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:53.956 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:54.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 51 16 67959 67943 83.2568 67.375 0.00284193 0.0119998 2024-09-02T10:06:54.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:54.077 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:54.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:54.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:54 smithi187 ceph-mon[20072]: pgmap v607: 47 pgs: 47 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-02T10:06:54.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:54 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:54.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:54 smithi187 ceph-mon[20072]: osdmap e251: 12 total, 12 up, 11 in 2024-09-02T10:06:54.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:54.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:54.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:54.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:54.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:54.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:54 smithi162 ceph-mon[20279]: pgmap v607: 47 pgs: 47 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-02T10:06:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:54 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:54 smithi162 ceph-mon[20279]: osdmap e251: 12 total, 12 up, 11 in 2024-09-02T10:06:54.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:54 smithi151 ceph-mon[16639]: pgmap v607: 47 pgs: 47 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 75 MiB/s wr, 1.20k op/s 2024-09-02T10:06:54.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:54 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:54.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:54 smithi151 ceph-mon[16639]: osdmap e251: 12 total, 12 up, 11 in 2024-09-02T10:06:54.404 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:54.404 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:54.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:54.465 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:54.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:54.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:54.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:54.594 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:54.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:54.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:54.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:54.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:54.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:54.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:54.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:54.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:54.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:54.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:54.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:55.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 52 16 68910 68894 82.7987 59.4375 0.0192719 0.012072 2024-09-02T10:06:55.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:55.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:06:55.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:55.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:55.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:06:55.217 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:06:55.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-02T10:06:55.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:55.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:55.253 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:55.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:55.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:55.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:55.424 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:06:55.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:55.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:55.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:55.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:55.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:55.625 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:55.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:55.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:55.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:55.750 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:55.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:55.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:55.857 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:55.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:55.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:55.979 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:55.979 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:55.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:56.008 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:56.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:56.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 53 16 70145 70129 82.6927 77.1875 0.00503373 0.0120884 2024-09-02T10:06:56.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:56.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:56.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:56.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:56.246 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:06:56.246 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:56 smithi151 ceph-mon[16639]: pgmap v609: 47 pgs: 47 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-02T10:06:56.246 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2648019795' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:06:56.264 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:56.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:56.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:56.395 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:56.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:56.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:06:56.497 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:06:56.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:56 smithi187 ceph-mon[20072]: pgmap v609: 47 pgs: 47 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-02T10:06:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2648019795' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:06:56.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:56.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:56.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:56.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:56 smithi162 ceph-mon[20279]: pgmap v609: 47 pgs: 47 active+clean; 4.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-02T10:06:56.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2648019795' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:06:56.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:56.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:56.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:56.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:56.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:06:56.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:56.793 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:56.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:56.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:56.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:56.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:56.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:56.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:56.969 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:06:57.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:57.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 54 16 71314 71298 82.5142 73.0625 0.00541012 0.0121142 2024-09-02T10:06:57.056 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:57.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:57.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:57.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:57.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:57.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:06:57.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:57.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:06:57.364 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2648019795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:06:57.364 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:57 smithi187 ceph-mon[20072]: osdmap e252: 12 total, 12 up, 11 in 2024-09-02T10:06:57.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:57.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2648019795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:06:57.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:57 smithi151 ceph-mon[16639]: osdmap e252: 12 total, 12 up, 11 in 2024-09-02T10:06:57.440 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:57.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:57.496 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:06:57.496 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:06:57.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:57.567 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:57.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:57.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2648019795' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:06:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:57 smithi162 ceph-mon[20279]: osdmap e252: 12 total, 12 up, 11 in 2024-09-02T10:06:57.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:06:57.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:57.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:06:57.796 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:57.820 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:57.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:06:57.917 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:06:57.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-02T10:06:57.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:57.946 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:57.946 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:58.010 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:58.027 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 55 16 72536 72520 82.4025 76.375 0.0137853 0.0121295 2024-09-02T10:06:58.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:06:58.072 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:58.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:58.073 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:06:58.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:58.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:58.198 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:58.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:58.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:06:58.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:06:58.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:58.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:58.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:06:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:58 smithi151 ceph-mon[16639]: pgmap v611: 47 pgs: 47 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-02T10:06:58.451 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:58.479 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:58.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:58.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:58 smithi187 ceph-mon[20072]: pgmap v611: 47 pgs: 47 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-02T10:06:58.524 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:06:58.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:06:58.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:58 smithi162 ceph-mon[20279]: pgmap v611: 47 pgs: 47 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 91 MiB/s wr, 1.46k op/s 2024-09-02T10:06:58.608 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:58.609 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:06:58.609 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:06:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:58.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:06:58.619 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:06:58.644 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:06:58.645 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:06:58.674 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:06:58.674 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:06:58.709 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:06:58.709 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:06:58.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:06:58.739 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:58.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:06:58.740 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:06:58.740 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:06:58.771 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:06:58.772 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:06:58.804 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:06:58.805 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:06:58.833 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:06:58.834 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:06:58.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:58.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:58.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:58.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:58.879 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:06:58.879 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:06:58.905 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:06:58.905 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:06:58.937 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:06:58.937 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:06:58.962 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:06:58.963 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:06:58.969 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:06:58.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:58.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:06:59.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 56 16 73766 73750 82.3037 76.875 0.00349056 0.0121441 2024-09-02T10:06:59.058 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:06:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:06:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:06:59.058 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:06:59.058 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:06:59.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:06:59.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:59.124 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:06:59.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:06:59.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:59.236 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:06:59.237 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:06:59.251 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:59.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:59.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:06:59.375 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:59.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:06:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3837692305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:59 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:59.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:06:59 smithi151 ceph-mon[16639]: osdmap e253: 12 total, 12 up, 11 in 2024-09-02T10:06:59.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:06:59.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:06:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:59.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:06:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3837692305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:59 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:06:59 smithi187 ceph-mon[20072]: osdmap e253: 12 total, 12 up, 11 in 2024-09-02T10:06:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3837692305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:06:59.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:59 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:06:59.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:06:59 smithi162 ceph-mon[20279]: osdmap e253: 12 total, 12 up, 11 in 2024-09-02T10:06:59.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:06:59.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:06:59.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:59.629 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:06:59.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:59.756 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:59.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:59.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:59.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:06:59.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:06:59.884 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:06:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:06:59.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:06:59.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:00.013 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:00.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:00.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 57 16 74939 74923 82.1458 73.3125 0.00669252 0.0121684 2024-09-02T10:07:00.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:00.116 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:07:00.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:00.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:00.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:00.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:00.271 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:00.272 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:00.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:00.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:00.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:00.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:00.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:00.496 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:00 smithi187 ceph-mon[20072]: pgmap v613: 47 pgs: 47 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-02T10:07:00.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:00.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:00.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:00 smithi162 ceph-mon[20279]: pgmap v613: 47 pgs: 47 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-02T10:07:00.597 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:07:00.597 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:07:00.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:07:00.597 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:07:00.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.597 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1131 2.0 e96/92 2:021aab7d:::benchmark_data_smithi187_45882_object1130:head v 96'61, mlcod=96'58)", 2024-09-02T10:07:00.597 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:07:00.597 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 324.69707116400002, 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077611448, 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898452+0000", 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8899999999999998e-07 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898480+0000", 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.817e-05 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975643+0000", 2024-09-02T10:07:00.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077163063000000004 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975651+0000", 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.028e-06 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975697+0000", 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5544999999999998e-05 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975710+0000", 2024-09-02T10:07:00.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2928000000000001e-05 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.976007+0000", 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029711499999999998 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.976063+0000", 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5810000000000003e-05 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.601 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1142 2.7 e96/92 2:e2cb33eb:::benchmark_data_smithi187_45882_object1141:head v 96'87, mlcod=96'83)", 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 324.68856881599999, 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18675391099999999, 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:07:00.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906955+0000", 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.181e-06 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906991+0000", 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6523999999999998e-05 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093330+0000", 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18633878100000001 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093335+0000", 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5909999999999998e-06 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093381+0000", 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6925999999999998e-05 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093396+0000", 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4216000000000001e-05 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093682+0000", 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028659900000000001 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093707+0000", 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5092999999999998e-05 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1442 2.0 e96/92 2:04aa890d:::benchmark_data_smithi187_45882_object1441:head v 96'80, mlcod=96'77)", 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 324.40615346200002, 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087334971999999997, 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.606 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8100000000000002e-07 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189467+0000", 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7173e-05 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276310+0000", 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.086843528000000003 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276315+0000", 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8670000000000002e-06 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276365+0000", 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9459999999999997e-05 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276382+0000", 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7045999999999999e-05 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276666+0000", 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028404500000000002 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276704+0000", 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8172e-05 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11303 2.7 e98/92 2:eeb591e6:::benchmark_data_smithi187_45882_object11302:head v 98'755, mlcod=98'751)", 2024-09-02T10:07:00.610 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:07:00.611 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 318.281175292, 2024-09-02T10:07:00.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.117126889, 2024-09-02T10:07:00.611 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.611 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.611 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314348+0000", 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8599999999999998e-07 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314428+0000", 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0412000000000002e-05 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430860+0000", 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.116432137 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430866+0000", 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.083e-06 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430918+0000", 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.215e-05 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430932+0000", 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3591000000000001e-05 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.431445+0000", 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051315900000000001 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.431474+0000", 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8671e-05 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.615 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11334 2.4 e98/92 2:22fc355e:::benchmark_data_smithi187_45882_object11333:head v 98'688, mlcod=98'684)", 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 318.241854886, 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.072501649000000001, 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353668+0000", 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5899999999999995e-07 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353720+0000", 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1539999999999998e-05 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425505+0000", 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.071785287000000003 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425512+0000", 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2980000000000003e-06 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425576+0000", 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3614000000000005e-05 2024-09-02T10:07:00.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425598+0000", 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2076e-05 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426142+0000", 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00054422300000000004 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426169+0000", 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6852e-05 2024-09-02T10:07:00.619 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11352 2.4 e98/92 2:2477406b:::benchmark_data_smithi187_45882_object11351:head v 98'690, mlcod=98'686)", 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 318.168844799, 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11620186, 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426679+0000", 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2580000000000001e-06 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426733+0000", 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4398999999999999e-05 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542090+0000", 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.115357134 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542105+0000", 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4596999999999999e-05 2024-09-02T10:07:00.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542170+0000", 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5376999999999994e-05 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542194+0000", 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3611000000000001e-05 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542825+0000", 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00063158400000000003 2024-09-02T10:07:00.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542879+0000", 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3900000000000002e-05 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11358 2.7 e98/92 2:e5a3134d:::benchmark_data_smithi187_45882_object11357:head v 98'756, mlcod=98'753)", 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 318.16473278500001, 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078694929999999996, 2024-09-02T10:07:00.624 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430790+0000", 2024-09-02T10:07:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7499999999999999e-07 2024-09-02T10:07:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430846+0000", 2024-09-02T10:07:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5544999999999998e-05 2024-09-02T10:07:00.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508940+0000", 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078093869999999996 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508946+0000", 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7120000000000003e-06 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508990+0000", 2024-09-02T10:07:00.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4095999999999999e-05 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509005+0000", 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5000999999999999e-05 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509456+0000", 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00045036799999999998 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509484+0000", 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8563000000000001e-05 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11782 2.4 e98/92 2:2417a9e9:::benchmark_data_smithi187_45882_object11781:head v 98'718, mlcod=98'714)", 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 317.85252979299997, 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12480622099999999, 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.629 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742993+0000", 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2399999999999997e-07 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.743050+0000", 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6356e-05 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867097+0000", 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.124047 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867103+0000", 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0399999999999998e-06 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867172+0000", 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8996000000000001e-05 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867187+0000", 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5218999999999999e-05 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867764+0000", 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00057719100000000001 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867799+0000", 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4694999999999999e-05 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11788 2.7 e98/92 2:ee0a4980:::benchmark_data_smithi187_45882_object11787:head v 98'788, mlcod=98'785)", 2024-09-02T10:07:00.633 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 317.84524602200003, 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.134742468, 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750278+0000", 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5430000000000001e-06 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750323+0000", 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4706000000000002e-05 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884234+0000", 2024-09-02T10:07:00.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13391187800000001 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884240+0000", 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4190000000000001e-06 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884280+0000", 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0012000000000002e-05 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884316+0000", 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6318e-05 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884977+0000", 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00066127600000000003 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.885019+0000", 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1316000000000002e-05 2024-09-02T10:07:00.637 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11794 2.4 e98/92 2:22bc5fbb:::benchmark_data_smithi187_45882_object11793:head v 98'719, mlcod=98'715)", 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 317.83817318299998, 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.125805317, 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.638 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757350+0000", 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5300000000000003e-07 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757408+0000", 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8291000000000001e-05 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882534+0000", 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.125126182 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882540+0000", 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0689999999999998e-06 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882583+0000", 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2962e-05 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882599+0000", 2024-09-02T10:07:00.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5718000000000001e-05 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.883121+0000", 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052199199999999999 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.883154+0000", 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3349999999999997e-05 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11897 2.4 e98/92 2:2a9822cc:::benchmark_data_smithi187_45882_object11896:head v 98'727, mlcod=98'723)", 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 317.63649297199999, 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088729108000000001, 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:07:00.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959030+0000", 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0200000000000001e-07 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959076+0000", 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5896000000000002e-05 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047130+0000", 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088054190000000004 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047135+0000", 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2379999999999997e-06 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047218+0000", 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2892000000000005e-05 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047233+0000", 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5126999999999999e-05 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047729+0000", 2024-09-02T10:07:00.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000495847 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047758+0000", 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9116e-05 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:13212 2.0 e98/92 2:00f0b824:::benchmark_data_smithi187_45882_object13211:head v 98'832, mlcod=98'829)", 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:07:00.647 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 316.82380803000001, 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.075308911000000006, 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771715+0000", 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1599999999999999e-06 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771775+0000", 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9979999999999998e-05 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846360+0000", 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.074584792999999996 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846367+0000", 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1489999999999997e-06 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846446+0000", 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8652999999999994e-05 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846461+0000", 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5254e-05 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846996+0000", 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053431400000000003 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.847023+0000", 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7608000000000001e-05 2024-09-02T10:07:00.651 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:13565 2.0 e98/92 2:0460cc71:::benchmark_data_smithi187_45882_object13564:head v 98'854, mlcod=98'850)", 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 316.53429555600002, 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29179487700000001, 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.652 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9400000000000005e-07 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061284+0000", 2024-09-02T10:07:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6251000000000001e-05 2024-09-02T10:07:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352274+0000", 2024-09-02T10:07:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29099082599999998 2024-09-02T10:07:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352280+0000", 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1950000000000002e-06 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352346+0000", 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5837e-05 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352360+0000", 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.433e-05 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352980+0000", 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00062060200000000005 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.353022+0000", 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1142e-05 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:22449 2.0 e104/92 2:0733b693:::benchmark_data_smithi187_45882_object22448:head v 104'1397, mlcod=104'1393)", 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 309.31481244999998, 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12946363599999999, 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280711+0000", 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1800000000000005e-07 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280779+0000", 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8644000000000002e-05 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409376+0000", 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12859704799999999 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409382+0000", 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5589999999999996e-06 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.660 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409441+0000", 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8746000000000003e-05 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409466+0000", 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5493000000000001e-05 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.410146+0000", 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00067948499999999996 2024-09-02T10:07:00.661 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.410173+0000", 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7943e-05 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:22536 2.0 e104/92 2:01400da2:::benchmark_data_smithi187_45882_object22535:head v 104'1399, mlcod=104'1395)", 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 309.153923679, 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077533989999999997, 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.662 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.663 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.47e-07 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441696+0000", 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6059999999999998e-05 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518264+0000", 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.076568862000000001 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518272+0000", 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5789999999999998e-06 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518333+0000", 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1159000000000002e-05 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518358+0000", 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5182e-05 2024-09-02T10:07:00.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.519104+0000", 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00074525100000000005 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.519133+0000", 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9050000000000001e-05 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:36974 2.7 e110/92 2:ea266fd4:::benchmark_data_smithi187_45882_object36973:head v 110'2332, mlcod=110'2328)", 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 297.995572928, 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080151230000000004, 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599953+0000", 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6090000000000001e-06 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599999+0000", 2024-09-02T10:07:00.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5561e-05 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679063+0000", 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.079064375000000006 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679068+0000", 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3619999999999997e-06 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679116+0000", 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8084999999999998e-05 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679132+0000", 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5264000000000002e-05 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.680070+0000", 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00093846200000000004 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.680101+0000", 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0511999999999999e-05 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:37371 2.0 e110/92 2:00cc6826:::benchmark_data_smithi187_45882_object37370:head v 110'2365, mlcod=110'2361)", 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 297.46719444899998, 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083380836999999999, 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.671 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128329+0000", 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.207e-06 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128397+0000", 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7869999999999999e-05 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210301+0000", 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081903702999999994 2024-09-02T10:07:00.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210307+0000", 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.737e-06 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210368+0000", 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0795000000000003e-05 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210383+0000", 2024-09-02T10:07:00.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4538999999999999e-05 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.211539+0000", 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011564310000000001 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.211709+0000", 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000169555 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:67961 2.7 e120/92 2:ed776389:::benchmark_data_smithi187_45882_object67960:head v 120'4269, mlcod=120'4266)", 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 270.68681699000001, 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.095923729999999999, 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908706+0000", 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1399999999999997e-07 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908770+0000", 2024-09-02T10:07:00.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4410000000000002e-05 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002909+0000", 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.094138870999999999 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002918+0000", 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3489999999999992e-06 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002977+0000", 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9299999999999998e-05 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002993+0000", 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6424000000000001e-05 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.004595+0000", 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001601725 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.004629+0000", 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4037e-05 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:78685 2.4 e124/92 2:26d5ebbe:::benchmark_data_smithi187_45882_object78684:head v 124'4867, mlcod=124'4864)", 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 260.14702779300001, 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.135791625, 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.680 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448496+0000", 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.257e-06 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448603+0000", 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106925 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.581994+0000", 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13339121900000001 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582001+0000", 2024-09-02T10:07:00.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7950000000000002e-06 2024-09-02T10:07:00.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582057+0000", 2024-09-02T10:07:00.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6724000000000001e-05 2024-09-02T10:07:00.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582074+0000", 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7037999999999998e-05 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.584242+0000", 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021673809999999999 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.584286+0000", 2024-09-02T10:07:00.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4286000000000003e-05 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:79093 2.4 e125/92 2:2f2be3a1:::benchmark_data_smithi187_45882_object79092:head v 125'4892, mlcod=125'4888)", 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 259.64259482300002, 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16453667699999999, 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:00.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952928+0000", 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:07:00.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952991+0000", 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3367999999999994e-05 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115366+0000", 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16237467799999999 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115374+0000", 2024-09-02T10:07:00.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6240000000000002e-06 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115428+0000", 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3885999999999999e-05 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115442+0000", 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4438e-05 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:07:00.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.117418+0000", 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019764600000000002 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.117464+0000", 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5694e-05 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:00.689 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:07:00.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:00 smithi151 ceph-mon[16639]: pgmap v613: 47 pgs: 47 active+clean; 4.3 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 70 MiB/s wr, 1.13k op/s 2024-09-02T10:07:00.715 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:00.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:07:00.725 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:07:00.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-02T10:07:00.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:00.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:00.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:00.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:00.867 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:00.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:00.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:00.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:01.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 58 16 76016 76000 81.8899 67.3125 0.0120985 0.0122065 2024-09-02T10:07:01.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:01.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:01.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:01.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:01.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:01.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:01.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:01.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:01.375 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:01.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:01.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:01.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:01 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:07:01.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:01 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:07:01.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:01 smithi187 ceph-mon[20072]: osdmap e254: 12 total, 12 up, 11 in 2024-09-02T10:07:01.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:07:01.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:01.484 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:01.502 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:01.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:01.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:01 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:07:01.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:01 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:07:01.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:01 smithi162 ceph-mon[20279]: osdmap e254: 12 total, 12 up, 11 in 2024-09-02T10:07:01.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:01.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:01 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:07:01.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:01 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:07:01.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:01 smithi151 ceph-mon[16639]: osdmap e254: 12 total, 12 up, 11 in 2024-09-02T10:07:01.632 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:01.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:01.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:01.766 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:01.767 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:01.783 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:07:01.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:01.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:01.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:01.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:02.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:07:02.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:02.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:02.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 59 16 77167 77151 81.7212 71.9375 0.00911024 0.0122317 2024-09-02T10:07:02.052 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:07:02.052 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:07:02.052 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:07:02.052 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:07:02.053 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:07:02.053 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:07:02.053 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:07:02.053 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:07:02.053 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:07:02.053 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:07:02.053 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:07:02.053 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:07:02.126 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:02.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:02.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:02.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:02.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:02.268 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:07:02.284 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:02.285 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24976.0:77479 3.8 3:1c071d43:::benchmark_data_smithi187_59989_object77478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:07:02.307301+0000", 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.011936385000000001, 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011960659, 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:07:02.321 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24976", 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 77479 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307301+0000", 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307301+0000", 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307302+0000", 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2759999999999999e-06 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307323+0000", 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1098999999999999e-05 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307336+0000", 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3546999999999999e-05 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307342+0000", 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.417e-06 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307387+0000", 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4761000000000001e-05 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307530+0000", 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014339600000000001 2024-09-02T10:07:02.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307681+0000", 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015084199999999999 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.310851+0000", 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031697800000000001 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.310860+0000", 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1810000000000007e-06 2024-09-02T10:07:02.325 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24976.0:77481 3.0 3:0f537931:::benchmark_data_smithi187_59989_object77480:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:07:02.307585+0000", 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.011652114999999999, 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011724574999999999, 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24976", 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:02.326 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 77481 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307585+0000", 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307585+0000", 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307585+0000", 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.61e-07 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307603+0000", 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7785e-05 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307612+0000", 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3189999999999995e-06 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307616+0000", 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3460000000000004e-06 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307667+0000", 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1285000000000001e-05 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307776+0000", 2024-09-02T10:07:02.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108354 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.307926+0000", 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000150414 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.311156+0000", 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032302300000000002 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:02.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.311589+0000", 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000432438 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24976.0:77473 3.e 3:7ae47364:::benchmark_data_smithi187_59989_object77472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:07:02.302387+0000", 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.016850032000000001, 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016965581, 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:07:02.331 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24976", 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 77473 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.302387+0000", 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.302387+0000", 2024-09-02T10:07:02.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.302388+0000", 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.22e-06 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.302410+0000", 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2342999999999999e-05 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.302423+0000", 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3010000000000001e-05 2024-09-02T10:07:02.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.302431+0000", 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.092e-06 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.302495+0000", 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4054999999999996e-05 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.302626+0000", 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013146300000000001 2024-09-02T10:07:02.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.302783+0000", 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000156835 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.305580+0000", 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027966369999999998 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24976.0:77475 3.8 3:1ba547eb:::benchmark_data_smithi187_59989_object77474:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e255)", 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:07:02.303367+0000", 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.015870254, 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016031376999999999, 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24976", 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 77475 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:07:02.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.303367+0000", 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.303367+0000", 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.303368+0000", 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7000000000000003e-07 2024-09-02T10:07:02.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.303390+0000", 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2272999999999998e-05 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.303399+0000", 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0680000000000003e-06 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.303404+0000", 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9089999999999999e-06 2024-09-02T10:07:02.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.303453+0000", 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8921999999999997e-05 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.303544+0000", 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1105000000000003e-05 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.303628+0000", 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3857999999999996e-05 2024-09-02T10:07:02.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.305998+0000", 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023704849999999999 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:02.306610+0000", 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000611503 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:07:02.340 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 4 2024-09-02T10:07:02.341 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:07:02.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:02.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:02.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:02 smithi187 ceph-mon[20072]: pgmap v615: 47 pgs: 47 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:07:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/317136883' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:02.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:02.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:02.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:02.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:02.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:02 smithi162 ceph-mon[20279]: pgmap v615: 47 pgs: 47 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:07:02.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/317136883' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:02.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:02.658 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:02.659 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:02.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:02 smithi151 ceph-mon[16639]: pgmap v615: 47 pgs: 47 active+clean; 4.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:07:02.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/317136883' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:02.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:02.787 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:02.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:02.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:02.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:02.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:03.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:03.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:07:03.028262+0000 min lat: 0.0010322 max lat: 0.311198 avg lat: 0.0122756 2024-09-02T10:07:03.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:07:03.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 60 16 78194 78178 81.4288 64.1875 0.00619044 0.0122756 2024-09-02T10:07:03.044 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:03.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:03.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:03.179 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:03.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:03.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:03.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:03.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:03.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:07:03.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:03.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:03 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:07:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:03 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:07:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/317136883' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:03 smithi151 ceph-mon[16639]: osdmap e255: 12 total, 12 up, 11 in 2024-09-02T10:07:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:03 smithi151 ceph-mon[16639]: 3.d scrub starts 2024-09-02T10:07:03.439 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:03.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:03.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:03 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:07:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:03 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:07:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/317136883' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:03 smithi187 ceph-mon[20072]: osdmap e255: 12 total, 12 up, 11 in 2024-09-02T10:07:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:03 smithi187 ceph-mon[20072]: 3.d scrub starts 2024-09-02T10:07:03.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:03.573 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:03.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:03.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:03.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:03 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:07:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:03 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:07:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/317136883' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:03 smithi162 ceph-mon[20279]: osdmap e255: 12 total, 12 up, 11 in 2024-09-02T10:07:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:03 smithi162 ceph-mon[20279]: 3.d scrub starts 2024-09-02T10:07:03.671 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:07:03.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:07:03.679 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:07:03.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-09-02T10:07:03.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:03.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:03.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:03.788 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:03.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:03.841 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:03.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:03.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:03.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:03.975 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:03.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:04.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 61 16 79464 79448 81.395 79.375 0.00525044 0.0122794 2024-09-02T10:07:04.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:04.111 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:04.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:04.205 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:07:04.205 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:07:04.205 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:07:04.205 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:07:04.205 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:07:04.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:04.239 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:04.240 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:04.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:04.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:04.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:04.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:07:04.458 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:07:04.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:04.482 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:07:04.482 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:07:04.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:04.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:04 smithi187 ceph-mon[20072]: pgmap v617: 47 pgs: 47 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-02T10:07:04.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:07:04.515 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:07:04.546 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:07:04.546 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:07:04.547 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:04 smithi151 ceph-mon[16639]: pgmap v617: 47 pgs: 47 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-02T10:07:04.556 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:04.577 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:07:04.577 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:07:04.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:04 smithi162 ceph-mon[20279]: pgmap v617: 47 pgs: 47 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-02T10:07:04.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:04.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:07:04.607 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:07:04.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:04.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:04.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:04.641 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:07:04.641 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:07:04.670 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:07:04.670 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:07:04.702 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:07:04.702 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:07:04.733 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:07:04.736 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:07:04.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:04.760 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:04.761 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:04.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:07:04.763 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:07:04.795 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:07:04.795 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:07:04.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:04.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:07:04.889 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:04.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:04.890 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:04.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:05.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:05.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:05.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 62 16 80580 80564 81.2071 69.75 0.0113415 0.0123076 2024-09-02T10:07:05.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:05.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:05.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:05.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:07:05.181 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:07:05.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:05.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:05.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:05.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:05.398 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:05.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:05.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:05 smithi151 ceph-mon[16639]: 3.1 scrub starts 2024-09-02T10:07:05.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2876985637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:05.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:05 smithi151 ceph-mon[16639]: 3.d scrub ok 2024-09-02T10:07:05.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:05.524 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:05.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:05.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:05.584 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:05 smithi187 ceph-mon[20072]: 3.1 scrub starts 2024-09-02T10:07:05.584 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2876985637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:05.584 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:05 smithi187 ceph-mon[20072]: 3.d scrub ok 2024-09-02T10:07:05.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:05 smithi162 ceph-mon[20279]: 3.1 scrub starts 2024-09-02T10:07:05.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2876985637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:05.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:05 smithi162 ceph-mon[20279]: 3.d scrub ok 2024-09-02T10:07:05.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:05.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:05.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:05.712 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:07:05.712 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:07:05.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:05.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:05.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:05.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:05.901 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:07:05.901 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:07:05.901 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:07:05.901 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:07:05.901 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:07:05.901 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:07:05.902 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:07:05.902 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:07:05.902 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:07:05.902 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:07:05.902 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:07:05.902 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:07:05.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:05.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:05.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:06.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:06.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 63 16 81470 81454 80.801 55.625 0.0658359 0.0123638 2024-09-02T10:07:06.049 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:06.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:06.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:06.181 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:06.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:06.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:06.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:06.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:06.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:06.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:06.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:06.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:06.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:06.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:06.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:06.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:06.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:06 smithi162 ceph-mon[20279]: pgmap v618: 47 pgs: 47 active+clean; 4.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:07:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:06 smithi162 ceph-mon[20279]: 3.e scrub starts 2024-09-02T10:07:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:06 smithi162 ceph-mon[20279]: 3.1 scrub ok 2024-09-02T10:07:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:06 smithi162 ceph-mon[20279]: 3.f scrub starts 2024-09-02T10:07:06.648 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:06 smithi187 ceph-mon[20072]: pgmap v618: 47 pgs: 47 active+clean; 4.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:07:06.648 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:06 smithi187 ceph-mon[20072]: 3.e scrub starts 2024-09-02T10:07:06.648 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:06 smithi187 ceph-mon[20072]: 3.1 scrub ok 2024-09-02T10:07:06.648 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:06 smithi187 ceph-mon[20072]: 3.f scrub starts 2024-09-02T10:07:06.675 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:06.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:06 smithi151 ceph-mon[16639]: pgmap v618: 47 pgs: 47 active+clean; 4.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:07:06.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:06 smithi151 ceph-mon[16639]: 3.e scrub starts 2024-09-02T10:07:06.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:06 smithi151 ceph-mon[16639]: 3.1 scrub ok 2024-09-02T10:07:06.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:06 smithi151 ceph-mon[16639]: 3.f scrub starts 2024-09-02T10:07:06.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:07:06.700 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:06.701 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:07:06.718 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:07:06.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-09-02T10:07:06.802 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:06.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:06.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:06.830 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:06.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:06.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:06.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:06.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:07.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 64 16 82629 82613 80.6702 72.4375 0.00997971 0.0123906 2024-09-02T10:07:07.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:07.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:07.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:07.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:07.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:07.216 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:07.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:07.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:07.301 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:07.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:07.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:07.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:07.366 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:07.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:07.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:07.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:07.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:07 smithi162 ceph-mon[20279]: 3.8 scrub starts 2024-09-02T10:07:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:07 smithi162 ceph-mon[20279]: 3.f scrub ok 2024-09-02T10:07:07.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:07.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:07 smithi151 ceph-mon[16639]: 3.8 scrub starts 2024-09-02T10:07:07.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:07 smithi151 ceph-mon[16639]: 3.f scrub ok 2024-09-02T10:07:07.623 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:07.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:07.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:07:07.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:07.755 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:07.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:07 smithi187 ceph-mon[20072]: 3.8 scrub starts 2024-09-02T10:07:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:07 smithi187 ceph-mon[20072]: 3.f scrub ok 2024-09-02T10:07:07.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:07.884 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:07.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:07.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:08.010 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:08.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:08.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:08.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 65 16 83857 83841 80.6098 76.75 0.00654493 0.0124001 2024-09-02T10:07:08.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:08.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:08.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:08.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:08.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:08.271 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:08.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:08.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:08.299 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:07:08.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:08.380 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:07:08.381 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:07:08.401 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:08.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:08.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:08.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:08.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:08 smithi151 ceph-mon[16639]: 3.e scrub ok 2024-09-02T10:07:08.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:08 smithi151 ceph-mon[16639]: pgmap v619: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-02T10:07:08.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:08 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:07:08.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:08 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:07:08.514 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3035222131' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:08.514 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:08 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:08.514 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:08 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:07:08.514 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:08 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:07:08.531 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:08.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:08.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:08 smithi162 ceph-mon[20279]: 3.e scrub ok 2024-09-02T10:07:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:08 smithi162 ceph-mon[20279]: pgmap v619: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-02T10:07:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:08 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:07:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:08 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:07:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3035222131' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:08 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:08 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:07:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:08 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:07:08.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:08.659 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:08.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:08.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:08 smithi187 ceph-mon[20072]: 3.e scrub ok 2024-09-02T10:07:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:08 smithi187 ceph-mon[20072]: pgmap v619: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-02T10:07:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:08 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:07:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:08 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:07:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3035222131' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:08 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:08 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:07:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:08 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:07:08.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:08.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:08.785 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:08.837 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:07:08.837 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:07:08.886 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:08.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:08.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:09.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:09.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 66 16 85096 85080 80.5617 77.4375 0.0154161 0.0124085 2024-09-02T10:07:09.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:09.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:09.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:09.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:09.171 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:09.172 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:09.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:09.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:09.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:09.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:09.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:09.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:09.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:09 smithi151 ceph-mon[16639]: 3.8 scrub ok 2024-09-02T10:07:09.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:09 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:09.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:09 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:07:09.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:09 smithi151 ceph-mon[16639]: osdmap e256: 12 total, 12 up, 11 in 2024-09-02T10:07:09.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:09 smithi151 ceph-mon[16639]: osdmap e257: 12 total, 12 up, 11 in 2024-09-02T10:07:09.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:09.559 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:09.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:09.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:09 smithi162 ceph-mon[20279]: 3.8 scrub ok 2024-09-02T10:07:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:09 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:09 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:07:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:09 smithi162 ceph-mon[20279]: osdmap e256: 12 total, 12 up, 11 in 2024-09-02T10:07:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:09 smithi162 ceph-mon[20279]: osdmap e257: 12 total, 12 up, 11 in 2024-09-02T10:07:09.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:09.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:07:09.687 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:09.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:09.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:09 smithi187 ceph-mon[20072]: 3.8 scrub ok 2024-09-02T10:07:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:09 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:09 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:07:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:09 smithi187 ceph-mon[20072]: osdmap e256: 12 total, 12 up, 11 in 2024-09-02T10:07:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:09 smithi187 ceph-mon[20072]: osdmap e257: 12 total, 12 up, 11 in 2024-09-02T10:07:09.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:09.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:09.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:09.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:09.912 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:09.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:09.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:10.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 67 16 86122 86106 80.3163 64.125 0.00395579 0.0124457 2024-09-02T10:07:10.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:10.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:10.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:10.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:10.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:10.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:10.198 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:10.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:10.328 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:10.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:10.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:10.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:10 smithi151 ceph-mon[16639]: pgmap v622: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-02T10:07:10.461 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:10.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:10.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:10.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:10.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:10.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:10.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:10 smithi162 ceph-mon[20279]: pgmap v622: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-02T10:07:10.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:10.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:10.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:10 smithi187 ceph-mon[20072]: pgmap v622: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-02T10:07:10.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:10.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:10.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:10.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:10.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:10.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:10.987 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:07:11.009 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:07:11.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-09-02T10:07:11.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 68 16 87044 87028 79.9825 57.625 0.0230652 0.012498 2024-09-02T10:07:11.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:11.092 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:11.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:11.194 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:11.220 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:11.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:11.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:11.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:11.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:11.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:11.490 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:11.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:11.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:11.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:11.623 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:11.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:11.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:11.668 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:11.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:11.753 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:11.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:11.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:11.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:11.880 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:11.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:11.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:12.008 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:12.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:12.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 69 16 88380 88364 80.0334 83.5 0.00421222 0.0124901 2024-09-02T10:07:12.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:12.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:12.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:12.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:07:12.239 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:12.264 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:12.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:12.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:12.391 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:12.392 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:12.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:07:12.437 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:07:12.460 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:07:12.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:07:12.460 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:12 smithi151 ceph-mon[16639]: pgmap v623: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-02T10:07:12.460 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/355510455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:12.485 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:07:12.485 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:07:12.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:12.516 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:12.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:12.541 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:07:12.542 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:07:12.569 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:07:12.569 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:07:12.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:12 smithi162 ceph-mon[20279]: pgmap v623: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-02T10:07:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/355510455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:12.599 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:07:12.599 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:07:12.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:12.632 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:07:12.632 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:07:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:12.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:12.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:12.664 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:07:12.665 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:07:12.696 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:07:12.696 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:07:12.728 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:07:12.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:07:12.744 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:07:12.744 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:07:12.744 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:07:12.744 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:07:12.744 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:07:12.744 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:07:12.744 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:07:12.745 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:07:12.745 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:07:12.745 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:07:12.745 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:07:12.745 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:07:12.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:12 smithi187 ceph-mon[20072]: pgmap v623: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-02T10:07:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/355510455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:12.762 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:07:12.762 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:07:12.783 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:12.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:12.791 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:07:12.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:07:12.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:12.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:12.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:12.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:13.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:13.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 70 16 89666 89650 80.0381 80.375 0.0101963 0.0124897 2024-09-02T10:07:13.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:13.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:13.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:13.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:13.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:13.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:13.231 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:07:13.231 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:07:13.231 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:07:13.231 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:07:13.232 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:07:13.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:13.310 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:13.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:13.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:13.437 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:13.438 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:13.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:13.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:13.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:13.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:13.692 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:13.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:13.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:13.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:13.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:13.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:13.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:13.954 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:13.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:14.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 71 15 90889 90874 79.9882 76.5 0.00462672 0.0124968 2024-09-02T10:07:14.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:14.081 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:14.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:14.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:14.208 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:14.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:14.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:14.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:14.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:14.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:14.470 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:14.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:14.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:14 smithi162 ceph-mon[20279]: pgmap v624: 47 pgs: 47 active+clean; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:07:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:14 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:07:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:14 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:07:14.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:14.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:14.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:14 smithi151 ceph-mon[16639]: pgmap v624: 47 pgs: 47 active+clean; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:07:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:14 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:07:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:14 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:07:14.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:14.726 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:14.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:14 smithi187 ceph-mon[20072]: pgmap v624: 47 pgs: 47 active+clean; 5.3 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:07:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:14 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:07:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:14 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:07:14.828 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:14.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:14.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:14.953 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:14.976 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:14.977 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:15.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 72 16 91859 91843 79.7184 60.5625 0.0146217 0.0125392 2024-09-02T10:07:15.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:15.106 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:15.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:07:15.144 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:07:15.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:15.236 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:15.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:15.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:15.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:15.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:15.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:15.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:15 smithi151 ceph-mon[16639]: osdmap e258: 12 total, 12 up, 11 in 2024-09-02T10:07:15.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:15.484 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:15.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:15.586 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:15.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:15 smithi162 ceph-mon[20279]: osdmap e258: 12 total, 12 up, 11 in 2024-09-02T10:07:15.613 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:15.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:15.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:15.667 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:15.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:07:15.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:15.739 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:15.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:15.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:15 smithi187 ceph-mon[20072]: osdmap e258: 12 total, 12 up, 11 in 2024-09-02T10:07:15.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:15.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:15.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:15.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:15.967 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:15.994 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:15.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:16.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 73 16 92831 92815 79.4585 60.75 0.0146294 0.0125805 2024-09-02T10:07:16.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:16.124 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:16.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:16.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:16.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:16.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:16.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:07:16.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:16.372 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:16.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:16.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:16.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:16.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:16.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:16 smithi162 ceph-mon[20279]: pgmap v626: 47 pgs: 47 active+clean; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:07:16.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:16.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:16.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:16.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:16.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:16 smithi151 ceph-mon[16639]: pgmap v626: 47 pgs: 47 active+clean; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:07:16.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:16 smithi187 ceph-mon[20072]: pgmap v626: 47 pgs: 47 active+clean; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:07:16.765 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:16.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:16.766 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:16.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:16.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:16.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:16.893 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:16.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:17.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:17.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:17.022 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:17.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 74 16 93823 93807 79.2225 62 0.0201579 0.0126183 2024-09-02T10:07:17.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:17.146 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:17.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:17.147 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:17.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:17.274 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:17.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:17.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:17.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:17.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:17.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:17.405 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:17.449 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:07:17.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:17.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:17.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:17.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:17.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2912614112' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:17.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:17.661 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:17.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2912614112' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:17.705 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2912614112' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:17.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:17.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:17.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:17.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:17.922 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:17.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:18.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:18.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 75 16 94907 94891 79.0694 67.75 0.00383724 0.012641 2024-09-02T10:07:18.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:18.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:18.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:18.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:18.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:18.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:18.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:18.305 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:18.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:18.407 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:18.431 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:18.432 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:18.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:18.557 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:18.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:18 smithi162 ceph-mon[20279]: pgmap v627: 47 pgs: 47 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-02T10:07:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:18 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2912614112' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:18 smithi162 ceph-mon[20279]: osdmap e259: 12 total, 12 up, 11 in 2024-09-02T10:07:18.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:18.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:18.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:18.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:18 smithi151 ceph-mon[16639]: pgmap v627: 47 pgs: 47 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-02T10:07:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:18 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2912614112' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:18 smithi151 ceph-mon[16639]: osdmap e259: 12 total, 12 up, 11 in 2024-09-02T10:07:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:18 smithi187 ceph-mon[20072]: pgmap v627: 47 pgs: 47 active+clean; 5.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s 2024-09-02T10:07:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:18 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2912614112' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:18 smithi187 ceph-mon[20072]: osdmap e259: 12 total, 12 up, 11 in 2024-09-02T10:07:18.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:18.809 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:18.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:18.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:18.937 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:18.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:19.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 76 16 95965 95949 78.899 66.125 0.0189283 0.0126676 2024-09-02T10:07:19.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:19.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:19.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:19.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:19.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:19.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:19.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:19.315 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:19.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:19.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:19.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:19.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:19.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:19.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:19.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:19.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:19.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:19.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:19.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:19.823 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:19.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:19.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:19.955 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:19.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:19.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:20.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 77 16 96900 96884 78.6332 58.4375 0.0060035 0.0127124 2024-09-02T10:07:20.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:20.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:20.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:20.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:20.144 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:07:20.144 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:07:20.145 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-02T10:07:20.145 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-02T10:07:20.145 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-02T10:07:20.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-02T10:07:20.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:20.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:20.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:20.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:20.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:20.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:20.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:20.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:20.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:20.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:20.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:20.487 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:07:20.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:20.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:20 smithi162 ceph-mon[20279]: pgmap v629: 47 pgs: 47 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:07:20.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:20 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:07:20.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:20 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:07:20.597 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:20.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:20.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:20.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:20 smithi151 ceph-mon[16639]: pgmap v629: 47 pgs: 47 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:07:20.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:20 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:07:20.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:20 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:07:20.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:20.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:20.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:20.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:20 smithi187 ceph-mon[20072]: pgmap v629: 47 pgs: 47 active+clean; 5.7 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:07:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:20 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:07:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:20 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:07:20.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:20.846 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:20.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:20.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:20.976 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:07:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:07:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:8373 2.1 e97/92 2:8eb82fd1:::benchmark_data_smithi187_45882_object8372:head v 97'536, mlcod=97'533)", 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.035752+0000", 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 340.93792637299998, 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16431514799999999, 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035752+0000", 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035752+0000", 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035753+0000", 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1999999999999998e-07 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.035789+0000", 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6365999999999997e-05 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.199174+0000", 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.163385155 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.199186+0000", 2024-09-02T10:07:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1626e-05 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.199261+0000", 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4898e-05 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.199284+0000", 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2804999999999999e-05 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.199999+0000", 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00071496199999999998 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.200067+0000", 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8416000000000004e-05 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:9159 2.8 e98/92 2:1c146a3c:::benchmark_data_smithi187_45882_object9158:head v 98'582, mlcod=98'580)", 2024-09-02T10:07:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:40.618746+0000", 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 340.35493268499999, 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.163030018, 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618746+0000", 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:20.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618746+0000", 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618746+0000", 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3499999999999996e-07 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.618850+0000", 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010351100000000001 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781192+0000", 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16234200400000001 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781196+0000", 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5290000000000002e-06 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:20.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781247+0000", 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1226e-05 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781276+0000", 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9014999999999999e-05 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781733+0000", 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045623300000000001 2024-09-02T10:07:20.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:40.781776+0000", 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2864999999999997e-05 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:27115 2.4 e105/92 2:211dbe49:::benchmark_data_smithi187_45882_object27114:head v 105'1674, mlcod=105'1671)", 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:54.684341+0000", 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 326.28933679900001, 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19103002599999999, 2024-09-02T10:07:20.986 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.684341+0000", 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.684341+0000", 2024-09-02T10:07:20.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.684342+0000", 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.684366+0000", 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3853999999999999e-05 2024-09-02T10:07:20.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.874351+0000", 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18998515999999999 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.874360+0000", 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7460000000000006e-06 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.874419+0000", 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8624e-05 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.874439+0000", 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0698000000000001e-05 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.875320+0000", 2024-09-02T10:07:20.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00088058199999999998 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:54.875371+0000", 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1518999999999999e-05 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:37137 2.8 e110/92 2:19842f3c:::benchmark_data_smithi187_45882_object37136:head v 110'2294, mlcod=110'2291)", 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:02.805961+0000", 2024-09-02T10:07:20.991 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 318.16771758900001, 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.160898445, 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.805961+0000", 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:20.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.805961+0000", 2024-09-02T10:07:20.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:20.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:20.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.805962+0000", 2024-09-02T10:07:20.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0100000000000003e-07 2024-09-02T10:07:20.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.806067+0000", 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105635 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.965743+0000", 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.159676075 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.965749+0000", 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5439999999999998e-06 2024-09-02T10:07:20.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.965794+0000", 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5435999999999997e-05 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.965808+0000", 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3380999999999999e-05 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:20.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.966824+0000", 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001015988 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:02.966859+0000", 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5485000000000003e-05 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:40737 2.c e111/92 2:307dfd04:::benchmark_data_smithi187_45882_object40736:head v 111'2615, mlcod=111'2612)", 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:05.722004+0000", 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 315.25167464600003, 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.168674661, 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.722004+0000", 2024-09-02T10:07:20.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.722004+0000", 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.722005+0000", 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.547e-06 2024-09-02T10:07:20.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.722112+0000", 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107323 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.889099+0000", 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16698626999999999 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:20.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.889108+0000", 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0469999999999999e-06 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.889176+0000", 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7788999999999998e-05 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.889198+0000", 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1970000000000001e-05 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.890635+0000", 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001437186 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:05.890678+0000", 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3529000000000003e-05 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54903 2.c e113/92 2:3cdee963:::benchmark_data_smithi187_45882_object54902:head v 113'3490, mlcod=113'3487)", 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.130749+0000", 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 303.84292936600002, 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.263935692, 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130749+0000", 2024-09-02T10:07:21.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130749+0000", 2024-09-02T10:07:21.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130750+0000", 2024-09-02T10:07:21.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1699999999999997e-07 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.130858+0000", 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108033 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.392762+0000", 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.261904201 2024-09-02T10:07:21.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.392768+0000", 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6810000000000001e-06 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.392838+0000", 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9361000000000002e-05 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.392859+0000", 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1509999999999999e-05 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.394641+0000", 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001781691 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.394685+0000", 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3498000000000003e-05 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.006 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:67911 2.4 e120/92 2:20b40a62:::benchmark_data_smithi187_45882_object67910:head v 120'4182, mlcod=120'4179)", 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:29.867694+0000", 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 291.10598442399998, 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23454814500000001, 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867694+0000", 2024-09-02T10:07:21.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:21.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867694+0000", 2024-09-02T10:07:21.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867695+0000", 2024-09-02T10:07:21.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2669999999999999e-06 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867729+0000", 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3627000000000002e-05 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.099825+0000", 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23209649800000001 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.099832+0000", 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8059999999999997e-06 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.099878+0000", 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6208999999999998e-05 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.099899+0000", 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0531e-05 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.102216+0000", 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023169240000000002 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.102242+0000", 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6282999999999999e-05 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.013 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:127291 2.8 e155/153 2:135f62a8:::benchmark_data_smithi187_45882_object29540:head v 155'7963, mlcod=105'1854)", 2024-09-02T10:07:21.014 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:44.998279+0000", 2024-09-02T10:07:21.014 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 215.975398719, 2024-09-02T10:07:21.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15613745700000001, 2024-09-02T10:07:21.014 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.014 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.014 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.014 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.998279+0000", 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.998279+0000", 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.998281+0000", 2024-09-02T10:07:21.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.296e-06 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:44.998287+0000", 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2990000000000002e-06 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.150388+0000", 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15210099099999999 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.150398+0000", 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0151e-05 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.150477+0000", 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8641999999999997e-05 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.150504+0000", 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7169999999999999e-05 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.154342+0000", 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003837549 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.154417+0000", 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5358999999999995e-05 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.018 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:141612 2.4 e169/131 2:218ceefe:::benchmark_data_smithi187_45882_object43861:head v 169'8782, mlcod=113'3590)", 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:14.899954+0000", 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 186.073724578, 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17027746399999999, 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.899954+0000", 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.899954+0000", 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.899955+0000", 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3990000000000001e-06 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:14.899965+0000", 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6069999999999997e-06 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.065161+0000", 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16519602799999999 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.065170+0000", 2024-09-02T10:07:21.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5380000000000008e-06 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.065278+0000", 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107441 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.065304+0000", 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6769000000000001e-05 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.070039+0000", 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047348110000000002 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:15.070231+0000", 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019187100000000001 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:152189 2.8 e181/153 2:1833c2b6:::benchmark_data_smithi187_45882_object54438:head v 181'9498, mlcod=114'3827)", 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:34.787034+0000", 2024-09-02T10:07:21.023 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 166.186644341, 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.170816568, 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.787034+0000", 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.787034+0000", 2024-09-02T10:07:21.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.787035+0000", 2024-09-02T10:07:21.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.09e-07 2024-09-02T10:07:21.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.787041+0000", 2024-09-02T10:07:21.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7819999999999996e-06 2024-09-02T10:07:21.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.954068+0000", 2024-09-02T10:07:21.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16702689800000001 2024-09-02T10:07:21.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.954084+0000", 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.592e-05 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.954292+0000", 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020735799999999999 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.954363+0000", 2024-09-02T10:07:21.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1605000000000003e-05 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.957819+0000", 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034561090000000002 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.957850+0000", 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1087000000000002e-05 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:167761 2.c e202/92 2:3f32c72e:::benchmark_data_smithi187_45882_object70010:head v 202'10555, mlcod=202'10552)", 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:03.800156+0000", 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 137.173521967, 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.148199367, 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.800156+0000", 2024-09-02T10:07:21.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.800156+0000", 2024-09-02T10:07:21.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.800157+0000", 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3900000000000003e-07 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.800165+0000", 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4529999999999996e-06 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.942683+0000", 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.142518123 2024-09-02T10:07:21.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.942689+0000", 2024-09-02T10:07:21.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6320000000000002e-06 2024-09-02T10:07:21.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.942820+0000", 2024-09-02T10:07:21.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013107200000000001 2024-09-02T10:07:21.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.942844+0000", 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3723e-05 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.948275+0000", 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054306700000000003 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:03.948356+0000", 2024-09-02T10:07:21.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0754999999999995e-05 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:193929 2.4 e215/188 2:2645495d:::benchmark_data_smithi187_45882_object96178:head v 215'12023, mlcod=215'12020)", 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:40.198754+0000", 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 100.77492386500001, 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20554975, 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.034 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.035 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:40.198754+0000", 2024-09-02T10:07:21.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.035 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.035 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:40.198754+0000", 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:40.198755+0000", 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4600000000000003e-07 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:40.198762+0000", 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1400000000000002e-06 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:40.399077+0000", 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20031413300000001 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:40.399085+0000", 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.901e-06 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:40.399357+0000", 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027194699999999998 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:40.399404+0000", 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6189000000000001e-05 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:40.404259+0000", 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004855425 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:40.404304+0000", 2024-09-02T10:07:21.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5068999999999998e-05 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:194907 2.8 e217/214 2:157c87af:::benchmark_data_smithi187_45882_object97156:head v 217'12188, mlcod=217'12185)", 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:42.113436+0000", 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 98.860242662999994, 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16862015999999999, 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.039 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.113436+0000", 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.113436+0000", 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.113436+0000", 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7000000000000004e-07 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.113442+0000", 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6910000000000002e-06 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.278401+0000", 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.164958722 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.278405+0000", 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2599999999999999e-06 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.278452+0000", 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7129000000000002e-05 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.278466+0000", 2024-09-02T10:07:21.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3434e-05 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.282012+0000", 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035462549999999999 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.282056+0000", 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3899000000000001e-05 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:194920 2.1 e217/200 2:867ed7f5:::benchmark_data_smithi187_45882_object97169:head v 217'12220, mlcod=217'12218)", 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:42.143347+0000", 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 98.830331670000007, 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16188584, 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.143347+0000", 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.143347+0000", 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.143348+0000", 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.142e-06 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.143355+0000", 2024-09-02T10:07:21.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4309999999999997e-06 2024-09-02T10:07:21.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.046 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.300128+0000", 2024-09-02T10:07:21.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15677295499999999 2024-09-02T10:07:21.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.300135+0000", 2024-09-02T10:07:21.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2439999999999996e-06 2024-09-02T10:07:21.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.300295+0000", 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015945400000000001 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.300340+0000", 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4854000000000001e-05 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.305176+0000", 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048363240000000004 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:42.305232+0000", 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6436e-05 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24976.0:12895 3.b e233/226 3:d7b97c02:::benchmark_data_smithi187_59989_object12894:head v 233'795, mlcod=233'791)", 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:06:11.029221+0000", 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 69.944456716999994, 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.2258809, 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.049 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.029221+0000", 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.029221+0000", 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.029222+0000", 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6199999999999997e-07 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.029245+0000", 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3227999999999999e-05 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.254270+0000", 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22502497699999999 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.254277+0000", 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8179999999999998e-06 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.254354+0000", 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6781999999999998e-05 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.254376+0000", 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1770999999999999e-05 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.255058+0000", 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068251 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.255102+0000", 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4051999999999998e-05 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.053 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:14010 3.6 3:62a51074:::benchmark_data_smithi187_59989_object14009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:06:11.914907+0000", 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 69.058771370000002, 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16507558899999999, 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 14010 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.914907+0000", 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.914907+0000", 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.914908+0000", 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.048e-06 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.914961+0000", 2024-09-02T10:07:21.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.274e-05 2024-09-02T10:07:21.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.056 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.056 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.914971+0000", 2024-09-02T10:07:21.056 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9380000000000004e-06 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.914975+0000", 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8890000000000001e-06 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.915025+0000", 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9440000000000001e-05 2024-09-02T10:07:21.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.915164+0000", 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013925499999999999 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.915308+0000", 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014425899999999999 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:21.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.916030+0000", 2024-09-02T10:07:21.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00072157999999999999 2024-09-02T10:07:21.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.059 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:21.059 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.936762+0000", 2024-09-02T10:07:21.059 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020732318999999999 2024-09-02T10:07:21.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.059 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.079923+0000", 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14316056699999999 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.079958+0000", 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4663999999999999e-05 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.060 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.079982+0000", 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4890000000000001e-05 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:14010 3.6 e233/226 ondisk, result = 0)", 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:06:11.916224+0000", 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 69.057454293000006, 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16376420899999999, 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:07:21.061 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.916224+0000", 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.916224+0000", 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.062 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.916225+0000", 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5199999999999996e-07 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.916229+0000", 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7870000000000001e-06 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.079875+0000", 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16364601000000001 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.063 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.079882+0000", 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1099999999999999e-06 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.079915+0000", 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3401000000000001e-05 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.079920+0000", 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5600000000000004e-06 2024-09-02T10:07:21.064 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.079988+0000", 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8589000000000003e-05 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:14020 3.6 3:658e24f1:::benchmark_data_smithi187_59989_object14019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:06:11.933241+0000", 2024-09-02T10:07:21.065 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 69.040437263000001, 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14808352699999999, 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 14020 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.066 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.933241+0000", 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.933241+0000", 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.933242+0000", 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0070000000000001e-06 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.067 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.933296+0000", 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3733999999999997e-05 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.933307+0000", 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1241e-05 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.068 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.933312+0000", 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3460000000000001e-06 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.933361+0000", 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8832000000000003e-05 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.933493+0000", 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013236000000000001 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.069 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.933641+0000", 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014737600000000001 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.934347+0000", 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00070574899999999998 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.938336+0000", 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039889890000000001 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.070 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.081253+0000", 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.142917462 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.081288+0000", 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5157999999999998e-05 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:12.081324+0000", 2024-09-02T10:07:21.071 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6273000000000003e-05 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24976.0:17974 3.b e234/226 3:d8b346fe:::benchmark_data_smithi187_59989_object17973:head v 234'1123, mlcod=234'1120)", 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:06:14.765161+0000", 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 66.208517236000006, 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.203681528, 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.072 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:14.765161+0000", 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:14.765161+0000", 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.073 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:14.765162+0000", 2024-09-02T10:07:21.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8299999999999996e-07 2024-09-02T10:07:21.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.074 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.074 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.074 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:14.765186+0000", 2024-09-02T10:07:21.074 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.391e-05 2024-09-02T10:07:21.074 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:14.967918+0000", 2024-09-02T10:07:21.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20273276800000001 2024-09-02T10:07:21.075 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.075 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.075 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.075 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:14.967924+0000", 2024-09-02T10:07:21.075 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8030000000000002e-06 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:14.967978+0000", 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3514000000000003e-05 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:14.968001+0000", 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3635e-05 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.076 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:14.968810+0000", 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080881499999999995 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:14.968842+0000", 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2400000000000001e-05 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.077 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24976.0:36136 3.b e238/226 3:da6bc728:::benchmark_data_smithi187_59989_object36135:head v 238'2261, mlcod=238'2258)", 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:06:27.831802+0000", 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 53.141875839999997, 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18183576400000001, 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:27.831802+0000", 2024-09-02T10:07:21.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:27.831802+0000", 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:27.831803+0000", 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2800000000000005e-07 2024-09-02T10:07:21.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:21.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:27.831826+0000", 2024-09-02T10:07:21.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2571000000000001e-05 2024-09-02T10:07:21.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:21.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:28.012493+0000", 2024-09-02T10:07:21.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.180666988 2024-09-02T10:07:21.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:28.012499+0000", 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8429999999999998e-06 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:28.012540+0000", 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1653000000000003e-05 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:28.012555+0000", 2024-09-02T10:07:21.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4218e-05 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:28.013604+0000", 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010491229999999999 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:28.013638+0000", 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4440000000000002e-05 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.082 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:21.083 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:21.083 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:07:21.083 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:21.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:21.083 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:07:21.084 INFO:teuthology.orchestra.run.smithi162.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-02T10:07:21.084 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:07:21.084 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 78 16 97668 97652 78.2404 48 0.0957229 0.0127751 2024-09-02T10:07:21.184 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:21.207 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:21.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:21.208 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:21.254 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:21.309 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:21.330 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:21.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:21.331 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:21.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:21.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:21.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:21.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:21.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:21.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:21.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:21.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:21 smithi162 ceph-mon[20279]: osdmap e260: 12 total, 12 up, 11 in 2024-09-02T10:07:21.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:21 smithi151 ceph-mon[16639]: osdmap e260: 12 total, 12 up, 11 in 2024-09-02T10:07:21.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:21.709 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:21.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:21.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:21 smithi187 ceph-mon[20072]: osdmap e260: 12 total, 12 up, 11 in 2024-09-02T10:07:21.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:21.836 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:21.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:21.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:21.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:21.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:21.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:21.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:07:22.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 79 16 98541 98525 77.9407 54.5625 0.010438 0.0128249 2024-09-02T10:07:22.072 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:22.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:22.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:22.098 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:22.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:22.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:22.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:22.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:22.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:22.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:22.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:22.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:22.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:07:22.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:22.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:22.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:22 smithi151 ceph-mon[16639]: pgmap v631: 47 pgs: 47 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-02T10:07:22.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:22 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:07:22.522 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:22 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:07:22.588 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:07:22.588 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:07:22.589 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:22.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:22.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:22 smithi162 ceph-mon[20279]: pgmap v631: 47 pgs: 47 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-02T10:07:22.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:22 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:07:22.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:22 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:07:22.624 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:22.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:22.628 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:07:22.628 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:07:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:22 smithi187 ceph-mon[20072]: pgmap v631: 47 pgs: 47 active+clean; 5.8 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 55 MiB/s wr, 882 op/s 2024-09-02T10:07:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:22 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:07:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:22 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:07:22.668 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:07:22.669 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:07:22.710 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:07:22.710 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:07:22.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:22.753 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:07:22.754 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:07:22.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:22.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:22.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:07:22.786 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:07:22.818 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:07:22.818 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:07:22.849 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:07:22.849 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:07:22.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:22.874 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:07:22.875 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:07:22.885 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:22.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:22.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:22.903 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:07:22.903 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:07:22.929 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:07:22.929 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:07:22.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:07:22.950 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:07:22.987 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:23.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:23.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:23.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:07:23.029881+0000 min lat: 0.0010322 max lat: 0.315006 avg lat: 0.0128422 2024-09-02T10:07:23.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:07:23.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 80 16 99571 99555 77.7711 64.375 0.00338409 0.0128422 2024-09-02T10:07:23.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:23.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:23.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:23.141 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:23.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:23.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:23.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:23.372 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:23.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:23.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:23.402 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:23.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:23.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:23.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:23.561 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:23 smithi187 ceph-mon[20072]: osdmap e261: 12 total, 12 up, 11 in 2024-09-02T10:07:23.561 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1688350014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:23.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:23.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:23.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:23.667 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:23.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:23 smithi151 ceph-mon[16639]: osdmap e261: 12 total, 12 up, 11 in 2024-09-02T10:07:23.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1688350014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:23.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:23.823 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:23 smithi162 ceph-mon[20279]: osdmap e261: 12 total, 12 up, 11 in 2024-09-02T10:07:23.823 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1688350014' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:23.838 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:23.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:23.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:23.940 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:23.970 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:23.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:24.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 81 16 100713 100697 77.692 71.375 0.0321637 0.0128647 2024-09-02T10:07:24.040 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:07:24.040 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:07:24.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:24.093 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:24.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:24.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:24.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:24.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:24.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:24.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:24.287 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:24.325 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:24.353 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:24.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:24.354 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:24.455 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:24.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:24.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:24.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:24.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:24.612 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:24.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:24.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:24 smithi151 ceph-mon[16639]: pgmap v633: 47 pgs: 47 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 55 MiB/s wr, 872 op/s 2024-09-02T10:07:24.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:24.741 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:24.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:24 smithi187 ceph-mon[20072]: pgmap v633: 47 pgs: 47 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 55 MiB/s wr, 872 op/s 2024-09-02T10:07:24.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:24.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:24 smithi162 ceph-mon[20279]: pgmap v633: 47 pgs: 47 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 55 MiB/s wr, 872 op/s 2024-09-02T10:07:24.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:24.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:24.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:07:24.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:24.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:24.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:25.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 82 16 101766 101750 77.5471 65.8125 0.00774459 0.0128905 2024-09-02T10:07:25.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:25.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:25.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:25.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:25.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:25.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:25.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:25.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:25.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:25.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:25.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:25.446 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:25.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:25.521 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:25.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:25.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:25.648 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:25.648 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:25.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:25.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:07:25.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:25.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:25.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:25.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:25.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:26.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:26.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:26.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:26.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 83 16 102636 102620 77.2678 54.375 0.0384215 0.0129259 2024-09-02T10:07:26.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:26.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:26.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:26.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:26.288 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:26.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:26.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:26.370 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:07:26.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:26.419 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:26.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:26.520 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:26.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:26 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:07:26.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:26 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:07:26.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:26 smithi151 ceph-mon[16639]: pgmap v634: 47 pgs: 47 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 53 MiB/s wr, 852 op/s 2024-09-02T10:07:26.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:26 smithi151 ceph-mon[16639]: osdmap e262: 12 total, 12 up, 11 in 2024-09-02T10:07:26.534 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/732735080' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:07:26.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:26.548 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:26.567 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:07:26.567 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:07:26.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:26.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:26.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:26.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:26 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:07:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:26 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:07:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:26 smithi187 ceph-mon[20072]: pgmap v634: 47 pgs: 47 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 53 MiB/s wr, 852 op/s 2024-09-02T10:07:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:26 smithi187 ceph-mon[20072]: osdmap e262: 12 total, 12 up, 11 in 2024-09-02T10:07:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/732735080' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:07:26.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:26.804 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:07:26.804 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:07:26.804 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-02T10:07:26.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:07:26.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:26.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:26.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:26.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:26 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:07:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:26 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:07:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:26 smithi162 ceph-mon[20279]: pgmap v634: 47 pgs: 47 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 53 MiB/s wr, 852 op/s 2024-09-02T10:07:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:26 smithi162 ceph-mon[20279]: osdmap e262: 12 total, 12 up, 11 in 2024-09-02T10:07:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/732735080' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:07:26.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:26.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:26.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:26.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:27.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 84 16 103832 103816 77.2378 74.75 0.020945 0.0129419 2024-09-02T10:07:27.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:27.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:27.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:27.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:27.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:27.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:27.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:27.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:27.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:27.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:27 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/732735080' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:07:27.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:27 smithi151 ceph-mon[16639]: osdmap e263: 12 total, 12 up, 11 in 2024-09-02T10:07:27.456 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:27.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:27.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:27.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:27.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:27.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:27.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:27.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:27.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:27.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:27 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/732735080' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:07:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:27 smithi187 ceph-mon[20072]: osdmap e263: 12 total, 12 up, 11 in 2024-09-02T10:07:27.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:27.834 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:27.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:27.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:27 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/732735080' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:07:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:27 smithi162 ceph-mon[20279]: osdmap e263: 12 total, 12 up, 11 in 2024-09-02T10:07:27.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:27.961 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:27.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:28.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:07:28.022 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:28.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 85 16 104883 104867 77.1019 65.6875 0.00544759 0.0129653 2024-09-02T10:07:28.058 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:28.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:28.092 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:28.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:28.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:28.223 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:28.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:28.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:28.353 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:28.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:28.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:28.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:28.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:28.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:28.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:28.613 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:28.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:28 smithi151 ceph-mon[16639]: pgmap v637: 47 pgs: 47 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:07:28.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:28 smithi151 ceph-mon[16639]: 3.b scrub starts 2024-09-02T10:07:28.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:28 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:07:28.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:28 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:07:28.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:28.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:28.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:28 smithi187 ceph-mon[20072]: pgmap v637: 47 pgs: 47 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:07:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:28 smithi187 ceph-mon[20072]: 3.b scrub starts 2024-09-02T10:07:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:28 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:07:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:28 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:07:28.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:28.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:28 smithi162 ceph-mon[20279]: pgmap v637: 47 pgs: 47 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 76 MiB/s wr, 1.22k op/s 2024-09-02T10:07:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:28 smithi162 ceph-mon[20279]: 3.b scrub starts 2024-09-02T10:07:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:28 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:07:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:28 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:07:28.866 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:28.868 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:28.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:28.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:28.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:29.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 86 16 105832 105816 76.895 59.3125 0.00912435 0.0129996 2024-09-02T10:07:29.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:29.130 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:29.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:29.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:29.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:07:29.260 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:29.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:29.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:29.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:29.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:29.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:29.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:29.537 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:29.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:29.593 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:29 smithi151 ceph-mon[16639]: 3.5 scrub starts 2024-09-02T10:07:29.594 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:29 smithi151 ceph-mon[16639]: 3.2 scrub starts 2024-09-02T10:07:29.594 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:29 smithi151 ceph-mon[16639]: 3.a scrub starts 2024-09-02T10:07:29.594 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:29 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:07:29.594 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:29 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:07:29.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:29.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:29.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:29.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:29.683 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:07:29.685 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:07:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:29 smithi187 ceph-mon[20072]: 3.5 scrub starts 2024-09-02T10:07:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:29 smithi187 ceph-mon[20072]: 3.2 scrub starts 2024-09-02T10:07:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:29 smithi187 ceph-mon[20072]: 3.a scrub starts 2024-09-02T10:07:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:29 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:07:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:29 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:07:29.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:29.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:29.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:29.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:29 smithi162 ceph-mon[20279]: 3.5 scrub starts 2024-09-02T10:07:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:29 smithi162 ceph-mon[20279]: 3.2 scrub starts 2024-09-02T10:07:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:29 smithi162 ceph-mon[20279]: 3.a scrub starts 2024-09-02T10:07:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:29 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:07:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:29 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:07:29.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:29.912 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:07:29.912 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:07:29.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:29.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:29.948 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:07:29.949 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:07:29.982 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:07:29.982 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:07:30.010 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:07:30.011 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:07:30.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 87 16 106655 106639 76.6023 51.4375 0.00542168 0.01305 2024-09-02T10:07:30.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:30.046 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:07:30.047 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:07:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:30.070 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:30.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:30.083 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:07:30.083 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:07:30.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:07:30.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:07:30.146 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:07:30.146 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:07:30.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:30.176 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:07:30.176 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:07:30.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:30.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:30.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:30.213 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:07:30.213 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:07:30.242 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:07:30.242 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:07:30.275 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:07:30.275 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:07:30.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:30.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:30.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:30.442 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:30.467 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:30.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:30.570 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:30.596 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:30.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:30.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:30 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:07:30.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:30 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:07:30.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:30 smithi151 ceph-mon[16639]: pgmap v638: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 58 MiB/s wr, 924 op/s 2024-09-02T10:07:30.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:30 smithi151 ceph-mon[16639]: 3.b scrub ok 2024-09-02T10:07:30.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:30 smithi151 ceph-mon[16639]: 3.7 scrub starts 2024-09-02T10:07:30.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:30 smithi151 ceph-mon[16639]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:07:30.632 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2273323048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:30.675 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":638,"stamp":"2024-09-02T10:07:29.152915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6780813856,"num_objects":103460,"num_object_clones":0,"num_object_copies":310380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103458,"num_write_kb":6621312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106310,"ondisk_log_size":106310,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":21333380,"kb_used_data":19763048,"kb_used_omap":17,"kb_used_meta":1570030,"kb_avail":1009863292,"statfs":{"total":1055945392128,"available":1034100011008,"internally_reserved":0,"allocated":20237361152,"data_stored":20228862706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1607711611},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":412221440,"num_objects":6290,"num_object_clones":0,"num_object_copies":18870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6290,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.802973"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":58,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.372663+0000","last_change":"2024-09-02T10:06:35.411836+0000","last_active":"2024-09-02T10:07:26.372663+0000","last_peered":"2024-09-02T10:07:26.372663+0000","last_clean":"2024-09-02T10:07:26.372663+0000","last_became_active":"2024-09-02T10:06:34.853129+0000","last_became_peered":"2024-09-02T10:06:34.853129+0000","last_unstale":"2024-09-02T10:07:26.372663+0000","last_undegraded":"2024-09-02T10:07:26.372663+0000","last_fullsized":"2024-09-02T10:07:26.372663+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.411763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.411763+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:52.412713+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":60,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.375102+0000","last_change":"2024-09-02T10:06:36.539700+0000","last_active":"2024-09-02T10:07:26.375102+0000","last_peered":"2024-09-02T10:07:26.375102+0000","last_clean":"2024-09-02T10:07:26.375102+0000","last_became_active":"2024-09-02T10:06:34.860248+0000","last_became_peered":"2024-09-02T10:06:34.860248+0000","last_unstale":"2024-09-02T10:07:26.375102+0000","last_undegraded":"2024-09-02T10:07:26.375102+0000","last_fullsized":"2024-09-02T10:07:26.375102+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:36.539650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:36.539650+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:57.048931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-02T10:07:22.350541+0000","last_change":"2024-09-02T10:06:37.123017+0000","last_active":"2024-09-02T10:07:22.350541+0000","last_peered":"2024-09-02T10:07:22.350541+0000","last_clean":"2024-09-02T10:07:22.350541+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:07:22.350541+0000","last_undegraded":"2024-09-02T10:07:22.350541+0000","last_fullsized":"2024-09-02T10:07:22.350541+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:37.122948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:37.122948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:42.474881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-02T10:07:22.350541+0000","last_change":"2024-09-02T10:06:36.130155+0000","last_active":"2024-09-02T10:07:22.350541+0000","last_peered":"2024-09-02T10:07:22.350541+0000","last_clean":"2024-09-02T10:07:22.350541+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:07:22.350541+0000","last_undegraded":"2024-09-02T10:07:22.350541+0000","last_fullsized":"2024-09-02T10:07:22.350541+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:36.130088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:36.130088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:55.421396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":56,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-02T10:07:22.353958+0000","last_change":"2024-09-02T10:06:35.749426+0000","last_active":"2024-09-02T10:07:22.353958+0000","last_peered":"2024-09-02T10:07:22.353958+0000","last_clean":"2024-09-02T10:07:22.353958+0000","last_became_active":"2024-09-02T10:06:34.852952+0000","last_became_peered":"2024-09-02T10:06:34.852952+0000","last_unstale":"2024-09-02T10:07:22.353958+0000","last_undegraded":"2024-09-02T10:07:22.353958+0000","last_fullsized":"2024-09-02T10:07:22.353958+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.749356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.749356+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:05.610484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":58,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.375335+0000","last_change":"2024-09-02T10:06:35.152787+0000","last_active":"2024-09-02T10:07:26.375335+0000","last_peered":"2024-09-02T10:07:26.375335+0000","last_clean":"2024-09-02T10:07:26.375335+0000","last_became_active":"2024-09-02T10:06:34.851298+0000","last_became_peered":"2024-09-02T10:06:34.851298+0000","last_unstale":"2024-09-02T10:07:26.375335+0000","last_undegraded":"2024-09-02T10:07:26.375335+0000","last_fullsized":"2024-09-02T10:07:26.375335+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.152605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.152605+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:36.176551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":417,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.371318+0000","last_change":"2024-09-02T10:06:35.086723+0000","last_active":"2024-09-02T10:07:26.371318+0000","last_peered":"2024-09-02T10:07:26.371318+0000","last_clean":"2024-09-02T10:07:26.371318+0000","last_became_active":"2024-09-02T10:06:34.859627+0000","last_became_peered":"2024-09-02T10:06:34.859627+0000","last_unstale":"2024-09-02T10:07:26.371318+0000","last_undegraded":"2024-09-02T10:07:26.371318+0000","last_fullsized":"2024-09-02T10:07:26.371318+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.086640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.086640+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:38.352824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":54,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-02T10:07:22.350639+0000","last_change":"2024-09-02T10:06:35.154338+0000","last_active":"2024-09-02T10:07:22.350639+0000","last_peered":"2024-09-02T10:07:22.350639+0000","last_clean":"2024-09-02T10:07:22.350639+0000","last_became_active":"2024-09-02T10:06:34.849680+0000","last_became_peered":"2024-09-02T10:06:34.849680+0000","last_unstale":"2024-09-02T10:07:22.350639+0000","last_undegraded":"2024-09-02T10:07:22.350639+0000","last_fullsized":"2024-09-02T10:07:22.350639+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.154262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.154262+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:03.820638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":58,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.375104+0000","last_change":"2024-09-02T10:06:35.567150+0000","last_active":"2024-09-02T10:07:26.375104+0000","last_peered":"2024-09-02T10:07:26.375104+0000","last_clean":"2024-09-02T10:07:26.375104+0000","last_became_active":"2024-09-02T10:06:34.850895+0000","last_became_peered":"2024-09-02T10:06:34.850895+0000","last_unstale":"2024-09-02T10:07:26.375104+0000","last_undegraded":"2024-09-02T10:07:26.375104+0000","last_fullsized":"2024-09-02T10:07:26.375104+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.567106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.567106+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:01.997964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":56,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-02T10:07:25.373950+0000","last_change":"2024-09-02T10:06:34.985266+0000","last_active":"2024-09-02T10:07:25.373950+0000","last_peered":"2024-09-02T10:07:25.373950+0000","last_clean":"2024-09-02T10:07:25.373950+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:07:25.373950+0000","last_undegraded":"2024-09-02T10:07:25.373950+0000","last_fullsized":"2024-09-02T10:07:25.373950+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.985198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.985198+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:47.131601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":305,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.425390+0000","last_change":"2024-09-02T10:06:01.163588+0000","last_active":"2024-09-02T10:07:26.425390+0000","last_peered":"2024-09-02T10:07:26.425390+0000","last_clean":"2024-09-02T10:07:26.425390+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:07:26.425390+0000","last_undegraded":"2024-09-02T10:07:26.425390+0000","last_fullsized":"2024-09-02T10:07:26.425390+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.163525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.163525+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:28.752619+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":391,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.374956+0000","last_change":"2024-09-02T10:06:34.583354+0000","last_active":"2024-09-02T10:07:26.374956+0000","last_peered":"2024-09-02T10:07:26.374956+0000","last_clean":"2024-09-02T10:07:26.374956+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:07:26.374956+0000","last_undegraded":"2024-09-02T10:07:26.374956+0000","last_fullsized":"2024-09-02T10:07:26.374956+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.583305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.583305+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:36.008223+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":327,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.371303+0000","last_change":"2024-09-02T10:06:03.254398+0000","last_active":"2024-09-02T10:07:26.371303+0000","last_peered":"2024-09-02T10:07:26.371303+0000","last_clean":"2024-09-02T10:07:26.371303+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:07:26.371303+0000","last_undegraded":"2024-09-02T10:07:26.371303+0000","last_fullsized":"2024-09-02T10:07:26.371303+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.254327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.254327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:30.078652+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":315,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.375635+0000","last_change":"2024-09-02T10:06:41.293858+0000","last_active":"2024-09-02T10:07:26.375635+0000","last_peered":"2024-09-02T10:07:26.375635+0000","last_clean":"2024-09-02T10:07:26.375635+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:07:26.375635+0000","last_undegraded":"2024-09-02T10:07:26.375635+0000","last_fullsized":"2024-09-02T10:07:26.375635+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:41.293787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:41.293787+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:05.071456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":320,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.371538+0000","last_change":"2024-09-02T10:06:01.152342+0000","last_active":"2024-09-02T10:07:26.371538+0000","last_peered":"2024-09-02T10:07:26.371538+0000","last_clean":"2024-09-02T10:07:26.371538+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:07:26.371538+0000","last_undegraded":"2024-09-02T10:07:26.371538+0000","last_fullsized":"2024-09-02T10:07:26.371538+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.152278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.152278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"263'6474","reported_seq":6558,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:27.031436+0000","last_change":"2024-09-02T10:05:56.584135+0000","last_active":"2024-09-02T10:07:27.031436+0000","last_peered":"2024-09-02T10:07:27.031436+0000","last_clean":"2024-09-02T10:07:27.031436+0000","last_became_active":"2024-09-02T10:05:56.583663+0000","last_became_peered":"2024-09-02T10:05:56.583663+0000","last_unstale":"2024-09-02T10:07:27.031436+0000","last_undegraded":"2024-09-02T10:07:27.031436+0000","last_fullsized":"2024-09-02T10:07:27.031436+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":6474,"log_dups_size":0,"ondisk_log_size":6474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:28.702077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424280064,"num_objects":6474,"num_object_clones":0,"num_object_copies":19422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6474,"num_write_kb":414336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":406,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.371584+0000","last_change":"2024-09-02T10:06:33.072793+0000","last_active":"2024-09-02T10:07:26.371584+0000","last_peered":"2024-09-02T10:07:26.371584+0000","last_clean":"2024-09-02T10:07:26.371584+0000","last_became_active":"2024-09-02T10:06:32.873327+0000","last_became_peered":"2024-09-02T10:06:32.873327+0000","last_unstale":"2024-09-02T10:07:26.371584+0000","last_undegraded":"2024-09-02T10:07:26.371584+0000","last_fullsized":"2024-09-02T10:07:26.371584+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.072734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.072734+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:59.473552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"263'6536","reported_seq":6674,"reported_epoch":263,"state":"active+clean+scrubbing","last_fresh":"2024-09-02T10:07:28.354337+0000","last_change":"2024-09-02T10:07:27.441571+0000","last_active":"2024-09-02T10:07:28.354337+0000","last_peered":"2024-09-02T10:07:28.354337+0000","last_clean":"2024-09-02T10:07:28.354337+0000","last_became_active":"2024-09-02T10:05:56.583278+0000","last_became_peered":"2024-09-02T10:05:56.583278+0000","last_unstale":"2024-09-02T10:07:28.354337+0000","last_undegraded":"2024-09-02T10:07:28.354337+0000","last_fullsized":"2024-09-02T10:07:28.354337+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":4899,"log_size":6536,"log_dups_size":0,"ondisk_log_size":6536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428343296,"num_objects":6536,"num_object_clones":0,"num_object_copies":19608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6536,"num_write_kb":418304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":528,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-02T10:07:25.373532+0000","last_change":"2024-09-02T10:06:32.998782+0000","last_active":"2024-09-02T10:07:25.373532+0000","last_peered":"2024-09-02T10:07:25.373532+0000","last_clean":"2024-09-02T10:07:25.373532+0000","last_became_active":"2024-09-02T10:06:32.871027+0000","last_became_peered":"2024-09-02T10:06:32.871027+0000","last_unstale":"2024-09-02T10:07:25.373532+0000","last_undegraded":"2024-09-02T10:07:25.373532+0000","last_fullsized":"2024-09-02T10:07:25.373532+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:32.998705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:32.998705+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:33.587122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"263'6462","reported_seq":6607,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.887688+0000","last_change":"2024-09-02T10:07:07.774469+0000","last_active":"2024-09-02T10:07:26.887688+0000","last_peered":"2024-09-02T10:07:26.887688+0000","last_clean":"2024-09-02T10:07:26.887688+0000","last_became_active":"2024-09-02T10:05:56.583249+0000","last_became_peered":"2024-09-02T10:05:56.583249+0000","last_unstale":"2024-09-02T10:07:26.887688+0000","last_undegraded":"2024-09-02T10:07:26.887688+0000","last_fullsized":"2024-09-02T10:07:26.887688+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5253","last_scrub_stamp":"2024-09-02T10:07:07.774339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:07.774339+0000","objects_scrubbed":5197,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:26.717286+0000","scrub_duration":1300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423493632,"num_objects":6462,"num_object_clones":0,"num_object_copies":19386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6462,"num_write_kb":413568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":164,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.375566+0000","last_change":"2024-09-02T10:06:33.026327+0000","last_active":"2024-09-02T10:07:26.375566+0000","last_peered":"2024-09-02T10:07:26.375566+0000","last_clean":"2024-09-02T10:07:26.375566+0000","last_became_active":"2024-09-02T10:06:32.870387+0000","last_became_peered":"2024-09-02T10:06:32.870387+0000","last_unstale":"2024-09-02T10:07:26.375566+0000","last_undegraded":"2024-09-02T10:07:26.375566+0000","last_fullsized":"2024-09-02T10:07:26.375566+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.026282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.026282+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:33.943423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"261'6279","reported_seq":6360,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-02T10:07:24.187724+0000","last_change":"2024-09-02T10:05:56.584669+0000","last_active":"2024-09-02T10:07:24.187724+0000","last_peered":"2024-09-02T10:07:24.187724+0000","last_clean":"2024-09-02T10:07:24.187724+0000","last_became_active":"2024-09-02T10:05:56.584313+0000","last_became_peered":"2024-09-02T10:05:56.584313+0000","last_unstale":"2024-09-02T10:07:24.187724+0000","last_undegraded":"2024-09-02T10:07:24.187724+0000","last_fullsized":"2024-09-02T10:07:24.187724+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":6279,"log_dups_size":0,"ondisk_log_size":6279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:28.205155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411500544,"num_objects":6279,"num_object_clones":0,"num_object_copies":18837,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6279,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6279,"num_write_kb":401856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":319,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.372503+0000","last_change":"2024-09-02T10:06:33.317499+0000","last_active":"2024-09-02T10:07:26.372503+0000","last_peered":"2024-09-02T10:07:26.372503+0000","last_clean":"2024-09-02T10:07:26.372503+0000","last_became_active":"2024-09-02T10:06:32.874387+0000","last_became_peered":"2024-09-02T10:06:32.874387+0000","last_unstale":"2024-09-02T10:07:26.372503+0000","last_undegraded":"2024-09-02T10:07:26.372503+0000","last_fullsized":"2024-09-02T10:07:26.372503+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.317426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.317426+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:38.072563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"263'6462","reported_seq":6546,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.496834+0000","last_change":"2024-09-02T10:05:56.579293+0000","last_active":"2024-09-02T10:07:26.496834+0000","last_peered":"2024-09-02T10:07:26.496834+0000","last_clean":"2024-09-02T10:07:26.496834+0000","last_became_active":"2024-09-02T10:05:56.578973+0000","last_became_peered":"2024-09-02T10:05:56.578973+0000","last_unstale":"2024-09-02T10:07:26.496834+0000","last_undegraded":"2024-09-02T10:07:26.496834+0000","last_fullsized":"2024-09-02T10:07:26.496834+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":6462,"log_dups_size":0,"ondisk_log_size":6462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:31.174196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423493632,"num_objects":6462,"num_object_clones":0,"num_object_copies":19386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6462,"num_write_kb":413568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":469,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.372624+0000","last_change":"2024-09-02T10:07:07.536258+0000","last_active":"2024-09-02T10:07:26.372624+0000","last_peered":"2024-09-02T10:07:26.372624+0000","last_clean":"2024-09-02T10:07:26.372624+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:07:26.372624+0000","last_undegraded":"2024-09-02T10:07:26.372624+0000","last_fullsized":"2024-09-02T10:07:26.372624+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:07.536194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:07.536194+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:25.174745+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"263'6531","reported_seq":6615,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:28.334421+0000","last_change":"2024-09-02T10:05:56.585087+0000","last_active":"2024-09-02T10:07:28.334421+0000","last_peered":"2024-09-02T10:07:28.334421+0000","last_clean":"2024-09-02T10:07:28.334421+0000","last_became_active":"2024-09-02T10:05:56.584888+0000","last_became_peered":"2024-09-02T10:05:56.584888+0000","last_unstale":"2024-09-02T10:07:28.334421+0000","last_undegraded":"2024-09-02T10:07:28.334421+0000","last_fullsized":"2024-09-02T10:07:28.334421+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":6531,"log_dups_size":0,"ondisk_log_size":6531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:29.414225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":428015616,"num_objects":6531,"num_object_clones":0,"num_object_copies":19593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6531,"num_write_kb":417984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":313,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.428781+0000","last_change":"2024-09-02T10:06:32.983641+0000","last_active":"2024-09-02T10:07:26.428781+0000","last_peered":"2024-09-02T10:07:26.428781+0000","last_clean":"2024-09-02T10:07:26.428781+0000","last_became_active":"2024-09-02T10:06:32.870858+0000","last_became_peered":"2024-09-02T10:06:32.870858+0000","last_unstale":"2024-09-02T10:07:26.428781+0000","last_undegraded":"2024-09-02T10:07:26.428781+0000","last_fullsized":"2024-09-02T10:07:26.428781+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:32.983547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:32.983547+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:40.190690+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"263'6716","reported_seq":6801,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:27.864047+0000","last_change":"2024-09-02T10:05:56.581266+0000","last_active":"2024-09-02T10:07:27.864047+0000","last_peered":"2024-09-02T10:07:27.864047+0000","last_clean":"2024-09-02T10:07:27.864047+0000","last_became_active":"2024-09-02T10:05:56.580910+0000","last_became_peered":"2024-09-02T10:05:56.580910+0000","last_unstale":"2024-09-02T10:07:27.864047+0000","last_undegraded":"2024-09-02T10:07:27.864047+0000","last_fullsized":"2024-09-02T10:07:27.864047+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:29.421323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440139776,"num_objects":6716,"num_object_clones":0,"num_object_copies":20148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6716,"num_write_kb":429824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":164,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.375831+0000","last_change":"2024-09-02T10:06:34.057029+0000","last_active":"2024-09-02T10:07:26.375831+0000","last_peered":"2024-09-02T10:07:26.375831+0000","last_clean":"2024-09-02T10:07:26.375831+0000","last_became_active":"2024-09-02T10:06:32.871735+0000","last_became_peered":"2024-09-02T10:06:32.871735+0000","last_unstale":"2024-09-02T10:07:26.375831+0000","last_undegraded":"2024-09-02T10:07:26.375831+0000","last_fullsized":"2024-09-02T10:07:26.375831+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.056985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.056985+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:58.472446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"263'6442","reported_seq":6526,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.831503+0000","last_change":"2024-09-02T10:05:56.690497+0000","last_active":"2024-09-02T10:07:26.831503+0000","last_peered":"2024-09-02T10:07:26.831503+0000","last_clean":"2024-09-02T10:07:26.831503+0000","last_became_active":"2024-09-02T10:05:56.690229+0000","last_became_peered":"2024-09-02T10:05:56.690229+0000","last_unstale":"2024-09-02T10:07:26.831503+0000","last_undegraded":"2024-09-02T10:07:26.831503+0000","last_fullsized":"2024-09-02T10:07:26.831503+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":6442,"log_dups_size":0,"ondisk_log_size":6442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422182912,"num_objects":6442,"num_object_clones":0,"num_object_copies":19326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6442,"num_write_kb":412288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":436,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.379032+0000","last_change":"2024-09-02T10:06:33.444679+0000","last_active":"2024-09-02T10:07:26.379032+0000","last_peered":"2024-09-02T10:07:26.379032+0000","last_clean":"2024-09-02T10:07:26.379032+0000","last_became_active":"2024-09-02T10:06:32.874081+0000","last_became_peered":"2024-09-02T10:06:32.874081+0000","last_unstale":"2024-09-02T10:07:26.379032+0000","last_undegraded":"2024-09-02T10:07:26.379032+0000","last_fullsized":"2024-09-02T10:07:26.379032+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.444635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.444635+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:57.766179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"261'6382","reported_seq":6463,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-02T10:07:23.973752+0000","last_change":"2024-09-02T10:05:56.582278+0000","last_active":"2024-09-02T10:07:23.973752+0000","last_peered":"2024-09-02T10:07:23.973752+0000","last_clean":"2024-09-02T10:07:23.973752+0000","last_became_active":"2024-09-02T10:05:56.581969+0000","last_became_peered":"2024-09-02T10:05:56.581969+0000","last_unstale":"2024-09-02T10:07:23.973752+0000","last_undegraded":"2024-09-02T10:07:23.973752+0000","last_fullsized":"2024-09-02T10:07:23.973752+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":6382,"log_dups_size":0,"ondisk_log_size":6382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:25.630714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418250752,"num_objects":6382,"num_object_clones":0,"num_object_copies":19146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6382,"num_write_kb":408448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":150,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.375551+0000","last_change":"2024-09-02T10:06:03.857986+0000","last_active":"2024-09-02T10:07:26.375551+0000","last_peered":"2024-09-02T10:07:26.375551+0000","last_clean":"2024-09-02T10:07:26.375551+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:07:26.375551+0000","last_undegraded":"2024-09-02T10:07:26.375551+0000","last_fullsized":"2024-09-02T10:07:26.375551+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:03.857913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:03.857913+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:26.940815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"263'6376","reported_seq":6459,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.918696+0000","last_change":"2024-09-02T10:05:56.876782+0000","last_active":"2024-09-02T10:07:26.918696+0000","last_peered":"2024-09-02T10:07:26.918696+0000","last_clean":"2024-09-02T10:07:26.918696+0000","last_became_active":"2024-09-02T10:05:56.876397+0000","last_became_peered":"2024-09-02T10:05:56.876397+0000","last_unstale":"2024-09-02T10:07:26.918696+0000","last_undegraded":"2024-09-02T10:07:26.918696+0000","last_fullsized":"2024-09-02T10:07:26.918696+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:31.212443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417857536,"num_objects":6376,"num_object_clones":0,"num_object_copies":19128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6376,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":162,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-02T10:07:25.373717+0000","last_change":"2024-09-02T10:06:01.039876+0000","last_active":"2024-09-02T10:07:25.373717+0000","last_peered":"2024-09-02T10:07:25.373717+0000","last_clean":"2024-09-02T10:07:25.373717+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:07:25.373717+0000","last_undegraded":"2024-09-02T10:07:25.373717+0000","last_fullsized":"2024-09-02T10:07:25.373717+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:01.039815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:01.039815+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:30.265758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"263'6407","reported_seq":6549,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.511124+0000","last_change":"2024-09-02T10:07:05.712175+0000","last_active":"2024-09-02T10:07:26.511124+0000","last_peered":"2024-09-02T10:07:26.511124+0000","last_clean":"2024-09-02T10:07:26.511124+0000","last_became_active":"2024-09-02T10:05:56.581229+0000","last_became_peered":"2024-09-02T10:05:56.581229+0000","last_unstale":"2024-09-02T10:07:26.511124+0000","last_undegraded":"2024-09-02T10:07:26.511124+0000","last_fullsized":"2024-09-02T10:07:26.511124+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5000","last_scrub_stamp":"2024-09-02T10:07:05.712114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:05.712114+0000","objects_scrubbed":4957,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:31.644516+0000","scrub_duration":1394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419889152,"num_objects":6407,"num_object_clones":0,"num_object_copies":19221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":410048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":378,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.375188+0000","last_change":"2024-09-02T10:06:15.608866+0000","last_active":"2024-09-02T10:07:26.375188+0000","last_peered":"2024-09-02T10:07:26.375188+0000","last_clean":"2024-09-02T10:07:26.375188+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:07:26.375188+0000","last_undegraded":"2024-09-02T10:07:26.375188+0000","last_fullsized":"2024-09-02T10:07:26.375188+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:15.608802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:15.608802+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:31.267880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"263'6660","reported_seq":6745,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:28.342698+0000","last_change":"2024-09-02T10:05:56.876935+0000","last_active":"2024-09-02T10:07:28.342698+0000","last_peered":"2024-09-02T10:07:28.342698+0000","last_clean":"2024-09-02T10:07:28.342698+0000","last_became_active":"2024-09-02T10:05:56.876573+0000","last_became_peered":"2024-09-02T10:05:56.876573+0000","last_unstale":"2024-09-02T10:07:28.342698+0000","last_undegraded":"2024-09-02T10:07:28.342698+0000","last_fullsized":"2024-09-02T10:07:28.342698+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":6660,"log_dups_size":0,"ondisk_log_size":6660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436469760,"num_objects":6660,"num_object_clones":0,"num_object_copies":19980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6660,"num_write_kb":426240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":173,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-02T10:07:25.372627+0000","last_change":"2024-09-02T10:06:04.062386+0000","last_active":"2024-09-02T10:07:25.372627+0000","last_peered":"2024-09-02T10:07:25.372627+0000","last_clean":"2024-09-02T10:07:25.372627+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:07:25.372627+0000","last_undegraded":"2024-09-02T10:07:25.372627+0000","last_fullsized":"2024-09-02T10:07:25.372627+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:04.062325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:04.062325+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:27.716198+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"262'6476","reported_seq":6616,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-02T10:07:25.828710+0000","last_change":"2024-09-02T10:07:04.549316+0000","last_active":"2024-09-02T10:07:25.828710+0000","last_peered":"2024-09-02T10:07:25.828710+0000","last_clean":"2024-09-02T10:07:25.828710+0000","last_became_active":"2024-09-02T10:05:56.876540+0000","last_became_peered":"2024-09-02T10:05:56.876540+0000","last_unstale":"2024-09-02T10:07:25.828710+0000","last_undegraded":"2024-09-02T10:07:25.828710+0000","last_fullsized":"2024-09-02T10:07:25.828710+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'4967","last_scrub_stamp":"2024-09-02T10:07:04.549227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:04.549227+0000","objects_scrubbed":4900,"log_size":6476,"log_dups_size":0,"ondisk_log_size":6476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:16.373684+0000","scrub_duration":1666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424411136,"num_objects":6476,"num_object_clones":0,"num_object_copies":19428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6476,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6476,"num_write_kb":414464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":407,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.378886+0000","last_change":"2024-09-02T10:06:04.265827+0000","last_active":"2024-09-02T10:07:26.378886+0000","last_peered":"2024-09-02T10:07:26.378886+0000","last_clean":"2024-09-02T10:07:26.378886+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:07:26.378886+0000","last_undegraded":"2024-09-02T10:07:26.378886+0000","last_fullsized":"2024-09-02T10:07:26.378886+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:04.265776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:04.265776+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:27.669257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"263'6452","reported_seq":6536,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.851327+0000","last_change":"2024-09-02T10:05:56.586356+0000","last_active":"2024-09-02T10:07:26.851327+0000","last_peered":"2024-09-02T10:07:26.851327+0000","last_clean":"2024-09-02T10:07:26.851327+0000","last_became_active":"2024-09-02T10:05:56.585883+0000","last_became_peered":"2024-09-02T10:05:56.585883+0000","last_unstale":"2024-09-02T10:07:26.851327+0000","last_undegraded":"2024-09-02T10:07:26.851327+0000","last_fullsized":"2024-09-02T10:07:26.851327+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:05:55.567155+0000","objects_scrubbed":0,"log_size":6452,"log_dups_size":0,"ondisk_log_size":6452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:27.107887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422838272,"num_objects":6452,"num_object_clones":0,"num_object_copies":19356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6452,"num_write_kb":412928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":324,"reported_epoch":261,"state":"active+clean","last_fresh":"2024-09-02T10:07:22.354048+0000","last_change":"2024-09-02T10:06:34.728115+0000","last_active":"2024-09-02T10:07:22.354048+0000","last_peered":"2024-09-02T10:07:22.354048+0000","last_clean":"2024-09-02T10:07:22.354048+0000","last_became_active":"2024-09-02T10:06:32.874830+0000","last_became_peered":"2024-09-02T10:06:32.874830+0000","last_unstale":"2024-09-02T10:07:22.354048+0000","last_undegraded":"2024-09-02T10:07:22.354048+0000","last_fullsized":"2024-09-02T10:07:22.354048+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.728059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.728059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:46.599903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"262'6363","reported_seq":6506,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-02T10:07:25.810477+0000","last_change":"2024-09-02T10:07:06.845356+0000","last_active":"2024-09-02T10:07:25.810477+0000","last_peered":"2024-09-02T10:07:25.810477+0000","last_clean":"2024-09-02T10:07:25.810477+0000","last_became_active":"2024-09-02T10:05:56.584450+0000","last_became_peered":"2024-09-02T10:05:56.584450+0000","last_unstale":"2024-09-02T10:07:25.810477+0000","last_undegraded":"2024-09-02T10:07:25.810477+0000","last_fullsized":"2024-09-02T10:07:25.810477+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5143","last_scrub_stamp":"2024-09-02T10:07:06.845283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:06.845283+0000","objects_scrubbed":5114,"log_size":6363,"log_dups_size":0,"ondisk_log_size":6363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:09.749906+0000","scrub_duration":1004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417005568,"num_objects":6363,"num_object_clones":0,"num_object_copies":19089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6363,"num_write_kb":407232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":187,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-09-02T10:07:25.374051+0000","last_change":"2024-09-02T10:06:34.005118+0000","last_active":"2024-09-02T10:07:25.374051+0000","last_peered":"2024-09-02T10:07:25.374051+0000","last_clean":"2024-09-02T10:07:25.374051+0000","last_became_active":"2024-09-02T10:06:32.876151+0000","last_became_peered":"2024-09-02T10:06:32.876151+0000","last_unstale":"2024-09-02T10:07:25.374051+0000","last_undegraded":"2024-09-02T10:07:25.374051+0000","last_fullsized":"2024-09-02T10:07:25.374051+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.005073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.005073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:44.452325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"263'6440","reported_seq":6583,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.953087+0000","last_change":"2024-09-02T10:07:06.655925+0000","last_active":"2024-09-02T10:07:26.953087+0000","last_peered":"2024-09-02T10:07:26.953087+0000","last_clean":"2024-09-02T10:07:26.953087+0000","last_became_active":"2024-09-02T10:05:56.585870+0000","last_became_peered":"2024-09-02T10:05:56.585870+0000","last_unstale":"2024-09-02T10:07:26.953087+0000","last_undegraded":"2024-09-02T10:07:26.953087+0000","last_fullsized":"2024-09-02T10:07:26.953087+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5095","last_scrub_stamp":"2024-09-02T10:07:06.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:06.655797+0000","objects_scrubbed":5054,"log_size":6440,"log_dups_size":0,"ondisk_log_size":6440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:08.831810+0000","scrub_duration":1179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422051840,"num_objects":6440,"num_object_clones":0,"num_object_copies":19320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6440,"num_write_kb":412160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":202,"reported_epoch":263,"state":"active+clean","last_fresh":"2024-09-02T10:07:26.379256+0000","last_change":"2024-09-02T10:06:33.485010+0000","last_active":"2024-09-02T10:07:26.379256+0000","last_peered":"2024-09-02T10:07:26.379256+0000","last_clean":"2024-09-02T10:07:26.379256+0000","last_became_active":"2024-09-02T10:06:32.875557+0000","last_became_peered":"2024-09-02T10:06:32.875557+0000","last_unstale":"2024-09-02T10:07:26.379256+0000","last_undegraded":"2024-09-02T10:07:26.379256+0000","last_fullsized":"2024-09-02T10:07:26.379256+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.484964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.484964+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:56.433474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":6780223488,"num_objects":103458,"num_object_clones":0,"num_object_copies":310374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103458,"num_write_kb":6621312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20320288768,"data_stored":20320288768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103458,"ondisk_log_size":103458,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383768,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3469656,"kb_used_data":3285764,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":90275496,"statfs":{"total":95995035648,"available":92442107904,"internally_reserved":0,"allocated":3364622336,"data_stored":3363851974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":188283332},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612701,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841634,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1428900,"kb_used_data":1263056,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92316252,"statfs":{"total":95995035648,"available":94531842048,"internally_reserved":0,"allocated":1293369344,"data_stored":1292594394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":169802176},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070567,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897864,"kb_used_data":804216,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92847288,"statfs":{"total":95995035648,"available":95075622912,"internally_reserved":0,"allocated":823517184,"data_stored":822751376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299498,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1779996,"kb_used_data":1640524,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91965156,"statfs":{"total":95995035648,"available":94172319744,"internally_reserved":0,"allocated":1679896576,"data_stored":1679122670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":142801354},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528431,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2184936,"kb_used_data":2073168,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91560216,"statfs":{"total":95995035648,"available":93757661184,"internally_reserved":0,"allocated":2122924032,"data_stored":2122149082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724660,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1764896,"kb_used_data":1657356,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91980256,"statfs":{"total":95995035648,"available":94187782144,"internally_reserved":0,"allocated":1697132544,"data_stored":1696358638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953593,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2928524,"kb_used_data":2802936,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":90816628,"statfs":{"total":95995035648,"available":92996227072,"internally_reserved":0,"allocated":2870206464,"data_stored":2869440656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149824,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2287400,"kb_used_data":2085456,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":91457752,"statfs":{"total":95995035648,"available":93652738048,"internally_reserved":0,"allocated":2135506944,"data_stored":2134731994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":206764488},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411459,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":576240,"kb_used_data":414552,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93168912,"statfs":{"total":95995035648,"available":95404965888,"internally_reserved":0,"allocated":424501248,"data_stored":423722746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574984,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1815456,"kb_used_data":1680144,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91929696,"statfs":{"total":95995035648,"available":94136008704,"internally_reserved":0,"allocated":1720467456,"data_stored":1719692506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738510,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2199512,"kb_used_data":2055876,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91545640,"statfs":{"total":95995035648,"available":93742735360,"internally_reserved":0,"allocated":2105217024,"data_stored":2104446670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2111635456,"data_stored":2111635456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1724907520,"data_stored":1724907520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":426442752,"data_stored":426442752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2150891520,"data_stored":2150891520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2884829184,"data_stored":2884829184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1709834240,"data_stored":1709834240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2127167488,"data_stored":2127167488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1678573568,"data_stored":1678573568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":821559296,"data_stored":821559296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1292828672,"data_stored":1292828672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3391619072,"data_stored":3391619072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:07:30.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 31 2024-09-02T10:07:30.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:30.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:30.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:30 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:07:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:30 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:07:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:30 smithi187 ceph-mon[20072]: pgmap v638: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 58 MiB/s wr, 924 op/s 2024-09-02T10:07:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:30 smithi187 ceph-mon[20072]: 3.b scrub ok 2024-09-02T10:07:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:30 smithi187 ceph-mon[20072]: 3.7 scrub starts 2024-09-02T10:07:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:30 smithi187 ceph-mon[20072]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:07:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2273323048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:30.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:30 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:07:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:30 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:07:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:30 smithi162 ceph-mon[20279]: pgmap v638: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.3 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 58 MiB/s wr, 924 op/s 2024-09-02T10:07:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:30 smithi162 ceph-mon[20279]: 3.b scrub ok 2024-09-02T10:07:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:30 smithi162 ceph-mon[20279]: 3.7 scrub starts 2024-09-02T10:07:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:30 smithi162 ceph-mon[20279]: from='client.25219 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:07:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2273323048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:30.852 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:30.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:30.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:30.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:30.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:30.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:31.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 88 16 107565 107549 76.3781 56.875 0.0312689 0.0130852 2024-09-02T10:07:31.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:31.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:31.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:31.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:31.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:31.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:31.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:31.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:31.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:31.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:31.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:31.501 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:31.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:31.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:31.634 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:31.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:31.653 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:31.682 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:31.685 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:31 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:07:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:31 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:07:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:31 smithi151 ceph-mon[16639]: 3.c scrub starts 2024-09-02T10:07:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:31 smithi151 ceph-mon[16639]: 3.5 scrub ok 2024-09-02T10:07:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:31 smithi151 ceph-mon[16639]: 3.2 scrub ok 2024-09-02T10:07:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:31 smithi151 ceph-mon[16639]: 3.a scrub ok 2024-09-02T10:07:31.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:31 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:07:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:31 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:07:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:31 smithi187 ceph-mon[20072]: 3.c scrub starts 2024-09-02T10:07:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:31 smithi187 ceph-mon[20072]: 3.5 scrub ok 2024-09-02T10:07:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:31 smithi187 ceph-mon[20072]: 3.2 scrub ok 2024-09-02T10:07:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:31 smithi187 ceph-mon[20072]: 3.a scrub ok 2024-09-02T10:07:31.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:31.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:31.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:31 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:07:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:31 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:07:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:31 smithi162 ceph-mon[20279]: 3.c scrub starts 2024-09-02T10:07:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:31 smithi162 ceph-mon[20279]: 3.5 scrub ok 2024-09-02T10:07:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:31 smithi162 ceph-mon[20279]: 3.2 scrub ok 2024-09-02T10:07:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:31 smithi162 ceph-mon[20279]: 3.a scrub ok 2024-09-02T10:07:31.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:31.898 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:31.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:32.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:32.030 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 89 16 108523 108507 76.1926 59.875 0.00919509 0.0131197 2024-09-02T10:07:32.031 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:32.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:32.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:32.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:32.169 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:32.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:32.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:32.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:32.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:32.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:32.428 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:32.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:32.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:32.554 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:32.555 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:32.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:32.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:32 smithi151 ceph-mon[16639]: 3.7 scrub ok 2024-09-02T10:07:32.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:32 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:07:32.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:32 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:07:32.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:32 smithi151 ceph-mon[16639]: pgmap v639: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 57 MiB/s wr, 912 op/s 2024-09-02T10:07:32.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:32 smithi151 ceph-mon[16639]: 3.6 scrub starts 2024-09-02T10:07:32.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:32.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:32.702 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:7841 2.2 e97/92 2:45ea9ad1:::benchmark_data_smithi187_45882_object7840:head v 97'525, mlcod=97'521)", 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:39.581770+0000", 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 353.11716005400001, 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21803672299999999, 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.581770+0000", 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.581770+0000", 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.581770+0000", 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6300000000000005e-07 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.581819+0000", 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8532999999999998e-05 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799209+0000", 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21738985399999999 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799213+0000", 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2830000000000003e-06 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799266+0000", 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2677999999999999e-05 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799283+0000", 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7192000000000001e-05 2024-09-02T10:07:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799767+0000", 2024-09-02T10:07:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000484537 2024-09-02T10:07:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:39.799806+0000", 2024-09-02T10:07:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8983000000000003e-05 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:11302 2.c e98/92 2:30c239d5:::benchmark_data_smithi187_45882_object11301:head v 98'706, mlcod=98'703)", 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:01:42.312851+0000", 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 350.38607867399998, 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21628996, 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.312851+0000", 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.312851+0000", 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.312852+0000", 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9400000000000004e-07 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.312909+0000", 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6777999999999997e-05 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.528456+0000", 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21554783999999999 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.528469+0000", 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2279999999999999e-05 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.528572+0000", 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010367899999999999 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.528587+0000", 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4841000000000001e-05 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.529109+0000", 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052206500000000005 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:01:42.529141+0000", 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1582999999999999e-05 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:54905 2.2 e113/92 2:470b2221:::benchmark_data_smithi187_45882_object54904:head v 113'3507, mlcod=113'3504)", 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.133347+0000", 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 315.56558247499999, 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26424163000000001, 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.133347+0000", 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.133347+0000", 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.133348+0000", 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5400000000000001e-07 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.133405+0000", 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7853000000000003e-05 2024-09-02T10:07:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.395793+0000", 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26238781799999999 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.395800+0000", 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2639999999999997e-06 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.395862+0000", 2024-09-02T10:07:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2212999999999995e-05 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.395880+0000", 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8236e-05 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.397552+0000", 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001671715 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.397589+0000", 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6977000000000002e-05 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:54924 2.6 2:6f9b4ae3:::benchmark_data_smithi187_45882_object54923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.150918+0000", 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 315.54801163799999, 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22757286900000001, 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 54924 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150918+0000", 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150918+0000", 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150920+0000", 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1160000000000002e-06 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150949+0000", 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9124000000000001e-05 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150964+0000", 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5111e-05 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.150971+0000", 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9349999999999997e-06 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.151021+0000", 2024-09-02T10:07:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0238999999999998e-05 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.151120+0000", 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8702999999999993e-05 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.151213+0000", 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.302e-05 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.152690+0000", 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001476761 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.153002+0000", 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031222999999999998 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.378379+0000", 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22537722900000001 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.378438+0000", 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9086999999999999e-05 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.378491+0000", 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2314000000000001e-05 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:55061 2.6 2:670fe1b2:::benchmark_data_smithi187_45882_object55060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.478779+0000", 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 315.22015105000003, 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33186097799999997, 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55061 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478779+0000", 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478779+0000", 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478780+0000", 2024-09-02T10:07:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7039999999999999e-06 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478820+0000", 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0166000000000001e-05 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478837+0000", 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.615e-05 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478843+0000", 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9519999999999997e-06 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.478902+0000", 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8517000000000002e-05 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.479044+0000", 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014225300000000001 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.479203+0000", 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015919400000000001 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.480839+0000", 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016358060000000001 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.482347+0000", 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015076200000000001 2024-09-02T10:07:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.810532+0000", 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32818559200000003 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.810591+0000", 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8059000000000001e-05 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.810639+0000", 2024-09-02T10:07:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8965000000000003e-05 2024-09-02T10:07:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:55087 2.9 2:9c3a08ac:::benchmark_data_smithi187_45882_object55086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:07:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:17.521103+0000", 2024-09-02T10:07:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 315.177826176, 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.239958381, 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55087 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521103+0000", 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521103+0000", 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521104+0000", 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6200000000000006e-07 2024-09-02T10:07:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521126+0000", 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1237e-05 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521137+0000", 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1243000000000001e-05 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521142+0000", 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.057e-06 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521179+0000", 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6770999999999997e-05 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521263+0000", 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4365999999999998e-05 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.521352+0000", 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9184000000000005e-05 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.522733+0000", 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013806129999999999 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.523181+0000", 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044799100000000001 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.760954+0000", 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23777305800000001 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.761014+0000", 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9953e-05 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:17.761062+0000", 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7945999999999998e-05 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:67911 2.4 2:20b40a62:::benchmark_data_smithi187_45882_object67910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:02:29.867114+0000", 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 302.83181508299998, 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.235349739, 2024-09-02T10:07:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 67911 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867114+0000", 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867114+0000", 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867116+0000", 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.491e-06 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867159+0000", 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2629999999999997e-05 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867171+0000", 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2342e-05 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867179+0000", 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7649999999999993e-06 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867242+0000", 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3178999999999999e-05 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867410+0000", 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000167823 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.867560+0000", 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015035499999999999 2024-09-02T10:07:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.870186+0000", 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002626354 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:29.870907+0000", 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000720679 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.102373+0000", 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.231466269 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.102417+0000", 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3509e-05 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:02:30.102464+0000", 2024-09-02T10:07:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7342999999999997e-05 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:152239 2.9 2:9e1d182d:::benchmark_data_smithi187_45882_object54488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:34.912114+0000", 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.786816055, 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.241466075, 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 152239 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912114+0000", 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912114+0000", 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912115+0000", 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5260000000000001e-06 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912120+0000", 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.045e-06 2024-09-02T10:07:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912133+0000", 2024-09-02T10:07:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.295e-05 2024-09-02T10:07:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912139+0000", 2024-09-02T10:07:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8769999999999998e-06 2024-09-02T10:07:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912187+0000", 2024-09-02T10:07:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7899000000000003e-05 2024-09-02T10:07:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912277+0000", 2024-09-02T10:07:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9938999999999994e-05 2024-09-02T10:07:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:07:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912348+0000", 2024-09-02T10:07:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1512000000000002e-05 2024-09-02T10:07:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.916871+0000", 2024-09-02T10:07:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045222650000000001 2024-09-02T10:07:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.917151+0000", 2024-09-02T10:07:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028088700000000001 2024-09-02T10:07:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153516+0000", 2024-09-02T10:07:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.236364713 2024-09-02T10:07:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153553+0000", 2024-09-02T10:07:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6801999999999998e-05 2024-09-02T10:07:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153580+0000", 2024-09-02T10:07:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6659999999999999e-05 2024-09-02T10:07:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:152239 2.9 e182/92 ondisk, result = 0)", 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:34.916225+0000", 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.78270422899999, 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.237360295, 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.916225+0000", 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.916225+0000", 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.916227+0000", 2024-09-02T10:07:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2699999999999999e-06 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.916232+0000", 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1479999999999997e-06 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153424+0000", 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23719195500000001 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153429+0000", 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2639999999999999e-06 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153506+0000", 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7212999999999994e-05 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153512+0000", 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2430000000000002e-06 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153586+0000", 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3201999999999995e-05 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(2.4 182/131 [PushReplyOp(2:27b97130:::benchmark_data_smithi187_45882_object78555:head)])", 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:35.629108+0000", 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.069821394, 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25949049200000002, 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.629108+0000", 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.629108+0000", 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.629109+0000", 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4600000000000003e-07 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.629113+0000", 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3440000000000003e-06 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.887962+0000", 2024-09-02T10:07:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25884861100000001 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.887969+0000", 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2559999999999996e-06 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.888057+0000", 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7391000000000004e-05 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.888599+0000", 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00054204399999999999 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:158092 2.c e191/92 2:360f665f:::benchmark_data_smithi187_45882_object60341:head v 191'9940, mlcod=191'9935)", 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:46.645549+0000", 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 166.05338100399999, 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.224523372, 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:46.645549+0000", 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:46.645549+0000", 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:46.645549+0000", 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7599999999999996e-07 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:46.645555+0000", 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3140000000000003e-06 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:46.856455+0000", 2024-09-02T10:07:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.210900373 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:46.856461+0000", 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6450000000000002e-06 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:46.856628+0000", 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016702000000000001 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:46.856673+0000", 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5701000000000002e-05 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:46.870027+0000", 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013353810000000001 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:46.870072+0000", 2024-09-02T10:07:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4733e-05 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:185641 2.6 2:67d73080:::benchmark_data_smithi187_45882_object87890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:28.359856+0000", 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 124.33907322, 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22303853600000001, 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 185641 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.359856+0000", 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.359856+0000", 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.359858+0000", 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3430000000000001e-06 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.359863+0000", 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.04e-06 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.359874+0000", 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1286e-05 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.359880+0000", 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2280000000000003e-06 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.359974+0000", 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4052e-05 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.360107+0000", 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013268000000000001 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.360217+0000", 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109885 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.365152+0000", 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049351550000000001 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.367916+0000", 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027642600000000002 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.582785+0000", 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21486914700000001 2024-09-02T10:07:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.582843+0000", 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8056000000000001e-05 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.582895+0000", 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1403999999999998e-05 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:185645 2.9 2:903990b2:::benchmark_data_smithi187_45882_object87894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:28.365374+0000", 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 124.333555825, 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23628532499999999, 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 185645 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.365374+0000", 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.365374+0000", 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.365375+0000", 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4929999999999999e-06 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.365382+0000", 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6059999999999999e-06 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.365397+0000", 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.499e-05 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.365404+0000", 2024-09-02T10:07:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3939999999999998e-06 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.365459+0000", 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4848000000000001e-05 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.365674+0000", 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021461000000000001 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.365821+0000", 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014757999999999999 2024-09-02T10:07:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.370472+0000", 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046502440000000004 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.370770+0000", 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029847800000000002 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.601590+0000", 2024-09-02T10:07:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23082018200000001 2024-09-02T10:07:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.781 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:32 smithi187 ceph-mon[20072]: 3.7 scrub ok 2024-09-02T10:07:32.782 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:32 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:07:32.782 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:32 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:07:32.782 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:32 smithi187 ceph-mon[20072]: pgmap v639: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 57 MiB/s wr, 912 op/s 2024-09-02T10:07:32.782 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:32 smithi187 ceph-mon[20072]: 3.6 scrub starts 2024-09-02T10:07:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.601627+0000", 2024-09-02T10:07:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6431999999999999e-05 2024-09-02T10:07:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.601659+0000", 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2468000000000002e-05 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:185649 2.9 2:91f1d57a:::benchmark_data_smithi187_45882_object87898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:28.374152+0000", 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 124.324777918, 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23692353099999999, 2024-09-02T10:07:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 185649 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374152+0000", 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374152+0000", 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374154+0000", 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.866e-06 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374160+0000", 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6159999999999998e-06 2024-09-02T10:07:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374172+0000", 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1936999999999999e-05 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374180+0000", 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8900000000000007e-06 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374235+0000", 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4558999999999997e-05 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374334+0000", 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9807000000000002e-05 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374439+0000", 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010433399999999999 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.382018+0000", 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075792450000000001 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.382505+0000", 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00048756100000000002 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.610992+0000", 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22848654099999999 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611029+0000", 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7431000000000001e-05 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611075+0000", 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5744000000000001e-05 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:185652 2.6 2:6e339a4a:::benchmark_data_smithi187_45882_object87901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-02T10:07:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:28.374195+0000", 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 124.324734303, 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.222138533, 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 185652 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374195+0000", 2024-09-02T10:07:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374195+0000", 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374196+0000", 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5199999999999998e-07 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374198+0000", 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9630000000000001e-06 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374202+0000", 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7879999999999997e-06 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374206+0000", 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.179e-06 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374270+0000", 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4672999999999996e-05 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374406+0000", 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013535700000000001 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:07:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374518+0000", 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011288 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.378431+0000", 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039123609999999996 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.379413+0000", 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00098254399999999991 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.596229+0000", 2024-09-02T10:07:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21681551900000001 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.596291+0000", 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1864000000000003e-05 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.596334+0000", 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3053000000000003e-05 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:185654 2.9 2:9a62ca65:::benchmark_data_smithi187_45882_object87903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:28.374759+0000", 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 124.32417039400001, 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23643328899999999, 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 185654 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374759+0000", 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374759+0000", 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374760+0000", 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6499999999999999e-07 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374763+0000", 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4599999999999999e-06 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374772+0000", 2024-09-02T10:07:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7199999999999995e-06 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374777+0000", 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5519999999999998e-06 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.378482+0000", 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037049370000000002 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.378627+0000", 2024-09-02T10:07:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145181 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.378744+0000", 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117475 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.382119+0000", 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033746779999999999 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.388940+0000", 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068215849999999998 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611131+0000", 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.222191004 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611165+0000", 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3868999999999997e-05 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611192+0000", 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7163000000000001e-05 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:185662 2.9 2:9f81ae8c:::benchmark_data_smithi187_45882_object87911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:28.383925+0000", 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 124.31500432199999, 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.227354481, 2024-09-02T10:07:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 185662 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.383925+0000", 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.383925+0000", 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.383926+0000", 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8999999999999996e-07 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.383929+0000", 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1820000000000002e-06 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.383939+0000", 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.995e-06 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.383947+0000", 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7090000000000002e-06 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.383995+0000", 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8607999999999999e-05 2024-09-02T10:07:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.384099+0000", 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104017 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.384208+0000", 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108253 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.388814+0000", 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046060060000000002 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.420133+0000", 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.031319193000000002 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611222+0000", 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19108897599999999 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611250+0000", 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7905000000000001e-05 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611280+0000", 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0046999999999999e-05 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:12888 3.4 3:25569122:::benchmark_data_smithi187_59989_object12887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:06:11.023080+0000", 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 81.675849850000006, 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24377531899999999, 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 12888 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023080+0000", 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023080+0000", 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023081+0000", 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1599999999999999e-06 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023124+0000", 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3371999999999997e-05 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023137+0000", 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2269999999999999e-05 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023142+0000", 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.48e-06 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023231+0000", 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9326000000000005e-05 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023350+0000", 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011908000000000001 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023487+0000", 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013629500000000001 2024-09-02T10:07:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024008+0000", 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052135500000000004 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024495+0000", 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000487109 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266794+0000", 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.242299019 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266826+0000", 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2268999999999999e-05 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266855+0000", 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8583999999999999e-05 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:12888 3.4 e233/226 ondisk, result = 0)", 2024-09-02T10:07:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:06:11.024330+0000", 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 81.674599086000001, 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.242529825, 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024330+0000", 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024330+0000", 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024331+0000", 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0299999999999999e-07 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024334+0000", 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1070000000000001e-06 2024-09-02T10:07:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266748+0000", 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24241418300000001 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266753+0000", 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7140000000000001e-06 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266786+0000", 2024-09-02T10:07:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3408000000000003e-05 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266791+0000", 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5879999999999999e-06 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266860+0000", 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9321999999999998e-05 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:92931 3.4 3:24dae35b:::benchmark_data_smithi187_59989_object92930:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:16.221469+0000", 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.477460937, 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16293174599999999, 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92931 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.221469+0000", 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.221469+0000", 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.221471+0000", 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8600000000000001e-06 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.221536+0000", 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4299000000000002e-05 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.221582+0000", 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6269e-05 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.221603+0000", 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1046000000000001e-05 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.221670+0000", 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6636000000000004e-05 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.221823+0000", 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015318800000000001 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.221969+0000", 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145842 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.224819+0000", 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028503529999999999 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.225371+0000", 2024-09-02T10:07:32.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00055151899999999999 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.384274+0000", 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.158902871 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.384344+0000", 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0307000000000003e-05 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:32.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:07:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:16.384400+0000", 2024-09-02T10:07:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6555999999999999e-05 2024-09-02T10:07:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:32.825 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:32.825 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:07:32.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:32.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:32 smithi162 ceph-mon[20279]: 3.7 scrub ok 2024-09-02T10:07:32.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:32 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:07:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:32 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:07:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:32 smithi162 ceph-mon[20279]: pgmap v639: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.4 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 57 MiB/s wr, 912 op/s 2024-09-02T10:07:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:32 smithi162 ceph-mon[20279]: 3.6 scrub starts 2024-09-02T10:07:32.848 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:32.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:32.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:32.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:32.979 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:32.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:32.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:33.031 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 90 8 109540 109532 76.0578 64.0625 0.00438754 0.0131426 2024-09-02T10:07:33.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:33.104 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:33.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:33.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:33.228 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:33.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:33.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:33.351 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:33.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:33.387 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:07:33.388 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 1 pgp_num to 31 2024-09-02T10:07:33.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:33.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:33.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:33.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:33.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:33.599 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:33.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:33.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:33.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:07:33.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:33 smithi151 ceph-mon[16639]: 3.c scrub ok 2024-09-02T10:07:33.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:33 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:07:33.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:33 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:07:33.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:33 smithi151 ceph-mon[16639]: 3.4 scrub starts 2024-09-02T10:07:33.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/141833636' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:33.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3856404785' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-02T10:07:33.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:33.722 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:33.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:33 smithi187 ceph-mon[20072]: 3.c scrub ok 2024-09-02T10:07:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:33 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:07:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:33 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:07:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:33 smithi187 ceph-mon[20072]: 3.4 scrub starts 2024-09-02T10:07:33.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/141833636' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:33.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3856404785' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-02T10:07:33.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:33 smithi162 ceph-mon[20279]: 3.c scrub ok 2024-09-02T10:07:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:33 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:07:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:33 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:07:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:33 smithi162 ceph-mon[20279]: 3.4 scrub starts 2024-09-02T10:07:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/141833636' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3856404785' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"} : dispatch 2024-09-02T10:07:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:33.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:33.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:33.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:33.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:33.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:34.031 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total time run: 90.6659 2024-09-02T10:07:34.031 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total writes made: 109540 2024-09-02T10:07:34.031 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Write size: 65536 2024-09-02T10:07:34.031 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object size: 65536 2024-09-02T10:07:34.031 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Bandwidth (MB/sec): 75.5108 2024-09-02T10:07:34.031 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Bandwidth: 16.4641 2024-09-02T10:07:34.031 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max bandwidth (MB/sec): 136.25 2024-09-02T10:07:34.032 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min bandwidth (MB/sec): 46.125 2024-09-02T10:07:34.032 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average IOPS: 1208 2024-09-02T10:07:34.032 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev IOPS: 263.425 2024-09-02T10:07:34.032 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max IOPS: 2180 2024-09-02T10:07:34.032 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min IOPS: 738 2024-09-02T10:07:34.032 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average Latency(s): 0.0131718 2024-09-02T10:07:34.032 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Latency(s): 0.0183556 2024-09-02T10:07:34.032 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max latency(s): 0.676625 2024-09-02T10:07:34.032 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min latency(s): 0.0010322 2024-09-02T10:07:34.036 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Cleaning up (deleting benchmark objects) 2024-09-02T10:07:34.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:34.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:34.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:34.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:34.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:07:34.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:34.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:34.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:34.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:34.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:34.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:34 smithi151 ceph-mon[16639]: pgmap v640: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-02T10:07:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:34 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:07:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:34 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:07:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/141833636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3856404785' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-02T10:07:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:34 smithi151 ceph-mon[16639]: osdmap e264: 12 total, 12 up, 11 in 2024-09-02T10:07:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:34 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:07:34.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:34 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:07:34.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:34 smithi151 ceph-mon[16639]: osdmap e265: 12 total, 12 up, 11 in 2024-09-02T10:07:34.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:34.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:34.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:34.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:34.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:34.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:34.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:34.740 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:34.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:34 smithi187 ceph-mon[20072]: pgmap v640: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-02T10:07:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:34 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:07:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:34 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:07:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/141833636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3856404785' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-02T10:07:34.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:34 smithi187 ceph-mon[20072]: osdmap e264: 12 total, 12 up, 11 in 2024-09-02T10:07:34.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:34 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:07:34.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:34 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:07:34.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:34 smithi187 ceph-mon[20072]: osdmap e265: 12 total, 12 up, 11 in 2024-09-02T10:07:34.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:34.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:34 smithi162 ceph-mon[20279]: pgmap v640: 47 pgs: 3 active+clean+scrubbing, 44 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-02T10:07:34.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:34 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:07:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:34 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:07:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/141833636' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3856404785' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "31"}]': finished 2024-09-02T10:07:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:34 smithi162 ceph-mon[20279]: osdmap e264: 12 total, 12 up, 11 in 2024-09-02T10:07:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:34 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:07:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:34 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:07:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:34 smithi162 ceph-mon[20279]: osdmap e265: 12 total, 12 up, 11 in 2024-09-02T10:07:34.863 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:34.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:34.965 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:34.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:34.990 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:35.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:35.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:35.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:35.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:35.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:35.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:35.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:35.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:35.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:35.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:35.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:35.470 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:35.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:35.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:35.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:35.617 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:35.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:35.618 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:35.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:35 smithi151 ceph-mon[16639]: 3.6 scrub ok 2024-09-02T10:07:35.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:35 smithi151 ceph-mon[16639]: 3.0 scrub starts 2024-09-02T10:07:35.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:35 smithi151 ceph-mon[16639]: 3.4 scrub ok 2024-09-02T10:07:35.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:35 smithi151 ceph-mon[16639]: 3.9 scrub starts 2024-09-02T10:07:35.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:35.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:35.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:35.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:35 smithi187 ceph-mon[20072]: 3.6 scrub ok 2024-09-02T10:07:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:35 smithi187 ceph-mon[20072]: 3.0 scrub starts 2024-09-02T10:07:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:35 smithi187 ceph-mon[20072]: 3.4 scrub ok 2024-09-02T10:07:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:35 smithi187 ceph-mon[20072]: 3.9 scrub starts 2024-09-02T10:07:35.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:35 smithi162 ceph-mon[20279]: 3.6 scrub ok 2024-09-02T10:07:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:35 smithi162 ceph-mon[20279]: 3.0 scrub starts 2024-09-02T10:07:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:35 smithi162 ceph-mon[20279]: 3.4 scrub ok 2024-09-02T10:07:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:35 smithi162 ceph-mon[20279]: 3.9 scrub starts 2024-09-02T10:07:35.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:35.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:35.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:35.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:35.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:35.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:35.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:35.997 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:36.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:36.120 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:36.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:36.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:36.249 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:36.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:36.304 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:36.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:36.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:36.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:36.461 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:36 smithi151 ceph-mon[16639]: 3.3 scrub starts 2024-09-02T10:07:36.461 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:36 smithi151 ceph-mon[16639]: pgmap v643: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 51 MiB/s wr, 821 op/s 2024-09-02T10:07:36.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:36.498 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:36.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:36.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:36.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:36.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:36.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:36.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:36.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:36 smithi187 ceph-mon[20072]: 3.3 scrub starts 2024-09-02T10:07:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:36 smithi187 ceph-mon[20072]: pgmap v643: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 51 MiB/s wr, 821 op/s 2024-09-02T10:07:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:36 smithi162 ceph-mon[20279]: 3.3 scrub starts 2024-09-02T10:07:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:36 smithi162 ceph-mon[20279]: pgmap v643: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 51 MiB/s wr, 821 op/s 2024-09-02T10:07:36.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:36.871 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:36.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:36.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:36.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:36.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:37.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:37.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:37.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:37.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:37.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:37.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:37.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:37.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:37.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:37.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:37.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:37.480 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:37.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:37.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:37.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:37.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:37.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:37.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:37 smithi151 ceph-mon[16639]: 3.0 scrub ok 2024-09-02T10:07:37.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:37 smithi151 ceph-mon[16639]: 3.3 scrub ok 2024-09-02T10:07:37.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:37 smithi151 ceph-mon[16639]: 3.9 scrub ok 2024-09-02T10:07:37.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:37.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:37 smithi187 ceph-mon[20072]: 3.0 scrub ok 2024-09-02T10:07:37.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:37 smithi187 ceph-mon[20072]: 3.3 scrub ok 2024-09-02T10:07:37.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:37 smithi187 ceph-mon[20072]: 3.9 scrub ok 2024-09-02T10:07:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:37.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:37.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:37.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:37 smithi162 ceph-mon[20279]: 3.0 scrub ok 2024-09-02T10:07:37.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:37 smithi162 ceph-mon[20279]: 3.3 scrub ok 2024-09-02T10:07:37.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:37 smithi162 ceph-mon[20279]: 3.9 scrub ok 2024-09-02T10:07:37.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:37.892 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:37.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:37.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:38.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:38.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:38.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:38.134 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:07:38.134 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:07:38.134 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:07:38.135 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:07:38.151 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:38.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:38.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:07:38.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:38.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:38.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:38.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:38.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:38.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:38.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:38.485 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:38.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:38.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:38.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:38.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:38.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:38.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:38.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:38.826 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:38.827 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:38.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:38 smithi151 ceph-mon[16639]: pgmap v644: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 127 B/s rd, 47 MiB/s wr, 924 op/s 2024-09-02T10:07:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:38 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:07:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:38 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:07:38.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:38.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:38 smithi187 ceph-mon[20072]: pgmap v644: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 127 B/s rd, 47 MiB/s wr, 924 op/s 2024-09-02T10:07:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:38 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:07:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:38 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:07:39.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:39.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:07:39.080 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:39.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:39.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:38 smithi162 ceph-mon[20279]: pgmap v644: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 127 B/s rd, 47 MiB/s wr, 924 op/s 2024-09-02T10:07:39.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:38 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:07:39.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:38 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:07:39.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:39.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:39.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:39.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:39.311 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:39.334 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:39.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:39.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:39.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:39.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:39.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:39.586 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:39.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:39.590 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:39.601 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:07:39.601 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:07:39.637 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:39.639 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:07:39.639 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:07:39.667 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:07:39.668 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:07:39.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:39.696 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:07:39.696 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:07:39.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:39.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:39.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:39.726 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:07:39.726 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:07:39.752 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:07:39.752 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:07:39.777 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:07:39.777 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:07:39.805 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:07:39.805 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:07:39.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:39.830 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:07:39.830 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:07:39.830 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:07:39.831 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:07:39.831 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:39 smithi151 ceph-mon[16639]: osdmap e266: 12 total, 12 up, 11 in 2024-09-02T10:07:39.831 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:39 smithi151 ceph-mon[16639]: osdmap e267: 12 total, 12 up, 11 in 2024-09-02T10:07:39.831 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3530984533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:39.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:39.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:39.855 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:07:39.855 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:07:39.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:07:39.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:07:39.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:07:39.913 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:07:39.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:39.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:39.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:07:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:07:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:39 smithi187 ceph-mon[20072]: osdmap e266: 12 total, 12 up, 11 in 2024-09-02T10:07:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:39 smithi187 ceph-mon[20072]: osdmap e267: 12 total, 12 up, 11 in 2024-09-02T10:07:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3530984533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:40.068 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:40.090 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:40.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:40.091 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:40.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:07:40.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:07:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:39 smithi162 ceph-mon[20279]: osdmap e266: 12 total, 12 up, 11 in 2024-09-02T10:07:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:39 smithi162 ceph-mon[20279]: osdmap e267: 12 total, 12 up, 11 in 2024-09-02T10:07:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3530984533' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:40.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:40.214 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:40.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:40.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:40.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:40.340 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:40.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:40.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:40.466 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:40.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:40.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:40.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:40.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:40.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:40.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:40.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:40.721 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:40.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:40.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:40.845 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:40.889 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:40 smithi151 ceph-mon[16639]: pgmap v647: 47 pgs: 47 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 177 B/s rd, 14 MiB/s wr, 611 op/s 2024-09-02T10:07:40.889 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:40 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:07:40.889 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:40 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:07:40.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:40.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:40.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:40 smithi187 ceph-mon[20072]: pgmap v647: 47 pgs: 47 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 177 B/s rd, 14 MiB/s wr, 611 op/s 2024-09-02T10:07:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:40 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:07:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:40 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:07:41.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:41.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:40 smithi162 ceph-mon[20279]: pgmap v647: 47 pgs: 47 active+clean; 6.6 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 177 B/s rd, 14 MiB/s wr, 611 op/s 2024-09-02T10:07:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:40 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:07:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:40 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:07:41.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:41.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:41.096 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:41.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:41.221 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:41.223 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:41.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:41.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:41.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:41.455 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:41.479 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:41.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:41.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:41.608 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:41.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:41.637 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:07:41.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:41.737 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:41.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:41.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:41.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:41.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:41.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2062774203' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:41.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:41 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:41.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:41.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:41.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:41.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2062774203' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:41 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:42.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2062774203' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:42.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:41 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:07:42.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:42.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:42.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:42.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:42.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:42.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:42.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:42.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:42.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:42.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:42.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:42.510 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:42.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:42.582 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:07:42.583 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:07:42.583 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-02T10:07:42.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:07:42.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:42.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:42.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:42.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:42.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:42.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:42.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:42.908 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:42.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:42.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:42 smithi151 ceph-mon[16639]: pgmap v648: 47 pgs: 47 active+clean; 6.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 140 B/s rd, 11 MiB/s wr, 639 op/s 2024-09-02T10:07:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:42 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:42 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:07:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:42 smithi151 ceph-mon[16639]: osdmap e268: 12 total, 12 up, 11 in 2024-09-02T10:07:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:42 smithi187 ceph-mon[20072]: pgmap v648: 47 pgs: 47 active+clean; 6.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 140 B/s rd, 11 MiB/s wr, 639 op/s 2024-09-02T10:07:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:42 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:42 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:07:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:42 smithi187 ceph-mon[20072]: osdmap e268: 12 total, 12 up, 11 in 2024-09-02T10:07:43.010 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:43.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:43.032 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:43.057 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:07:43.057 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:07:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:42 smithi162 ceph-mon[20279]: pgmap v648: 47 pgs: 47 active+clean; 6.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 140 B/s rd, 11 MiB/s wr, 639 op/s 2024-09-02T10:07:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:42 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:42 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:07:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:42 smithi162 ceph-mon[20279]: osdmap e268: 12 total, 12 up, 11 in 2024-09-02T10:07:43.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:43.160 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:43.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:43.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:43.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:43.291 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:43.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:43.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:43.417 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:43.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:43.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:43.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:43.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:43.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:43.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:43.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:43.670 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:43.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:07:43.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:43.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:43.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:43.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:43.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:07:43.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:43.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:44.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:44.051 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:44.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:44.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:44.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:44.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:44.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:44.312 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:44.314 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:44.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:44.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:44.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:44.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:44.562 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:44.570 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:44.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:44.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:44.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:44.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:44.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:44.823 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:44.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:44.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:44 smithi151 ceph-mon[16639]: pgmap v650: 47 pgs: 47 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 11 KiB/s wr, 784 op/s 2024-09-02T10:07:44.861 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:44 smithi151 ceph-mon[16639]: osdmap e269: 12 total, 12 up, 11 in 2024-09-02T10:07:44.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:44.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:44.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:44 smithi187 ceph-mon[20072]: pgmap v650: 47 pgs: 47 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 11 KiB/s wr, 784 op/s 2024-09-02T10:07:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:44 smithi187 ceph-mon[20072]: osdmap e269: 12 total, 12 up, 11 in 2024-09-02T10:07:45.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:45.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:45.074 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:45.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:44 smithi162 ceph-mon[20279]: pgmap v650: 47 pgs: 47 active+clean; 6.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 11 KiB/s wr, 784 op/s 2024-09-02T10:07:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:44 smithi162 ceph-mon[20279]: osdmap e269: 12 total, 12 up, 11 in 2024-09-02T10:07:45.167 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:45.174 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:45.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:45.195 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:45.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:45.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:45.320 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:45.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:45.347 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:07:45.348 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:07:45.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:45.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:45.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:45.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:45.584 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:45.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:45.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:45.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:45.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:45.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:45.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:45.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:45.844 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:45.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:45.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:45.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:45.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:46.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:46.098 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:46.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:46.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:46.109 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:07:46.110 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:07:46.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:46.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:46.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:46.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:46.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:46.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:46.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:46.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:46.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:46.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:07:46.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:46.619 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:46.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:46.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:46.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:46.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:46.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:46.846 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:46.852 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:46 smithi151 ceph-mon[16639]: pgmap v652: 47 pgs: 47 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 837 op/s 2024-09-02T10:07:46.852 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:46 smithi151 ceph-mon[16639]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:07:46.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:07:46.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:46.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:46.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:46 smithi187 ceph-mon[20072]: pgmap v652: 47 pgs: 47 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 837 op/s 2024-09-02T10:07:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:46 smithi187 ceph-mon[20072]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:07:47.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:47.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:46 smithi162 ceph-mon[20279]: pgmap v652: 47 pgs: 47 active+clean; 6.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 837 op/s 2024-09-02T10:07:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:46 smithi162 ceph-mon[20279]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:07:47.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:47.132 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:47.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:47.137 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:07:47.137 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:07:47.161 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:07:47.161 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:07:47.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:07:47.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:07:47.211 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:07:47.211 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:07:47.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:47.236 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:07:47.237 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:07:47.262 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:47.263 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:07:47.263 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:07:47.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:47.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:47.292 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:07:47.292 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:07:47.317 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:07:47.318 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:07:47.348 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:07:47.348 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:07:47.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:47.372 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:07:47.372 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:07:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:47.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:47.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:47.402 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:07:47.403 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:07:47.428 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:07:47.428 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:07:47.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:47.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:47.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:47.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:47.650 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:47.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:47.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1466758554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:47.670 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":652,"stamp":"2024-09-02T10:07:45.156219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6688735800,"num_objects":102056,"num_object_clones":0,"num_object_copies":306168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102056,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117028,"num_write_kb":7010561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119880,"ondisk_log_size":119880,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":21315156,"kb_used_data":19671300,"kb_used_omap":17,"kb_used_meta":1643630,"kb_avail":1009881516,"statfs":{"total":1055945392128,"available":1034118672384,"internally_reserved":0,"allocated":20143411200,"data_stored":20134629498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1683078011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-345571328,"num_objects":-5273,"num_object_clones":0,"num_object_copies":-15819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5273,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.298372"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":69,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.636381+0000","last_change":"2024-09-02T10:06:35.411836+0000","last_active":"2024-09-02T10:07:41.636381+0000","last_peered":"2024-09-02T10:07:41.636381+0000","last_clean":"2024-09-02T10:07:41.636381+0000","last_became_active":"2024-09-02T10:06:34.853129+0000","last_became_peered":"2024-09-02T10:06:34.853129+0000","last_unstale":"2024-09-02T10:07:41.636381+0000","last_undegraded":"2024-09-02T10:07:41.636381+0000","last_fullsized":"2024-09-02T10:07:41.636381+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.411763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.411763+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:53.117589+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":71,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.641996+0000","last_change":"2024-09-02T10:06:36.539700+0000","last_active":"2024-09-02T10:07:41.641996+0000","last_peered":"2024-09-02T10:07:41.641996+0000","last_clean":"2024-09-02T10:07:41.641996+0000","last_became_active":"2024-09-02T10:06:34.860248+0000","last_became_peered":"2024-09-02T10:06:34.860248+0000","last_unstale":"2024-09-02T10:07:41.641996+0000","last_undegraded":"2024-09-02T10:07:41.641996+0000","last_fullsized":"2024-09-02T10:07:41.641996+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:36.539650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:36.539650+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:49.640292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":71,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-02T10:07:43.860129+0000","last_change":"2024-09-02T10:06:37.123017+0000","last_active":"2024-09-02T10:07:43.860129+0000","last_peered":"2024-09-02T10:07:43.860129+0000","last_clean":"2024-09-02T10:07:43.860129+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:07:43.860129+0000","last_undegraded":"2024-09-02T10:07:43.860129+0000","last_fullsized":"2024-09-02T10:07:43.860129+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:37.122948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:37.122948+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":71,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-02T10:07:43.860057+0000","last_change":"2024-09-02T10:06:36.130155+0000","last_active":"2024-09-02T10:07:43.860057+0000","last_peered":"2024-09-02T10:07:43.860057+0000","last_clean":"2024-09-02T10:07:43.860057+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:07:43.860057+0000","last_undegraded":"2024-09-02T10:07:43.860057+0000","last_fullsized":"2024-09-02T10:07:43.860057+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:36.130088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:36.130088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:59.742254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":73,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-02T10:07:43.866128+0000","last_change":"2024-09-02T10:06:35.749426+0000","last_active":"2024-09-02T10:07:43.866128+0000","last_peered":"2024-09-02T10:07:43.866128+0000","last_clean":"2024-09-02T10:07:43.866128+0000","last_became_active":"2024-09-02T10:06:34.852952+0000","last_became_peered":"2024-09-02T10:06:34.852952+0000","last_unstale":"2024-09-02T10:07:43.866128+0000","last_undegraded":"2024-09-02T10:07:43.866128+0000","last_fullsized":"2024-09-02T10:07:43.866128+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.749356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.749356+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:51.822172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":75,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:39.188463+0000","last_change":"2024-09-02T10:07:39.188463+0000","last_active":"2024-09-02T10:07:39.188463+0000","last_peered":"2024-09-02T10:07:39.188463+0000","last_clean":"2024-09-02T10:07:39.188463+0000","last_became_active":"2024-09-02T10:06:34.851298+0000","last_became_peered":"2024-09-02T10:06:34.851298+0000","last_unstale":"2024-09-02T10:07:39.188463+0000","last_undegraded":"2024-09-02T10:07:39.188463+0000","last_fullsized":"2024-09-02T10:07:39.188463+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:39.188419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:39.188419+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:48.116976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":428,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.636062+0000","last_change":"2024-09-02T10:06:35.086723+0000","last_active":"2024-09-02T10:07:41.636062+0000","last_peered":"2024-09-02T10:07:41.636062+0000","last_clean":"2024-09-02T10:07:41.636062+0000","last_became_active":"2024-09-02T10:06:34.859627+0000","last_became_peered":"2024-09-02T10:06:34.859627+0000","last_unstale":"2024-09-02T10:07:41.636062+0000","last_undegraded":"2024-09-02T10:07:41.636062+0000","last_fullsized":"2024-09-02T10:07:41.636062+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.086640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.086640+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:00.340025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":71,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-02T10:07:43.860167+0000","last_change":"2024-09-02T10:06:35.154338+0000","last_active":"2024-09-02T10:07:43.860167+0000","last_peered":"2024-09-02T10:07:43.860167+0000","last_clean":"2024-09-02T10:07:43.860167+0000","last_became_active":"2024-09-02T10:06:34.849680+0000","last_became_peered":"2024-09-02T10:06:34.849680+0000","last_unstale":"2024-09-02T10:07:43.860167+0000","last_undegraded":"2024-09-02T10:07:43.860167+0000","last_fullsized":"2024-09-02T10:07:43.860167+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.154262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.154262+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:48.631282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":69,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.643199+0000","last_change":"2024-09-02T10:06:35.567150+0000","last_active":"2024-09-02T10:07:41.643199+0000","last_peered":"2024-09-02T10:07:41.643199+0000","last_clean":"2024-09-02T10:07:41.643199+0000","last_became_active":"2024-09-02T10:06:34.850895+0000","last_became_peered":"2024-09-02T10:06:34.850895+0000","last_unstale":"2024-09-02T10:07:41.643199+0000","last_undegraded":"2024-09-02T10:07:41.643199+0000","last_fullsized":"2024-09-02T10:07:41.643199+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.567106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.567106+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:01.241139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":67,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:38.860359+0000","last_change":"2024-09-02T10:06:34.985266+0000","last_active":"2024-09-02T10:07:38.860359+0000","last_peered":"2024-09-02T10:07:38.860359+0000","last_clean":"2024-09-02T10:07:38.860359+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:07:38.860359+0000","last_undegraded":"2024-09-02T10:07:38.860359+0000","last_fullsized":"2024-09-02T10:07:38.860359+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.985198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.985198+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:50.914676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":324,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.640794+0000","last_change":"2024-09-02T10:07:28.918922+0000","last_active":"2024-09-02T10:07:41.640794+0000","last_peered":"2024-09-02T10:07:41.640794+0000","last_clean":"2024-09-02T10:07:41.640794+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:07:41.640794+0000","last_undegraded":"2024-09-02T10:07:41.640794+0000","last_fullsized":"2024-09-02T10:07:41.640794+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:28.918851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:28.918851+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:40.479112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":402,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.641962+0000","last_change":"2024-09-02T10:06:34.583354+0000","last_active":"2024-09-02T10:07:41.641962+0000","last_peered":"2024-09-02T10:07:41.641962+0000","last_clean":"2024-09-02T10:07:41.641962+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:07:41.641962+0000","last_undegraded":"2024-09-02T10:07:41.641962+0000","last_fullsized":"2024-09-02T10:07:41.641962+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.583305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.583305+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:49.795240+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":346,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.636062+0000","last_change":"2024-09-02T10:07:31.059528+0000","last_active":"2024-09-02T10:07:41.636062+0000","last_peered":"2024-09-02T10:07:41.636062+0000","last_clean":"2024-09-02T10:07:41.636062+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:07:41.636062+0000","last_undegraded":"2024-09-02T10:07:41.636062+0000","last_fullsized":"2024-09-02T10:07:41.636062+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:31.059464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:31.059464+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:39.131834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":324,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:38.867201+0000","last_change":"2024-09-02T10:06:41.293858+0000","last_active":"2024-09-02T10:07:38.867201+0000","last_peered":"2024-09-02T10:07:38.867201+0000","last_clean":"2024-09-02T10:07:38.867201+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:07:38.867201+0000","last_undegraded":"2024-09-02T10:07:38.867201+0000","last_fullsized":"2024-09-02T10:07:38.867201+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:41.293787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:41.293787+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:57.536691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":339,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.636065+0000","last_change":"2024-09-02T10:07:32.127860+0000","last_active":"2024-09-02T10:07:41.636065+0000","last_peered":"2024-09-02T10:07:41.636065+0000","last_clean":"2024-09-02T10:07:41.636065+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:07:41.636065+0000","last_undegraded":"2024-09-02T10:07:41.636065+0000","last_fullsized":"2024-09-02T10:07:41.636065+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:32.127803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:32.127803+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:48.053291+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"268'7303","reported_seq":7471,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:42.056065+0000","last_change":"2024-09-02T10:07:30.761771+0000","last_active":"2024-09-02T10:07:42.056065+0000","last_peered":"2024-09-02T10:07:42.056065+0000","last_clean":"2024-09-02T10:07:42.056065+0000","last_became_active":"2024-09-02T10:05:56.583663+0000","last_became_peered":"2024-09-02T10:05:56.583663+0000","last_unstale":"2024-09-02T10:07:42.056065+0000","last_undegraded":"2024-09-02T10:07:42.056065+0000","last_fullsized":"2024-09-02T10:07:42.056065+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6702","last_scrub_stamp":"2024-09-02T10:07:30.761550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.761550+0000","objects_scrubbed":6632,"log_size":7303,"log_dups_size":0,"ondisk_log_size":7303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:51.824511+0000","scrub_duration":1999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418185216,"num_objects":6381,"num_object_clones":0,"num_object_copies":19143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7303,"num_write_kb":437888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":417,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.636192+0000","last_change":"2024-09-02T10:06:33.072793+0000","last_active":"2024-09-02T10:07:41.636192+0000","last_peered":"2024-09-02T10:07:41.636192+0000","last_clean":"2024-09-02T10:07:41.636192+0000","last_became_active":"2024-09-02T10:06:32.873327+0000","last_became_peered":"2024-09-02T10:06:32.873327+0000","last_unstale":"2024-09-02T10:07:41.636192+0000","last_undegraded":"2024-09-02T10:07:41.636192+0000","last_fullsized":"2024-09-02T10:07:41.636192+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.072734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.072734+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:57.640290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"268'7326","reported_seq":7493,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:43.356591+0000","last_change":"2024-09-02T10:07:29.219779+0000","last_active":"2024-09-02T10:07:43.356591+0000","last_peered":"2024-09-02T10:07:43.356591+0000","last_clean":"2024-09-02T10:07:43.356591+0000","last_became_active":"2024-09-02T10:05:56.583278+0000","last_became_peered":"2024-09-02T10:05:56.583278+0000","last_unstale":"2024-09-02T10:07:43.356591+0000","last_undegraded":"2024-09-02T10:07:43.356591+0000","last_fullsized":"2024-09-02T10:07:43.356591+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6573","last_scrub_stamp":"2024-09-02T10:07:29.219665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:29.219665+0000","objects_scrubbed":6515,"log_size":7326,"log_dups_size":0,"ondisk_log_size":7326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:37.367524+0000","scrub_duration":1777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412221440,"num_objects":6290,"num_object_clones":0,"num_object_copies":18870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7326,"num_write_kb":435712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":539,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:38.861333+0000","last_change":"2024-09-02T10:06:32.998782+0000","last_active":"2024-09-02T10:07:38.861333+0000","last_peered":"2024-09-02T10:07:38.861333+0000","last_clean":"2024-09-02T10:07:38.861333+0000","last_became_active":"2024-09-02T10:06:32.871027+0000","last_became_peered":"2024-09-02T10:06:32.871027+0000","last_unstale":"2024-09-02T10:07:38.861333+0000","last_undegraded":"2024-09-02T10:07:38.861333+0000","last_fullsized":"2024-09-02T10:07:38.861333+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:32.998705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:32.998705+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:01.309334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"268'7232","reported_seq":7387,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.950516+0000","last_change":"2024-09-02T10:07:07.774469+0000","last_active":"2024-09-02T10:07:41.950516+0000","last_peered":"2024-09-02T10:07:41.950516+0000","last_clean":"2024-09-02T10:07:41.950516+0000","last_became_active":"2024-09-02T10:05:56.583249+0000","last_became_peered":"2024-09-02T10:05:56.583249+0000","last_unstale":"2024-09-02T10:07:41.950516+0000","last_undegraded":"2024-09-02T10:07:41.950516+0000","last_fullsized":"2024-09-02T10:07:41.950516+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5253","last_scrub_stamp":"2024-09-02T10:07:07.774339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:07.774339+0000","objects_scrubbed":5197,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:26.717286+0000","scrub_duration":1300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419692544,"num_objects":6404,"num_object_clones":0,"num_object_copies":19212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7232,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":175,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.639382+0000","last_change":"2024-09-02T10:06:33.026327+0000","last_active":"2024-09-02T10:07:41.639382+0000","last_peered":"2024-09-02T10:07:41.639382+0000","last_clean":"2024-09-02T10:07:41.639382+0000","last_became_active":"2024-09-02T10:06:32.870387+0000","last_became_peered":"2024-09-02T10:06:32.870387+0000","last_unstale":"2024-09-02T10:07:41.639382+0000","last_undegraded":"2024-09-02T10:07:41.639382+0000","last_fullsized":"2024-09-02T10:07:41.639382+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.026282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.026282+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:01.622377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"269'7430","reported_seq":7604,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-02T10:07:44.242959+0000","last_change":"2024-09-02T10:07:36.281686+0000","last_active":"2024-09-02T10:07:44.242959+0000","last_peered":"2024-09-02T10:07:44.242959+0000","last_clean":"2024-09-02T10:07:44.242959+0000","last_became_active":"2024-09-02T10:05:56.584313+0000","last_became_peered":"2024-09-02T10:05:56.584313+0000","last_unstale":"2024-09-02T10:07:44.242959+0000","last_undegraded":"2024-09-02T10:07:44.242959+0000","last_fullsized":"2024-09-02T10:07:44.242959+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6942","last_scrub_stamp":"2024-09-02T10:07:36.281628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:36.281628+0000","objects_scrubbed":6782,"log_size":7430,"log_dups_size":0,"ondisk_log_size":7430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:02.324822+0000","scrub_duration":1988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409731072,"num_objects":6252,"num_object_clones":0,"num_object_copies":18756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7430,"num_write_kb":437824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":330,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.636151+0000","last_change":"2024-09-02T10:06:33.317499+0000","last_active":"2024-09-02T10:07:41.636151+0000","last_peered":"2024-09-02T10:07:41.636151+0000","last_clean":"2024-09-02T10:07:41.636151+0000","last_became_active":"2024-09-02T10:06:32.874387+0000","last_became_peered":"2024-09-02T10:06:32.874387+0000","last_unstale":"2024-09-02T10:07:41.636151+0000","last_undegraded":"2024-09-02T10:07:41.636151+0000","last_fullsized":"2024-09-02T10:07:41.636151+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.317426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.317426+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:02.728906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"267'7262","reported_seq":7430,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.518191+0000","last_change":"2024-09-02T10:07:34.096922+0000","last_active":"2024-09-02T10:07:41.518191+0000","last_peered":"2024-09-02T10:07:41.518191+0000","last_clean":"2024-09-02T10:07:41.518191+0000","last_became_active":"2024-09-02T10:05:56.578973+0000","last_became_peered":"2024-09-02T10:05:56.578973+0000","last_unstale":"2024-09-02T10:07:41.518191+0000","last_undegraded":"2024-09-02T10:07:41.518191+0000","last_fullsized":"2024-09-02T10:07:41.518191+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6853","last_scrub_stamp":"2024-09-02T10:07:34.096729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:34.096729+0000","objects_scrubbed":6787,"log_size":7262,"log_dups_size":0,"ondisk_log_size":7262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:48.699432+0000","scrub_duration":2896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421789696,"num_objects":6436,"num_object_clones":0,"num_object_copies":19308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7262,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":480,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.636316+0000","last_change":"2024-09-02T10:07:07.536258+0000","last_active":"2024-09-02T10:07:41.636316+0000","last_peered":"2024-09-02T10:07:41.636316+0000","last_clean":"2024-09-02T10:07:41.636316+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:07:41.636316+0000","last_undegraded":"2024-09-02T10:07:41.636316+0000","last_fullsized":"2024-09-02T10:07:41.636316+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:07.536194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:07.536194+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:27.817134+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"268'7311","reported_seq":7479,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:43.361019+0000","last_change":"2024-09-02T10:07:30.887849+0000","last_active":"2024-09-02T10:07:43.361019+0000","last_peered":"2024-09-02T10:07:43.361019+0000","last_clean":"2024-09-02T10:07:43.361019+0000","last_became_active":"2024-09-02T10:05:56.584888+0000","last_became_peered":"2024-09-02T10:05:56.584888+0000","last_unstale":"2024-09-02T10:07:43.361019+0000","last_undegraded":"2024-09-02T10:07:43.361019+0000","last_fullsized":"2024-09-02T10:07:43.361019+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6680","last_scrub_stamp":"2024-09-02T10:07:30.887770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.887770+0000","objects_scrubbed":6647,"log_size":7311,"log_dups_size":0,"ondisk_log_size":7311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:34.316233+0000","scrub_duration":1437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412024832,"num_objects":6287,"num_object_clones":0,"num_object_copies":18861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7311,"num_write_kb":435136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":324,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.643444+0000","last_change":"2024-09-02T10:06:32.983641+0000","last_active":"2024-09-02T10:07:41.643444+0000","last_peered":"2024-09-02T10:07:41.643444+0000","last_clean":"2024-09-02T10:07:41.643444+0000","last_became_active":"2024-09-02T10:06:32.870858+0000","last_became_peered":"2024-09-02T10:06:32.870858+0000","last_unstale":"2024-09-02T10:07:41.643444+0000","last_undegraded":"2024-09-02T10:07:41.643444+0000","last_fullsized":"2024-09-02T10:07:41.643444+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:32.983547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:32.983547+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:00.900960+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"268'7553","reported_seq":7727,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:42.868680+0000","last_change":"2024-09-02T10:07:34.254248+0000","last_active":"2024-09-02T10:07:42.868680+0000","last_peered":"2024-09-02T10:07:42.868680+0000","last_clean":"2024-09-02T10:07:42.868680+0000","last_became_active":"2024-09-02T10:05:56.580910+0000","last_became_peered":"2024-09-02T10:05:56.580910+0000","last_unstale":"2024-09-02T10:07:42.868680+0000","last_undegraded":"2024-09-02T10:07:42.868680+0000","last_fullsized":"2024-09-02T10:07:42.868680+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'7048","last_scrub_stamp":"2024-09-02T10:07:34.254184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:34.254184+0000","objects_scrubbed":7013,"log_size":7553,"log_dups_size":0,"ondisk_log_size":7553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:54.308941+0000","scrub_duration":1842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426049536,"num_objects":6501,"num_object_clones":0,"num_object_copies":19503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7553,"num_write_kb":449728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":175,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.637468+0000","last_change":"2024-09-02T10:06:34.057029+0000","last_active":"2024-09-02T10:07:41.637468+0000","last_peered":"2024-09-02T10:07:41.637468+0000","last_clean":"2024-09-02T10:07:41.637468+0000","last_became_active":"2024-09-02T10:06:32.871735+0000","last_became_peered":"2024-09-02T10:06:32.871735+0000","last_unstale":"2024-09-02T10:07:41.637468+0000","last_undegraded":"2024-09-02T10:07:41.637468+0000","last_fullsized":"2024-09-02T10:07:41.637468+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.056985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.056985+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:03.996035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"268'7237","reported_seq":7406,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.860086+0000","last_change":"2024-09-02T10:07:30.154080+0000","last_active":"2024-09-02T10:07:41.860086+0000","last_peered":"2024-09-02T10:07:41.860086+0000","last_clean":"2024-09-02T10:07:41.860086+0000","last_became_active":"2024-09-02T10:05:56.690229+0000","last_became_peered":"2024-09-02T10:05:56.690229+0000","last_unstale":"2024-09-02T10:07:41.860086+0000","last_undegraded":"2024-09-02T10:07:41.860086+0000","last_fullsized":"2024-09-02T10:07:41.860086+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6652","last_scrub_stamp":"2024-09-02T10:07:30.153975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.153975+0000","objects_scrubbed":6605,"log_size":7237,"log_dups_size":0,"ondisk_log_size":7237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:59.384309+0000","scrub_duration":2047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418447360,"num_objects":6385,"num_object_clones":0,"num_object_copies":19155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7237,"num_write_kb":435904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":447,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.633385+0000","last_change":"2024-09-02T10:06:33.444679+0000","last_active":"2024-09-02T10:07:41.633385+0000","last_peered":"2024-09-02T10:07:41.633385+0000","last_clean":"2024-09-02T10:07:41.633385+0000","last_became_active":"2024-09-02T10:06:32.874081+0000","last_became_peered":"2024-09-02T10:06:32.874081+0000","last_unstale":"2024-09-02T10:07:41.633385+0000","last_undegraded":"2024-09-02T10:07:41.633385+0000","last_fullsized":"2024-09-02T10:07:41.633385+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.444635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.444635+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:45.927168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"269'7536","reported_seq":7710,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-02T10:07:44.210382+0000","last_change":"2024-09-02T10:07:35.873554+0000","last_active":"2024-09-02T10:07:44.210382+0000","last_peered":"2024-09-02T10:07:44.210382+0000","last_clean":"2024-09-02T10:07:44.210382+0000","last_became_active":"2024-09-02T10:05:56.581969+0000","last_became_peered":"2024-09-02T10:05:56.581969+0000","last_unstale":"2024-09-02T10:07:44.210382+0000","last_undegraded":"2024-09-02T10:07:44.210382+0000","last_fullsized":"2024-09-02T10:07:44.210382+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'7026","last_scrub_stamp":"2024-09-02T10:07:35.873463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:35.873463+0000","objects_scrubbed":6888,"log_size":7536,"log_dups_size":0,"ondisk_log_size":7536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:03.009740+0000","scrub_duration":1412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415105024,"num_objects":6334,"num_object_clones":0,"num_object_copies":19002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7536,"num_write_kb":443840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":169,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.637263+0000","last_change":"2024-09-02T10:07:29.964663+0000","last_active":"2024-09-02T10:07:41.637263+0000","last_peered":"2024-09-02T10:07:41.637263+0000","last_clean":"2024-09-02T10:07:41.637263+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:07:41.637263+0000","last_undegraded":"2024-09-02T10:07:41.637263+0000","last_fullsized":"2024-09-02T10:07:41.637263+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:29.964607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:29.964607+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:30.312469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"268'7134","reported_seq":7303,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.954252+0000","last_change":"2024-09-02T10:07:35.803712+0000","last_active":"2024-09-02T10:07:41.954252+0000","last_peered":"2024-09-02T10:07:41.954252+0000","last_clean":"2024-09-02T10:07:41.954252+0000","last_became_active":"2024-09-02T10:05:56.876397+0000","last_became_peered":"2024-09-02T10:05:56.876397+0000","last_unstale":"2024-09-02T10:07:41.954252+0000","last_undegraded":"2024-09-02T10:07:41.954252+0000","last_fullsized":"2024-09-02T10:07:41.954252+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6802","last_scrub_stamp":"2024-09-02T10:07:35.803575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:35.803575+0000","objects_scrubbed":6687,"log_size":7134,"log_dups_size":0,"ondisk_log_size":7134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:04.747241+0000","scrub_duration":1595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413401088,"num_objects":6308,"num_object_clones":0,"num_object_copies":18924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7134,"num_write_kb":430144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":181,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:38.860412+0000","last_change":"2024-09-02T10:07:33.808790+0000","last_active":"2024-09-02T10:07:38.860412+0000","last_peered":"2024-09-02T10:07:38.860412+0000","last_clean":"2024-09-02T10:07:38.860412+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:07:38.860412+0000","last_undegraded":"2024-09-02T10:07:38.860412+0000","last_fullsized":"2024-09-02T10:07:38.860412+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:33.808730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:33.808730+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:57.847747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"267'7201","reported_seq":7351,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.511118+0000","last_change":"2024-09-02T10:07:05.712175+0000","last_active":"2024-09-02T10:07:41.511118+0000","last_peered":"2024-09-02T10:07:41.511118+0000","last_clean":"2024-09-02T10:07:41.511118+0000","last_became_active":"2024-09-02T10:05:56.581229+0000","last_became_peered":"2024-09-02T10:05:56.581229+0000","last_unstale":"2024-09-02T10:07:41.511118+0000","last_undegraded":"2024-09-02T10:07:41.511118+0000","last_fullsized":"2024-09-02T10:07:41.511118+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5000","last_scrub_stamp":"2024-09-02T10:07:05.712114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:05.712114+0000","objects_scrubbed":4957,"log_size":7201,"log_dups_size":0,"ondisk_log_size":7201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:31.644516+0000","scrub_duration":1394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419758080,"num_objects":6405,"num_object_clones":0,"num_object_copies":19215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7201,"num_write_kb":435392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":397,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.642080+0000","last_change":"2024-09-02T10:07:37.379407+0000","last_active":"2024-09-02T10:07:41.642080+0000","last_peered":"2024-09-02T10:07:41.642080+0000","last_clean":"2024-09-02T10:07:41.642080+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:07:41.642080+0000","last_undegraded":"2024-09-02T10:07:41.642080+0000","last_fullsized":"2024-09-02T10:07:41.642080+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:37.379359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:37.379359+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:49.767013+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"268'7473","reported_seq":7644,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:43.321248+0000","last_change":"2024-09-02T10:07:30.485290+0000","last_active":"2024-09-02T10:07:43.321248+0000","last_peered":"2024-09-02T10:07:43.321248+0000","last_clean":"2024-09-02T10:07:43.321248+0000","last_became_active":"2024-09-02T10:05:56.876573+0000","last_became_peered":"2024-09-02T10:05:56.876573+0000","last_unstale":"2024-09-02T10:07:43.321248+0000","last_undegraded":"2024-09-02T10:07:43.321248+0000","last_fullsized":"2024-09-02T10:07:43.321248+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6788","last_scrub_stamp":"2024-09-02T10:07:30.485127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.485127+0000","objects_scrubbed":6730,"log_size":7473,"log_dups_size":0,"ondisk_log_size":7473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:31.184390+0000","scrub_duration":2030,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420675584,"num_objects":6419,"num_object_clones":0,"num_object_copies":19257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7473,"num_write_kb":444544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":192,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:38.860186+0000","last_change":"2024-09-02T10:07:28.824347+0000","last_active":"2024-09-02T10:07:38.860186+0000","last_peered":"2024-09-02T10:07:38.860186+0000","last_clean":"2024-09-02T10:07:38.860186+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:07:38.860186+0000","last_undegraded":"2024-09-02T10:07:38.860186+0000","last_fullsized":"2024-09-02T10:07:38.860186+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:28.824291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:28.824291+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:45.861891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"267'7347","reported_seq":7499,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:40.859942+0000","last_change":"2024-09-02T10:07:04.549316+0000","last_active":"2024-09-02T10:07:40.859942+0000","last_peered":"2024-09-02T10:07:40.859942+0000","last_clean":"2024-09-02T10:07:40.859942+0000","last_became_active":"2024-09-02T10:05:56.876540+0000","last_became_peered":"2024-09-02T10:05:56.876540+0000","last_unstale":"2024-09-02T10:07:40.859942+0000","last_undegraded":"2024-09-02T10:07:40.859942+0000","last_fullsized":"2024-09-02T10:07:40.859942+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'4967","last_scrub_stamp":"2024-09-02T10:07:04.549227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:04.549227+0000","objects_scrubbed":4900,"log_size":7347,"log_dups_size":0,"ondisk_log_size":7347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:16.373684+0000","scrub_duration":1666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":429391896,"num_objects":6553,"num_object_clones":0,"num_object_copies":19659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6553,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7347,"num_write_kb":444737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":426,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.633336+0000","last_change":"2024-09-02T10:07:27.771848+0000","last_active":"2024-09-02T10:07:41.633336+0000","last_peered":"2024-09-02T10:07:41.633336+0000","last_clean":"2024-09-02T10:07:41.633336+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:07:41.633336+0000","last_undegraded":"2024-09-02T10:07:41.633336+0000","last_fullsized":"2024-09-02T10:07:41.633336+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:27.771782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:27.771782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:42.542912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"268'7255","reported_seq":7424,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.819503+0000","last_change":"2024-09-02T10:07:31.723546+0000","last_active":"2024-09-02T10:07:41.819503+0000","last_peered":"2024-09-02T10:07:41.819503+0000","last_clean":"2024-09-02T10:07:41.819503+0000","last_became_active":"2024-09-02T10:05:56.585883+0000","last_became_peered":"2024-09-02T10:05:56.585883+0000","last_unstale":"2024-09-02T10:07:41.819503+0000","last_undegraded":"2024-09-02T10:07:41.819503+0000","last_fullsized":"2024-09-02T10:07:41.819503+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6718","last_scrub_stamp":"2024-09-02T10:07:31.723385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:31.723385+0000","objects_scrubbed":6667,"log_size":7255,"log_dups_size":0,"ondisk_log_size":7255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:59.746574+0000","scrub_duration":1623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417660928,"num_objects":6373,"num_object_clones":0,"num_object_copies":19119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7255,"num_write_kb":436096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":341,"reported_epoch":269,"state":"active+clean","last_fresh":"2024-09-02T10:07:43.866076+0000","last_change":"2024-09-02T10:06:34.728115+0000","last_active":"2024-09-02T10:07:43.866076+0000","last_peered":"2024-09-02T10:07:43.866076+0000","last_clean":"2024-09-02T10:07:43.866076+0000","last_became_active":"2024-09-02T10:06:32.874830+0000","last_became_peered":"2024-09-02T10:06:32.874830+0000","last_unstale":"2024-09-02T10:07:43.866076+0000","last_undegraded":"2024-09-02T10:07:43.866076+0000","last_fullsized":"2024-09-02T10:07:43.866076+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.728059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.728059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:00.290725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"267'7196","reported_seq":7349,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:40.876710+0000","last_change":"2024-09-02T10:07:06.845356+0000","last_active":"2024-09-02T10:07:40.876710+0000","last_peered":"2024-09-02T10:07:40.876710+0000","last_clean":"2024-09-02T10:07:40.876710+0000","last_became_active":"2024-09-02T10:05:56.584450+0000","last_became_peered":"2024-09-02T10:05:56.584450+0000","last_unstale":"2024-09-02T10:07:40.876710+0000","last_undegraded":"2024-09-02T10:07:40.876710+0000","last_fullsized":"2024-09-02T10:07:40.876710+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5143","last_scrub_stamp":"2024-09-02T10:07:06.845283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:06.845283+0000","objects_scrubbed":5114,"log_size":7196,"log_dups_size":0,"ondisk_log_size":7196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:09.749906+0000","scrub_duration":1004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418381824,"num_objects":6384,"num_object_clones":0,"num_object_copies":19152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7196,"num_write_kb":434560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":198,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-02T10:07:38.860540+0000","last_change":"2024-09-02T10:06:34.005118+0000","last_active":"2024-09-02T10:07:38.860540+0000","last_peered":"2024-09-02T10:07:38.860540+0000","last_clean":"2024-09-02T10:07:38.860540+0000","last_became_active":"2024-09-02T10:06:32.876151+0000","last_became_peered":"2024-09-02T10:06:32.876151+0000","last_unstale":"2024-09-02T10:07:38.860540+0000","last_undegraded":"2024-09-02T10:07:38.860540+0000","last_fullsized":"2024-09-02T10:07:38.860540+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.005073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.005073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:48.829803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"268'7232","reported_seq":7385,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.935864+0000","last_change":"2024-09-02T10:07:06.655925+0000","last_active":"2024-09-02T10:07:41.935864+0000","last_peered":"2024-09-02T10:07:41.935864+0000","last_clean":"2024-09-02T10:07:41.935864+0000","last_became_active":"2024-09-02T10:05:56.585870+0000","last_became_peered":"2024-09-02T10:05:56.585870+0000","last_unstale":"2024-09-02T10:07:41.935864+0000","last_undegraded":"2024-09-02T10:07:41.935864+0000","last_fullsized":"2024-09-02T10:07:41.935864+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5095","last_scrub_stamp":"2024-09-02T10:07:06.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:06.655797+0000","objects_scrubbed":5054,"log_size":7232,"log_dups_size":0,"ondisk_log_size":7232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:08.831810+0000","scrub_duration":1179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415629312,"num_objects":6342,"num_object_clones":0,"num_object_copies":19026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7232,"num_write_kb":434368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":213,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-02T10:07:41.636122+0000","last_change":"2024-09-02T10:06:33.485010+0000","last_active":"2024-09-02T10:07:41.636122+0000","last_peered":"2024-09-02T10:07:41.636122+0000","last_clean":"2024-09-02T10:07:41.636122+0000","last_became_active":"2024-09-02T10:06:32.875557+0000","last_became_peered":"2024-09-02T10:06:32.875557+0000","last_unstale":"2024-09-02T10:07:41.636122+0000","last_undegraded":"2024-09-02T10:07:41.636122+0000","last_fullsized":"2024-09-02T10:07:41.636122+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.484964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.484964+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:01.399206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":6688145432,"num_objects":102054,"num_object_clones":0,"num_object_copies":306162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102054,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":117028,"num_write_kb":7010561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20030697472,"data_stored":20030685256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117028,"ondisk_log_size":117028,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383771,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3511576,"kb_used_data":3295744,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":90233576,"statfs":{"total":95995035648,"available":92399181824,"internally_reserved":0,"allocated":3374841856,"data_stored":3374048934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":220985796},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612704,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841637,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1387812,"kb_used_data":1226128,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92357340,"statfs":{"total":95995035648,"available":94573916160,"internally_reserved":0,"allocated":1255555072,"data_stored":1254753490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":165542336},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070571,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903600,"kb_used_data":800216,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":92841552,"statfs":{"total":95995035648,"available":95069749248,"internally_reserved":0,"allocated":819421184,"data_stored":818619078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299501,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1791388,"kb_used_data":1656076,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":91953764,"statfs":{"total":95995035648,"available":94160654336,"internally_reserved":0,"allocated":1695821824,"data_stored":1695021286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":138541514},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528434,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2185056,"kb_used_data":2059468,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91560096,"statfs":{"total":95995035648,"available":93757538304,"internally_reserved":0,"allocated":2108895232,"data_stored":2108097722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":128580039},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724663,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1769436,"kb_used_data":1666060,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":91975716,"statfs":{"total":95995035648,"available":94183133184,"internally_reserved":0,"allocated":1706045440,"data_stored":1705244902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953597,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2964124,"kb_used_data":2824652,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":90781028,"statfs":{"total":95995035648,"available":92959772672,"internally_reserved":0,"allocated":2892443648,"data_stored":2891646138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149827,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2226012,"kb_used_data":2024076,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":91519140,"statfs":{"total":95995035648,"available":93715599360,"internally_reserved":0,"allocated":2072653824,"data_stored":2071856314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":206764488},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411462,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":575844,"kb_used_data":410004,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93169308,"statfs":{"total":95995035648,"available":95405371392,"internally_reserved":0,"allocated":419844096,"data_stored":419043034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1791260,"kb_used_data":1643468,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91953892,"statfs":{"total":95995035648,"available":94160785408,"internally_reserved":0,"allocated":1682911232,"data_stored":1682113722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738513,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2209048,"kb_used_data":2065408,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91536104,"statfs":{"total":95995035648,"available":93732970496,"internally_reserved":0,"allocated":2114977792,"data_stored":2114184878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2105081856,"data_stored":2105081856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1672151040,"data_stored":1672151040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":416022528,"data_stored":416022528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2069954560,"data_stored":2069954560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2869493760,"data_stored":2869493760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1691947008,"data_stored":1691942936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2097283072,"data_stored":2097283072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1692864512,"data_stored":1692860440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":816250880,"data_stored":816250880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1252986880,"data_stored":1252982808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":3346661376,"data_stored":3346661376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:07:47.673 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-02T10:07:47.673 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pg_num 41 2024-09-02T10:07:47.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:47.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:47.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:47.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:47.888 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:47.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:47.915 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:47.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1466758554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:48.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:48.043 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:48.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:48.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:48.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1466758554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:48.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:48.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:48.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:48.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:48.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:48.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:48.400 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:48.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:48.425 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:48.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:48.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:48.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:48.569 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:48.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:48.678 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:48.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:48.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:48.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:48.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:48.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:48.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:48 smithi151 ceph-mon[16639]: pgmap v653: 47 pgs: 47 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-02T10:07:48.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:48.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:48.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:48.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:48 smithi187 ceph-mon[20072]: pgmap v653: 47 pgs: 47 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-02T10:07:49.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:49.061 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:49.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:49.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:49.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:48 smithi162 ceph-mon[20279]: pgmap v653: 47 pgs: 47 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s 2024-09-02T10:07:49.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:49.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:49.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:49.187 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:49.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:49.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:49.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:49.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:49.423 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:49.450 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:49.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:49.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:49.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:49.578 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:49.579 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:07:49.579 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:07:49.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.579 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:124065 3.b 3:de2b4cfe:::benchmark_data_smithi187_59989_object14522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-02T10:07:49.579 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:49.565415+0000", 2024-09-02T10:07:49.579 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.011641556000000001, 2024-09-02T10:07:49.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011674441000000001, 2024-09-02T10:07:49.579 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 124065 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:49.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.565415+0000", 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.565415+0000", 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.565416+0000", 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.629e-06 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.565422+0000", 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4219999999999999e-06 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.565438+0000", 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5845000000000002e-05 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.565447+0000", 2024-09-02T10:07:49.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8279999999999998e-06 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.565508+0000", 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0989999999999997e-05 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.565650+0000", 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014195599999999999 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.565782+0000", 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131116 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.569549+0000", 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037678640000000001 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.570129+0000", 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057927900000000004 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:49.584 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:124072 3.2 3:47865f00:::benchmark_data_smithi187_59989_object14529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:49.571664+0000", 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0053919129999999999, 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055063289999999999, 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:49.585 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 124072 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.571664+0000", 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.571664+0000", 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.571666+0000", 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1990000000000001e-06 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.571670+0000", 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6550000000000003e-06 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.571682+0000", 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2099000000000001e-05 2024-09-02T10:07:49.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.571689+0000", 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0849999999999999e-06 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.571757+0000", 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7755000000000004e-05 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:49.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.571859+0000", 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101499 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.571952+0000", 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3511999999999995e-05 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.575709+0000", 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037564320000000001 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:49.589 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:124056 3.2 e269/226)", 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:49.563462+0000", 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.013594601, 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013768173, 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.563462+0000", 2024-09-02T10:07:49.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.563462+0000", 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.563462+0000", 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9200000000000001e-07 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.563466+0000", 2024-09-02T10:07:49.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9700000000000001e-06 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.575507+0000", 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012040456 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.575513+0000", 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6239999999999996e-06 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:49.592 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:124056 3.2 3:41ad7bbf:::benchmark_data_smithi187_59989_object14513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e269)", 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:49.558855+0000", 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.018201302999999999, 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018417085, 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 124056 2024-09-02T10:07:49.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.558855+0000", 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.558855+0000", 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.558857+0000", 2024-09-02T10:07:49.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8929999999999999e-06 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.558863+0000", 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0809999999999999e-06 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.558882+0000", 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.91e-05 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:07:49.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.558890+0000", 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0469999999999994e-06 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.559016+0000", 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126072 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.559130+0000", 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114258 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.559243+0000", 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112923 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.563165+0000", 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039212179999999998 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:49.564294+0000", 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011292120000000001 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:49.597 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:49.598 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:49.598 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:49.598 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:07:49.598 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 4 2024-09-02T10:07:49.598 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:07:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:49.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:49.599 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:49.649 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:07:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3577027085' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:49.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:49.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:49.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:49.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:49.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:49.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:49.952 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:49.975 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:49.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:49.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3577027085' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:50.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:50.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3577027085' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:07:50.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:50.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:50.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:50.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:50.227 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:50.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:50.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:50.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:50.353 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:50.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:50.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:50.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:50.478 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:50.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:50.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:07:50.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:50.604 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:50.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:50.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:50.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:50.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:50.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:07:50.830 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:50.854 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:50.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:50 smithi151 ceph-mon[16639]: pgmap v654: 47 pgs: 47 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-02T10:07:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:50 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3577027085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:50 smithi151 ceph-mon[16639]: osdmap e270: 12 total, 12 up, 11 in 2024-09-02T10:07:50.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:50.981 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:50.981 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:50 smithi187 ceph-mon[20072]: pgmap v654: 47 pgs: 47 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-02T10:07:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:50 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3577027085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:50 smithi187 ceph-mon[20072]: osdmap e270: 12 total, 12 up, 11 in 2024-09-02T10:07:51.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:51.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:50 smithi162 ceph-mon[20279]: pgmap v654: 47 pgs: 47 active+clean; 6.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.05k op/s 2024-09-02T10:07:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:50 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3577027085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:07:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:50 smithi162 ceph-mon[20279]: osdmap e270: 12 total, 12 up, 11 in 2024-09-02T10:07:51.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:51.108 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:51.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:51.237 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:51.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:51.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:51.252 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pg_num 41 < pg_num_max 32 2024-09-02T10:07:51.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:51.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:51.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:51.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:51.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:51.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:51.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:51.617 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:51.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:51.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:51.743 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:51.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:51.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:51.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:51.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:51.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:51.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1092401450' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-02T10:07:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:51 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-02T10:07:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:07:51.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:52.005 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:52.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1092401450' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-02T10:07:52.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:51 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-02T10:07:52.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:07:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:52.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:07:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1092401450' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-02T10:07:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:51 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "41"} : dispatch 2024-09-02T10:07:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:07:52.111 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:52.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:52.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:52.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:52.265 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:52.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:52.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:52.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:52.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:52.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:52.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:52.521 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:52.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:52.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:52.626 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:52.627 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:52.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:52.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:52.760 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:52.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:52.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:52.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:52.889 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:52.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:52 smithi187 ceph-mon[20072]: pgmap v656: 47 pgs: 47 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-09-02T10:07:52.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:52.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:52.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:52.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:52 smithi151 ceph-mon[16639]: pgmap v656: 47 pgs: 47 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-09-02T10:07:52.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:52 smithi162 ceph-mon[20279]: pgmap v656: 47 pgs: 47 active+clean; 5.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 867 op/s 2024-09-02T10:07:53.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:53.039 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:53.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:53.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:53.169 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:53.169 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:53.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:53.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:53.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:53.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:53.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:53.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:53.475 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:07:53.475 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:07:53.475 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:07:53.475 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:07:53.476 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:07:53.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:53.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:53.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:53.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:53.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:53.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:53.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:53.810 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:53.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:53.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:53.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:53.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:54.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:54.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:54.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:54.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:54.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:54.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:54.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:54.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:54.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:54.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:54.425 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:54.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:07:54.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:54.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:54.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:54.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:54.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:54.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:54.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:07:54.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:54.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:54.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:54 smithi187 ceph-mon[20072]: pgmap v657: 47 pgs: 47 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 946 op/s 2024-09-02T10:07:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:54 smithi187 ceph-mon[20072]: osdmap e271: 12 total, 12 up, 11 in 2024-09-02T10:07:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1730608420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:54.800 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:07:54.801 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:07:54.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:54.832 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:54.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:54.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:54.834 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:07:54.834 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:07:54.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:07:54.862 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:07:54.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:07:54.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:07:54.918 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:07:54.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:07:54.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:54 smithi151 ceph-mon[16639]: pgmap v657: 47 pgs: 47 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 946 op/s 2024-09-02T10:07:54.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:54 smithi151 ceph-mon[16639]: osdmap e271: 12 total, 12 up, 11 in 2024-09-02T10:07:54.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1730608420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:54.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:54.942 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:07:54.942 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:07:54.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:54.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:54.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:54.964 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:07:54.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:07:54.993 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:07:54.993 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:07:55.023 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:07:55.023 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:07:55.051 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:07:55.052 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:07:55.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:55.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:07:55.077 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:07:55.087 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:55.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:55.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:54 smithi162 ceph-mon[20279]: pgmap v657: 47 pgs: 47 active+clean; 5.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 946 op/s 2024-09-02T10:07:55.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:54 smithi162 ceph-mon[20279]: osdmap e271: 12 total, 12 up, 11 in 2024-09-02T10:07:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1730608420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:07:55.104 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:07:55.104 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:07:55.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:55.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:55.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:55.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:55.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:55.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:55.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:55.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:55.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:55.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:55.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:55.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:55.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:55.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:55.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:55.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:55.730 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:55.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:55.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:55.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:07:55.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:55.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:07:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:55 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:07:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:55 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:07:55.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:55.987 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:55.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:55 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:07:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:55 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:07:56.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:56.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:55 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:07:56.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:55 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:07:56.117 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:56.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:56.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:56.131 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:56.132 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:56.218 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:56.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:56.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:56.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:56.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:56.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:56.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:56.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:56.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:56.495 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:56.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:56.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:56.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:56.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:56.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:56.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:56.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:56.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:56.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:56.884 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:56.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:56.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:56 smithi151 ceph-mon[16639]: pgmap v659: 47 pgs: 47 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 822 op/s 2024-09-02T10:07:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:56 smithi151 ceph-mon[16639]: osdmap e272: 12 total, 12 up, 11 in 2024-09-02T10:07:56.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:56 smithi187 ceph-mon[20072]: pgmap v659: 47 pgs: 47 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 822 op/s 2024-09-02T10:07:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:56 smithi187 ceph-mon[20072]: osdmap e272: 12 total, 12 up, 11 in 2024-09-02T10:07:57.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:57.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:57.013 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:57.039 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:07:57.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:07:57.040 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-02T10:07:57.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:07:57.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:56 smithi162 ceph-mon[20279]: pgmap v659: 47 pgs: 47 active+clean; 5.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 822 op/s 2024-09-02T10:07:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:56 smithi162 ceph-mon[20279]: osdmap e272: 12 total, 12 up, 11 in 2024-09-02T10:07:57.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:57.130 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:07:57.130 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:07:57.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:57.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:57.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:07:57.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:57.265 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:57.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:57.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:57.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:57.391 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:57.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:57.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:07:57.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:57.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:07:57.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:07:57.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:57.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:07:57.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:57.793 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:57.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:57.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:07:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:57.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:07:57.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:07:58.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:07:58.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:58.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:58.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:58.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:58.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:07:58.099 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:58.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:58.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:58.192 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:58.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:58.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:07:58.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:58.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:58.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:07:58.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:07:58.424 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:07:58.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:58.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:07:58.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:07:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:58.572 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:58.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:07:58.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:58.695 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:07:58.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:58.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:58.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:58.725 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:07:58.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:58.827 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:58.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:58.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:58.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:58 smithi162 ceph-mon[20279]: pgmap v661: 47 pgs: 47 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-02T10:07:58.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2230407328' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:07:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:58 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:07:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:58 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:07:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:58 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:07:58.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:07:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:58 smithi151 ceph-mon[16639]: pgmap v661: 47 pgs: 47 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-02T10:07:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2230407328' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:07:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:58 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:07:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:58 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:07:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:58 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:07:58.958 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:07:58.960 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:07:58.960 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 273, 2024-09-02T10:07:58.960 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:07:58.960 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:07:58.960 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:07:58.662904+0000", 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:02:47.267122+0000", 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5804032, 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: "nodeep-scrub", 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:07:58.961 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 3, 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "jewel", 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:07:58.962 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:07:58.963 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:07:58.964 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "264", 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "241", 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:07:58.965 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:07:58.966 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:07:58.967 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:07:58.968 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 1.9199999570846558, 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 1.9199999570846558, 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 1.7699999809265137, 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 1.7699999809265137, 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 3, 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_1", 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:05:55.119628+0000", 2024-09-02T10:07:58.969 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:07:58.970 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:07:58.970 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:07:58.970 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:07:58.970 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:07:58.970 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:07:58.970 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:07:58.970 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:07:58.971 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:07:58.971 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:07:58.972 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "230", 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:07:58.973 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:07:58.974 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:07:58.975 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.2400000095367432, 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.2400000095367432, 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.059999942779541, 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.059999942779541, 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.976 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 243, 2024-09-02T10:07:58.977 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.978 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.979 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.980 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:07:58.981 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.982 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 243, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:58.983 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.984 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:58.985 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.985 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.985 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:07:58.985 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:07:58.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.985 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.985 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:07:58.985 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:07:58.986 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.986 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.986 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:58.986 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.986 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:07:58.989 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 241, 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.990 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.991 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.993 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.993 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:07:58.993 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:07:58.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.993 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.993 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:07:58.993 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:07:58.993 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.994 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:07:58.995 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 241, 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.996 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:58.998 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:58.999 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:59.000 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:07:59.002 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:59.003 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 243, 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:07:59.004 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:07:59.005 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:07:59.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 243, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:59.009 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.010 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:59.011 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:07:59.013 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:59.014 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 241, 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.015 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.016 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.017 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.018 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:59.019 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:07:59.020 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 243, 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:07:59.021 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:07:59.022 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.023 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:07:59.024 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 243, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:59.026 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.028 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.029 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:59.030 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 241, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:59.031 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.032 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.033 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:07:59.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.036 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:07:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:07:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-09-02T10:07:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-09-02T10:07:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:59.037 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 124, 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:07:59.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.040 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:07:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:07:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:07:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:07:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.042 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.043 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 243, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:07:59.044 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:07:59.045 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.046 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.047 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:07:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:07:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:07:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:07:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:07:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:07:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:07:59.048 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.049 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:06:49.960342+0000", 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.050 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:06:41.073038+0000", 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.051 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:07:19.425111+0000", 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:06:39.086534+0000", 2024-09-02T10:07:59.052 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:07:37.884256+0000", 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:07:59.053 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:07:13.352349+0000", 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.054 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:07:21.655885+0000", 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:07:54.831646+0000", 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:07:59.055 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:06:34.636427+0000", 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.056 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:06:59.885058+0000", 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 65536, 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:07:24.683309+0000", 2024-09-02T10:07:59.057 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:07:07.596420+0000", 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [], 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-09-02T10:07:59.058 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [], 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:07:59.059 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:07:59.060 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:07:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:07:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:07:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:07:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:07:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:07:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:07:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:07:59.061 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:07:59.061 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:07:59.062 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:59.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:07:59.062 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:07:59.062 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:07:59.062 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:58 smithi187 ceph-mon[20072]: pgmap v661: 47 pgs: 47 active+clean; 5.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-02T10:07:59.063 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2230407328' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:07:59.063 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:58 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:07:59.063 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:58 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:07:59.063 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:58 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:07:59.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:07:59.189 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:59.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:07:59.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:59.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:59.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:59.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:59.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:59.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:59.443 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:59.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:59.570 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:59.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:59.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:07:59.698 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:59.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:59.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:07:59.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:07:59.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:07:59.829 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:59 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:59.829 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:59 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:07:59.829 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:59 smithi162 ceph-mon[20279]: osdmap e273: 12 total, 12 up, 11 in 2024-09-02T10:07:59.829 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:59 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:07:59.829 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:59 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:07:59.829 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:59 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:07:59.829 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1963064490' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:07:59.829 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:59 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:07:59.830 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:07:59 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:07:59.866 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:07:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:59.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:07:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:59 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:07:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:59 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:07:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:59 smithi151 ceph-mon[16639]: osdmap e273: 12 total, 12 up, 11 in 2024-09-02T10:07:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:07:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:07:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:07:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1963064490' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:07:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:59 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:07:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:07:59 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:07:59.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:07:59.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:07:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:07:59.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:59 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:59 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:08:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:59 smithi187 ceph-mon[20072]: osdmap e273: 12 total, 12 up, 11 in 2024-09-02T10:08:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:59 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:08:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:59 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:08:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:59 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:08:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1963064490' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:08:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:59 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:08:00.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:07:59 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:08:00.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:00.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:00.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:00.131 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:08:00.131 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:08:00.154 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:00.157 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:08:00.157 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:08:00.179 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:08:00.180 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:08:00.205 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:08:00.206 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:08:00.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:00.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:08:00.234 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:08:00.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:00.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:00.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:00.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:08:00.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:08:00.292 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:08:00.292 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:08:00.317 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:08:00.317 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:08:00.342 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:08:00.342 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:08:00.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:00.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:08:00.368 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:08:00.381 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:00.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:00.392 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:08:00.392 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:08:00.419 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:08:00.419 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:08:00.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:00.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:00.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:00.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:00.637 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:00.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:00.716 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:08:00.717 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:08:00.717 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:08:00.717 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:08:00.717 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:08:00.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:00.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:00.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: pgmap v663: 47 pgs: 47 active+clean; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-09-02T10:08:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:08:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:08:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:08:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:08:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: osdmap e274: 12 total, 12 up, 11 in 2024-09-02T10:08:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2749814411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:00.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:08:00.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:00 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:08:00.859 INFO:tasks.thrashosds.thrasher:j is {'epoch': 273, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:07:58.662904+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:02:47.267122+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9199999570846558, 'score_stable': 1.9199999570846558, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 1.7699999809265137, 'raw_score_stable': 1.7699999809265137, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-02T10:05:55.119628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 241, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 241, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 241, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 241, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 124, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:06:49.960342+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:06:41.073038+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:07:19.425111+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:06:39.086534+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:07:37.884256+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:07:13.352349+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:07:21.655885+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:07:54.831646+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:06:34.636427+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:06:59.885058+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-02T10:07:24.683309+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:07:07.596420+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:08:00.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:08:00.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:00.891 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:00.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:00.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:00.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: pgmap v663: 47 pgs: 47 active+clean; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-09-02T10:08:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:08:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:08:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:08:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:08:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: osdmap e274: 12 total, 12 up, 11 in 2024-09-02T10:08:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2749814411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:08:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:00 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:08:00.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: pgmap v663: 47 pgs: 47 active+clean; 5.4 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 942 op/s 2024-09-02T10:08:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:08:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:08:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:08:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:08:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: osdmap e274: 12 total, 12 up, 11 in 2024-09-02T10:08:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2749814411' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:08:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:00 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:08:01.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:01.017 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:01.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:01.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:01.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:01.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:01.276 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:01.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:01.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:01.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:01.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:01.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:08:01.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:01.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:08:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:01.524 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:01.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:01.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:01.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:01.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:01.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:01.752 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:01.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:01.791 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:01.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:01.892 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:01.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:01.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:01.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:02.011 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:02.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:02.019 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:02.039 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:02.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:02.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:02.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:02.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:02.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:02.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:02.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:02.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:02.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:02.337 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:08:02.337 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:08:02.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:08:02.338 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:08:02.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.338 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "replica_scrub(pg: 1.2,from:0'0,to:0'0,epoch:104/103,start:1:40000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-02T10:08:02.338 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:50.941506+0000", 2024-09-02T10:08:02.338 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 371.39232977400002, 2024-09-02T10:08:02.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.3952948, 2024-09-02T10:08:02.339 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.339 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.339 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:08:02.339 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941506+0000", 2024-09-02T10:08:02.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941506+0000", 2024-09-02T10:08:02.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.342 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.342 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941510+0000", 2024-09-02T10:08:02.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7570000000000002e-06 2024-09-02T10:08:02.342 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:50.941517+0000", 2024-09-02T10:08:02.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0249999999999997e-06 2024-09-02T10:08:02.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336499+0000", 2024-09-02T10:08:02.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.39498246199999998 2024-09-02T10:08:02.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336515+0000", 2024-09-02T10:08:02.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.562e-05 2024-09-02T10:08:02.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336678+0000", 2024-09-02T10:08:02.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016350699999999999 2024-09-02T10:08:02.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336718+0000", 2024-09-02T10:08:02.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0278999999999998e-05 2024-09-02T10:08:02.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:51.336801+0000", 2024-09-02T10:08:02.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2150000000000005e-05 2024-09-02T10:08:02.348 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.348 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.349 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.349 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127455 2.d 2.f151987d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:08:02.349 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.256858+0000", 2024-09-02T10:08:02.349 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 257.07697748599998, 2024-09-02T10:08:02.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96215398699999999, 2024-09-02T10:08:02.350 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.350 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.350 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:08:02.350 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.351 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.351 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.351 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127455 2024-09-02T10:08:02.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.351 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.352 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256858+0000", 2024-09-02T10:08:02.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256858+0000", 2024-09-02T10:08:02.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256860+0000", 2024-09-02T10:08:02.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5039999999999999e-06 2024-09-02T10:08:02.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256866+0000", 2024-09-02T10:08:02.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0959999999999997e-06 2024-09-02T10:08:02.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256875+0000", 2024-09-02T10:08:02.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1940000000000004e-06 2024-09-02T10:08:02.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256884+0000", 2024-09-02T10:08:02.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3030000000000005e-06 2024-09-02T10:08:02.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256938+0000", 2024-09-02T10:08:02.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4259999999999999e-05 2024-09-02T10:08:02.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256949+0000", 2024-09-02T10:08:02.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0624e-05 2024-09-02T10:08:02.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.218985+0000", 2024-09-02T10:08:02.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96203561699999995 2024-09-02T10:08:02.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219012+0000", 2024-09-02T10:08:02.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7389000000000002e-05 2024-09-02T10:08:02.361 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.361 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.361 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.362 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127536 2.d 2.4160433d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:08:02.362 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.711015+0000", 2024-09-02T10:08:02.362 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 256.62282030900002, 2024-09-02T10:08:02.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.50804419700000003, 2024-09-02T10:08:02.363 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.363 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.363 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:08:02.363 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.364 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.364 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.364 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127536 2024-09-02T10:08:02.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.364 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711015+0000", 2024-09-02T10:08:02.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711015+0000", 2024-09-02T10:08:02.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711017+0000", 2024-09-02T10:08:02.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9400000000000001e-06 2024-09-02T10:08:02.367 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.368 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.368 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711026+0000", 2024-09-02T10:08:02.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3729999999999993e-06 2024-09-02T10:08:02.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711044+0000", 2024-09-02T10:08:02.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8017e-05 2024-09-02T10:08:02.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711068+0000", 2024-09-02T10:08:02.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4510000000000001e-05 2024-09-02T10:08:02.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711155+0000", 2024-09-02T10:08:02.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7280999999999993e-05 2024-09-02T10:08:02.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711167+0000", 2024-09-02T10:08:02.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1497e-05 2024-09-02T10:08:02.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.372 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219052+0000", 2024-09-02T10:08:02.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.50788502000000002 2024-09-02T10:08:02.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219059+0000", 2024-09-02T10:08:02.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.559e-06 2024-09-02T10:08:02.374 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.374 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.374 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.375 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127544 2.d 2.969bbf1d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:08:02.377 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.788037+0000", 2024-09-02T10:08:02.377 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 256.54579822599999, 2024-09-02T10:08:02.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.43104598700000002, 2024-09-02T10:08:02.378 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.378 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.378 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:08:02.378 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.378 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.378 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.379 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127544 2024-09-02T10:08:02.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.379 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788037+0000", 2024-09-02T10:08:02.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788037+0000", 2024-09-02T10:08:02.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788039+0000", 2024-09-02T10:08:02.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0760000000000001e-06 2024-09-02T10:08:02.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788049+0000", 2024-09-02T10:08:02.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4150000000000005e-06 2024-09-02T10:08:02.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788070+0000", 2024-09-02T10:08:02.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0809999999999999e-05 2024-09-02T10:08:02.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788079+0000", 2024-09-02T10:08:02.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8320000000000001e-06 2024-09-02T10:08:02.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788139+0000", 2024-09-02T10:08:02.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9336000000000002e-05 2024-09-02T10:08:02.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788147+0000", 2024-09-02T10:08:02.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7940000000000003e-06 2024-09-02T10:08:02.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219079+0000", 2024-09-02T10:08:02.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.43093289000000001 2024-09-02T10:08:02.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219083+0000", 2024-09-02T10:08:02.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8340000000000001e-06 2024-09-02T10:08:02.388 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.388 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.389 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.389 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127455 2.d 2:be198a8f:::benchmark_data_smithi187_45882_object29704:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-09-02T10:08:02.389 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:47.249583+0000", 2024-09-02T10:08:02.390 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 255.08425209699999, 2024-09-02T10:08:02.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30158227999999998, 2024-09-02T10:08:02.390 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.390 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.390 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.390 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.391 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.391 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.391 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127455 2024-09-02T10:08:02.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.391 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249583+0000", 2024-09-02T10:08:02.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249583+0000", 2024-09-02T10:08:02.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249586+0000", 2024-09-02T10:08:02.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3070000000000001e-06 2024-09-02T10:08:02.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249595+0000", 2024-09-02T10:08:02.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3219999999999999e-06 2024-09-02T10:08:02.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249611+0000", 2024-09-02T10:08:02.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5908000000000001e-05 2024-09-02T10:08:02.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249625+0000", 2024-09-02T10:08:02.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3954999999999999e-05 2024-09-02T10:08:02.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250314+0000", 2024-09-02T10:08:02.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00068951199999999998 2024-09-02T10:08:02.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250328+0000", 2024-09-02T10:08:02.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3358e-05 2024-09-02T10:08:02.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.516250+0000", 2024-09-02T10:08:02.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26592224800000003 2024-09-02T10:08:02.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.518815+0000", 2024-09-02T10:08:02.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025647310000000002 2024-09-02T10:08:02.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.518985+0000", 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017049599999999999 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531298+0000", 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012313098 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531372+0000", 2024-09-02T10:08:02.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3818000000000004e-05 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.549895+0000", 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018523234999999999 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.549957+0000", 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1729000000000005e-05 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.551166+0000", 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001208563 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127536 2.d 2:bcc20682:::benchmark_data_smithi187_45882_object29785:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:47.249649+0000", 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 255.084186499, 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30254320299999998, 2024-09-02T10:08:02.403 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127536 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249649+0000", 2024-09-02T10:08:02.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249649+0000", 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249650+0000", 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.37e-07 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249654+0000", 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9929999999999997e-06 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249662+0000", 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6139999999999992e-06 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249668+0000", 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8479999999999997e-06 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250366+0000", 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069775700000000002 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250375+0000", 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.285e-06 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.523344+0000", 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27296902099999998 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.523517+0000", 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017299000000000001 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.523683+0000", 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016615299999999999 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531437+0000", 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.007753551 2024-09-02T10:08:02.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531462+0000", 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4754000000000001e-05 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.551213+0000", 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.019751898 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.551247+0000", 2024-09-02T10:08:02.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3800000000000002e-05 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.552192+0000", 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00094500200000000002 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127544 2.d 2:b8fdd969:::benchmark_data_smithi187_45882_object29793:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e157)", 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:47.249677+0000", 2024-09-02T10:08:02.410 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 255.08415824599999, 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30345529100000002, 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127544 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249677+0000", 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249677+0000", 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249678+0000", 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5499999999999998e-07 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249680+0000", 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.17e-06 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249688+0000", 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6000000000000001e-06 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249693+0000", 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0989999999999998e-06 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250429+0000", 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073661199999999999 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250440+0000", 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0710000000000001e-05 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.527337+0000", 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27689745300000002 2024-09-02T10:08:02.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.527498+0000", 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000160342 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.527654+0000", 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000156021 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.535483+0000", 2024-09-02T10:08:02.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078291300000000001 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.537825+0000", 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023424259999999999 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.552232+0000", 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014406672000000001 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.552266+0000", 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4202000000000002e-05 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.553133+0000", 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00086639900000000001 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127547 2.d 2:bf1255db:::benchmark_data_smithi187_45882_object29796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:47.249699+0000", 2024-09-02T10:08:02.417 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 255.084136323, 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.34997614900000001, 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127547 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249699+0000", 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249699+0000", 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249700+0000", 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3000000000000001e-07 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249702+0000", 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.933e-06 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249709+0000", 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3309999999999998e-06 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249714+0000", 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0880000000000002e-06 2024-09-02T10:08:02.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250467+0000", 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075314800000000003 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250476+0000", 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6170000000000007e-06 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531604+0000", 2024-09-02T10:08:02.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28112852799999999 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531742+0000", 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137353 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531881+0000", 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013981700000000001 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.535629+0000", 2024-09-02T10:08:02.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037478059999999998 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.538014+0000", 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002384976 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.598001+0000", 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059986314999999998 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.598060+0000", 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9583999999999997e-05 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.599675+0000", 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001615223 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.424 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.d 183/157 [PushReplyOp(2:b0969168:::benchmark_data_smithi187_45882_object60943:head)])", 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:36.457663+0000", 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 205.87617261400001, 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36433320800000002, 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457663+0000", 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457663+0000", 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457664+0000", 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.13e-06 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457670+0000", 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2910000000000004e-06 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.820382+0000", 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.362711703 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.820397+0000", 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5423000000000001e-05 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.820576+0000", 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017897 2024-09-02T10:08:02.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.821996+0000", 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014196910000000001 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163245 2.d 2:b53557ce:::benchmark_data_smithi187_45882_object65494:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027769+0000", 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 185.30606678699999, 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36269144800000003, 2024-09-02T10:08:02.428 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163245 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027769+0000", 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027769+0000", 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027771+0000", 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7020000000000001e-06 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027781+0000", 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6700000000000006e-06 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027798+0000", 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6416999999999999e-05 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027810+0000", 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2568000000000001e-05 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028091+0000", 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028051500000000001 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028103+0000", 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2768999999999999e-05 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390352+0000", 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36224896000000001 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390460+0000", 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107847 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163286 2.d 2:b26ad3af:::benchmark_data_smithi187_45882_object65535:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027840+0000", 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 185.30599544099999, 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38364109499999999, 2024-09-02T10:08:02.433 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163286 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027840+0000", 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027840+0000", 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027841+0000", 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3600000000000003e-07 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027844+0000", 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4170000000000001e-06 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027856+0000", 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1659e-05 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027862+0000", 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5089999999999999e-06 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028144+0000", 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028157300000000002 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028154+0000", 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7010000000000008e-06 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390697+0000", 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36254281199999999 2024-09-02T10:08:02.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390784+0000", 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7717000000000001e-05 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390867+0000", 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2432e-05 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.401652+0000", 2024-09-02T10:08:02.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010785041 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.401689+0000", 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7502000000000001e-05 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411205+0000", 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0095160409999999994 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411244+0000", 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8296000000000001e-05 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411481+0000", 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023765900000000001 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163296 2.d 2:b1ae2ae4:::benchmark_data_smithi187_45882_object65545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:08:02.440 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 185.30596490900001, 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38880961400000003, 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163296 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4200000000000002e-07 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027883+0000", 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2315e-05 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027926+0000", 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2339000000000002e-05 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027946+0000", 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0801999999999998e-05 2024-09-02T10:08:02.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028193+0000", 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024658900000000002 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028198+0000", 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6009999999999997e-06 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.394381+0000", 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36618306299999998 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.394457+0000", 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6509999999999998e-05 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.394532+0000", 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4348e-05 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402216+0000", 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076842819999999997 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405720+0000", 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035040480000000001 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416431+0000", 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010711178999999999 2024-09-02T10:08:02.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416463+0000", 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2008999999999998e-05 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416680+0000", 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021718700000000001 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163297 2.d 2:b94d3aa0:::benchmark_data_smithi187_45882_object65546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027961+0000", 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 185.30587425900001, 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38891082999999999, 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163297 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.448 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027961+0000", 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027961+0000", 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027962+0000", 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8800000000000002e-07 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027967+0000", 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6299999999999997e-06 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027975+0000", 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7819999999999999e-06 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027981+0000", 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8309999999999997e-06 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028219+0000", 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000237774 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028223+0000", 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5980000000000001e-06 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.397985+0000", 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36976200399999998 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.398112+0000", 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127565 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.398213+0000", 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100368 2024-09-02T10:08:02.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405733+0000", 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0075208200000000001 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411164+0000", 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054306110000000001 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416727+0000", 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055628049999999997 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416766+0000", 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9609e-05 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416872+0000", 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105645 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.454 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163303 2.d 2:b483d011:::benchmark_data_smithi187_45882_object65552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 185.30584470700001, 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.39586157599999999, 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.455 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163303 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7800000000000002e-07 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027995+0000", 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3500000000000001e-06 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028001+0000", 2024-09-02T10:08:02.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2859999999999996e-06 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028006+0000", 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6430000000000003e-06 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028242+0000", 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023680199999999999 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028246+0000", 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2260000000000001e-06 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402251+0000", 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.374004893 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402324+0000", 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3100000000000001e-05 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402391+0000", 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7261999999999994e-05 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411493+0000", 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091022120000000001 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416394+0000", 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049004870000000002 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.423496+0000", 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071020859999999996 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.423554+0000", 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8511000000000003e-05 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.423852+0000", 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029824000000000002 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163305 2.d 2:bdea317b:::benchmark_data_smithi187_45882_object65554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 185.305811834, 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40774826600000003, 2024-09-02T10:08:02.462 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163305 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6199999999999998e-07 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028027+0000", 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9299999999999999e-06 2024-09-02T10:08:02.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028034+0000", 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5039999999999999e-06 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028038+0000", 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6739999999999996e-06 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028260+0000", 2024-09-02T10:08:02.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022132900000000001 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028263+0000", 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.163e-06 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405770+0000", 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.377506856 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405877+0000", 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010699799999999999 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405954+0000", 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7732999999999996e-05 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411501+0000", 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005546212 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.425267+0000", 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013766327 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435398+0000", 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010131506 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435457+0000", 2024-09-02T10:08:02.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8248000000000002e-05 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435772+0000", 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031532400000000002 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163342 2.d 2:b4bdf525:::benchmark_data_smithi187_45882_object65591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.028055+0000", 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 185.30578029500001, 2024-09-02T10:08:02.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40808169100000002, 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163342 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028055+0000", 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028055+0000", 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028056+0000", 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7999999999999996e-07 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028059+0000", 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1109999999999999e-06 2024-09-02T10:08:02.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028065+0000", 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6189999999999996e-06 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028071+0000", 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4410000000000001e-06 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028292+0000", 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221406 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028296+0000", 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3840000000000001e-06 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.408774+0000", 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38047832599999998 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.408848+0000", 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4216999999999998e-05 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.408915+0000", 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7244999999999997e-05 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416361+0000", 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074451420000000001 2024-09-02T10:08:02.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.425326+0000", 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089654829999999998 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435842+0000", 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010515531999999999 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435934+0000", 2024-09-02T10:08:02.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:02.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2621000000000006e-05 2024-09-02T10:08:02.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.436137+0000", 2024-09-02T10:08:02.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020268400000000001 2024-09-02T10:08:02.477 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.477 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.477 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(2.d e219,198 2:b2e165ef:::benchmark_data_smithi187_45882_object46440:head,171'8780)", 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:45.488577+0000", 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 136.84525852499999, 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57263216900000002, 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488577+0000", 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488577+0000", 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488578+0000", 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000004e-07 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488584+0000", 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2720000000000003e-06 2024-09-02T10:08:02.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.060736+0000", 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57215209199999995 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.060751+0000", 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5387e-05 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.060841+0000", 2024-09-02T10:08:02.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9415999999999999e-05 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.061209+0000", 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036841100000000002 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(2.d e219,198 2:b1206fd5:::benchmark_data_smithi187_45882_object46449:head,171'8781)", 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:45.492423+0000", 2024-09-02T10:08:02.481 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 136.84141233, 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57615407699999999, 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492423+0000", 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492423+0000", 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492430+0000", 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6959999999999999e-06 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492440+0000", 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9299999999999998e-06 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068235+0000", 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57579494799999997 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068243+0000", 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4330000000000004e-06 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068306+0000", 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2854000000000005e-05 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068577+0000", 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027121600000000001 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:97423 3.b e260/226 3:da6c5fce:::benchmark_data_smithi187_59989_object97422:head v 260'6024, mlcod=260'6020)", 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:20.582891+0000", 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 41.750944852000003, 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26149520100000001, 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.485 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.582891+0000", 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.582891+0000", 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.582892+0000", 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.392e-06 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.582941+0000", 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8860000000000003e-05 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.841651+0000", 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.258709723 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.841654+0000", 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3579999999999999e-06 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.841703+0000", 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9141999999999998e-05 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.841725+0000", 2024-09-02T10:08:02.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1348e-05 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.844344+0000", 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026198749999999998 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.844386+0000", 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1502999999999999e-05 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "replica_scrub(pg: 3.6,from:0'0,to:263'6750,epoch:263/226,start:3:6f96281c:::benchmark_data_smithi187_59989_object88170:0,end:3:6fcee8f7:::benchmark_data_smithi187_59989_object42531:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:33.294203+0000", 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.039632640000001, 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.75891445099999999, 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:33.294203+0000", 2024-09-02T10:08:02.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:33.294203+0000", 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:33.294203+0000", 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6499999999999999e-07 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:02.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:33.294209+0000", 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7490000000000004e-06 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:34.052988+0000", 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.75877908199999999 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:34.052998+0000", 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3689999999999996e-06 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:02.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:34.053074+0000", 2024-09-02T10:08:02.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6249999999999997e-05 2024-09-02T10:08:02.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:02.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:34.053089+0000", 2024-09-02T10:08:02.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4943000000000001e-05 2024-09-02T10:08:02.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:02.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:02.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:02.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:34.053117+0000", 2024-09-02T10:08:02.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8493e-05 2024-09-02T10:08:02.494 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.495 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:02.495 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:02.495 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:08:02.522 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:02.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:02.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:02.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:02.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:02.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:02.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:02.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:02.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:02.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:02.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:02.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:02.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: pgmap v665: 47 pgs: 47 active+clean; 5.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 910 op/s 2024-09-02T10:08:02.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:08:02.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:08:02.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:08:02.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:08:02.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:08:02.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:08:02.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:08:02.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:08:02.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:08:02.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:02 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:08:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: pgmap v665: 47 pgs: 47 active+clean; 5.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 910 op/s 2024-09-02T10:08:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:08:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:08:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:08:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:08:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:08:03.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:08:03.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:08:03.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:08:03.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:08:03.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:02 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:08:03.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:03.032 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:03.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: pgmap v665: 47 pgs: 47 active+clean; 5.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 910 op/s 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:08:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:02 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:08:03.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:03.167 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:03.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:03.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:03.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:03.299 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:03.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:03.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:03.322 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:08:03.323 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:08:03.401 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:03.425 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:03.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:03.426 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:03.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:08:03.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:03.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:03.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:03.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:03.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:03.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:03.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:03.683 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:08:03.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:03.795 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:03.805 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:03.806 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:03.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:03.932 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:03.933 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:03.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:03 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:08:03.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:03 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:08:03.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:03 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:08:03.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:03 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:08:03.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:03 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:08:03.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:03 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:08:03.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3877549148' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:03.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:08:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:03 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:08:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:03 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:08:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:03 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:08:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:03 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:08:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:03 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:08:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:03 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:08:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3877549148' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:04.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:04.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:04.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:04.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:03 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:08:04.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:03 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:08:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:03 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:08:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:03 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:08:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:03 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:08:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:03 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:08:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3877549148' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:04.168 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:04.203 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:04.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:04.304 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:04.331 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:04.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:04.332 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:04.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:04.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:04.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:04.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:04.600 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:04.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:04.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:04.735 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:04.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:04.774 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":666,"stamp":"2024-09-02T10:08:03.159943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5525799480,"num_objects":84311,"num_object_clones":0,"num_object_copies":252933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84311,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134773,"num_write_kb":7010561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137625,"ondisk_log_size":137625,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":17992452,"kb_used_data":16237028,"kb_used_omap":17,"kb_used_meta":1754670,"kb_avail":1013204220,"statfs":{"total":1055945392128,"available":1037521121280,"internally_reserved":0,"allocated":16626716672,"data_stored":16618176712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1796782971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,1,2,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-531562496,"num_objects":-8111,"num_object_clones":0,"num_object_copies":-24333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8111,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.500939"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":89,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:00.309394+0000","last_change":"2024-09-02T10:08:00.309394+0000","last_active":"2024-09-02T10:08:00.309394+0000","last_peered":"2024-09-02T10:08:00.309394+0000","last_clean":"2024-09-02T10:08:00.309394+0000","last_became_active":"2024-09-02T10:06:34.853129+0000","last_became_peered":"2024-09-02T10:06:34.853129+0000","last_unstale":"2024-09-02T10:08:00.309394+0000","last_undegraded":"2024-09-02T10:08:00.309394+0000","last_fullsized":"2024-09-02T10:08:00.309394+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:00.309314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:00.309314+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:12.355302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":83,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.888392+0000","last_change":"2024-09-02T10:06:36.539700+0000","last_active":"2024-09-02T10:07:59.888392+0000","last_peered":"2024-09-02T10:07:59.888392+0000","last_clean":"2024-09-02T10:07:59.888392+0000","last_became_active":"2024-09-02T10:06:34.860248+0000","last_became_peered":"2024-09-02T10:06:34.860248+0000","last_unstale":"2024-09-02T10:07:59.888392+0000","last_undegraded":"2024-09-02T10:07:59.888392+0000","last_fullsized":"2024-09-02T10:07:59.888392+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:36.539650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:36.539650+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:03.335546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":87,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.237326+0000","last_change":"2024-09-02T10:07:59.237326+0000","last_active":"2024-09-02T10:07:59.237326+0000","last_peered":"2024-09-02T10:07:59.237326+0000","last_clean":"2024-09-02T10:07:59.237326+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:07:59.237326+0000","last_undegraded":"2024-09-02T10:07:59.237326+0000","last_fullsized":"2024-09-02T10:07:59.237326+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:59.237277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:59.237277+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:23.463935+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":79,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-02T10:07:58.690344+0000","last_change":"2024-09-02T10:06:36.130155+0000","last_active":"2024-09-02T10:07:58.690344+0000","last_peered":"2024-09-02T10:07:58.690344+0000","last_clean":"2024-09-02T10:07:58.690344+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:07:58.690344+0000","last_undegraded":"2024-09-02T10:07:58.690344+0000","last_fullsized":"2024-09-02T10:07:58.690344+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:36.130088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:36.130088+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:07:59.742254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":81,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-02T10:07:58.706291+0000","last_change":"2024-09-02T10:06:35.749426+0000","last_active":"2024-09-02T10:07:58.706291+0000","last_peered":"2024-09-02T10:07:58.706291+0000","last_clean":"2024-09-02T10:07:58.706291+0000","last_became_active":"2024-09-02T10:06:34.852952+0000","last_became_peered":"2024-09-02T10:06:34.852952+0000","last_unstale":"2024-09-02T10:07:58.706291+0000","last_undegraded":"2024-09-02T10:07:58.706291+0000","last_fullsized":"2024-09-02T10:07:58.706291+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.749356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.749356+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":89,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.882603+0000","last_change":"2024-09-02T10:07:39.188463+0000","last_active":"2024-09-02T10:07:59.882603+0000","last_peered":"2024-09-02T10:07:59.882603+0000","last_clean":"2024-09-02T10:07:59.882603+0000","last_became_active":"2024-09-02T10:06:34.851298+0000","last_became_peered":"2024-09-02T10:06:34.851298+0000","last_unstale":"2024-09-02T10:07:59.882603+0000","last_undegraded":"2024-09-02T10:07:59.882603+0000","last_fullsized":"2024-09-02T10:07:59.882603+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:39.188419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:39.188419+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:48.116976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":448,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:01.621452+0000","last_change":"2024-09-02T10:08:01.621452+0000","last_active":"2024-09-02T10:08:01.621452+0000","last_peered":"2024-09-02T10:08:01.621452+0000","last_clean":"2024-09-02T10:08:01.621452+0000","last_became_active":"2024-09-02T10:06:34.859627+0000","last_became_peered":"2024-09-02T10:06:34.859627+0000","last_unstale":"2024-09-02T10:08:01.621452+0000","last_undegraded":"2024-09-02T10:08:01.621452+0000","last_fullsized":"2024-09-02T10:08:01.621452+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:01.621368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:01.621368+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:10.066322+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":79,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-02T10:07:58.690352+0000","last_change":"2024-09-02T10:06:35.154338+0000","last_active":"2024-09-02T10:07:58.690352+0000","last_peered":"2024-09-02T10:07:58.690352+0000","last_clean":"2024-09-02T10:07:58.690352+0000","last_became_active":"2024-09-02T10:06:34.849680+0000","last_became_peered":"2024-09-02T10:06:34.849680+0000","last_unstale":"2024-09-02T10:07:58.690352+0000","last_undegraded":"2024-09-02T10:07:58.690352+0000","last_fullsized":"2024-09-02T10:07:58.690352+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:35.154262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:35.154262+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:02.210417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":89,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:02.403736+0000","last_change":"2024-09-02T10:08:02.403736+0000","last_active":"2024-09-02T10:08:02.403736+0000","last_peered":"2024-09-02T10:08:02.403736+0000","last_clean":"2024-09-02T10:08:02.403736+0000","last_became_active":"2024-09-02T10:06:34.850895+0000","last_became_peered":"2024-09-02T10:06:34.850895+0000","last_unstale":"2024-09-02T10:08:02.403736+0000","last_undegraded":"2024-09-02T10:08:02.403736+0000","last_fullsized":"2024-09-02T10:08:02.403736+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:02.403664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:02.403664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:13.841777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":81,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.884533+0000","last_change":"2024-09-02T10:06:34.985266+0000","last_active":"2024-09-02T10:07:59.884533+0000","last_peered":"2024-09-02T10:07:59.884533+0000","last_clean":"2024-09-02T10:07:59.884533+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:07:59.884533+0000","last_undegraded":"2024-09-02T10:07:59.884533+0000","last_fullsized":"2024-09-02T10:07:59.884533+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.985198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.985198+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":336,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.917090+0000","last_change":"2024-09-02T10:07:28.918922+0000","last_active":"2024-09-02T10:07:59.917090+0000","last_peered":"2024-09-02T10:07:59.917090+0000","last_clean":"2024-09-02T10:07:59.917090+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:07:59.917090+0000","last_undegraded":"2024-09-02T10:07:59.917090+0000","last_fullsized":"2024-09-02T10:07:59.917090+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:28.918851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:28.918851+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:40.479112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":422,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:01.381603+0000","last_change":"2024-09-02T10:08:01.381603+0000","last_active":"2024-09-02T10:08:01.381603+0000","last_peered":"2024-09-02T10:08:01.381603+0000","last_clean":"2024-09-02T10:08:01.381603+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:08:01.381603+0000","last_undegraded":"2024-09-02T10:08:01.381603+0000","last_fullsized":"2024-09-02T10:08:01.381603+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:01.381551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:01.381551+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:15.888878+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":358,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.886609+0000","last_change":"2024-09-02T10:07:31.059528+0000","last_active":"2024-09-02T10:07:59.886609+0000","last_peered":"2024-09-02T10:07:59.886609+0000","last_clean":"2024-09-02T10:07:59.886609+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:07:59.886609+0000","last_undegraded":"2024-09-02T10:07:59.886609+0000","last_fullsized":"2024-09-02T10:07:59.886609+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:31.059464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:31.059464+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:39.131834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":346,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:01.316267+0000","last_change":"2024-09-02T10:08:01.316267+0000","last_active":"2024-09-02T10:08:01.316267+0000","last_peered":"2024-09-02T10:08:01.316267+0000","last_clean":"2024-09-02T10:08:01.316267+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:08:01.316267+0000","last_undegraded":"2024-09-02T10:08:01.316267+0000","last_fullsized":"2024-09-02T10:08:01.316267+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:01.316149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:01.316149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:16.729683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":351,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.886443+0000","last_change":"2024-09-02T10:07:32.127860+0000","last_active":"2024-09-02T10:07:59.886443+0000","last_peered":"2024-09-02T10:07:59.886443+0000","last_clean":"2024-09-02T10:07:59.886443+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:07:59.886443+0000","last_undegraded":"2024-09-02T10:07:59.886443+0000","last_fullsized":"2024-09-02T10:07:59.886443+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:32.127803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:32.127803+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:48.053291+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"274'8503","reported_seq":8683,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:02.024369+0000","last_change":"2024-09-02T10:07:30.761771+0000","last_active":"2024-09-02T10:08:02.024369+0000","last_peered":"2024-09-02T10:08:02.024369+0000","last_clean":"2024-09-02T10:08:02.024369+0000","last_became_active":"2024-09-02T10:05:56.583663+0000","last_became_peered":"2024-09-02T10:05:56.583663+0000","last_unstale":"2024-09-02T10:08:02.024369+0000","last_undegraded":"2024-09-02T10:08:02.024369+0000","last_fullsized":"2024-09-02T10:08:02.024369+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6702","last_scrub_stamp":"2024-09-02T10:07:30.761550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.761550+0000","objects_scrubbed":6632,"log_size":8503,"log_dups_size":0,"ondisk_log_size":8503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:51.824511+0000","scrub_duration":1999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339542016,"num_objects":5181,"num_object_clones":0,"num_object_copies":15543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8503,"num_write_kb":437888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":437,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.886521+0000","last_change":"2024-09-02T10:07:59.408627+0000","last_active":"2024-09-02T10:07:59.886521+0000","last_peered":"2024-09-02T10:07:59.886521+0000","last_clean":"2024-09-02T10:07:59.886521+0000","last_became_active":"2024-09-02T10:06:32.873327+0000","last_became_peered":"2024-09-02T10:06:32.873327+0000","last_unstale":"2024-09-02T10:07:59.886521+0000","last_undegraded":"2024-09-02T10:07:59.886521+0000","last_fullsized":"2024-09-02T10:07:59.886521+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:59.408564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:59.408564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:15.132163+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"272'8248","reported_seq":8425,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-02T10:07:58.296406+0000","last_change":"2024-09-02T10:07:29.219779+0000","last_active":"2024-09-02T10:07:58.296406+0000","last_peered":"2024-09-02T10:07:58.296406+0000","last_clean":"2024-09-02T10:07:58.296406+0000","last_became_active":"2024-09-02T10:05:56.583278+0000","last_became_peered":"2024-09-02T10:05:56.583278+0000","last_unstale":"2024-09-02T10:07:58.296406+0000","last_undegraded":"2024-09-02T10:07:58.296406+0000","last_fullsized":"2024-09-02T10:07:58.296406+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6573","last_scrub_stamp":"2024-09-02T10:07:29.219665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:29.219665+0000","objects_scrubbed":6515,"log_size":8248,"log_dups_size":0,"ondisk_log_size":8248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:37.367524+0000","scrub_duration":1777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351797248,"num_objects":5368,"num_object_clones":0,"num_object_copies":16104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8248,"num_write_kb":435712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":553,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.883518+0000","last_change":"2024-09-02T10:06:32.998782+0000","last_active":"2024-09-02T10:07:59.883518+0000","last_peered":"2024-09-02T10:07:59.883518+0000","last_clean":"2024-09-02T10:07:59.883518+0000","last_became_active":"2024-09-02T10:06:32.871027+0000","last_became_peered":"2024-09-02T10:06:32.871027+0000","last_unstale":"2024-09-02T10:07:59.883518+0000","last_undegraded":"2024-09-02T10:07:59.883518+0000","last_fullsized":"2024-09-02T10:07:59.883518+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:32.998705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:32.998705+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:01.309334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"274'8427","reported_seq":8594,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:01.970374+0000","last_change":"2024-09-02T10:07:07.774469+0000","last_active":"2024-09-02T10:08:01.970374+0000","last_peered":"2024-09-02T10:08:01.970374+0000","last_clean":"2024-09-02T10:08:01.970374+0000","last_became_active":"2024-09-02T10:05:56.583249+0000","last_became_peered":"2024-09-02T10:05:56.583249+0000","last_unstale":"2024-09-02T10:08:01.970374+0000","last_undegraded":"2024-09-02T10:08:01.970374+0000","last_fullsized":"2024-09-02T10:08:01.970374+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5253","last_scrub_stamp":"2024-09-02T10:07:07.774339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:07.774339+0000","objects_scrubbed":5197,"log_size":8427,"log_dups_size":0,"ondisk_log_size":8427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:26.717286+0000","scrub_duration":1300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341377024,"num_objects":5209,"num_object_clones":0,"num_object_copies":15627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8427,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":187,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.886093+0000","last_change":"2024-09-02T10:06:33.026327+0000","last_active":"2024-09-02T10:07:59.886093+0000","last_peered":"2024-09-02T10:07:59.886093+0000","last_clean":"2024-09-02T10:07:59.886093+0000","last_became_active":"2024-09-02T10:06:32.870387+0000","last_became_peered":"2024-09-02T10:06:32.870387+0000","last_unstale":"2024-09-02T10:07:59.886093+0000","last_undegraded":"2024-09-02T10:07:59.886093+0000","last_fullsized":"2024-09-02T10:07:59.886093+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.026282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.026282+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:01.622377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"273'8352","reported_seq":8534,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.248979+0000","last_change":"2024-09-02T10:07:36.281686+0000","last_active":"2024-09-02T10:07:59.248979+0000","last_peered":"2024-09-02T10:07:59.248979+0000","last_clean":"2024-09-02T10:07:59.248979+0000","last_became_active":"2024-09-02T10:05:56.584313+0000","last_became_peered":"2024-09-02T10:05:56.584313+0000","last_unstale":"2024-09-02T10:07:59.248979+0000","last_undegraded":"2024-09-02T10:07:59.248979+0000","last_fullsized":"2024-09-02T10:07:59.248979+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6942","last_scrub_stamp":"2024-09-02T10:07:36.281628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:36.281628+0000","objects_scrubbed":6782,"log_size":8352,"log_dups_size":0,"ondisk_log_size":8352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:02.324822+0000","scrub_duration":1988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349306880,"num_objects":5330,"num_object_clones":0,"num_object_copies":15990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8352,"num_write_kb":437824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":342,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.886574+0000","last_change":"2024-09-02T10:06:33.317499+0000","last_active":"2024-09-02T10:07:59.886574+0000","last_peered":"2024-09-02T10:07:59.886574+0000","last_clean":"2024-09-02T10:07:59.886574+0000","last_became_active":"2024-09-02T10:06:32.874387+0000","last_became_peered":"2024-09-02T10:06:32.874387+0000","last_unstale":"2024-09-02T10:07:59.886574+0000","last_undegraded":"2024-09-02T10:07:59.886574+0000","last_fullsized":"2024-09-02T10:07:59.886574+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.317426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.317426+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:02.728906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.6","version":"274'8463","reported_seq":8645,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:01.425166+0000","last_change":"2024-09-02T10:07:34.096922+0000","last_active":"2024-09-02T10:08:01.425166+0000","last_peered":"2024-09-02T10:08:01.425166+0000","last_clean":"2024-09-02T10:08:01.425166+0000","last_became_active":"2024-09-02T10:05:56.578973+0000","last_became_peered":"2024-09-02T10:05:56.578973+0000","last_unstale":"2024-09-02T10:08:01.425166+0000","last_undegraded":"2024-09-02T10:08:01.425166+0000","last_fullsized":"2024-09-02T10:08:01.425166+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6853","last_scrub_stamp":"2024-09-02T10:07:34.096729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:34.096729+0000","objects_scrubbed":6787,"log_size":8463,"log_dups_size":0,"ondisk_log_size":8463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:48.699432+0000","scrub_duration":2896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343080960,"num_objects":5235,"num_object_clones":0,"num_object_copies":15705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8463,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":492,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.886602+0000","last_change":"2024-09-02T10:07:07.536258+0000","last_active":"2024-09-02T10:07:59.886602+0000","last_peered":"2024-09-02T10:07:59.886602+0000","last_clean":"2024-09-02T10:07:59.886602+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:07:59.886602+0000","last_undegraded":"2024-09-02T10:07:59.886602+0000","last_fullsized":"2024-09-02T10:07:59.886602+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:07.536194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:07.536194+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:27.817134+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"272'8205","reported_seq":8382,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-02T10:07:58.328158+0000","last_change":"2024-09-02T10:07:30.887849+0000","last_active":"2024-09-02T10:07:58.328158+0000","last_peered":"2024-09-02T10:07:58.328158+0000","last_clean":"2024-09-02T10:07:58.328158+0000","last_became_active":"2024-09-02T10:05:56.584888+0000","last_became_peered":"2024-09-02T10:05:56.584888+0000","last_unstale":"2024-09-02T10:07:58.328158+0000","last_undegraded":"2024-09-02T10:07:58.328158+0000","last_fullsized":"2024-09-02T10:07:58.328158+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6680","last_scrub_stamp":"2024-09-02T10:07:30.887770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.887770+0000","objects_scrubbed":6647,"log_size":8205,"log_dups_size":0,"ondisk_log_size":8205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:34.316233+0000","scrub_duration":1437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353435648,"num_objects":5393,"num_object_clones":0,"num_object_copies":16179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8205,"num_write_kb":435136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":336,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.926129+0000","last_change":"2024-09-02T10:06:32.983641+0000","last_active":"2024-09-02T10:07:59.926129+0000","last_peered":"2024-09-02T10:07:59.926129+0000","last_clean":"2024-09-02T10:07:59.926129+0000","last_became_active":"2024-09-02T10:06:32.870858+0000","last_became_peered":"2024-09-02T10:06:32.870858+0000","last_unstale":"2024-09-02T10:07:59.926129+0000","last_undegraded":"2024-09-02T10:07:59.926129+0000","last_fullsized":"2024-09-02T10:07:59.926129+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:32.983547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:32.983547+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:00.900960+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,11],"acting":[2,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"274'8817","reported_seq":9002,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:03.099040+0000","last_change":"2024-09-02T10:07:34.254248+0000","last_active":"2024-09-02T10:08:03.099040+0000","last_peered":"2024-09-02T10:08:03.099040+0000","last_clean":"2024-09-02T10:08:03.099040+0000","last_became_active":"2024-09-02T10:05:56.580910+0000","last_became_peered":"2024-09-02T10:05:56.580910+0000","last_unstale":"2024-09-02T10:08:03.099040+0000","last_undegraded":"2024-09-02T10:08:03.099040+0000","last_fullsized":"2024-09-02T10:08:03.099040+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'7048","last_scrub_stamp":"2024-09-02T10:07:34.254184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:34.254184+0000","objects_scrubbed":7013,"log_size":8817,"log_dups_size":0,"ondisk_log_size":8817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:54.308941+0000","scrub_duration":1842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343212032,"num_objects":5237,"num_object_clones":0,"num_object_copies":15711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8817,"num_write_kb":449728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":187,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.886178+0000","last_change":"2024-09-02T10:06:34.057029+0000","last_active":"2024-09-02T10:07:59.886178+0000","last_peered":"2024-09-02T10:07:59.886178+0000","last_clean":"2024-09-02T10:07:59.886178+0000","last_became_active":"2024-09-02T10:06:32.871735+0000","last_became_peered":"2024-09-02T10:06:32.871735+0000","last_unstale":"2024-09-02T10:07:59.886178+0000","last_undegraded":"2024-09-02T10:07:59.886178+0000","last_fullsized":"2024-09-02T10:07:59.886178+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.056985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.056985+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:03.996035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"274'8414","reported_seq":8595,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:01.861783+0000","last_change":"2024-09-02T10:07:30.154080+0000","last_active":"2024-09-02T10:08:01.861783+0000","last_peered":"2024-09-02T10:08:01.861783+0000","last_clean":"2024-09-02T10:08:01.861783+0000","last_became_active":"2024-09-02T10:05:56.690229+0000","last_became_peered":"2024-09-02T10:05:56.690229+0000","last_unstale":"2024-09-02T10:08:01.861783+0000","last_undegraded":"2024-09-02T10:08:01.861783+0000","last_fullsized":"2024-09-02T10:08:01.861783+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6652","last_scrub_stamp":"2024-09-02T10:07:30.153975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.153975+0000","objects_scrubbed":6605,"log_size":8414,"log_dups_size":0,"ondisk_log_size":8414,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:59.384309+0000","scrub_duration":2047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341311488,"num_objects":5208,"num_object_clones":0,"num_object_copies":15624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8414,"num_write_kb":435904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":459,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.887907+0000","last_change":"2024-09-02T10:06:33.444679+0000","last_active":"2024-09-02T10:07:59.887907+0000","last_peered":"2024-09-02T10:07:59.887907+0000","last_clean":"2024-09-02T10:07:59.887907+0000","last_became_active":"2024-09-02T10:06:32.874081+0000","last_became_peered":"2024-09-02T10:06:32.874081+0000","last_unstale":"2024-09-02T10:07:59.887907+0000","last_undegraded":"2024-09-02T10:07:59.887907+0000","last_fullsized":"2024-09-02T10:07:59.887907+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.444635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.444635+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:00.792892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"273'8452","reported_seq":8633,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.205886+0000","last_change":"2024-09-02T10:07:35.873554+0000","last_active":"2024-09-02T10:07:59.205886+0000","last_peered":"2024-09-02T10:07:59.205886+0000","last_clean":"2024-09-02T10:07:59.205886+0000","last_became_active":"2024-09-02T10:05:56.581969+0000","last_became_peered":"2024-09-02T10:05:56.581969+0000","last_unstale":"2024-09-02T10:07:59.205886+0000","last_undegraded":"2024-09-02T10:07:59.205886+0000","last_fullsized":"2024-09-02T10:07:59.205886+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'7026","last_scrub_stamp":"2024-09-02T10:07:35.873463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:35.873463+0000","objects_scrubbed":6888,"log_size":8452,"log_dups_size":0,"ondisk_log_size":8452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:03.009740+0000","scrub_duration":1412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355074048,"num_objects":5418,"num_object_clones":0,"num_object_copies":16254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8452,"num_write_kb":443840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":181,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.886048+0000","last_change":"2024-09-02T10:07:29.964663+0000","last_active":"2024-09-02T10:07:59.886048+0000","last_peered":"2024-09-02T10:07:59.886048+0000","last_clean":"2024-09-02T10:07:59.886048+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:07:59.886048+0000","last_undegraded":"2024-09-02T10:07:59.886048+0000","last_fullsized":"2024-09-02T10:07:59.886048+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:29.964607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:29.964607+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:30.312469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"274'8323","reported_seq":8505,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:01.934182+0000","last_change":"2024-09-02T10:07:35.803712+0000","last_active":"2024-09-02T10:08:01.934182+0000","last_peered":"2024-09-02T10:08:01.934182+0000","last_clean":"2024-09-02T10:08:01.934182+0000","last_became_active":"2024-09-02T10:05:56.876397+0000","last_became_peered":"2024-09-02T10:05:56.876397+0000","last_unstale":"2024-09-02T10:08:01.934182+0000","last_undegraded":"2024-09-02T10:08:01.934182+0000","last_fullsized":"2024-09-02T10:08:01.934182+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6802","last_scrub_stamp":"2024-09-02T10:07:35.803575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:35.803575+0000","objects_scrubbed":6687,"log_size":8323,"log_dups_size":0,"ondisk_log_size":8323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:04.747241+0000","scrub_duration":1595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335478784,"num_objects":5119,"num_object_clones":0,"num_object_copies":15357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8323,"num_write_kb":430144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":195,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.883605+0000","last_change":"2024-09-02T10:07:33.808790+0000","last_active":"2024-09-02T10:07:59.883605+0000","last_peered":"2024-09-02T10:07:59.883605+0000","last_clean":"2024-09-02T10:07:59.883605+0000","last_became_active":"2024-09-02T10:04:47.929173+0000","last_became_peered":"2024-09-02T10:04:47.929173+0000","last_unstale":"2024-09-02T10:07:59.883605+0000","last_undegraded":"2024-09-02T10:07:59.883605+0000","last_fullsized":"2024-09-02T10:07:59.883605+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:33.808730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:33.808730+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:57.847747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"274'8364","reported_seq":8529,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:01.535932+0000","last_change":"2024-09-02T10:07:05.712175+0000","last_active":"2024-09-02T10:08:01.535932+0000","last_peered":"2024-09-02T10:08:01.535932+0000","last_clean":"2024-09-02T10:08:01.535932+0000","last_became_active":"2024-09-02T10:05:56.581229+0000","last_became_peered":"2024-09-02T10:05:56.581229+0000","last_unstale":"2024-09-02T10:08:01.535932+0000","last_undegraded":"2024-09-02T10:08:01.535932+0000","last_fullsized":"2024-09-02T10:08:01.535932+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5000","last_scrub_stamp":"2024-09-02T10:07:05.712114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:05.712114+0000","objects_scrubbed":4957,"log_size":8364,"log_dups_size":0,"ondisk_log_size":8364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:31.644516+0000","scrub_duration":1394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343539712,"num_objects":5242,"num_object_clones":0,"num_object_copies":15726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8364,"num_write_kb":435392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,11],"acting":[0,4,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":409,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.888342+0000","last_change":"2024-09-02T10:07:37.379407+0000","last_active":"2024-09-02T10:07:59.888342+0000","last_peered":"2024-09-02T10:07:59.888342+0000","last_clean":"2024-09-02T10:07:59.888342+0000","last_became_active":"2024-09-02T10:02:40.781655+0000","last_became_peered":"2024-09-02T10:02:40.781655+0000","last_unstale":"2024-09-02T10:07:59.888342+0000","last_undegraded":"2024-09-02T10:07:59.888342+0000","last_fullsized":"2024-09-02T10:07:59.888342+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:37.379359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:37.379359+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:49.767013+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"272'8338","reported_seq":8515,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-02T10:07:58.374589+0000","last_change":"2024-09-02T10:07:30.485290+0000","last_active":"2024-09-02T10:07:58.374589+0000","last_peered":"2024-09-02T10:07:58.374589+0000","last_clean":"2024-09-02T10:07:58.374589+0000","last_became_active":"2024-09-02T10:05:56.876573+0000","last_became_peered":"2024-09-02T10:05:56.876573+0000","last_unstale":"2024-09-02T10:07:58.374589+0000","last_undegraded":"2024-09-02T10:07:58.374589+0000","last_fullsized":"2024-09-02T10:07:58.374589+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6788","last_scrub_stamp":"2024-09-02T10:07:30.485127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.485127+0000","objects_scrubbed":6730,"log_size":8338,"log_dups_size":0,"ondisk_log_size":8338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:31.184390+0000","scrub_duration":2030,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363986944,"num_objects":5554,"num_object_clones":0,"num_object_copies":16662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8338,"num_write_kb":444544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":206,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.883396+0000","last_change":"2024-09-02T10:07:28.824347+0000","last_active":"2024-09-02T10:07:59.883396+0000","last_peered":"2024-09-02T10:07:59.883396+0000","last_clean":"2024-09-02T10:07:59.883396+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:07:59.883396+0000","last_undegraded":"2024-09-02T10:07:59.883396+0000","last_fullsized":"2024-09-02T10:07:59.883396+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:28.824291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:28.824291+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:45.861891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"274'8593","reported_seq":8759,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:00.879233+0000","last_change":"2024-09-02T10:07:04.549316+0000","last_active":"2024-09-02T10:08:00.879233+0000","last_peered":"2024-09-02T10:08:00.879233+0000","last_clean":"2024-09-02T10:08:00.879233+0000","last_became_active":"2024-09-02T10:05:56.876540+0000","last_became_peered":"2024-09-02T10:05:56.876540+0000","last_unstale":"2024-09-02T10:08:00.879233+0000","last_undegraded":"2024-09-02T10:08:00.879233+0000","last_fullsized":"2024-09-02T10:08:00.879233+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'4967","last_scrub_stamp":"2024-09-02T10:07:04.549227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:04.549227+0000","objects_scrubbed":4900,"log_size":8593,"log_dups_size":0,"ondisk_log_size":8593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:16.373684+0000","scrub_duration":1666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347734040,"num_objects":5307,"num_object_clones":0,"num_object_copies":15921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5307,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8593,"num_write_kb":444737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":438,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.887813+0000","last_change":"2024-09-02T10:07:27.771848+0000","last_active":"2024-09-02T10:07:59.887813+0000","last_peered":"2024-09-02T10:07:59.887813+0000","last_clean":"2024-09-02T10:07:59.887813+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:07:59.887813+0000","last_undegraded":"2024-09-02T10:07:59.887813+0000","last_fullsized":"2024-09-02T10:07:59.887813+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:27.771782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:27.771782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:42.542912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"274'8486","reported_seq":8666,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:01.856866+0000","last_change":"2024-09-02T10:07:31.723546+0000","last_active":"2024-09-02T10:08:01.856866+0000","last_peered":"2024-09-02T10:08:01.856866+0000","last_clean":"2024-09-02T10:08:01.856866+0000","last_became_active":"2024-09-02T10:05:56.585883+0000","last_became_peered":"2024-09-02T10:05:56.585883+0000","last_unstale":"2024-09-02T10:08:01.856866+0000","last_undegraded":"2024-09-02T10:08:01.856866+0000","last_fullsized":"2024-09-02T10:08:01.856866+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6718","last_scrub_stamp":"2024-09-02T10:07:31.723385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:31.723385+0000","objects_scrubbed":6667,"log_size":8486,"log_dups_size":0,"ondisk_log_size":8486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:59.746574+0000","scrub_duration":1623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336986112,"num_objects":5142,"num_object_clones":0,"num_object_copies":15426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8486,"num_write_kb":436096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":349,"reported_epoch":273,"state":"active+clean","last_fresh":"2024-09-02T10:07:58.709588+0000","last_change":"2024-09-02T10:06:34.728115+0000","last_active":"2024-09-02T10:07:58.709588+0000","last_peered":"2024-09-02T10:07:58.709588+0000","last_clean":"2024-09-02T10:07:58.709588+0000","last_became_active":"2024-09-02T10:06:32.874830+0000","last_became_peered":"2024-09-02T10:06:32.874830+0000","last_unstale":"2024-09-02T10:07:58.709588+0000","last_undegraded":"2024-09-02T10:07:58.709588+0000","last_fullsized":"2024-09-02T10:07:58.709588+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.728059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.728059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:00.290725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"274'8367","reported_seq":8533,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:00.894120+0000","last_change":"2024-09-02T10:07:06.845356+0000","last_active":"2024-09-02T10:08:00.894120+0000","last_peered":"2024-09-02T10:08:00.894120+0000","last_clean":"2024-09-02T10:08:00.894120+0000","last_became_active":"2024-09-02T10:05:56.584450+0000","last_became_peered":"2024-09-02T10:05:56.584450+0000","last_unstale":"2024-09-02T10:08:00.894120+0000","last_undegraded":"2024-09-02T10:08:00.894120+0000","last_fullsized":"2024-09-02T10:08:00.894120+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5143","last_scrub_stamp":"2024-09-02T10:07:06.845283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:06.845283+0000","objects_scrubbed":5114,"log_size":8367,"log_dups_size":0,"ondisk_log_size":8367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:09.749906+0000","scrub_duration":1004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341639168,"num_objects":5213,"num_object_clones":0,"num_object_copies":15639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8367,"num_write_kb":434560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":212,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:07:59.884656+0000","last_change":"2024-09-02T10:06:34.005118+0000","last_active":"2024-09-02T10:07:59.884656+0000","last_peered":"2024-09-02T10:07:59.884656+0000","last_clean":"2024-09-02T10:07:59.884656+0000","last_became_active":"2024-09-02T10:06:32.876151+0000","last_became_peered":"2024-09-02T10:06:32.876151+0000","last_unstale":"2024-09-02T10:07:59.884656+0000","last_undegraded":"2024-09-02T10:07:59.884656+0000","last_fullsized":"2024-09-02T10:07:59.884656+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:34.005073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:34.005073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:02.913922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"274'8421","reported_seq":8586,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-09-02T10:08:01.954037+0000","last_change":"2024-09-02T10:07:06.655925+0000","last_active":"2024-09-02T10:08:01.954037+0000","last_peered":"2024-09-02T10:08:01.954037+0000","last_clean":"2024-09-02T10:08:01.954037+0000","last_became_active":"2024-09-02T10:05:56.585870+0000","last_became_peered":"2024-09-02T10:05:56.585870+0000","last_unstale":"2024-09-02T10:08:01.954037+0000","last_undegraded":"2024-09-02T10:08:01.954037+0000","last_fullsized":"2024-09-02T10:08:01.954037+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5095","last_scrub_stamp":"2024-09-02T10:07:06.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:06.655797+0000","objects_scrubbed":5054,"log_size":8421,"log_dups_size":0,"ondisk_log_size":8421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:08.831810+0000","scrub_duration":1179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337707008,"num_objects":5153,"num_object_clones":0,"num_object_copies":15459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8421,"num_write_kb":434368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,0],"acting":[11,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":221,"reported_epoch":272,"state":"active+clean","last_fresh":"2024-09-02T10:07:55.661128+0000","last_change":"2024-09-02T10:06:33.485010+0000","last_active":"2024-09-02T10:07:55.661128+0000","last_peered":"2024-09-02T10:07:55.661128+0000","last_clean":"2024-09-02T10:07:55.661128+0000","last_became_active":"2024-09-02T10:06:32.875557+0000","last_became_peered":"2024-09-02T10:06:32.875557+0000","last_unstale":"2024-09-02T10:07:55.661128+0000","last_undegraded":"2024-09-02T10:07:55.661128+0000","last_fullsized":"2024-09-02T10:07:55.661128+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:06:33.484964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:06:33.484964+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:01.399206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":5525209112,"num_objects":84309,"num_object_clones":0,"num_object_copies":252927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":134773,"num_write_kb":7010561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16519081984,"data_stored":16519069768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134773,"ondisk_log_size":134773,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383775,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2950632,"kb_used_data":2676320,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":90794520,"statfs":{"total":95995035648,"available":92973588480,"internally_reserved":0,"allocated":2740551680,"data_stored":2739793560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":280689092},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841641,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1152180,"kb_used_data":994660,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92592972,"statfs":{"total":95995035648,"available":94815203328,"internally_reserved":0,"allocated":1018531840,"data_stored":1017769648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":161282496},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070574,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":794844,"kb_used_data":683144,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":92950308,"statfs":{"total":95995035648,"available":95181115392,"internally_reserved":0,"allocated":699539456,"data_stored":698719016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299505,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1518392,"kb_used_data":1360684,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":92226760,"statfs":{"total":95995035648,"available":94440202240,"internally_reserved":0,"allocated":1393340416,"data_stored":1392574672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":161282506},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528438,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1791600,"kb_used_data":1674336,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":91953552,"statfs":{"total":95995035648,"available":94160437248,"internally_reserved":0,"allocated":1714520064,"data_stored":1713761944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":120060359},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724667,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1455420,"kb_used_data":1347884,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92289732,"statfs":{"total":95995035648,"available":94504685568,"internally_reserved":0,"allocated":1380233216,"data_stored":1379467472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953600,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2566860,"kb_used_data":2419068,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":91178292,"statfs":{"total":95995035648,"available":93366571008,"internally_reserved":0,"allocated":2477125632,"data_stored":2476309788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149830,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1941772,"kb_used_data":1735676,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":91803380,"statfs":{"total":95995035648,"available":94006661120,"internally_reserved":0,"allocated":1777332224,"data_stored":1776516380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":211024328},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411466,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":504696,"kb_used_data":334696,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":93240456,"statfs":{"total":95995035648,"available":95478226944,"internally_reserved":0,"allocated":342728704,"data_stored":341967032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574992,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1485744,"kb_used_data":1333792,"kb_used_omap":1,"kb_used_meta":151934,"kb_avail":92259408,"statfs":{"total":95995035648,"available":94473633792,"internally_reserved":0,"allocated":1365803008,"data_stored":1365044888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":155580869},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738517,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1830312,"kb_used_data":1676768,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":91914840,"statfs":{"total":95995035648,"available":94120796160,"internally_reserved":0,"allocated":1717010432,"data_stored":1716252312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1712455680,"data_stored":1712455680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1352859648,"data_stored":1352859648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":338558976,"data_stored":338558976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1773928448,"data_stored":1773928448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2453995520,"data_stored":2453995520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1370689536,"data_stored":1370685464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1705639936,"data_stored":1705639936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1375866880,"data_stored":1375862808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":697171968,"data_stored":697171968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1015943168,"data_stored":1015939096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":2721972224,"data_stored":2721972224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:08:04.777 INFO:tasks.thrashosds.thrasher:Setting 1.f to [10, 5, 0, 6, 9, 4] 2024-09-02T10:08:04.777 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.f', '10', '5', '0', '6', '9', '4'] 2024-09-02T10:08:04.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 1.f 10 5 0 6 9 4 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:138572 3.c 3:3bcd9083:::benchmark_data_smithi187_59989_object29029:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e275)", 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:04.773324+0000", 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.013261017999999999, 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013290577, 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:04.788 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 138572 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.773324+0000", 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.773324+0000", 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.773326+0000", 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.488e-06 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.773331+0000", 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5650000000000001e-06 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.773346+0000", 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4905999999999999e-05 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.773353+0000", 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0029999999999997e-06 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.773467+0000", 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113495 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.773667+0000", 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020069599999999999 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.773847+0000", 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017980399999999999 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.778211+0000", 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043643029999999999 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.779470+0000", 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001258861 2024-09-02T10:08:04.792 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:138587 3.4 e275/226 3:2d7df9f8:::benchmark_data_smithi187_59989_object29044:head v 275'8925, mlcod=275'8923)", 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:04.785237+0000", 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0013481260000000001, 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001453642, 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.785237+0000", 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.785237+0000", 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.785238+0000", 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.09e-07 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.785244+0000", 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8270000000000003e-06 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.785277+0000", 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3299000000000001e-05 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.785284+0000", 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9199999999999998e-06 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.785377+0000", 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3398000000000003e-05 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:04.785425+0000", 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7469000000000002e-05 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-02T10:08:04.796 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:08:04.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:04.868 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:04.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:04.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:04 smithi151 ceph-mon[16639]: pgmap v666: 47 pgs: 47 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-02T10:08:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:04 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:08:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:04 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:08:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:04 smithi151 ceph-mon[16639]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:08:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:04 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:04 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:04 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:08:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:04 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:08:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:04 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3877549148' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:04 smithi151 ceph-mon[16639]: osdmap e275: 12 total, 12 up, 11 in 2024-09-02T10:08:04.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:05.002 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:05.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:05.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:04 smithi187 ceph-mon[20072]: pgmap v666: 47 pgs: 47 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-02T10:08:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:04 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:08:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:04 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:08:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:04 smithi187 ceph-mon[20072]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:08:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:04 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:04 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:04 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:08:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:04 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:08:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:04 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3877549148' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:05.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:04 smithi187 ceph-mon[20072]: osdmap e275: 12 total, 12 up, 11 in 2024-09-02T10:08:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:04 smithi162 ceph-mon[20279]: pgmap v666: 47 pgs: 47 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s 2024-09-02T10:08:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:04 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:08:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:04 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:08:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:04 smithi162 ceph-mon[20279]: from='client.15675 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:08:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:04 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:04 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:04 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:08:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:04 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:08:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:04 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3877549148' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:04 smithi162 ceph-mon[20279]: osdmap e275: 12 total, 12 up, 11 in 2024-09-02T10:08:05.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:05.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:05.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:05.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:05.210 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:05.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:05.267 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:05.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:05.368 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:05.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:05.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:05.393 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:05.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:05.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:05.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:05.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:08:05.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:05.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:05.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:05.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:05.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:05.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:05.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:05.772 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:05.772 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:05.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:08:05.787 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:08:05.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:05 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:08:05.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:05 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:08:05.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:05 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:08:05.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:05 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:08:05.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:05 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:08:05.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:05 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:08:05.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:05 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:08:05.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:05 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:08:05.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:05 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:08:05.787 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:05 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:08:05.819 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:08:05.819 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:08:05.845 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:08:05.846 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:08:05.846 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:05.869 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:08:05.869 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:08:05.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:05.895 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:08:05.895 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:08:05.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:05.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:05.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:05.921 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:08:05.922 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:08:05.976 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:05 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:08:05.976 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:05 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:08:05.976 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:05 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:08:05.976 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:05 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:08:05.976 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:05 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:08:05.976 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:05 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:08:05.976 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:05 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:08:05.977 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:05 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:08:05.977 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:05 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:08:05.977 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:05 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:08:06.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:05 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:08:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:05 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:08:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:05 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:08:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:05 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:08:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:05 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:08:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:05 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:08:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:05 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:08:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:05 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:08:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:05 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:08:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:05 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:08:06.024 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:06.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:06.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:06.154 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:06.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:06.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:06.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:08:06.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:08:06.179 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:08:06.180 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:08:06.206 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:08:06.206 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:08:06.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:08:06.234 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:08:06.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:06.266 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:08:06.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:08:06.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:06.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:06.296 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:08:06.297 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:08:06.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:06.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:06.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:06.490 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:06.490 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:06.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:06.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:06.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:06.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:06.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:06.668 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:06.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:06.692 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.f pg_upmap_items mapping to [10->5,0->6,9->4] 2024-09-02T10:08:06.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:06.794 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:06.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:06.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:06.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:06.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:06.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:06.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:06 smithi151 ceph-mon[16639]: pgmap v668: 47 pgs: 47 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-09-02T10:08:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4020417286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:06 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:08:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:06 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:08:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/453156820' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [10, 5, 0, 6, 9, 4]} : dispatch 2024-09-02T10:08:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:06 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [10, 5, 0, 6, 9, 4]} : dispatch 2024-09-02T10:08:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:06 smithi187 ceph-mon[20072]: pgmap v668: 47 pgs: 47 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-09-02T10:08:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4020417286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:06 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:08:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:06 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:08:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:07.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:07.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/453156820' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [10, 5, 0, 6, 9, 4]} : dispatch 2024-09-02T10:08:07.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:06 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [10, 5, 0, 6, 9, 4]} : dispatch 2024-09-02T10:08:07.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:07.046 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:07.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:07.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:07.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:06 smithi162 ceph-mon[20279]: pgmap v668: 47 pgs: 47 active+clean; 5.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 827 op/s 2024-09-02T10:08:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4020417286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:06 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:08:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:06 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:08:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/453156820' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [10, 5, 0, 6, 9, 4]} : dispatch 2024-09-02T10:08:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:06 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [10, 5, 0, 6, 9, 4]} : dispatch 2024-09-02T10:08:07.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:07.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:07.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:07.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:08:07.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:07.305 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:07.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:07.407 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:07.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:07.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:08:07.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:07.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:07.557 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:07.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:07.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:07.681 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:07.682 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:07.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:07.794 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:07.820 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:07.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:07.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:07.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:07.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:07.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:07 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [10, 5, 0, 6, 9, 4]}]': finished 2024-09-02T10:08:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:07 smithi187 ceph-mon[20072]: osdmap e276: 12 total, 12 up, 11 in 2024-09-02T10:08:08.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:08.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:07 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [10, 5, 0, 6, 9, 4]}]': finished 2024-09-02T10:08:08.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:07 smithi162 ceph-mon[20279]: osdmap e276: 12 total, 12 up, 11 in 2024-09-02T10:08:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:08.078 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:08.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:08.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:08.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:07 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [10, 5, 0, 6, 9, 4]}]': finished 2024-09-02T10:08:08.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:07 smithi151 ceph-mon[16639]: osdmap e276: 12 total, 12 up, 11 in 2024-09-02T10:08:08.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:08.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:08.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:08.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:08.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:08.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:08.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:08.467 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:08.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:08.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:08.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:08.605 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:08.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:08.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:08.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:08.732 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:08.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:08.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:08.800 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:08.800 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:08.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:08.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:08 smithi162 ceph-mon[20279]: pgmap v670: 47 pgs: 47 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-02T10:08:08.861 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:08.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:08 smithi151 ceph-mon[16639]: pgmap v670: 47 pgs: 47 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-02T10:08:08.963 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:08.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:08.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:08.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:08 smithi187 ceph-mon[20072]: pgmap v670: 47 pgs: 47 active+clean; 4.9 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s 2024-09-02T10:08:09.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:09.122 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:09.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:09.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:09.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:09.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:08:09.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:09.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:09.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:09.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:09.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:09.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:09.494 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:09.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:09.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:09.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:08:09.596 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:09.619 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:09.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:09.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:09.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:09.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:09.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:09.891 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:09.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:09.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:09.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:09.949 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:10.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:10.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:10.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:10.126 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:10.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:10.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:10.148 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:09 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:09 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:09 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:08:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:09 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:08:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:09 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:08:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:09 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:08:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:09 smithi151 ceph-mon[16639]: pgmap v671: 47 pgs: 47 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-02T10:08:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:09 smithi151 ceph-mon[16639]: osdmap e277: 12 total, 12 up, 11 in 2024-09-02T10:08:10.191 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:09 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:10.191 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:09 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:10.191 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:09 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:08:10.191 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:09 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:08:10.191 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:09 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:08:10.191 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:09 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:08:10.192 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:09 smithi162 ceph-mon[20279]: pgmap v671: 47 pgs: 47 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-02T10:08:10.192 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:09 smithi162 ceph-mon[20279]: osdmap e277: 12 total, 12 up, 11 in 2024-09-02T10:08:10.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:09 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:09 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:09 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:08:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:09 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:08:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:09 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:08:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:09 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:08:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:09 smithi187 ceph-mon[20072]: pgmap v671: 47 pgs: 47 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s 2024-09-02T10:08:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:09 smithi187 ceph-mon[20072]: osdmap e277: 12 total, 12 up, 11 in 2024-09-02T10:08:10.274 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:10.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:10.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:10.376 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:10.404 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:10.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:10.464 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:10.465 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:10.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:10.537 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:10.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:10.639 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:10.643 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:10.671 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:10.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:10.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:10.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:10.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:10.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:10.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:10.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:11.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:11.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:11.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:11.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:10 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:11.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:10 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:11.106 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:10 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:11.106 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:10 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:11.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:08:11.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:11.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:11.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:11.246 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:08:11.246 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:08:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:10 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:10 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:11.270 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:08:11.270 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:08:11.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:11.301 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:08:11.302 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:08:11.322 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:11.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:11.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:11.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:08:11.330 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:08:11.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:08:11.356 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:08:11.382 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:08:11.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:08:11.411 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:08:11.411 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:08:11.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:11.445 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:08:11.445 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:08:11.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:11.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:11.472 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:08:11.472 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:08:11.501 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:08:11.501 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:08:11.508 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:11.527 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:08:11.527 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:08:11.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:11.556 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:08:11.557 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:08:11.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:11.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:11.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:11.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:11.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:11.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:11.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:11.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:11.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:11.886 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:08:11.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:11.971 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:11.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:12.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:12.081 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:08:12.082 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:08:12.082 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:08:12.082 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:08:12.082 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:08:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:12.109 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:12.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:12.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:11 smithi151 ceph-mon[16639]: osdmap e278: 12 total, 12 up, 11 in 2024-09-02T10:08:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2877153484' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:08:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:11 smithi151 ceph-mon[16639]: pgmap v674: 47 pgs: 47 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-09-02T10:08:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/619743854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:11 smithi151 ceph-mon[16639]: 3.e scrub starts 2024-09-02T10:08:12.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:12.251 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:12.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:11 smithi187 ceph-mon[20072]: osdmap e278: 12 total, 12 up, 11 in 2024-09-02T10:08:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2877153484' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:08:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:11 smithi187 ceph-mon[20072]: pgmap v674: 47 pgs: 47 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-09-02T10:08:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/619743854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:11 smithi187 ceph-mon[20072]: 3.e scrub starts 2024-09-02T10:08:12.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:11 smithi162 ceph-mon[20279]: osdmap e278: 12 total, 12 up, 11 in 2024-09-02T10:08:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2877153484' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:08:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:11 smithi162 ceph-mon[20279]: pgmap v674: 47 pgs: 47 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 875 op/s 2024-09-02T10:08:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/619743854' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:11 smithi162 ceph-mon[20279]: 3.e scrub starts 2024-09-02T10:08:12.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:12.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:12.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:12.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:12.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:12.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:12.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:12.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:12.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:12.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:12.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:12.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:12.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:12.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:12.777 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:08:12.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:12.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:12.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:12.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:12.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:13.005 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:13.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:13.092 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:13.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:12 smithi162 ceph-mon[20279]: 3.f scrub starts 2024-09-02T10:08:13.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:12 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2877153484' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:08:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:12 smithi162 ceph-mon[20279]: osdmap e279: 12 total, 12 up, 11 in 2024-09-02T10:08:13.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:13.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:13.130 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:13.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:12 smithi151 ceph-mon[16639]: 3.f scrub starts 2024-09-02T10:08:13.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:12 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:13.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2877153484' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:08:13.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:12 smithi151 ceph-mon[16639]: osdmap e279: 12 total, 12 up, 11 in 2024-09-02T10:08:13.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:13.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:13.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:13.255 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:12 smithi187 ceph-mon[20072]: 3.f scrub starts 2024-09-02T10:08:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:12 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2877153484' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:08:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:12 smithi187 ceph-mon[20072]: osdmap e279: 12 total, 12 up, 11 in 2024-09-02T10:08:13.356 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:13.379 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:13.380 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:13.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:13.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:13.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:13.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:13.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:13.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:13.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:13.629 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:08:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:08:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:08:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:08:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:1129 2.7 2:eb0d96c7:::benchmark_data_smithi187_45882_object1128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-02T10:08:13.629 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:35.893627+0000", 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 397.732287289, 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20029433599999999, 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1129 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893627+0000", 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893627+0000", 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893628+0000", 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1200000000000001e-07 2024-09-02T10:08:13.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893695+0000", 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7444e-05 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893707+0000", 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2024999999999999e-05 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893713+0000", 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3650000000000003e-06 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893761+0000", 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8760000000000001e-05 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893846+0000", 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4368000000000003e-05 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893930+0000", 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4179999999999997e-05 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.894256+0000", 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032631199999999999 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.908716+0000", 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014459929999999999 2024-09-02T10:08:13.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093866+0000", 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18514956099999999 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093896+0000", 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0238000000000001e-05 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093921+0000", 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5241000000000001e-05 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:1142 2.7 2:e2cb33eb:::benchmark_data_smithi187_45882_object1141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:35.906559+0000", 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 397.71935503399999, 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20112611999999999, 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.636 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1142 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906559+0000", 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906559+0000", 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906560+0000", 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2780000000000001e-06 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906583+0000", 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2288000000000001e-05 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906594+0000", 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1235e-05 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906599+0000", 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.006e-06 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906640+0000", 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0808e-05 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906719+0000", 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9271999999999996e-05 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906808+0000", 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9093999999999998e-05 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907153+0000", 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034475 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093837+0000", 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18668417100000001 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.107626+0000", 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013788761 2024-09-02T10:08:13.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.107659+0000", 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3714000000000003e-05 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.107685+0000", 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5743000000000001e-05 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:1129 2.7 e96/92 ondisk, result = 0)", 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:35.907472+0000", 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 397.71844178399999, 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18645468200000001, 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907472+0000", 2024-09-02T10:08:13.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907472+0000", 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907473+0000", 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8400000000000004e-07 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907476+0000", 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1769999999999998e-06 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093797+0000", 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.186320872 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093803+0000", 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9880000000000001e-06 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093857+0000", 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4190000000000001e-05 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093862+0000", 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5290000000000002e-06 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093927+0000", 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5341999999999999e-05 2024-09-02T10:08:13.646 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:8376 2.2 e97/92 2:42cf9cd7:::benchmark_data_smithi187_45882_object8375:head v 97'548, mlcod=97'545)", 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:40.037608+0000", 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 393.58830638900002, 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18157076599999999, 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.037608+0000", 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.037608+0000", 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.037608+0000", 2024-09-02T10:08:13.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0999999999999998e-07 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.037636+0000", 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7625999999999999e-05 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.218489+0000", 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18085346899999999 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.218495+0000", 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6840000000000001e-06 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.218555+0000", 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9864000000000001e-05 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.218575+0000", 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0304000000000001e-05 2024-09-02T10:08:13.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.219120+0000", 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00054517499999999996 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.219179+0000", 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8034e-05 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.7 137/131 [PushReplyOp(2:e86168c2:::benchmark_data_smithi187_45882_object9180:head)])", 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:05.394929+0000", 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 308.23098532500001, 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.63519735200000005, 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.394929+0000", 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.394929+0000", 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.394929+0000", 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3e-07 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.394935+0000", 2024-09-02T10:08:13.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0690000000000001e-06 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:06.026536+0000", 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.631601419 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:06.026562+0000", 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5559999999999999e-05 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:06.026668+0000", 2024-09-02T10:08:13.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106269 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:06.030126+0000", 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034584049999999999 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:97742 2.2 e137/92 2:492b4420:::benchmark_data_smithi187_45882_object97741:head v 137'6271, mlcod=137'6269)", 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:05.413861+0000", 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 308.21205295700003, 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.263774603, 2024-09-02T10:08:13.655 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413861+0000", 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413861+0000", 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413863+0000", 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4169999999999999e-06 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413889+0000", 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6381e-05 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.675134+0000", 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26124523599999999 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.675143+0000", 2024-09-02T10:08:13.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7790000000000001e-06 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.675219+0000", 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6085000000000001e-05 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.675236+0000", 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6748999999999998e-05 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677595+0000", 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023591269999999999 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677636+0000", 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0828999999999998e-05 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127481 2.7 2:e856f541:::benchmark_data_smithi187_45882_object29730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:08:13.660 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.321173+0000", 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 268.304741095, 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.180644579, 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127481 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.321173+0000", 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.321173+0000", 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.321174+0000", 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.223e-06 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.321178+0000", 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8840000000000004e-06 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.321187+0000", 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1600000000000004e-06 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.321192+0000", 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6709999999999998e-06 2024-09-02T10:08:13.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.325336+0000", 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041440169999999998 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.325446+0000", 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109825 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:08:13.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.325529+0000", 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2965000000000003e-05 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.334720+0000", 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.009190709 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.336407+0000", 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016877680000000001 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.500342+0000", 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.163934464 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.500383+0000", 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0871000000000002e-05 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.501818+0000", 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001435022 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.666 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:127481 2.7 e156/131 ondisk, result = 0)", 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.329201+0000", 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 268.296712947, 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17262190299999999, 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.329201+0000", 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.329201+0000", 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.329202+0000", 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.156e-06 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.329207+0000", 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.296e-06 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.500278+0000", 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17107123199999999 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.500284+0000", 2024-09-02T10:08:13.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9390000000000002e-06 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.500331+0000", 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7009999999999999e-05 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.500337+0000", 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4840000000000001e-06 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.501823+0000", 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001485786 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127503 2.7 2:e04e2dbe:::benchmark_data_smithi187_45882_object29752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.471479+0000", 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 268.15443473099998, 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19658355, 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:13.671 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127503 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471479+0000", 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471479+0000", 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471482+0000", 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.097e-06 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471490+0000", 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1419999999999993e-06 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471504+0000", 2024-09-02T10:08:13.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4005999999999999e-05 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471513+0000", 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0040000000000005e-06 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.472019+0000", 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050612499999999996 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.472135+0000", 2024-09-02T10:08:13.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115798 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.472233+0000", 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8450999999999996e-05 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.475025+0000", 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027922329999999999 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.481083+0000", 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060575769999999998 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.666323+0000", 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18524028300000001 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.666409+0000", 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5414000000000006e-05 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.668063+0000", 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00165442 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127504 2.7 2:e9c5417b:::benchmark_data_smithi187_45882_object29753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.490634+0000", 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 268.13527986399998, 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.187364009, 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.677 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127504 2024-09-02T10:08:13.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.490634+0000", 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.490634+0000", 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.490636+0000", 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.564e-06 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.490644+0000", 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.14e-06 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.490656+0000", 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1582e-05 2024-09-02T10:08:13.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.490663+0000", 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2570000000000001e-06 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.495865+0000", 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0052023629999999998 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.496060+0000", 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019475100000000001 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.496147+0000", 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6569000000000004e-05 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.499002+0000", 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002855528 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.506225+0000", 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072231170000000003 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.676690+0000", 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.170464849 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.676774+0000", 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3955999999999994e-05 2024-09-02T10:08:13.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.677998+0000", 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001224333 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127528 2.7 2:e0932104:::benchmark_data_smithi187_45882_object29777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.657520+0000", 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 267.96839388199999, 2024-09-02T10:08:13.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67433600000000005, 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127528 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657520+0000", 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657520+0000", 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657522+0000", 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3400000000000001e-06 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657527+0000", 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0599999999999998e-06 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657538+0000", 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1555e-05 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657561+0000", 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2456e-05 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.658124+0000", 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056373299999999996 2024-09-02T10:08:13.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.658258+0000", 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133101 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.658396+0000", 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000138901 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.666255+0000", 2024-09-02T10:08:13.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078582219999999998 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.668130+0000", 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00187503 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.330974+0000", 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.66284469899999998 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.331011+0000", 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6532000000000002e-05 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.331856+0000", 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00084537100000000003 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.7 156/131 [PushReplyOp(2:e2e982ef:::benchmark_data_smithi187_45882_object43210:head)])", 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.912154+0000", 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 267.71376001800002, 2024-09-02T10:08:13.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32342126199999999, 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912154+0000", 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912154+0000", 2024-09-02T10:08:13.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912155+0000", 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2810000000000001e-06 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912161+0000", 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.412e-06 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.233179+0000", 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32101774300000002 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.233188+0000", 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4930000000000004e-06 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.233262+0000", 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3827999999999999e-05 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.235575+0000", 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002313505 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:163310 2.7 e198/176 ondisk, result = 0)", 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:56.079360+0000", 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 197.54655404799999, 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.184014859, 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:08:13.695 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:56.079360+0000", 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:56.079360+0000", 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:56.079361+0000", 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9600000000000008e-07 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:56.079371+0000", 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5999999999999996e-06 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:56.263162+0000", 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.183790809 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:56.263175+0000", 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.379e-05 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:56.263279+0000", 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010376499999999999 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:56.263302+0000", 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2566e-05 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:56.263375+0000", 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3332999999999997e-05 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:176710 2.7 2:eb3142a2:::benchmark_data_smithi187_45882_object78959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e208)", 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:16.778140+0000", 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 176.847773762, 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17981702099999999, 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:08:13.699 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 176710 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.778140+0000", 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.778140+0000", 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.778141+0000", 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0599999999999999e-07 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.778145+0000", 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0570000000000003e-06 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.778155+0000", 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3579999999999992e-06 2024-09-02T10:08:13.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.778160+0000", 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8029999999999996e-06 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.778197+0000", 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7224999999999997e-05 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.778272+0000", 2024-09-02T10:08:13.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4850000000000003e-05 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.778338+0000", 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6449e-05 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.781632+0000", 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032942850000000001 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.782029+0000", 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00039655199999999999 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.957868+0000", 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17583935000000001 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.957926+0000", 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8078999999999997e-05 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:16.957957+0000", 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1106999999999999e-05 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:11932 3.a 3:56401dcf:::benchmark_data_smithi187_59989_object11931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:06:10.400201+0000", 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 123.22571299099999, 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17407112799999999, 2024-09-02T10:08:13.705 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 11932 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.400201+0000", 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.400201+0000", 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.400202+0000", 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3200000000000001e-06 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.400261+0000", 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8366999999999999e-05 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.400271+0000", 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0596e-05 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.400279+0000", 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0840000000000003e-06 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.400329+0000", 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0278000000000002e-05 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.400410+0000", 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1285000000000005e-05 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.400521+0000", 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110697 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.401143+0000", 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00062262999999999999 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.401396+0000", 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025285500000000002 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.574160+0000", 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.172763476 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.574222+0000", 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1801e-05 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:10.574272+0000", 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0739000000000003e-05 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:47600 3.a 3:5b7ae563:::benchmark_data_smithi187_59989_object47599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:06:36.595403+0000", 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.030511111999999, 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.191540089, 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.711 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 47600 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.595403+0000", 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.595403+0000", 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.595405+0000", 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5489999999999999e-06 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.595433+0000", 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8904999999999999e-05 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.595446+0000", 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2651e-05 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.595453+0000", 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9840000000000004e-06 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.595559+0000", 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105988 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.595669+0000", 2024-09-02T10:08:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110337 2024-09-02T10:08:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:08:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.595813+0000", 2024-09-02T10:08:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014338099999999999 2024-09-02T10:08:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.597457+0000", 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016437190000000001 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.603105+0000", 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00564818 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.786828+0000", 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18372319100000001 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.786893+0000", 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5434999999999999e-05 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.786943+0000", 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9768999999999997e-05 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.717 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:47611 3.a 3:5baedb0a:::benchmark_data_smithi187_59989_object47610:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:06:36.600709+0000", 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.025204959999996, 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19098784399999999, 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 47611 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.600709+0000", 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.600709+0000", 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.600711+0000", 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4419999999999999e-06 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.600797+0000", 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6063000000000006e-05 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.600811+0000", 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3939e-05 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.600818+0000", 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9859999999999997e-06 2024-09-02T10:08:13.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.600883+0000", 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5797999999999996e-05 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.600963+0000", 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9640000000000003e-05 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.601046+0000", 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2573999999999999e-05 2024-09-02T10:08:13.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.602534+0000", 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001488601 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.606003+0000", 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034691019999999999 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.791576+0000", 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.185572664 2024-09-02T10:08:13.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.791645+0000", 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9151000000000003e-05 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:36.791697+0000", 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1884e-05 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:75424 3.a 3:58480fd1:::benchmark_data_smithi187_59989_object75423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.266315758000005, 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22582144000000001, 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75424 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.724 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359600+0000", 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3820000000000001e-06 2024-09-02T10:08:13.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359629+0000", 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8775000000000001e-05 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359640+0000", 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1151e-05 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359648+0000", 2024-09-02T10:08:13.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0620000000000001e-06 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359688+0000", 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9882000000000002e-05 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359791+0000", 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010327699999999999 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:08:13.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359876+0000", 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5418000000000001e-05 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.361865+0000", 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019887289999999998 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362104+0000", 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023905199999999999 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585329+0000", 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223224432 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585374+0000", 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4972999999999998e-05 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585420+0000", 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6307000000000002e-05 2024-09-02T10:08:13.729 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24976.0:75424 3.a e253/226 ondisk, result = 0)", 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 73.263610217999997, 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22312427800000001, 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362305+0000", 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9100000000000003e-07 2024-09-02T10:08:13.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362309+0000", 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2899999999999996e-06 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585259+0000", 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22295003499999999 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585266+0000", 2024-09-02T10:08:13.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6460000000000004e-06 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585311+0000", 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5802e-05 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585321+0000", 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3090000000000001e-06 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585428+0000", 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010750499999999999 2024-09-02T10:08:13.733 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:145692 3.a 3:5311ad09:::benchmark_data_smithi187_59989_object36149:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:12.713337+0000", 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.912577588, 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17524968699999999, 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:13.734 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145692 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.713337+0000", 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.713337+0000", 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.713338+0000", 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4649999999999999e-06 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.713344+0000", 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1750000000000002e-06 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.713361+0000", 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6727000000000001e-05 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.713369+0000", 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8520000000000004e-06 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.713440+0000", 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1635000000000001e-05 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.713561+0000", 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012047199999999999 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.713670+0000", 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109212 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.718055+0000", 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043844610000000001 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.718880+0000", 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00082516800000000004 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.888431+0000", 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16955098599999999 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.888532+0000", 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101405 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:12.888586+0000", 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4129000000000003e-05 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.740 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:13.740 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:13.740 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:08:13.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:13.762 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:13.763 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:13.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:13.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:13.894 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:13.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:13.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:14.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:14.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:14.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:14.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:14.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:14.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:14.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:14.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:13 smithi151 ceph-mon[16639]: pgmap v676: 47 pgs: 47 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 807 op/s 2024-09-02T10:08:14.195 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:13 smithi162 ceph-mon[20279]: pgmap v676: 47 pgs: 47 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 807 op/s 2024-09-02T10:08:14.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:13 smithi187 ceph-mon[20072]: pgmap v676: 47 pgs: 47 active+clean; 4.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 807 op/s 2024-09-02T10:08:14.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:14.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:14.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:14.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:14.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:14.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:14.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:14.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:08:14.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:08:14.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:14.519 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:14.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:14.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:14.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:14.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:14.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:14.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:14.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:14.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:14.838 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:14.868 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:14.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:14.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:14.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:14.939 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:08:14.939 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:08:14.939 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-02T10:08:14.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 110 0.05 2024-09-02T10:08:15.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:15.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:15.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:15.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:15.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:15.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:15.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:08:15.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:15.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:15.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:15.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:15.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:15.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:15.509 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:08:15.509 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:08:15.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.509 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:147893 3.d 3:b4c2d5fb:::benchmark_data_smithi187_59989_object38350:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:15.457622+0000", 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.039189871000000001, 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039220613000000001, 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 147893 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.457622+0000", 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.457622+0000", 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.457622+0000", 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2099999999999996e-07 2024-09-02T10:08:15.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.457627+0000", 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2980000000000002e-06 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.457636+0000", 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0289999999999994e-06 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.457643+0000", 2024-09-02T10:08:15.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8159999999999996e-06 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.457696+0000", 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3122999999999999e-05 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.457831+0000", 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013537900000000001 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,9", 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.457922+0000", 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0886999999999999e-05 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.464143+0000", 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062206359999999999 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.467441+0000", 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032987659999999999 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.514 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:147924 3.f 3:fa315a4c:::benchmark_data_smithi187_59989_object38381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:15.490541+0000", 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0062708809999999999, 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0063515020000000002, 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:08:15.515 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 147924 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490541+0000", 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490541+0000", 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:15.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490543+0000", 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9989999999999998e-06 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490549+0000", 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0839999999999997e-06 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490577+0000", 2024-09-02T10:08:15.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7858999999999999e-05 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490585+0000", 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8650000000000001e-06 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490650+0000", 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5333999999999995e-05 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:15.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490782+0000", 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013233599999999999 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 8,11", 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490887+0000", 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104831 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:147925 3.f 3:ff4e1c30:::benchmark_data_smithi187_59989_object38382:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:15.490647+0000", 2024-09-02T10:08:15.519 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0061648600000000003, 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062817369999999999, 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 147925 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490647+0000", 2024-09-02T10:08:15.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490647+0000", 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490647+0000", 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7999999999999995e-07 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:15.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490651+0000", 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3840000000000001e-06 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490660+0000", 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8440000000000004e-06 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.490665+0000", 2024-09-02T10:08:15.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3410000000000002e-06 2024-09-02T10:08:15.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495288+0000", 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046230730000000001 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495404+0000", 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116063 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 8,11", 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495522+0000", 2024-09-02T10:08:15.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011822799999999999 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24976.0:147924 3.f e279/226)", 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:15.495310+0000", 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.001501589, 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001650838, 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:15.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495310+0000", 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495310+0000", 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495311+0000", 2024-09-02T10:08:15.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.16e-07 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495315+0000", 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1910000000000002e-06 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495341+0000", 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5491999999999999e-05 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:15.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495347+0000", 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7399999999999998e-06 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24976.0:147924 3.f e279/226)", 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:15.495479+0000", 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0013322049999999999, 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001500985, 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-02T10:08:15.528 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495479+0000", 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495479+0000", 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495480+0000", 2024-09-02T10:08:15.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0900000000000001e-07 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495484+0000", 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0609999999999997e-06 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495510+0000", 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5789999999999999e-05 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:08:15.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:15.495517+0000", 2024-09-02T10:08:15.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2710000000000003e-06 2024-09-02T10:08:15.531 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.531 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:08:15.531 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.531 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:08:15.531 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:08:15.531 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 5 2024-09-02T10:08:15.531 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:08:15.565 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:15.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:08:15.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:08:15.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:08:15.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:08:15.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:08:15.574 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:08:15.574 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:08:15.574 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:08:15.574 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:08:15.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:08:15.578 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:08:15.578 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:08:15.589 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:15.590 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:15.610 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:08:15.610 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:08:15.641 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:08:15.641 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:08:15.673 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:08:15.673 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:08:15.676 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:15.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:15.709 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:08:15.709 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:08:15.713 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:15.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:15.737 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:15.742 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:08:15.742 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:08:15.770 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:08:15.770 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:08:15.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:15.840 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:15.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:15.841 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:15.942 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:15.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:15.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:15.968 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:16.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:16.093 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:16.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:16.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:16.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:16.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:16.291 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:16 smithi162 ceph-mon[20279]: pgmap v677: 47 pgs: 47 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-09-02T10:08:16.291 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/645835167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:16.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:16.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:08:16.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:16.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:16.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:16 smithi151 ceph-mon[16639]: pgmap v677: 47 pgs: 47 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-09-02T10:08:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/645835167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:16.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:16.475 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:16.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:16 smithi187 ceph-mon[20072]: pgmap v677: 47 pgs: 47 active+clean; 4.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 832 op/s 2024-09-02T10:08:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/645835167' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:16.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:16.601 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:16.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:16.643 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:16.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:16.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:16.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:16.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:16.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:16.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:16.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:16.985 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:16.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:17.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:17.122 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:17.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:17.181 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:17.181 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:17.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:17.227 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:08:17.249 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:17.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:17.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2501298020' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:17.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:17.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:17.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:17.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:17.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2501298020' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:17.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:17.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:17.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:17.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2501298020' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:17.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:17.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:17.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:17.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:17.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:17.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:17.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:17.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:17.883 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:17.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:17.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:17.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:08:17.971 INFO:teuthology.orchestra.run.smithi151.stdout:moved 2 / 141 (1.41844%) 2024-09-02T10:08:17.971 INFO:teuthology.orchestra.run.smithi151.stdout:avg 12.8182 2024-09-02T10:08:17.971 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.69066 -> 2.44273 (expected baseline 3.41363) 2024-09-02T10:08:17.971 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.2 with 8 -> 9 pgs (0.624113 -> 0.702128 * mean) 2024-09-02T10:08:17.971 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 17 -> 17 pgs (1.32624 -> 1.32624 * mean) 2024-09-02T10:08:17.971 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:08:17.971 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:08:17.971 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 0.05 2024-09-02T10:08:17.972 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:08:17.972 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0151 2024-09-02T10:08:17.972 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0166 2024-09-02T10:08:17.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11 weight 1.0000 -> 0.9500 2024-09-02T10:08:17.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 weight 1.0000 -> 0.9500 2024-09-02T10:08:17.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 1.0000 -> 0.9500 2024-09-02T10:08:17.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 1.0000 -> 0.9500 2024-09-02T10:08:17.972 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:08:17.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:18.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:18.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:18.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:18.149 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:18.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:18.188 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 10:08:17 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: 2024-09-02T10:08:17.969+0000 7fd7798ae640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-02T10:08:18.222 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:18 smithi162 ceph-mon[20279]: pgmap v678: 47 pgs: 47 active+clean; 4.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-09-02T10:08:18.222 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:18 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:18.222 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2501298020' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:18.223 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:18 smithi162 ceph-mon[20279]: osdmap e280: 12 total, 12 up, 11 in 2024-09-02T10:08:18.238 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:08:18.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:18.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:18.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:18.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:18.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:18.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:18.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:18.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:18 smithi187 ceph-mon[20072]: pgmap v678: 47 pgs: 47 active+clean; 4.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-09-02T10:08:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:18 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2501298020' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:18 smithi187 ceph-mon[20072]: osdmap e280: 12 total, 12 up, 11 in 2024-09-02T10:08:18.525 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:18.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:18.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:18.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:18.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:18 smithi151 ceph-mon[16639]: pgmap v678: 47 pgs: 47 active+clean; 4.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-09-02T10:08:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:18 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2501298020' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:18 smithi151 ceph-mon[16639]: osdmap e280: 12 total, 12 up, 11 in 2024-09-02T10:08:18.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:18.762 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:18.762 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:18.772 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:18.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:18.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:18.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:18.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:18.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:18.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:19.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:19.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:19.121 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-02T10:08:19.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-02T10:08:19.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:19.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:19.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:19.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:19.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:08:19.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:19.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:19.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:19.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:19.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:19.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:19.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:19 smithi151 ceph-mon[16639]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:19.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:08:19.495 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:19 smithi187 ceph-mon[20072]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:19.496 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:19.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:19.551 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:19.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:19.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:19.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:19 smithi162 ceph-mon[20279]: from='client.25330 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:19.653 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:19.677 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:19.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:19.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:19.779 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:19.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:19.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:19.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:19.912 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:19.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:19.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:20.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:20.073 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:20.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:20.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:20.175 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:20.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:20.203 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:20.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:20.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:20.331 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:20.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:20.369 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:20.369 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:20.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:20 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:20.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:20 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:20.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:20 smithi151 ceph-mon[16639]: pgmap v680: 47 pgs: 47 active+clean; 4.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-09-02T10:08:20.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:20 smithi151 ceph-mon[16639]: osdmap e281: 12 total, 12 up, 11 in 2024-09-02T10:08:20.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:08:20.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:20.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:20.460 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:20.475 INFO:teuthology.orchestra.run.smithi151.stdout:moved 13 / 141 (9.21986%) 2024-09-02T10:08:20.475 INFO:teuthology.orchestra.run.smithi151.stdout:avg 12.8182 2024-09-02T10:08:20.475 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.69066 -> 2.9486 (expected baseline 3.41363) 2024-09-02T10:08:20.475 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.2 with 8 -> 10 pgs (0.624113 -> 0.780142 * mean) 2024-09-02T10:08:20.475 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 17 -> 17 pgs (1.32624 -> 1.32624 * mean) 2024-09-02T10:08:20.475 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:08:20.475 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:08:20.476 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-09-02T10:08:20.476 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:08:20.476 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0149 2024-09-02T10:08:20.476 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0164 2024-09-02T10:08:20.476 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11 weight 1.0000 -> 0.5534 2024-09-02T10:08:20.476 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 weight 1.0000 -> 0.6395 2024-09-02T10:08:20.477 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 1.0000 -> 0.8729 2024-09-02T10:08:20.477 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 1.0000 -> 0.8906 2024-09-02T10:08:20.477 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:08:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:20 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:20 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:20 smithi187 ceph-mon[20072]: pgmap v680: 47 pgs: 47 active+clean; 4.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-09-02T10:08:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:20 smithi187 ceph-mon[20072]: osdmap e281: 12 total, 12 up, 11 in 2024-09-02T10:08:20.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:20.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:20.595 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:20.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:20 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:20 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:20 smithi162 ceph-mon[20279]: pgmap v680: 47 pgs: 47 active+clean; 4.3 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 935 op/s 2024-09-02T10:08:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:20 smithi162 ceph-mon[20279]: osdmap e281: 12 total, 12 up, 11 in 2024-09-02T10:08:20.688 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 10:08:20 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: 2024-09-02T10:08:20.473+0000 7fd7798ae640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-02T10:08:20.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:20.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:20.720 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:20.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:20.849 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:20.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:20.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:20.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:20.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:21.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:21.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:21.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:21.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:08:21.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:21.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:21.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:21.263 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:21.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:21.394 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:21.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:21.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:21 smithi187 ceph-mon[20072]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/366550064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:21.509 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:21.521 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:21.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:21.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:21 smithi162 ceph-mon[20279]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:21.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/366550064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:21.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:21 smithi151 ceph-mon[16639]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:21.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/366550064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:21.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:21.644 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-02T10:08:21.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-09-02T10:08:21.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:21.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:21.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:21.758 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:08:21.758 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:08:21.758 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:08:21.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:08:21.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:08:21.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:08:21.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:08:21.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:08:21.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:08:21.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:08:21.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:08:21.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:08:21.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:08:21.788 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:21.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:21.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:21.921 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:21.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:22.010 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:08:22.010 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:08:22.011 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:08:22.011 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:08:22.011 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:08:22.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:22.049 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:22.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:22.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:22.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:22.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:22.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:22.307 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:22.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:22.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:22.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:22.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:22.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:22 smithi151 ceph-mon[16639]: pgmap v682: 47 pgs: 47 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 740 op/s 2024-09-02T10:08:22.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:22 smithi162 ceph-mon[20279]: pgmap v682: 47 pgs: 47 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 740 op/s 2024-09-02T10:08:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:22 smithi187 ceph-mon[20072]: pgmap v682: 47 pgs: 47 active+clean; 4.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 740 op/s 2024-09-02T10:08:22.534 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:22.560 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:22.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:22.568 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:22.663 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:22.689 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:22.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:22.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:22.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:22.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:08:22.824 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:22.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:22.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:22.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:22.957 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:22.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:23.058 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:23.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:23.088 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:23.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:23.101 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:23.126 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:23.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:23.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:23.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:23.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:23.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:23.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:23.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:23.454 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:23.478 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:23.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:23.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:23.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:23.606 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:23.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:23.683 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:08:23.683 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:08:23.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:08:23.683 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:08:23.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:36428 2.b 2:da93149a:::benchmark_data_smithi187_45882_object36427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:02.079845+0000", 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.60055722599998, 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22416934999999999, 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.684 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36428 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079845+0000", 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079845+0000", 2024-09-02T10:08:23.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079846+0000", 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3900000000000004e-07 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079867+0000", 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1290000000000001e-05 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079878+0000", 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0677e-05 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.079887+0000", 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1109999999999997e-06 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.081179+0000", 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001292444 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.081311+0000", 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000131855 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.081411+0000", 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9522e-05 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.082483+0000", 2024-09-02T10:08:23.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010728630000000001 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.143788+0000", 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.061304751999999997 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.303899+0000", 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16011112 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.303951+0000", 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1687999999999997e-05 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.304014+0000", 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3288999999999997e-05 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:54932 2.b 2:d9ab64ae:::benchmark_data_smithi187_45882_object54931:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:08:23.690 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.160766+0000", 2024-09-02T10:08:23.691 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 366.51963588900003, 2024-09-02T10:08:23.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26274712500000003, 2024-09-02T10:08:23.691 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.691 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.691 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.691 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.691 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.691 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.691 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 54932 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160766+0000", 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160766+0000", 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160767+0000", 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2e-07 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160785+0000", 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8139e-05 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160792+0000", 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5109999999999997e-06 2024-09-02T10:08:23.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160797+0000", 2024-09-02T10:08:23.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2030000000000002e-06 2024-09-02T10:08:23.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160835+0000", 2024-09-02T10:08:23.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8591999999999999e-05 2024-09-02T10:08:23.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.160936+0000", 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100828 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.161018+0000", 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1940999999999994e-05 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162378+0000", 2024-09-02T10:08:23.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001359968 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162459+0000", 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0600999999999996e-05 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423449+0000", 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26099011300000002 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423482+0000", 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.379e-05 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423513+0000", 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0919e-05 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:54932 2.b e113/92 ondisk, result = 0)", 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:08:23.697 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 366.51750965399998, 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26062987399999998, 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:08:23.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162893+0000", 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4199999999999995e-07 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.162896+0000", 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.23e-06 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423394+0000", 2024-09-02T10:08:23.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26049795999999997 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423399+0000", 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4360000000000004e-06 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423440+0000", 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1412999999999998e-05 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423445+0000", 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1499999999999998e-06 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.423522+0000", 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7043000000000001e-05 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:55061 2.6 e113/92 2:670fe1b2:::benchmark_data_smithi187_45882_object55060:head v 113'3388, mlcod=113'3385)", 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.479402+0000", 2024-09-02T10:08:23.702 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 366.20100042399997, 2024-09-02T10:08:23.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33103881099999999, 2024-09-02T10:08:23.703 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.703 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.703 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.703 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479402+0000", 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479402+0000", 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479403+0000", 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3600000000000002e-07 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.479437+0000", 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4811999999999998e-05 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809044+0000", 2024-09-02T10:08:23.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32960664000000001 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809050+0000", 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7679999999999997e-06 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809098+0000", 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7859999999999999e-05 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.809112+0000", 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4579000000000001e-05 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.810407+0000", 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012949649999999999 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.810441+0000", 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3250999999999997e-05 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.707 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55063 2.b 2:d2962897:::benchmark_data_smithi187_45882_object55062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.481328+0000", 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 366.19907404600002, 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.292836235, 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.708 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55063 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481328+0000", 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481328+0000", 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481329+0000", 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0500000000000003e-07 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481351+0000", 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2506000000000001e-05 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481361+0000", 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0105999999999999e-05 2024-09-02T10:08:23.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481366+0000", 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8119999999999998e-06 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481408+0000", 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1689e-05 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481498+0000", 2024-09-02T10:08:23.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0192000000000007e-05 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.481585+0000", 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7214999999999998e-05 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.482940+0000", 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013545709999999999 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483445+0000", 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050479699999999997 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774099+0000", 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29065394700000002 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774135+0000", 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6665999999999998e-05 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774164+0000", 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9028999999999999e-05 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:55063 2.b e113/92 ondisk, result = 0)", 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.483081+0000", 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 366.19732107099998, 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29108941100000002, 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.714 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483081+0000", 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483081+0000", 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483082+0000", 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6700000000000003e-07 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.483085+0000", 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3009999999999998e-06 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774042+0000", 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29095665500000001 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774047+0000", 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8629999999999996e-06 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774087+0000", 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9206999999999999e-05 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774093+0000", 2024-09-02T10:08:23.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6710000000000002e-06 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774170+0000", 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7146999999999998e-05 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55077 2.b 2:d13d7ad6:::benchmark_data_smithi187_45882_object55076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:08:23.718 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.497835+0000", 2024-09-02T10:08:23.719 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 366.18256685099999, 2024-09-02T10:08:23.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29033786299999997, 2024-09-02T10:08:23.719 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.719 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.719 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.719 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.719 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.719 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.719 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55077 2024-09-02T10:08:23.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497835+0000", 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497835+0000", 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.720 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497836+0000", 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.06e-07 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497866+0000", 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9967999999999998e-05 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497875+0000", 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9090000000000006e-06 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497882+0000", 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1409999999999999e-06 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.497929+0000", 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6683e-05 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498008+0000", 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9110000000000007e-05 2024-09-02T10:08:23.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:08:23.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498101+0000", 2024-09-02T10:08:23.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3379000000000002e-05 2024-09-02T10:08:23.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.499478+0000", 2024-09-02T10:08:23.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001376637 2024-09-02T10:08:23.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.500299+0000", 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00082144399999999995 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.788094+0000", 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28779512299999999 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.788131+0000", 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6924000000000001e-05 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.788173+0000", 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1838999999999997e-05 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55083 2.5 2:a3eaecaa:::benchmark_data_smithi187_45882_object55082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:08:23.725 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.511889+0000", 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 366.16851270900003, 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.254154883, 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55083 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511889+0000", 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511889+0000", 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511890+0000", 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.046e-06 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511910+0000", 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9162999999999999e-05 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511921+0000", 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0995000000000001e-05 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511926+0000", 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2650000000000004e-06 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.511967+0000", 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1461999999999997e-05 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.512047+0000", 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9907999999999995e-05 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.512138+0000", 2024-09-02T10:08:23.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0909999999999995e-05 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.513495+0000", 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001356307 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.513828+0000", 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00033362600000000001 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765934+0000", 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25210628000000002 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765996+0000", 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1614999999999999e-05 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.766044+0000", 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8306000000000001e-05 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.731 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:55086 2.f 2:f2dac436:::benchmark_data_smithi187_45882_object55085:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.518890+0000", 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 366.16151227400002, 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25559369300000001, 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.732 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55086 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518890+0000", 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518890+0000", 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518891+0000", 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8199999999999998e-07 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518909+0000", 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8649000000000001e-05 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518920+0000", 2024-09-02T10:08:23.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0560000000000001e-05 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518925+0000", 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0830000000000003e-06 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.518974+0000", 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8979e-05 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.519056+0000", 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1796000000000001e-05 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.519137+0000", 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1422999999999996e-05 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520575+0000", 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014377159999999999 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520789+0000", 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00021378200000000001 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774432+0000", 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25364359800000003 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774462+0000", 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9416000000000001e-05 2024-09-02T10:08:23.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.774484+0000", 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1809000000000001e-05 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:55083 2.5 e113/92 ondisk, result = 0)", 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.520803+0000", 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 366.15959902200001, 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24524790699999999, 2024-09-02T10:08:23.738 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520803+0000", 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520803+0000", 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520804+0000", 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1200000000000003e-07 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.520807+0000", 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.523e-06 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765868+0000", 2024-09-02T10:08:23.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.245060377 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765872+0000", 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6949999999999999e-06 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765920+0000", 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7593000000000003e-05 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.765929+0000", 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6260000000000001e-06 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.766051+0000", 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122481 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:97743 2.5 2:a58175a7:::benchmark_data_smithi187_45882_object97742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-02T10:08:23.742 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:05.424082+0000", 2024-09-02T10:08:23.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:23.744 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 318.25632009200001, 2024-09-02T10:08:23.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.60300915200000005, 2024-09-02T10:08:23.744 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.744 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.744 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.744 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.744 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.744 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.744 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 97743 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424082+0000", 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424082+0000", 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424084+0000", 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6160000000000001e-06 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424109+0000", 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5466999999999999e-05 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424122+0000", 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2765999999999999e-05 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424129+0000", 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2599999999999999e-06 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424197+0000", 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7782999999999998e-05 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424308+0000", 2024-09-02T10:08:23.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000110645 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.424453+0000", 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014499099999999999 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427671+0000", 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032188590000000001 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.555569+0000", 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.127897921 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027018+0000", 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.47144856699999999 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027061+0000", 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3092e-05 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027091+0000", 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0185e-05 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:97743 2.5 e137/92 ondisk, result = 0)", 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:05.427175+0000", 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 318.25322670000003, 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.59992140800000004, 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.750 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427175+0000", 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427175+0000", 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427177+0000", 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.511e-06 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.427183+0000", 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4629999999999998e-06 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.026965+0000", 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.59978128600000002 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.026969+0000", 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1810000000000003e-06 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027007+0000", 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8557999999999999e-05 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027013+0000", 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7649999999999999e-06 2024-09-02T10:08:23.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.027097+0000", 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3644000000000001e-05 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:117060 2.f 2:f36bc4f3:::benchmark_data_smithi187_45882_object19309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:29.622070+0000", 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 294.05833240999999, 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.266536263, 2024-09-02T10:08:23.754 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 117060 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622070+0000", 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622070+0000", 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622071+0000", 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.001e-06 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622075+0000", 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1509999999999997e-06 2024-09-02T10:08:23.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622084+0000", 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8580000000000006e-06 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622088+0000", 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5970000000000002e-06 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622136+0000", 2024-09-02T10:08:23.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7803999999999998e-05 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622256+0000", 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119537 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622368+0000", 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011192900000000001 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.625537+0000", 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003168911 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.637552+0000", 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012015965999999999 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.888531+0000", 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25097857000000001 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.888578+0000", 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7426000000000003e-05 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.888606+0000", 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7512999999999999e-05 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.760 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:163327 2.f 2:f9a47073:::benchmark_data_smithi187_45882_object65576:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:56.455190+0000", 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 207.22521229700001, 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.422262621, 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 163327 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455190+0000", 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455190+0000", 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455192+0000", 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.889e-06 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455200+0000", 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7710000000000006e-06 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455216+0000", 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6943999999999999e-05 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455224+0000", 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1550000000000001e-06 2024-09-02T10:08:23.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455280+0000", 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6443999999999997e-05 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455368+0000", 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8307999999999995e-05 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:08:23.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455446+0000", 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7698e-05 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.459097+0000", 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036506360000000001 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.459525+0000", 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00042830199999999998 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877381+0000", 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41785591900000002 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877420+0000", 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8832999999999999e-05 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877452+0000", 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2722000000000003e-05 2024-09-02T10:08:23.766 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:164051 2.b 2:dd30ecaf:::benchmark_data_smithi187_45882_object66300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:58.383444+0000", 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.29695778300001, 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26956966300000001, 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.767 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 164051 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383444+0000", 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383444+0000", 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383446+0000", 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.347e-06 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383452+0000", 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.088e-06 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383469+0000", 2024-09-02T10:08:23.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7153999999999999e-05 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383478+0000", 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9109999999999999e-06 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383537+0000", 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8687999999999998e-05 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383632+0000", 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5619999999999996e-05 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.383712+0000", 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9435000000000001e-05 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387376+0000", 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003664488 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387533+0000", 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015662400000000001 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652930+0000", 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26539721100000002 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652964+0000", 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4489999999999997e-05 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.653014+0000", 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9607000000000001e-05 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:164051 2.b e200/92 ondisk, result = 0)", 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.292735497, 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26535247699999998, 2024-09-02T10:08:23.773 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387667+0000", 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1799999999999995e-07 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.387670+0000", 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8219999999999998e-06 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652865+0000", 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26519509200000002 2024-09-02T10:08:23.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652872+0000", 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3509999999999996e-06 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652921+0000", 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8124999999999997e-05 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.652926+0000", 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.682e-06 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:58.653019+0000", 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2886999999999993e-05 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "rep_scrubmap(2.b e92 from shard 7)", 2024-09-02T10:08:23.777 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:42.047060+0000", 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 161.63334208699999, 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.229439527, 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.047060+0000", 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.047060+0000", 2024-09-02T10:08:23.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.047061+0000", 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7899999999999997e-07 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.047075+0000", 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4046000000000001e-05 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.271919+0000", 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22484417700000001 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.271925+0000", 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0190000000000003e-06 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.276438+0000", 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045127470000000001 2024-09-02T10:08:23.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.276447+0000", 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2539999999999998e-06 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.276500+0000", 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2404999999999997e-05 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:195477 2.f 2:f32fe7b9:::benchmark_data_smithi187_45882_object97726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:43.014428+0000", 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.66597387100001, 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43313299799999999, 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 195477 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014428+0000", 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014428+0000", 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014430+0000", 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.55e-06 2024-09-02T10:08:23.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014459+0000", 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9539e-05 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014470+0000", 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1002000000000001e-05 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.014477+0000", 2024-09-02T10:08:23.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0720000000000002e-06 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.019153+0000", 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046751120000000004 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.019267+0000", 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011444500000000001 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:08:23.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.019338+0000", 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1087999999999994e-05 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.024391+0000", 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005052506 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.025167+0000", 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00077600499999999997 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447446+0000", 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42227959799999998 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447512+0000", 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5617000000000006e-05 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447561+0000", 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9463999999999999e-05 2024-09-02T10:08:23.787 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:195477 2.f e217/92 ondisk, result = 0)", 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:43.026172+0000", 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.654230555, 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42139491699999998, 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.026172+0000", 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.026172+0000", 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.026172+0000", 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8999999999999996e-07 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.026175+0000", 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2610000000000002e-06 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447363+0000", 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42118752500000001 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447370+0000", 2024-09-02T10:08:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.126e-06 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447428+0000", 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7776999999999999e-05 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447438+0000", 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.039e-05 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.447567+0000", 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000128248 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24976.0:109536 3.9 3:9af67999:::benchmark_data_smithi187_59989_object109535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:07:33.018363+0000", 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 50.662039612999997, 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.368789486, 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:23.792 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24976", 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 109536 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018363+0000", 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:23.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018363+0000", 2024-09-02T10:08:23.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:23.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:23.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018364+0000", 2024-09-02T10:08:23.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0750000000000001e-06 2024-09-02T10:08:23.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:23.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018385+0000", 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1044e-05 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018396+0000", 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1812e-05 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018403+0000", 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9580000000000001e-06 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021486+0000", 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030823600000000001 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021631+0000", 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014475600000000001 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021787+0000", 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015694600000000001 2024-09-02T10:08:23.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.024639+0000", 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028514159999999998 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.025503+0000", 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086444899999999999 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.387070+0000", 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36156629699999998 2024-09-02T10:08:23.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.387107+0000", 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7620000000000002e-05 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.387152+0000", 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4753000000000003e-05 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:23.798 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:23.799 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:08:23.832 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:23.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:23.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:23.959 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:23.960 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:24.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:24.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:24.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:24.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:24.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:24.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:24.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:24.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:24 smithi151 ceph-mon[16639]: pgmap v683: 47 pgs: 47 active+clean; 4.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 851 op/s 2024-09-02T10:08:24.246 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4224233421' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:08:24.246 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:24 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'58531','3':'54708','4':'42137','11':'36430'}"} : dispatch 2024-09-02T10:08:24.250 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:08:24.250 INFO:teuthology.orchestra.run.smithi151.stdout:moved 13 / 141 (9.21986%) 2024-09-02T10:08:24.250 INFO:teuthology.orchestra.run.smithi151.stdout:avg 12.8182 2024-09-02T10:08:24.250 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.69066 -> 2.9486 (expected baseline 3.41363) 2024-09-02T10:08:24.250 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.2 with 8 -> 10 pgs (0.624113 -> 0.780142 * mean) 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 17 -> 17 pgs (1.32624 -> 1.32624 * mean) 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0142 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0156 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11 weight 1.0000 -> 0.5559 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 weight 1.0000 -> 0.6430 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 1.0000 -> 0.8348 2024-09-02T10:08:24.251 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 1.0000 -> 0.8931 2024-09-02T10:08:24.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:24.338 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:24 smithi187 ceph-mon[20072]: pgmap v683: 47 pgs: 47 active+clean; 4.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 851 op/s 2024-09-02T10:08:24.338 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4224233421' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:08:24.338 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:24 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'58531','3':'54708','4':'42137','11':'36430'}"} : dispatch 2024-09-02T10:08:24.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:24.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:24.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:24.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:24.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:24.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:08:24.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:24.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:24.593 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:24 smithi162 ceph-mon[20279]: pgmap v683: 47 pgs: 47 active+clean; 4.1 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 851 op/s 2024-09-02T10:08:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4224233421' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:08:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:24 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'58531','3':'54708','4':'42137','11':'36430'}"} : dispatch 2024-09-02T10:08:24.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:24.717 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:24.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:24.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:24.807 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:24.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:24.840 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:24.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:24.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:24.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:24.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:24.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:25.066 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:25.087 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:25.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:25.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:25.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:25.211 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:25.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:25.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:25.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:25.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:25.459 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:25.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:25.461 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-02T10:08:25.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-09-02T10:08:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:25 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:25 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:25 smithi187 ceph-mon[20072]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:25 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:25.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:25 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:25.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:25 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:25.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4224233421' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:08:25.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'58531','3':'54708','4':'42137','11':'36430'}"}]': finished 2024-09-02T10:08:25.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:25 smithi187 ceph-mon[20072]: osdmap e282: 12 total, 12 up, 11 in 2024-09-02T10:08:25.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:25 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:25.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:25 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:25.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:25 smithi151 ceph-mon[16639]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:25.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:25 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:25.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:25 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:25.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:25 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:25.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4224233421' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:08:25.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'58531','3':'54708','4':'42137','11':'36430'}"}]': finished 2024-09-02T10:08:25.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:25 smithi151 ceph-mon[16639]: osdmap e282: 12 total, 12 up, 11 in 2024-09-02T10:08:25.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:25.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:25.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:25.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:25 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:25 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:25 smithi162 ceph-mon[20279]: from='client.25354 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:25 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:25 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:25 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4224233421' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:08:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'58531','3':'54708','4':'42137','11':'36430'}"}]': finished 2024-09-02T10:08:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:25 smithi162 ceph-mon[20279]: osdmap e282: 12 total, 12 up, 11 in 2024-09-02T10:08:25.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:25.709 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:25.710 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:25.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:25.835 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:25.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:25.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:25.957 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:25.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:25.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:26.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:26.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:26.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:26.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:26.206 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:26.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:26.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:08:26.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:26.329 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:26.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:26.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:26.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:26 smithi151 ceph-mon[16639]: pgmap v685: 47 pgs: 1 peering, 46 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 258 B/s rd, 6.2 KiB/s wr, 699 op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-02T10:08:26.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:26 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:08:26.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:26 smithi151 ceph-mon[16639]: osdmap e283: 12 total, 12 up, 11 in 2024-09-02T10:08:26.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:26 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:08:26.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:26 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:08:26.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:26.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:26.464 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:26 smithi187 ceph-mon[20072]: pgmap v685: 47 pgs: 1 peering, 46 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 258 B/s rd, 6.2 KiB/s wr, 699 op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-02T10:08:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:26 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:08:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:26 smithi187 ceph-mon[20072]: osdmap e283: 12 total, 12 up, 11 in 2024-09-02T10:08:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:26 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:08:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:26 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:08:26.514 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:26.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:26.579 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:26.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:26.580 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:26 smithi162 ceph-mon[20279]: pgmap v685: 47 pgs: 1 peering, 46 active+clean; 4.0 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 258 B/s rd, 6.2 KiB/s wr, 699 op/s; 1.5 KiB/s, 0 objects/s recovering 2024-09-02T10:08:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:26 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:08:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:26 smithi162 ceph-mon[20279]: osdmap e283: 12 total, 12 up, 11 in 2024-09-02T10:08:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:26 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:08:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:26 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:08:26.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:26.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:26.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:26.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:26.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:26.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:26.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:26.942 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:26.965 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:26.966 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:27.048 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:08:27.048 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:08:27.048 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:08:27.049 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:08:27.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:27.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:27.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:27.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:27.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:27.203 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:08:27.203 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:08:27.224 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:27.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:27.239 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:08:27.239 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:08:27.276 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:08:27.276 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:08:27.310 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:08:27.311 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:08:27.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:27.331 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 141 (0%) 2024-09-02T10:08:27.331 INFO:teuthology.orchestra.run.smithi151.stdout:avg 12.8182 2024-09-02T10:08:27.331 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.9486 -> 2.9486 (expected baseline 3.41363) 2024-09-02T10:08:27.331 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.4 with 9 -> 9 pgs (0.702128 -> 0.702128 * mean) 2024-09-02T10:08:27.331 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.0 with 17 -> 17 pgs (1.32624 -> 1.32624 * mean) 2024-09-02T10:08:27.331 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:08:27.331 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-09-02T10:08:27.332 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-09-02T10:08:27.332 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:08:27.332 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0138 2024-09-02T10:08:27.332 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.1380 2024-09-02T10:08:27.332 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:08:27.344 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:08:27.345 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:08:27.352 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:27.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:27.381 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:08:27.384 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:08:27.415 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:08:27.415 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:08:27.449 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:08:27.450 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:08:27.454 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:27.478 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:27.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:27.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:08:27.480 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:08:27.480 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:27 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:08:27.480 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:27 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:08:27.481 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:27 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:08:27.481 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:27 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:08:27.481 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:27 smithi151 ceph-mon[16639]: osdmap e284: 12 total, 12 up, 11 in 2024-09-02T10:08:27.481 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2131039280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:27.481 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:27 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:08:27.481 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:27 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:08:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:27 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:08:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:27 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:08:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:27 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:08:27.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:27 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:08:27.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:27 smithi187 ceph-mon[20072]: osdmap e284: 12 total, 12 up, 11 in 2024-09-02T10:08:27.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2131039280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:27.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:27 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:08:27.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:27 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:08:27.509 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:08:27.509 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:08:27.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:08:27.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:08:27.570 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:08:27.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:08:27.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:27.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:27 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:08:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:27 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:08:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:27 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:08:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:27 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:08:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:27 smithi162 ceph-mon[20279]: osdmap e284: 12 total, 12 up, 11 in 2024-09-02T10:08:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2131039280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:27 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:08:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:27 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:08:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:27.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:27.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:27.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:27.746 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:27.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:27.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:08:27.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:27.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:08:27.872 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:27.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:27.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:27.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:27.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:27.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:28.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:28.101 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:28.123 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:28.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:28.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:28.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:28.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:28.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:28.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:28.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:28.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:28.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:28.494 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:28 smithi187 ceph-mon[20072]: pgmap v688: 47 pgs: 1 peering, 46 active+clean; 3.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 1023 B/s rd, 33 KiB/s wr, 757 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-09-02T10:08:28.494 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:28 smithi187 ceph-mon[20072]: from='client.15747 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:28.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:28.505 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:28.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:28.540 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:28 smithi151 ceph-mon[16639]: pgmap v688: 47 pgs: 1 peering, 46 active+clean; 3.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 1023 B/s rd, 33 KiB/s wr, 757 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-09-02T10:08:28.541 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:28 smithi151 ceph-mon[16639]: from='client.15747 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:28.541 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-02T10:08:28.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-02T10:08:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:28 smithi162 ceph-mon[20279]: pgmap v688: 47 pgs: 1 peering, 46 active+clean; 3.9 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 1023 B/s rd, 33 KiB/s wr, 757 op/s; 7.8 KiB/s, 0 objects/s recovering 2024-09-02T10:08:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:28 smithi162 ceph-mon[20279]: from='client.15747 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:28.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:28.610 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:28.610 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:28.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:28.634 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:28.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:28.756 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:28.757 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:28.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:28.882 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:28.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:28.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:28.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:29.006 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:29.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:29.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:29.093 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:29.107 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:29.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:29.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:29.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:29.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:29.231 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:29.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:29.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:29.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:08:29.356 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:29.402 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:29.404 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:29.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:29.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:29.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:29.615 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:29.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:29.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:29.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:29.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:29.789 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:29.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:29.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:29.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:29.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:29.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:29 smithi187 ceph-mon[20072]: pgmap v689: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 peering, 43 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 1023 B/s rd, 33 KiB/s wr, 528 op/s; 13598/191214 objects degraded (7.111%); 7895/191214 objects misplaced (4.129%); 445 KiB/s, 7 objects/s recovering 2024-09-02T10:08:30.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:30.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:30.049 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:30.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:30.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:30.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:30.178 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:30.178 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:30.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:29 smithi151 ceph-mon[16639]: pgmap v689: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 peering, 43 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 1023 B/s rd, 33 KiB/s wr, 528 op/s; 13598/191214 objects degraded (7.111%); 7895/191214 objects misplaced (4.129%); 445 KiB/s, 7 objects/s recovering 2024-09-02T10:08:30.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:30.302 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:30.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:29 smithi162 ceph-mon[20279]: pgmap v689: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 1 peering, 43 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 1023 B/s rd, 33 KiB/s wr, 528 op/s; 13598/191214 objects degraded (7.111%); 7895/191214 objects misplaced (4.129%); 445 KiB/s, 7 objects/s recovering 2024-09-02T10:08:30.404 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:30.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:30.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:30.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:30.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:30.552 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:30.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:30.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:30.673 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 141 (0%) 2024-09-02T10:08:30.674 INFO:teuthology.orchestra.run.smithi151.stdout:avg 12.8182 2024-09-02T10:08:30.674 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.9486 -> 2.9486 (expected baseline 3.41363) 2024-09-02T10:08:30.674 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.4 with 9 -> 9 pgs (0.702128 -> 0.702128 * mean) 2024-09-02T10:08:30.674 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.0 with 17 -> 17 pgs (1.32624 -> 1.32624 * mean) 2024-09-02T10:08:30.674 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:08:30.674 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-09-02T10:08:30.675 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 0.05 2024-09-02T10:08:30.675 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:08:30.675 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0137 2024-09-02T10:08:30.675 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.1368 2024-09-02T10:08:30.675 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:08:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:30.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:30.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:30.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:30.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:30.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:30.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:30.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:08:30.896 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:08:30.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:30.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:30 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 13598/191214 objects degraded (7.111%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1109513448' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:30 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:30 smithi151 ceph-mon[16639]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:30.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:30.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:31.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:08:31.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:31.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:31.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:31.076 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:31.165 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:31.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:31.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:31.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:30 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 13598/191214 objects degraded (7.111%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1109513448' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:30 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:30 smithi187 ceph-mon[20072]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:31.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:31.337 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:31.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:31.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:30 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 13598/191214 objects degraded (7.111%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1109513448' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:30 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:30 smithi162 ceph-mon[20279]: from='client.15759 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:08:31.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:31.463 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:31.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:31.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:31.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:31.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:31.682 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:31.682 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:31.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:31.722 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:31.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:31.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:31.845 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:31.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:31.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:31.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:31.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:31.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:32.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:32.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:32.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:32.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:31 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:08:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:31 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:31 smithi151 ceph-mon[16639]: osdmap e285: 12 total, 12 up, 11 in 2024-09-02T10:08:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:31 smithi151 ceph-mon[16639]: pgmap v691: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 389 op/s; 13598/189795 objects degraded (7.165%); 7895/189795 objects misplaced (4.160%); 435 KiB/s, 7 objects/s recovering 2024-09-02T10:08:32.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:32.229 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:32.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:31 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:08:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:31 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:31 smithi187 ceph-mon[20072]: osdmap e285: 12 total, 12 up, 11 in 2024-09-02T10:08:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:31 smithi187 ceph-mon[20072]: pgmap v691: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 389 op/s; 13598/189795 objects degraded (7.165%); 7895/189795 objects misplaced (4.160%); 435 KiB/s, 7 objects/s recovering 2024-09-02T10:08:32.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:32.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:08:32.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:31 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:08:32.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:31 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:32.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:31 smithi162 ceph-mon[20279]: osdmap e285: 12 total, 12 up, 11 in 2024-09-02T10:08:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:31 smithi162 ceph-mon[20279]: pgmap v691: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 389 op/s; 13598/189795 objects degraded (7.165%); 7895/189795 objects misplaced (4.160%); 435 KiB/s, 7 objects/s recovering 2024-09-02T10:08:32.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:32.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:32.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:32.479 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:32.483 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:32.484 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:32.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:32.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:32.612 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:32.627 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:32.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:32.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:32.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:32.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:32.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:32.873 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:32.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:32.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:32.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:32.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:32.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:32 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:08:33.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:32 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:08:33.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:33.108 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:08:33.108 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:08:33.120 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:33.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:33.138 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:08:33.138 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:08:33.163 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:33.163 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:33.168 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:08:33.168 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:08:33.199 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:08:33.199 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:08:33.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:33.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:08:33.234 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:08:33.247 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:33.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:33.267 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:08:33.268 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:08:33.300 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:08:33.300 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:08:33.332 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:08:33.332 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:08:33.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:32 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:08:33.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:33.359 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:08:33.359 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:08:33.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:33.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:33.395 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:08:33.395 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:08:33.424 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:08:33.424 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:08:33.457 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:08:33.457 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:08:33.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:33.496 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:33.497 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:33.598 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:33.620 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:33.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:33.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:33.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:33.745 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:33.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:33.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:33.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:33.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:33.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:08:33.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:34.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:34.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:34.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2045483879' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:33 smithi162 ceph-mon[20279]: pgmap v692: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 480 op/s; 51/185397 objects degraded (0.028%); 15388/185397 objects misplaced (8.300%); 2.9 MiB/s, 46 objects/s recovering 2024-09-02T10:08:34.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:34.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:34.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:34.169 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:34.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2045483879' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:33 smithi151 ceph-mon[16639]: pgmap v692: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 480 op/s; 51/185397 objects degraded (0.028%); 15388/185397 objects misplaced (8.300%); 2.9 MiB/s, 46 objects/s recovering 2024-09-02T10:08:34.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:34.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:34.252 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2045483879' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:33 smithi187 ceph-mon[20072]: pgmap v692: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 480 op/s; 51/185397 objects degraded (0.028%); 15388/185397 objects misplaced (8.300%); 2.9 MiB/s, 46 objects/s recovering 2024-09-02T10:08:34.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:34.376 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:34.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:34.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:08:34.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:34.505 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:34.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:34.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:34.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:34.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:34.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:34.684 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:08:34.684 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1131 2.0 e96/92 2:021aab7d:::benchmark_data_smithi187_45882_object1130:head v 96'61, mlcod=96'58)", 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 418.78341819000002, 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077611448, 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.685 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898451+0000", 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898452+0000", 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8899999999999998e-07 2024-09-02T10:08:34.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.898480+0000", 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.817e-05 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975643+0000", 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077163063000000004 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975651+0000", 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.028e-06 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975697+0000", 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5544999999999998e-05 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.975710+0000", 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2928000000000001e-05 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.976007+0000", 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00029711499999999998 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.976063+0000", 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5810000000000003e-05 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1142 2.7 e96/92 2:e2cb33eb:::benchmark_data_smithi187_45882_object1141:head v 96'87, mlcod=96'83)", 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 418.77491584199998, 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18675391099999999, 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:08:34.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906953+0000", 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906955+0000", 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.181e-06 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:35.906991+0000", 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6523999999999998e-05 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093330+0000", 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18633878100000001 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093335+0000", 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5909999999999998e-06 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093381+0000", 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6925999999999998e-05 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093396+0000", 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4216000000000001e-05 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093682+0000", 2024-09-02T10:08:34.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028659900000000001 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.093707+0000", 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5092999999999998e-05 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:1442 2.0 e96/92 2:04aa890d:::benchmark_data_smithi187_45882_object1441:head v 96'80, mlcod=96'77)", 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 418.49250048800002, 2024-09-02T10:08:34.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087334971999999997, 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189369+0000", 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8100000000000002e-07 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.189467+0000", 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7173e-05 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276310+0000", 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.086843528000000003 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276315+0000", 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8670000000000002e-06 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276365+0000", 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9459999999999997e-05 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276382+0000", 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7045999999999999e-05 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276666+0000", 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028404500000000002 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:36.276704+0000", 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8172e-05 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.698 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.699 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.699 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11303 2.7 e98/92 2:eeb591e6:::benchmark_data_smithi187_45882_object11302:head v 98'755, mlcod=98'751)", 2024-09-02T10:08:34.699 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:08:34.699 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 412.367522318, 2024-09-02T10:08:34.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.117126889, 2024-09-02T10:08:34.700 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.700 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.700 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.700 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:08:34.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314347+0000", 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314348+0000", 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8599999999999998e-07 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.314428+0000", 2024-09-02T10:08:34.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0412000000000002e-05 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430860+0000", 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.116432137 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430866+0000", 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.083e-06 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430918+0000", 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.215e-05 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430932+0000", 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3591000000000001e-05 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.431445+0000", 2024-09-02T10:08:34.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051315900000000001 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.431474+0000", 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8671e-05 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11334 2.4 e98/92 2:22fc355e:::benchmark_data_smithi187_45882_object11333:head v 98'688, mlcod=98'684)", 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:08:34.704 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 412.328201912, 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.072501649000000001, 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353667+0000", 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353668+0000", 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5899999999999995e-07 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.353720+0000", 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1539999999999998e-05 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425505+0000", 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.071785287000000003 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425512+0000", 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2980000000000003e-06 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425576+0000", 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3614000000000005e-05 2024-09-02T10:08:34.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.425598+0000", 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2076e-05 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426142+0000", 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00054422300000000004 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426169+0000", 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6852e-05 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11352 2.4 e98/92 2:2477406b:::benchmark_data_smithi187_45882_object11351:head v 98'690, mlcod=98'686)", 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 412.255191825, 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.11620186, 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.709 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426677+0000", 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426679+0000", 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2580000000000001e-06 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.426733+0000", 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4398999999999999e-05 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542090+0000", 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.115357134 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542105+0000", 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4596999999999999e-05 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542170+0000", 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5376999999999994e-05 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542194+0000", 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3611000000000001e-05 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542825+0000", 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00063158400000000003 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.542879+0000", 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3900000000000002e-05 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.713 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11358 2.7 e98/92 2:e5a3134d:::benchmark_data_smithi187_45882_object11357:head v 98'756, mlcod=98'753)", 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 412.25107981100001, 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078694929999999996, 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430789+0000", 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430790+0000", 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7499999999999999e-07 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.430846+0000", 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5544999999999998e-05 2024-09-02T10:08:34.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508940+0000", 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078093869999999996 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508946+0000", 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7120000000000003e-06 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.508990+0000", 2024-09-02T10:08:34.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4095999999999999e-05 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509005+0000", 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5000999999999999e-05 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509456+0000", 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00045036799999999998 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.509484+0000", 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8563000000000001e-05 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11782 2.4 e98/92 2:2417a9e9:::benchmark_data_smithi187_45882_object11781:head v 98'718, mlcod=98'714)", 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 411.93887681899997, 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12480622099999999, 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.718 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742992+0000", 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.742993+0000", 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2399999999999997e-07 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.743050+0000", 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6356e-05 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867097+0000", 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.124047 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867103+0000", 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0399999999999998e-06 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867172+0000", 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8996000000000001e-05 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867187+0000", 2024-09-02T10:08:34.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5218999999999999e-05 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867764+0000", 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00057719100000000001 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.867799+0000", 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4694999999999999e-05 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11788 2.7 e98/92 2:ee0a4980:::benchmark_data_smithi187_45882_object11787:head v 98'788, mlcod=98'785)", 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 411.93159304800002, 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.134742468, 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750276+0000", 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750278+0000", 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5430000000000001e-06 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.750323+0000", 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4706000000000002e-05 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884234+0000", 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13391187800000001 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884240+0000", 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4190000000000001e-06 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884280+0000", 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0012000000000002e-05 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884316+0000", 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6318e-05 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.884977+0000", 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00066127600000000003 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.885019+0000", 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1316000000000002e-05 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11794 2.4 e98/92 2:22bc5fbb:::benchmark_data_smithi187_45882_object11793:head v 98'719, mlcod=98'715)", 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 411.92452020899998, 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.125805317, 2024-09-02T10:08:34.727 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757349+0000", 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757350+0000", 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5300000000000003e-07 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.757408+0000", 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8291000000000001e-05 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882534+0000", 2024-09-02T10:08:34.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.125126182 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882540+0000", 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0689999999999998e-06 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882583+0000", 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2962e-05 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.882599+0000", 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5718000000000001e-05 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.883121+0000", 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052199199999999999 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.883154+0000", 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3349999999999997e-05 2024-09-02T10:08:34.731 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:11897 2.4 e98/92 2:2a9822cc:::benchmark_data_smithi187_45882_object11896:head v 98'727, mlcod=98'723)", 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 411.72283999799998, 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088729108000000001, 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.732 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:08:34.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959029+0000", 2024-09-02T10:08:34.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959030+0000", 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0200000000000001e-07 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.959076+0000", 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5896000000000002e-05 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047130+0000", 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088054190000000004 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047135+0000", 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2379999999999997e-06 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047218+0000", 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2892000000000005e-05 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047233+0000", 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5126999999999999e-05 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047729+0000", 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000495847 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.047758+0000", 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9116e-05 2024-09-02T10:08:34.736 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.737 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.737 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.740 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:13212 2.0 e98/92 2:00f0b824:::benchmark_data_smithi187_45882_object13211:head v 98'832, mlcod=98'829)", 2024-09-02T10:08:34.740 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:08:34.740 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 410.91015505600001, 2024-09-02T10:08:34.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.075308911000000006, 2024-09-02T10:08:34.740 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.740 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.740 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771714+0000", 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771715+0000", 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1599999999999999e-06 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.771775+0000", 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9979999999999998e-05 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846360+0000", 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.074584792999999996 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846367+0000", 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1489999999999997e-06 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846446+0000", 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8652999999999994e-05 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846461+0000", 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5254e-05 2024-09-02T10:08:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.846996+0000", 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053431400000000003 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:43.847023+0000", 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7608000000000001e-05 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:13565 2.0 e98/92 2:0460cc71:::benchmark_data_smithi187_45882_object13564:head v 98'854, mlcod=98'850)", 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 410.62064258200002, 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29179487700000001, 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061227+0000", 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9400000000000005e-07 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.061284+0000", 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6251000000000001e-05 2024-09-02T10:08:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352274+0000", 2024-09-02T10:08:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29099082599999998 2024-09-02T10:08:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352280+0000", 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1950000000000002e-06 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352346+0000", 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5837e-05 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352360+0000", 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.433e-05 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.352980+0000", 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00062060200000000005 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:44.353022+0000", 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1142e-05 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:22449 2.0 e104/92 2:0733b693:::benchmark_data_smithi187_45882_object22448:head v 104'1397, mlcod=104'1393)", 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 403.40115947599998, 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12946363599999999, 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280710+0000", 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280711+0000", 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1800000000000005e-07 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.280779+0000", 2024-09-02T10:08:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8644000000000002e-05 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409376+0000", 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12859704799999999 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409382+0000", 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5589999999999996e-06 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409441+0000", 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8746000000000003e-05 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.409466+0000", 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5493000000000001e-05 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.410146+0000", 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00067948499999999996 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.410173+0000", 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7943e-05 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:22536 2.0 e104/92 2:01400da2:::benchmark_data_smithi187_45882_object22535:head v 104'1399, mlcod=104'1395)", 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 403.240270705, 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.077533989999999997, 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441599+0000", 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.47e-07 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.441696+0000", 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6059999999999998e-05 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518264+0000", 2024-09-02T10:08:34.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.076568862000000001 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518272+0000", 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5789999999999998e-06 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518333+0000", 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1159000000000002e-05 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.518358+0000", 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5182e-05 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.519104+0000", 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00074525100000000005 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:51.519133+0000", 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9050000000000001e-05 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.759 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:36974 2.7 e110/92 2:ea266fd4:::benchmark_data_smithi187_45882_object36973:head v 110'2332, mlcod=110'2328)", 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 392.081919954, 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.080151230000000004, 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599949+0000", 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599953+0000", 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6090000000000001e-06 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.599999+0000", 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5561e-05 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679063+0000", 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.079064375000000006 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679068+0000", 2024-09-02T10:08:34.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3619999999999997e-06 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679116+0000", 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8084999999999998e-05 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.679132+0000", 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5264000000000002e-05 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.680070+0000", 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00093846200000000004 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:02.680101+0000", 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0511999999999999e-05 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:37371 2.0 e110/92 2:00cc6826:::benchmark_data_smithi187_45882_object37370:head v 110'2365, mlcod=110'2361)", 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 391.55354147499997, 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083380836999999999, 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128328+0000", 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128329+0000", 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.207e-06 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.128397+0000", 2024-09-02T10:08:34.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7869999999999999e-05 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210301+0000", 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.081903702999999994 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210307+0000", 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.737e-06 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210368+0000", 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0795000000000003e-05 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.210383+0000", 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4538999999999999e-05 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.211539+0000", 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011564310000000001 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:03.211709+0000", 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000169555 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:67961 2.7 e120/92 2:ed776389:::benchmark_data_smithi187_45882_object67960:head v 120'4269, mlcod=120'4266)", 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 364.77316401600001, 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.095923729999999999, 2024-09-02T10:08:34.769 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908705+0000", 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908706+0000", 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1399999999999997e-07 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:29.908770+0000", 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4410000000000002e-05 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002909+0000", 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.094138870999999999 2024-09-02T10:08:34.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002918+0000", 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3489999999999992e-06 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002977+0000", 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9299999999999998e-05 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.002993+0000", 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6424000000000001e-05 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.004595+0000", 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001601725 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:30.004629+0000", 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4037e-05 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.773 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:78685 2.4 e124/92 2:26d5ebbe:::benchmark_data_smithi187_45882_object78684:head v 124'4867, mlcod=124'4864)", 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 354.23337481900001, 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.135791625, 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:08:34.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448496+0000", 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.257e-06 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448603+0000", 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106925 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.581994+0000", 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13339121900000001 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582001+0000", 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7950000000000002e-06 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582057+0000", 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6724000000000001e-05 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582074+0000", 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7037999999999998e-05 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.584242+0000", 2024-09-02T10:08:34.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021673809999999999 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.584286+0000", 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4286000000000003e-05 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:79093 2.4 e125/92 2:2f2be3a1:::benchmark_data_smithi187_45882_object79092:head v 125'4892, mlcod=125'4888)", 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 353.72894184900002, 2024-09-02T10:08:34.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16453667699999999, 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:34.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952928+0000", 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952991+0000", 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3367999999999994e-05 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:34.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115366+0000", 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16237467799999999 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115374+0000", 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6240000000000002e-06 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115428+0000", 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3885999999999999e-05 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:34.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115442+0000", 2024-09-02T10:08:34.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4438e-05 2024-09-02T10:08:34.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:34.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.117418+0000", 2024-09-02T10:08:34.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019764600000000002 2024-09-02T10:08:34.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:34.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:34.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:34.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.117464+0000", 2024-09-02T10:08:34.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5694e-05 2024-09-02T10:08:34.783 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.783 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.783 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.783 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:34.783 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:34.783 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:08:34.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:34.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:34.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:34.832 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:34.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:34.948 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:34.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:35.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:35.074 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:35.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:35.175 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:35.198 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:35.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:35.199 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:35.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:35.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:35.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:35.325 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:35.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:35.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:35.510 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:35.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:35.551 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:35 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 50/182901 objects degraded (0.027%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:35.553 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:08:35.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:35 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 50/182901 objects degraded (0.027%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:35.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:35.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:35.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:35.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:35.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:35 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 50/182901 objects degraded (0.027%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:35.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:35.777 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:35.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:35.778 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:35.843 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:35.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:35.907 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:35.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:36.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:36.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:36.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:36.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:36.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:36.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:36.194 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:36.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:36.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:36.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:36.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:36.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:08:36.358 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:08:36.358 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:08:36.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.358 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24976.0:160266 3.8 3:1238eaab:::benchmark_data_smithi187_59989_object50723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e285)", 2024-09-02T10:08:36.358 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:08:36.347504+0000", 2024-09-02T10:08:36.358 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0090454130000000004, 2024-09-02T10:08:36.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0090712580000000004, 2024-09-02T10:08:36.358 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:36.358 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:36.358 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24976", 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 160266 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.347504+0000", 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:36.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.347504+0000", 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.347505+0000", 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7300000000000005e-07 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.347508+0000", 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.72e-06 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.347517+0000", 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8340000000000005e-06 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.347522+0000", 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8799999999999999e-06 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.347557+0000", 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4839999999999998e-05 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.347635+0000", 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8008000000000003e-05 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.347696+0000", 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1346000000000005e-05 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.351459+0000", 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037627960000000001 2024-09-02T10:08:36.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.351810+0000", 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000351119 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:160257 3.4 e285/226 3:2b739413:::benchmark_data_smithi187_59989_object50714:head v 285'10322, mlcod=285'10318)", 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:08:36.330883+0000", 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.025665855000000001, 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.025738852, 2024-09-02T10:08:36.363 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.330883+0000", 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.330883+0000", 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.330884+0000", 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3300000000000001e-07 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.330890+0000", 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6489999999999996e-06 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.352857+0000", 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021967370999999999 2024-09-02T10:08:36.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.352863+0000", 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9850000000000003e-06 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.352904+0000", 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0543999999999997e-05 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.352917+0000", 2024-09-02T10:08:36.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2904e-05 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:160268 3.1 e285/283 3:8bc7c0fa:::benchmark_data_smithi187_59989_object50725:head v 285'9925, mlcod=245'3146)", 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:08:36.353457+0000", 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0030922359999999999, 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003194161, 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:36.367 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353457+0000", 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353457+0000", 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:36.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353458+0000", 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5099999999999999e-07 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353462+0000", 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.78e-06 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353476+0000", 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4015999999999999e-05 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:36.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353483+0000", 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5490000000000003e-06 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353540+0000", 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6830999999999998e-05 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353560+0000", 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0378e-05 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:36.370 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:160270 3.9 e285/226 3:94fce43a:::benchmark_data_smithi187_59989_object50727:head v 285'10021, mlcod=285'10019)", 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:08:36.353624+0000", 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0029256040000000001, 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003054041, 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:36.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353624+0000", 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353624+0000", 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353624+0000", 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5000000000000002e-07 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353629+0000", 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.921e-06 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353660+0000", 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1260000000000002e-05 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353667+0000", 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1289999999999998e-06 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:36.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353715+0000", 2024-09-02T10:08:36.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7555999999999997e-05 2024-09-02T10:08:36.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:36.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:36.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:36.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:36.353737+0000", 2024-09-02T10:08:36.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1829000000000001e-05 2024-09-02T10:08:36.374 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:36.376 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:36.376 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:36.376 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:36.376 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:08:36.376 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 4 2024-09-02T10:08:36.376 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:08:36.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:36 smithi151 ceph-mon[16639]: pgmap v693: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 480 op/s; 50/182901 objects degraded (0.027%); 15247/182901 objects misplaced (8.336%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:08:36.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:36.450 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:36.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:36.450 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:36 smithi187 ceph-mon[20072]: pgmap v693: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 480 op/s; 50/182901 objects degraded (0.027%); 15247/182901 objects misplaced (8.336%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:08:36.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:36.573 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:36.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:36.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:36 smithi162 ceph-mon[20279]: pgmap v693: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.7 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 480 op/s; 50/182901 objects degraded (0.027%); 15247/182901 objects misplaced (8.336%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:08:36.645 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:08:36.645 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:08:36.646 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-02T10:08:36.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:08:36.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:36.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:36.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:36.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:36.830 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:36.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:36.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:36.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:36.979 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:36.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:37.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:08:37.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:37.104 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:37.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:37.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:37.229 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:37.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:37.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:37.353 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:37.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:37.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:37.369 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:37.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:37.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:37.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:37.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:37.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:37.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:37.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:37.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:37.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:37.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:37.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:37.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:37.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:37.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:37.918 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:08:37.918 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:08:37.918 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:08:37.918 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:08:37.918 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:08:37.987 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:38.009 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:38.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:38.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:38.134 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:38.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:38.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:38.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:38.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:38.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:38.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:38.393 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:38.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:38.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:38.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:38.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:38.522 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:38.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:38.544 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:38 smithi187 ceph-mon[20072]: pgmap v694: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 534 op/s; 50/177726 objects degraded (0.028%); 14729/177726 objects misplaced (8.287%); 5.3 MiB/s, 85 objects/s recovering 2024-09-02T10:08:38.579 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:08:38.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:38 smithi162 ceph-mon[20279]: pgmap v694: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 534 op/s; 50/177726 objects degraded (0.028%); 14729/177726 objects misplaced (8.287%); 5.3 MiB/s, 85 objects/s recovering 2024-09-02T10:08:38.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:38.648 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:38.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:38.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:38 smithi151 ceph-mon[16639]: pgmap v694: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 534 op/s; 50/177726 objects degraded (0.028%); 14729/177726 objects misplaced (8.287%); 5.3 MiB/s, 85 objects/s recovering 2024-09-02T10:08:38.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:38.780 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:38.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:38.865 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:08:38.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:38.905 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:38.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:39.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:39.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:39.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:39.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:39.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:39.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:39.165 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:39.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:39.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:39.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:39.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:39.405 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:08:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:08:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:08:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:08:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:11302 2.c 2:30c239d5:::benchmark_data_smithi187_45882_object11301:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e98)", 2024-09-02T10:08:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.312510+0000", 2024-09-02T10:08:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 417.09060508599998, 2024-09-02T10:08:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22924329199999999, 2024-09-02T10:08:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.406 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 11302 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.312510+0000", 2024-09-02T10:08:39.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.312510+0000", 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.312511+0000", 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4270000000000001e-06 2024-09-02T10:08:39.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.312532+0000", 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0525000000000001e-05 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.312545+0000", 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3254999999999999e-05 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.312551+0000", 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4139999999999999e-06 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.312596+0000", 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4875e-05 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.312676+0000", 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9685999999999996e-05 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.312761+0000", 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4816999999999998e-05 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:39.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.313283+0000", 2024-09-02T10:08:39.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052176100000000003 2024-09-02T10:08:39.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.529425+0000", 2024-09-02T10:08:39.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21614260900000001 2024-09-02T10:08:39.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.541681+0000", 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012256065 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.541729+0000", 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7213999999999999e-05 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.541753+0000", 2024-09-02T10:08:39.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4644e-05 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:11302 2.c e98/92 ondisk, result = 0)", 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:42.313526+0000", 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 417.08958850900001, 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22823324, 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.414 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.415 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:39.415 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.313526+0000", 2024-09-02T10:08:39.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.313526+0000", 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.313527+0000", 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9999999999999997e-07 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.313530+0000", 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5950000000000001e-06 2024-09-02T10:08:39.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.541608+0000", 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22807871399999999 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.541614+0000", 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1090000000000003e-06 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.541669+0000", 2024-09-02T10:08:39.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4938999999999997e-05 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.541676+0000", 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7700000000000004e-06 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:42.541760+0000", 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3512999999999998e-05 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:54903 2.c 2:3cdee963:::benchmark_data_smithi187_45882_object54902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.130442+0000", 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 382.27267253999997, 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.264513322, 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 54903 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.419 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130442+0000", 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130442+0000", 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130444+0000", 2024-09-02T10:08:39.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.418e-06 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130465+0000", 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1396e-05 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130479+0000", 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4222999999999999e-05 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130487+0000", 2024-09-02T10:08:39.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.678e-06 2024-09-02T10:08:39.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130542+0000", 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5089000000000001e-05 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130628+0000", 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6173000000000003e-05 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:08:39.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130715+0000", 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6714999999999999e-05 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.132228+0000", 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0015130740000000001 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.135665+0000", 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034371010000000001 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.394891+0000", 2024-09-02T10:08:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25922587499999999 2024-09-02T10:08:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.394927+0000", 2024-09-02T10:08:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5917000000000002e-05 2024-09-02T10:08:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.394956+0000", 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8663e-05 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:54922 2.a 2:5831fc15:::benchmark_data_smithi187_45882_object54921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.148488+0000", 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 382.25462671499997, 2024-09-02T10:08:39.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23997639400000001, 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 54922 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148488+0000", 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148488+0000", 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148490+0000", 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1459999999999999e-06 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148519+0000", 2024-09-02T10:08:39.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8144e-05 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148537+0000", 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8776000000000001e-05 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148547+0000", 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4520000000000003e-06 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148603+0000", 2024-09-02T10:08:39.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6718000000000001e-05 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148725+0000", 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121307 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148875+0000", 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014978999999999999 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:39.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.150687+0000", 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001812241 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.150836+0000", 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014881200000000001 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.388375+0000", 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23753954699999999 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.388426+0000", 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1286999999999999e-05 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.388465+0000", 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8173999999999997e-05 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:55077 2.b e113/92 2:d13d7ad6:::benchmark_data_smithi187_45882_object55076:head v 113'3410, mlcod=113'3407)", 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.498205+0000", 2024-09-02T10:08:39.432 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.90491044200002, 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27578873599999998, 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498205+0000", 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498205+0000", 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498205+0000", 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8899999999999998e-07 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498229+0000", 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3920000000000001e-05 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.772173+0000", 2024-09-02T10:08:39.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27394390200000002 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.772181+0000", 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1890000000000007e-06 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.772247+0000", 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5660000000000005e-05 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.772271+0000", 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4040999999999999e-05 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.773959+0000", 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001688457 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.773993+0000", 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3778000000000001e-05 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.436 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPush(2.0 137/130 [PushOp(2:05d70a15:::benchmark_data_smithi187_45882_object14416:head, version: 98'898, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:05d70a15:::benchmark_data_smithi187_45882_object14416:head@98'898, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:05.850569+0000", 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 333.55254600500001, 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.474171917, 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.850569+0000", 2024-09-02T10:08:39.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.850569+0000", 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.850570+0000", 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7700000000000004e-07 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.850593+0000", 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3478000000000001e-05 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.322911+0000", 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.472317875 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.322941+0000", 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9618999999999999e-05 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.323115+0000", 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017406100000000001 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.323124+0000", 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6730000000000004e-06 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.324741+0000", 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001616534 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.440 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPush(2.e 156/153 [PushOp(2:78cd3671:::benchmark_data_smithi187_45882_object29812:head, version: 105'1897, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:78cd3671:::benchmark_data_smithi187_45882_object29812:head@105'1897, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:45.773857+0000", 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 293.62925831699999, 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23415799800000001, 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:45.773857+0000", 2024-09-02T10:08:39.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:45.773857+0000", 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:45.773857+0000", 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5199999999999996e-07 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:45.773887+0000", 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9974000000000001e-05 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:46.006264+0000", 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.232376527 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:46.006273+0000", 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5950000000000005e-06 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:46.006386+0000", 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011239 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:46.006401+0000", 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5145e-05 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:46.008015+0000", 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0016137149999999999 2024-09-02T10:08:39.444 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:145583 2.c 2:308f5bfe:::benchmark_data_smithi187_45882_object47832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:22.433050+0000", 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 256.97006534899998, 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22512525999999999, 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:39.445 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 145583 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.433050+0000", 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.433050+0000", 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.433051+0000", 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.857e-06 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.433058+0000", 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.054e-06 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.433068+0000", 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0964e-05 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.433076+0000", 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1740000000000003e-06 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.433136+0000", 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0318000000000001e-05 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.433274+0000", 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013819 2024-09-02T10:08:39.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.433359+0000", 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4529000000000003e-05 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.440001+0000", 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0066422069999999998 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.446173+0000", 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0061725829999999997 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.658067+0000", 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.211893742 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.658126+0000", 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8591000000000002e-05 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.658175+0000", 2024-09-02T10:08:39.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9051000000000002e-05 2024-09-02T10:08:39.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.451 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.451 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:145587 2.c 2:3a6c68f7:::benchmark_data_smithi187_45882_object47836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-09-02T10:08:39.451 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:22.434248+0000", 2024-09-02T10:08:39.451 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 256.96886653799999, 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.224040355, 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 145587 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.434248+0000", 2024-09-02T10:08:39.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.434248+0000", 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.434250+0000", 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1069999999999999e-06 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.434254+0000", 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3000000000000003e-06 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.434265+0000", 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1311999999999999e-05 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.434272+0000", 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5420000000000002e-06 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.436759+0000", 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00248731 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.436858+0000", 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9117000000000007e-05 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.436930+0000", 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1939000000000004e-05 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.440129+0000", 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031990510000000001 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.454920+0000", 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014791067 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.658219+0000", 2024-09-02T10:08:39.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20329882399999999 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.658253+0000", 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3515e-05 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:22.658289+0000", 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6270999999999999e-05 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:156211 2.a 2:582e1c60:::benchmark_data_smithi187_45882_object58460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:43.251374+0000", 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 236.15174055099999, 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30373849200000003, 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 156211 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.458 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251374+0000", 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251374+0000", 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251375+0000", 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3200000000000003e-07 2024-09-02T10:08:39.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251380+0000", 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4689999999999999e-06 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251389+0000", 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9889999999999998e-06 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251395+0000", 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7599999999999999e-06 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251433+0000", 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8529999999999999e-05 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251512+0000", 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8590999999999993e-05 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251577+0000", 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5259999999999995e-05 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.254848+0000", 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003271178 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.254978+0000", 2024-09-02T10:08:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:39.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:39.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:39.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013031100000000001 2024-09-02T10:08:39.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555047+0000", 2024-09-02T10:08:39.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30006905299999997 2024-09-02T10:08:39.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555082+0000", 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4354999999999999e-05 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555113+0000", 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1063999999999999e-05 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:156211 2.a e188/92 ondisk, result = 0)", 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:43.255213+0000", 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 236.14790182799999, 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29990493899999998, 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.255213+0000", 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.255213+0000", 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.255214+0000", 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6700000000000003e-07 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.255220+0000", 2024-09-02T10:08:39.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0449999999999997e-06 2024-09-02T10:08:39.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.554994+0000", 2024-09-02T10:08:39.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.299774394 2024-09-02T10:08:39.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.554999+0000", 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.186e-06 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555038+0000", 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8933000000000001e-05 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555044+0000", 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.327e-06 2024-09-02T10:08:39.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555118+0000", 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4387000000000005e-05 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:158092 2.c 2:360f665f:::benchmark_data_smithi187_45882_object60341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:46.645236+0000", 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 232.75787849899999, 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22514566699999999, 2024-09-02T10:08:39.471 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 158092 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.645236+0000", 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.645236+0000", 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.645238+0000", 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3659999999999999e-06 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.645242+0000", 2024-09-02T10:08:39.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.053e-06 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.645252+0000", 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8479999999999996e-06 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.645259+0000", 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0389999999999999e-06 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.645313+0000", 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4199000000000001e-05 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.645399+0000", 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6042000000000001e-05 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.645474+0000", 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5271999999999994e-05 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.649066+0000", 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035919810000000002 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.657613+0000", 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0085471509999999994 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.870284+0000", 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21267063 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.870340+0000", 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5519999999999997e-05 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:46.870382+0000", 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2565999999999999e-05 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.477 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:194200 2.a 2:5976d7b0:::benchmark_data_smithi187_45882_object96449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:40.797959+0000", 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 178.60515550100001, 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57870098699999994, 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194200 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797959+0000", 2024-09-02T10:08:39.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797959+0000", 2024-09-02T10:08:39.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797961+0000", 2024-09-02T10:08:39.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8670000000000001e-06 2024-09-02T10:08:39.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797969+0000", 2024-09-02T10:08:39.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5510000000000002e-06 2024-09-02T10:08:39.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797987+0000", 2024-09-02T10:08:39.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7691000000000001e-05 2024-09-02T10:08:39.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797995+0000", 2024-09-02T10:08:39.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7749999999999998e-06 2024-09-02T10:08:39.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.798058+0000", 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2309000000000003e-05 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.798148+0000", 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0781999999999999e-05 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.798225+0000", 2024-09-02T10:08:39.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6240000000000002e-05 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.801976+0000", 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003751009 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802761+0000", 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00078486799999999998 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376548+0000", 2024-09-02T10:08:39.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57378783700000002 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376616+0000", 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7465000000000005e-05 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376660+0000", 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4592999999999998e-05 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.484 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:194200 2.a e216/92 ondisk, result = 0)", 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:40.802119+0000", 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 178.60099598100001, 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57454802699999996, 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802119+0000", 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802119+0000", 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802120+0000", 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1119999999999999e-06 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802126+0000", 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9309999999999996e-06 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376473+0000", 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57434650499999995 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376478+0000", 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8109999999999999e-06 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376533+0000", 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4347e-05 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376541+0000", 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0039999999999999e-06 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376667+0000", 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000126317 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.488 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "replica_scrub(pg: 2.b,from:0'0,to:217'12114,epoch:217/92,start:2:d33335b9:::benchmark_data_smithi187_45882_object97191:0,end:2:daefa704:::benchmark_data_smithi187_45882_object97716:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:42.045131+0000", 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 177.35798413800001, 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.222551576, 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.045131+0000", 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.045131+0000", 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.045132+0000", 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4399999999999999e-07 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.045138+0000", 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0809999999999999e-06 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.267532+0000", 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.222394393 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.267543+0000", 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0736e-05 2024-09-02T10:08:39.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.267631+0000", 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.763e-05 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.267644+0000", 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3797e-05 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:42.267682+0000", 2024-09-02T10:08:39.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8095e-05 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:12905 3.9 e233/226 3:91cb3b60:::benchmark_data_smithi187_59989_object12904:head v 233'820, mlcod=233'816)", 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:06:11.039079+0000", 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 148.364035473, 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.222962982, 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.039079+0000", 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.039079+0000", 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.039080+0000", 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7100000000000001e-07 2024-09-02T10:08:39.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.039104+0000", 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3414e-05 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.261271+0000", 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.222167738 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.261276+0000", 2024-09-02T10:08:39.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9180000000000002e-06 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.261342+0000", 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5541000000000001e-05 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.261359+0000", 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7408e-05 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.262007+0000", 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00064818199999999997 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.262042+0000", 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5009999999999999e-05 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:12910 3.9 e233/226 3:97976269:::benchmark_data_smithi187_59989_object12909:head v 233'821, mlcod=233'817)", 2024-09-02T10:08:39.497 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:06:11.042431+0000", 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 148.36068372400001, 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22110782000000001, 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.042431+0000", 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.042431+0000", 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.042432+0000", 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5600000000000005e-07 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.042452+0000", 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9853000000000001e-05 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.262960+0000", 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22050778700000001 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.262966+0000", 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2430000000000002e-06 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.262998+0000", 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2132999999999999e-05 2024-09-02T10:08:39.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.263010+0000", 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1749000000000001e-05 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.263506+0000", 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00049605799999999996 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.263539+0000", 2024-09-02T10:08:39.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3241000000000002e-05 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:109536 3.9 e263/226 3:9af67999:::benchmark_data_smithi187_59989_object109535:head v 263'6841, mlcod=263'6837)", 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:07:33.021899+0000", 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 66.381216120999994, 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.365039328, 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021899+0000", 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021899+0000", 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021900+0000", 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2799999999999995e-07 2024-09-02T10:08:39.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021935+0000", 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4860999999999997e-05 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.384110+0000", 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36217508799999998 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.384116+0000", 2024-09-02T10:08:39.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5919999999999997e-06 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.384204+0000", 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7767000000000002e-05 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.384221+0000", 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6699000000000001e-05 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:08:39.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.386906+0000", 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026849679999999998 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.386938+0000", 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2524999999999998e-05 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:154914 3.7 e281/226)", 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:08:24.043568+0000", 2024-09-02T10:08:39.506 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.359547266, 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26404111200000002, 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.043568+0000", 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.043568+0000", 2024-09-02T10:08:39.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.043568+0000", 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2799999999999995e-07 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.043574+0000", 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.096e-06 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.301227+0000", 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25765300000000002 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.301234+0000", 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1180000000000003e-06 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.307590+0000", 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063568690000000002 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.307609+0000", 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8301e-05 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:154922 3.7 e281/226)", 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:08:24.052011+0000", 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.351103589999999, 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25561977200000002, 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:08:39.510 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.052011+0000", 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.052011+0000", 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:08:39.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.052012+0000", 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0800000000000001e-07 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.052013+0000", 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.903e-06 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.301255+0000", 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24924117300000001 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.301259+0000", 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0520000000000003e-06 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.307624+0000", 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063653850000000003 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.307631+0000", 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0509999999999999e-06 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.513 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.514 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.514 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:08:39.514 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:08:39.514 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:08:39.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:39.589 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:39.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:39.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:39.716 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:39.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:39.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:39.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:39.842 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:39.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:39.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:39.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:39.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:39.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:39.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:40.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:40.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:40.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:39 smithi151 ceph-mon[16639]: pgmap v695: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 510 op/s; 70/175872 objects degraded (0.040%); 14571/175872 objects misplaced (8.285%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:08:40.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/294794933' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:08:40.190 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:08:40.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:40.227 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:40.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:39 smithi187 ceph-mon[20072]: pgmap v695: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 510 op/s; 70/175872 objects degraded (0.040%); 14571/175872 objects misplaced (8.285%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:08:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/294794933' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:08:40.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:08:40.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:40.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:39 smithi162 ceph-mon[20279]: pgmap v695: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.6 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 510 op/s; 70/175872 objects degraded (0.040%); 14571/175872 objects misplaced (8.285%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:08:40.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/294794933' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:08:40.352 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:40.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:40.454 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:40.477 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:40.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:40.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:40.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:40.604 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:40.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:40.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:40.736 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:40.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:40.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:40.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:40.861 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:40.861 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:40.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:40.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:40.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:41.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:41.113 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:41.114 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:40 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/294794933' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:08:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:40 smithi151 ceph-mon[16639]: osdmap e286: 12 total, 12 up, 11 in 2024-09-02T10:08:41.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:41.240 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:41.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:40 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/294794933' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:08:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:40 smithi187 ceph-mon[20072]: osdmap e286: 12 total, 12 up, 11 in 2024-09-02T10:08:41.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:40 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/294794933' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:08:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:40 smithi162 ceph-mon[20279]: osdmap e286: 12 total, 12 up, 11 in 2024-09-02T10:08:41.368 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:41.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:41.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:41.492 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:41.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:41.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:41.616 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:41.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:41.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:41.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:41.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:41.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:41.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:41.872 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:41.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:41.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:41.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:41.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:42.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:42.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:42.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:41 smithi151 ceph-mon[16639]: pgmap v697: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 527 op/s; 70/173964 objects degraded (0.040%); 14571/173964 objects misplaced (8.376%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:08:42.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:42.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:41 smithi187 ceph-mon[20072]: pgmap v697: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 527 op/s; 70/173964 objects degraded (0.040%); 14571/173964 objects misplaced (8.376%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:08:42.257 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:42.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:42.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:42.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:41 smithi162 ceph-mon[20279]: pgmap v697: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.5 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 527 op/s; 70/173964 objects degraded (0.040%); 14571/173964 objects misplaced (8.376%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:08:42.359 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:42.382 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:42.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:42.383 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:42.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:42.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:42.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:42.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:42.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:42.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:42.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:42.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:42.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:42.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:42.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:42.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:42.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:42.999 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:43.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:43.024 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:43.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:43.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:43.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:43.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:42 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 70/173964 objects degraded (0.040%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:43.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:42 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 70/173964 objects degraded (0.040%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:43.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:43.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:43.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:42 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 70/173964 objects degraded (0.040%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:43.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:43.412 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:43.413 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:43.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:43.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:43.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:43.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:43.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:43.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:43.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:43.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:43.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:43.799 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:43.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:43.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:43.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:43.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:44.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:44.052 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:44.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:44.154 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:44.177 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:44.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:44.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:43 smithi151 ceph-mon[16639]: pgmap v698: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 529 op/s; 70/169518 objects degraded (0.041%); 14062/169518 objects misplaced (8.295%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:08:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:43 smithi187 ceph-mon[20072]: pgmap v698: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 529 op/s; 70/169518 objects degraded (0.041%); 14062/169518 objects misplaced (8.295%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:08:44.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:44.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:44.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:44.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:44.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:43 smithi162 ceph-mon[20279]: pgmap v698: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 529 op/s; 70/169518 objects degraded (0.041%); 14062/169518 objects misplaced (8.295%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:08:44.401 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:44.426 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:44.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:44.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:44.550 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:44.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:44.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:44.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:44.675 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:44.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:44.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:44.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:44.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:44.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:44.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:44.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:45.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:45.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:45.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:45.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:45.193 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:45.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:45.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:45.319 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:45.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:45.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:45.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:45.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:45.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:45.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:45.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:45.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:45.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:45.700 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:45.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:45.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:45.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:45.828 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:45.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:45.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:45.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:46.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:46.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:46.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:46.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:46.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:46.214 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:46.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:46.316 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:46.338 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:46.339 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:46.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:46.462 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:46.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:46 smithi187 ceph-mon[20072]: pgmap v699: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 527 op/s; 70/167067 objects degraded (0.042%); 13906/167067 objects misplaced (8.324%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:08:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:46 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:46 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:46.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:46.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:46.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:46.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:46.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:46 smithi162 ceph-mon[20279]: pgmap v699: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 527 op/s; 70/167067 objects degraded (0.042%); 13906/167067 objects misplaced (8.324%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:08:46.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:46 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:46.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:46 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:46 smithi151 ceph-mon[16639]: pgmap v699: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.4 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 527 op/s; 70/167067 objects degraded (0.042%); 13906/167067 objects misplaced (8.324%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:08:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:46 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:46 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:46.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:46.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:46.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:46.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:46.852 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:46.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:46.954 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:46.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:46.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:47.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:47.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:47.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:47.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:47.237 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:47.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:47.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:47.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:47.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:47.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:47.363 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:47.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:47.487 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:47.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:47 smithi187 ceph-mon[20072]: osdmap e287: 12 total, 12 up, 11 in 2024-09-02T10:08:47.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:47.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:47 smithi162 ceph-mon[20279]: osdmap e287: 12 total, 12 up, 11 in 2024-09-02T10:08:47.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:47.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:47.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:47 smithi151 ceph-mon[16639]: osdmap e287: 12 total, 12 up, 11 in 2024-09-02T10:08:47.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:47.737 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:47.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:47.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:47.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:47.864 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:47.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:47.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:47.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:47.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:47.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:48.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:48.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:48.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:48.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:48.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:48.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:48.362 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:48.387 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:48.388 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:48.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:48 smithi187 ceph-mon[20072]: pgmap v701: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 72/161787 objects degraded (0.045%); 13382/161787 objects misplaced (8.271%); 6.5 MiB/s, 103 objects/s recovering 2024-09-02T10:08:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:48 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 72/161787 objects degraded (0.045%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:48.512 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:48.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:48 smithi162 ceph-mon[20279]: pgmap v701: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 72/161787 objects degraded (0.045%); 13382/161787 objects misplaced (8.271%); 6.5 MiB/s, 103 objects/s recovering 2024-09-02T10:08:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:48 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 72/161787 objects degraded (0.045%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:48.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:48.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:48.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:48.652 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:48 smithi151 ceph-mon[16639]: pgmap v701: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.3 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 586 op/s; 72/161787 objects degraded (0.045%); 13382/161787 objects misplaced (8.271%); 6.5 MiB/s, 103 objects/s recovering 2024-09-02T10:08:48.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:48 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 72/161787 objects degraded (0.045%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:48.716 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:48.720 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:48.752 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:48.774 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:48.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:48.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:48.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:48.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:48.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:49.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:49.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:49.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:49.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:49.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:49.167 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:49.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:49.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:49.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:49.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:49.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:49.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:49.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:49.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:49.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:49.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:49.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:49.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:49.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:49.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:49.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:49.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:49.927 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:49.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:49.953 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:50.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:50.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:50.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:50.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:50.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:50.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:50.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:50.302 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:50.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:50.337 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:50.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:50.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:50.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:50.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:50.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:50 smithi187 ceph-mon[20072]: pgmap v702: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 546 op/s; 93/159663 objects degraded (0.058%); 13223/159663 objects misplaced (8.282%); 6.5 MiB/s, 104 objects/s recovering 2024-09-02T10:08:50.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:50.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:50 smithi162 ceph-mon[20279]: pgmap v702: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 546 op/s; 93/159663 objects degraded (0.058%); 13223/159663 objects misplaced (8.282%); 6.5 MiB/s, 104 objects/s recovering 2024-09-02T10:08:50.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:50.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:50 smithi151 ceph-mon[16639]: pgmap v702: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 546 op/s; 93/159663 objects degraded (0.058%); 13223/159663 objects misplaced (8.282%); 6.5 MiB/s, 104 objects/s recovering 2024-09-02T10:08:50.701 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:50.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:08:50.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:50.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:50.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:50.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:50.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:50.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:50.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:50.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:50.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:51.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:51.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:51.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:51.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:51.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:51.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:51.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:51.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:51.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:51.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:51.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:51.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:51.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:51.521 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:51 smithi151 ceph-mon[16639]: osdmap e288: 12 total, 12 up, 11 in 2024-09-02T10:08:51.525 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:51.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:51.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:51 smithi162 ceph-mon[20279]: osdmap e288: 12 total, 12 up, 11 in 2024-09-02T10:08:51.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:51.651 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:51.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:51.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:51 smithi187 ceph-mon[20072]: osdmap e288: 12 total, 12 up, 11 in 2024-09-02T10:08:51.775 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:51.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:51.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:51.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:51.903 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:51.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:52.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:52.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:52.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:52.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:52.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:52.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:52.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:52.252 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:08:52.252 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 288/283 [PushReplyOp(3:7b51765f:::benchmark_data_smithi187_59989_object59290:head)])", 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:52.225021+0000", 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.025060764999999999, 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.025086071000000001, 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.225021+0000", 2024-09-02T10:08:52.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.225021+0000", 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.225022+0000", 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6029999999999999e-06 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:08:52.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.225030+0000", 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6890000000000004e-06 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.225051+0000", 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1141999999999998e-05 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.225060+0000", 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1120000000000002e-06 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.255 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 288/283 [PushReplyOp(3:7b51765f:::benchmark_data_smithi187_59989_object59290:head)])", 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:52.230132+0000", 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.019949317000000001, 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020018424999999999, 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:08:52.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.230132+0000", 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.230132+0000", 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.230134+0000", 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5230000000000001e-06 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:08:52.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.230142+0000", 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8589999999999994e-06 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.230197+0000", 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4593999999999999e-05 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.230202+0000", 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2320000000000001e-06 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.258 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:168815 3.e 3.97cb433e (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e288)", 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:52.233460+0000", 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.016620942, 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016731670000000001, 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:08:52.259 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168815 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.233460+0000", 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.233460+0000", 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.233462+0000", 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7269999999999999e-06 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.233467+0000", 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3539999999999999e-06 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.233477+0000", 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3610000000000007e-06 2024-09-02T10:08:52.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.233484+0000", 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2629999999999997e-06 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:168842 3.1 3:8eda046b:::benchmark_data_smithi187_59989_object59299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:52.243195+0000", 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.006886601, 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.007050143, 2024-09-02T10:08:52.262 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168842 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:08:52.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.243195+0000", 2024-09-02T10:08:52.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.243195+0000", 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.243197+0000", 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6450000000000001e-06 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.243206+0000", 2024-09-02T10:08:52.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5839999999999995e-06 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.243222+0000", 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5974999999999999e-05 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.243232+0000", 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6949999999999995e-06 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:08:52.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.245516+0000", 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002283995 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.245625+0000", 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010902499999999999 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.245800+0000", 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175558 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:52.267 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:168844 3.6 3.241f6de6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-02T10:08:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:52.244837+0000", 2024-09-02T10:08:52.268 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0052441639999999999, 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054670780000000002, 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168844 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244837+0000", 2024-09-02T10:08:52.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244837+0000", 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244839+0000", 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5149999999999999e-06 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:08:52.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244845+0000", 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2090000000000002e-06 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244868+0000", 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2881999999999999e-05 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244876+0000", 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6719999999999993e-06 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:52.271 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:168845 3.6 3.e2578156 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:52.244892+0000", 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0051892850000000001, 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054552050000000003, 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168845 2024-09-02T10:08:52.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244892+0000", 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244892+0000", 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:08:52.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244893+0000", 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6800000000000001e-07 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244895+0000", 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.599e-06 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244909+0000", 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4137e-05 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244915+0000", 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8019999999999997e-06 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:168847 3.1 3.b98b79b1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:52.244940+0000", 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0051410070000000004, 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054500089999999996, 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:08:52.275 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168847 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244940+0000", 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244940+0000", 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244941+0000", 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8800000000000003e-07 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244944+0000", 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0299999999999998e-06 2024-09-02T10:08:52.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244952+0000", 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2220000000000002e-06 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.244958+0000", 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4210000000000003e-06 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.278 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:168768 3.e e288/283)", 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:52.247038+0000", 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0030428830000000001, 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033937260000000001, 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.247038+0000", 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:08:52.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.247038+0000", 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.247040+0000", 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.234e-06 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.247046+0000", 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0970000000000002e-06 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.247067+0000", 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1299999999999999e-05 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.247076+0000", 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2739999999999996e-06 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:168768 3.e 3:7e7764cd:::benchmark_data_smithi187_59989_object59225:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e288)", 2024-09-02T10:08:52.281 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:52.198613+0000", 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.051467884999999998, 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.05186056, 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168768 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:08:52.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198613+0000", 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198613+0000", 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198615+0000", 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5629999999999999e-06 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198621+0000", 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6570000000000002e-06 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198630+0000", 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5570000000000001e-06 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198637+0000", 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3249999999999996e-06 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.221828+0000", 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023191485000000001 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.221907+0000", 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9017000000000006e-05 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.222005+0000", 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7625000000000001e-05 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.285 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:168822 3.6 3:67d3f3da:::benchmark_data_smithi187_59989_object59279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:52.198691+0000", 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.051389914000000002, 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051845635000000001, 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:08:52.286 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 168822 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198691+0000", 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198691+0000", 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:08:52.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198692+0000", 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8100000000000002e-07 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198695+0000", 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.224e-06 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198710+0000", 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4195999999999999e-05 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.198715+0000", 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4779999999999999e-06 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.226054+0000", 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027338995000000001 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.226185+0000", 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013122399999999999 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.226311+0000", 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012579799999999999 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:52.233372+0000", 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070610860000000003 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:08:52.290 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 10 2024-09-02T10:08:52.291 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:08:52.291 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:52.315 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:52.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:52.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:52 smithi151 ceph-mon[16639]: pgmap v704: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 495 op/s; 93/157614 objects degraded (0.059%); 13223/157614 objects misplaced (8.389%); 5.0 MiB/s, 80 objects/s recovering 2024-09-02T10:08:52.439 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:52.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:52.541 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:52.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:52.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:52.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:52.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:52 smithi162 ceph-mon[20279]: pgmap v704: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 495 op/s; 93/157614 objects degraded (0.059%); 13223/157614 objects misplaced (8.389%); 5.0 MiB/s, 80 objects/s recovering 2024-09-02T10:08:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:52 smithi187 ceph-mon[20072]: pgmap v704: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.2 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 495 op/s; 93/157614 objects degraded (0.059%); 13223/157614 objects misplaced (8.389%); 5.0 MiB/s, 80 objects/s recovering 2024-09-02T10:08:52.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:52.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:52.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:52.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:52.840 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:52.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:52.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:52.964 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:52.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:53.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:08:53.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:08:53.095 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:08:53.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:53.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:53.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:53.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:53.223 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:53.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:53.232 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:08:53.233 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:08:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 288, 2024-09-02T10:08:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:08:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:08:53.233 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:08:50.240578+0000", 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:02:47.267122+0000", 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5801984, 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "noscrub", 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:08:53.234 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 3, 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "jewel", 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:08:53.235 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:08:53.236 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:08:53.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "264", 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "241", 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:08:53.238 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:08:53.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:08:53.240 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.7000000476837158, 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.7000000476837158, 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.4800000190734863, 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.4800000190734863, 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:08:53.241 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 3, 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_1", 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:05:55.119628+0000", 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:08:53.242 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:08:53.243 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "230", 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:08:53.244 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:08:53.245 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:08:53.246 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.2400000095367432, 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.2400000095367432, 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.059999942779541, 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.059999942779541, 2024-09-02T10:08:53.247 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.8931121826171875, 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.248 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 283, 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.249 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:08:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:08:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.250 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.251 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.252 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.253 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.254 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 282, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.255 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.256 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.257 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.258 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.259 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.260 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 282, 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.261 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.262 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.263 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.264 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:08:53.265 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.83477783203125, 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.266 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 283, 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.267 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:08:53.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.271 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.6429595947265625, 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.272 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 243, 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.273 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.274 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:08:53.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.277 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:08:53.278 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 243, 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:08:53.279 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:08:53.280 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:08:53.281 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:08:53.282 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:08:53.283 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 282, 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.284 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.285 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.287 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.288 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:08:53.288 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:08:53.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.288 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.289 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 282, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.290 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.292 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.293 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:08:53.294 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:08:53.295 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 282, 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.299 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:08:53.300 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.301 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 241, 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:08:53.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:08:53.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:08:53.304 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:08:53.305 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.306 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.307 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 124, 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.308 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.309 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.310 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.311 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.313 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:08:53.313 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:08:53.313 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:08:53.313 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:08:53.313 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.555877685546875, 2024-09-02T10:08:53.313 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:08:53.313 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:08:53.313 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:08:53.313 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 243, 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.314 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.315 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:08:53.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:08:53.318 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:07:59.075621+0000", 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:08:10.078089+0000", 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:08:53.320 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:08:45.283102+0000", 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.321 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:08:08.825777+0000", 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:07:37.884256+0000", 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:08:24.204028+0000", 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.323 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:08:23.730716+0000", 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.324 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:07:54.831646+0000", 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:08:03.511504+0000", 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:08:06.121230+0000", 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:08:53.326 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 65536, 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:08:49.852283+0000", 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:08:53.327 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:08:18.655771+0000", 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [], 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [ 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "3.1", 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-09-02T10:08:53.328 INFO:teuthology.orchestra.run.smithi151.stdout: 11 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "3.7", 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: 3, 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: 7 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "3.e", 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:08:53.329 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: 3 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:08:53.330 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:08:53.332 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:08:53.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:53.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:53.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:53.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3048236152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3302059604' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:08:53.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:53.497 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:53.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:53.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3048236152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3302059604' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:08:53.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3048236152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:08:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3302059604' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:08:53.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:53.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:53.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:53.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:53.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:53.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:53.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:53.882 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:53.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:53.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:54.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:54.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:54.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:54.152 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:54.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:54.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:54.277 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:54.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:54.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:54.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:54.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:54.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:54 smithi187 ceph-mon[20072]: pgmap v705: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 588 op/s; 123/152928 objects degraded (0.080%); 12700/152928 objects misplaced (8.305%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:08:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:54 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 123/152928 objects degraded (0.080%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:54.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:54.543 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:54.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:54.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:54 smithi162 ceph-mon[20279]: pgmap v705: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 588 op/s; 123/152928 objects degraded (0.080%); 12700/152928 objects misplaced (8.305%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:08:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:54 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 123/152928 objects degraded (0.080%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:54.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:54.669 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:54.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:54 smithi151 ceph-mon[16639]: pgmap v705: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 588 op/s; 123/152928 objects degraded (0.080%); 12700/152928 objects misplaced (8.305%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:08:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:54 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 123/152928 objects degraded (0.080%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:54.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:54.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:54.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:54.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:08:54.907 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:54.929 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:54.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:55.031 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:55.052 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:55.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:55.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:55.154 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:55.177 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:55.179 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:55.270 INFO:tasks.thrashosds.thrasher:j is {'epoch': 288, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:08:50.240578+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:02:47.267122+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.700000047683716, 'score_stable': 2.700000047683716, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.4800000190734863, 'raw_score_stable': 2.4800000190734863, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-02T10:05:55.119628+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '230', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 0.8931121826171875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 283, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 282, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 282, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 0.83477783203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 283, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 1, 'weight': 0.6429595947265625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 282, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 282, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 282, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 241, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 124, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 1, 'weight': 0.555877685546875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:07:59.075621+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:08:10.078089+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:08:45.283102+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:08:08.825777+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:07:37.884256+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:08:24.204028+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:08:23.730716+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:07:54.831646+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:08:03.511504+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:08:06.121230+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-02T10:08:49.852283+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:08:18.655771+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.1', 'osds': [0, 11]}, {'pgid': '3.7', 'osds': [3, 7]}, {'pgid': '3.e', 'osds': [0, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:08:55.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:08:55.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:55.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:55.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:55.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:55.309 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:55.363 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:08:55.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:55.428 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:55.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:55.509 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:55 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:55.509 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:55 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:55.509 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:55 smithi151 ceph-mon[16639]: osdmap e289: 12 total, 12 up, 11 in 2024-09-02T10:08:55.530 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:55.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:55.552 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:55.552 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:55.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:55 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:55 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:55 smithi162 ceph-mon[20279]: osdmap e289: 12 total, 12 up, 11 in 2024-09-02T10:08:55.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:55.674 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:55.675 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:55 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:55 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:55 smithi187 ceph-mon[20072]: osdmap e289: 12 total, 12 up, 11 in 2024-09-02T10:08:55.775 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:08:55.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:55.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:08:55.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:55.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:55.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:56.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:56.065 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:56.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:56.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:56.189 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:56.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:56.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:08:56.291 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:56.312 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:56.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:08:56.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:56 smithi151 ceph-mon[16639]: pgmap v707: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 471 op/s; 133/150456 objects degraded (0.088%); 12538/150456 objects misplaced (8.333%); 4.8 MiB/s, 76 objects/s recovering 2024-09-02T10:08:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:56 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:08:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:56 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:08:56.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:56.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:56.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:56.574 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:56.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:56.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:56 smithi162 ceph-mon[20279]: pgmap v707: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 471 op/s; 133/150456 objects degraded (0.088%); 12538/150456 objects misplaced (8.333%); 4.8 MiB/s, 76 objects/s recovering 2024-09-02T10:08:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:56 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:08:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:56 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:08:56.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:56.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:56.702 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:56.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:56 smithi187 ceph-mon[20072]: pgmap v707: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.1 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 471 op/s; 133/150456 objects degraded (0.088%); 12538/150456 objects misplaced (8.333%); 4.8 MiB/s, 76 objects/s recovering 2024-09-02T10:08:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:56 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:08:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:56 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:08:56.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:56.828 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:56.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:56.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:56.888 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:08:56.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:56.953 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:56.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:57.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:57.077 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:57.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:57.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:57.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:57.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:57.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:57.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:57.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:57.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:57.462 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:57.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:57.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:57.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:57.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:57.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/723179129' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:57.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:57 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:57 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:57 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:57 smithi162 ceph-mon[20279]: osdmap e290: 12 total, 12 up, 11 in 2024-09-02T10:08:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/723179129' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:57 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:57 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:57 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:57 smithi151 ceph-mon[16639]: osdmap e290: 12 total, 12 up, 11 in 2024-09-02T10:08:57.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:57.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:57.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/723179129' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:57 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:08:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:57 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:08:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:57 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:08:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:57 smithi187 ceph-mon[20072]: osdmap e290: 12 total, 12 up, 11 in 2024-09-02T10:08:57.814 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:57.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:57.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:57.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:57.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:57.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:57.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:58.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:08:58.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:58.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:58.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:08:58.179 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:08:58.179 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:08:58.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:08:58.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:58.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:08:58.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:58.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:58.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:58 smithi151 ceph-mon[16639]: pgmap v709: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 133/145695 objects degraded (0.091%); 12050/145695 objects misplaced (8.271%); 7.2 MiB/s, 115 objects/s recovering 2024-09-02T10:08:58.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:08:58.488 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:58.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:58.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:08:58.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:08:58.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:58 smithi162 ceph-mon[20279]: pgmap v709: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 133/145695 objects degraded (0.091%); 12050/145695 objects misplaced (8.271%); 7.2 MiB/s, 115 objects/s recovering 2024-09-02T10:08:58.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:58.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:08:58.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:58.741 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:58.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:58 smithi187 ceph-mon[20072]: pgmap v709: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 3.0 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 620 op/s; 133/145695 objects degraded (0.091%); 12050/145695 objects misplaced (8.271%); 7.2 MiB/s, 115 objects/s recovering 2024-09-02T10:08:58.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:58.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:58.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:58.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:08:59.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:59.006 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:59.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:08:59.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:59.139 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:59.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:59.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:59.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:08:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:59.267 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:59.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:08:59.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:08:59.398 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:59.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:08:59.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:08:59.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:08:59.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:59.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:08:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 133/145695 objects degraded (0.091%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:59.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:08:59 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:08:59.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:08:59.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:59.657 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:59.657 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:08:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 133/145695 objects degraded (0.091%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:08:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:08:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 133/145695 objects degraded (0.091%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:08:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:08:59 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:08:59.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:08:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:59.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:08:59.782 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:08:59.883 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:08:59.906 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:08:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:08:59.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:00.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:00.029 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:00.030 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:00.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:00.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:00.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:00.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:00.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:00.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:00.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:00.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:00.417 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:00.417 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:00.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:00.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:00.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:00.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:00 smithi162 ceph-mon[20279]: pgmap v710: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 140/143820 objects degraded (0.097%); 11892/143820 objects misplaced (8.269%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:09:00.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:00.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:00.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:00.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:00.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:00 smithi151 ceph-mon[16639]: pgmap v710: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 140/143820 objects degraded (0.097%); 11892/143820 objects misplaced (8.269%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:09:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:00 smithi187 ceph-mon[20072]: pgmap v710: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 140/143820 objects degraded (0.097%); 11892/143820 objects misplaced (8.269%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:09:00.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:00.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:00.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:00.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:00.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:00.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:00.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:01.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:01.072 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:01.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:01.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:01.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:01.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:01.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:01.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:01.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:01.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:01.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:01.465 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:01.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:01.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:01.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:01.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:09:01.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:09:01.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:01.729 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:01.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:09:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:09:01.830 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:01.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:09:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:09:01.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:01.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:01.852 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:01.953 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:01.976 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:01.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:02.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:02.101 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:02.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:02.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:02.227 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:02.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:02.228 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:02.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:02.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:02.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:02.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:02.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:02.496 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:02.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:02.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:02.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:02.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:02.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:02 smithi151 ceph-mon[16639]: pgmap v711: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 461 op/s; 140/141855 objects degraded (0.099%); 11892/141855 objects misplaced (8.383%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:09:02.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:02.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:02.747 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:02.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:02 smithi187 ceph-mon[20072]: pgmap v711: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 461 op/s; 140/141855 objects degraded (0.099%); 11892/141855 objects misplaced (8.383%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:09:02.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:02 smithi162 ceph-mon[20279]: pgmap v711: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.9 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 461 op/s; 140/141855 objects degraded (0.099%); 11892/141855 objects misplaced (8.383%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:09:02.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:02.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:02.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:02.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:03.013 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:03.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:03.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:03.137 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:03.138 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:03.239 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:03.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:03.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:03.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:03.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:03.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:03.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:03.526 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:03.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:03.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:03.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:03.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:03.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:03.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:03.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:03.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:03.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:03.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:03.912 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:03.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:04.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:04.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:04.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:04.038 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:04.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:04.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:04.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:04.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:04.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:04.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:04.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:04.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:04.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:04.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:04.544 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:04.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:04.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:04.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:04.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:04.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:04 smithi151 ceph-mon[16639]: pgmap v712: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.8 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 140/137373 objects degraded (0.102%); 11359/137373 objects misplaced (8.269%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:09:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 140/137373 objects degraded (0.102%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:04 smithi187 ceph-mon[20072]: pgmap v712: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.8 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 140/137373 objects degraded (0.102%); 11359/137373 objects misplaced (8.269%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:09:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 140/137373 objects degraded (0.102%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:04.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:04.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:04.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:04.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:04.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:04 smithi162 ceph-mon[20279]: pgmap v712: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.8 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 525 op/s; 140/137373 objects degraded (0.102%); 11359/137373 objects misplaced (8.269%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:09:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 140/137373 objects degraded (0.102%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:04.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:04.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:04.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:05.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:05.060 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:05.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:05.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:05.185 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:05.187 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:05.288 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:05.318 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:05.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:05.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:05.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:05.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:05.445 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:05.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:05.570 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:05.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:05.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:05.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:05.694 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:05.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:05.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:05.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:05.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:05.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:05.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:05.948 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:05.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:05.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:06.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:06.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:06.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:06.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:06.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:06.213 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:06.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:06.214 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:06.315 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:06.338 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:06.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:06.340 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:06.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:06.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:06.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:06.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:06.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:06.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:06.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:06 smithi151 ceph-mon[16639]: pgmap v713: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 153/134784 objects degraded (0.114%); 11189/134784 objects misplaced (8.301%); 6.2 MiB/s, 98 objects/s recovering 2024-09-02T10:09:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:09:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:09:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:09:06.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:06.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:06.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:06.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:06 smithi187 ceph-mon[20072]: pgmap v713: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 153/134784 objects degraded (0.114%); 11189/134784 objects misplaced (8.301%); 6.2 MiB/s, 98 objects/s recovering 2024-09-02T10:09:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:09:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:09:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:09:06.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:06.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:06 smithi162 ceph-mon[20279]: pgmap v713: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.7 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 522 op/s; 153/134784 objects degraded (0.114%); 11189/134784 objects misplaced (8.301%); 6.2 MiB/s, 98 objects/s recovering 2024-09-02T10:09:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:09:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:09:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:09:06.849 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:06.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:06.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:06.975 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:06.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:06.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:07.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:07.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:07.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:07.201 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:07.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:07.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:07.226 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:07.327 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:07.349 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:07.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:07.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:07.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:07.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:07.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:07.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:07.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:07.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:07.601 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:07.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:07.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:07.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:07.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:07.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:07.860 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:07.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:07.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:07.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:07.984 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:07.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:07.985 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:08.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:08.110 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:08.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:08.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:08.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:08.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:08.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:08.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:08.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:08.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:08.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:08.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:08.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:08.633 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:08.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:08 smithi151 ceph-mon[16639]: pgmap v714: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 153/129474 objects degraded (0.118%); 10657/129474 objects misplaced (8.231%); 8.1 MiB/s, 130 objects/s recovering 2024-09-02T10:09:08.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:08 smithi187 ceph-mon[20072]: pgmap v714: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 153/129474 objects degraded (0.118%); 10657/129474 objects misplaced (8.231%); 8.1 MiB/s, 130 objects/s recovering 2024-09-02T10:09:08.758 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:08.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:08.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:08.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:08 smithi162 ceph-mon[20279]: pgmap v714: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 648 op/s; 153/129474 objects degraded (0.118%); 10657/129474 objects misplaced (8.231%); 8.1 MiB/s, 130 objects/s recovering 2024-09-02T10:09:08.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:08.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:08.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:08.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:08.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:09.009 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:09.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:09.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:09.111 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:09.132 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:09.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:09.133 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:09.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:09.258 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:09.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:09.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:09.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:09.386 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:09.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:09.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:09.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:09.515 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:09.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:09.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:09.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:09.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 153/129474 objects degraded (0.118%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:09.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 153/129474 objects degraded (0.118%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:09.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:09.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:09.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 153/129474 objects degraded (0.118%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:09.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:09.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:09.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:10.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:10.039 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:10.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:10.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:10.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:10.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:10.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:10.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:10.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:10.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:10.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:10.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:10.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:10.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:10.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:10.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:10.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:10.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:10.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:10.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:10.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:10 smithi151 ceph-mon[16639]: pgmap v715: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 513 op/s; 187/127203 objects degraded (0.147%); 10473/127203 objects misplaced (8.233%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:09:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:10 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:09:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:10 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:09:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:10 smithi151 ceph-mon[16639]: osdmap e291: 12 total, 12 up, 11 in 2024-09-02T10:09:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:10 smithi187 ceph-mon[20072]: pgmap v715: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 513 op/s; 187/127203 objects degraded (0.147%); 10473/127203 objects misplaced (8.233%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:09:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:10 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:09:10.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:10 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:09:10.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:10 smithi187 ceph-mon[20072]: osdmap e291: 12 total, 12 up, 11 in 2024-09-02T10:09:10.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:10.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:10.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:10.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:10 smithi162 ceph-mon[20279]: pgmap v715: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.6 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 513 op/s; 187/127203 objects degraded (0.147%); 10473/127203 objects misplaced (8.233%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:09:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:10 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:09:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:10 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:09:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:10 smithi162 ceph-mon[20279]: osdmap e291: 12 total, 12 up, 11 in 2024-09-02T10:09:10.920 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:10.948 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:10.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:11.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:11.075 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:11.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:11.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:11.202 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:11.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:11.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:11.340 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:11.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:11.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:11.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:11.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:11.534 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:09:11.556 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:09:11.577 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:11.597 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:09:11.599 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:11.601 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:11.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:11.724 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:11.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:11.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:11.848 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:11.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:11.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:11.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:11.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:11.974 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:12.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:12.096 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:12.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:12.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:12.223 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:12.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:12.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:12.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:12.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:12.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:12.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:12.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:12.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:12.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:12.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:12.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:12.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:12 smithi151 ceph-mon[16639]: pgmap v717: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.5 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 187/125070 objects degraded (0.150%); 10473/125070 objects misplaced (8.374%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:09:12.699 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:09:12.699 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:09:12.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:12.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:09:12.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:09:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:12 smithi187 ceph-mon[20072]: pgmap v717: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.5 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 187/125070 objects degraded (0.150%); 10473/125070 objects misplaced (8.374%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:09:12.756 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:12.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:12.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:12.772 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:09:12.773 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:09:12.813 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:09:12.813 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:09:12.843 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:09:12.844 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:09:12.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:12 smithi162 ceph-mon[20279]: pgmap v717: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.5 GiB data, 9.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 187/125070 objects degraded (0.150%); 10473/125070 objects misplaced (8.374%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:09:12.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:12.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:12.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:12.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:09:12.886 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:09:12.917 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:09:12.917 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:09:12.935 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:09:12.936 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:09:12.952 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:09:12.953 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:09:12.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:12.985 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:09:12.988 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:09:13.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:13.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:13.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:13.019 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:09:13.019 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:09:13.053 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:09:13.054 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:09:13.089 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:09:13.090 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:09:13.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:13.142 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:13.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:13.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:13.269 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:13.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:13.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:13.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:13.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:09:13.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:13.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:13.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:13.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:13.531 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:13.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:13.654 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:13.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:13 smithi151 ceph-mon[16639]: from='client.15801 172.21.15.151:0/323287046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:09:13.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:13 smithi187 ceph-mon[20072]: from='client.15801 172.21.15.151:0/323287046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:09:13.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:13.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:13.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:13.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:13 smithi162 ceph-mon[20279]: from='client.15801 172.21.15.151:0/323287046' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:09:13.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:13.911 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:13.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:13.982 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:09:14.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:14.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:14.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:14.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:14.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:14.174 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:14.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:14.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:14.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:14.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:14.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:14.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:14.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:14.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:14.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:14.558 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:14.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:14.574 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":717,"stamp":"2024-09-02T10:09:11.182614+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2732589624,"num_objects":41690,"num_object_clones":0,"num_object_copies":125070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":10473,"num_objects_unfound":0,"num_objects_dirty":41690,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177394,"num_write_kb":7010561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3929,"num_bytes_recovered":257163264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163746,"ondisk_log_size":163746,"up":141,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":10388940,"kb_used_data":8442456,"kb_used_omap":17,"kb_used_meta":1946286,"kb_avail":1020807732,"statfs":{"total":1055945392128,"available":1045307117568,"internally_reserved":0,"allocated":8645074944,"data_stored":8642087194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17541,"internal_metadata":1992997755},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-366673920,"num_objects":-5595,"num_object_clones":0,"num_object_copies":-16785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":47,"num_objects_misplaced":-1419,"num_objects_unfound":0,"num_objects_dirty":-5595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5595,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1010,"num_bytes_recovered":66191360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002511"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":36,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.885194+0000","last_change":"2024-09-02T10:08:26.064320+0000","last_active":"2024-09-02T10:08:56.885194+0000","last_peered":"2024-09-02T10:08:56.885194+0000","last_clean":"2024-09-02T10:08:56.885194+0000","last_became_active":"2024-09-02T10:08:25.262897+0000","last_became_peered":"2024-09-02T10:08:25.262897+0000","last_unstale":"2024-09-02T10:08:56.885194+0000","last_undegraded":"2024-09-02T10:08:56.885194+0000","last_fullsized":"2024-09-02T10:08:56.885194+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:26.064259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:26.064259+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:39.500115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,9],"acting":[8,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":131,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.891048+0000","last_change":"2024-09-02T10:08:25.265285+0000","last_active":"2024-09-02T10:08:56.891048+0000","last_peered":"2024-09-02T10:08:56.891048+0000","last_clean":"2024-09-02T10:08:56.891048+0000","last_became_active":"2024-09-02T10:08:25.264923+0000","last_became_peered":"2024-09-02T10:08:25.264923+0000","last_unstale":"2024-09-02T10:08:56.891048+0000","last_undegraded":"2024-09-02T10:08:56.891048+0000","last_fullsized":"2024-09-02T10:08:56.891048+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:04.416697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:04.416697+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:13.321346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":122,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.885164+0000","last_change":"2024-09-02T10:07:59.237326+0000","last_active":"2024-09-02T10:08:56.885164+0000","last_peered":"2024-09-02T10:08:56.885164+0000","last_clean":"2024-09-02T10:08:56.885164+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:08:56.885164+0000","last_undegraded":"2024-09-02T10:08:56.885164+0000","last_fullsized":"2024-09-02T10:08:56.885164+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:59.237277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:59.237277+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:23.463935+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":122,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.885107+0000","last_change":"2024-09-02T10:08:03.295634+0000","last_active":"2024-09-02T10:08:56.885107+0000","last_peered":"2024-09-02T10:08:56.885107+0000","last_clean":"2024-09-02T10:08:56.885107+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:08:56.885107+0000","last_undegraded":"2024-09-02T10:08:56.885107+0000","last_fullsized":"2024-09-02T10:08:56.885107+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:03.295588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:03.295588+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:20.620240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":41,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-02T10:09:09.895555+0000","last_change":"2024-09-02T10:08:27.128433+0000","last_active":"2024-09-02T10:09:09.895555+0000","last_peered":"2024-09-02T10:09:09.895555+0000","last_clean":"2024-09-02T10:09:09.895555+0000","last_became_active":"2024-09-02T10:08:25.264164+0000","last_became_peered":"2024-09-02T10:08:25.264164+0000","last_unstale":"2024-09-02T10:09:09.895555+0000","last_undegraded":"2024-09-02T10:09:09.895555+0000","last_fullsized":"2024-09-02T10:09:09.895555+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:27.128380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:27.128380+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:43.068400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":122,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:57.015148+0000","last_change":"2024-09-02T10:07:39.188463+0000","last_active":"2024-09-02T10:08:57.015148+0000","last_peered":"2024-09-02T10:08:57.015148+0000","last_clean":"2024-09-02T10:08:57.015148+0000","last_became_active":"2024-09-02T10:06:34.851298+0000","last_became_peered":"2024-09-02T10:06:34.851298+0000","last_unstale":"2024-09-02T10:08:57.015148+0000","last_undegraded":"2024-09-02T10:08:57.015148+0000","last_fullsized":"2024-09-02T10:08:57.015148+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:39.188419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:39.188419+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":481,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.887781+0000","last_change":"2024-09-02T10:08:01.621452+0000","last_active":"2024-09-02T10:08:56.887781+0000","last_peered":"2024-09-02T10:08:56.887781+0000","last_clean":"2024-09-02T10:08:56.887781+0000","last_became_active":"2024-09-02T10:06:34.859627+0000","last_became_peered":"2024-09-02T10:06:34.859627+0000","last_unstale":"2024-09-02T10:08:56.887781+0000","last_undegraded":"2024-09-02T10:08:56.887781+0000","last_fullsized":"2024-09-02T10:08:56.887781+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:01.621368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:01.621368+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:10.066322+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":122,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.885268+0000","last_change":"2024-09-02T10:08:02.301918+0000","last_active":"2024-09-02T10:08:56.885268+0000","last_peered":"2024-09-02T10:08:56.885268+0000","last_clean":"2024-09-02T10:08:56.885268+0000","last_became_active":"2024-09-02T10:06:34.849680+0000","last_became_peered":"2024-09-02T10:06:34.849680+0000","last_unstale":"2024-09-02T10:08:56.885268+0000","last_undegraded":"2024-09-02T10:08:56.885268+0000","last_fullsized":"2024-09-02T10:08:56.885268+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:02.301851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:02.301851+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:10.641338+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":122,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.890871+0000","last_change":"2024-09-02T10:08:02.403736+0000","last_active":"2024-09-02T10:08:56.890871+0000","last_peered":"2024-09-02T10:08:56.890871+0000","last_clean":"2024-09-02T10:08:56.890871+0000","last_became_active":"2024-09-02T10:06:34.850895+0000","last_became_peered":"2024-09-02T10:06:34.850895+0000","last_unstale":"2024-09-02T10:08:56.890871+0000","last_undegraded":"2024-09-02T10:08:56.890871+0000","last_fullsized":"2024-09-02T10:08:56.890871+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:02.403664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:02.403664+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:13.841777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":124,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-02T10:09:09.895814+0000","last_change":"2024-09-02T10:08:02.003252+0000","last_active":"2024-09-02T10:09:09.895814+0000","last_peered":"2024-09-02T10:09:09.895814+0000","last_clean":"2024-09-02T10:09:09.895814+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:09:09.895814+0000","last_undegraded":"2024-09-02T10:09:09.895814+0000","last_fullsized":"2024-09-02T10:09:09.895814+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:02.003171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:02.003171+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:10.888013+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":368,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.891759+0000","last_change":"2024-09-02T10:07:28.918922+0000","last_active":"2024-09-02T10:08:56.891759+0000","last_peered":"2024-09-02T10:08:56.891759+0000","last_clean":"2024-09-02T10:08:56.891759+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:08:56.891759+0000","last_undegraded":"2024-09-02T10:08:56.891759+0000","last_fullsized":"2024-09-02T10:08:56.891759+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:28.918851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:28.918851+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:57.183051+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":454,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.890620+0000","last_change":"2024-09-02T10:08:01.381603+0000","last_active":"2024-09-02T10:08:56.890620+0000","last_peered":"2024-09-02T10:08:56.890620+0000","last_clean":"2024-09-02T10:08:56.890620+0000","last_became_active":"2024-09-02T10:02:48.287566+0000","last_became_peered":"2024-09-02T10:02:48.287566+0000","last_unstale":"2024-09-02T10:08:56.890620+0000","last_undegraded":"2024-09-02T10:08:56.890620+0000","last_fullsized":"2024-09-02T10:08:56.890620+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:01.381551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:01.381551+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:15.888878+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":390,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.887843+0000","last_change":"2024-09-02T10:07:31.059528+0000","last_active":"2024-09-02T10:08:56.887843+0000","last_peered":"2024-09-02T10:08:56.887843+0000","last_clean":"2024-09-02T10:08:56.887843+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:08:56.887843+0000","last_undegraded":"2024-09-02T10:08:56.887843+0000","last_fullsized":"2024-09-02T10:08:56.887843+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:31.059464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:31.059464+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":378,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:57.015275+0000","last_change":"2024-09-02T10:08:01.316267+0000","last_active":"2024-09-02T10:08:57.015275+0000","last_peered":"2024-09-02T10:08:57.015275+0000","last_clean":"2024-09-02T10:08:57.015275+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:08:57.015275+0000","last_undegraded":"2024-09-02T10:08:57.015275+0000","last_fullsized":"2024-09-02T10:08:57.015275+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:01.316149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:01.316149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:16.729683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":383,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.887751+0000","last_change":"2024-09-02T10:07:32.127860+0000","last_active":"2024-09-02T10:08:56.887751+0000","last_peered":"2024-09-02T10:08:56.887751+0000","last_clean":"2024-09-02T10:08:56.887751+0000","last_became_active":"2024-09-02T10:02:48.304718+0000","last_became_peered":"2024-09-02T10:02:48.304718+0000","last_unstale":"2024-09-02T10:08:56.887751+0000","last_undegraded":"2024-09-02T10:08:56.887751+0000","last_fullsized":"2024-09-02T10:08:56.887751+0000","mapping_epoch":130,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:32.127803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:32.127803+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"290'11086","reported_seq":11299,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:06.949634+0000","last_change":"2024-09-02T10:07:30.761771+0000","last_active":"2024-09-02T10:09:06.949634+0000","last_peered":"2024-09-02T10:09:06.949634+0000","last_clean":"2024-09-02T10:09:06.949634+0000","last_became_active":"2024-09-02T10:05:56.583663+0000","last_became_peered":"2024-09-02T10:05:56.583663+0000","last_unstale":"2024-09-02T10:09:06.949634+0000","last_undegraded":"2024-09-02T10:09:06.949634+0000","last_fullsized":"2024-09-02T10:09:06.949634+0000","mapping_epoch":226,"log_start":"233'1000","ondisk_log_start":"233'1000","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6702","last_scrub_stamp":"2024-09-02T10:07:30.761550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.761550+0000","objects_scrubbed":6632,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:09.635288+0000","scrub_duration":1999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170262528,"num_objects":2598,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11086,"num_write_kb":437888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":470,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.887832+0000","last_change":"2024-09-02T10:07:59.408627+0000","last_active":"2024-09-02T10:08:56.887832+0000","last_peered":"2024-09-02T10:08:56.887832+0000","last_clean":"2024-09-02T10:08:56.887832+0000","last_became_active":"2024-09-02T10:06:32.873327+0000","last_became_peered":"2024-09-02T10:06:32.873327+0000","last_unstale":"2024-09-02T10:08:56.887832+0000","last_undegraded":"2024-09-02T10:08:56.887832+0000","last_fullsized":"2024-09-02T10:08:56.887832+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:59.408564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:59.408564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:15.132163+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"290'11023","reported_seq":11236,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:08.500469+0000","last_change":"2024-09-02T10:07:29.219779+0000","last_active":"2024-09-02T10:09:08.500469+0000","last_peered":"2024-09-02T10:09:08.500469+0000","last_clean":"2024-09-02T10:09:08.500469+0000","last_became_active":"2024-09-02T10:05:56.583278+0000","last_became_peered":"2024-09-02T10:05:56.583278+0000","last_unstale":"2024-09-02T10:09:08.500469+0000","last_undegraded":"2024-09-02T10:09:08.500469+0000","last_fullsized":"2024-09-02T10:09:08.500469+0000","mapping_epoch":226,"log_start":"233'1000","ondisk_log_start":"233'1000","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6573","last_scrub_stamp":"2024-09-02T10:07:29.219665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:29.219665+0000","objects_scrubbed":6515,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169934848,"num_objects":2593,"num_object_clones":0,"num_object_copies":7779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11023,"num_write_kb":435712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":596,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-02T10:09:09.895619+0000","last_change":"2024-09-02T10:08:08.952478+0000","last_active":"2024-09-02T10:09:09.895619+0000","last_peered":"2024-09-02T10:09:09.895619+0000","last_clean":"2024-09-02T10:09:09.895619+0000","last_became_active":"2024-09-02T10:06:32.871027+0000","last_became_peered":"2024-09-02T10:06:32.871027+0000","last_unstale":"2024-09-02T10:09:09.895619+0000","last_undegraded":"2024-09-02T10:09:09.895619+0000","last_fullsized":"2024-09-02T10:09:09.895619+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:08.952413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:08.952413+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:22.804020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"290'10984","reported_seq":11183,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:06.926526+0000","last_change":"2024-09-02T10:07:07.774469+0000","last_active":"2024-09-02T10:09:06.926526+0000","last_peered":"2024-09-02T10:09:06.926526+0000","last_clean":"2024-09-02T10:09:06.926526+0000","last_became_active":"2024-09-02T10:05:56.583249+0000","last_became_peered":"2024-09-02T10:05:56.583249+0000","last_unstale":"2024-09-02T10:09:06.926526+0000","last_undegraded":"2024-09-02T10:09:06.926526+0000","last_fullsized":"2024-09-02T10:09:06.926526+0000","mapping_epoch":226,"log_start":"233'900","ondisk_log_start":"233'900","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5253","last_scrub_stamp":"2024-09-02T10:07:07.774339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:07.774339+0000","objects_scrubbed":5197,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1300,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173801472,"num_objects":2652,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10984,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":227,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.901587+0000","last_change":"2024-09-02T10:08:06.076632+0000","last_active":"2024-09-02T10:08:56.901587+0000","last_peered":"2024-09-02T10:08:56.901587+0000","last_clean":"2024-09-02T10:08:56.901587+0000","last_became_active":"2024-09-02T10:06:32.870387+0000","last_became_peered":"2024-09-02T10:06:32.870387+0000","last_unstale":"2024-09-02T10:08:56.901587+0000","last_undegraded":"2024-09-02T10:08:56.901587+0000","last_fullsized":"2024-09-02T10:08:56.901587+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:06.076562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:06.076562+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:19.771015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"290'11144","reported_seq":11360,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:09.235143+0000","last_change":"2024-09-02T10:07:36.281686+0000","last_active":"2024-09-02T10:09:09.235143+0000","last_peered":"2024-09-02T10:09:09.235143+0000","last_clean":"2024-09-02T10:09:09.235143+0000","last_became_active":"2024-09-02T10:05:56.584313+0000","last_became_peered":"2024-09-02T10:05:56.584313+0000","last_unstale":"2024-09-02T10:09:09.235143+0000","last_undegraded":"2024-09-02T10:09:09.235143+0000","last_fullsized":"2024-09-02T10:09:09.235143+0000","mapping_epoch":226,"log_start":"234'1100","ondisk_log_start":"234'1100","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6942","last_scrub_stamp":"2024-09-02T10:07:36.281628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:36.281628+0000","objects_scrubbed":6782,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1988,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166330368,"num_objects":2538,"num_object_clones":0,"num_object_copies":7614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11144,"num_write_kb":437824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":37,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:57.015172+0000","last_change":"2024-09-02T10:08:25.382153+0000","last_active":"2024-09-02T10:08:57.015172+0000","last_peered":"2024-09-02T10:08:57.015172+0000","last_clean":"2024-09-02T10:08:57.015172+0000","last_became_active":"2024-09-02T10:08:25.264766+0000","last_became_peered":"2024-09-02T10:08:25.264766+0000","last_unstale":"2024-09-02T10:08:57.015172+0000","last_undegraded":"2024-09-02T10:08:57.015172+0000","last_fullsized":"2024-09-02T10:08:57.015172+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:25.382103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:25.382103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:55.029769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"290'11048","reported_seq":11262,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:06.978014+0000","last_change":"2024-09-02T10:07:34.096922+0000","last_active":"2024-09-02T10:09:06.978014+0000","last_peered":"2024-09-02T10:09:06.978014+0000","last_clean":"2024-09-02T10:09:06.978014+0000","last_became_active":"2024-09-02T10:05:56.578973+0000","last_became_peered":"2024-09-02T10:05:56.578973+0000","last_unstale":"2024-09-02T10:09:06.978014+0000","last_undegraded":"2024-09-02T10:09:06.978014+0000","last_fullsized":"2024-09-02T10:09:06.978014+0000","mapping_epoch":226,"log_start":"233'1000","ondisk_log_start":"233'1000","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6853","last_scrub_stamp":"2024-09-02T10:07:34.096729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:34.096729+0000","objects_scrubbed":6787,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173670400,"num_objects":2650,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11048,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":525,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.888075+0000","last_change":"2024-09-02T10:07:07.536258+0000","last_active":"2024-09-02T10:08:56.888075+0000","last_peered":"2024-09-02T10:08:56.888075+0000","last_clean":"2024-09-02T10:08:56.888075+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:08:56.888075+0000","last_undegraded":"2024-09-02T10:08:56.888075+0000","last_fullsized":"2024-09-02T10:08:56.888075+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:07.536194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:07.536194+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"290'11001","reported_seq":13669,"reported_epoch":290,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:09:08.526027+0000","last_change":"2024-09-02T10:08:26.621106+0000","last_active":"2024-09-02T10:09:08.526027+0000","last_peered":"2024-09-02T10:09:08.526027+0000","last_clean":"2024-09-02T10:08:24.056426+0000","last_became_active":"2024-09-02T10:08:26.550955+0000","last_became_peered":"2024-09-02T10:08:26.550955+0000","last_unstale":"2024-09-02T10:09:08.526027+0000","last_undegraded":"2024-09-02T10:08:26.292304+0000","last_fullsized":"2024-09-02T10:08:26.259077+0000","mapping_epoch":283,"log_start":"233'1000","ondisk_log_start":"233'1000","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6680","last_scrub_stamp":"2024-09-02T10:07:30.887770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.887770+0000","objects_scrubbed":6647,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170196992,"num_objects":2597,"num_object_clones":0,"num_object_copies":7791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":126,"num_objects_misplaced":2596,"num_objects_unfound":0,"num_objects_dirty":2597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11001,"num_write_kb":435136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1226,"num_bytes_recovered":80084992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7],"avail_no_missing":["3","7","11"],"object_location_counts":[{"shards":"3,7","objects":-125},{"shards":"3,7,11","objects":2596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":384,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.896362+0000","last_change":"2024-09-02T10:08:25.265558+0000","last_active":"2024-09-02T10:08:56.896362+0000","last_peered":"2024-09-02T10:08:56.896362+0000","last_clean":"2024-09-02T10:08:56.896362+0000","last_became_active":"2024-09-02T10:08:25.265218+0000","last_became_peered":"2024-09-02T10:08:25.265218+0000","last_unstale":"2024-09-02T10:08:56.896362+0000","last_undegraded":"2024-09-02T10:08:56.896362+0000","last_fullsized":"2024-09-02T10:08:56.896362+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:04.181537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:04.181537+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:31.855618+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"290'11396","reported_seq":11614,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:08.166383+0000","last_change":"2024-09-02T10:07:34.254248+0000","last_active":"2024-09-02T10:09:08.166383+0000","last_peered":"2024-09-02T10:09:08.166383+0000","last_clean":"2024-09-02T10:09:08.166383+0000","last_became_active":"2024-09-02T10:05:56.580910+0000","last_became_peered":"2024-09-02T10:05:56.580910+0000","last_unstale":"2024-09-02T10:09:08.166383+0000","last_undegraded":"2024-09-02T10:09:08.166383+0000","last_fullsized":"2024-09-02T10:09:08.166383+0000","mapping_epoch":226,"log_start":"234'1300","ondisk_log_start":"234'1300","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'7048","last_scrub_stamp":"2024-09-02T10:07:34.254184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:34.254184+0000","objects_scrubbed":7013,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:09.354308+0000","scrub_duration":1842,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174194688,"num_objects":2658,"num_object_clones":0,"num_object_copies":7974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11396,"num_write_kb":449728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":227,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.901721+0000","last_change":"2024-09-02T10:08:09.094198+0000","last_active":"2024-09-02T10:08:56.901721+0000","last_peered":"2024-09-02T10:08:56.901721+0000","last_clean":"2024-09-02T10:08:56.901721+0000","last_became_active":"2024-09-02T10:06:32.871735+0000","last_became_peered":"2024-09-02T10:06:32.871735+0000","last_unstale":"2024-09-02T10:08:56.901721+0000","last_undegraded":"2024-09-02T10:08:56.901721+0000","last_fullsized":"2024-09-02T10:08:56.901721+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:09.094146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:09.094146+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:27.879919+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"290'11059","reported_seq":11272,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:06.856013+0000","last_change":"2024-09-02T10:07:30.154080+0000","last_active":"2024-09-02T10:09:06.856013+0000","last_peered":"2024-09-02T10:09:06.856013+0000","last_clean":"2024-09-02T10:09:06.856013+0000","last_became_active":"2024-09-02T10:05:56.690229+0000","last_became_peered":"2024-09-02T10:05:56.690229+0000","last_unstale":"2024-09-02T10:09:06.856013+0000","last_undegraded":"2024-09-02T10:09:06.856013+0000","last_fullsized":"2024-09-02T10:09:06.856013+0000","mapping_epoch":226,"log_start":"233'1000","ondisk_log_start":"233'1000","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6652","last_scrub_stamp":"2024-09-02T10:07:30.153975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.153975+0000","objects_scrubbed":6605,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167968768,"num_objects":2563,"num_object_clones":0,"num_object_copies":7689,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2563,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11059,"num_write_kb":435904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":507,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.888839+0000","last_change":"2024-09-02T10:08:25.315919+0000","last_active":"2024-09-02T10:08:56.888839+0000","last_peered":"2024-09-02T10:08:56.888839+0000","last_clean":"2024-09-02T10:08:56.888839+0000","last_became_active":"2024-09-02T10:08:25.315580+0000","last_became_peered":"2024-09-02T10:08:25.315580+0000","last_unstale":"2024-09-02T10:08:56.888839+0000","last_undegraded":"2024-09-02T10:08:56.888839+0000","last_fullsized":"2024-09-02T10:08:56.888839+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:04.721466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:04.721466+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:07.635341+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"290'11243","reported_seq":11458,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:09.266958+0000","last_change":"2024-09-02T10:07:35.873554+0000","last_active":"2024-09-02T10:09:09.266958+0000","last_peered":"2024-09-02T10:09:09.266958+0000","last_clean":"2024-09-02T10:09:09.266958+0000","last_became_active":"2024-09-02T10:05:56.581969+0000","last_became_peered":"2024-09-02T10:05:56.581969+0000","last_unstale":"2024-09-02T10:09:09.266958+0000","last_undegraded":"2024-09-02T10:09:09.266958+0000","last_fullsized":"2024-09-02T10:09:09.266958+0000","mapping_epoch":226,"log_start":"234'1200","ondisk_log_start":"234'1200","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'7026","last_scrub_stamp":"2024-09-02T10:07:35.873463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:35.873463+0000","objects_scrubbed":6888,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172163072,"num_objects":2627,"num_object_clones":0,"num_object_copies":7881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11243,"num_write_kb":443840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":214,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.901727+0000","last_change":"2024-09-02T10:07:29.964663+0000","last_active":"2024-09-02T10:08:56.901727+0000","last_peered":"2024-09-02T10:08:56.901727+0000","last_clean":"2024-09-02T10:08:56.901727+0000","last_became_active":"2024-09-02T10:04:57.021178+0000","last_became_peered":"2024-09-02T10:04:57.021178+0000","last_unstale":"2024-09-02T10:08:56.901727+0000","last_undegraded":"2024-09-02T10:08:56.901727+0000","last_fullsized":"2024-09-02T10:08:56.901727+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:29.964607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:29.964607+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.0","version":"290'10844","reported_seq":11058,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:06.994121+0000","last_change":"2024-09-02T10:07:35.803712+0000","last_active":"2024-09-02T10:09:06.994121+0000","last_peered":"2024-09-02T10:09:06.994121+0000","last_clean":"2024-09-02T10:09:06.994121+0000","last_became_active":"2024-09-02T10:05:56.876397+0000","last_became_peered":"2024-09-02T10:05:56.876397+0000","last_unstale":"2024-09-02T10:09:06.994121+0000","last_undegraded":"2024-09-02T10:09:06.994121+0000","last_fullsized":"2024-09-02T10:09:06.994121+0000","mapping_epoch":226,"log_start":"233'800","ondisk_log_start":"233'800","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6802","last_scrub_stamp":"2024-09-02T10:07:35.803575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:35.803575+0000","objects_scrubbed":6687,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170262528,"num_objects":2598,"num_object_clones":0,"num_object_copies":7794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10844,"num_write_kb":430144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":237,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-02T10:09:09.895649+0000","last_change":"2024-09-02T10:08:25.263472+0000","last_active":"2024-09-02T10:09:09.895649+0000","last_peered":"2024-09-02T10:09:09.895649+0000","last_clean":"2024-09-02T10:09:09.895649+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:09:09.895649+0000","last_undegraded":"2024-09-02T10:09:09.895649+0000","last_fullsized":"2024-09-02T10:09:09.895649+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:33.808730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:33.808730+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"290'10935","reported_seq":13792,"reported_epoch":290,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:09:07.128518+0000","last_change":"2024-09-02T10:08:26.607400+0000","last_active":"2024-09-02T10:09:07.128518+0000","last_peered":"2024-09-02T10:09:07.128518+0000","last_clean":"2024-09-02T10:08:24.179939+0000","last_became_active":"2024-09-02T10:08:26.550894+0000","last_became_peered":"2024-09-02T10:08:26.550894+0000","last_unstale":"2024-09-02T10:09:07.128518+0000","last_undegraded":"2024-09-02T10:08:26.301889+0000","last_fullsized":"2024-09-02T10:08:26.259596+0000","mapping_epoch":283,"log_start":"233'900","ondisk_log_start":"233'900","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5000","last_scrub_stamp":"2024-09-02T10:07:05.712114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:05.712114+0000","objects_scrubbed":4957,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175046656,"num_objects":2671,"num_object_clones":0,"num_object_copies":8013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":61,"num_objects_misplaced":2628,"num_objects_unfound":0,"num_objects_dirty":2671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10935,"num_write_kb":435392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1328,"num_bytes_recovered":86966272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,5],"acting":[0,11],"avail_no_missing":["0","4","11"],"object_location_counts":[{"shards":"0,4,11","objects":2628},{"shards":"0,11","objects":-18}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":449,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.890689+0000","last_change":"2024-09-02T10:08:25.264478+0000","last_active":"2024-09-02T10:08:56.890689+0000","last_peered":"2024-09-02T10:08:56.890689+0000","last_clean":"2024-09-02T10:08:56.890689+0000","last_became_active":"2024-09-02T10:08:25.264099+0000","last_became_peered":"2024-09-02T10:08:25.264099+0000","last_unstale":"2024-09-02T10:08:56.890689+0000","last_undegraded":"2024-09-02T10:08:56.890689+0000","last_fullsized":"2024-09-02T10:08:56.890689+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:37.379359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:37.379359+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:08:59.365638+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"290'11230","reported_seq":11446,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:08.510954+0000","last_change":"2024-09-02T10:07:30.485290+0000","last_active":"2024-09-02T10:09:08.510954+0000","last_peered":"2024-09-02T10:09:08.510954+0000","last_clean":"2024-09-02T10:09:08.510954+0000","last_became_active":"2024-09-02T10:05:56.876573+0000","last_became_peered":"2024-09-02T10:05:56.876573+0000","last_unstale":"2024-09-02T10:09:08.510954+0000","last_undegraded":"2024-09-02T10:09:08.510954+0000","last_fullsized":"2024-09-02T10:09:08.510954+0000","mapping_epoch":226,"log_start":"234'1200","ondisk_log_start":"234'1200","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6788","last_scrub_stamp":"2024-09-02T10:07:30.485127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.485127+0000","objects_scrubbed":6730,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2030,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174456832,"num_objects":2662,"num_object_clones":0,"num_object_copies":7986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11230,"num_write_kb":444544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":241,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-02T10:09:09.895414+0000","last_change":"2024-09-02T10:07:28.824347+0000","last_active":"2024-09-02T10:09:09.895414+0000","last_peered":"2024-09-02T10:09:09.895414+0000","last_clean":"2024-09-02T10:09:09.895414+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:09:09.895414+0000","last_undegraded":"2024-09-02T10:09:09.895414+0000","last_fullsized":"2024-09-02T10:09:09.895414+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:28.824291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:28.824291+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"291'11287","reported_seq":11488,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-02T10:09:10.925775+0000","last_change":"2024-09-02T10:07:04.549316+0000","last_active":"2024-09-02T10:09:10.925775+0000","last_peered":"2024-09-02T10:09:10.925775+0000","last_clean":"2024-09-02T10:09:10.925775+0000","last_became_active":"2024-09-02T10:05:56.876540+0000","last_became_peered":"2024-09-02T10:05:56.876540+0000","last_unstale":"2024-09-02T10:09:10.925775+0000","last_undegraded":"2024-09-02T10:09:10.925775+0000","last_fullsized":"2024-09-02T10:09:10.925775+0000","mapping_epoch":226,"log_start":"234'1200","ondisk_log_start":"234'1200","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'4967","last_scrub_stamp":"2024-09-02T10:07:04.549227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:04.549227+0000","objects_scrubbed":4900,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171180056,"num_objects":2613,"num_object_clones":0,"num_object_copies":7839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11287,"num_write_kb":444737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":470,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.888738+0000","last_change":"2024-09-02T10:07:27.771848+0000","last_active":"2024-09-02T10:08:56.888738+0000","last_peered":"2024-09-02T10:08:56.888738+0000","last_clean":"2024-09-02T10:08:56.888738+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:08:56.888738+0000","last_undegraded":"2024-09-02T10:08:56.888738+0000","last_fullsized":"2024-09-02T10:08:56.888738+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:27.771782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:27.771782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"290'11066","reported_seq":11280,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:09:06.891948+0000","last_change":"2024-09-02T10:07:31.723546+0000","last_active":"2024-09-02T10:09:06.891948+0000","last_peered":"2024-09-02T10:09:06.891948+0000","last_clean":"2024-09-02T10:09:06.891948+0000","last_became_active":"2024-09-02T10:05:56.585883+0000","last_became_peered":"2024-09-02T10:05:56.585883+0000","last_unstale":"2024-09-02T10:09:06.891948+0000","last_undegraded":"2024-09-02T10:09:06.891948+0000","last_fullsized":"2024-09-02T10:09:06.891948+0000","mapping_epoch":226,"log_start":"233'1000","ondisk_log_start":"233'1000","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6718","last_scrub_stamp":"2024-09-02T10:07:31.723385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:31.723385+0000","objects_scrubbed":6667,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167903232,"num_objects":2562,"num_object_clones":0,"num_object_copies":7686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11066,"num_write_kb":436096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":38,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-02T10:09:09.895950+0000","last_change":"2024-09-02T10:08:26.088128+0000","last_active":"2024-09-02T10:09:09.895950+0000","last_peered":"2024-09-02T10:09:09.895950+0000","last_clean":"2024-09-02T10:09:09.895950+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:09:09.895950+0000","last_undegraded":"2024-09-02T10:09:09.895950+0000","last_fullsized":"2024-09-02T10:09:09.895950+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:26.088081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:26.088081+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:52.862650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"291'11100","reported_seq":11305,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-02T10:09:10.947277+0000","last_change":"2024-09-02T10:08:11.942333+0000","last_active":"2024-09-02T10:09:10.947277+0000","last_peered":"2024-09-02T10:09:10.947277+0000","last_clean":"2024-09-02T10:09:10.947277+0000","last_became_active":"2024-09-02T10:05:56.584450+0000","last_became_peered":"2024-09-02T10:05:56.584450+0000","last_unstale":"2024-09-02T10:09:10.947277+0000","last_undegraded":"2024-09-02T10:09:10.947277+0000","last_fullsized":"2024-09-02T10:09:10.947277+0000","mapping_epoch":226,"log_start":"233'1000","ondisk_log_start":"233'1000","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5143","last_scrub_stamp":"2024-09-02T10:07:06.845283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:06.845283+0000","objects_scrubbed":99,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162529280,"num_objects":2480,"num_object_clones":0,"num_object_copies":7440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11100,"num_write_kb":434560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":254,"reported_epoch":291,"state":"active+clean","last_fresh":"2024-09-02T10:09:09.895915+0000","last_change":"2024-09-02T10:08:04.938089+0000","last_active":"2024-09-02T10:09:09.895915+0000","last_peered":"2024-09-02T10:09:09.895915+0000","last_clean":"2024-09-02T10:09:09.895915+0000","last_became_active":"2024-09-02T10:06:32.876151+0000","last_became_peered":"2024-09-02T10:06:32.876151+0000","last_unstale":"2024-09-02T10:09:09.895915+0000","last_undegraded":"2024-09-02T10:09:09.895915+0000","last_fullsized":"2024-09-02T10:09:09.895915+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:04.938027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:04.938027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:25.463231+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"290'10948","reported_seq":15289,"reported_epoch":290,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:09:07.139908+0000","last_change":"2024-09-02T10:08:26.668214+0000","last_active":"2024-09-02T10:09:07.139908+0000","last_peered":"2024-09-02T10:09:07.139908+0000","last_clean":"2024-09-02T10:08:24.105685+0000","last_became_active":"2024-09-02T10:08:26.603797+0000","last_became_peered":"2024-09-02T10:08:26.603797+0000","last_unstale":"2024-09-02T10:09:07.139908+0000","last_undegraded":"2024-09-02T10:09:07.139908+0000","last_fullsized":"2024-09-02T10:08:26.259622+0000","mapping_epoch":283,"log_start":"233'900","ondisk_log_start":"233'900","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5095","last_scrub_stamp":"2024-09-02T10:07:06.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:06.655797+0000","objects_scrubbed":949,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172097536,"num_objects":2626,"num_object_clones":0,"num_object_copies":7878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5249,"num_objects_unfound":0,"num_objects_dirty":2626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10948,"num_write_kb":434368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1375,"num_bytes_recovered":90112000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,3],"acting":[0,3],"avail_no_missing":["0","4","11"],"object_location_counts":[{"shards":"0,3","objects":1},{"shards":"0,3,4,11","objects":1},{"shards":"0,4,11","objects":2624}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":266,"reported_epoch":290,"state":"active+clean","last_fresh":"2024-09-02T10:08:56.885855+0000","last_change":"2024-09-02T10:08:02.302316+0000","last_active":"2024-09-02T10:08:56.885855+0000","last_peered":"2024-09-02T10:08:56.885855+0000","last_clean":"2024-09-02T10:08:56.885855+0000","last_became_active":"2024-09-02T10:06:32.875557+0000","last_became_peered":"2024-09-02T10:06:32.875557+0000","last_unstale":"2024-09-02T10:08:56.885855+0000","last_undegraded":"2024-09-02T10:08:56.885855+0000","last_fullsized":"2024-09-02T10:08:56.885855+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:02.302231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:02.302231+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:09:24.149374+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":2731999256,"num_objects":41688,"num_object_clones":0,"num_object_copies":125064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":10473,"num_objects_unfound":0,"num_objects_dirty":41688,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":177394,"num_write_kb":7010561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3929,"num_bytes_recovered":257163264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8570548224,"data_stored":8570536008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160894,"ondisk_log_size":160894,"up":48,"acting":45,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383788,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1793480,"kb_used_data":1531832,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":91951672,"statfs":{"total":95995035648,"available":94158512128,"internally_reserved":0,"allocated":1568595968,"data_stored":1568326839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":267909572},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532612721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841654,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":713804,"kb_used_data":511868,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93031348,"statfs":{"total":95995035648,"available":95264100352,"internally_reserved":0,"allocated":524152832,"data_stored":523879631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":206764480},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070588,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":447752,"kb_used_data":326328,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93297400,"statfs":{"total":95995035648,"available":95536537600,"internally_reserved":0,"allocated":334159872,"data_stored":333890743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299519,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":816868,"kb_used_data":651024,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92928284,"statfs":{"total":95995035648,"available":95158562816,"internally_reserved":0,"allocated":666648576,"data_stored":666367364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":169802186},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528451,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":987848,"kb_used_data":848376,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":92757304,"statfs":{"total":95995035648,"available":94983479296,"internally_reserved":0,"allocated":868737024,"data_stored":868467895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142801351},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724680,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":831828,"kb_used_data":688196,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92913324,"statfs":{"total":95995035648,"available":95143243776,"internally_reserved":0,"allocated":704712704,"data_stored":704435951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953614,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1494920,"kb_used_data":1347128,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":92250232,"statfs":{"total":95995035648,"available":94464237568,"internally_reserved":0,"allocated":1379459072,"data_stored":1379189943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149844,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1098120,"kb_used_data":836472,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92647032,"statfs":{"total":95995035648,"available":94870560768,"internally_reserved":0,"allocated":856547328,"data_stored":856278199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":267909576},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411479,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325840,"kb_used_data":164160,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93419312,"statfs":{"total":95995035648,"available":95661375488,"internally_reserved":0,"allocated":168099840,"data_stored":167827159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575005,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":839176,"kb_used_data":677496,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92905976,"statfs":{"total":95995035648,"available":95135719424,"internally_reserved":0,"allocated":693755904,"data_stored":693486775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738530,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039304,"kb_used_data":859576,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":92705848,"statfs":{"total":95995035648,"available":94930788352,"internally_reserved":0,"allocated":880205824,"data_stored":879936695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":871301120,"data_stored":871301120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":684654592,"data_stored":684654592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":166658048,"data_stored":166658048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":843579392,"data_stored":843579392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1368719360,"data_stored":1368719360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":696193024,"data_stored":696188952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":863830016,"data_stored":863830016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":661262336,"data_stored":661258264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":332595200,"data_stored":332595200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":522326016,"data_stored":522321944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1559429120,"data_stored":1559429120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:09:14.577 INFO:tasks.thrashosds.thrasher:Setting 1.9 to [1, 7, 10, 9, 0, 5] 2024-09-02T10:09:14.577 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.9', '1', '7', '10', '9', '0', '5'] 2024-09-02T10:09:14.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 1.9 1 7 10 9 0 5 2024-09-02T10:09:14.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:14.683 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:14.684 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:14.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:14 smithi151 ceph-mon[16639]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:09:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:14 smithi151 ceph-mon[16639]: pgmap v718: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.5 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 187/120768 objects degraded (0.155%); 9969/120768 objects misplaced (8.255%); 6.2 MiB/s, 99 objects/s recovering 2024-09-02T10:09:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 187/120768 objects degraded (0.155%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:14 smithi187 ceph-mon[20072]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:09:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:14 smithi187 ceph-mon[20072]: pgmap v718: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.5 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 187/120768 objects degraded (0.155%); 9969/120768 objects misplaced (8.255%); 6.2 MiB/s, 99 objects/s recovering 2024-09-02T10:09:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 187/120768 objects degraded (0.155%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:14.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:14.807 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:14.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:14 smithi162 ceph-mon[20279]: from='client.15795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:09:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:14 smithi162 ceph-mon[20279]: pgmap v718: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.5 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 187/120768 objects degraded (0.155%); 9969/120768 objects misplaced (8.255%); 6.2 MiB/s, 99 objects/s recovering 2024-09-02T10:09:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 187/120768 objects degraded (0.155%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:14.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:14.930 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:14.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:14.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:15.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:15.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:15.058 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:15.078 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:09:15.093 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:09:15.093 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:09:15.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:09:15.093 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:09:15.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.093 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24976.0:109490 3.b e263/226 3:d613b4ca:::benchmark_data_smithi187_59989_object109489:head v 263'6808, mlcod=263'6805)", 2024-09-02T10:09:15.093 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:32.978241+0000", 2024-09-02T10:09:15.093 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 102.112140443, 2024-09-02T10:09:15.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.53835022099999996, 2024-09-02T10:09:15.093 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.094 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.094 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:09:15.094 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978241+0000", 2024-09-02T10:09:15.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978241+0000", 2024-09-02T10:09:15.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978242+0000", 2024-09-02T10:09:15.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.44e-07 2024-09-02T10:09:15.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978264+0000", 2024-09-02T10:09:15.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2433e-05 2024-09-02T10:09:15.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.513667+0000", 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.53540241300000002 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.513682+0000", 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5447e-05 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.097 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.513769+0000", 2024-09-02T10:09:15.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7028999999999996e-05 2024-09-02T10:09:15.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:09:15.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.513788+0000", 2024-09-02T10:09:15.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.942e-05 2024-09-02T10:09:15.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.516553+0000", 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027647320000000002 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.516591+0000", 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8203e-05 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:154979 3.1 3.edb17651 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.263594+0000", 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.826787793000001, 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99702418999999998, 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154979 2024-09-02T10:09:15.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263594+0000", 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263594+0000", 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263596+0000", 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.782e-06 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263604+0000", 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1240000000000005e-06 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.102 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263619+0000", 2024-09-02T10:09:15.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5542000000000001e-05 2024-09-02T10:09:15.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263634+0000", 2024-09-02T10:09:15.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4473e-05 2024-09-02T10:09:15.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263716+0000", 2024-09-02T10:09:15.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1929999999999997e-05 2024-09-02T10:09:15.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:09:15.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263729+0000", 2024-09-02T10:09:15.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3038e-05 2024-09-02T10:09:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260590+0000", 2024-09-02T10:09:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.996860996 2024-09-02T10:09:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260618+0000", 2024-09-02T10:09:15.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8305000000000001e-05 2024-09-02T10:09:15.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.107 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.107 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.107 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:154986 3.e 3.aae52e7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.287904+0000", 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.802477946000003, 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97255893000000004, 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154986 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.108 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287904+0000", 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287904+0000", 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287905+0000", 2024-09-02T10:09:15.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.497e-06 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287912+0000", 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9999999999999999e-06 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287932+0000", 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9681e-05 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287941+0000", 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4679999999999998e-06 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.288009+0000", 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7584999999999998e-05 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.288020+0000", 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0696000000000001e-05 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260431+0000", 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97241165799999996 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260463+0000", 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1344999999999999e-05 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.113 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155002 3.1 3.ef033051 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:09:15.113 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.312679+0000", 2024-09-02T10:09:15.113 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.777702257000001, 2024-09-02T10:09:15.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.94808788799999999, 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155002 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312679+0000", 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312679+0000", 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312681+0000", 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.367e-06 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312686+0000", 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3460000000000001e-06 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312698+0000", 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1854e-05 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312717+0000", 2024-09-02T10:09:15.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8736000000000002e-05 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312773+0000", 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6093e-05 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312781+0000", 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8900000000000007e-06 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260684+0000", 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.94790293400000003 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260767+0000", 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3667999999999999e-05 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155008 3.e 3.a75576ae (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.318646+0000", 2024-09-02T10:09:15.118 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.771735862, 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.941864861, 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155008 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318646+0000", 2024-09-02T10:09:15.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318646+0000", 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318647+0000", 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.232e-06 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318652+0000", 2024-09-02T10:09:15.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1499999999999998e-06 2024-09-02T10:09:15.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318668+0000", 2024-09-02T10:09:15.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5542000000000001e-05 2024-09-02T10:09:15.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318676+0000", 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4989999999999994e-06 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318753+0000", 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7109999999999999e-05 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318764+0000", 2024-09-02T10:09:15.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0397000000000001e-05 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260492+0000", 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.94172796400000003 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260511+0000", 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8967e-05 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.123 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155020 3.1 3.bfce4141 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.337044+0000", 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.753337459000001, 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.92377231199999998, 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:09:15.124 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155020 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337044+0000", 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337044+0000", 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337045+0000", 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8999999999999995e-07 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337049+0000", 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337057+0000", 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1610000000000003e-06 2024-09-02T10:09:15.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337061+0000", 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5279999999999997e-06 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337104+0000", 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2446e-05 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337112+0000", 2024-09-02T10:09:15.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9140000000000008e-06 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260807+0000", 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.92369535000000003 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260816+0000", 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4930000000000004e-06 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.128 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.129 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155022 3.1 3.961fe4e1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:09:15.129 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.345927+0000", 2024-09-02T10:09:15.129 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.744454838999999, 2024-09-02T10:09:15.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91494058099999998, 2024-09-02T10:09:15.129 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.129 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.129 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155022 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345927+0000", 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345927+0000", 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345928+0000", 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.071e-06 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345932+0000", 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9079999999999997e-06 2024-09-02T10:09:15.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345941+0000", 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8319999999999995e-06 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345945+0000", 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0840000000000002e-06 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345990+0000", 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5074999999999998e-05 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345997+0000", 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6090000000000003e-06 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260857+0000", 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91485953099999995 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260867+0000", 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0471e-05 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155038 3.1 3.9395eab1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.393719+0000", 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.696662285000002, 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86722825800000003, 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.134 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155038 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393719+0000", 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393719+0000", 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393720+0000", 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9800000000000002e-07 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393724+0000", 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.873e-06 2024-09-02T10:09:15.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393733+0000", 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8689999999999993e-06 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393739+0000", 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4350000000000002e-06 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393784+0000", 2024-09-02T10:09:15.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4614999999999999e-05 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393792+0000", 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.537e-06 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260938+0000", 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86714591900000004 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260948+0000", 2024-09-02T10:09:15.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0012e-05 2024-09-02T10:09:15.139 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.139 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.139 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.139 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155069 3.1 3.babc3b41 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:09:15.139 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.560041+0000", 2024-09-02T10:09:15.139 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.530340166999999, 2024-09-02T10:09:15.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70095849499999996, 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155069 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560041+0000", 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560041+0000", 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560043+0000", 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.209e-06 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560047+0000", 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8439999999999997e-06 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560059+0000", 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1732e-05 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560066+0000", 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7870000000000004e-06 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560147+0000", 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1073000000000001e-05 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560159+0000", 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2398999999999999e-05 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260991+0000", 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70083152199999998 2024-09-02T10:09:15.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.261000+0000", 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9290000000000004e-06 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155076 3.e 3.59c2747e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.799566+0000", 2024-09-02T10:09:15.144 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.290815160999998, 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46109792199999999, 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155076 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799566+0000", 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799566+0000", 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799568+0000", 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5990000000000001e-06 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799574+0000", 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7570000000000001e-06 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799587+0000", 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3127e-05 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799594+0000", 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4030000000000001e-06 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799652+0000", 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8124000000000001e-05 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799660+0000", 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7090000000000002e-06 2024-09-02T10:09:15.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260593+0000", 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46093320900000001 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260664+0000", 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0994000000000005e-05 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 285/283 [PushReplyOp(3:714100ed:::benchmark_data_smithi187_59989_object50041:head)])", 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:35.141214+0000", 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 39.949167498000001, 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.267076916, 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:35.141214+0000", 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:35.141214+0000", 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:35.141215+0000", 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5799999999999999e-07 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:35.141219+0000", 2024-09-02T10:09:15.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9269999999999998e-06 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:35.141233+0000", 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4122e-05 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:35.141240+0000", 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9290000000000001e-06 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:35.405549+0000", 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26430952200000002 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:35.408291+0000", 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002741758 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 285/283 [PushReplyOp(3:73bf05e4:::benchmark_data_smithi187_59989_object51570:head)])", 2024-09-02T10:09:15.153 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:38.071375+0000", 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 37.019006314999999, 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29147010499999998, 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.071375+0000", 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.071375+0000", 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.071376+0000", 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1429999999999999e-06 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.071385+0000", 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3280000000000006e-06 2024-09-02T10:09:15.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.071421+0000", 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6705999999999997e-05 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.071433+0000", 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1093e-05 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.356990+0000", 2024-09-02T10:09:15.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28555697200000002 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.362845+0000", 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0058558630000000002 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 285/283 [PushReplyOp(3:71716769:::benchmark_data_smithi187_59989_object51646:head)])", 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:38.296966+0000", 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.793415404000001, 2024-09-02T10:09:15.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26676754600000002, 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.296966+0000", 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.296966+0000", 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:15.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.296967+0000", 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1299999999999999e-07 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.296973+0000", 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1160000000000004e-06 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.297005+0000", 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1578999999999997e-05 2024-09-02T10:09:15.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.297013+0000", 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9580000000000007e-06 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.557150+0000", 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.260137022 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.563734+0000", 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065840580000000003 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 285/283 [PushReplyOp(3:758904ec:::benchmark_data_smithi187_59989_object51812:head)])", 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:38.550347+0000", 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.540034771000002, 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.286552896, 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.550347+0000", 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.550347+0000", 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.550348+0000", 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8400000000000004e-07 2024-09-02T10:09:15.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.550353+0000", 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.463e-06 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.550384+0000", 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0939999999999999e-05 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.550392+0000", 2024-09-02T10:09:15.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6799999999999993e-06 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.832953+0000", 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.282561071 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.836900+0000", 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039470579999999998 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 285/283 [PushReplyOp(3:713d2e03:::benchmark_data_smithi187_59989_object51866:head)])", 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:38.615472+0000", 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.474909132999997, 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35497819400000002, 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615472+0000", 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615472+0000", 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615473+0000", 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2399999999999997e-07 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615480+0000", 2024-09-02T10:09:15.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4350000000000002e-06 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615509+0000", 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9081000000000002e-05 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615516+0000", 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2119999999999997e-06 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.968094+0000", 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.352578265 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.970451+0000", 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023563770000000002 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 285/283 [PushReplyOp(3:713d2e03:::benchmark_data_smithi187_59989_object51866:head)])", 2024-09-02T10:09:15.170 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:38.617571+0000", 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.472810297999999, 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37077970100000002, 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617571+0000", 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617571+0000", 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617572+0000", 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0999999999999997e-07 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617579+0000", 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.37e-06 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617619+0000", 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.002e-05 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617627+0000", 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9750000000000007e-06 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.981676+0000", 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.364049503 2024-09-02T10:09:15.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.988351+0000", 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0066749230000000001 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 285/283 [PushReplyOp(3:7a9575a2:::benchmark_data_smithi187_59989_object51949:head)])", 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:38.740612+0000", 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.349769997999999, 2024-09-02T10:09:15.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28385495799999999, 2024-09-02T10:09:15.175 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.175 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.175 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.175 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.740612+0000", 2024-09-02T10:09:15.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.740612+0000", 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.740613+0000", 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.37e-06 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.740622+0000", 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6300000000000004e-06 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.740661+0000", 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9348999999999999e-05 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.740670+0000", 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9819999999999997e-06 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:39.018726+0000", 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27805612499999999 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:39.024467+0000", 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057405019999999998 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 285/283 [PushReplyOp(3:7a9575a2:::benchmark_data_smithi187_59989_object51949:head)])", 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:38.741347+0000", 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 36.349034437, 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27730445199999998, 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.741347+0000", 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.741347+0000", 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.741348+0000", 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0900000000000001e-07 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.741353+0000", 2024-09-02T10:09:15.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.66e-06 2024-09-02T10:09:15.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.741378+0000", 2024-09-02T10:09:15.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4621e-05 2024-09-02T10:09:15.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.741383+0000", 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1610000000000002e-06 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:39.015932+0000", 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27454835599999999 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:39.018652+0000", 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027200449999999999 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.182 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 286/283 [PushReplyOp(3:79ed81d9:::benchmark_data_smithi187_59989_object55678:head)])", 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:45.857626+0000", 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.232755141999998, 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30484871099999999, 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857626+0000", 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857626+0000", 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857627+0000", 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3600000000000003e-07 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857636+0000", 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7369999999999995e-06 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857687+0000", 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0785999999999998e-05 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857695+0000", 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.337e-06 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:46.157534+0000", 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29983937500000002 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:46.162475+0000", 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049408400000000002 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.186 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:181165 3.1 3:863dffc8:::benchmark_data_smithi187_59989_object71622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:09:14.757056+0000", 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.33332585599999998, 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.274718715, 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 181165 2024-09-02T10:09:15.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.757056+0000", 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.757056+0000", 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:09:15.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.757058+0000", 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0190000000000001e-06 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.757067+0000", 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2920000000000001e-06 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.757084+0000", 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6976000000000001e-05 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.757095+0000", 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0852e-05 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.761296+0000", 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042010809999999997 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.761456+0000", 2024-09-02T10:09:15.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000159953 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.761616+0000", 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015981099999999999 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.767758+0000", 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0061427529999999999 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:14.773078+0000", 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0053192990000000004 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.030147+0000", 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25706958699999999 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.030230+0000", 2024-09-02T10:09:15.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2565999999999995e-05 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.031774+0000", 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015445260000000001 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:09:15.193 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:09:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:15.195 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:15.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:15.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:15.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:15.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:15.318 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:15.419 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:15.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:15.449 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:15.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:15.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:15.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:15.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:15.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:15.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:15.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:15.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:15.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:15.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:15.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:15.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:15.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:15.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:15.970 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:15.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:09:16.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:16.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:16.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:16.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:16.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:16.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:16.340 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:16.366 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:16.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:16.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:16.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:16.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:16.505 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:16.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:16.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:16.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:16.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:16.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:16 smithi187 ceph-mon[20072]: pgmap v719: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.4 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 575 op/s; 199/117525 objects degraded (0.169%); 9789/117525 objects misplaced (8.329%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:09:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:16.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:16.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:16.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:16 smithi162 ceph-mon[20279]: pgmap v719: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.4 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 575 op/s; 199/117525 objects degraded (0.169%); 9789/117525 objects misplaced (8.329%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:09:16.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:16.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:16.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:16.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:16.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:16 smithi151 ceph-mon[16639]: pgmap v719: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.4 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 575 op/s; 199/117525 objects degraded (0.169%); 9789/117525 objects misplaced (8.329%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:09:16.997 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:17.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:17.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:17.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:17.121 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:17.143 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:17.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:17.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:17.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:17.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:17.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:17.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:17.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:17.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:17.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:17.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:17.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:17.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:17.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:17.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:17.665 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:17.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:17.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:17.790 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:17.792 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:17.892 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:17.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:17.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:17.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:18.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:18.042 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:18.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:18.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:18.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:18.173 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:18.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:18.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:18.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:18.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:18.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:18.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:18.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:18.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:18.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:18.572 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:18.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:18.676 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:18.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:18 smithi151 ceph-mon[16639]: pgmap v720: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.3 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 199/114120 objects degraded (0.174%); 9789/114120 objects misplaced (8.578%); 4.1 MiB/s, 65 objects/s recovering 2024-09-02T10:09:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1074045018' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:09:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:18 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:09:18.699 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:18.700 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:18 smithi187 ceph-mon[20072]: pgmap v720: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.3 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 199/114120 objects degraded (0.174%); 9789/114120 objects misplaced (8.578%); 4.1 MiB/s, 65 objects/s recovering 2024-09-02T10:09:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1074045018' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:09:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:18 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:09:18.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:18.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:18.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:18.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:18.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:18 smithi162 ceph-mon[20279]: pgmap v720: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.3 GiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 511 op/s; 199/114120 objects degraded (0.174%); 9789/114120 objects misplaced (8.578%); 4.1 MiB/s, 65 objects/s recovering 2024-09-02T10:09:18.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1074045018' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:09:18.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:18 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:09:18.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:18.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:18.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:19.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:19.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:19.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:19.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:19.227 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:19.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:19.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:19.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:19.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:19.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:19.495 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:19.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:19.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:19.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:19.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:19.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:19.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:19 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:09:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:19 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:09:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:19 smithi187 ceph-mon[20072]: osdmap e292: 12 total, 12 up, 11 in 2024-09-02T10:09:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:19 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:09:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:19 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:09:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:19 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:09:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:19 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:09:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 199/114120 objects degraded (0.174%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:19 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:09:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:19 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:09:19.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:19.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:19 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:09:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:19 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:09:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:19 smithi162 ceph-mon[20279]: osdmap e292: 12 total, 12 up, 11 in 2024-09-02T10:09:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:19 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:09:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:19 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:09:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:19 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:09:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:19 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:09:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 199/114120 objects degraded (0.174%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:19 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:09:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:19 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:09:19.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:19.892 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:19.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:19.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:19 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:09:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:19 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:09:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:19 smithi151 ceph-mon[16639]: osdmap e292: 12 total, 12 up, 11 in 2024-09-02T10:09:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:19 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:09:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:19 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:09:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:19 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:09:19.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:19 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:09:19.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 199/114120 objects degraded (0.174%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:19.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:19 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:09:19.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:19 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:09:19.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:20.018 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:20.019 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:20.120 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:20.151 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:20.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:20.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:20.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:20.278 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:20.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:20.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:20.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:20.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:20.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:20.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:20.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:20.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:20.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:20.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:20.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:20 smithi187 ceph-mon[20072]: pgmap v722: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.2 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 558 op/s; 199/110547 objects degraded (0.180%); 9063/110547 objects misplaced (8.198%); 7.1 MiB/s, 113 objects/s recovering 2024-09-02T10:09:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:20 smithi187 ceph-mon[20072]: osdmap e293: 12 total, 12 up, 11 in 2024-09-02T10:09:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:20 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:09:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:20 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:09:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:20 smithi187 ceph-mon[20072]: 3.9 scrub starts 2024-09-02T10:09:20.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:20.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:20.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:20.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:20.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:20 smithi162 ceph-mon[20279]: pgmap v722: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.2 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 558 op/s; 199/110547 objects degraded (0.180%); 9063/110547 objects misplaced (8.198%); 7.1 MiB/s, 113 objects/s recovering 2024-09-02T10:09:20.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:20 smithi162 ceph-mon[20279]: osdmap e293: 12 total, 12 up, 11 in 2024-09-02T10:09:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:20 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:09:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:20 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:09:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:20 smithi162 ceph-mon[20279]: 3.9 scrub starts 2024-09-02T10:09:20.889 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:20.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:20.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:20.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:20 smithi151 ceph-mon[16639]: pgmap v722: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.2 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 558 op/s; 199/110547 objects degraded (0.180%); 9063/110547 objects misplaced (8.198%); 7.1 MiB/s, 113 objects/s recovering 2024-09-02T10:09:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:20 smithi151 ceph-mon[16639]: osdmap e293: 12 total, 12 up, 11 in 2024-09-02T10:09:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:20 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:09:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:20 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:09:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:20 smithi151 ceph-mon[16639]: 3.9 scrub starts 2024-09-02T10:09:21.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:21.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:21.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:21.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:21.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:21.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:21.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:21.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:21.292 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:21.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:21.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:21.418 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:21.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:21.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:21.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:21.542 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:21.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:21.644 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:21.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:21.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:21 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:09:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:21 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:09:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:21 smithi187 ceph-mon[20072]: 3.9 scrub ok 2024-09-02T10:09:21.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:21.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:21.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:21.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:21 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:09:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:21 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:09:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:21 smithi162 ceph-mon[20279]: 3.9 scrub ok 2024-09-02T10:09:21.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:21.916 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:21.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:21.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:21 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:09:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:21 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:09:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:21 smithi151 ceph-mon[16639]: 3.9 scrub ok 2024-09-02T10:09:22.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:22.041 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:22.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:22.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:22.165 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:22.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:22.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:22.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:22.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:22.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:22.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:22.425 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:22.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:22.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:22.559 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:22.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:22.661 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:22.686 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:22.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:22.687 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:22.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:22 smithi162 ceph-mon[20279]: pgmap v724: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.2 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 519 op/s; 199/108285 objects degraded (0.184%); 9063/108285 objects misplaced (8.370%); 5.6 MiB/s, 89 objects/s recovering 2024-09-02T10:09:22.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:22 smithi162 ceph-mon[20279]: 3.a scrub starts 2024-09-02T10:09:22.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:22 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:09:22.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:22 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:09:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:22 smithi187 ceph-mon[20072]: pgmap v724: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.2 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 519 op/s; 199/108285 objects degraded (0.184%); 9063/108285 objects misplaced (8.370%); 5.6 MiB/s, 89 objects/s recovering 2024-09-02T10:09:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:22 smithi187 ceph-mon[20072]: 3.a scrub starts 2024-09-02T10:09:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:22 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:09:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:22 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:09:22.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:22.811 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:22.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:22.859 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:09:22.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:22 smithi151 ceph-mon[16639]: pgmap v724: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.2 GiB data, 9.0 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 519 op/s; 199/108285 objects degraded (0.184%); 9063/108285 objects misplaced (8.370%); 5.6 MiB/s, 89 objects/s recovering 2024-09-02T10:09:22.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:22 smithi151 ceph-mon[16639]: 3.a scrub starts 2024-09-02T10:09:22.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:22 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:09:22.860 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:22 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:09:22.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:22.937 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:22.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:23.038 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:23.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:23.061 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:23.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:23.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:23.188 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:23.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:23.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:23.316 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:23.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:23.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:23.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:23.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:23.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:23.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:23.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:23.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:23.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:23.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:23.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:23 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:09:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:23 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:09:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:23 smithi187 ceph-mon[20072]: 3.a scrub ok 2024-09-02T10:09:23.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:23.838 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:23.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:23.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:23 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:09:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:23 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:09:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:23 smithi162 ceph-mon[20279]: 3.a scrub ok 2024-09-02T10:09:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:23 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:09:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:23 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:09:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:23 smithi151 ceph-mon[16639]: 3.a scrub ok 2024-09-02T10:09:23.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:23.964 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:23.965 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:24.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:24.089 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:24.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:24.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:24.212 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:24.213 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:24.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:24.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:24.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:24.338 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:24.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:24.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:24.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:24.462 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:24.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:24.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:24.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:24.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:24.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:24.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:24.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:24 smithi187 ceph-mon[20072]: pgmap v725: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 199/103725 objects degraded (0.192%); 8503/103725 objects misplaced (8.198%); 7.3 MiB/s, 116 objects/s recovering 2024-09-02T10:09:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:24 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:09:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:24 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:09:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:24 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:09:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:24 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:09:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:24 smithi187 ceph-mon[20072]: osdmap e294: 12 total, 12 up, 11 in 2024-09-02T10:09:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 199/103725 objects degraded (0.192%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:24 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:09:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:24 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:09:24.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:24 smithi162 ceph-mon[20279]: pgmap v725: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 199/103725 objects degraded (0.192%); 8503/103725 objects misplaced (8.198%); 7.3 MiB/s, 116 objects/s recovering 2024-09-02T10:09:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:24 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:09:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:24 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:09:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:24 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:09:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:24 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:09:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:24 smithi162 ceph-mon[20279]: osdmap e294: 12 total, 12 up, 11 in 2024-09-02T10:09:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 199/103725 objects degraded (0.192%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:24 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:09:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:24 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:09:24.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:24.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:24.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:24 smithi151 ceph-mon[16639]: pgmap v725: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 574 op/s; 199/103725 objects degraded (0.192%); 8503/103725 objects misplaced (8.198%); 7.3 MiB/s, 116 objects/s recovering 2024-09-02T10:09:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:24 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:09:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:24 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:09:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:24 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:09:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:24 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:09:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:24 smithi151 ceph-mon[16639]: osdmap e294: 12 total, 12 up, 11 in 2024-09-02T10:09:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 199/103725 objects degraded (0.192%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:09:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:24 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:09:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:24 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:09:24.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:24.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:24.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:24.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:25.070 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:09:25.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:25.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:25.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:25.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:25.249 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:25.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:25.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:25.375 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:25.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:25.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:09:25.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:25.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:25.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:25.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:25.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:25.630 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:25.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:25.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:25.717 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:25 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:09:25.717 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:25 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:09:25.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:25 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:09:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:25 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:09:25.758 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:25.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:25 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:09:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:25 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:09:25.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:25.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:25.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:25.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:26.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:26.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:26.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:26.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:26.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:26.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:26.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:26.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:26.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:26.292 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:26.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:26.430 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:26.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:26.500 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.9 pg_upmap_items mapping to [1->7,10->9,0->5] 2024-09-02T10:09:26.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:26.556 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:26.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:26.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:26.681 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:26.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:26 smithi187 ceph-mon[20072]: pgmap v727: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 566 op/s; 199/100551 objects degraded (0.198%); 8326/100551 objects misplaced (8.280%); 6.9 MiB/s, 109 objects/s recovering 2024-09-02T10:09:26.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:26 smithi187 ceph-mon[20072]: osdmap e295: 12 total, 12 up, 11 in 2024-09-02T10:09:26.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:26 smithi187 ceph-mon[20072]: 3.3 scrub starts 2024-09-02T10:09:26.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:26 smithi187 ceph-mon[20072]: 3.3 scrub ok 2024-09-02T10:09:26.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2900825142' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [1, 7, 10, 9, 0, 5]} : dispatch 2024-09-02T10:09:26.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:26 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [1, 7, 10, 9, 0, 5]} : dispatch 2024-09-02T10:09:26.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:26.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:26.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:26.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:26.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:26 smithi162 ceph-mon[20279]: pgmap v727: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 566 op/s; 199/100551 objects degraded (0.198%); 8326/100551 objects misplaced (8.280%); 6.9 MiB/s, 109 objects/s recovering 2024-09-02T10:09:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:26 smithi162 ceph-mon[20279]: osdmap e295: 12 total, 12 up, 11 in 2024-09-02T10:09:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:26 smithi162 ceph-mon[20279]: 3.3 scrub starts 2024-09-02T10:09:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:26 smithi162 ceph-mon[20279]: 3.3 scrub ok 2024-09-02T10:09:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2900825142' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [1, 7, 10, 9, 0, 5]} : dispatch 2024-09-02T10:09:26.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:26 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [1, 7, 10, 9, 0, 5]} : dispatch 2024-09-02T10:09:26.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:26.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:26 smithi151 ceph-mon[16639]: pgmap v727: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.0 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 566 op/s; 199/100551 objects degraded (0.198%); 8326/100551 objects misplaced (8.280%); 6.9 MiB/s, 109 objects/s recovering 2024-09-02T10:09:26.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:26 smithi151 ceph-mon[16639]: osdmap e295: 12 total, 12 up, 11 in 2024-09-02T10:09:26.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:26 smithi151 ceph-mon[16639]: 3.3 scrub starts 2024-09-02T10:09:26.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:26 smithi151 ceph-mon[16639]: 3.3 scrub ok 2024-09-02T10:09:26.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2900825142' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [1, 7, 10, 9, 0, 5]} : dispatch 2024-09-02T10:09:26.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:26 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [1, 7, 10, 9, 0, 5]} : dispatch 2024-09-02T10:09:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:26.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:26.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:27.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:27.080 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:27.081 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:27.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:27.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:27.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:27.206 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:27.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:27.323 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:09:27.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:27.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:27.394 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:09:27.480 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:27.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:27.504 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:27.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:27.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:27.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:27.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:27 smithi187 ceph-mon[20072]: 3.4 scrub starts 2024-09-02T10:09:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:27 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [1, 7, 10, 9, 0, 5]}]': finished 2024-09-02T10:09:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:27 smithi187 ceph-mon[20072]: osdmap e296: 12 total, 12 up, 11 in 2024-09-02T10:09:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:27 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:09:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:27 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:09:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:27 smithi187 ceph-mon[20072]: 3.4 scrub ok 2024-09-02T10:09:27.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:27.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:27.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:27.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:27 smithi162 ceph-mon[20279]: 3.4 scrub starts 2024-09-02T10:09:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:27 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [1, 7, 10, 9, 0, 5]}]': finished 2024-09-02T10:09:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:27 smithi162 ceph-mon[20279]: osdmap e296: 12 total, 12 up, 11 in 2024-09-02T10:09:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:27 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:09:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:27 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:09:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:27 smithi162 ceph-mon[20279]: 3.4 scrub ok 2024-09-02T10:09:27.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:27.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:27.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:27.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:27 smithi151 ceph-mon[16639]: 3.4 scrub starts 2024-09-02T10:09:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:27 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.9", "id": [1, 7, 10, 9, 0, 5]}]': finished 2024-09-02T10:09:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:27 smithi151 ceph-mon[16639]: osdmap e296: 12 total, 12 up, 11 in 2024-09-02T10:09:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:27 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:09:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:27 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:09:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:27 smithi151 ceph-mon[16639]: 3.4 scrub ok 2024-09-02T10:09:27.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:28.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:28.020 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:28.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:28.149 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:28.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:28.251 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:28.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:28.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:28.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:28.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:28.403 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:28.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:28.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:28.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:28.528 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:28.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:28.529 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:28.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:28.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:28.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:28.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:28 smithi187 ceph-mon[20072]: pgmap v730: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.0 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 199/96789 objects degraded (0.206%); 8326/96789 objects misplaced (8.602%); 5.8 MiB/s, 92 objects/s recovering 2024-09-02T10:09:28.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:28.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:28.789 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:28.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:28.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:28 smithi162 ceph-mon[20279]: pgmap v730: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.0 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 199/96789 objects degraded (0.206%); 8326/96789 objects misplaced (8.602%); 5.8 MiB/s, 92 objects/s recovering 2024-09-02T10:09:28.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:28.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:28.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:28.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:28 smithi151 ceph-mon[16639]: pgmap v730: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 2.0 GiB data, 8.3 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 199/96789 objects degraded (0.206%); 8326/96789 objects misplaced (8.602%); 5.8 MiB/s, 92 objects/s recovering 2024-09-02T10:09:29.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:29.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:29.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:29.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:29.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:29.200 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:29.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:29.201 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:29.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:29.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:29.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:29.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:29.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:09:29.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:29.447 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:29.448 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:29.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:29.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:29.570 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:29.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:29 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:09:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:29 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:09:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:29 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:09:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:29 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:09:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:29 smithi151 ceph-mon[16639]: osdmap e297: 12 total, 12 up, 11 in 2024-09-02T10:09:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:29 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 199/96789 objects degraded (0.206%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:29.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:29.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:29.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:29.831 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:29.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:29.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:29 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:09:29.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:29 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:09:29.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:29 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:09:29.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:29 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:09:29.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:29 smithi162 ceph-mon[20279]: osdmap e297: 12 total, 12 up, 11 in 2024-09-02T10:09:29.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:29 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 199/96789 objects degraded (0.206%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:29.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:29.958 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:29.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:29.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:29 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:09:30.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:29 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:09:30.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:29 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:09:30.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:29 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:09:30.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:29 smithi187 ceph-mon[20072]: osdmap e297: 12 total, 12 up, 11 in 2024-09-02T10:09:30.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:29 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 199/96789 objects degraded (0.206%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:30.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:30.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:30.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:30.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:30.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:30.227 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:30.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:30.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:30.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:30.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:30.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:30.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:30.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:30.486 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:30.487 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:30.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:30.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:30.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:30.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:30 smithi151 ceph-mon[16639]: pgmap v732: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.9 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 200/92820 objects degraded (0.215%); 7609/92820 objects misplaced (8.198%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:09:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4151995687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:09:30.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:30.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:30.743 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:30.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:30.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:30.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:30 smithi162 ceph-mon[20279]: pgmap v732: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.9 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 200/92820 objects degraded (0.215%); 7609/92820 objects misplaced (8.198%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:09:30.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4151995687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:09:30.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:30.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:30.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:30 smithi187 ceph-mon[20072]: pgmap v732: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.9 GiB data, 8.1 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 608 op/s; 200/92820 objects degraded (0.215%); 7609/92820 objects misplaced (8.198%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:09:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4151995687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:09:31.007 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:31.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:31.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:31.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:31.134 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:31.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:31.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:31.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:31.261 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:31.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:31.311 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:09:31.311 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:09:31.311 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:09:31.311 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:09:31.311 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:09:31.311 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:09:31.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:09:31.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:09:31.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:09:31.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:09:31.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:09:31.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:09:31.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:09:31.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:09:31.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:09:31.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:09:31.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:09:31.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:09:31.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:09:31.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:09:31.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:09:31.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:09:31.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:09:31.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:09:31.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:31.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:31.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:31.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:31.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:31.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:31.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:31.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:31.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:31.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:31.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:31.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:31.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:31.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:31.889 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:31.911 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:31.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:32.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:32.037 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:32.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:32.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:32.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:32.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:32.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:32.291 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11] out_osds: [10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:09:32.291 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:09:32.291 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-09-02T10:09:32.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd in 10 2024-09-02T10:09:32.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:32.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:32.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:32.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:32.439 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:32.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:32.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:32.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:32.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:32.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:32.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:32.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:32.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:32.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:32.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:32 smithi162 ceph-mon[20279]: pgmap v733: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.8 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 200/90585 objects degraded (0.221%); 7609/90585 objects misplaced (8.400%); 5.5 MiB/s, 87 objects/s recovering 2024-09-02T10:09:32.848 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:32.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:32.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:32.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:32 smithi151 ceph-mon[16639]: pgmap v733: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.8 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 200/90585 objects degraded (0.221%); 7609/90585 objects misplaced (8.400%); 5.5 MiB/s, 87 objects/s recovering 2024-09-02T10:09:32.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:32.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:32.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:32.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:32 smithi187 ceph-mon[20072]: pgmap v733: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.8 GiB data, 8.0 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 553 op/s; 200/90585 objects degraded (0.221%); 7609/90585 objects misplaced (8.400%); 5.5 MiB/s, 87 objects/s recovering 2024-09-02T10:09:33.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:33.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:33.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:33.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:33.198 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:33.220 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:33.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:33.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:33.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:33.346 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:33.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:33.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:33.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:33.472 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:33.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:33.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:33.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:33.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:33.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:33.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:33.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:33.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:33.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:33.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:33.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:33.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:33.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:34.009 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:34.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:34.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:34.136 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:34.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:34.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:34.259 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:34.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:34.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:34.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:34.388 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:34.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:34.489 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:34.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:34.514 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:34.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:34.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:34.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:34.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:34.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:34.775 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:34.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:34 smithi162 ceph-mon[20279]: pgmap v734: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.8 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 200/86259 objects degraded (0.232%); 7029/86259 objects misplaced (8.149%); 7.5 MiB/s, 119 objects/s recovering 2024-09-02T10:09:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:34 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:09:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:34 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:09:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:34 smithi162 ceph-mon[20279]: osdmap e298: 12 total, 12 up, 11 in 2024-09-02T10:09:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:34 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 200/86259 objects degraded (0.232%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:34 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:09:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:34 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:09:34.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:34.903 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:34.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:34.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:34 smithi151 ceph-mon[16639]: pgmap v734: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.8 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 200/86259 objects degraded (0.232%); 7029/86259 objects misplaced (8.149%); 7.5 MiB/s, 119 objects/s recovering 2024-09-02T10:09:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:34 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:09:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:34 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:09:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:34 smithi151 ceph-mon[16639]: osdmap e298: 12 total, 12 up, 11 in 2024-09-02T10:09:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:34 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 200/86259 objects degraded (0.232%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:34 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:09:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:34 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:09:35.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:34 smithi187 ceph-mon[20072]: pgmap v734: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.8 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 618 op/s; 200/86259 objects degraded (0.232%); 7029/86259 objects misplaced (8.149%); 7.5 MiB/s, 119 objects/s recovering 2024-09-02T10:09:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:34 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:09:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:34 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:09:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:34 smithi187 ceph-mon[20072]: osdmap e298: 12 total, 12 up, 11 in 2024-09-02T10:09:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:34 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 200/86259 objects degraded (0.232%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:34 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:09:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:34 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:09:35.033 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:35.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:35.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:35.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:35.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:35.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:35.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:35.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:35.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:35.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:35.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:35.415 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:35.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:35.517 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:35.539 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:35.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:35.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:35.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:35.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:35.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:35.778 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:35.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:35.804 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:35.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:35 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:09:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:35 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:09:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:35 smithi162 ceph-mon[20279]: osdmap e299: 12 total, 12 up, 11 in 2024-09-02T10:09:35.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:35.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:35 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:09:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:35 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:09:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:35 smithi151 ceph-mon[16639]: osdmap e299: 12 total, 12 up, 11 in 2024-09-02T10:09:35.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:35.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:35 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:09:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:35 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:09:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:35 smithi187 ceph-mon[20072]: osdmap e299: 12 total, 12 up, 11 in 2024-09-02T10:09:36.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:36.064 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:36.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:36.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:36.191 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:36.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:36.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:36.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:36.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:36.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:36.443 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:36.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:36.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:36.572 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:36.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:36.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:36.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:36.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:36.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:36.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:36.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:36.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:36 smithi162 ceph-mon[20279]: pgmap v737: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.7 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 200/83202 objects degraded (0.240%); 6858/83202 objects misplaced (8.243%); 6.9 MiB/s, 110 objects/s recovering 2024-09-02T10:09:37.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:37.072 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:36 smithi187 ceph-mon[20072]: pgmap v737: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.7 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 200/83202 objects degraded (0.240%); 6858/83202 objects misplaced (8.243%); 6.9 MiB/s, 110 objects/s recovering 2024-09-02T10:09:37.072 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:36 smithi151 ceph-mon[16639]: pgmap v737: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.7 GiB data, 7.6 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 565 op/s; 200/83202 objects degraded (0.240%); 6858/83202 objects misplaced (8.243%); 6.9 MiB/s, 110 objects/s recovering 2024-09-02T10:09:37.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:37.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:37.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:37.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:37.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:37.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:37.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:37.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:37.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:37.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:37.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:37.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:37.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:37.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:37.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:37.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:37.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:37.752 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:37.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:37.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:37.879 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:37.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:37.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:38.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:38.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:38.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:38.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:38.139 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:38.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:38.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:38.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:38.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:38.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:38.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:38.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:38.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:38.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:38.526 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:38.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:38.628 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:38.650 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:38.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:38.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:38.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:38.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:38 smithi162 ceph-mon[20279]: pgmap v738: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.6 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 200/79398 objects degraded (0.252%); 6858/79398 objects misplaced (8.637%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:09:38.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:38.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:38.907 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:38.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:38 smithi151 ceph-mon[16639]: pgmap v738: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.6 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 200/79398 objects degraded (0.252%); 6858/79398 objects misplaced (8.637%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:09:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:38 smithi187 ceph-mon[20072]: pgmap v738: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.6 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 559 op/s; 200/79398 objects degraded (0.252%); 6858/79398 objects misplaced (8.637%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:09:39.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:39.034 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:39.035 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:39.136 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:39.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:39.161 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:39.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:39.287 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:39.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:39.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:39.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:39.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:39.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:39.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:39.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:39.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:39.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:39.678 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:39.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:39.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:39.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:39.808 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:39.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:39.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:39 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:09:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:39 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:09:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 200/79398 objects degraded (0.252%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:39.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:39.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:39.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:39.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:39 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:09:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:39 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:09:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 200/79398 objects degraded (0.252%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:39 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:09:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:39 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:09:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 200/79398 objects degraded (0.252%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:40.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:40.056 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:40.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:40.057 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:40.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:40.180 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:40.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:40.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:40.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:40.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:40.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:40.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:40.441 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:40.441 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:40.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:40.569 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:40.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:40.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:40.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:40.698 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:40.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:40.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:40.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:40.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:40.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:40 smithi162 ceph-mon[20279]: pgmap v739: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 225/75369 objects degraded (0.299%); 6119/75369 objects misplaced (8.119%); 8.6 MiB/s, 137 objects/s recovering 2024-09-02T10:09:40.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:40.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:40 smithi151 ceph-mon[16639]: pgmap v739: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 225/75369 objects degraded (0.299%); 6119/75369 objects misplaced (8.119%); 8.6 MiB/s, 137 objects/s recovering 2024-09-02T10:09:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:40.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:40.957 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:40 smithi187 ceph-mon[20072]: pgmap v739: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 633 op/s; 225/75369 objects degraded (0.299%); 6119/75369 objects misplaced (8.119%); 8.6 MiB/s, 137 objects/s recovering 2024-09-02T10:09:41.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:41.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:41.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:41.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:41.214 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:41.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:41.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:41.337 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:41.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:41.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:41.464 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:41.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:41.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:41.587 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:41.588 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:41.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:41.716 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:41.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:41.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:41.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:41.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:41.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:41.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:41.968 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:41.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:41.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:42.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:42.091 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:42.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:42.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:42.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:42.217 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:42.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:42.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:42.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:42.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:42.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:42.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:42.472 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:42.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:42.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:42.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:42.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:42.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:42.734 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:42.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:42.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:42.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:42 smithi151 ceph-mon[16639]: pgmap v740: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 225/73098 objects degraded (0.308%); 6119/73098 objects misplaced (8.371%); 5.6 MiB/s, 88 objects/s recovering 2024-09-02T10:09:42.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:42 smithi162 ceph-mon[20279]: pgmap v740: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 225/73098 objects degraded (0.308%); 6119/73098 objects misplaced (8.371%); 5.6 MiB/s, 88 objects/s recovering 2024-09-02T10:09:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:42.866 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:42.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:42.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:42.993 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:42.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:42 smithi187 ceph-mon[20072]: pgmap v740: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.5 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 548 op/s; 225/73098 objects degraded (0.308%); 6119/73098 objects misplaced (8.371%); 5.6 MiB/s, 88 objects/s recovering 2024-09-02T10:09:43.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:43.122 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:43.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:43.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:43.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:43.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:43.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:43.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:43.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:43.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:43.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:43.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:43.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:43.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:43.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:43.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:43.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:43.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:43.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:43.784 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:43.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:43.908 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:43.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:43.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:44.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:44.032 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:44.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:44.074 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:09:44.074 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:09:44.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:44.155 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:44.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:44.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:44.285 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:44.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:44.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:44.412 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:44.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:44.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:44.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:44.537 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:44.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:44.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:44.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:44.663 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:44.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:44.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:44.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:44.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:44 smithi162 ceph-mon[20279]: pgmap v741: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.4 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 225/68505 objects degraded (0.328%); 5500/68505 objects misplaced (8.029%); 7.4 MiB/s, 117 objects/s recovering 2024-09-02T10:09:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:44 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:09:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:44 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:09:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:44 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 225/68505 objects degraded (0.328%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:44.888 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:44.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:09:44.911 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:44.912 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:44.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:44 smithi151 ceph-mon[16639]: pgmap v741: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.4 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 225/68505 objects degraded (0.328%); 5500/68505 objects misplaced (8.029%); 7.4 MiB/s, 117 objects/s recovering 2024-09-02T10:09:44.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:44 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:09:44.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:44 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:09:44.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:44 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 225/68505 objects degraded (0.328%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:44 smithi187 ceph-mon[20072]: pgmap v741: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.4 GiB data, 6.8 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 592 op/s; 225/68505 objects degraded (0.328%); 5500/68505 objects misplaced (8.029%); 7.4 MiB/s, 117 objects/s recovering 2024-09-02T10:09:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:44 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:09:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:44 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:09:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:44 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 225/68505 objects degraded (0.328%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:45.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:45.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:45.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:45.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:45.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:45.175 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:45.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:45.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:45.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:45.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:45.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:45.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:45.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:45.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:45.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:45.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:45.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:45.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:45.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:45.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:45.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:45.810 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:45.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:45.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:45.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:45.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:45.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:45.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:46.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:46.088 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:46.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:46.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:46.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:46.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:46.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:46.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:46.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:46.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:46.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:46.478 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:46.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:46.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:46.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:46.602 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:46.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:46.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:46.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:46.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:46.828 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:46.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:46 smithi162 ceph-mon[20279]: pgmap v742: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.3 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 573 op/s; 233/64881 objects degraded (0.359%); 5299/64881 objects misplaced (8.167%); 6.8 MiB/s, 108 objects/s recovering 2024-09-02T10:09:46.852 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:46.853 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:46 smithi151 ceph-mon[16639]: pgmap v742: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.3 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 573 op/s; 233/64881 objects degraded (0.359%); 5299/64881 objects misplaced (8.167%); 6.8 MiB/s, 108 objects/s recovering 2024-09-02T10:09:46.953 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:46.975 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:46.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:46 smithi187 ceph-mon[20072]: pgmap v742: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.3 GiB data, 6.6 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 573 op/s; 233/64881 objects degraded (0.359%); 5299/64881 objects misplaced (8.167%); 6.8 MiB/s, 108 objects/s recovering 2024-09-02T10:09:47.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:47.101 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:47.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:47.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:47.225 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:47.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:47.327 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:47.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:47.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:47.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:47.474 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:47.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:47.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:47.603 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:47.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:47.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:47.729 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:47.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:47.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:47.855 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:47.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:47.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:47.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:47.982 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:47.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:48.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:48.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:48.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:48.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:48.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:48.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:48.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:48.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:48.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:48.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:48.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:48.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:48.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:48.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:48.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:48.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:48.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:48.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:48.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:48.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:48.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:48.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:48 smithi162 ceph-mon[20279]: pgmap v743: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.2 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 233/60798 objects degraded (0.383%); 5299/60798 objects misplaced (8.716%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:09:48.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:48.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:48.876 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:48.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:48 smithi151 ceph-mon[16639]: pgmap v743: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.2 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 233/60798 objects degraded (0.383%); 5299/60798 objects misplaced (8.716%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:09:48.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:49.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:49.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:48 smithi187 ceph-mon[20072]: pgmap v743: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.2 GiB data, 6.4 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 233/60798 objects degraded (0.383%); 5299/60798 objects misplaced (8.716%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:09:49.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:49.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:49.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:49.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:49.258 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:49.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:49.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:49.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:49.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:49.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:49.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:49.521 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:49.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:49.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:49.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:49.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:49.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:49.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:49.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:49.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:49.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:49.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:49 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 233/60798 objects degraded (0.383%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:49.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:49.911 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:49.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:49.912 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:49.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:49 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 233/60798 objects degraded (0.383%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:49 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 233/60798 objects degraded (0.383%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:50.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:50.039 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:50.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:50.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:50.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:50.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:50.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:50.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:50.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:50.290 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:50.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:50.412 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:50.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:50.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:50.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:50.540 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:50.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:50.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:50.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:50.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:50.664 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:50.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:50.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:50.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:50.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:50.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:50.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:50 smithi162 ceph-mon[20279]: pgmap v744: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.2 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 267/56793 objects degraded (0.470%); 4484/56793 objects misplaced (7.895%); 9.2 MiB/s, 146 objects/s recovering 2024-09-02T10:09:50.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:50.912 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:50 smithi151 ceph-mon[16639]: pgmap v744: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.2 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 267/56793 objects degraded (0.470%); 4484/56793 objects misplaced (7.895%); 9.2 MiB/s, 146 objects/s recovering 2024-09-02T10:09:50.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:50.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:50.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:50.971 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:09:50.971 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:09:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:50 smithi187 ceph-mon[20072]: pgmap v744: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.2 GiB data, 6.1 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 627 op/s; 267/56793 objects degraded (0.470%); 4484/56793 objects misplaced (7.895%); 9.2 MiB/s, 146 objects/s recovering 2024-09-02T10:09:51.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:09:51.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:51.056 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:51.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:51.083 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:09:51.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:51.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:51.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:51.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:51.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:51.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:51.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:51.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:51.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:51.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:51.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:51.574 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:51.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:51.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:51.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:51.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:51.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:51.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:51.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:51.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:51.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:51.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:52.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:52.122 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:52.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:52.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:52.248 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:52.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:52.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:52.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:52.379 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:52.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:52.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:52.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:52.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:52.505 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:52.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:52.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:52.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:52 smithi151 ceph-mon[16639]: pgmap v745: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 267/54354 objects degraded (0.491%); 4484/54354 objects misplaced (8.250%); 6.3 MiB/s, 101 objects/s recovering 2024-09-02T10:09:52.737 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:52.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:52.762 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:52.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:52 smithi162 ceph-mon[20279]: pgmap v745: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 267/54354 objects degraded (0.491%); 4484/54354 objects misplaced (8.250%); 6.3 MiB/s, 101 objects/s recovering 2024-09-02T10:09:52.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:52.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:52.887 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:52.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:52 smithi187 ceph-mon[20072]: pgmap v745: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.1 GiB data, 6.0 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 583 op/s; 267/54354 objects degraded (0.491%); 4484/54354 objects misplaced (8.250%); 6.3 MiB/s, 101 objects/s recovering 2024-09-02T10:09:53.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:53.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:53.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:53.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:53.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:53.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:53.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:53.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:53.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:53.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:53.403 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:53.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:53.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:53.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:53.527 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:53.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:53.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:53.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:53.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:53.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:53.759 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:53.786 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:53.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:53.888 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:53.911 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:53.914 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:54.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:54.037 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:54.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:54.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:54.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:54.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:54.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:54.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:54.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:54.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:54.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:54.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:54.437 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:54.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:54.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:54.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:54.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:54 smithi151 ceph-mon[16639]: pgmap v746: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.0 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 267/49518 objects degraded (0.539%); 3824/49518 objects misplaced (7.722%); 8.6 MiB/s, 137 objects/s recovering 2024-09-02T10:09:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:54 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 267/49518 objects degraded (0.539%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:54.689 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:54.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:54.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:54.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:54.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:54.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:54 smithi162 ceph-mon[20279]: pgmap v746: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.0 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 267/49518 objects degraded (0.539%); 3824/49518 objects misplaced (7.722%); 8.6 MiB/s, 137 objects/s recovering 2024-09-02T10:09:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:54 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 267/49518 objects degraded (0.539%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:54.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:54.939 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:54.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:54 smithi187 ceph-mon[20072]: pgmap v746: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 1.0 GiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 654 op/s; 267/49518 objects degraded (0.539%); 3824/49518 objects misplaced (7.722%); 8.6 MiB/s, 137 objects/s recovering 2024-09-02T10:09:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:54 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 267/49518 objects degraded (0.539%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:55.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:55.070 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:55.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:55.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:55.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:55.199 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:55.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:55.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:55.323 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:55.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:55.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:55.425 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:55.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:55.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:55.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:55.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:55.574 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:55.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:55.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:55.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:55.700 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:55.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:55.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:55.802 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:55.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:55.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:55.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:55 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:09:55.926 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:55 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:09:55.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:55.953 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:55.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:55 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:09:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:55 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:09:56.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:56.079 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:56.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:56.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:55 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:09:56.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:55 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:09:56.180 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:56.206 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:56.206 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:56.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:56.331 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:56.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:56.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:56.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:56.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:56.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:56.559 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:56.589 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:56.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:56.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:56.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:56.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:56.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:56.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:56 smithi162 ceph-mon[20279]: pgmap v747: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 954 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 273/45792 objects degraded (0.596%); 3617/45792 objects misplaced (7.899%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:09:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:56 smithi162 ceph-mon[20279]: osdmap e300: 12 total, 12 up, 11 in 2024-09-02T10:09:56.848 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:56.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:56 smithi151 ceph-mon[16639]: pgmap v747: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 954 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 273/45792 objects degraded (0.596%); 3617/45792 objects misplaced (7.899%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:09:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:56 smithi151 ceph-mon[16639]: osdmap e300: 12 total, 12 up, 11 in 2024-09-02T10:09:56.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:56.977 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:56.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:56 smithi187 ceph-mon[20072]: pgmap v747: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 954 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 630 op/s; 273/45792 objects degraded (0.596%); 3617/45792 objects misplaced (7.899%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:09:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:56 smithi187 ceph-mon[20072]: osdmap e300: 12 total, 12 up, 11 in 2024-09-02T10:09:57.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:57.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:57.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:57.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:57.231 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:57.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:57.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:57.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:57.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:57.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:57.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:57.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:57.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:57.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:57.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:57.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:57.630 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:57.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:57.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:57.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:57.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:09:57.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:57.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:09:57.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:09:58.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:58.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:09:58.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:58.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:58.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:58.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:57 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:09:57.924+0000 7fe3ffed4640 -1 mon.a@0(leader).osd e300 definitely_dead 0 2024-09-02T10:09:58.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:09:58.283 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:58.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:09:58.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:09:58.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:58.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:58.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:09:58.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:09:58.531 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:58.532 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:09:58.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:58.658 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:58.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:58.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:09:58.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:58.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:58.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:09:58.892 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:58.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:58 smithi151 ceph-mon[16639]: pgmap v749: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 868 MiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 273/41628 objects degraded (0.656%); 3617/41628 objects misplaced (8.689%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:09:58.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1969878349' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:09:58.898 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:58 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:09:58.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:58.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:58 smithi187 ceph-mon[20072]: pgmap v749: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 868 MiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 273/41628 objects degraded (0.656%); 3617/41628 objects misplaced (8.689%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:09:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1969878349' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:09:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:58 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:09:59.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:09:59.043 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:59.044 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:09:59.092 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:09:59.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:58 smithi162 ceph-mon[20279]: pgmap v749: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 868 MiB data, 5.3 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 638 op/s; 273/41628 objects degraded (0.656%); 3617/41628 objects misplaced (8.689%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:09:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1969878349' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:09:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:58 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:09:59.144 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:59.165 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:59.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:59.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:09:59.289 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:59.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:09:59.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:09:59.412 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:59.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:09:59.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:09:59.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:09:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:59.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:09:59.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:59.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:59.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:59.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:09:59.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:09:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:59.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:09:59.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:09:59.910 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:09:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:09:59.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:09:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-02T10:09:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: osdmap e301: 12 total, 12 up, 12 in 2024-09-02T10:09:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3256364561' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:09:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 273/41628 objects degraded (0.656%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:09:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:09:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3256364561' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:09:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: osdmap e302: 12 total, 12 up, 12 in 2024-09-02T10:09:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:09:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:09:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:09:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:09:59 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:10:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-02T10:10:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: osdmap e301: 12 total, 12 up, 12 in 2024-09-02T10:10:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3256364561' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:10:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 273/41628 objects degraded (0.656%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:10:00.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3256364561' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:10:00.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: osdmap e302: 12 total, 12 up, 12 in 2024-09-02T10:10:00.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:10:00.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:10:00.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:10:00.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:09:59 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:10:00.012 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:00.034 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:00.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:00.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:00.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-02T10:10:00.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: osdmap e301: 12 total, 12 up, 12 in 2024-09-02T10:10:00.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3256364561' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:10:00.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 273/41628 objects degraded (0.656%), 2 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:00.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:10:00.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3256364561' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:10:00.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: osdmap e302: 12 total, 12 up, 12 in 2024-09-02T10:10:00.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:10:00.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:10:00.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:10:00.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:09:59 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:10:00.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:00.160 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:00.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:00.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:00.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:00.286 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:00.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:00.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:00.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:00.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:00.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:00.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:00.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:00.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:00.681 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:00.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:10:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:10:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: pgmap v752: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 776 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 681 op/s; 279/37245 objects degraded (0.749%); 2722/37245 objects misplaced (7.308%); 9.1 MiB/s, 145 objects/s recovering 2024-09-02T10:10:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:10:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:10:00.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: 3.1 starting backfill to osd.10 from (0'0,0'0] MAX to 300'12886 2024-09-02T10:10:00.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: osdmap e303: 12 total, 12 up, 12 in 2024-09-02T10:10:00.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: Health detail: HEALTH_WARN Degraded data redundancy: 279/37245 objects degraded (0.749%), 2 pgs degraded, 3 pgs undersized 2024-09-02T10:10:00.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: [WRN] PG_DEGRADED: Degraded data redundancy: 279/37245 objects degraded (0.749%), 2 pgs degraded, 3 pgs undersized 2024-09-02T10:10:00.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: pg 3.1 is stuck undersized for 92s, current state active+recovering+undersized+degraded+remapped, last acting [0,11] 2024-09-02T10:10:00.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: pg 3.7 is stuck undersized for 92s, current state active+recovering+undersized+degraded+remapped, last acting [3,7] 2024-09-02T10:10:00.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:00 smithi187 ceph-mon[20072]: pg 3.e is stuck undersized for 92s, current state active+recovering+undersized+remapped, last acting [0,3] 2024-09-02T10:10:00.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:00.805 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:00.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:00.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:00.932 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:00.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:00.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: pgmap v752: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 776 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 681 op/s; 279/37245 objects degraded (0.749%); 2722/37245 objects misplaced (7.308%); 9.1 MiB/s, 145 objects/s recovering 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: 3.1 starting backfill to osd.10 from (0'0,0'0] MAX to 300'12886 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: osdmap e303: 12 total, 12 up, 12 in 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: Health detail: HEALTH_WARN Degraded data redundancy: 279/37245 objects degraded (0.749%), 2 pgs degraded, 3 pgs undersized 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: [WRN] PG_DEGRADED: Degraded data redundancy: 279/37245 objects degraded (0.749%), 2 pgs degraded, 3 pgs undersized 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: pg 3.1 is stuck undersized for 92s, current state active+recovering+undersized+degraded+remapped, last acting [0,11] 2024-09-02T10:10:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: pg 3.7 is stuck undersized for 92s, current state active+recovering+undersized+degraded+remapped, last acting [3,7] 2024-09-02T10:10:00.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:00 smithi151 ceph-mon[16639]: pg 3.e is stuck undersized for 92s, current state active+recovering+undersized+remapped, last acting [0,3] 2024-09-02T10:10:01.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:01.067 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:01.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: pgmap v752: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 776 MiB data, 5.1 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 681 op/s; 279/37245 objects degraded (0.749%); 2722/37245 objects misplaced (7.308%); 9.1 MiB/s, 145 objects/s recovering 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: 3.1 starting backfill to osd.10 from (0'0,0'0] MAX to 300'12886 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: osdmap e303: 12 total, 12 up, 12 in 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: Health detail: HEALTH_WARN Degraded data redundancy: 279/37245 objects degraded (0.749%), 2 pgs degraded, 3 pgs undersized 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: [WRN] PG_DEGRADED: Degraded data redundancy: 279/37245 objects degraded (0.749%), 2 pgs degraded, 3 pgs undersized 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: pg 3.1 is stuck undersized for 92s, current state active+recovering+undersized+degraded+remapped, last acting [0,11] 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: pg 3.7 is stuck undersized for 92s, current state active+recovering+undersized+degraded+remapped, last acting [3,7] 2024-09-02T10:10:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:00 smithi162 ceph-mon[20279]: pg 3.e is stuck undersized for 92s, current state active+recovering+undersized+remapped, last acting [0,3] 2024-09-02T10:10:01.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:01.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:01.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:01.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:01.319 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:01.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:01.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:01.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:01.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:01.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:01.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:01.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:01.571 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:01.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:01.703 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:01.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:01.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:01.834 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:01.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:01.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:01.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:01 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:10:01.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:01 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:10:01.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:01 smithi151 ceph-mon[16639]: 3.8 scrub starts 2024-09-02T10:10:01.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:01 smithi151 ceph-mon[16639]: 3.8 scrub ok 2024-09-02T10:10:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:01.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:01.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:01 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:10:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:01 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:10:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:01 smithi187 ceph-mon[20072]: 3.8 scrub starts 2024-09-02T10:10:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:01 smithi187 ceph-mon[20072]: 3.8 scrub ok 2024-09-02T10:10:02.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:02.072 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.10. 2024-09-02T10:10:02.093 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:02.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:02.094 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:02.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:01 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:10:02.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:01 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:10:02.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:01 smithi162 ceph-mon[20279]: 3.8 scrub starts 2024-09-02T10:10:02.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:01 smithi162 ceph-mon[20279]: 3.8 scrub ok 2024-09-02T10:10:02.194 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:02.215 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:02.216 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:02.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:02.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:02.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:02.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:02.451 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:02.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:02.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:02.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:02.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:02.601 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:02.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:02.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:02.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:02.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:02.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:02.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:02.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:02.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:02.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:02 smithi151 ceph-mon[16639]: pgmap v754: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 736 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 279/35331 objects degraded (0.790%); 2722/35331 objects misplaced (7.704%); 7.5 MiB/s, 119 objects/s recovering 2024-09-02T10:10:02.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:02.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:02.987 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:02 smithi187 ceph-mon[20072]: pgmap v754: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 736 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 279/35331 objects degraded (0.790%); 2722/35331 objects misplaced (7.704%); 7.5 MiB/s, 119 objects/s recovering 2024-09-02T10:10:03.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:03.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:02 smithi162 ceph-mon[20279]: pgmap v754: 47 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 736 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 621 op/s; 279/35331 objects degraded (0.790%); 2722/35331 objects misplaced (7.704%); 7.5 MiB/s, 119 objects/s recovering 2024-09-02T10:10:03.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:03.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:03.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:03.236 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:03.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:03.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:03.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:03.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:03.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:03.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:03.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:03.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:03.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:03.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:03.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:03.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:03.764 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:03.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:03.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:03.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:03.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:03.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:04.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:04.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:04.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:04.155 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:04.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:04.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:04.279 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:04.280 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:04.381 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:04.412 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:04.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:04.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:04.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:04.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:04.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:04.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:04.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:04.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:04.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:04.763 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:04.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:04.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:04.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:04.908 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:04.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:04 smithi151 ceph-mon[16639]: pgmap v755: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 667 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 218/31977 objects degraded (0.682%); 2219/31977 objects misplaced (6.939%); 10 MiB/s, 166 objects/s recovering 2024-09-02T10:10:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 218/31977 objects degraded (0.682%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:04 smithi187 ceph-mon[20072]: pgmap v755: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 667 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 218/31977 objects degraded (0.682%); 2219/31977 objects misplaced (6.939%); 10 MiB/s, 166 objects/s recovering 2024-09-02T10:10:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 218/31977 objects degraded (0.682%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:05.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:05.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:05.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:05.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:05.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:04 smithi162 ceph-mon[20279]: pgmap v755: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 667 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 218/31977 objects degraded (0.682%); 2219/31977 objects misplaced (6.939%); 10 MiB/s, 166 objects/s recovering 2024-09-02T10:10:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 218/31977 objects degraded (0.682%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:05.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:05.164 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:05.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:05.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:05.292 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:05.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:05.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:05.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:05.418 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:05.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:05.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:05.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:05.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:05.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:05.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:05.672 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:05.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:05.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:05.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:05.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:05.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:05.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:05.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:05.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:05.924 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:06.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:06.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:06.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:06.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:06.185 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:06.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:06.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:06.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:06.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:06.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:06.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:06.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:06.447 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:06.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:06.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:06.585 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:06.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:06.687 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:06.710 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:06.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:06.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:06.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:06.835 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:06.836 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:06.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:06.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:06 smithi151 ceph-mon[16639]: pgmap v756: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 614 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 218/29463 objects degraded (0.740%); 1998/29463 objects misplaced (6.781%); 7.3 MiB/s, 116 objects/s recovering 2024-09-02T10:10:06.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:06 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:10:06.965 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:06.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:06.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:06 smithi187 ceph-mon[20072]: pgmap v756: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 614 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 218/29463 objects degraded (0.740%); 1998/29463 objects misplaced (6.781%); 7.3 MiB/s, 116 objects/s recovering 2024-09-02T10:10:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:06 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:10:07.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:07.093 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:07.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:06 smithi162 ceph-mon[20279]: pgmap v756: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 614 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 218/29463 objects degraded (0.740%); 1998/29463 objects misplaced (6.781%); 7.3 MiB/s, 116 objects/s recovering 2024-09-02T10:10:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:06 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:10:07.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:07.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:07.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:07.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:07.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:07.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:07.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:07.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:07.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:07.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:07.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:07.632 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:07.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:07.647 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:10:07.647 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:10:07.647 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:10:07.647 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:10:07.647 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:10:07.648 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:10:07.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:10:07.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:10:07.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:10:07.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:10:07.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:10:07.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:10:07.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:10:07.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:10:07.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:07.755 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:07.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:07.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:07.885 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:07.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:07.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2885581960' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:07.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2885581960' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:08.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:08.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:08.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:08.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2885581960' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:08.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:08.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:08.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:08.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:08.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:08.289 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:08.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:08.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:08.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:08.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:08.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:08.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:08.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:08.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:08.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:08.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:08.670 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:08.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:08.796 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:08.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:08.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:08.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:08.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:08.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:08 smithi151 ceph-mon[16639]: pgmap v757: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 528 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 477 op/s; 218/25341 objects degraded (0.860%); 1998/25341 objects misplaced (7.884%); 4.1 MiB/s, 65 objects/s recovering 2024-09-02T10:10:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:08 smithi187 ceph-mon[20072]: pgmap v757: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 528 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 477 op/s; 218/25341 objects degraded (0.860%); 1998/25341 objects misplaced (7.884%); 4.1 MiB/s, 65 objects/s recovering 2024-09-02T10:10:09.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:09.053 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:09.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:09.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:09.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:08 smithi162 ceph-mon[20279]: pgmap v757: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 528 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 477 op/s; 218/25341 objects degraded (0.860%); 1998/25341 objects misplaced (7.884%); 4.1 MiB/s, 65 objects/s recovering 2024-09-02T10:10:09.155 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:09.178 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:09.179 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:09.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:09.306 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:09.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:09.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:09.437 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:09.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:09.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:09.565 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:09.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:09.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:09.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:09.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:09.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:09.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:09.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:09.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 218/25341 objects degraded (0.860%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:10:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:10:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:10:09.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:09.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:09.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 218/25341 objects degraded (0.860%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:10:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:10:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:10:10.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:10.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 218/25341 objects degraded (0.860%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:10:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:10:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:10:10.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:10.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:10.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:10.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:10.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:10.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:10.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:10.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:10.372 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:10.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:10.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:10.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:10.504 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:10.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:10.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:10.633 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:10.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:10.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:10.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:10.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:10.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:10.759 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:10.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:10.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:10.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:10 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:10:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:10 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:10:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:10 smithi151 ceph-mon[16639]: osdmap e304: 12 total, 12 up, 12 in 2024-09-02T10:10:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:10:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:10:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:10:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:10:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:10:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:10:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:10:10.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:10 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:10:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:10 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:10:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:10 smithi187 ceph-mon[20072]: osdmap e304: 12 total, 12 up, 12 in 2024-09-02T10:10:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:10:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:10:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:10:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:10:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:10:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:10:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:10:11.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:11.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:11.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:10 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:10:11.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:10 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:10:11.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:10 smithi162 ceph-mon[20279]: osdmap e304: 12 total, 12 up, 12 in 2024-09-02T10:10:11.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:10:11.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:10:11.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:10:11.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:10:11.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:10:11.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:10:11.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:10:11.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:11.144 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:11.145 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:11.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:11.275 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:11.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:11.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:11.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:11.400 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:11.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:11.502 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:11.526 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:11.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:11.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:11.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:11.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:11.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:11.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:11.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:11.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:11.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:11.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:12.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:12.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:12.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:12.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:12.182 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:12.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:12.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:12.268 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-09-02T10:10:12.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:12.305 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:12.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:12.306 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:12.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:12.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:12.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:12.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:12.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:12.567 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:12.568 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:12.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:12.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:12.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:12.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:12.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:12.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:12.911 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:12 smithi151 ceph-mon[16639]: pgmap v760: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 44 active+clean; 385 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 218/18462 objects degraded (1.181%); 1207/18462 objects misplaced (6.538%); 7.4 MiB/s, 117 objects/s recovering 2024-09-02T10:10:12.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:12.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:12.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:12.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:12 smithi187 ceph-mon[20072]: pgmap v760: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 44 active+clean; 385 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 218/18462 objects degraded (1.181%); 1207/18462 objects misplaced (6.538%); 7.4 MiB/s, 117 objects/s recovering 2024-09-02T10:10:13.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:13.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:12 smithi162 ceph-mon[20279]: pgmap v760: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 44 active+clean; 385 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 218/18462 objects degraded (1.181%); 1207/18462 objects misplaced (6.538%); 7.4 MiB/s, 117 objects/s recovering 2024-09-02T10:10:13.104 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:13.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:13.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:13.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:13.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:13.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:13.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:13.366 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:13.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:13.421 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:13.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:13.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:13.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:13.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:13.632 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:13.633 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:13.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:13.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:13.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:13.759 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:13.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:13.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:13.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:13.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:14.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:14.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:14.127 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:10:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:10:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:10:14.127 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:10:14.127 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:10:14.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:14.155 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:14.156 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:14.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:14.281 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:14.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:14.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:14.383 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:14.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:14.409 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:14.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:14.534 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:14.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:14.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:14.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:14.657 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:14.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:14.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:14.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:14.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:14.786 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:14.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:14.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:14.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:14.913 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:14 smithi151 ceph-mon[16639]: pgmap v761: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 281 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 218/13482 objects degraded (1.617%); 607/13482 objects misplaced (4.502%); 7.7 MiB/s, 129 objects/s recovering 2024-09-02T10:10:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 218/13482 objects degraded (1.617%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:14.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:10:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:14 smithi187 ceph-mon[20072]: pgmap v761: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 281 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 218/13482 objects degraded (1.617%); 607/13482 objects misplaced (4.502%); 7.7 MiB/s, 129 objects/s recovering 2024-09-02T10:10:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 218/13482 objects degraded (1.617%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:15.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:15.043 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:15.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:14 smithi162 ceph-mon[20279]: pgmap v761: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 281 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 218/13482 objects degraded (1.617%); 607/13482 objects misplaced (4.502%); 7.7 MiB/s, 129 objects/s recovering 2024-09-02T10:10:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 218/13482 objects degraded (1.617%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:15.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:15.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:15.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:15.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:15.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:15.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:15.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:15.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:15.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:15.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:15.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:15.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:10:15.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:15.601 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:15.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:15.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:15.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:15.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:15.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:15.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:15.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:15.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:15.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:15.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:16.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:16.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:16.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:16.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:16.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:16.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:16.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:16.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:16.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:16.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:16.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:16.500 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:16.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:16.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:16.628 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:16.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:16.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:16.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:16.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:16.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:16.906 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:16.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:16 smithi151 ceph-mon[16639]: pgmap v762: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 199 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 664 op/s; 218/9525 objects degraded (2.289%); 370/9525 objects misplaced (3.885%); 7.8 MiB/s, 133 objects/s recovering 2024-09-02T10:10:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:16 smithi187 ceph-mon[20072]: pgmap v762: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 199 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 664 op/s; 218/9525 objects degraded (2.289%); 370/9525 objects misplaced (3.885%); 7.8 MiB/s, 133 objects/s recovering 2024-09-02T10:10:17.006 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:17.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:17.032 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:17.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:16 smithi162 ceph-mon[20279]: pgmap v762: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 199 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 664 op/s; 218/9525 objects degraded (2.289%); 370/9525 objects misplaced (3.885%); 7.8 MiB/s, 133 objects/s recovering 2024-09-02T10:10:17.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:17.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:17.161 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:17.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:17.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:17.269 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:10:17.269 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:10:17.269 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-02T10:10:17.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:10:17.283 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:17.284 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:17.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:17.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:17.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:17.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:17.533 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:17.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:17.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:10:17.600 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Removed 109540 objects 2024-09-02T10:10:17.600 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Clean up completed and total clean up time :163.563 2024-09-02T10:10:17.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:17.638 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:17.640 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:17.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:17.656 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:17.656 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:17.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:17.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:17.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:17.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:17.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:17.905 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:17.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:17.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:18.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:18.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:18.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:18.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:18.164 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:18.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:18.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:18.287 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:18.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:18.288 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:18.337 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:18.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:18.412 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:18.412 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:18.469 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:10:18.469 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:10:18.495 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:10:18.495 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:10:18.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:18.524 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:10:18.524 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:10:18.534 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:18.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:18.549 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:10:18.549 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:10:18.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:10:18.574 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:10:18.601 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:10:18.601 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:10:18.622 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:10:18.623 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:10:18.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:18.644 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:10:18.645 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:10:18.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:18.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:18.667 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:10:18.667 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:10:18.693 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:10:18.693 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:10:18.715 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:10:18.715 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:10:18.715 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:18 smithi151 ceph-mon[16639]: pgmap v763: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 104 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 218/4983 objects degraded (4.375%); 370/4983 objects misplaced (7.425%); 7.8 MiB/s, 133 objects/s recovering 2024-09-02T10:10:18.716 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3245165057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:18.739 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:10:18.739 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:10:18.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:18.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:18.783 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:18.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:18.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:18.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:18 smithi187 ceph-mon[20072]: pgmap v763: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 104 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 218/4983 objects degraded (4.375%); 370/4983 objects misplaced (7.425%); 7.8 MiB/s, 133 objects/s recovering 2024-09-02T10:10:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3245165057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:19.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:19.037 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:19.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:18 smithi162 ceph-mon[20279]: pgmap v763: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 104 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 678 op/s; 218/4983 objects degraded (4.375%); 370/4983 objects misplaced (7.425%); 7.8 MiB/s, 133 objects/s recovering 2024-09-02T10:10:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3245165057' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:10:19.119 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:10:19.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:19.165 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:19.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:19.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:19.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:19.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:19.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:19.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:19.412 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:19.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:19.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:19.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:19.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:19.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:19.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:19.659 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:19.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:19.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:19.783 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:19.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:19.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:19.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:19.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:19.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 218/4983 objects degraded (4.375%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 218/4983 objects degraded (4.375%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:20.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:20.035 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:20.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:20.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 218/4983 objects degraded (4.375%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:20.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:20.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:20.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:20.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:20.284 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:20.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:20.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:20.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:20.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:20.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:20.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:20.532 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:20.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:20.601 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:20.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:20.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:20.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:20.676 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:10:20.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:20.779 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:20.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:20.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:20.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:20.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:20.904 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:20 smithi151 ceph-mon[16639]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:10:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:20 smithi151 ceph-mon[16639]: pgmap v764: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 44 active+clean; 27 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 218/1254 objects degraded (17.384%); 5.0 MiB/s, 152 objects/s recovering 2024-09-02T10:10:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1552837158' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:10:21.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:20 smithi187 ceph-mon[20072]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:10:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:20 smithi187 ceph-mon[20072]: pgmap v764: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 44 active+clean; 27 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 218/1254 objects degraded (17.384%); 5.0 MiB/s, 152 objects/s recovering 2024-09-02T10:10:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1552837158' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:10:21.031 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:21.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:21.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:20 smithi162 ceph-mon[20279]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:10:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:20 smithi162 ceph-mon[20279]: pgmap v764: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 44 active+clean; 27 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 659 op/s; 218/1254 objects degraded (17.384%); 5.0 MiB/s, 152 objects/s recovering 2024-09-02T10:10:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1552837158' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:10:21.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:21.155 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:21.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:21.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:21.279 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:21.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:21.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:21.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:21.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:21.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:21.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:21.527 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:21.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:21.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:21.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:21.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:21.652 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:21.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:21.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:21.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:21.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:21.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:21.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:22.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:21 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:10:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1552837158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:10:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:21 smithi187 ceph-mon[20072]: osdmap e305: 12 total, 12 up, 12 in 2024-09-02T10:10:22.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:22.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:22.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:21 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:10:22.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1552837158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:10:22.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:21 smithi162 ceph-mon[20279]: osdmap e305: 12 total, 12 up, 12 in 2024-09-02T10:10:22.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:22.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:22.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:21 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:10:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1552837158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:10:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:21 smithi151 ceph-mon[16639]: osdmap e305: 12 total, 12 up, 12 in 2024-09-02T10:10:22.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:22.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:22.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:22.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:22.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:22.395 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:22.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:22.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:22.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:22.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:22.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:22.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:22.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:22.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:22.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:22.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:22.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:22.900 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:22.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:22 smithi162 ceph-mon[20279]: pgmap v766: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 44 active+clean; 4.7 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 218/204 objects degraded (106.863%); 5.0 MiB/s, 152 objects/s recovering 2024-09-02T10:10:23.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:22 smithi187 ceph-mon[20072]: pgmap v766: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 44 active+clean; 4.7 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 218/204 objects degraded (106.863%); 5.0 MiB/s, 152 objects/s recovering 2024-09-02T10:10:23.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:23.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:23.126 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:23.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:23.149 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:23.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:23.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:22 smithi151 ceph-mon[16639]: pgmap v766: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+remapped, 44 active+clean; 4.7 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 218/204 objects degraded (106.863%); 5.0 MiB/s, 152 objects/s recovering 2024-09-02T10:10:23.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:23.272 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:23.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:23.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:23.375 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:23.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:23.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:23.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:23.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:23.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:23.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:23.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:23.650 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:23.651 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:23.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:23.774 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:23.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:23.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:23.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:23.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:23.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:24.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:24.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:24.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:24.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:24.147 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:24.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:24.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:24.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:24.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:24.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:24.274 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:24.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:24.396 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:24.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:24.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:24.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:24.521 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:24.522 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:24.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:24.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:24.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:24.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:24.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:24.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:24.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:24.771 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:24.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:24.893 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:24.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:24.894 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:24.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:24 smithi187 ceph-mon[20072]: pgmap v767: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 449 op/s; 218/6 objects degraded (3633.333%); 2.6 MiB/s, 193 objects/s recovering 2024-09-02T10:10:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 218/6 objects degraded (3633.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:25.017 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:25.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:25.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:25.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:24 smithi162 ceph-mon[20279]: pgmap v767: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 449 op/s; 218/6 objects degraded (3633.333%); 2.6 MiB/s, 193 objects/s recovering 2024-09-02T10:10:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 218/6 objects degraded (3633.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:25.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:25.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:24 smithi151 ceph-mon[16639]: pgmap v767: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 449 op/s; 218/6 objects degraded (3633.333%); 2.6 MiB/s, 193 objects/s recovering 2024-09-02T10:10:25.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 218/6 objects degraded (3633.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:25.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:25.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:25.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:25.265 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:25.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:25.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:25.388 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:25.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:25.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:25.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:25.512 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:25.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:25.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:25.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:25.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:25.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:25.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:25.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:25.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:25.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:25.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:25.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:26.008 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:26.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:26.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:26.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:26.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:26.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:26.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:26.256 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:26.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:26.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:26.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:26.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:26.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:26.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:26.504 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:26.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:26.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:26.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:26.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:26.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:26.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:26.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:26.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:26.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:26.878 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:26.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:26.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:27.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:27.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:27.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:26 smithi187 ceph-mon[20072]: pgmap v768: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 317 op/s; 218/6 objects degraded (3633.333%); 1.2 MiB/s, 220 objects/s recovering 2024-09-02T10:10:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:26 smithi162 ceph-mon[20279]: pgmap v768: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 317 op/s; 218/6 objects degraded (3633.333%); 1.2 MiB/s, 220 objects/s recovering 2024-09-02T10:10:27.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:27.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:27.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:27.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:26 smithi151 ceph-mon[16639]: pgmap v768: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 317 op/s; 218/6 objects degraded (3633.333%); 1.2 MiB/s, 220 objects/s recovering 2024-09-02T10:10:27.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:27.250 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:27.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:27.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:27.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:27.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:27.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:27.499 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:27.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:27.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:27.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:27.624 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:27.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:27.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:27.747 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:27.748 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:27.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:27.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:27.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:27.978 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:28.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:28.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:28.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:28.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:28.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:28.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:28.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:28.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:28.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:28.375 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:28.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:28.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:28.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:28.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:28.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:28.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:28.625 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:28.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:28.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:28.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:28.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:28.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:28.875 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:28.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:28.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:28.998 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:28.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: pgmap v769: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 165 op/s; 218/6 objects degraded (3633.333%); 1.2 MiB/s, 220 objects/s recovering 2024-09-02T10:10:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=898851933,config=2353167246,health=2611793219,kv=458811069}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2773047387,logm=28704325}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1521712989}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1224091477}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833597020}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3103165203}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1962235060}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3549035449}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259413083}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665159833}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491247785}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3891170195}) 2024-09-02T10:10:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847873942}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200425152}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496888923}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104273}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944778524}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605066914}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2219098352}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299654446}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3068194176}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339897777}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041284375}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=774603211}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=177659145}) 2024-09-02T10:10:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=159302026}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362360395}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2686359116}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871463516}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095272834}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=983733671}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1591985223}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323369826}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1301134251}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731568039}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630776320}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=26,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=40} crc {logm=132334208,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=2479507091,mon_config_key=1950856820}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3645238497}) 2024-09-02T10:10:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2522305430}) 2024-09-02T10:10:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=51,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=31} crc {mon_config_key=1936471371,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=1242040061,osdmap=4232539326}) 2024-09-02T10:10:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=76} crc {osdmap=3975666182}) 2024-09-02T10:10:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: pgmap v769: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 165 op/s; 218/6 objects degraded (3633.333%); 1.2 MiB/s, 220 objects/s recovering 2024-09-02T10:10:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=898851933,config=2353167246,health=2611793219,kv=458811069}) 2024-09-02T10:10:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2773047387,logm=28704325}) 2024-09-02T10:10:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1521712989}) 2024-09-02T10:10:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1224091477}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833597020}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3103165203}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1962235060}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3549035449}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259413083}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665159833}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491247785}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3891170195}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847873942}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200425152}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496888923}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104273}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944778524}) 2024-09-02T10:10:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605066914}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2219098352}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299654446}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3068194176}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339897777}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041284375}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=774603211}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=177659145}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=159302026}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362360395}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2686359116}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871463516}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095272834}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=983733671}) 2024-09-02T10:10:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1591985223}) 2024-09-02T10:10:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323369826}) 2024-09-02T10:10:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1301134251}) 2024-09-02T10:10:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731568039}) 2024-09-02T10:10:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630776320}) 2024-09-02T10:10:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=26,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=40} crc {logm=132334208,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=2479507091,mon_config_key=1950856820}) 2024-09-02T10:10:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3645238497}) 2024-09-02T10:10:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2522305430}) 2024-09-02T10:10:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=51,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=31} crc {mon_config_key=1936471371,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=1242040061,osdmap=4232539326}) 2024-09-02T10:10:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=76} crc {osdmap=3975666182}) 2024-09-02T10:10:29.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:29.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:29.123 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: pgmap v769: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 165 op/s; 218/6 objects degraded (3633.333%); 1.2 MiB/s, 220 objects/s recovering 2024-09-02T10:10:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=57} crc {auth=898851933,config=2353167246,health=2611793219,kv=458811069}) 2024-09-02T10:10:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=2773047387,logm=28704325}) 2024-09-02T10:10:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1521712989}) 2024-09-02T10:10:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1224091477}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=833597020}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3103165203}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1962235060}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3549035449}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259413083}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665159833}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491247785}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3891170195}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847873942}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200425152}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496888923}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104273}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944778524}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605066914}) 2024-09-02T10:10:29.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2219098352}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2299654446}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3068194176}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339897777}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4041284375}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=774603211}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=177659145}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=159302026}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362360395}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2686359116}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3871463516}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3095272834}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=983733671}) 2024-09-02T10:10:29.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1591985223}) 2024-09-02T10:10:29.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2323369826}) 2024-09-02T10:10:29.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1301134251}) 2024-09-02T10:10:29.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2731568039}) 2024-09-02T10:10:29.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3630776320}) 2024-09-02T10:10:29.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=26,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=40} crc {logm=132334208,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=2479507091,mon_config_key=1950856820}) 2024-09-02T10:10:29.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3645238497}) 2024-09-02T10:10:29.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2522305430}) 2024-09-02T10:10:29.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=51,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=31} crc {mon_config_key=1936471371,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=1242040061,osdmap=4232539326}) 2024-09-02T10:10:29.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=76} crc {osdmap=3975666182}) 2024-09-02T10:10:29.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:29.245 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:29.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:29.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:29.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:29.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:29.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:29.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:29.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:29.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:29.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:29.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:29.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:29.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:29.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:29.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:29.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:29.869 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:29.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:29.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:29.995 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:29.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:30.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:30.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:30.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:30.122 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:30.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:29 smithi151 ceph-mon[16639]: pgmap v770: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 41 op/s; 218/6 objects degraded (3633.333%); 0 B/s, 292 objects/s recovering 2024-09-02T10:10:30.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:30.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:30.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:30.248 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:29 smithi187 ceph-mon[20072]: pgmap v770: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 41 op/s; 218/6 objects degraded (3633.333%); 0 B/s, 292 objects/s recovering 2024-09-02T10:10:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:29 smithi162 ceph-mon[20279]: pgmap v770: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 41 op/s; 218/6 objects degraded (3633.333%); 0 B/s, 292 objects/s recovering 2024-09-02T10:10:30.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:30.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:30.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:30.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:30.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:30.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:30.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:30.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:30.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:30.626 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:30.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:30.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:30.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:30.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:30.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:30.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:30.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:30.873 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:30.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:30.955 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":763,"stamp":"2024-09-02T10:10:17.207286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":109249080,"num_objects":1661,"num_object_clones":0,"num_object_copies":4983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":218,"num_objects_misplaced":370,"num_objects_unfound":0,"num_objects_dirty":1661,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217423,"num_write_kb":7010561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11582,"num_bytes_recovered":753336320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163575,"ondisk_log_size":163575,"up":141,"acting":139,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":146,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":3598536,"kb_used_data":1403972,"kb_used_omap":18,"kb_used_meta":2194029,"kb_avail":1121343288,"statfs":{"total":1151940427776,"available":1148255526912,"internally_reserved":0,"allocated":1437667328,"data_stored":1434204512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":2246686011},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,8,4,8,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-444727296,"num_objects":-6786,"num_object_clones":0,"num_object_copies":-20358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1628,"num_objects_unfound":0,"num_objects_dirty":-6786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6786,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1334,"num_bytes_recovered":82051072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002456"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":79,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.655100+0000","last_change":"2024-09-02T10:09:58.904271+0000","last_active":"2024-09-02T10:10:09.655100+0000","last_peered":"2024-09-02T10:10:09.655100+0000","last_clean":"2024-09-02T10:10:09.655100+0000","last_became_active":"2024-09-02T10:09:58.903895+0000","last_became_peered":"2024-09-02T10:09:58.903895+0000","last_unstale":"2024-09-02T10:10:09.655100+0000","last_undegraded":"2024-09-02T10:10:09.655100+0000","last_fullsized":"2024-09-02T10:10:09.655100+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:43.348293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:43.348293+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:45.049119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":167,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.665337+0000","last_change":"2024-09-02T10:09:19.689068+0000","last_active":"2024-09-02T10:10:09.665337+0000","last_peered":"2024-09-02T10:10:09.665337+0000","last_clean":"2024-09-02T10:10:09.665337+0000","last_became_active":"2024-09-02T10:08:25.264923+0000","last_became_peered":"2024-09-02T10:08:25.264923+0000","last_unstale":"2024-09-02T10:10:09.665337+0000","last_undegraded":"2024-09-02T10:10:09.665337+0000","last_fullsized":"2024-09-02T10:10:09.665337+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:19.689004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:19.689004+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:34.223304+0000","scrub_duration":348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":158,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.654948+0000","last_change":"2024-09-02T10:09:24.214465+0000","last_active":"2024-09-02T10:10:09.654948+0000","last_peered":"2024-09-02T10:10:09.654948+0000","last_clean":"2024-09-02T10:10:09.654948+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:10:09.654948+0000","last_undegraded":"2024-09-02T10:10:09.654948+0000","last_fullsized":"2024-09-02T10:10:09.654948+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:24.214401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:24.214401+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:49.081655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":158,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.654949+0000","last_change":"2024-09-02T10:09:22.169230+0000","last_active":"2024-09-02T10:10:09.654949+0000","last_peered":"2024-09-02T10:10:09.654949+0000","last_clean":"2024-09-02T10:10:09.654949+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:10:09.654949+0000","last_undegraded":"2024-09-02T10:10:09.654949+0000","last_fullsized":"2024-09-02T10:10:09.654949+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:22.169181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:22.169181+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:45.550139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":67,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.659043+0000","last_change":"2024-09-02T10:08:27.128433+0000","last_active":"2024-09-02T10:10:09.659043+0000","last_peered":"2024-09-02T10:10:09.659043+0000","last_clean":"2024-09-02T10:10:09.659043+0000","last_became_active":"2024-09-02T10:08:25.264164+0000","last_became_peered":"2024-09-02T10:08:25.264164+0000","last_unstale":"2024-09-02T10:10:09.659043+0000","last_undegraded":"2024-09-02T10:10:09.659043+0000","last_fullsized":"2024-09-02T10:10:09.659043+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:27.128380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:27.128380+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":16,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.727298+0000","last_change":"2024-09-02T10:09:58.922811+0000","last_active":"2024-09-02T10:10:09.727298+0000","last_peered":"2024-09-02T10:10:09.727298+0000","last_clean":"2024-09-02T10:10:09.727298+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:10:09.727298+0000","last_undegraded":"2024-09-02T10:10:09.727298+0000","last_fullsized":"2024-09-02T10:10:09.727298+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:39.188419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:39.188419+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":509,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.664229+0000","last_change":"2024-09-02T10:08:01.621452+0000","last_active":"2024-09-02T10:10:09.664229+0000","last_peered":"2024-09-02T10:10:09.664229+0000","last_clean":"2024-09-02T10:10:09.664229+0000","last_became_active":"2024-09-02T10:06:34.859627+0000","last_became_peered":"2024-09-02T10:06:34.859627+0000","last_unstale":"2024-09-02T10:10:09.664229+0000","last_undegraded":"2024-09-02T10:10:09.664229+0000","last_fullsized":"2024-09-02T10:10:09.664229+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:01.621368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:01.621368+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":165,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.655076+0000","last_change":"2024-09-02T10:09:58.904087+0000","last_active":"2024-09-02T10:10:09.655076+0000","last_peered":"2024-09-02T10:10:09.655076+0000","last_clean":"2024-09-02T10:10:09.655076+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:10:09.655076+0000","last_undegraded":"2024-09-02T10:10:09.655076+0000","last_fullsized":"2024-09-02T10:10:09.655076+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:19.165310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:19.165310+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:33.420958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":158,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.665405+0000","last_change":"2024-09-02T10:09:22.418576+0000","last_active":"2024-09-02T10:10:09.665405+0000","last_peered":"2024-09-02T10:10:09.665405+0000","last_clean":"2024-09-02T10:10:09.665405+0000","last_became_active":"2024-09-02T10:06:34.850895+0000","last_became_peered":"2024-09-02T10:06:34.850895+0000","last_unstale":"2024-09-02T10:10:09.665405+0000","last_undegraded":"2024-09-02T10:10:09.665405+0000","last_fullsized":"2024-09-02T10:10:09.665405+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:22.418387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:22.418387+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:49.629738+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":150,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.659269+0000","last_change":"2024-09-02T10:08:02.003252+0000","last_active":"2024-09-02T10:10:09.659269+0000","last_peered":"2024-09-02T10:10:09.659269+0000","last_clean":"2024-09-02T10:10:09.659269+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:10:09.659269+0000","last_undegraded":"2024-09-02T10:10:09.659269+0000","last_fullsized":"2024-09-02T10:10:09.659269+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:02.003171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:02.003171+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":405,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.727774+0000","last_change":"2024-09-02T10:09:23.308289+0000","last_active":"2024-09-02T10:10:09.727774+0000","last_peered":"2024-09-02T10:10:09.727774+0000","last_clean":"2024-09-02T10:10:09.727774+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:10:09.727774+0000","last_undegraded":"2024-09-02T10:10:09.727774+0000","last_fullsized":"2024-09-02T10:10:09.727774+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:23.308207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:23.308207+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:28.710466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":498,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.665157+0000","last_change":"2024-09-02T10:09:58.906637+0000","last_active":"2024-09-02T10:10:09.665157+0000","last_peered":"2024-09-02T10:10:09.665157+0000","last_clean":"2024-09-02T10:10:09.665157+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:10:09.665157+0000","last_undegraded":"2024-09-02T10:10:09.665157+0000","last_fullsized":"2024-09-02T10:10:09.665157+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:28.449597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:28.449597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:39.262909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":427,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.664261+0000","last_change":"2024-09-02T10:09:59.084780+0000","last_active":"2024-09-02T10:10:09.664261+0000","last_peered":"2024-09-02T10:10:09.664261+0000","last_clean":"2024-09-02T10:10:09.664261+0000","last_became_active":"2024-09-02T10:04:21.422329+0000","last_became_peered":"2024-09-02T10:04:21.422329+0000","last_unstale":"2024-09-02T10:10:09.664261+0000","last_undegraded":"2024-09-02T10:10:09.664261+0000","last_fullsized":"2024-09-02T10:10:09.664261+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:59.084727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:59.084727+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:59.830442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":407,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.666711+0000","last_change":"2024-09-02T10:08:01.316267+0000","last_active":"2024-09-02T10:10:09.666711+0000","last_peered":"2024-09-02T10:10:09.666711+0000","last_clean":"2024-09-02T10:10:09.666711+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:10:09.666711+0000","last_undegraded":"2024-09-02T10:10:09.666711+0000","last_fullsized":"2024-09-02T10:10:09.666711+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:01.316149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:01.316149+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":427,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.664235+0000","last_change":"2024-09-02T10:10:00.055429+0000","last_active":"2024-09-02T10:10:09.664235+0000","last_peered":"2024-09-02T10:10:09.664235+0000","last_clean":"2024-09-02T10:10:09.664235+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:10:09.664235+0000","last_undegraded":"2024-09-02T10:10:09.664235+0000","last_fullsized":"2024-09-02T10:10:09.664235+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:00.055376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:00.055376+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:07.958091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"304'13672","reported_seq":13943,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:17.125077+0000","last_change":"2024-09-02T10:09:22.686192+0000","last_active":"2024-09-02T10:10:17.125077+0000","last_peered":"2024-09-02T10:10:17.125077+0000","last_clean":"2024-09-02T10:10:17.125077+0000","last_became_active":"2024-09-02T10:05:56.583663+0000","last_became_peered":"2024-09-02T10:05:56.583663+0000","last_unstale":"2024-09-02T10:10:17.125077+0000","last_undegraded":"2024-09-02T10:10:17.125077+0000","last_fullsized":"2024-09-02T10:10:17.125077+0000","mapping_epoch":226,"log_start":"248'3600","ondisk_log_start":"248'3600","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'11606","last_scrub_stamp":"2024-09-02T10:09:22.686018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:09:22.686018+0000","objects_scrubbed":2091,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:33.902492+0000","scrub_duration":826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13672,"num_write_kb":437888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":498,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.664359+0000","last_change":"2024-09-02T10:07:59.408627+0000","last_active":"2024-09-02T10:10:09.664359+0000","last_peered":"2024-09-02T10:10:09.664359+0000","last_clean":"2024-09-02T10:10:09.664359+0000","last_became_active":"2024-09-02T10:06:32.873327+0000","last_became_peered":"2024-09-02T10:06:32.873327+0000","last_unstale":"2024-09-02T10:10:09.664359+0000","last_undegraded":"2024-09-02T10:10:09.664359+0000","last_fullsized":"2024-09-02T10:10:09.664359+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:59.408564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:59.408564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.b","version":"304'13453","reported_seq":13695,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:13.767170+0000","last_change":"2024-09-02T10:07:29.219779+0000","last_active":"2024-09-02T10:10:13.767170+0000","last_peered":"2024-09-02T10:10:13.767170+0000","last_clean":"2024-09-02T10:10:13.767170+0000","last_became_active":"2024-09-02T10:05:56.583278+0000","last_became_peered":"2024-09-02T10:05:56.583278+0000","last_unstale":"2024-09-02T10:10:13.767170+0000","last_undegraded":"2024-09-02T10:10:13.767170+0000","last_fullsized":"2024-09-02T10:10:13.767170+0000","mapping_epoch":226,"log_start":"247'3400","ondisk_log_start":"247'3400","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6573","last_scrub_stamp":"2024-09-02T10:07:29.219665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:29.219665+0000","objects_scrubbed":6515,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10682368,"num_objects":163,"num_object_clones":0,"num_object_copies":489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13453,"num_write_kb":435712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":622,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.662083+0000","last_change":"2024-09-02T10:08:08.952478+0000","last_active":"2024-09-02T10:10:09.662083+0000","last_peered":"2024-09-02T10:10:09.662083+0000","last_clean":"2024-09-02T10:10:09.662083+0000","last_became_active":"2024-09-02T10:06:32.871027+0000","last_became_peered":"2024-09-02T10:06:32.871027+0000","last_unstale":"2024-09-02T10:10:09.662083+0000","last_undegraded":"2024-09-02T10:10:09.662083+0000","last_fullsized":"2024-09-02T10:10:09.662083+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:08.952413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:08.952413+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"304'13620","reported_seq":13863,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:16.997165+0000","last_change":"2024-09-02T10:10:00.592115+0000","last_active":"2024-09-02T10:10:16.997165+0000","last_peered":"2024-09-02T10:10:16.997165+0000","last_clean":"2024-09-02T10:10:16.997165+0000","last_became_active":"2024-09-02T10:05:56.583249+0000","last_became_peered":"2024-09-02T10:05:56.583249+0000","last_unstale":"2024-09-02T10:10:16.997165+0000","last_undegraded":"2024-09-02T10:10:16.997165+0000","last_fullsized":"2024-09-02T10:10:16.997165+0000","mapping_epoch":226,"log_start":"248'3600","ondisk_log_start":"248'3600","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'12982","last_scrub_stamp":"2024-09-02T10:10:00.592026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:10:00.592026+0000","objects_scrubbed":659,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:07.818252+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13620,"num_write_kb":436352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,4],"acting":[11,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":256,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.666279+0000","last_change":"2024-09-02T10:08:06.076632+0000","last_active":"2024-09-02T10:10:09.666279+0000","last_peered":"2024-09-02T10:10:09.666279+0000","last_clean":"2024-09-02T10:10:09.666279+0000","last_became_active":"2024-09-02T10:06:32.870387+0000","last_became_peered":"2024-09-02T10:06:32.870387+0000","last_unstale":"2024-09-02T10:10:09.666279+0000","last_undegraded":"2024-09-02T10:10:09.666279+0000","last_fullsized":"2024-09-02T10:10:09.666279+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:06.076562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:06.076562+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.9","version":"304'13565","reported_seq":13839,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:14.251008+0000","last_change":"2024-09-02T10:09:20.667075+0000","last_active":"2024-09-02T10:10:14.251008+0000","last_peered":"2024-09-02T10:10:14.251008+0000","last_clean":"2024-09-02T10:10:14.251008+0000","last_became_active":"2024-09-02T10:05:56.584313+0000","last_became_peered":"2024-09-02T10:05:56.584313+0000","last_unstale":"2024-09-02T10:10:14.251008+0000","last_undegraded":"2024-09-02T10:10:14.251008+0000","last_fullsized":"2024-09-02T10:10:14.251008+0000","mapping_epoch":226,"log_start":"247'3500","ondisk_log_start":"247'3500","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"293'11554","last_scrub_stamp":"2024-09-02T10:09:20.667001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:09:20.667001+0000","objects_scrubbed":2136,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:45.883046+0000","scrub_duration":494,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13565,"num_write_kb":437824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":65,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.663844+0000","last_change":"2024-09-02T10:08:25.382153+0000","last_active":"2024-09-02T10:10:09.663844+0000","last_peered":"2024-09-02T10:10:09.663844+0000","last_clean":"2024-09-02T10:10:09.663844+0000","last_became_active":"2024-09-02T10:08:25.264766+0000","last_became_peered":"2024-09-02T10:08:25.264766+0000","last_unstale":"2024-09-02T10:10:09.663844+0000","last_undegraded":"2024-09-02T10:10:09.663844+0000","last_fullsized":"2024-09-02T10:10:09.663844+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:25.382103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:25.382103+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"304'13492","reported_seq":13734,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:12.487171+0000","last_change":"2024-09-02T10:07:34.096922+0000","last_active":"2024-09-02T10:10:12.487171+0000","last_peered":"2024-09-02T10:10:12.487171+0000","last_clean":"2024-09-02T10:10:12.487171+0000","last_became_active":"2024-09-02T10:05:56.578973+0000","last_became_peered":"2024-09-02T10:05:56.578973+0000","last_unstale":"2024-09-02T10:10:12.487171+0000","last_undegraded":"2024-09-02T10:10:12.487171+0000","last_fullsized":"2024-09-02T10:10:12.487171+0000","mapping_epoch":226,"log_start":"247'3400","ondisk_log_start":"247'3400","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6853","last_scrub_stamp":"2024-09-02T10:07:34.096729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:34.096729+0000","objects_scrubbed":6787,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2896,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13500416,"num_objects":206,"num_object_clones":0,"num_object_copies":618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13492,"num_write_kb":438336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":554,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.659087+0000","last_change":"2024-09-02T10:07:07.536258+0000","last_active":"2024-09-02T10:10:09.659087+0000","last_peered":"2024-09-02T10:10:09.659087+0000","last_clean":"2024-09-02T10:10:09.659087+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:10:09.659087+0000","last_undegraded":"2024-09-02T10:10:09.659087+0000","last_fullsized":"2024-09-02T10:10:09.659087+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:07.536194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:07.536194+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.7","version":"304'13454","reported_seq":21193,"reported_epoch":304,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:10:13.794259+0000","last_change":"2024-09-02T10:08:26.621106+0000","last_active":"2024-09-02T10:10:13.794259+0000","last_peered":"2024-09-02T10:10:13.794259+0000","last_clean":"2024-09-02T10:08:24.056426+0000","last_became_active":"2024-09-02T10:08:26.550955+0000","last_became_peered":"2024-09-02T10:08:26.550955+0000","last_unstale":"2024-09-02T10:10:13.794259+0000","last_undegraded":"2024-09-02T10:08:26.292304+0000","last_fullsized":"2024-09-02T10:08:26.259077+0000","mapping_epoch":283,"log_start":"247'3400","ondisk_log_start":"247'3400","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6680","last_scrub_stamp":"2024-09-02T10:07:30.887770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.887770+0000","objects_scrubbed":6647,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1437,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9437184,"num_objects":144,"num_object_clones":0,"num_object_copies":432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":218,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13454,"num_write_kb":435136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3748,"num_bytes_recovered":244187136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7],"avail_no_missing":["3","7","11"],"object_location_counts":[{"shards":"3,7","objects":-74}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":16,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.727299+0000","last_change":"2024-09-02T10:09:58.926585+0000","last_active":"2024-09-02T10:10:09.727299+0000","last_peered":"2024-09-02T10:10:09.727299+0000","last_clean":"2024-09-02T10:10:09.727299+0000","last_became_active":"2024-09-02T10:09:58.926356+0000","last_became_peered":"2024-09-02T10:09:58.926356+0000","last_unstale":"2024-09-02T10:10:09.727299+0000","last_undegraded":"2024-09-02T10:10:09.727299+0000","last_fullsized":"2024-09-02T10:10:09.727299+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:38.456958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:38.456958+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:39.328825+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.4","version":"304'13871","reported_seq":14146,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:13.192800+0000","last_change":"2024-09-02T10:09:27.097581+0000","last_active":"2024-09-02T10:10:13.192800+0000","last_peered":"2024-09-02T10:10:13.192800+0000","last_clean":"2024-09-02T10:10:13.192800+0000","last_became_active":"2024-09-02T10:05:56.580910+0000","last_became_peered":"2024-09-02T10:05:56.580910+0000","last_unstale":"2024-09-02T10:10:13.192800+0000","last_undegraded":"2024-09-02T10:10:13.192800+0000","last_fullsized":"2024-09-02T10:10:13.192800+0000","mapping_epoch":226,"log_start":"248'3800","ondisk_log_start":"248'3800","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"296'12071","last_scrub_stamp":"2024-09-02T10:09:27.097309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:09:27.097309+0000","objects_scrubbed":1998,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:46.073116+0000","scrub_duration":629,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11993088,"num_objects":183,"num_object_clones":0,"num_object_copies":549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13871,"num_write_kb":449728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":24,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.727389+0000","last_change":"2024-09-02T10:09:59.384188+0000","last_active":"2024-09-02T10:10:09.727389+0000","last_peered":"2024-09-02T10:10:09.727389+0000","last_clean":"2024-09-02T10:10:09.727389+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:10:09.727389+0000","last_undegraded":"2024-09-02T10:10:09.727389+0000","last_fullsized":"2024-09-02T10:10:09.727389+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:59.384140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:59.384140+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:11.700113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.5","version":"304'13610","reported_seq":13852,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:16.954246+0000","last_change":"2024-09-02T10:07:30.154080+0000","last_active":"2024-09-02T10:10:16.954246+0000","last_peered":"2024-09-02T10:10:16.954246+0000","last_clean":"2024-09-02T10:10:16.954246+0000","last_became_active":"2024-09-02T10:05:56.690229+0000","last_became_peered":"2024-09-02T10:05:56.690229+0000","last_unstale":"2024-09-02T10:10:16.954246+0000","last_undegraded":"2024-09-02T10:10:16.954246+0000","last_fullsized":"2024-09-02T10:10:16.954246+0000","mapping_epoch":226,"log_start":"248'3600","ondisk_log_start":"248'3600","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6652","last_scrub_stamp":"2024-09-02T10:07:30.153975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.153975+0000","objects_scrubbed":6605,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2047,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13610,"num_write_kb":435904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":550,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.655277+0000","last_change":"2024-09-02T10:09:58.913028+0000","last_active":"2024-09-02T10:10:09.655277+0000","last_peered":"2024-09-02T10:10:09.655277+0000","last_clean":"2024-09-02T10:10:09.655277+0000","last_became_active":"2024-09-02T10:09:58.912625+0000","last_became_peered":"2024-09-02T10:09:58.912625+0000","last_unstale":"2024-09-02T10:10:09.655277+0000","last_undegraded":"2024-09-02T10:10:09.655277+0000","last_fullsized":"2024-09-02T10:10:09.655277+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:26.917637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:26.917637+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:49.287568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.3","version":"304'13736","reported_seq":14009,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:14.305747+0000","last_change":"2024-09-02T10:09:25.958061+0000","last_active":"2024-09-02T10:10:14.305747+0000","last_peered":"2024-09-02T10:10:14.305747+0000","last_clean":"2024-09-02T10:10:14.305747+0000","last_became_active":"2024-09-02T10:05:56.581969+0000","last_became_peered":"2024-09-02T10:05:56.581969+0000","last_unstale":"2024-09-02T10:10:14.305747+0000","last_undegraded":"2024-09-02T10:10:14.305747+0000","last_fullsized":"2024-09-02T10:10:14.305747+0000","mapping_epoch":226,"log_start":"248'3700","ondisk_log_start":"248'3700","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'11881","last_scrub_stamp":"2024-09-02T10:09:25.957816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:09:25.957816+0000","objects_scrubbed":1994,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:26.262454+0000","scrub_duration":400,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8781824,"num_objects":134,"num_object_clones":0,"num_object_copies":402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13736,"num_write_kb":443840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":16,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.727361+0000","last_change":"2024-09-02T10:09:58.922637+0000","last_active":"2024-09-02T10:10:09.727361+0000","last_peered":"2024-09-02T10:10:09.727361+0000","last_clean":"2024-09-02T10:10:09.727361+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:10:09.727361+0000","last_undegraded":"2024-09-02T10:10:09.727361+0000","last_fullsized":"2024-09-02T10:10:09.727361+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:29.964607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:29.964607+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"304'13432","reported_seq":13675,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:17.036732+0000","last_change":"2024-09-02T10:07:35.803712+0000","last_active":"2024-09-02T10:10:17.036732+0000","last_peered":"2024-09-02T10:10:17.036732+0000","last_clean":"2024-09-02T10:10:17.036732+0000","last_became_active":"2024-09-02T10:05:56.876397+0000","last_became_peered":"2024-09-02T10:05:56.876397+0000","last_unstale":"2024-09-02T10:10:17.036732+0000","last_undegraded":"2024-09-02T10:10:17.036732+0000","last_fullsized":"2024-09-02T10:10:17.036732+0000","mapping_epoch":226,"log_start":"247'3400","ondisk_log_start":"247'3400","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'6802","last_scrub_stamp":"2024-09-02T10:07:35.803575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:35.803575+0000","objects_scrubbed":6687,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":655360,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13432,"num_write_kb":430144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,6],"acting":[11,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":263,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.659159+0000","last_change":"2024-09-02T10:08:25.263472+0000","last_active":"2024-09-02T10:10:09.659159+0000","last_peered":"2024-09-02T10:10:09.659159+0000","last_clean":"2024-09-02T10:10:09.659159+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:10:09.659159+0000","last_undegraded":"2024-09-02T10:10:09.659159+0000","last_fullsized":"2024-09-02T10:10:09.659159+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:33.808730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:33.808730+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"304'13407","reported_seq":21680,"reported_epoch":304,"state":"active+recovering+remapped","last_fresh":"2024-09-02T10:10:12.504904+0000","last_change":"2024-09-02T10:10:00.476812+0000","last_active":"2024-09-02T10:10:12.504904+0000","last_peered":"2024-09-02T10:10:12.504904+0000","last_clean":"2024-09-02T10:08:24.179939+0000","last_became_active":"2024-09-02T10:10:00.385882+0000","last_became_peered":"2024-09-02T10:10:00.385882+0000","last_unstale":"2024-09-02T10:10:12.504904+0000","last_undegraded":"2024-09-02T10:10:12.504904+0000","last_fullsized":"2024-09-02T10:10:12.504904+0000","mapping_epoch":302,"log_start":"247'3400","ondisk_log_start":"247'3400","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5000","last_scrub_stamp":"2024-09-02T10:07:05.712114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:05.712114+0000","objects_scrubbed":4957,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13041664,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13407,"num_write_kb":435392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4019,"num_bytes_recovered":259129344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,11],"acting":[0,11,5],"avail_no_missing":["0","4","11"],"object_location_counts":[{"shards":"0,5,11","objects":199}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":486,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.665218+0000","last_change":"2024-09-02T10:09:20.395796+0000","last_active":"2024-09-02T10:10:09.665218+0000","last_peered":"2024-09-02T10:10:09.665218+0000","last_clean":"2024-09-02T10:10:09.665218+0000","last_became_active":"2024-09-02T10:08:25.264099+0000","last_became_peered":"2024-09-02T10:08:25.264099+0000","last_unstale":"2024-09-02T10:10:09.665218+0000","last_undegraded":"2024-09-02T10:10:09.665218+0000","last_fullsized":"2024-09-02T10:10:09.665218+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:20.395685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:20.395685+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:27.201725+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"304'13734","reported_seq":13978,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:13.756078+0000","last_change":"2024-09-02T10:07:30.485290+0000","last_active":"2024-09-02T10:10:13.756078+0000","last_peered":"2024-09-02T10:10:13.756078+0000","last_clean":"2024-09-02T10:10:13.756078+0000","last_became_active":"2024-09-02T10:05:56.876573+0000","last_became_peered":"2024-09-02T10:05:56.876573+0000","last_unstale":"2024-09-02T10:10:13.756078+0000","last_undegraded":"2024-09-02T10:10:13.756078+0000","last_fullsized":"2024-09-02T10:10:13.756078+0000","mapping_epoch":226,"log_start":"248'3700","ondisk_log_start":"248'3700","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6788","last_scrub_stamp":"2024-09-02T10:07:30.485127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:30.485127+0000","objects_scrubbed":6730,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2030,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10354688,"num_objects":158,"num_object_clones":0,"num_object_copies":474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13734,"num_write_kb":444544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":267,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.658988+0000","last_change":"2024-09-02T10:07:28.824347+0000","last_active":"2024-09-02T10:10:09.658988+0000","last_peered":"2024-09-02T10:10:09.658988+0000","last_clean":"2024-09-02T10:10:09.658988+0000","last_became_active":"2024-09-02T10:04:47.941933+0000","last_became_peered":"2024-09-02T10:04:47.941933+0000","last_unstale":"2024-09-02T10:10:09.658988+0000","last_undegraded":"2024-09-02T10:10:09.658988+0000","last_fullsized":"2024-09-02T10:10:09.658988+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:07:28.824291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:07:28.824291+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"304'13843","reported_seq":14070,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:15.948875+0000","last_change":"2024-09-02T10:07:04.549316+0000","last_active":"2024-09-02T10:10:15.948875+0000","last_peered":"2024-09-02T10:10:15.948875+0000","last_clean":"2024-09-02T10:10:15.948875+0000","last_became_active":"2024-09-02T10:05:56.876540+0000","last_became_peered":"2024-09-02T10:05:56.876540+0000","last_unstale":"2024-09-02T10:10:15.948875+0000","last_undegraded":"2024-09-02T10:10:15.948875+0000","last_fullsized":"2024-09-02T10:10:15.948875+0000","mapping_epoch":226,"log_start":"248'3800","ondisk_log_start":"248'3800","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'4967","last_scrub_stamp":"2024-09-02T10:07:04.549227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:04.549227+0000","objects_scrubbed":4900,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1666,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3670040,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13843,"num_write_kb":444737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,9],"acting":[7,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":507,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.655270+0000","last_change":"2024-09-02T10:09:18.768450+0000","last_active":"2024-09-02T10:10:09.655270+0000","last_peered":"2024-09-02T10:10:09.655270+0000","last_clean":"2024-09-02T10:10:09.655270+0000","last_became_active":"2024-09-02T10:02:42.784108+0000","last_became_peered":"2024-09-02T10:02:42.784108+0000","last_unstale":"2024-09-02T10:10:09.655270+0000","last_undegraded":"2024-09-02T10:10:09.655270+0000","last_fullsized":"2024-09-02T10:10:09.655270+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:18.768407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:18.768407+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:10:34.372512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"304'13612","reported_seq":13854,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:16.938754+0000","last_change":"2024-09-02T10:07:31.723546+0000","last_active":"2024-09-02T10:10:16.938754+0000","last_peered":"2024-09-02T10:10:16.938754+0000","last_clean":"2024-09-02T10:10:16.938754+0000","last_became_active":"2024-09-02T10:05:56.585883+0000","last_became_peered":"2024-09-02T10:05:56.585883+0000","last_unstale":"2024-09-02T10:10:16.938754+0000","last_undegraded":"2024-09-02T10:10:16.938754+0000","last_fullsized":"2024-09-02T10:10:16.938754+0000","mapping_epoch":226,"log_start":"248'3600","ondisk_log_start":"248'3600","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'6718","last_scrub_stamp":"2024-09-02T10:07:31.723385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:31.723385+0000","objects_scrubbed":6667,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1623,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13612,"num_write_kb":436096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":64,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.659332+0000","last_change":"2024-09-02T10:08:26.088128+0000","last_active":"2024-09-02T10:10:09.659332+0000","last_peered":"2024-09-02T10:10:09.659332+0000","last_clean":"2024-09-02T10:10:09.659332+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:10:09.659332+0000","last_undegraded":"2024-09-02T10:10:09.659332+0000","last_fullsized":"2024-09-02T10:10:09.659332+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:26.088081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:26.088081+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"304'13538","reported_seq":13769,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:15.960149+0000","last_change":"2024-09-02T10:08:11.942333+0000","last_active":"2024-09-02T10:10:15.960149+0000","last_peered":"2024-09-02T10:10:15.960149+0000","last_clean":"2024-09-02T10:10:15.960149+0000","last_became_active":"2024-09-02T10:05:56.584450+0000","last_became_peered":"2024-09-02T10:05:56.584450+0000","last_unstale":"2024-09-02T10:10:15.960149+0000","last_undegraded":"2024-09-02T10:10:15.960149+0000","last_fullsized":"2024-09-02T10:10:15.960149+0000","mapping_epoch":226,"log_start":"248'3500","ondisk_log_start":"248'3500","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5143","last_scrub_stamp":"2024-09-02T10:07:06.845283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:06.845283+0000","objects_scrubbed":99,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13538,"num_write_kb":434560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":281,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.659369+0000","last_change":"2024-09-02T10:08:04.938089+0000","last_active":"2024-09-02T10:10:09.659369+0000","last_peered":"2024-09-02T10:10:09.659369+0000","last_clean":"2024-09-02T10:10:09.659369+0000","last_became_active":"2024-09-02T10:06:32.876151+0000","last_became_peered":"2024-09-02T10:06:32.876151+0000","last_unstale":"2024-09-02T10:10:09.659369+0000","last_undegraded":"2024-09-02T10:10:09.659369+0000","last_fullsized":"2024-09-02T10:10:09.659369+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:04.938027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:04.938027+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"304'13384","reported_seq":25072,"reported_epoch":304,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:10:12.512117+0000","last_change":"2024-09-02T10:08:26.668214+0000","last_active":"2024-09-02T10:10:12.512117+0000","last_peered":"2024-09-02T10:10:12.512117+0000","last_clean":"2024-09-02T10:08:24.105685+0000","last_became_active":"2024-09-02T10:08:26.603797+0000","last_became_peered":"2024-09-02T10:08:26.603797+0000","last_unstale":"2024-09-02T10:10:12.512117+0000","last_undegraded":"2024-09-02T10:10:12.512117+0000","last_fullsized":"2024-09-02T10:08:26.259622+0000","mapping_epoch":283,"log_start":"246'3300","ondisk_log_start":"246'3300","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'5095","last_scrub_stamp":"2024-09-02T10:07:06.655797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:05:55.567155+0000","last_clean_scrub_stamp":"2024-09-02T10:07:06.655797+0000","objects_scrubbed":949,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12451840,"num_objects":190,"num_object_clones":0,"num_object_copies":570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":370,"num_objects_unfound":0,"num_objects_dirty":190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13384,"num_write_kb":434368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3815,"num_bytes_recovered":250019840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,3],"acting":[0,3],"avail_no_missing":["0","4","11"],"object_location_counts":[{"shards":"0,3","objects":5},{"shards":"0,4,11","objects":185}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":294,"reported_epoch":304,"state":"active+clean","last_fresh":"2024-09-02T10:10:09.666305+0000","last_change":"2024-09-02T10:08:02.302316+0000","last_active":"2024-09-02T10:10:09.666305+0000","last_peered":"2024-09-02T10:10:09.666305+0000","last_clean":"2024-09-02T10:10:09.666305+0000","last_became_active":"2024-09-02T10:06:32.875557+0000","last_became_peered":"2024-09-02T10:06:32.875557+0000","last_unstale":"2024-09-02T10:10:09.666305+0000","last_undegraded":"2024-09-02T10:10:09.666305+0000","last_fullsized":"2024-09-02T10:10:09.666305+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:02.302231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:02.302231+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":108658712,"num_objects":1659,"num_object_clones":0,"num_object_copies":4977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":218,"num_objects_misplaced":370,"num_objects_unfound":0,"num_objects_dirty":1659,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":217423,"num_write_kb":7010561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11582,"num_bytes_recovered":753336320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1354772480,"data_stored":1354760264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160723,"ondisk_log_size":160723,"up":48,"acting":46,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383802,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":804728,"kb_used_data":525032,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92940424,"statfs":{"total":95995035648,"available":95170994176,"internally_reserved":0,"allocated":537632768,"data_stored":537346106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":286390724},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612735,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":49976,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93695176,"statfs":{"total":95995035648,"available":95943860224,"internally_reserved":0,"allocated":761856,"data_stored":475194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":50395584},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841667,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":278396,"kb_used_data":40364,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93466756,"statfs":{"total":95995035648,"available":95709958144,"internally_reserved":0,"allocated":41332736,"data_stored":41042002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":243726784},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070601,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":142584,"kb_used_data":21160,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93602568,"statfs":{"total":95995035648,"available":95849029632,"internally_reserved":0,"allocated":21667840,"data_stored":21381178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299532,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":177284,"kb_used_data":19764,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93567868,"statfs":{"total":95995035648,"available":95813496832,"internally_reserved":0,"allocated":20238336,"data_stored":19944050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":161282506},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528465,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":152440,"kb_used_data":8808,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93592712,"statfs":{"total":95995035648,"available":95838937088,"internally_reserved":0,"allocated":9019392,"data_stored":8732730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":147061191},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724694,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":186820,"kb_used_data":7028,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93558332,"statfs":{"total":95995035648,"available":95803731968,"internally_reserved":0,"allocated":7196672,"data_stored":6902386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953627,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":755704,"kb_used_data":571816,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92989448,"statfs":{"total":95995035648,"available":95221194752,"internally_reserved":0,"allocated":585539584,"data_stored":585252922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149857,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":361656,"kb_used_data":63656,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":93383496,"statfs":{"total":95995035648,"available":95624699904,"internally_reserved":0,"allocated":65183744,"data_stored":64897082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":304871880},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411492,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":170692,"kb_used_data":13168,"kb_used_omap":1,"kb_used_meta":157502,"kb_avail":93574460,"statfs":{"total":95995035648,"available":95820247040,"internally_reserved":0,"allocated":13484032,"data_stored":13193818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":161282501},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575018,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":232056,"kb_used_data":48168,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":93513096,"statfs":{"total":95995035648,"available":95757410304,"internally_reserved":0,"allocated":49324032,"data_stored":49037370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738543,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":286200,"kb_used_data":84264,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93458952,"statfs":{"total":95995035648,"available":95701966848,"internally_reserved":0,"allocated":86286336,"data_stored":85999674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77856768,"data_stored":77856768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":47644672,"data_stored":47644672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12255232,"data_stored":12255232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50790400,"data_stored":50790400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":570359808,"data_stored":570359808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4526080,"data_stored":4522008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2359296,"data_stored":2359296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":13176832,"data_stored":13172760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":15859712,"data_stored":15859712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":35328000,"data_stored":35323928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":524615680,"data_stored":524615680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:10:30.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 41 2024-09-02T10:10:30.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:30.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:30.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:31.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:31.120 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:31.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:31.121 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:31.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:30 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 218/6 objects degraded (3633.333%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:31.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:31.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:31.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:30 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 218/6 objects degraded (3633.333%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:31.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:30 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 218/6 objects degraded (3633.333%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:31.346 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:31.367 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:31.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:31.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:31.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:31.492 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:31.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:31.615 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:31.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:31.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:31.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:31.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:31.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:31.864 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:31.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:31.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:31.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:31.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:32.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:32.112 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:32.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:32.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:31 smithi151 ceph-mon[16639]: pgmap v771: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 15 op/s; 218/6 objects degraded (3633.333%); 0 B/s, 278 objects/s recovering 2024-09-02T10:10:32.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:32.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:32.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:31 smithi187 ceph-mon[20072]: pgmap v771: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 15 op/s; 218/6 objects degraded (3633.333%); 0 B/s, 278 objects/s recovering 2024-09-02T10:10:32.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:32.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:31 smithi162 ceph-mon[20279]: pgmap v771: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 1 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 15 op/s; 218/6 objects degraded (3633.333%); 0 B/s, 278 objects/s recovering 2024-09-02T10:10:32.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:32.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:32.462 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:32.485 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:32.485 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:32.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:32.609 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:32.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:32.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:32.733 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:32.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:32.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:32.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:32.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:32.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:32.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:32.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:32.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:33.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:33.105 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:33.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:33.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:33.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:33.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:33.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:33.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:33.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:33.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:33.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:33.485 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:33.509 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:33.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:33.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:33.633 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:33.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:33.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:33.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:33.756 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:33.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:33.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:33.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:33.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:33.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:33.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:10:33.983 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:34.005 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:34.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:34.006 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:34.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:34.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:34.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:34.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:34.250 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:34.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:34.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:34.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:34.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:34.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:34.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:34.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:34.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:34.499 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:34.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:34 smithi162 ceph-mon[20279]: pgmap v772: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 5 op/s; 218/6 objects degraded (3633.333%); 0 B/s, 293 objects/s recovering 2024-09-02T10:10:34.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:34.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:34.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:34.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:34 smithi151 ceph-mon[16639]: pgmap v772: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 5 op/s; 218/6 objects degraded (3633.333%); 0 B/s, 293 objects/s recovering 2024-09-02T10:10:34.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:34.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:34.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:34 smithi187 ceph-mon[20072]: pgmap v772: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 5 op/s; 218/6 objects degraded (3633.333%); 0 B/s, 293 objects/s recovering 2024-09-02T10:10:34.846 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:34.866 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:34.867 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:34.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:34.990 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:34.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:35.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:35.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:35.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:35.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:35.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:35.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:35.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:35.362 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:35.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:35.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:35.485 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:35.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:35.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:35.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:35.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:35.609 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:35.609 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:35.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:35.731 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:35.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:35.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:35.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:35.854 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:35.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:35.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:35.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:35.979 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:35.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:35.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:36.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:36.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:36.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:36.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:36.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:36.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:36.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:36.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:36.349 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:36.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:36.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:36 smithi151 ceph-mon[16639]: pgmap v773: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 269 objects/s recovering 2024-09-02T10:10:36.451 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:36.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:36.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:36.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:36.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:36.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:36 smithi162 ceph-mon[20279]: pgmap v773: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 269 objects/s recovering 2024-09-02T10:10:36.601 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:36.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:36.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:36.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:36.725 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:36.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:36 smithi187 ceph-mon[20072]: pgmap v773: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 269 objects/s recovering 2024-09-02T10:10:36.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:36.848 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:36.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:36.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:36.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:36.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:36.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:37.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:37.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:37.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:37.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:37.222 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:37.223 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:37.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:37.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:37.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:37.346 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:37.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:37.468 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:37.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:37.469 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:37.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:37.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:37 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:10:37.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:37 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:10:37.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:37 smithi162 ceph-mon[20279]: osdmap e306: 12 total, 12 up, 12 in 2024-09-02T10:10:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:37.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:37.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:37.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:37 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:10:37.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:37 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:10:37.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:37 smithi151 ceph-mon[16639]: osdmap e306: 12 total, 12 up, 12 in 2024-09-02T10:10:37.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:37.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:37.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:37.719 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:37.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:37 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:10:37.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:37 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:10:37.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:37 smithi187 ceph-mon[20072]: osdmap e306: 12 total, 12 up, 12 in 2024-09-02T10:10:37.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:37.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:37.842 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:37.843 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:37.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:37.966 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:37.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:38.068 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:38.089 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:38.090 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:38.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:38.213 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:38.214 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:38.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:38.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:38.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:38.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:38.464 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:38.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:38.527 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:38 smithi151 ceph-mon[16639]: pgmap v775: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 270 objects/s recovering 2024-09-02T10:10:38.565 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:38.587 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:38.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:38.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:38 smithi162 ceph-mon[20279]: pgmap v775: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 270 objects/s recovering 2024-09-02T10:10:38.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:38.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:38.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:38.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:38 smithi187 ceph-mon[20072]: pgmap v775: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 270 objects/s recovering 2024-09-02T10:10:38.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:38.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:38.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:38.833 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:38.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:38.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:38.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:38.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:39.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:39.081 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:39.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:39.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:39.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:39.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:39.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:39.328 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:39.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:39.377 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:10:39.377 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:10:39.429 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:39.450 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:39.451 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:39.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:39.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:39.574 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:39.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:10:39.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:10:39.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:39 smithi162 ceph-mon[20279]: osdmap e307: 12 total, 12 up, 12 in 2024-09-02T10:10:39.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 218/6 objects degraded (3633.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:39.675 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:10:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:10:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:39 smithi151 ceph-mon[16639]: osdmap e307: 12 total, 12 up, 12 in 2024-09-02T10:10:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 218/6 objects degraded (3633.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:39.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:39.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:39.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:10:39.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:10:39.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:39 smithi187 ceph-mon[20072]: osdmap e307: 12 total, 12 up, 12 in 2024-09-02T10:10:39.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 218/6 objects degraded (3633.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:39.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:39.822 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:39.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:39.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:39.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:39.945 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:39.946 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:40.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:40.068 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:40.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:40.069 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:40.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:40.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:40.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:40.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:40.319 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:40.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:40.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:40.445 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:40.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:40.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:40.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:10:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:40.569 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:40.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:40.586 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:40 smithi151 ceph-mon[16639]: pgmap v777: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 385 objects/s recovering 2024-09-02T10:10:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:40 smithi162 ceph-mon[20279]: pgmap v777: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 385 objects/s recovering 2024-09-02T10:10:40.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:40.692 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:40.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:40.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:40 smithi187 ceph-mon[20072]: pgmap v777: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 385 objects/s recovering 2024-09-02T10:10:40.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:40.817 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:40.817 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:40.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:40.942 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:40.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:40.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:41.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:41.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:41.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:41.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:41.190 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:41.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:41.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:41.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:41.318 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:41.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:41.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:41.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:41.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:41.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:41.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:41.571 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:41.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:41.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:41.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:41 smithi162 ceph-mon[20279]: osdmap e308: 12 total, 12 up, 12 in 2024-09-02T10:10:41.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:41.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:41 smithi151 ceph-mon[16639]: osdmap e308: 12 total, 12 up, 12 in 2024-09-02T10:10:41.694 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:41.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:41.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:41 smithi187 ceph-mon[20072]: osdmap e308: 12 total, 12 up, 12 in 2024-09-02T10:10:41.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:41.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:41.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:41.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:41.947 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:41.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:42.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:42.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:42.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:42.174 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:42.196 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:42.197 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:42.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:42.325 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:42.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:42.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:42.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:42.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:42.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:42.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:42.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:42 smithi162 ceph-mon[20279]: pgmap v779: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 318 objects/s recovering 2024-09-02T10:10:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:42 smithi162 ceph-mon[20279]: osdmap e309: 12 total, 12 up, 12 in 2024-09-02T10:10:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:42 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:10:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:42 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:10:42.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:42.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:42 smithi151 ceph-mon[16639]: pgmap v779: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 318 objects/s recovering 2024-09-02T10:10:42.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:42 smithi151 ceph-mon[16639]: osdmap e309: 12 total, 12 up, 12 in 2024-09-02T10:10:42.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:42 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:10:42.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:42 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:10:42.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:42.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:42.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:42 smithi187 ceph-mon[20072]: pgmap v779: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+remapped, 44 active+clean; 577 KiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 218/6 objects degraded (3633.333%); 0 B/s, 318 objects/s recovering 2024-09-02T10:10:42.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:42 smithi187 ceph-mon[20072]: osdmap e309: 12 total, 12 up, 12 in 2024-09-02T10:10:42.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:42 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:10:42.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:42 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:10:42.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:42.827 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:42.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:42.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:42.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:42.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:42.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:43.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:43.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:43.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:43.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:43.199 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:43.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:43.200 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:43.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:43.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:43.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:43.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:43.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:43.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:43.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:43.576 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:43.577 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:43 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:10:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:43 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:10:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:43 smithi162 ceph-mon[20279]: osdmap e310: 12 total, 12 up, 12 in 2024-09-02T10:10:43.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:43.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:43 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:10:43.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:43 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:10:43.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:43 smithi151 ceph-mon[16639]: osdmap e310: 12 total, 12 up, 12 in 2024-09-02T10:10:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:43.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:43.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:43.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:43 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:10:43.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:43 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:10:43.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:43 smithi187 ceph-mon[20072]: osdmap e310: 12 total, 12 up, 12 in 2024-09-02T10:10:43.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:43.830 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:43.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:43.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:43.957 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:43.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:44.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:44.082 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:44.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:44.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:44.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:44.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:44.282 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:44.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:44.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:10:44.328 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:44.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:44.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:44.330 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:44.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:44 smithi151 ceph-mon[16639]: pgmap v782: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 222/6 objects degraded (3700.000%); 0 B/s, 265 objects/s recovering 2024-09-02T10:10:44.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:44 smithi151 ceph-mon[16639]: osdmap e311: 12 total, 12 up, 12 in 2024-09-02T10:10:44.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:44 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 222/6 objects degraded (3700.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:44.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:44.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:44.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:44.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:44.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:44.582 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:44.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:44.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:44 smithi162 ceph-mon[20279]: pgmap v782: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 222/6 objects degraded (3700.000%); 0 B/s, 265 objects/s recovering 2024-09-02T10:10:44.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:44 smithi162 ceph-mon[20279]: osdmap e311: 12 total, 12 up, 12 in 2024-09-02T10:10:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:44 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 222/6 objects degraded (3700.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:44.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:44.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:44.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:44.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:44 smithi187 ceph-mon[20072]: pgmap v782: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 222/6 objects degraded (3700.000%); 0 B/s, 265 objects/s recovering 2024-09-02T10:10:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:44 smithi187 ceph-mon[20072]: osdmap e311: 12 total, 12 up, 12 in 2024-09-02T10:10:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:44 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 222/6 objects degraded (3700.000%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:10:44.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:44.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:44.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:44.830 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:44.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:44.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:44.953 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:44.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:45.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:45.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:45.076 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:45.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:45.204 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:45.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:45.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:45.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:45.331 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:45.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:45.356 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pgp_num 41 > pg_num 31 2024-09-02T10:10:45.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:45.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:45.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:45.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:45.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:45.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:45.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:45.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:45.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:45.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:45.830 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:45.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:45.933 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:45.959 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:45.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:45.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:46.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:46.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:46.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:10:46.141 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-02T10:10:46.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-02T10:10:46.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:46.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:46.206 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:46.207 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:46.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:46.313 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:10:46.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:46.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:46.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:46 smithi151 ceph-mon[16639]: pgmap v784: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4/6 objects degraded (66.667%); 0 B/s, 228 objects/s recovering 2024-09-02T10:10:46.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1938264802' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-02T10:10:46.433 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2274936136' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:10:46.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:46.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:46.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:46.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:46.585 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:46.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:46.586 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:46.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:46 smithi162 ceph-mon[20279]: pgmap v784: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4/6 objects degraded (66.667%); 0 B/s, 228 objects/s recovering 2024-09-02T10:10:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1938264802' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-02T10:10:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2274936136' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:10:46.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:46.714 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:46.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:46.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:46 smithi187 ceph-mon[20072]: pgmap v784: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4/6 objects degraded (66.667%); 0 B/s, 228 objects/s recovering 2024-09-02T10:10:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1938264802' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "41"} : dispatch 2024-09-02T10:10:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2274936136' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:10:46.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:46.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:46.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:46.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:46.906 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:10:46.907 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:10:46.931 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:10:46.932 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:10:46.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:46.955 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:46.966 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:46.966 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:46.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:46.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:46.974 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:10:47.003 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:10:47.004 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:10:47.027 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:10:47.027 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:10:47.052 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:10:47.053 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:10:47.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:47.075 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:10:47.075 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:10:47.093 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:47.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:47.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:47.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:10:47.099 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:10:47.132 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:10:47.133 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:10:47.154 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:10:47.154 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:10:47.177 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:10:47.177 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:10:47.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:47.200 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:10:47.201 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:10:47.216 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:47.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:47.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:47.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:47.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:47.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:47.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:47.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:47.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:47.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:47.576 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:47 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:10:47.577 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2274936136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:10:47.577 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:47 smithi151 ceph-mon[16639]: osdmap e312: 12 total, 12 up, 12 in 2024-09-02T10:10:47.577 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3846414505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:47.591 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:47.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:47.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:47.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:47 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:10:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2274936136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:10:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:47 smithi162 ceph-mon[20279]: osdmap e312: 12 total, 12 up, 12 in 2024-09-02T10:10:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3846414505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:47.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:47.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:47.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:47.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:47.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:47 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:10:47.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2274936136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:10:47.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:47 smithi187 ceph-mon[20072]: osdmap e312: 12 total, 12 up, 12 in 2024-09-02T10:10:47.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3846414505' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:47.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:47.844 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:47.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:47.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:47.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:47.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:47.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:48.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:48.096 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:48.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:48.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:48.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:48.224 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:48.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:48.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:48.352 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:48.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:48.433 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:10:48.433 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:10:48.434 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:10:48.434 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:10:48.455 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:48.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:48.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:48.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:48.584 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:48 smithi162 ceph-mon[20279]: pgmap v786: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4/6 objects degraded (66.667%); 0 B/s, 190 objects/s recovering 2024-09-02T10:10:48.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:48.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:48.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:48 smithi151 ceph-mon[16639]: pgmap v786: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4/6 objects degraded (66.667%); 0 B/s, 190 objects/s recovering 2024-09-02T10:10:48.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:48.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:48.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:48 smithi187 ceph-mon[20072]: pgmap v786: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4/6 objects degraded (66.667%); 0 B/s, 190 objects/s recovering 2024-09-02T10:10:48.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:48.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:48.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:48.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:48.988 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:48.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:49.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:49.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:10:49.115 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:49.116 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:49.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:49.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:49.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:49.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:49.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:49.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:49.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:49.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:49.494 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:49.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:49.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:49 smithi162 ceph-mon[20279]: osdmap e313: 12 total, 12 up, 12 in 2024-09-02T10:10:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:49 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:10:49.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:49.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:49.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:49 smithi151 ceph-mon[16639]: osdmap e313: 12 total, 12 up, 12 in 2024-09-02T10:10:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:49 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:10:49.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:10:49.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:49.741 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:49.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:49 smithi187 ceph-mon[20072]: osdmap e313: 12 total, 12 up, 12 in 2024-09-02T10:10:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:49 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 4/6 objects degraded (66.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:10:49.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:49.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:49.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:49.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:49.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:49.990 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:50.066 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:50.085 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:50.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:50.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:50.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:50.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:50.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:50.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:50.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:50.343 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_1' removed 2024-09-02T10:10:50.372 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:50.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:50.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:50.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:50.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:50.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:50 smithi162 ceph-mon[20279]: pgmap v788: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 214 objects/s recovering 2024-09-02T10:10:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:50 smithi162 ceph-mon[20279]: osdmap e314: 12 total, 12 up, 12 in 2024-09-02T10:10:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3026284163' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:10:50.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:50.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:50.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:50.644 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:50 smithi151 ceph-mon[16639]: pgmap v788: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 214 objects/s recovering 2024-09-02T10:10:50.644 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:50 smithi151 ceph-mon[16639]: osdmap e314: 12 total, 12 up, 12 in 2024-09-02T10:10:50.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3026284163' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:10:50.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:50.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:50.752 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:50.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:50 smithi187 ceph-mon[20072]: pgmap v788: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 214 objects/s recovering 2024-09-02T10:10:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:50 smithi187 ceph-mon[20072]: osdmap e314: 12 total, 12 up, 12 in 2024-09-02T10:10:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3026284163' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:10:50.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:50.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:50.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:50.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:51.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:51.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:51.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:51.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:51.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:51.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:51.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:51.142 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:10:51.142 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:10:51.142 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-02T10:10:51.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 1000 0.05 2024-09-02T10:10:51.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:51.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:51.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:51.251 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:51.327 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:10:51.328 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:10:51.328 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:10:51.328 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:10:51.328 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:10:51.348 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:10:51.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:51.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:51.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:51.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:51.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:51.497 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:51.498 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:51.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:51.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:51.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:51.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3026284163' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:10:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:51 smithi151 ceph-mon[16639]: osdmap e315: 12 total, 12 up, 12 in 2024-09-02T10:10:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3294274309' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:10:51.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:10:51.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:51.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:10:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3026284163' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:10:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:51 smithi187 ceph-mon[20072]: osdmap e315: 12 total, 12 up, 12 in 2024-09-02T10:10:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3294274309' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:10:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3026284163' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:10:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:51 smithi162 ceph-mon[20279]: osdmap e315: 12 total, 12 up, 12 in 2024-09-02T10:10:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3294274309' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:10:51.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:51.874 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:51.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:51.900 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-02T10:10:51.901 INFO:tasks.radosbench:Beginning radosbench... 2024-09-02T10:10:51.901 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-02T10:10:51.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-02T10:10:51.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:10:51.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:52.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:52.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:52.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:52.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:52.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:52.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:52.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:52.252 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:52.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:52.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:52.380 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:52.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:52.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:52.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:52.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:52.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:52.512 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:52 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:10:52.512 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:52 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:10:52.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:52 smithi151 ceph-mon[16639]: pgmap v791: 31 pgs: 31 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:52.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:52 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-02T10:10:52.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:52 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:10:52.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3294274309' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:10:52.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:52 smithi151 ceph-mon[16639]: osdmap e316: 12 total, 12 up, 12 in 2024-09-02T10:10:52.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:52.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:52.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:52 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:10:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:52 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:10:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:52 smithi187 ceph-mon[20072]: pgmap v791: 31 pgs: 31 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:52 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-02T10:10:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:52 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:10:52.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3294274309' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:10:52.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:52 smithi187 ceph-mon[20072]: osdmap e316: 12 total, 12 up, 12 in 2024-09-02T10:10:52.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:52 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:10:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:52 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:10:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:52 smithi162 ceph-mon[20279]: pgmap v791: 31 pgs: 31 active+clean; 577 KiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:52 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-02T10:10:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:52 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:10:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3294274309' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:10:52.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:52 smithi162 ceph-mon[20279]: osdmap e316: 12 total, 12 up, 12 in 2024-09-02T10:10:52.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:52.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:52.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:52.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:52.887 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:52.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:52.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:53.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:53.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:53.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:53.140 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:53.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:53.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:53.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:10:53.213 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:53.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:53.261 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:53.261 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:53.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:53.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:53.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:53.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:53.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:53.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:53 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:10:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:53 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:10:53.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:53.520 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:53.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:53.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:53.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:53.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:53.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:53 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:10:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:53 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:10:53.776 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:53.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:53.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:53 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:10:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:53 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:10:53.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:53.903 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:53.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:53.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:54.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:54.031 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:54.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:54.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:54.129 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:10:54.129 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:10:54.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:54.150 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:10:54.152 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:10:54.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:54.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:54.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:54.185 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:10:54.186 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:10:54.206 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:10:54.206 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:10:54.227 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:10:54.227 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:10:54.248 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:10:54.249 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:10:54.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:54.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:10:54.271 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:10:54.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:54.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:54.292 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:10:54.293 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:10:54.314 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:10:54.314 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:10:54.336 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:10:54.336 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:10:54.357 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:10:54.357 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:10:54.360 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 93 (0%) 2024-09-02T10:10:54.360 INFO:teuthology.orchestra.run.smithi151.stdout:avg 7.75 2024-09-02T10:10:54.361 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 3.00347 -> 3.00347 (expected baseline 2.66536) 2024-09-02T10:10:54.361 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.4 with 3 -> 3 pgs (0.387097 -> 0.387097 * mean) 2024-09-02T10:10:54.361 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 12 -> 12 pgs (1.54839 -> 1.54839 * mean) 2024-09-02T10:10:54.361 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:10:54.361 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-09-02T10:10:54.361 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 0.05 2024-09-02T10:10:54.361 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:10:54.361 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0024 2024-09-02T10:10:54.361 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0241 2024-09-02T10:10:54.361 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 0.8931 -> 0.9431 2024-09-02T10:10:54.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:54 smithi151 ceph-mon[16639]: pgmap v793: 31 pgs: 31 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:54.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:54 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:10:54.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:54 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:10:54.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:54 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:10:54.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:54 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:10:54.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/447118703' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:54.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:54 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'61807'}"} : dispatch 2024-09-02T10:10:54.379 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:10:54.379 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:10:54.383 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:54.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:54.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:54.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:54.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:54.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:54.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:10:54.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:54.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:54.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:54.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:54 smithi187 ceph-mon[20072]: pgmap v793: 31 pgs: 31 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:54 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:10:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:54 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:10:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:54 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:10:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:54 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:10:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/447118703' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:54 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'61807'}"} : dispatch 2024-09-02T10:10:54.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:54.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:54.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:54 smithi162 ceph-mon[20279]: pgmap v793: 31 pgs: 31 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:54 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:10:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:54 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:10:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:54 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:10:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:54 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:10:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/447118703' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:10:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:54 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'61807'}"} : dispatch 2024-09-02T10:10:54.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:54.913 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:54.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:55.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:55.041 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:55.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:55.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:55.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:55.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:55.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:55.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:55.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:55.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:55.367 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:10:55.367 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:10:55.367 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:10:55.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:10:55.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:10:55.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:10:55.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:10:55.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'61807'}"}]': finished 2024-09-02T10:10:55.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: osdmap e317: 12 total, 12 up, 12 in 2024-09-02T10:10:55.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:10:55.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:10:55.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:10:55.368 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:55 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:10:55.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:55.415 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:55.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:55.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:55.517 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:10:55.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:55.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:55.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:10:55.613 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-02T10:10:55.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 0.05 2024-09-02T10:10:55.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:10:55.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:10:55.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:10:55.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:10:55.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:10:55.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:10:55.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:10:55.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'61807'}"}]': finished 2024-09-02T10:10:55.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: osdmap e317: 12 total, 12 up, 12 in 2024-09-02T10:10:55.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:10:55.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:10:55.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:10:55.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:55 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:10:55.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:55.669 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:55.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:55.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:10:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'61807'}"}]': finished 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: osdmap e317: 12 total, 12 up, 12 in 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:10:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:55 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:10:55.769 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:55.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:55.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:55.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:55.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:55.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:55.926 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:55.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:56.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:56.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:56.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:56.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:10:56.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:56.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:56.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:56.243 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:10:56.243 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:10:56.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:56.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:56.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:56.372 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_2' created 2024-09-02T10:10:56.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:10:56.429 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:56.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: pgmap v795: 31 pgs: 31 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:10:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:10:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1647552815' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-02T10:10:56.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:56.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:56.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:56.558 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:56.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:56.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:56.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:56.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: pgmap v795: 31 pgs: 31 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:10:56.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1647552815' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-02T10:10:56.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:56.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:56.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:56.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: pgmap v795: 31 pgs: 31 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:10:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:10:56.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1647552815' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16} : dispatch 2024-09-02T10:10:56.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:56.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:56.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:57.031 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:57.056 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:57.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:57.057 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:57.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:57.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-02T10:10:57.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:57.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:57.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:57.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:57.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:57.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:57.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:10:57.433 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:57.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:10:57.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1647552815' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-02T10:10:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:57 smithi151 ceph-mon[16639]: osdmap e318: 12 total, 12 up, 12 in 2024-09-02T10:10:57.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:57.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:57.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:57.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:57.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:57.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:57.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:57.704 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:57.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:10:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1647552815' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-02T10:10:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:57 smithi187 ceph-mon[20072]: osdmap e318: 12 total, 12 up, 12 in 2024-09-02T10:10:57.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:57.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:57.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:57.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:57.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1647552815' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-02T10:10:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:57 smithi162 ceph-mon[20279]: osdmap e318: 12 total, 12 up, 12 in 2024-09-02T10:10:57.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:57.932 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:57.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:58.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:58.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:58.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:58.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:58.185 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:58.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:58.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:10:58.313 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:58.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:58.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:10:58.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:58.441 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:58.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:58.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:58.570 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:58.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:58.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:58.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:58.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:58 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:10:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:58 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:10:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:58 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:10:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:58 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:10:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:58 smithi151 ceph-mon[16639]: pgmap v797: 47 pgs: 4 creating+peering, 12 unknown, 31 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:58 smithi151 ceph-mon[16639]: osdmap e319: 12 total, 12 up, 12 in 2024-09-02T10:10:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:58.699 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:58.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:58 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:10:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:58 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:10:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:58 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:10:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:58 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:10:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:58 smithi187 ceph-mon[20072]: pgmap v797: 47 pgs: 4 creating+peering, 12 unknown, 31 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:58 smithi187 ceph-mon[20072]: osdmap e319: 12 total, 12 up, 12 in 2024-09-02T10:10:58.761 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:58.800 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:10:58.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:58.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:10:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:58 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:10:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:58 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:10:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:58 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:10:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:58 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:10:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:58 smithi162 ceph-mon[20279]: pgmap v797: 47 pgs: 4 creating+peering, 12 unknown, 31 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:10:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:58 smithi162 ceph-mon[20279]: osdmap e319: 12 total, 12 up, 12 in 2024-09-02T10:10:58.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:58.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:58.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:59.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:10:59.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:59.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:59.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:10:59.175 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:59.202 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:59.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:59.303 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:10:59.321 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:59.322 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:10:59.328 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:59.330 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:10:59.392 INFO:teuthology.orchestra.run.smithi151.stdout:moved 1 / 141 (0.70922%) 2024-09-02T10:10:59.393 INFO:teuthology.orchestra.run.smithi151.stdout:avg 11.75 2024-09-02T10:10:59.393 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 3.72212 -> 3.89711 (expected baseline 3.28189) 2024-09-02T10:10:59.393 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.11 with 6 -> 6 pgs (0.510638 -> 0.510638 * mean) 2024-09-02T10:10:59.393 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.1 with 17 -> 17 pgs (1.44681 -> 1.44681 * mean) 2024-09-02T10:10:59.393 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:10:59.393 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:10:59.393 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 0.05 2024-09-02T10:10:59.393 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:10:59.393 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0024 2024-09-02T10:10:59.394 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0026 2024-09-02T10:10:59.394 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 weight 0.6430 -> 0.5930 2024-09-02T10:10:59.394 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 0.8348 -> 0.7848 2024-09-02T10:10:59.394 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11 weight 0.5559 -> 0.5059 2024-09-02T10:10:59.394 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 0.9431 -> 0.9931 2024-09-02T10:10:59.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:59.456 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:59.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:59.559 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:10:59.584 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:10:59.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:59.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:10:59.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:59 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:10:59.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:59 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:10:59.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:59 smithi151 ceph-mon[16639]: osdmap e320: 12 total, 12 up, 12 in 2024-09-02T10:10:59.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:59 smithi151 ceph-mon[16639]: from='client.15906 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:10:59.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:59 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65083','3':'51432','4':'38861','11':'33154'}"} : dispatch 2024-09-02T10:10:59.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:59 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:10:59.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:10:59 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:10:59.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:10:59.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:59.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:10:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:59 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:10:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:59 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:10:59.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:59 smithi187 ceph-mon[20072]: osdmap e320: 12 total, 12 up, 12 in 2024-09-02T10:10:59.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:59 smithi187 ceph-mon[20072]: from='client.15906 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:10:59.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:59 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65083','3':'51432','4':'38861','11':'33154'}"} : dispatch 2024-09-02T10:10:59.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:59 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:10:59.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:10:59 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:10:59.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:10:59.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:10:59.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:59.837 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:10:59.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:59 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:10:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:59 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:10:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:59 smithi162 ceph-mon[20279]: osdmap e320: 12 total, 12 up, 12 in 2024-09-02T10:10:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:59 smithi162 ceph-mon[20279]: from='client.15906 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:10:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:59 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65083','3':'51432','4':'38861','11':'33154'}"} : dispatch 2024-09-02T10:10:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:59 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:10:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:10:59 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:10:59.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:10:59.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:10:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:10:59.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:00.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:00.086 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:00.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:00.087 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:00.187 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:00.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:00.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:00.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:00.336 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:00.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:00.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:00.400 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:11:00.421 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-02T10:11:00.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-02T10:11:00.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:00.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:00.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:00.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:00.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:00.592 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:00.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:00.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:00 smithi151 ceph-mon[16639]: pgmap v800: 47 pgs: 4 creating+peering, 4 unknown, 39 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:11:00.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:00 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:11:00.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:00 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:11:00.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:00 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:11:00.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:00 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:11:00.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:00 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65083','3':'51432','4':'38861','11':'33154'}"}]': finished 2024-09-02T10:11:00.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:00 smithi151 ceph-mon[16639]: osdmap e321: 12 total, 12 up, 12 in 2024-09-02T10:11:00.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1542719817' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:00.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:00.718 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:00.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:00 smithi187 ceph-mon[20072]: pgmap v800: 47 pgs: 4 creating+peering, 4 unknown, 39 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:11:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:00 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:11:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:00 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:11:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:00 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:11:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:00 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:11:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:00 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65083','3':'51432','4':'38861','11':'33154'}"}]': finished 2024-09-02T10:11:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:00 smithi187 ceph-mon[20072]: osdmap e321: 12 total, 12 up, 12 in 2024-09-02T10:11:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1542719817' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:00.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:11:00.821 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:00 smithi162 ceph-mon[20279]: pgmap v800: 47 pgs: 4 creating+peering, 4 unknown, 39 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:11:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:00 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:11:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:00 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:11:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:00 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:11:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:00 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:11:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:00 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65083','3':'51432','4':'38861','11':'33154'}"}]': finished 2024-09-02T10:11:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:00 smithi162 ceph-mon[20279]: osdmap e321: 12 total, 12 up, 12 in 2024-09-02T10:11:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1542719817' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:00.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:00.847 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:00.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:00.975 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:00.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:00.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:01.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:01.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:01.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:01.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:01.226 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:01.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:01.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:01.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:01.355 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:01.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:01.402 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-02T10:11:01.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:01.468 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:11:01.468 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:11:01.468 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:11:01.468 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:11:01.468 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1542719817' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:01.468 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: osdmap e322: 12 total, 12 up, 12 in 2024-09-02T10:11:01.468 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:11:01.468 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:11:01.469 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:11:01.469 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:11:01.469 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2030334967' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:11:01.469 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:11:01.469 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:01 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:11:01.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:01.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:01.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:01.606 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:01.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:01.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:01.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:01.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:11:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:11:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:11:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:11:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1542719817' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:01.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: osdmap e322: 12 total, 12 up, 12 in 2024-09-02T10:11:01.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:11:01.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:11:01.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:11:01.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:11:01.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2030334967' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:11:01.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:11:01.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:01 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:11:01.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1542719817' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: osdmap e322: 12 total, 12 up, 12 in 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2030334967' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:11:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:11:01.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:01 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:11:01.858 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:01.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:01.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:01.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:01.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:01.988 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:02.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:02.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:02.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:02.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:02.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:02.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:02.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:02.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:02.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:02.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:02.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:02.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:02.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:02.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:02.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:02 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:11:02.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:02 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:11:02.494 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:02 smithi151 ceph-mon[16639]: pgmap v803: 47 pgs: 4 creating+peering, 43 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:11:02.495 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2030334967' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:11:02.495 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:02 smithi151 ceph-mon[16639]: osdmap e323: 12 total, 12 up, 12 in 2024-09-02T10:11:02.495 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:02 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:11:02.495 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:02 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:11:02.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:02.618 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:02.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:02.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:02.709 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:11:02.709 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:11:02.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:02.746 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:02.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:02.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:02 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:11:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:02 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:11:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:02 smithi187 ceph-mon[20072]: pgmap v803: 47 pgs: 4 creating+peering, 43 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:11:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2030334967' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:11:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:02 smithi187 ceph-mon[20072]: osdmap e323: 12 total, 12 up, 12 in 2024-09-02T10:11:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:02 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:11:02.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:02 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:11:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:02 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:11:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:02 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:11:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:02 smithi162 ceph-mon[20279]: pgmap v803: 47 pgs: 4 creating+peering, 43 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:11:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2030334967' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:11:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:02 smithi162 ceph-mon[20279]: osdmap e323: 12 total, 12 up, 12 in 2024-09-02T10:11:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:02 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:11:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:02 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:11:02.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:02.874 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:02.875 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:02.954 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:02.960 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:02.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:03.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:03.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:03.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:03.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:03.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:03.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:03.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:03.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:03.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:03.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:03.376 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:03.377 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:03.438 DEBUG:teuthology.orchestra.run.smithi187:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-02T10:11:03.441 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-02T10:11:03.441 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-02T10:11:03.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:03.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:03.501 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:hints = 1 2024-09-02T10:11:03.501 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-02T10:11:03.501 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object prefix: benchmark_data_smithi187_76542 2024-09-02T10:11:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:03.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:03.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:11:03.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 0 0 0 0 0 0 - 0 2024-09-02T10:11:03.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:11:03.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:03.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:03.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:03.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:03 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:11:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:03 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:11:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:03 smithi151 ceph-mon[16639]: osdmap e324: 12 total, 12 up, 12 in 2024-09-02T10:11:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:03 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:11:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:03 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:11:03.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:03.755 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:03.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:03 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:11:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:03 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:11:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:03 smithi187 ceph-mon[20072]: osdmap e324: 12 total, 12 up, 12 in 2024-09-02T10:11:03.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:03 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:11:03.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:03 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:11:03.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:11:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:03 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:11:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:03 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:11:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:03 smithi162 ceph-mon[20279]: osdmap e324: 12 total, 12 up, 12 in 2024-09-02T10:11:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:03 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:11:03.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:03 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:11:03.857 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:03.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:03.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:03.891 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:03.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:04.015 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:04.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:04.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:04.090 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:04.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:04.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:04.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:04.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:04.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:04.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:04.273 INFO:teuthology.orchestra.run.smithi151.stdout:moved 9 / 141 (6.38298%) 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:avg 11.75 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 3.89711 -> 5.06828 (expected baseline 3.28189) 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.4 with 6 -> 1 pgs (0.510638 -> 0.0851064 * mean) 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.1 with 17 -> 17 pgs (1.44681 -> 1.44681 * mean) 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0024 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0026 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 weight 0.5930 -> 0.2086 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 0.7848 -> 0.5622 2024-09-02T10:11:04.274 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11 weight 0.5059 -> 0.4022 2024-09-02T10:11:04.275 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 0.9931 -> 1.0000 2024-09-02T10:11:04.275 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:11:04.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:04.389 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 10:11:04 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: 2024-09-02T10:11:04.272+0000 7fd7798ae640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-02T10:11:04.395 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:04.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:04.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:04.501 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 1 16 1535 1519 94.9323 94.9375 0.00533801 0.0105025 2024-09-02T10:11:04.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:04.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:04.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:04.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:04.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:04.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:04.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:04 smithi151 ceph-mon[16639]: pgmap v806: 47 pgs: 47 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:11:04.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:04 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:11:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:04 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:11:04.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:04 smithi187 ceph-mon[20072]: pgmap v806: 47 pgs: 47 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:11:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:04 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:11:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:04 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:11:04.777 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:04.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:04.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:04 smithi162 ceph-mon[20279]: pgmap v806: 47 pgs: 47 active+clean; 577 KiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:11:04.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:04 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:11:04.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:04 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:11:04.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:04.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:04.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:05.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:05.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:05.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:05.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:05.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:05.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:05.155 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:05.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:11:05.191 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:11:05.216 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:11:05.217 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:11:05.240 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:11:05.240 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:11:05.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:05.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:11:05.265 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:11:05.283 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:05.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:05.290 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:11:05.290 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:11:05.315 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:11:05.315 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:11:05.352 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:11:05.352 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:11:05.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:05.388 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:11:05.389 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:11:05.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:05.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:05.413 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:11:05.414 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:11:05.441 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:11:05.441 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:11:05.442 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:05.455 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:11:05.455 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:11:05.455 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:11:05.455 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:11:05.455 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:11:05.471 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:11:05.471 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:11:05.497 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:11:05.498 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:11:05.501 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 2 16 3183 3167 98.9623 103 0.00137745 0.0100642 2024-09-02T10:11:05.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:05.534 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:05.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:05.570 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:05 smithi151 ceph-mon[16639]: from='client.25513 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:11:05.570 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:05 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:11:05.570 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:05 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:11:05.570 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1651689131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:05.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:05.659 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:05.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:05 smithi187 ceph-mon[20072]: from='client.25513 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:11:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:05 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:11:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:05 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:11:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1651689131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:05.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:05.785 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:05.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:05.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:05 smithi162 ceph-mon[20279]: from='client.25513 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:11:05.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:05 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:11:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:05 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:11:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1651689131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:05.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:05.911 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:05.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:05.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:06.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:06.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:06.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:06.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:06.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:06.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:06.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:06.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:06.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:06.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:06.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:06.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:06.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:06.468 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-02T10:11:06.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 3.0 2024-09-02T10:11:06.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 3 16 4468 4452 92.7437 80.3125 0.00135613 0.0107293 2024-09-02T10:11:06.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:06.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:06.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:06.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:06.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:06.670 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:06.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:06.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:06 smithi151 ceph-mon[16639]: pgmap v807: 47 pgs: 47 active+clean; 9.4 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 24 op/s 2024-09-02T10:11:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:06 smithi187 ceph-mon[20072]: pgmap v807: 47 pgs: 47 active+clean; 9.4 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 24 op/s 2024-09-02T10:11:06.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:06.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:06.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:06.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:06 smithi162 ceph-mon[20279]: pgmap v807: 47 pgs: 47 active+clean; 9.4 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s wr, 24 op/s 2024-09-02T10:11:06.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:06.932 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:06.933 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:07.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:07.058 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:07.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:07.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:07.183 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:07.185 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:07.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:07.307 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:07.308 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:07.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:07.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:07.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:07.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 4 16 6137 6121 95.6337 104.312 0.012607 0.0104351 2024-09-02T10:11:07.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:07.560 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:07.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:07.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:07.689 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:07.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:07.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:07.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:07.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:07.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:07.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:07.945 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:07.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:07.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:08.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:08.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:08.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:08.078 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:08.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:08.200 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:08.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:08.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:11:08.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:08.326 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:08.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:08.350 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:08.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:08.434 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:11:08.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:08.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:08.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 5 16 8335 8319 103.98 137.375 0.0135192 0.00960138 2024-09-02T10:11:08.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:08.592 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:08.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:08 smithi151 ceph-mon[16639]: pgmap v808: 47 pgs: 47 active+clean; 127 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 297 op/s 2024-09-02T10:11:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:08 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:11:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:08 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:11:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:08 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:11:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:08 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:11:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1336510260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:11:08.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:08.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:08.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:08 smithi187 ceph-mon[20072]: pgmap v808: 47 pgs: 47 active+clean; 127 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 297 op/s 2024-09-02T10:11:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:08 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:11:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:08 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:11:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:08 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:11:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:08 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:11:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1336510260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:11:08.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:08.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:08.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:08.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:08 smithi162 ceph-mon[20279]: pgmap v808: 47 pgs: 47 active+clean; 127 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 19 MiB/s wr, 297 op/s 2024-09-02T10:11:08.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:08 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:11:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:08 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:11:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:08 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:11:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:08 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:11:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1336510260' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:11:08.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:08.970 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:08.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:08.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:08.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:11:09.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:09.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:09.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:09.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:09.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:09.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:09.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:09.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:09.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:09.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:09.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:09.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:09.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:09.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:09.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 6 16 10162 10146 105.68 114.188 0.00770153 0.00945021 2024-09-02T10:11:09.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:09.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:09.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:09 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1336510260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:11:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:09 smithi151 ceph-mon[16639]: osdmap e325: 12 total, 12 up, 12 in 2024-09-02T10:11:09.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:09.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:09.735 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:09 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1336510260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:11:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:09 smithi187 ceph-mon[20072]: osdmap e325: 12 total, 12 up, 12 in 2024-09-02T10:11:09.836 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:09.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:09 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:09.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1336510260' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:11:09.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:09 smithi162 ceph-mon[20279]: osdmap e325: 12 total, 12 up, 12 in 2024-09-02T10:11:09.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:09.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:09.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:09.880 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:09.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:09.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:09.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:09.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:10.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:10.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:10.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:10.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:10.236 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:10.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:10.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:10.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:10.369 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:10.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:10.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:10.457 INFO:teuthology.orchestra.run.smithi151.stdout:moved 9 / 141 (6.38298%) 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:avg 11.75 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 3.89711 -> 4.71036 (expected baseline 3.28189) 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.4 with 6 -> 3 pgs (0.510638 -> 0.255319 * mean) 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.1 with 17 -> 15 pgs (1.44681 -> 1.2766 * mean) 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0032 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0035 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 weight 0.5930 -> 0.2863 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 weight 1.0000 -> 0.7452 2024-09-02T10:11:10.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 0.7848 -> 0.6009 2024-09-02T10:11:10.459 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11 weight 0.5059 -> 0.4314 2024-09-02T10:11:10.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:10.492 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:10.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:10.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:10.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 7 16 11558 11542 103.046 87.25 0.0170866 0.009646 2024-09-02T10:11:10.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:10.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:10 smithi151 ceph-mon[16639]: pgmap v810: 47 pgs: 47 active+clean; 365 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 745 op/s 2024-09-02T10:11:10.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:10 smithi151 ceph-mon[16639]: from='client.25528 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:11:10.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'48839','3':'39379','4':'18765','11':'28269'}"} : dispatch 2024-09-02T10:11:10.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:10 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:11:10.617 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:10.618 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:10.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:10.741 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:10.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:10.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:10 smithi187 ceph-mon[20072]: pgmap v810: 47 pgs: 47 active+clean; 365 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 745 op/s 2024-09-02T10:11:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:10 smithi187 ceph-mon[20072]: from='client.25528 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:11:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'48839','3':'39379','4':'18765','11':'28269'}"} : dispatch 2024-09-02T10:11:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:10 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:11:10.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:10 smithi162 ceph-mon[20279]: pgmap v810: 47 pgs: 47 active+clean; 365 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 745 op/s 2024-09-02T10:11:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:10 smithi162 ceph-mon[20279]: from='client.25528 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:11:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'1':'48839','3':'39379','4':'18765','11':'28269'}"} : dispatch 2024-09-02T10:11:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:10 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:11:10.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:10.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:10.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:10.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:10.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:10.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:10.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:11.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:11.112 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:11.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:11.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:11.236 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:11.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:11.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:11.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:11.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:11.361 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:11.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:11.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:11.484 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-02T10:11:11.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-09-02T10:11:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:11.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:11.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:11.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 8 16 11584 11568 90.368 1.625 0.00172704 0.00983595 2024-09-02T10:11:11.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:11.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:11.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:11.648 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:11:11.648 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:11:11.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:11:11.648 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:11:11.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.648 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:97423 3.b 3:da6c5fce:::benchmark_data_smithi187_59989_object97422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-02T10:11:11.648 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:20.582338+0000", 2024-09-02T10:11:11.648 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 230.02558174800001, 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26220150199999998, 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 97423 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.582338+0000", 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.582338+0000", 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.582344+0000", 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9270000000000002e-06 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.582393+0000", 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9150000000000002e-05 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.582421+0000", 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8014999999999999e-05 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.582435+0000", 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3546999999999999e-05 2024-09-02T10:11:11.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.582507+0000", 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2068000000000002e-05 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.582597+0000", 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0517000000000001e-05 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:11:11.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.582686+0000", 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8451999999999999e-05 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.585108+0000", 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024219749999999998 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.585433+0000", 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00032488699999999999 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.844441+0000", 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25900793599999999 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.844499+0000", 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8111e-05 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:20.844540+0000", 2024-09-02T10:11:11.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0917000000000001e-05 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:97754 3.2 3:44adf9ba:::benchmark_data_smithi187_59989_object97753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:21.111344+0000", 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 229.49657564200001, 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26476896900000002, 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.655 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 97754 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.111344+0000", 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.111344+0000", 2024-09-02T10:11:11.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.111346+0000", 2024-09-02T10:11:11.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.455e-06 2024-09-02T10:11:11.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.111416+0000", 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0479000000000001e-05 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.111430+0000", 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4076999999999999e-05 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.111439+0000", 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6479999999999992e-06 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.111519+0000", 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.017e-05 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.111629+0000", 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110373 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.111767+0000", 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013775899999999999 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.114575+0000", 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002808175 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.114598+0000", 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2840000000000002e-05 2024-09-02T10:11:11.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.376039+0000", 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26144063000000001 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.376073+0000", 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4331e-05 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.376113+0000", 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0031999999999999e-05 2024-09-02T10:11:11.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:97754 3.2 e260/226 ondisk, result = 0)", 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:21.115072+0000", 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 229.49284762600001, 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26104606800000002, 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.115072+0000", 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.115072+0000", 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.115073+0000", 2024-09-02T10:11:11.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5099999999999999e-07 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.115077+0000", 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2440000000000004e-06 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.375996+0000", 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26091896999999997 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.375999+0000", 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0460000000000001e-06 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.376030+0000", 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1080000000000001e-05 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.376036+0000", 2024-09-02T10:11:11.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2399999999999998e-06 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:21.376118+0000", 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2737000000000004e-05 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:109490 3.b 3:d613b4ca:::benchmark_data_smithi187_59989_object109489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:32.977730+0000", 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.63018949799999, 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.53911734600000005, 2024-09-02T10:11:11.666 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109490 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977730+0000", 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977730+0000", 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977732+0000", 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.497e-06 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977772+0000", 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9984000000000002e-05 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977788+0000", 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6146000000000001e-05 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977796+0000", 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5549999999999997e-06 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977869+0000", 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3405999999999995e-05 2024-09-02T10:11:11.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978004+0000", 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000134912 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978159+0000", 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000154635 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.981783+0000", 2024-09-02T10:11:11.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036246579999999998 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.982194+0000", 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041032900000000002 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.516775+0000", 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.53458172199999998 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.516815+0000", 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9320000000000003e-05 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.516848+0000", 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3182000000000002e-05 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:109515 3.7 3:e47edbba:::benchmark_data_smithi187_59989_object109514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-02T10:11:11.672 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:33.000140+0000", 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.60777970699999, 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.66181097899999997, 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109515 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000140+0000", 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000140+0000", 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000142+0000", 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.652e-06 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000185+0000", 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2762000000000002e-05 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000199+0000", 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4448e-05 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000208+0000", 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5380000000000002e-06 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000277+0000", 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9122000000000007e-05 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000429+0000", 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015264900000000001 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000612+0000", 2024-09-02T10:11:11.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018222299999999999 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.007079+0000", 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064675729999999999 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409419+0000", 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40233940000000001 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661860+0000", 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25244123099999999 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661908+0000", 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8597999999999998e-05 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661951+0000", 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2783e-05 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:109515 3.7 e263/226 ondisk, result = 0)", 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:33.004435+0000", 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.60348496200001, 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40498933300000001, 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004435+0000", 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004435+0000", 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004436+0000", 2024-09-02T10:11:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0599999999999999e-07 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004444+0000", 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0960000000000001e-06 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409298+0000", 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.404853835 2024-09-02T10:11:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409307+0000", 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2180000000000005e-06 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409404+0000", 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6917999999999996e-05 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409413+0000", 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9299999999999992e-06 2024-09-02T10:11:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409424+0000", 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.153e-05 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:109515 3.7 e263/226 ondisk, result = 0)", 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:33.004543+0000", 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.60337719500001, 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.65741479000000003, 2024-09-02T10:11:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004543+0000", 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004543+0000", 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004543+0000", 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7199999999999999e-07 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004549+0000", 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4430000000000002e-06 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661760+0000", 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.65721097299999998 2024-09-02T10:11:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661768+0000", 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3690000000000007e-06 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661842+0000", 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4200000000000001e-05 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661854+0000", 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1608e-05 2024-09-02T10:11:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661958+0000", 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103625 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:109529 3.2 3:4d5fc6c5:::benchmark_data_smithi187_59989_object109528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:33.012817+0000", 2024-09-02T10:11:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.59510241699999, 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.676394685, 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109529 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012817+0000", 2024-09-02T10:11:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012817+0000", 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012819+0000", 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6640000000000001e-06 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012865+0000", 2024-09-02T10:11:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5345e-05 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012886+0000", 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999999e-05 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012896+0000", 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4010000000000003e-06 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012963+0000", 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7180000000000004e-05 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.013114+0000", 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015087100000000001 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.013264+0000", 2024-09-02T10:11:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015018599999999999 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.016396+0000", 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031319469999999999 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.016510+0000", 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011406 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689111+0000", 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67260133600000005 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689165+0000", 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4035e-05 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689212+0000", 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6810000000000001e-05 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:109529 3.2 e263/226 ondisk, result = 0)", 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:33.018049+0000", 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.58987076299999, 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67117003600000003, 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.018049+0000", 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.018049+0000", 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.018050+0000", 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.018055+0000", 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8790000000000002e-06 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689017+0000", 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67096189699999997 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689027+0000", 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0617e-05 2024-09-02T10:11:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689094+0000", 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6507000000000005e-05 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689105+0000", 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1164e-05 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689219+0000", 2024-09-02T10:11:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011430999999999999 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155062 3.2 3:4fe18cc9:::benchmark_data_smithi187_59989_object45519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.476035+0000", 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 166.13188516599999, 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.292284565, 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155062 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.476035+0000", 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.476035+0000", 2024-09-02T10:11:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.476036+0000", 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.502e-06 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.476043+0000", 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.657e-06 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.476058+0000", 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5324000000000001e-05 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.476065+0000", 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1489999999999997e-06 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.476119+0000", 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3701e-05 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.476206+0000", 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6672e-05 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.476276+0000", 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0668999999999997e-05 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.479752+0000", 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034751679999999998 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.480030+0000", 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027877199999999999 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.768247+0000", 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28821702700000001 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.768285+0000", 2024-09-02T10:11:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8078999999999999e-05 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.768319+0000", 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3844999999999999e-05 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:154914 3.7 3:e3087e7e:::benchmark_data_smithi187_59989_object45371:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292575+0000", 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 164.31534484599999, 2024-09-02T10:11:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29543513199999999, 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154914 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292575+0000", 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292575+0000", 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292577+0000", 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.103e-06 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292586+0000", 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0019999999999995e-06 2024-09-02T10:11:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292604+0000", 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8326999999999999e-05 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292618+0000", 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3237e-05 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294561+0000", 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001942788 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294575+0000", 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4076e-05 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.587841+0000", 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29326616700000002 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.588010+0000", 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016943199999999999 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:154922 3.7 3:e3f98aef:::benchmark_data_smithi187_59989_object45379:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292652+0000", 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 164.31526802100001, 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29827295999999998, 2024-09-02T10:11:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154922 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292652+0000", 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292652+0000", 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292652+0000", 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0800000000000005e-07 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292656+0000", 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1559999999999999e-06 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292666+0000", 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9850000000000001e-06 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292671+0000", 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4060000000000004e-06 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294602+0000", 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001931449 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294609+0000", 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0999999999999998e-06 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.590721+0000", 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.296111387 2024-09-02T10:11:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.590925+0000", 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000203969 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:154987 3.7 3:e9a05f93:::benchmark_data_smithi187_59989_object45444:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292679+0000", 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 164.315240596, 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31734095899999998, 2024-09-02T10:11:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154987 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292679+0000", 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292679+0000", 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292680+0000", 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7500000000000001e-07 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292682+0000", 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2840000000000001e-06 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292689+0000", 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6540000000000002e-06 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292693+0000", 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5879999999999999e-06 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294669+0000", 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019755459999999999 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294678+0000", 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4849999999999998e-06 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.590957+0000", 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29627842999999998 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.591095+0000", 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013822899999999999 2024-09-02T10:11:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.591239+0000", 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144518 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.606991+0000", 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015751363000000001 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.607059+0000", 2024-09-02T10:11:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8637e-05 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.607077+0000", 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7581999999999999e-05 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.607146+0000", 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8596000000000005e-05 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.610020+0000", 2024-09-02T10:11:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002874672 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155049 3.7 3:e346e101:::benchmark_data_smithi187_59989_object45506:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292699+0000", 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 164.31522047199999, 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33504108799999999, 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155049 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292699+0000", 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292699+0000", 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292700+0000", 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7700000000000001e-07 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292702+0000", 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9180000000000001e-06 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292719+0000", 2024-09-02T10:11:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6994999999999999e-05 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292723+0000", 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6380000000000003e-06 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294733+0000", 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020092690000000002 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294740+0000", 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4490000000000002e-06 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.602174+0000", 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30743415000000002 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.602323+0000", 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014901700000000001 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.602441+0000", 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117873 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.621007+0000", 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018566426 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.621054+0000", 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6263000000000001e-05 2024-09-02T10:11:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.624195+0000", 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031411410000000001 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.624316+0000", 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120695 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627741+0000", 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034249770000000001 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155068 3.7 3:e666e28e:::benchmark_data_smithi187_59989_object45525:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 164.31518996400001, 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33844332199999999, 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155068 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8599999999999999e-07 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292733+0000", 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4530000000000001e-06 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292739+0000", 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3090000000000001e-06 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292745+0000", 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8420000000000001e-06 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294766+0000", 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020206989999999999 2024-09-02T10:11:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294769+0000", 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8099999999999999e-06 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.615962+0000", 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32119220799999998 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.616132+0000", 2024-09-02T10:11:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000170359 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.616296+0000", 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016439699999999999 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627817+0000", 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011520871 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627838+0000", 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0364999999999999e-05 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627892+0000", 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4667999999999999e-05 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627957+0000", 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4299000000000002e-05 2024-09-02T10:11:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631173+0000", 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003216656 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155089 3.7 3:e3752e01:::benchmark_data_smithi187_59989_object45546:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:11:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 164.31516929599999, 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35327147199999998, 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155089 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0199999999999998e-07 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292753+0000", 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.182e-06 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292759+0000", 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6450000000000002e-06 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292763+0000", 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4530000000000004e-06 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294792+0000", 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020289150000000001 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294796+0000", 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5149999999999998e-06 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631221+0000", 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33642498300000001 2024-09-02T10:11:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631396+0000", 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175866 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631567+0000", 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017052899999999999 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642330+0000", 2024-09-02T10:11:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010763403 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642513+0000", 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018293399999999999 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642560+0000", 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6344000000000002e-05 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642615+0000", 2024-09-02T10:11:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5634000000000003e-05 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.646022+0000", 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034067670000000002 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155155 3.7 3:e495e8d5:::benchmark_data_smithi187_59989_object45612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.682259+0000", 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 163.92566067999999, 2024-09-02T10:11:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29771890600000001, 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155155 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.682259+0000", 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.682259+0000", 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.682261+0000", 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.623e-06 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.682266+0000", 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5450000000000003e-06 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.682276+0000", 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3109999999999995e-06 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.682282+0000", 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7449999999999998e-06 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.696559+0000", 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014276308999999999 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.696628+0000", 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8910000000000003e-05 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.727073+0000", 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.030445699 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.727178+0000", 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010477299999999999 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.727333+0000", 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015526900000000001 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.741759+0000", 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014425523000000001 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.746715+0000", 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049560009999999998 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.976525+0000", 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22980989399999999 2024-09-02T10:11:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.976574+0000", 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9598000000000002e-05 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.979978+0000", 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034037059999999998 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155175 3.7 3:e0a2507c:::benchmark_data_smithi187_59989_object45632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.727976+0000", 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 163.879944362, 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25465184699999999, 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155175 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.727976+0000", 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.727976+0000", 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.727977+0000", 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8339999999999999e-06 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.727984+0000", 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5649999999999998e-06 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.727998+0000", 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4341e-05 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.728008+0000", 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4020000000000008e-06 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.741803+0000", 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013795112999999999 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.742193+0000", 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038985400000000003 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.742346+0000", 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000153196 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.756939+0000", 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014592765000000001 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.756964+0000", 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5063999999999999e-05 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.980061+0000", 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.223097199 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.980120+0000", 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9181000000000001e-05 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.982627+0000", 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002507333 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155193 3.7 3:e5ffd3d5:::benchmark_data_smithi187_59989_object45650:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e284)", 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.758993+0000", 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 163.84892718699999, 2024-09-02T10:11:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24240935999999999, 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155193 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.758993+0000", 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.758993+0000", 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.758995+0000", 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3709999999999998e-06 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.759003+0000", 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4699999999999996e-06 2024-09-02T10:11:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.759021+0000", 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8703e-05 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.759031+0000", 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6639999999999993e-06 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.759115+0000", 2024-09-02T10:11:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3888999999999996e-05 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.759207+0000", 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2127999999999996e-05 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:27.001385+0000", 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.242178316 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:27.001402+0000", 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6818999999999999e-05 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155193 3.7 3:e5ffd3d5:::benchmark_data_smithi187_59989_object45650:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e284)", 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.770480+0000", 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 163.837439588, 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26340817700000002, 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:11:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155193 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.770480+0000", 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:11:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.770480+0000", 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.770481+0000", 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8599999999999999e-07 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.770482+0000", 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4589999999999999e-06 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:11:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.770487+0000", 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4240000000000001e-06 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.770491+0000", 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.501e-06 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.770685+0000", 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000193804 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.770796+0000", 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111347 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:27.001605+0000", 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23080893299999999 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:27.001724+0000", 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119422 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:27.001871+0000", 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014664100000000001 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:27.012244+0000", 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010372609 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:27.012285+0000", 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1727999999999998e-05 2024-09-02T10:11:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:27.030774+0000", 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018488800999999999 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:27.030829+0000", 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4866e-05 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:27.033888+0000", 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030593560000000001 2024-09-02T10:11:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:11.768 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:11:11.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'48839','3':'39379','4':'18765','11':'28269'}"}]': finished 2024-09-02T10:11:11.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:11 smithi151 ceph-mon[16639]: osdmap e326: 12 total, 12 up, 12 in 2024-09-02T10:11:11.768 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:11.769 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'48839','3':'39379','4':'18765','11':'28269'}"}]': finished 2024-09-02T10:11:11.769 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:11 smithi187 ceph-mon[20072]: osdmap e326: 12 total, 12 up, 12 in 2024-09-02T10:11:11.790 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:11.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:11.791 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:11.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:11:11.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'48839','3':'39379','4':'18765','11':'28269'}"}]': finished 2024-09-02T10:11:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:11 smithi162 ceph-mon[20279]: osdmap e326: 12 total, 12 up, 12 in 2024-09-02T10:11:11.849 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:11.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:11.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:11.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:12.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:12.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:12.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:12.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:12.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:12.161 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:12.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:12.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:12.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:12.286 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:12.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:12.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:12.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:12.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 9 16 11599 11583 80.4312 0.9375 0.00166682 0.0106921 2024-09-02T10:11:12.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:12.542 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:12.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:12.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:12.670 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:12.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:12.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:12 smithi187 ceph-mon[20072]: pgmap v812: 47 pgs: 1 peering, 46 active+clean; 493 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 984 op/s 2024-09-02T10:11:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:12 smithi187 ceph-mon[20072]: osdmap e327: 12 total, 12 up, 12 in 2024-09-02T10:11:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:11:12.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:11:12.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:12 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:11:12.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:11:12.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:11:12.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:11:12.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:12.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:12.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:12.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:12 smithi151 ceph-mon[16639]: pgmap v812: 47 pgs: 1 peering, 46 active+clean; 493 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 984 op/s 2024-09-02T10:11:12.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:12 smithi151 ceph-mon[16639]: osdmap e327: 12 total, 12 up, 12 in 2024-09-02T10:11:12.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:11:12.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:11:12.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:12 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:11:12.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:11:12.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:11:12.812 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:11:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:12 smithi162 ceph-mon[20279]: pgmap v812: 47 pgs: 1 peering, 46 active+clean; 493 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 984 op/s 2024-09-02T10:11:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:12 smithi162 ceph-mon[20279]: osdmap e327: 12 total, 12 up, 12 in 2024-09-02T10:11:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:11:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:11:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:12 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:11:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:11:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:11:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:11:12.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:11:12.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:12.910 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:12.922 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:12.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:12.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:13.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:13.048 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:13.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:13.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:13.148 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:13.150 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:13.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:13.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:13.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:13.299 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:13.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:13.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:13.402 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:13.427 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:13.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:13.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 10 16 13020 13004 81.2686 88.8125 0.0128578 0.0122917 2024-09-02T10:11:13.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:13.554 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:13.555 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:13.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:13 smithi162 ceph-mon[20279]: osdmap e328: 12 total, 12 up, 12 in 2024-09-02T10:11:13.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:13 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:11:13.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:13 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:11:13.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:13.681 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:13.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:13.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:13 smithi187 ceph-mon[20072]: osdmap e328: 12 total, 12 up, 12 in 2024-09-02T10:11:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:13 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:11:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:13 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:11:13.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:13.811 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:13.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:13.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:13.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:13 smithi151 ceph-mon[16639]: osdmap e328: 12 total, 12 up, 12 in 2024-09-02T10:11:13.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:13 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:11:13.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:13 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:11:13.940 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:13.941 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:14.020 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:11:14.020 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:13446 4.e 4:79bcdcc2:::benchmark_data_smithi187_76542_object13445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:13.846823+0000", 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.042622396, 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042659186000000002, 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:14.021 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 13446 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.846823+0000", 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.846823+0000", 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:14.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.846823+0000", 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.61e-07 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.846910+0000", 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6432000000000002e-05 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.846920+0000", 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0349e-05 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:14.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.846926+0000", 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6869999999999999e-06 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.846956+0000", 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0069e-05 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.847024+0000", 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8448999999999995e-05 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,10", 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.847098+0000", 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4233999999999994e-05 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.847699+0000", 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00060082699999999996 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:13.848003+0000", 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00030404100000000001 2024-09-02T10:11:14.025 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:14.026 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:14.026 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:14.026 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:14.026 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:11:14.026 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-02T10:11:14.026 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:11:14.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:14.070 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:14.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:14.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:14.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:14.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:14.204 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:14.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:14.332 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:14.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:14.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:14.434 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:14.462 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:14.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:14.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 11 16 14423 14407 81.8515 87.6875 0.0126212 0.0122032 2024-09-02T10:11:14.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:14.590 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:14.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:14.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:14 smithi151 ceph-mon[16639]: pgmap v815: 47 pgs: 1 remapped+peering, 1 peering, 45 active+clean; 679 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 67 KiB/s, 0 objects/s recovering 2024-09-02T10:11:14.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:14 smithi151 ceph-mon[16639]: osdmap e329: 12 total, 12 up, 12 in 2024-09-02T10:11:14.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:14.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:14.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:14.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:14 smithi187 ceph-mon[20072]: pgmap v815: 47 pgs: 1 remapped+peering, 1 peering, 45 active+clean; 679 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 67 KiB/s, 0 objects/s recovering 2024-09-02T10:11:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:14 smithi187 ceph-mon[20072]: osdmap e329: 12 total, 12 up, 12 in 2024-09-02T10:11:14.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:14.835 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:11:14.836 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:11:14.836 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:11:14.836 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:11:14.836 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:11:14.836 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:11:14.836 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:11:14.836 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:11:14.836 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:11:14.836 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:11:14.836 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:11:14.837 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:11:14.837 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:11:14.837 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:11:14.837 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:11:14.838 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:11:14.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:14 smithi162 ceph-mon[20279]: pgmap v815: 47 pgs: 1 remapped+peering, 1 peering, 45 active+clean; 679 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 67 KiB/s, 0 objects/s recovering 2024-09-02T10:11:14.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:14 smithi162 ceph-mon[20279]: osdmap e329: 12 total, 12 up, 12 in 2024-09-02T10:11:14.848 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:14.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:14.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:14.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:11:14.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:11:14.897 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:11:14.897 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:11:14.922 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:11:14.922 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:11:14.946 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:11:14.946 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:11:14.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:14.975 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:14.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:15.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:15.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:15.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:15.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:15.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:15.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:15.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:11:15.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:15.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:15.361 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:15.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:15.484 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:15.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:15.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 12 16 15741 15725 81.8946 82.375 0.00533721 0.0120853 2024-09-02T10:11:15.559 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:15.586 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:15.616 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:15.617 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:15.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/292800787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:15.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:15.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:15.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/292800787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:15.843 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/292800787' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:15.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:15.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:15.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:15.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:16.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:16.003 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:16.017 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:16.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:16.115 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:11:16.115 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:11:16.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:16.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:16.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:16.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:16.261 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:16.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:16.363 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:16.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:16.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:16.394 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:16.488 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:11:16.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:16.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 13 16 17440 17424 83.7626 106.188 0.0177552 0.0119241 2024-09-02T10:11:16.518 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:16.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:16.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:16.643 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:16.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:16.733 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:16 smithi162 ceph-mon[20279]: pgmap v817: 47 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 1 peering, 43 active+clean; 766 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 1188/36747 objects misplaced (3.233%); 3.0 MiB/s, 47 objects/s recovering 2024-09-02T10:11:16.733 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/576090222' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:16.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:16 smithi187 ceph-mon[20072]: pgmap v817: 47 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 1 peering, 43 active+clean; 766 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 1188/36747 objects misplaced (3.233%); 3.0 MiB/s, 47 objects/s recovering 2024-09-02T10:11:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/576090222' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:16.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:11:16.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:16.769 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:16.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:16.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:16.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:16.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:16.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:16.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:16 smithi151 ceph-mon[16639]: pgmap v817: 47 pgs: 2 active+recovering+undersized+remapped, 1 remapped+peering, 1 peering, 43 active+clean; 766 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 1188/36747 objects misplaced (3.233%); 3.0 MiB/s, 47 objects/s recovering 2024-09-02T10:11:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/576090222' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:16.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:17.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:17.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:17.065 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:17.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:17.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:17.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:17.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:17.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:17.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:17.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:17.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:17.365 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:17.387 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:17.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:17.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:17.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 14 16 17651 17635 78.7214 13.1875 0.064077 0.0120915 2024-09-02T10:11:17.511 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:17.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:17.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:17 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:17.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/576090222' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:17.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:17 smithi151 ceph-mon[16639]: osdmap e330: 12 total, 12 up, 12 in 2024-09-02T10:11:17.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:17 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:11:17.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:17 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:11:17.591 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:11:17.591 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:11:17.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:17.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:17.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:17.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:17.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:17 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/576090222' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:17 smithi187 ceph-mon[20072]: osdmap e330: 12 total, 12 up, 12 in 2024-09-02T10:11:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:17 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:11:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:17 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:11:17.764 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:17.765 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:17 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/576090222' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:17 smithi162 ceph-mon[20279]: osdmap e330: 12 total, 12 up, 12 in 2024-09-02T10:11:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:17 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:11:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:17 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:11:17.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:17.889 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:17.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:17.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:18.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:18.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:18.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:18.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:18.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:18.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:18.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:18.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:18.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:18.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:11:18.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:18.393 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:18.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:18.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:18.502 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 15 16 18671 18655 77.723 63.75 0.00196012 0.0128527 2024-09-02T10:11:18.507 INFO:teuthology.orchestra.run.smithi151.stdout:moved 11 / 141 (7.80142%) 2024-09-02T10:11:18.507 INFO:teuthology.orchestra.run.smithi151.stdout:avg 11.75 2024-09-02T10:11:18.507 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 4.71036 -> 5.19816 (expected baseline 3.28189) 2024-09-02T10:11:18.507 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.4 with 3 -> 1 pgs (0.255319 -> 0.0851064 * mean) 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.7 with 18 -> 20 pgs (1.53191 -> 1.70213 * mean) 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0049 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0053 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4 weight 0.2863 -> 0.1626 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6 weight 1.0000 -> 0.7025 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1 weight 0.7452 -> 0.5699 2024-09-02T10:11:18.508 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 0.9931 -> 1.0000 2024-09-02T10:11:18.524 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:18.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:18.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:11:18.573 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:18.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:18.648 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:18.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:18.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:18.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:18 smithi187 ceph-mon[20072]: pgmap v819: 47 pgs: 1 peering, 2 active+recovering+undersized+remapped, 44 active+clean; 925 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 1188/44385 objects misplaced (2.677%); 11 MiB/s, 168 objects/s recovering 2024-09-02T10:11:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:18 smithi187 ceph-mon[20072]: osdmap e331: 12 total, 12 up, 12 in 2024-09-02T10:11:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:18 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:11:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:18 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65536','1':'37346','4':'10656','6':'46036'}"} : dispatch 2024-09-02T10:11:18.770 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:18.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:18.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:18 smithi162 ceph-mon[20279]: pgmap v819: 47 pgs: 1 peering, 2 active+recovering+undersized+remapped, 44 active+clean; 925 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 1188/44385 objects misplaced (2.677%); 11 MiB/s, 168 objects/s recovering 2024-09-02T10:11:18.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:18 smithi162 ceph-mon[20279]: osdmap e331: 12 total, 12 up, 12 in 2024-09-02T10:11:18.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:18 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:11:18.818 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:18 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65536','1':'37346','4':'10656','6':'46036'}"} : dispatch 2024-09-02T10:11:18.832 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:18 smithi151 ceph-mon[16639]: pgmap v819: 47 pgs: 1 peering, 2 active+recovering+undersized+remapped, 44 active+clean; 925 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 75 MiB/s wr, 1.20k op/s; 1188/44385 objects misplaced (2.677%); 11 MiB/s, 168 objects/s recovering 2024-09-02T10:11:18.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:18 smithi151 ceph-mon[16639]: osdmap e331: 12 total, 12 up, 12 in 2024-09-02T10:11:18.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:18 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:11:18.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:18 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'65536','1':'37346','4':'10656','6':'46036'}"} : dispatch 2024-09-02T10:11:18.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:18.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:18.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:18.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:19.018 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:19.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:19.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:19.027 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:11:19.027 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:11:19.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:19.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:19.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:19.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:19.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:11:19.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:19.265 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:19.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:19.333 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:19.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:19.389 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:19.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:19.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:19.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:19.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 16 16 18694 18678 72.955 1.4375 0.371712 0.0129279 2024-09-02T10:11:19.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:19.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:19.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:19 smithi162 ceph-mon[20279]: from='client.15969 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:11:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:19 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536','1':'37346','4':'10656','6':'46036'}"}]': finished 2024-09-02T10:11:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:19 smithi162 ceph-mon[20279]: osdmap e332: 12 total, 12 up, 12 in 2024-09-02T10:11:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:19 smithi162 ceph-mon[20279]: osdmap e333: 12 total, 12 up, 12 in 2024-09-02T10:11:19.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:19.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:19.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:19.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:11:19.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:19.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:19 smithi151 ceph-mon[16639]: from='client.15969 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:11:19.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:19 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536','1':'37346','4':'10656','6':'46036'}"}]': finished 2024-09-02T10:11:19.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:19 smithi151 ceph-mon[16639]: osdmap e332: 12 total, 12 up, 12 in 2024-09-02T10:11:19.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:19 smithi151 ceph-mon[16639]: osdmap e333: 12 total, 12 up, 12 in 2024-09-02T10:11:19.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:19.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:19.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:19.879 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:11:19.879 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:11:19.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:19.891 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:11:19.891 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:11:19.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:19.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:19.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:19.911 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:11:19.912 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:11:19.917 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:19.937 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:11:19.937 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:11:19.981 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:11:19.981 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:11:20.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:19 smithi187 ceph-mon[20072]: from='client.15969 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:11:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:19 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'65536','1':'37346','4':'10656','6':'46036'}"}]': finished 2024-09-02T10:11:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:19 smithi187 ceph-mon[20072]: osdmap e332: 12 total, 12 up, 12 in 2024-09-02T10:11:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:19 smithi187 ceph-mon[20072]: osdmap e333: 12 total, 12 up, 12 in 2024-09-02T10:11:20.024 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:11:20.024 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:11:20.046 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:20.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:20.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:20.059 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:11:20.059 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:11:20.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:20.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:20.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:20.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:20.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:20.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:20.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:20.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:20.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:20.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:20.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:20.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 17 16 19427 19411 71.3582 45.8125 0.00861347 0.0140021 2024-09-02T10:11:20.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:20.561 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:20.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:20.662 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:20.689 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:20.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:20.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:20.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:20.814 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:20.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:20 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:11:20.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:20 smithi162 ceph-mon[20279]: pgmap v823: 47 pgs: 2 active+undersized+remapped, 1 peering, 44 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 30 MiB/s, 483 objects/s recovering 2024-09-02T10:11:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/372085585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:20 smithi162 ceph-mon[20279]: osdmap e334: 12 total, 12 up, 12 in 2024-09-02T10:11:20.847 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:11:20.848 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:11:20.848 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:11:20.848 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:11:20.848 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:11:20.886 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:11:20.887 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:11:20.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:20 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:11:20.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:20 smithi151 ceph-mon[16639]: pgmap v823: 47 pgs: 2 active+undersized+remapped, 1 peering, 44 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 30 MiB/s, 483 objects/s recovering 2024-09-02T10:11:20.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/372085585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:20.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:20 smithi151 ceph-mon[16639]: osdmap e334: 12 total, 12 up, 12 in 2024-09-02T10:11:20.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:20.921 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:11:20.921 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:11:20.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:20.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:20.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:20.955 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:11:20.955 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:11:20.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:11:20.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:11:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:20 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:11:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:20 smithi187 ceph-mon[20072]: pgmap v823: 47 pgs: 2 active+undersized+remapped, 1 peering, 44 active+clean; 1.1 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 30 MiB/s, 483 objects/s recovering 2024-09-02T10:11:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/372085585' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:20 smithi187 ceph-mon[20072]: osdmap e334: 12 total, 12 up, 12 in 2024-09-02T10:11:21.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:11:21.022 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:11:21.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:21.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:21.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:21.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:21.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:21.210 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:21.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:21.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:21.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:21.334 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:21.335 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:21.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:21.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:21.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:21.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 18 16 20427 20411 70.8658 62.5 0.00645054 0.014091 2024-09-02T10:11:21.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:11:21.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:21.614 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:21.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:21.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:21.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:21.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:21.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:21.834 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:21.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:21.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:21.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:21.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:22.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:22.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:22.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:22.134 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:22.135 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:22.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:22.260 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:22.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:22.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:22.385 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:22.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:22.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:22.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:22.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 19 16 21515 21499 70.7147 68 0.0126215 0.0141323 2024-09-02T10:11:22.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:22.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:22.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:22.624 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:22.645 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:22.647 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:22.677 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:22 smithi162 ceph-mon[20279]: pgmap v825: 47 pgs: 1 remapped+peering, 2 active+undersized+remapped, 1 peering, 43 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 857 op/s; 19 MiB/s, 296 objects/s recovering 2024-09-02T10:11:22.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:22.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:22.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:22.808 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:22 smithi187 ceph-mon[20072]: pgmap v825: 47 pgs: 1 remapped+peering, 2 active+undersized+remapped, 1 peering, 43 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 857 op/s; 19 MiB/s, 296 objects/s recovering 2024-09-02T10:11:22.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:22.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:22.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:22 smithi151 ceph-mon[16639]: pgmap v825: 47 pgs: 1 remapped+peering, 2 active+undersized+remapped, 1 peering, 43 active+clean; 1.1 GiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 857 op/s; 19 MiB/s, 296 objects/s recovering 2024-09-02T10:11:23.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:23.045 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:23.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:23.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:23.159 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:11:23.159 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:11:23.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:11:23.159 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:11:23.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:8321 2.3 2:c249931d:::benchmark_data_smithi187_45882_object8320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 582.71763944700001, 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25867863899999999, 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 8321 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.160 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:11:23.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:11:23.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958035+0000", 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8100000000000003e-07 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958104+0000", 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8448000000000006e-05 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958111+0000", 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0469999999999996e-06 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958115+0000", 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9689999999999996e-06 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958155+0000", 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0398999999999997e-05 2024-09-02T10:11:23.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958225+0000", 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9578000000000004e-05 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958316+0000", 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1249000000000007e-05 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958733+0000", 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041638500000000002 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.959078+0000", 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000345855 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216656+0000", 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25757725999999997 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216689+0000", 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3368999999999998e-05 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216714+0000", 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4598999999999999e-05 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.166 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:8321 2.3 e97/92 ondisk, result = 0)", 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 582.71677959299996, 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25782361500000001, 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958895+0000", 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1600000000000001e-07 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:39.958898+0000", 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.066e-06 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216621+0000", 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25772234500000002 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216625+0000", 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2989999999999998e-06 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216650+0000", 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4939e-05 2024-09-02T10:11:23.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216653+0000", 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1559999999999999e-06 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:40.216718+0000", 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5294000000000002e-05 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.170 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:54905 2.2 2:470b2221:::benchmark_data_smithi187_45882_object54904:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.133024+0000", 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 545.54265037699997, 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26483498300000002, 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54905 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133024+0000", 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133024+0000", 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133026+0000", 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.517e-06 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133067+0000", 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1332999999999999e-05 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133081+0000", 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4272e-05 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133088+0000", 2024-09-02T10:11:23.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3600000000000001e-06 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133138+0000", 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0037999999999998e-05 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133240+0000", 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102081 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.133332+0000", 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2132999999999994e-05 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.134852+0000", 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001519963 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.135073+0000", 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022145800000000001 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.397790+0000", 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26271683099999998 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.397827+0000", 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7119000000000002e-05 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.397859+0000", 2024-09-02T10:11:23.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1878000000000002e-05 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:54937 2.3 2:c9a686f1:::benchmark_data_smithi187_45882_object54936:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 545.50425584899995, 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23343879200000001, 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.177 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 54937 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171419+0000", 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1500000000000003e-07 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171466+0000", 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6523000000000002e-05 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171477+0000", 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0784e-05 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171482+0000", 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2079999999999999e-06 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171521+0000", 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8736999999999999e-05 2024-09-02T10:11:23.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171596+0000", 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5042000000000004e-05 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.171675+0000", 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9369999999999994e-05 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.173027+0000", 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013518409999999999 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.273746+0000", 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.100718883 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.404789+0000", 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.13104327700000001 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.404822+0000", 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2811000000000001e-05 2024-09-02T10:11:23.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.404857+0000", 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5401000000000002e-05 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:55038 2.3 2:c94b451a:::benchmark_data_smithi187_45882_object55037:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:02:17.453569+0000", 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 545.22210495000002, 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33233287500000003, 2024-09-02T10:11:23.183 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55038 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453569+0000", 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453569+0000", 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453571+0000", 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.702e-06 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453616+0000", 2024-09-02T10:11:23.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5033000000000001e-05 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453635+0000", 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8595000000000001e-05 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453644+0000", 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0100000000000001e-06 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453705+0000", 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0816000000000002e-05 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453835+0000", 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013086799999999999 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.453981+0000", 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014516900000000001 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.455854+0000", 2024-09-02T10:11:23.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001873537 2024-09-02T10:11:23.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.456078+0000", 2024-09-02T10:11:23.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000224083 2024-09-02T10:11:23.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.785794+0000", 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32971594999999998 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.785853+0000", 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9083999999999999e-05 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:02:17.785902+0000", 2024-09-02T10:11:23.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9027999999999999e-05 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:97742 2.2 2:492b4420:::benchmark_data_smithi187_45882_object97741:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:05.413372+0000", 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 497.26230223900001, 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26451705399999997, 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.190 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 97742 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413372+0000", 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413372+0000", 2024-09-02T10:11:23.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413373+0000", 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0200000000000001e-07 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413450+0000", 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6959000000000006e-05 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413460+0000", 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.842e-06 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413465+0000", 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8749999999999999e-06 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413516+0000", 2024-09-02T10:11:23.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1252000000000003e-05 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413625+0000", 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010952500000000001 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413745+0000", 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011921000000000001 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.416627+0000", 2024-09-02T10:11:23.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028824359999999999 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.427846+0000", 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011219084000000001 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677815+0000", 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.249968879 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677858+0000", 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3084000000000003e-05 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677889+0000", 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1106000000000003e-05 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:141512 2.3 2:c7378ca5:::benchmark_data_smithi187_45882_object43761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:14.693802+0000", 2024-09-02T10:11:23.196 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 427.98187201399998, 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37122683899999998, 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141512 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693802+0000", 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693802+0000", 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693803+0000", 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2500000000000005e-07 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693807+0000", 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.681e-06 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693815+0000", 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9140000000000008e-06 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693821+0000", 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3090000000000001e-06 2024-09-02T10:11:23.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693879+0000", 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7862000000000003e-05 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.693996+0000", 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116896 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694103+0000", 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010771400000000001 2024-09-02T10:11:23.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.698406+0000", 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043025820000000001 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.703236+0000", 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048299390000000001 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.064923+0000", 2024-09-02T10:11:23.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36168711999999997 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.064970+0000", 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7067000000000002e-05 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.065029+0000", 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9030000000000002e-05 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:156215 2.2 2:47721b3f:::benchmark_data_smithi187_45882_object58464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:43.294501+0000", 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 399.38117337699998, 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27888434200000001, 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 156215 2024-09-02T10:11:23.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294501+0000", 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294501+0000", 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294503+0000", 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.525e-06 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294509+0000", 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.905e-06 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294527+0000", 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7744000000000002e-05 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294534+0000", 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0160000000000003e-06 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294588+0000", 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3582000000000003e-05 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294669+0000", 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1450000000000001e-05 2024-09-02T10:11:23.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,6", 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.294743+0000", 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3330000000000004e-05 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.298500+0000", 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037571219999999999 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.298571+0000", 2024-09-02T10:11:23.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1128000000000001e-05 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573304+0000", 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27473348800000003 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573343+0000", 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8288999999999999e-05 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573385+0000", 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2762999999999997e-05 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:156215 2.2 e188/92 ondisk, result = 0)", 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:43.299839+0000", 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 399.375834989, 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27355232099999999, 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.209 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.299839+0000", 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.299839+0000", 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.299840+0000", 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3799999999999997e-07 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.299845+0000", 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8999999999999997e-06 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573233+0000", 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27338810800000002 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573241+0000", 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7950000000000008e-06 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573293+0000", 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2540999999999997e-05 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573301+0000", 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3900000000000004e-06 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.573392+0000", 2024-09-02T10:11:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1049000000000002e-05 2024-09-02T10:11:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.213 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:195489 2.3 2:c2cfeaf0:::benchmark_data_smithi187_45882_object97738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-02T10:11:23.216 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:43.024659+0000", 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 339.65101583400002, 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.298585832, 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 195489 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024659+0000", 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024659+0000", 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024660+0000", 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000001e-07 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024664+0000", 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5120000000000002e-06 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024676+0000", 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2391e-05 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.024683+0000", 2024-09-02T10:11:23.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2480000000000001e-06 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.028947+0000", 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042639560000000002 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029054+0000", 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107088 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:11:23.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029120+0000", 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5988999999999994e-05 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.033127+0000", 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040077639999999996 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.038683+0000", 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055557139999999998 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.323168+0000", 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.284485031 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.323211+0000", 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3260999999999998e-05 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.323244+0000", 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2928000000000001e-05 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.222 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:12911 3.5 3:a2419a0b:::benchmark_data_smithi187_59989_object12910:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:06:11.043329+0000", 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 311.63234581799998, 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22769582599999999, 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:23.223 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 12911 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043329+0000", 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043329+0000", 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043330+0000", 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0380000000000001e-06 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043373+0000", 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3662000000000003e-05 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043384+0000", 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0804e-05 2024-09-02T10:11:23.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043391+0000", 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8630000000000002e-06 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043432+0000", 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0735999999999998e-05 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043511+0000", 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9149999999999999e-05 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.043594+0000", 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2744000000000006e-05 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.044220+0000", 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00062675399999999998 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.176189+0000", 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.131968694 2024-09-02T10:11:23.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.270931+0000", 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094741994999999996 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.270981+0000", 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0244000000000002e-05 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:06:11.271024+0000", 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3142000000000001e-05 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:75407 3.5 3:a97a52c4:::benchmark_data_smithi187_59989_object75406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.346108+0000", 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 262.32956610700001, 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.258318889, 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.229 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75407 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.346108+0000", 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.346108+0000", 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.346109+0000", 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1850000000000001e-06 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.346144+0000", 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4983e-05 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.346158+0000", 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3903000000000001e-05 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.346166+0000", 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6799999999999993e-06 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.346212+0000", 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5658000000000002e-05 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.346306+0000", 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3819000000000004e-05 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.346393+0000", 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7665999999999997e-05 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.348546+0000", 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021531359999999999 2024-09-02T10:11:23.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.349246+0000", 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00069975799999999998 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.604359+0000", 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25511324099999999 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.604394+0000", 2024-09-02T10:11:23.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5108999999999999e-05 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.604427+0000", 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2750999999999999e-05 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:75428 3.c 3:3c55ad64:::benchmark_data_smithi187_59989_object75427:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.362025+0000", 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 262.31364910299999, 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.227022746, 2024-09-02T10:11:23.235 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75428 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362025+0000", 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362025+0000", 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362027+0000", 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0540000000000002e-06 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362077+0000", 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9657000000000002e-05 2024-09-02T10:11:23.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362094+0000", 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7339000000000001e-05 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362103+0000", 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1050000000000001e-06 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362165+0000", 2024-09-02T10:11:23.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1272000000000006e-05 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362298+0000", 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013336199999999999 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362450+0000", 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015155299999999999 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.365656+0000", 2024-09-02T10:11:23.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032059609999999998 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.366569+0000", 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00091302099999999997 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.588845+0000", 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22227601899999999 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.588994+0000", 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000149149 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.589048+0000", 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4253999999999999e-05 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:75432 3.5 3:ad1a80df:::benchmark_data_smithi187_59989_object75431:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-02T10:11:23.241 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.364957+0000", 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 262.31071776099998, 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30528142200000002, 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75432 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.364957+0000", 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.364957+0000", 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.364958+0000", 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.39e-06 2024-09-02T10:11:23.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.365026+0000", 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8371e-05 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.365039+0000", 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2673e-05 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.365047+0000", 2024-09-02T10:11:23.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.537e-06 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.365101+0000", 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4407999999999998e-05 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.365215+0000", 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113922 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.365363+0000", 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014791500000000001 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.368237+0000", 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002874007 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.368257+0000", 2024-09-02T10:11:23.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0233000000000001e-05 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.670125+0000", 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30186744199999999 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.670186+0000", 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1235000000000006e-05 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.670238+0000", 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2289e-05 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:97449 3.d e260/226 3:bab3d302:::benchmark_data_smithi187_59989_object97448:head v 260'6122, mlcod=260'6119)", 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:20.619203+0000", 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 242.05647108299999, 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.269348327, 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.248 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.619203+0000", 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.619203+0000", 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.619204+0000", 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6700000000000003e-07 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.619228+0000", 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3669999999999999e-05 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.885625+0000", 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26639728800000001 2024-09-02T10:11:23.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.885632+0000", 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3069999999999997e-06 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.885689+0000", 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6953000000000002e-05 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.885713+0000", 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3615e-05 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.888506+0000", 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027926549999999998 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:20.888552+0000", 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6072000000000002e-05 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.252 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:181224 3.c 3:3deba549:::benchmark_data_smithi187_59989_object71681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:09:15.161840+0000", 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 127.513834312, 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23356724600000001, 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 181224 2024-09-02T10:11:23.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.161840+0000", 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.161840+0000", 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.161841+0000", 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2049999999999999e-06 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.161847+0000", 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2739999999999998e-06 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.161859+0000", 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2873e-05 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.161866+0000", 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0190000000000001e-06 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.161924+0000", 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7482999999999998e-05 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.162051+0000", 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012699599999999999 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.162172+0000", 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012143500000000001 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.166162+0000", 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039901099999999998 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.166211+0000", 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8378999999999999e-05 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.395222+0000", 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22901160800000001 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.395363+0000", 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000140815 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.395407+0000", 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4048999999999998e-05 2024-09-02T10:11:23.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:188910 3.5 3:a65ef79e:::benchmark_data_smithi187_59989_object79367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:09:28.166076+0000", 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 114.50959865999999, 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26426439099999999, 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:23.259 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 188910 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.166076+0000", 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.166076+0000", 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.166077+0000", 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.601e-06 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.166084+0000", 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4620000000000001e-06 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.166097+0000", 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2789000000000001e-05 2024-09-02T10:11:23.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.166104+0000", 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9759999999999998e-06 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.166155+0000", 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1359000000000001e-05 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.166300+0000", 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014520000000000001 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.166427+0000", 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126981 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.171556+0000", 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051285599999999999 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.189542+0000", 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017986089 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.430262+0000", 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24072000099999999 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.430306+0000", 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4415000000000001e-05 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:28.430340+0000", 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3958000000000003e-05 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:219075 3.c 3:3c470eda:::benchmark_data_smithi187_59989_object109532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:10:17.230619+0000", 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 65.445054933999998, 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.302553452, 2024-09-02T10:11:23.265 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 219075 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.230619+0000", 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.230619+0000", 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.230620+0000", 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5899999999999995e-07 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.230625+0000", 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3019999999999996e-06 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.230635+0000", 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0267e-05 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.230642+0000", 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8349999999999998e-06 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.230689+0000", 2024-09-02T10:11:23.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7670999999999998e-05 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.230801+0000", 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112105 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.230907+0000", 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105264 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:23.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.236713+0000", 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0058066259999999996 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.236892+0000", 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000179144 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.533106+0000", 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29621398999999998 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:23.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.533142+0000", 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5408999999999999e-05 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.533173+0000", 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1080000000000001e-05 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24976.0:219075 3.c e304/226 ondisk, result = 0)", 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:10:17.236234+0000", 2024-09-02T10:11:23.271 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 65.439440093000002, 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.296942488, 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.236234+0000", 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.236234+0000", 2024-09-02T10:11:23.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.236235+0000", 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0999999999999998e-07 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.236240+0000", 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8380000000000001e-06 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.533040+0000", 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29679992799999999 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.533047+0000", 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8900000000000001e-06 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.533096+0000", 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9023000000000001e-05 2024-09-02T10:11:23.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.533103+0000", 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.934e-06 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.533177+0000", 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4165000000000006e-05 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(3.1 e304,302 3:809ccb94:::benchmark_data_smithi187_59989_object6970:head,268'7225)", 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:10:17.690000+0000", 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 64.985674282999994, 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28270331100000001, 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.690000+0000", 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.690000+0000", 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.690002+0000", 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.39e-06 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.690011+0000", 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6299999999999993e-06 2024-09-02T10:11:23.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.972464+0000", 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28245280299999997 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.972481+0000", 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6841999999999999e-05 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.972635+0000", 2024-09-02T10:11:23.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015449300000000001 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.972642+0000", 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.08e-06 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:10:17.972703+0000", 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1073000000000003e-05 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.279 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:23.280 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:23.280 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:11:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:23.281 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:23.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:23.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:23.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:23.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:23.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:23.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:11:23.503184+0000 min lat: 0.00103182 max lat: 2.06751 avg lat: 0.014135 2024-09-02T10:11:23.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:11:23.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 20 16 22612 22596 70.6068 68.5625 0.0132683 0.014135 2024-09-02T10:11:23.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:23.536 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:23.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:23.628 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:23 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:11:23.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:23.674 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:23.675 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:23.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:23.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:23.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:23.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:23 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:11:23.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:23 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:11:23.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:23.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:23.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:24.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:24.069 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:24.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:24.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:11:24.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:24.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:24.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:24.256 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:11:24.256 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:11:24.257 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_2 2024-09-02T10:11:24.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:11:24.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:24.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:24.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:24.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:24.455 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:24.460 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:24.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:24.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 21 15 24024 24009 71.4495 88.3125 0.0147768 0.0139854 2024-09-02T10:11:24.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:24.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:24.593 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:24.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:24 smithi151 ceph-mon[16639]: pgmap v826: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+undersized+remapped, 41 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 357 B/s rd, 60 MiB/s wr, 952 op/s; 1141/60729 objects degraded (1.879%); 2702/60729 objects misplaced (4.449%); 22 MiB/s, 358 objects/s recovering 2024-09-02T10:11:24.662 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:24 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 1141/60729 objects degraded (1.879%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:24.662 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:24 smithi151 ceph-mon[16639]: osdmap e335: 12 total, 12 up, 12 in 2024-09-02T10:11:24.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:24.717 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:24.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:24.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:24.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:24 smithi162 ceph-mon[20279]: pgmap v826: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+undersized+remapped, 41 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 357 B/s rd, 60 MiB/s wr, 952 op/s; 1141/60729 objects degraded (1.879%); 2702/60729 objects misplaced (4.449%); 22 MiB/s, 358 objects/s recovering 2024-09-02T10:11:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:24 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 1141/60729 objects degraded (1.879%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:24 smithi162 ceph-mon[20279]: osdmap e335: 12 total, 12 up, 12 in 2024-09-02T10:11:24.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:24.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:24.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:24.976 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:24.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:24.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:24.984 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:11:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:11:24.984 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:11:24.984 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:11:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:24 smithi187 ceph-mon[20072]: pgmap v826: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 1 remapped+peering, 1 active+undersized+remapped, 41 active+clean; 1.2 GiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 357 B/s rd, 60 MiB/s wr, 952 op/s; 1141/60729 objects degraded (1.879%); 2702/60729 objects misplaced (4.449%); 22 MiB/s, 358 objects/s recovering 2024-09-02T10:11:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:24 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 1141/60729 objects degraded (1.879%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:24 smithi187 ceph-mon[20072]: osdmap e335: 12 total, 12 up, 12 in 2024-09-02T10:11:25.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:25.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:25.101 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:25.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:25.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:25.232 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:25.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:25.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:25.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:25.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:25.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:25.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:25.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:25.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:25.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:25.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:25.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 22 16 25025 25009 71.0425 62.5 0.00454786 0.0140662 2024-09-02T10:11:25.532 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:11:25.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:25.618 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:25.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:25.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:25.743 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:25.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:25.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:11:25.827 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:25 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:11:25.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:25 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:11:25.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3795344425' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:11:25.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:25.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:25.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:25.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:25.881 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:25 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:11:25.881 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:25 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:11:25.881 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3795344425' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:11:25.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:25.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:25.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:25 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:11:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:25 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:11:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3795344425' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:11:26.088 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:26.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:26.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:26.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:26.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:26.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:26.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:26.319 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:26.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:26.386 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:26.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:26.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:26.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 23 16 25905 25889 70.3448 55 0.00912157 0.0142094 2024-09-02T10:11:26.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:26.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:26.629 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:11:26.629 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:11:26.629 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:11:26.629 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:11:26.629 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:11:26.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:26.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:26.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:26.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:26 smithi151 ceph-mon[16639]: pgmap v828: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 324 B/s rd, 41 MiB/s wr, 661 op/s; 1141/65106 objects degraded (1.753%); 5175/65106 objects misplaced (7.949%); 18 MiB/s, 295 objects/s recovering 2024-09-02T10:11:26.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:26 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:26.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3795344425' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:11:26.745 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:26 smithi151 ceph-mon[16639]: osdmap e336: 12 total, 12 up, 12 in 2024-09-02T10:11:26.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:26.821 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:26.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:26.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:26 smithi162 ceph-mon[20279]: pgmap v828: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 324 B/s rd, 41 MiB/s wr, 661 op/s; 1141/65106 objects degraded (1.753%); 5175/65106 objects misplaced (7.949%); 18 MiB/s, 295 objects/s recovering 2024-09-02T10:11:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:26 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3795344425' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:11:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:26 smithi162 ceph-mon[20279]: osdmap e336: 12 total, 12 up, 12 in 2024-09-02T10:11:26.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:26.947 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:26.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:26.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:26 smithi187 ceph-mon[20072]: pgmap v828: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.3 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 324 B/s rd, 41 MiB/s wr, 661 op/s; 1141/65106 objects degraded (1.753%); 5175/65106 objects misplaced (7.949%); 18 MiB/s, 295 objects/s recovering 2024-09-02T10:11:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:26 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3795344425' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:11:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:26 smithi187 ceph-mon[20072]: osdmap e336: 12 total, 12 up, 12 in 2024-09-02T10:11:27.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:27.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:27.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:27.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:27.180 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:11:27.181 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:11:27.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:27.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:27.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:27.341 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:27.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:27.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:11:27.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:27.475 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:27.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:27.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 24 16 27163 27147 70.6895 78.625 0.00156815 0.0141325 2024-09-02T10:11:27.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:11:27.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:27.601 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:27.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:27.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:27.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:27.706 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:27.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:27.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:27.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:27.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:27.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:27.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:27.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:28.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:28.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:28.002 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:28.050 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":829,"stamp":"2024-09-02T10:11:25.533826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1422636682,"num_objects":21702,"num_object_clones":0,"num_object_copies":65106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1141,"num_objects_misplaced":5175,"num_objects_unfound":0,"num_objects_dirty":21702,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":21704,"num_write_kb":1388913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4026,"num_bytes_recovered":264122101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24553,"ondisk_log_size":24553,"up":141,"acting":134,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":153,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":6874568,"kb_used_data":4569268,"kb_used_omap":18,"kb_used_meta":2305069,"kb_avail":1118067256,"statfs":{"total":1151940427776,"available":1144900870144,"internally_reserved":0,"allocated":4678930432,"data_stored":4675013847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":2360390971},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,0,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":273167466,"num_objects":4168,"num_object_clones":0,"num_object_copies":12504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1141,"num_objects_misplaced":5175,"num_objects_unfound":0,"num_objects_dirty":4168,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4171,"num_write_kb":266801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1865,"num_bytes_recovered":122236661,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.313046"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":150,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-02T10:11:23.919499+0000","last_change":"2024-09-02T10:10:52.609809+0000","last_active":"2024-09-02T10:11:23.919499+0000","last_peered":"2024-09-02T10:11:23.919499+0000","last_clean":"2024-09-02T10:11:23.919499+0000","last_became_active":"2024-09-02T10:09:58.903895+0000","last_became_peered":"2024-09-02T10:09:58.903895+0000","last_unstale":"2024-09-02T10:11:23.919499+0000","last_undegraded":"2024-09-02T10:11:23.919499+0000","last_fullsized":"2024-09-02T10:11:23.919499+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:52.609761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:52.609761+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:02.709076+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":21,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.931736+0000","last_change":"2024-09-02T10:11:19.931678+0000","last_active":"2024-09-02T10:11:19.931736+0000","last_peered":"2024-09-02T10:11:19.931736+0000","last_clean":"2024-09-02T10:11:19.931736+0000","last_became_active":"2024-09-02T10:11:18.920647+0000","last_became_peered":"2024-09-02T10:11:18.920647+0000","last_unstale":"2024-09-02T10:11:19.931736+0000","last_undegraded":"2024-09-02T10:11:19.931736+0000","last_fullsized":"2024-09-02T10:11:19.931736+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:55.322310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:55.322310+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-02T10:11:24.109856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,10],"acting":[2,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":228,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-02T10:11:23.919139+0000","last_change":"2024-09-02T10:11:03.720364+0000","last_active":"2024-09-02T10:11:23.919139+0000","last_peered":"2024-09-02T10:11:23.919139+0000","last_clean":"2024-09-02T10:11:23.919139+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:11:23.919139+0000","last_undegraded":"2024-09-02T10:11:23.919139+0000","last_fullsized":"2024-09-02T10:11:23.919139+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:03.720317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:03.720317+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:16.507003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":228,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-02T10:11:23.919269+0000","last_change":"2024-09-02T10:10:54.656216+0000","last_active":"2024-09-02T10:11:23.919269+0000","last_peered":"2024-09-02T10:11:23.919269+0000","last_clean":"2024-09-02T10:11:23.919269+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:11:23.919269+0000","last_undegraded":"2024-09-02T10:11:23.919269+0000","last_fullsized":"2024-09-02T10:11:23.919269+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.656165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.656165+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:24.539265+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":142,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.942506+0000","last_change":"2024-09-02T10:11:18.920901+0000","last_active":"2024-09-02T10:11:19.942506+0000","last_peered":"2024-09-02T10:11:19.942506+0000","last_clean":"2024-09-02T10:11:19.942506+0000","last_became_active":"2024-09-02T10:11:18.920488+0000","last_became_peered":"2024-09-02T10:11:18.920488+0000","last_unstale":"2024-09-02T10:11:19.942506+0000","last_undegraded":"2024-09-02T10:11:19.942506+0000","last_fullsized":"2024-09-02T10:11:19.942506+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:07.462786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:07.462786+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:22.886310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":84,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.933302+0000","last_change":"2024-09-02T10:10:58.372721+0000","last_active":"2024-09-02T10:11:19.933302+0000","last_peered":"2024-09-02T10:11:19.933302+0000","last_clean":"2024-09-02T10:11:19.933302+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:11:19.933302+0000","last_undegraded":"2024-09-02T10:11:19.933302+0000","last_fullsized":"2024-09-02T10:11:19.933302+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:58.372641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:58.372641+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:21.412502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":577,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.943051+0000","last_change":"2024-09-02T10:10:54.892467+0000","last_active":"2024-09-02T10:11:19.943051+0000","last_peered":"2024-09-02T10:11:19.943051+0000","last_clean":"2024-09-02T10:11:19.943051+0000","last_became_active":"2024-09-02T10:06:34.859627+0000","last_became_peered":"2024-09-02T10:06:34.859627+0000","last_unstale":"2024-09-02T10:11:19.943051+0000","last_undegraded":"2024-09-02T10:11:19.943051+0000","last_fullsized":"2024-09-02T10:11:19.943051+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.892371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.892371+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:05.539128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":236,"reported_epoch":335,"state":"active+clean","last_fresh":"2024-09-02T10:11:23.919542+0000","last_change":"2024-09-02T10:10:55.684692+0000","last_active":"2024-09-02T10:11:23.919542+0000","last_peered":"2024-09-02T10:11:23.919542+0000","last_clean":"2024-09-02T10:11:23.919542+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:11:23.919542+0000","last_undegraded":"2024-09-02T10:11:23.919542+0000","last_fullsized":"2024-09-02T10:11:23.919542+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:55.684623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:55.684623+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:56.065453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":233,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.932592+0000","last_change":"2024-09-02T10:11:11.497630+0000","last_active":"2024-09-02T10:11:19.932592+0000","last_peered":"2024-09-02T10:11:19.932592+0000","last_clean":"2024-09-02T10:11:19.932592+0000","last_became_active":"2024-09-02T10:11:11.497246+0000","last_became_peered":"2024-09-02T10:11:11.497246+0000","last_unstale":"2024-09-02T10:11:19.932592+0000","last_undegraded":"2024-09-02T10:11:19.932592+0000","last_fullsized":"2024-09-02T10:11:19.932592+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:07.248083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:07.248083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:36.348505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":217,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.942166+0000","last_change":"2024-09-02T10:10:55.422138+0000","last_active":"2024-09-02T10:11:19.942166+0000","last_peered":"2024-09-02T10:11:19.942166+0000","last_clean":"2024-09-02T10:11:19.942166+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:11:19.942166+0000","last_undegraded":"2024-09-02T10:11:19.942166+0000","last_fullsized":"2024-09-02T10:11:19.942166+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:55.422074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:55.422074+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:13.928736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":473,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.939184+0000","last_change":"2024-09-02T10:10:57.026011+0000","last_active":"2024-09-02T10:11:19.939184+0000","last_peered":"2024-09-02T10:11:19.939184+0000","last_clean":"2024-09-02T10:11:19.939184+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:11:19.939184+0000","last_undegraded":"2024-09-02T10:11:19.939184+0000","last_fullsized":"2024-09-02T10:11:19.939184+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:57.025960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:57.025960+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:16.853607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":566,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.933135+0000","last_change":"2024-09-02T10:10:59.358111+0000","last_active":"2024-09-02T10:11:19.933135+0000","last_peered":"2024-09-02T10:11:19.933135+0000","last_clean":"2024-09-02T10:11:19.933135+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:11:19.933135+0000","last_undegraded":"2024-09-02T10:11:19.933135+0000","last_fullsized":"2024-09-02T10:11:19.933135+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:59.358063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:59.358063+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:29.124978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":502,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.943569+0000","last_change":"2024-09-02T10:11:11.468687+0000","last_active":"2024-09-02T10:11:19.943569+0000","last_peered":"2024-09-02T10:11:19.943569+0000","last_clean":"2024-09-02T10:11:19.943569+0000","last_became_active":"2024-09-02T10:11:11.468355+0000","last_became_peered":"2024-09-02T10:11:11.468355+0000","last_unstale":"2024-09-02T10:11:19.943569+0000","last_undegraded":"2024-09-02T10:11:19.943569+0000","last_fullsized":"2024-09-02T10:11:19.943569+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:00.896593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:00.896593+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:08.856040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":475,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.932406+0000","last_change":"2024-09-02T10:11:00.720591+0000","last_active":"2024-09-02T10:11:19.932406+0000","last_peered":"2024-09-02T10:11:19.932406+0000","last_clean":"2024-09-02T10:11:19.932406+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:11:19.932406+0000","last_undegraded":"2024-09-02T10:11:19.932406+0000","last_fullsized":"2024-09-02T10:11:19.932406+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:00.720541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:00.720541+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:09.602998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":487,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.943164+0000","last_change":"2024-09-02T10:10:00.055429+0000","last_active":"2024-09-02T10:11:19.943164+0000","last_peered":"2024-09-02T10:11:19.943164+0000","last_clean":"2024-09-02T10:11:19.943164+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:11:19.943164+0000","last_undegraded":"2024-09-02T10:11:19.943164+0000","last_fullsized":"2024-09-02T10:11:19.943164+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:00.055376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:00.055376+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:20.801266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"334'1391","reported_seq":1434,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:23.107396+0000","last_change":"2024-09-02T10:10:57.383623+0000","last_active":"2024-09-02T10:11:23.107396+0000","last_peered":"2024-09-02T10:11:23.107396+0000","last_clean":"2024-09-02T10:11:23.107396+0000","last_became_active":"2024-09-02T10:10:57.383259+0000","last_became_peered":"2024-09-02T10:10:57.383259+0000","last_unstale":"2024-09-02T10:11:23.107396+0000","last_undegraded":"2024-09-02T10:11:23.107396+0000","last_fullsized":"2024-09-02T10:11:23.107396+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1391,"log_dups_size":0,"ondisk_log_size":1391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:17.654125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91160576,"num_objects":1391,"num_object_clones":0,"num_object_copies":4173,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1391,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1391,"num_write_kb":89024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,11],"acting":[9,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":567,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.943246+0000","last_change":"2024-09-02T10:10:59.881117+0000","last_active":"2024-09-02T10:11:19.943246+0000","last_peered":"2024-09-02T10:11:19.943246+0000","last_clean":"2024-09-02T10:11:19.943246+0000","last_became_active":"2024-09-02T10:06:32.873327+0000","last_became_peered":"2024-09-02T10:06:32.873327+0000","last_unstale":"2024-09-02T10:11:19.943246+0000","last_undegraded":"2024-09-02T10:11:19.943246+0000","last_fullsized":"2024-09-02T10:11:19.943246+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:59.881053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:59.881053+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:05.354025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"334'1427","reported_seq":2939,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:23.803736+0000","last_change":"2024-09-02T10:11:19.936905+0000","last_active":"2024-09-02T10:11:23.803736+0000","last_peered":"2024-09-02T10:11:23.803736+0000","last_clean":"2024-09-02T10:11:23.803736+0000","last_became_active":"2024-09-02T10:11:19.934231+0000","last_became_peered":"2024-09-02T10:11:19.934231+0000","last_unstale":"2024-09-02T10:11:23.803736+0000","last_undegraded":"2024-09-02T10:11:23.803736+0000","last_fullsized":"2024-09-02T10:11:23.803736+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1427,"log_dups_size":0,"ondisk_log_size":1427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:21.826450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93519872,"num_objects":1427,"num_object_clones":0,"num_object_copies":4281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1427,"num_write_kb":91328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":47775744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":690,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.941357+0000","last_change":"2024-09-02T10:11:02.444385+0000","last_active":"2024-09-02T10:11:19.941357+0000","last_peered":"2024-09-02T10:11:19.941357+0000","last_clean":"2024-09-02T10:11:19.941357+0000","last_became_active":"2024-09-02T10:06:32.871027+0000","last_became_peered":"2024-09-02T10:06:32.871027+0000","last_unstale":"2024-09-02T10:11:19.941357+0000","last_undegraded":"2024-09-02T10:11:19.941357+0000","last_fullsized":"2024-09-02T10:11:19.941357+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:02.444317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:02.444317+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:26.961398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"334'1432","reported_seq":1475,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:23.249597+0000","last_change":"2024-09-02T10:10:57.383040+0000","last_active":"2024-09-02T10:11:23.249597+0000","last_peered":"2024-09-02T10:11:23.249597+0000","last_clean":"2024-09-02T10:11:23.249597+0000","last_became_active":"2024-09-02T10:10:57.382683+0000","last_became_peered":"2024-09-02T10:10:57.382683+0000","last_unstale":"2024-09-02T10:11:23.249597+0000","last_undegraded":"2024-09-02T10:11:23.249597+0000","last_fullsized":"2024-09-02T10:11:23.249597+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1432,"log_dups_size":0,"ondisk_log_size":1432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:18.987975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93847552,"num_objects":1432,"num_object_clones":0,"num_object_copies":4296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1432,"num_write_kb":91648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":331,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.941431+0000","last_change":"2024-09-02T10:11:11.472721+0000","last_active":"2024-09-02T10:11:19.941431+0000","last_peered":"2024-09-02T10:11:19.941431+0000","last_clean":"2024-09-02T10:11:19.941431+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:11:19.941431+0000","last_undegraded":"2024-09-02T10:11:19.941431+0000","last_fullsized":"2024-09-02T10:11:19.941431+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.278963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.278963+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:59.180799+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"335'1525","reported_seq":2459,"reported_epoch":335,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:24.314651+0000","last_change":"2024-09-02T10:11:19.999770+0000","last_active":"2024-09-02T10:11:24.314651+0000","last_peered":"2024-09-02T10:11:24.314651+0000","last_clean":"2024-09-02T10:11:18.483729+0000","last_became_active":"2024-09-02T10:11:19.986269+0000","last_became_peered":"2024-09-02T10:11:19.986269+0000","last_unstale":"2024-09-02T10:11:24.314651+0000","last_undegraded":"2024-09-02T10:11:24.314651+0000","last_fullsized":"2024-09-02T10:11:19.931168+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1525,"log_dups_size":0,"ondisk_log_size":1525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99942400,"num_objects":1525,"num_object_clones":0,"num_object_copies":4575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":756,"num_objects_unfound":0,"num_objects_dirty":1525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1525,"num_write_kb":97600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":29032448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10],"avail_no_missing":["4","6","10"],"object_location_counts":[{"shards":"4,6,10","objects":756},{"shards":"4,10","objects":769}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":134,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.941757+0000","last_change":"2024-09-02T10:10:55.733177+0000","last_active":"2024-09-02T10:11:19.941757+0000","last_peered":"2024-09-02T10:11:19.941757+0000","last_clean":"2024-09-02T10:11:19.941757+0000","last_became_active":"2024-09-02T10:08:25.264766+0000","last_became_peered":"2024-09-02T10:08:25.264766+0000","last_unstale":"2024-09-02T10:11:19.941757+0000","last_undegraded":"2024-09-02T10:11:19.941757+0000","last_fullsized":"2024-09-02T10:11:19.941757+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:55.733125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:55.733125+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:05.466258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"334'1355","reported_seq":1398,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:21.969517+0000","last_change":"2024-09-02T10:10:57.935507+0000","last_active":"2024-09-02T10:11:21.969517+0000","last_peered":"2024-09-02T10:11:21.969517+0000","last_clean":"2024-09-02T10:11:21.969517+0000","last_became_active":"2024-09-02T10:10:57.935152+0000","last_became_peered":"2024-09-02T10:10:57.935152+0000","last_unstale":"2024-09-02T10:11:21.969517+0000","last_undegraded":"2024-09-02T10:11:21.969517+0000","last_fullsized":"2024-09-02T10:11:21.969517+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1355,"log_dups_size":0,"ondisk_log_size":1355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:19.506090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88801280,"num_objects":1355,"num_object_clones":0,"num_object_copies":4065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1355,"num_write_kb":86720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":622,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.933099+0000","last_change":"2024-09-02T10:10:56.874596+0000","last_active":"2024-09-02T10:11:19.933099+0000","last_peered":"2024-09-02T10:11:19.933099+0000","last_clean":"2024-09-02T10:11:19.933099+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:11:19.933099+0000","last_undegraded":"2024-09-02T10:11:19.933099+0000","last_fullsized":"2024-09-02T10:11:19.933099+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:56.874512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.874512+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:17.006393+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"334'1206","reported_seq":1248,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:20.986094+0000","last_change":"2024-09-02T10:10:57.384205+0000","last_active":"2024-09-02T10:11:20.986094+0000","last_peered":"2024-09-02T10:11:20.986094+0000","last_clean":"2024-09-02T10:11:20.986094+0000","last_became_active":"2024-09-02T10:10:57.383868+0000","last_became_peered":"2024-09-02T10:10:57.383868+0000","last_unstale":"2024-09-02T10:11:20.986094+0000","last_undegraded":"2024-09-02T10:11:20.986094+0000","last_fullsized":"2024-09-02T10:11:20.986094+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1206,"log_dups_size":0,"ondisk_log_size":1206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:23.782012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79036416,"num_objects":1206,"num_object_clones":0,"num_object_copies":3618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1206,"num_write_kb":77184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":91,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.939508+0000","last_change":"2024-09-02T10:11:18.921335+0000","last_active":"2024-09-02T10:11:19.939508+0000","last_peered":"2024-09-02T10:11:19.939508+0000","last_clean":"2024-09-02T10:11:19.939508+0000","last_became_active":"2024-09-02T10:11:18.920967+0000","last_became_peered":"2024-09-02T10:11:18.920967+0000","last_unstale":"2024-09-02T10:11:19.939508+0000","last_undegraded":"2024-09-02T10:11:19.939508+0000","last_fullsized":"2024-09-02T10:11:19.939508+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:04.459478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:04.459478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:21.373605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"334'1251","reported_seq":1292,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:20.971605+0000","last_change":"2024-09-02T10:10:57.385534+0000","last_active":"2024-09-02T10:11:20.971605+0000","last_peered":"2024-09-02T10:11:20.971605+0000","last_clean":"2024-09-02T10:11:20.971605+0000","last_became_active":"2024-09-02T10:10:57.384786+0000","last_became_peered":"2024-09-02T10:10:57.384786+0000","last_unstale":"2024-09-02T10:11:20.971605+0000","last_undegraded":"2024-09-02T10:11:20.971605+0000","last_fullsized":"2024-09-02T10:11:20.971605+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1251,"log_dups_size":0,"ondisk_log_size":1251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:21.785342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81985536,"num_objects":1251,"num_object_clones":0,"num_object_copies":3753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1251,"num_write_kb":80064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":84,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.933809+0000","last_change":"2024-09-02T10:09:59.384188+0000","last_active":"2024-09-02T10:11:19.933809+0000","last_peered":"2024-09-02T10:11:19.933809+0000","last_clean":"2024-09-02T10:11:19.933809+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:11:19.933809+0000","last_undegraded":"2024-09-02T10:11:19.933809+0000","last_fullsized":"2024-09-02T10:11:19.933809+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:59.384140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:59.384140+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:24.706091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"334'1401","reported_seq":1443,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:23.126176+0000","last_change":"2024-09-02T10:10:57.935515+0000","last_active":"2024-09-02T10:11:23.126176+0000","last_peered":"2024-09-02T10:11:23.126176+0000","last_clean":"2024-09-02T10:11:23.126176+0000","last_became_active":"2024-09-02T10:10:57.935035+0000","last_became_peered":"2024-09-02T10:10:57.935035+0000","last_unstale":"2024-09-02T10:11:23.126176+0000","last_undegraded":"2024-09-02T10:11:23.126176+0000","last_fullsized":"2024-09-02T10:11:23.126176+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1401,"log_dups_size":0,"ondisk_log_size":1401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:25.601388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91815936,"num_objects":1401,"num_object_clones":0,"num_object_copies":4203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1401,"num_write_kb":89664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,8],"acting":[9,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":618,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.932204+0000","last_change":"2024-09-02T10:11:00.739069+0000","last_active":"2024-09-02T10:11:19.932204+0000","last_peered":"2024-09-02T10:11:19.932204+0000","last_clean":"2024-09-02T10:11:19.932204+0000","last_became_active":"2024-09-02T10:09:58.912625+0000","last_became_peered":"2024-09-02T10:09:58.912625+0000","last_unstale":"2024-09-02T10:11:19.932204+0000","last_undegraded":"2024-09-02T10:11:19.932204+0000","last_fullsized":"2024-09-02T10:11:19.932204+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:00.739009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:00.739009+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:01.566889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"334'1347","reported_seq":3301,"reported_epoch":334,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:22.574107+0000","last_change":"2024-09-02T10:11:19.986521+0000","last_active":"2024-09-02T10:11:22.574107+0000","last_peered":"2024-09-02T10:11:22.574107+0000","last_clean":"2024-09-02T10:11:10.452592+0000","last_became_active":"2024-09-02T10:11:19.972638+0000","last_became_peered":"2024-09-02T10:11:19.972638+0000","last_unstale":"2024-09-02T10:11:22.574107+0000","last_undegraded":"2024-09-02T10:11:22.574107+0000","last_fullsized":"2024-09-02T10:11:19.931585+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1347,"log_dups_size":0,"ondisk_log_size":1347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88276992,"num_objects":1347,"num_object_clones":0,"num_object_copies":4041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":895,"num_objects_unfound":0,"num_objects_dirty":1347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1347,"num_write_kb":86208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":952,"num_bytes_recovered":62390272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,7],"acting":[2,10],"avail_no_missing":["2","1","3","10"],"object_location_counts":[{"shards":"1,2,3,10","objects":418},{"shards":"1,2,10","objects":477},{"shards":"2,10","objects":452}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":84,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.933373+0000","last_change":"2024-09-02T10:10:59.341544+0000","last_active":"2024-09-02T10:11:19.933373+0000","last_peered":"2024-09-02T10:11:19.933373+0000","last_clean":"2024-09-02T10:11:19.933373+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:11:19.933373+0000","last_undegraded":"2024-09-02T10:11:19.933373+0000","last_fullsized":"2024-09-02T10:11:19.933373+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:59.341491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:59.341491+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:24.333902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"334'1282","reported_seq":2828,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:21.533837+0000","last_change":"2024-09-02T10:11:17.509154+0000","last_active":"2024-09-02T10:11:21.533837+0000","last_peered":"2024-09-02T10:11:21.533837+0000","last_clean":"2024-09-02T10:11:21.533837+0000","last_became_active":"2024-09-02T10:11:17.507402+0000","last_became_peered":"2024-09-02T10:11:17.507402+0000","last_unstale":"2024-09-02T10:11:21.533837+0000","last_undegraded":"2024-09-02T10:11:21.533837+0000","last_fullsized":"2024-09-02T10:11:21.533837+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:57.355073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84017152,"num_objects":1282,"num_object_clones":0,"num_object_copies":3846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1282,"num_write_kb":82048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":746,"num_bytes_recovered":48889856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":331,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.945230+0000","last_change":"2024-09-02T10:10:53.401665+0000","last_active":"2024-09-02T10:11:19.945230+0000","last_peered":"2024-09-02T10:11:19.945230+0000","last_clean":"2024-09-02T10:11:19.945230+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:11:19.945230+0000","last_undegraded":"2024-09-02T10:11:19.945230+0000","last_fullsized":"2024-09-02T10:11:19.945230+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:53.401592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:53.401592+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:02.359527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"334'1363","reported_seq":1406,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:23.086628+0000","last_change":"2024-09-02T10:10:57.934887+0000","last_active":"2024-09-02T10:11:23.086628+0000","last_peered":"2024-09-02T10:11:23.086628+0000","last_clean":"2024-09-02T10:11:23.086628+0000","last_became_active":"2024-09-02T10:10:57.934516+0000","last_became_peered":"2024-09-02T10:10:57.934516+0000","last_unstale":"2024-09-02T10:11:23.086628+0000","last_undegraded":"2024-09-02T10:11:23.086628+0000","last_fullsized":"2024-09-02T10:11:23.086628+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1363,"log_dups_size":0,"ondisk_log_size":1363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:07.849897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89325568,"num_objects":1363,"num_object_clones":0,"num_object_copies":4089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1363,"num_write_kb":87232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":561,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.933392+0000","last_change":"2024-09-02T10:11:18.921403+0000","last_active":"2024-09-02T10:11:19.933392+0000","last_peered":"2024-09-02T10:11:19.933392+0000","last_clean":"2024-09-02T10:11:19.933392+0000","last_became_active":"2024-09-02T10:11:18.920790+0000","last_became_peered":"2024-09-02T10:11:18.920790+0000","last_unstale":"2024-09-02T10:11:19.933392+0000","last_undegraded":"2024-09-02T10:11:19.933392+0000","last_fullsized":"2024-09-02T10:11:19.933392+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.330529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.330529+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:00.786830+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,10],"acting":[1,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"334'1252","reported_seq":1296,"reported_epoch":334,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:11:21.535138+0000","last_change":"2024-09-02T10:11:19.986066+0000","last_active":"2024-09-02T10:11:21.535138+0000","last_peered":"2024-09-02T10:11:21.535138+0000","last_clean":"2024-09-02T10:11:18.505664+0000","last_became_active":"2024-09-02T10:11:19.983441+0000","last_became_peered":"2024-09-02T10:11:19.983441+0000","last_unstale":"2024-09-02T10:11:21.535138+0000","last_undegraded":"2024-09-02T10:11:19.938592+0000","last_fullsized":"2024-09-02T10:11:19.933419+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1252,"log_dups_size":0,"ondisk_log_size":1252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82051072,"num_objects":1252,"num_object_clones":0,"num_object_copies":3756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1141,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1252,"num_write_kb":80128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0],"acting":[10,7],"avail_no_missing":["10","6","7"],"object_location_counts":[{"shards":"6,7,10","objects":1141},{"shards":"7,10","objects":111}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":342,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.941637+0000","last_change":"2024-09-02T10:11:11.468234+0000","last_active":"2024-09-02T10:11:19.941637+0000","last_peered":"2024-09-02T10:11:19.941637+0000","last_clean":"2024-09-02T10:11:19.941637+0000","last_became_active":"2024-09-02T10:11:11.467851+0000","last_became_peered":"2024-09-02T10:11:11.467851+0000","last_unstale":"2024-09-02T10:11:19.941637+0000","last_undegraded":"2024-09-02T10:11:19.941637+0000","last_fullsized":"2024-09-02T10:11:19.941637+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.437492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.437492+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:57.835987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"334'1396","reported_seq":2197,"reported_epoch":334,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:23.256759+0000","last_change":"2024-09-02T10:11:19.985238+0000","last_active":"2024-09-02T10:11:23.256759+0000","last_peered":"2024-09-02T10:11:23.256759+0000","last_clean":"2024-09-02T10:11:18.470398+0000","last_became_active":"2024-09-02T10:11:19.970400+0000","last_became_peered":"2024-09-02T10:11:19.970400+0000","last_unstale":"2024-09-02T10:11:23.256759+0000","last_undegraded":"2024-09-02T10:11:23.256759+0000","last_fullsized":"2024-09-02T10:11:19.932307+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1396,"log_dups_size":0,"ondisk_log_size":1396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91488256,"num_objects":1396,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":768,"num_objects_unfound":0,"num_objects_dirty":1396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1396,"num_write_kb":89344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":24772608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,7],"acting":[1,10],"avail_no_missing":["1","6","10"],"object_location_counts":[{"shards":"1,6,10","objects":768},{"shards":"1,10","objects":628}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":581,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.932096+0000","last_change":"2024-09-02T10:11:11.467461+0000","last_active":"2024-09-02T10:11:19.932096+0000","last_peered":"2024-09-02T10:11:19.932096+0000","last_clean":"2024-09-02T10:11:19.932096+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:11:19.932096+0000","last_undegraded":"2024-09-02T10:11:19.932096+0000","last_fullsized":"2024-09-02T10:11:19.932096+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:58.678413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:58.678413+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:05.764576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"334'1346","reported_seq":1981,"reported_epoch":334,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:22.523774+0000","last_change":"2024-09-02T10:11:19.990934+0000","last_active":"2024-09-02T10:11:22.523774+0000","last_peered":"2024-09-02T10:11:22.523774+0000","last_clean":"2024-09-02T10:11:18.507166+0000","last_became_active":"2024-09-02T10:11:19.976381+0000","last_became_peered":"2024-09-02T10:11:19.976381+0000","last_unstale":"2024-09-02T10:11:22.523774+0000","last_undegraded":"2024-09-02T10:11:22.523774+0000","last_fullsized":"2024-09-02T10:11:19.933237+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1346,"log_dups_size":0,"ondisk_log_size":1346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88211456,"num_objects":1346,"num_object_clones":0,"num_object_copies":4038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":875,"num_objects_unfound":0,"num_objects_dirty":1346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1346,"num_write_kb":86144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":295,"num_bytes_recovered":19333120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1","objects":471},{"shards":"0,1,4","objects":875}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":124,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.942031+0000","last_change":"2024-09-02T10:08:26.088128+0000","last_active":"2024-09-02T10:11:19.942031+0000","last_peered":"2024-09-02T10:11:19.942031+0000","last_clean":"2024-09-02T10:11:19.942031+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:11:19.942031+0000","last_undegraded":"2024-09-02T10:11:19.942031+0000","last_fullsized":"2024-09-02T10:11:19.942031+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:26.088081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:26.088081+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:22.525474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"334'1383","reported_seq":2028,"reported_epoch":334,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:22.054559+0000","last_change":"2024-09-02T10:11:19.982947+0000","last_active":"2024-09-02T10:11:22.054559+0000","last_peered":"2024-09-02T10:11:22.054559+0000","last_clean":"2024-09-02T10:11:18.511255+0000","last_became_active":"2024-09-02T10:11:19.976057+0000","last_became_peered":"2024-09-02T10:11:19.976057+0000","last_unstale":"2024-09-02T10:11:22.054559+0000","last_undegraded":"2024-09-02T10:11:22.054559+0000","last_fullsized":"2024-09-02T10:11:19.933001+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1383,"log_dups_size":0,"ondisk_log_size":1383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90636288,"num_objects":1383,"num_object_clones":0,"num_object_copies":4149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":932,"num_objects_unfound":0,"num_objects_dirty":1383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1383,"num_write_kb":88512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":299,"num_bytes_recovered":19595264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7],"avail_no_missing":["11","6","7"],"object_location_counts":[{"shards":"6,7,11","objects":932},{"shards":"7,11","objects":451}],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":355,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.942270+0000","last_change":"2024-09-02T10:11:01.465118+0000","last_active":"2024-09-02T10:11:19.942270+0000","last_peered":"2024-09-02T10:11:19.942270+0000","last_clean":"2024-09-02T10:11:19.942270+0000","last_became_active":"2024-09-02T10:11:00.404482+0000","last_became_peered":"2024-09-02T10:11:00.404482+0000","last_unstale":"2024-09-02T10:11:19.942270+0000","last_undegraded":"2024-09-02T10:11:19.942270+0000","last_fullsized":"2024-09-02T10:11:19.942270+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:01.465055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:01.465055+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:22.934648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"334'1344","reported_seq":1757,"reported_epoch":334,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:23.251927+0000","last_change":"2024-09-02T10:11:20.025398+0000","last_active":"2024-09-02T10:11:23.251927+0000","last_peered":"2024-09-02T10:11:23.251927+0000","last_clean":"2024-09-02T10:11:18.496918+0000","last_became_active":"2024-09-02T10:11:19.995839+0000","last_became_peered":"2024-09-02T10:11:19.995839+0000","last_unstale":"2024-09-02T10:11:23.251927+0000","last_undegraded":"2024-09-02T10:11:23.251927+0000","last_fullsized":"2024-09-02T10:11:19.933979+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":1344,"log_dups_size":0,"ondisk_log_size":1344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":1344,"num_object_clones":0,"num_object_copies":4032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":949,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1344,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":11993088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6],"avail_no_missing":["1","4","6"],"object_location_counts":[{"shards":"1,4,6","objects":949},{"shards":"1,6","objects":395}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":110,"reported_epoch":334,"state":"active+clean","last_fresh":"2024-09-02T10:11:19.932460+0000","last_change":"2024-09-02T10:11:18.507873+0000","last_active":"2024-09-02T10:11:19.932460+0000","last_peered":"2024-09-02T10:11:19.932460+0000","last_clean":"2024-09-02T10:11:19.932460+0000","last_became_active":"2024-09-02T10:11:11.472186+0000","last_became_peered":"2024-09-02T10:11:11.472186+0000","last_unstale":"2024-09-02T10:11:19.932460+0000","last_undegraded":"2024-09-02T10:11:19.932460+0000","last_fullsized":"2024-09-02T10:11:19.932460+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.010964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.010964+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"user requested scrub scheduled @ 2024-09-02T10:11:22.650041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163840,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":1422196736,"num_objects":21701,"num_object_clones":0,"num_object_copies":65103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1141,"num_objects_misplaced":5175,"num_objects_unfound":0,"num_objects_dirty":21701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21701,"num_write_kb":1388864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4025,"num_bytes_recovered":263782400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4469620736,"data_stored":4469620736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21701,"ondisk_log_size":21701,"up":48,"acting":41,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":439946,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":339701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383815,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":538844,"kb_used_data":259148,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93206308,"statfs":{"total":95995035648,"available":95443259392,"internally_reserved":0,"allocated":265367552,"data_stored":265042424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":286390724},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612748,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":470368,"kb_used_data":394764,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93274784,"statfs":{"total":95995035648,"available":95513378816,"internally_reserved":0,"allocated":404238336,"data_stored":403913208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":77396416},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841681,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":622688,"kb_used_data":397132,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":93122464,"statfs":{"total":95995035648,"available":95357403136,"internally_reserved":0,"allocated":406663168,"data_stored":406338040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":230947264},"hb_peers":[0,1,2,3,4,5,6,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070615,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":398172,"kb_used_data":272588,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93346980,"statfs":{"total":95995035648,"available":95587307520,"internally_reserved":0,"allocated":279130112,"data_stored":278804984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299545,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":597028,"kb_used_data":408980,"kb_used_omap":1,"kb_used_meta":188030,"kb_avail":93148124,"statfs":{"total":95995035648,"available":95383678976,"internally_reserved":0,"allocated":418795520,"data_stored":418466840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":192543178},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528479,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":707936,"kb_used_data":546252,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93037216,"statfs":{"total":95995035648,"available":95270109184,"internally_reserved":0,"allocated":559362048,"data_stored":559036920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":165542343},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724707,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348836,"kb_used_data":169108,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93396316,"statfs":{"total":95995035648,"available":95637827584,"internally_reserved":0,"allocated":173166592,"data_stored":172837912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953641,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":820204,"kb_used_data":644632,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92924948,"statfs":{"total":95995035648,"available":95155146752,"internally_reserved":0,"allocated":660103168,"data_stored":659773647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149871,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":575580,"kb_used_data":251468,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":93169572,"statfs":{"total":95995035648,"available":95405641728,"internally_reserved":0,"allocated":257503232,"data_stored":257178104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":331872712},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411506,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334564,"kb_used_data":168724,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93410588,"statfs":{"total":95995035648,"available":95652442112,"internally_reserved":0,"allocated":172773376,"data_stored":172444696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":908124,"kb_used_data":706188,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92837028,"statfs":{"total":95995035648,"available":95065116672,"internally_reserved":0,"allocated":723136512,"data_stored":722811384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738557,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":552224,"kb_used_data":350284,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93192928,"statfs":{"total":95995035648,"available":95429558272,"internally_reserved":0,"allocated":358690816,"data_stored":358365688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":383320064,"data_stored":383320064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":753401856,"data_stored":753401856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":174981120,"data_stored":174981120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":276824064,"data_stored":276824064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":288817152,"data_stored":288817152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":174587904,"data_stored":174587904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":567803904,"data_stored":567803904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":431554560,"data_stored":431554560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":288030720,"data_stored":288030720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":444989440,"data_stored":444989440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":419364864,"data_stored":419364864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":265945088,"data_stored":265945088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:11:28.053 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-02T10:11:28.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set unique_pool_2 pg_num 26 2024-09-02T10:11:28.102 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:28.115 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:28.130 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:28.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:28.225 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:11:28.225 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:1129 2.7 2:eb0d96c7:::benchmark_data_smithi187_45882_object1128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:35.893627+0000", 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 592.32989425300002, 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20029433599999999, 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:28.226 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1129 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893627+0000", 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893627+0000", 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893628+0000", 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1200000000000001e-07 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893695+0000", 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7444e-05 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893707+0000", 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2024999999999999e-05 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893713+0000", 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3650000000000003e-06 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893761+0000", 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8760000000000001e-05 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893846+0000", 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4368000000000003e-05 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:11:28.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.893930+0000", 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4179999999999997e-05 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.894256+0000", 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032631199999999999 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.908716+0000", 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014459929999999999 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.230 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093866+0000", 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18514956099999999 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093896+0000", 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0238000000000001e-05 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093921+0000", 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5241000000000001e-05 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:1142 2.7 2:e2cb33eb:::benchmark_data_smithi187_45882_object1141:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:01:35.906559+0000", 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 592.31696199800001, 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20112611999999999, 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 1142 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.232 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906559+0000", 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906559+0000", 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906560+0000", 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2780000000000001e-06 2024-09-02T10:11:28.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906583+0000", 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2288000000000001e-05 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906594+0000", 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1235e-05 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906599+0000", 2024-09-02T10:11:28.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.006e-06 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906640+0000", 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0808e-05 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906719+0000", 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9271999999999996e-05 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.906808+0000", 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9093999999999998e-05 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:35.907153+0000", 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034475 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.093837+0000", 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18668417100000001 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.107626+0000", 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013788761 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.107659+0000", 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3714000000000003e-05 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:01:36.107685+0000", 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5743000000000001e-05 2024-09-02T10:11:28.237 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.7 137/131 [PushReplyOp(2:e86168c2:::benchmark_data_smithi187_45882_object9180:head)])", 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:05.394929+0000", 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 502.82859228900003, 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.63519735200000005, 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.394929+0000", 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.394929+0000", 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.394929+0000", 2024-09-02T10:11:28.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3e-07 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.394935+0000", 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0690000000000001e-06 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:06.026536+0000", 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.631601419 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:06.026562+0000", 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5559999999999999e-05 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:06.026668+0000", 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000106269 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:06.030126+0000", 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034584049999999999 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.242 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:97742 2.2 e137/92 2:492b4420:::benchmark_data_smithi187_45882_object97741:head v 137'6271, mlcod=137'6269)", 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:05.413861+0000", 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 502.80965992099999, 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.263774603, 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413861+0000", 2024-09-02T10:11:28.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413861+0000", 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413863+0000", 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4169999999999999e-06 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.413889+0000", 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6381e-05 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.675134+0000", 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26124523599999999 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.675143+0000", 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7790000000000001e-06 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.245 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.675219+0000", 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6085000000000001e-05 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.675236+0000", 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6748999999999998e-05 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677595+0000", 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023591269999999999 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:05.677636+0000", 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0828999999999998e-05 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127503 2.7 2:e04e2dbe:::benchmark_data_smithi187_45882_object29752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.471479+0000", 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 462.752041695, 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19658355, 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.247 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127503 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471479+0000", 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471479+0000", 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471482+0000", 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.097e-06 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471490+0000", 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1419999999999993e-06 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471504+0000", 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4005999999999999e-05 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.471513+0000", 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0040000000000005e-06 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.472019+0000", 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050612499999999996 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.472135+0000", 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115798 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.250 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.472233+0000", 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8450999999999996e-05 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.475025+0000", 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027922329999999999 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.481083+0000", 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060575769999999998 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.666323+0000", 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18524028300000001 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.666409+0000", 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5414000000000006e-05 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.668063+0000", 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00165442 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.252 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127528 2.7 2:e0932104:::benchmark_data_smithi187_45882_object29777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.657520+0000", 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 462.56600084600001, 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67433600000000005, 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127528 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.253 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657520+0000", 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657520+0000", 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657522+0000", 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3400000000000001e-06 2024-09-02T10:11:28.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657527+0000", 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0599999999999998e-06 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657538+0000", 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1555e-05 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657561+0000", 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2456e-05 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.658124+0000", 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056373299999999996 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.658258+0000", 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133101 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.658396+0000", 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000138901 2024-09-02T10:11:28.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.666255+0000", 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078582219999999998 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.668130+0000", 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00187503 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.330974+0000", 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.66284469899999998 2024-09-02T10:11:28.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.331011+0000", 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6532000000000002e-05 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.331856+0000", 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00084537100000000003 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.7 156/131 [PushReplyOp(2:e2e982ef:::benchmark_data_smithi187_45882_object43210:head)])", 2024-09-02T10:11:28.258 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.912154+0000", 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 462.31136698199998, 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32342126199999999, 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912154+0000", 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912154+0000", 2024-09-02T10:11:28.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912155+0000", 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2810000000000001e-06 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912161+0000", 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.412e-06 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.233179+0000", 2024-09-02T10:11:28.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32101774300000002 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.233188+0000", 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4930000000000004e-06 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.233262+0000", 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3827999999999999e-05 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.235575+0000", 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002313505 2024-09-02T10:11:28.261 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:75424 3.a 3:58480fd1:::benchmark_data_smithi187_59989_object75423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 267.86392272199998, 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22582144000000001, 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75424 2024-09-02T10:11:28.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359600+0000", 2024-09-02T10:11:28.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3820000000000001e-06 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359629+0000", 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8775000000000001e-05 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359640+0000", 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1151e-05 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359648+0000", 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0620000000000001e-06 2024-09-02T10:11:28.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359688+0000", 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9882000000000002e-05 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359791+0000", 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010327699999999999 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359876+0000", 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5418000000000001e-05 2024-09-02T10:11:28.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.361865+0000", 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019887289999999998 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362104+0000", 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023905199999999999 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585329+0000", 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223224432 2024-09-02T10:11:28.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585374+0000", 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4972999999999998e-05 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585420+0000", 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6307000000000002e-05 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.267 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24976.0:75424 3.a e253/226 ondisk, result = 0)", 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 267.86121718200002, 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22312427800000001, 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362305+0000", 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9100000000000003e-07 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362309+0000", 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2899999999999996e-06 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585259+0000", 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22295003499999999 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585266+0000", 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6460000000000004e-06 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585311+0000", 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5802e-05 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585321+0000", 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3090000000000001e-06 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585428+0000", 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010750499999999999 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:155062 3.2 e282/226 3:4fe18cc9:::benchmark_data_smithi187_59989_object45519:head v 282'9762, mlcod=282'9758)", 2024-09-02T10:11:28.271 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 183.74701382500001, 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29158908, 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476508+0000", 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1500000000000003e-07 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476514+0000", 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6559999999999997e-06 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.763870+0000", 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28735592300000001 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.763885+0000", 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4823e-05 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.764037+0000", 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015244400000000001 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.764077+0000", 2024-09-02T10:11:28.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0371999999999999e-05 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.768047+0000", 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039701650000000003 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.768096+0000", 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8881999999999997e-05 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.275 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:181206 3.3 e291/226 3:c5ec1ffd:::benchmark_data_smithi187_59989_object71663:head v 291'11456, mlcod=291'11454)", 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 133.217318138, 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37650431099999998, 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006204+0000", 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6000000000000002e-07 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006211+0000", 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7880000000000001e-06 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377595+0000", 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.371383927 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377601+0000", 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0769999999999996e-06 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377649+0000", 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8368000000000002e-05 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:11:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377669+0000", 2024-09-02T10:11:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0258000000000002e-05 2024-09-02T10:11:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.382663+0000", 2024-09-02T10:11:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049941880000000001 2024-09-02T10:11:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.382707+0000", 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3844999999999998e-05 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3855 4.a 4:57b917f5:::benchmark_data_smithi187_76542_object3854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.314204411999999, 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22371366500000001, 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.281 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3855 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909327+0000", 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9559999999999992e-06 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909360+0000", 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3556999999999997e-05 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909377+0000", 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6283e-05 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909385+0000", 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4030000000000007e-06 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909444+0000", 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9203000000000002e-05 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909571+0000", 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127324 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909715+0000", 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143888 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.910145+0000", 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042936599999999998 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.917291+0000", 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071467290000000001 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132953+0000", 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.215661517 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132989+0000", 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6123e-05 2024-09-02T10:11:28.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.133030+0000", 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1316000000000002e-05 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3871 4.a 4:530be978:::benchmark_data_smithi187_76542_object3870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.307428604999998, 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21808112499999999, 2024-09-02T10:11:28.287 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3871 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1899999999999998e-07 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916113+0000", 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9800999999999999e-05 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916118+0000", 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916121+0000", 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3330000000000001e-06 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916168+0000", 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7147000000000001e-05 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916291+0000", 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122934 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916427+0000", 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135337 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916951+0000", 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052393700000000004 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.291 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924167+0000", 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072167580000000002 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134113+0000", 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20994539000000001 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134147+0000", 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4149000000000001e-05 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134174+0000", 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6798999999999999e-05 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3887 4.a 4:5466a52d:::benchmark_data_smithi187_76542_object3886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.301770314999999, 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21253008300000001, 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.293 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3887 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921752+0000", 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4160000000000001e-06 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921813+0000", 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0420000000000001e-05 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921822+0000", 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4299999999999995e-06 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921827+0000", 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9819999999999999e-06 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921867+0000", 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0071000000000003e-05 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921944+0000", 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6533999999999996e-05 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922025+0000", 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1601999999999996e-05 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922360+0000", 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033462200000000002 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924931+0000", 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025713160000000001 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134215+0000", 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209284054 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134247+0000", 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2147999999999998e-05 2024-09-02T10:11:28.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134281+0000", 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3488000000000002e-05 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3898 4.a 4:55cc6e50:::benchmark_data_smithi187_76542_object3897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.296774382999999, 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20761331199999999, 2024-09-02T10:11:28.299 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3898 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926748+0000", 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6599999999999994e-07 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926786+0000", 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8506e-05 2024-09-02T10:11:28.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926798+0000", 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.221e-05 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926804+0000", 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3140000000000003e-06 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926853+0000", 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8804999999999998e-05 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926965+0000", 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112256 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927110+0000", 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014520099999999999 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927488+0000", 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000378332 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929483+0000", 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019947670000000002 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134309+0000", 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20482557300000001 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.304 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134337+0000", 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8334999999999999e-05 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134360+0000", 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3047000000000001e-05 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3901 4.a 4:53342710:::benchmark_data_smithi187_76542_object3900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:11:28.305 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.294946290999999, 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20590491499999999, 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3901 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.306 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928576+0000", 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1699999999999997e-07 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.307 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928631+0000", 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5276999999999999e-05 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928642+0000", 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1195999999999999e-05 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928649+0000", 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.669e-06 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.308 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928691+0000", 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1868e-05 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928778+0000", 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7416999999999993e-05 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928866+0000", 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8471e-05 2024-09-02T10:11:28.309 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929233+0000", 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036694499999999998 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.942995+0000", 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013761572999999999 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134423+0000", 2024-09-02T10:11:28.310 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19142793799999999 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134461+0000", 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7839000000000001e-05 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134480+0000", 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9004999999999998e-05 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.311 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:6012 4.5 4:a5fa00f8:::benchmark_data_smithi187_76542_object6011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.960513289000001, 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209785581, 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6012 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.312 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263009+0000", 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0139999999999999e-06 2024-09-02T10:11:28.313 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263120+0000", 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011099000000000001 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263134+0000", 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4467999999999999e-05 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263141+0000", 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9820000000000002e-06 2024-09-02T10:11:28.314 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263387+0000", 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024598300000000002 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263500+0000", 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113174 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:11:28.315 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263599+0000", 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8974999999999993e-05 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264127+0000", 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052785799999999997 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264358+0000", 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023054600000000001 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:28.316 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472691+0000", 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20833321599999999 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472750+0000", 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8897000000000002e-05 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472793+0000", 2024-09-02T10:11:28.317 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3477999999999999e-05 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:24517 4.c e335/333 4:3b00f749:::benchmark_data_smithi187_76542_object24516:head v 335'1520, mlcod=335'1515)", 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:24.995923+0000", 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.2275979339999998, 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20209185700000001, 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.318 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:24.995923+0000", 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:24.995923+0000", 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:24.995924+0000", 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4199999999999995e-07 2024-09-02T10:11:28.319 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.320 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:24.995987+0000", 2024-09-02T10:11:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2753e-05 2024-09-02T10:11:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.197029+0000", 2024-09-02T10:11:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20104265499999999 2024-09-02T10:11:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.321 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.197036+0000", 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6290000000000004e-06 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.197124+0000", 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8319000000000006e-05 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.197141+0000", 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6965000000000001e-05 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.322 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.197980+0000", 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00083849399999999996 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.198015+0000", 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5299999999999997e-05 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.323 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:24533 4.c e335/333 4:3a0221c2:::benchmark_data_smithi187_76542_object24532:head v 335'1521, mlcod=335'1517)", 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 3.2124802369999998, 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212837002, 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.324 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011042+0000", 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1650000000000001e-06 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011129+0000", 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7443999999999998e-05 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.325 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222953+0000", 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21182389400000001 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222960+0000", 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8569999999999997e-06 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223010+0000", 2024-09-02T10:11:28.326 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0182000000000002e-05 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223032+0000", 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1699999999999999e-05 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223838+0000", 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080572200000000004 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223878+0000", 2024-09-02T10:11:28.327 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0037999999999999e-05 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:25165 4.c e336/333 4:3c3f6316:::benchmark_data_smithi187_76542_object25164:head v 336'1563, mlcod=335'1560)", 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.5782506870000002, 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204647308, 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.328 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645271+0000", 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9999999999999996e-07 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.329 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645443+0000", 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017179700000000001 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848893+0000", 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203449884 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848900+0000", 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8360000000000003e-06 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.330 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848961+0000", 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1163999999999999e-05 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848982+0000", 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1403000000000002e-05 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849888+0000", 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000905469 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:11:28.331 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849918+0000", 2024-09-02T10:11:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9855000000000001e-05 2024-09-02T10:11:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:11:28.332 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:11:28.332 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:11:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:28.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:28.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:28.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:28.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:28.456 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:28.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 25 16 28464 28448 71.1142 81.3125 0.0441949 0.0140506 2024-09-02T10:11:28.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:28.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:28.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:28.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:28.715 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:11:28.715 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:11:28.721 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:28.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:28.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:28.725 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:28.741 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:11:28.741 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:11:28.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:11:28.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:11:28.768 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:28.792 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:11:28.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:11:28.815 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:11:28.815 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:11:28.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:28 smithi151 ceph-mon[16639]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:11:28.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:28 smithi151 ceph-mon[16639]: pgmap v830: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.5 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 280 B/s rd, 51 MiB/s wr, 815 op/s; 1141/71301 objects degraded (1.600%); 4335/71301 objects misplaced (6.080%); 23 MiB/s, 370 objects/s recovering 2024-09-02T10:11:28.816 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:28 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:11:28.816 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:28 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:11:28.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:28.837 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:11:28.838 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:11:28.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:28 smithi162 ceph-mon[20279]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:11:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:28 smithi162 ceph-mon[20279]: pgmap v830: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.5 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 280 B/s rd, 51 MiB/s wr, 815 op/s; 1141/71301 objects degraded (1.600%); 4335/71301 objects misplaced (6.080%); 23 MiB/s, 370 objects/s recovering 2024-09-02T10:11:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:28 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:11:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:28 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:11:28.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:28.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:28.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:28.860 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:11:28.860 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:11:28.883 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:11:28.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:11:28.905 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:11:28.905 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:11:28.934 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:11:28.935 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:11:28.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:28.966 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:11:28.967 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:11:28.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:28.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:28.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:28.996 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:11:28.997 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:11:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:28 smithi187 ceph-mon[20072]: from='client.25604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:11:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:28 smithi187 ceph-mon[20072]: pgmap v830: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.5 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 280 B/s rd, 51 MiB/s wr, 815 op/s; 1141/71301 objects degraded (1.600%); 4335/71301 objects misplaced (6.080%); 23 MiB/s, 370 objects/s recovering 2024-09-02T10:11:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:28 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:11:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:28 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:11:29.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:29.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:29.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:29.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:29.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:29.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:29.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:29.325 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:29.346 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:29.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:29.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:29.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:29.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:29.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:29.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:29.503 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 26 16 28847 28831 69.2996 23.9375 0.0249672 0.0141899 2024-09-02T10:11:29.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:29.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:29.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:29.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:29 smithi162 ceph-mon[20279]: osdmap e337: 12 total, 12 up, 12 in 2024-09-02T10:11:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1402351855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:29 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1141/71301 objects degraded (1.600%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:29 smithi162 ceph-mon[20279]: osdmap e338: 12 total, 12 up, 12 in 2024-09-02T10:11:29.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:29.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:29.716 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:29 smithi151 ceph-mon[16639]: osdmap e337: 12 total, 12 up, 12 in 2024-09-02T10:11:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1402351855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:29 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1141/71301 objects degraded (1.600%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:29.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:29 smithi151 ceph-mon[16639]: osdmap e338: 12 total, 12 up, 12 in 2024-09-02T10:11:29.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:29.842 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:29.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:29.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:29.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:29.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:29.978 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:29 smithi187 ceph-mon[20072]: osdmap e337: 12 total, 12 up, 12 in 2024-09-02T10:11:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1402351855' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:29 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1141/71301 objects degraded (1.600%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:29 smithi187 ceph-mon[20072]: osdmap e338: 12 total, 12 up, 12 in 2024-09-02T10:11:30.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:30.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:30.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:30.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:11:30.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:30.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:30.242 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:30.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:30.370 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:30.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:30.434 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:30.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:30.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:30.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:30.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:30.504 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 27 16 29848 29832 69.0499 62.5625 0.00730363 0.0144722 2024-09-02T10:11:30.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:30 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:11:30.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:30.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:30.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:30.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:30.754 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:30.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:30.833 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:30 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:11:30.833 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:30 smithi162 ceph-mon[20279]: pgmap v833: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 1141/80907 objects degraded (1.410%); 1707/80907 objects misplaced (2.110%); 46 MiB/s, 737 objects/s recovering 2024-09-02T10:11:30.833 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:30 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:11:30.834 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:30 smithi162 ceph-mon[20279]: osdmap e339: 12 total, 12 up, 12 in 2024-09-02T10:11:30.834 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/562685289' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:11:30.834 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:30 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:11:30.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:30.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:30.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:30 smithi151 ceph-mon[16639]: pgmap v833: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 1141/80907 objects degraded (1.410%); 1707/80907 objects misplaced (2.110%); 46 MiB/s, 737 objects/s recovering 2024-09-02T10:11:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:30 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:11:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:30 smithi151 ceph-mon[16639]: osdmap e339: 12 total, 12 up, 12 in 2024-09-02T10:11:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/562685289' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:11:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:30 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:11:30.938 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 4 pg_num to 26 2024-09-02T10:11:30.946 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:11:30.946 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:11:30.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:31.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:30 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:11:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:30 smithi187 ceph-mon[20072]: pgmap v833: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.6 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s; 1141/80907 objects degraded (1.410%); 1707/80907 objects misplaced (2.110%); 46 MiB/s, 737 objects/s recovering 2024-09-02T10:11:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:30 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:11:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:30 smithi187 ceph-mon[20072]: osdmap e339: 12 total, 12 up, 12 in 2024-09-02T10:11:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/562685289' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:11:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:30 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:11:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:31.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:31.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:31.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:31.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:31.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:31.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:31.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:31.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:31.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:31.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:31.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:31.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:31.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:31.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:31.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:31.504 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 28 16 30249 30233 67.4788 25.0625 0.158057 0.0146132 2024-09-02T10:11:31.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:31.624 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:31.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:31.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:11:31.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:31.747 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:31.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:31.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:11:31.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:31 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:11:31.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:31 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:11:31.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:31 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:11:31.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:31 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:11:31.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:31 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-02T10:11:31.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:31 smithi151 ceph-mon[16639]: osdmap e340: 12 total, 12 up, 12 in 2024-09-02T10:11:31.816 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:11:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:31 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:11:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:31 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:11:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:31 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:11:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:31 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:11:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:31 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-02T10:11:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:31 smithi162 ceph-mon[20279]: osdmap e340: 12 total, 12 up, 12 in 2024-09-02T10:11:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:11:31.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:31.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:31.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:31.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:31.883 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:31.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:31 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:11:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:31 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:11:32.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:31 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:11:32.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:31 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:11:32.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:31 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num", "val": "26"}]': finished 2024-09-02T10:11:32.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:31 smithi187 ceph-mon[20072]: osdmap e340: 12 total, 12 up, 12 in 2024-09-02T10:11:32.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:11:32.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:32.027 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:32.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:32.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:32.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:32.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:32.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:32.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:32.275 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:32.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:32.351 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:11:32.351 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:11:32.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:32.400 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:32.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:32.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:32.504 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 29 16 30305 30289 65.2726 3.5 0.0210255 0.0148151 2024-09-02T10:11:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:32.524 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:32.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:32 smithi162 ceph-mon[20279]: pgmap v836: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 1141/83619 objects degraded (1.365%); 1193/83619 objects misplaced (1.427%); 49 MiB/s, 785 objects/s recovering 2024-09-02T10:11:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:32 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-02T10:11:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:32 smithi162 ceph-mon[20279]: osdmap e341: 12 total, 12 up, 12 in 2024-09-02T10:11:32.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:32.647 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:32.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:32.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:32.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:32.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:32.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:32.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:32.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:32.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:32 smithi151 ceph-mon[16639]: pgmap v836: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 1141/83619 objects degraded (1.365%); 1193/83619 objects misplaced (1.427%); 49 MiB/s, 785 objects/s recovering 2024-09-02T10:11:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:32 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-02T10:11:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:32 smithi151 ceph-mon[16639]: osdmap e341: 12 total, 12 up, 12 in 2024-09-02T10:11:33.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:32 smithi187 ceph-mon[20072]: pgmap v836: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 40 active+clean; 1.7 GiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.03k op/s; 1141/83619 objects degraded (1.365%); 1193/83619 objects misplaced (1.427%); 49 MiB/s, 785 objects/s recovering 2024-09-02T10:11:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:32 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-02T10:11:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:32 smithi187 ceph-mon[20072]: osdmap e341: 12 total, 12 up, 12 in 2024-09-02T10:11:33.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:33.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:33.036 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:33.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:11:33.137 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:33.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:33.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:33.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:33.282 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:33.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:33.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:33.388 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:11:33.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:33.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:33.504 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 30 16 30597 30581 63.705 18.25 0.198815 0.0154935 2024-09-02T10:11:33.509 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:33.531 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:33.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:33.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:33.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:33.654 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:33.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:33.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:33.780 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:33.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:33.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:33.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:33.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:33.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:34.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:34.015 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:11:34.015 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: osdmap e342: 12 total, 12 up, 12 in 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: 4.4 scrub starts 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: pgmap v839: 57 pgs: 3 peering, 8 unknown, 4 active+recovering+undersized+remapped, 42 active+clean; 1.8 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 681 op/s; 1309/89100 objects misplaced (1.469%); 24 MiB/s, 384 objects/s recovering 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: 4.4 scrub ok 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: 4.2 scrub starts 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: 4.2 scrub ok 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1655429268' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1141/83619 objects degraded (1.365%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-02T10:11:34.025 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1655429268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:34.026 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:33 smithi162 ceph-mon[20279]: osdmap e343: 12 total, 12 up, 12 in 2024-09-02T10:11:34.028 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:34.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:34.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:34.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:34.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:34.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:34.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: osdmap e342: 12 total, 12 up, 12 in 2024-09-02T10:11:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: 4.4 scrub starts 2024-09-02T10:11:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: pgmap v839: 57 pgs: 3 peering, 8 unknown, 4 active+recovering+undersized+remapped, 42 active+clean; 1.8 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 681 op/s; 1309/89100 objects misplaced (1.469%); 24 MiB/s, 384 objects/s recovering 2024-09-02T10:11:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:11:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: 4.4 scrub ok 2024-09-02T10:11:34.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: 4.2 scrub starts 2024-09-02T10:11:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: 4.2 scrub ok 2024-09-02T10:11:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1655429268' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1141/83619 objects degraded (1.365%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:11:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-02T10:11:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1655429268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:34.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:33 smithi151 ceph-mon[16639]: osdmap e343: 12 total, 12 up, 12 in 2024-09-02T10:11:34.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: osdmap e342: 12 total, 12 up, 12 in 2024-09-02T10:11:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: 4.4 scrub starts 2024-09-02T10:11:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: pgmap v839: 57 pgs: 3 peering, 8 unknown, 4 active+recovering+undersized+remapped, 42 active+clean; 1.8 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 681 op/s; 1309/89100 objects misplaced (1.469%); 24 MiB/s, 384 objects/s recovering 2024-09-02T10:11:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:11:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: 4.4 scrub ok 2024-09-02T10:11:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: 4.2 scrub starts 2024-09-02T10:11:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: 4.2 scrub ok 2024-09-02T10:11:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1655429268' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1141/83619 objects degraded (1.365%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:11:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-02T10:11:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1655429268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:33 smithi187 ceph-mon[20072]: osdmap e343: 12 total, 12 up, 12 in 2024-09-02T10:11:34.275 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:34.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:34.351 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:11:34.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:34.399 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:34.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:34.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:34.504 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 31 16 30666 30650 61.7891 4.3125 0.00151106 0.0155167 2024-09-02T10:11:34.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:34.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:34.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:34.672 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:34.673 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:34.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:34.796 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:34.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:34.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:34.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:34.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:34.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:11:35.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:35.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:35.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:35.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:35.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:35.167 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:35.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:35.197 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: 4.9 scrub starts 2024-09-02T10:11:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: 4.9 scrub ok 2024-09-02T10:11:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: 4.0 scrub starts 2024-09-02T10:11:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: 4.7 scrub starts 2024-09-02T10:11:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: 4.1 scrub starts 2024-09-02T10:11:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1141/83619 objects degraded (1.365%), 1 pg degraded) 2024-09-02T10:11:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:11:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: 4.6 scrub starts 2024-09-02T10:11:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: 4.6 scrub ok 2024-09-02T10:11:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: 4.7 scrub ok 2024-09-02T10:11:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:34 smithi187 ceph-mon[20072]: osdmap e344: 12 total, 12 up, 12 in 2024-09-02T10:11:35.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:35.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:35.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:35.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:35.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: 4.9 scrub starts 2024-09-02T10:11:35.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: 4.9 scrub ok 2024-09-02T10:11:35.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: 4.0 scrub starts 2024-09-02T10:11:35.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: 4.7 scrub starts 2024-09-02T10:11:35.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: 4.1 scrub starts 2024-09-02T10:11:35.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1141/83619 objects degraded (1.365%), 1 pg degraded) 2024-09-02T10:11:35.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:11:35.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: 4.6 scrub starts 2024-09-02T10:11:35.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: 4.6 scrub ok 2024-09-02T10:11:35.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: 4.7 scrub ok 2024-09-02T10:11:35.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:34 smithi151 ceph-mon[16639]: osdmap e344: 12 total, 12 up, 12 in 2024-09-02T10:11:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: 4.9 scrub starts 2024-09-02T10:11:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: 4.9 scrub ok 2024-09-02T10:11:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: 4.0 scrub starts 2024-09-02T10:11:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: 4.7 scrub starts 2024-09-02T10:11:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: 4.1 scrub starts 2024-09-02T10:11:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1141/83619 objects degraded (1.365%), 1 pg degraded) 2024-09-02T10:11:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:11:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: 4.6 scrub starts 2024-09-02T10:11:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: 4.6 scrub ok 2024-09-02T10:11:35.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: 4.7 scrub ok 2024-09-02T10:11:35.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:34 smithi162 ceph-mon[20279]: osdmap e344: 12 total, 12 up, 12 in 2024-09-02T10:11:35.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:35.417 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:35.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:35.504 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 32 16 30666 30650 59.8581 0 - 0.0155167 2024-09-02T10:11:35.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:35.542 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:35.543 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:35.644 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:35.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:35.669 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:35.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:35.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:35.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:35.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:35.916 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:35.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:35.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:36.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:36.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:36.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:36.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:36.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:36.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:36.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:35 smithi151 ceph-mon[16639]: 4.0 scrub ok 2024-09-02T10:11:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:35 smithi151 ceph-mon[16639]: 4.1 scrub ok 2024-09-02T10:11:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:35 smithi151 ceph-mon[16639]: 4.3 scrub starts 2024-09-02T10:11:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:35 smithi151 ceph-mon[16639]: 4.8 scrub starts 2024-09-02T10:11:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:35 smithi151 ceph-mon[16639]: 4.3 scrub ok 2024-09-02T10:11:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:35 smithi151 ceph-mon[16639]: 4.8 scrub ok 2024-09-02T10:11:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:35 smithi151 ceph-mon[16639]: pgmap v842: 57 pgs: 3 peering, 7 unknown, 1 active+recovering+undersized+remapped, 46 active+clean; 1.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 636 op/s; 599/91269 objects misplaced (0.656%); 27 MiB/s, 433 objects/s recovering 2024-09-02T10:11:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:35 smithi151 ceph-mon[16639]: osdmap e345: 12 total, 12 up, 12 in 2024-09-02T10:11:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:35 smithi187 ceph-mon[20072]: 4.0 scrub ok 2024-09-02T10:11:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:35 smithi187 ceph-mon[20072]: 4.1 scrub ok 2024-09-02T10:11:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:35 smithi187 ceph-mon[20072]: 4.3 scrub starts 2024-09-02T10:11:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:35 smithi187 ceph-mon[20072]: 4.8 scrub starts 2024-09-02T10:11:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:35 smithi187 ceph-mon[20072]: 4.3 scrub ok 2024-09-02T10:11:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:35 smithi187 ceph-mon[20072]: 4.8 scrub ok 2024-09-02T10:11:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:35 smithi187 ceph-mon[20072]: pgmap v842: 57 pgs: 3 peering, 7 unknown, 1 active+recovering+undersized+remapped, 46 active+clean; 1.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 636 op/s; 599/91269 objects misplaced (0.656%); 27 MiB/s, 433 objects/s recovering 2024-09-02T10:11:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:35 smithi187 ceph-mon[20072]: osdmap e345: 12 total, 12 up, 12 in 2024-09-02T10:11:36.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:36.300 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:36.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:36.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:35 smithi162 ceph-mon[20279]: 4.0 scrub ok 2024-09-02T10:11:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:35 smithi162 ceph-mon[20279]: 4.1 scrub ok 2024-09-02T10:11:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:35 smithi162 ceph-mon[20279]: 4.3 scrub starts 2024-09-02T10:11:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:35 smithi162 ceph-mon[20279]: 4.8 scrub starts 2024-09-02T10:11:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:35 smithi162 ceph-mon[20279]: 4.3 scrub ok 2024-09-02T10:11:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:35 smithi162 ceph-mon[20279]: 4.8 scrub ok 2024-09-02T10:11:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:35 smithi162 ceph-mon[20279]: pgmap v842: 57 pgs: 3 peering, 7 unknown, 1 active+recovering+undersized+remapped, 46 active+clean; 1.9 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 636 op/s; 599/91269 objects misplaced (0.656%); 27 MiB/s, 433 objects/s recovering 2024-09-02T10:11:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:35 smithi162 ceph-mon[20279]: osdmap e345: 12 total, 12 up, 12 in 2024-09-02T10:11:36.402 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:36.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:36.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:36.504 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 33 16 30895 30879 58.4779 7.15625 0.00222036 0.0169738 2024-09-02T10:11:36.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:36.548 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:36.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:36.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:36.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:36.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:36.758 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:36.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:36.802 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:36.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:36.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:36.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:36.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:37.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:37.059 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:37.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:37.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:37.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:37.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:37.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:37.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:37.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:36 smithi151 ceph-mon[16639]: 4.5 scrub starts 2024-09-02T10:11:37.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:36 smithi151 ceph-mon[16639]: 4.5 scrub ok 2024-09-02T10:11:37.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:36 smithi151 ceph-mon[16639]: 4.17 starting backfill to osd.1 from (0'0,325'746] MIN to 342'1885 2024-09-02T10:11:37.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:36 smithi151 ceph-mon[16639]: osdmap e346: 12 total, 12 up, 12 in 2024-09-02T10:11:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:36 smithi187 ceph-mon[20072]: 4.5 scrub starts 2024-09-02T10:11:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:36 smithi187 ceph-mon[20072]: 4.5 scrub ok 2024-09-02T10:11:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:36 smithi187 ceph-mon[20072]: 4.17 starting backfill to osd.1 from (0'0,325'746] MIN to 342'1885 2024-09-02T10:11:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:36 smithi187 ceph-mon[20072]: osdmap e346: 12 total, 12 up, 12 in 2024-09-02T10:11:37.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:37.324 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:37.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:37.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:37.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:36 smithi162 ceph-mon[20279]: 4.5 scrub starts 2024-09-02T10:11:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:36 smithi162 ceph-mon[20279]: 4.5 scrub ok 2024-09-02T10:11:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:36 smithi162 ceph-mon[20279]: 4.17 starting backfill to osd.1 from (0'0,325'746] MIN to 342'1885 2024-09-02T10:11:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:36 smithi162 ceph-mon[20279]: osdmap e346: 12 total, 12 up, 12 in 2024-09-02T10:11:37.395 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:11:37.396 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:11:37.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:37.433 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:11:37.433 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:11:37.448 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:37.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:37.467 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:11:37.468 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:11:37.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:11:37.500 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:11:37.504 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 34 16 31592 31576 58.0391 43.5625 0.00641848 0.0172196 2024-09-02T10:11:37.529 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:11:37.529 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:11:37.550 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:37.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:11:37.560 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:11:37.574 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:37.576 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:37.587 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:11:37.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:11:37.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:11:37.619 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:11:37.653 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:11:37.653 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:11:37.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:37.687 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:11:37.687 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:11:37.701 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:37.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:37.724 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:11:37.724 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:11:37.758 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:11:37.758 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:11:37.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:37.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:37.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:37.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:11:37.942 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:37.970 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:37.971 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:38.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:38.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:38.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:38.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:38.145 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:37 smithi187 ceph-mon[20072]: pgmap v845: 57 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 peering, 3 unknown, 1 active+remapped+backfill_toofull, 49 active+clean; 1.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2022/86088 objects degraded (2.349%); 6039/86088 objects misplaced (7.015%) 2024-09-02T10:11:38.145 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/582273660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:38.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:37 smithi151 ceph-mon[16639]: pgmap v845: 57 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 peering, 3 unknown, 1 active+remapped+backfill_toofull, 49 active+clean; 1.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2022/86088 objects degraded (2.349%); 6039/86088 objects misplaced (7.015%) 2024-09-02T10:11:38.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/582273660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:38.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:38.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:38.231 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:38.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:38.333 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:38.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:37 smithi162 ceph-mon[20279]: pgmap v845: 57 pgs: 1 activating+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 peering, 3 unknown, 1 active+remapped+backfill_toofull, 49 active+clean; 1.8 GiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2022/86088 objects degraded (2.349%); 6039/86088 objects misplaced (7.015%) 2024-09-02T10:11:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/582273660' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:38.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:38.370 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:38.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:38.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:38.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:38.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:38.504 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 35 16 32707 32691 58.3718 69.6875 0.00704649 0.0171231 2024-09-02T10:11:38.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:38.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:38.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:38.694 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:38.747 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:38.773 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:38.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:38.810 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:11:38.810 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:11:38.810 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:11:38.810 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:11:38.875 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:38.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:38.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:38.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:39.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:39.026 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:39.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:39.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:39.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:39.149 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:38 smithi187 ceph-mon[20072]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-02T10:11:39.149 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:38 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 2022/86088 objects degraded (2.349%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:39.149 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:38 smithi187 ceph-mon[20072]: osdmap e347: 12 total, 12 up, 12 in 2024-09-02T10:11:39.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:38 smithi151 ceph-mon[16639]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-02T10:11:39.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:38 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 2022/86088 objects degraded (2.349%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:39.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:38 smithi151 ceph-mon[16639]: osdmap e347: 12 total, 12 up, 12 in 2024-09-02T10:11:39.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:39.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:39.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:39.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:39.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:39.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:38 smithi162 ceph-mon[20279]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-02T10:11:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:38 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 2022/86088 objects degraded (2.349%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:38 smithi162 ceph-mon[20279]: osdmap e347: 12 total, 12 up, 12 in 2024-09-02T10:11:39.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:39.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:39.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:39.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 36 16 33639 33623 58.3683 58.25 0.0128817 0.0171191 2024-09-02T10:11:39.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:39.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:39.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:39.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:39.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:11:39.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:39.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:39.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:39.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:39.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:39.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:39.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:39.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:39.996 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:11:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:39.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:40.003 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:40.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:40.124 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:40.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:40.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:39 smithi187 ceph-mon[20072]: pgmap v847: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 1.9 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3934/94791 objects degraded (4.150%); 14534/94791 objects misplaced (15.333%) 2024-09-02T10:11:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1160235473' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:11:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:39 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:11:40.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:40.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:40.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:39 smithi162 ceph-mon[20279]: pgmap v847: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 1.9 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3934/94791 objects degraded (4.150%); 14534/94791 objects misplaced (15.333%) 2024-09-02T10:11:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1160235473' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:11:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:39 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:11:40.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:39 smithi151 ceph-mon[16639]: pgmap v847: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 1.9 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3934/94791 objects degraded (4.150%); 14534/94791 objects misplaced (15.333%) 2024-09-02T10:11:40.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1160235473' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:11:40.348 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:39 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:11:40.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:40.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:40.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:40.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:40.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:40.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 37 16 34984 34968 59.0625 84.0625 0.00310374 0.0169176 2024-09-02T10:11:40.533 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:40.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:40.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:40.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:11:40.625 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:11:40.625 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:11:40.625 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:11:40.625 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:11:40.625 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:11:40.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:40.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:40.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:40.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:40.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:40.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:40.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:40.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:40.940 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:40.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:41.033 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:41.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:41.065 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:41.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:41.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:41.195 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:41.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:41.196 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:40 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:40 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:11:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:40 smithi187 ceph-mon[20072]: osdmap e348: 12 total, 12 up, 12 in 2024-09-02T10:11:41.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:41.324 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:41.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:41.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:40 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:41.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:40 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:11:41.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:40 smithi151 ceph-mon[16639]: osdmap e348: 12 total, 12 up, 12 in 2024-09-02T10:11:41.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:40 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:41.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:40 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:11:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:40 smithi162 ceph-mon[20279]: osdmap e348: 12 total, 12 up, 12 in 2024-09-02T10:11:41.401 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:11:41.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:41.454 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:41.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:41.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:41.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 38 16 36353 36337 59.7597 85.5625 0.0147873 0.0167267 2024-09-02T10:11:41.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:41.581 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:41.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:41.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:41.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:41.693 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:11:41.694 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:11:41.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:41.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:41.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:11:41.732 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:11:41.748 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:41.760 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:11:41.760 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:11:41.760 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-02T10:11:41.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:11:41.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:11:41.775 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:11:41.810 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:11:41.810 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:11:41.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:41.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:11:41.842 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:11:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:41.848 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:41.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:41.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:11:41.876 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:11:41.909 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:11:41.909 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:11:41.943 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:11:41.943 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:11:41.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:41.974 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:41.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:41.981 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:11:41.982 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:11:42.019 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:11:42.020 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:11:42.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:42.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:11:42.077 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:11:42.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:11:42.114 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:11:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:42.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:42.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:42.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:41 smithi187 ceph-mon[20072]: pgmap v849: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1912/97134 objects degraded (1.968%); 15806/97134 objects misplaced (16.272%) 2024-09-02T10:11:42.181 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/286600570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:42.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:41 smithi151 ceph-mon[16639]: pgmap v849: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1912/97134 objects degraded (1.968%); 15806/97134 objects misplaced (16.272%) 2024-09-02T10:11:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/286600570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:42.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:42.250 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:42.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:42.303 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:11:42.303 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:11:42.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:41 smithi162 ceph-mon[20279]: pgmap v849: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.0 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1912/97134 objects degraded (1.968%); 15806/97134 objects misplaced (16.272%) 2024-09-02T10:11:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/286600570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:42.350 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:42.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:42.378 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:42.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:42.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:42.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:42.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 39 16 37496 37480 60.059 71.4375 0.00789363 0.0166381 2024-09-02T10:11:42.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:42.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:42.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:42.615 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:42.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:42.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:42.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:42.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:42.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:42.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:42.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:11:42.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:42.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:42.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:42.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:42.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:11:43.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:43.041 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:43.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:43.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:43.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:43.169 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:43.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:43.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:43.260 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:43.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:43.297 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:43.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:43.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:43 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1912/107076 objects degraded (1.786%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:43 smithi151 ceph-mon[16639]: osdmap e349: 12 total, 12 up, 12 in 2024-09-02T10:11:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:43.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:43.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:43.487 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:43.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:11:43.504925+0000 min lat: 0.00103182 max lat: 3.02533 avg lat: 0.0166335 2024-09-02T10:11:43.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:11:43.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 40 16 38460 38444 60.0637 60.25 0.0181754 0.0166335 2024-09-02T10:11:43.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:43.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:43.568 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:43.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:43 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1912/107076 objects degraded (1.786%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:43 smithi162 ceph-mon[20279]: osdmap e349: 12 total, 12 up, 12 in 2024-09-02T10:11:43.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:43 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1912/107076 objects degraded (1.786%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:43.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:43 smithi187 ceph-mon[20072]: osdmap e349: 12 total, 12 up, 12 in 2024-09-02T10:11:43.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:43.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:43.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:43.764 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:11:43.764 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:11:43.764 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 349, 2024-09-02T10:11:43.764 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:11:43.328521+0000", 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:09:58.599885+0000", 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5801984, 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "noscrub", 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:11:43.765 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 4, 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "jewel", 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:11:43.766 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:11:43.767 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:11:43.768 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "264", 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "241", 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:11:43.769 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:11:43.770 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:11:43.771 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.7000000476837158, 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.7000000476837158, 2024-09-02T10:11:43.772 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.4800000190734863, 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.4800000190734863, 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 4, 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_2", 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:10:55.979431+0000", 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:11:43.773 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 26, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 26, 2024-09-02T10:11:43.774 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 26, 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 26, 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 26, 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "343", 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:11:43.775 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "341", 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:11:43.776 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.777 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:11:43.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 1.8500000238418579, 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.7699999809265137, 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 1.8500000238418579, 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.7699999809265137, 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.779 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 344, 2024-09-02T10:11:43.780 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.781 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.782 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.782 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.782 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.782 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.783 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.784 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.785 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:11:43.786 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:11:43.786 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:11:43.786 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:11:43.786 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.786 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.786 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.786 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.786 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.569854736328125, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 345, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.788 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.789 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:11:43.790 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.791 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.792 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:11:43.793 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 344, 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:11:43.794 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:11:43.795 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.796 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.6008758544921875, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 343, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.801 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.802 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:11:43.803 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.16259765625, 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.804 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 340, 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:11:43.805 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:11:43.806 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:11:43.807 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:11:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:11:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:11:43.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.809 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.809 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:11:43.809 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:11:43.809 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.809 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.809 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.810 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 344, 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:11:43.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.812 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.814 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.815 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.816 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.70245361328125, 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 326, 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:11:43.817 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:11:43.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.819 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.820 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:11:43.820 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:11:43.820 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.820 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.820 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.820 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.820 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.820 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:11:43.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:43.821 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:11:43.822 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:11:43.822 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:11:43.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.822 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.822 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:11:43.822 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.823 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 344, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.825 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.826 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:11:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.828 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 301, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.829 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.831 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:11:43.832 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.833 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.834 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 348, 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:11:43.835 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:11:43.836 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:11:43.837 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:11:43.838 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.840 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:11:43.840 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:11:43.840 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.840 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.840 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:11:43.840 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.840 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.840 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.840 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:11:43.840 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 344, 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:11:43.841 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.842 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.843 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:11:43.844 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:11:43.845 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.4313507080078125, 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 341, 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.846 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:11:43.847 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:11:43.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.847 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.847 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:11:43.847 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:11:43.847 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.847 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.847 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.848 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:11:43.850 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.851 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:10:37.448727+0000", 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:10:53.624029+0000", 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.852 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:30.441749+0000", 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.853 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:10:41.724933+0000", 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:17.125049+0000", 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:11:43.854 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:01.991219+0000", 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.855 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:01.349928+0000", 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:13.177789+0000", 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:11:43.856 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:10:36.000893+0000", 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.857 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.858 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:10:50.538534+0000", 2024-09-02T10:11:43.858 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.858 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:24.748345+0000", 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:11:43.860 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:10:42.056175+0000", 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [], 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [ 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "4.10", 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-09-02T10:11:43.861 INFO:teuthology.orchestra.run.smithi151.stdout: 1 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "4.11", 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: 5, 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: 6 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "4.13", 2024-09-02T10:11:43.862 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: 1 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "4.14", 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: 2, 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: 7 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "4.15", 2024-09-02T10:11:43.863 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: 6 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "4.16", 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: 0 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.864 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "4.17", 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: 7, 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: 0 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "4.18", 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: 7, 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: 0 2024-09-02T10:11:43.865 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "4.19", 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: 0, 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: 2 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:11:43.866 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:11:43.867 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:11:43.868 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:11:43.869 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:11:43.869 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:11:43.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:43.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:43.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:43.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:44.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:44.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:44.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:44.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:44.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:44.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:44.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:44.272 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:44.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:44.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:44.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:44.407 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:44.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:11:44.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 41 16 38953 38937 59.3502 30.8125 0.00877402 0.0168418 2024-09-02T10:11:44.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:44.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:44.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:44.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:44.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:44.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:44.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:44 smithi151 ceph-mon[16639]: pgmap v850: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.2 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 207 op/s; 1912/107076 objects degraded (1.786%); 10426/107076 objects misplaced (9.737%); 19 MiB/s, 300 objects/s recovering 2024-09-02T10:11:44.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/818718191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:11:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:44 smithi151 ceph-mon[16639]: osdmap e350: 12 total, 12 up, 12 in 2024-09-02T10:11:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:44 smithi187 ceph-mon[20072]: pgmap v850: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.2 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 207 op/s; 1912/107076 objects degraded (1.786%); 10426/107076 objects misplaced (9.737%); 19 MiB/s, 300 objects/s recovering 2024-09-02T10:11:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/818718191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:11:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:44 smithi187 ceph-mon[20072]: osdmap e350: 12 total, 12 up, 12 in 2024-09-02T10:11:44.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:44.805 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:44.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:44.829 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:44 smithi162 ceph-mon[20279]: pgmap v850: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.2 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s wr, 207 op/s; 1912/107076 objects degraded (1.786%); 10426/107076 objects misplaced (9.737%); 19 MiB/s, 300 objects/s recovering 2024-09-02T10:11:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/818718191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:11:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:44 smithi162 ceph-mon[20279]: osdmap e350: 12 total, 12 up, 12 in 2024-09-02T10:11:44.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:44.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:11:44.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:44.938 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:45.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:45.061 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:45.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:45.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:45.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:45.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:45.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:45.349 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:45.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:45.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:45.389 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:11:45.390 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:11:45.407 INFO:tasks.thrashosds.thrasher:j is {'epoch': 349, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:11:43.328521+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:09:58.599885+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.700000047683716, 'score_stable': 2.700000047683716, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.4800000190734863, 'raw_score_stable': 2.4800000190734863, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-02T10:10:55.979431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '343', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '341', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.850000023841858, 'score_stable': 2.7699999809265137, 'optimal_score': 1, 'raw_score_acting': 1.850000023841858, 'raw_score_stable': 2.7699999809265137, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 344, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 0.569854736328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 345, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 344, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 0.6008758544921875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 343, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 1, 'weight': 0.16259765625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 340, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 344, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 0.70245361328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 326, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 344, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 301, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 348, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 344, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 1, 'weight': 0.4313507080078125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 341, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:10:37.448727+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:10:53.624029+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:30.441749+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:10:41.724933+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:17.125049+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:01.991219+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:01.349928+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:13.177789+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:10:36.000893+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:10:50.538534+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:24.748345+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:10:42.056175+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '4.10', 'osds': [0, 1]}, {'pgid': '4.11', 'osds': [5, 6]}, {'pgid': '4.13', 'osds': [0, 1]}, {'pgid': '4.14', 'osds': [2, 7]}, {'pgid': '4.15', 'osds': [10, 6]}, {'pgid': '4.16', 'osds': [1, 0]}, {'pgid': '4.17', 'osds': [7, 10, 0]}, {'pgid': '4.18', 'osds': [7, 0]}, {'pgid': '4.19', 'osds': [0, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:11:45.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:11:45.450 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:45.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:45.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:45.479 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:45.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 42 16 40397 40381 60.0857 90.25 0.00295044 0.0166174 2024-09-02T10:11:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:45 smithi187 ceph-mon[20072]: 4.12 scrub starts 2024-09-02T10:11:45.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:45.599 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:45.600 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:45.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:45 smithi151 ceph-mon[16639]: 4.12 scrub starts 2024-09-02T10:11:45.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:45.722 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:45.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:45.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:45.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:45 smithi162 ceph-mon[20279]: 4.12 scrub starts 2024-09-02T10:11:45.848 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:45.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:45.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:45.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:45.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:45.972 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:46.063 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:46.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:46.094 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:46.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:46.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:46.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:46.219 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:46.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:46.220 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:46.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:46.344 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:46.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:46.356 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:11:46.380 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:11:46.445 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:46.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:46.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:46.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:46.477 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:46 smithi151 ceph-mon[16639]: pgmap v853: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 975 op/s; 1912/112383 objects degraded (1.701%); 10426/112383 objects misplaced (9.277%); 34 MiB/s, 551 objects/s recovering 2024-09-02T10:11:46.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:46 smithi151 ceph-mon[16639]: osdmap e351: 12 total, 12 up, 12 in 2024-09-02T10:11:46.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2174120123' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:46.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3423899821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:46.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 43 16 40669 40653 59.0837 17 0.0140357 0.0169195 2024-09-02T10:11:46.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:46.608 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:46.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:46.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:46.723 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:46.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:46.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:46.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:46 smithi187 ceph-mon[20072]: pgmap v853: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 975 op/s; 1912/112383 objects degraded (1.701%); 10426/112383 objects misplaced (9.277%); 34 MiB/s, 551 objects/s recovering 2024-09-02T10:11:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:46 smithi187 ceph-mon[20072]: osdmap e351: 12 total, 12 up, 12 in 2024-09-02T10:11:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2174120123' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3423899821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:46.834 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:11:46.834 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:11:46.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:46.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:46 smithi162 ceph-mon[20279]: pgmap v853: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 975 op/s; 1912/112383 objects degraded (1.701%); 10426/112383 objects misplaced (9.277%); 34 MiB/s, 551 objects/s recovering 2024-09-02T10:11:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:46 smithi162 ceph-mon[20279]: osdmap e351: 12 total, 12 up, 12 in 2024-09-02T10:11:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2174120123' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:11:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3423899821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:11:46.858 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:11:46.858 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:11:46.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:46.864 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:46.874 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:11:46.874 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:11:46.880 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:11:46.880 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:11:46.909 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:11:46.909 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:11:46.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:11:46.940 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:11:46.964 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:46.975 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:11:46.975 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:11:46.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:47.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:47.010 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:11:47.010 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:11:47.042 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:11:47.043 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:11:47.067 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:11:47.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:11:47.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:47.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:11:47.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:11:47.128 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:47.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:47.139 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:11:47.139 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:11:47.177 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:11:47.177 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:11:47.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:47.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:47.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:47.263 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:47.283 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:27094 2.d 2:b197dc2d:::benchmark_data_smithi187_45882_object27093:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:01:54.667076+0000", 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 592.61269561500001, 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23479106199999999, 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.284 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27094 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.667076+0000", 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.667076+0000", 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.667077+0000", 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.11e-06 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.667095+0000", 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8117999999999998e-05 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.667107+0000", 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1678999999999999e-05 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.667112+0000", 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3399999999999997e-06 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.667167+0000", 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4981000000000001e-05 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.287 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.667285+0000", 2024-09-02T10:11:47.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118447 2024-09-02T10:11:47.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,4", 2024-09-02T10:11:47.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.667424+0000", 2024-09-02T10:11:47.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013901899999999999 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.668385+0000", 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000960136 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.668886+0000", 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050128200000000003 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.901755+0000", 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23286900999999999 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.901802+0000", 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.727e-05 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:01:54.901867+0000", 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4670000000000003e-05 2024-09-02T10:11:47.290 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:89423 2.d 2:b52cef48:::benchmark_data_smithi187_45882_object89422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:55.043958+0000", 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 532.23581376899995, 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21652831, 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.291 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 89423 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.043958+0000", 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.043958+0000", 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.043959+0000", 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7719999999999999e-06 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.043983+0000", 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3768999999999999e-05 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.293 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.044001+0000", 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7394e-05 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.044008+0000", 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3939999999999998e-06 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.044081+0000", 2024-09-02T10:11:47.294 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3238999999999995e-05 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.044227+0000", 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000145463 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,4", 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.044363+0000", 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013671199999999999 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:47.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.046741+0000", 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023773090000000002 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.064737+0000", 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017996814 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.260409+0000", 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19567180000000001 2024-09-02T10:11:47.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.260453+0000", 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3325000000000003e-05 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:55.260486+0000", 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3318999999999997e-05 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:117060 2.f e147/92 2:f36bc4f3:::benchmark_data_smithi187_45882_object19309:head v 147'7327, mlcod=147'7324)", 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:29.622453+0000", 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 497.65731838200003, 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26359092499999998, 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622453+0000", 2024-09-02T10:11:47.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622453+0000", 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622454+0000", 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7499999999999999e-07 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.622459+0000", 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8450000000000002e-06 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.882507+0000", 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26004827899999999 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.882512+0000", 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8570000000000003e-06 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.882602+0000", 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0004000000000001e-05 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.882628+0000", 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6373999999999999e-05 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.886004+0000", 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033755679999999998 2024-09-02T10:11:47.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:29.886044+0000", 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0222999999999998e-05 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:163319 2.f e198/92 2:fc89819c:::benchmark_data_smithi187_45882_object65568:head v 198'10220, mlcod=198'10217)", 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:56.124308+0000", 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 411.15546331199999, 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.334735424, 2024-09-02T10:11:47.302 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.124308+0000", 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.124308+0000", 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.124309+0000", 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.16e-07 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.124314+0000", 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.49e-06 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455559+0000", 2024-09-02T10:11:47.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33124411199999998 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455563+0000", 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4880000000000001e-06 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455605+0000", 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1980000000000001e-05 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.305 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455622+0000", 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6753999999999999e-05 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.459008+0000", 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033862549999999999 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.459044+0000", 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5429000000000002e-05 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.306 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:163327 2.f e198/92 2:f9a47073:::benchmark_data_smithi187_45882_object65576:head v 198'10221, mlcod=198'10218)", 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:56.455531+0000", 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 410.82424074300002, 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42168928999999999, 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455531+0000", 2024-09-02T10:11:47.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455531+0000", 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455531+0000", 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0900000000000001e-07 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.455544+0000", 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2347999999999999e-05 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.873648+0000", 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41810388399999998 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.873660+0000", 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2113000000000001e-05 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.873775+0000", 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000114862 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.873814+0000", 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9168999999999997e-05 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877190+0000", 2024-09-02T10:11:47.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003376109 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:56.877220+0000", 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0096000000000001e-05 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:168004 2.d e203/198 2:b1d0904c:::benchmark_data_smithi187_45882_object70253:head v 203'10242, mlcod=155'7720)", 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:04.136721+0000", 2024-09-02T10:11:47.311 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 403.14305000600001, 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.233264363, 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:04.136721+0000", 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:04.136721+0000", 2024-09-02T10:11:47.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:04.136722+0000", 2024-09-02T10:11:47.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8800000000000002e-07 2024-09-02T10:11:47.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:04.136729+0000", 2024-09-02T10:11:47.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.99e-06 2024-09-02T10:11:47.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:04.364619+0000", 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22788939799999999 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:04.364627+0000", 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5089999999999993e-06 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:04.364686+0000", 2024-09-02T10:11:47.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8767000000000002e-05 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:04.364712+0000", 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6610000000000001e-05 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:04.369953+0000", 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0052405259999999997 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:04.369986+0000", 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2774999999999997e-05 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGRecoveryDelete(2.d e217,198 2:b0e57f56:::benchmark_data_smithi187_45882_object40480:head,164'8403)", 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:43.127447+0000", 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 364.15232405799998, 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21266674599999999, 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.317 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.127447+0000", 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.127447+0000", 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.127448+0000", 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3399999999999999e-07 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.127452+0000", 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2560000000000004e-06 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.340033+0000", 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21258033100000001 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.340037+0000", 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0579999999999999e-06 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.340077+0000", 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0222999999999998e-05 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.340083+0000", 2024-09-02T10:11:47.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5100000000000004e-06 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.340114+0000", 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0734000000000001e-05 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGRecoveryDelete(2.d e217,198 2:bf35c8fc:::benchmark_data_smithi187_45882_object42344:head,166'8528)", 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:43.959775+0000", 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 363.319996049, 2024-09-02T10:11:47.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42357175699999999, 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.959775+0000", 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.322 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.959775+0000", 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.959776+0000", 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7999999999999993e-07 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:43.959783+0000", 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.827e-06 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.323 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:44.383208+0000", 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42342490500000002 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:44.383216+0000", 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6090000000000003e-06 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:44.383300+0000", 2024-09-02T10:11:47.324 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4432999999999996e-05 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:44.383310+0000", 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0003000000000001e-05 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:44.383347+0000", 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6999999999999998e-05 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.325 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:12909 3.d e233/226 3:b7ff8135:::benchmark_data_smithi187_59989_object12908:head v 233'843, mlcod=233'840)", 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:06:11.042245+0000", 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 336.23752680000001, 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21531315300000001, 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.042245+0000", 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.042245+0000", 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.042245+0000", 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8199999999999998e-07 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.042305+0000", 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9632000000000001e-05 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.256224+0000", 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.213918736 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.256230+0000", 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2149999999999998e-06 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.256292+0000", 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1505000000000002e-05 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.256316+0000", 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4321999999999999e-05 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.257522+0000", 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012062920000000001 2024-09-02T10:11:47.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:06:11.257558+0000", 2024-09-02T10:11:47.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5568999999999998e-05 2024-09-02T10:11:47.330 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.330 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.331 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:75430 3.d e253/226 3:b0cdaf1d:::benchmark_data_smithi187_59989_object75429:head v 253'4693, mlcod=253'4690)", 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:07:00.363886+0000", 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 286.91588520800002, 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.225089915, 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:00.363886+0000", 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:00.363886+0000", 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:00.363887+0000", 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8299999999999996e-07 2024-09-02T10:11:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:00.363942+0000", 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5189999999999998e-05 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:00.586794+0000", 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.222851461 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:00.586801+0000", 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1099999999999997e-06 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:00.586866+0000", 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5396999999999997e-05 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:00.586891+0000", 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5225e-05 2024-09-02T10:11:47.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:00.588933+0000", 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002041289 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:00.588976+0000", 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3560000000000003e-05 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.336 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:189485 3.d e297/226 3:b04681eb:::benchmark_data_smithi187_59989_object79942:head v 297'11911, mlcod=297'11908)", 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:09:29.145057+0000", 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 138.13471479899999, 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26159663799999999, 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:29.145057+0000", 2024-09-02T10:11:47.337 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:29.145057+0000", 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:29.145057+0000", 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9299999999999996e-07 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:29.145072+0000", 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4663e-05 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:29.402919+0000", 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.257846562 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:29.402926+0000", 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3010000000000001e-06 2024-09-02T10:11:47.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:29.403004+0000", 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7585999999999999e-05 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:29.403021+0000", 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7073999999999999e-05 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:29.406625+0000", 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036047399999999999 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:29.406653+0000", 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7818999999999999e-05 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(1.a,GRANT,e:316,from:4,reservation_nonce:0)", 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:10:53.324055+0000", 2024-09-02T10:11:47.341 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 53.955716637000002, 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.952292107, 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:10:53.324055+0000", 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:10:53.324055+0000", 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:10:53.324056+0000", 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1400000000000001e-06 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:10:53.324062+0000", 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7239999999999997e-06 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:10:54.276153+0000", 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.95209184099999999 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:10:54.276171+0000", 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.772e-05 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:10:54.276286+0000", 2024-09-02T10:11:47.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011474 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:10:54.276303+0000", 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7283e-05 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:10:54.276347+0000", 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3659000000000003e-05 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.345 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:11556 4.d 4:b36fbd37:::benchmark_data_smithi187_76542_object11555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:10.472238+0000", 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 36.807533411000001, 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25507339499999998, 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:47.346 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 11556 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472238+0000", 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472238+0000", 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472239+0000", 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2099999999999996e-07 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472257+0000", 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.808e-05 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472267+0000", 2024-09-02T10:11:47.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0447e-05 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472273+0000", 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8100000000000003e-06 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472317+0000", 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3727999999999999e-05 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472402+0000", 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4790999999999995e-05 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,11", 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472493+0000", 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1399999999999999e-05 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.473081+0000", 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000588063 2024-09-02T10:11:47.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.473610+0000", 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00052937600000000002 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.727240+0000", 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25363005799999999 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.727275+0000", 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4183000000000001e-05 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.727311+0000", 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6738e-05 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:11571 4.d 4:b88a9f8d:::benchmark_data_smithi187_76542_object11570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:11:47.352 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:10.795158+0000", 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 36.484613035000002, 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.49190283099999998, 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 11571 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795158+0000", 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795158+0000", 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795161+0000", 2024-09-02T10:11:47.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2400000000000002e-06 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795204+0000", 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2951999999999998e-05 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795235+0000", 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1832999999999998e-05 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795241+0000", 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2129999999999999e-06 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795296+0000", 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5022000000000003e-05 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795397+0000", 2024-09-02T10:11:47.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010155899999999999 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,11", 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795482+0000", 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4738000000000001e-05 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.796037+0000", 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00055539100000000002 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.796398+0000", 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036089500000000002 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.286938+0000", 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.49054011600000003 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.287018+0000", 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9672999999999994e-05 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.287061+0000", 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3198999999999998e-05 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:21192 4.2 4:4719e8c3:::benchmark_data_smithi187_76542_object21191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:22.130259+0000", 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 25.14951194, 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21566385299999999, 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.359 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 21192 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.130259+0000", 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.130259+0000", 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.130260+0000", 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0999999999999997e-07 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.130281+0000", 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0652000000000001e-05 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.130292+0000", 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0829000000000001e-05 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.130299+0000", 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8669999999999996e-06 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.132166+0000", 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001867563 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.132317+0000", 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015082600000000001 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:11:47.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.132456+0000", 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013885199999999999 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.133808+0000", 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013525099999999999 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.135831+0000", 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020223340000000002 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.345831+0000", 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.210000306 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.345883+0000", 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2264e-05 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.345923+0000", 2024-09-02T10:11:47.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0040000000000003e-05 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:21197 4.2 4:4284590f:::benchmark_data_smithi187_76542_object21196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:22.132256+0000", 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 25.147515707, 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21376477699999999, 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:47.366 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 21197 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.132256+0000", 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.132256+0000", 2024-09-02T10:11:47.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.132257+0000", 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0699999999999999e-06 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.132278+0000", 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1528000000000001e-05 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.132287+0000", 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0019999999999995e-06 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.132294+0000", 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6560000000000003e-06 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.133835+0000", 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0015408710000000001 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.134217+0000", 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00038199499999999999 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.134366+0000", 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014946800000000001 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.135733+0000", 2024-09-02T10:11:47.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00136699 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.140801+0000", 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050673039999999999 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.345955+0000", 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20515478100000001 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.345989+0000", 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4103999999999997e-05 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.346020+0000", 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1007999999999999e-05 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.372 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:21207 4.2 4:432a33fa:::benchmark_data_smithi187_76542_object21206:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:22.139102+0000", 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 25.140669722999998, 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20754537100000001, 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 21207 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.139102+0000", 2024-09-02T10:11:47.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:47.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.139102+0000", 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.139103+0000", 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2279999999999999e-06 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.139122+0000", 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.933e-05 2024-09-02T10:11:47.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.139134+0000", 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1411e-05 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.139139+0000", 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1240000000000004e-06 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.139192+0000", 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3464999999999997e-05 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.139321+0000", 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012886199999999999 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.139469+0000", 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014779800000000001 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.140466+0000", 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00099738299999999991 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.157023+0000", 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016556797000000002 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.346592+0000", 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18956931799999999 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.346624+0000", 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1186000000000003e-05 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.346647+0000", 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3468999999999998e-05 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:24530 4.2 4:46839913:::benchmark_data_smithi187_76542_object24529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:25.006957+0000", 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 22.272814504999999, 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20920845399999999, 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:47.381 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 24530 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.006957+0000", 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.006957+0000", 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.006958+0000", 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0499999999999999e-06 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.006976+0000", 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7975e-05 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.006987+0000", 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0604e-05 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.006993+0000", 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6560000000000003e-06 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.007048+0000", 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4629999999999997e-05 2024-09-02T10:11:47.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.007128+0000", 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9896999999999998e-05 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.007214+0000", 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6292000000000007e-05 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:47.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.008102+0000", 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00088760499999999997 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.008396+0000", 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000293928 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.215974+0000", 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.207578282 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.216034+0000", 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0352000000000001e-05 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.216165+0000", 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013118299999999999 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.388 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25507.0:24530 4.2 e335/318 ondisk, result = 0)", 2024-09-02T10:11:47.388 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:25.008478+0000", 2024-09-02T10:11:47.388 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 22.271292937999998, 2024-09-02T10:11:47.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.207694777, 2024-09-02T10:11:47.388 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.388 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.008478+0000", 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.008478+0000", 2024-09-02T10:11:47.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.008479+0000", 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0800000000000004e-07 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.008483+0000", 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8369999999999999e-06 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.215888+0000", 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20740551500000001 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.215899+0000", 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.065e-05 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.215959+0000", 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9729999999999999e-05 2024-09-02T10:11:47.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.215968+0000", 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1349999999999998e-06 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.216173+0000", 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00020530200000000001 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:34740 4.2 4:42d82b8a:::benchmark_data_smithi187_76542_object34739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:40.308640+0000", 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 6.9711313300000004, 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.092384994999999998, 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 34740 2024-09-02T10:11:47.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.308640+0000", 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.308640+0000", 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.308641+0000", 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.403e-06 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.308665+0000", 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3059999999999999e-05 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.308679+0000", 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4285e-05 2024-09-02T10:11:47.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.308684+0000", 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1440000000000002e-06 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.308729+0000", 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4767e-05 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.308855+0000", 2024-09-02T10:11:47.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012645399999999999 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.308944+0000", 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9030000000000006e-05 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.309939+0000", 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00099527399999999999 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.310432+0000", 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000492356 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.400922+0000", 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.090490599000000005 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.400983+0000", 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0093000000000002e-05 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:47.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:40.401025+0000", 2024-09-02T10:11:47.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2530000000000001e-05 2024-09-02T10:11:47.399 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.399 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.399 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.399 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:47.399 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:47.399 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:11:47.401 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:47.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:47.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:47.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 44 16 41775 41759 59.3118 69.125 0.00264922 0.0168412 2024-09-02T10:11:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:47 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2174120123' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:47 smithi187 ceph-mon[20072]: osdmap e352: 12 total, 12 up, 12 in 2024-09-02T10:11:47.526 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:47.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:47.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:47.654 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:47.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:47.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:47.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:47 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2174120123' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:47 smithi151 ceph-mon[16639]: osdmap e352: 12 total, 12 up, 12 in 2024-09-02T10:11:47.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:47.778 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:47.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:47.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:47.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:47 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:47.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2174120123' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:11:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:47 smithi162 ceph-mon[20279]: osdmap e352: 12 total, 12 up, 12 in 2024-09-02T10:11:47.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:47.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:47.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:48.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:48.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:48.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:48.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:48.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:48.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:48.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:48.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:48.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:48.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:48.274 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:48.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:11:48.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:48.425 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:48.426 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:48.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 45 16 42133 42117 58.491 22.375 0.00224399 0.0170676 2024-09-02T10:11:48.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:48.548 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:48.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:48.636 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:48.644 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:48 smithi151 ceph-mon[16639]: 4.14 scrub starts 2024-09-02T10:11:48.644 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:48 smithi151 ceph-mon[16639]: from='client.25615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:11:48.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:48 smithi151 ceph-mon[16639]: pgmap v856: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s; 1912/116559 objects degraded (1.640%); 7800/116559 objects misplaced (6.692%); 19 MiB/s, 296 objects/s recovering 2024-09-02T10:11:48.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:48 smithi151 ceph-mon[16639]: osdmap e353: 12 total, 12 up, 12 in 2024-09-02T10:11:48.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:48.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:48.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:48.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:48 smithi187 ceph-mon[20072]: 4.14 scrub starts 2024-09-02T10:11:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:48 smithi187 ceph-mon[20072]: from='client.25615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:11:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:48 smithi187 ceph-mon[20072]: pgmap v856: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s; 1912/116559 objects degraded (1.640%); 7800/116559 objects misplaced (6.692%); 19 MiB/s, 296 objects/s recovering 2024-09-02T10:11:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:48 smithi187 ceph-mon[20072]: osdmap e353: 12 total, 12 up, 12 in 2024-09-02T10:11:48.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:48.795 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:48.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:48.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:48.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:48 smithi162 ceph-mon[20279]: 4.14 scrub starts 2024-09-02T10:11:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:48 smithi162 ceph-mon[20279]: from='client.25615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:11:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:48 smithi162 ceph-mon[20279]: pgmap v856: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 47 active+clean; 2.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 790 op/s; 1912/116559 objects degraded (1.640%); 7800/116559 objects misplaced (6.692%); 19 MiB/s, 296 objects/s recovering 2024-09-02T10:11:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:48 smithi162 ceph-mon[20279]: osdmap e353: 12 total, 12 up, 12 in 2024-09-02T10:11:48.897 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:48.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:48.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:49.016 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":855,"stamp":"2024-09-02T10:11:46.356854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2494464544,"num_objects":38058,"num_object_clones":0,"num_object_copies":114174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":9506,"num_objects_unfound":0,"num_objects_dirty":38058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28584,"num_write_kb":1829216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11421,"num_bytes_recovered":748421120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59828,"ondisk_log_size":59828,"up":171,"acting":162,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":200,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":10784216,"kb_used_data":8278980,"kb_used_omap":18,"kb_used_meta":2505005,"kb_avail":1114157608,"statfs":{"total":1151940427776,"available":1140897390592,"internally_reserved":0,"allocated":8477675520,"data_stored":8473020220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":2565125435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,2,3,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":372244480,"num_objects":5680,"num_object_clones":0,"num_object_copies":17040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6300,"num_objects_unfound":0,"num_objects_dirty":5680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5680,"num_write_kb":363520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3364,"num_bytes_recovered":220463104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.130990"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":178,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.331677+0000","last_change":"2024-09-02T10:10:52.609809+0000","last_active":"2024-09-02T10:11:43.331677+0000","last_peered":"2024-09-02T10:11:43.331677+0000","last_clean":"2024-09-02T10:11:43.331677+0000","last_became_active":"2024-09-02T10:09:58.903895+0000","last_became_peered":"2024-09-02T10:09:58.903895+0000","last_unstale":"2024-09-02T10:11:43.331677+0000","last_undegraded":"2024-09-02T10:11:43.331677+0000","last_fullsized":"2024-09-02T10:11:43.331677+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:52.609761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:52.609761+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:02.709076+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.18","version":"351'2270","reported_seq":7447,"reported_epoch":351,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:45.997854+0000","last_change":"2024-09-02T10:11:36.071317+0000","last_active":"2024-09-02T10:11:45.997854+0000","last_peered":"2024-09-02T10:11:45.997854+0000","last_clean":"2024-09-02T10:11:33.782776+0000","last_became_active":"2024-09-02T10:11:36.060324+0000","last_became_peered":"2024-09-02T10:11:36.060324+0000","last_unstale":"2024-09-02T10:11:45.997854+0000","last_undegraded":"2024-09-02T10:11:45.997854+0000","last_fullsized":"2024-09-02T10:11:35.927097+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2270,"log_dups_size":0,"ondisk_log_size":2270,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84967424,"num_objects":1297,"num_object_clones":0,"num_object_copies":3891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1071,"num_objects_unfound":0,"num_objects_dirty":1297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1297,"num_write_kb":82976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1270,"num_bytes_recovered":83197952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[7,0],"avail_no_missing":["7","9","11"],"object_location_counts":[{"shards":"0,7","objects":940},{"shards":"7,9,11","objects":357}],"blocked_by":[],"up_primary":1,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":60,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.998687+0000","last_change":"2024-09-02T10:11:29.264235+0000","last_active":"2024-09-02T10:11:39.998687+0000","last_peered":"2024-09-02T10:11:39.998687+0000","last_clean":"2024-09-02T10:11:39.998687+0000","last_became_active":"2024-09-02T10:11:18.920647+0000","last_became_peered":"2024-09-02T10:11:18.920647+0000","last_unstale":"2024-09-02T10:11:39.998687+0000","last_undegraded":"2024-09-02T10:11:39.998687+0000","last_fullsized":"2024-09-02T10:11:39.998687+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:29.264185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:29.264185+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:53.100224+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,10],"acting":[2,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"348'2102","reported_seq":5911,"reported_epoch":348,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:42.540417+0000","last_change":"2024-09-02T10:11:36.110765+0000","last_active":"2024-09-02T10:11:42.540417+0000","last_peered":"2024-09-02T10:11:42.540417+0000","last_clean":"2024-09-02T10:11:33.762656+0000","last_became_active":"2024-09-02T10:11:36.081233+0000","last_became_peered":"2024-09-02T10:11:36.081233+0000","last_unstale":"2024-09-02T10:11:42.540417+0000","last_undegraded":"2024-09-02T10:11:42.540417+0000","last_fullsized":"2024-09-02T10:11:35.925082+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76316672,"num_objects":1165,"num_object_clones":0,"num_object_copies":3495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1010,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1165,"num_write_kb":74528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1052,"num_bytes_recovered":68943872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[0,2],"avail_no_missing":["0","1","4","6"],"object_location_counts":[{"shards":"0,1,6","objects":505},{"shards":"0,2","objects":660}],"blocked_by":[],"up_primary":10,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":257,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.331525+0000","last_change":"2024-09-02T10:11:03.720364+0000","last_active":"2024-09-02T10:11:43.331525+0000","last_peered":"2024-09-02T10:11:43.331525+0000","last_clean":"2024-09-02T10:11:43.331525+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:11:43.331525+0000","last_undegraded":"2024-09-02T10:11:43.331525+0000","last_fullsized":"2024-09-02T10:11:43.331525+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:03.720317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:03.720317+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:16.507003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":257,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.331360+0000","last_change":"2024-09-02T10:10:54.656216+0000","last_active":"2024-09-02T10:11:43.331360+0000","last_peered":"2024-09-02T10:11:43.331360+0000","last_clean":"2024-09-02T10:11:43.331360+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:11:43.331360+0000","last_undegraded":"2024-09-02T10:11:43.331360+0000","last_fullsized":"2024-09-02T10:11:43.331360+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.656165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.656165+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:24.539265+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":176,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-02T10:11:45.355379+0000","last_change":"2024-09-02T10:11:18.920901+0000","last_active":"2024-09-02T10:11:45.355379+0000","last_peered":"2024-09-02T10:11:45.355379+0000","last_clean":"2024-09-02T10:11:45.355379+0000","last_became_active":"2024-09-02T10:11:18.920488+0000","last_became_peered":"2024-09-02T10:11:18.920488+0000","last_unstale":"2024-09-02T10:11:45.355379+0000","last_undegraded":"2024-09-02T10:11:45.355379+0000","last_fullsized":"2024-09-02T10:11:45.355379+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:07.462786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:07.462786+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:22.886310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":112,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.997485+0000","last_change":"2024-09-02T10:10:58.372721+0000","last_active":"2024-09-02T10:11:39.997485+0000","last_peered":"2024-09-02T10:11:39.997485+0000","last_clean":"2024-09-02T10:11:39.997485+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:11:39.997485+0000","last_undegraded":"2024-09-02T10:11:39.997485+0000","last_fullsized":"2024-09-02T10:11:39.997485+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:58.372641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:58.372641+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:21.412502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":606,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.993048+0000","last_change":"2024-09-02T10:10:54.892467+0000","last_active":"2024-09-02T10:11:39.993048+0000","last_peered":"2024-09-02T10:11:39.993048+0000","last_clean":"2024-09-02T10:11:39.993048+0000","last_became_active":"2024-09-02T10:06:34.859627+0000","last_became_peered":"2024-09-02T10:06:34.859627+0000","last_unstale":"2024-09-02T10:11:39.993048+0000","last_undegraded":"2024-09-02T10:11:39.993048+0000","last_fullsized":"2024-09-02T10:11:39.993048+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.892371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.892371+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:05.539128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"348'2090","reported_seq":3980,"reported_epoch":348,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:43.133191+0000","last_change":"2024-09-02T10:11:36.038974+0000","last_active":"2024-09-02T10:11:43.133191+0000","last_peered":"2024-09-02T10:11:43.133191+0000","last_clean":"2024-09-02T10:11:33.750445+0000","last_became_active":"2024-09-02T10:11:36.024454+0000","last_became_peered":"2024-09-02T10:11:36.024454+0000","last_unstale":"2024-09-02T10:11:43.133191+0000","last_undegraded":"2024-09-02T10:11:43.133191+0000","last_fullsized":"2024-09-02T10:11:35.926968+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2090,"log_dups_size":0,"ondisk_log_size":2090,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75563008,"num_objects":1153,"num_object_clones":0,"num_object_copies":3459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1153,"num_write_kb":73792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":903,"num_bytes_recovered":59179008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,8],"avail_no_missing":["9","5","8"],"object_location_counts":[{"shards":"5,8,9","objects":26},{"shards":"8,9","objects":1127}],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":264,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.331773+0000","last_change":"2024-09-02T10:10:55.684692+0000","last_active":"2024-09-02T10:11:43.331773+0000","last_peered":"2024-09-02T10:11:43.331773+0000","last_clean":"2024-09-02T10:11:43.331773+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:11:43.331773+0000","last_undegraded":"2024-09-02T10:11:43.331773+0000","last_fullsized":"2024-09-02T10:11:43.331773+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:55.684623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:55.684623+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:56.065453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.13","version":"348'2123","reported_seq":3428,"reported_epoch":348,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:42.540534+0000","last_change":"2024-09-02T10:11:36.069089+0000","last_active":"2024-09-02T10:11:42.540534+0000","last_peered":"2024-09-02T10:11:42.540534+0000","last_clean":"2024-09-02T10:11:33.759677+0000","last_became_active":"2024-09-02T10:11:36.041297+0000","last_became_peered":"2024-09-02T10:11:36.041297+0000","last_unstale":"2024-09-02T10:11:42.540534+0000","last_undegraded":"2024-09-02T10:11:42.540534+0000","last_fullsized":"2024-09-02T10:11:35.926663+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2123,"log_dups_size":0,"ondisk_log_size":2123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77103104,"num_objects":1177,"num_object_clones":0,"num_object_copies":3531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1084,"num_objects_unfound":0,"num_objects_dirty":1177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1177,"num_write_kb":75296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":408,"num_bytes_recovered":26738688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[0,1],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1","objects":635},{"shards":"0,1,7","objects":542}],"blocked_by":[],"up_primary":9,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":262,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.997207+0000","last_change":"2024-09-02T10:11:11.497630+0000","last_active":"2024-09-02T10:11:39.997207+0000","last_peered":"2024-09-02T10:11:39.997207+0000","last_clean":"2024-09-02T10:11:39.997207+0000","last_became_active":"2024-09-02T10:11:11.497246+0000","last_became_peered":"2024-09-02T10:11:11.497246+0000","last_unstale":"2024-09-02T10:11:39.997207+0000","last_undegraded":"2024-09-02T10:11:39.997207+0000","last_fullsized":"2024-09-02T10:11:39.997207+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:07.248083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:07.248083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:36.348505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"348'2067","reported_seq":3953,"reported_epoch":348,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:42.538262+0000","last_change":"2024-09-02T10:11:36.056890+0000","last_active":"2024-09-02T10:11:42.538262+0000","last_peered":"2024-09-02T10:11:42.538262+0000","last_clean":"2024-09-02T10:11:33.716131+0000","last_became_active":"2024-09-02T10:11:36.012873+0000","last_became_peered":"2024-09-02T10:11:36.012873+0000","last_unstale":"2024-09-02T10:11:42.538262+0000","last_undegraded":"2024-09-02T10:11:42.538262+0000","last_fullsized":"2024-09-02T10:11:35.925089+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74809344,"num_objects":1142,"num_object_clones":0,"num_object_copies":3426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1467,"num_objects_unfound":0,"num_objects_dirty":1142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1142,"num_write_kb":73056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":29556736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[0,1],"avail_no_missing":["0","7","8"],"object_location_counts":[{"shards":"0,1","objects":653},{"shards":"0,7,8","objects":489}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":252,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-02T10:11:45.356226+0000","last_change":"2024-09-02T10:10:55.422138+0000","last_active":"2024-09-02T10:11:45.356226+0000","last_peered":"2024-09-02T10:11:45.356226+0000","last_clean":"2024-09-02T10:11:45.356226+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:11:45.356226+0000","last_undegraded":"2024-09-02T10:11:45.356226+0000","last_fullsized":"2024-09-02T10:11:45.356226+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:55.422074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:55.422074+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:13.928736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"348'2152","reported_seq":3418,"reported_epoch":348,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:42.008513+0000","last_change":"2024-09-02T10:11:36.036865+0000","last_active":"2024-09-02T10:11:42.008513+0000","last_peered":"2024-09-02T10:11:42.008513+0000","last_clean":"2024-09-02T10:11:33.720833+0000","last_became_active":"2024-09-02T10:11:36.008050+0000","last_became_peered":"2024-09-02T10:11:36.008050+0000","last_unstale":"2024-09-02T10:11:42.008513+0000","last_undegraded":"2024-09-02T10:11:42.008513+0000","last_fullsized":"2024-09-02T10:11:35.925577+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2152,"log_dups_size":0,"ondisk_log_size":2152,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77955072,"num_objects":1190,"num_object_clones":0,"num_object_copies":3570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1229,"num_objects_unfound":0,"num_objects_dirty":1190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1190,"num_write_kb":76128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":394,"num_bytes_recovered":25821184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[5,6],"avail_no_missing":["5","6","11"],"object_location_counts":[{"shards":"3,5,6,11","objects":1},{"shards":"5,6","objects":575},{"shards":"5,6,11","objects":614}],"blocked_by":[],"up_primary":3,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":501,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:40.010618+0000","last_change":"2024-09-02T10:10:57.026011+0000","last_active":"2024-09-02T10:11:40.010618+0000","last_peered":"2024-09-02T10:11:40.010618+0000","last_clean":"2024-09-02T10:11:40.010618+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:11:40.010618+0000","last_undegraded":"2024-09-02T10:11:40.010618+0000","last_fullsized":"2024-09-02T10:11:40.010618+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:57.025960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:57.025960+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:16.853607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"348'2117","reported_seq":2197,"reported_epoch":348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:11:43.141660+0000","last_change":"2024-09-02T10:11:36.991451+0000","last_active":"2024-09-02T10:11:43.141660+0000","last_peered":"2024-09-02T10:11:43.141660+0000","last_clean":"2024-09-02T10:11:33.751443+0000","last_became_active":"2024-09-02T10:11:36.977541+0000","last_became_peered":"2024-09-02T10:11:36.977541+0000","last_unstale":"2024-09-02T10:11:43.141660+0000","last_undegraded":"2024-09-02T10:11:36.974300+0000","last_fullsized":"2024-09-02T10:11:36.928520+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2117,"log_dups_size":0,"ondisk_log_size":2117,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77594624,"num_objects":1184,"num_object_clones":0,"num_object_copies":3552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":956,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1184,"num_write_kb":75776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[1,0],"avail_no_missing":["1","2","9"],"object_location_counts":[{"shards":"0,1","objects":228},{"shards":"1,2,9","objects":956}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":594,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.997249+0000","last_change":"2024-09-02T10:10:59.358111+0000","last_active":"2024-09-02T10:11:39.997249+0000","last_peered":"2024-09-02T10:11:39.997249+0000","last_clean":"2024-09-02T10:11:39.997249+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:11:39.997249+0000","last_undegraded":"2024-09-02T10:11:39.997249+0000","last_fullsized":"2024-09-02T10:11:39.997249+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:59.358063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:59.358063+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:29.124978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"351'2177","reported_seq":3772,"reported_epoch":351,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-02T10:11:45.842063+0000","last_change":"2024-09-02T10:11:35.954711+0000","last_active":"2024-09-02T10:11:45.842063+0000","last_peered":"2024-09-02T10:11:45.842063+0000","last_clean":"2024-09-02T10:11:33.717559+0000","last_became_active":"2024-09-02T10:11:35.951452+0000","last_became_peered":"2024-09-02T10:11:35.951452+0000","last_unstale":"2024-09-02T10:11:45.842063+0000","last_undegraded":"2024-09-02T10:11:45.842063+0000","last_fullsized":"2024-09-02T10:11:45.842063+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81395712,"num_objects":1242,"num_object_clones":0,"num_object_copies":3726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1242,"num_objects_unfound":0,"num_objects_dirty":1242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1242,"num_write_kb":79488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":24444928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,10],"acting":[7,10,0],"avail_no_missing":["7","0","10"],"object_location_counts":[{"shards":"0,7,10","objects":1242}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":530,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.993197+0000","last_change":"2024-09-02T10:11:11.468687+0000","last_active":"2024-09-02T10:11:39.993197+0000","last_peered":"2024-09-02T10:11:39.993197+0000","last_clean":"2024-09-02T10:11:39.993197+0000","last_became_active":"2024-09-02T10:11:11.468355+0000","last_became_peered":"2024-09-02T10:11:11.468355+0000","last_unstale":"2024-09-02T10:11:39.993197+0000","last_undegraded":"2024-09-02T10:11:39.993197+0000","last_fullsized":"2024-09-02T10:11:39.993197+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:00.896593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:00.896593+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:08.856040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"348'2128","reported_seq":7428,"reported_epoch":348,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:42.580317+0000","last_change":"2024-09-02T10:11:36.002906+0000","last_active":"2024-09-02T10:11:42.580317+0000","last_peered":"2024-09-02T10:11:42.580317+0000","last_clean":"2024-09-02T10:11:33.885096+0000","last_became_active":"2024-09-02T10:11:35.990824+0000","last_became_peered":"2024-09-02T10:11:35.990824+0000","last_unstale":"2024-09-02T10:11:42.580317+0000","last_undegraded":"2024-09-02T10:11:42.580317+0000","last_fullsized":"2024-09-02T10:11:35.927090+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2128,"log_dups_size":0,"ondisk_log_size":2128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77004800,"num_objects":1175,"num_object_clones":0,"num_object_copies":3525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":197,"num_objects_unfound":0,"num_objects_dirty":1175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1175,"num_write_kb":75200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1682,"num_bytes_recovered":110198784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[2,7],"avail_no_missing":["2","1","7","10"],"object_location_counts":[{"shards":"2,7","objects":978},{"shards":"2,7,10","objects":197}],"blocked_by":[],"up_primary":9,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":503,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.996693+0000","last_change":"2024-09-02T10:11:00.720591+0000","last_active":"2024-09-02T10:11:39.996693+0000","last_peered":"2024-09-02T10:11:39.996693+0000","last_clean":"2024-09-02T10:11:39.996693+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:11:39.996693+0000","last_undegraded":"2024-09-02T10:11:39.996693+0000","last_fullsized":"2024-09-02T10:11:39.996693+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:00.720541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:00.720541+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:09.602998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"348'2056","reported_seq":4706,"reported_epoch":348,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:11:41.534585+0000","last_change":"2024-09-02T10:11:36.033325+0000","last_active":"2024-09-02T10:11:41.534585+0000","last_peered":"2024-09-02T10:11:41.534585+0000","last_clean":"2024-09-02T10:11:18.505664+0000","last_became_active":"2024-09-02T10:11:35.993712+0000","last_became_peered":"2024-09-02T10:11:35.993712+0000","last_unstale":"2024-09-02T10:11:41.534585+0000","last_undegraded":"2024-09-02T10:11:41.534585+0000","last_fullsized":"2024-09-02T10:11:35.927169+0000","mapping_epoch":344,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2056,"log_dups_size":0,"ondisk_log_size":2056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74055680,"num_objects":1130,"num_object_clones":0,"num_object_copies":3390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1224,"num_objects_unfound":0,"num_objects_dirty":1130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1130,"num_write_kb":72320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":643,"num_bytes_recovered":42139648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[10,6],"avail_no_missing":["10","7"],"object_location_counts":[{"shards":"0,6,7,10","objects":337},{"shards":"0,7,10","objects":34},{"shards":"6,7,10","objects":224},{"shards":"6,10","objects":535}],"blocked_by":[],"up_primary":8,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":523,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.993182+0000","last_change":"2024-09-02T10:11:30.646296+0000","last_active":"2024-09-02T10:11:39.993182+0000","last_peered":"2024-09-02T10:11:39.993182+0000","last_clean":"2024-09-02T10:11:39.993182+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:11:39.993182+0000","last_undegraded":"2024-09-02T10:11:39.993182+0000","last_fullsized":"2024-09-02T10:11:39.993182+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:30.646251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:30.646251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:42.404524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"348'2352","reported_seq":2426,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.046982+0000","last_change":"2024-09-02T10:10:57.383623+0000","last_active":"2024-09-02T10:11:43.046982+0000","last_peered":"2024-09-02T10:11:43.046982+0000","last_clean":"2024-09-02T10:11:43.046982+0000","last_became_active":"2024-09-02T10:10:57.383259+0000","last_became_peered":"2024-09-02T10:10:57.383259+0000","last_unstale":"2024-09-02T10:11:43.046982+0000","last_undegraded":"2024-09-02T10:11:43.046982+0000","last_fullsized":"2024-09-02T10:11:43.046982+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2352,"log_dups_size":0,"ondisk_log_size":2352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:03.284820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2352,"num_write_kb":150528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,11],"acting":[9,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":595,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.993195+0000","last_change":"2024-09-02T10:10:59.881117+0000","last_active":"2024-09-02T10:11:39.993195+0000","last_peered":"2024-09-02T10:11:39.993195+0000","last_clean":"2024-09-02T10:11:39.993195+0000","last_became_active":"2024-09-02T10:06:32.873327+0000","last_became_peered":"2024-09-02T10:06:32.873327+0000","last_unstale":"2024-09-02T10:11:39.993195+0000","last_undegraded":"2024-09-02T10:11:39.993195+0000","last_fullsized":"2024-09-02T10:11:39.993195+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:59.881053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:59.881053+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:05.354025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"349'2417","reported_seq":3962,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.631387+0000","last_change":"2024-09-02T10:11:19.936905+0000","last_active":"2024-09-02T10:11:43.631387+0000","last_peered":"2024-09-02T10:11:43.631387+0000","last_clean":"2024-09-02T10:11:43.631387+0000","last_became_active":"2024-09-02T10:11:19.934231+0000","last_became_peered":"2024-09-02T10:11:19.934231+0000","last_unstale":"2024-09-02T10:11:43.631387+0000","last_undegraded":"2024-09-02T10:11:43.631387+0000","last_fullsized":"2024-09-02T10:11:43.631387+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2417,"log_dups_size":0,"ondisk_log_size":2417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:00.023826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158400512,"num_objects":2417,"num_object_clones":0,"num_object_copies":7251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2417,"num_write_kb":154688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":47775744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":724,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-02T10:11:45.355203+0000","last_change":"2024-09-02T10:11:02.444385+0000","last_active":"2024-09-02T10:11:45.355203+0000","last_peered":"2024-09-02T10:11:45.355203+0000","last_clean":"2024-09-02T10:11:45.355203+0000","last_became_active":"2024-09-02T10:06:32.871027+0000","last_became_peered":"2024-09-02T10:06:32.871027+0000","last_unstale":"2024-09-02T10:11:45.355203+0000","last_undegraded":"2024-09-02T10:11:45.355203+0000","last_fullsized":"2024-09-02T10:11:45.355203+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:02.444317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:02.444317+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:26.961398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"348'2434","reported_seq":2508,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.192949+0000","last_change":"2024-09-02T10:10:57.383040+0000","last_active":"2024-09-02T10:11:43.192949+0000","last_peered":"2024-09-02T10:11:43.192949+0000","last_clean":"2024-09-02T10:11:43.192949+0000","last_became_active":"2024-09-02T10:10:57.382683+0000","last_became_peered":"2024-09-02T10:10:57.382683+0000","last_unstale":"2024-09-02T10:11:43.192949+0000","last_undegraded":"2024-09-02T10:11:43.192949+0000","last_fullsized":"2024-09-02T10:11:43.192949+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2434,"log_dups_size":0,"ondisk_log_size":2434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:04.793656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159514624,"num_objects":2434,"num_object_clones":0,"num_object_copies":7302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2434,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2434,"num_write_kb":155776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":360,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:40.001352+0000","last_change":"2024-09-02T10:11:11.472721+0000","last_active":"2024-09-02T10:11:40.001352+0000","last_peered":"2024-09-02T10:11:40.001352+0000","last_clean":"2024-09-02T10:11:40.001352+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:11:40.001352+0000","last_undegraded":"2024-09-02T10:11:40.001352+0000","last_fullsized":"2024-09-02T10:11:40.001352+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.278963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.278963+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:59.180799+0000","scrub_duration":955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"349'2478","reported_seq":4961,"reported_epoch":349,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.857125+0000","last_change":"2024-09-02T10:11:31.957137+0000","last_active":"2024-09-02T10:11:43.857125+0000","last_peered":"2024-09-02T10:11:43.857125+0000","last_clean":"2024-09-02T10:11:43.857125+0000","last_became_active":"2024-09-02T10:11:31.954359+0000","last_became_peered":"2024-09-02T10:11:31.954359+0000","last_unstale":"2024-09-02T10:11:43.857125+0000","last_undegraded":"2024-09-02T10:11:43.857125+0000","last_fullsized":"2024-09-02T10:11:43.857125+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2478,"log_dups_size":0,"ondisk_log_size":2478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:20.565447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":7434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2478,"num_write_kb":158592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1199,"num_bytes_recovered":78577664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":162,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.995873+0000","last_change":"2024-09-02T10:10:55.733177+0000","last_active":"2024-09-02T10:11:39.995873+0000","last_peered":"2024-09-02T10:11:39.995873+0000","last_clean":"2024-09-02T10:11:39.995873+0000","last_became_active":"2024-09-02T10:08:25.264766+0000","last_became_peered":"2024-09-02T10:08:25.264766+0000","last_unstale":"2024-09-02T10:11:39.995873+0000","last_undegraded":"2024-09-02T10:11:39.995873+0000","last_fullsized":"2024-09-02T10:11:39.995873+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:55.733125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:55.733125+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:05.466258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"348'2107","reported_seq":2204,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:42.040211+0000","last_change":"2024-09-02T10:11:34.727957+0000","last_active":"2024-09-02T10:11:42.040211+0000","last_peered":"2024-09-02T10:11:42.040211+0000","last_clean":"2024-09-02T10:11:42.040211+0000","last_became_active":"2024-09-02T10:11:32.951191+0000","last_became_peered":"2024-09-02T10:11:32.951191+0000","last_unstale":"2024-09-02T10:11:42.040211+0000","last_undegraded":"2024-09-02T10:11:42.040211+0000","last_fullsized":"2024-09-02T10:11:42.040211+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1937","last_scrub_stamp":"2024-09-02T10:11:34.727882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.727882+0000","objects_scrubbed":936,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:42.143902+0000","scrub_duration":939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72548352,"num_objects":1107,"num_object_clones":0,"num_object_copies":3321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":650,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.998052+0000","last_change":"2024-09-02T10:10:56.874596+0000","last_active":"2024-09-02T10:11:39.998052+0000","last_peered":"2024-09-02T10:11:39.998052+0000","last_clean":"2024-09-02T10:11:39.998052+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:11:39.998052+0000","last_undegraded":"2024-09-02T10:11:39.998052+0000","last_fullsized":"2024-09-02T10:11:39.998052+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:56.874512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.874512+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:17.006393+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"351'2174","reported_seq":2276,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-02T10:11:45.817685+0000","last_change":"2024-09-02T10:11:34.502637+0000","last_active":"2024-09-02T10:11:45.817685+0000","last_peered":"2024-09-02T10:11:45.817685+0000","last_clean":"2024-09-02T10:11:45.817685+0000","last_became_active":"2024-09-02T10:11:32.955679+0000","last_became_peered":"2024-09-02T10:11:32.955679+0000","last_unstale":"2024-09-02T10:11:45.817685+0000","last_undegraded":"2024-09-02T10:11:45.817685+0000","last_fullsized":"2024-09-02T10:11:45.817685+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1865","last_scrub_stamp":"2024-09-02T10:11:34.502539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.502539+0000","objects_scrubbed":937,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:45.346063+0000","scrub_duration":786,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81723392,"num_objects":1247,"num_object_clones":0,"num_object_copies":3741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":310,"num_write_kb":19840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":119,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.997973+0000","last_change":"2024-09-02T10:11:18.921335+0000","last_active":"2024-09-02T10:11:39.997973+0000","last_peered":"2024-09-02T10:11:39.997973+0000","last_clean":"2024-09-02T10:11:39.997973+0000","last_became_active":"2024-09-02T10:11:18.920967+0000","last_became_peered":"2024-09-02T10:11:18.920967+0000","last_unstale":"2024-09-02T10:11:39.997973+0000","last_undegraded":"2024-09-02T10:11:39.997973+0000","last_fullsized":"2024-09-02T10:11:39.997973+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:04.459478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:04.459478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:21.373605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"351'2194","reported_seq":2297,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-02T10:11:45.621323+0000","last_change":"2024-09-02T10:11:34.835252+0000","last_active":"2024-09-02T10:11:45.621323+0000","last_peered":"2024-09-02T10:11:45.621323+0000","last_clean":"2024-09-02T10:11:45.621323+0000","last_became_active":"2024-09-02T10:11:32.955684+0000","last_became_peered":"2024-09-02T10:11:32.955684+0000","last_unstale":"2024-09-02T10:11:45.621323+0000","last_undegraded":"2024-09-02T10:11:45.621323+0000","last_fullsized":"2024-09-02T10:11:45.621323+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1901","last_scrub_stamp":"2024-09-02T10:11:34.835164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.835164+0000","objects_scrubbed":968,"log_size":2194,"log_dups_size":0,"ondisk_log_size":2194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:57.368702+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82640896,"num_objects":1261,"num_object_clones":0,"num_object_copies":3783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":293,"num_write_kb":18752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":112,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.998302+0000","last_change":"2024-09-02T10:09:59.384188+0000","last_active":"2024-09-02T10:11:39.998302+0000","last_peered":"2024-09-02T10:11:39.998302+0000","last_clean":"2024-09-02T10:11:39.998302+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:11:39.998302+0000","last_undegraded":"2024-09-02T10:11:39.998302+0000","last_fullsized":"2024-09-02T10:11:39.998302+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:09:59.384140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:09:59.384140+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"348'2142","reported_seq":2239,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.003152+0000","last_change":"2024-09-02T10:11:33.363660+0000","last_active":"2024-09-02T10:11:43.003152+0000","last_peered":"2024-09-02T10:11:43.003152+0000","last_clean":"2024-09-02T10:11:43.003152+0000","last_became_active":"2024-09-02T10:11:32.953702+0000","last_became_peered":"2024-09-02T10:11:32.953702+0000","last_unstale":"2024-09-02T10:11:43.003152+0000","last_undegraded":"2024-09-02T10:11:43.003152+0000","last_fullsized":"2024-09-02T10:11:43.003152+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1882","last_scrub_stamp":"2024-09-02T10:11:33.363574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:33.363574+0000","objects_scrubbed":965,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:57.111356+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80281600,"num_objects":1225,"num_object_clones":0,"num_object_copies":3675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":260,"num_write_kb":16640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,8],"acting":[9,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":646,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.998715+0000","last_change":"2024-09-02T10:11:00.739069+0000","last_active":"2024-09-02T10:11:39.998715+0000","last_peered":"2024-09-02T10:11:39.998715+0000","last_clean":"2024-09-02T10:11:39.998715+0000","last_became_active":"2024-09-02T10:09:58.912625+0000","last_became_peered":"2024-09-02T10:09:58.912625+0000","last_unstale":"2024-09-02T10:11:39.998715+0000","last_undegraded":"2024-09-02T10:11:39.998715+0000","last_fullsized":"2024-09-02T10:11:39.998715+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:00.739009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:00.739009+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:01.566889+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"348'2146","reported_seq":5945,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:42.507991+0000","last_change":"2024-09-02T10:11:33.267368+0000","last_active":"2024-09-02T10:11:42.507991+0000","last_peered":"2024-09-02T10:11:42.507991+0000","last_clean":"2024-09-02T10:11:42.507991+0000","last_became_active":"2024-09-02T10:11:32.948422+0000","last_became_peered":"2024-09-02T10:11:32.948422+0000","last_unstale":"2024-09-02T10:11:42.507991+0000","last_undegraded":"2024-09-02T10:11:42.507991+0000","last_fullsized":"2024-09-02T10:11:42.507991+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1912","last_scrub_stamp":"2024-09-02T10:11:33.267241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:33.267241+0000","objects_scrubbed":974,"log_size":2146,"log_dups_size":0,"ondisk_log_size":2146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:34.436780+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79167488,"num_objects":1208,"num_object_clones":0,"num_object_copies":3624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":234,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,7],"acting":[2,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":112,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.997610+0000","last_change":"2024-09-02T10:10:59.341544+0000","last_active":"2024-09-02T10:11:39.997610+0000","last_peered":"2024-09-02T10:11:39.997610+0000","last_clean":"2024-09-02T10:11:39.997610+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:11:39.997610+0000","last_undegraded":"2024-09-02T10:11:39.997610+0000","last_fullsized":"2024-09-02T10:11:39.997610+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:59.341491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:59.341491+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:24.333902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"348'2064","reported_seq":3664,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:41.536170+0000","last_change":"2024-09-02T10:11:34.391701+0000","last_active":"2024-09-02T10:11:41.536170+0000","last_peered":"2024-09-02T10:11:41.536170+0000","last_clean":"2024-09-02T10:11:41.536170+0000","last_became_active":"2024-09-02T10:11:32.951628+0000","last_became_peered":"2024-09-02T10:11:32.951628+0000","last_unstale":"2024-09-02T10:11:41.536170+0000","last_undegraded":"2024-09-02T10:11:41.536170+0000","last_fullsized":"2024-09-02T10:11:41.536170+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1881","last_scrub_stamp":"2024-09-02T10:11:34.391635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.391635+0000","objects_scrubbed":926,"log_size":2064,"log_dups_size":0,"ondisk_log_size":2064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:38.303878+0000","scrub_duration":612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72679424,"num_objects":1109,"num_object_clones":0,"num_object_copies":3327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":183,"num_write_kb":11712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":365,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-02T10:11:45.355902+0000","last_change":"2024-09-02T10:10:53.401665+0000","last_active":"2024-09-02T10:11:45.355902+0000","last_peered":"2024-09-02T10:11:45.355902+0000","last_clean":"2024-09-02T10:11:45.355902+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:11:45.355902+0000","last_undegraded":"2024-09-02T10:11:45.355902+0000","last_fullsized":"2024-09-02T10:11:45.355902+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:53.401592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:53.401592+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:02.359527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"348'2117","reported_seq":2215,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:42.844028+0000","last_change":"2024-09-02T10:11:34.358626+0000","last_active":"2024-09-02T10:11:42.844028+0000","last_peered":"2024-09-02T10:11:42.844028+0000","last_clean":"2024-09-02T10:11:42.844028+0000","last_became_active":"2024-09-02T10:11:32.967808+0000","last_became_peered":"2024-09-02T10:11:32.967808+0000","last_unstale":"2024-09-02T10:11:42.844028+0000","last_undegraded":"2024-09-02T10:11:42.844028+0000","last_fullsized":"2024-09-02T10:11:42.844028+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1880","last_scrub_stamp":"2024-09-02T10:11:34.358535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.358535+0000","objects_scrubbed":941,"log_size":2117,"log_dups_size":0,"ondisk_log_size":2117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:55.173674+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77201408,"num_objects":1178,"num_object_clones":0,"num_object_copies":3534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":237,"num_write_kb":15168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":589,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.997425+0000","last_change":"2024-09-02T10:11:18.921403+0000","last_active":"2024-09-02T10:11:39.997425+0000","last_peered":"2024-09-02T10:11:39.997425+0000","last_clean":"2024-09-02T10:11:39.997425+0000","last_became_active":"2024-09-02T10:11:18.920790+0000","last_became_peered":"2024-09-02T10:11:18.920790+0000","last_unstale":"2024-09-02T10:11:39.997425+0000","last_undegraded":"2024-09-02T10:11:39.997425+0000","last_fullsized":"2024-09-02T10:11:39.997425+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.330529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.330529+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:00.786830+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,10],"acting":[1,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"348'2059","reported_seq":3995,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:41.510715+0000","last_change":"2024-09-02T10:11:35.890951+0000","last_active":"2024-09-02T10:11:41.510715+0000","last_peered":"2024-09-02T10:11:41.510715+0000","last_clean":"2024-09-02T10:11:41.510715+0000","last_became_active":"2024-09-02T10:11:34.930403+0000","last_became_peered":"2024-09-02T10:11:34.930403+0000","last_unstale":"2024-09-02T10:11:41.510715+0000","last_undegraded":"2024-09-02T10:11:41.510715+0000","last_fullsized":"2024-09-02T10:11:41.510715+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1863","last_scrub_stamp":"2024-09-02T10:11:35.890838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:35.890838+0000","objects_scrubbed":957,"log_size":2059,"log_dups_size":0,"ondisk_log_size":2059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:02.222800+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75563008,"num_objects":1153,"num_object_clones":0,"num_object_copies":3459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":196,"num_write_kb":12544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0],"acting":[10,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":376,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-02T10:11:45.355328+0000","last_change":"2024-09-02T10:11:11.468234+0000","last_active":"2024-09-02T10:11:45.355328+0000","last_peered":"2024-09-02T10:11:45.355328+0000","last_clean":"2024-09-02T10:11:45.355328+0000","last_became_active":"2024-09-02T10:11:11.467851+0000","last_became_peered":"2024-09-02T10:11:11.467851+0000","last_unstale":"2024-09-02T10:11:45.355328+0000","last_undegraded":"2024-09-02T10:11:45.355328+0000","last_fullsized":"2024-09-02T10:11:45.355328+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.437492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.437492+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:11:57.835987+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"348'2373","reported_seq":4748,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.257381+0000","last_change":"2024-09-02T10:11:29.940818+0000","last_active":"2024-09-02T10:11:43.257381+0000","last_peered":"2024-09-02T10:11:43.257381+0000","last_clean":"2024-09-02T10:11:43.257381+0000","last_became_active":"2024-09-02T10:11:29.938756+0000","last_became_peered":"2024-09-02T10:11:29.938756+0000","last_unstale":"2024-09-02T10:11:43.257381+0000","last_undegraded":"2024-09-02T10:11:43.257381+0000","last_fullsized":"2024-09-02T10:11:43.257381+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2373,"log_dups_size":0,"ondisk_log_size":2373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:25.199650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155516928,"num_objects":2373,"num_object_clones":0,"num_object_copies":7119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2373,"num_write_kb":151872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1146,"num_bytes_recovered":75104256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,7],"acting":[1,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":610,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.998618+0000","last_change":"2024-09-02T10:11:11.467461+0000","last_active":"2024-09-02T10:11:39.998618+0000","last_peered":"2024-09-02T10:11:39.998618+0000","last_clean":"2024-09-02T10:11:39.998618+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:11:39.998618+0000","last_undegraded":"2024-09-02T10:11:39.998618+0000","last_fullsized":"2024-09-02T10:11:39.998618+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:58.678413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:58.678413+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:05.764576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"348'2363","reported_seq":4791,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:42.516915+0000","last_change":"2024-09-02T10:11:28.950121+0000","last_active":"2024-09-02T10:11:42.516915+0000","last_peered":"2024-09-02T10:11:42.516915+0000","last_clean":"2024-09-02T10:11:42.516915+0000","last_became_active":"2024-09-02T10:11:28.936098+0000","last_became_peered":"2024-09-02T10:11:28.936098+0000","last_unstale":"2024-09-02T10:11:42.516915+0000","last_undegraded":"2024-09-02T10:11:42.516915+0000","last_fullsized":"2024-09-02T10:11:42.516915+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":2363,"log_dups_size":0,"ondisk_log_size":2363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:21.614086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154861568,"num_objects":2363,"num_object_clones":0,"num_object_copies":7089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2363,"num_write_kb":151232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1171,"num_bytes_recovered":76742656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":158,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-02T10:11:45.356260+0000","last_change":"2024-09-02T10:08:26.088128+0000","last_active":"2024-09-02T10:11:45.356260+0000","last_peered":"2024-09-02T10:11:45.356260+0000","last_clean":"2024-09-02T10:11:45.356260+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:11:45.356260+0000","last_undegraded":"2024-09-02T10:11:45.356260+0000","last_fullsized":"2024-09-02T10:11:45.356260+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:26.088081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:26.088081+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"348'2172","reported_seq":4741,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:42.026334+0000","last_change":"2024-09-02T10:11:34.867424+0000","last_active":"2024-09-02T10:11:42.026334+0000","last_peered":"2024-09-02T10:11:42.026334+0000","last_clean":"2024-09-02T10:11:42.026334+0000","last_became_active":"2024-09-02T10:11:32.955100+0000","last_became_peered":"2024-09-02T10:11:32.955100+0000","last_unstale":"2024-09-02T10:11:42.026334+0000","last_undegraded":"2024-09-02T10:11:42.026334+0000","last_fullsized":"2024-09-02T10:11:42.026334+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1959","last_scrub_stamp":"2024-09-02T10:11:34.867321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.867321+0000","objects_scrubbed":963,"log_size":2172,"log_dups_size":0,"ondisk_log_size":2172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:54.964856+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77070336,"num_objects":1176,"num_object_clones":0,"num_object_copies":3528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":213,"num_write_kb":13632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":390,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-02T10:11:45.356324+0000","last_change":"2024-09-02T10:11:01.465118+0000","last_active":"2024-09-02T10:11:45.356324+0000","last_peered":"2024-09-02T10:11:45.356324+0000","last_clean":"2024-09-02T10:11:45.356324+0000","last_became_active":"2024-09-02T10:11:00.404482+0000","last_became_peered":"2024-09-02T10:11:00.404482+0000","last_unstale":"2024-09-02T10:11:45.356324+0000","last_undegraded":"2024-09-02T10:11:45.356324+0000","last_fullsized":"2024-09-02T10:11:45.356324+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:01.465055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:01.465055+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:22.934648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"348'2102","reported_seq":4468,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:43.142533+0000","last_change":"2024-09-02T10:11:33.677020+0000","last_active":"2024-09-02T10:11:43.142533+0000","last_peered":"2024-09-02T10:11:43.142533+0000","last_clean":"2024-09-02T10:11:43.142533+0000","last_became_active":"2024-09-02T10:11:32.951840+0000","last_became_peered":"2024-09-02T10:11:32.951840+0000","last_unstale":"2024-09-02T10:11:43.142533+0000","last_undegraded":"2024-09-02T10:11:43.142533+0000","last_fullsized":"2024-09-02T10:11:43.142533+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1887","last_scrub_stamp":"2024-09-02T10:11:33.676961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:33.676961+0000","objects_scrubbed":905,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:57.486731+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73400320,"num_objects":1120,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":215,"num_write_kb":13760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":149,"reported_epoch":348,"state":"active+clean","last_fresh":"2024-09-02T10:11:39.994811+0000","last_change":"2024-09-02T10:11:27.692069+0000","last_active":"2024-09-02T10:11:39.994811+0000","last_peered":"2024-09-02T10:11:39.994811+0000","last_clean":"2024-09-02T10:11:39.994811+0000","last_became_active":"2024-09-02T10:11:11.472186+0000","last_became_peered":"2024-09-02T10:11:11.472186+0000","last_unstale":"2024-09-02T10:11:39.994811+0000","last_undegraded":"2024-09-02T10:11:39.994811+0000","last_fullsized":"2024-09-02T10:11:39.994811+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:27.692014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:27.692014+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:31.198340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":2493874176,"num_objects":38056,"num_object_clones":0,"num_object_copies":114168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1912,"num_objects_misplaced":9506,"num_objects_unfound":0,"num_objects_dirty":38056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28584,"num_write_kb":1829216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11421,"num_bytes_recovered":748421120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8358002688,"data_stored":8358002688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56976,"ondisk_log_size":56976,"up":78,"acting":69,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383819,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":748744,"kb_used_data":420468,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92996408,"statfs":{"total":95995035648,"available":95228321792,"internally_reserved":0,"allocated":430559232,"data_stored":430173975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":336132548},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612752,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799364,"kb_used_data":736244,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92945788,"statfs":{"total":95995035648,"available":95176486912,"internally_reserved":0,"allocated":753913856,"data_stored":753528599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":64616896},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841685,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1279492,"kb_used_data":1013684,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92465660,"statfs":{"total":95995035648,"available":94684835840,"internally_reserved":0,"allocated":1038012416,"data_stored":1037627159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":272169408},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070619,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":615188,"kb_used_data":493760,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93129964,"statfs":{"total":95995035648,"available":95365083136,"internally_reserved":0,"allocated":505610240,"data_stored":505220681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299551,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1370804,"kb_used_data":1150816,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":92374348,"statfs":{"total":95995035648,"available":94591332352,"internally_reserved":0,"allocated":1178435584,"data_stored":1178033819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":225245642},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528483,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":655748,"kb_used_data":494068,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93089404,"statfs":{"total":95995035648,"available":95323549696,"internally_reserved":0,"allocated":505925632,"data_stored":505540375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":165542343},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724711,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491024,"kb_used_data":315452,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":93254128,"statfs":{"total":95995035648,"available":95492227072,"internally_reserved":0,"allocated":323022848,"data_stored":322634039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953645,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":718032,"kb_used_data":538304,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93027120,"statfs":{"total":95995035648,"available":95259770880,"internally_reserved":0,"allocated":551223296,"data_stored":550833737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":184023493},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149875,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":930180,"kb_used_data":614388,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92814972,"statfs":{"total":95995035648,"available":95042531328,"internally_reserved":0,"allocated":629133312,"data_stored":628748055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":323353032},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411510,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":626384,"kb_used_data":424444,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":93118768,"statfs":{"total":95995035648,"available":95353618432,"internally_reserved":0,"allocated":434630656,"data_stored":434241847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575036,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1403908,"kb_used_data":1188084,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92341244,"statfs":{"total":95995035648,"available":94557433856,"internally_reserved":0,"allocated":1216598016,"data_stored":1216212759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738561,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1145348,"kb_used_data":889268,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":92599804,"statfs":{"total":95995035648,"available":94822199296,"internally_reserved":0,"allocated":910610432,"data_stored":910225175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":262207941},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":939917312,"data_stored":939917312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1241513984,"data_stored":1241513984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":449118208,"data_stored":449118208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":652410880,"data_stored":652410880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":254476288,"data_stored":254476288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":326828032,"data_stored":326828032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":516816896,"data_stored":516816896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1182269440,"data_stored":1182269440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":511705088,"data_stored":511705088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1074528256,"data_stored":1074528256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":775880704,"data_stored":775880704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":432537600,"data_stored":432537600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:11:49.019 INFO:tasks.thrashosds.thrasher:Setting 1.1e to [11, 10, 9] 2024-09-02T10:11:49.019 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.1e', '11', '10', '9'] 2024-09-02T10:11:49.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap 1.1e 11 10 9 2024-09-02T10:11:49.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:49.068 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:49.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:49.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:49.190 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:49.191 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:49.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:49.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:49.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:49.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:49.356 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:49.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:49 smithi151 ceph-mon[16639]: osdmap e354: 12 total, 12 up, 12 in 2024-09-02T10:11:49.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:49 smithi151 ceph-mon[16639]: 4.15 scrub starts 2024-09-02T10:11:49.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:49 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1912/116559 objects degraded (1.640%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:49.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:49 smithi151 ceph-mon[16639]: osdmap e355: 12 total, 12 up, 12 in 2024-09-02T10:11:49.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:49.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:49.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:49.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:49.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 46 16 42799 42783 58.1242 41.625 0.0017551 0.0171963 2024-09-02T10:11:49.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:11:49.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:49.580 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:49.581 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:49.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:49.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:49.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:49.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:49 smithi187 ceph-mon[20072]: osdmap e354: 12 total, 12 up, 12 in 2024-09-02T10:11:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:49 smithi187 ceph-mon[20072]: 4.15 scrub starts 2024-09-02T10:11:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:49 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1912/116559 objects degraded (1.640%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:49 smithi187 ceph-mon[20072]: osdmap e355: 12 total, 12 up, 12 in 2024-09-02T10:11:49.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:49.831 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:49.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:49.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:49 smithi162 ceph-mon[20279]: osdmap e354: 12 total, 12 up, 12 in 2024-09-02T10:11:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:49 smithi162 ceph-mon[20279]: 4.15 scrub starts 2024-09-02T10:11:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:49 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1912/116559 objects degraded (1.640%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:11:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:49 smithi162 ceph-mon[20279]: osdmap e355: 12 total, 12 up, 12 in 2024-09-02T10:11:49.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:49.954 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:49.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:49.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:50.007 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:11:50.007 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:11:50.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:42418 4.e e355/340 4:7b120e8c:::benchmark_data_smithi187_76542_object42417:head v 355'2704, mlcod=355'2701)", 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:49.127349+0000", 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.014989794000000001, 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.015007983000000001, 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.127349+0000", 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:50.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.127349+0000", 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.127354+0000", 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1980000000000003e-06 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.127400+0000", 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6275e-05 2024-09-02T10:11:50.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.141601+0000", 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014201213000000001 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.141608+0000", 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9240000000000001e-06 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.141665+0000", 2024-09-02T10:11:50.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6609e-05 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.141686+0000", 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1705999999999999e-05 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:11:50.011 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:11:50.056 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:50.082 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:50.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:50.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:50.205 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:11:50.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:50.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:50.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:50.333 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:50.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:50.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:50.381 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.1e pg_upmap mapping to [11,10,9] 2024-09-02T10:11:50.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:50.458 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:50.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:50.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 47 16 44071 44055 58.5789 79.5 0.0142495 0.0170528 2024-09-02T10:11:50.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:11:50.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:50.581 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:50.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:50.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:50.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:50 smithi151 ceph-mon[16639]: 4.19 scrub starts 2024-09-02T10:11:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:50 smithi151 ceph-mon[16639]: 4.10 scrub starts 2024-09-02T10:11:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:50 smithi151 ceph-mon[16639]: pgmap v860: 57 pgs: 1 creating+peering, 2 active+undersized+remapped, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 980 op/s; 5712/123816 objects misplaced (4.613%); 49 MiB/s, 777 objects/s recovering 2024-09-02T10:11:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:50 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1912/116559 objects degraded (1.640%), 1 pg degraded) 2024-09-02T10:11:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1469384731' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.1e", "id": [11, 10, 9]} : dispatch 2024-09-02T10:11:50.705 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:50.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:50 smithi187 ceph-mon[20072]: 4.19 scrub starts 2024-09-02T10:11:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:50 smithi187 ceph-mon[20072]: 4.10 scrub starts 2024-09-02T10:11:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:50 smithi187 ceph-mon[20072]: pgmap v860: 57 pgs: 1 creating+peering, 2 active+undersized+remapped, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 980 op/s; 5712/123816 objects misplaced (4.613%); 49 MiB/s, 777 objects/s recovering 2024-09-02T10:11:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:50 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1912/116559 objects degraded (1.640%), 1 pg degraded) 2024-09-02T10:11:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1469384731' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.1e", "id": [11, 10, 9]} : dispatch 2024-09-02T10:11:50.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:50.829 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:50.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:50.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:50.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:50 smithi162 ceph-mon[20279]: 4.19 scrub starts 2024-09-02T10:11:50.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:50 smithi162 ceph-mon[20279]: 4.10 scrub starts 2024-09-02T10:11:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:50 smithi162 ceph-mon[20279]: pgmap v860: 57 pgs: 1 creating+peering, 2 active+undersized+remapped, 4 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 49 active+clean; 2.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 980 op/s; 5712/123816 objects misplaced (4.613%); 49 MiB/s, 777 objects/s recovering 2024-09-02T10:11:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:50 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1912/116559 objects degraded (1.640%), 1 pg degraded) 2024-09-02T10:11:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1469384731' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.1e", "id": [11, 10, 9]} : dispatch 2024-09-02T10:11:50.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:50.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:50.952 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:51.053 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:51.074 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:51.076 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:51.177 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:51.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:51.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:51.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:51.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:51.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:51.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:51.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:51.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:51.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:51.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:51.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 48 16 44392 44376 57.7764 20.0625 0.0204117 0.017302 2024-09-02T10:11:51.547 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:11:51.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:51.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:51.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:51 smithi151 ceph-mon[16639]: 4.12 scrub starts 2024-09-02T10:11:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1469384731' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1e", "id": [11, 10, 9]}]': finished 2024-09-02T10:11:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:51 smithi151 ceph-mon[16639]: osdmap e356: 12 total, 12 up, 12 in 2024-09-02T10:11:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:51 smithi151 ceph-mon[16639]: osdmap e357: 12 total, 12 up, 12 in 2024-09-02T10:11:51.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:51.749 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:51.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:51 smithi187 ceph-mon[20072]: 4.12 scrub starts 2024-09-02T10:11:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1469384731' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1e", "id": [11, 10, 9]}]': finished 2024-09-02T10:11:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:51 smithi187 ceph-mon[20072]: osdmap e356: 12 total, 12 up, 12 in 2024-09-02T10:11:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:51 smithi187 ceph-mon[20072]: osdmap e357: 12 total, 12 up, 12 in 2024-09-02T10:11:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:51 smithi162 ceph-mon[20279]: 4.12 scrub starts 2024-09-02T10:11:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1469384731' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.1e", "id": [11, 10, 9]}]': finished 2024-09-02T10:11:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:51 smithi162 ceph-mon[20279]: osdmap e356: 12 total, 12 up, 12 in 2024-09-02T10:11:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:51 smithi162 ceph-mon[20279]: osdmap e357: 12 total, 12 up, 12 in 2024-09-02T10:11:51.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:51.876 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:51.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:51.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:51.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:51.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:52.017 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:52.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:52.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:52.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:52.145 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:52.145 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:52.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:52.269 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:52.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:52.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:52.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:52.398 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:52.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:52.399 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:52.399 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:11:52.399 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:11:52.399 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:11:52.399 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:11:52.399 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:11:52.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:52.505 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 49 16 45726 45710 58.2987 83.375 0.0110947 0.0171439 2024-09-02T10:11:52.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:52.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:52.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:52.648 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:52.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:52.668 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:52 smithi187 ceph-mon[20072]: 4.13 scrub starts 2024-09-02T10:11:52.668 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:52 smithi187 ceph-mon[20072]: pgmap v862: 57 pgs: 1 creating+peering, 2 active+undersized+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 50 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 759 op/s; 5618/125679 objects misplaced (4.470%); 30 MiB/s, 483 objects/s recovering 2024-09-02T10:11:52.668 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:52 smithi187 ceph-mon[20072]: 4.18 scrub starts 2024-09-02T10:11:52.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:52 smithi151 ceph-mon[16639]: 4.13 scrub starts 2024-09-02T10:11:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:52 smithi151 ceph-mon[16639]: pgmap v862: 57 pgs: 1 creating+peering, 2 active+undersized+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 50 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 759 op/s; 5618/125679 objects misplaced (4.470%); 30 MiB/s, 483 objects/s recovering 2024-09-02T10:11:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:52 smithi151 ceph-mon[16639]: 4.18 scrub starts 2024-09-02T10:11:52.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:52 smithi162 ceph-mon[20279]: 4.13 scrub starts 2024-09-02T10:11:52.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:52 smithi162 ceph-mon[20279]: pgmap v862: 57 pgs: 1 creating+peering, 2 active+undersized+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 50 active+clean; 2.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 759 op/s; 5618/125679 objects misplaced (4.470%); 30 MiB/s, 483 objects/s recovering 2024-09-02T10:11:52.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:52 smithi162 ceph-mon[20279]: 4.18 scrub starts 2024-09-02T10:11:52.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:52.779 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:52.780 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:52.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:52.914 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:52.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:53.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:53.040 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:53.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:53.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:53.164 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:53.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:53.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:53.294 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:53.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:53.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:53.421 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:53.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:53.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:53.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:11:53.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 50 16 46849 46833 58.5364 70.1875 0.00457582 0.0170771 2024-09-02T10:11:53.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:53.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:53.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:53.632 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:11:53.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:53 smithi151 ceph-mon[16639]: 4.14 scrub starts 2024-09-02T10:11:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:53 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:11:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:53 smithi151 ceph-mon[16639]: 4.11 scrub starts 2024-09-02T10:11:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3800377243' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:11:53.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:53.692 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:53 smithi187 ceph-mon[20072]: 4.14 scrub starts 2024-09-02T10:11:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:53 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:11:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:53 smithi187 ceph-mon[20072]: 4.11 scrub starts 2024-09-02T10:11:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3800377243' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:11:53.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:53.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:53.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:53.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:53.831 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:53.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:53 smithi162 ceph-mon[20279]: 4.14 scrub starts 2024-09-02T10:11:53.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:53 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:11:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:53 smithi162 ceph-mon[20279]: 4.11 scrub starts 2024-09-02T10:11:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3800377243' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:11:53.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:53.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:53.952 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:53.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:54.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:54.077 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:54.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:54.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:54.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:54.208 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:54.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:54.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:54.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:54.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:54.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:54.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:54.409 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:11:54.409 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:11:54.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:11:54.409 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:11:54.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.409 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:78685 2.4 e124/92 2:26d5ebbe:::benchmark_data_smithi187_45882_object78684:head v 124'4867, mlcod=124'4864)", 2024-09-02T10:11:54.409 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:11:54.409 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 553.95798289899994, 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.135791625, 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448494+0000", 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448496+0000", 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.257e-06 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.448603+0000", 2024-09-02T10:11:54.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106925 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.581994+0000", 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13339121900000001 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582001+0000", 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7950000000000002e-06 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582057+0000", 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6724000000000001e-05 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.582074+0000", 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7037999999999998e-05 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.584242+0000", 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021673809999999999 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.584286+0000", 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4286000000000003e-05 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:79093 2.4 e125/92 2:2f2be3a1:::benchmark_data_smithi187_45882_object79092:head v 125'4892, mlcod=125'4888)", 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:11:54.414 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 553.45354992900002, 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16453667699999999, 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952927+0000", 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952928+0000", 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:40.952991+0000", 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3367999999999994e-05 2024-09-02T10:11:54.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115366+0000", 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16237467799999999 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115374+0000", 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6240000000000002e-06 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115428+0000", 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3885999999999999e-05 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.115442+0000", 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4438e-05 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.117418+0000", 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019764600000000002 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:41.117464+0000", 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5694e-05 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:3929 4.5 e324/318 4:af3910d4:::benchmark_data_smithi187_76542_object3928:head v 324'255, mlcod=324'252)", 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:05.948100+0000", 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 48.458376966000003, 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18627370900000001, 2024-09-02T10:11:54.419 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:05.948100+0000", 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:05.948100+0000", 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:05.948101+0000", 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4799999999999997e-07 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:05.948156+0000", 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5145999999999997e-05 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.133889+0000", 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.185732598 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.133894+0000", 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8729999999999998e-06 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.133939+0000", 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5148999999999998e-05 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.133954+0000", 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5067e-05 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.134338+0000", 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00038381500000000002 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.134374+0000", 2024-09-02T10:11:54.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6313000000000003e-05 2024-09-02T10:11:54.424 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.424 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.424 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.424 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:4347 4.e e324/318 4:78dcb871:::benchmark_data_smithi187_76542_object4346:head v 324'265, mlcod=324'262)", 2024-09-02T10:11:54.424 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:06.332751+0000", 2024-09-02T10:11:54.424 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 48.073726354000001, 2024-09-02T10:11:54.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.128106001, 2024-09-02T10:11:54.425 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.332751+0000", 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.332751+0000", 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.332752+0000", 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7800000000000001e-07 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.332800+0000", 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8782000000000002e-05 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.460430+0000", 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12762952599999999 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.460435+0000", 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0150000000000003e-06 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.460477+0000", 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2163999999999998e-05 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.460491+0000", 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3855e-05 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.460822+0000", 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00033122400000000003 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.460857+0000", 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4857000000000002e-05 2024-09-02T10:11:54.429 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:20287 4.a e334/333 4:5a358598:::benchmark_data_smithi187_76542_object20286:head v 334'1255, mlcod=324'144)", 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:21.271947+0000", 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 33.134529899, 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13754444399999999, 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.430 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.271947+0000", 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.271947+0000", 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.271949+0000", 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1379999999999999e-06 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.271990+0000", 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1248000000000002e-05 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.408497+0000", 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13650664900000001 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.408504+0000", 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0260000000000002e-06 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.408565+0000", 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1915000000000006e-05 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.408595+0000", 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9258999999999999e-05 2024-09-02T10:11:54.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.409455+0000", 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086051300000000003 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.409492+0000", 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6696000000000002e-05 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.434 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:20297 4.4 e334/333 4:21d0dd1e:::benchmark_data_smithi187_76542_object20296:head v 334'1271, mlcod=324'122)", 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:21.281157+0000", 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 33.125320315000003, 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.130660207, 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.281157+0000", 2024-09-02T10:11:54.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.281157+0000", 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.281162+0000", 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5020000000000003e-06 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.281205+0000", 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3914e-05 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.411087+0000", 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.129881675 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.411092+0000", 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1610000000000002e-06 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.411142+0000", 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0158000000000003e-05 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.411162+0000", 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9924000000000001e-05 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.411775+0000", 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00061285500000000004 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.411817+0000", 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2018000000000003e-05 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:20625 4.4 e334/333 4:2eb01bf7:::benchmark_data_smithi187_76542_object20624:head v 334'1292, mlcod=324'138)", 2024-09-02T10:11:54.439 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:21.629995+0000", 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 32.776482784000002, 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12886773000000001, 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.629995+0000", 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.629995+0000", 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.629995+0000", 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9999999999999997e-07 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.630053+0000", 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8026999999999999e-05 2024-09-02T10:11:54.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.757691+0000", 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12763792700000001 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.757699+0000", 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5050000000000001e-06 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.757765+0000", 2024-09-02T10:11:54.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6573000000000001e-05 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.757799+0000", 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3637000000000003e-05 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.758818+0000", 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010188910000000001 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:21.758862+0000", 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4570000000000002e-05 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:21294 4.5 4:a77738c3:::benchmark_data_smithi187_76542_object21293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:22.244803+0000", 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 32.161674634999997, 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13570411700000001, 2024-09-02T10:11:54.444 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 21294 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.244803+0000", 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.244803+0000", 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.244804+0000", 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1510000000000001e-06 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.244823+0000", 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9014000000000001e-05 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.244835+0000", 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1593000000000001e-05 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.244840+0000", 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4269999999999999e-06 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.244899+0000", 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8674000000000001e-05 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.244994+0000", 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5084e-05 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:54.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.245093+0000", 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9302999999999994e-05 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.245852+0000", 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00075946900000000003 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.350175+0000", 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.104322515 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.379661+0000", 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.029486500999999998 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.379709+0000", 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7926999999999997e-05 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:22.380507+0000", 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00079745900000000004 2024-09-02T10:11:54.450 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:22571 4.7 4:e4d9c144:::benchmark_data_smithi187_76542_object22570:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:23.374541+0000", 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 31.031935922999999, 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.1315769, 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:54.451 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 22571 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.374541+0000", 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.374541+0000", 2024-09-02T10:11:54.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.374543+0000", 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.37e-06 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.374566+0000", 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3553999999999998e-05 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.374578+0000", 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1924e-05 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.374585+0000", 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.331e-06 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.374627+0000", 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2645999999999998e-05 2024-09-02T10:11:54.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.374717+0000", 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9788000000000002e-05 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.374805+0000", 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7992999999999996e-05 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:54.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.375779+0000", 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00097391700000000003 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.378290+0000", 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025106410000000001 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.506048+0000", 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12775883699999999 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.506083+0000", 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4279999999999997e-05 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.506118+0000", 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5618999999999999e-05 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.457 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:22572 4.5 4:a168cbef:::benchmark_data_smithi187_76542_object22571:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:23.376429+0000", 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 31.030047939999999, 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.132586491, 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 22572 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.376429+0000", 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.376429+0000", 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.376430+0000", 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0079999999999999e-06 2024-09-02T10:11:54.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.376453+0000", 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2637999999999999e-05 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.376466+0000", 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3191e-05 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.376473+0000", 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2089999999999999e-06 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.376742+0000", 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00026842100000000002 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.376839+0000", 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7527000000000003e-05 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.376994+0000", 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015475800000000001 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.377815+0000", 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00082049 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.505464+0000", 2024-09-02T10:11:54.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.127649606 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.508102+0000", 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026377390000000001 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.508167+0000", 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4880000000000002e-05 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.509016+0000", 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00084902399999999998 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25507.0:22571 4.7 e334/330 ondisk, result = 0)", 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:23.378059+0000", 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 31.028418883000001, 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.128065664, 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.464 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.378059+0000", 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.378059+0000", 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.378059+0000", 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.23e-07 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.378065+0000", 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7969999999999997e-06 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.505972+0000", 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12790695499999999 2024-09-02T10:11:54.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.505979+0000", 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6239999999999996e-06 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.506038+0000", 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9654999999999997e-05 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.506045+0000", 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.268e-06 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.506124+0000", 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9542000000000005e-05 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25507.0:22572 4.5 e334/333 ondisk, result = 0)", 2024-09-02T10:11:54.468 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:23.378120+0000", 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 31.028356939999998, 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.127354623, 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.378120+0000", 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.378120+0000", 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.378121+0000", 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8700000000000001e-07 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.378124+0000", 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7870000000000002e-06 2024-09-02T10:11:54.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.505386+0000", 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.127262666 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.505391+0000", 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9400000000000001e-06 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.505445+0000", 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3380000000000001e-05 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.505455+0000", 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0838e-05 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:23.505475+0000", 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9624999999999999e-05 2024-09-02T10:11:54.472 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:25387 4.a e336/333 4:51410dd3:::benchmark_data_smithi187_76542_object25386:head v 336'1582, mlcod=325'647)", 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 28.442234622000001, 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20680125099999999, 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964244+0000", 2024-09-02T10:11:54.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5900000000000005e-07 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964284+0000", 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0487e-05 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169804+0000", 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20551941600000001 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169809+0000", 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6319999999999996e-06 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169867+0000", 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7938e-05 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169895+0000", 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7430999999999999e-05 2024-09-02T10:11:54.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.171002+0000", 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011072790000000001 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.171044+0000", 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2109e-05 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:25663 4.a e336/333 4:5a73eb61:::benchmark_data_smithi187_76542_object25662:head v 336'1596, mlcod=325'705)", 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:26.320475+0000", 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 28.086002169, 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14677976700000001, 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.320475+0000", 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.320475+0000", 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.320476+0000", 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0699999999999997e-07 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.320547+0000", 2024-09-02T10:11:54.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0920000000000005e-05 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.465845+0000", 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14529815500000001 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.465852+0000", 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7270000000000001e-06 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.465900+0000", 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8803e-05 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.465928+0000", 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7869999999999999e-05 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.467206+0000", 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001277381 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.467255+0000", 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9404000000000003e-05 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:32958 4.5 4:a717a701:::benchmark_data_smithi187_76542_object32957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:38.731478+0000", 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.674999746999999, 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.153665837, 2024-09-02T10:11:54.482 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 32958 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731478+0000", 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731478+0000", 2024-09-02T10:11:54.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731479+0000", 2024-09-02T10:11:54.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.795e-06 2024-09-02T10:11:54.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:11:54.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731505+0000", 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5527000000000001e-05 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731523+0000", 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7546000000000001e-05 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731531+0000", 2024-09-02T10:11:54.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7239999999999998e-06 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731591+0000", 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0154e-05 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731694+0000", 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010283 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:54.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731796+0000", 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010157699999999999 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.732584+0000", 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000787947 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.733556+0000", 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000972152 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.885024+0000", 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15146827700000001 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.885092+0000", 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7997999999999995e-05 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.885143+0000", 2024-09-02T10:11:54.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1310000000000002e-05 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:32978 4.7 4:e2ead9b4:::benchmark_data_smithi187_76542_object32977:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:38.758642+0000", 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 15.647835719, 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12652171100000001, 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:54.490 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 32978 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.758642+0000", 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.758642+0000", 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.758643+0000", 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8279999999999999e-06 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.758668+0000", 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4669e-05 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.758686+0000", 2024-09-02T10:11:54.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8232000000000001e-05 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.758695+0000", 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4509999999999992e-06 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.758756+0000", 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1459999999999998e-05 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.758898+0000", 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014125800000000001 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.759037+0000", 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000139309 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.759933+0000", 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00089586199999999998 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.760307+0000", 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00037460599999999998 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.885068+0000", 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.124760339 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.885121+0000", 2024-09-02T10:11:54.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3560000000000002e-05 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.885163+0000", 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2137e-05 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:43771 4.7 4:e2758974:::benchmark_data_smithi187_76542_object43770:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-02T10:11:54.496 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:50.237599+0000", 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 4.1688788629999998, 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.121042658, 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 43771 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.237599+0000", 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.237599+0000", 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.237600+0000", 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.229e-06 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.237625+0000", 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5065000000000001e-05 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.237637+0000", 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.183e-05 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.237644+0000", 2024-09-02T10:11:54.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6199999999999999e-06 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.237697+0000", 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3204000000000001e-05 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.237809+0000", 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111811 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.237937+0000", 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012815 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.238820+0000", 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00088280800000000001 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.241644+0000", 2024-09-02T10:11:54.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002823281 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.358539+0000", 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.116895951 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.358602+0000", 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2402000000000003e-05 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.358641+0000", 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9307000000000002e-05 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:44087 4.7 4:e44e1657:::benchmark_data_smithi187_76542_object44086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:50.597202+0000", 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 3.8092758230000001, 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20155887, 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.503 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 44087 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597202+0000", 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597202+0000", 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597203+0000", 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1969999999999999e-06 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597224+0000", 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1421999999999999e-05 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597236+0000", 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2155e-05 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597244+0000", 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6410000000000002e-06 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597293+0000", 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8990999999999999e-05 2024-09-02T10:11:54.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597387+0000", 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3700000000000001e-05 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597478+0000", 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1479999999999998e-05 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:54.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.598211+0000", 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00073308599999999998 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.598723+0000", 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051141900000000004 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.798694+0000", 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19997105900000001 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.798730+0000", 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6631999999999998e-05 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.798760+0000", 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0088000000000001e-05 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.509 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:45746 4.7 4:e2330a36:::benchmark_data_smithi187_76542_object45745:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:52.553916+0000", 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 1.8525617620000001, 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13671781399999999, 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 45746 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.553916+0000", 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.553916+0000", 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.553917+0000", 2024-09-02T10:11:54.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.483e-06 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.553938+0000", 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0854000000000001e-05 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.553953+0000", 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5478999999999999e-05 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.553960+0000", 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1099999999999999e-06 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.554017+0000", 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7404000000000001e-05 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.554118+0000", 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010097799999999999 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.513 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.554209+0000", 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0830999999999998e-05 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.555019+0000", 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00081046999999999998 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.661717+0000", 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.106697418 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.690569+0000", 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.028852802 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.690604+0000", 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4900999999999997e-05 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.690633+0000", 2024-09-02T10:11:54.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9084000000000001e-05 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:45764 4.5 4:a04d9e22:::benchmark_data_smithi187_76542_object45763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e357)", 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:52.578372+0000", 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 1.82810575, 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13083120300000001, 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:11:54.516 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 45764 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.578372+0000", 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.578372+0000", 2024-09-02T10:11:54.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.578373+0000", 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5489999999999999e-06 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.578393+0000", 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9649000000000001e-05 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.578408+0000", 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4931e-05 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.578416+0000", 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8229999999999995e-06 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.578466+0000", 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0435000000000001e-05 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.578542+0000", 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6229999999999994e-05 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.578621+0000", 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8343000000000005e-05 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:11:54.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.579621+0000", 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010004199999999999 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.708696+0000", 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.129075211 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.709132+0000", 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00043585899999999998 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.709169+0000", 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6489999999999998e-05 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:52.709203+0000", 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4263e-05 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:54.522 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:11:54.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:54 smithi187 ceph-mon[20072]: pgmap v864: 57 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 52 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 421 B/s rd, 35 MiB/s wr, 556 op/s; 5618/131283 objects misplaced (4.279%); 1.7 MiB/s, 27 objects/s recovering 2024-09-02T10:11:54.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:54 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:54.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3800377243' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:11:54.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:54 smithi187 ceph-mon[20072]: osdmap e358: 12 total, 12 up, 12 in 2024-09-02T10:11:54.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:54 smithi187 ceph-mon[20072]: osdmap e359: 12 total, 12 up, 12 in 2024-09-02T10:11:54.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:54 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:11:54.523 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:54 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:11:54.524 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 51 16 48043 48027 58.8517 74.625 0.00269189 0.0169685 2024-09-02T10:11:54.551 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:54.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:54.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:11:54.653 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:54.679 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:54.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:54.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:54.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:54 smithi151 ceph-mon[16639]: pgmap v864: 57 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 52 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 421 B/s rd, 35 MiB/s wr, 556 op/s; 5618/131283 objects misplaced (4.279%); 1.7 MiB/s, 27 objects/s recovering 2024-09-02T10:11:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:54 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3800377243' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:11:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:54 smithi151 ceph-mon[16639]: osdmap e358: 12 total, 12 up, 12 in 2024-09-02T10:11:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:54 smithi151 ceph-mon[16639]: osdmap e359: 12 total, 12 up, 12 in 2024-09-02T10:11:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:54 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:11:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:54 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:11:54.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:54.805 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:54.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:54 smithi162 ceph-mon[20279]: pgmap v864: 57 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 52 active+clean; 2.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 421 B/s rd, 35 MiB/s wr, 556 op/s; 5618/131283 objects misplaced (4.279%); 1.7 MiB/s, 27 objects/s recovering 2024-09-02T10:11:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:54 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:11:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3800377243' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:11:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:54 smithi162 ceph-mon[20279]: osdmap e358: 12 total, 12 up, 12 in 2024-09-02T10:11:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:54 smithi162 ceph-mon[20279]: osdmap e359: 12 total, 12 up, 12 in 2024-09-02T10:11:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:54 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:11:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:54 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:11:54.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:54.929 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:54.930 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:55.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:55.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:55.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:55.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:55.190 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:55.192 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:55.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:55.319 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:55.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:55.320 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:55.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:55.445 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:55.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:55.446 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:55.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 52 16 49325 49309 59.2607 80.125 0.00274619 0.0168649 2024-09-02T10:11:55.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:11:55.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:55.567 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:55.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:55.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:55.691 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:55.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:55.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:55.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:55.794 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:55.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:55.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:55.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:55.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:55.950 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:55.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:56.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:56.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:56.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:56.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:56.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:56.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:56.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:56.219 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:55 smithi187 ceph-mon[20072]: 4.12 scrub starts 2024-09-02T10:11:56.219 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:55 smithi187 ceph-mon[20072]: 4.15 scrub starts 2024-09-02T10:11:56.219 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:55 smithi187 ceph-mon[20072]: 4.12 scrub ok 2024-09-02T10:11:56.219 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:55 smithi187 ceph-mon[20072]: 4.15 scrub ok 2024-09-02T10:11:56.219 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:55 smithi187 ceph-mon[20072]: osdmap e360: 12 total, 12 up, 12 in 2024-09-02T10:11:56.219 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:55 smithi187 ceph-mon[20072]: 4.10 scrub starts 2024-09-02T10:11:56.219 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:55 smithi187 ceph-mon[20072]: 4.19 scrub starts 2024-09-02T10:11:56.219 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:55 smithi187 ceph-mon[20072]: 4.19 scrub ok 2024-09-02T10:11:56.220 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:55 smithi187 ceph-mon[20072]: pgmap v868: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 65 MiB/s wr, 1.03k op/s; 3159/138069 objects misplaced (2.288%); 14 MiB/s, 227 objects/s recovering 2024-09-02T10:11:56.220 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:55 smithi187 ceph-mon[20072]: 4.10 scrub ok 2024-09-02T10:11:56.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:11:56.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:56.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:11:56.344 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:11:56.344 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:11:56.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:49894 4.1 4:80bf992e:::benchmark_data_smithi187_76542_object49893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:56.323795+0000", 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.001060973, 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010915600000000001, 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 49894 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:56.345 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:56.323795+0000", 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:56.323795+0000", 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:11:56.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:56.323797+0000", 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2359999999999999e-06 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:56.323825+0000", 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7267000000000002e-05 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:56.323852+0000", 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6991999999999998e-05 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:56.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:56.323861+0000", 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4120000000000007e-06 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:56.323940+0000", 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9177000000000004e-05 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:56.324076+0000", 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000135229 2024-09-02T10:11:56.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:56.324222+0000", 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000146759 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:11:56.349 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:11:56.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:55 smithi162 ceph-mon[20279]: 4.12 scrub starts 2024-09-02T10:11:56.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:55 smithi162 ceph-mon[20279]: 4.15 scrub starts 2024-09-02T10:11:56.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:55 smithi162 ceph-mon[20279]: 4.12 scrub ok 2024-09-02T10:11:56.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:55 smithi162 ceph-mon[20279]: 4.15 scrub ok 2024-09-02T10:11:56.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:55 smithi162 ceph-mon[20279]: osdmap e360: 12 total, 12 up, 12 in 2024-09-02T10:11:56.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:55 smithi162 ceph-mon[20279]: 4.10 scrub starts 2024-09-02T10:11:56.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:55 smithi162 ceph-mon[20279]: 4.19 scrub starts 2024-09-02T10:11:56.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:55 smithi162 ceph-mon[20279]: 4.19 scrub ok 2024-09-02T10:11:56.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:55 smithi162 ceph-mon[20279]: pgmap v868: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 65 MiB/s wr, 1.03k op/s; 3159/138069 objects misplaced (2.288%); 14 MiB/s, 227 objects/s recovering 2024-09-02T10:11:56.350 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:55 smithi162 ceph-mon[20279]: 4.10 scrub ok 2024-09-02T10:11:56.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:55 smithi151 ceph-mon[16639]: 4.12 scrub starts 2024-09-02T10:11:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:55 smithi151 ceph-mon[16639]: 4.15 scrub starts 2024-09-02T10:11:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:55 smithi151 ceph-mon[16639]: 4.12 scrub ok 2024-09-02T10:11:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:55 smithi151 ceph-mon[16639]: 4.15 scrub ok 2024-09-02T10:11:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:55 smithi151 ceph-mon[16639]: osdmap e360: 12 total, 12 up, 12 in 2024-09-02T10:11:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:55 smithi151 ceph-mon[16639]: 4.10 scrub starts 2024-09-02T10:11:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:55 smithi151 ceph-mon[16639]: 4.19 scrub starts 2024-09-02T10:11:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:55 smithi151 ceph-mon[16639]: 4.19 scrub ok 2024-09-02T10:11:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:55 smithi151 ceph-mon[16639]: pgmap v868: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 2.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 65 MiB/s wr, 1.03k op/s; 3159/138069 objects misplaced (2.288%); 14 MiB/s, 227 objects/s recovering 2024-09-02T10:11:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:55 smithi151 ceph-mon[16639]: 4.10 scrub ok 2024-09-02T10:11:56.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:56.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:56.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 53 16 50158 50142 59.1248 52.0625 0.00285244 0.0169046 2024-09-02T10:11:56.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:56.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:56.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:56.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:56.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:56.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:56.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:56.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:56.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:56.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:56.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:11:56.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:56.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:11:57.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:57.110 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:57.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:57.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:57.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:57.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:57.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:57.346 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:57.371 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:57.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:57.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:57.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:57.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:57.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 54 16 51287 51271 59.3365 70.5625 0.0694163 0.016836 2024-09-02T10:11:57.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:57.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:57.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:57.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:57.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:57.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:57.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:57.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:57.874 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:57.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:57.875 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:57.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:57.997 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:57.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:57.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:58.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:58.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:58.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:58.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:11:58.252 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:58.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:11:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:57 smithi187 ceph-mon[20072]: 4.18 scrub starts 2024-09-02T10:11:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:57 smithi187 ceph-mon[20072]: 4.13 scrub starts 2024-09-02T10:11:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:57 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:11:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:57 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:11:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:57 smithi187 ceph-mon[20072]: 4.18 scrub ok 2024-09-02T10:11:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:57 smithi187 ceph-mon[20072]: osdmap e361: 12 total, 12 up, 12 in 2024-09-02T10:11:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:57 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:11:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:57 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:11:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:57 smithi187 ceph-mon[20072]: 4.13 scrub ok 2024-09-02T10:11:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:57 smithi187 ceph-mon[20072]: pgmap v870: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 583 op/s; 3342/142956 objects misplaced (2.338%) 2024-09-02T10:11:58.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:57 smithi162 ceph-mon[20279]: 4.18 scrub starts 2024-09-02T10:11:58.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:57 smithi162 ceph-mon[20279]: 4.13 scrub starts 2024-09-02T10:11:58.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:57 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:11:58.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:57 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:11:58.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:57 smithi162 ceph-mon[20279]: 4.18 scrub ok 2024-09-02T10:11:58.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:57 smithi162 ceph-mon[20279]: osdmap e361: 12 total, 12 up, 12 in 2024-09-02T10:11:58.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:57 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:11:58.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:57 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:11:58.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:57 smithi162 ceph-mon[20279]: 4.13 scrub ok 2024-09-02T10:11:58.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:57 smithi162 ceph-mon[20279]: pgmap v870: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 583 op/s; 3342/142956 objects misplaced (2.338%) 2024-09-02T10:11:58.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:11:58.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:58.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:11:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:57 smithi151 ceph-mon[16639]: 4.18 scrub starts 2024-09-02T10:11:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 4.13 scrub starts 2024-09-02T10:11:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:11:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:11:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 4.18 scrub ok 2024-09-02T10:11:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: osdmap e361: 12 total, 12 up, 12 in 2024-09-02T10:11:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:11:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:11:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 4.13 scrub ok 2024-09-02T10:11:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: pgmap v870: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 2.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 36 MiB/s wr, 583 op/s; 3342/142956 objects misplaced (2.338%) 2024-09-02T10:11:58.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:11:58.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:58.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:58.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:11:58.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 55 16 52177 52161 59.269 55.625 0.0518337 0.0168538 2024-09-02T10:11:58.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:58.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:58.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:58.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:58.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:58.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:58.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:11:58.875 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:58.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:58.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:11:58.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:11:59.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:59.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:59.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:11:59.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:59.125 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:59.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:59.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:59.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:59.250 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:59.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:59.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: 4.14 scrub starts 2024-09-02T10:11:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:11:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:11:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:11:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:11:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:11:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:11:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: 4.14 scrub ok 2024-09-02T10:11:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: 4.11 scrub starts 2024-09-02T10:11:59.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: osdmap e362: 12 total, 12 up, 12 in 2024-09-02T10:11:59.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: 4.11 scrub ok 2024-09-02T10:11:59.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:58 smithi187 ceph-mon[20072]: osdmap e363: 12 total, 12 up, 12 in 2024-09-02T10:11:59.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: 4.14 scrub starts 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: 4.14 scrub ok 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: 4.11 scrub starts 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: osdmap e362: 12 total, 12 up, 12 in 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: 4.11 scrub ok 2024-09-02T10:11:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:58 smithi162 ceph-mon[20279]: osdmap e363: 12 total, 12 up, 12 in 2024-09-02T10:11:59.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:11:59.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:11:59.376 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:59.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:11:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 4.14 scrub starts 2024-09-02T10:11:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:11:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:11:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:11:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:11:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:11:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:11:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 4.14 scrub ok 2024-09-02T10:11:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 4.11 scrub starts 2024-09-02T10:11:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: osdmap e362: 12 total, 12 up, 12 in 2024-09-02T10:11:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: 4.11 scrub ok 2024-09-02T10:11:59.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:58 smithi151 ceph-mon[16639]: osdmap e363: 12 total, 12 up, 12 in 2024-09-02T10:11:59.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:59.498 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:59.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:59.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 56 16 53295 53279 59.4583 69.875 0.0290208 0.0168075 2024-09-02T10:11:59.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:11:59.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:11:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:59.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:11:59.726 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:11:59.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:11:59.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:11:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:59.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:11:59.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:11:59.880 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:11:59.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:11:59.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:11:59.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:00.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:00.015 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:00.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:00.142 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:00.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:00.145 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:59 smithi187 ceph-mon[20072]: pgmap v873: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2441/153399 objects misplaced (1.591%) 2024-09-02T10:12:00.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:59 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:12:00.146 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:11:59 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:12:00.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:00.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:00.266 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:00.273 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:12:00.273 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:12:00.273 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:12:00.273 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:12:00.273 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:12:00.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:59 smithi162 ceph-mon[20279]: pgmap v873: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2441/153399 objects misplaced (1.591%) 2024-09-02T10:12:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:59 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:12:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:11:59 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:12:00.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:00.387 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:00.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:00.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:59 smithi151 ceph-mon[16639]: pgmap v873: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2441/153399 objects misplaced (1.591%) 2024-09-02T10:12:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:59 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:12:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:11:59 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:12:00.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:00.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 57 16 54414 54398 59.642 69.9375 0.00442931 0.0167596 2024-09-02T10:12:00.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:00.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:00.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:00.642 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:00.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:00.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:00.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:00.768 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:00.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:00.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:00.897 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:00.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:00.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:01.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:01.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:01.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:01.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:12:01.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:01.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:01.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:00 smithi187 ceph-mon[20072]: osdmap e364: 12 total, 12 up, 12 in 2024-09-02T10:12:01.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:01.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:01.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:01.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:00 smithi151 ceph-mon[16639]: osdmap e364: 12 total, 12 up, 12 in 2024-09-02T10:12:01.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:00 smithi162 ceph-mon[20279]: osdmap e364: 12 total, 12 up, 12 in 2024-09-02T10:12:01.368 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:12:01.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:01.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:01.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:01.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 58 16 55597 55581 59.8884 73.9375 0.00585569 0.0166907 2024-09-02T10:12:01.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:01.549 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:01.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:01.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:01.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:01.672 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:01.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:01.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:01.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:01.803 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:01.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:01.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:01.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:02.042 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:02.070 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:02.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:02.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:02.078 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:01 smithi151 ceph-mon[16639]: pgmap v875: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2611/156162 objects misplaced (1.672%) 2024-09-02T10:12:02.141 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:12:02.141 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:12:02.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:54903 2.c 2:3cdee963:::benchmark_data_smithi187_45882_object54902:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.130442+0000", 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 584.77899223999998, 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.264513322, 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 54903 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.142 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130442+0000", 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130442+0000", 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130444+0000", 2024-09-02T10:12:02.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.418e-06 2024-09-02T10:12:02.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130465+0000", 2024-09-02T10:12:02.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1396e-05 2024-09-02T10:12:02.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130479+0000", 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4222999999999999e-05 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130487+0000", 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.678e-06 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130542+0000", 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5089000000000001e-05 2024-09-02T10:12:02.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130628+0000", 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6173000000000003e-05 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.130715+0000", 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6714999999999999e-05 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.132228+0000", 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0015130740000000001 2024-09-02T10:12:02.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.135665+0000", 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034371010000000001 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.394891+0000", 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25922587499999999 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.394927+0000", 2024-09-02T10:12:02.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5917000000000002e-05 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.394956+0000", 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8663e-05 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:54922 2.a 2:5831fc15:::benchmark_data_smithi187_45882_object54921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e113)", 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.148488+0000", 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 584.76094641500003, 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23997639400000001, 2024-09-02T10:12:02.148 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 54922 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148488+0000", 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148488+0000", 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148490+0000", 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1459999999999999e-06 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148519+0000", 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8144e-05 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148537+0000", 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8776000000000001e-05 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148547+0000", 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4520000000000003e-06 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148603+0000", 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6718000000000001e-05 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148725+0000", 2024-09-02T10:12:02.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121307 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.148875+0000", 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014978999999999999 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.150687+0000", 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001812241 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.150836+0000", 2024-09-02T10:12:02.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014881200000000001 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.388375+0000", 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23753954699999999 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.388426+0000", 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1286999999999999e-05 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.388465+0000", 2024-09-02T10:12:02.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8173999999999997e-05 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24451.0:55077 2.b e113/92 2:d13d7ad6:::benchmark_data_smithi187_45882_object55076:head v 113'3410, mlcod=113'3407)", 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:02:17.498205+0000", 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 584.41123014200002, 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27578873599999998, 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498205+0000", 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498205+0000", 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498205+0000", 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8899999999999998e-07 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.498229+0000", 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3920000000000001e-05 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.772173+0000", 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27394390200000002 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.772181+0000", 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1890000000000007e-06 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.772247+0000", 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5660000000000005e-05 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.772271+0000", 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4040999999999999e-05 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.773959+0000", 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001688457 2024-09-02T10:12:02.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:02:17.773993+0000", 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3778000000000001e-05 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPush(2.0 137/130 [PushOp(2:05d70a15:::benchmark_data_smithi187_45882_object14416:head, version: 98'898, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:05d70a15:::benchmark_data_smithi187_45882_object14416:head@98'898, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:03:05.850569+0000", 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 536.05886570500002, 2024-09-02T10:12:02.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.474171917, 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.850569+0000", 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.850569+0000", 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.850570+0000", 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7700000000000004e-07 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:05.850593+0000", 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3478000000000001e-05 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.322911+0000", 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.472317875 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.322941+0000", 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9618999999999999e-05 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.323115+0000", 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017406100000000001 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.323124+0000", 2024-09-02T10:12:02.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6730000000000004e-06 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:03:06.324741+0000", 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001616534 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:156211 2.a 2:582e1c60:::benchmark_data_smithi187_45882_object58460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:43.251374+0000", 2024-09-02T10:12:02.162 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 438.65806025099999, 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30373849200000003, 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 156211 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.163 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251374+0000", 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251374+0000", 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251375+0000", 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3200000000000003e-07 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251380+0000", 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4689999999999999e-06 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251389+0000", 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9889999999999998e-06 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251395+0000", 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7599999999999999e-06 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251433+0000", 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8529999999999999e-05 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251512+0000", 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8590999999999993e-05 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.251577+0000", 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5259999999999995e-05 2024-09-02T10:12:02.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.254848+0000", 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003271178 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.254978+0000", 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013031100000000001 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555047+0000", 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30006905299999997 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555082+0000", 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4354999999999999e-05 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555113+0000", 2024-09-02T10:12:02.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1063999999999999e-05 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:156211 2.a e188/92 ondisk, result = 0)", 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:04:43.255213+0000", 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 438.65422152799999, 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29990493899999998, 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:12:02.169 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.255213+0000", 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.255213+0000", 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.255214+0000", 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6700000000000003e-07 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.255220+0000", 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0449999999999997e-06 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.554994+0000", 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.299774394 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.554999+0000", 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.186e-06 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555038+0000", 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8933000000000001e-05 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555044+0000", 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.327e-06 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:04:43.555118+0000", 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4387000000000005e-05 2024-09-02T10:12:02.172 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:194200 2.a 2:5976d7b0:::benchmark_data_smithi187_45882_object96449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:40.797959+0000", 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.11147520100002, 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57870098699999994, 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:12:02.173 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194200 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797959+0000", 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797959+0000", 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.174 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797961+0000", 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8670000000000001e-06 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797969+0000", 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5510000000000002e-06 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797987+0000", 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7691000000000001e-05 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.175 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797995+0000", 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7749999999999998e-06 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.798058+0000", 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2309000000000003e-05 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.798148+0000", 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0781999999999999e-05 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.798225+0000", 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6240000000000002e-05 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.801976+0000", 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003751009 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802761+0000", 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00078486799999999998 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376548+0000", 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57378783700000002 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376616+0000", 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7465000000000005e-05 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376660+0000", 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4592999999999998e-05 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.178 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:194200 2.a e216/92 ondisk, result = 0)", 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:40.802119+0000", 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.10731568099999, 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57454802699999996, 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802119+0000", 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802119+0000", 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802120+0000", 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1119999999999999e-06 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802126+0000", 2024-09-02T10:12:02.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9309999999999996e-06 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376473+0000", 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57434650499999995 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376478+0000", 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8109999999999999e-06 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.181 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376533+0000", 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4347e-05 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376541+0000", 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0039999999999999e-06 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376667+0000", 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000126317 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.182 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:109536 3.9 e263/226 3:9af67999:::benchmark_data_smithi187_59989_object109535:head v 263'6841, mlcod=263'6837)", 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:07:33.021899+0000", 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 268.88753582099997, 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.365039328, 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021899+0000", 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021899+0000", 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021900+0000", 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2799999999999995e-07 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021935+0000", 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4860999999999997e-05 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.384110+0000", 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36217508799999998 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.384116+0000", 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5919999999999997e-06 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.384204+0000", 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7767000000000002e-05 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.384221+0000", 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6699000000000001e-05 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.386906+0000", 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026849679999999998 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.386938+0000", 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2524999999999998e-05 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.187 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:154914 3.7 e281/226)", 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:08:24.043568+0000", 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 217.865866966, 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26404111200000002, 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.043568+0000", 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.188 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.043568+0000", 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.043568+0000", 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2799999999999995e-07 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.043574+0000", 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.096e-06 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.189 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.301227+0000", 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25765300000000002 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.301234+0000", 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1180000000000003e-06 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.307590+0000", 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063568690000000002 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.307609+0000", 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8301e-05 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:154922 3.7 e281/226)", 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:08:24.052011+0000", 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 217.85742329000001, 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25561977200000002, 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.191 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.052011+0000", 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.052011+0000", 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.052012+0000", 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0800000000000001e-07 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.052013+0000", 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.903e-06 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.301255+0000", 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24924117300000001 2024-09-02T10:12:02.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.301259+0000", 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0520000000000003e-06 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.307624+0000", 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0063653850000000003 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:08:24.307631+0000", 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0509999999999999e-06 2024-09-02T10:12:02.194 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:181165 3.1 e291/283 3:863dffc8:::benchmark_data_smithi187_59989_object71622:head v 291'11208, mlcod=252'4439)", 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:09:14.761756+0000", 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 167.14767891400001, 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.260975868, 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:14.761756+0000", 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:14.761756+0000", 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:14.761756+0000", 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.68e-07 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:14.761762+0000", 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4999999999999999e-06 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.018884+0000", 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25712180800000001 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.018889+0000", 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0799999999999996e-06 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.018944+0000", 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4936999999999999e-05 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.018961+0000", 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6877000000000001e-05 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.022699+0000", 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003738388 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.022732+0000", 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2610000000000001e-05 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:181199 3.f e291/226 3:fb925b3d:::benchmark_data_smithi187_59989_object71656:head v 291'11268, mlcod=291'11266)", 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:09:14.950707+0000", 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 166.95872813, 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42617851899999998, 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:14.950707+0000", 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:14.950707+0000", 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:14.950707+0000", 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9400000000000005e-07 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:14.950712+0000", 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9660000000000004e-06 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.372548+0000", 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.421835866 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.372558+0000", 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0368e-05 2024-09-02T10:12:02.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.372634+0000", 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5782000000000001e-05 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.372649+0000", 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4822e-05 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.376846+0000", 2024-09-02T10:12:02.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041968400000000003 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.376885+0000", 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9181000000000003e-05 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.24976.0:181220 3.1 e291/283 3:87d3bc93:::benchmark_data_smithi187_59989_object71677:head v 291'11212, mlcod=252'4457)", 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:09:15.131828+0000", 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 166.77760707499999, 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28126711700000001, 2024-09-02T10:12:02.204 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.131828+0000", 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.131828+0000", 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.131829+0000", 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.418e-06 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.131838+0000", 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1670000000000005e-06 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.404038+0000", 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27219950399999998 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.404045+0000", 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1389999999999998e-06 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.404102+0000", 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7429999999999997e-05 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.404125+0000", 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3085e-05 2024-09-02T10:12:02.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.413035+0000", 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0089096069999999999 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:09:15.413095+0000", 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9766999999999999e-05 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:11556 4.d e326/318 4:b36fbd37:::benchmark_data_smithi187_76542_object11555:head v 326'722, mlcod=325'719)", 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:10.472581+0000", 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.436853755000001, 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25455072200000001, 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472581+0000", 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472581+0000", 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472582+0000", 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.198e-06 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.472609+0000", 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6951e-05 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.726413+0000", 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25380428300000002 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.726419+0000", 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3759999999999999e-06 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.726476+0000", 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7784000000000001e-05 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.726501+0000", 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4515000000000002e-05 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.727104+0000", 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000603336 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.727132+0000", 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7279000000000001e-05 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:11571 4.d e326/318 4:b88a9f8d:::benchmark_data_smithi187_76542_object11570:head v 326'724, mlcod=326'721)", 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:10.795584+0000", 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 51.113850364000001, 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.49112458599999997, 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.213 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795584+0000", 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795584+0000", 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795585+0000", 2024-09-02T10:12:02.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7899999999999998e-07 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795608+0000", 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3408e-05 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285722+0000", 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.490114102 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285750+0000", 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7104e-05 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285822+0000", 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2354999999999994e-05 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285855+0000", 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2731000000000002e-05 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.286659+0000", 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00080397800000000001 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.286709+0000", 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0229000000000003e-05 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.217 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:44058 4.d e356/318 4:b86e3ed1:::benchmark_data_smithi187_76542_object44057:head v 356'2735, mlcod=356'2731)", 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:50.483169+0000", 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 11.426265408999999, 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.350330796, 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.483169+0000", 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.483169+0000", 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.483170+0000", 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6499999999999998e-07 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.483187+0000", 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7360999999999999e-05 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.831996+0000", 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34880852200000001 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.832002+0000", 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3180000000000003e-06 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.832054+0000", 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1700999999999998e-05 2024-09-02T10:12:02.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.832078+0000", 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3954000000000001e-05 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.833455+0000", 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013775770000000001 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.221 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.833500+0000", 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4598000000000002e-05 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:44070 4.1 4:85d0390b:::benchmark_data_smithi187_76542_object44069:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:50.499351+0000", 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 11.410083289999999, 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34735302699999998, 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.222 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 44070 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.499351+0000", 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.499351+0000", 2024-09-02T10:12:02.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.499353+0000", 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.161e-06 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.499373+0000", 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0375999999999999e-05 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.224 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.499386+0000", 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2864000000000001e-05 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.499392+0000", 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6250000000000001e-06 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.499444+0000", 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1854999999999997e-05 2024-09-02T10:12:02.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.499538+0000", 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3722999999999997e-05 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.499625+0000", 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7053999999999997e-05 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.500401+0000", 2024-09-02T10:12:02.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00077572399999999999 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.500730+0000", 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000329704 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.846609+0000", 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34587902300000001 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.846665+0000", 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.503e-05 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.846704+0000", 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9888000000000002e-05 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:44109 4.d e356/318 4:bf74cfb0:::benchmark_data_smithi187_76542_object44108:head v 356'2737, mlcod=356'2734)", 2024-09-02T10:12:02.228 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:50.833973+0000", 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 11.075461599, 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30307105200000001, 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.833973+0000", 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.833973+0000", 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.833974+0000", 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1600000000000001e-07 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.833995+0000", 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1243999999999999e-05 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:51.135478+0000", 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30148317600000002 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:51.135487+0000", 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7010000000000002e-06 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:51.135563+0000", 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6514999999999995e-05 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:51.135591+0000", 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7467999999999999e-05 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:51.137002+0000", 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001411696 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:51.137044+0000", 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1635999999999999e-05 2024-09-02T10:12:02.232 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:44116 4.1 4:81bb743d:::benchmark_data_smithi187_76542_object44115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:50.844830+0000", 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 11.064605128, 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.371922686, 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:12:02.233 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 44116 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.844830+0000", 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.844830+0000", 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.844831+0000", 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2359999999999999e-06 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.844858+0000", 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7376e-05 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.844871+0000", 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2481e-05 2024-09-02T10:12:02.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.844876+0000", 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8390000000000003e-06 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.844935+0000", 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8122999999999999e-05 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.845065+0000", 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013026199999999999 2024-09-02T10:12:02.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.845208+0000", 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014317099999999999 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.846031+0000", 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00082312399999999995 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.846746+0000", 2024-09-02T10:12:02.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00071508800000000001 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:51.216653+0000", 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36990643299999998 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:51.216710+0000", 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7309000000000003e-05 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:12:02.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:12:02.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:12:02.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:51.216752+0000", 2024-09-02T10:12:02.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2243999999999997e-05 2024-09-02T10:12:02.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.239 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:12:02.239 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:12:02.239 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:12:02.239 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:01 smithi187 ceph-mon[20072]: pgmap v875: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2611/156162 objects misplaced (1.672%) 2024-09-02T10:12:02.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:02.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:02.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:02.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:01 smithi162 ceph-mon[20279]: pgmap v875: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2611/156162 objects misplaced (1.672%) 2024-09-02T10:12:02.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:02.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:02.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:02.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:02.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 59 16 56918 56902 60.2726 82.5625 0.0581379 0.0165727 2024-09-02T10:12:02.533 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:02.534 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:02.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:02.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:02.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:02.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:02.788 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:02.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:02.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:02.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:02.912 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:02.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:03.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:03.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:03.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:03.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:03.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:03.166 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:03.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:03.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:03.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:12:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:02 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:12:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:02 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:12:03.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:03.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:03.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:03.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:02 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:12:03.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:02 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:12:03.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:03.412 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:03.413 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:02 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:12:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:02 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:12:03.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:12:03.506486+0000 min lat: 0.00103182 max lat: 3.02533 avg lat: 0.0165847 2024-09-02T10:12:03.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:12:03.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 60 16 57685 57669 60.0669 47.9375 0.00281436 0.0165847 2024-09-02T10:12:03.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:03.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:03.536 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:03.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:03.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:03.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:03.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:03.782 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:03.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:03.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:03.905 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:03.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:04.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:04.032 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:04.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:04.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:04.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:04.165 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:04.236 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:03 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:12:04.236 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:03 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:12:04.236 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:03 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:12:04.237 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:03 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:12:04.237 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:03 smithi162 ceph-mon[20279]: osdmap e365: 12 total, 12 up, 12 in 2024-09-02T10:12:04.237 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:03 smithi162 ceph-mon[20279]: pgmap v877: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2611/164154 objects misplaced (1.591%) 2024-09-02T10:12:04.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:03 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:12:04.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:03 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:12:04.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:03 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:12:04.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:03 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:12:04.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:03 smithi151 ceph-mon[16639]: osdmap e365: 12 total, 12 up, 12 in 2024-09-02T10:12:04.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:03 smithi151 ceph-mon[16639]: pgmap v877: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2611/164154 objects misplaced (1.591%) 2024-09-02T10:12:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:03 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:12:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:03 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:12:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:03 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:12:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:03 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:12:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:03 smithi187 ceph-mon[20072]: osdmap e365: 12 total, 12 up, 12 in 2024-09-02T10:12:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:03 smithi187 ceph-mon[20072]: pgmap v877: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2611/164154 objects misplaced (1.591%) 2024-09-02T10:12:04.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:04.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:04.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:04.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:04.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:04.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:04.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:04.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 61 16 58538 58522 59.9561 53.3125 0.0564729 0.0166384 2024-09-02T10:12:04.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:04.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:04.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:04.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:04.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:04.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:04.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:04.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:04.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:04.804 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:04.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:04.928 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:04.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:04.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:05.020 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:05.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:05.053 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:05.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:05.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:05.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:04 smithi151 ceph-mon[16639]: osdmap e366: 12 total, 12 up, 12 in 2024-09-02T10:12:05.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:05.186 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:05.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:04 smithi187 ceph-mon[20072]: osdmap e366: 12 total, 12 up, 12 in 2024-09-02T10:12:05.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:05.310 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:05.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:04 smithi162 ceph-mon[20279]: osdmap e366: 12 total, 12 up, 12 in 2024-09-02T10:12:05.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:05.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:05.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:05.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:05.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 62 16 59953 59937 60.4153 88.4375 0.00316572 0.0165472 2024-09-02T10:12:05.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:05.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:05.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:05.675 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:05.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:05.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:05.799 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:05.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:05.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:05.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:05.953 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:05.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:06.055 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:06.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:06.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:06.184 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:06.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:06.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:05 smithi187 ceph-mon[20072]: 4.16 scrub starts 2024-09-02T10:12:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:05 smithi187 ceph-mon[20072]: 4.16 scrub ok 2024-09-02T10:12:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:05 smithi187 ceph-mon[20072]: 4.c scrub starts 2024-09-02T10:12:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:05 smithi187 ceph-mon[20072]: pgmap v879: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 723 op/s; 2611/170826 objects misplaced (1.528%) 2024-09-02T10:12:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:05 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:12:06.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:05 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:12:06.285 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:05 smithi151 ceph-mon[16639]: 4.16 scrub starts 2024-09-02T10:12:06.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:05 smithi151 ceph-mon[16639]: 4.16 scrub ok 2024-09-02T10:12:06.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:05 smithi151 ceph-mon[16639]: 4.c scrub starts 2024-09-02T10:12:06.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:05 smithi151 ceph-mon[16639]: pgmap v879: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 723 op/s; 2611/170826 objects misplaced (1.528%) 2024-09-02T10:12:06.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:05 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:12:06.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:05 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:12:06.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:06.334 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:06.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:06.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:05 smithi162 ceph-mon[20279]: 4.16 scrub starts 2024-09-02T10:12:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:05 smithi162 ceph-mon[20279]: 4.16 scrub ok 2024-09-02T10:12:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:05 smithi162 ceph-mon[20279]: 4.c scrub starts 2024-09-02T10:12:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:05 smithi162 ceph-mon[20279]: pgmap v879: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfill_toofull, 55 active+clean; 3.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 MiB/s wr, 723 op/s; 2611/170826 objects misplaced (1.528%) 2024-09-02T10:12:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:05 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:12:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:05 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:12:06.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:06.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:06.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:06.506 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 63 16 60938 60922 60.4335 61.5625 0.00371144 0.0165394 2024-09-02T10:12:06.560 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:06.564 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:06.583 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:06.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:06.585 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:12:06.585 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:12:06.615 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:12:06.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:12:06.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:12:06.643 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:12:06.668 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:12:06.668 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:12:06.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:06.691 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:12:06.692 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:12:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:06.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:06.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:06.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:12:06.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:12:06.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:12:06.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:12:06.764 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:12:06.764 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:12:06.789 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:12:06.789 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:12:06.810 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:06.816 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:12:06.816 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:12:06.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:06.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:06.839 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:12:06.840 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:12:06.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:12:06.866 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:12:06.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:06.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:06.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:07.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:12:07.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:07.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:07.089 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:06 smithi151 ceph-mon[16639]: 4.f scrub starts 2024-09-02T10:12:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:06 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:12:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:06 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:12:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:06 smithi151 ceph-mon[16639]: 4.c scrub ok 2024-09-02T10:12:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1368805152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:07.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:07.218 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:07.219 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:06 smithi187 ceph-mon[20072]: 4.f scrub starts 2024-09-02T10:12:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:06 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:12:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:06 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:12:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:06 smithi187 ceph-mon[20072]: 4.c scrub ok 2024-09-02T10:12:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1368805152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:07.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:06 smithi162 ceph-mon[20279]: 4.f scrub starts 2024-09-02T10:12:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:06 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:12:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:06 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:12:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:06 smithi162 ceph-mon[20279]: 4.c scrub ok 2024-09-02T10:12:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1368805152' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:07.347 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:07.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:07.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:07.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:07.477 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:07.478 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:07.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 64 16 62114 62098 60.6376 73.5 0.00487458 0.0164857 2024-09-02T10:12:07.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:07.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:07.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:07.709 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:07.732 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:07.733 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:07.781 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:12:07.781 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:12:07.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:07.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:07.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:07.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:07.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:07.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:07.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:08.091 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:08.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:08.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:08.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:08.245 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:08.246 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:07 smithi187 ceph-mon[20072]: 4.f scrub ok 2024-09-02T10:12:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:07 smithi187 ceph-mon[20072]: 4.d scrub starts 2024-09-02T10:12:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:07 smithi187 ceph-mon[20072]: pgmap v880: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 55 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 939 op/s; 2759/175257 objects misplaced (1.574%); 25 KiB/s, 0 objects/s recovering 2024-09-02T10:12:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:07 smithi162 ceph-mon[20279]: 4.f scrub ok 2024-09-02T10:12:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:07 smithi162 ceph-mon[20279]: 4.d scrub starts 2024-09-02T10:12:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:07 smithi162 ceph-mon[20279]: pgmap v880: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 55 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 939 op/s; 2759/175257 objects misplaced (1.574%); 25 KiB/s, 0 objects/s recovering 2024-09-02T10:12:08.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:08.370 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:08.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:08.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:07 smithi151 ceph-mon[16639]: 4.f scrub ok 2024-09-02T10:12:08.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:07 smithi151 ceph-mon[16639]: 4.d scrub starts 2024-09-02T10:12:08.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:07 smithi151 ceph-mon[16639]: pgmap v880: 57 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 55 active+clean; 3.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 939 op/s; 2759/175257 objects misplaced (1.574%); 25 KiB/s, 0 objects/s recovering 2024-09-02T10:12:08.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:08.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:08.499 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:08.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:08.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 65 16 63526 63510 61.0623 88.25 0.0243619 0.0163719 2024-09-02T10:12:08.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:08.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:08.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:08.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:08.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:08.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:08.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:08.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:08.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:08.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:08.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:09.024 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:09.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:09.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:12:09.046 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:09.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:09.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:09.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:09.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:08 smithi187 ceph-mon[20072]: 4.d scrub ok 2024-09-02T10:12:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:08 smithi187 ceph-mon[20072]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-02T10:12:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:08 smithi187 ceph-mon[20072]: osdmap e367: 12 total, 12 up, 12 in 2024-09-02T10:12:09.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:09.286 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:09.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:09.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:08 smithi162 ceph-mon[20279]: 4.d scrub ok 2024-09-02T10:12:09.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:08 smithi162 ceph-mon[20279]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-02T10:12:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:08 smithi162 ceph-mon[20279]: osdmap e367: 12 total, 12 up, 12 in 2024-09-02T10:12:09.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:08 smithi151 ceph-mon[16639]: 4.d scrub ok 2024-09-02T10:12:09.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:08 smithi151 ceph-mon[16639]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-02T10:12:09.359 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:08 smithi151 ceph-mon[16639]: osdmap e367: 12 total, 12 up, 12 in 2024-09-02T10:12:09.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:09.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:09.412 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:09.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 66 16 65001 64985 61.5337 92.1875 0.0618012 0.016232 2024-09-02T10:12:09.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:09.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:09.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:09.536 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:09.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:09.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:09.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:09.668 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:09.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:09.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:09.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:09.870 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:12:09.870 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:12:09.870 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-02T10:12:09.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:12:09.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:09.920 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:09.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:10.013 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:12:10.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:10.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:10.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:10.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:10.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:10.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:10.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:10.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:10 smithi151 ceph-mon[16639]: pgmap v882: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s; 1851/186072 objects misplaced (0.995%) 2024-09-02T10:12:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:10 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:12:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:10 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:12:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:10 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:12:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:10 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:12:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3884382370' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:10 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:10 smithi187 ceph-mon[20072]: pgmap v882: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s; 1851/186072 objects misplaced (0.995%) 2024-09-02T10:12:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:10 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:12:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:10 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:12:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:10 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:12:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:10 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:12:10.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3884382370' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:10.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:10 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:10.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:10.309 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:10.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:10.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:10 smithi162 ceph-mon[20279]: pgmap v882: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 3.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s; 1851/186072 objects misplaced (0.995%) 2024-09-02T10:12:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:10 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:12:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:10 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:12:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:10 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:12:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:10 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:12:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3884382370' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:10 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:10.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:10.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:10.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:10.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 67 16 66135 66119 61.6731 70.875 0.00878276 0.0162087 2024-09-02T10:12:10.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:10.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:10.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:10.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:12:10.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:10.696 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:10.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:10.797 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:10.819 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:10.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:10.820 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:10.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:10.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:10.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:11.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:11.079 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:11.080 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:11.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:11.205 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:11.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:11 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:12:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:11 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:12:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:11 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:12:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:11 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:12:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:11 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:12:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:11 smithi187 ceph-mon[20072]: osdmap e368: 12 total, 12 up, 12 in 2024-09-02T10:12:11.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:11 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:12:11.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:11 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:12:11.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:11.332 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:11.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:11.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:11 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:12:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:11 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:12:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:11 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:12:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:11 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:12:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:11 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:12:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:11 smithi162 ceph-mon[20279]: osdmap e368: 12 total, 12 up, 12 in 2024-09-02T10:12:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:11 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:12:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:11 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:12:11.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:11 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:12:11.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:11 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:12:11.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:11 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:12:11.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:11 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:12:11.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:11 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:12:11.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:11 smithi151 ceph-mon[16639]: osdmap e368: 12 total, 12 up, 12 in 2024-09-02T10:12:11.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:11 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:12:11.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:11 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:12:11.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:11.457 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:11.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:11.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 68 16 67546 67530 62.0629 88.1875 0.003804 0.0161076 2024-09-02T10:12:11.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:11.586 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:11.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:11.686 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:12:11.686 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:12:11.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:11.705 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:11.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:11.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:11.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:11.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:11.861 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:11.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:11.963 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:11.990 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:11.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:11.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:12.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:12.116 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:12.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:12.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:12.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:12.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:12.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:12 smithi162 ceph-mon[20279]: pgmap v884: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 679 op/s; 1243/189633 objects misplaced (0.655%); 555 KiB/s, 8 objects/s recovering 2024-09-02T10:12:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:12 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:12:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:12 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:12:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:12:12.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:12.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:12.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:12.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:12.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:12 smithi151 ceph-mon[16639]: pgmap v884: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 679 op/s; 1243/189633 objects misplaced (0.655%); 555 KiB/s, 8 objects/s recovering 2024-09-02T10:12:12.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:12 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:12:12.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:12 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:12:12.434 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:12:12.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:12:12.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:12 smithi187 ceph-mon[20072]: pgmap v884: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 3.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 679 op/s; 1243/189633 objects misplaced (0.655%); 555 KiB/s, 8 objects/s recovering 2024-09-02T10:12:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:12 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:12:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:12 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:12:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:12:12.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 69 16 68753 68737 62.2566 75.4375 0.00181358 0.0160467 2024-09-02T10:12:12.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:12.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:12.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:12.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:12.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:12.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:12.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:12.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:12.786 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:12.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:12.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:12.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:12.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:12.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:13.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:13.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:13.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:13.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:13.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:13.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:13.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:13.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:13.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:13.303 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:13.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:13.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:13.314 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:12:13.314 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:12:13.314 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:12:13.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:12:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:12:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:12:13.361 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:13.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:13.433 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:13.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:13.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:13.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:12:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:12:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:12:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:12:13.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 70 16 69932 69916 62.4199 73.6875 0.00895141 0.0160146 2024-09-02T10:12:13.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:13.558 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:13.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:13.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:13.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:13.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:13.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:13.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:13.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:13.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:13.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:13.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:13.975 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:12:13.975 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:12:14.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:14.101 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:14.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:14.131 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:12:14.132 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:12:14.160 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:12:14.160 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:12:14.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:12:14.191 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:12:14.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:14.221 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:12:14.221 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:12:14.221 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:14 smithi151 ceph-mon[16639]: pgmap v885: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 969 op/s; 1243/199650 objects misplaced (0.623%); 504 KiB/s, 7 objects/s recovering 2024-09-02T10:12:14.221 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/974933567' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:14.222 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:14 smithi151 ceph-mon[16639]: osdmap e369: 12 total, 12 up, 12 in 2024-09-02T10:12:14.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:14.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:14.255 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:12:14.255 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:12:14.256 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:14.288 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:12:14.288 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:12:14.318 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:12:14.318 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:12:14.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:14.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:14 smithi162 ceph-mon[20279]: pgmap v885: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 969 op/s; 1243/199650 objects misplaced (0.623%); 504 KiB/s, 7 objects/s recovering 2024-09-02T10:12:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/974933567' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:14 smithi162 ceph-mon[20279]: osdmap e369: 12 total, 12 up, 12 in 2024-09-02T10:12:14.345 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:12:14.345 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:12:14.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:14.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:14.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:14.374 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:12:14.374 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:12:14.403 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:12:14.403 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:12:14.433 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:12:14.433 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:12:14.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:14.464 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:12:14.464 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:12:14.486 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:14.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:14.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:14 smithi187 ceph-mon[20072]: pgmap v885: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 969 op/s; 1243/199650 objects misplaced (0.623%); 504 KiB/s, 7 objects/s recovering 2024-09-02T10:12:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/974933567' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:14 smithi187 ceph-mon[20072]: osdmap e369: 12 total, 12 up, 12 in 2024-09-02T10:12:14.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 71 16 71293 71277 62.7387 85.0625 0.00367382 0.0159325 2024-09-02T10:12:14.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:14.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:14.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:14.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:14.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:14.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:14.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:14.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:12:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:14.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:14.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:14.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:15.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:15.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:15.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:15.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:15 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:12:15.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:15 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:12:15.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:15.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:15.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:15.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:15.271 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:15.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:15.330 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:15 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:12:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:15 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:12:15.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:15.402 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:15.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:15.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:15 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:12:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:15 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:12:15.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 72 16 72813 72797 63.1867 95 0.00591699 0.0158212 2024-09-02T10:12:15.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:15.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:15.639 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:15.662 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:15.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:15.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:15.788 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:15.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:15.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:15.915 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:15.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:15.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:16.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:16.044 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:16.045 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:16.146 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:16.171 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:16.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:16.273 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:16.294 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:16.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:16.296 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:16 smithi151 ceph-mon[16639]: osdmap e370: 12 total, 12 up, 12 in 2024-09-02T10:12:16.296 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:16 smithi151 ceph-mon[16639]: pgmap v888: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 1243/207366 objects misplaced (0.599%); 7.4 MiB/s, 118 objects/s recovering 2024-09-02T10:12:16.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:16 smithi162 ceph-mon[20279]: osdmap e370: 12 total, 12 up, 12 in 2024-09-02T10:12:16.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:16 smithi162 ceph-mon[20279]: pgmap v888: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 1243/207366 objects misplaced (0.599%); 7.4 MiB/s, 118 objects/s recovering 2024-09-02T10:12:16.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:16.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:16.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:16 smithi187 ceph-mon[20072]: osdmap e370: 12 total, 12 up, 12 in 2024-09-02T10:12:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:16 smithi187 ceph-mon[20072]: pgmap v888: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.18k op/s; 1243/207366 objects misplaced (0.599%); 7.4 MiB/s, 118 objects/s recovering 2024-09-02T10:12:16.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 73 16 74083 74067 63.4083 79.375 0.010205 0.0157657 2024-09-02T10:12:16.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:16.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:16.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:16.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:16.677 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:16.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:16.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:16.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:16.803 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:16.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:16.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:16.927 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:16.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:17.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:17.045 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:12:17.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:17.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:17.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:17.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:17.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:17.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:17.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:17.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:17.307 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2093479786' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:12:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:17 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:12:17.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2093479786' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:12:17.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:17 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:12:17.356 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:12:17.356 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:12:17.356 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 371, 2024-09-02T10:12:17.356 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:12:17.356 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:12:17.356 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:12:17.035331+0000", 2024-09-02T10:12:17.356 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:12:17.356 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:09:58.599885+0000", 2024-09-02T10:12:17.356 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5801984, 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "noscrub", 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:12:17.357 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 4, 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:12:17.358 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:12:17.359 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.360 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "264", 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "241", 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:12:17.361 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:12:17.362 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:12:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:12:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:12:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:12:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:12:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:12:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:12:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:12:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:12:17.363 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:12:17.364 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.7000000476837158, 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.7000000476837158, 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.92000001668930054, 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.4800000190734863, 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.4800000190734863, 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 4, 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_2", 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:10:55.979431+0000", 2024-09-02T10:12:17.365 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 26, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 26, 2024-09-02T10:12:17.366 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 26, 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 26, 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 26, 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "343", 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:12:17.367 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "341", 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:12:17.368 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:12:17.369 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:12:17.370 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.7699999809265137, 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.7699999809265137, 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.7699999809265137, 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.7699999809265137, 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:12:17.371 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 365, 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.372 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.373 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.374 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.374 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:12:17.374 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:12:17.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.375 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.376 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.569854736328125, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 364, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.378 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.379 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.382 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:12:17.383 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 349, 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.385 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.385 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:12:17.385 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:12:17.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.385 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.385 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:12:17.385 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:12:17.385 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:12:17.386 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:12:17.387 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.388 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.389 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.6008758544921875, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 356, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.390 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.391 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:12:17.393 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:12:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:12:17.396 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:12:17.397 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.16259765625, 2024-09-02T10:12:17.398 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 340, 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:12:17.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:12:17.400 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:12:17.401 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.402 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:12:17.403 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.404 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 355, 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.405 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:12:17.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.407 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.408 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:12:17.409 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.70245361328125, 2024-09-02T10:12:17.410 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 326, 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:12:17.411 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.414 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.415 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.415 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:12:17.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:17.416 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:12:17.417 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:12:17.418 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 370, 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.420 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:12:17.422 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.423 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.424 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 353, 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:12:17.425 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:12:17.426 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:12:17.427 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.428 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 354, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.430 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.431 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:12:17.434 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.435 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 354, 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.438 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.438 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:12:17.438 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:12:17.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.438 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.438 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:12:17.438 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.439 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.440 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.441 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.4313507080078125, 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:12:17.442 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 356, 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.443 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.444 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.445 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.446 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:12:17.447 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:56.587101+0000", 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:10:53.624029+0000", 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:12:17.449 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:30.441749+0000", 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.450 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:59.590807+0000", 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:17.125049+0000", 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:01.991219+0000", 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.452 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:12:02.435629+0000", 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.453 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:12:14.237200+0000", 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.454 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:53.974448+0000", 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:10:50.538534+0000", 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:12:17.455 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:24.748345+0000", 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:12:17.456 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:11:57.828343+0000", 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.1e", 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: 11, 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: 9 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:12:17.457 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [], 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:12:17.458 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:12:17.459 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:12:17.460 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:12:17.460 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:12:17.460 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:12:17.460 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:12:17.460 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:12:17.460 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:12:17.460 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:12:17.460 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:12:17.460 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:17.460 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:12:17.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:17.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2093479786' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:12:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:17 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:12:17.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 74 15 75116 75101 63.4247 64.625 0.0188736 0.0157374 2024-09-02T10:12:17.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:17.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:17.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:17.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:17.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:17.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:17.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:17.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:17.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:12:17.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:17.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:17.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:17.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:17.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:17.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:17.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:18.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:18.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:18.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:18.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:18.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:18 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:18.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:18 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:12:18.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:18 smithi151 ceph-mon[16639]: osdmap e371: 12 total, 12 up, 12 in 2024-09-02T10:12:18.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:18 smithi151 ceph-mon[16639]: pgmap v890: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 178/213039 objects misplaced (0.084%); 9.8 MiB/s, 157 objects/s recovering 2024-09-02T10:12:18.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/767194720' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:12:18.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:18.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:18.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:18.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:18 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:18 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:12:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:18 smithi162 ceph-mon[20279]: osdmap e371: 12 total, 12 up, 12 in 2024-09-02T10:12:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:18 smithi162 ceph-mon[20279]: pgmap v890: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 178/213039 objects misplaced (0.084%); 9.8 MiB/s, 157 objects/s recovering 2024-09-02T10:12:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/767194720' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:12:18.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:18.386 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:18.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:18.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:18 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:18 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:12:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:18 smithi187 ceph-mon[20072]: osdmap e371: 12 total, 12 up, 12 in 2024-09-02T10:12:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:18 smithi187 ceph-mon[20072]: pgmap v890: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 178/213039 objects misplaced (0.084%); 9.8 MiB/s, 157 objects/s recovering 2024-09-02T10:12:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/767194720' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:12:18.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 75 16 75983 75967 63.3006 54.125 0.00378512 0.0157911 2024-09-02T10:12:18.512 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:18.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:18.527 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:18.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:18.636 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:18.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:18.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:18.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:18.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:18.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:18.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:18.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:18.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:19.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:19.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:19.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:19.146 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:19.147 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:19.148 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:12:19.148 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:12:19.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.148 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:76671 4.9 4:9519b51c:::benchmark_data_smithi187_76542_object76670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-02T10:12:19.148 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:19.107503+0000", 2024-09-02T10:12:19.148 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.038152780999999997, 2024-09-02T10:12:19.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038185172000000003, 2024-09-02T10:12:19.148 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:19.148 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76671 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.107503+0000", 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.107503+0000", 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.107505+0000", 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.779e-06 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.107540+0000", 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5065999999999999e-05 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.107555+0000", 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4955e-05 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.107562+0000", 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8739999999999997e-06 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.107641+0000", 2024-09-02T10:12:19.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9252000000000006e-05 2024-09-02T10:12:19.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:19.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.107778+0000", 2024-09-02T10:12:19.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013644899999999999 2024-09-02T10:12:19.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:12:19.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.107912+0000", 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013437599999999999 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.109480+0000", 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00156829 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.109905+0000", 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042442000000000001 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:19.153 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:76730 4.a 4:5398cb8d:::benchmark_data_smithi187_76542_object76729:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e373)", 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:19.144593+0000", 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0010628199999999999, 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001167749, 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 76730 2024-09-02T10:12:19.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.144593+0000", 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.144593+0000", 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.144594+0000", 2024-09-02T10:12:19.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2249999999999999e-06 2024-09-02T10:12:19.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:19.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.144642+0000", 2024-09-02T10:12:19.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7092000000000002e-05 2024-09-02T10:12:19.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:19.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.144653+0000", 2024-09-02T10:12:19.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1790000000000001e-05 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.144661+0000", 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4479999999999997e-06 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.144726+0000", 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5431000000000004e-05 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.144835+0000", 2024-09-02T10:12:19.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010911 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:19.144929+0000", 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3682000000000002e-05 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 2 2024-09-02T10:12:19.158 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:12:19.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:19.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:19 smithi151 ceph-mon[16639]: osdmap e372: 12 total, 12 up, 12 in 2024-09-02T10:12:19.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:19 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:12:19.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:19 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:12:19.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:19 smithi151 ceph-mon[16639]: osdmap e373: 12 total, 12 up, 12 in 2024-09-02T10:12:19.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:19.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:19.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:19 smithi162 ceph-mon[20279]: osdmap e372: 12 total, 12 up, 12 in 2024-09-02T10:12:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:19 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:12:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:19 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:12:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:19 smithi162 ceph-mon[20279]: osdmap e373: 12 total, 12 up, 12 in 2024-09-02T10:12:19.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:19.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:19.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:19.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:19 smithi187 ceph-mon[20072]: osdmap e372: 12 total, 12 up, 12 in 2024-09-02T10:12:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:19 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:12:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:19 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:12:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:19 smithi187 ceph-mon[20072]: osdmap e373: 12 total, 12 up, 12 in 2024-09-02T10:12:19.507 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 76 16 77342 77326 63.5853 84.9375 0.00246432 0.0157213 2024-09-02T10:12:19.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:19.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:19.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:19.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:19.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:19.672 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:19.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:19.750 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:12:19.750 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:12:19.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:19.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:12:19.786 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:12:19.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:19.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:19.820 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:12:19.820 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:12:19.849 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:12:19.849 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:12:19.879 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:12:19.879 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:12:19.903 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:19.911 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:12:19.911 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:12:19.928 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:19.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:19.947 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:12:19.947 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:12:19.979 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:12:19.979 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:12:20.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:12:20.015 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:12:20.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:20.051 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:12:20.051 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:12:20.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:20.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:20.086 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:12:20.086 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:12:20.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:20 smithi151 ceph-mon[16639]: 4.17 scrub starts 2024-09-02T10:12:20.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:20 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:12:20.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:20 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:12:20.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:20 smithi151 ceph-mon[16639]: pgmap v893: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s; 178/224223 objects misplaced (0.079%); 14 MiB/s, 223 objects/s recovering 2024-09-02T10:12:20.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/46785327' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:20.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:12:20.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:12:20.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:20.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:20.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:20.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:12:20.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:20.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:20.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:20.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:20.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:20 smithi162 ceph-mon[20279]: 4.17 scrub starts 2024-09-02T10:12:20.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:20 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:12:20.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:20 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:12:20.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:20 smithi162 ceph-mon[20279]: pgmap v893: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s; 178/224223 objects misplaced (0.079%); 14 MiB/s, 223 objects/s recovering 2024-09-02T10:12:20.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/46785327' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:20.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:20.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:20.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:20.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:20 smithi187 ceph-mon[20072]: 4.17 scrub starts 2024-09-02T10:12:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:20 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:12:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:20 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:12:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:20 smithi187 ceph-mon[20072]: pgmap v893: 57 pgs: 1 active+remapped+backfilling, 56 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s; 178/224223 objects misplaced (0.079%); 14 MiB/s, 223 objects/s recovering 2024-09-02T10:12:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/46785327' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:20.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 77 16 78928 78912 64.0467 99.125 0.00850682 0.0156054 2024-09-02T10:12:20.570 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:20.593 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:20.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:20.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:20.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:12:20.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:20.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:20.832 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:20.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:20.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:20.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:20.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:20.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:20.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:21.081 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:21.111 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:21.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:21.213 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:21.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:21.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:21.236 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:21.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:21.360 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:21.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:21.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:21.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:21.486 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:21.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:21.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 78 16 79982 79966 64.0701 65.875 0.00561123 0.0156028 2024-09-02T10:12:21.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:21.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:21.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:21.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:21.738 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:21.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:21.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:21.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:21.864 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:21.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:21.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:21.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:21.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:22.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:22.111 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:22.112 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:22.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:22.236 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:22.237 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:22.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:22.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:22.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:22.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:22.404 INFO:tasks.thrashosds.thrasher:j is {'epoch': 371, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:12:17.035331+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:09:58.599885+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.700000047683716, 'score_stable': 2.700000047683716, 'optimal_score': 0.9200000166893005, 'raw_score_acting': 2.4800000190734863, 'raw_score_stable': 2.4800000190734863, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-02T10:10:55.979431+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '343', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '341', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.7699999809265137, 'score_stable': 2.7699999809265137, 'optimal_score': 1, 'raw_score_acting': 2.7699999809265137, 'raw_score_stable': 2.7699999809265137, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 365, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 0.569854736328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 364, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 349, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 0.6008758544921875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 1, 'weight': 0.16259765625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 340, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 355, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 0.70245361328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 326, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 370, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 353, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 354, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 354, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 1, 'weight': 0.4313507080078125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:56.587101+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:10:53.624029+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:30.441749+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:59.590807+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:17.125049+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:01.991219+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:12:02.435629+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:12:14.237200+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:53.974448+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:10:50.538534+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:24.748345+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:11:57.828343+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1e', 'osds': [11, 10, 9]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:12:22.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:12:22.405 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:22.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:22.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:22.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:22.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 79 16 80749 80733 63.8658 47.9375 0.0250064 0.0156521 2024-09-02T10:12:22.577 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:22 smithi151 ceph-mon[16639]: pgmap v894: 57 pgs: 57 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 12 MiB/s, 184 objects/s recovering 2024-09-02T10:12:22.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:22 smithi162 ceph-mon[20279]: pgmap v894: 57 pgs: 57 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 12 MiB/s, 184 objects/s recovering 2024-09-02T10:12:22.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:22.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:22.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:22.650 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:22 smithi187 ceph-mon[20072]: pgmap v894: 57 pgs: 57 active+clean; 4.6 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s; 12 MiB/s, 184 objects/s recovering 2024-09-02T10:12:22.689 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:22.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:22.748 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:22.749 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:22.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:22.872 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:22.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:22.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:22.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:22.995 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:22.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:22.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:23.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:23.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:23.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:23.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:23.257 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:23.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:23.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:12:23.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:23.380 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:23.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:23.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:23.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:12:23.508109+0000 min lat: 0.00103182 max lat: 3.02533 avg lat: 0.0157114 2024-09-02T10:12:23.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:12:23.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 80 16 81470 81454 63.6307 45.0625 0.0171338 0.0157114 2024-09-02T10:12:23.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:23.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:23.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:23.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:23.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:23.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:23.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:23.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:23.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:23.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:23.893 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:23.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:23.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:23.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:24.017 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:24.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:24.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:24.119 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:24.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:24.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:24.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:24.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:24.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:24.310 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:12:24.363 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:12:24.364 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:12:24.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:24.409 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:24.410 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:24.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 81 16 82691 82675 63.7872 76.3125 0.0107506 0.015671 2024-09-02T10:12:24.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:24.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:24.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:24.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:24 smithi162 ceph-mon[20279]: pgmap v895: 57 pgs: 57 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 1.7 MiB/s, 26 objects/s recovering 2024-09-02T10:12:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2097453227' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:24 smithi162 ceph-mon[20279]: 4.17 scrub starts 2024-09-02T10:12:24.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:24 smithi151 ceph-mon[16639]: pgmap v895: 57 pgs: 57 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 1.7 MiB/s, 26 objects/s recovering 2024-09-02T10:12:24.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2097453227' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:24.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:24 smithi151 ceph-mon[16639]: 4.17 scrub starts 2024-09-02T10:12:24.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:24.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:24.670 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:24 smithi187 ceph-mon[20072]: pgmap v895: 57 pgs: 57 active+clean; 4.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.32k op/s; 1.7 MiB/s, 26 objects/s recovering 2024-09-02T10:12:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2097453227' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:24 smithi187 ceph-mon[20072]: 4.17 scrub starts 2024-09-02T10:12:24.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:24.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:24.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:24.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:24.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:24.927 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:25.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:25.051 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:25.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:25.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:25.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:25.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:25.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:25.309 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:25.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:25.310 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:25.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:12:25.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:25.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:25.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:25.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 82 15 84025 84010 64.0268 83.4375 0.00724232 0.0156135 2024-09-02T10:12:25.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:25.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:25 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2097453227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:12:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:25 smithi162 ceph-mon[20279]: osdmap e374: 12 total, 12 up, 12 in 2024-09-02T10:12:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:25.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:25.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:25 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:25.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2097453227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:12:25.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:25 smithi151 ceph-mon[16639]: osdmap e374: 12 total, 12 up, 12 in 2024-09-02T10:12:25.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:25.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:25.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:25.727 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:25 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2097453227' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:12:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:25 smithi187 ceph-mon[20072]: osdmap e374: 12 total, 12 up, 12 in 2024-09-02T10:12:25.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:25.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:25.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:25.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:25.977 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:25.977 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:25.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:25.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:26.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:26.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:26.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:26.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:26.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:26.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:26.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:26.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:26.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:26.374 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:26.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:26.498 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:26.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:26.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 83 16 85246 85230 64.1739 76.25 0.0238278 0.0155758 2024-09-02T10:12:26.587 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:12:26.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:12:26.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:26 smithi151 ceph-mon[16639]: pgmap v897: 57 pgs: 57 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-09-02T10:12:26.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:26 smithi162 ceph-mon[20279]: pgmap v897: 57 pgs: 57 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-09-02T10:12:26.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:26.616 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:12:26.616 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:12:26.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:26.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:26.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:26.644 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:12:26.644 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:12:26.673 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:12:26.673 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:12:26.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:12:26.700 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:12:26.727 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:12:26.727 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:12:26.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:26.750 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:26.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:26 smithi187 ceph-mon[20072]: pgmap v897: 57 pgs: 57 active+clean; 4.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.30k op/s; 1.4 MiB/s, 22 objects/s recovering 2024-09-02T10:12:26.757 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:12:26.757 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:12:26.781 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:12:26.781 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:12:26.806 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:12:26.806 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:12:26.833 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:12:26.833 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:12:26.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:26.857 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:12:26.857 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:12:26.874 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:26.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:26.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:26.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:12:26.885 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:12:26.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:26.998 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:26.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:26.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:27.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:27.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:27.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:27.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:27.249 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:27.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:27.302 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-02T10:12:27.266+0000 7efc18868640 0 --2- 172.21.15.151:0/3694852222 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7efc14101a60 0x7efc14101e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-02T10:12:27.311 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:27.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:27.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:27.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:27.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:27.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:27.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 84 16 86399 86383 64.2678 72.0625 0.0108578 0.0155542 2024-09-02T10:12:27.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:27.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:27.572 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:12:27.572 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:12:27.572 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:27 smithi151 ceph-mon[16639]: osdmap e375: 12 total, 12 up, 12 in 2024-09-02T10:12:27.572 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/141883757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:27.575 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:12:27.576 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:12:27.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:12:27.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:12:27.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:27 smithi162 ceph-mon[20279]: osdmap e375: 12 total, 12 up, 12 in 2024-09-02T10:12:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/141883757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:27.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:27.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:27.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:12:27.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:12:27.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:27 smithi187 ceph-mon[20072]: osdmap e375: 12 total, 12 up, 12 in 2024-09-02T10:12:27.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/141883757' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:27.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:27.789 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:27.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:27.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:27.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:27.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:28.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:12:28.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:28.043 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:28.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:28.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:28.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:28.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:28.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:28.296 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:28.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:28.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:28.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:28.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:28.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:28.508 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 85 15 87904 87889 64.619 94.125 0.0104768 0.0154711 2024-09-02T10:12:28.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:28.551 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:28.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:28.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:28.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:28 smithi162 ceph-mon[20279]: pgmap v899: 57 pgs: 57 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 974 op/s; 1.3 MiB/s, 20 objects/s recovering 2024-09-02T10:12:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:28 smithi162 ceph-mon[20279]: from='client.16098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:12:28.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:28.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:28.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:28.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:28.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:28 smithi151 ceph-mon[16639]: pgmap v899: 57 pgs: 57 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 974 op/s; 1.3 MiB/s, 20 objects/s recovering 2024-09-02T10:12:28.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:28 smithi151 ceph-mon[16639]: from='client.16098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:12:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:28 smithi187 ceph-mon[20072]: pgmap v899: 57 pgs: 57 active+clean; 5.0 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 974 op/s; 1.3 MiB/s, 20 objects/s recovering 2024-09-02T10:12:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:28 smithi187 ceph-mon[20072]: from='client.16098 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:12:28.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:28.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:28.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:28.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:28.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:28.940 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:28.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:29.042 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:29.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:29.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:29.073 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:29.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:29.195 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:29.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:29.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:29.318 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:29.319 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:29.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:29.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:29.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:29.509 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 86 16 88840 88824 64.547 58.4375 0.00207038 0.0154584 2024-09-02T10:12:29.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:29.566 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:29.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:29.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:29.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:29.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:29.691 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:29.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:29.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:29.817 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:29.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:29.818 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:29.919 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:29.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:29.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:30.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:30.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:30.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:30.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:30.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:30.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:30.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:29 smithi187 ceph-mon[20072]: pgmap v900: 57 pgs: 57 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-02T10:12:30.294 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:29 smithi151 ceph-mon[16639]: pgmap v900: 57 pgs: 57 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-02T10:12:30.307 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:30.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:30.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:30.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:29 smithi162 ceph-mon[20279]: pgmap v900: 57 pgs: 57 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.32k op/s 2024-09-02T10:12:30.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:30.453 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:30.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:30.509 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 87 16 89560 89544 64.3223 45 0.00681644 0.0155271 2024-09-02T10:12:30.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:30.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:30.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:30.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:30.715 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:30.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:30.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:30.840 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:30.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:30.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:30.966 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:30.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:30.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:31.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:31.090 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:31.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:31.136 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:31.148 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":899,"stamp":"2024-09-02T10:12:27.247626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5409866272,"num_objects":82541,"num_object_clones":0,"num_object_copies":247623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58720,"num_write_kb":3758080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6443,"num_bytes_recovered":422248448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104270,"ondisk_log_size":104270,"up":171,"acting":171,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19205916,"kb_used_data":16267460,"kb_used_omap":18,"kb_used_meta":2938157,"kb_avail":1105735908,"statfs":{"total":1151940427776,"available":1132273569792,"internally_reserved":0,"allocated":16657879040,"data_stored":16651868711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":3008673083},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":511180800,"num_objects":7800,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-178,"num_objects_unfound":0,"num_objects_dirty":7800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":499200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":165,"num_bytes_recovered":10813440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001662"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":59,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.336754+0000","last_change":"2024-09-02T10:11:57.772410+0000","last_active":"2024-09-02T10:12:26.336754+0000","last_peered":"2024-09-02T10:12:26.336754+0000","last_clean":"2024-09-02T10:12:26.336754+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:12:26.336754+0000","last_undegraded":"2024-09-02T10:12:26.336754+0000","last_fullsized":"2024-09-02T10:12:26.336754+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:57.772340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:57.772340+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:11.809584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.18","version":"373'3558","reported_seq":10244,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.209404+0000","last_change":"2024-09-02T10:11:56.850886+0000","last_active":"2024-09-02T10:12:23.209404+0000","last_peered":"2024-09-02T10:12:23.209404+0000","last_clean":"2024-09-02T10:12:23.209404+0000","last_became_active":"2024-09-02T10:11:51.387879+0000","last_became_peered":"2024-09-02T10:11:51.387879+0000","last_unstale":"2024-09-02T10:12:23.209404+0000","last_undegraded":"2024-09-02T10:12:23.209404+0000","last_fullsized":"2024-09-02T10:12:23.209404+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"360'2577","last_scrub_stamp":"2024-09-02T10:11:56.850539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:56.850539+0000","objects_scrubbed":1624,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:04.872390+0000","scrub_duration":443,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170852352,"num_objects":2607,"num_object_clones":0,"num_object_copies":7821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":114,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.338501+0000","last_change":"2024-09-02T10:11:29.264235+0000","last_active":"2024-09-02T10:12:26.338501+0000","last_peered":"2024-09-02T10:12:26.338501+0000","last_clean":"2024-09-02T10:12:26.338501+0000","last_became_active":"2024-09-02T10:11:18.920647+0000","last_became_peered":"2024-09-02T10:11:18.920647+0000","last_unstale":"2024-09-02T10:12:26.338501+0000","last_undegraded":"2024-09-02T10:12:26.338501+0000","last_fullsized":"2024-09-02T10:12:26.338501+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:29.264185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:29.264185+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:53.100224+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,10],"acting":[2,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"375'3573","reported_seq":8988,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.534139+0000","last_change":"2024-09-02T10:11:55.239829+0000","last_active":"2024-09-02T10:12:26.534139+0000","last_peered":"2024-09-02T10:12:26.534139+0000","last_clean":"2024-09-02T10:12:26.534139+0000","last_became_active":"2024-09-02T10:11:48.946623+0000","last_became_peered":"2024-09-02T10:11:48.946623+0000","last_unstale":"2024-09-02T10:12:26.534139+0000","last_undegraded":"2024-09-02T10:12:26.534139+0000","last_fullsized":"2024-09-02T10:12:26.534139+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":5,"last_scrub":"360'2457","last_scrub_stamp":"2024-09-02T10:11:55.239752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:55.239752+0000","objects_scrubbed":1558,"log_size":3573,"log_dups_size":0,"ondisk_log_size":3573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:14.775025+0000","scrub_duration":230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175636480,"num_objects":2680,"num_object_clones":0,"num_object_copies":8040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1122,"num_write_kb":71808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":305,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:18.936415+0000","last_change":"2024-09-02T10:11:03.720364+0000","last_active":"2024-09-02T10:12:18.936415+0000","last_peered":"2024-09-02T10:12:18.936415+0000","last_clean":"2024-09-02T10:12:18.936415+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:12:18.936415+0000","last_undegraded":"2024-09-02T10:12:18.936415+0000","last_fullsized":"2024-09-02T10:12:18.936415+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:03.720317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:03.720317+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:23.733894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":305,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:18.936254+0000","last_change":"2024-09-02T10:10:54.656216+0000","last_active":"2024-09-02T10:12:18.936254+0000","last_peered":"2024-09-02T10:12:18.936254+0000","last_clean":"2024-09-02T10:12:18.936254+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:12:18.936254+0000","last_undegraded":"2024-09-02T10:12:18.936254+0000","last_fullsized":"2024-09-02T10:12:18.936254+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.656165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.656165+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:24.539265+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":222,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:24.309254+0000","last_change":"2024-09-02T10:11:18.920901+0000","last_active":"2024-09-02T10:12:24.309254+0000","last_peered":"2024-09-02T10:12:24.309254+0000","last_clean":"2024-09-02T10:12:24.309254+0000","last_became_active":"2024-09-02T10:11:18.920488+0000","last_became_peered":"2024-09-02T10:11:18.920488+0000","last_unstale":"2024-09-02T10:12:24.309254+0000","last_undegraded":"2024-09-02T10:12:24.309254+0000","last_fullsized":"2024-09-02T10:12:24.309254+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:07.462786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:07.462786+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":167,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.331784+0000","last_change":"2024-09-02T10:10:58.372721+0000","last_active":"2024-09-02T10:12:26.331784+0000","last_peered":"2024-09-02T10:12:26.331784+0000","last_clean":"2024-09-02T10:12:26.331784+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:12:26.331784+0000","last_undegraded":"2024-09-02T10:12:26.331784+0000","last_fullsized":"2024-09-02T10:12:26.331784+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:58.372641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:58.372641+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":668,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.343455+0000","last_change":"2024-09-02T10:12:10.602620+0000","last_active":"2024-09-02T10:12:26.343455+0000","last_peered":"2024-09-02T10:12:26.343455+0000","last_clean":"2024-09-02T10:12:26.343455+0000","last_became_active":"2024-09-02T10:06:34.859627+0000","last_became_peered":"2024-09-02T10:06:34.859627+0000","last_unstale":"2024-09-02T10:12:26.343455+0000","last_undegraded":"2024-09-02T10:12:26.343455+0000","last_fullsized":"2024-09-02T10:12:26.343455+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:10.602556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:10.602556+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:26.649915+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.12","version":"373'3435","reported_seq":5464,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.085708+0000","last_change":"2024-09-02T10:11:54.863926+0000","last_active":"2024-09-02T10:12:23.085708+0000","last_peered":"2024-09-02T10:12:23.085708+0000","last_clean":"2024-09-02T10:12:23.085708+0000","last_became_active":"2024-09-02T10:11:44.335569+0000","last_became_peered":"2024-09-02T10:11:44.335569+0000","last_unstale":"2024-09-02T10:12:23.085708+0000","last_undegraded":"2024-09-02T10:12:23.085708+0000","last_fullsized":"2024-09-02T10:12:23.085708+0000","mapping_epoch":349,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":5,"last_scrub":"359'2405","last_scrub_stamp":"2024-09-02T10:11:54.863621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:54.863621+0000","objects_scrubbed":1438,"log_size":3435,"log_dups_size":0,"ondisk_log_size":3435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:55.569913+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":7434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1040,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,8],"acting":[9,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":320,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:18.936598+0000","last_change":"2024-09-02T10:11:57.667394+0000","last_active":"2024-09-02T10:12:18.936598+0000","last_peered":"2024-09-02T10:12:18.936598+0000","last_clean":"2024-09-02T10:12:18.936598+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:12:18.936598+0000","last_undegraded":"2024-09-02T10:12:18.936598+0000","last_fullsized":"2024-09-02T10:12:18.936598+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:57.667349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:57.667349+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:09.776096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.13","version":"373'3517","reported_seq":6535,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.090180+0000","last_change":"2024-09-02T10:11:57.133980+0000","last_active":"2024-09-02T10:12:23.090180+0000","last_peered":"2024-09-02T10:12:23.090180+0000","last_clean":"2024-09-02T10:12:23.090180+0000","last_became_active":"2024-09-02T10:11:48.946652+0000","last_became_peered":"2024-09-02T10:11:48.946652+0000","last_unstale":"2024-09-02T10:12:23.090180+0000","last_undegraded":"2024-09-02T10:12:23.090180+0000","last_fullsized":"2024-09-02T10:12:23.090180+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":5,"last_scrub":"361'2553","last_scrub_stamp":"2024-09-02T10:11:57.133827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:57.133827+0000","objects_scrubbed":1583,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:24.666141+0000","scrub_duration":572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167575552,"num_objects":2557,"num_object_clones":0,"num_object_copies":7671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":974,"num_write_kb":62336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,5],"acting":[9,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":312,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:18.942036+0000","last_change":"2024-09-02T10:11:11.497630+0000","last_active":"2024-09-02T10:12:18.942036+0000","last_peered":"2024-09-02T10:12:18.942036+0000","last_clean":"2024-09-02T10:12:18.942036+0000","last_became_active":"2024-09-02T10:11:11.497246+0000","last_became_peered":"2024-09-02T10:11:11.497246+0000","last_unstale":"2024-09-02T10:12:18.942036+0000","last_undegraded":"2024-09-02T10:12:18.942036+0000","last_fullsized":"2024-09-02T10:12:18.942036+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:07.248083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:07.248083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:36.348505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"373'3425","reported_seq":7354,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.695583+0000","last_change":"2024-09-02T10:11:55.398378+0000","last_active":"2024-09-02T10:12:23.695583+0000","last_peered":"2024-09-02T10:12:23.695583+0000","last_clean":"2024-09-02T10:12:23.695583+0000","last_became_active":"2024-09-02T10:11:48.931086+0000","last_became_peered":"2024-09-02T10:11:48.931086+0000","last_unstale":"2024-09-02T10:12:23.695583+0000","last_undegraded":"2024-09-02T10:12:23.695583+0000","last_fullsized":"2024-09-02T10:12:23.695583+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":5,"last_scrub":"360'2425","last_scrub_stamp":"2024-09-02T10:11:55.398278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:55.398278+0000","objects_scrubbed":1495,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:13.800446+0000","scrub_duration":434,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163905536,"num_objects":2501,"num_object_clones":0,"num_object_copies":7503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1006,"num_write_kb":64384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":298,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:24.310094+0000","last_change":"2024-09-02T10:10:55.422138+0000","last_active":"2024-09-02T10:12:24.310094+0000","last_peered":"2024-09-02T10:12:24.310094+0000","last_clean":"2024-09-02T10:12:24.310094+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:12:24.310094+0000","last_undegraded":"2024-09-02T10:12:24.310094+0000","last_fullsized":"2024-09-02T10:12:24.310094+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:55.422074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:55.422074+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"373'3595","reported_seq":6792,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.709928+0000","last_change":"2024-09-02T10:11:58.282219+0000","last_active":"2024-09-02T10:12:23.709928+0000","last_peered":"2024-09-02T10:12:23.709928+0000","last_clean":"2024-09-02T10:12:23.709928+0000","last_became_active":"2024-09-02T10:11:51.386810+0000","last_became_peered":"2024-09-02T10:11:51.386810+0000","last_unstale":"2024-09-02T10:12:23.709928+0000","last_undegraded":"2024-09-02T10:12:23.709928+0000","last_fullsized":"2024-09-02T10:12:23.709928+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"362'2657","last_scrub_stamp":"2024-09-02T10:11:58.281957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:58.281957+0000","objects_scrubbed":1726,"log_size":3595,"log_dups_size":0,"ondisk_log_size":3595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:27.629850+0000","scrub_duration":272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174981120,"num_objects":2670,"num_object_clones":0,"num_object_copies":8010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":944,"num_write_kb":60416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":556,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.339406+0000","last_change":"2024-09-02T10:10:57.026011+0000","last_active":"2024-09-02T10:12:26.339406+0000","last_peered":"2024-09-02T10:12:26.339406+0000","last_clean":"2024-09-02T10:12:26.339406+0000","last_became_active":"2024-09-02T10:02:42.797838+0000","last_became_peered":"2024-09-02T10:02:42.797838+0000","last_unstale":"2024-09-02T10:12:26.339406+0000","last_undegraded":"2024-09-02T10:12:26.339406+0000","last_fullsized":"2024-09-02T10:12:26.339406+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:57.025960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:57.025960+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:29.377724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"373'3471","reported_seq":7460,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:22.509168+0000","last_change":"2024-09-02T10:12:05.011107+0000","last_active":"2024-09-02T10:12:22.509168+0000","last_peered":"2024-09-02T10:12:22.509168+0000","last_clean":"2024-09-02T10:12:22.509168+0000","last_became_active":"2024-09-02T10:12:03.986919+0000","last_became_peered":"2024-09-02T10:12:03.986919+0000","last_unstale":"2024-09-02T10:12:22.509168+0000","last_undegraded":"2024-09-02T10:12:22.509168+0000","last_fullsized":"2024-09-02T10:12:22.509168+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":5,"last_scrub":"366'2781","last_scrub_stamp":"2024-09-02T10:12:05.010972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:12:05.010972+0000","objects_scrubbed":1849,"log_size":3471,"log_dups_size":0,"ondisk_log_size":3471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:30.757601+0000","scrub_duration":361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166723584,"num_objects":2544,"num_object_clones":0,"num_object_copies":7632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2544,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":695,"num_write_kb":44480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":645,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:18.942326+0000","last_change":"2024-09-02T10:10:59.358111+0000","last_active":"2024-09-02T10:12:18.942326+0000","last_peered":"2024-09-02T10:12:18.942326+0000","last_clean":"2024-09-02T10:12:18.942326+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:12:18.942326+0000","last_undegraded":"2024-09-02T10:12:18.942326+0000","last_fullsized":"2024-09-02T10:12:18.942326+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:59.358063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:59.358063+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:29.124978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"374'3553","reported_seq":8864,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:25.945142+0000","last_change":"2024-09-02T10:12:23.449801+0000","last_active":"2024-09-02T10:12:25.945142+0000","last_peered":"2024-09-02T10:12:25.945142+0000","last_clean":"2024-09-02T10:12:25.945142+0000","last_became_active":"2024-09-02T10:12:18.058766+0000","last_became_peered":"2024-09-02T10:12:18.058766+0000","last_unstale":"2024-09-02T10:12:25.945142+0000","last_undegraded":"2024-09-02T10:12:25.945142+0000","last_fullsized":"2024-09-02T10:12:25.945142+0000","mapping_epoch":371,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":3553,"log_dups_size":0,"ondisk_log_size":3553,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-02T10:12:28.396386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171573248,"num_objects":2618,"num_object_clones":0,"num_object_copies":7854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2618,"num_write_kb":167552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2198,"num_bytes_recovered":144048128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,10],"acting":[7,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":593,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.343593+0000","last_change":"2024-09-02T10:12:11.591279+0000","last_active":"2024-09-02T10:12:26.343593+0000","last_peered":"2024-09-02T10:12:26.343593+0000","last_clean":"2024-09-02T10:12:26.343593+0000","last_became_active":"2024-09-02T10:11:11.468355+0000","last_became_peered":"2024-09-02T10:11:11.468355+0000","last_unstale":"2024-09-02T10:12:26.343593+0000","last_undegraded":"2024-09-02T10:12:26.343593+0000","last_fullsized":"2024-09-02T10:12:26.343593+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:11.591235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:11.591235+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:13.664755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"373'3457","reported_seq":9241,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.110953+0000","last_change":"2024-09-02T10:11:57.840608+0000","last_active":"2024-09-02T10:12:23.110953+0000","last_peered":"2024-09-02T10:12:23.110953+0000","last_clean":"2024-09-02T10:12:23.110953+0000","last_became_active":"2024-09-02T10:11:46.378103+0000","last_became_peered":"2024-09-02T10:11:46.378103+0000","last_unstale":"2024-09-02T10:12:23.110953+0000","last_undegraded":"2024-09-02T10:12:23.110953+0000","last_fullsized":"2024-09-02T10:12:23.110953+0000","mapping_epoch":351,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":352,"parent":"0.0","parent_split_bits":5,"last_scrub":"361'2577","last_scrub_stamp":"2024-09-02T10:11:57.840556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:57.840556+0000","objects_scrubbed":1607,"log_size":3457,"log_dups_size":0,"ondisk_log_size":3457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:24.858628+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163119104,"num_objects":2489,"num_object_clones":0,"num_object_copies":7467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":882,"num_write_kb":56448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,2],"acting":[9,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":562,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:18.940127+0000","last_change":"2024-09-02T10:12:09.770402+0000","last_active":"2024-09-02T10:12:18.940127+0000","last_peered":"2024-09-02T10:12:18.940127+0000","last_clean":"2024-09-02T10:12:18.940127+0000","last_became_active":"2024-09-02T10:02:42.786836+0000","last_became_peered":"2024-09-02T10:02:42.786836+0000","last_unstale":"2024-09-02T10:12:18.940127+0000","last_undegraded":"2024-09-02T10:12:18.940127+0000","last_fullsized":"2024-09-02T10:12:18.940127+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:09.770356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:09.770356+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:16.385274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"373'3434","reported_seq":7994,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:24.255661+0000","last_change":"2024-09-02T10:11:54.907320+0000","last_active":"2024-09-02T10:12:24.255661+0000","last_peered":"2024-09-02T10:12:24.255661+0000","last_clean":"2024-09-02T10:12:24.255661+0000","last_became_active":"2024-09-02T10:11:48.405502+0000","last_became_peered":"2024-09-02T10:11:48.405502+0000","last_unstale":"2024-09-02T10:12:24.255661+0000","last_undegraded":"2024-09-02T10:12:24.255661+0000","last_fullsized":"2024-09-02T10:12:24.255661+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":341,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"359'2420","last_scrub_stamp":"2024-09-02T10:11:54.907230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:54.907230+0000","objects_scrubbed":1467,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:59.997228+0000","scrub_duration":258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163053568,"num_objects":2488,"num_object_clones":0,"num_object_copies":7464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1021,"num_write_kb":65344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":578,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.343462+0000","last_change":"2024-09-02T10:11:30.646296+0000","last_active":"2024-09-02T10:12:26.343462+0000","last_peered":"2024-09-02T10:12:26.343462+0000","last_clean":"2024-09-02T10:12:26.343462+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:12:26.343462+0000","last_undegraded":"2024-09-02T10:12:26.343462+0000","last_fullsized":"2024-09-02T10:12:26.343462+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:30.646251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:30.646251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:42.404524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"373'5010","reported_seq":5178,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.086924+0000","last_change":"2024-09-02T10:12:07.500747+0000","last_active":"2024-09-02T10:12:23.086924+0000","last_peered":"2024-09-02T10:12:23.086924+0000","last_clean":"2024-09-02T10:12:23.086924+0000","last_became_active":"2024-09-02T10:10:57.383259+0000","last_became_peered":"2024-09-02T10:10:57.383259+0000","last_unstale":"2024-09-02T10:12:23.086924+0000","last_undegraded":"2024-09-02T10:12:23.086924+0000","last_fullsized":"2024-09-02T10:12:23.086924+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'3814","last_scrub_stamp":"2024-09-02T10:12:07.500602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:12:07.500602+0000","objects_scrubbed":3780,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:22.215137+0000","scrub_duration":951,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328335360,"num_objects":5010,"num_object_clones":0,"num_object_copies":15030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5010,"num_write_kb":320640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,11],"acting":[9,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":657,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.343566+0000","last_change":"2024-09-02T10:12:09.559678+0000","last_active":"2024-09-02T10:12:26.343566+0000","last_peered":"2024-09-02T10:12:26.343566+0000","last_clean":"2024-09-02T10:12:26.343566+0000","last_became_active":"2024-09-02T10:06:32.873327+0000","last_became_peered":"2024-09-02T10:06:32.873327+0000","last_unstale":"2024-09-02T10:12:26.343566+0000","last_undegraded":"2024-09-02T10:12:26.343566+0000","last_fullsized":"2024-09-02T10:12:26.343566+0000","mapping_epoch":182,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:09.559616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:09.559616+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:15.245551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.c","version":"373'5116","reported_seq":6756,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.691842+0000","last_change":"2024-09-02T10:12:06.253952+0000","last_active":"2024-09-02T10:12:23.691842+0000","last_peered":"2024-09-02T10:12:23.691842+0000","last_clean":"2024-09-02T10:12:23.691842+0000","last_became_active":"2024-09-02T10:11:19.934231+0000","last_became_peered":"2024-09-02T10:11:19.934231+0000","last_unstale":"2024-09-02T10:12:23.691842+0000","last_undegraded":"2024-09-02T10:12:23.691842+0000","last_fullsized":"2024-09-02T10:12:23.691842+0000","mapping_epoch":333,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":334,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'3782","last_scrub_stamp":"2024-09-02T10:12:06.253637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:12:06.253637+0000","objects_scrubbed":3737,"log_size":5116,"log_dups_size":0,"ondisk_log_size":5116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:07.659801+0000","scrub_duration":1162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335282176,"num_objects":5116,"num_object_clones":0,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5116,"num_write_kb":327424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":47775744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,8],"acting":[3,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":770,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:24.308887+0000","last_change":"2024-09-02T10:11:02.444385+0000","last_active":"2024-09-02T10:12:24.308887+0000","last_peered":"2024-09-02T10:12:24.308887+0000","last_clean":"2024-09-02T10:12:24.308887+0000","last_became_active":"2024-09-02T10:06:32.871027+0000","last_became_peered":"2024-09-02T10:06:32.871027+0000","last_unstale":"2024-09-02T10:12:24.308887+0000","last_undegraded":"2024-09-02T10:12:24.308887+0000","last_fullsized":"2024-09-02T10:12:24.308887+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:02.444317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:02.444317+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:26.961398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"373'5216","reported_seq":5388,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.202943+0000","last_change":"2024-09-02T10:12:06.461383+0000","last_active":"2024-09-02T10:12:23.202943+0000","last_peered":"2024-09-02T10:12:23.202943+0000","last_clean":"2024-09-02T10:12:23.202943+0000","last_became_active":"2024-09-02T10:10:57.382683+0000","last_became_peered":"2024-09-02T10:10:57.382683+0000","last_unstale":"2024-09-02T10:12:23.202943+0000","last_undegraded":"2024-09-02T10:12:23.202943+0000","last_fullsized":"2024-09-02T10:12:23.202943+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'3917","last_scrub_stamp":"2024-09-02T10:12:06.461205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:12:06.461205+0000","objects_scrubbed":3896,"log_size":5216,"log_dups_size":0,"ondisk_log_size":5216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:24.298110+0000","scrub_duration":961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341835776,"num_objects":5216,"num_object_clones":0,"num_object_copies":15648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5216,"num_write_kb":333824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,9],"acting":[1,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":418,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:18.940755+0000","last_change":"2024-09-02T10:12:01.573316+0000","last_active":"2024-09-02T10:12:18.940755+0000","last_peered":"2024-09-02T10:12:18.940755+0000","last_clean":"2024-09-02T10:12:18.940755+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:12:18.940755+0000","last_undegraded":"2024-09-02T10:12:18.940755+0000","last_fullsized":"2024-09-02T10:12:18.940755+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:01.573268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:01.573268+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:17.808446+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.e","version":"373'5223","reported_seq":7756,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:24.313393+0000","last_change":"2024-09-02T10:11:31.957137+0000","last_active":"2024-09-02T10:12:24.313393+0000","last_peered":"2024-09-02T10:12:24.313393+0000","last_clean":"2024-09-02T10:12:24.313393+0000","last_became_active":"2024-09-02T10:11:31.954359+0000","last_became_peered":"2024-09-02T10:11:31.954359+0000","last_unstale":"2024-09-02T10:12:24.313393+0000","last_undegraded":"2024-09-02T10:12:24.313393+0000","last_fullsized":"2024-09-02T10:12:24.313393+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:28.102846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342294528,"num_objects":5223,"num_object_clones":0,"num_object_copies":15669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5223,"num_write_kb":334272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1199,"num_bytes_recovered":78577664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":220,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:18.940171+0000","last_change":"2024-09-02T10:12:05.659085+0000","last_active":"2024-09-02T10:12:18.940171+0000","last_peered":"2024-09-02T10:12:18.940171+0000","last_clean":"2024-09-02T10:12:18.940171+0000","last_became_active":"2024-09-02T10:08:25.264766+0000","last_became_peered":"2024-09-02T10:08:25.264766+0000","last_unstale":"2024-09-02T10:12:18.940171+0000","last_undegraded":"2024-09-02T10:12:18.940171+0000","last_fullsized":"2024-09-02T10:12:18.940171+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:05.659033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:05.659033+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:33.397534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"375'3703","reported_seq":3854,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.888231+0000","last_change":"2024-09-02T10:11:34.727957+0000","last_active":"2024-09-02T10:12:26.888231+0000","last_peered":"2024-09-02T10:12:26.888231+0000","last_clean":"2024-09-02T10:12:26.888231+0000","last_became_active":"2024-09-02T10:11:32.951191+0000","last_became_peered":"2024-09-02T10:11:32.951191+0000","last_unstale":"2024-09-02T10:12:26.888231+0000","last_undegraded":"2024-09-02T10:12:26.888231+0000","last_fullsized":"2024-09-02T10:12:26.888231+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1937","last_scrub_stamp":"2024-09-02T10:11:34.727882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.727882+0000","objects_scrubbed":936,"log_size":3703,"log_dups_size":0,"ondisk_log_size":3703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:42.143902+0000","scrub_duration":939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177143808,"num_objects":2703,"num_object_clones":0,"num_object_copies":8109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1767,"num_write_kb":113088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":705,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.336711+0000","last_change":"2024-09-02T10:10:56.874596+0000","last_active":"2024-09-02T10:12:26.336711+0000","last_peered":"2024-09-02T10:12:26.336711+0000","last_clean":"2024-09-02T10:12:26.336711+0000","last_became_active":"2024-09-02T10:02:40.777057+0000","last_became_peered":"2024-09-02T10:02:40.777057+0000","last_unstale":"2024-09-02T10:12:26.336711+0000","last_undegraded":"2024-09-02T10:12:26.336711+0000","last_fullsized":"2024-09-02T10:12:26.336711+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:56.874512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.874512+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:28.867381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.0","version":"374'3540","reported_seq":3687,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:25.952388+0000","last_change":"2024-09-02T10:11:34.502637+0000","last_active":"2024-09-02T10:12:25.952388+0000","last_peered":"2024-09-02T10:12:25.952388+0000","last_clean":"2024-09-02T10:12:25.952388+0000","last_became_active":"2024-09-02T10:11:32.955679+0000","last_became_peered":"2024-09-02T10:11:32.955679+0000","last_unstale":"2024-09-02T10:12:25.952388+0000","last_undegraded":"2024-09-02T10:12:25.952388+0000","last_fullsized":"2024-09-02T10:12:25.952388+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1865","last_scrub_stamp":"2024-09-02T10:11:34.502539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.502539+0000","objects_scrubbed":937,"log_size":3540,"log_dups_size":0,"ondisk_log_size":3540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:45.346063+0000","scrub_duration":786,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171245568,"num_objects":2613,"num_object_clones":0,"num_object_copies":7839,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2613,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1676,"num_write_kb":107264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":174,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.331959+0000","last_change":"2024-09-02T10:11:18.921335+0000","last_active":"2024-09-02T10:12:26.331959+0000","last_peered":"2024-09-02T10:12:26.331959+0000","last_clean":"2024-09-02T10:12:26.331959+0000","last_became_active":"2024-09-02T10:11:18.920967+0000","last_became_peered":"2024-09-02T10:11:18.920967+0000","last_unstale":"2024-09-02T10:12:26.331959+0000","last_undegraded":"2024-09-02T10:12:26.331959+0000","last_fullsized":"2024-09-02T10:12:26.331959+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:04.459478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:04.459478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:29.779520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,0],"acting":[10,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.3","version":"374'3536","reported_seq":3685,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:25.867974+0000","last_change":"2024-09-02T10:11:34.835252+0000","last_active":"2024-09-02T10:12:25.867974+0000","last_peered":"2024-09-02T10:12:25.867974+0000","last_clean":"2024-09-02T10:12:25.867974+0000","last_became_active":"2024-09-02T10:11:32.955684+0000","last_became_peered":"2024-09-02T10:11:32.955684+0000","last_unstale":"2024-09-02T10:12:25.867974+0000","last_undegraded":"2024-09-02T10:12:25.867974+0000","last_fullsized":"2024-09-02T10:12:25.867974+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1901","last_scrub_stamp":"2024-09-02T10:11:34.835164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.835164+0000","objects_scrubbed":968,"log_size":3536,"log_dups_size":0,"ondisk_log_size":3536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:57.368702+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170590208,"num_objects":2603,"num_object_clones":0,"num_object_copies":7809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1635,"num_write_kb":104640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":175,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.332006+0000","last_change":"2024-09-02T10:11:57.058122+0000","last_active":"2024-09-02T10:12:26.332006+0000","last_peered":"2024-09-02T10:12:26.332006+0000","last_clean":"2024-09-02T10:12:26.332006+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:12:26.332006+0000","last_undegraded":"2024-09-02T10:12:26.332006+0000","last_fullsized":"2024-09-02T10:12:26.332006+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:57.057873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:57.057873+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:02.290524+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.2","version":"373'3496","reported_seq":3644,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.093054+0000","last_change":"2024-09-02T10:11:33.363660+0000","last_active":"2024-09-02T10:12:23.093054+0000","last_peered":"2024-09-02T10:12:23.093054+0000","last_clean":"2024-09-02T10:12:23.093054+0000","last_became_active":"2024-09-02T10:11:32.953702+0000","last_became_peered":"2024-09-02T10:11:32.953702+0000","last_unstale":"2024-09-02T10:12:23.093054+0000","last_undegraded":"2024-09-02T10:12:23.093054+0000","last_fullsized":"2024-09-02T10:12:23.093054+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1882","last_scrub_stamp":"2024-09-02T10:11:33.363574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:33.363574+0000","objects_scrubbed":965,"log_size":3496,"log_dups_size":0,"ondisk_log_size":3496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:57.111356+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169017344,"num_objects":2579,"num_object_clones":0,"num_object_copies":7737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1614,"num_write_kb":103296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,8],"acting":[9,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":708,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.348369+0000","last_change":"2024-09-02T10:12:05.334909+0000","last_active":"2024-09-02T10:12:26.348369+0000","last_peered":"2024-09-02T10:12:26.348369+0000","last_clean":"2024-09-02T10:12:26.348369+0000","last_became_active":"2024-09-02T10:09:58.912625+0000","last_became_peered":"2024-09-02T10:09:58.912625+0000","last_unstale":"2024-09-02T10:12:26.348369+0000","last_undegraded":"2024-09-02T10:12:26.348369+0000","last_fullsized":"2024-09-02T10:12:26.348369+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:05.334854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:05.334854+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:23.503446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"375'3669","reported_seq":7522,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.750006+0000","last_change":"2024-09-02T10:11:33.267368+0000","last_active":"2024-09-02T10:12:26.750006+0000","last_peered":"2024-09-02T10:12:26.750006+0000","last_clean":"2024-09-02T10:12:26.750006+0000","last_became_active":"2024-09-02T10:11:32.948422+0000","last_became_peered":"2024-09-02T10:11:32.948422+0000","last_unstale":"2024-09-02T10:12:26.750006+0000","last_undegraded":"2024-09-02T10:12:26.750006+0000","last_fullsized":"2024-09-02T10:12:26.750006+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1912","last_scrub_stamp":"2024-09-02T10:11:33.267241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:33.267241+0000","objects_scrubbed":974,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:34.436780+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178978816,"num_objects":2731,"num_object_clones":0,"num_object_copies":8193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1757,"num_write_kb":112448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,7],"acting":[2,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":167,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.331858+0000","last_change":"2024-09-02T10:10:59.341544+0000","last_active":"2024-09-02T10:12:26.331858+0000","last_peered":"2024-09-02T10:12:26.331858+0000","last_clean":"2024-09-02T10:12:26.331858+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:12:26.331858+0000","last_undegraded":"2024-09-02T10:12:26.331858+0000","last_fullsized":"2024-09-02T10:12:26.331858+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:59.341491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:59.341491+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"375'3585","reported_seq":5239,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.552303+0000","last_change":"2024-09-02T10:11:34.391701+0000","last_active":"2024-09-02T10:12:26.552303+0000","last_peered":"2024-09-02T10:12:26.552303+0000","last_clean":"2024-09-02T10:12:26.552303+0000","last_became_active":"2024-09-02T10:11:32.951628+0000","last_became_peered":"2024-09-02T10:11:32.951628+0000","last_unstale":"2024-09-02T10:12:26.552303+0000","last_undegraded":"2024-09-02T10:12:26.552303+0000","last_fullsized":"2024-09-02T10:12:26.552303+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1881","last_scrub_stamp":"2024-09-02T10:11:34.391635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.391635+0000","objects_scrubbed":926,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:38.303878+0000","scrub_duration":612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172359680,"num_objects":2630,"num_object_clones":0,"num_object_copies":7890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1704,"num_write_kb":109056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":411,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:24.309277+0000","last_change":"2024-09-02T10:10:53.401665+0000","last_active":"2024-09-02T10:12:24.309277+0000","last_peered":"2024-09-02T10:12:24.309277+0000","last_clean":"2024-09-02T10:12:24.309277+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:12:24.309277+0000","last_undegraded":"2024-09-02T10:12:24.309277+0000","last_fullsized":"2024-09-02T10:12:24.309277+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:53.401592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:53.401592+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:25.370113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"373'3449","reported_seq":3597,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.111411+0000","last_change":"2024-09-02T10:11:34.358626+0000","last_active":"2024-09-02T10:12:23.111411+0000","last_peered":"2024-09-02T10:12:23.111411+0000","last_clean":"2024-09-02T10:12:23.111411+0000","last_became_active":"2024-09-02T10:11:32.967808+0000","last_became_peered":"2024-09-02T10:11:32.967808+0000","last_unstale":"2024-09-02T10:12:23.111411+0000","last_undegraded":"2024-09-02T10:12:23.111411+0000","last_fullsized":"2024-09-02T10:12:23.111411+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1880","last_scrub_stamp":"2024-09-02T10:11:34.358535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.358535+0000","objects_scrubbed":941,"log_size":3449,"log_dups_size":0,"ondisk_log_size":3449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:55.173674+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164495360,"num_objects":2510,"num_object_clones":0,"num_object_copies":7530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1569,"num_write_kb":100416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,2],"acting":[9,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":648,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:18.944520+0000","last_change":"2024-09-02T10:12:02.444114+0000","last_active":"2024-09-02T10:12:18.944520+0000","last_peered":"2024-09-02T10:12:18.944520+0000","last_clean":"2024-09-02T10:12:18.944520+0000","last_became_active":"2024-09-02T10:11:18.920790+0000","last_became_peered":"2024-09-02T10:11:18.920790+0000","last_unstale":"2024-09-02T10:12:18.944520+0000","last_undegraded":"2024-09-02T10:12:18.944520+0000","last_fullsized":"2024-09-02T10:12:18.944520+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:02.444037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:02.444037+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:05.296600+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,10],"acting":[1,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"375'3630","reported_seq":5620,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.500480+0000","last_change":"2024-09-02T10:11:35.890951+0000","last_active":"2024-09-02T10:12:26.500480+0000","last_peered":"2024-09-02T10:12:26.500480+0000","last_clean":"2024-09-02T10:12:26.500480+0000","last_became_active":"2024-09-02T10:11:34.930403+0000","last_became_peered":"2024-09-02T10:11:34.930403+0000","last_unstale":"2024-09-02T10:12:26.500480+0000","last_undegraded":"2024-09-02T10:12:26.500480+0000","last_fullsized":"2024-09-02T10:12:26.500480+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1863","last_scrub_stamp":"2024-09-02T10:11:35.890838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:35.890838+0000","objects_scrubbed":957,"log_size":3630,"log_dups_size":0,"ondisk_log_size":3630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:02.222800+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178520064,"num_objects":2724,"num_object_clones":0,"num_object_copies":8172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1767,"num_write_kb":113088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,0],"acting":[10,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":422,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:24.309160+0000","last_change":"2024-09-02T10:11:11.468234+0000","last_active":"2024-09-02T10:12:24.309160+0000","last_peered":"2024-09-02T10:12:24.309160+0000","last_clean":"2024-09-02T10:12:24.309160+0000","last_became_active":"2024-09-02T10:11:11.467851+0000","last_became_peered":"2024-09-02T10:11:11.467851+0000","last_unstale":"2024-09-02T10:12:24.309160+0000","last_undegraded":"2024-09-02T10:12:24.309160+0000","last_fullsized":"2024-09-02T10:12:24.309160+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:10:54.437492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:10:54.437492+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:24.342292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"373'5008","reported_seq":7432,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.128068+0000","last_change":"2024-09-02T10:11:29.940818+0000","last_active":"2024-09-02T10:12:23.128068+0000","last_peered":"2024-09-02T10:12:23.128068+0000","last_clean":"2024-09-02T10:12:23.128068+0000","last_became_active":"2024-09-02T10:11:29.938756+0000","last_became_peered":"2024-09-02T10:11:29.938756+0000","last_unstale":"2024-09-02T10:12:23.128068+0000","last_undegraded":"2024-09-02T10:12:23.128068+0000","last_fullsized":"2024-09-02T10:12:23.128068+0000","mapping_epoch":338,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":5008,"log_dups_size":0,"ondisk_log_size":5008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:25.199650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328204288,"num_objects":5008,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5008,"num_write_kb":320512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1146,"num_bytes_recovered":75104256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,7],"acting":[1,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":672,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:26.348336+0000","last_change":"2024-09-02T10:12:09.327787+0000","last_active":"2024-09-02T10:12:26.348336+0000","last_peered":"2024-09-02T10:12:26.348336+0000","last_clean":"2024-09-02T10:12:26.348336+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:12:26.348336+0000","last_undegraded":"2024-09-02T10:12:26.348336+0000","last_fullsized":"2024-09-02T10:12:26.348336+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:09.327734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:09.327734+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:13:22.080053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"373'5105","reported_seq":7582,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:22.532097+0000","last_change":"2024-09-02T10:11:28.950121+0000","last_active":"2024-09-02T10:12:22.532097+0000","last_peered":"2024-09-02T10:12:22.532097+0000","last_clean":"2024-09-02T10:12:22.532097+0000","last_became_active":"2024-09-02T10:11:28.936098+0000","last_became_peered":"2024-09-02T10:11:28.936098+0000","last_unstale":"2024-09-02T10:12:22.532097+0000","last_undegraded":"2024-09-02T10:12:22.532097+0000","last_fullsized":"2024-09-02T10:12:22.532097+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:10:56.365261+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334561280,"num_objects":5105,"num_object_clones":0,"num_object_copies":15315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5105,"num_write_kb":326720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1171,"num_bytes_recovered":76742656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":204,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:24.310080+0000","last_change":"2024-09-02T10:08:26.088128+0000","last_active":"2024-09-02T10:12:24.310080+0000","last_peered":"2024-09-02T10:12:24.310080+0000","last_clean":"2024-09-02T10:12:24.310080+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:12:24.310080+0000","last_undegraded":"2024-09-02T10:12:24.310080+0000","last_fullsized":"2024-09-02T10:12:24.310080+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:08:26.088081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:08:26.088081+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"375'3746","reported_seq":6369,"reported_epoch":375,"state":"active+clean","last_fresh":"2024-09-02T10:12:27.011426+0000","last_change":"2024-09-02T10:11:34.867424+0000","last_active":"2024-09-02T10:12:27.011426+0000","last_peered":"2024-09-02T10:12:27.011426+0000","last_clean":"2024-09-02T10:12:27.011426+0000","last_became_active":"2024-09-02T10:11:32.955100+0000","last_became_peered":"2024-09-02T10:11:32.955100+0000","last_unstale":"2024-09-02T10:12:27.011426+0000","last_undegraded":"2024-09-02T10:12:27.011426+0000","last_fullsized":"2024-09-02T10:12:27.011426+0000","mapping_epoch":337,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1959","last_scrub_stamp":"2024-09-02T10:11:34.867321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:34.867321+0000","objects_scrubbed":963,"log_size":3746,"log_dups_size":0,"ondisk_log_size":3746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:54.964856+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180224000,"num_objects":2750,"num_object_clones":0,"num_object_copies":8250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1787,"num_write_kb":114368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,9],"acting":[11,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":436,"reported_epoch":374,"state":"active+clean","last_fresh":"2024-09-02T10:12:24.310142+0000","last_change":"2024-09-02T10:11:01.465118+0000","last_active":"2024-09-02T10:12:24.310142+0000","last_peered":"2024-09-02T10:12:24.310142+0000","last_clean":"2024-09-02T10:12:24.310142+0000","last_became_active":"2024-09-02T10:11:00.404482+0000","last_became_peered":"2024-09-02T10:11:00.404482+0000","last_unstale":"2024-09-02T10:12:24.310142+0000","last_undegraded":"2024-09-02T10:12:24.310142+0000","last_fullsized":"2024-09-02T10:12:24.310142+0000","mapping_epoch":321,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:01.465055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:01.465055+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"373'3368","reported_seq":5784,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:23.117319+0000","last_change":"2024-09-02T10:11:33.677020+0000","last_active":"2024-09-02T10:12:23.117319+0000","last_peered":"2024-09-02T10:12:23.117319+0000","last_clean":"2024-09-02T10:12:23.117319+0000","last_became_active":"2024-09-02T10:11:32.951840+0000","last_became_peered":"2024-09-02T10:11:32.951840+0000","last_unstale":"2024-09-02T10:12:23.117319+0000","last_undegraded":"2024-09-02T10:12:23.117319+0000","last_fullsized":"2024-09-02T10:12:23.117319+0000","mapping_epoch":341,"log_start":"0'0","ondisk_log_start":"0'0","created":318,"last_epoch_clean":342,"parent":"0.0","parent_split_bits":0,"last_scrub":"342'1887","last_scrub_stamp":"2024-09-02T10:11:33.676961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:10:56.365261+0000","last_clean_scrub_stamp":"2024-09-02T10:11:33.676961+0000","objects_scrubbed":905,"log_size":3368,"log_dups_size":0,"ondisk_log_size":3368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:57.486731+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":156368896,"num_objects":2386,"num_object_clones":0,"num_object_copies":7158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2386,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1481,"num_write_kb":94784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":199,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-02T10:12:18.940329+0000","last_change":"2024-09-02T10:11:27.692069+0000","last_active":"2024-09-02T10:12:18.940329+0000","last_peered":"2024-09-02T10:12:18.940329+0000","last_clean":"2024-09-02T10:12:18.940329+0000","last_became_active":"2024-09-02T10:11:11.472186+0000","last_became_peered":"2024-09-02T10:11:11.472186+0000","last_unstale":"2024-09-02T10:12:18.940329+0000","last_undegraded":"2024-09-02T10:12:18.940329+0000","last_fullsized":"2024-09-02T10:12:18.940329+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:27.692014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:27.692014+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:12:31.198340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":26,"stat_sum":{"num_bytes":5409275904,"num_objects":82539,"num_object_clones":0,"num_object_copies":247617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58720,"num_write_kb":3758080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6443,"num_bytes_recovered":422248448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16621699072,"data_stored":16621699072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101418,"ondisk_log_size":101418,"up":78,"acting":78,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383828,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1008784,"kb_used_data":688824,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92736368,"statfs":{"total":95995035648,"available":94962040832,"internally_reserved":0,"allocated":705355776,"data_stored":704850137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":327612868},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612761,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1634048,"kb_used_data":1526444,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":92111104,"statfs":{"total":95995035648,"available":94321770496,"internally_reserved":0,"allocated":1563078656,"data_stored":1562577530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":110098880},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841693,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2449648,"kb_used_data":2103328,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91295504,"statfs":{"total":95995035648,"available":93486596096,"internally_reserved":0,"allocated":2153807872,"data_stored":2153311309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070627,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1135344,"kb_used_data":973664,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92609808,"statfs":{"total":95995035648,"available":94832443392,"internally_reserved":0,"allocated":997031936,"data_stored":996535373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299560,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2343176,"kb_used_data":2041268,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91401976,"statfs":{"total":95995035648,"available":93595623424,"internally_reserved":0,"allocated":2090258432,"data_stored":2089753754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":309131722},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528492,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1201212,"kb_used_data":1017324,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92543940,"statfs":{"total":95995035648,"available":94764994560,"internally_reserved":0,"allocated":1041739776,"data_stored":1041238650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":188283335},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724720,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":717268,"kb_used_data":519488,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93027884,"statfs":{"total":95995035648,"available":95260553216,"internally_reserved":0,"allocated":531955712,"data_stored":531446521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953653,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":662960,"kb_used_data":469344,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93082192,"statfs":{"total":95995035648,"available":95316164608,"internally_reserved":0,"allocated":480608256,"data_stored":480111693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149883,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1990960,"kb_used_data":1639072,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91754192,"statfs":{"total":95995035648,"available":93956292608,"internally_reserved":0,"allocated":1678409728,"data_stored":1677913165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":360315336},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411519,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1216532,"kb_used_data":1022912,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92528620,"statfs":{"total":95995035648,"available":94749306880,"internally_reserved":0,"allocated":1047461888,"data_stored":1046952697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2776240,"kb_used_data":2488224,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":90968912,"statfs":{"total":95995035648,"available":93152165888,"internally_reserved":0,"allocated":2547941376,"data_stored":2547444813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738569,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2069744,"kb_used_data":1777568,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91675408,"statfs":{"total":95995035648,"available":93875617792,"internally_reserved":0,"allocated":1820229632,"data_stored":1819733069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1828651008,"data_stored":1828651008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2565079040,"data_stored":2565079040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1051262976,"data_stored":1051262976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1688272896,"data_stored":1688272896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":342294528,"data_stored":342294528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":531496960,"data_stored":531496960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1050607616,"data_stored":1050607616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2112290816,"data_stored":2112290816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1004863488,"data_stored":1004863488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":2166226944,"data_stored":2166226944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1576271872,"data_stored":1576271872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":704380928,"data_stored":704380928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:12:31.151 INFO:tasks.thrashosds.thrasher:Setting 1.7 to [6, 4, 7, 5, 10, 11] 2024-09-02T10:12:31.151 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.7', '6', '4', '7', '5', '10', '11'] 2024-09-02T10:12:31.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 1.7 6 4 7 5 10 11 2024-09-02T10:12:31.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:31.213 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:31.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:31.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:30 smithi187 ceph-mon[20072]: 4.17 scrub starts 2024-09-02T10:12:31.258 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:30 smithi151 ceph-mon[16639]: 4.17 scrub starts 2024-09-02T10:12:31.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:31.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:31.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:31.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:30 smithi162 ceph-mon[20279]: 4.17 scrub starts 2024-09-02T10:12:31.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:31.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:31.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:31.509 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 88 16 90323 90307 64.1332 47.6875 0.00391342 0.0155837 2024-09-02T10:12:31.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:31.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:31.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:31.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:31.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:31.725 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:31.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:31.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:31.735 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:31.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:31.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:31.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:31.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:31.977 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:31.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:32.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:32.101 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:32.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:32.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:31 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:12:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:31 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:12:32.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:31 smithi151 ceph-mon[16639]: pgmap v901: 57 pgs: 57 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-02T10:12:32.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:32.225 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:32.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:32.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:31 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:12:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:31 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:12:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:31 smithi187 ceph-mon[20072]: pgmap v901: 57 pgs: 57 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-02T10:12:32.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:32.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:12:32.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:31 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:12:32.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:31 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:12:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:31 smithi162 ceph-mon[20279]: pgmap v901: 57 pgs: 57 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.09k op/s 2024-09-02T10:12:32.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:32.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:32.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:32.489 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:32.490 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:32.509 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 89 16 91123 91107 63.9744 50 0.0145831 0.0156259 2024-09-02T10:12:32.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:32.615 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:32.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:32.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:32.739 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:32.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:32.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:32.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:32.864 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:32.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:32.963 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:12:32.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:33.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:33.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:33.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:33.134 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:33.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:33.236 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:33.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2052896643' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:12:33.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:32 smithi151 ceph-mon[16639]: osdmap e376: 12 total, 12 up, 12 in 2024-09-02T10:12:33.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:32 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:12:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2052896643' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:12:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:32 smithi187 ceph-mon[20072]: osdmap e376: 12 total, 12 up, 12 in 2024-09-02T10:12:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:32 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:12:33.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:33.264 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:33.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2052896643' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:12:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:32 smithi162 ceph-mon[20279]: osdmap e376: 12 total, 12 up, 12 in 2024-09-02T10:12:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:32 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:12:33.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:33.386 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:33.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:33.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:33.509 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 90 10 92333 92323 64.1079 76 0.011645 0.0155925 2024-09-02T10:12:33.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:33.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:33.584 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:152239 2.9 2:9e1d182d:::benchmark_data_smithi187_45882_object54488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:34.912114+0000", 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 478.66975995299998, 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.241466075, 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.585 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 152239 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912114+0000", 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912114+0000", 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912115+0000", 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5260000000000001e-06 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912120+0000", 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.045e-06 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912133+0000", 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.295e-05 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912139+0000", 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8769999999999998e-06 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912187+0000", 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7899000000000003e-05 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912277+0000", 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9938999999999994e-05 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.912348+0000", 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1512000000000002e-05 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.916871+0000", 2024-09-02T10:12:33.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045222650000000001 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.917151+0000", 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028088700000000001 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153516+0000", 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.236364713 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153553+0000", 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6801999999999998e-05 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153580+0000", 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6659999999999999e-05 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:152239 2.9 e182/92 ondisk, result = 0)", 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:34.916225+0000", 2024-09-02T10:12:33.591 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 478.665648127, 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.237360295, 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.916225+0000", 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.916225+0000", 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.916227+0000", 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2699999999999999e-06 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:34.916232+0000", 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1479999999999997e-06 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153424+0000", 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23719195500000001 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153429+0000", 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2639999999999999e-06 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153506+0000", 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7212999999999994e-05 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153512+0000", 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2430000000000002e-06 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.153586+0000", 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3201999999999995e-05 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.595 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(2.4 182/131 [PushReplyOp(2:27b97130:::benchmark_data_smithi187_45882_object78555:head)])", 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:35.629108+0000", 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 477.95276529199998, 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25949049200000002, 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.629108+0000", 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.629108+0000", 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.629109+0000", 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4600000000000003e-07 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.629113+0000", 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3440000000000003e-06 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.887962+0000", 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25884861100000001 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.887969+0000", 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2559999999999996e-06 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.888057+0000", 2024-09-02T10:12:33.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7391000000000004e-05 2024-09-02T10:12:33.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:35.888599+0000", 2024-09-02T10:12:33.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00054204399999999999 2024-09-02T10:12:33.600 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.600 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.600 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.600 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:185649 2.9 2:91f1d57a:::benchmark_data_smithi187_45882_object87898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:28.374152+0000", 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 425.207721816, 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23692353099999999, 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 185649 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374152+0000", 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374152+0000", 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374154+0000", 2024-09-02T10:12:33.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.866e-06 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374160+0000", 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6159999999999998e-06 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374172+0000", 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1936999999999999e-05 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374180+0000", 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8900000000000007e-06 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374235+0000", 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4558999999999997e-05 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374334+0000", 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9807000000000002e-05 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374439+0000", 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010433399999999999 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.382018+0000", 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075792450000000001 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.382505+0000", 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00048756100000000002 2024-09-02T10:12:33.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.610992+0000", 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22848654099999999 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611029+0000", 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7431000000000001e-05 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611075+0000", 2024-09-02T10:12:33.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5744000000000001e-05 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:185654 2.9 2:9a62ca65:::benchmark_data_smithi187_45882_object87903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e211)", 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:28.374759+0000", 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 425.20711429200003, 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23643328899999999, 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.607 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 185654 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374759+0000", 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374759+0000", 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374760+0000", 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6499999999999999e-07 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374763+0000", 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4599999999999999e-06 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374772+0000", 2024-09-02T10:12:33.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7199999999999995e-06 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.374777+0000", 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5519999999999998e-06 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.378482+0000", 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037049370000000002 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.378627+0000", 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145181 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.378744+0000", 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117475 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.382119+0000", 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033746779999999999 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.388940+0000", 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068215849999999998 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611131+0000", 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.222191004 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611165+0000", 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3868999999999997e-05 2024-09-02T10:12:33.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:28.611192+0000", 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7163000000000001e-05 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:12888 3.4 3:25569122:::benchmark_data_smithi187_59989_object12887:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:06:11.023080+0000", 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 382.55879374800003, 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24377531899999999, 2024-09-02T10:12:33.613 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 12888 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023080+0000", 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023080+0000", 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023081+0000", 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1599999999999999e-06 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023124+0000", 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3371999999999997e-05 2024-09-02T10:12:33.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023137+0000", 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2269999999999999e-05 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023142+0000", 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.48e-06 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023231+0000", 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9326000000000005e-05 2024-09-02T10:12:33.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023350+0000", 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011908000000000001 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.023487+0000", 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013629500000000001 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024008+0000", 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052135500000000004 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024495+0000", 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000487109 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266794+0000", 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.242299019 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266826+0000", 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2268999999999999e-05 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266855+0000", 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8583999999999999e-05 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.619 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:12888 3.4 e233/226 ondisk, result = 0)", 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:06:11.024330+0000", 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 382.55754298400001, 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.242529825, 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024330+0000", 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024330+0000", 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024331+0000", 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0299999999999999e-07 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.024334+0000", 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1070000000000001e-06 2024-09-02T10:12:33.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266748+0000", 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24241418300000001 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266753+0000", 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7140000000000001e-06 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266786+0000", 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3408000000000003e-05 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266791+0000", 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5879999999999999e-06 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:06:11.266860+0000", 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9321999999999998e-05 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "rep_scrubmap(3.4 e226 from shard 5)", 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:32.970366+0000", 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 300.61150785699999, 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.45714139399999998, 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.970366+0000", 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.970366+0000", 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.970366+0000", 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1800000000000005e-07 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.970394+0000", 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.722e-05 2024-09-02T10:12:33.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.427289+0000", 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.456895524 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.427298+0000", 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7299999999999994e-06 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.427367+0000", 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8763999999999994e-05 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.427382+0000", 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5041999999999999e-05 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.427507+0000", 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125296 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11531 4.9 4:91b88f89:::benchmark_data_smithi187_76542_object11530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.453697+0000", 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 83.128176268000004, 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.254224538, 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11531 2024-09-02T10:12:33.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.453697+0000", 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.453697+0000", 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.453698+0000", 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2589999999999999e-06 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.453741+0000", 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2327999999999999e-05 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.453752+0000", 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1124000000000001e-05 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.453759+0000", 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1049999999999997e-06 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.453818+0000", 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8801999999999997e-05 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.453951+0000", 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013299000000000001 2024-09-02T10:12:33.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,6", 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.454100+0000", 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014928599999999999 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.454839+0000", 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073917399999999999 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.458039+0000", 2024-09-02T10:12:33.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003199974 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.707764+0000", 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24972475799999999 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.707868+0000", 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103805 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.707922+0000", 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3933e-05 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11549 4.f 4:fdcceb31:::benchmark_data_smithi187_76542_object11548:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.466618+0000", 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 83.115255079999997, 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25702618900000002, 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.634 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11549 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466618+0000", 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466618+0000", 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466619+0000", 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6099999999999999e-07 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466662+0000", 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2951000000000003e-05 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466671+0000", 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9600000000000006e-06 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466676+0000", 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0019999999999997e-06 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466738+0000", 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1978999999999997e-05 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466884+0000", 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014551599999999999 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.467035+0000", 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015158899999999999 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.468034+0000", 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00099856899999999993 2024-09-02T10:12:33.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.468192+0000", 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000157918 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.723541+0000", 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25534909900000002 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.723597+0000", 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6008999999999999e-05 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.723645+0000", 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7636000000000003e-05 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11564 4.f 4:f03605e0:::benchmark_data_smithi187_76542_object11563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.523090+0000", 2024-09-02T10:12:33.640 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 83.058783378000001, 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26754456500000001, 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11564 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523090+0000", 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523090+0000", 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523091+0000", 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3069999999999999e-06 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523141+0000", 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9268000000000003e-05 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523152+0000", 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0915e-05 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523157+0000", 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7450000000000001e-06 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523203+0000", 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5537000000000001e-05 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523312+0000", 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010900000000000001 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523417+0000", 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105166 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.524047+0000", 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063040700000000004 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.524435+0000", 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038750699999999999 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.790567+0000", 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.266131952 2024-09-02T10:12:33.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.790609+0000", 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1706999999999999e-05 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.790635+0000", 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6054000000000002e-05 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.646 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:25334 4.a 4:5b4ee755:::benchmark_data_smithi187_76542_object25333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:25.930302+0000", 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 67.651571923999995, 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24139142099999999, 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 25334 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.930302+0000", 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.930302+0000", 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.930303+0000", 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.206e-06 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.930357+0000", 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4558999999999997e-05 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.930369+0000", 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2140999999999999e-05 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.930377+0000", 2024-09-02T10:12:33.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2740000000000002e-06 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.930714+0000", 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033763199999999998 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.930919+0000", 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020493400000000001 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:12:33.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.931093+0000", 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000173363 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.932168+0000", 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001075517 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:25.934167+0000", 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001999018 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:26.171209+0000", 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23704217799999999 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:26.171248+0000", 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8349000000000002e-05 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:26.171693+0000", 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044525000000000001 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.652 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:41912 4.f 4:f03c9d73:::benchmark_data_smithi187_76542_object41911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.767259+0000", 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.814614237000001, 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52722899400000001, 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:33.653 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 41912 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767259+0000", 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767259+0000", 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767260+0000", 2024-09-02T10:12:33.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2999999999999999e-07 2024-09-02T10:12:33.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767309+0000", 2024-09-02T10:12:33.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8523999999999998e-05 2024-09-02T10:12:33.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:33.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767321+0000", 2024-09-02T10:12:33.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-02T10:12:33.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767331+0000", 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.01e-05 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767384+0000", 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3012e-05 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767516+0000", 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131166 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767648+0000", 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000132106 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.768976+0000", 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013279559999999999 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.774494+0000", 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055181090000000002 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294421+0000", 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51992769000000005 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294459+0000", 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7832e-05 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294488+0000", 2024-09-02T10:12:33.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8980999999999999e-05 2024-09-02T10:12:33.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.661 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.661 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.18 353/344 [PushOp(4:1c2dd2e1:::benchmark_data_smithi187_76542_object24960:head, version: 335'1638, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1c2dd2e1:::benchmark_data_smithi187_76542_object24960:head@335'1638, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:12:33.661 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:12:33.661 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 45.804980501999999, 2024-09-02T10:12:33.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29217855399999998, 2024-09-02T10:12:33.661 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8299999999999997e-07 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.777026+0000", 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013260400000000001 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068430+0000", 2024-09-02T10:12:33.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29140371700000001 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068439+0000", 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2909999999999996e-06 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068496+0000", 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6597e-05 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068504+0000", 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2199999999999992e-06 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.069072+0000", 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056764200000000004 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:44135 4.f 4:fa3c3350:::benchmark_data_smithi187_76542_object44134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-02T10:12:33.665 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:50.893955+0000", 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.687918228000001, 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32054305900000002, 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 44135 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.893955+0000", 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.893955+0000", 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.893957+0000", 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.604e-06 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894004+0000", 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6853e-05 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894015+0000", 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.129e-05 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894021+0000", 2024-09-02T10:12:33.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6049999999999997e-06 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894061+0000", 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0114999999999998e-05 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894142+0000", 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1724000000000006e-05 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:12:33.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894228+0000", 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5729000000000005e-05 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.895543+0000", 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001314747 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900651+0000", 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051081329999999999 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.214312+0000", 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31366078800000002 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.214448+0000", 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013631299999999999 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.214498+0000", 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0158000000000003e-05 2024-09-02T10:12:33.671 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(4.16 356/345 [PushReplyOp(4:6d00321c:::benchmark_data_smithi187_76542_object7271:head)])", 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:50.900564+0000", 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.681309386999999, 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31795428799999997, 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900564+0000", 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900564+0000", 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900565+0000", 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-02T10:12:33.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900569+0000", 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7049999999999998e-06 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.217556+0000", 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31698648099999999 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.217564+0000", 2024-09-02T10:12:33.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.943e-06 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.217625+0000", 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0806e-05 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.218518+0000", 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00089372300000000002 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90780 4.f 4:f9a425ff:::benchmark_data_smithi187_76542_object90779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.016735+0000", 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.5651382620000001, 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28756859099999998, 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90780 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016735+0000", 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016735+0000", 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016736+0000", 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.540000000000001e-07 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016790+0000", 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3933e-05 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016802+0000", 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1605e-05 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016807+0000", 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0869999999999997e-06 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016846+0000", 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9502000000000002e-05 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016939+0000", 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2520000000000002e-05 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.017047+0000", 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010813400000000001 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.020060+0000", 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030132090000000002 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.031359+0000", 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011299346 2024-09-02T10:12:33.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304201+0000", 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.272841522 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304259+0000", 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7732000000000002e-05 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304304+0000", 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5046999999999997e-05 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90797 4.f 4:f65eb26e:::benchmark_data_smithi187_76542_object90796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.028192+0000", 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.5536816500000001, 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.276265435, 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.682 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90797 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028192+0000", 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028192+0000", 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028193+0000", 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3740000000000001e-06 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028251+0000", 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8097000000000003e-05 2024-09-02T10:12:33.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028264+0000", 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2223e-05 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028270+0000", 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6290000000000004e-06 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028310+0000", 2024-09-02T10:12:33.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9527000000000003e-05 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028410+0000", 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9959999999999998e-05 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028501+0000", 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0976000000000004e-05 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.030870+0000", 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002369491 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034987+0000", 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041169179999999998 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304357+0000", 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26936986000000002 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304419+0000", 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1709000000000002e-05 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304457+0000", 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8671000000000003e-05 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.688 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90800 4.f 4:f0d21d3f:::benchmark_data_smithi187_76542_object90799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.032169+0000", 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.549704763, 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27243329199999999, 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90800 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032169+0000", 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032169+0000", 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032170+0000", 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7699999999999992e-07 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032221+0000", 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1365e-05 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032233+0000", 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2102e-05 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032240+0000", 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6669999999999999e-06 2024-09-02T10:12:33.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032277+0000", 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7024000000000003e-05 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032359+0000", 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2163999999999995e-05 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032440+0000", 2024-09-02T10:12:33.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0870000000000003e-05 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034544+0000", 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021037590000000002 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300042+0000", 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26549787899999999 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:33.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304502+0000", 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044609840000000003 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304561+0000", 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8257999999999997e-05 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304602+0000", 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1242999999999998e-05 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:90800 4.f e376/318 ondisk, result = 0)", 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.546943687, 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26511866899999997, 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:12:33.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8500000000000002e-07 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034933+0000", 2024-09-02T10:12:33.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2229999999999999e-06 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.298510+0000", 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26357608700000001 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.298516+0000", 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.657e-06 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300029+0000", 2024-09-02T10:12:33.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015128190000000001 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300036+0000", 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9280000000000004e-06 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300048+0000", 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2469999999999999e-05 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.698 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:33.699 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:33.699 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:12:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:33.700 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:33.700 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:33.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:33.825 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:33.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:33.926 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:33.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:33.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:34.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:34.072 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:34.072 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:34.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:34.196 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:34.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:34.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:33 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:33 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:12:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:33 smithi187 ceph-mon[20072]: osdmap e377: 12 total, 12 up, 12 in 2024-09-02T10:12:34.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:33 smithi187 ceph-mon[20072]: pgmap v904: 57 pgs: 57 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-02T10:12:34.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:34.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:34.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:33 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:33 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:12:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:33 smithi162 ceph-mon[20279]: osdmap e377: 12 total, 12 up, 12 in 2024-09-02T10:12:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:33 smithi162 ceph-mon[20279]: pgmap v904: 57 pgs: 57 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-02T10:12:34.422 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:33 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:33 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:12:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:33 smithi151 ceph-mon[16639]: osdmap e377: 12 total, 12 up, 12 in 2024-09-02T10:12:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:33 smithi151 ceph-mon[16639]: pgmap v904: 57 pgs: 57 active+clean; 5.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-02T10:12:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:34.444 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:34.445 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:34.509 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total time run: 90.7871 2024-09-02T10:12:34.509 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total writes made: 92333 2024-09-02T10:12:34.509 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Write size: 65536 2024-09-02T10:12:34.509 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object size: 65536 2024-09-02T10:12:34.509 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Bandwidth (MB/sec): 63.5642 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Bandwidth: 28.676 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max bandwidth (MB/sec): 137.375 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min bandwidth (MB/sec): 0 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average IOPS: 1017 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev IOPS: 458.827 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max IOPS: 2198 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min IOPS: 0 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average Latency(s): 0.0156315 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Latency(s): 0.0638852 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max latency(s): 3.02533 2024-09-02T10:12:34.510 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min latency(s): 0.00103182 2024-09-02T10:12:34.516 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Cleaning up (deleting benchmark objects) 2024-09-02T10:12:34.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:34.573 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:34.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:34.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:34.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:34.697 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:34.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:34.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:34.822 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:34.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:34.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:34.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:34.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:34.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:34.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:35.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:35.073 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:35.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:35.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:12:35.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:35.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:35.221 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:35.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:35.255 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.7 pg_upmap_items mapping to [6->4,7->5,10->11] 2024-09-02T10:12:35.256 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:34 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:12:35.256 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:34 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:12:35.256 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4027407097' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [6, 4, 7, 5, 10, 11]} : dispatch 2024-09-02T10:12:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:34 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:12:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:34 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:12:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4027407097' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [6, 4, 7, 5, 10, 11]} : dispatch 2024-09-02T10:12:35.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:35.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:34 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:12:35.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:34 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:12:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4027407097' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [6, 4, 7, 5, 10, 11]} : dispatch 2024-09-02T10:12:35.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:35.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:35.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:35.457 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:35.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:35.483 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:35.584 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:35.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:35.606 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:35.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:35.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:35.731 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:35.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:35.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:35.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:35.860 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:35.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:35.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:35.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:35.987 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:35.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:36.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:36.112 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:36.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:36.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:36.237 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:36.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:36.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:35 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:12:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:35 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:12:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:35 smithi187 ceph-mon[20072]: 4.b scrub starts 2024-09-02T10:12:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:35 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:12:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:35 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:12:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4027407097' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [6, 4, 7, 5, 10, 11]}]': finished 2024-09-02T10:12:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:35 smithi187 ceph-mon[20072]: osdmap e378: 12 total, 12 up, 12 in 2024-09-02T10:12:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:35 smithi187 ceph-mon[20072]: 4.e scrub starts 2024-09-02T10:12:36.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:35 smithi187 ceph-mon[20072]: pgmap v906: 57 pgs: 57 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 857 op/s 2024-09-02T10:12:36.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:35 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:12:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:35 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:12:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:35 smithi162 ceph-mon[20279]: 4.b scrub starts 2024-09-02T10:12:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:35 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:12:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:35 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:12:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4027407097' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [6, 4, 7, 5, 10, 11]}]': finished 2024-09-02T10:12:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:35 smithi162 ceph-mon[20279]: osdmap e378: 12 total, 12 up, 12 in 2024-09-02T10:12:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:35 smithi162 ceph-mon[20279]: 4.e scrub starts 2024-09-02T10:12:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:35 smithi162 ceph-mon[20279]: pgmap v906: 57 pgs: 57 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 857 op/s 2024-09-02T10:12:36.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:35 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:12:36.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:35 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:12:36.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:35 smithi151 ceph-mon[16639]: 4.b scrub starts 2024-09-02T10:12:36.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:35 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:12:36.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:35 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:12:36.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4027407097' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [6, 4, 7, 5, 10, 11]}]': finished 2024-09-02T10:12:36.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:35 smithi151 ceph-mon[16639]: osdmap e378: 12 total, 12 up, 12 in 2024-09-02T10:12:36.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:35 smithi151 ceph-mon[16639]: 4.e scrub starts 2024-09-02T10:12:36.362 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:35 smithi151 ceph-mon[16639]: pgmap v906: 57 pgs: 57 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 857 op/s 2024-09-02T10:12:36.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:36.364 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:36.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:36.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:36.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:36.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:36.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:36.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:36.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:36.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:36.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:36.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:36.755 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:36.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:36.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:36.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:36.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:36.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:37.012 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:37.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:37.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:37.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:37.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:37.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:37.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:36 smithi151 ceph-mon[16639]: 4.17 scrub starts 2024-09-02T10:12:37.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:36 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:12:37.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:36 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:12:37.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:36 smithi151 ceph-mon[16639]: 4.17 scrub ok 2024-09-02T10:12:37.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:36 smithi151 ceph-mon[16639]: osdmap e379: 12 total, 12 up, 12 in 2024-09-02T10:12:37.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:36 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:12:37.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:36 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:12:37.178 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:36 smithi151 ceph-mon[16639]: 4.b scrub ok 2024-09-02T10:12:37.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:36 smithi187 ceph-mon[20072]: 4.17 scrub starts 2024-09-02T10:12:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:36 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:12:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:36 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:12:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:36 smithi187 ceph-mon[20072]: 4.17 scrub ok 2024-09-02T10:12:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:36 smithi187 ceph-mon[20072]: osdmap e379: 12 total, 12 up, 12 in 2024-09-02T10:12:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:36 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:12:37.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:36 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:12:37.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:36 smithi187 ceph-mon[20072]: 4.b scrub ok 2024-09-02T10:12:37.266 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:37.269 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:37.270 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:37.278 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:36 smithi162 ceph-mon[20279]: 4.17 scrub starts 2024-09-02T10:12:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:36 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:12:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:36 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:12:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:36 smithi162 ceph-mon[20279]: 4.17 scrub ok 2024-09-02T10:12:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:36 smithi162 ceph-mon[20279]: osdmap e379: 12 total, 12 up, 12 in 2024-09-02T10:12:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:36 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:12:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:36 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:12:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:36 smithi162 ceph-mon[20279]: 4.b scrub ok 2024-09-02T10:12:37.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:37.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:37.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:37.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:37.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:37.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:37.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:37.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:37.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:37.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:37.771 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:37.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:37.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:37.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:37.881 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:12:37.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:12:37.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:37.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:37.908 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:12:37.908 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:12:37.934 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:12:37.934 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:12:37.958 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:12:37.958 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:12:37.985 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:12:37.985 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:12:38.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:38.014 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:12:38.014 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:12:38.022 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:38.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:38.047 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:12:38.047 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:12:38.078 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:12:38.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:12:38.110 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:12:38.110 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:12:38.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:37 smithi151 ceph-mon[16639]: 4.e scrub ok 2024-09-02T10:12:38.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:37 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:12:38.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:37 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:12:38.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:37 smithi151 ceph-mon[16639]: pgmap v908: 57 pgs: 57 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 356 op/s 2024-09-02T10:12:38.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1107936900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:38.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:38.144 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:12:38.144 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:12:38.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:38.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:38.173 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:12:38.173 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:12:38.237 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:12:38.238 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:12:38.250 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:37 smithi187 ceph-mon[20072]: 4.e scrub ok 2024-09-02T10:12:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:37 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:12:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:37 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:12:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:37 smithi187 ceph-mon[20072]: pgmap v908: 57 pgs: 57 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 356 op/s 2024-09-02T10:12:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1107936900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:38.277 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:38.278 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:38.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:37 smithi162 ceph-mon[20279]: 4.e scrub ok 2024-09-02T10:12:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:37 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:12:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:37 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:12:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:37 smithi162 ceph-mon[20279]: pgmap v908: 57 pgs: 57 active+clean; 5.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s wr, 356 op/s 2024-09-02T10:12:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1107936900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:38.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:38.403 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:38.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:38.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:38.533 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:38.534 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:38.595 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:12:38.595 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:12:38.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:38.657 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:38.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:38.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:38.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:38.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:38.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:38.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:38.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:38.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:39.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:39.044 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:39.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:39.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:39.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:39.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:39.171 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:39.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:39 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:12:39.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:39 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:12:39.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:12:39.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:39.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:39.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:39.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:39 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:12:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:39 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:12:39.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:12:39.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:39.424 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:39.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:39 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:12:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:39 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:12:39.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:39.548 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:39.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:39.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:39.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:39.673 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:39.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:39.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:39.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:39.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:39.804 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:39.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:39.928 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:39.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:39.995 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:40.015 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:40.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:40.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-02T10:12:40.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]} : dispatch 2024-09-02T10:12:40.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:40 smithi151 ceph-mon[16639]: pgmap v909: 57 pgs: 57 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 162 B/s rd, 5.2 MiB/s wr, 686 op/s 2024-09-02T10:12:40.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:40.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:40.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:40.193 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:40.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:40.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:40.317 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:40.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:40.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-02T10:12:40.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]} : dispatch 2024-09-02T10:12:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:40 smithi162 ceph-mon[20279]: pgmap v909: 57 pgs: 57 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 162 B/s rd, 5.2 MiB/s wr, 686 op/s 2024-09-02T10:12:40.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:40.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:40.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]} : dispatch 2024-09-02T10:12:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]} : dispatch 2024-09-02T10:12:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:40 smithi187 ceph-mon[20072]: pgmap v909: 57 pgs: 57 active+clean; 5.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 162 B/s rd, 5.2 MiB/s wr, 686 op/s 2024-09-02T10:12:40.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:40.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:40.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:40.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:40.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:40.692 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:40.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:40.794 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:40.808 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:12:40.808 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:12:40.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:40.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:40.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:40.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:40.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:41.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:41.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:41.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:41.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:41.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:41 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:12:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:41 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:12:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:41 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:12:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:41 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:12:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:41 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-02T10:12:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:41 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-09-02T10:12:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:41 smithi151 ceph-mon[16639]: osdmap e380: 12 total, 12 up, 12 in 2024-09-02T10:12:41.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:41.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:41.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:41.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:41.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:41.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:41 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:12:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:41 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:12:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:41 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:12:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:41 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:12:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:41 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-02T10:12:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:41 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-09-02T10:12:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:41 smithi162 ceph-mon[20279]: osdmap e380: 12 total, 12 up, 12 in 2024-09-02T10:12:41.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:41.436 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:41.437 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:41 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:12:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:41 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:12:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:41 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:12:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:41 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:12:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:41 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-02T10:12:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:41 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-09-02T10:12:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:41 smithi187 ceph-mon[20072]: osdmap e380: 12 total, 12 up, 12 in 2024-09-02T10:12:41.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:41.559 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:41.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:41.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:41.683 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:41.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:41.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:41.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:41.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:41.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:41.830 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:12:41.831 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:12:41.831 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-09-02T10:12:41.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd out 2 2024-09-02T10:12:41.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:12:41.912 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:41.935 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:41.936 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:42.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:42.059 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:12:42.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:42.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:42.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:42.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:42.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:42.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:42.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: 4.a scrub starts 2024-09-02T10:12:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:12:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:12:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: osdmap e381: 12 total, 12 up, 12 in 2024-09-02T10:12:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3136070581' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: pgmap v912: 57 pgs: 1 peering, 56 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 B/s wr, 469 op/s 2024-09-02T10:12:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: 4.a scrub ok 2024-09-02T10:12:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:12:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:12:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3136070581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:12:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:42 smithi151 ceph-mon[16639]: osdmap e382: 12 total, 12 up, 12 in 2024-09-02T10:12:42.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:42.309 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:42.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:42.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: 4.a scrub starts 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: osdmap e381: 12 total, 12 up, 12 in 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3136070581' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: pgmap v912: 57 pgs: 1 peering, 56 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 B/s wr, 469 op/s 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: 4.a scrub ok 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3136070581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:12:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:42 smithi162 ceph-mon[20279]: osdmap e382: 12 total, 12 up, 12 in 2024-09-02T10:12:42.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:42.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:12:42.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:42.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:42.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: 4.a scrub starts 2024-09-02T10:12:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:12:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:12:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: osdmap e381: 12 total, 12 up, 12 in 2024-09-02T10:12:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3136070581' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:12:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: pgmap v912: 57 pgs: 1 peering, 56 active+clean; 5.3 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 B/s wr, 469 op/s 2024-09-02T10:12:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: 4.a scrub ok 2024-09-02T10:12:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:12:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:12:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3136070581' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:12:42.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:42 smithi187 ceph-mon[20072]: osdmap e382: 12 total, 12 up, 12 in 2024-09-02T10:12:42.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:42.563 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:42.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:42.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:42.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:42.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:42.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:42.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:42.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:42.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:42.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:42.958 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:42.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:42.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:43.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:43.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:43.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:43.183 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:43.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:43.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:43.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:43.231 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:43.241 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:43.245 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:43.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:43.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:43.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:43.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:43.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:43 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:12:43.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:43 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:12:43.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:43 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:12:43.373 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:43 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:12:43.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:43.468 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:43.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:43 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:12:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:43 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:12:43.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:43.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:43.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:43.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:43.725 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:43.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:43.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:43.849 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:43.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:43.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:43.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:43.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:43.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:43.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:44.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:44.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:44.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:44.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:44.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:44.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:44.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:44 smithi162 ceph-mon[20279]: 4.1 scrub starts 2024-09-02T10:12:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:44 smithi162 ceph-mon[20279]: 4.1 scrub ok 2024-09-02T10:12:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:44 smithi162 ceph-mon[20279]: pgmap v914: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.4 MiB/s wr, 1.08k op/s; 20/256107 objects degraded (0.008%); 5322/256107 objects misplaced (2.078%); 21 KiB/s, 0 objects/s recovering 2024-09-02T10:12:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:44 smithi162 ceph-mon[20279]: 4.4 scrub starts 2024-09-02T10:12:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:44 smithi162 ceph-mon[20279]: 4.4 scrub ok 2024-09-02T10:12:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:44 smithi162 ceph-mon[20279]: osdmap e383: 12 total, 12 up, 12 in 2024-09-02T10:12:44.357 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:44.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:44 smithi151 ceph-mon[16639]: 4.1 scrub starts 2024-09-02T10:12:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:44 smithi151 ceph-mon[16639]: 4.1 scrub ok 2024-09-02T10:12:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:44 smithi151 ceph-mon[16639]: pgmap v914: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.4 MiB/s wr, 1.08k op/s; 20/256107 objects degraded (0.008%); 5322/256107 objects misplaced (2.078%); 21 KiB/s, 0 objects/s recovering 2024-09-02T10:12:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:44 smithi151 ceph-mon[16639]: 4.4 scrub starts 2024-09-02T10:12:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:44 smithi151 ceph-mon[16639]: 4.4 scrub ok 2024-09-02T10:12:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:44 smithi151 ceph-mon[16639]: osdmap e383: 12 total, 12 up, 12 in 2024-09-02T10:12:44.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:44.486 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:44.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:44.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:44 smithi187 ceph-mon[20072]: 4.1 scrub starts 2024-09-02T10:12:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:44 smithi187 ceph-mon[20072]: 4.1 scrub ok 2024-09-02T10:12:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:44 smithi187 ceph-mon[20072]: pgmap v914: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 5.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 2.4 MiB/s wr, 1.08k op/s; 20/256107 objects degraded (0.008%); 5322/256107 objects misplaced (2.078%); 21 KiB/s, 0 objects/s recovering 2024-09-02T10:12:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:44 smithi187 ceph-mon[20072]: 4.4 scrub starts 2024-09-02T10:12:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:44 smithi187 ceph-mon[20072]: 4.4 scrub ok 2024-09-02T10:12:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:44 smithi187 ceph-mon[20072]: osdmap e383: 12 total, 12 up, 12 in 2024-09-02T10:12:44.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:44.612 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:44.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:44.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:44.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:44.737 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:44.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:44.770 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:12:44.770 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:12:44.798 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:12:44.798 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:12:44.829 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:12:44.830 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:12:44.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:44.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:12:44.862 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:12:44.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:44.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:44.899 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:12:44.900 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:12:44.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:12:44.927 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:12:44.956 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:12:44.957 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:12:44.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:44.985 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:12:44.985 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:12:44.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:44.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:44.993 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:45.011 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:12:45.011 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:12:45.040 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:12:45.040 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:12:45.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:12:45.072 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:12:45.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:45.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:12:45.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:12:45.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:45.119 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:45.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:12:45.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:45.244 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:45.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:45.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:45 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 20/256107 objects degraded (0.008%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:12:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1366633615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:45.346 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:45.364 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:45 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 20/256107 objects degraded (0.008%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:12:45.365 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1366633615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:45.370 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:45.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:45.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:45.497 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:45.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:45 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 20/256107 objects degraded (0.008%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:12:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1366633615' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:45.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:45.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:45.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:45.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:45 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:12:45.274+0000 7fe3ffed4640 -1 mon.a@0(leader).osd e383 definitely_dead 0 2024-09-02T10:12:45.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:45.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:45.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:45.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:45.885 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:45.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:45.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:46.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:46.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:46.084 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.2. 2024-09-02T10:12:46.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:46.137 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:46.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:46.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:46.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:46.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:46.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:46.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:46 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:12:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:46 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:12:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:46 smithi162 ceph-mon[20279]: pgmap v916: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 20/251952 objects degraded (0.008%); 5322/251952 objects misplaced (2.112%); 25 KiB/s, 0 objects/s recovering 2024-09-02T10:12:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/29524897' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-02T10:12:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:46 smithi162 ceph-mon[20279]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-02T10:12:46.362 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:46.384 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:46.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:46.385 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:46.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:46 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:12:46.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:46 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:12:46.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:46 smithi151 ceph-mon[16639]: pgmap v916: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 20/251952 objects degraded (0.008%); 5322/251952 objects misplaced (2.112%); 25 KiB/s, 0 objects/s recovering 2024-09-02T10:12:46.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/29524897' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-02T10:12:46.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:46 smithi151 ceph-mon[16639]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-02T10:12:46.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:46 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:12:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:46 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:12:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:46 smithi187 ceph-mon[20072]: pgmap v916: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 754 op/s; 20/251952 objects degraded (0.008%); 5322/251952 objects misplaced (2.112%); 25 KiB/s, 0 objects/s recovering 2024-09-02T10:12:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/29524897' entity='client.admin' cmd={"prefix": "osd out", "ids": ["2"]} : dispatch 2024-09-02T10:12:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:46 smithi187 ceph-mon[20072]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-02T10:12:46.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:46.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:46.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:46.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:46.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:46.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:46.634 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:97747 2.1 2:8268f1a3:::benchmark_data_smithi187_45882_object97746:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:05.429537+0000", 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 581.19960399700005, 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75786974200000001, 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:46.635 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 97747 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429537+0000", 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429537+0000", 2024-09-02T10:12:46.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429538+0000", 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0690000000000001e-06 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429603+0000", 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4875000000000005e-05 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429614+0000", 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0604999999999999e-05 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.429620+0000", 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7070000000000003e-06 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.432050+0000", 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002429381 2024-09-02T10:12:46.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.432155+0000", 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010553899999999999 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,3", 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.432282+0000", 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012681899999999999 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:46.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.435226+0000", 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029439179999999998 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.436254+0000", 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010281400000000001 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187337+0000", 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75108334499999996 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:46.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187376+0000", 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8974000000000003e-05 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187407+0000", 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0369999999999999e-05 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24451.0:97747 2.1 e137/92 ondisk, result = 0)", 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:05.435888+0000", 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 581.19325310800002, 2024-09-02T10:12:46.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75152487700000004, 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.435888+0000", 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.435888+0000", 2024-09-02T10:12:46.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.435889+0000", 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8999999999999996e-07 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:05.435893+0000", 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3279999999999999e-06 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187266+0000", 2024-09-02T10:12:46.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75137313999999999 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187271+0000", 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3709999999999999e-06 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187326+0000", 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4203999999999998e-05 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187333+0000", 2024-09-02T10:12:46.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8800000000000002e-06 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:06.187413+0000", 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0264000000000003e-05 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:127445 2.1 2.eb347b31 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:12:46.645 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:45.230005+0000", 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 541.39913637999996, 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99042561699999998, 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 127445 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.230005+0000", 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.230005+0000", 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.230006+0000", 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.795e-06 2024-09-02T10:12:46.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.230016+0000", 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4420000000000004e-06 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.230032+0000", 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5849e-05 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.230041+0000", 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8890000000000008e-06 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.230255+0000", 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000214279 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.230282+0000", 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7368999999999998e-05 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220399+0000", 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99011653200000005 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220430+0000", 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1461999999999998e-05 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.650 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:127451 2.1 2.b4f93701 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:45.249229+0000", 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 541.37991255700001, 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97134549000000003, 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 127451 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249229+0000", 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249229+0000", 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249230+0000", 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.204e-06 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249236+0000", 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1999999999999999e-06 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249250+0000", 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4248e-05 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249257+0000", 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1670000000000002e-06 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249329+0000", 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2089999999999996e-05 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249338+0000", 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4980000000000006e-06 2024-09-02T10:12:46.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220565+0000", 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.971227112 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220574+0000", 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9709999999999993e-06 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.655 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:127452 2.1 2.b2636371 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:45.249552+0000", 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 541.37958923600002, 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97105341300000003, 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 127452 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249552+0000", 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249552+0000", 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249553+0000", 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5899999999999995e-07 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249556+0000", 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8970000000000001e-06 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249564+0000", 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3830000000000003e-06 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249573+0000", 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7390000000000005e-06 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249617+0000", 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4468999999999997e-05 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.249624+0000", 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5780000000000004e-06 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220599+0000", 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97097536299999998 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220605+0000", 2024-09-02T10:12:46.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2249999999999997e-06 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:127469 2.1 2.fdb38031 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:45.291954+0000", 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 541.33718667400001, 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.92871181999999997, 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.660 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 127469 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.291954+0000", 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.291954+0000", 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.291956+0000", 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.694e-06 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.291962+0000", 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1070000000000001e-06 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.291974+0000", 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2179e-05 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.291981+0000", 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7220000000000002e-06 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.292083+0000", 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010189899999999999 2024-09-02T10:12:46.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.292093+0000", 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6500000000000008e-06 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220659+0000", 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.92856649499999999 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220666+0000", 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0740000000000004e-06 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:127490 2.1 2.7f417ca1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:45.345646+0000", 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 541.28349486800005, 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.87505255000000004, 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 127490 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.345646+0000", 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.345646+0000", 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.345647+0000", 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6199999999999996e-07 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.345652+0000", 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6269999999999999e-06 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.345662+0000", 2024-09-02T10:12:46.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0389e-05 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.345668+0000", 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2009999999999996e-06 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.345732+0000", 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3830000000000004e-05 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.345739+0000", 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5599999999999999e-06 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220692+0000", 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.87495371899999996 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220699+0000", 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000002e-06 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.669 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:127494 2.1 2.44dea571 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:45.355941+0000", 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 541.27320036200001, 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86479569599999995, 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 127494 2024-09-02T10:12:46.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.355941+0000", 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.355941+0000", 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.355941+0000", 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-02T10:12:46.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.355945+0000", 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8829999999999999e-06 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.355956+0000", 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0397999999999999e-05 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.355962+0000", 2024-09-02T10:12:46.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8610000000000003e-06 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.356015+0000", 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2877999999999998e-05 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.356022+0000", 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0609999999999998e-06 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220730+0000", 2024-09-02T10:12:46.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86470844099999999 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220736+0000", 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3940000000000001e-06 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:127497 2.1 2.6cba70d1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:45.393138+0000", 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 541.23600316500006, 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.82763076599999996, 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.674 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 127497 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.393138+0000", 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.393138+0000", 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.393140+0000", 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9910000000000001e-06 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.393147+0000", 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8900000000000001e-06 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.393163+0000", 2024-09-02T10:12:46.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6336000000000001e-05 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.393175+0000", 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.164e-05 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.393245+0000", 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9827999999999996e-05 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.393253+0000", 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7520000000000002e-06 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220762+0000", 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.82750892200000004 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.220769+0000", 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4069999999999998e-06 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.679 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24451.0:127528 2.7 e156/131 2:e0932104:::benchmark_data_smithi187_45882_object29777:head v 156'7970, mlcod=111'2666)", 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:03:45.658476+0000", 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 540.97066490899999, 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67227498600000002, 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.658476+0000", 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.658476+0000", 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.658477+0000", 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9999999999999996e-07 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:45.658483+0000", 2024-09-02T10:12:46.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1850000000000001e-06 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.327542+0000", 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.66905830200000005 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.327565+0000", 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3574000000000002e-05 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.327654+0000", 2024-09-02T10:12:46.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8524999999999997e-05 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.327679+0000", 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4918000000000001e-05 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.330712+0000", 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003033376 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:03:46.330751+0000", 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9205999999999997e-05 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:158538 2.e 2:7b7285b6:::benchmark_data_smithi187_45882_object60787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:48.063842+0000", 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 478.56529867299997, 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31157125600000002, 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.684 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 158538 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063842+0000", 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063842+0000", 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063843+0000", 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4099999999999999e-07 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063845+0000", 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1969999999999999e-06 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063852+0000", 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2139999999999999e-06 2024-09-02T10:12:46.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063857+0000", 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8679999999999998e-06 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.065509+0000", 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00165193 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.065514+0000", 2024-09-02T10:12:46.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.203e-06 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.343219+0000", 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27770477500000001 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.343377+0000", 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000157665 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:12:46.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.343535+0000", 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015809 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.351593+0000", 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080579429999999997 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.372900+0000", 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021307058 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.372946+0000", 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6341999999999998e-05 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.373000+0000", 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3934999999999998e-05 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.375414+0000", 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024135950000000002 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24451.0:158542 2.e 2:7afc3b5f:::benchmark_data_smithi187_45882_object60791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:04:48.063864+0000", 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 478.56527695300002, 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.332372365, 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24451", 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 158542 2024-09-02T10:12:46.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063864+0000", 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063864+0000", 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063865+0000", 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1399999999999997e-07 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063867+0000", 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.249e-06 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063873+0000", 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8980000000000001e-06 2024-09-02T10:12:46.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.063878+0000", 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0980000000000001e-06 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.065550+0000", 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016725360000000001 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.065559+0000", 2024-09-02T10:12:46.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.149e-06 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.354320+0000", 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28876076899999997 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.354466+0000", 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014552900000000001 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:12:46.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.354607+0000", 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014085899999999999 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.360626+0000", 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060190069999999998 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.375483+0000", 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014856962 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.391270+0000", 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015787359000000001 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.391401+0000", 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130698 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:04:48.396236+0000", 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048358380000000003 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.697 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.698 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(2.e e217,198 2:7ca24583:::benchmark_data_smithi187_45882_object59839:head,190'9890)", 2024-09-02T10:12:46.698 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:05:43.490663+0000", 2024-09-02T10:12:46.698 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 423.13847763699999, 2024-09-02T10:12:46.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.341834682, 2024-09-02T10:12:46.698 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.698 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.698 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:12:46.698 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:43.490663+0000", 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:43.490663+0000", 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:43.490664+0000", 2024-09-02T10:12:46.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7700000000000004e-07 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:43.490668+0000", 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2610000000000004e-06 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:43.832168+0000", 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34149969099999999 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:43.832182+0000", 2024-09-02T10:12:46.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.396e-05 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:43.832260+0000", 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8467000000000006e-05 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:05:43.832498+0000", 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000237626 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.1d,QREQUEST,e:292,from:1,reservation_nonce:2)", 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:09:19.341165+0000", 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 207.287975569, 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34585376800000001, 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:19.341165+0000", 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:19.341165+0000", 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:19.341167+0000", 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.437e-06 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:19.341174+0000", 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3579999999999997e-06 2024-09-02T10:12:46.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:19.686442+0000", 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34526779400000002 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:19.686452+0000", 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7869999999999996e-06 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:19.686769+0000", 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031724699999999999 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:19.686822+0000", 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2774e-05 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:19.687019+0000", 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000197371 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.705 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.706 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.706 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:41902 4.14 e353/351 4:2ffda43f:::benchmark_data_smithi187_76542_object41901:head v 353'2264, mlcod=353'2262)", 2024-09-02T10:12:46.706 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.743167+0000", 2024-09-02T10:12:46.706 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 58.885973610999997, 2024-09-02T10:12:46.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50111127099999997, 2024-09-02T10:12:46.706 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743167+0000", 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743167+0000", 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743168+0000", 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8599999999999996e-07 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743268+0000", 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9846000000000006e-05 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243322+0000", 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50005331600000003 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243327+0000", 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.057e-06 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243388+0000", 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0948e-05 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243407+0000", 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9040000000000001e-05 2024-09-02T10:12:46.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.244241+0000", 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00083470799999999998 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.244279+0000", 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7370000000000003e-05 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90150 4.4 4:22e9747b:::benchmark_data_smithi187_76542_object90149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.040493+0000", 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.588647876, 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.181265486, 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90150 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.711 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040493+0000", 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040493+0000", 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040495+0000", 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4130000000000001e-06 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040588+0000", 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3113000000000001e-05 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040600+0000", 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1834000000000001e-05 2024-09-02T10:12:46.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040605+0000", 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7049999999999996e-06 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040652+0000", 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6643e-05 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040741+0000", 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8797999999999999e-05 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040826+0000", 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5191000000000005e-05 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042413+0000", 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001587605 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042531+0000", 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117157 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221691+0000", 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17916051699999999 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221731+0000", 2024-09-02T10:12:46.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9650999999999997e-05 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221759+0000", 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7858999999999999e-05 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:90150 4.4 e375/341 ondisk, result = 0)", 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.042430+0000", 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.586711040000001, 2024-09-02T10:12:46.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17933455100000001, 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042430+0000", 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042430+0000", 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042431+0000", 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9999999999999996e-07 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042439+0000", 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4810000000000006e-06 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221645+0000", 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17920517899999999 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221649+0000", 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.865e-06 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221683+0000", 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3324000000000001e-05 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221688+0000", 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.887e-06 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221765+0000", 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6915000000000005e-05 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90160 4.4 4:2097a6ca:::benchmark_data_smithi187_76542_object90159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.048271+0000", 2024-09-02T10:12:46.721 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.580869746999999, 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17505789499999999, 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90160 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048271+0000", 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048271+0000", 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048276+0000", 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.938e-06 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048319+0000", 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2793999999999997e-05 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048330+0000", 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.118e-05 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048335+0000", 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2660000000000001e-06 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048380+0000", 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4733e-05 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048474+0000", 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3494999999999998e-05 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048562+0000", 2024-09-02T10:12:46.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8528000000000003e-05 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.049898+0000", 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001336159 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050229+0000", 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033104400000000001 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:12:46.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223275+0000", 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.173045588 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223306+0000", 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0621999999999999e-05 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223329+0000", 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3547999999999999e-05 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.727 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.728 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.728 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:90160 4.4 e375/341 ondisk, result = 0)", 2024-09-02T10:12:46.728 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:12:46.728 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.578785225000001, 2024-09-02T10:12:46.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.172980419, 2024-09-02T10:12:46.728 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.728 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.728 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4600000000000005e-07 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050360+0000", 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.253e-06 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223231+0000", 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.172871466 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223235+0000", 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7309999999999999e-06 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223266+0000", 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1443999999999999e-05 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223272+0000", 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3249999999999998e-06 2024-09-02T10:12:46.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223336+0000", 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4653999999999995e-05 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:90175 4.19 e375/354 4:988493fd:::benchmark_data_smithi187_76542_object90174:head v 375'3736, mlcod=375'3734)", 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.066480+0000", 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.562660736, 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.188910198, 2024-09-02T10:12:46.732 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066480+0000", 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066480+0000", 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:12:46.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066482+0000", 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3939999999999999e-06 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066535+0000", 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3510000000000001e-05 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253612+0000", 2024-09-02T10:12:46.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18707629000000001 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253619+0000", 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9719999999999992e-06 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253697+0000", 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7881999999999997e-05 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:12:46.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253728+0000", 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1118000000000003e-05 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.255346+0000", 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016170609999999999 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.255391+0000", 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4971e-05 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.736 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.737 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.737 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:12:46.737 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:12:46.737 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:12:46.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:46.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:46.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:46.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:46.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:46.883 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:46.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:46.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:47.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:47.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:47.110 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:47.133 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:47.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:47.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:47.257 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:47.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:47.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/29524897' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-02T10:12:47.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:47 smithi162 ceph-mon[20279]: osdmap e384: 12 total, 12 up, 11 in 2024-09-02T10:12:47.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:47.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:47.382 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:47.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:47.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/29524897' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-02T10:12:47.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:47 smithi151 ceph-mon[16639]: osdmap e384: 12 total, 12 up, 11 in 2024-09-02T10:12:47.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:47.504 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:47.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/29524897' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-02T10:12:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:47 smithi187 ceph-mon[20072]: osdmap e384: 12 total, 12 up, 11 in 2024-09-02T10:12:47.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:47.627 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:47.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:47.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:47.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:47.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:47.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:47.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:47.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:47.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:47.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:47.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:47.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:47.999 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:48.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:48.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:48.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:48.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:48.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:48.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:48.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:48 smithi151 ceph-mon[16639]: osdmap e385: 12 total, 12 up, 11 in 2024-09-02T10:12:48.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:48 smithi151 ceph-mon[16639]: pgmap v919: 57 pgs: 1 unknown, 5 peering, 2 active+recovering+undersized+degraded+remapped, 49 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 29/248736 objects degraded (0.012%); 7631/248736 objects misplaced (3.068%); 3.7 MiB/s, 59 objects/s recovering 2024-09-02T10:12:48.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:48 smithi151 ceph-mon[16639]: osdmap e386: 12 total, 12 up, 11 in 2024-09-02T10:12:48.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:48.267 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:48.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:48.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:48.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:48.391 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:48.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:48.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:48.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:12:48.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:48 smithi187 ceph-mon[20072]: osdmap e385: 12 total, 12 up, 11 in 2024-09-02T10:12:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:48 smithi187 ceph-mon[20072]: pgmap v919: 57 pgs: 1 unknown, 5 peering, 2 active+recovering+undersized+degraded+remapped, 49 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 29/248736 objects degraded (0.012%); 7631/248736 objects misplaced (3.068%); 3.7 MiB/s, 59 objects/s recovering 2024-09-02T10:12:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:48 smithi187 ceph-mon[20072]: osdmap e386: 12 total, 12 up, 11 in 2024-09-02T10:12:48.521 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:48.522 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:48 smithi162 ceph-mon[20279]: osdmap e385: 12 total, 12 up, 11 in 2024-09-02T10:12:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:48 smithi162 ceph-mon[20279]: pgmap v919: 57 pgs: 1 unknown, 5 peering, 2 active+recovering+undersized+degraded+remapped, 49 active+clean; 5.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 29/248736 objects degraded (0.012%); 7631/248736 objects misplaced (3.068%); 3.7 MiB/s, 59 objects/s recovering 2024-09-02T10:12:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:48 smithi162 ceph-mon[20279]: osdmap e386: 12 total, 12 up, 11 in 2024-09-02T10:12:48.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:48.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:48.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:48.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:48.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:48.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:48.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:48.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:48.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:49.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:49.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:49.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:49.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:49.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:49.168 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:49.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:49.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:49.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:49.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:49.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:49.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:49.434 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:49.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:49.435 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:49.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:49 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:12:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:49 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:12:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:49 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:12:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:49 smithi151 ceph-mon[16639]: osdmap e387: 12 total, 12 up, 11 in 2024-09-02T10:12:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:49 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:12:49.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:49 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:12:49.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:49 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:12:49.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:49 smithi187 ceph-mon[20072]: osdmap e387: 12 total, 12 up, 11 in 2024-09-02T10:12:49.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:49.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:49.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:49.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:49.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:49 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:12:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:49 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:12:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:49 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:12:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:49 smithi162 ceph-mon[20279]: osdmap e387: 12 total, 12 up, 11 in 2024-09-02T10:12:49.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:49.694 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:49.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:49.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:49.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:49.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:49.833 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:49.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:49.956 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:49.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:49.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:50.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:50.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:50.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:50.188 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:50.215 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:50.216 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:50.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:50.340 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:50.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:50 smithi151 ceph-mon[16639]: pgmap v922: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 unknown, 5 peering, 3 active+recovering+undersized+degraded+remapped, 45 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 8663/242979 objects degraded (3.565%); 9839/242979 objects misplaced (4.049%); 8.3 MiB/s, 133 objects/s recovering 2024-09-02T10:12:50.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:50.468 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:50.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:50 smithi187 ceph-mon[20072]: pgmap v922: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 unknown, 5 peering, 3 active+recovering+undersized+degraded+remapped, 45 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 8663/242979 objects degraded (3.565%); 9839/242979 objects misplaced (4.049%); 8.3 MiB/s, 133 objects/s recovering 2024-09-02T10:12:50.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:50.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:50 smithi162 ceph-mon[20279]: pgmap v922: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 unknown, 5 peering, 3 active+recovering+undersized+degraded+remapped, 45 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 747 op/s; 8663/242979 objects degraded (3.565%); 9839/242979 objects misplaced (4.049%); 8.3 MiB/s, 133 objects/s recovering 2024-09-02T10:12:50.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:50.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:50.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:50.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:50.725 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:50.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:50.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:50.853 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:50.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:50.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:50.982 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:50.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:50.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:51.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:51.108 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:51.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:51.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:51.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:51.239 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:51.240 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:51.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:51.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:51.372 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:51.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:51.386 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:51 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 8663/242979 objects degraded (3.565%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:12:51.386 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:12:51.386 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:12:51.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:51.498 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:51.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:51.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:51 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 8663/242979 objects degraded (3.565%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:12:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:12:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:12:51.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:51 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 8663/242979 objects degraded (3.565%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:12:51.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:12:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:12:51.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:51.632 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:51.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:51.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:51.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:51.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:51.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:51.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:51.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:51.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:52.018 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:52.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:52.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:52.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:52.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:52.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:52.272 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:52.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:52.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:52.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:52.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:52.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:52.407 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:52 smithi187 ceph-mon[20072]: osdmap e388: 12 total, 12 up, 11 in 2024-09-02T10:12:52.407 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:52 smithi187 ceph-mon[20072]: pgmap v924: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 unknown, 5 peering, 3 active+recovering+undersized+degraded+remapped, 45 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 595 op/s; 8663/241293 objects degraded (3.590%); 9547/241293 objects misplaced (3.957%); 7.7 MiB/s, 124 objects/s recovering 2024-09-02T10:12:52.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:52 smithi151 ceph-mon[16639]: osdmap e388: 12 total, 12 up, 11 in 2024-09-02T10:12:52.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:52 smithi151 ceph-mon[16639]: pgmap v924: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 unknown, 5 peering, 3 active+recovering+undersized+degraded+remapped, 45 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 595 op/s; 8663/241293 objects degraded (3.590%); 9547/241293 objects misplaced (3.957%); 7.7 MiB/s, 124 objects/s recovering 2024-09-02T10:12:52.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:52 smithi162 ceph-mon[20279]: osdmap e388: 12 total, 12 up, 11 in 2024-09-02T10:12:52.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:52 smithi162 ceph-mon[20279]: pgmap v924: 57 pgs: 1 activating+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 unknown, 5 peering, 3 active+recovering+undersized+degraded+remapped, 45 active+clean; 4.9 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 595 op/s; 8663/241293 objects degraded (3.590%); 9547/241293 objects misplaced (3.957%); 7.7 MiB/s, 124 objects/s recovering 2024-09-02T10:12:52.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:52.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:52.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:52.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:52.652 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:52.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:52.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:12:52.681 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:52.718 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:52.754 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:52.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:52.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:52.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:52.879 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:52.908 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:52.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:53.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:53.033 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:53.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:53.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:53.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:53.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:53.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:53.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:53.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:53.295 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:53.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:53.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:53.427 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:53.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:53 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:12:53.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:53.553 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:53.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:53.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:53.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:53.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:53 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:12:53.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:53.795 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:12:53.796 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:12:53.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:53.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:53.825 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:12:53.825 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:12:53.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:53 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:12:53.861 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:12:53.888 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:12:53.888 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:12:53.898 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:12:53.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:53.918 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:12:53.918 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:12:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:53.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:53.942 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:53.942 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:12:53.943 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:12:53.944 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:12:53.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:12:53.973 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:12:54.001 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:12:54.002 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:12:54.032 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:12:54.033 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:12:54.043 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:54.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:12:54.059 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:12:54.070 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:54.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:54.086 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:12:54.086 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:12:54.115 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:12:54.115 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:12:54.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:54.194 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:54.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:54.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:54.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:54.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:54.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:54.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:54.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:54.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:54.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:54.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:54.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:54.585 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:54.686 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:54 smithi151 ceph-mon[16639]: pgmap v925: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 3087/234795 objects degraded (1.315%); 18397/234795 objects misplaced (7.835%); 18 MiB/s, 292 objects/s recovering 2024-09-02T10:12:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/329857111' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:12:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:54 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:12:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:54 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:12:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/879509382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:54 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/329857111' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:12:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:54 smithi151 ceph-mon[16639]: osdmap e389: 12 total, 12 up, 11 in 2024-09-02T10:12:54.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:54.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:54.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:54 smithi187 ceph-mon[20072]: pgmap v925: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 3087/234795 objects degraded (1.315%); 18397/234795 objects misplaced (7.835%); 18 MiB/s, 292 objects/s recovering 2024-09-02T10:12:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/329857111' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:12:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:54 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:12:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:54 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:12:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/879509382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:54 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/329857111' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:12:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:54 smithi187 ceph-mon[20072]: osdmap e389: 12 total, 12 up, 11 in 2024-09-02T10:12:54.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:54.833 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:54.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:54.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:54 smithi162 ceph-mon[20279]: pgmap v925: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 3087/234795 objects degraded (1.315%); 18397/234795 objects misplaced (7.835%); 18 MiB/s, 292 objects/s recovering 2024-09-02T10:12:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/329857111' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:12:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:54 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:12:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:54 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:12:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/879509382' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:12:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:54 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:12:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/329857111' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:12:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:54 smithi162 ceph-mon[20279]: osdmap e389: 12 total, 12 up, 11 in 2024-09-02T10:12:54.925 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:12:54.926 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:12:54.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:54.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:54.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:54.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:55.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:55.094 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:55.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:55.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:55.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:55.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:55.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:55.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:55.362 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:55.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:55.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:55.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:55.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:55.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:55.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:55.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:55.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:55.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:55 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 3087/229062 objects degraded (1.348%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:12:55.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:55.745 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:55.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:55 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 3087/229062 objects degraded (1.348%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:12:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:55 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 3087/229062 objects degraded (1.348%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:12:55.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:55.870 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:55.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:55.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:55.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:55.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:56.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:56.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:56.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:56.221 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:56.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:56.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:56.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:56.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:56.387 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:56.389 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:56.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:56.514 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:56.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:56.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:56.641 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:56.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:56 smithi151 ceph-mon[16639]: pgmap v927: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 734 op/s; 3087/229062 objects degraded (1.348%); 18160/229062 objects misplaced (7.928%); 17 MiB/s, 279 objects/s recovering 2024-09-02T10:12:56.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:12:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:56 smithi187 ceph-mon[20072]: pgmap v927: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 734 op/s; 3087/229062 objects degraded (1.348%); 18160/229062 objects misplaced (7.928%); 17 MiB/s, 279 objects/s recovering 2024-09-02T10:12:56.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:56.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:12:56.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:56 smithi162 ceph-mon[20279]: pgmap v927: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.7 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 734 op/s; 3087/229062 objects degraded (1.348%); 18160/229062 objects misplaced (7.928%); 17 MiB/s, 279 objects/s recovering 2024-09-02T10:12:56.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:12:56.895 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:56.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:12:56.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:57.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:57.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:57.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:12:57.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:57.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:57.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:12:57.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:12:57.277 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:57.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:57.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:12:57.307 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:12:57.307 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:12:57.308 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-09-02T10:12:57.308 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-02T10:12:57.308 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-02T10:12:57.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-09-02T10:12:57.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:12:57.409 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:57.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:12:57.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:57.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:57.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:57.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:57.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:57.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:57.695 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:12:57.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:57.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:57.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:57.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:57.920 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:57.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:58.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:58.043 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:58.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:58.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:58.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:58.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:58.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:58.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:58.276 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:12:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-02T10:12:58.276 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:12:58.305 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:58.306 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:58.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:58.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:58.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:58.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:58.555 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:58.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:58.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:58.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:58.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:58.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:12:58 smithi151 ceph-mon[16639]: pgmap v928: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 3087/225540 objects degraded (1.369%); 17395/225540 objects misplaced (7.713%); 20 MiB/s, 316 objects/s recovering 2024-09-02T10:12:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:12:58 smithi187 ceph-mon[20072]: pgmap v928: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 3087/225540 objects degraded (1.369%); 17395/225540 objects misplaced (7.713%); 20 MiB/s, 316 objects/s recovering 2024-09-02T10:12:58.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:12:58.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:58.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:12:58.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:12:58 smithi162 ceph-mon[20279]: pgmap v928: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.6 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 3087/225540 objects degraded (1.369%); 17395/225540 objects misplaced (7.713%); 20 MiB/s, 316 objects/s recovering 2024-09-02T10:12:58.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:12:58.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:58.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:12:59.039 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:59.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:59.072 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:59.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:59.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:12:59.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:12:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:59.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:12:59.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:12:59.340 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:59.341 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:12:59.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:59.466 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:59.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:59.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:59.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:12:59.603 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:12:59.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:59.605 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:12:59.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:12:59.730 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:59.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:59.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:12:59.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:59.856 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:59.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:12:59.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:12:59.981 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:12:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:12:59.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:00.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:00.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:00.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:00.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:00.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:00.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:00.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:00.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:13:00.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:00.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:00.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:00.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:00.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:00.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:00.500 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:00.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:00.623 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:00.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:00 smithi151 ceph-mon[16639]: pgmap v929: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 913 op/s; 3154/219000 objects degraded (1.440%); 15750/219000 objects misplaced (7.192%); 29 MiB/s, 469 objects/s recovering 2024-09-02T10:13:00.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:00.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:00.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:00 smithi187 ceph-mon[20072]: pgmap v929: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 913 op/s; 3154/219000 objects degraded (1.440%); 15750/219000 objects misplaced (7.192%); 29 MiB/s, 469 objects/s recovering 2024-09-02T10:13:00.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:00 smithi162 ceph-mon[20279]: pgmap v929: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.5 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 913 op/s; 3154/219000 objects degraded (1.440%); 15750/219000 objects misplaced (7.192%); 29 MiB/s, 469 objects/s recovering 2024-09-02T10:13:00.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:00.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:00.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:00.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:00.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:01.002 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:01.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:01.054 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:01.105 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:01.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:01.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:01.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:01.254 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:01.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:01.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:01.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:01.388 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:01.389 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:01.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:01.516 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:01.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:01.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:01.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:01.642 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:01.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:01.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:01.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:01 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 3154/217008 objects degraded (1.453%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:01.773 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:01.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:01.792 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:01.792 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:01.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:01 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 3154/217008 objects degraded (1.453%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:01.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:01 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 3154/217008 objects degraded (1.453%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:01.875 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:01.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:01.903 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:02.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:02.029 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:02.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:02.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:02.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:02.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:02.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:02.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:13:02.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:02.281 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:02.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:02.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:02.383 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:02.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:02.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:02.414 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:02.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:02.545 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:02.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:02.546 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:02.647 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:02.670 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:02.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:02.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:02 smithi151 ceph-mon[16639]: pgmap v930: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 809 op/s; 3154/217008 objects degraded (1.453%); 15297/217008 objects misplaced (7.049%); 27 MiB/s, 426 objects/s recovering 2024-09-02T10:13:02.753 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:02 smithi187 ceph-mon[20072]: pgmap v930: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 809 op/s; 3154/217008 objects degraded (1.453%); 15297/217008 objects misplaced (7.049%); 27 MiB/s, 426 objects/s recovering 2024-09-02T10:13:02.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:13:02.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:02 smithi162 ceph-mon[20279]: pgmap v930: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.4 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 809 op/s; 3154/217008 objects degraded (1.453%); 15297/217008 objects misplaced (7.049%); 27 MiB/s, 426 objects/s recovering 2024-09-02T10:13:02.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:02.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:02.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:02.895 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:02.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:13:02.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:02.978 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:02.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:02.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:03.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:03.108 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:03.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:03.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:03.237 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:03.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:03.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:03.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:03.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:03.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:03.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:03.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:03.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:03.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:03.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:03.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:03.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:03.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:03.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:03.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:03.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:03.887 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:03.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:03.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:04.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:04.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:04.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:04.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:04.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:04.141 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:04.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:04.266 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:04.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:04.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:04.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:04.392 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:04.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:04.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:04.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:04.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:04.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:04.650 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:04.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:04.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:04 smithi151 ceph-mon[16639]: pgmap v931: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 3159/212133 objects degraded (1.489%); 13638/212133 objects misplaced (6.429%); 29 MiB/s, 468 objects/s recovering 2024-09-02T10:13:04.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:04 smithi187 ceph-mon[20072]: pgmap v931: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 3159/212133 objects degraded (1.489%); 13638/212133 objects misplaced (6.429%); 29 MiB/s, 468 objects/s recovering 2024-09-02T10:13:04.776 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:04.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:04.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:04.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:04 smithi162 ceph-mon[20279]: pgmap v931: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.3 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 755 op/s; 3159/212133 objects degraded (1.489%); 13638/212133 objects misplaced (6.429%); 29 MiB/s, 468 objects/s recovering 2024-09-02T10:13:04.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:04.904 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:04.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:05.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:05.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:05.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:05.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:05.167 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:05.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:05.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:05.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:05.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:05.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:05.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:05.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:05.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:05.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:05.567 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:05.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:05.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:05.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:05.695 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:05.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:05.797 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:05.828 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:05.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:05.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:05.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:05.961 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:05.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:06.049 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:06.063 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:06.068 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:06.094 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:06.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:06.196 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:06.221 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:06.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:06.323 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:06.350 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:06.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:06.452 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:06.479 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:06.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:06.484 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:13:06.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:06.605 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:06.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:06.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:06.732 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:06.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:06 smithi187 ceph-mon[20072]: pgmap v932: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 3159/207513 objects degraded (1.522%); 13249/207513 objects misplaced (6.385%); 27 MiB/s, 427 objects/s recovering 2024-09-02T10:13:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4120803225' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:06 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:13:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:13:06.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:06.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:06 smithi162 ceph-mon[20279]: pgmap v932: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 3159/207513 objects degraded (1.522%); 13249/207513 objects misplaced (6.385%); 27 MiB/s, 427 objects/s recovering 2024-09-02T10:13:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4120803225' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:06 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:13:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:13:06.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:06.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:06.876 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:06 smithi151 ceph-mon[16639]: pgmap v932: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.2 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 672 op/s; 3159/207513 objects degraded (1.522%); 13249/207513 objects misplaced (6.385%); 27 MiB/s, 427 objects/s recovering 2024-09-02T10:13:06.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4120803225' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:06.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:06 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:06.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:13:06.877 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:13:06.965 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:06.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:06.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:06.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:07.094 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:07.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:07.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:07.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:07.252 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:07.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:07.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:07.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:07.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:07.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:07.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:07.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:07.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:07.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:07.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:07.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:07 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:07 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:13:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:07 smithi187 ceph-mon[20072]: osdmap e390: 12 total, 12 up, 11 in 2024-09-02T10:13:07.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:07.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:07.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:07.801 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:13:07.801 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:13:07.801 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-02T10:13:07.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-02T10:13:07.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:07 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:07.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:07 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:13:07.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:07 smithi151 ceph-mon[16639]: osdmap e390: 12 total, 12 up, 11 in 2024-09-02T10:13:07.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:07 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:07.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:07 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:13:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:07 smithi162 ceph-mon[20279]: osdmap e390: 12 total, 12 up, 11 in 2024-09-02T10:13:07.876 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:07.909 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:07.909 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:08.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:08.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:08.034 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:08.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:08.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:08.163 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:08.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:08.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:08.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:08.294 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:08.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:08.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:08.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:08.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:08.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:08.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:08.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:08.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:08.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:08.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:08.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:08 smithi187 ceph-mon[20072]: pgmap v934: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 3157/203754 objects degraded (1.549%); 12270/203754 objects misplaced (6.022%); 32 MiB/s, 505 objects/s recovering 2024-09-02T10:13:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:08 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 3157/203754 objects degraded (1.549%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:08.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:08.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:08.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:08.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:08 smithi162 ceph-mon[20279]: pgmap v934: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 3157/203754 objects degraded (1.549%); 12270/203754 objects misplaced (6.022%); 32 MiB/s, 505 objects/s recovering 2024-09-02T10:13:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:08 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 3157/203754 objects degraded (1.549%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:08.875 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:08 smithi151 ceph-mon[16639]: pgmap v934: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.1 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 0 B/s wr, 726 op/s; 3157/203754 objects degraded (1.549%); 12270/203754 objects misplaced (6.022%); 32 MiB/s, 505 objects/s recovering 2024-09-02T10:13:08.875 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:08 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 3157/203754 objects degraded (1.549%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:08.927 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:08.956 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:08.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:09.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:09.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:09.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:09.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:09.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:09.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:09.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:09.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:09.353 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:09.354 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:09.456 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:09.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:09.486 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:09.486 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:09.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:09.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:09.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:09.727 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:09.727 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:09.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:09.755 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:09.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:09.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:09.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:09.886 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:09.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:09.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:10.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:10.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:10.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:10.151 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:10.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:10.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:09 smithi151 ceph-mon[16639]: osdmap e391: 12 total, 12 up, 11 in 2024-09-02T10:13:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:09 smithi151 ceph-mon[16639]: pgmap v936: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 918 op/s; 3301/194952 objects degraded (1.693%); 10475/194952 objects misplaced (5.373%); 36 MiB/s, 584 objects/s recovering 2024-09-02T10:13:10.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:09 smithi187 ceph-mon[20072]: osdmap e391: 12 total, 12 up, 11 in 2024-09-02T10:13:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:09 smithi187 ceph-mon[20072]: pgmap v936: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 918 op/s; 3301/194952 objects degraded (1.693%); 10475/194952 objects misplaced (5.373%); 36 MiB/s, 584 objects/s recovering 2024-09-02T10:13:10.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:10.277 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:09 smithi162 ceph-mon[20279]: osdmap e391: 12 total, 12 up, 11 in 2024-09-02T10:13:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:09 smithi162 ceph-mon[20279]: pgmap v936: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 4.0 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 918 op/s; 3301/194952 objects degraded (1.693%); 10475/194952 objects misplaced (5.373%); 36 MiB/s, 584 objects/s recovering 2024-09-02T10:13:10.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:10.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:10.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:10.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:10.540 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:10.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:10.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:10.667 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:10.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:10.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:10.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:10.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:10.840 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:13:10.840 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:13:10.840 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:13:10.840 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:13:10.840 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:13:10.840 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:13:10.841 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:13:10.841 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:13:10.841 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:13:10.841 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:13:10.841 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:13:10.841 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:13:10.847 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:10.858 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:13:10.859 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:13:10.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:13:10.886 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:13:10.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:10.912 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:13:10.912 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:13:10.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:10.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:10.937 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:13:10.938 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:13:10.969 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:13:10.969 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:13:10.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/477978170' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:10.997 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:13:10.997 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:13:11.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:11.047 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:11.048 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:11.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:11.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:11.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/477978170' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:11.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:11.295 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:11.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:11.296 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:11.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/477978170' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:11.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:11.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:11.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:11.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:13:11.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:11.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:11.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:11.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:11.689 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:11.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:11.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:11.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:11.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:11.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:11.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:11.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:12.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:12.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:12.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:12.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:12.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:12.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:12.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:12.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:11 smithi187 ceph-mon[20072]: pgmap v937: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 820 op/s; 3301/192426 objects degraded (1.715%); 9982/192426 objects misplaced (5.187%); 27 MiB/s, 439 objects/s recovering 2024-09-02T10:13:12.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:12.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:11 smithi162 ceph-mon[20279]: pgmap v937: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 820 op/s; 3301/192426 objects degraded (1.715%); 9982/192426 objects misplaced (5.187%); 27 MiB/s, 439 objects/s recovering 2024-09-02T10:13:12.345 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:12.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:11 smithi151 ceph-mon[16639]: pgmap v937: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.9 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 820 op/s; 3301/192426 objects degraded (1.715%); 9982/192426 objects misplaced (5.187%); 27 MiB/s, 439 objects/s recovering 2024-09-02T10:13:12.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:12.472 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:12.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:12.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:12.604 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:12.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:12.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:12.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:12.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:12.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:12.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:12.863 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:12.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:12.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:12.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:12.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:13.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:13.123 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:13.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:13.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:13.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:13.252 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:13.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:13.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:13.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:13:13.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:13.381 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:13.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:13.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:13:13.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:13 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:13:13.512 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:13.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:13.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:13 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:13:13.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:13.647 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:13.648 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:13.661 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:13.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:13.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:13.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:13.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:13.909 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:13.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:14.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:14.035 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:14.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:14.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:14.161 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:14.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:14.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:14.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:14.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:14.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:14.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:14.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:14.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:14.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:14.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:14 smithi151 ceph-mon[16639]: pgmap v938: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 900 op/s; 3373/185898 objects degraded (1.814%); 7840/185898 objects misplaced (4.217%); 41 MiB/s, 649 objects/s recovering 2024-09-02T10:13:14.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 3301/192426 objects degraded (1.715%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:14 smithi187 ceph-mon[20072]: pgmap v938: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 900 op/s; 3373/185898 objects degraded (1.814%); 7840/185898 objects misplaced (4.217%); 41 MiB/s, 649 objects/s recovering 2024-09-02T10:13:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 3301/192426 objects degraded (1.715%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:14.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:14.536 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:14.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:14 smithi162 ceph-mon[20279]: pgmap v938: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.8 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 900 op/s; 3373/185898 objects degraded (1.814%); 7840/185898 objects misplaced (4.217%); 41 MiB/s, 649 objects/s recovering 2024-09-02T10:13:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 3301/192426 objects degraded (1.715%), 8 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:14.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:14.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:14.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:14.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:14.750 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:14.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:14.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:14.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:14.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:14.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:14.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:14.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:15.036 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:15.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:15.068 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:15.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:15.200 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:15.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:15.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:15.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:15.306 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:13:15.307 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:13:15.307 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:13:15.307 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:13:15.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:15.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:15.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:15.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:15 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:13:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:15 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:13:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:15 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:13:15.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:15.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:15.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-02T10:13:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:15 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:13:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:15 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:13:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:15 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:13:15.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:15.581 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:15.582 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:15 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:13:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:15 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:13:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:15 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:13:15.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:15.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:15.706 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:15.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:15.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:15.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:15.833 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:15.934 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:15.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:15.964 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:15.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:16.036 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:16.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:16.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:16.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:16.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:13:16.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:16.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:16.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:16.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:16.273 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:13:16.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:16.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:13:16.353 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:16.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:16 smithi151 ceph-mon[16639]: pgmap v939: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 978 op/s; 3373/180519 objects degraded (1.869%); 7538/180519 objects misplaced (4.176%); 36 MiB/s, 568 objects/s recovering 2024-09-02T10:13:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:16 smithi151 ceph-mon[16639]: osdmap e392: 12 total, 12 up, 11 in 2024-09-02T10:13:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:16 smithi151 ceph-mon[16639]: from='client.16167 172.21.15.151:0/2586226234' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:13:16.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:16.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:16.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:16.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:16.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:16 smithi162 ceph-mon[20279]: pgmap v939: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 978 op/s; 3373/180519 objects degraded (1.869%); 7538/180519 objects misplaced (4.176%); 36 MiB/s, 568 objects/s recovering 2024-09-02T10:13:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:16 smithi162 ceph-mon[20279]: osdmap e392: 12 total, 12 up, 11 in 2024-09-02T10:13:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:16 smithi162 ceph-mon[20279]: from='client.16167 172.21.15.151:0/2586226234' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:13:16.622 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:16.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:16.709 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:16.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:16 smithi187 ceph-mon[20072]: pgmap v939: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.7 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 0 B/s wr, 978 op/s; 3373/180519 objects degraded (1.869%); 7538/180519 objects misplaced (4.176%); 36 MiB/s, 568 objects/s recovering 2024-09-02T10:13:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:16 smithi187 ceph-mon[20072]: osdmap e392: 12 total, 12 up, 11 in 2024-09-02T10:13:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:16 smithi187 ceph-mon[20072]: from='client.16167 172.21.15.151:0/2586226234' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:13:16.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:16.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:16.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:16.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:16.894 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:16.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:16.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:17.023 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:17.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:17.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:17.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:17.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:17.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:17.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:17.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:17.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:17.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:17.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:17.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:17.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:17.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:17.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:17.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:17.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:17 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:17.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:17 smithi162 ceph-mon[20279]: from='client.16167 172.21.15.151:0/2586226234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:13:17.541 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:17 smithi162 ceph-mon[20279]: osdmap e393: 12 total, 12 up, 11 in 2024-09-02T10:13:17.557 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:17.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:17.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:17.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:17.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:17.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:17 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:17.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:17 smithi151 ceph-mon[16639]: from='client.16167 172.21.15.151:0/2586226234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:13:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:17 smithi151 ceph-mon[16639]: osdmap e393: 12 total, 12 up, 11 in 2024-09-02T10:13:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:17 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:17 smithi187 ceph-mon[20072]: from='client.16167 172.21.15.151:0/2586226234' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:13:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:17 smithi187 ceph-mon[20072]: osdmap e393: 12 total, 12 up, 11 in 2024-09-02T10:13:17.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:17.814 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:17.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:17.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:17.845 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:13:17.845 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:13:17.845 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:13:17.845 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:13:17.845 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:13:17.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:17.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:17.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:18.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:18.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-02T10:13:18.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:18.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:18.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:18.218 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:18.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:18.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:18.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:18.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:18 smithi151 ceph-mon[16639]: 4.8 scrub starts 2024-09-02T10:13:18.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:18 smithi151 ceph-mon[16639]: pgmap v942: 57 pgs: 1 peering, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 1188/177228 objects degraded (0.670%); 8464/177228 objects misplaced (4.776%); 29 MiB/s, 502 objects/s recovering 2024-09-02T10:13:18.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:18 smithi151 ceph-mon[16639]: 4.8 scrub ok 2024-09-02T10:13:18.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:18.364 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:18.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:18.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:18.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:18.493 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:18.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:18.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:18.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:18 smithi162 ceph-mon[20279]: 4.8 scrub starts 2024-09-02T10:13:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:18 smithi162 ceph-mon[20279]: pgmap v942: 57 pgs: 1 peering, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 1188/177228 objects degraded (0.670%); 8464/177228 objects misplaced (4.776%); 29 MiB/s, 502 objects/s recovering 2024-09-02T10:13:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:18 smithi162 ceph-mon[20279]: 4.8 scrub ok 2024-09-02T10:13:18.625 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:18.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:18.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:13:18.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:18 smithi187 ceph-mon[20072]: 4.8 scrub starts 2024-09-02T10:13:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:18 smithi187 ceph-mon[20072]: pgmap v942: 57 pgs: 1 peering, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 1188/177228 objects degraded (0.670%); 8464/177228 objects misplaced (4.776%); 29 MiB/s, 502 objects/s recovering 2024-09-02T10:13:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:18 smithi187 ceph-mon[20072]: 4.8 scrub ok 2024-09-02T10:13:18.757 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:18.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:18.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:18.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:18.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:18.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:19.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:19.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:19.028 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:19.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:19.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:19.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:19.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:19.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:19.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:19.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:19.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:13:19.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:19.404 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:19.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:19.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:19 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:13:19.487 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:19 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:13:19.488 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1188/177228 objects degraded (0.670%), 7 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:19.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:19.533 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:19.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:19.534 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:19.612 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:13:19.612 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:13:19.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:13:19.612 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:13:19.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.612 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127455 2.d 2.f151987d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:13:19.612 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.256858+0000", 2024-09-02T10:13:19.612 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 574.35191025799998, 2024-09-02T10:13:19.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96215398699999999, 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127455 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256858+0000", 2024-09-02T10:13:19.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256858+0000", 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256860+0000", 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5039999999999999e-06 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256866+0000", 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0959999999999997e-06 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256875+0000", 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1940000000000004e-06 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256884+0000", 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3030000000000005e-06 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256938+0000", 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4259999999999999e-05 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.256949+0000", 2024-09-02T10:13:19.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0624e-05 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.218985+0000", 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96203561699999995 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219012+0000", 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7389000000000002e-05 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.617 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127536 2.d 2.4160433d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.711015+0000", 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 573.89775308100002, 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.50804419700000003, 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:19.618 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.619 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.619 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.619 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127536 2024-09-02T10:13:19.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.619 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711015+0000", 2024-09-02T10:13:19.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711015+0000", 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711017+0000", 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9400000000000001e-06 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711026+0000", 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3729999999999993e-06 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711044+0000", 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8017e-05 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711068+0000", 2024-09-02T10:13:19.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4510000000000001e-05 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711155+0000", 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7280999999999993e-05 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.711167+0000", 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1497e-05 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219052+0000", 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.50788502000000002 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219059+0000", 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.559e-06 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127544 2.d 2.969bbf1d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:13:19.623 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.788037+0000", 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 573.82073099800004, 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.43104598700000002, 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127544 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788037+0000", 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788037+0000", 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788039+0000", 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0760000000000001e-06 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788049+0000", 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4150000000000005e-06 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788070+0000", 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0809999999999999e-05 2024-09-02T10:13:19.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788079+0000", 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8320000000000001e-06 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788139+0000", 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9336000000000002e-05 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.788147+0000", 2024-09-02T10:13:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7940000000000003e-06 2024-09-02T10:13:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219079+0000", 2024-09-02T10:13:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.43093289000000001 2024-09-02T10:13:19.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.219083+0000", 2024-09-02T10:13:19.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8340000000000001e-06 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127547 2.d 2:bf1255db:::benchmark_data_smithi187_45882_object29796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:47.249699+0000", 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 572.359069095, 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.34997614900000001, 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.630 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127547 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249699+0000", 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249699+0000", 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249700+0000", 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3000000000000001e-07 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249702+0000", 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.933e-06 2024-09-02T10:13:19.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249709+0000", 2024-09-02T10:13:19.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3309999999999998e-06 2024-09-02T10:13:19.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.249714+0000", 2024-09-02T10:13:19.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0880000000000002e-06 2024-09-02T10:13:19.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250467+0000", 2024-09-02T10:13:19.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00075314800000000003 2024-09-02T10:13:19.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.250476+0000", 2024-09-02T10:13:19.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6170000000000007e-06 2024-09-02T10:13:19.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531604+0000", 2024-09-02T10:13:19.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28112852799999999 2024-09-02T10:13:19.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531742+0000", 2024-09-02T10:13:19.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000137353 2024-09-02T10:13:19.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:13:19.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.531881+0000", 2024-09-02T10:13:19.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013981700000000001 2024-09-02T10:13:19.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:19.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.535629+0000", 2024-09-02T10:13:19.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037478059999999998 2024-09-02T10:13:19.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.538014+0000", 2024-09-02T10:13:19.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002384976 2024-09-02T10:13:19.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.598001+0000", 2024-09-02T10:13:19.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.059986314999999998 2024-09-02T10:13:19.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:19.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.598060+0000", 2024-09-02T10:13:19.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9583999999999997e-05 2024-09-02T10:13:19.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.599675+0000", 2024-09-02T10:13:19.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001615223 2024-09-02T10:13:19.643 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.643 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.643 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.644 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.d 183/157 [PushReplyOp(2:b0969168:::benchmark_data_smithi187_45882_object60943:head)])", 2024-09-02T10:13:19.644 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:36.457663+0000", 2024-09-02T10:13:19.644 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 523.15110538600004, 2024-09-02T10:13:19.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36433320800000002, 2024-09-02T10:13:19.645 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.645 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.645 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:19.645 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457663+0000", 2024-09-02T10:13:19.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457663+0000", 2024-09-02T10:13:19.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457664+0000", 2024-09-02T10:13:19.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.13e-06 2024-09-02T10:13:19.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.457670+0000", 2024-09-02T10:13:19.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2910000000000004e-06 2024-09-02T10:13:19.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.820382+0000", 2024-09-02T10:13:19.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.362711703 2024-09-02T10:13:19.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.820397+0000", 2024-09-02T10:13:19.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5423000000000001e-05 2024-09-02T10:13:19.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.820576+0000", 2024-09-02T10:13:19.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017897 2024-09-02T10:13:19.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:36.821996+0000", 2024-09-02T10:13:19.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014196910000000001 2024-09-02T10:13:19.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.653 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.654 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163245 2.d 2:b53557ce:::benchmark_data_smithi187_45882_object65494:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:13:19.654 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027769+0000", 2024-09-02T10:13:19.654 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 502.58099955900002, 2024-09-02T10:13:19.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36269144800000003, 2024-09-02T10:13:19.654 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.655 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.655 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:19.655 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.655 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.655 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.656 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163245 2024-09-02T10:13:19.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.656 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027769+0000", 2024-09-02T10:13:19.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027769+0000", 2024-09-02T10:13:19.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027771+0000", 2024-09-02T10:13:19.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7020000000000001e-06 2024-09-02T10:13:19.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027781+0000", 2024-09-02T10:13:19.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6700000000000006e-06 2024-09-02T10:13:19.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027798+0000", 2024-09-02T10:13:19.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6416999999999999e-05 2024-09-02T10:13:19.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027810+0000", 2024-09-02T10:13:19.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2568000000000001e-05 2024-09-02T10:13:19.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028091+0000", 2024-09-02T10:13:19.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028051500000000001 2024-09-02T10:13:19.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028103+0000", 2024-09-02T10:13:19.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2768999999999999e-05 2024-09-02T10:13:19.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390352+0000", 2024-09-02T10:13:19.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36224896000000001 2024-09-02T10:13:19.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390460+0000", 2024-09-02T10:13:19.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107847 2024-09-02T10:13:19.666 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.666 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.666 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.667 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163286 2.d 2:b26ad3af:::benchmark_data_smithi187_45882_object65535:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:13:19.667 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027840+0000", 2024-09-02T10:13:19.667 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 502.58092821299999, 2024-09-02T10:13:19.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38364109499999999, 2024-09-02T10:13:19.668 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.668 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.668 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:19.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.669 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.669 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163286 2024-09-02T10:13:19.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.669 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027840+0000", 2024-09-02T10:13:19.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027840+0000", 2024-09-02T10:13:19.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027841+0000", 2024-09-02T10:13:19.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3600000000000003e-07 2024-09-02T10:13:19.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027844+0000", 2024-09-02T10:13:19.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4170000000000001e-06 2024-09-02T10:13:19.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027856+0000", 2024-09-02T10:13:19.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1659e-05 2024-09-02T10:13:19.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027862+0000", 2024-09-02T10:13:19.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5089999999999999e-06 2024-09-02T10:13:19.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028144+0000", 2024-09-02T10:13:19.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028157300000000002 2024-09-02T10:13:19.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028154+0000", 2024-09-02T10:13:19.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7010000000000008e-06 2024-09-02T10:13:19.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390697+0000", 2024-09-02T10:13:19.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36254281199999999 2024-09-02T10:13:19.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390784+0000", 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7717000000000001e-05 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.390867+0000", 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2432e-05 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.401652+0000", 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010785041 2024-09-02T10:13:19.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.401689+0000", 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7502000000000001e-05 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411205+0000", 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0095160409999999994 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:19.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411244+0000", 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8296000000000001e-05 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411481+0000", 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023765900000000001 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163296 2.d 2:b1ae2ae4:::benchmark_data_smithi187_45882_object65545:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:13:19.681 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 502.58089768100001, 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38880961400000003, 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163296 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027871+0000", 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4200000000000002e-07 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027883+0000", 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2315e-05 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027926+0000", 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2339000000000002e-05 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027946+0000", 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0801999999999998e-05 2024-09-02T10:13:19.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028193+0000", 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024658900000000002 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028198+0000", 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6009999999999997e-06 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.394381+0000", 2024-09-02T10:13:19.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36618306299999998 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.394457+0000", 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6509999999999998e-05 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.394532+0000", 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4348e-05 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402216+0000", 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076842819999999997 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405720+0000", 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035040480000000001 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416431+0000", 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010711178999999999 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416463+0000", 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2008999999999998e-05 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416680+0000", 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021718700000000001 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163297 2.d 2:b94d3aa0:::benchmark_data_smithi187_45882_object65546:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027961+0000", 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 502.58080703100001, 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38891082999999999, 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163297 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.689 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027961+0000", 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027961+0000", 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027962+0000", 2024-09-02T10:13:19.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8800000000000002e-07 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027967+0000", 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6299999999999997e-06 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027975+0000", 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7819999999999999e-06 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027981+0000", 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8309999999999997e-06 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028219+0000", 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000237774 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028223+0000", 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5980000000000001e-06 2024-09-02T10:13:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.397985+0000", 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36976200399999998 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.398112+0000", 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127565 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:13:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.398213+0000", 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100368 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405733+0000", 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0075208200000000001 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411164+0000", 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054306110000000001 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416727+0000", 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055628049999999997 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416766+0000", 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9609e-05 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416872+0000", 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105645 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163303 2.d 2:b483d011:::benchmark_data_smithi187_45882_object65552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 502.58077747900001, 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.39586157599999999, 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163303 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027991+0000", 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7800000000000002e-07 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.027995+0000", 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3500000000000001e-06 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028001+0000", 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2859999999999996e-06 2024-09-02T10:13:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028006+0000", 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6430000000000003e-06 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028242+0000", 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023680199999999999 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028246+0000", 2024-09-02T10:13:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2260000000000001e-06 2024-09-02T10:13:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402251+0000", 2024-09-02T10:13:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.374004893 2024-09-02T10:13:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:19.701 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:19 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:13:19.701 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:19 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:13:19.701 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1188/177228 objects degraded (0.670%), 7 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402324+0000", 2024-09-02T10:13:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3100000000000001e-05 2024-09-02T10:13:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.402391+0000", 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7261999999999994e-05 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411493+0000", 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091022120000000001 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416394+0000", 2024-09-02T10:13:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049004870000000002 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.423496+0000", 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071020859999999996 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.423554+0000", 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8511000000000003e-05 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.423852+0000", 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00029824000000000002 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163305 2.d 2:bdea317b:::benchmark_data_smithi187_45882_object65554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 502.580744606, 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40774826600000003, 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163305 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028024+0000", 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6199999999999998e-07 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028027+0000", 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9299999999999999e-06 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028034+0000", 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5039999999999999e-06 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028038+0000", 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6739999999999996e-06 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028260+0000", 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00022132900000000001 2024-09-02T10:13:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028263+0000", 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.163e-06 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405770+0000", 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.377506856 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405877+0000", 2024-09-02T10:13:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010699799999999999 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.405954+0000", 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7732999999999996e-05 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.411501+0000", 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.005546212 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.425267+0000", 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013766327 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435398+0000", 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010131506 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435457+0000", 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8248000000000002e-05 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435772+0000", 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031532400000000002 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:163342 2.d 2:b4bdf525:::benchmark_data_smithi187_45882_object65591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:57.028055+0000", 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 502.58071306699998, 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.40808169100000002, 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 163342 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028055+0000", 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028055+0000", 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028056+0000", 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7999999999999996e-07 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028059+0000", 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1109999999999999e-06 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028065+0000", 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6189999999999996e-06 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028071+0000", 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4410000000000001e-06 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028292+0000", 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221406 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.028296+0000", 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3840000000000001e-06 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.408774+0000", 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38047832599999998 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.408848+0000", 2024-09-02T10:13:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4216999999999998e-05 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.408915+0000", 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7244999999999997e-05 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.416361+0000", 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0074451420000000001 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.425326+0000", 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0089654829999999998 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435842+0000", 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010515531999999999 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.435934+0000", 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2621000000000006e-05 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:57.436137+0000", 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020268400000000001 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(2.d e219,198 2:b2e165ef:::benchmark_data_smithi187_45882_object46440:head,171'8780)", 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:45.488577+0000", 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 454.12019129700002, 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57263216900000002, 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488577+0000", 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488577+0000", 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488578+0000", 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9100000000000004e-07 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.488584+0000", 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2720000000000003e-06 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.060736+0000", 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57215209199999995 2024-09-02T10:13:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.060751+0000", 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5387e-05 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.060841+0000", 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9415999999999999e-05 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.061209+0000", 2024-09-02T10:13:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036841100000000002 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(2.d e219,198 2:b1206fd5:::benchmark_data_smithi187_45882_object46449:head,171'8781)", 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:45.492423+0000", 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 454.11634510200003, 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57615407699999999, 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492423+0000", 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492423+0000", 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492430+0000", 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6959999999999999e-06 2024-09-02T10:13:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:45.492440+0000", 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9299999999999998e-06 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068235+0000", 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.57579494799999997 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068243+0000", 2024-09-02T10:13:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4330000000000004e-06 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068306+0000", 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2854000000000005e-05 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:46.068577+0000", 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027121600000000001 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "replica_scrub(pg: 3.6,from:0'0,to:263'6750,epoch:263/226,start:3:6f96281c:::benchmark_data_smithi187_59989_object88170:0,end:3:6fcee8f7:::benchmark_data_smithi187_59989_object42531:0,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:33.294203+0000", 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 346.31456541199998, 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.75891445099999999, 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:33.294203+0000", 2024-09-02T10:13:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:33.294203+0000", 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:33.294203+0000", 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6499999999999999e-07 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:33.294209+0000", 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7490000000000004e-06 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:34.052988+0000", 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.75877908199999999 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:34.052998+0000", 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3689999999999996e-06 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:34.053074+0000", 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6249999999999997e-05 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:34.053089+0000", 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4943000000000001e-05 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:34.053117+0000", 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8493e-05 2024-09-02T10:13:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:181206 3.3 3:c5ec1ffd:::benchmark_data_smithi187_59989_object71663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:09:15.005860+0000", 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 244.60290798299999, 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.377987352, 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:13:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 181206 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.005860+0000", 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.005860+0000", 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.005862+0000", 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5689999999999999e-06 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.005869+0000", 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4769999999999997e-06 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.005886+0000", 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6606e-05 2024-09-02T10:13:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.005893+0000", 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.498e-06 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.005945+0000", 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1551999999999997e-05 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006037+0000", 2024-09-02T10:13:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2404000000000005e-05 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006115+0000", 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7645000000000006e-05 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.010047+0000", 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039317919999999999 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.011084+0000", 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010374329999999999 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.383657+0000", 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37257285800000001 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.383803+0000", 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000145389 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.383848+0000", 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5129000000000001e-05 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:11553 4.e 4:77913b74:::benchmark_data_smithi187_76542_object11552:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:10.469148+0000", 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 129.13962046699999, 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38080176300000002, 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 11553 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.469148+0000", 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.469148+0000", 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.469149+0000", 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.407e-06 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.469198+0000", 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8758999999999999e-05 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.469213+0000", 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5071e-05 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.469222+0000", 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7800000000000006e-06 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.469287+0000", 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5162999999999999e-05 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.469427+0000", 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014042500000000001 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,10", 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.469574+0000", 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014693 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.470314+0000", 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073999299999999999 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.470482+0000", 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016726499999999999 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.849883+0000", 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37940108099999997 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.849921+0000", 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8294000000000003e-05 2024-09-02T10:13:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.849950+0000", 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8595e-05 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.25507.0:11553 4.e e326/318 ondisk, result = 0)", 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:10.470435+0000", 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 129.13833374999999, 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37951973900000002, 2024-09-02T10:13:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.470435+0000", 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.470435+0000", 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.470435+0000", 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5199999999999996e-07 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.470440+0000", 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2989999999999998e-06 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.849811+0000", 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37937090200000001 2024-09-02T10:13:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.849818+0000", 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6399999999999997e-06 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.849872+0000", 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4289000000000001e-05 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.849879+0000", 2024-09-02T10:13:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5400000000000001e-06 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.849954+0000", 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5316999999999997e-05 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:75261 4.e 4:7d9bdb56:::benchmark_data_smithi187_76542_object75260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:17.679207+0000", 2024-09-02T10:13:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 61.929561272000001, 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.39759054999999999, 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75261 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679207+0000", 2024-09-02T10:13:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679207+0000", 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679209+0000", 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.722e-06 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679269+0000", 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0007000000000003e-05 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679280+0000", 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1413999999999999e-05 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679295+0000", 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4572999999999999e-05 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679367+0000", 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1791000000000004e-05 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679550+0000", 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000183019 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,10", 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679705+0000", 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015513500000000001 2024-09-02T10:13:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.681783+0000", 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020785370000000001 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.682235+0000", 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045192700000000003 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076685+0000", 2024-09-02T10:13:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.39444963199999999 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076746+0000", 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1613999999999997e-05 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076798+0000", 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1178999999999999e-05 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.25507.0:75261 4.e e371/340 ondisk, result = 0)", 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:17.682016+0000", 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 61.926752532000002, 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.394789379, 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.682016+0000", 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.682016+0000", 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.682016+0000", 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.99e-07 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.682020+0000", 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1829999999999998e-06 2024-09-02T10:13:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076598+0000", 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.394578759 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076606+0000", 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3089999999999998e-06 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076669+0000", 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3819999999999995e-05 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076679+0000", 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5279999999999992e-06 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076805+0000", 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126181 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:19.758 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:13:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:19 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:13:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:19 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:13:19.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1188/177228 objects degraded (0.670%), 7 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:19.784 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:19.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:19.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:19.911 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:19.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:20.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:20.043 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:20.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:20.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:20.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:20.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:20.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:20.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:20.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:20.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:20.416 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:20.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:20.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:20.505 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:20 smithi162 ceph-mon[20279]: pgmap v943: 57 pgs: 1 peering, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 944 op/s; 1271/169746 objects degraded (0.749%); 6319/169746 objects misplaced (3.723%); 42 MiB/s, 705 objects/s recovering 2024-09-02T10:13:20.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:20.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:13:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:20.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:20.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:20.632 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:20 smithi151 ceph-mon[16639]: pgmap v943: 57 pgs: 1 peering, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 944 op/s; 1271/169746 objects degraded (0.749%); 6319/169746 objects misplaced (3.723%); 42 MiB/s, 705 objects/s recovering 2024-09-02T10:13:20.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:20.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:20.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:20 smithi187 ceph-mon[20072]: pgmap v943: 57 pgs: 1 peering, 7 active+recovering+undersized+degraded+remapped, 49 active+clean; 3.5 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 944 op/s; 1271/169746 objects degraded (0.749%); 6319/169746 objects misplaced (3.723%); 42 MiB/s, 705 objects/s recovering 2024-09-02T10:13:20.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:20.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:20.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:20.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:20.808 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:20.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:20.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:20.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:21.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:21.044 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:21.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:21.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:21.167 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:21.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:21.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:21.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:21.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:21.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:21.385 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:21.385 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:21.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:21.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:21.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:21.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:21.548 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:21.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:21.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:21 smithi162 ceph-mon[20279]: osdmap e394: 12 total, 12 up, 11 in 2024-09-02T10:13:21.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:21 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:13:21.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:21 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:13:21.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/647197644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:21.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:21.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:21.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:21.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:21 smithi151 ceph-mon[16639]: osdmap e394: 12 total, 12 up, 11 in 2024-09-02T10:13:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:21 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:13:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:21 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:13:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/647197644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:21 smithi187 ceph-mon[20072]: osdmap e394: 12 total, 12 up, 11 in 2024-09-02T10:13:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:21 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:13:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:21 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:13:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/647197644' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:13:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:13:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:13:21.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:13:21.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:13:21.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:13:21.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:13:21.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:13:21.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:13:21.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:13:21.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:13:21.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:13:21.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:13:21.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:13:21.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:13:21.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:13:21.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:13:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:13:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:13:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:13:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:13:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:13:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:13:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:13:21.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:21.802 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:21.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:21.904 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:21.926 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:21.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:22.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:22.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:13:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:22.051 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:22.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:22.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:22.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:22.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:22.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:22.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:22.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:22.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:22.304 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:22.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:22.426 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:22.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:22.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:22.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:22.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:22.559 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:22.559 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:22 smithi162 ceph-mon[20279]: 4.c scrub starts 2024-09-02T10:13:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:22 smithi162 ceph-mon[20279]: pgmap v945: 57 pgs: 7 active+recovering+undersized+degraded+remapped, 50 active+clean; 3.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 725 op/s; 1271/167463 objects degraded (0.759%); 6319/167463 objects misplaced (3.773%); 31 MiB/s, 545 objects/s recovering 2024-09-02T10:13:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:22 smithi162 ceph-mon[20279]: osdmap e395: 12 total, 12 up, 11 in 2024-09-02T10:13:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:22 smithi162 ceph-mon[20279]: 4.10 scrub starts 2024-09-02T10:13:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:22 smithi187 ceph-mon[20072]: 4.c scrub starts 2024-09-02T10:13:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:22 smithi187 ceph-mon[20072]: pgmap v945: 57 pgs: 7 active+recovering+undersized+degraded+remapped, 50 active+clean; 3.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 725 op/s; 1271/167463 objects degraded (0.759%); 6319/167463 objects misplaced (3.773%); 31 MiB/s, 545 objects/s recovering 2024-09-02T10:13:22.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:22 smithi187 ceph-mon[20072]: osdmap e395: 12 total, 12 up, 11 in 2024-09-02T10:13:22.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:22 smithi187 ceph-mon[20072]: 4.10 scrub starts 2024-09-02T10:13:22.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:22.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:22 smithi151 ceph-mon[16639]: 4.c scrub starts 2024-09-02T10:13:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:22 smithi151 ceph-mon[16639]: pgmap v945: 57 pgs: 7 active+recovering+undersized+degraded+remapped, 50 active+clean; 3.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 725 op/s; 1271/167463 objects degraded (0.759%); 6319/167463 objects misplaced (3.773%); 31 MiB/s, 545 objects/s recovering 2024-09-02T10:13:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:22 smithi151 ceph-mon[16639]: osdmap e395: 12 total, 12 up, 11 in 2024-09-02T10:13:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:22 smithi151 ceph-mon[16639]: 4.10 scrub starts 2024-09-02T10:13:22.696 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:22.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:22.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:22.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:22.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:22.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:22.860 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:22.860 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:22.927 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:22.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:22.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:22.959 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:23.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:23.087 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:23.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:23.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:23.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:23.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:23.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:23.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:23.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:23.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:23.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:23.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:23.415 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:23 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:13:23.415 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:23 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:13:23.415 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:23 smithi187 ceph-mon[20072]: 4.c scrub ok 2024-09-02T10:13:23.415 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:23 smithi187 ceph-mon[20072]: osdmap e396: 12 total, 12 up, 11 in 2024-09-02T10:13:23.415 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:23 smithi187 ceph-mon[20072]: 4.10 scrub ok 2024-09-02T10:13:23.416 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:23 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:13:23.416 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:23 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:13:23.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:23.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:23.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:23.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:23.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:13:23.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:23 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:13:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:23 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:13:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:23 smithi162 ceph-mon[20279]: 4.c scrub ok 2024-09-02T10:13:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:23 smithi162 ceph-mon[20279]: osdmap e396: 12 total, 12 up, 11 in 2024-09-02T10:13:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:23 smithi162 ceph-mon[20279]: 4.10 scrub ok 2024-09-02T10:13:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:23 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:13:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:23 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:13:23.605 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:23.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:23.606 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:23 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:13:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:23 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:13:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:23 smithi151 ceph-mon[16639]: 4.c scrub ok 2024-09-02T10:13:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:23 smithi151 ceph-mon[16639]: osdmap e396: 12 total, 12 up, 11 in 2024-09-02T10:13:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:23 smithi151 ceph-mon[16639]: 4.10 scrub ok 2024-09-02T10:13:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:23 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:13:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:23 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:13:23.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:23.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:23.737 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:23.743 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:23.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:23.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:23.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:23.889 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:23.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:23.992 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:23.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:24.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:24.118 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:24.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:24.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:24.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:24.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:24.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:24.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:24.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:24.376 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:24.437 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:24.437 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:24.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:24.503 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:24.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:24 smithi162 ceph-mon[20279]: pgmap v948: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 3.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 1318/164490 objects degraded (0.801%); 5543/164490 objects misplaced (3.370%); 29 MiB/s, 532 objects/s recovering 2024-09-02T10:13:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:24 smithi162 ceph-mon[20279]: osdmap e397: 12 total, 12 up, 11 in 2024-09-02T10:13:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:24 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:13:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:24 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:13:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1318/164490 objects degraded (0.801%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:24.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:24.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:24 smithi151 ceph-mon[16639]: pgmap v948: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 3.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 1318/164490 objects degraded (0.801%); 5543/164490 objects misplaced (3.370%); 29 MiB/s, 532 objects/s recovering 2024-09-02T10:13:24.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:24 smithi151 ceph-mon[16639]: osdmap e397: 12 total, 12 up, 11 in 2024-09-02T10:13:24.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:24 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:13:24.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:24 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:13:24.609 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1318/164490 objects degraded (0.801%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:24.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:24.634 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:24.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:24 smithi187 ceph-mon[20072]: pgmap v948: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 3.3 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 707 op/s; 1318/164490 objects degraded (0.801%); 5543/164490 objects misplaced (3.370%); 29 MiB/s, 532 objects/s recovering 2024-09-02T10:13:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:24 smithi187 ceph-mon[20072]: osdmap e397: 12 total, 12 up, 11 in 2024-09-02T10:13:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:24 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:13:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:24 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:13:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1318/164490 objects degraded (0.801%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:24.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:24.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:24.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:24.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:24.889 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:24.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:24.991 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:25.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:25.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:25.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:25.119 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:25.139 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:25.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:25.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:13:25.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:25.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:25.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:25.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:25.396 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:25.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:25.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:25.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:25.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:25.529 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:25.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.4 reset_purged_snaps_last 2024-09-02T10:13:25.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:25.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:25.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:25.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:25.791 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:25.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:25.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:25.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:25.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:25.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:26.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:26.056 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:26.057 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:26.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:26.076 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:26.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:26.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:26.205 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:26.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:26.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:26.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:26.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:26.432 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:26.454 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:26.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:26.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:26.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:26.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:26.578 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:26.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:26 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:13:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:26 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:13:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:26 smithi162 ceph-mon[20279]: pgmap v950: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 1315/158751 objects degraded (0.828%); 4564/158751 objects misplaced (2.875%); 21 MiB/s, 581 objects/s recovering 2024-09-02T10:13:26.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:13:26.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:26 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:13:26.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:26 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:13:26.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:26 smithi151 ceph-mon[16639]: pgmap v950: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 1315/158751 objects degraded (0.828%); 4564/158751 objects misplaced (2.875%); 21 MiB/s, 581 objects/s recovering 2024-09-02T10:13:26.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:26.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:26.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:26 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:13:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:26 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:13:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:26 smithi187 ceph-mon[20072]: pgmap v950: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 660 op/s; 1315/158751 objects degraded (0.828%); 4564/158751 objects misplaced (2.875%); 21 MiB/s, 581 objects/s recovering 2024-09-02T10:13:26.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:26.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:26.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:26.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:26.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:26.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:26.952 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:27.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:27.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:27.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:27.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:27.198 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:27.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:27.301 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:27.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:27.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:27.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:27.351 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:27.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:27.449 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:27.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:27.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:27.574 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:27.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:27.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:27.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:27 smithi162 ceph-mon[20279]: 4.9 scrub starts 2024-09-02T10:13:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:27 smithi162 ceph-mon[20279]: 4.9 scrub ok 2024-09-02T10:13:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:13:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:13:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:27 smithi162 ceph-mon[20279]: osdmap e398: 12 total, 12 up, 11 in 2024-09-02T10:13:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:13:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:13:27.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:13:27.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:27.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:27.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:27 smithi187 ceph-mon[20072]: 4.9 scrub starts 2024-09-02T10:13:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:27 smithi187 ceph-mon[20072]: 4.9 scrub ok 2024-09-02T10:13:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:13:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:13:27.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:27 smithi187 ceph-mon[20072]: osdmap e398: 12 total, 12 up, 11 in 2024-09-02T10:13:27.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:13:27.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:13:27.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:27.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:27.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:27.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:27.894 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:27 smithi151 ceph-mon[16639]: 4.9 scrub starts 2024-09-02T10:13:27.894 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:27 smithi151 ceph-mon[16639]: 4.9 scrub ok 2024-09-02T10:13:27.894 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:13:27.894 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:13:27.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:27 smithi151 ceph-mon[16639]: osdmap e398: 12 total, 12 up, 11 in 2024-09-02T10:13:27.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:13:27.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:13:27.932 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:27.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:27.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:28.009 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:28.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:28.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:28.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:28.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:28.149 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:28.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:28.204 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:13:28.204 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:13:28.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:28.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:28.239 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:13:28.239 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:13:28.266 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:13:28.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:13:28.295 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:13:28.295 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:13:28.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:13:28.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:13:28.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:28.353 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:13:28.353 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:13:28.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:28.354 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:28.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:28.382 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:13:28.382 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:13:28.413 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:13:28.413 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:13:28.445 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:13:28.446 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:13:28.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:28.480 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:28.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:28.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:28.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:13:28.481 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:13:28.494 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:13:28.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:13:28.519 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:13:28.544 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:13:28.544 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:13:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:13:28.545 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:13:28.545 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:13:28.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:28.587 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:13:28.588 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:13:28.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:28.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:28.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:28.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: pgmap v952: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 1315/155301 objects degraded (0.847%); 4165/155301 objects misplaced (2.682%); 22 MiB/s, 552 objects/s recovering 2024-09-02T10:13:28.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2374101219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:28.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: osdmap e399: 12 total, 12 up, 11 in 2024-09-02T10:13:28.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: 4.15 scrub starts 2024-09-02T10:13:28.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:13:28.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:13:28.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: 4.11 scrub starts 2024-09-02T10:13:28.740 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: 4.15 scrub ok 2024-09-02T10:13:28.741 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3428728017' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:28.741 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:13:28.741 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:28 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:13:28.743 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:28.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:28.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:28.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: pgmap v952: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 1315/155301 objects degraded (0.847%); 4165/155301 objects misplaced (2.682%); 22 MiB/s, 552 objects/s recovering 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2374101219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: osdmap e399: 12 total, 12 up, 11 in 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: 4.15 scrub starts 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: 4.11 scrub starts 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: 4.15 scrub ok 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3428728017' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:13:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:28 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:13:28.873 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:28.874 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:28.975 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:29.005 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:29.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: pgmap v952: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 53 active+clean; 3.2 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 683 op/s; 1315/155301 objects degraded (0.847%); 4165/155301 objects misplaced (2.682%); 22 MiB/s, 552 objects/s recovering 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2374101219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: osdmap e399: 12 total, 12 up, 11 in 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: 4.15 scrub starts 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: 4.11 scrub starts 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: 4.15 scrub ok 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3428728017' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:13:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:28 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:13:29.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:29.134 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:29.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:29.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:29.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:29.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:29.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:29.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:29.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:13:29.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:29.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:29.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:29.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:29.518 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:29.519 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:29.569 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:29.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:29.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:29.645 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:29.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:29.772 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:29.773 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:29 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:13:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:29 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:13:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2374101219' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:13:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:29 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:13:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:29 smithi162 ceph-mon[20279]: osdmap e400: 12 total, 12 up, 11 in 2024-09-02T10:13:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:29 smithi162 ceph-mon[20279]: 4.11 scrub ok 2024-09-02T10:13:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:29 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1315/155301 objects degraded (0.847%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:29 smithi162 ceph-mon[20279]: osdmap e401: 12 total, 12 up, 11 in 2024-09-02T10:13:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:29 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:13:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:29 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:13:29.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:29.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:29.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:29 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:13:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:29 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:13:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2374101219' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:13:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:29 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:13:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:29 smithi151 ceph-mon[16639]: osdmap e400: 12 total, 12 up, 11 in 2024-09-02T10:13:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:29 smithi151 ceph-mon[16639]: 4.11 scrub ok 2024-09-02T10:13:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:29 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1315/155301 objects degraded (0.847%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:29 smithi151 ceph-mon[16639]: osdmap e401: 12 total, 12 up, 11 in 2024-09-02T10:13:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:29 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:13:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:29 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:13:29.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:29 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:13:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:29 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:13:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2374101219' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:13:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:29 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:13:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:29 smithi187 ceph-mon[20072]: osdmap e400: 12 total, 12 up, 11 in 2024-09-02T10:13:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:29 smithi187 ceph-mon[20072]: 4.11 scrub ok 2024-09-02T10:13:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:29 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1315/155301 objects degraded (0.847%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:30.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:29 smithi187 ceph-mon[20072]: osdmap e401: 12 total, 12 up, 11 in 2024-09-02T10:13:30.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:29 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:13:30.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:29 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:13:30.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:30.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:30.121 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:13:30.121 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:13:30.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:13:30.121 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:13:30.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.121 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127528 2.7 2:e0932104:::benchmark_data_smithi187_45882_object29777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:13:30.121 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.657520+0000", 2024-09-02T10:13:30.121 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 584.46071160400004, 2024-09-02T10:13:30.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.67433600000000005, 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127528 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657520+0000", 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657520+0000", 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657522+0000", 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3400000000000001e-06 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657527+0000", 2024-09-02T10:13:30.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0599999999999998e-06 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657538+0000", 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1555e-05 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.657561+0000", 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2456e-05 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.658124+0000", 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056373299999999996 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.658258+0000", 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133101 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.658396+0000", 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000138901 2024-09-02T10:13:30.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.666255+0000", 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078582219999999998 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.668130+0000", 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00187503 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.330974+0000", 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.66284469899999998 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.331011+0000", 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6532000000000002e-05 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.331856+0000", 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00084537100000000003 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.127 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.128 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.128 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.7 156/131 [PushReplyOp(2:e2e982ef:::benchmark_data_smithi187_45882_object43210:head)])", 2024-09-02T10:13:30.128 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.912154+0000", 2024-09-02T10:13:30.128 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 584.20607773999996, 2024-09-02T10:13:30.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32342126199999999, 2024-09-02T10:13:30.128 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.128 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.128 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:30.129 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912154+0000", 2024-09-02T10:13:30.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912154+0000", 2024-09-02T10:13:30.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912155+0000", 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2810000000000001e-06 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.912161+0000", 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.412e-06 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.233179+0000", 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32101774300000002 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.233188+0000", 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4930000000000004e-06 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.233262+0000", 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3827999999999999e-05 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.235575+0000", 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002313505 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:75424 3.a 3:58480fd1:::benchmark_data_smithi187_59989_object75423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 389.75863348000001, 2024-09-02T10:13:30.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22582144000000001, 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75424 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:13:30.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359600+0000", 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3820000000000001e-06 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359629+0000", 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8775000000000001e-05 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359640+0000", 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1151e-05 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359648+0000", 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0620000000000001e-06 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359688+0000", 2024-09-02T10:13:30.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9882000000000002e-05 2024-09-02T10:13:30.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:30.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359791+0000", 2024-09-02T10:13:30.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010327699999999999 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359876+0000", 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5418000000000001e-05 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.361865+0000", 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019887289999999998 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362104+0000", 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023905199999999999 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585329+0000", 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223224432 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585374+0000", 2024-09-02T10:13:30.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4972999999999998e-05 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585420+0000", 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6307000000000002e-05 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24976.0:75424 3.a e253/226 ondisk, result = 0)", 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 389.75592793999999, 2024-09-02T10:13:30.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22312427800000001, 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362305+0000", 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9100000000000003e-07 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362309+0000", 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2899999999999996e-06 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585259+0000", 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22295003499999999 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585266+0000", 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6460000000000004e-06 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585311+0000", 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5802e-05 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585321+0000", 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3090000000000001e-06 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585428+0000", 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010750499999999999 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.144 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:155062 3.2 e282/226 3:4fe18cc9:::benchmark_data_smithi187_59989_object45519:head v 282'9762, mlcod=282'9758)", 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 305.64172458299998, 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29158908, 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476508+0000", 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1500000000000003e-07 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476514+0000", 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6559999999999997e-06 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.763870+0000", 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28735592300000001 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.763885+0000", 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4823e-05 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.764037+0000", 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015244400000000001 2024-09-02T10:13:30.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.764077+0000", 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0371999999999999e-05 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.768047+0000", 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039701650000000003 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.768096+0000", 2024-09-02T10:13:30.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8881999999999997e-05 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:181206 3.3 e291/226 3:c5ec1ffd:::benchmark_data_smithi187_59989_object71663:head v 291'11456, mlcod=291'11454)", 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 255.112028896, 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37650431099999998, 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006204+0000", 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6000000000000002e-07 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006211+0000", 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7880000000000001e-06 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377595+0000", 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.371383927 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377601+0000", 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0769999999999996e-06 2024-09-02T10:13:30.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377649+0000", 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8368000000000002e-05 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377669+0000", 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0258000000000002e-05 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.382663+0000", 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049941880000000001 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.382707+0000", 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3844999999999998e-05 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.153 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3855 4.a 4:57b917f5:::benchmark_data_smithi187_76542_object3854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 144.20891517000001, 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22371366500000001, 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3855 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909327+0000", 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9559999999999992e-06 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909360+0000", 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3556999999999997e-05 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909377+0000", 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6283e-05 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909385+0000", 2024-09-02T10:13:30.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4030000000000007e-06 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909444+0000", 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9203000000000002e-05 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909571+0000", 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127324 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909715+0000", 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143888 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.910145+0000", 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042936599999999998 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.917291+0000", 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071467290000000001 2024-09-02T10:13:30.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132953+0000", 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.215661517 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132989+0000", 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6123e-05 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.133030+0000", 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1316000000000002e-05 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3871 4.a 4:530be978:::benchmark_data_smithi187_76542_object3870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 144.20213936299999, 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21808112499999999, 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.160 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3871 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1899999999999998e-07 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916113+0000", 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9800999999999999e-05 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.162 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916118+0000", 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916121+0000", 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3330000000000001e-06 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916168+0000", 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7147000000000001e-05 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916291+0000", 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122934 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916427+0000", 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135337 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916951+0000", 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052393700000000004 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924167+0000", 2024-09-02T10:13:30.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072167580000000002 2024-09-02T10:13:30.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134113+0000", 2024-09-02T10:13:30.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20994539000000001 2024-09-02T10:13:30.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134147+0000", 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4149000000000001e-05 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134174+0000", 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6798999999999999e-05 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3887 4.a 4:5466a52d:::benchmark_data_smithi187_76542_object3886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 144.196481073, 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21253008300000001, 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3887 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921752+0000", 2024-09-02T10:13:30.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4160000000000001e-06 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921813+0000", 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0420000000000001e-05 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921822+0000", 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4299999999999995e-06 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921827+0000", 2024-09-02T10:13:30.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9819999999999999e-06 2024-09-02T10:13:30.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921867+0000", 2024-09-02T10:13:30.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0071000000000003e-05 2024-09-02T10:13:30.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921944+0000", 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6533999999999996e-05 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922025+0000", 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1601999999999996e-05 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922360+0000", 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033462200000000002 2024-09-02T10:13:30.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924931+0000", 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025713160000000001 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134215+0000", 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209284054 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134247+0000", 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2147999999999998e-05 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134281+0000", 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3488000000000002e-05 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3898 4.a 4:55cc6e50:::benchmark_data_smithi187_76542_object3897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:13:30.173 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 144.19148514099999, 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20761331199999999, 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3898 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:13:30.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926748+0000", 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6599999999999994e-07 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926786+0000", 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8506e-05 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926798+0000", 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.221e-05 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926804+0000", 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3140000000000003e-06 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926853+0000", 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8804999999999998e-05 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926965+0000", 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112256 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927110+0000", 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014520099999999999 2024-09-02T10:13:30.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927488+0000", 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000378332 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929483+0000", 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019947670000000002 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134309+0000", 2024-09-02T10:13:30.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20482557300000001 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134337+0000", 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8334999999999999e-05 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134360+0000", 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3047000000000001e-05 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.180 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3901 4.a 4:53342710:::benchmark_data_smithi187_76542_object3900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:13:30.181 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:13:30.181 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 144.189657049, 2024-09-02T10:13:30.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20590491499999999, 2024-09-02T10:13:30.181 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.181 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.181 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.181 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:30.181 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:30.181 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3901 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928576+0000", 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1699999999999997e-07 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928631+0000", 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5276999999999999e-05 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928642+0000", 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1195999999999999e-05 2024-09-02T10:13:30.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928649+0000", 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.669e-06 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928691+0000", 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1868e-05 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928778+0000", 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7416999999999993e-05 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928866+0000", 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8471e-05 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929233+0000", 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036694499999999998 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.942995+0000", 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013761572999999999 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134423+0000", 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19142793799999999 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134461+0000", 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7839000000000001e-05 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134480+0000", 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9004999999999998e-05 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:6012 4.5 4:a5fa00f8:::benchmark_data_smithi187_76542_object6011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 142.85522404700001, 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209785581, 2024-09-02T10:13:30.187 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6012 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263009+0000", 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0139999999999999e-06 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263120+0000", 2024-09-02T10:13:30.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011099000000000001 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263134+0000", 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4467999999999999e-05 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263141+0000", 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9820000000000002e-06 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263387+0000", 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024598300000000002 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263500+0000", 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113174 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263599+0000", 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8974999999999993e-05 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264127+0000", 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052785799999999997 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264358+0000", 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023054600000000001 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472691+0000", 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20833321599999999 2024-09-02T10:13:30.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472750+0000", 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8897000000000002e-05 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472793+0000", 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3477999999999999e-05 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:24533 4.c e335/333 4:3a0221c2:::benchmark_data_smithi187_76542_object24532:head v 335'1521, mlcod=335'1517)", 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 125.107190995, 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212837002, 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011042+0000", 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1650000000000001e-06 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011129+0000", 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7443999999999998e-05 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222953+0000", 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21182389400000001 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222960+0000", 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8569999999999997e-06 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223010+0000", 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0182000000000002e-05 2024-09-02T10:13:30.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223032+0000", 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1699999999999999e-05 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223838+0000", 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080572200000000004 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223878+0000", 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0037999999999999e-05 2024-09-02T10:13:30.197 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:25165 4.c e336/333 4:3c3f6316:::benchmark_data_smithi187_76542_object25164:head v 336'1563, mlcod=335'1560)", 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 124.472961445, 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204647308, 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645271+0000", 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9999999999999996e-07 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645443+0000", 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017179700000000001 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848893+0000", 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203449884 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848900+0000", 2024-09-02T10:13:30.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8360000000000003e-06 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848961+0000", 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1163999999999999e-05 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848982+0000", 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1403000000000002e-05 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.201 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849888+0000", 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000905469 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849918+0000", 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9855000000000001e-05 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.202 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:52446 4.15 e362/353 4:acabd05d:::benchmark_data_smithi187_76542_object52445:head v 362'2522, mlcod=362'2519)", 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 91.424832085000006, 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.220391959, 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9599999999999999e-07 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693457+0000", 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6582000000000001e-05 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912812+0000", 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21935476500000001 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912819+0000", 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4989999999999997e-06 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912871+0000", 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1388000000000003e-05 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912887+0000", 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5911999999999999e-05 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.913759+0000", 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087215899999999996 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.913792+0000", 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2957999999999999e-05 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:58518 4.1 e366/341 4:8490730e:::benchmark_data_smithi187_76542_object58517:head v 366'2807, mlcod=366'2804)", 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 85.753103858000003, 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209866793, 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.207 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365129+0000", 2024-09-02T10:13:30.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7100000000000001e-07 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365181+0000", 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1816e-05 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573805+0000", 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.2086249 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573812+0000", 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1349999999999997e-06 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573866+0000", 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4373000000000003e-05 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573888+0000", 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2008999999999999e-05 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.574955+0000", 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010673519999999999 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.574995+0000", 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9536999999999998e-05 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.211 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:75251 4.11 e371/356 4:8ba31329:::benchmark_data_smithi187_76542_object75250:head v 371'3396, mlcod=371'3394)", 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 72.452774837999996, 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37157874099999999, 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665458+0000", 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9400000000000004e-07 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665521+0000", 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2685e-05 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035523+0000", 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37000295799999999 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035542+0000", 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8692999999999999e-05 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035709+0000", 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016700999999999999 2024-09-02T10:13:30.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035745+0000", 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5913e-05 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.037004+0000", 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001258459 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.037036+0000", 2024-09-02T10:13:30.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2228999999999999e-05 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:90191 4.15 e375/353 4:afce227e:::benchmark_data_smithi187_76542_object90190:head v 375'3680, mlcod=375'3677)", 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.019985292000001, 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20444974699999999, 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9299999999999997e-07 2024-09-02T10:13:30.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098273+0000", 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5378999999999999e-05 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301201+0000", 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20292787800000001 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301213+0000", 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860000000000001e-05 2024-09-02T10:13:30.218 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301360+0000", 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014643600000000001 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301383+0000", 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2782999999999999e-05 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302662+0000", 2024-09-02T10:13:30.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012796470000000001 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302696+0000", 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3970999999999997e-05 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126679 4.1 e392/341 4:86205e4c:::benchmark_data_smithi187_76542_object34343:head v 392'4951, mlcod=392'4948)", 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.650425021, 2024-09-02T10:13:30.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44507702900000001, 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467808+0000", 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.277e-06 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467816+0000", 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3690000000000007e-06 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910880+0000", 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44306368899999998 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910889+0000", 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0380000000000005e-06 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910948+0000", 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8498000000000001e-05 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910970+0000", 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2580000000000001e-05 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912839+0000", 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018688120000000001 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912884+0000", 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4765999999999998e-05 2024-09-02T10:13:30.224 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126802 4.1 e392/341 4:81a050df:::benchmark_data_smithi187_76542_object34466:head v 392'4952, mlcod=392'4949)", 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.207381645, 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25345677799999999, 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:30.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910851+0000", 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0690000000000001e-06 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910859+0000", 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.233e-06 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162387+0000", 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25152825099999998 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162402+0000", 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4905999999999999e-05 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162538+0000", 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013591800000000001 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162581+0000", 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2991000000000002e-05 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164270+0000", 2024-09-02T10:13:30.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016893419999999999 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164307+0000", 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7067999999999998e-05 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:30.229 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:13:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:30.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:30.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:30.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:30.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:30.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:30.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:30.484 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:30.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:30.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:30.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:30.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:30.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:30.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:30.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:30.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:30.746 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:30 smithi162 ceph-mon[20279]: pgmap v956: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 3.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 906 op/s; 1340/147867 objects degraded (0.906%); 3967/147867 objects misplaced (2.683%); 8.9 MiB/s, 267 objects/s recovering 2024-09-02T10:13:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:30 smithi162 ceph-mon[20279]: osdmap e402: 12 total, 12 up, 11 in 2024-09-02T10:13:30.846 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:30.874 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:30.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:30.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:30 smithi151 ceph-mon[16639]: pgmap v956: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 3.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 906 op/s; 1340/147867 objects degraded (0.906%); 3967/147867 objects misplaced (2.683%); 8.9 MiB/s, 267 objects/s recovering 2024-09-02T10:13:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:30 smithi151 ceph-mon[16639]: osdmap e402: 12 total, 12 up, 11 in 2024-09-02T10:13:30.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:30.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:13:30.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:31.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:30 smithi187 ceph-mon[20072]: pgmap v956: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 3.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 906 op/s; 1340/147867 objects degraded (0.906%); 3967/147867 objects misplaced (2.683%); 8.9 MiB/s, 267 objects/s recovering 2024-09-02T10:13:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:30 smithi187 ceph-mon[20072]: osdmap e402: 12 total, 12 up, 11 in 2024-09-02T10:13:31.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:31.126 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:31.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:31.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:31.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:31.257 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:31.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:31.291 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:31.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-02T10:13:31.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:31.386 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:31.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:31.387 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:31.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:31.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:31.514 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:31.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:31.640 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:31.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:31 smithi151 ceph-mon[16639]: 4.18 scrub starts 2024-09-02T10:13:31.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:31.765 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:31.767 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:31.813 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:31 smithi162 ceph-mon[20279]: 4.18 scrub starts 2024-09-02T10:13:31.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:31.893 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:31.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:31.931 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:13:31.931 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:13:31.931 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.931 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:139417 4.3 4:c2165a9e:::benchmark_data_smithi187_76542_object47081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e402)", 2024-09-02T10:13:31.931 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:31.921552+0000", 2024-09-02T10:13:31.931 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0084808820000000004, 2024-09-02T10:13:31.931 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.008502852, 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 139417 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.921552+0000", 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:31.932 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.921552+0000", 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.921553+0000", 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8400000000000002e-07 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.921557+0000", 2024-09-02T10:13:31.933 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5210000000000004e-06 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.921570+0000", 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2595999999999999e-05 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.921577+0000", 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7739999999999999e-06 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:31.934 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.921626+0000", 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9323999999999997e-05 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.921760+0000", 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013383899999999999 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.921865+0000", 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105579 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.935 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.923647+0000", 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017818300000000001 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.923775+0000", 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127713 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.936 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:139423 4.16 e402/365 4:6e4a1ab6:::benchmark_data_smithi187_76542_object47087:head v 402'5301, mlcod=402'5298)", 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:31.929521+0000", 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00051216399999999996, 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00057974299999999997, 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.929521+0000", 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.937 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.929521+0000", 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.929521+0000", 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9400000000000005e-07 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.929527+0000", 2024-09-02T10:13:31.938 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8900000000000004e-06 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.929545+0000", 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7638999999999998e-05 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.929552+0000", 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.153e-06 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:31.939 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.929594+0000", 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1581000000000001e-05 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:31.929607+0000", 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3249e-05 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-02T10:13:31.940 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:13:31.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:31 smithi187 ceph-mon[20072]: 4.18 scrub starts 2024-09-02T10:13:32.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:32.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:32.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:32.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:32.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:32.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:32.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:32.282 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:32.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:32.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:32.383 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:32.407 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:32.407 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:32.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:32.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:32.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:32.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:32.658 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:32.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:32.716 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:32.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:32.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:32.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:32 smithi162 ceph-mon[20279]: 4.18 scrub ok 2024-09-02T10:13:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:32 smithi162 ceph-mon[20279]: pgmap v958: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 3.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 1340/145698 objects degraded (0.920%); 3967/145698 objects misplaced (2.723%); 2.6 MiB/s, 167 objects/s recovering 2024-09-02T10:13:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:32 smithi162 ceph-mon[20279]: 4.f scrub starts 2024-09-02T10:13:32.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:32 smithi151 ceph-mon[16639]: 4.18 scrub ok 2024-09-02T10:13:32.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:32 smithi151 ceph-mon[16639]: pgmap v958: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 3.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 1340/145698 objects degraded (0.920%); 3967/145698 objects misplaced (2.723%); 2.6 MiB/s, 167 objects/s recovering 2024-09-02T10:13:32.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:32 smithi151 ceph-mon[16639]: 4.f scrub starts 2024-09-02T10:13:32.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:32.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:32.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:32 smithi187 ceph-mon[20072]: 4.18 scrub ok 2024-09-02T10:13:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:32 smithi187 ceph-mon[20072]: pgmap v958: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 3.0 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 800 op/s; 1340/145698 objects degraded (0.920%); 3967/145698 objects misplaced (2.723%); 2.6 MiB/s, 167 objects/s recovering 2024-09-02T10:13:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:32 smithi187 ceph-mon[20072]: 4.f scrub starts 2024-09-02T10:13:33.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:33.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:33.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:33.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:33.173 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:33.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:33.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:33.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:13:33.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:33.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:33.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:33.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:33.431 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:33.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:33.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:33.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:33.536 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:33 smithi162 ceph-mon[20279]: 4.f scrub ok 2024-09-02T10:13:33.559 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:33.560 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:33.618 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:33.661 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:33.689 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:33.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:33.690 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:33.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:33.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:33.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:33.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:33.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:33 smithi151 ceph-mon[16639]: 4.f scrub ok 2024-09-02T10:13:33.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:33.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:33.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:33 smithi187 ceph-mon[20072]: 4.f scrub ok 2024-09-02T10:13:34.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:34.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:34.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:34.162 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:13:34.162 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:13:34.163 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:13:34.163 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:13:34.163 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:13:34.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:34.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:13:34.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:34.217 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:34.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:34.319 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:34.351 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:34.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:34.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:34.478 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:34.480 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:34.581 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:34.607 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:34.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:34.608 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:34.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:13:34.708 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:34.740 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:34.741 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:34.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:34 smithi162 ceph-mon[20279]: pgmap v959: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 1441/141891 objects degraded (1.016%); 3248/141891 objects misplaced (2.289%); 8.5 MiB/s, 223 objects/s recovering 2024-09-02T10:13:34.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:34 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:13:34.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:34 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:13:34.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:34 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1441/141891 objects degraded (1.016%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:34.792 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:34 smithi162 ceph-mon[20279]: osdmap e403: 12 total, 12 up, 11 in 2024-09-02T10:13:34.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:13:34.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:34.898 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:34.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:34 smithi151 ceph-mon[16639]: pgmap v959: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 1441/141891 objects degraded (1.016%); 3248/141891 objects misplaced (2.289%); 8.5 MiB/s, 223 objects/s recovering 2024-09-02T10:13:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:34 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:13:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:34 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:13:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:34 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1441/141891 objects degraded (1.016%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:34 smithi151 ceph-mon[16639]: osdmap e403: 12 total, 12 up, 11 in 2024-09-02T10:13:35.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:34 smithi187 ceph-mon[20072]: pgmap v959: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.9 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 774 op/s; 1441/141891 objects degraded (1.016%); 3248/141891 objects misplaced (2.289%); 8.5 MiB/s, 223 objects/s recovering 2024-09-02T10:13:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:34 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:13:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:34 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:13:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:34 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1441/141891 objects degraded (1.016%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:34 smithi187 ceph-mon[20072]: osdmap e403: 12 total, 12 up, 11 in 2024-09-02T10:13:35.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:35.027 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:35.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:35.140 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:13:35.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:35.155 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:35.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:35.278 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:35.279 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:35.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:35.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:35.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:35.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:35.528 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:35.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:35.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:35.630 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:35.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:35.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:35.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:35.739 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:13:35.739 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127423 2.0 2:06eddedb:::benchmark_data_smithi187_45882_object29672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.199335+0000", 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 590.51209437299997, 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0271604599999999, 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.740 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127423 2024-09-02T10:13:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199335+0000", 2024-09-02T10:13:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199335+0000", 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199337+0000", 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.564e-06 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199343+0000", 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7760000000000003e-06 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199354+0000", 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1321000000000001e-05 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199362+0000", 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6040000000000004e-06 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199422+0000", 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0145000000000001e-05 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199562+0000", 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013979699999999999 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.199712+0000", 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015045799999999999 2024-09-02T10:13:35.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.209645+0000", 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0099324600000000006 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.217410+0000", 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077656410000000002 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232716+0000", 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015306024 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232726+0000", 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3230000000000004e-06 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226479+0000", 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99375334900000001 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226496+0000", 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6997999999999999e-05 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127440 2.0 2.9f94cd30 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 590.49180020899996, 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0069557170000001, 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.747 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127440 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219630+0000", 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.63e-07 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219633+0000", 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.818e-06 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219639+0000", 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8000000000000004e-06 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.219645+0000", 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3980000000000004e-06 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232802+0000", 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013156805000000001 2024-09-02T10:13:35.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.232812+0000", 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0394000000000001e-05 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226575+0000", 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.99376249500000002 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226585+0000", 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0543999999999999e-05 2024-09-02T10:13:35.751 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127423 2.0 2.db7bb760 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.275687+0000", 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 590.43574255299995, 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95098331899999999, 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:35.752 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127423 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275687+0000", 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275687+0000", 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275689+0000", 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5829999999999999e-06 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275696+0000", 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1359999999999999e-06 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275707+0000", 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1024e-05 2024-09-02T10:13:35.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275715+0000", 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9750000000000007e-06 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279013+0000", 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003297688 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279020+0000", 2024-09-02T10:13:35.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0790000000000003e-06 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226644+0000", 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94762439200000004 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226671+0000", 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6441999999999999e-05 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.756 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.757 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127440 2.0 2.9f94cd30 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:13:35.757 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:13:35.757 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 590.43569898700002, 2024-09-02T10:13:35.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.95100121400000004, 2024-09-02T10:13:35.757 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.757 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.757 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:35.757 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127440 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:13:35.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275731+0000", 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6499999999999999e-07 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275734+0000", 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2299999999999998e-06 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275740+0000", 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8970000000000002e-06 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.275745+0000", 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.814e-06 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279036+0000", 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003290837 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279041+0000", 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4769999999999997e-06 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226722+0000", 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94768102099999996 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226732+0000", 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0473e-05 2024-09-02T10:13:35.761 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127464 2.0 2.e26bb170 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.277404+0000", 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 590.43402625700003, 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94936489599999996, 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:35.762 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127464 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277404+0000", 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277404+0000", 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277405+0000", 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4799999999999997e-07 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277410+0000", 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0529999999999997e-06 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277419+0000", 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6420000000000002e-06 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.277424+0000", 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.831e-06 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279299+0000", 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018751639999999999 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.279303+0000", 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.143e-06 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226761+0000", 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.94745774000000005 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226768+0000", 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3749999999999997e-06 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.766 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127473 2.0 2.e700b5b0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.298011+0000", 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 590.41341917199998, 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.92878800299999997, 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:35.767 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127473 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298011+0000", 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298011+0000", 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298012+0000", 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.412e-06 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298018+0000", 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9190000000000004e-06 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298030+0000", 2024-09-02T10:13:35.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2262000000000001e-05 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298035+0000", 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8969999999999999e-06 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298101+0000", 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5538999999999997e-05 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.298112+0000", 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1017e-05 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226792+0000", 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.92868043 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226799+0000", 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5270000000000004e-06 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.771 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:127507 2.0 2.51c134f0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:45.510346+0000", 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 590.20108402899996, 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.71648088300000001, 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:35.772 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127507 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510346+0000", 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510346+0000", 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510347+0000", 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2869999999999999e-06 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510353+0000", 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9290000000000003e-06 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510376+0000", 2024-09-02T10:13:35.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3251000000000002e-05 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510384+0000", 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5129999999999999e-06 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510437+0000", 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3245000000000003e-05 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:13:35.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:45.510446+0000", 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9910000000000008e-06 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226821+0000", 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.716374493 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.226827+0000", 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1739999999999997e-06 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.776 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(2.8 157/153 [PushReplyOp(2:10f8d703:::benchmark_data_smithi187_45882_object31118:head)])", 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:03:46.348475+0000", 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 589.36295455100003, 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0031679010000001, 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348475+0000", 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348475+0000", 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348476+0000", 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2900000000000002e-07 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:46.348484+0000", 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.695e-06 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.265961+0000", 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91747688900000002 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.265975+0000", 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3981e-05 2024-09-02T10:13:35.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.347454+0000", 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.081478748000000004 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:03:47.351643+0000", 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041896590000000001 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24451.0:141512 2.3 e169/92 2:c7378ca5:::benchmark_data_smithi187_45882_object43761:head v 169'8968, mlcod=169'8965)", 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:14.694189+0000", 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 561.01724105300002, 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.365733013, 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694189+0000", 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694189+0000", 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694190+0000", 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1209999999999999e-06 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:14.694197+0000", 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5700000000000004e-06 2024-09-02T10:13:35.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056209+0000", 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36201126900000002 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056216+0000", 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5390000000000002e-06 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056294+0000", 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7521000000000005e-05 2024-09-02T10:13:35.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.056309+0000", 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4691e-05 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.059878+0000", 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003569582 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:15.059922+0000", 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3720000000000002e-05 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:156192 2.8 2:133c662a:::benchmark_data_smithi187_45882_object58441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:43.142777+0000", 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 532.56865265700003, 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.60071848299999997, 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:35.785 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 156192 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142777+0000", 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142777+0000", 2024-09-02T10:13:35.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142779+0000", 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.655e-06 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142784+0000", 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5350000000000004e-06 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142798+0000", 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3525e-05 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.142805+0000", 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4050000000000003e-06 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.147079+0000", 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042733989999999998 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.147202+0000", 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012349799999999999 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,11", 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.147292+0000", 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9788000000000002e-05 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151073+0000", 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037812150000000001 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.153573+0000", 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025002380000000001 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742725+0000", 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.58915150900000002 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742781+0000", 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6039000000000003e-05 2024-09-02T10:13:35.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.743496+0000", 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00071467700000000004 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:156192 2.8 e188/153 ondisk, result = 0)", 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:04:43.151803+0000", 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 532.55962725400002, 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.59169890800000002, 2024-09-02T10:13:35.791 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.792 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.792 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:13:35.792 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151803+0000", 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151803+0000", 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151804+0000", 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5960000000000001e-06 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.151817+0000", 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2957e-05 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.740670+0000", 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.58885322699999998 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.740675+0000", 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7310000000000001e-06 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742713+0000", 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020379510000000001 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.742719+0000", 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2979999999999997e-06 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:04:43.743501+0000", 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00078214799999999998 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24451.0:195494 2.0 2:019f944c:::benchmark_data_smithi187_45882_object97743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:43.029456+0000", 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 472.681973997, 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.66465173200000005, 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.796 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24451", 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 195494 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029456+0000", 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029456+0000", 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029457+0000", 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.609e-06 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029464+0000", 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1240000000000002e-06 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029479+0000", 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5832999999999999e-05 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029487+0000", 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6149999999999999e-06 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029544+0000", 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6965000000000001e-05 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029647+0000", 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103219 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.029750+0000", 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102916 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034083+0000", 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043323980000000003 2024-09-02T10:13:35.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034644+0000", 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056196000000000002 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694038+0000", 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65939311300000003 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694078+0000", 2024-09-02T10:13:35.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0043999999999998e-05 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694108+0000", 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9935999999999999e-05 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24451.0:195494 2.0 e217/216 ondisk, result = 0)", 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:05:43.034227+0000", 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 472.67720288700002, 2024-09-02T10:13:35.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65988618399999999, 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034227+0000", 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034227+0000", 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034228+0000", 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1459999999999999e-06 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.034232+0000", 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9149999999999998e-06 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.693957+0000", 2024-09-02T10:13:35.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.65972478499999998 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.693968+0000", 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1198e-05 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694026+0000", 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7663e-05 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694033+0000", 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9750000000000001e-06 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:05:43.694113+0000", 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0501999999999996e-05 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:181199 3.f 3:fb925b3d:::benchmark_data_smithi187_59989_object71656:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:09:14.950521+0000", 2024-09-02T10:13:35.806 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 260.76090871700001, 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.43135058500000001, 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 181199 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.950521+0000", 2024-09-02T10:13:35.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.950521+0000", 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.950522+0000", 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8199999999999998e-07 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.950526+0000", 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7510000000000002e-06 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.950535+0000", 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0359999999999995e-06 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.950539+0000", 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.459e-06 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.950571+0000", 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1650999999999999e-05 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.950645+0000", 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3610999999999997e-05 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 8,11", 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.950719+0000", 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4725e-05 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.953878+0000", 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031591229999999998 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:14.953930+0000", 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1882000000000002e-05 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.381770+0000", 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42783937999999999 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.381825+0000", 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5395000000000001e-05 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.381872+0000", 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6690000000000002e-05 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.812 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:11574 4.3 4:c9ca2732:::benchmark_data_smithi187_76542_object11573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:10.852243+0000", 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 144.85918642499999, 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.97128315799999998, 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 11574 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.852243+0000", 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.852243+0000", 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.852245+0000", 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6929999999999999e-06 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.852275+0000", 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0213999999999999e-05 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.852292+0000", 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6521e-05 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.852299+0000", 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1380000000000001e-06 2024-09-02T10:13:35.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.852359+0000", 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0411000000000001e-05 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.852484+0000", 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012506400000000001 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.852626+0000", 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014185399999999999 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.853350+0000", 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00072326699999999999 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:10.853928+0000", 2024-09-02T10:13:35.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00057877999999999998 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:11.823436+0000", 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.96950731800000001 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:11.823489+0000", 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3681000000000003e-05 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:11.823527+0000", 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7217e-05 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:92328 4.17 4:e9ed0e66:::benchmark_data_smithi187_76542_object92327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:33.496001+0000", 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 62.215428660000001, 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46501651100000002, 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.819 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 92328 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.496001+0000", 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.496001+0000", 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.496008+0000", 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3269999999999997e-06 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.496056+0000", 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8832000000000003e-05 2024-09-02T10:13:35.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.496069+0000", 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2702e-05 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.496075+0000", 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4389999999999997e-06 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.496124+0000", 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8545999999999999e-05 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.496218+0000", 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3661999999999999e-05 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.496305+0000", 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6934999999999994e-05 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:35.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.497565+0000", 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001260681 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.498040+0000", 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00047434000000000002 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.960933+0000", 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46289352700000003 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.960980+0000", 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7040999999999999e-05 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.961018+0000", 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7478999999999998e-05 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.25507.0:92328 4.17 e377/371 ondisk, result = 0)", 2024-09-02T10:13:35.825 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:33.498256+0000", 2024-09-02T10:13:35.826 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 62.213173634, 2024-09-02T10:13:35.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:35.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46276857399999999, 2024-09-02T10:13:35.827 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.827 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.827 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:13:35.827 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.498256+0000", 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.498256+0000", 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.498257+0000", 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5999999999999999e-07 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.498259+0000", 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2419999999999999e-06 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.960844+0000", 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46258493699999997 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.960854+0000", 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9410000000000002e-06 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.960920+0000", 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6131999999999996e-05 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.960929+0000", 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9439999999999994e-06 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:33.961025+0000", 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5917999999999999e-05 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:103690 4.17 4:eb8e19c4:::benchmark_data_smithi187_76542_object11354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:46.092127+0000", 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.619303250000002, 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29783116399999998, 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.831 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 103690 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.092127+0000", 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.092127+0000", 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.092129+0000", 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9599999999999999e-06 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.092135+0000", 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1480000000000003e-06 2024-09-02T10:13:35.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.092148+0000", 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2907e-05 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.092157+0000", 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5210000000000008e-06 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.093975+0000", 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018174969999999999 2024-09-02T10:13:35.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.094118+0000", 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014365800000000001 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 5,10", 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.094264+0000", 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000146129 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.095767+0000", 2024-09-02T10:13:35.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015024210000000001 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.099582+0000", 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003815639 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.388475+0000", 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28889283900000001 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.388543+0000", 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7342999999999995e-05 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.389958+0000", 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014151019999999999 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.25507.0:103690 4.17 e384/381 ondisk, result = 0)", 2024-09-02T10:13:35.837 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:46.095856+0000", 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.615573392000002, 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.294109169, 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.095856+0000", 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.095856+0000", 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.095857+0000", 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5799999999999998e-07 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.095862+0000", 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3019999999999996e-06 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.382912+0000", 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28705068900000003 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.382919+0000", 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8129999999999998e-06 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.388458+0000", 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055388479999999999 2024-09-02T10:13:35.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.388468+0000", 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6970000000000005e-06 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:46.389966+0000", 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001497962 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.841 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(4.6 e394,385 4:62590fd1:::benchmark_data_smithi187_76542_object7488:head,380'3977)", 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:20.321898+0000", 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 15.389531351, 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.339685711, 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:20.321898+0000", 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:20.321898+0000", 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:20.321899+0000", 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0800000000000004e-07 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:20.321905+0000", 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.728e-06 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:20.661472+0000", 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33956667000000001 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:20.661480+0000", 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5420000000000005e-06 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:20.661532+0000", 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2138999999999997e-05 2024-09-02T10:13:35.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:20.661538+0000", 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5339999999999999e-06 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:20.661584+0000", 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6390000000000001e-05 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:13:35.845 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:13:35.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:35.857 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:35.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:35.985 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:35.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:35.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:36.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:36.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:36.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:36.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:36.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:36.243 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:36.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:36.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:36.345 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:36.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:36.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:36.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:36.504 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:36.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:36.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-02T10:13:36.559 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:36.565 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:36.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:36.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:36.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:36.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:36.717 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:13:36.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:36.758 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:36 smithi151 ceph-mon[16639]: pgmap v961: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 1441/133068 objects degraded (1.083%); 3248/133068 objects misplaced (2.441%); 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:13:36.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:36.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:36.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:36 smithi162 ceph-mon[20279]: pgmap v961: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 1441/133068 objects degraded (1.083%); 3248/133068 objects misplaced (2.441%); 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:13:36.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:36.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:36.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:36.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:36.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:36 smithi187 ceph-mon[20072]: pgmap v961: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.7 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 781 op/s; 1441/133068 objects degraded (1.083%); 3248/133068 objects misplaced (2.441%); 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:13:37.016 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:37.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:37.062 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:37.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:37.143 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:37.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:37.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:37.269 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:37.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:37.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:37.402 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:37.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:37.403 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:37.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:37.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:37.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:37.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:37.607 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:13:37.607 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:13:37.607 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:13:37.607 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:13:37.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:37.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/88384143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:37.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:37.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:37.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/88384143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:37.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:37.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:37.833 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:37.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:37.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/88384143' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:37.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:37.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:37.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:37.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:38.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:38.093 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:38.094 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:38.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:38.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:38.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:38.255 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:38.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:38.352 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:38.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:38.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:38.381 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:13:38.381 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:13:38.413 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:13:38.414 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:13:38.423 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:13:38.445 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:13:38.446 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:13:38.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:38.476 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:13:38.477 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:13:38.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:38.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:38.505 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:13:38.506 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:13:38.536 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:13:38.537 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:13:38.548 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:13:38.569 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:13:38.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:13:38.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:38.602 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:13:38.602 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:13:38.606 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:38.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:38.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:38.633 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:13:38.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:13:38.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:38 smithi151 ceph-mon[16639]: pgmap v962: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 782 op/s; 1441/129072 objects degraded (1.116%); 2821/129072 objects misplaced (2.186%); 8.2 MiB/s, 130 objects/s recovering 2024-09-02T10:13:38.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:38 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:13:38.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:38 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:13:38.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3353381183' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:13:38.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:38 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:13:38.668 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:13:38.668 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:13:38.691 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:13:38.691 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:13:38.709 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:38.717 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:13:38.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:13:38.731 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:38.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:38.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:38.741 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:38.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:38.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:38 smithi162 ceph-mon[20279]: pgmap v962: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 782 op/s; 1441/129072 objects degraded (1.116%); 2821/129072 objects misplaced (2.186%); 8.2 MiB/s, 130 objects/s recovering 2024-09-02T10:13:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:38 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:13:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:38 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:13:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3353381183' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:13:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:38 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:13:38.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:38.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:38.868 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:38 smithi187 ceph-mon[20072]: pgmap v962: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.6 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 782 op/s; 1441/129072 objects degraded (1.116%); 2821/129072 objects misplaced (2.186%); 8.2 MiB/s, 130 objects/s recovering 2024-09-02T10:13:38.868 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:38 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:13:38.868 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:38 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:13:38.868 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3353381183' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:13:38.868 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:38 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:13:38.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:38.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:38.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:39.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:39.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:39.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:39.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:39.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:39.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:39.253 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:39.338 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:39.338 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:39.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:39.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:39.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:39.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:39.511 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:39.512 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:39.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:39.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:39.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:39.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:39.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:39 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:39.686 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:39 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:13:39.686 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:39 smithi151 ceph-mon[16639]: osdmap e404: 12 total, 12 up, 11 in 2024-09-02T10:13:39.686 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1441/129072 objects degraded (1.116%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:39.686 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:39 smithi151 ceph-mon[16639]: osdmap e405: 12 total, 12 up, 11 in 2024-09-02T10:13:39.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:39.766 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:39.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:39 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:39 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:13:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:39 smithi162 ceph-mon[20279]: osdmap e404: 12 total, 12 up, 11 in 2024-09-02T10:13:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1441/129072 objects degraded (1.116%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:39 smithi162 ceph-mon[20279]: osdmap e405: 12 total, 12 up, 11 in 2024-09-02T10:13:39.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:39.896 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:39.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:39.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:39.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:39 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:39 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:13:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:39 smithi187 ceph-mon[20072]: osdmap e404: 12 total, 12 up, 11 in 2024-09-02T10:13:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1441/129072 objects degraded (1.116%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:13:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:39 smithi187 ceph-mon[20072]: osdmap e405: 12 total, 12 up, 11 in 2024-09-02T10:13:40.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:40.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:40.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:40.126 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:40.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:40.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:40.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:40.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:13:40.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:40.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:40.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:40.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:40.408 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:40.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:40.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:40.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:40.536 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:40.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:40.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:40.584 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:40.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:40.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:40.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:40.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:40 smithi151 ceph-mon[16639]: pgmap v965: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s; 1588/120171 objects degraded (1.321%); 2521/120171 objects misplaced (2.098%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:13:40.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:40.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:40.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:40.802 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:40.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:40 smithi162 ceph-mon[20279]: pgmap v965: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s; 1588/120171 objects degraded (1.321%); 2521/120171 objects misplaced (2.098%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:13:40.874 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:40 smithi187 ceph-mon[20072]: pgmap v965: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 1.21k op/s; 1588/120171 objects degraded (1.321%); 2521/120171 objects misplaced (2.098%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:13:40.903 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:40.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-02T10:13:40.962 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:40.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:40.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:41.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:41.097 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:41.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:41.196 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:41.196 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:41.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:41.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:41.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:41.229 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:41.323 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:41.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:41.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:41.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:41.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:41.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:41.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:41.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:41.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:41.615 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:41.617 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:41.717 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:41.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:13:41.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:41.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:41.745 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:41.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:41 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:13:41.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:41 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:13:41.842 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:41 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:13:41.842 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:41 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:13:41.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:41 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:13:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:41 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:13:41.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:41.871 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:41.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:41.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:13:41.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:41.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:41.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:41.998 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:42.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:42.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:42.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:42.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:42.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:13:42.217 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:42.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:42.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:42.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:42.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:42.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:42.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:42.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:42.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:42.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:42.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:42.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:42.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:42.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:42.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:42.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:42.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:42.656 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:42 smithi187 ceph-mon[20072]: pgmap v966: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 1588/117474 objects degraded (1.352%); 2521/117474 objects misplaced (2.146%); 5.0 MiB/s, 79 objects/s recovering 2024-09-02T10:13:42.656 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:42 smithi187 ceph-mon[20072]: osdmap e406: 12 total, 12 up, 11 in 2024-09-02T10:13:42.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:42.772 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:42.772 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:42.773 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:42 smithi151 ceph-mon[16639]: pgmap v966: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 1588/117474 objects degraded (1.352%); 2521/117474 objects misplaced (2.146%); 5.0 MiB/s, 79 objects/s recovering 2024-09-02T10:13:42.773 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:42 smithi151 ceph-mon[16639]: osdmap e406: 12 total, 12 up, 11 in 2024-09-02T10:13:42.773 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:42.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:42.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.8 reset_purged_snaps_last 2024-09-02T10:13:42.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:42.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:42 smithi162 ceph-mon[20279]: pgmap v966: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.4 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 1588/117474 objects degraded (1.352%); 2521/117474 objects misplaced (2.146%); 5.0 MiB/s, 79 objects/s recovering 2024-09-02T10:13:42.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:42 smithi162 ceph-mon[20279]: osdmap e406: 12 total, 12 up, 11 in 2024-09-02T10:13:42.876 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:42.883 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:42.900 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:42.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:42.943 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:43.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:43.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:43.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:43.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:43.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:43.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:43.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:43.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:43.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:43.281 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:43.282 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:43.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:43.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:43.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:43.457 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:13:43.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:43.546 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:43.547 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:43.648 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:43.672 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:43.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:43.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:43.809 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:43.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:43.815 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:43.912 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:43.937 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:43.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:44.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:44.065 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:44.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:44.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:44.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:44.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:44.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:44.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:44.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:44.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:44.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:44.345 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:44.345 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:44.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:44.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:44.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:44.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:44.575 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:44.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:44.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:44.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:44.699 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:44.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:44.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:44.822 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:44.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:44.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:44 smithi162 ceph-mon[20279]: pgmap v968: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 888 op/s; 1374/113076 objects degraded (1.215%); 2170/113076 objects misplaced (1.919%); 3.6 MiB/s, 177 objects/s recovering 2024-09-02T10:13:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:44 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1374/113076 objects degraded (1.215%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:44 smithi162 ceph-mon[20279]: osdmap e407: 12 total, 12 up, 11 in 2024-09-02T10:13:44.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:44.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:44 smithi151 ceph-mon[16639]: pgmap v968: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 888 op/s; 1374/113076 objects degraded (1.215%); 2170/113076 objects misplaced (1.919%); 3.6 MiB/s, 177 objects/s recovering 2024-09-02T10:13:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:44 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1374/113076 objects degraded (1.215%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:44 smithi151 ceph-mon[16639]: osdmap e407: 12 total, 12 up, 11 in 2024-09-02T10:13:44.939 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:44 smithi187 ceph-mon[20072]: pgmap v968: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.3 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 888 op/s; 1374/113076 objects degraded (1.215%); 2170/113076 objects misplaced (1.919%); 3.6 MiB/s, 177 objects/s recovering 2024-09-02T10:13:44.939 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:44 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1374/113076 objects degraded (1.215%), 2 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:44.940 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:44 smithi187 ceph-mon[20072]: osdmap e407: 12 total, 12 up, 11 in 2024-09-02T10:13:44.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:44.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:44.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:45.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:45.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:45.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:45.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:45.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:13:45.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:45.196 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:45.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:45.298 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:45.320 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:45.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:45.422 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:45.423 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:45.450 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:45.451 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:45.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:45.574 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:45.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:45.575 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:45.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:45.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:45.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:45.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:45.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:45.822 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:45.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:45.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:45.945 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:45.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:46.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:45 smithi151 ceph-mon[16639]: osdmap e408: 12 total, 12 up, 11 in 2024-09-02T10:13:46.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:45 smithi151 ceph-mon[16639]: pgmap v971: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.1 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 903 op/s; 1374/103902 objects degraded (1.322%); 2170/103902 objects misplaced (2.089%); 2.0 MiB/s, 152 objects/s recovering 2024-09-02T10:13:46.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2490324668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:46.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:46.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:46.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:46.088 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:46.089 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:46.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:46.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:46.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:46.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:45 smithi187 ceph-mon[20072]: osdmap e408: 12 total, 12 up, 11 in 2024-09-02T10:13:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:45 smithi187 ceph-mon[20072]: pgmap v971: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.1 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 903 op/s; 1374/103902 objects degraded (1.322%); 2170/103902 objects misplaced (2.089%); 2.0 MiB/s, 152 objects/s recovering 2024-09-02T10:13:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2490324668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:46.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:46.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:13:46.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:13:46.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:13:46.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:13:46.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:13:46.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:13:46.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:13:46.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:13:46.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:13:46.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:13:46.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:13:46.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:13:46.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:13:46.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:13:46.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:13:46.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:13:46.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:13:46.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:13:46.314 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:13:46.314 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:13:46.314 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:13:46.314 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:13:46.314 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:13:46.314 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:13:46.334 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:46.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:46.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:45 smithi162 ceph-mon[20279]: osdmap e408: 12 total, 12 up, 11 in 2024-09-02T10:13:46.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:45 smithi162 ceph-mon[20279]: pgmap v971: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 55 active+clean; 2.1 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 903 op/s; 1374/103902 objects degraded (1.322%); 2170/103902 objects misplaced (2.089%); 2.0 MiB/s, 152 objects/s recovering 2024-09-02T10:13:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2490324668' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:46.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:46.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:46.462 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:46.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:46.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:46.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:46.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:46.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:46.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:46.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:46.778 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:13:46.814 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:46.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:46.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:46.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:46.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:46.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:46 smithi187 ceph-mon[20072]: osdmap e409: 12 total, 12 up, 11 in 2024-09-02T10:13:47.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:47.103 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:47.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:47.124 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:47.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:47.231 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:47.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:47.232 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:47.247 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:46 smithi151 ceph-mon[16639]: osdmap e409: 12 total, 12 up, 11 in 2024-09-02T10:13:47.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:47.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:46 smithi162 ceph-mon[20279]: osdmap e409: 12 total, 12 up, 11 in 2024-09-02T10:13:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:47.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:47.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:47.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:47.497 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:47.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:47.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:47.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:47.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:47.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:47.662 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:13:47.662 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:13:47.662 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.662 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:153756 4.4 4:2736ea6e:::benchmark_data_smithi187_76542_object61420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-09-02T10:13:47.662 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:47.657700+0000", 2024-09-02T10:13:47.662 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.003283481, 2024-09-02T10:13:47.662 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033051909999999999, 2024-09-02T10:13:47.663 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:47.663 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:47.663 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:13:47.663 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:47.663 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:47.663 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:47.663 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 153756 2024-09-02T10:13:47.663 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.663 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:47.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.657700+0000", 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.657700+0000", 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:47.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.657702+0000", 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9429999999999999e-06 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.657707+0000", 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.843e-06 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.657717+0000", 2024-09-02T10:13:47.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0129000000000001e-05 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.657724+0000", 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.365e-06 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.657978+0000", 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000254462 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658084+0000", 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010618199999999999 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,8", 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658178+0000", 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4115000000000003e-05 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:47.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:153759 4.5 4:a4229c34:::benchmark_data_smithi187_76542_object61423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:47.658086+0000", 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0028972189999999999, 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029597590000000002, 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 153759 2024-09-02T10:13:47.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658086+0000", 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658086+0000", 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658087+0000", 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7399999999999991e-07 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658092+0000", 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.685e-06 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658104+0000", 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.171e-05 2024-09-02T10:13:47.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658111+0000", 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8299999999999998e-06 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658179+0000", 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8497999999999994e-05 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658279+0000", 2024-09-02T10:13:47.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9610000000000003e-05 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.658369+0000", 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0344000000000001e-05 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 2 2024-09-02T10:13:47.672 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:13:47.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:47.766 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:47.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:47.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:47.894 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:47.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:47.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:47.976 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:13:47.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:48.016 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:48.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:48.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:48.120 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:48.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:48.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:48.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:47 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:13:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:47 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:13:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:47 smithi187 ceph-mon[20072]: osdmap e410: 12 total, 12 up, 11 in 2024-09-02T10:13:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:47 smithi187 ceph-mon[20072]: pgmap v974: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 2.0 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 1018/100578 objects degraded (1.012%); 2170/100578 objects misplaced (2.158%); 0 B/s, 88 objects/s recovering 2024-09-02T10:13:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1228842214' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:48.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:48.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:47 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:13:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:47 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:13:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:47 smithi162 ceph-mon[20279]: osdmap e410: 12 total, 12 up, 11 in 2024-09-02T10:13:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:47 smithi162 ceph-mon[20279]: pgmap v974: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 2.0 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 1018/100578 objects degraded (1.012%); 2170/100578 objects misplaced (2.158%); 0 B/s, 88 objects/s recovering 2024-09-02T10:13:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1228842214' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:48.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:48.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:13:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:48.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:48.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:47 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:13:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:47 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:13:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:47 smithi151 ceph-mon[16639]: osdmap e410: 12 total, 12 up, 11 in 2024-09-02T10:13:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:47 smithi151 ceph-mon[16639]: pgmap v974: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 2.0 GiB data, 9.5 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 1018/100578 objects degraded (1.012%); 2170/100578 objects misplaced (2.158%); 0 B/s, 88 objects/s recovering 2024-09-02T10:13:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1228842214' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:13:48.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:48.548 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:48.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:48.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:48.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:48.676 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:48.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:48.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:48.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:48.790 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:48.803 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:48.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:48.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:48.934 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:48.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:48.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:49.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:49.059 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:49.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:49.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:49.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:49.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:49.202 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:48 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:49.202 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1228842214' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:13:49.202 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:48 smithi187 ceph-mon[20072]: osdmap e411: 12 total, 12 up, 11 in 2024-09-02T10:13:49.202 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:48 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1018/100578 objects degraded (1.012%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:49.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:49.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:13:49.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:49.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:13:49.338 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:13:49.338 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:13:49.338 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:13:49.338 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:13:49.339 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:48 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:49.339 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1228842214' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:13:49.339 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:48 smithi151 ceph-mon[16639]: osdmap e411: 12 total, 12 up, 11 in 2024-09-02T10:13:49.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:48 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1018/100578 objects degraded (1.012%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:49.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:48 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1228842214' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:13:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:48 smithi162 ceph-mon[20279]: osdmap e411: 12 total, 12 up, 11 in 2024-09-02T10:13:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:48 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1018/100578 objects degraded (1.012%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:49.413 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:49.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:49.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:49.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:49.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:49.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:49.566 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:49.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:49.667 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:49.692 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:49.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:49.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:49.794 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:49.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:49.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:49.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:49.950 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:49.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:50.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:50.063 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:13:50.079 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:50.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:50.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:50.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:50.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:50.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:49 smithi187 ceph-mon[20072]: pgmap v976: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.9 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 835 op/s; 1073/93111 objects degraded (1.152%); 1888/93111 objects misplaced (2.028%); 3.3 MiB/s, 135 objects/s recovering 2024-09-02T10:13:50.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:50.338 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:50.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:50.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:49 smithi162 ceph-mon[20279]: pgmap v976: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.9 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 835 op/s; 1073/93111 objects degraded (1.152%); 1888/93111 objects misplaced (2.028%); 3.3 MiB/s, 135 objects/s recovering 2024-09-02T10:13:50.428 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:50.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:49 smithi151 ceph-mon[16639]: pgmap v976: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.9 GiB data, 9.1 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 835 op/s; 1073/93111 objects degraded (1.152%); 1888/93111 objects misplaced (2.028%); 3.3 MiB/s, 135 objects/s recovering 2024-09-02T10:13:50.440 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:50.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:50.468 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:50.468 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:50.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.9 reset_purged_snaps_last 2024-09-02T10:13:50.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:50.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:50.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:50.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:50.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:50.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:50.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:50.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:50.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:50.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:50.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:50.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:50.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:50.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:50.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:50.992 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:13:50.992 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:13:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:13:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:13:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:3929 4.5 e324/318 4:af3910d4:::benchmark_data_smithi187_76542_object3928:head v 324'255, mlcod=324'252)", 2024-09-02T10:13:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:05.948100+0000", 2024-09-02T10:13:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 165.04164607000001, 2024-09-02T10:13:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18627370900000001, 2024-09-02T10:13:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:50.993 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:05.948100+0000", 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:05.948100+0000", 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:50.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:05.948101+0000", 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4799999999999997e-07 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:05.948156+0000", 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5145999999999997e-05 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.133889+0000", 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.185732598 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.133894+0000", 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8729999999999998e-06 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.133939+0000", 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5148999999999998e-05 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.133954+0000", 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5067e-05 2024-09-02T10:13:50.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.134338+0000", 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00038381500000000002 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:06.134374+0000", 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6313000000000003e-05 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:25387 4.a e336/333 4:51410dd3:::benchmark_data_smithi187_76542_object25386:head v 336'1582, mlcod=325'647)", 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 145.02550372600001, 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20680125099999999, 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964244+0000", 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5900000000000005e-07 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964284+0000", 2024-09-02T10:13:50.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0487e-05 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169804+0000", 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20551941600000001 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169809+0000", 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6319999999999996e-06 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169867+0000", 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7938e-05 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169895+0000", 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7430999999999999e-05 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.171002+0000", 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011072790000000001 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.171044+0000", 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2109e-05 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:32958 4.5 4:a717a701:::benchmark_data_smithi187_76542_object32957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e346)", 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:38.731478+0000", 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 132.258268851, 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.153665837, 2024-09-02T10:13:51.002 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 32958 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731478+0000", 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731478+0000", 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731479+0000", 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.795e-06 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731505+0000", 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5527000000000001e-05 2024-09-02T10:13:51.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731523+0000", 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7546000000000001e-05 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731531+0000", 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7239999999999998e-06 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731591+0000", 2024-09-02T10:13:51.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0154e-05 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731694+0000", 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010283 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.731796+0000", 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010157699999999999 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:13:51.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.732584+0000", 2024-09-02T10:13:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000787947 2024-09-02T10:13:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.733556+0000", 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000972152 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.885024+0000", 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15146827700000001 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.885092+0000", 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7997999999999995e-05 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:38.885143+0000", 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1310000000000002e-05 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:44087 4.7 4:e44e1657:::benchmark_data_smithi187_76542_object44086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:50.597202+0000", 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 120.392544927, 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20155887, 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.009 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 44087 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597202+0000", 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597202+0000", 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597203+0000", 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1969999999999999e-06 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597224+0000", 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1421999999999999e-05 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597236+0000", 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2155e-05 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597244+0000", 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6410000000000002e-06 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597293+0000", 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8990999999999999e-05 2024-09-02T10:13:51.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597387+0000", 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3700000000000001e-05 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597478+0000", 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1479999999999998e-05 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.598211+0000", 2024-09-02T10:13:51.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00073308599999999998 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.598723+0000", 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051141900000000004 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.798694+0000", 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19997105900000001 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.798730+0000", 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6631999999999998e-05 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.798760+0000", 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0088000000000001e-05 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:58507 4.19 4:9b45fe1a:::benchmark_data_smithi187_76542_object58506:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:04.359480+0000", 2024-09-02T10:13:51.015 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 106.63026618000001, 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.17705659700000001, 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 58507 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.359480+0000", 2024-09-02T10:13:51.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.359480+0000", 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.359481+0000", 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8700000000000005e-07 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.359500+0000", 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9213999999999999e-05 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.359507+0000", 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.72e-06 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.359512+0000", 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.967e-06 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.359580+0000", 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7780000000000005e-05 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.359718+0000", 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013861099999999999 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.359876+0000", 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015789299999999999 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.361190+0000", 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013142270000000001 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.361819+0000", 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00062846799999999997 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.536456+0000", 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.17463699999999999 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.536493+0000", 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6927000000000001e-05 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:04.536537+0000", 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4103000000000001e-05 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.021 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:88797 4.19 4:98f3844c:::benchmark_data_smithi187_76542_object88796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:29.219562+0000", 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 81.770184069999999, 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.163081488, 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 88797 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.219562+0000", 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.219562+0000", 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.219564+0000", 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.195e-06 2024-09-02T10:13:51.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.219583+0000", 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9375e-05 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.219596+0000", 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2525e-05 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.219601+0000", 2024-09-02T10:13:51.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4999999999999999e-06 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.219655+0000", 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4107000000000002e-05 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.219757+0000", 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101628 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-02T10:13:51.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.219846+0000", 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8818000000000002e-05 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.221033+0000", 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001187373 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.221639+0000", 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00060632000000000004 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.382570+0000", 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16093048500000001 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.382611+0000", 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0831999999999998e-05 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.382644+0000", 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3330000000000001e-05 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.027 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:88800 4.7 4:e0e7d000:::benchmark_data_smithi187_76542_object88799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:29.222529+0000", 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 81.767217105, 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16928864199999999, 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 88800 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.028 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.222529+0000", 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.222529+0000", 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.222530+0000", 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3600000000000003e-07 2024-09-02T10:13:51.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.222547+0000", 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7238999999999999e-05 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.222557+0000", 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0124e-05 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.222562+0000", 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9370000000000003e-06 2024-09-02T10:13:51.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.222605+0000", 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3028000000000002e-05 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.222690+0000", 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4553000000000002e-05 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.222775+0000", 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5210000000000006e-05 2024-09-02T10:13:51.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.223960+0000", 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001184907 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.224445+0000", 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048470000000000002 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.391754+0000", 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16730915599999999 2024-09-02T10:13:51.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.391790+0000", 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6222e-05 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.391818+0000", 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.783e-05 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25507.0:88800 4.7 e375/341 ondisk, result = 0)", 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:29.224304+0000", 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 81.765442153999999, 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16751985799999999, 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.224304+0000", 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.224304+0000", 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.224305+0000", 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5499999999999998e-07 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.224307+0000", 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5189999999999999e-06 2024-09-02T10:13:51.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.391712+0000", 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16740506799999999 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.391717+0000", 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5789999999999997e-06 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.391746+0000", 2024-09-02T10:13:51.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9238999999999999e-05 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.391750+0000", 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2570000000000001e-06 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:29.391824+0000", 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3740999999999998e-05 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90175 4.19 4:988493fd:::benchmark_data_smithi187_76542_object90174:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:31.066034+0000", 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 79.923712351000006, 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18949429600000001, 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90175 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.066034+0000", 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.066034+0000", 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.066035+0000", 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1999999999999999e-06 2024-09-02T10:13:51.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.066056+0000", 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0632000000000002e-05 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.066068+0000", 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1678000000000001e-05 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.066074+0000", 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8619999999999999e-06 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.066122+0000", 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8572000000000002e-05 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.066216+0000", 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4182000000000001e-05 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,2", 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.066302+0000", 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5741999999999993e-05 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.067510+0000", 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012075359999999999 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.084854+0000", 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017344239000000001 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.255471+0000", 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.17061702400000001 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.255504+0000", 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3501000000000003e-05 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.255528+0000", 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4128e-05 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.043 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:92945 4.5 4:a1c5a0f5:::benchmark_data_smithi187_76542_object609:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:34.858377+0000", 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 76.131369390000003, 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15841824299999999, 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.044 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 92945 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.858377+0000", 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.858377+0000", 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.858378+0000", 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2610000000000001e-06 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.858384+0000", 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7409999999999998e-06 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.858399+0000", 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4457000000000001e-05 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.858406+0000", 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1040000000000001e-06 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.858451+0000", 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5543000000000001e-05 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.858529+0000", 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7581000000000001e-05 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:13:51.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.858602+0000", 2024-09-02T10:13:51.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3687000000000001e-05 2024-09-02T10:13:51.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.859885+0000", 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001282409 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:34.860681+0000", 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000796183 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:35.016680+0000", 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15599897099999999 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:35.016748+0000", 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8332999999999998e-05 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:35.016795+0000", 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6972999999999999e-05 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.050 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103692 4.19 4.40dd9139 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 64.894003335999997, 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.993593167, 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103692 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.051 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095744+0000", 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6899999999999996e-07 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095748+0000", 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4499999999999997e-06 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095758+0000", 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2140000000000002e-06 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095764+0000", 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5690000000000001e-06 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095808+0000", 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3588999999999999e-05 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095814+0000", 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2299999999999996e-06 2024-09-02T10:13:51.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089305+0000", 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99349078599999996 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089336+0000", 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.146e-05 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:103721 4.e e384/340 4:78ce6951:::benchmark_data_smithi187_76542_object11385:head v 384'6547, mlcod=384'6544)", 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 64.597953368999995, 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.265184527, 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391794+0000", 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4000000000000001e-07 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391799+0000", 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2009999999999998e-06 2024-09-02T10:13:51.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654392+0000", 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26259298599999997 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654398+0000", 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7880000000000003e-06 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654446+0000", 2024-09-02T10:13:51.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8183000000000003e-05 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654470+0000", 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4093000000000001e-05 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.656943+0000", 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024726489999999999 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.656978+0000", 2024-09-02T10:13:51.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4887e-05 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103725 4.19 4.a40a4399 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 64.553258690000007, 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65288606400000004, 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.060 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103725 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436489+0000", 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2750000000000001e-06 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436494+0000", 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4660000000000001e-06 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436506+0000", 2024-09-02T10:13:51.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2123000000000001e-05 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436512+0000", 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6749999999999996e-06 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436566+0000", 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3285000000000002e-05 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436572+0000", 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6019999999999996e-06 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089367+0000", 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65279432500000001 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089374+0000", 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3130000000000001e-06 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.064 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103743 4.19 4.222eb379 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 64.320418880999995, 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42007966200000002, 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103743 2024-09-02T10:13:51.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669329+0000", 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.221e-06 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669334+0000", 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1340000000000003e-06 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669346+0000", 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2155e-05 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669351+0000", 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9810000000000003e-06 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669433+0000", 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1617000000000001e-05 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669445+0000", 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.207e-05 2024-09-02T10:13:51.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089400+0000", 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41995552600000002 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089407+0000", 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9580000000000001e-06 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103767 4.19 4.bc594c79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 64.219964504000004, 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31965745400000001, 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103767 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769785+0000", 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5299999999999999e-06 2024-09-02T10:13:51.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769796+0000", 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.164e-05 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769823+0000", 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7055000000000001e-05 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769828+0000", 2024-09-02T10:13:51.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0350000000000001e-06 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769891+0000", 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2973999999999997e-05 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769903+0000", 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1569999999999999e-05 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089432+0000", 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31952966900000002 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089439+0000", 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9809999999999997e-06 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:108214 4.19 4:9c55397a:::benchmark_data_smithi187_76542_object15878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:52.632759+0000", 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 58.356987701999998, 2024-09-02T10:13:51.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15162389700000001, 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 108214 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.632759+0000", 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.632759+0000", 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.632760+0000", 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2249999999999999e-06 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.632766+0000", 2024-09-02T10:13:51.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0519999999999998e-06 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.632774+0000", 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2810000000000008e-06 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.632780+0000", 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.356e-06 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.632838+0000", 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7969e-05 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.632993+0000", 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015523100000000001 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.633113+0000", 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120429 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.635165+0000", 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020517550000000002 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.704376+0000", 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.069210674 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.783256+0000", 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.078880659000000006 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.783318+0000", 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1919000000000001e-05 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:52.784383+0000", 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001064347 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.080 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPushReply(4.19 393/385 [PushReplyOp(4:9d13bd2c:::benchmark_data_smithi187_76542_object44187:head)])", 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:19.725374+0000", 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 31.264372874999999, 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.205654219, 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.725374+0000", 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.725374+0000", 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.725374+0000", 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.06e-07 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.725378+0000", 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8929999999999998e-06 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.929578+0000", 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.204199979 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.929582+0000", 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.481e-06 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.929627+0000", 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4759000000000003e-05 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.931028+0000", 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001400601 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.084 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:130549 4.17 e394/392 4:e9e2ab89:::benchmark_data_smithi187_76542_object38213:head v 394'4984, mlcod=394'4981)", 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:20.417677+0000", 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 30.572069455000001, 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14944665700000001, 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.417677+0000", 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.417677+0000", 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.417678+0000", 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4600000000000003e-07 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.417683+0000", 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3380000000000004e-06 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.565485+0000", 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14780211200000001 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.565489+0000", 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0080000000000004e-06 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.565537+0000", 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8078999999999998e-05 2024-09-02T10:13:51.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.565559+0000", 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1690000000000001e-05 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.567089+0000", 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001530004 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.567124+0000", 2024-09-02T10:13:51.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4579999999999998e-05 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPushReply(4.19 402/385 [PushReplyOp(4:9eb94d79:::benchmark_data_smithi187_76542_object73210:head)])", 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:30.224750+0000", 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 20.764996473, 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.193412206, 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.089 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.224750+0000", 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.224750+0000", 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.224751+0000", 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2799999999999996e-07 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.224754+0000", 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2200000000000001e-06 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.417127+0000", 2024-09-02T10:13:51.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19237320899999999 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.417134+0000", 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8809999999999998e-06 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.417186+0000", 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2537000000000002e-05 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.418162+0000", 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00097573099999999997 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:154071 4.a e410/338 4:531ff552:::benchmark_data_smithi187_76542_object61735:head v 410'9496, mlcod=410'9492)", 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:47.947177+0000", 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 3.042569903, 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20590476799999999, 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:13:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.947177+0000", 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.947177+0000", 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.947177+0000", 2024-09-02T10:13:51.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9100000000000004e-07 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.947182+0000", 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.724e-06 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.149287+0000", 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20210461299999999 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:13:51.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.149293+0000", 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2400000000000004e-06 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.149369+0000", 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6324999999999999e-05 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.149392+0000", 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.285e-05 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.153036+0000", 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036444310000000001 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.153081+0000", 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4994000000000003e-05 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:13:51.099 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:13:51.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:51.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:51.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:51.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:51.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:51.258 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:51.259 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:51.312 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:51.316 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:51.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:51.385 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:51.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:51.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:51.511 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:51.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:51.512 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:51.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:51.634 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:51.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:51.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:51.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:51.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:51.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:51.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:51.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:51.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:52.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:52.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:52.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:52.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:52.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:52.153 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:52.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:52.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:52.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:52.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:52.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:52.422 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:52.423 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:13:52.423 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:13:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:52.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:52.447 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:13:52.447 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:13:52.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:13:52.479 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:13:52.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:13:52.515 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:13:52.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:52 smithi151 ceph-mon[16639]: pgmap v977: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.8 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 1073/90606 objects degraded (1.184%); 1888/90606 objects misplaced (2.084%); 2.4 MiB/s, 97 objects/s recovering 2024-09-02T10:13:52.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:52.546 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:13:52.546 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:13:52.553 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:52.554 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:52.575 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:13:52.576 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:13:52.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:52 smithi162 ceph-mon[20279]: pgmap v977: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.8 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 1073/90606 objects degraded (1.184%); 1888/90606 objects misplaced (2.084%); 2.4 MiB/s, 97 objects/s recovering 2024-09-02T10:13:52.596 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:52 smithi187 ceph-mon[20072]: pgmap v977: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.8 GiB data, 8.9 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 738 op/s; 1073/90606 objects degraded (1.184%); 1888/90606 objects misplaced (2.084%); 2.4 MiB/s, 97 objects/s recovering 2024-09-02T10:13:52.598 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:13:52.608 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:13:52.608 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:13:52.647 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:13:52.647 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:13:52.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:52.678 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:13:52.678 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:13:52.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:52.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:52.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:52.713 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:13:52.713 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:13:52.743 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:13:52.743 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:13:52.774 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:13:52.775 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:13:52.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:52.821 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:52.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:52.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:52.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:52.946 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:52.952 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:52.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:52.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:53.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:53.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:53.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:53.184 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:53.210 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:53.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:53.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:13:53.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.11 reset_purged_snaps_last 2024-09-02T10:13:53.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:53.337 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:53.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:53.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:53.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:53.464 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:53.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:53.504 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:53.504 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:53.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:53.589 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:53.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:53.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:53.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2349101611' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:53.657 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2349101611' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:53.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:53.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:53.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:53.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2349101611' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:13:53.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:53.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:53.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:53.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:53.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:53.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:53.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:53.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:54.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:54.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:54.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:54.140 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:54.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:13:54.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:54.248 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:54.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:54.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:54.375 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:54.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:54.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:54.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:54 smithi151 ceph-mon[16639]: pgmap v978: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.8 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 1175/86226 objects degraded (1.363%); 1581/86226 objects misplaced (1.834%); 3.7 MiB/s, 107 objects/s recovering 2024-09-02T10:13:54.417 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:54 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1175/86226 objects degraded (1.363%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:54.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:54 smithi187 ceph-mon[20072]: pgmap v978: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.8 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 1175/86226 objects degraded (1.363%); 1581/86226 objects misplaced (1.834%); 3.7 MiB/s, 107 objects/s recovering 2024-09-02T10:13:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:54 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1175/86226 objects degraded (1.363%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:54.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:54.508 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:54.558 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:54 smithi162 ceph-mon[20279]: pgmap v978: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.8 GiB data, 8.6 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 806 op/s; 1175/86226 objects degraded (1.363%); 1581/86226 objects misplaced (1.834%); 3.7 MiB/s, 107 objects/s recovering 2024-09-02T10:13:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:54 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1175/86226 objects degraded (1.363%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:54.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:54.638 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:54.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:54.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:54.763 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:54.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:54.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:54.767 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:54.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:54.888 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:54.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:54.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:55.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:55.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:55.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:55.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:55.145 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:55.146 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:55.163 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:55.163 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:55.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:55.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:55.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:55.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:55.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:55.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:55.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:55.421 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:13:55.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:13:55.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:55.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:13:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:55 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4192447156' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:13:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:55 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:13:55.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:13:55.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:55.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:13:55.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:55 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4192447156' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:13:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:55 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:13:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:55 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4192447156' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:13:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:55 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:13:55.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:55.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:55.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:55.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:55.928 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:55.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:55.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:56.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:56.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:56.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:56.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:13:56.165 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:56.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:13:56.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:56.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:56.198 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:56.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:56.320 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:56.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:56.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:56.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:56.443 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:13:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:56.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:56.462 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:56.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:13:56.588 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:56.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:13:56.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:56 smithi162 ceph-mon[20279]: pgmap v979: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.6 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 996 op/s; 1175/76554 objects degraded (1.535%); 1581/76554 objects misplaced (2.065%); 3.2 MiB/s, 52 objects/s recovering 2024-09-02T10:13:56.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:56 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:56 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:13:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:56 smithi162 ceph-mon[20279]: osdmap e412: 12 total, 12 up, 11 in 2024-09-02T10:13:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:56 smithi162 ceph-mon[20279]: 4.5 scrub starts 2024-09-02T10:13:56.613 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:56 smithi187 ceph-mon[20072]: pgmap v979: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.6 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 996 op/s; 1175/76554 objects degraded (1.535%); 1581/76554 objects misplaced (2.065%); 3.2 MiB/s, 52 objects/s recovering 2024-09-02T10:13:56.613 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:56 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:56.613 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:56 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:13:56.613 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:56 smithi187 ceph-mon[20072]: osdmap e412: 12 total, 12 up, 11 in 2024-09-02T10:13:56.613 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:56 smithi187 ceph-mon[20072]: 4.5 scrub starts 2024-09-02T10:13:56.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:56 smithi151 ceph-mon[16639]: pgmap v979: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.6 GiB data, 8.2 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 996 op/s; 1175/76554 objects degraded (1.535%); 1581/76554 objects misplaced (2.065%); 3.2 MiB/s, 52 objects/s recovering 2024-09-02T10:13:56.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:56 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:13:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:56 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:13:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:56 smithi151 ceph-mon[16639]: osdmap e412: 12 total, 12 up, 11 in 2024-09-02T10:13:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:56 smithi151 ceph-mon[16639]: 4.5 scrub starts 2024-09-02T10:13:56.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:56.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:56.721 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:56.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:56.846 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:56.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:56.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:56.971 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:56.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:56.983 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:13:56.983 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:13:57.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:13:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:57.095 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:57.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:13:57.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:57.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:57.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:57.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:57.352 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:57.353 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:57.454 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:57.485 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:57.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:57.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:57.615 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:57.616 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:57.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:57 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:13:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:57 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:13:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:57 smithi151 ceph-mon[16639]: 4.5 scrub ok 2024-09-02T10:13:57.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:57.738 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:57.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:57.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:57 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:13:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:57 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:13:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:57 smithi187 ceph-mon[20072]: 4.5 scrub ok 2024-09-02T10:13:57.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:57.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:57 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:13:57.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:57 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:13:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:57 smithi162 ceph-mon[20279]: 4.5 scrub ok 2024-09-02T10:13:57.861 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:57.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:57.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:57.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:57.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:58.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:58.111 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:58.112 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:58.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:13:58.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:58.236 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:13:58.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:58.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:58.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:58.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:13:58.483 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:58.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:58.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:13:58.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:58.612 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:58.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:58.613 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:58 smithi151 ceph-mon[16639]: pgmap v981: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.5 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 964 op/s; 1175/72549 objects degraded (1.620%); 1581/72549 objects misplaced (2.179%); 1.4 MiB/s, 22 objects/s recovering 2024-09-02T10:13:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:58 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:13:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:58 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:13:58.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:58.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:58.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:58 smithi187 ceph-mon[20072]: pgmap v981: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.5 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 964 op/s; 1175/72549 objects degraded (1.620%); 1581/72549 objects misplaced (2.179%); 1.4 MiB/s, 22 objects/s recovering 2024-09-02T10:13:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:58 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:13:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:58 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:13:58.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.10 reset_purged_snaps_last 2024-09-02T10:13:58.798 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:58.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:58.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:58 smithi162 ceph-mon[20279]: pgmap v981: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.5 GiB data, 7.8 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 964 op/s; 1175/72549 objects degraded (1.620%); 1581/72549 objects misplaced (2.179%); 1.4 MiB/s, 22 objects/s recovering 2024-09-02T10:13:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:58 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:13:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:58 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:13:58.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:58.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:58.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:13:59.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:59.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:59.002 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:13:59.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:59.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:59.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:59.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:13:59.261 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:59.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:59.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:13:59.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:59.387 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:59.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:59.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:59.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:13:59.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1175/72549 objects degraded (1.620%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:59.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:59 smithi151 ceph-mon[16639]: osdmap e413: 12 total, 12 up, 11 in 2024-09-02T10:13:59.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:59 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:13:59.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:13:59 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:13:59.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:13:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:59.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:13:59.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:13:59.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:59.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:13:59.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:13:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1175/72549 objects degraded (1.620%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:59 smithi187 ceph-mon[20072]: osdmap e413: 12 total, 12 up, 11 in 2024-09-02T10:13:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:59 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:13:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:13:59 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:13:59.770 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:13:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:59.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:13:59.834 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:13:59.835 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:13:59.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1175/72549 objects degraded (1.620%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:13:59.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:59 smithi162 ceph-mon[20279]: osdmap e413: 12 total, 12 up, 11 in 2024-09-02T10:13:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:59 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:13:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:13:59 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:13:59.848 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:13:59.860 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:13:59.860 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:13:59.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:13:59.890 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:13:59.890 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:13:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:13:59.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:13:59.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:13:59.925 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:13:59.926 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:13:59.953 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:13:59.953 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:13:59.986 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:13:59.986 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:14:00.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:00.017 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:14:00.018 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:14:00.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:00.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:00.048 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:14:00.048 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:14:00.075 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:14:00.075 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:14:00.108 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:14:00.109 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:14:00.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:00.141 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:14:00.141 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:14:00.161 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:00.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:00.169 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:14:00.169 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:14:00.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:00.290 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:00.291 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:00.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:00.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:00.419 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:00.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:00.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:00.545 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:00.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:00.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:00.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:00 smithi151 ceph-mon[16639]: pgmap v983: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.3 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 1282/64155 objects degraded (1.998%); 1261/64155 objects misplaced (1.966%); 3.3 MiB/s, 52 objects/s recovering 2024-09-02T10:14:00.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:00 smithi151 ceph-mon[16639]: 4.e scrub starts 2024-09-02T10:14:00.635 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3635627127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:00.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:00.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:00.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:00.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:00 smithi187 ceph-mon[20072]: pgmap v983: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.3 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 1282/64155 objects degraded (1.998%); 1261/64155 objects misplaced (1.966%); 3.3 MiB/s, 52 objects/s recovering 2024-09-02T10:14:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:00 smithi187 ceph-mon[20072]: 4.e scrub starts 2024-09-02T10:14:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3635627127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:00.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:00.795 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:00.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:00 smithi162 ceph-mon[20279]: pgmap v983: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.3 GiB data, 7.4 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.10k op/s; 1282/64155 objects degraded (1.998%); 1261/64155 objects misplaced (1.966%); 3.3 MiB/s, 52 objects/s recovering 2024-09-02T10:14:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:00 smithi162 ceph-mon[20279]: 4.e scrub starts 2024-09-02T10:14:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3635627127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:00.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:00.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:00.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:01.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:01.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:01.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:01.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:01.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:01.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:01.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:01.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:01.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:14:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:01.305 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:01.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:01.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:01.428 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:01.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:01.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:01.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:01.558 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:01.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:01.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:01.687 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:01.688 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:01.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:01 smithi151 ceph-mon[16639]: 4.a scrub starts 2024-09-02T10:14:01.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:01 smithi151 ceph-mon[16639]: 4.e scrub ok 2024-09-02T10:14:01.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:01 smithi151 ceph-mon[16639]: 4.a scrub ok 2024-09-02T10:14:01.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:01 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:14:01.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:01 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:14:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:01 smithi187 ceph-mon[20072]: 4.a scrub starts 2024-09-02T10:14:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:01 smithi187 ceph-mon[20072]: 4.e scrub ok 2024-09-02T10:14:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:01 smithi187 ceph-mon[20072]: 4.a scrub ok 2024-09-02T10:14:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:01 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:14:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:01 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:14:01.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:01.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:01.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:01.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:01 smithi162 ceph-mon[20279]: 4.a scrub starts 2024-09-02T10:14:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:01 smithi162 ceph-mon[20279]: 4.e scrub ok 2024-09-02T10:14:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:01 smithi162 ceph-mon[20279]: 4.a scrub ok 2024-09-02T10:14:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:01 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:14:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:01 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:14:01.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:01.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:01.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:01.948 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:02.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:02.074 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:02.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:02.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:02.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:02.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:02.206 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:02.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:14:02.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:02.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:02.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:02.456 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:02.477 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:02.479 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:02 smithi187 ceph-mon[20072]: pgmap v984: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.3 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 1282/61611 objects degraded (2.081%); 1261/61611 objects misplaced (2.047%); 1.7 MiB/s, 26 objects/s recovering 2024-09-02T10:14:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:02 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:14:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:02 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:14:02.579 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:02.608 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:02.610 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:02 smithi151 ceph-mon[16639]: pgmap v984: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.3 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 1282/61611 objects degraded (2.081%); 1261/61611 objects misplaced (2.047%); 1.7 MiB/s, 26 objects/s recovering 2024-09-02T10:14:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:02 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:14:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:02 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:14:02.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:02.736 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:02.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:02.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:02.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:02.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:02 smithi162 ceph-mon[20279]: pgmap v984: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.3 GiB data, 7.2 GiB used, 976 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 1282/61611 objects degraded (2.081%); 1261/61611 objects misplaced (2.047%); 1.7 MiB/s, 26 objects/s recovering 2024-09-02T10:14:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:02 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:14:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:02 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:14:02.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:02.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:02.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:02.971 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:02.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:03.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:03.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:03.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:03.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:03.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:03.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:03.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:03.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:03.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:03.480 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:03.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:03.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:03.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:03.637 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:03.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:03.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:03.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:03.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:03.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:03.894 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:03.894 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:03.901 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:03.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:04.020 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:04.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:04.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:04.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:04.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:04.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:04.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:04.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:04.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:04.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:04.417 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:04.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:04.492 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:04 smithi187 ceph-mon[20072]: 4.7 scrub starts 2024-09-02T10:14:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:04 smithi187 ceph-mon[20072]: 4.7 scrub ok 2024-09-02T10:14:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:04 smithi187 ceph-mon[20072]: pgmap v985: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.2 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 772 op/s; 1286/58002 objects degraded (2.217%); 980/58002 objects misplaced (1.690%); 3.8 MiB/s, 61 objects/s recovering 2024-09-02T10:14:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:04 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:14:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1286/58002 objects degraded (2.217%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:04.519 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:04.540 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:04.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:04.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:04.617 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:04 smithi151 ceph-mon[16639]: 4.7 scrub starts 2024-09-02T10:14:04.617 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:04 smithi151 ceph-mon[16639]: 4.7 scrub ok 2024-09-02T10:14:04.617 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:04 smithi151 ceph-mon[16639]: pgmap v985: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.2 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 772 op/s; 1286/58002 objects degraded (2.217%); 980/58002 objects misplaced (1.690%); 3.8 MiB/s, 61 objects/s recovering 2024-09-02T10:14:04.617 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:04 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:14:04.617 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1286/58002 objects degraded (2.217%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:04.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:04.666 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:04.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:04.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:04.796 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:04.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:04 smithi162 ceph-mon[20279]: 4.7 scrub starts 2024-09-02T10:14:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:04 smithi162 ceph-mon[20279]: 4.7 scrub ok 2024-09-02T10:14:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:04 smithi162 ceph-mon[20279]: pgmap v985: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.2 GiB data, 6.9 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 772 op/s; 1286/58002 objects degraded (2.217%); 980/58002 objects misplaced (1.690%); 3.8 MiB/s, 61 objects/s recovering 2024-09-02T10:14:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:04 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:14:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1286/58002 objects degraded (2.217%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:04.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:04.929 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:04.930 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:05.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:05.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:05.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:05.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:05.197 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:05.198 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:05.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:05.324 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:05.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:05.370 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:14:05.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:05.449 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:05.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:05.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:170576 4.b 4:d1890c03:::benchmark_data_smithi187_76542_object78240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:05.471033+0000", 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.030025677000000001, 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.030048376000000002, 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:14:05.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170576 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.471033+0000", 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:14:05.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.471033+0000", 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.471035+0000", 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0449999999999999e-06 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.471042+0000", 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.516e-06 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.471059+0000", 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7232e-05 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.471069+0000", 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6379999999999999e-06 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.476264+0000", 2024-09-02T10:14:05.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051951949999999997 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.476397+0000", 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013293199999999999 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,9", 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.476543+0000", 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014664300000000001 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:14:05.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.490513+0000", 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013969709 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.492205+0000", 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016921130000000001 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:170587 4.b 4:d939612a:::benchmark_data_smithi187_76542_object78251:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e414)", 2024-09-02T10:14:05.508 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:05.480901+0000", 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.020158220000000001, 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020231153000000002, 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 170587 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:14:05.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.480901+0000", 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.480901+0000", 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.480902+0000", 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1650000000000001e-06 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.480907+0000", 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9640000000000002e-06 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.480917+0000", 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7950000000000002e-06 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.480921+0000", 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7280000000000003e-06 2024-09-02T10:14:05.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.494219+0000", 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013297365 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.494356+0000", 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013738300000000001 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,9", 2024-09-02T10:14:05.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.494508+0000", 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015193100000000001 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(4.b 414/381 [PushReplyOp(4:d50cb3ba:::benchmark_data_smithi187_76542_object78727:head)])", 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:05.494392+0000", 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0066670999999999996, 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0067751629999999998, 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:14:05.513 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.494392+0000", 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.494392+0000", 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.494393+0000", 2024-09-02T10:14:05.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0249999999999999e-06 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.494400+0000", 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3309999999999998e-06 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.494433+0000", 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2453999999999998e-05 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.494442+0000", 2024-09-02T10:14:05.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1929999999999999e-06 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:170576 4.b e414/381)", 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:05.498751+0000", 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0023080959999999999, 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024375180000000001, 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:14:05.516 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.498751+0000", 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.498751+0000", 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.498752+0000", 2024-09-02T10:14:05.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7000000000000003e-07 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.498757+0000", 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8749999999999999e-06 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.498788+0000", 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1189000000000002e-05 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.498795+0000", 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.832e-06 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:170587 4.b e414/381)", 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:05.499971+0000", 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0010880270000000001, 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012394350000000001, 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:14:05.519 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.499971+0000", 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.499971+0000", 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.499972+0000", 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4200000000000004e-07 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.499978+0000", 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6829999999999996e-06 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.500009+0000", 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1702999999999998e-05 2024-09-02T10:14:05.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:05.500017+0000", 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9689999999999994e-06 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 5 2024-09-02T10:14:05.522 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:14:05.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:05.578 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:05.580 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:05 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:14:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2625992050' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:05.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:05.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:05 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:14:05.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2625992050' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:05.702 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:05.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:05 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:14:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2625992050' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:05.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:05.835 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:05.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:05.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:05.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:14:05.938 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:05.962 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:05.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:06.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:06.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:06.090 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:06.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:06.216 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:06.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:06.218 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:06.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:06.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:06.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:06.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:06 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:14:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:06 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:14:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:06 smithi151 ceph-mon[16639]: pgmap v986: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.0 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 912 op/s; 1286/49581 objects degraded (2.594%); 980/49581 objects misplaced (1.977%); 3.1 MiB/s, 49 objects/s recovering 2024-09-02T10:14:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:06 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:14:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2625992050' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:06 smithi151 ceph-mon[16639]: osdmap e414: 12 total, 12 up, 11 in 2024-09-02T10:14:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:14:06.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:14:06.452 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:06.476 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:06.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:06.477 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:06.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:06.606 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:06.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:06.608 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:06.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:06.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:06.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:06.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:06 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:14:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:06 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:14:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:06 smithi187 ceph-mon[20072]: pgmap v986: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.0 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 912 op/s; 1286/49581 objects degraded (2.594%); 980/49581 objects misplaced (1.977%); 3.1 MiB/s, 49 objects/s recovering 2024-09-02T10:14:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:06 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:14:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2625992050' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:06 smithi187 ceph-mon[20072]: osdmap e414: 12 total, 12 up, 11 in 2024-09-02T10:14:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:14:06.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:14:06.839 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:06 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:14:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:06 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:14:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:06 smithi162 ceph-mon[20279]: pgmap v986: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 1.0 GiB data, 6.5 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 912 op/s; 1286/49581 objects degraded (2.594%); 980/49581 objects misplaced (1.977%); 3.1 MiB/s, 49 objects/s recovering 2024-09-02T10:14:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:06 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:14:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2625992050' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:06 smithi162 ceph-mon[20279]: osdmap e414: 12 total, 12 up, 11 in 2024-09-02T10:14:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:14:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:14:06.864 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:06.865 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:06.966 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:06.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:06.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:07.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:07.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:07.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:07.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:14:07.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:07.253 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:07.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:07.258 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:07.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:07.377 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:07.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:07.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:07.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:07.505 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:07.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:07.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:07.630 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:07.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:07 smithi151 ceph-mon[16639]: osdmap e415: 12 total, 12 up, 11 in 2024-09-02T10:14:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:07 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:14:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:07 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:14:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:07 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:14:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:07 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:14:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:07 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:14:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:07 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:14:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:07 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:14:07.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:07.755 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:07.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:07 smithi187 ceph-mon[20072]: osdmap e415: 12 total, 12 up, 11 in 2024-09-02T10:14:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:07 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:14:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:07 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:14:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:07 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:14:07.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:07 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:14:07.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:07 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:14:07.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:07 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:14:07.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:07 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:14:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:07 smithi162 ceph-mon[20279]: osdmap e415: 12 total, 12 up, 11 in 2024-09-02T10:14:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:07 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:14:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:07 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:14:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:07 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:14:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:07 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:14:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:07 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:14:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:07 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:14:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:07 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:14:07.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:07.885 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:07.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:07.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:14:07.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:14:07.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:14:07.927 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:14:07.960 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:14:07.960 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:14:07.970 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:07.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:07.994 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:14:07.994 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:14:08.012 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:08.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:08.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:08.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:14:08.022 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:14:08.054 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:14:08.054 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:14:08.089 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:14:08.089 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:14:08.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:08.123 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:14:08.123 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:14:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:08.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:08.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:08.153 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:14:08.153 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:14:08.182 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:14:08.182 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:14:08.210 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:14:08.233 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:14:08.242 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:14:08.243 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:14:08.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:08.270 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:08.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:08.286 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:14:08.286 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:14:08.287 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 3, 4, 5, 6, 7, 8, 9, 11, 10] 2024-09-02T10:14:08.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd out 4 2024-09-02T10:14:08.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:08.396 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:08.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:08.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:08.525 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:08.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:08.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:08.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:08 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:14:08.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:08 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:14:08.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:08 smithi151 ceph-mon[16639]: pgmap v989: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 948 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 778 op/s; 1286/45468 objects degraded (2.828%); 980/45468 objects misplaced (2.155%); 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:14:08.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:08 smithi151 ceph-mon[16639]: osdmap e416: 12 total, 12 up, 11 in 2024-09-02T10:14:08.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3165769525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:08.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:14:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:08.656 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:08.657 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:08 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:14:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:08 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:14:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:08 smithi187 ceph-mon[20072]: pgmap v989: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 948 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 778 op/s; 1286/45468 objects degraded (2.828%); 980/45468 objects misplaced (2.155%); 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:14:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:08 smithi187 ceph-mon[20072]: osdmap e416: 12 total, 12 up, 11 in 2024-09-02T10:14:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3165769525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:08.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:08.782 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:08.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:08.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:08 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:14:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:08 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:14:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:08 smithi162 ceph-mon[20279]: pgmap v989: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 948 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 0 B/s wr, 778 op/s; 1286/45468 objects degraded (2.828%); 980/45468 objects misplaced (2.155%); 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:14:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:08 smithi162 ceph-mon[20279]: osdmap e416: 12 total, 12 up, 11 in 2024-09-02T10:14:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3165769525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:08.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:08.907 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:08.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:09.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:09.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:09.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:09.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:09.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:09.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:09.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:09.170 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:14:09.170 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:14:09.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:09.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:09.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:09.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:09.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:09.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:09.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1286/45468 objects degraded (2.828%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:09.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:09 smithi151 ceph-mon[16639]: osdmap e417: 12 total, 12 up, 11 in 2024-09-02T10:14:09.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:09.553 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:09.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:09.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:09.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:09.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:09.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1286/45468 objects degraded (2.828%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:09 smithi187 ceph-mon[20072]: osdmap e417: 12 total, 12 up, 11 in 2024-09-02T10:14:09.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:09.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:09.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:09.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:09.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1286/45468 objects degraded (2.828%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:09 smithi162 ceph-mon[20279]: osdmap e417: 12 total, 12 up, 11 in 2024-09-02T10:14:09.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:14:09.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:09.933 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:09.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:09.966 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:09.978 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:10.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:10.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:10.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:10.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:10.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:10.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:10.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:10.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:10.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:10.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:10.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:10.453 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:10.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:10.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:10.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:10.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:10.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:10.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:10 smithi151 ceph-mon[16639]: 4.19 scrub starts 2024-09-02T10:14:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:10 smithi151 ceph-mon[16639]: 4.19 scrub ok 2024-09-02T10:14:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:10 smithi151 ceph-mon[16639]: pgmap v992: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 754 MiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s; 1292/36171 objects degraded (3.572%); 710/36171 objects misplaced (1.963%); 4.1 MiB/s, 65 objects/s recovering 2024-09-02T10:14:10.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:10.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:10 smithi187 ceph-mon[20072]: 4.19 scrub starts 2024-09-02T10:14:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:10 smithi187 ceph-mon[20072]: 4.19 scrub ok 2024-09-02T10:14:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:10 smithi187 ceph-mon[20072]: pgmap v992: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 754 MiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s; 1292/36171 objects degraded (3.572%); 710/36171 objects misplaced (1.963%); 4.1 MiB/s, 65 objects/s recovering 2024-09-02T10:14:10.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:10.841 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:10.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:10.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:10 smithi162 ceph-mon[20279]: 4.19 scrub starts 2024-09-02T10:14:10.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:10 smithi162 ceph-mon[20279]: 4.19 scrub ok 2024-09-02T10:14:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:10 smithi162 ceph-mon[20279]: pgmap v992: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 754 MiB data, 5.7 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s; 1292/36171 objects degraded (3.572%); 710/36171 objects misplaced (1.963%); 4.1 MiB/s, 65 objects/s recovering 2024-09-02T10:14:10.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:10.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:10.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:11.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:11.092 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:11.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:11.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:14:11.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:11.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:11.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:11.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:11.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:11.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:11.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:11.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:11.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:11.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:11.577 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:11.605 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:11.606 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:11.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:11 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:14:11.667 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:11 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:14:11.667 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:11 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:14:11.667 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:11 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:14:11.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:11.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:11.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:11 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:14:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:11 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:14:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:11 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:14:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:11 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:14:11.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:11 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:14:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:11 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:14:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:11 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:14:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:11 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:14:11.866 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:11.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:11.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:11.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:11.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:11.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:11.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:12.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:12.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:12.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:12.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:12.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:12.245 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:12.245 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:12.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:12.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:12.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:12.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:12.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:12.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:12.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:12.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:12.630 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:12.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:12.675 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:12 smithi151 ceph-mon[16639]: pgmap v993: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 696 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 913 op/s; 1292/33399 objects degraded (3.868%); 710/33399 objects misplaced (2.126%); 2.8 MiB/s, 44 objects/s recovering 2024-09-02T10:14:12.675 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:12 smithi151 ceph-mon[16639]: 4.1 scrub starts 2024-09-02T10:14:12.675 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:12 smithi151 ceph-mon[16639]: 4.1 scrub ok 2024-09-02T10:14:12.675 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:12 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:14:12.675 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:12 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:14:12.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:12 smithi187 ceph-mon[20072]: pgmap v993: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 696 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 913 op/s; 1292/33399 objects degraded (3.868%); 710/33399 objects misplaced (2.126%); 2.8 MiB/s, 44 objects/s recovering 2024-09-02T10:14:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:12 smithi187 ceph-mon[20072]: 4.1 scrub starts 2024-09-02T10:14:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:12 smithi187 ceph-mon[20072]: 4.1 scrub ok 2024-09-02T10:14:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:12 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:14:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:12 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:14:12.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:12.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:12.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:12 smithi162 ceph-mon[20279]: pgmap v993: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 696 MiB data, 5.5 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 913 op/s; 1292/33399 objects degraded (3.868%); 710/33399 objects misplaced (2.126%); 2.8 MiB/s, 44 objects/s recovering 2024-09-02T10:14:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:12 smithi162 ceph-mon[20279]: 4.1 scrub starts 2024-09-02T10:14:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:12 smithi162 ceph-mon[20279]: 4.1 scrub ok 2024-09-02T10:14:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:12 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:14:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:12 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:14:12.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:12.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:12.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:12.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:13.020 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:13.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:13.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:13.044 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:13.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:13.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:13.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:13.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:13.278 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:13.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:13.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:13.409 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:13.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:13.426 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:14:13.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:13.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:13.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:13.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:13.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:13.665 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:13.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:13.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:13 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:14:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:13 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:14:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:13 smithi151 ceph-mon[16639]: osdmap e418: 12 total, 12 up, 11 in 2024-09-02T10:14:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2159537379' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:14:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:13 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:14:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:13 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:14:13.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:13 smithi187 ceph-mon[20072]: osdmap e418: 12 total, 12 up, 11 in 2024-09-02T10:14:13.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2159537379' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:14:13.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:13.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:13.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:13 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:14:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:13 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:14:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:13 smithi162 ceph-mon[20279]: osdmap e418: 12 total, 12 up, 11 in 2024-09-02T10:14:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2159537379' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:14:13.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:13.917 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:13.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:14.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:14.042 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:14.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:14.144 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:14.165 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:14.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:14.166 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:14.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:13 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:14:13.895+0000 7fe3ffed4640 -1 mon.a@0(leader).osd e419 definitely_dead 0 2024-09-02T10:14:14.267 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:14.293 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:14.293 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:14.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:14.416 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:14.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:14.440 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.4. 2024-09-02T10:14:14.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:14.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:14.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:14.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:14.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:14.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:14.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:14.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:14.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:14 smithi151 ceph-mon[16639]: pgmap v995: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 611 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 898 op/s; 1372/29292 objects degraded (4.684%); 420/29292 objects misplaced (1.434%); 4.9 MiB/s, 78 objects/s recovering 2024-09-02T10:14:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:14 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2159537379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:14:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:14 smithi151 ceph-mon[16639]: osdmap e419: 12 total, 12 up, 11 in 2024-09-02T10:14:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3189516163' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-02T10:14:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:14 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-02T10:14:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:14 smithi151 ceph-mon[16639]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-02T10:14:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1372/29292 objects degraded (4.684%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:14 smithi187 ceph-mon[20072]: pgmap v995: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 611 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 898 op/s; 1372/29292 objects degraded (4.684%); 420/29292 objects misplaced (1.434%); 4.9 MiB/s, 78 objects/s recovering 2024-09-02T10:14:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:14 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2159537379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:14:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:14 smithi187 ceph-mon[20072]: osdmap e419: 12 total, 12 up, 11 in 2024-09-02T10:14:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3189516163' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-02T10:14:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:14 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-02T10:14:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:14 smithi187 ceph-mon[20072]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-02T10:14:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1372/29292 objects degraded (4.684%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:14.766 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:14.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:14.788 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:14 smithi162 ceph-mon[20279]: pgmap v995: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 611 MiB data, 5.2 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 898 op/s; 1372/29292 objects degraded (4.684%); 420/29292 objects misplaced (1.434%); 4.9 MiB/s, 78 objects/s recovering 2024-09-02T10:14:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:14 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2159537379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:14:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:14 smithi162 ceph-mon[20279]: osdmap e419: 12 total, 12 up, 11 in 2024-09-02T10:14:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3189516163' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-02T10:14:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:14 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd out", "ids": ["4"]} : dispatch 2024-09-02T10:14:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:14 smithi162 ceph-mon[20279]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-02T10:14:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1372/29292 objects degraded (4.684%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:14.889 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:14.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:14.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:14.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:15.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:15.036 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:15.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:15.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:15.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:15.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:15.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:15.261 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:15.282 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:15.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:15.283 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:15.290 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:14:15.290 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:14:15.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:15.405 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:15.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:15.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:15.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:15.528 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:15.529 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:15.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:15.654 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:15.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:15.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:15 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-02T10:14:15.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:15 smithi151 ceph-mon[16639]: osdmap e420: 12 total, 12 up, 10 in 2024-09-02T10:14:15.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:15 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-02T10:14:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:15 smithi187 ceph-mon[20072]: osdmap e420: 12 total, 12 up, 10 in 2024-09-02T10:14:15.777 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:15.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:15 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-02T10:14:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:15 smithi162 ceph-mon[20279]: osdmap e420: 12 total, 12 up, 10 in 2024-09-02T10:14:15.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:15.899 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:15.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:15.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:16.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:16.026 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:16.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:16.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:16.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:16.153 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:16.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:16.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:16.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:16.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:16.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:16.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:16.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:16.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:16.402 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:16.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:16.524 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:16.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:16.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:16.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:16.650 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:16.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:16 smithi151 ceph-mon[16639]: pgmap v998: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 424 MiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 880 op/s; 1372/20316 objects degraded (6.753%); 420/20316 objects misplaced (2.067%); 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:14:16.669 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:16 smithi151 ceph-mon[16639]: osdmap e421: 12 total, 12 up, 10 in 2024-09-02T10:14:16.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:14:16.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:14:16.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:16 smithi187 ceph-mon[20072]: pgmap v998: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 424 MiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 880 op/s; 1372/20316 objects degraded (6.753%); 420/20316 objects misplaced (2.067%); 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:14:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:16 smithi187 ceph-mon[20072]: osdmap e421: 12 total, 12 up, 10 in 2024-09-02T10:14:16.760 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:16.773 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:16.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:16.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:16 smithi162 ceph-mon[20279]: pgmap v998: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 424 MiB data, 4.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 880 op/s; 1372/20316 objects degraded (6.753%); 420/20316 objects misplaced (2.067%); 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:14:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:16 smithi162 ceph-mon[20279]: osdmap e421: 12 total, 12 up, 10 in 2024-09-02T10:14:16.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:16.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:16.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:17.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:17.024 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:17.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:17.126 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:17.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:17.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:17.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:17.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:17.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:17.341 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:14:17.341 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:14:17.368 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:17.369 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:17.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:17.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:17.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:17.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:17.459 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:17 smithi151 ceph-mon[16639]: 4.e starting backfill to osd.5 from (0'0,0'0] MAX to 419'11337 2024-09-02T10:14:17.460 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:17 smithi151 ceph-mon[16639]: osdmap e422: 12 total, 12 up, 10 in 2024-09-02T10:14:17.460 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/889402852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:17.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:17.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:17.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:17.570 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:14:17.570 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:14:17.601 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:14:17.601 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:14:17.629 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:14:17.630 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:14:17.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:17.662 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:14:17.663 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:14:17.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:17.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:17.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:17.697 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:14:17.697 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:14:17.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:14:17.731 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:14:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:17 smithi187 ceph-mon[20072]: 4.e starting backfill to osd.5 from (0'0,0'0] MAX to 419'11337 2024-09-02T10:14:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:17 smithi187 ceph-mon[20072]: osdmap e422: 12 total, 12 up, 10 in 2024-09-02T10:14:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/889402852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:17.764 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:14:17.764 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:14:17.791 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:14:17.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:14:17.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:17.819 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:14:17.819 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:14:17.821 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:17.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:17.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:17 smithi162 ceph-mon[20279]: 4.e starting backfill to osd.5 from (0'0,0'0] MAX to 419'11337 2024-09-02T10:14:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:17 smithi162 ceph-mon[20279]: osdmap e422: 12 total, 12 up, 10 in 2024-09-02T10:14:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/889402852' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:17.848 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:14:17.849 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:14:17.881 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:14:17.881 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:14:17.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:17.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:17.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:18.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:18.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:18.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:18.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:18.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:18.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:18.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:18.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:18.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:18.339 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:18.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:18.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:18.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:18.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:18.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:18.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:18.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:18.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:18.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:18.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:18.713 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:18.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:18 smithi187 ceph-mon[20072]: pgmap v1001: 57 pgs: 1 activating+remapped, 1 active+recovering+undersized+degraded+remapped, 55 active+clean; 372 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 953 op/s; 1372/17850 objects degraded (7.686%); 745/17850 objects misplaced (4.174%) 2024-09-02T10:14:18.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:18.838 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:18.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:18.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:18 smithi162 ceph-mon[20279]: pgmap v1001: 57 pgs: 1 activating+remapped, 1 active+recovering+undersized+degraded+remapped, 55 active+clean; 372 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 953 op/s; 1372/17850 objects degraded (7.686%); 745/17850 objects misplaced (4.174%) 2024-09-02T10:14:18.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:18 smithi151 ceph-mon[16639]: pgmap v1001: 57 pgs: 1 activating+remapped, 1 active+recovering+undersized+degraded+remapped, 55 active+clean; 372 MiB data, 4.2 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 953 op/s; 1372/17850 objects degraded (7.686%); 745/17850 objects misplaced (4.174%) 2024-09-02T10:14:18.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:18.962 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:18.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:18.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:19.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:19.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:19.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:19.109 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:180152 4.f 4:f22b14b5:::benchmark_data_smithi187_76542_object87816:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:17.982179+0000", 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0096214379999999995, 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0096419450000000007, 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:14:19.110 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 180152 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.982179+0000", 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.982179+0000", 2024-09-02T10:14:19.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.982180+0000", 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4729999999999999e-06 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.982185+0000", 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0289999999999999e-06 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.982192+0000", 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9680000000000006e-06 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.982198+0000", 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3619999999999997e-06 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.982236+0000", 2024-09-02T10:14:19.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.82e-05 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.982347+0000", 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111218 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.982459+0000", 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000111437 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:14:19.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.987322+0000", 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048630629999999999 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.989475+0000", 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021529790000000002 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:180156 4.18 4:197d2109:::benchmark_data_smithi187_76542_object87820:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-02T10:14:19.115 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:17.985062+0000", 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0067381760000000002, 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006806968, 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 180156 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:14:19.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.985062+0000", 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.985062+0000", 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.985063+0000", 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5099999999999998e-07 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.985070+0000", 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6019999999999996e-06 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.985091+0000", 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1257e-05 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.985098+0000", 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8719999999999996e-06 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.985207+0000", 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108954 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.985317+0000", 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110543 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,8", 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.985417+0000", 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9648999999999994e-05 2024-09-02T10:14:19.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.988538+0000", 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003121342 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:17.988552+0000", 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3845e-05 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:19.120 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:14:19.121 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 2 2024-09-02T10:14:19.121 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:14:19.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:19.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:19.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:19.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:19.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:19.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:19.335 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:19.335 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:19.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:19.438 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:14:19.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:19.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:19.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:19.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:19.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:19.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:19.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:19.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:19.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:19.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3110550455' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1372/17850 objects degraded (7.686%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:19 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3110550455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:19 smithi187 ceph-mon[20072]: osdmap e423: 12 total, 12 up, 10 in 2024-09-02T10:14:19.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3110550455' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:19.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1372/17850 objects degraded (7.686%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:19.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:19 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:19.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3110550455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:19.759 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:19 smithi151 ceph-mon[16639]: osdmap e423: 12 total, 12 up, 10 in 2024-09-02T10:14:19.806 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:19.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:19.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:19.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3110550455' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1372/17850 objects degraded (7.686%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:19 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3110550455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:19 smithi162 ceph-mon[20279]: osdmap e423: 12 total, 12 up, 10 in 2024-09-02T10:14:19.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:19.951 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:19.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:20.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:20.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:20.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:20.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:20.207 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:20.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:20.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:20.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:20.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:20.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:20.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:20.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:20.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:20.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:20.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:20.588 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:20.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:20.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:20.715 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:20.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:20 smithi187 ceph-mon[20072]: pgmap v1003: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+remapped, 55 active+clean; 284 MiB data, 4.1 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 460 op/s; 1385/13632 objects degraded (10.160%); 325/13632 objects misplaced (2.384%); 5.3 MiB/s, 85 objects/s recovering 2024-09-02T10:14:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:20 smithi187 ceph-mon[20072]: osdmap e424: 12 total, 12 up, 10 in 2024-09-02T10:14:20.817 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:20.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:20 smithi162 ceph-mon[20279]: pgmap v1003: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+remapped, 55 active+clean; 284 MiB data, 4.1 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 460 op/s; 1385/13632 objects degraded (10.160%); 325/13632 objects misplaced (2.384%); 5.3 MiB/s, 85 objects/s recovering 2024-09-02T10:14:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:20 smithi162 ceph-mon[20279]: osdmap e424: 12 total, 12 up, 10 in 2024-09-02T10:14:20.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:20.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:20 smithi151 ceph-mon[16639]: pgmap v1003: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+remapped, 55 active+clean; 284 MiB data, 4.1 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 460 op/s; 1385/13632 objects degraded (10.160%); 325/13632 objects misplaced (2.384%); 5.3 MiB/s, 85 objects/s recovering 2024-09-02T10:14:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:20 smithi151 ceph-mon[16639]: osdmap e424: 12 total, 12 up, 10 in 2024-09-02T10:14:20.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:20.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:20.986 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:21.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:14:21.022 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:14:21.023 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:14:21.023 INFO:tasks.thrashosds.thrasher:Reweighting osd 8 to 0.7323247257175832 2024-09-02T10:14:21.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 0.7323247257175832 2024-09-02T10:14:21.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:21.111 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:21.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:21.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:21.238 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:21.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:21.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:21.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:21.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:21.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:21.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:21.502 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:21.603 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:21.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:21.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:21.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:21.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:21.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:21.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:21.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:21.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:21.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:22.013 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:22.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:22.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:22.136 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:22.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:22.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:22.271 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:22.272 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:22.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:22.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:22.402 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:22.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:22.504 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:22.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:22.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:22.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:22.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:22.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:22.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:22.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:22 smithi151 ceph-mon[16639]: pgmap v1005: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+remapped, 55 active+clean; 266 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 432 op/s; 1385/12747 objects degraded (10.865%); 325/12747 objects misplaced (2.550%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:14:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:22 smithi187 ceph-mon[20072]: pgmap v1005: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+remapped, 55 active+clean; 266 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 432 op/s; 1385/12747 objects degraded (10.865%); 325/12747 objects misplaced (2.550%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:14:22.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:22.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:22.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:22.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:14:22.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:22 smithi162 ceph-mon[20279]: pgmap v1005: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 activating+remapped, 55 active+clean; 266 MiB data, 4.0 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 432 op/s; 1385/12747 objects degraded (10.865%); 325/12747 objects misplaced (2.550%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:14:22.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:22.905 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:22.916 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:22.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:22.927 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:23.018 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:23.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:23.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:23.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:23.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:23.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:23.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:23.303 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:23.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:23.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:23.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:23.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:23.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:23.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:23.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:23.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:23.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:23.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:23.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:23.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:23.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:23.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:23.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:23.945 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:14:23.945 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:14:23.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:23.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:23.948 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:23.965 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:14:23.966 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:14:23.971 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 0.732325 (bb79) 2024-09-02T10:14:23.990 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:14:23.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:14:24.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:14:24.015 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:14:24.040 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:14:24.040 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:14:24.049 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:24.060 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:14:24.060 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:14:24.070 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:24.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:24.081 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:14:24.081 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:14:24.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:14:24.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:14:24.124 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:14:24.124 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:14:24.147 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:14:24.147 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:14:24.170 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:14:24.170 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:14:24.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:24.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:24.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:24.196 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:14:24.196 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:14:24.204 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:24.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:24.317 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:24.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:24.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:24.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:14:24.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:24.441 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:24.442 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:24.537 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Removed 92333 objects 2024-09-02T10:14:24.537 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Clean up completed and total clean up time :110.021 2024-09-02T10:14:24.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:24.564 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:24.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:24.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:24.687 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:24.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:24 smithi187 ceph-mon[20072]: pgmap v1006: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 188 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 506 op/s; 1385/9024 objects degraded (15.348%); 6.1 MiB/s, 157 objects/s recovering 2024-09-02T10:14:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/379896626' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 0.7323247257175832} : dispatch 2024-09-02T10:14:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3366152669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1385/9024 objects degraded (15.348%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/379896626' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.7323247257175832}]': finished 2024-09-02T10:14:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:24 smithi187 ceph-mon[20072]: osdmap e425: 12 total, 12 up, 10 in 2024-09-02T10:14:24.785 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-02T10:14:24.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-02T10:14:24.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:24.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:24 smithi151 ceph-mon[16639]: pgmap v1006: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 188 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 506 op/s; 1385/9024 objects degraded (15.348%); 6.1 MiB/s, 157 objects/s recovering 2024-09-02T10:14:24.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/379896626' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 0.7323247257175832} : dispatch 2024-09-02T10:14:24.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3366152669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:24.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1385/9024 objects degraded (15.348%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:24.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/379896626' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.7323247257175832}]': finished 2024-09-02T10:14:24.809 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:24 smithi151 ceph-mon[16639]: osdmap e425: 12 total, 12 up, 10 in 2024-09-02T10:14:24.812 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:24.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:24.813 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:24 smithi162 ceph-mon[20279]: pgmap v1006: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 188 MiB data, 3.8 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 506 op/s; 1385/9024 objects degraded (15.348%); 6.1 MiB/s, 157 objects/s recovering 2024-09-02T10:14:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/379896626' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 0.7323247257175832} : dispatch 2024-09-02T10:14:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3366152669' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1385/9024 objects degraded (15.348%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/379896626' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 0.7323247257175832}]': finished 2024-09-02T10:14:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:24 smithi162 ceph-mon[20279]: osdmap e425: 12 total, 12 up, 10 in 2024-09-02T10:14:24.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:24.935 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:24.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:25.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:25.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:25.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:25.162 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:25.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:25.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:25.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:25.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:25.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:25.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:25.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:25.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:25.471 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:14:25.471 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:14:25.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:25.553 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:25.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:25.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:25.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:25.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:25.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:25.803 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:25.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:25.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:25.930 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:25.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:25.932 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:26.033 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:26.056 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:26.057 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:26.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:26.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:26.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:25 smithi187 ceph-mon[20072]: osdmap e426: 12 total, 12 up, 10 in 2024-09-02T10:14:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:25 smithi187 ceph-mon[20072]: pgmap v1009: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 62 MiB data, 3.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 593 op/s; 1385/2940 objects degraded (47.109%); 2.7 MiB/s, 110 objects/s recovering 2024-09-02T10:14:26.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:26.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:26.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:26.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:25 smithi162 ceph-mon[20279]: osdmap e426: 12 total, 12 up, 10 in 2024-09-02T10:14:26.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:25 smithi162 ceph-mon[20279]: pgmap v1009: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 62 MiB data, 3.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 593 op/s; 1385/2940 objects degraded (47.109%); 2.7 MiB/s, 110 objects/s recovering 2024-09-02T10:14:26.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:26.430 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:26.430 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:26.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:25 smithi151 ceph-mon[16639]: osdmap e426: 12 total, 12 up, 10 in 2024-09-02T10:14:26.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:25 smithi151 ceph-mon[16639]: pgmap v1009: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 62 MiB data, 3.5 GiB used, 890 GiB / 894 GiB avail; 0 B/s wr, 593 op/s; 1385/2940 objects degraded (47.109%); 2.7 MiB/s, 110 objects/s recovering 2024-09-02T10:14:26.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:26.553 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:26.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:26.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:26.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:26.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:26.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:26.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:26.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:26.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:26.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:26.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:26.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:27.028 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:27.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:27.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:27.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:27.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:27.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:26 smithi187 ceph-mon[20072]: osdmap e427: 12 total, 12 up, 10 in 2024-09-02T10:14:27.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:27.299 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:27.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:26 smithi162 ceph-mon[20279]: osdmap e427: 12 total, 12 up, 10 in 2024-09-02T10:14:27.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:27.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:27.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:27.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:26 smithi151 ceph-mon[16639]: osdmap e427: 12 total, 12 up, 10 in 2024-09-02T10:14:27.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:27.552 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:27.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:27.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:27.675 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:27.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:27.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:27.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:27.800 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:27.800 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:27.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:27.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:27.926 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:27.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:14:28.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:28.049 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:28.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:28.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:28.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:28.173 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:28.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:28.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:28.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:27 smithi151 ceph-mon[16639]: pgmap v1011: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 17 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 664 op/s; 1385/786 objects degraded (176.209%); 2.7 MiB/s, 110 objects/s recovering 2024-09-02T10:14:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:27 smithi187 ceph-mon[20072]: pgmap v1011: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 17 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 664 op/s; 1385/786 objects degraded (176.209%); 2.7 MiB/s, 110 objects/s recovering 2024-09-02T10:14:28.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:28.296 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:28.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:28.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:27 smithi162 ceph-mon[20279]: pgmap v1011: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 17 MiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 664 op/s; 1385/786 objects degraded (176.209%); 2.7 MiB/s, 110 objects/s recovering 2024-09-02T10:14:28.362 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:28.376 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:28.376 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:28.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:28.425 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:28.427 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:28.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:28.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:28.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:28.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:28.678 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:28.679 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:28.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:28.803 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:28.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:28.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:28.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:28.928 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:28.929 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:29.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:29.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:29.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:29.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:29.143 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:14:29.143 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:14:29.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:28 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1385/786 objects degraded (176.209%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:29.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:29.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:29.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:29.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:28 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1385/786 objects degraded (176.209%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:29.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:29.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:29.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:28 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1385/786 objects degraded (176.209%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:14:29.401 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:29.423 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:29.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:29.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:29.525 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:29.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:29.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:29.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:29.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:29.671 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:29.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:29.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:29.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:29.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:29.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:29.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:29.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:29.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:29.944 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:14:29.944 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:14:29.945 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-02T10:14:29.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:14:30.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:30.046 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:30.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:30.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:30.118 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:29 smithi151 ceph-mon[16639]: pgmap v1012: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 55 active+clean; 577 KiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 500 op/s; 576/6 objects degraded (9600.000%); 0 B/s, 195 objects/s recovering 2024-09-02T10:14:30.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:30.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:14:30.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:30.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:29 smithi187 ceph-mon[20072]: pgmap v1012: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 55 active+clean; 577 KiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 500 op/s; 576/6 objects degraded (9600.000%); 0 B/s, 195 objects/s recovering 2024-09-02T10:14:30.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:30.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:14:30.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:29 smithi162 ceph-mon[20279]: pgmap v1012: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 55 active+clean; 577 KiB data, 3.3 GiB used, 891 GiB / 894 GiB avail; 0 B/s wr, 500 op/s; 576/6 objects degraded (9600.000%); 0 B/s, 195 objects/s recovering 2024-09-02T10:14:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:30.355 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:30.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:30.456 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:30.477 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:30.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:30.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:30.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:30.586 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:30.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:30.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:30.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:30.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:30.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:30.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:30.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:30.834 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:30.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:30.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:30.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:30.868 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:30.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:30.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:30.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:30.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:31.001 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:14:31.002 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_2' removed 2024-09-02T10:14:31.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:31.103 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:31.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:31.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:31.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:31.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:31.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:31.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2816397239' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:14:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/12493119' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:14:31.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2816397239' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:14:31.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/12493119' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:14:31.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:31.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2816397239' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:14:31.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/12493119' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:14:31.350 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:31.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:31.452 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:31.473 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:31.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:31.492 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:14:31.492 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:14:31.492 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 428, 2024-09-02T10:14:31.492 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:14:31.492 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:14:31.492 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:14:30.994245+0000", 2024-09-02T10:14:31.492 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:14:14.425747+0000", 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5804032, 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "nodeep-scrub", 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:14:31.493 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 4, 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:14:31.494 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:14:31.495 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:14:31.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "264", 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "241", 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:14:31.497 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:14:31.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:14:31.499 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 3.1099998950958252, 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 3.1099998950958252, 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.82999998331069946, 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.5799999237060547, 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.5799999237060547, 2024-09-02T10:14:31.500 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:14:31.501 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 425, 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:14:31.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:14:31.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:14:31.504 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:14:31.505 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.506 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.569854736328125, 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:14:31.507 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 425, 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:14:31.508 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:14:31.509 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.510 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.511 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.512 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 349, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.513 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.514 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.515 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:14:31.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:14:31.517 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.518 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.6008758544921875, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 356, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.520 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.521 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.522 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.523 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:14:31.524 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 378, 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.525 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.526 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.527 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.528 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.529 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.529 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:14:31.529 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:14:31.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:14:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.531 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 384, 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:14:31.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:14:31.535 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:14:31.536 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.70245361328125, 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.537 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 420, 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.538 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.539 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.542 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 425, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.543 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:14:31.548 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.7323150634765625, 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 353, 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.551 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:14:31.553 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:14:31.554 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 426, 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:14:31.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:14:31.556 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:14:31.557 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.558 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:14:31.559 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.560 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 423, 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.561 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.562 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.565 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.4313507080078125, 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 425, 2024-09-02T10:14:31.566 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:14:31.568 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:14:31.569 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:14:31.570 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:14:31.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:13:20.446996+0000", 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.572 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:13:26.200346+0000", 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.573 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 65536, 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:14:05.529662+0000", 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:14:11.753416+0000", 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 10656, 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:13:29.410158+0000", 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.575 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:13:46.945389+0000", 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:14:31.576 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:13:27.202708+0000", 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:13:40.754703+0000", 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.577 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:13:22.761129+0000", 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.578 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:13:28.365180+0000", 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:14:07.120017+0000", 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:13:22.013872+0000", 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-09-02T10:14:31.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.1e", 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: 11, 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: 9 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [ 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.7", 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: "mappings": [ 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:14:31.581 INFO:teuthology.orchestra.run.smithi151.stdout: "from": 10, 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: "to": 11 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [], 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:14:31.582 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:14:31.583 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:14:31.584 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:14:31.585 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:14:31.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:14:31.585 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:14:31.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:31.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:31.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:31.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:31.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:31.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:31.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:31.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:31.853 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:31.854 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:31.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:31.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:31.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:32.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:32.102 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:32.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:32.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:32.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:32.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:32.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:32.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:31 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2816397239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:14:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/12493119' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:14:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:31 smithi187 ceph-mon[20072]: osdmap e428: 12 total, 12 up, 10 in 2024-09-02T10:14:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:31 smithi187 ceph-mon[20072]: pgmap v1014: 31 pgs: 31 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:32.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3463996082' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:14:32.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:31 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2816397239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:14:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/12493119' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:14:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:31 smithi162 ceph-mon[20279]: osdmap e428: 12 total, 12 up, 10 in 2024-09-02T10:14:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:31 smithi162 ceph-mon[20279]: pgmap v1014: 31 pgs: 31 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3463996082' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:14:32.349 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:32.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:32.350 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:31 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2816397239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:14:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/12493119' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:14:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:31 smithi151 ceph-mon[16639]: osdmap e428: 12 total, 12 up, 10 in 2024-09-02T10:14:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:31 smithi151 ceph-mon[16639]: pgmap v1014: 31 pgs: 31 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3463996082' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:14:32.451 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:32.474 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:32.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:32.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:32.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:32.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:32.603 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:32.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:32.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:32.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:32.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:32.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:32.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:32.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:32.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:32.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:32.976 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:32.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:33.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:33.102 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:33.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:33.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:33.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:33.230 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:33.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:32 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:14:33.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:32 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:14:33.252 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:32 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:14:33.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:32 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:14:33.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:32 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 576/6 objects degraded (9600.000%), 1 pg degraded, 1 pg undersized) 2024-09-02T10:14:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:32 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:14:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:33 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:14:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:33 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:14:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:33 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:14:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:33 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 576/6 objects degraded (9600.000%), 1 pg degraded, 1 pg undersized) 2024-09-02T10:14:33.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:32 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:14:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:32 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:14:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:32 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:14:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:32 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:14:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:32 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 576/6 objects degraded (9600.000%), 1 pg degraded, 1 pg undersized) 2024-09-02T10:14:33.356 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:33.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:33.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:33.481 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:33.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:33.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:33.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:33.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:33.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:33.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:33.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:33.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:33.740 INFO:tasks.thrashosds.thrasher:j is {'epoch': 428, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:14:30.994245+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:14:14.425747+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 4, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.109999895095825, 'score_stable': 3.109999895095825, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5799999237060547, 'raw_score_stable': 2.5799999237060547, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 425, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 0.569854736328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 425, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 349, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 0.6008758544921875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 378, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 384, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 0.70245361328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 425, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 0.7323150634765625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 353, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 426, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 423, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 1, 'weight': 0.4313507080078125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 425, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:13:20.446996+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:13:26.200346+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-02T10:14:05.529662+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:14:11.753416+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 10656, 'last_purged_snaps_scrub': '2024-09-02T10:13:29.410158+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:13:46.945389+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:13:27.202708+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:13:40.754703+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:13:22.761129+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:13:28.365180+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:14:07.120017+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:13:22.013872+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1e', 'osds': [11, 10, 9]}], 'pg_upmap_items': [{'pgid': '1.7', 'mappings': [{'from': 10, 'to': 11}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:14:33.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:14:33.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:33.857 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:33.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:33.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:33.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:33.971 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:14:33.972 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:14:33.985 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:33.986 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:33.997 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:14:33.997 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:14:34.018 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:14:34.018 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:14:34.038 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:14:34.039 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:14:34.061 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:14:34.061 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:14:34.082 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:14:34.083 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:14:34.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:34.104 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:14:34.104 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:14:34.113 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:34.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:34.126 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:14:34.126 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:14:34.147 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:14:34.147 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:14:34.168 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:14:34.169 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:14:34.181 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:34.193 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:14:34.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:14:34.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:34.217 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-02T10:14:34.218 INFO:tasks.radosbench:Beginning radosbench... 2024-09-02T10:14:34.218 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-02T10:14:34.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-02T10:14:34.220 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:14:34.220 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:14:34.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:33 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:14:34.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:33 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:14:34.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:33 smithi151 ceph-mon[16639]: pgmap v1015: 31 pgs: 31 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:34.229 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2571550386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:34.230 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:33 smithi151 ceph-mon[16639]: osdmap e429: 12 total, 12 up, 10 in 2024-09-02T10:14:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:34.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:33 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:14:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:33 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:14:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:33 smithi187 ceph-mon[20072]: pgmap v1015: 31 pgs: 31 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2571550386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:33 smithi187 ceph-mon[20072]: osdmap e429: 12 total, 12 up, 10 in 2024-09-02T10:14:34.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:33 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:14:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:33 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:14:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:33 smithi162 ceph-mon[20279]: pgmap v1015: 31 pgs: 31 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2571550386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:33 smithi162 ceph-mon[20279]: osdmap e429: 12 total, 12 up, 10 in 2024-09-02T10:14:34.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:34.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:34.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:34.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:34.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:34.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:34.619 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:34.620 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:34.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:34.747 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:34.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:34.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:34.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:34.876 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:34.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:34.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:35.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:35.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:35.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:35.129 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:35.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:35.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:35.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:35.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:35.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:35 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:14:35.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:35 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:14:35.276 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:35 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:14:35.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:35 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:14:35.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:35 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:14:35.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:35 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:14:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:35 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:14:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:35 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:14:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:35 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:14:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:35 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:14:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:35 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:14:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:35 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:14:35.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:35.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:35.382 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:35.393 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:14:35.393 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:14:35.393 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:14:35.393 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:14:35.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:35.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:35 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:14:35.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:35 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:14:35.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:35 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:14:35.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:35 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:14:35.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:35 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:14:35.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:35 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:14:35.509 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:35.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:35.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:35.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:35.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:35.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:35.737 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:35.763 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:35.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:35.764 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:35.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:35.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:35.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:35.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:36.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:36.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:36.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:36.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:36.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:36 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:14:36.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:36 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:14:36.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:36 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:14:36.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:36 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:14:36.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:36 smithi151 ceph-mon[16639]: osdmap e430: 12 total, 12 up, 10 in 2024-09-02T10:14:36.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:36 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:14:36.134 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:36 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:14:36.135 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:36 smithi151 ceph-mon[16639]: pgmap v1018: 31 pgs: 31 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:36.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:36.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:36.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:36.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:36.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:36.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:36.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:36 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:14:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:36 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:14:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:36 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:14:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:36 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:14:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:36 smithi162 ceph-mon[20279]: osdmap e430: 12 total, 12 up, 10 in 2024-09-02T10:14:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:36 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:14:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:36 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:14:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:36 smithi162 ceph-mon[20279]: pgmap v1018: 31 pgs: 31 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:36.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:36.396 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:36.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:36.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:36.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:36 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:14:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:36 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:14:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:36 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:14:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:36 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:14:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:36 smithi187 ceph-mon[20072]: osdmap e430: 12 total, 12 up, 10 in 2024-09-02T10:14:36.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:36 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:14:36.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:36 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:14:36.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:36 smithi187 ceph-mon[20072]: pgmap v1018: 31 pgs: 31 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:36.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:36.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:36.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:36.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:36.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:36.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:36.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:36.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:36.825 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:36.825 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:36.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:14:36.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:36.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:36.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:36.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:37.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:37.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:37.046 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:37.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:37.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:37 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:14:37.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:37 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:14:37.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:37 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:14:37.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:37 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:14:37.111 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:37 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:14:37.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:37 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:14:37.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:37 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:14:37.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:37 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:14:37.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:37.173 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:37.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:37.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:37.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:37.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:37.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:37.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:14:37.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:37 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:14:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:37 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:14:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:37 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:14:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:37 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:14:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:37 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:14:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:37 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:14:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:37 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:14:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:37 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:14:37.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:37.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:37.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:37.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:37 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:14:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:37 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:14:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:37 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:14:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:37 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:14:37.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:37 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:14:37.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:37 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:14:37.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:37 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:14:37.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:37 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:14:37.521 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:37.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:37.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:37.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:37.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:37.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:37.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:37.799 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:37.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:37.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:37.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:37.924 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:37.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:37.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:37.932 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:38.024 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_3' created 2024-09-02T10:14:38.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:38.054 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:38.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:38.056 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:14:38.057 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:14:38.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:38.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:38.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:38.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:14:38.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:38 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:14:38.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:38 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:14:38.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:38 smithi151 ceph-mon[16639]: pgmap v1019: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:38.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/523871362' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-02T10:14:38.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:38.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:38.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:38 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:14:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:38 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:14:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:38 smithi162 ceph-mon[20279]: pgmap v1019: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/523871362' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-02T10:14:38.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:38.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:38.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:38 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:14:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:38 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:14:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:38 smithi187 ceph-mon[20072]: pgmap v1019: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/523871362' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16} : dispatch 2024-09-02T10:14:38.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:38.553 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:38.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:38.554 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:38.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:38.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:38.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:38.690 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:14:38.690 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:14:38.690 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:14:38.690 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:14:38.690 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:14:38.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:38.806 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:38.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:38.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:38.934 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:38.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:39.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:39.065 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-02T10:14:39.063+0000 7f6bd6404640 0 --2- 172.21.15.151:0/983616608 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7f6bd0101a40 0x7f6bd0197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-02T10:14:39.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:39.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:39.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:39.093 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:14:39.093 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:14:39.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:14:39.114 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:14:39.187 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:39.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/523871362' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-02T10:14:39.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:39 smithi151 ceph-mon[16639]: osdmap e431: 12 total, 12 up, 10 in 2024-09-02T10:14:39.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:39 smithi151 ceph-mon[16639]: from='client.16368 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:14:39.214 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:39.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:39.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:39.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:39.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:39.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:39.341 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:14:39.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/523871362' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-02T10:14:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:39 smithi162 ceph-mon[20279]: osdmap e431: 12 total, 12 up, 10 in 2024-09-02T10:14:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:39 smithi162 ceph-mon[20279]: from='client.16368 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:14:39.361 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:14:39.362 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:14:39.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:14:39.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:14:39.404 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:14:39.404 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:14:39.427 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:14:39.427 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:14:39.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:39.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:14:39.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:14:39.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:39.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:39.471 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:14:39.471 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:14:39.493 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:14:39.494 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:14:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/523871362' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-02T10:14:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:39 smithi187 ceph-mon[20072]: osdmap e431: 12 total, 12 up, 10 in 2024-09-02T10:14:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:39 smithi187 ceph-mon[20072]: from='client.16368 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:14:39.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:14:39.515 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:14:39.536 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:14:39.537 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:14:39.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:39.590 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:39.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:39.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:39.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:39.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:39.720 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:39.757 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1020,"stamp":"2024-09-02T10:14:38.025063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":3392484,"kb_used_data":373412,"kb_used_omap":15,"kb_used_meta":3018864,"kb_avail":934059036,"statfs":{"total":959950356480,"available":956476452864,"internally_reserved":0,"allocated":382373888,"data_stored":378126445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3091317169},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.747597"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":186,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:36.326372+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:14:36.326372+0000","last_peered":"2024-09-02T10:14:36.326372+0000","last_clean":"2024-09-02T10:14:36.326372+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:14:36.326372+0000","last_undegraded":"2024-09-02T10:14:36.326372+0000","last_fullsized":"2024-09-02T10:14:36.326372+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:56.714204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":217,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293320+0000","last_change":"2024-09-02T10:14:33.239009+0000","last_active":"2024-09-02T10:14:35.293320+0000","last_peered":"2024-09-02T10:14:35.293320+0000","last_clean":"2024-09-02T10:14:35.293320+0000","last_became_active":"2024-09-02T10:12:47.212512+0000","last_became_peered":"2024-09-02T10:12:47.212512+0000","last_unstale":"2024-09-02T10:14:35.293320+0000","last_undegraded":"2024-09-02T10:14:35.293320+0000","last_fullsized":"2024-09-02T10:14:35.293320+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:33.238949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:33.238949+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:56.403772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,6],"acting":[5,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":434,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-02T10:14:33.975756+0000","last_change":"2024-09-02T10:13:59.249166+0000","last_active":"2024-09-02T10:14:33.975756+0000","last_peered":"2024-09-02T10:14:33.975756+0000","last_clean":"2024-09-02T10:14:33.975756+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:14:33.975756+0000","last_undegraded":"2024-09-02T10:14:33.975756+0000","last_fullsized":"2024-09-02T10:14:33.975756+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:59.249105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:59.249105+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:02.787049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":434,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-02T10:14:33.975796+0000","last_change":"2024-09-02T10:14:01.321873+0000","last_active":"2024-09-02T10:14:33.975796+0000","last_peered":"2024-09-02T10:14:33.975796+0000","last_clean":"2024-09-02T10:14:33.975796+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:14:33.975796+0000","last_undegraded":"2024-09-02T10:14:33.975796+0000","last_fullsized":"2024-09-02T10:14:33.975796+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:01.321818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:01.321818+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:18.881384+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":343,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.292380+0000","last_change":"2024-09-02T10:14:01.105137+0000","last_active":"2024-09-02T10:14:35.292380+0000","last_peered":"2024-09-02T10:14:35.292380+0000","last_clean":"2024-09-02T10:14:35.292380+0000","last_became_active":"2024-09-02T10:11:18.920488+0000","last_became_peered":"2024-09-02T10:11:18.920488+0000","last_unstale":"2024-09-02T10:14:35.292380+0000","last_undegraded":"2024-09-02T10:14:35.292380+0000","last_fullsized":"2024-09-02T10:14:35.292380+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:01.105083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:01.105083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:03.020721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":293,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293762+0000","last_change":"2024-09-02T10:14:11.689851+0000","last_active":"2024-09-02T10:14:35.293762+0000","last_peered":"2024-09-02T10:14:35.293762+0000","last_clean":"2024-09-02T10:14:35.293762+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:14:35.293762+0000","last_undegraded":"2024-09-02T10:14:35.293762+0000","last_fullsized":"2024-09-02T10:14:35.293762+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.689767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:19.591338+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":790,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293220+0000","last_change":"2024-09-02T10:12:47.098906+0000","last_active":"2024-09-02T10:14:35.293220+0000","last_peered":"2024-09-02T10:14:35.293220+0000","last_clean":"2024-09-02T10:14:35.293220+0000","last_became_active":"2024-09-02T10:12:47.091891+0000","last_became_peered":"2024-09-02T10:12:47.091891+0000","last_unstale":"2024-09-02T10:14:35.293220+0000","last_undegraded":"2024-09-02T10:14:35.293220+0000","last_fullsized":"2024-09-02T10:14:35.293220+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:10.602556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:10.602556+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":441,"reported_epoch":429,"state":"active+clean","last_fresh":"2024-09-02T10:14:33.976057+0000","last_change":"2024-09-02T10:13:33.498600+0000","last_active":"2024-09-02T10:14:33.976057+0000","last_peered":"2024-09-02T10:14:33.976057+0000","last_clean":"2024-09-02T10:14:33.976057+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:14:33.976057+0000","last_undegraded":"2024-09-02T10:14:33.976057+0000","last_fullsized":"2024-09-02T10:14:33.976057+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:33.498535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:33.498535+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:14:57.741728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":439,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-02T10:14:31.294024+0000","last_change":"2024-09-02T10:14:04.633772+0000","last_active":"2024-09-02T10:14:31.294024+0000","last_peered":"2024-09-02T10:14:31.294024+0000","last_clean":"2024-09-02T10:14:31.294024+0000","last_became_active":"2024-09-02T10:11:11.497246+0000","last_became_peered":"2024-09-02T10:11:11.497246+0000","last_unstale":"2024-09-02T10:14:31.294024+0000","last_undegraded":"2024-09-02T10:14:31.294024+0000","last_fullsized":"2024-09-02T10:14:31.294024+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:04.633686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:04.633686+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:28.648762+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":419,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.292414+0000","last_change":"2024-09-02T10:13:37.418287+0000","last_active":"2024-09-02T10:14:35.292414+0000","last_peered":"2024-09-02T10:14:35.292414+0000","last_clean":"2024-09-02T10:14:35.292414+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:14:35.292414+0000","last_undegraded":"2024-09-02T10:14:35.292414+0000","last_fullsized":"2024-09-02T10:14:35.292414+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:37.418216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:37.418216+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:14:37.813610+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":270,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.292363+0000","last_change":"2024-09-02T10:12:47.317126+0000","last_active":"2024-09-02T10:14:35.292363+0000","last_peered":"2024-09-02T10:14:35.292363+0000","last_clean":"2024-09-02T10:14:35.292363+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:14:35.292363+0000","last_undegraded":"2024-09-02T10:14:35.292363+0000","last_fullsized":"2024-09-02T10:14:35.292363+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:47.317082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:47.317082+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":772,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-02T10:14:31.292836+0000","last_change":"2024-09-02T10:13:55.581470+0000","last_active":"2024-09-02T10:14:31.292836+0000","last_peered":"2024-09-02T10:14:31.292836+0000","last_clean":"2024-09-02T10:14:31.292836+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:14:31.292836+0000","last_undegraded":"2024-09-02T10:14:31.292836+0000","last_fullsized":"2024-09-02T10:14:31.292836+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:55.581403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:55.581403+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:14:59.483473+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":712,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293400+0000","last_change":"2024-09-02T10:14:35.252590+0000","last_active":"2024-09-02T10:14:35.293400+0000","last_peered":"2024-09-02T10:14:35.293400+0000","last_clean":"2024-09-02T10:14:35.293400+0000","last_became_active":"2024-09-02T10:11:11.468355+0000","last_became_peered":"2024-09-02T10:11:11.468355+0000","last_unstale":"2024-09-02T10:14:35.293400+0000","last_undegraded":"2024-09-02T10:14:35.293400+0000","last_fullsized":"2024-09-02T10:14:35.293400+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.252526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.252526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:58.626048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":692,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.007407+0000","last_change":"2024-09-02T10:14:34.665490+0000","last_active":"2024-09-02T10:14:35.007407+0000","last_peered":"2024-09-02T10:14:35.007407+0000","last_clean":"2024-09-02T10:14:35.007407+0000","last_became_active":"2024-09-02T10:14:24.979372+0000","last_became_peered":"2024-09-02T10:14:24.979372+0000","last_unstale":"2024-09-02T10:14:35.007407+0000","last_undegraded":"2024-09-02T10:14:35.007407+0000","last_fullsized":"2024-09-02T10:14:35.007407+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.665438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.665438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:58.191772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1],"acting":[0,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":697,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293222+0000","last_change":"2024-09-02T10:14:34.217697+0000","last_active":"2024-09-02T10:14:35.293222+0000","last_peered":"2024-09-02T10:14:35.293222+0000","last_clean":"2024-09-02T10:14:35.293222+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:14:35.293222+0000","last_undegraded":"2024-09-02T10:14:35.293222+0000","last_fullsized":"2024-09-02T10:14:35.293222+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.217624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.217624+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:02.148154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":775,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293352+0000","last_change":"2024-09-02T10:12:47.092718+0000","last_active":"2024-09-02T10:14:35.293352+0000","last_peered":"2024-09-02T10:14:35.293352+0000","last_clean":"2024-09-02T10:14:35.293352+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:14:35.293352+0000","last_undegraded":"2024-09-02T10:14:35.293352+0000","last_fullsized":"2024-09-02T10:14:35.293352+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:09.559616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:09.559616+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":890,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.292322+0000","last_change":"2024-09-02T10:12:47.091705+0000","last_active":"2024-09-02T10:14:35.292322+0000","last_peered":"2024-09-02T10:14:35.292322+0000","last_clean":"2024-09-02T10:14:35.292322+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:14:35.292322+0000","last_undegraded":"2024-09-02T10:14:35.292322+0000","last_fullsized":"2024-09-02T10:14:35.292322+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:11:02.444317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:11:02.444317+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":541,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293525+0000","last_change":"2024-09-02T10:14:31.581643+0000","last_active":"2024-09-02T10:14:35.293525+0000","last_peered":"2024-09-02T10:14:35.293525+0000","last_clean":"2024-09-02T10:14:35.293525+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:14:35.293525+0000","last_undegraded":"2024-09-02T10:14:35.293525+0000","last_fullsized":"2024-09-02T10:14:35.293525+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.581597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.581597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.091930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":350,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.007430+0000","last_change":"2024-09-02T10:14:31.668649+0000","last_active":"2024-09-02T10:14:35.007430+0000","last_peered":"2024-09-02T10:14:35.007430+0000","last_clean":"2024-09-02T10:14:35.007430+0000","last_became_active":"2024-09-02T10:12:47.112475+0000","last_became_peered":"2024-09-02T10:12:47.112475+0000","last_unstale":"2024-09-02T10:14:35.007430+0000","last_undegraded":"2024-09-02T10:14:35.007430+0000","last_fullsized":"2024-09-02T10:14:35.007430+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.668602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.668602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:32.955289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":839,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293275+0000","last_change":"2024-09-02T10:14:24.978488+0000","last_active":"2024-09-02T10:14:35.293275+0000","last_peered":"2024-09-02T10:14:35.293275+0000","last_clean":"2024-09-02T10:14:35.293275+0000","last_became_active":"2024-09-02T10:14:24.978135+0000","last_became_peered":"2024-09-02T10:14:24.978135+0000","last_unstale":"2024-09-02T10:14:35.293275+0000","last_undegraded":"2024-09-02T10:14:35.293275+0000","last_fullsized":"2024-09-02T10:14:35.293275+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:04.292348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:04.292348+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:11.581892+0000","scrub_duration":971,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":300,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293889+0000","last_change":"2024-09-02T10:12:47.093970+0000","last_active":"2024-09-02T10:14:35.293889+0000","last_peered":"2024-09-02T10:14:35.293889+0000","last_clean":"2024-09-02T10:14:35.293889+0000","last_became_active":"2024-09-02T10:12:47.093728+0000","last_became_peered":"2024-09-02T10:12:47.093728+0000","last_unstale":"2024-09-02T10:14:35.293889+0000","last_undegraded":"2024-09-02T10:14:35.293889+0000","last_fullsized":"2024-09-02T10:14:35.293889+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:39.814764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:39.814764+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":294,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293881+0000","last_change":"2024-09-02T10:14:09.768298+0000","last_active":"2024-09-02T10:14:35.293881+0000","last_peered":"2024-09-02T10:14:35.293881+0000","last_clean":"2024-09-02T10:14:35.293881+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:14:35.293881+0000","last_undegraded":"2024-09-02T10:14:35.293881+0000","last_fullsized":"2024-09-02T10:14:35.293881+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:09.768061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:09.768061+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:27.302501+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":30,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293440+0000","last_change":"2024-09-02T10:14:15.450212+0000","last_active":"2024-09-02T10:14:35.293440+0000","last_peered":"2024-09-02T10:14:35.293440+0000","last_clean":"2024-09-02T10:14:35.293440+0000","last_became_active":"2024-09-02T10:14:15.449831+0000","last_became_peered":"2024-09-02T10:14:15.449831+0000","last_unstale":"2024-09-02T10:14:35.293440+0000","last_undegraded":"2024-09-02T10:14:35.293440+0000","last_fullsized":"2024-09-02T10:14:35.293440+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:57.422564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:57.422564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:24.085156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":290,"reported_epoch":430,"state":"active+clean+scrubbing","last_fresh":"2024-09-02T10:14:36.622645+0000","last_change":"2024-09-02T10:14:36.621315+0000","last_active":"2024-09-02T10:14:36.622645+0000","last_peered":"2024-09-02T10:14:36.622645+0000","last_clean":"2024-09-02T10:14:36.622645+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:14:36.622645+0000","last_undegraded":"2024-09-02T10:14:36.622645+0000","last_fullsized":"2024-09-02T10:14:36.622645+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:40.792141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:40.792141+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":532,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.292522+0000","last_change":"2024-09-02T10:12:34.308797+0000","last_active":"2024-09-02T10:14:35.292522+0000","last_peered":"2024-09-02T10:14:35.292522+0000","last_clean":"2024-09-02T10:14:35.292522+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:14:35.292522+0000","last_undegraded":"2024-09-02T10:14:35.292522+0000","last_fullsized":"2024-09-02T10:14:35.292522+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:12:34.308704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:12:34.308704+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":774,"reported_epoch":428,"state":"active+clean","last_fresh":"2024-09-02T10:14:31.293240+0000","last_change":"2024-09-02T10:14:24.978876+0000","last_active":"2024-09-02T10:14:31.293240+0000","last_peered":"2024-09-02T10:14:31.293240+0000","last_clean":"2024-09-02T10:14:31.293240+0000","last_became_active":"2024-09-02T10:14:24.978484+0000","last_became_peered":"2024-09-02T10:14:24.978484+0000","last_unstale":"2024-09-02T10:14:31.293240+0000","last_undegraded":"2024-09-02T10:14:31.293240+0000","last_fullsized":"2024-09-02T10:14:31.293240+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:27.525135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:27.525135+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:14:34.684099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":543,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.292322+0000","last_change":"2024-09-02T10:13:28.492313+0000","last_active":"2024-09-02T10:14:35.292322+0000","last_peered":"2024-09-02T10:14:35.292322+0000","last_clean":"2024-09-02T10:14:35.292322+0000","last_became_active":"2024-09-02T10:11:11.467851+0000","last_became_peered":"2024-09-02T10:11:11.467851+0000","last_unstale":"2024-09-02T10:14:35.292322+0000","last_undegraded":"2024-09-02T10:14:35.292322+0000","last_fullsized":"2024-09-02T10:14:35.292322+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:28.492170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:28.492170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":799,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.293543+0000","last_change":"2024-09-02T10:14:34.284618+0000","last_active":"2024-09-02T10:14:35.293543+0000","last_peered":"2024-09-02T10:14:35.293543+0000","last_clean":"2024-09-02T10:14:35.293543+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:14:35.293543+0000","last_undegraded":"2024-09-02T10:14:35.293543+0000","last_fullsized":"2024-09-02T10:14:35.293543+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.284563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.284563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:59.753379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":325,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.292375+0000","last_change":"2024-09-02T10:13:23.407152+0000","last_active":"2024-09-02T10:14:35.292375+0000","last_peered":"2024-09-02T10:14:35.292375+0000","last_clean":"2024-09-02T10:14:35.292375+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:14:35.292375+0000","last_undegraded":"2024-09-02T10:14:35.292375+0000","last_fullsized":"2024-09-02T10:14:35.292375+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:23.407096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:23.407096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":564,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.292473+0000","last_change":"2024-09-02T10:14:24.978625+0000","last_active":"2024-09-02T10:14:35.292473+0000","last_peered":"2024-09-02T10:14:35.292473+0000","last_clean":"2024-09-02T10:14:35.292473+0000","last_became_active":"2024-09-02T10:14:24.978245+0000","last_became_peered":"2024-09-02T10:14:24.978245+0000","last_unstale":"2024-09-02T10:14:35.292473+0000","last_undegraded":"2024-09-02T10:14:35.292473+0000","last_fullsized":"2024-09-02T10:14:35.292473+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.120173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.120173+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:37.111735+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-02T10:14:38.017507+0000","last_change":"2024-09-02T10:14:38.017507+0000","last_active":"2024-09-02T10:14:38.017507+0000","last_peered":"2024-09-02T10:14:38.017507+0000","last_clean":"2024-09-02T10:14:38.017507+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-02T10:14:38.017507+0000","last_undegraded":"2024-09-02T10:14:38.017507+0000","last_fullsized":"2024-09-02T10:14:38.017507+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":330,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-09-02T10:14:35.617172+0000","last_change":"2024-09-02T10:14:35.617172+0000","last_active":"2024-09-02T10:14:35.617172+0000","last_peered":"2024-09-02T10:14:35.617172+0000","last_clean":"2024-09-02T10:14:35.617172+0000","last_became_active":"2024-09-02T10:11:11.472186+0000","last_became_peered":"2024-09-02T10:11:11.472186+0000","last_unstale":"2024-09-02T10:14:35.617172+0000","last_undegraded":"2024-09-02T10:14:35.617172+0000","last_fullsized":"2024-09-02T10:14:35.617172+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.617088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.617088+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:00.397366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321096,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93424056,"statfs":{"total":95995035648,"available":95666233344,"internally_reserved":0,"allocated":1163264,"data_stored":736191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":327612868},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612787,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":144968,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600184,"statfs":{"total":95995035648,"available":95846588416,"internally_reserved":0,"allocated":1368064,"data_stored":937439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[0,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841720,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":389440,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93355712,"statfs":{"total":95995035648,"available":95596249088,"internally_reserved":0,"allocated":1490944,"data_stored":1063871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":397277632},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070653,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190572,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554580,"statfs":{"total":95995035648,"available":95799889920,"internally_reserved":0,"allocated":1138688,"data_stored":721313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299586,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":371264,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93373888,"statfs":{"total":95995035648,"available":95614861312,"internally_reserved":0,"allocated":1355776,"data_stored":930000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":378796490},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528518,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":195076,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93550076,"statfs":{"total":95995035648,"available":95795277824,"internally_reserved":0,"allocated":1490944,"data_stored":1063871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":198244807},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724747,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307408,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93437744,"statfs":{"total":95995035648,"available":95680249856,"internally_reserved":0,"allocated":1368064,"data_stored":937439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149909,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401580,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93343572,"statfs":{"total":95995035648,"available":95583817728,"internally_reserved":0,"allocated":1138688,"data_stored":721313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":410057160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411545,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575070,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":717736,"kb_used_data":361688,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93027416,"statfs":{"total":95995035648,"available":95260073984,"internally_reserved":0,"allocated":370368512,"data_stored":369951137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738597,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353344,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93391808,"statfs":{"total":95995035648,"available":95633211392,"internally_reserved":0,"allocated":1490944,"data_stored":1063871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:14:39.760 INFO:tasks.thrashosds.thrasher:No pools; doing nothing 2024-09-02T10:14:39.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:39.847 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:39.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:39.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:39.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:39.975 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:39.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:40.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:40.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:40.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:40.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:40.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:40.226 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:40.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:40.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:40 smithi162 ceph-mon[20279]: osdmap e432: 12 total, 12 up, 10 in 2024-09-02T10:14:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/983616608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:40 smithi162 ceph-mon[20279]: pgmap v1022: 47 pgs: 15 unknown, 1 creating+peering, 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:40.355 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:40.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:40.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:40 smithi151 ceph-mon[16639]: osdmap e432: 12 total, 12 up, 10 in 2024-09-02T10:14:40.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/983616608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:40.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:40 smithi151 ceph-mon[16639]: pgmap v1022: 47 pgs: 15 unknown, 1 creating+peering, 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:40.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:40.482 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:40.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:40.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:40 smithi187 ceph-mon[20072]: osdmap e432: 12 total, 12 up, 10 in 2024-09-02T10:14:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/983616608' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:40 smithi187 ceph-mon[20072]: pgmap v1022: 47 pgs: 15 unknown, 1 creating+peering, 1 active+clean+scrubbing, 30 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:40.584 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:40.609 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:40.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:40.611 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:40.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:40.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:40.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:40.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:40.862 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:40.863 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:40.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:40.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:40.991 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:41.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:14:41.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:41.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:41.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:41.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:41.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:41.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:41.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:41.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:41.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:41.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:41 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:14:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:41 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:14:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:41 smithi162 ceph-mon[20279]: osdmap e433: 12 total, 12 up, 10 in 2024-09-02T10:14:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:41 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:14:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:41 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:14:41.364 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:41.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:41 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:14:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:41 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:14:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:41 smithi151 ceph-mon[16639]: osdmap e433: 12 total, 12 up, 10 in 2024-09-02T10:14:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:41 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:14:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:41 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:14:41.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:41.473 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:41.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:41.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:41.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-02T10:14:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:41 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:14:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:41 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:14:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:41 smithi187 ceph-mon[20072]: osdmap e433: 12 total, 12 up, 10 in 2024-09-02T10:14:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:41 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:14:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:41 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:14:41.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:41.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:41.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:41.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:41.744 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:41.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:41.846 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:41.872 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:41.873 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:41.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:42.000 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:42.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:42.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:42.029 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:42.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:42 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:14:42.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:42 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:14:42.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:42 smithi151 ceph-mon[16639]: pgmap v1024: 47 pgs: 13 unknown, 1 creating+peering, 1 active+clean+scrubbing, 32 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:42.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:42 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:14:42.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:42 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:14:42.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:42.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:42.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:42.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:42.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:42.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:42.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:42.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:42 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:14:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:42 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:14:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:42 smithi162 ceph-mon[20279]: pgmap v1024: 47 pgs: 13 unknown, 1 creating+peering, 1 active+clean+scrubbing, 32 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:42 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:14:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:42 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:14:42.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:42.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:42.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:42.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:42.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:14:42.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:42.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:42.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:42 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:14:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:42 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:14:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:42 smithi187 ceph-mon[20072]: pgmap v1024: 47 pgs: 13 unknown, 1 creating+peering, 1 active+clean+scrubbing, 32 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:42 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:14:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:42 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:14:42.528 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:42.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:42.626 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:42.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:42.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:42.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:42.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:42.755 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:42.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:42.880 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:42.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:42.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:43.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:43.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:43.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:43.053 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-02T10:14:43.053 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:14:43.084 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:14:43.084 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:14:43.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:43.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:43.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:43.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:43.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:43.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:43.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:43.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:43 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:14:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:43 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:14:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:43 smithi162 ceph-mon[20279]: osdmap e434: 12 total, 12 up, 10 in 2024-09-02T10:14:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3080539185' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2394782287' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:14:43.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:43.383 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:43.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:43 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:14:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:43 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:14:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:43 smithi151 ceph-mon[16639]: osdmap e434: 12 total, 12 up, 10 in 2024-09-02T10:14:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3080539185' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2394782287' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:14:43.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:43 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:14:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:43 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:14:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:43 smithi187 ceph-mon[20072]: osdmap e434: 12 total, 12 up, 10 in 2024-09-02T10:14:43.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3080539185' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:43.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2394782287' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:14:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:43.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:43.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:43.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:43.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:43.636 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:43.683 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:14:43.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:43.763 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:43.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:43.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:43.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:43.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:43.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:43.891 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:43.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:44.018 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:44.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:44.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:44.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:44.144 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:44.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:44.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:44.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:44.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:44.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:44 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:14:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:44 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:14:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3080539185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2394782287' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:14:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:44 smithi162 ceph-mon[20279]: osdmap e435: 12 total, 12 up, 10 in 2024-09-02T10:14:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:44 smithi162 ceph-mon[20279]: pgmap v1027: 47 pgs: 1 creating+peering, 46 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:44.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:44.396 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:44.397 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:44.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:44 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:14:44.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:44 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:14:44.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3080539185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:44.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2394782287' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:14:44.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:44 smithi151 ceph-mon[16639]: osdmap e435: 12 total, 12 up, 10 in 2024-09-02T10:14:44.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:44 smithi151 ceph-mon[16639]: pgmap v1027: 47 pgs: 1 creating+peering, 46 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:44.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:44 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:14:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:44 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:14:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3080539185' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2394782287' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:14:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:44 smithi187 ceph-mon[20072]: osdmap e435: 12 total, 12 up, 10 in 2024-09-02T10:14:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:44 smithi187 ceph-mon[20072]: pgmap v1027: 47 pgs: 1 creating+peering, 46 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:44.523 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:44.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:44.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:44.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:44.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:44.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:44.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:44.761 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:14:44.761 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:14:44.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:44.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:44.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:44.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:44.904 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:44.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:44.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:45.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:45.031 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:45.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:45.124 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:45.127 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:45.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:45.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:45.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:45 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:14:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:45 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:14:45.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:45.284 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:45.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:45 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:14:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:45 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:14:45.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:45.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:45.412 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:45 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:14:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:45 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:14:45.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:45.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:45.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:45.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:45.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:45.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:45.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:45.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:45.763 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:45.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:45.789 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:45.876 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:14:45.876 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:14:45.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:45.915 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:45.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:46.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:46.041 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:46.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:46.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:14:46.077 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:14:46.098 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:14:46.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:14:46.100 DEBUG:teuthology.orchestra.run.smithi187:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-02T10:14:46.101 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-02T10:14:46.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:14:46.103 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-02T10:14:46.104 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-02T10:14:46.119 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:14:46.119 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:14:46.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:46 smithi151 ceph-mon[16639]: osdmap e436: 12 total, 12 up, 10 in 2024-09-02T10:14:46.119 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:46 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:14:46.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:46 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:14:46.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:46 smithi151 ceph-mon[16639]: pgmap v1029: 47 pgs: 47 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:46.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:46 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:14:46.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:46 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:14:46.120 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4279761008' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:46.139 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:14:46.140 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:14:46.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:46.161 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:14:46.161 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:14:46.165 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:46.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:46.167 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:hints = 1 2024-09-02T10:14:46.167 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-02T10:14:46.167 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object prefix: benchmark_data_smithi187_89405 2024-09-02T10:14:46.167 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:14:46.167 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 0 0 0 0 0 0 - 0 2024-09-02T10:14:46.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:14:46.190 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:14:46.213 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:14:46.214 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:14:46.235 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:14:46.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:14:46.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:14:46.256 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:14:46.267 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:46.277 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:14:46.277 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:14:46.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:46.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:46.290 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:46.297 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:14:46.298 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:14:46.321 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:14:46.321 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:14:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:46 smithi162 ceph-mon[20279]: osdmap e436: 12 total, 12 up, 10 in 2024-09-02T10:14:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:46 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:14:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:46 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:14:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:46 smithi162 ceph-mon[20279]: pgmap v1029: 47 pgs: 47 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:46 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:14:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:46 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:14:46.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4279761008' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:46.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:46.413 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:46.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:46.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:46.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:14:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:46 smithi187 ceph-mon[20072]: osdmap e436: 12 total, 12 up, 10 in 2024-09-02T10:14:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:46 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:14:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:46 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:14:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:46 smithi187 ceph-mon[20072]: pgmap v1029: 47 pgs: 47 active+clean; 577 KiB data, 3.2 GiB used, 891 GiB / 894 GiB avail 2024-09-02T10:14:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:46 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:14:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:46 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:14:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4279761008' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:46.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:46.536 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:46.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:46.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:46.666 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:46.667 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:46.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:46.796 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:46.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:46.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:46.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:46.924 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:46.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:46.925 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:47.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:47.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:47.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:47.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:47.107 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:47.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:47.167 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 1 16 1544 1528 95.4951 95.5 0.00148979 0.0103148 2024-09-02T10:14:47.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:47.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:47.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:47.307 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:47.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:47.308 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:47.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:47 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:14:47.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:47.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:47.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:47.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:47.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:47 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:14:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:47 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:14:47.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:47.562 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:47.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:47.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:14:47.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:47.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:47.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:47.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:47.826 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:47.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:47.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:47.952 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:47.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:47.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:48.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:48.066 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:48.078 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:48.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:48.167 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 2 16 3251 3235 101.088 106.688 0.0025111 0.00986305 2024-09-02T10:14:48.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:48.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:48.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:48.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:48.314 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:14:48.314 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:14:48.333 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:48.334 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:48.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:48 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:14:48.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:48 smithi162 ceph-mon[20279]: osdmap e437: 12 total, 12 up, 10 in 2024-09-02T10:14:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:48 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:14:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:48 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:14:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:48 smithi162 ceph-mon[20279]: pgmap v1031: 47 pgs: 47 active+clean; 28 MiB data, 3.2 GiB used, 891 GiB / 894 GiB avail; 5.1 MiB/s wr, 82 op/s 2024-09-02T10:14:48.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:48 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:14:48.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:48 smithi151 ceph-mon[16639]: osdmap e437: 12 total, 12 up, 10 in 2024-09-02T10:14:48.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:48 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:14:48.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:48 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:14:48.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:48 smithi151 ceph-mon[16639]: pgmap v1031: 47 pgs: 47 active+clean; 28 MiB data, 3.2 GiB used, 891 GiB / 894 GiB avail; 5.1 MiB/s wr, 82 op/s 2024-09-02T10:14:48.434 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:48.456 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:48.457 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:48 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:14:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:48 smithi187 ceph-mon[20072]: osdmap e437: 12 total, 12 up, 10 in 2024-09-02T10:14:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:48 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:14:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:48 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:14:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:48 smithi187 ceph-mon[20072]: pgmap v1031: 47 pgs: 47 active+clean; 28 MiB data, 3.2 GiB used, 891 GiB / 894 GiB avail; 5.1 MiB/s wr, 82 op/s 2024-09-02T10:14:48.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:48.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:48.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:48.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:48.712 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:48.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:48.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:48.842 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:48.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:48.881 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:48.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:48.969 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:48.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:49.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:49.095 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:49.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:14:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:49.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:49.167 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 3 16 5189 5173 107.763 121.125 0.0551217 0.00913066 2024-09-02T10:14:49.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:49.221 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:49.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:49.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:49 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:14:49.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:49 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:14:49.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:49 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:14:49.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:49 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:14:49.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:49 smithi151 ceph-mon[16639]: from='client.16407 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:14:49.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:49.340 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1031,"stamp":"2024-09-02T10:14:47.280622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":28836384,"num_objects":433,"num_object_clones":0,"num_object_copies":1299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":431,"num_write_kb":27584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3283,"ondisk_log_size":3283,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":3362012,"kb_used_data":342940,"kb_used_omap":15,"kb_used_meta":3018864,"kb_avail":934089508,"statfs":{"total":959950356480,"available":956507656192,"internally_reserved":0,"allocated":351170560,"data_stored":346668723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3091317169},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28246016,"num_objects":431,"num_object_clones":0,"num_object_copies":1293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":431,"num_write_kb":27584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.235501"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":200,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-09-02T10:14:47.074300+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:14:47.074300+0000","last_peered":"2024-09-02T10:14:47.074300+0000","last_clean":"2024-09-02T10:14:47.074300+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:14:47.074300+0000","last_undegraded":"2024-09-02T10:14:47.074300+0000","last_fullsized":"2024-09-02T10:14:47.074300+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:56.714204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":229,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158165+0000","last_change":"2024-09-02T10:14:33.239009+0000","last_active":"2024-09-02T10:14:45.158165+0000","last_peered":"2024-09-02T10:14:45.158165+0000","last_clean":"2024-09-02T10:14:45.158165+0000","last_became_active":"2024-09-02T10:12:47.212512+0000","last_became_peered":"2024-09-02T10:12:47.212512+0000","last_unstale":"2024-09-02T10:14:45.158165+0000","last_undegraded":"2024-09-02T10:14:45.158165+0000","last_fullsized":"2024-09-02T10:14:45.158165+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:33.238949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:33.238949+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:56.403772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,6],"acting":[5,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":446,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-02T10:14:43.058587+0000","last_change":"2024-09-02T10:13:59.249166+0000","last_active":"2024-09-02T10:14:43.058587+0000","last_peered":"2024-09-02T10:14:43.058587+0000","last_clean":"2024-09-02T10:14:43.058587+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:14:43.058587+0000","last_undegraded":"2024-09-02T10:14:43.058587+0000","last_fullsized":"2024-09-02T10:14:43.058587+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:59.249105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:59.249105+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:02.787049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":446,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-02T10:14:43.058531+0000","last_change":"2024-09-02T10:14:01.321873+0000","last_active":"2024-09-02T10:14:43.058531+0000","last_peered":"2024-09-02T10:14:43.058531+0000","last_clean":"2024-09-02T10:14:43.058531+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:14:43.058531+0000","last_undegraded":"2024-09-02T10:14:43.058531+0000","last_fullsized":"2024-09-02T10:14:43.058531+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:01.321818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:01.321818+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:18.881384+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":355,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158520+0000","last_change":"2024-09-02T10:14:01.105137+0000","last_active":"2024-09-02T10:14:45.158520+0000","last_peered":"2024-09-02T10:14:45.158520+0000","last_clean":"2024-09-02T10:14:45.158520+0000","last_became_active":"2024-09-02T10:11:18.920488+0000","last_became_peered":"2024-09-02T10:11:18.920488+0000","last_unstale":"2024-09-02T10:14:45.158520+0000","last_undegraded":"2024-09-02T10:14:45.158520+0000","last_fullsized":"2024-09-02T10:14:45.158520+0000","mapping_epoch":332,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:01.105083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:01.105083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:03.020721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,5],"acting":[7,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":305,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158209+0000","last_change":"2024-09-02T10:14:11.689851+0000","last_active":"2024-09-02T10:14:45.158209+0000","last_peered":"2024-09-02T10:14:45.158209+0000","last_clean":"2024-09-02T10:14:45.158209+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:14:45.158209+0000","last_undegraded":"2024-09-02T10:14:45.158209+0000","last_fullsized":"2024-09-02T10:14:45.158209+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.689767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:19.591338+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":810,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.351955+0000","last_change":"2024-09-02T10:14:45.351955+0000","last_active":"2024-09-02T10:14:45.351955+0000","last_peered":"2024-09-02T10:14:45.351955+0000","last_clean":"2024-09-02T10:14:45.351955+0000","last_became_active":"2024-09-02T10:12:47.091891+0000","last_became_peered":"2024-09-02T10:12:47.091891+0000","last_unstale":"2024-09-02T10:14:45.351955+0000","last_undegraded":"2024-09-02T10:14:45.351955+0000","last_fullsized":"2024-09-02T10:14:45.351955+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:45.351862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:45.351862+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:00.571756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":453,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-02T10:14:43.058734+0000","last_change":"2024-09-02T10:13:33.498600+0000","last_active":"2024-09-02T10:14:43.058734+0000","last_peered":"2024-09-02T10:14:43.058734+0000","last_clean":"2024-09-02T10:14:43.058734+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:14:43.058734+0000","last_undegraded":"2024-09-02T10:14:43.058734+0000","last_fullsized":"2024-09-02T10:14:43.058734+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:33.498535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:33.498535+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:14:57.741728+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":453,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-02T10:14:43.054371+0000","last_change":"2024-09-02T10:14:04.633772+0000","last_active":"2024-09-02T10:14:43.054371+0000","last_peered":"2024-09-02T10:14:43.054371+0000","last_clean":"2024-09-02T10:14:43.054371+0000","last_became_active":"2024-09-02T10:11:11.497246+0000","last_became_peered":"2024-09-02T10:11:11.497246+0000","last_unstale":"2024-09-02T10:14:43.054371+0000","last_undegraded":"2024-09-02T10:14:43.054371+0000","last_fullsized":"2024-09-02T10:14:43.054371+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:04.633686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:04.633686+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:28.648762+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,11],"acting":[1,0,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":431,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158348+0000","last_change":"2024-09-02T10:13:37.418287+0000","last_active":"2024-09-02T10:14:45.158348+0000","last_peered":"2024-09-02T10:14:45.158348+0000","last_clean":"2024-09-02T10:14:45.158348+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:14:45.158348+0000","last_undegraded":"2024-09-02T10:14:45.158348+0000","last_fullsized":"2024-09-02T10:14:45.158348+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:37.418216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:37.418216+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":290,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.203339+0000","last_change":"2024-09-02T10:14:45.203339+0000","last_active":"2024-09-02T10:14:45.203339+0000","last_peered":"2024-09-02T10:14:45.203339+0000","last_clean":"2024-09-02T10:14:45.203339+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:14:45.203339+0000","last_undegraded":"2024-09-02T10:14:45.203339+0000","last_fullsized":"2024-09-02T10:14:45.203339+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:45.203278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:45.203278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:51.594287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":786,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-02T10:14:43.054176+0000","last_change":"2024-09-02T10:13:55.581470+0000","last_active":"2024-09-02T10:14:43.054176+0000","last_peered":"2024-09-02T10:14:43.054176+0000","last_clean":"2024-09-02T10:14:43.054176+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:14:43.054176+0000","last_undegraded":"2024-09-02T10:14:43.054176+0000","last_fullsized":"2024-09-02T10:14:43.054176+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:55.581403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:55.581403+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:14:59.483473+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":724,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.157919+0000","last_change":"2024-09-02T10:14:35.252590+0000","last_active":"2024-09-02T10:14:45.157919+0000","last_peered":"2024-09-02T10:14:45.157919+0000","last_clean":"2024-09-02T10:14:45.157919+0000","last_became_active":"2024-09-02T10:11:11.468355+0000","last_became_peered":"2024-09-02T10:11:11.468355+0000","last_unstale":"2024-09-02T10:14:45.157919+0000","last_undegraded":"2024-09-02T10:14:45.157919+0000","last_fullsized":"2024-09-02T10:14:45.157919+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.252526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.252526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:58.626048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":700,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-09-02T10:14:42.047158+0000","last_change":"2024-09-02T10:14:34.665490+0000","last_active":"2024-09-02T10:14:42.047158+0000","last_peered":"2024-09-02T10:14:42.047158+0000","last_clean":"2024-09-02T10:14:42.047158+0000","last_became_active":"2024-09-02T10:14:24.979372+0000","last_became_peered":"2024-09-02T10:14:24.979372+0000","last_unstale":"2024-09-02T10:14:42.047158+0000","last_undegraded":"2024-09-02T10:14:42.047158+0000","last_fullsized":"2024-09-02T10:14:42.047158+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.665438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.665438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:58.191772+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1],"acting":[0,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":709,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158440+0000","last_change":"2024-09-02T10:14:34.217697+0000","last_active":"2024-09-02T10:14:45.158440+0000","last_peered":"2024-09-02T10:14:45.158440+0000","last_clean":"2024-09-02T10:14:45.158440+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:14:45.158440+0000","last_undegraded":"2024-09-02T10:14:45.158440+0000","last_fullsized":"2024-09-02T10:14:45.158440+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.217624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.217624+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:02.148154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":16,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-09-02T10:14:42.047275+0000","last_change":"2024-09-02T10:14:39.037453+0000","last_active":"2024-09-02T10:14:42.047275+0000","last_peered":"2024-09-02T10:14:42.047275+0000","last_clean":"2024-09-02T10:14:42.047275+0000","last_became_active":"2024-09-02T10:14:39.037228+0000","last_became_peered":"2024-09-02T10:14:39.037228+0000","last_unstale":"2024-09-02T10:14:42.047275+0000","last_undegraded":"2024-09-02T10:14:42.047275+0000","last_fullsized":"2024-09-02T10:14:42.047275+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:02.128741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":795,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158506+0000","last_change":"2024-09-02T10:14:41.306927+0000","last_active":"2024-09-02T10:14:45.158506+0000","last_peered":"2024-09-02T10:14:45.158506+0000","last_clean":"2024-09-02T10:14:45.158506+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:14:45.158506+0000","last_undegraded":"2024-09-02T10:14:45.158506+0000","last_fullsized":"2024-09-02T10:14:45.158506+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:41.306868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:41.306868+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:04.992960+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"437'101","reported_seq":124,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-09-02T10:14:47.074634+0000","last_change":"2024-09-02T10:14:39.040351+0000","last_active":"2024-09-02T10:14:47.074634+0000","last_peered":"2024-09-02T10:14:47.074634+0000","last_clean":"2024-09-02T10:14:47.074634+0000","last_became_active":"2024-09-02T10:14:39.040025+0000","last_became_peered":"2024-09-02T10:14:39.040025+0000","last_unstale":"2024-09-02T10:14:47.074634+0000","last_undegraded":"2024-09-02T10:14:47.074634+0000","last_fullsized":"2024-09-02T10:14:47.074634+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:54.413941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":910,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158687+0000","last_change":"2024-09-02T10:14:36.143055+0000","last_active":"2024-09-02T10:14:45.158687+0000","last_peered":"2024-09-02T10:14:45.158687+0000","last_clean":"2024-09-02T10:14:45.158687+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:14:45.158687+0000","last_undegraded":"2024-09-02T10:14:45.158687+0000","last_fullsized":"2024-09-02T10:14:45.158687+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.142993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.142993+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.623989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":16,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-09-02T10:14:42.364281+0000","last_change":"2024-09-02T10:14:39.038506+0000","last_active":"2024-09-02T10:14:42.364281+0000","last_peered":"2024-09-02T10:14:42.364281+0000","last_clean":"2024-09-02T10:14:42.364281+0000","last_became_active":"2024-09-02T10:14:39.038164+0000","last_became_peered":"2024-09-02T10:14:39.038164+0000","last_unstale":"2024-09-02T10:14:42.364281+0000","last_undegraded":"2024-09-02T10:14:42.364281+0000","last_fullsized":"2024-09-02T10:14:42.364281+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:52.835965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":549,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-09-02T10:14:42.364078+0000","last_change":"2024-09-02T10:14:31.581643+0000","last_active":"2024-09-02T10:14:42.364078+0000","last_peered":"2024-09-02T10:14:42.364078+0000","last_clean":"2024-09-02T10:14:42.364078+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:14:42.364078+0000","last_undegraded":"2024-09-02T10:14:42.364078+0000","last_fullsized":"2024-09-02T10:14:42.364078+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.581597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.581597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.091930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"436'73","reported_seq":94,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:46.983801+0000","last_change":"2024-09-02T10:14:39.037573+0000","last_active":"2024-09-02T10:14:46.983801+0000","last_peered":"2024-09-02T10:14:46.983801+0000","last_clean":"2024-09-02T10:14:46.983801+0000","last_became_active":"2024-09-02T10:14:39.037121+0000","last_became_peered":"2024-09-02T10:14:39.037121+0000","last_unstale":"2024-09-02T10:14:46.983801+0000","last_undegraded":"2024-09-02T10:14:46.983801+0000","last_fullsized":"2024-09-02T10:14:46.983801+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:01.026599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784128,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73,"num_write_kb":4672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":358,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-09-02T10:14:42.047960+0000","last_change":"2024-09-02T10:14:31.668649+0000","last_active":"2024-09-02T10:14:42.047960+0000","last_peered":"2024-09-02T10:14:42.047960+0000","last_clean":"2024-09-02T10:14:42.047960+0000","last_became_active":"2024-09-02T10:12:47.112475+0000","last_became_peered":"2024-09-02T10:12:47.112475+0000","last_unstale":"2024-09-02T10:14:42.047960+0000","last_undegraded":"2024-09-02T10:14:42.047960+0000","last_fullsized":"2024-09-02T10:14:42.047960+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.668602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.668602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:32.955289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"436'17","reported_seq":38,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:46.608954+0000","last_change":"2024-09-02T10:14:39.642395+0000","last_active":"2024-09-02T10:14:46.608954+0000","last_peered":"2024-09-02T10:14:46.608954+0000","last_clean":"2024-09-02T10:14:46.608954+0000","last_became_active":"2024-09-02T10:14:39.641995+0000","last_became_peered":"2024-09-02T10:14:39.641995+0000","last_unstale":"2024-09-02T10:14:46.608954+0000","last_undegraded":"2024-09-02T10:14:46.608954+0000","last_fullsized":"2024-09-02T10:14:46.608954+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:55.658135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":853,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-09-02T10:14:47.074229+0000","last_change":"2024-09-02T10:14:24.978488+0000","last_active":"2024-09-02T10:14:47.074229+0000","last_peered":"2024-09-02T10:14:47.074229+0000","last_clean":"2024-09-02T10:14:47.074229+0000","last_became_active":"2024-09-02T10:14:24.978135+0000","last_became_peered":"2024-09-02T10:14:24.978135+0000","last_unstale":"2024-09-02T10:14:47.074229+0000","last_undegraded":"2024-09-02T10:14:47.074229+0000","last_fullsized":"2024-09-02T10:14:47.074229+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:04.292348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:04.292348+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:11.581892+0000","scrub_duration":971,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,6],"acting":[11,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":19,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-02T10:14:43.053802+0000","last_change":"2024-09-02T10:14:39.034903+0000","last_active":"2024-09-02T10:14:43.053802+0000","last_peered":"2024-09-02T10:14:43.053802+0000","last_clean":"2024-09-02T10:14:43.053802+0000","last_became_active":"2024-09-02T10:14:39.034636+0000","last_became_peered":"2024-09-02T10:14:39.034636+0000","last_unstale":"2024-09-02T10:14:43.053802+0000","last_undegraded":"2024-09-02T10:14:43.053802+0000","last_fullsized":"2024-09-02T10:14:43.053802+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:07.235133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":320,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158209+0000","last_change":"2024-09-02T10:14:40.701835+0000","last_active":"2024-09-02T10:14:45.158209+0000","last_peered":"2024-09-02T10:14:45.158209+0000","last_clean":"2024-09-02T10:14:45.158209+0000","last_became_active":"2024-09-02T10:12:47.093728+0000","last_became_peered":"2024-09-02T10:12:47.093728+0000","last_unstale":"2024-09-02T10:14:45.158209+0000","last_undegraded":"2024-09-02T10:14:45.158209+0000","last_fullsized":"2024-09-02T10:14:45.158209+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.701773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.701773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:07.019321+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,7],"acting":[10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"437'68","reported_seq":90,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-09-02T10:14:47.191154+0000","last_change":"2024-09-02T10:14:39.038158+0000","last_active":"2024-09-02T10:14:47.191154+0000","last_peered":"2024-09-02T10:14:47.191154+0000","last_clean":"2024-09-02T10:14:47.191154+0000","last_became_active":"2024-09-02T10:14:39.037822+0000","last_became_peered":"2024-09-02T10:14:39.037822+0000","last_unstale":"2024-09-02T10:14:47.191154+0000","last_undegraded":"2024-09-02T10:14:47.191154+0000","last_fullsized":"2024-09-02T10:14:47.191154+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:47.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":4352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":306,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158145+0000","last_change":"2024-09-02T10:14:09.768298+0000","last_active":"2024-09-02T10:14:45.158145+0000","last_peered":"2024-09-02T10:14:45.158145+0000","last_clean":"2024-09-02T10:14:45.158145+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:14:45.158145+0000","last_undegraded":"2024-09-02T10:14:45.158145+0000","last_fullsized":"2024-09-02T10:14:45.158145+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:09.768061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:09.768061+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:27.302501+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":16,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-09-02T10:14:42.047836+0000","last_change":"2024-09-02T10:14:39.035303+0000","last_active":"2024-09-02T10:14:42.047836+0000","last_peered":"2024-09-02T10:14:42.047836+0000","last_clean":"2024-09-02T10:14:42.047836+0000","last_became_active":"2024-09-02T10:14:39.035100+0000","last_became_peered":"2024-09-02T10:14:39.035100+0000","last_unstale":"2024-09-02T10:14:42.047836+0000","last_undegraded":"2024-09-02T10:14:42.047836+0000","last_fullsized":"2024-09-02T10:14:42.047836+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.857725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":44,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-09-02T10:14:47.073547+0000","last_change":"2024-09-02T10:14:15.450212+0000","last_active":"2024-09-02T10:14:47.073547+0000","last_peered":"2024-09-02T10:14:47.073547+0000","last_clean":"2024-09-02T10:14:47.073547+0000","last_became_active":"2024-09-02T10:14:15.449831+0000","last_became_peered":"2024-09-02T10:14:15.449831+0000","last_unstale":"2024-09-02T10:14:47.073547+0000","last_undegraded":"2024-09-02T10:14:47.073547+0000","last_fullsized":"2024-09-02T10:14:47.073547+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:57.422564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:57.422564+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:24.085156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,5],"acting":[6,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":16,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-09-02T10:14:42.047424+0000","last_change":"2024-09-02T10:14:39.641746+0000","last_active":"2024-09-02T10:14:42.047424+0000","last_peered":"2024-09-02T10:14:42.047424+0000","last_clean":"2024-09-02T10:14:42.047424+0000","last_became_active":"2024-09-02T10:14:39.641478+0000","last_became_peered":"2024-09-02T10:14:39.641478+0000","last_unstale":"2024-09-02T10:14:42.047424+0000","last_undegraded":"2024-09-02T10:14:42.047424+0000","last_fullsized":"2024-09-02T10:14:42.047424+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:01.557107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":307,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158344+0000","last_change":"2024-09-02T10:14:36.623992+0000","last_active":"2024-09-02T10:14:45.158344+0000","last_peered":"2024-09-02T10:14:45.158344+0000","last_clean":"2024-09-02T10:14:45.158344+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:14:45.158344+0000","last_undegraded":"2024-09-02T10:14:45.158344+0000","last_fullsized":"2024-09-02T10:14:45.158344+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.623930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.623930+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.218010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"436'90","reported_seq":110,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:46.965308+0000","last_change":"2024-09-02T10:14:39.038216+0000","last_active":"2024-09-02T10:14:46.965308+0000","last_peered":"2024-09-02T10:14:46.965308+0000","last_clean":"2024-09-02T10:14:46.965308+0000","last_became_active":"2024-09-02T10:14:39.037882+0000","last_became_peered":"2024-09-02T10:14:39.037882+0000","last_unstale":"2024-09-02T10:14:46.965308+0000","last_undegraded":"2024-09-02T10:14:46.965308+0000","last_fullsized":"2024-09-02T10:14:46.965308+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:43.534011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5898240,"num_objects":90,"num_object_clones":0,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90,"num_write_kb":5760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":552,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158632+0000","last_change":"2024-09-02T10:14:40.192356+0000","last_active":"2024-09-02T10:14:45.158632+0000","last_peered":"2024-09-02T10:14:45.158632+0000","last_clean":"2024-09-02T10:14:45.158632+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:14:45.158632+0000","last_undegraded":"2024-09-02T10:14:45.158632+0000","last_fullsized":"2024-09-02T10:14:45.158632+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.192297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.192297+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.932788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"436'65","reported_seq":85,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:46.980319+0000","last_change":"2024-09-02T10:14:39.642277+0000","last_active":"2024-09-02T10:14:46.980319+0000","last_peered":"2024-09-02T10:14:46.980319+0000","last_clean":"2024-09-02T10:14:46.980319+0000","last_became_active":"2024-09-02T10:14:39.641932+0000","last_became_peered":"2024-09-02T10:14:39.641932+0000","last_unstale":"2024-09-02T10:14:46.980319+0000","last_undegraded":"2024-09-02T10:14:46.980319+0000","last_fullsized":"2024-09-02T10:14:46.980319+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:55.249885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4259840,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65,"num_write_kb":4160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":796,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-02T10:14:43.054242+0000","last_change":"2024-09-02T10:14:35.804085+0000","last_active":"2024-09-02T10:14:43.054242+0000","last_peered":"2024-09-02T10:14:43.054242+0000","last_clean":"2024-09-02T10:14:43.054242+0000","last_became_active":"2024-09-02T10:14:24.978484+0000","last_became_peered":"2024-09-02T10:14:24.978484+0000","last_unstale":"2024-09-02T10:14:43.054242+0000","last_undegraded":"2024-09-02T10:14:43.054242+0000","last_fullsized":"2024-09-02T10:14:43.054242+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.804014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.804014+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:52.685249+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"436'17","reported_seq":38,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:46.615104+0000","last_change":"2024-09-02T10:14:39.038104+0000","last_active":"2024-09-02T10:14:46.615104+0000","last_peered":"2024-09-02T10:14:46.615104+0000","last_clean":"2024-09-02T10:14:46.615104+0000","last_became_active":"2024-09-02T10:14:39.037755+0000","last_became_peered":"2024-09-02T10:14:39.037755+0000","last_unstale":"2024-09-02T10:14:46.615104+0000","last_undegraded":"2024-09-02T10:14:46.615104+0000","last_fullsized":"2024-09-02T10:14:46.615104+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:00.080239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1114112,"num_objects":17,"num_object_clones":0,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":1088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":555,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158239+0000","last_change":"2024-09-02T10:13:28.492313+0000","last_active":"2024-09-02T10:14:45.158239+0000","last_peered":"2024-09-02T10:14:45.158239+0000","last_clean":"2024-09-02T10:14:45.158239+0000","last_became_active":"2024-09-02T10:11:11.467851+0000","last_became_peered":"2024-09-02T10:11:11.467851+0000","last_unstale":"2024-09-02T10:14:45.158239+0000","last_undegraded":"2024-09-02T10:14:45.158239+0000","last_fullsized":"2024-09-02T10:14:45.158239+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:28.492170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:28.492170+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":16,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-09-02T10:14:42.364204+0000","last_change":"2024-09-02T10:14:39.039399+0000","last_active":"2024-09-02T10:14:42.364204+0000","last_peered":"2024-09-02T10:14:42.364204+0000","last_clean":"2024-09-02T10:14:42.364204+0000","last_became_active":"2024-09-02T10:14:39.039060+0000","last_became_peered":"2024-09-02T10:14:39.039060+0000","last_unstale":"2024-09-02T10:14:42.364204+0000","last_undegraded":"2024-09-02T10:14:42.364204+0000","last_fullsized":"2024-09-02T10:14:42.364204+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:45.602961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":813,"reported_epoch":437,"state":"active+clean","last_fresh":"2024-09-02T10:14:47.073600+0000","last_change":"2024-09-02T10:14:34.284618+0000","last_active":"2024-09-02T10:14:47.073600+0000","last_peered":"2024-09-02T10:14:47.073600+0000","last_clean":"2024-09-02T10:14:47.073600+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:14:47.073600+0000","last_undegraded":"2024-09-02T10:14:47.073600+0000","last_fullsized":"2024-09-02T10:14:47.073600+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.284563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.284563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:59.753379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":19,"reported_epoch":435,"state":"active+clean","last_fresh":"2024-09-02T10:14:43.058539+0000","last_change":"2024-09-02T10:14:39.036854+0000","last_active":"2024-09-02T10:14:43.058539+0000","last_peered":"2024-09-02T10:14:43.058539+0000","last_clean":"2024-09-02T10:14:43.058539+0000","last_became_active":"2024-09-02T10:14:39.036453+0000","last_became_peered":"2024-09-02T10:14:39.036453+0000","last_unstale":"2024-09-02T10:14:43.058539+0000","last_undegraded":"2024-09-02T10:14:43.058539+0000","last_fullsized":"2024-09-02T10:14:43.058539+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:57.124824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":337,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158764+0000","last_change":"2024-09-02T10:13:23.407152+0000","last_active":"2024-09-02T10:14:45.158764+0000","last_peered":"2024-09-02T10:14:45.158764+0000","last_clean":"2024-09-02T10:14:45.158764+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:14:45.158764+0000","last_undegraded":"2024-09-02T10:14:45.158764+0000","last_fullsized":"2024-09-02T10:14:45.158764+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:13:23.407096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:13:23.407096+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":21,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158687+0000","last_change":"2024-09-02T10:14:39.037960+0000","last_active":"2024-09-02T10:14:45.158687+0000","last_peered":"2024-09-02T10:14:45.158687+0000","last_clean":"2024-09-02T10:14:45.158687+0000","last_became_active":"2024-09-02T10:14:39.037567+0000","last_became_peered":"2024-09-02T10:14:39.037567+0000","last_unstale":"2024-09-02T10:14:45.158687+0000","last_undegraded":"2024-09-02T10:14:45.158687+0000","last_fullsized":"2024-09-02T10:14:45.158687+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.027618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":576,"reported_epoch":436,"state":"active+clean","last_fresh":"2024-09-02T10:14:45.158429+0000","last_change":"2024-09-02T10:14:24.978625+0000","last_active":"2024-09-02T10:14:45.158429+0000","last_peered":"2024-09-02T10:14:45.158429+0000","last_clean":"2024-09-02T10:14:45.158429+0000","last_became_active":"2024-09-02T10:14:24.978245+0000","last_became_peered":"2024-09-02T10:14:24.978245+0000","last_unstale":"2024-09-02T10:14:45.158429+0000","last_undegraded":"2024-09-02T10:14:45.158429+0000","last_fullsized":"2024-09-02T10:14:45.158429+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.120173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.120173+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:37.111735+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":16,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-09-02T10:14:42.047015+0000","last_change":"2024-09-02T10:14:39.036825+0000","last_active":"2024-09-02T10:14:42.047015+0000","last_peered":"2024-09-02T10:14:42.047015+0000","last_clean":"2024-09-02T10:14:42.047015+0000","last_became_active":"2024-09-02T10:14:39.036496+0000","last_became_peered":"2024-09-02T10:14:39.036496+0000","last_unstale":"2024-09-02T10:14:42.047015+0000","last_undegraded":"2024-09-02T10:14:42.047015+0000","last_fullsized":"2024-09-02T10:14:42.047015+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:44.097220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,10,9],"acting":[0,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":338,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-09-02T10:14:42.047335+0000","last_change":"2024-09-02T10:14:35.617172+0000","last_active":"2024-09-02T10:14:42.047335+0000","last_peered":"2024-09-02T10:14:42.047335+0000","last_clean":"2024-09-02T10:14:42.047335+0000","last_became_active":"2024-09-02T10:11:11.472186+0000","last_became_peered":"2024-09-02T10:11:11.472186+0000","last_unstale":"2024-09-02T10:14:42.047335+0000","last_undegraded":"2024-09-02T10:14:42.047335+0000","last_fullsized":"2024-09-02T10:14:42.047335+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.617088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.617088+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:00.397366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,9],"acting":[0,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":28246016,"num_objects":431,"num_object_clones":0,"num_object_copies":1293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":431,"num_write_kb":27584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":99876864,"data_stored":99876864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":431,"ondisk_log_size":431,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":201248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":323016,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":93422136,"statfs":{"total":95995035648,"available":95664267264,"internally_reserved":0,"allocated":3137536,"data_stored":2684468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":327612868},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612789,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":145044,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93600108,"statfs":{"total":95995035648,"available":95846510592,"internally_reserved":0,"allocated":1441792,"data_stored":985172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":147061184},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841721,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":389492,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93355660,"statfs":{"total":95995035648,"available":95596195840,"internally_reserved":0,"allocated":1540096,"data_stored":1095706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":397277632},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070655,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190656,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93554496,"statfs":{"total":95995035648,"available":95799803904,"internally_reserved":0,"allocated":1224704,"data_stored":776120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299588,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":371348,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93373804,"statfs":{"total":95995035648,"available":95614775296,"internally_reserved":0,"allocated":1441792,"data_stored":985172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":378796490},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528520,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":197708,"kb_used_data":4088,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":93547444,"statfs":{"total":95995035648,"available":95792582656,"internally_reserved":0,"allocated":4186112,"data_stored":3733044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":198244807},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724749,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310288,"kb_used_data":4224,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93434864,"statfs":{"total":95995035648,"available":95677300736,"internally_reserved":0,"allocated":4325376,"data_stored":3868756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953681,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149911,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401668,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93343484,"statfs":{"total":95995035648,"available":95583727616,"internally_reserved":0,"allocated":1224704,"data_stored":776120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":410057160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411547,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575072,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":679396,"kb_used_data":323348,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93065756,"statfs":{"total":95995035648,"available":95299334144,"internally_reserved":0,"allocated":331108352,"data_stored":330668459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738598,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353396,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93391756,"statfs":{"total":95995035648,"available":95633158144,"internally_reserved":0,"allocated":1540096,"data_stored":1095706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":35651584,"data_stored":35651584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":31981568,"data_stored":31981568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":7602176,"data_stored":7602176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24641536,"data_stored":24641536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:14:49.343 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-02T10:14:49.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pg_num 51 2024-09-02T10:14:49.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:49 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:14:49.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:49 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:14:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:49 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:14:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:49 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:14:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:49 smithi162 ceph-mon[20279]: from='client.16407 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:14:49.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:49.346 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:49.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:49.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:49.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:49.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:49 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:14:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:49 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:14:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:49 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:14:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:49 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:14:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:49 smithi187 ceph-mon[20072]: from='client.16407 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:14:49.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:49.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:49.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:49.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:49.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:49.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:49.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:49.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:49.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:49.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:49.876 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:14:49.876 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:14:49.876 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:14:49.876 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:14:49.877 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:14:49.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:49.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:49.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:50.091 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:14:50.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:50.115 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:50.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:50.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:50.167 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 4 16 6677 6661 104.071 93 0.0145768 0.00958399 2024-09-02T10:14:50.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:50 smithi151 ceph-mon[16639]: osdmap e438: 12 total, 12 up, 10 in 2024-09-02T10:14:50.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2336087490' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:14:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:50 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:14:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:50 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:14:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:50 smithi151 ceph-mon[16639]: pgmap v1033: 47 pgs: 47 active+clean; 113 MiB data, 3.4 GiB used, 891 GiB / 894 GiB avail; 19 MiB/s wr, 299 op/s 2024-09-02T10:14:50.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:50.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:50.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:50.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:50.366 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:50.367 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:50.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:50.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:50.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:50.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:50 smithi187 ceph-mon[20072]: osdmap e438: 12 total, 12 up, 10 in 2024-09-02T10:14:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2336087490' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:14:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:50 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:14:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:50 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:14:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:50 smithi187 ceph-mon[20072]: pgmap v1033: 47 pgs: 47 active+clean; 113 MiB data, 3.4 GiB used, 891 GiB / 894 GiB avail; 19 MiB/s wr, 299 op/s 2024-09-02T10:14:50.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:50 smithi162 ceph-mon[20279]: osdmap e438: 12 total, 12 up, 10 in 2024-09-02T10:14:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2336087490' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:14:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:50 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:14:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:50 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:14:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:50 smithi162 ceph-mon[20279]: pgmap v1033: 47 pgs: 47 active+clean; 113 MiB data, 3.4 GiB used, 891 GiB / 894 GiB avail; 19 MiB/s wr, 299 op/s 2024-09-02T10:14:50.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:50.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:50.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:50.737 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:50.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:50.751 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:50.752 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:50.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:14:50.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:50.860 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:50.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:50.962 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:50.985 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:50.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:51.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:51.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:51.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:51.167 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 5 16 7706 7690 96.1174 64.3125 0.0402117 0.0102199 2024-09-02T10:14:51.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:51.236 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:51.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:51.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:51.316 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:51 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:51.316 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2336087490' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:14:51.316 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:51 smithi151 ceph-mon[16639]: osdmap e439: 12 total, 12 up, 10 in 2024-09-02T10:14:51.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:51.360 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:51.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:51.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:51.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:51.490 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:51.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:51 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2336087490' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:14:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:51 smithi187 ceph-mon[20072]: osdmap e439: 12 total, 12 up, 10 in 2024-09-02T10:14:51.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:51 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2336087490' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:14:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:51 smithi162 ceph-mon[20279]: osdmap e439: 12 total, 12 up, 10 in 2024-09-02T10:14:51.609 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:14:51.609 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:14:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:51.619 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:51.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:51.637 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:14:51.638 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:14:51.663 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:14:51.663 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:14:51.690 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:14:51.690 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:14:51.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:51.720 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:14:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:14:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:51.744 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:51.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:51.744 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:14:51.745 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:14:51.769 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:14:51.769 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:14:51.793 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:14:51.793 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:14:51.814 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:14:51.814 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:14:51.835 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:14:51.835 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:14:51.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:51.855 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:14:51.855 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:14:51.866 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:51.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:51.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:51.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:51.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:52.083 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:14:52.083 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:14:52.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:52.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:52.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:52.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:52.168 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 6 16 9182 9166 95.4712 92.25 0.00162584 0.0103977 2024-09-02T10:14:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:52 smithi151 ceph-mon[16639]: pgmap v1035: 47 pgs: 47 active+clean; 163 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 27 MiB/s wr, 433 op/s 2024-09-02T10:14:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3605032379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2893913159' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-02T10:14:52.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:52.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:52.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:52.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:52.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:52.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:52.407 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:52 smithi187 ceph-mon[20072]: pgmap v1035: 47 pgs: 47 active+clean; 163 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 27 MiB/s wr, 433 op/s 2024-09-02T10:14:52.407 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3605032379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:52.407 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2893913159' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-02T10:14:52.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:52.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:52 smithi162 ceph-mon[20279]: pgmap v1035: 47 pgs: 47 active+clean; 163 MiB data, 3.6 GiB used, 890 GiB / 894 GiB avail; 27 MiB/s wr, 433 op/s 2024-09-02T10:14:52.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3605032379' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:52.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2893913159' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "51"} : dispatch 2024-09-02T10:14:52.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:52.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:52.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:52.628 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:52.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:52.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:52.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:52.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:52.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:52.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:52.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:52.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:53.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:53.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:53.106 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pg_num 51 < pg_num_max 32 2024-09-02T10:14:53.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:53.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:53.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:53.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:53.168 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 7 16 11101 11085 98.9651 119.938 0.0265398 0.0100896 2024-09-02T10:14:53.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:53.269 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:53.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:53.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:53.372 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:53.397 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:53.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:53.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:53.522 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:53.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:53.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:53.624 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:53.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:53.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:53.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:53.747 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:53.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:53.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:53.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:53.893 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:14:53.893 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:14:53.899 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:53.900 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:54.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:54.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:54.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:54.023 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:54.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:14:54.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:54.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:54.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:54.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:54.168 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 8 16 12344 12328 96.3046 77.6875 0.00371956 0.0103759 2024-09-02T10:14:54.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:54.271 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:54.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:54.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:54.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:54.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:54.400 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:54 smithi151 ceph-mon[16639]: pgmap v1036: 47 pgs: 47 active+clean; 555 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-02T10:14:54.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:54.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:54.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:54.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:14:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:54 smithi162 ceph-mon[20279]: pgmap v1036: 47 pgs: 47 active+clean; 555 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-02T10:14:54.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:54.654 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:54.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:54.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:54 smithi187 ceph-mon[20072]: pgmap v1036: 47 pgs: 47 active+clean; 555 MiB data, 4.7 GiB used, 889 GiB / 894 GiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-02T10:14:54.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:54.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:54.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:54.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:54.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:54.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:55.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:14:55.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:14:55.040 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:55.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:55.073 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:55.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:55.168 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 9 16 13802 13786 95.7283 91.125 0.00886415 0.0104362 2024-09-02T10:14:55.169 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:55.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:55.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:55.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:55.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:55.302 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:55.361 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:14:55.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:55.426 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:55.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:55.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:55.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:55.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:55.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:55.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:55.674 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:55.681 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:55.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:55.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:55.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:55.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:55.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:55.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:55.910 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:14:55.910 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:14:55.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:55.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:56.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:56.065 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:56.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:56.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:56.168 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 10 16 15604 15588 97.417 112.625 0.0105218 0.0102555 2024-09-02T10:14:56.193 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:14:56.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:14:56.197 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:56.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:56.219 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:14:56.220 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:14:56.245 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:14:56.245 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:14:56.270 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:14:56.271 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:14:56.299 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:14:56.299 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:14:56.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:56.324 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:14:56.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:14:56.326 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:56.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:56.342 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:14:56.353 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:14:56.353 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:14:56.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:14:56.384 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:14:56.410 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:14:56.410 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:14:56.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:56.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:14:56.435 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:14:56.455 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:56.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:56.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:56.460 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:14:56.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:14:56.483 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:14:56.483 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:14:56.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:56.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:56.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:56 smithi162 ceph-mon[20279]: pgmap v1037: 47 pgs: 47 active+clean; 614 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:14:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2142948158' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2569960362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:56.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:56 smithi151 ceph-mon[16639]: pgmap v1037: 47 pgs: 47 active+clean; 614 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:14:56.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2142948158' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:56.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2569960362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:56.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:14:56.685 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:56.709 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:56.710 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:56 smithi187 ceph-mon[20072]: pgmap v1037: 47 pgs: 47 active+clean; 614 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:14:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2142948158' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:14:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2569960362' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:14:56.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:56.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:56.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:56.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:14:56.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:56.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:56.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:14:57.001 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:14:57.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:57.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:57.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:57.168 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 11 16 17629 17613 100.066 126.562 0.00473405 0.00997795 2024-09-02T10:14:57.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:57.218 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:57.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:57.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:57.349 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:57.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:57.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:57.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:57.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:57.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:57.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:57.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:14:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:57 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2142948158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:57 smithi162 ceph-mon[20279]: osdmap e440: 12 total, 12 up, 10 in 2024-09-02T10:14:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:57.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:57.596 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:14:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:57 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2142948158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:57 smithi151 ceph-mon[16639]: osdmap e440: 12 total, 12 up, 10 in 2024-09-02T10:14:57.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:57.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:57.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:57.728 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:57 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:14:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2142948158' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:14:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:57 smithi187 ceph-mon[20072]: osdmap e440: 12 total, 12 up, 10 in 2024-09-02T10:14:57.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:14:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:57.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:57.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:14:57.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:57.980 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:57.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:57.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:58.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:58.110 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:58.110 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:58.168 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 12 16 18704 18688 97.3253 67.1875 0.00203914 0.010266 2024-09-02T10:14:58.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:58.237 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:58.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:58.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:14:58.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:58.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:14:58.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:14:58.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:58.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:58.492 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:14:58.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:14:58.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:58 smithi162 ceph-mon[20279]: pgmap v1039: 47 pgs: 47 active+clean; 871 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-02T10:14:58.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:58.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:14:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:58 smithi151 ceph-mon[16639]: pgmap v1039: 47 pgs: 47 active+clean; 871 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-02T10:14:58.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:14:58.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:58.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:14:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:58 smithi187 ceph-mon[20072]: pgmap v1039: 47 pgs: 47 active+clean; 871 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 95 MiB/s wr, 1.52k op/s 2024-09-02T10:14:58.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:58.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:58.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:58.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:58.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:14:59.013 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:59.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:14:59.115 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:59.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:59.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:59.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:59.168 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 13 16 20126 20110 96.6747 88.875 0.0232802 0.0103337 2024-09-02T10:14:59.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:14:59.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:14:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:59.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:14:59.324 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:14:59.325 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:14:59.325 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:14:59.325 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:14:59.325 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:14:59.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:14:59.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:59.393 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:59.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:59.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:59.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:59.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:59.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:14:59.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:14:59.661 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:59.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:14:59.669 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:14:59.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:14:59.789 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:14:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:59.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:14:59.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:14:59.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:14:59.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:14:59.923 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:00.024 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:15:00.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-09-02T10:15:00.025 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:00.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:00.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:59 smithi151 ceph-mon[16639]: osdmap e441: 12 total, 12 up, 10 in 2024-09-02T10:15:00.053 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:14:59 smithi151 ceph-mon[16639]: pgmap v1041: 47 pgs: 47 active+clean; 1.1 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-02T10:15:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:00.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:00.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:15:00.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:00.168 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 14 16 22143 22127 98.7731 126.062 0.00890395 0.0101176 2024-09-02T10:15:00.190 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:00.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:00.228 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:15:00.228 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:15:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:59 smithi187 ceph-mon[20072]: osdmap e441: 12 total, 12 up, 10 in 2024-09-02T10:15:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:14:59 smithi187 ceph-mon[20072]: pgmap v1041: 47 pgs: 47 active+clean; 1.1 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-02T10:15:00.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:15:00.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:15:00.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:59 smithi162 ceph-mon[20279]: osdmap e441: 12 total, 12 up, 10 in 2024-09-02T10:15:00.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:14:59 smithi162 ceph-mon[20279]: pgmap v1041: 47 pgs: 47 active+clean; 1.1 GiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 114 MiB/s wr, 1.83k op/s 2024-09-02T10:15:00.278 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:15:00.279 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:15:00.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:00.301 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:15:00.301 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:15:00.315 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:00.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:00.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:15:00.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:15:00.323 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:00.346 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:15:00.346 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:15:00.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:15:00.368 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:15:00.379 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:15:00.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:15:00.390 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:15:00.390 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:15:00.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:00.421 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:15:00.422 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:15:00.447 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:00.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:00.450 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:15:00.450 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:15:00.476 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:15:00.476 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:15:00.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:15:00.500 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:15:00.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:00.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:00.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:00.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:00.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:00.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:00.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:00.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:00.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:00.825 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:00.915 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:00.915 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:00.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:00.953 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:00.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:01.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:01.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:01.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:01.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3758633599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:01.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 15 16 23308 23292 97.0421 72.8125 0.0108983 0.0102947 2024-09-02T10:15:01.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:01.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:01.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:01.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:01.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:01.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:01.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:01.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3758633599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:01.442 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:01.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:01.470 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3758633599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:01.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:01.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:01.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:01.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:15:01.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:01.719 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:01.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:01.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:01.849 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:01.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:01.882 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:15:01.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:01.976 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:01.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:02.024 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-09-02T10:15:02.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:02.105 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:02.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:02.168 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 16 16 24765 24749 96.6679 91.0625 0.00552425 0.0103377 2024-09-02T10:15:02.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:02.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:02.230 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:02.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:02 smithi162 ceph-mon[20279]: pgmap v1042: 47 pgs: 47 active+clean; 1.1 GiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:15:02.304 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3094309481' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:15:02.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:02.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:02 smithi151 ceph-mon[16639]: pgmap v1042: 47 pgs: 47 active+clean; 1.1 GiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:15:02.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3094309481' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:15:02.356 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:02.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:02.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:02.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:02.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:02.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:02 smithi187 ceph-mon[20072]: pgmap v1042: 47 pgs: 47 active+clean; 1.1 GiB data, 6.5 GiB used, 888 GiB / 894 GiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:15:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3094309481' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:15:02.537 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:02.537 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:02.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:02.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:02.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:02.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:02.740 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:02.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:02.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:02.863 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:02.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:02.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:02.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-09-02T10:15:02.963 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:02.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:02.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:02.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:03.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:03.119 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:03.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:03.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 17 16 25911 25895 95.1943 71.625 0.0199236 0.0104524 2024-09-02T10:15:03.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:03.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:03.243 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:03.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:15:03.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3094309481' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:15:03.273 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:03 smithi162 ceph-mon[20279]: osdmap e442: 12 total, 12 up, 10 in 2024-09-02T10:15:03.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3094309481' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:15:03.279 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:03 smithi151 ceph-mon[16639]: osdmap e442: 12 total, 12 up, 10 in 2024-09-02T10:15:03.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:03.372 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:03.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:03.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:03.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:03.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3094309481' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:15:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:03 smithi187 ceph-mon[20072]: osdmap e442: 12 total, 12 up, 10 in 2024-09-02T10:15:03.566 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:03.570 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:15:03.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:03.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:03.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:03.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:03.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:03.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:03.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:03.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:03.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:03.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:04.009 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:04.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:04.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:04.046 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:15:04.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:15:04.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:04 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:04.090 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:04 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:04.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:04 smithi151 ceph-mon[16639]: osdmap e443: 12 total, 12 up, 10 in 2024-09-02T10:15:04.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:04 smithi151 ceph-mon[16639]: pgmap v1045: 47 pgs: 47 active+clean; 1.5 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-09-02T10:15:04.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:04 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3786164539' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:04.091 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:04 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:04.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:04.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:04.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:04.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 18 16 27196 27180 94.367 80.3125 0.00191664 0.0105639 2024-09-02T10:15:04.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:04.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:04.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:04.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:04 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:04 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:04 smithi162 ceph-mon[20279]: osdmap e443: 12 total, 12 up, 10 in 2024-09-02T10:15:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:04 smithi162 ceph-mon[20279]: pgmap v1045: 47 pgs: 47 active+clean; 1.5 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-09-02T10:15:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:04 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3786164539' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:04.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:04 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:04.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:04.382 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:04.382 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:04.387 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:04.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:04.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:04 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:04 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:04 smithi187 ceph-mon[20072]: osdmap e443: 12 total, 12 up, 10 in 2024-09-02T10:15:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:04 smithi187 ceph-mon[20072]: pgmap v1045: 47 pgs: 47 active+clean; 1.5 GiB data, 7.6 GiB used, 886 GiB / 894 GiB avail; 107 MiB/s wr, 1.72k op/s 2024-09-02T10:15:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:04 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3786164539' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:04 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:04.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:04.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:04.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:04.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:04.641 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:04.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:04.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:04.771 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:04.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:04.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:04.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:04.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:04.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:05.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:05.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:05.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:05.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:05.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:05.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:05.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 19 16 28504 28488 93.7026 81.75 0.0138718 0.0106631 2024-09-02T10:15:05.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:05 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:05 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:15:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:05 smithi151 ceph-mon[16639]: osdmap e444: 12 total, 12 up, 10 in 2024-09-02T10:15:05.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:05.282 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:05.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:05.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:05 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:05 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:15:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:05 smithi162 ceph-mon[20279]: osdmap e444: 12 total, 12 up, 10 in 2024-09-02T10:15:05.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:05.409 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:05.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:05.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:05.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:05 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:05.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:05 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:15:05.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:05 smithi187 ceph-mon[20072]: osdmap e444: 12 total, 12 up, 10 in 2024-09-02T10:15:05.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:05.534 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:05.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:05.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:15:05.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:05.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:05.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:05.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:05.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:05.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:05.783 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:05.857 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:15:05.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:05.908 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:05.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:05.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:06.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:06.030 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:06.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:06.045 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-02T10:15:06.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:06.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:06.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:06.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:06.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:15:06.169129+0000 min lat: 0.00106838 max lat: 0.2766 avg lat: 0.01078 2024-09-02T10:15:06.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:15:06.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 20 16 29587 29571 92.4015 67.6875 0.0210432 0.01078 2024-09-02T10:15:06.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:06.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:06 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:15:06.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:06 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:15:06.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:06 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:15:06.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:06 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:15:06.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3251615907' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:15:06.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:06 smithi162 ceph-mon[20279]: pgmap v1047: 47 pgs: 47 active+clean; 1.5 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:15:06.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:06.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:06.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:06.365 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:06.365 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:06.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:06.399 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:06 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:15:06.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:06 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:15:06.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:06 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:15:06.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:06 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:15:06.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3251615907' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:15:06.400 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:06 smithi151 ceph-mon[16639]: pgmap v1047: 47 pgs: 47 active+clean; 1.5 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:15:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:06.401 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:06.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:06.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:06 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:15:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:06 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:15:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:06 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:15:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:06 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:15:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3251615907' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:15:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:06 smithi187 ceph-mon[20072]: pgmap v1047: 47 pgs: 47 active+clean; 1.5 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:15:06.524 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:06.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:06.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:06.648 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:06.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:06.750 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:06.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:06.772 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:06.873 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:06.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:06.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:06.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:07.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:07.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:07.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3251615907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:15:07.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:07 smithi151 ceph-mon[16639]: osdmap e445: 12 total, 12 up, 10 in 2024-09-02T10:15:07.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:07 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:15:07.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:07 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:15:07.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:07 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:15:07.121 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:07.143 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:07.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:07.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 21 16 29703 29687 88.3464 7.25 0.00218302 0.010839 2024-09-02T10:15:07.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:07.266 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:07.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:07.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:07.319 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3251615907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:15:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:07 smithi162 ceph-mon[20279]: osdmap e445: 12 total, 12 up, 10 in 2024-09-02T10:15:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:07 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:15:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:07 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:15:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:07 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:15:07.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:07.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:07.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:07.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3251615907' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:15:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:07 smithi187 ceph-mon[20072]: osdmap e445: 12 total, 12 up, 10 in 2024-09-02T10:15:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:07 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:15:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:07 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:15:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:07 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:15:07.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:07.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:07.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:15:07.613 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:07.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:07.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:07.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:07.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:07.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:07.795 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:15:07.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:07.864 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:15:07.864 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:15:07.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:07.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:07.887 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:15:07.888 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:15:07.907 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:15:07.907 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:15:07.928 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:15:07.928 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:15:07.957 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:15:07.958 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:15:07.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:07.993 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:15:07.993 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:15:08.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:08.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:08.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:15:08.015 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:15:08.037 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:15:08.037 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:15:08.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:15:08.058 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:15:08.080 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:15:08.080 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:15:08.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:15:08.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:15:08.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:08.123 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:15:08.123 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:15:08.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:08.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:08.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 22 16 29703 29687 84.3303 0 - 0.010839 2024-09-02T10:15:08.209 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: osdmap e446: 12 total, 12 up, 10 in 2024-09-02T10:15:08.209 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: pgmap v1050: 47 pgs: 3 peering, 44 active+clean; 1.7 GiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 54 MiB/s wr, 856 op/s 2024-09-02T10:15:08.210 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:15:08.210 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:15:08.210 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1609254515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:08.210 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:15:08.210 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:15:08.210 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:15:08.210 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:15:08.210 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:15:08.210 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:15:08.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:08.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:08.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:08.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:08.347 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:29967 5.c e447/446 5:3b8342d1:::benchmark_data_smithi187_89405_object29966:head v 447'1795, mlcod=436'6)", 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:08.344733+0000", 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0013973130000000001, 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001423417, 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:08.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.344733+0000", 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.344733+0000", 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.344734+0000", 2024-09-02T10:15:08.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2099999999999996e-07 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.344816+0000", 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1984999999999995e-05 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.344844+0000", 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8078000000000001e-05 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.344851+0000", 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9519999999999997e-06 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.344915+0000", 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3670999999999994e-05 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.344945+0000", 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9940000000000001e-05 2024-09-02T10:15:08.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:29966 5.9 e447/431 5:99822ad3:::benchmark_data_smithi187_89405_object29965:head v 447'1951, mlcod=447'1947)", 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:08.345210+0000", 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00092039200000000002, 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010011040000000001, 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.352 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.345210+0000", 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.345210+0000", 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.345211+0000", 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4199999999999995e-07 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.345233+0000", 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2095000000000001e-05 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.345266+0000", 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2759000000000003e-05 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.345272+0000", 2024-09-02T10:15:08.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6930000000000001e-06 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.345332+0000", 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9525999999999999e-05 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:08.345349+0000", 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7390000000000001e-05 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:08.355 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:08.356 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:08.356 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:15:08.356 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-02T10:15:08.356 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:15:08.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:08.387 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: osdmap e446: 12 total, 12 up, 10 in 2024-09-02T10:15:08.387 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: pgmap v1050: 47 pgs: 3 peering, 44 active+clean; 1.7 GiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 54 MiB/s wr, 856 op/s 2024-09-02T10:15:08.387 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:15:08.387 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:15:08.387 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1609254515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:08.387 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:15:08.388 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:15:08.388 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:15:08.388 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:15:08.388 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:15:08.388 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:15:08.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:08.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:08.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-09-02T10:15:08.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: osdmap e446: 12 total, 12 up, 10 in 2024-09-02T10:15:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: pgmap v1050: 47 pgs: 3 peering, 44 active+clean; 1.7 GiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 54 MiB/s wr, 856 op/s 2024-09-02T10:15:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:15:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:15:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1609254515' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:15:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:15:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:15:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:15:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:15:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:15:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:08.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:08.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:08.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:08.679 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:08.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:08.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:08.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:15:08.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:08.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:08.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:08.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:08.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:08.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:09.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:09.074 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:09.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:15:09.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:09.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 23 16 31588 31572 85.7856 58.9062 0.00740631 0.0116448 2024-09-02T10:15:09.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:09.202 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:09.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:09.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:09.240 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:09.254 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:09.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:09.331 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:09.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:09.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:09 smithi162 ceph-mon[20279]: osdmap e447: 12 total, 12 up, 10 in 2024-09-02T10:15:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:15:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:15:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:15:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:15:09.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:09 smithi151 ceph-mon[16639]: osdmap e447: 12 total, 12 up, 10 in 2024-09-02T10:15:09.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:15:09.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:15:09.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:15:09.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:15:09.424 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:15:09.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:09.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:09.460 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:09 smithi187 ceph-mon[20072]: osdmap e447: 12 total, 12 up, 10 in 2024-09-02T10:15:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:15:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:15:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:15:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:15:09.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:09.583 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:09.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:09.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:09.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:09.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:09.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:09.843 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:09.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:09.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:09.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:09.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:09.983 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:15:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:15:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:15:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:15:09.983 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:15:10.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:10.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:10.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:10.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:10.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 24 15 33135 33120 86.2421 96.75 0.00626524 0.011587 2024-09-02T10:15:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:10 smithi151 ceph-mon[16639]: pgmap v1052: 47 pgs: 1 remapped+peering, 3 peering, 43 active+clean; 1.8 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 59 MiB/s wr, 942 op/s 2024-09-02T10:15:10.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:10.226 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:10.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:10.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:10.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:10.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:10 smithi162 ceph-mon[20279]: pgmap v1052: 47 pgs: 1 remapped+peering, 3 peering, 43 active+clean; 1.8 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 59 MiB/s wr, 942 op/s 2024-09-02T10:15:10.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:10.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:10.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:10.476 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:10.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:10.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:10.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:10 smithi187 ceph-mon[20072]: pgmap v1052: 47 pgs: 1 remapped+peering, 3 peering, 43 active+clean; 1.8 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 59 MiB/s wr, 942 op/s 2024-09-02T10:15:10.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:10.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:10.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:10.674 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:15:10.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:10.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:10.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:10.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:10.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:10.855 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:10.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:10.956 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:10.966 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:15:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:10.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:10.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-02T10:15:11.073 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:15:11.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:11.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:11.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:11.169 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 25 16 34218 34202 85.4972 67.625 0.00270561 0.0116902 2024-09-02T10:15:11.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:15:11.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:11.239 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:11.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:11.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:11.323 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3154296203' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:11.323 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/198354975' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:15:11.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3154296203' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/198354975' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:15:11.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:11.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:11.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:11.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:11.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:11.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3154296203' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/198354975' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:15:11.510 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:15:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:15:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:15:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:15:11.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:75424 3.a 3:58480fd1:::benchmark_data_smithi187_59989_object75423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 491.148890612, 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22582144000000001, 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75424 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.511 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:15:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:15:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359600+0000", 2024-09-02T10:15:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3820000000000001e-06 2024-09-02T10:15:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359629+0000", 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8775000000000001e-05 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359640+0000", 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1151e-05 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359648+0000", 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0620000000000001e-06 2024-09-02T10:15:11.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359688+0000", 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9882000000000002e-05 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359791+0000", 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010327699999999999 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359876+0000", 2024-09-02T10:15:11.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5418000000000001e-05 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.361865+0000", 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019887289999999998 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362104+0000", 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023905199999999999 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585329+0000", 2024-09-02T10:15:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223224432 2024-09-02T10:15:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585374+0000", 2024-09-02T10:15:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4972999999999998e-05 2024-09-02T10:15:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585420+0000", 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6307000000000002e-05 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24976.0:75424 3.a e253/226 ondisk, result = 0)", 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:15:11.518 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 491.14618507199998, 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22312427800000001, 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:15:11.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362305+0000", 2024-09-02T10:15:11.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9100000000000003e-07 2024-09-02T10:15:11.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362309+0000", 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2899999999999996e-06 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585259+0000", 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22295003499999999 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585266+0000", 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6460000000000004e-06 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585311+0000", 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5802e-05 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585321+0000", 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3090000000000001e-06 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585428+0000", 2024-09-02T10:15:11.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010750499999999999 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:155062 3.2 e282/226 3:4fe18cc9:::benchmark_data_smithi187_59989_object45519:head v 282'9762, mlcod=282'9758)", 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 407.03198171499997, 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29158908, 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.523 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476508+0000", 2024-09-02T10:15:11.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1500000000000003e-07 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476514+0000", 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6559999999999997e-06 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.763870+0000", 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28735592300000001 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.763885+0000", 2024-09-02T10:15:11.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4823e-05 2024-09-02T10:15:11.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.764037+0000", 2024-09-02T10:15:11.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015244400000000001 2024-09-02T10:15:11.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.764077+0000", 2024-09-02T10:15:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0371999999999999e-05 2024-09-02T10:15:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.768047+0000", 2024-09-02T10:15:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039701650000000003 2024-09-02T10:15:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.768096+0000", 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8881999999999997e-05 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:181206 3.3 e291/226 3:c5ec1ffd:::benchmark_data_smithi187_59989_object71663:head v 291'11456, mlcod=291'11454)", 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 356.50228602800001, 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37650431099999998, 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.528 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006204+0000", 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6000000000000002e-07 2024-09-02T10:15:11.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006211+0000", 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7880000000000001e-06 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377595+0000", 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.371383927 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377601+0000", 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0769999999999996e-06 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377649+0000", 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8368000000000002e-05 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377669+0000", 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0258000000000002e-05 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.382663+0000", 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049941880000000001 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.382707+0000", 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3844999999999998e-05 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3855 4.a 4:57b917f5:::benchmark_data_smithi187_76542_object3854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:15:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:15:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 245.599172302, 2024-09-02T10:15:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22371366500000001, 2024-09-02T10:15:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:15:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:15:11.533 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3855 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909327+0000", 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9559999999999992e-06 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909360+0000", 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3556999999999997e-05 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909377+0000", 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6283e-05 2024-09-02T10:15:11.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909385+0000", 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4030000000000007e-06 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909444+0000", 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9203000000000002e-05 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.536 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909571+0000", 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127324 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909715+0000", 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143888 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.910145+0000", 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042936599999999998 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.537 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.917291+0000", 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071467290000000001 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132953+0000", 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.215661517 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132989+0000", 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6123e-05 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.538 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.133030+0000", 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1316000000000002e-05 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3871 4.a 4:530be978:::benchmark_data_smithi187_76542_object3870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 245.592396495, 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21808112499999999, 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:15:11.539 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3871 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.540 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1899999999999998e-07 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916113+0000", 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9800999999999999e-05 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916118+0000", 2024-09-02T10:15:11.541 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916121+0000", 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3330000000000001e-06 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916168+0000", 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7147000000000001e-05 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.542 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916291+0000", 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122934 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916427+0000", 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135337 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916951+0000", 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052393700000000004 2024-09-02T10:15:11.543 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924167+0000", 2024-09-02T10:15:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072167580000000002 2024-09-02T10:15:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134113+0000", 2024-09-02T10:15:11.544 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20994539000000001 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134147+0000", 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4149000000000001e-05 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134174+0000", 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6798999999999999e-05 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.545 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3887 4.a 4:5466a52d:::benchmark_data_smithi187_76542_object3886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 245.58673820499999, 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21253008300000001, 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3887 2024-09-02T10:15:11.546 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.547 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921752+0000", 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4160000000000001e-06 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921813+0000", 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0420000000000001e-05 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921822+0000", 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4299999999999995e-06 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.548 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921827+0000", 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9819999999999999e-06 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921867+0000", 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0071000000000003e-05 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921944+0000", 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6533999999999996e-05 2024-09-02T10:15:11.549 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922025+0000", 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1601999999999996e-05 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922360+0000", 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033462200000000002 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924931+0000", 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025713160000000001 2024-09-02T10:15:11.550 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134215+0000", 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209284054 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134247+0000", 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2147999999999998e-05 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134281+0000", 2024-09-02T10:15:11.551 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3488000000000002e-05 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3898 4.a 4:55cc6e50:::benchmark_data_smithi187_76542_object3897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 245.581742273, 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20761331199999999, 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.552 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3898 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.553 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926748+0000", 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6599999999999994e-07 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926786+0000", 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8506e-05 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.554 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926798+0000", 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.221e-05 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926804+0000", 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3140000000000003e-06 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926853+0000", 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8804999999999998e-05 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.555 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926965+0000", 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112256 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927110+0000", 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014520099999999999 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:15:11.556 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927488+0000", 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000378332 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929483+0000", 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019947670000000002 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134309+0000", 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20482557300000001 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.557 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134337+0000", 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8334999999999999e-05 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134360+0000", 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3047000000000001e-05 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3901 4.a 4:53342710:::benchmark_data_smithi187_76542_object3900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:15:11.558 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 245.57991418099999, 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20590491499999999, 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3901 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928576+0000", 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1699999999999997e-07 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928631+0000", 2024-09-02T10:15:11.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5276999999999999e-05 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928642+0000", 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1195999999999999e-05 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928649+0000", 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.669e-06 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928691+0000", 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1868e-05 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928778+0000", 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7416999999999993e-05 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928866+0000", 2024-09-02T10:15:11.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8471e-05 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929233+0000", 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036694499999999998 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.942995+0000", 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013761572999999999 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134423+0000", 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19142793799999999 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134461+0000", 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7839000000000001e-05 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134480+0000", 2024-09-02T10:15:11.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9004999999999998e-05 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:6012 4.5 4:a5fa00f8:::benchmark_data_smithi187_76542_object6011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 244.245481179, 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209785581, 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:15:11.565 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6012 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:15:11.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263009+0000", 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0139999999999999e-06 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263120+0000", 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011099000000000001 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263134+0000", 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4467999999999999e-05 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263141+0000", 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9820000000000002e-06 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263387+0000", 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024598300000000002 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.568 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263500+0000", 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113174 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263599+0000", 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8974999999999993e-05 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264127+0000", 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052785799999999997 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264358+0000", 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023054600000000001 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472691+0000", 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20833321599999999 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472750+0000", 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8897000000000002e-05 2024-09-02T10:15:11.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472793+0000", 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3477999999999999e-05 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:24533 4.c e335/333 4:3a0221c2:::benchmark_data_smithi187_76542_object24532:head v 335'1521, mlcod=335'1517)", 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 226.49744812700001, 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212837002, 2024-09-02T10:15:11.571 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011042+0000", 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1650000000000001e-06 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011129+0000", 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7443999999999998e-05 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222953+0000", 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21182389400000001 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222960+0000", 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8569999999999997e-06 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223010+0000", 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0182000000000002e-05 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223032+0000", 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1699999999999999e-05 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223838+0000", 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080572200000000004 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223878+0000", 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0037999999999999e-05 2024-09-02T10:15:11.575 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:25165 4.c e336/333 4:3c3f6316:::benchmark_data_smithi187_76542_object25164:head v 336'1563, mlcod=335'1560)", 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 225.863218577, 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204647308, 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645271+0000", 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9999999999999996e-07 2024-09-02T10:15:11.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645443+0000", 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017179700000000001 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848893+0000", 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203449884 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848900+0000", 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8360000000000003e-06 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848961+0000", 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1163999999999999e-05 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848982+0000", 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1403000000000002e-05 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849888+0000", 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000905469 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849918+0000", 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9855000000000001e-05 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:52446 4.15 e362/353 4:acabd05d:::benchmark_data_smithi187_76542_object52445:head v 362'2522, mlcod=362'2519)", 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 192.81508921700001, 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.220391959, 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9599999999999999e-07 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693457+0000", 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6582000000000001e-05 2024-09-02T10:15:11.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912812+0000", 2024-09-02T10:15:11.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21935476500000001 2024-09-02T10:15:11.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912819+0000", 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4989999999999997e-06 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912871+0000", 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1388000000000003e-05 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912887+0000", 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5911999999999999e-05 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.913759+0000", 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087215899999999996 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.913792+0000", 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2957999999999999e-05 2024-09-02T10:15:11.585 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:58518 4.1 e366/341 4:8490730e:::benchmark_data_smithi187_76542_object58517:head v 366'2807, mlcod=366'2804)", 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 187.14336098999999, 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209866793, 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.586 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365129+0000", 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7100000000000001e-07 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365181+0000", 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1816e-05 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573805+0000", 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.2086249 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573812+0000", 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1349999999999997e-06 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573866+0000", 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4373000000000003e-05 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573888+0000", 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2008999999999999e-05 2024-09-02T10:15:11.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.574955+0000", 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010673519999999999 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.574995+0000", 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9536999999999998e-05 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:75251 4.11 e371/356 4:8ba31329:::benchmark_data_smithi187_76542_object75250:head v 371'3396, mlcod=371'3394)", 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 173.84303197, 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37157874099999999, 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665458+0000", 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9400000000000004e-07 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665521+0000", 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2685e-05 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035523+0000", 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37000295799999999 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035542+0000", 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8692999999999999e-05 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035709+0000", 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016700999999999999 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035745+0000", 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5913e-05 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.037004+0000", 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001258459 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.037036+0000", 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2228999999999999e-05 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:90191 4.15 e375/353 4:afce227e:::benchmark_data_smithi187_76542_object90190:head v 375'3680, mlcod=375'3677)", 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 160.41024242399999, 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20444974699999999, 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.595 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:15:11.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9299999999999997e-07 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098273+0000", 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5378999999999999e-05 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301201+0000", 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20292787800000001 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301213+0000", 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860000000000001e-05 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301360+0000", 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014643600000000001 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301383+0000", 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2782999999999999e-05 2024-09-02T10:15:11.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302662+0000", 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012796470000000001 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302696+0000", 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3970999999999997e-05 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126679 4.1 e392/341 4:86205e4c:::benchmark_data_smithi187_76542_object34343:head v 392'4951, mlcod=392'4948)", 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 116.04068215300001, 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44507702900000001, 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467808+0000", 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.277e-06 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467816+0000", 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3690000000000007e-06 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910880+0000", 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44306368899999998 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910889+0000", 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0380000000000005e-06 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910948+0000", 2024-09-02T10:15:11.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8498000000000001e-05 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910970+0000", 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2580000000000001e-05 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912839+0000", 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018688120000000001 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912884+0000", 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4765999999999998e-05 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126802 4.1 e392/341 4:81a050df:::benchmark_data_smithi187_76542_object34466:head v 392'4952, mlcod=392'4949)", 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.597638777, 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25345677799999999, 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.604 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910851+0000", 2024-09-02T10:15:11.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0690000000000001e-06 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910859+0000", 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.233e-06 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162387+0000", 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25152825099999998 2024-09-02T10:15:11.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162402+0000", 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4905999999999999e-05 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162538+0000", 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013591800000000001 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162581+0000", 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2991000000000002e-05 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164270+0000", 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016893419999999999 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164307+0000", 2024-09-02T10:15:11.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7067999999999998e-05 2024-09-02T10:15:11.609 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.609 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.609 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.609 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:5213 5.2 5:463fe113:::benchmark_data_smithi187_89405_object5212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e438)", 2024-09-02T10:15:11.609 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:14:49.191418+0000", 2024-09-02T10:15:11.609 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.317070529999999, 2024-09-02T10:15:11.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19806263399999999, 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5213 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.191418+0000", 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.191418+0000", 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.191419+0000", 2024-09-02T10:15:11.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1100000000000004e-07 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.191475+0000", 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5374000000000002e-05 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.191484+0000", 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3030000000000005e-06 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.191491+0000", 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9059999999999996e-06 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.191540+0000", 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9123999999999999e-05 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.191612+0000", 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1450000000000002e-05 2024-09-02T10:15:11.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.191688+0000", 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6333000000000003e-05 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.192105+0000", 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041732599999999998 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.192348+0000", 2024-09-02T10:15:11.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024291100000000001 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.389377+0000", 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.197028389 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.389434+0000", 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7224e-05 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:49.389481+0000", 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7382999999999997e-05 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:18211 5.5 e440/431 5:a447aadf:::benchmark_data_smithi187_89405_object18210:head v 440'1137, mlcod=440'1135)", 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:14:57.748140+0000", 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 13.760348869, 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19505925299999999, 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:11.616 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:57.748140+0000", 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:57.748140+0000", 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:11.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:57.748141+0000", 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.82e-07 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:57.748193+0000", 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2720999999999999e-05 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:57.942209+0000", 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.194015353 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:57.942220+0000", 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1123e-05 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:57.942313+0000", 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3158999999999994e-05 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:57.942345+0000", 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2245e-05 2024-09-02T10:15:11.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:57.943176+0000", 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00083033 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:57.943199+0000", 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3839999999999999e-05 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:11.620 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:11.621 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:15:11.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:11.652 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:11.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:11.653 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:11.754 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:11.780 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:11.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:11.781 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:11.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:11.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:11.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:12.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:12.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:12.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:12 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:12 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:12 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:15:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:12 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:15:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3154296203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:15:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/198354975' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:15:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:12 smithi162 ceph-mon[20279]: osdmap e448: 12 total, 12 up, 10 in 2024-09-02T10:15:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:12 smithi162 ceph-mon[20279]: pgmap v1054: 47 pgs: 1 remapped+peering, 2 peering, 44 active+clean; 1.8 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 60 MiB/s wr, 961 op/s 2024-09-02T10:15:12.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:12.169 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:12.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:12.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 26 16 35739 35723 85.8648 95.0625 0.0114629 0.0116337 2024-09-02T10:15:12.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:12.300 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:12.301 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:12.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:12.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:12.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:12.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:12 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:12 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:12 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:15:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:12 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:15:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3154296203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:15:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/198354975' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:15:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:12 smithi151 ceph-mon[16639]: osdmap e448: 12 total, 12 up, 10 in 2024-09-02T10:15:12.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:12 smithi151 ceph-mon[16639]: pgmap v1054: 47 pgs: 1 remapped+peering, 2 peering, 44 active+clean; 1.8 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 60 MiB/s wr, 961 op/s 2024-09-02T10:15:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:12 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:12 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:12 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:15:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:12 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:15:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3154296203' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:15:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/198354975' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:15:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:12 smithi187 ceph-mon[20072]: osdmap e448: 12 total, 12 up, 10 in 2024-09-02T10:15:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:12 smithi187 ceph-mon[20072]: pgmap v1054: 47 pgs: 1 remapped+peering, 2 peering, 44 active+clean; 1.8 GiB data, 8.9 GiB used, 885 GiB / 894 GiB avail; 60 MiB/s wr, 961 op/s 2024-09-02T10:15:12.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:12.562 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:12.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:12.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:12.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:12.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:12.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:12.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:12.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:12.822 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:12.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:12.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:12.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:12.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:13.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:13.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:13.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:13.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:13.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 27 16 37122 37106 85.8858 86.4375 0.00334921 0.0116375 2024-09-02T10:15:13.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:13.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:13.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:13.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:13.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:13 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:13.333 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:13 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:13.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:13 smithi162 ceph-mon[20279]: osdmap e449: 12 total, 12 up, 10 in 2024-09-02T10:15:13.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:13 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:15:13.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:13 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:15:13.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:13 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:15:13.334 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:13 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:15:13.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:13.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:13.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:13.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:15:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:13 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:13 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:13 smithi151 ceph-mon[16639]: osdmap e449: 12 total, 12 up, 10 in 2024-09-02T10:15:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:13 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:15:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:13 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:15:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:13 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:15:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:13 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:15:13.440 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:13.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:13.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:13.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:13 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:13 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:13 smithi187 ceph-mon[20072]: osdmap e449: 12 total, 12 up, 10 in 2024-09-02T10:15:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:13 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:15:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:13 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:15:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:13 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:15:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:13 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:15:13.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:13.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:13.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:13.690 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:15:13.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:13.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:13.723 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:13.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:13.853 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:13.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:13.854 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:13.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:13.985 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:13.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:13.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:14.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:14.111 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:14.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:14.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:14.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 28 16 38643 38627 86.2133 95.0625 0.00560908 0.0115915 2024-09-02T10:15:14.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:38671 5.9 5:986f3a17:::benchmark_data_smithi187_89405_object38670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:14.222348+0000", 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.011249962000000001, 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011276112, 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:15:14.235 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 38671 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.222348+0000", 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.222348+0000", 2024-09-02T10:15:14.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.222350+0000", 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.468e-06 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.222435+0000", 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5161999999999995e-05 2024-09-02T10:15:14.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.222448+0000", 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3559e-05 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.222455+0000", 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3069999999999999e-06 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:15:14.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.222499+0000", 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4231999999999999e-05 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.222608+0000", 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000109188 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.222704+0000", 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5696e-05 2024-09-02T10:15:14.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.224110+0000", 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014067039999999999 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:14.224681+0000", 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00057040300000000003 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:14.240 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:15:14.241 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:14.241 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:15:14.241 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:15:14.241 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-02T10:15:14.241 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:15:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:14.242 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:14.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:14.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-09-02T10:15:14.270 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:14.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:14 smithi162 ceph-mon[20279]: pgmap v1056: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.2 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s; 1363/105702 objects misplaced (1.289%); 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:15:14.368 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:14.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:14.402 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:14 smithi151 ceph-mon[16639]: pgmap v1056: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.2 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s; 1363/105702 objects misplaced (1.289%); 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:15:14.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:14.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:14.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:14 smithi187 ceph-mon[20072]: pgmap v1056: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.2 GiB data, 9.7 GiB used, 884 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s; 1363/105702 objects misplaced (1.289%); 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:15:14.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:14.625 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:14.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:14.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:14.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:14.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:14.758 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:14.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:14.886 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:14.887 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:14.910 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:15:14.911 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:15:14.911 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:14.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:15:14.940 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:15:14.940 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:15:14.968 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:15:14.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:15:14.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:14.997 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:15:14.997 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:15:15.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:15.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:15.028 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:15:15.028 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:15:15.056 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:15:15.057 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:15:15.087 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:15:15.087 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:15:15.118 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:15:15.119 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:15:15.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:15.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:15.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:15.156 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:15:15.156 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:15:15.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 29 16 40157 40141 86.5031 94.625 0.00917736 0.0115513 2024-09-02T10:15:15.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:15:15.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:15:15.199 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2657988368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:15.211 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:15:15.211 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:15:15.232 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:15:15.233 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:15:15.235 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:15:15.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:15.279 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:15.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:15.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2657988368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:15.380 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:15.409 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:15.410 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2657988368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:15.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:15.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:15.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:15.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:15.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:15.663 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:15.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:15.739 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:15.739 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:15.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:15.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:15.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:15.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:15:15.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:15.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:15.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:15.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:16.025 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:16.054 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:16.055 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:16.095 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:16 smithi151 ceph-mon[16639]: pgmap v1057: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.2 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 58 MiB/s wr, 931 op/s; 1363/108321 objects misplaced (1.258%); 3.6 MiB/s, 56 objects/s recovering 2024-09-02T10:15:16.095 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2245582559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:15:16.102 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-02T10:15:16.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:16.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 30 16 41458 41442 86.3297 81.3125 0.0191608 0.0115645 2024-09-02T10:15:16.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:16.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:16.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:16.310 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:16.311 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:16 smithi162 ceph-mon[20279]: pgmap v1057: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.2 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 58 MiB/s wr, 931 op/s; 1363/108321 objects misplaced (1.258%); 3.6 MiB/s, 56 objects/s recovering 2024-09-02T10:15:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2245582559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:15:16.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:16.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:16.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:16.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:16 smithi187 ceph-mon[20072]: pgmap v1057: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.2 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 58 MiB/s wr, 931 op/s; 1363/108321 objects misplaced (1.258%); 3.6 MiB/s, 56 objects/s recovering 2024-09-02T10:15:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2245582559' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:15:16.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:16.570 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:16.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:16.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:16.698 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:16.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:16.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:16.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:16.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:16.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:16.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:16.950 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:16.951 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:17.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:15:17.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:17.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:17.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:17.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 31 16 42891 42875 86.4337 89.5625 0.00659627 0.0115601 2024-09-02T10:15:17.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:17.207 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:17.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:17.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:17.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:17.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:17.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:17 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:17.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:17 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:17.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2245582559' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:15:17.347 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:17 smithi162 ceph-mon[20279]: osdmap e450: 12 total, 12 up, 10 in 2024-09-02T10:15:17.379 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:15:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:17 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:17 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2245582559' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:15:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:17 smithi151 ceph-mon[16639]: osdmap e450: 12 total, 12 up, 10 in 2024-09-02T10:15:17.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:17.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:17.466 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:17 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:17 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2245582559' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:15:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:17 smithi187 ceph-mon[20072]: osdmap e450: 12 total, 12 up, 10 in 2024-09-02T10:15:17.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:17.590 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:17.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:17.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:17.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-09-02T10:15:17.695 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:17.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:17.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:17.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:17.847 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:17.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:17.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:17.952 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:17.952 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:17.976 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:17.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:17.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:18.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:18.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:18.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:18.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 32 16 44046 44030 85.9882 72.1875 0.0058317 0.011595 2024-09-02T10:15:18.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:18.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:18.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:18.244 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:18 smithi151 ceph-mon[16639]: pgmap v1059: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s; 1363/119238 objects misplaced (1.143%); 4.2 MiB/s, 65 objects/s recovering 2024-09-02T10:15:18.244 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:18 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:18.245 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:18 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:18.285 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:18.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:18.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:18.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:18.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:18.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:18.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:18 smithi187 ceph-mon[20072]: pgmap v1059: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s; 1363/119238 objects misplaced (1.143%); 4.2 MiB/s, 65 objects/s recovering 2024-09-02T10:15:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:18 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:18 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:18.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:18.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:18 smithi162 ceph-mon[20279]: pgmap v1059: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 99 MiB/s wr, 1.58k op/s; 1363/119238 objects misplaced (1.143%); 4.2 MiB/s, 65 objects/s recovering 2024-09-02T10:15:18.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:18 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:18.589 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:18 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:18.604 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:18.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:18.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:18.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:15:18.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:18.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:18.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:18.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:18.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:18.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:18.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:18.991 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:19.009 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:19.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:19.010 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:19.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:19.118 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:15:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:19.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:19.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:19.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 33 16 45336 45320 85.8255 80.625 0.00778955 0.0116446 2024-09-02T10:15:19.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:19.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:15:19.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:19.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:19.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:19.398 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:19.399 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:19.413 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:19 smithi187 ceph-mon[20072]: osdmap e451: 12 total, 12 up, 10 in 2024-09-02T10:15:19.414 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3520848638' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:19.414 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:19 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:19.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:19 smithi151 ceph-mon[16639]: osdmap e451: 12 total, 12 up, 10 in 2024-09-02T10:15:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3520848638' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:19 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:19.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:19.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:19.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:19.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:19.561 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:19.561 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:19.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:19 smithi162 ceph-mon[20279]: osdmap e451: 12 total, 12 up, 10 in 2024-09-02T10:15:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3520848638' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:19 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:19.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:19.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:19.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:19.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:19.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:19.788 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:19.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:19.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:19.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:20.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:20.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:20.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:20.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:20.122 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-02T10:15:20.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:20.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 34 16 46449 46433 85.347 69.5625 0.0113171 0.0117103 2024-09-02T10:15:20.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:20.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:20.176 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:20.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:20 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:15:20.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:20 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:15:20.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:20 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:20.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1479114050' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:15:20.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:20 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:15:20.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:20 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:15:20.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:20 smithi187 ceph-mon[20072]: osdmap e452: 12 total, 12 up, 10 in 2024-09-02T10:15:20.269 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:20 smithi187 ceph-mon[20072]: pgmap v1062: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s; 811/128358 objects misplaced (0.632%); 5.8 MiB/s, 92 objects/s recovering 2024-09-02T10:15:20.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:20.310 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:20.311 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:20.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:15:20.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:20.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:20 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:15:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:20 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:15:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:20 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1479114050' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:15:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:20 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:15:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:20 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:15:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:20 smithi151 ceph-mon[16639]: osdmap e452: 12 total, 12 up, 10 in 2024-09-02T10:15:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:20 smithi151 ceph-mon[16639]: pgmap v1062: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s; 811/128358 objects misplaced (0.632%); 5.8 MiB/s, 92 objects/s recovering 2024-09-02T10:15:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:20.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:20.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:20.579 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:20.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:20.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:20 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:15:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:20 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:15:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:20 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1479114050' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:15:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:20 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:15:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:20 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:15:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:20 smithi162 ceph-mon[20279]: osdmap e452: 12 total, 12 up, 10 in 2024-09-02T10:15:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:20 smithi162 ceph-mon[20279]: pgmap v1062: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 79 MiB/s wr, 1.26k op/s; 811/128358 objects misplaced (0.632%); 5.8 MiB/s, 92 objects/s recovering 2024-09-02T10:15:20.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:20.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:20.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:20.723 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:20.727 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:20.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:20.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:20.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:20.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:20.977 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:20.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:20.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:21.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:21.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:21.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:21.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 35 16 47651 47635 85.0548 75.125 0.0104131 0.0117469 2024-09-02T10:15:21.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:21.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:21.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:21.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:21.338 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:21.338 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:21.361 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:21.361 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:21.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:21 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:15:21.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:21 smithi151 ceph-mon[16639]: osdmap e453: 12 total, 12 up, 10 in 2024-09-02T10:15:21.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:21.483 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:21.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:21 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:15:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:21 smithi187 ceph-mon[20072]: osdmap e453: 12 total, 12 up, 10 in 2024-09-02T10:15:21.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:21.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:21 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:15:21.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:21 smithi162 ceph-mon[20279]: osdmap e453: 12 total, 12 up, 10 in 2024-09-02T10:15:21.610 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:21.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:21.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:21.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:21.735 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:21.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:21.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:21.837 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:21.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:21.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:21.865 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:21.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:21.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:21.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:21.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:22.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:15:22.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:22.117 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:22.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:22.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 36 15 48801 48786 84.6903 71.9375 0.00332612 0.0118018 2024-09-02T10:15:22.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:22.245 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:22.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:22.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:22.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:22.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:22.407 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:22 smithi151 ceph-mon[16639]: osdmap e454: 12 total, 12 up, 10 in 2024-09-02T10:15:22.408 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:22 smithi151 ceph-mon[16639]: pgmap v1065: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 61 MiB/s wr, 969 op/s; 811/130878 objects misplaced (0.620%); 8.6 MiB/s, 137 objects/s recovering 2024-09-02T10:15:22.453 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:22.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:22.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:22 smithi162 ceph-mon[20279]: osdmap e454: 12 total, 12 up, 10 in 2024-09-02T10:15:22.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:22 smithi162 ceph-mon[20279]: pgmap v1065: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 61 MiB/s wr, 969 op/s; 811/130878 objects misplaced (0.620%); 8.6 MiB/s, 137 objects/s recovering 2024-09-02T10:15:22.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:22.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:22 smithi187 ceph-mon[20072]: osdmap e454: 12 total, 12 up, 10 in 2024-09-02T10:15:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:22 smithi187 ceph-mon[20072]: pgmap v1065: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 61 MiB/s wr, 969 op/s; 811/130878 objects misplaced (0.620%); 8.6 MiB/s, 137 objects/s recovering 2024-09-02T10:15:22.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:22.630 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:22.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:22.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:22.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:22.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:22.762 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:22.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:22.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:22.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:22.960 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:22.960 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:22.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:23.008 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:23.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:23.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:23.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:23.149 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:23.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:23.165 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:23.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-09-02T10:15:23.170 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 37 16 50070 50054 84.5431 79.25 0.00474289 0.0118213 2024-09-02T10:15:23.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:23.271 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:23.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:23.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:23.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:23.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:23.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:23.519 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:23.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:23.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:23.642 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:23.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:23.711 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:23.711 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:15:23.712 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:15:23.735 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:15:23.735 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:15:23.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:23.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:15:23.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:15:23.766 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:23.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:23.780 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:15:23.780 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:15:23.802 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:15:23.802 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:15:23.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:15:23.823 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:15:23.843 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:15:23.843 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:15:23.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:15:23.866 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:15:23.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:23.887 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:15:23.887 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:15:23.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:23.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:23.908 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:15:23.908 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:15:23.928 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:15:23.929 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:15:23.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:15:23.949 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:15:23.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:24.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:24.014 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:24.115 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:24.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:24.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:24.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 38 16 50254 50238 82.6209 11.5 0.00376817 0.0120911 2024-09-02T10:15:24.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:24.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:24.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:24.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:24.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:24 smithi151 ceph-mon[16639]: osdmap e455: 12 total, 12 up, 10 in 2024-09-02T10:15:24.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:24 smithi151 ceph-mon[16639]: pgmap v1067: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s; 12 MiB/s, 194 objects/s recovering 2024-09-02T10:15:24.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3073024517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:24.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:24.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:24.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:24.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:15:24.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:24 smithi187 ceph-mon[20072]: osdmap e455: 12 total, 12 up, 10 in 2024-09-02T10:15:24.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:24 smithi187 ceph-mon[20072]: pgmap v1067: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s; 12 MiB/s, 194 objects/s recovering 2024-09-02T10:15:24.433 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3073024517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:24.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:24.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:24.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:24 smithi162 ceph-mon[20279]: osdmap e455: 12 total, 12 up, 10 in 2024-09-02T10:15:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:24 smithi162 ceph-mon[20279]: pgmap v1067: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 85 MiB/s wr, 1.36k op/s; 12 MiB/s, 194 objects/s recovering 2024-09-02T10:15:24.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3073024517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:24.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:24.658 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:24.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:24.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:24.786 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:24.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:24.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:15:24.889 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:24.902 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:24.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:24.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:24.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:25.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:25.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:25.052 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:25.145 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-02T10:15:25.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:25.164 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:25.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 39 16 51815 51799 83.0035 97.5625 0.00652484 0.0120402 2024-09-02T10:15:25.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:25.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:25.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:25.314 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:25.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:25.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:25.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:25.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:25 smithi187 ceph-mon[20072]: osdmap e456: 12 total, 12 up, 10 in 2024-09-02T10:15:25.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2397119605' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:15:25.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:25 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:15:25.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:25 smithi151 ceph-mon[16639]: osdmap e456: 12 total, 12 up, 10 in 2024-09-02T10:15:25.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2397119605' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:15:25.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:25 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:15:25.445 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:25.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:25.446 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:25.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:25.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:25.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:25.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:25.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:25 smithi162 ceph-mon[20279]: osdmap e456: 12 total, 12 up, 10 in 2024-09-02T10:15:25.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2397119605' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:15:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:25 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:15:25.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:25.694 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:25.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:25.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:25.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:25.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:25.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:25.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:25.952 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:25.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:25.963 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:52783 5.e 5:75e8219b:::benchmark_data_smithi187_89405_object52782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e457)", 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:25.958971+0000", 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0020385160000000002, 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002512827, 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:25.964 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 52783 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.958971+0000", 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.958971+0000", 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:25.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.958972+0000", 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0759999999999999e-06 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.958992+0000", 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0879000000000001e-05 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959010+0000", 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7119999999999999e-05 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959017+0000", 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3799999999999996e-06 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.960052+0000", 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001035331 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.960174+0000", 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012195 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,8", 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.960290+0000", 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116116 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:25.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:52784 5.b 5:d903c985:::benchmark_data_smithi187_89405_object52783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e457)", 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:25.959343+0000", 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.001665655, 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022012080000000001, 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 52784 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.969 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959343+0000", 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959343+0000", 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:25.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959344+0000", 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5899999999999995e-07 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959366+0000", 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1671999999999999e-05 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959480+0000", 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011386499999999999 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959487+0000", 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8390000000000001e-06 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959586+0000", 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9092000000000006e-05 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959769+0000", 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018373700000000001 2024-09-02T10:15:25.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959900+0000", 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000130648 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:52785 5.b 5.8f07423b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e457)", 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:25.959510+0000", 2024-09-02T10:15:25.973 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0014993199999999999, 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021000989999999998, 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 52785 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959510+0000", 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:25.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959510+0000", 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959510+0000", 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1099999999999995e-07 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:25.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959530+0000", 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9477e-05 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959538+0000", 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8439999999999998e-06 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.959543+0000", 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9880000000000004e-06 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:52787 5.e 5.9e15b4fe (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e457)", 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:25.961221+0000", 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 4294967295.9997883, 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00043800399999999998, 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 52787 2024-09-02T10:15:25.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.961221+0000", 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.961221+0000", 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.961222+0000", 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.164e-06 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.961241+0000", 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9477e-05 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.961253+0000", 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1897e-05 2024-09-02T10:15:25.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:25.961260+0000", 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9720000000000003e-06 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 4 2024-09-02T10:15:25.980 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:15:26.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:26.077 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:26.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:26.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:26.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-09-02T10:15:26.153 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:15:26.171 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:15:26.171122+0000 min lat: 0.00106838 max lat: 2.22758 avg lat: 0.012053 2024-09-02T10:15:26.171 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:15:26.171 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 40 16 53093 53077 82.9251 79.875 0.019492 0.012053 2024-09-02T10:15:26.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:26.208 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:26.209 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:26.310 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:26.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:26.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:26.384 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:26 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:15:26.384 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:26 smithi151 ceph-mon[16639]: osdmap e457: 12 total, 12 up, 10 in 2024-09-02T10:15:26.384 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:26 smithi151 ceph-mon[16639]: pgmap v1070: 47 pgs: 1 creating+peering, 46 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 88 MiB/s wr, 1.41k op/s; 12 MiB/s, 194 objects/s recovering 2024-09-02T10:15:26.384 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1965436751' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:26.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:26 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:15:26.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:26 smithi187 ceph-mon[20072]: osdmap e457: 12 total, 12 up, 10 in 2024-09-02T10:15:26.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:26 smithi187 ceph-mon[20072]: pgmap v1070: 47 pgs: 1 creating+peering, 46 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 88 MiB/s wr, 1.41k op/s; 12 MiB/s, 194 objects/s recovering 2024-09-02T10:15:26.430 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1965436751' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:26.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:26.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:26.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:26.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:26.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:26 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:15:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:26 smithi162 ceph-mon[20279]: osdmap e457: 12 total, 12 up, 10 in 2024-09-02T10:15:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:26 smithi162 ceph-mon[20279]: pgmap v1070: 47 pgs: 1 creating+peering, 46 active+clean; 3.0 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 88 MiB/s wr, 1.41k op/s; 12 MiB/s, 194 objects/s recovering 2024-09-02T10:15:26.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1965436751' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:26.596 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:26.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:26.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:15:26.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:26.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:26.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:26.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:26.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:26.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:26.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:15:26.889 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:26.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:26.969 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:26.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:26.985 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:27.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:27.091 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:27.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:27.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:27 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:27.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1965436751' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:15:27.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:27 smithi151 ceph-mon[16639]: osdmap e458: 12 total, 12 up, 10 in 2024-09-02T10:15:27.171 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 41 16 54132 54116 82.4863 64.9375 0.117024 0.0121134 2024-09-02T10:15:27.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:27.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:27.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:27.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:27.339 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:27.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:27.390 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:27 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:27.390 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1965436751' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:15:27.390 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:27 smithi187 ceph-mon[20072]: osdmap e458: 12 total, 12 up, 10 in 2024-09-02T10:15:27.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:27.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:27.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:27.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:27.486 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:27.534 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:15:27.535 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:15:27.535 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:15:27.535 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:15:27.535 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:15:27.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:27.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:27 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:27.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1965436751' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:15:27.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:27 smithi162 ceph-mon[20279]: osdmap e458: 12 total, 12 up, 10 in 2024-09-02T10:15:27.605 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:27.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:27.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:27.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:27.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:27.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:27.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:27.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:27.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:27.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:27.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:27.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:27.985 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:27.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:28.067 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:15:28.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:15:28.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:28.091 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:15:28.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:15:28.112 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:28.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:28.113 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:28.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:15:28.116 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:15:28.138 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:15:28.138 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:15:28.165 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:15:28.165 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:15:28.165 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-09-02T10:15:28.171 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 42 16 55270 55254 82.2156 71.125 0.00730847 0.0121555 2024-09-02T10:15:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: pgmap v1072: 47 pgs: 1 creating+peering, 46 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 52 MiB/s wr, 825 op/s 2024-09-02T10:15:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/493914059' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:15:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:15:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=180854463,config=2353167246,health=603722711,kv=853135788}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=1995160670,logm=1723682647}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2669023058}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=475651330}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1034779663}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3423425374}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54376247}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3804616831}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175577981}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=519503402}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3865492210}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4249675416' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393550209}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131081676}) 2024-09-02T10:15:28.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3282269535}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028664213}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887007085}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=202664029}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=244134159}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318801996}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3966661474}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=569944808}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705403301}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=688956402}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1423789428}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021865979}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1675484431}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1120646216}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2582483399}) 2024-09-02T10:15:28.190 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137416019}) 2024-09-02T10:15:28.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2645412490}) 2024-09-02T10:15:28.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2728761551}) 2024-09-02T10:15:28.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1771482702}) 2024-09-02T10:15:28.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328893496}) 2024-09-02T10:15:28.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1793414356}) 2024-09-02T10:15:28.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3606522877}) 2024-09-02T10:15:28.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2695593771}) 2024-09-02T10:15:28.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4037211467}) 2024-09-02T10:15:28.191 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3168653237}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1157630798}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273668802}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277900412}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072642004}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053132881}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1458154925}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692857350}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=909810973}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=2643223816,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=366209522}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=11,mon_config_key=89} crc {mgrstat=768194395,mon_config_key=1253762902}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=90438608}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3161779108}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=2,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=80} crc {mon_config_key=295492351,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=2694930058,osdmap=4260863667}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2844983920}) 2024-09-02T10:15:28.192 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=32} crc {osdmap=2897088474}) 2024-09-02T10:15:28.193 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:15:28.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:15:28.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:28.219 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:15:28.219 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:15:28.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:28.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:28.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:28.242 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:15:28.242 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:15:28.255 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: pgmap v1072: 47 pgs: 1 creating+peering, 46 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 52 MiB/s wr, 825 op/s 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/493914059' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=180854463,config=2353167246,health=603722711,kv=853135788}) 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=1995160670,logm=1723682647}) 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2669023058}) 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=475651330}) 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1034779663}) 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3423425374}) 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54376247}) 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3804616831}) 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175577981}) 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=519503402}) 2024-09-02T10:15:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3865492210}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4249675416' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393550209}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131081676}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3282269535}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028664213}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887007085}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=202664029}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=244134159}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318801996}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3966661474}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=569944808}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705403301}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=688956402}) 2024-09-02T10:15:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1423789428}) 2024-09-02T10:15:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021865979}) 2024-09-02T10:15:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1675484431}) 2024-09-02T10:15:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1120646216}) 2024-09-02T10:15:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2582483399}) 2024-09-02T10:15:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137416019}) 2024-09-02T10:15:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2645412490}) 2024-09-02T10:15:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2728761551}) 2024-09-02T10:15:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1771482702}) 2024-09-02T10:15:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328893496}) 2024-09-02T10:15:28.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1793414356}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3606522877}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2695593771}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4037211467}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3168653237}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1157630798}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273668802}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277900412}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072642004}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053132881}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1458154925}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692857350}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=909810973}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=2643223816,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=366209522}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=11,mon_config_key=89} crc {mgrstat=768194395,mon_config_key=1253762902}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=90438608}) 2024-09-02T10:15:28.259 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3161779108}) 2024-09-02T10:15:28.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=2,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=80} crc {mon_config_key=295492351,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=2694930058,osdmap=4260863667}) 2024-09-02T10:15:28.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2844983920}) 2024-09-02T10:15:28.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=32} crc {osdmap=2897088474}) 2024-09-02T10:15:28.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:15:28.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:15:28.288 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:15:28.288 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:15:28.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:15:28.310 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:15:28.311 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:15:28.336 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:15:28.336 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:15:28.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:28.363 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:28.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:28.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:28.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:28.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:28.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:28.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: pgmap v1072: 47 pgs: 1 creating+peering, 46 active+clean; 3.2 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 52 MiB/s wr, 825 op/s 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/493914059' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=56} crc {auth=180854463,config=2353167246,health=603722711,kv=853135788}) 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=1995160670,logm=1723682647}) 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2669023058}) 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=475651330}) 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1034779663}) 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3423425374}) 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54376247}) 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3804616831}) 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175577981}) 2024-09-02T10:15:28.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=519503402}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3865492210}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4249675416' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393550209}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131081676}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3282269535}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4028664213}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3887007085}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=202664029}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=244134159}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=318801996}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3966661474}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=569944808}) 2024-09-02T10:15:28.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=705403301}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=688956402}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1423789428}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3021865979}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1675484431}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1120646216}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2582483399}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137416019}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2645412490}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2728761551}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1771482702}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3328893496}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1793414356}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3606522877}) 2024-09-02T10:15:28.598 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2695593771}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4037211467}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3168653237}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1157630798}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1273668802}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277900412}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4072642004}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2053132881}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1458154925}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692857350}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=909810973}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=2643223816,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=366209522}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=11,mon_config_key=89} crc {mgrstat=768194395,mon_config_key=1253762902}) 2024-09-02T10:15:28.599 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=90438608}) 2024-09-02T10:15:28.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3161779108}) 2024-09-02T10:15:28.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=2,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=80} crc {mon_config_key=295492351,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=2694930058,osdmap=4260863667}) 2024-09-02T10:15:28.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2844983920}) 2024-09-02T10:15:28.600 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=32} crc {osdmap=2897088474}) 2024-09-02T10:15:28.617 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:28.623 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:28.624 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:28.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:28.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:28.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:28.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:28.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:28.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:28.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:28.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:15:28.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-09-02T10:15:28.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:29.007 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:29.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:29.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:29.127 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:29.127 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:29.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:29.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:29.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:29.171 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 43 16 56217 56201 81.68 59.1875 0.00616149 0.0122348 2024-09-02T10:15:29.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:29 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:15:29.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:29 smithi151 ceph-mon[16639]: osdmap e459: 12 total, 12 up, 10 in 2024-09-02T10:15:29.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:29 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:15:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:29 smithi187 ceph-mon[20072]: osdmap e459: 12 total, 12 up, 10 in 2024-09-02T10:15:29.264 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:29.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:29.265 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:29.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:29.395 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:29.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:29.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:29.525 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:29.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:29.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:29 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:15:29.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:29 smithi162 ceph-mon[20279]: osdmap e459: 12 total, 12 up, 10 in 2024-09-02T10:15:29.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:29.696 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:29.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:29.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:29.740 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:29.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:29.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:29.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:29.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:29.944 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:29.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:29.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:30.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:30.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:30.072 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:30.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:30.171 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 44 16 57537 57521 81.6985 82.5 0.00894426 0.0122342 2024-09-02T10:15:30.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:30.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:30.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:30.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:30.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:30.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:30.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:30.388 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:30 smithi151 ceph-mon[16639]: osdmap e460: 12 total, 12 up, 10 in 2024-09-02T10:15:30.388 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:30 smithi151 ceph-mon[16639]: pgmap v1075: 47 pgs: 1 peering, 46 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s; 15 KiB/s, 0 objects/s recovering 2024-09-02T10:15:30.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:30.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:30.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:30 smithi187 ceph-mon[20072]: osdmap e460: 12 total, 12 up, 10 in 2024-09-02T10:15:30.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:30 smithi187 ceph-mon[20072]: pgmap v1075: 47 pgs: 1 peering, 46 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s; 15 KiB/s, 0 objects/s recovering 2024-09-02T10:15:30.547 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:15:30.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:30.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:30.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:30.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:30.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:30 smithi162 ceph-mon[20279]: osdmap e460: 12 total, 12 up, 10 in 2024-09-02T10:15:30.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:30 smithi162 ceph-mon[20279]: pgmap v1075: 47 pgs: 1 peering, 46 active+clean; 3.3 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 77 MiB/s wr, 1.23k op/s; 15 KiB/s, 0 objects/s recovering 2024-09-02T10:15:30.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:30.718 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:30.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:30.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:30.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:30.855 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:30.904 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:30.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:30.984 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:30.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:30.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:31.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:31.109 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:31.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:31.171 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 45 16 58880 58864 81.7481 83.9375 0.00321507 0.0122251 2024-09-02T10:15:31.176 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:15:31.178 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-09-02T10:15:31.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:31.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:31.241 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:31.291 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4036992648' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:15:31.291 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:31 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:15:31.291 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2360284761' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:31.291 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:31 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:31.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:31.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:31.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4036992648' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:15:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:31 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:15:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2360284761' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:31 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:31.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:31.512 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:31.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4036992648' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:15:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:31 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:15:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2360284761' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:31 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:15:31.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:31.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:15:31.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:31.638 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:31.638 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:31.670 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:31.670 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:31.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:31.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:31.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:31.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:31.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:31.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:31.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:32.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:32.022 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:32.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:32.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:32.152 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:32.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:32.171 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 46 16 59986 59970 81.4735 69.125 0.0129472 0.0122688 2024-09-02T10:15:32.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:32.279 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:32.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:32.372 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:15:32.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:32.406 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:32.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:32.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:32.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:32 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:32 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:15:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:32 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:15:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:32 smithi151 ceph-mon[16639]: osdmap e461: 12 total, 12 up, 10 in 2024-09-02T10:15:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:32 smithi151 ceph-mon[16639]: pgmap v1077: 47 pgs: 1 peering, 46 active+clean; 3.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 12 KiB/s, 0 objects/s recovering 2024-09-02T10:15:32.442 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:32 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:32.442 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:32 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:15:32.442 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:32 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:15:32.442 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:32 smithi187 ceph-mon[20072]: osdmap e461: 12 total, 12 up, 10 in 2024-09-02T10:15:32.442 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:32 smithi187 ceph-mon[20072]: pgmap v1077: 47 pgs: 1 peering, 46 active+clean; 3.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 12 KiB/s, 0 objects/s recovering 2024-09-02T10:15:32.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:32.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:32.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:32.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-09-02T10:15:32.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:32 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:32 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:15:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:32 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:15:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:32 smithi162 ceph-mon[20279]: osdmap e461: 12 total, 12 up, 10 in 2024-09-02T10:15:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:32 smithi162 ceph-mon[20279]: pgmap v1077: 47 pgs: 1 peering, 46 active+clean; 3.4 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 12 KiB/s, 0 objects/s recovering 2024-09-02T10:15:32.600 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:32.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:32.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:32.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:32.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:32.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:32.787 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:32.792 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:32.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:32.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:32.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:32.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:33.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:33.059 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:33.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:33.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:33.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 47 15 61457 61442 81.6973 92 0.00772625 0.0122348 2024-09-02T10:15:33.174 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:15:33.174 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:15:33.181 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:33.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:33.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:33.212 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:15:33.213 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:15:33.238 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:15:33.239 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:15:33.261 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:15:33.261 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:15:33.286 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:15:33.286 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:15:33.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:33.313 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:15:33.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:15:33.323 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:33.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:33.341 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:15:33.341 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:15:33.367 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:33.368 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:33.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:15:33.377 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:15:33.415 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:15:33.416 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:15:33.425 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:33.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:33 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:15:33.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:33 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:15:33.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2601886650' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:33.446 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:15:33.446 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:15:33.453 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:33.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:33.477 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:15:33.477 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:15:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:33 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:15:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:33 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:15:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2601886650' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:33.537 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:15:33.537 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:15:33.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:33.583 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:33.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:33.584 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:33 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:15:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:33 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:15:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2601886650' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:33.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:33.710 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:33.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:33.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:33.845 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:33.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:33.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:33.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:33.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:34.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:15:34.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:34.097 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:34.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:34.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:15:34.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 48 16 62737 62721 81.6605 79.9375 0.00381313 0.0122369 2024-09-02T10:15:34.196 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-09-02T10:15:34.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:34.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:34.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:34.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:34.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:34.330 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:34.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:34.350 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:34.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:34 smithi151 ceph-mon[16639]: pgmap v1078: 47 pgs: 1 peering, 46 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 78 MiB/s wr, 1.24k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-02T10:15:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2958330226' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:15:34.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:34.453 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:34 smithi187 ceph-mon[20072]: pgmap v1078: 47 pgs: 1 peering, 46 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 78 MiB/s wr, 1.24k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-02T10:15:34.453 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2958330226' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:15:34.473 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:34.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:34.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:34 smithi162 ceph-mon[20279]: pgmap v1078: 47 pgs: 1 peering, 46 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 78 MiB/s wr, 1.24k op/s; 11 KiB/s, 0 objects/s recovering 2024-09-02T10:15:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2958330226' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:15:34.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:34.598 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:34.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:34.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:34.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:34.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:34.845 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:34.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:34.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:34.883 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:15:34.883 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:15:34.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:15:34.883 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:15:34.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.883 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:25387 4.a e336/333 4:51410dd3:::benchmark_data_smithi187_76542_object25386:head v 336'1582, mlcod=325'647)", 2024-09-02T10:15:34.883 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:15:34.883 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 248.91667763500001, 2024-09-02T10:15:34.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20680125099999999, 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:15:34.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964244+0000", 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5900000000000005e-07 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964284+0000", 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0487e-05 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169804+0000", 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20551941600000001 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169809+0000", 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6319999999999996e-06 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169867+0000", 2024-09-02T10:15:34.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7938e-05 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169895+0000", 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7430999999999999e-05 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.171002+0000", 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011072790000000001 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.171044+0000", 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2109e-05 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:44087 4.7 4:e44e1657:::benchmark_data_smithi187_76542_object44086:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:50.597202+0000", 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 224.28371883599999, 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20155887, 2024-09-02T10:15:34.888 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 44087 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597202+0000", 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597202+0000", 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597203+0000", 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1969999999999999e-06 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597224+0000", 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1421999999999999e-05 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597236+0000", 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2155e-05 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597244+0000", 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6410000000000002e-06 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597293+0000", 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8990999999999999e-05 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597387+0000", 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3700000000000001e-05 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:15:34.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.597478+0000", 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1479999999999998e-05 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.598211+0000", 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00073308599999999998 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.598723+0000", 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00051141900000000004 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.798694+0000", 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19997105900000001 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.798730+0000", 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6631999999999998e-05 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:50.798760+0000", 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0088000000000001e-05 2024-09-02T10:15:34.894 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103692 4.19 4.40dd9139 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 168.785177245, 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.993593167, 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:15:34.895 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103692 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095744+0000", 2024-09-02T10:15:34.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6899999999999996e-07 2024-09-02T10:15:34.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095748+0000", 2024-09-02T10:15:34.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4499999999999997e-06 2024-09-02T10:15:34.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.898 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095758+0000", 2024-09-02T10:15:34.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2140000000000002e-06 2024-09-02T10:15:34.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095764+0000", 2024-09-02T10:15:34.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5690000000000001e-06 2024-09-02T10:15:34.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095808+0000", 2024-09-02T10:15:34.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3588999999999999e-05 2024-09-02T10:15:34.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095814+0000", 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2299999999999996e-06 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089305+0000", 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99349078599999996 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089336+0000", 2024-09-02T10:15:34.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.146e-05 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:103721 4.e e384/340 4:78ce6951:::benchmark_data_smithi187_76542_object11385:head v 384'6547, mlcod=384'6544)", 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 168.48912727800001, 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.265184527, 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.901 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391794+0000", 2024-09-02T10:15:34.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4000000000000001e-07 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391799+0000", 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2009999999999998e-06 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654392+0000", 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26259298599999997 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654398+0000", 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7880000000000003e-06 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654446+0000", 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8183000000000003e-05 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654470+0000", 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4093000000000001e-05 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.656943+0000", 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024726489999999999 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.656978+0000", 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4887e-05 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.905 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103725 4.19 4.a40a4399 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 168.44443259900001, 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65288606400000004, 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103725 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436489+0000", 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2750000000000001e-06 2024-09-02T10:15:34.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436494+0000", 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4660000000000001e-06 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436506+0000", 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2123000000000001e-05 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436512+0000", 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6749999999999996e-06 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436566+0000", 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3285000000000002e-05 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436572+0000", 2024-09-02T10:15:34.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6019999999999996e-06 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089367+0000", 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65279432500000001 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089374+0000", 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3130000000000001e-06 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103743 4.19 4.222eb379 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 168.21159279, 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42007966200000002, 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:34.911 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103743 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669329+0000", 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.221e-06 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669334+0000", 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1340000000000003e-06 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669346+0000", 2024-09-02T10:15:34.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2155e-05 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669351+0000", 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9810000000000003e-06 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669433+0000", 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1617000000000001e-05 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669445+0000", 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.207e-05 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089400+0000", 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41995552600000002 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089407+0000", 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9580000000000001e-06 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.915 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103767 4.19 4.bc594c79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 168.11113841299999, 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31965745400000001, 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:34.916 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103767 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769785+0000", 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5299999999999999e-06 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769796+0000", 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.164e-05 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769823+0000", 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7055000000000001e-05 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769828+0000", 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0350000000000001e-06 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769891+0000", 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2973999999999997e-05 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769903+0000", 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1569999999999999e-05 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089432+0000", 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31952966900000002 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089439+0000", 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9809999999999997e-06 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.920 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPushReply(4.19 393/385 [PushReplyOp(4:9d13bd2c:::benchmark_data_smithi187_76542_object44187:head)])", 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:19.725374+0000", 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 135.15554678399999, 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.205654219, 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.725374+0000", 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.725374+0000", 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.725374+0000", 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.06e-07 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.725378+0000", 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8929999999999998e-06 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.929578+0000", 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.204199979 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.929582+0000", 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.481e-06 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.929627+0000", 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4759000000000003e-05 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:19.931028+0000", 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001400601 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPushReply(4.19 402/385 [PushReplyOp(4:9eb94d79:::benchmark_data_smithi187_76542_object73210:head)])", 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:30.224750+0000", 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 124.656170382, 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.193412206, 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.924 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.224750+0000", 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.224750+0000", 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.224751+0000", 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2799999999999996e-07 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.224754+0000", 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2200000000000001e-06 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.417127+0000", 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19237320899999999 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.417134+0000", 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8809999999999998e-06 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.417186+0000", 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2537000000000002e-05 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:30.418162+0000", 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00097573099999999997 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.927 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:154071 4.a e410/338 4:531ff552:::benchmark_data_smithi187_76542_object61735:head v 410'9496, mlcod=410'9492)", 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:47.947177+0000", 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 106.933743812, 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20590476799999999, 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.947177+0000", 2024-09-02T10:15:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.947177+0000", 2024-09-02T10:15:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.947177+0000", 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9100000000000004e-07 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:47.947182+0000", 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.724e-06 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.149287+0000", 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20210461299999999 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.149293+0000", 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2400000000000004e-06 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.149369+0000", 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6324999999999999e-05 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.149392+0000", 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.285e-05 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.153036+0000", 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036444310000000001 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:48.153081+0000", 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4994000000000003e-05 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.932 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:181097 4.7 4:e7273704:::benchmark_data_smithi187_76542_object88761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:19.485221+0000", 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 75.395699809000007, 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23570571100000001, 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 181097 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485221+0000", 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485221+0000", 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485222+0000", 2024-09-02T10:15:34.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.155e-06 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485226+0000", 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2409999999999997e-06 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485239+0000", 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2782000000000001e-05 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485245+0000", 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9830000000000001e-06 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485290+0000", 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5024999999999997e-05 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485380+0000", 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9802999999999994e-05 2024-09-02T10:15:34.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485451+0000", 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1474e-05 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.487590+0000", 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021388309999999999 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.674878+0000", 2024-09-02T10:15:34.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18728809299999999 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.720859+0000", 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.045981517999999999 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.720895+0000", 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5024999999999998e-05 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.720926+0000", 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1780999999999999e-05 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:183023 4.e 4:7b0884ad:::benchmark_data_smithi187_76542_object90687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:21.990649+0000", 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 72.890270928999996, 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.209310995, 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.939 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183023 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990649+0000", 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990649+0000", 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990651+0000", 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.083e-06 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990655+0000", 2024-09-02T10:15:34.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6070000000000001e-06 2024-09-02T10:15:34.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990666+0000", 2024-09-02T10:15:34.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0400999999999999e-05 2024-09-02T10:15:34.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990671+0000", 2024-09-02T10:15:34.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1800000000000004e-06 2024-09-02T10:15:34.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990712+0000", 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1507000000000001e-05 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990793+0000", 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1162000000000007e-05 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990876+0000", 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2533000000000004e-05 2024-09-02T10:15:34.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.994433+0000", 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035574809999999999 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.995109+0000", 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00067595600000000002 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:22.199869+0000", 2024-09-02T10:15:34.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20475953599999999 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:22.199920+0000", 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1403999999999998e-05 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:22.199960+0000", 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0145000000000003e-05 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.946 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:196 5.0 5:05575aab:::benchmark_data_smithi187_89405_object195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:46.406087+0000", 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 48.474833240999999, 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.194724022, 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 196 2024-09-02T10:15:34.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.406087+0000", 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.406087+0000", 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.406088+0000", 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7900000000000007e-07 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.406105+0000", 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7125e-05 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.406116+0000", 2024-09-02T10:15:34.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0249e-05 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.406122+0000", 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9380000000000003e-06 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.406166+0000", 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3163e-05 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.406236+0000", 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0673999999999995e-05 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.406318+0000", 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1323999999999996e-05 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.406621+0000", 2024-09-02T10:15:34.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00030372199999999998 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.407039+0000", 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00041785200000000002 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.600711+0000", 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19367231600000001 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.600763+0000", 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.113e-05 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:46.600811+0000", 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8550000000000001e-05 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.953 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:5213 5.2 e438/431 5:463fe113:::benchmark_data_smithi187_89405_object5212:head v 438'281, mlcod=438'277)", 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:49.191750+0000", 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 45.689170816000001, 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19744609499999999, 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:49.191750+0000", 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.954 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:49.191750+0000", 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:49.191750+0000", 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9200000000000001e-07 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:49.191830+0000", 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0196000000000003e-05 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:49.388652+0000", 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19682185699999999 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:49.388657+0000", 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5569999999999997e-06 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:49.388703+0000", 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6610999999999998e-05 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:49.388717+0000", 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3971999999999999e-05 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:49.389166+0000", 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00044894299999999999 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:49.389196+0000", 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9467000000000001e-05 2024-09-02T10:15:34.957 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:17907 5.f e440/431 5:f141e4c3:::benchmark_data_smithi187_89405_object17906:head v 440'1151, mlcod=440'1147)", 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.430772+0000", 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 37.450147917000002, 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19904446100000001, 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.430772+0000", 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.430772+0000", 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.430773+0000", 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1200000000000003e-07 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.430870+0000", 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7147999999999999e-05 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.628789+0000", 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19791883900000001 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.628798+0000", 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7930000000000003e-06 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.628875+0000", 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7194000000000007e-05 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.628897+0000", 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2348e-05 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.629766+0000", 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086888199999999997 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.629817+0000", 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0645e-05 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:18186 5.0 5:0af1a60d:::benchmark_data_smithi187_89405_object18185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.734715+0000", 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 37.146205747000003, 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.211869428, 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.962 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 18186 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734715+0000", 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734715+0000", 2024-09-02T10:15:34.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734716+0000", 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.012e-06 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734736+0000", 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0123999999999999e-05 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734751+0000", 2024-09-02T10:15:34.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4955e-05 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734779+0000", 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8138000000000001e-05 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734850+0000", 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0975000000000004e-05 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734970+0000", 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119677 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735115+0000", 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014499000000000001 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.736868+0000", 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001753065 2024-09-02T10:15:34.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.737123+0000", 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000255732 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946471+0000", 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20934799300000001 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946538+0000", 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6598000000000002e-05 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946584+0000", 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6168999999999998e-05 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:18189 5.0 5:053524f9:::benchmark_data_smithi187_89405_object18188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-02T10:15:34.968 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.735686+0000", 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 37.145234430000002, 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.211005743, 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 18189 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735686+0000", 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735686+0000", 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735687+0000", 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0419999999999999e-06 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735713+0000", 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5760000000000001e-05 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735728+0000", 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5184000000000001e-05 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735736+0000", 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5789999999999998e-06 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735969+0000", 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00023298500000000001 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.736084+0000", 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115158 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.736174+0000", 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0140000000000001e-05 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.737138+0000", 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00096445099999999996 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.749747+0000", 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012608546 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946627+0000", 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.196880686 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946661+0000", 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3976000000000001e-05 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946692+0000", 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0236e-05 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.974 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:18212 5.0 5:0dc8f405:::benchmark_data_smithi187_89405_object18211:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.748444+0000", 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 37.132476463000003, 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19837127199999999, 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 18212 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.748444+0000", 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.748444+0000", 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.748445+0000", 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0720000000000001e-06 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.748467+0000", 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1997e-05 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.748478+0000", 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1031e-05 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.748485+0000", 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2370000000000003e-06 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.748533+0000", 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7958999999999999e-05 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.748614+0000", 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0283000000000004e-05 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.748700+0000", 2024-09-02T10:15:34.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6033999999999997e-05 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.749484+0000", 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00078456299999999995 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.755688+0000", 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0062035739999999999 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946729+0000", 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19104126799999999 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946784+0000", 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.469e-05 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946815+0000", 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1563999999999998e-05 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.980 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:22168 5.8 e441/431 5:166a9b24:::benchmark_data_smithi187_89405_object22167:head v 441'1424, mlcod=441'1421)", 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:00.180685+0000", 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 34.700235890999998, 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.211436923, 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.180685+0000", 2024-09-02T10:15:34.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.180685+0000", 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.180685+0000", 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4000000000000001e-07 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.180785+0000", 2024-09-02T10:15:34.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9597000000000002e-05 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.390284+0000", 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20949867599999999 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.390291+0000", 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6190000000000002e-06 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.390379+0000", 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7923999999999994e-05 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.390399+0000", 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.965e-05 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.392072+0000", 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001673477 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.392121+0000", 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9240000000000003e-05 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:36410 5.4 5:20a8e25f:::benchmark_data_smithi187_89405_object36409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:12.596921+0000", 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 22.283999289, 2024-09-02T10:15:34.985 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20586489699999999, 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36410 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.596921+0000", 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.986 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.596921+0000", 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.596922+0000", 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.54e-07 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:34.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.596940+0000", 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8522999999999999e-05 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.596949+0000", 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4349999999999997e-06 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.596954+0000", 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.011e-06 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.597007+0000", 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2580000000000001e-05 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.597154+0000", 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014725900000000001 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.597283+0000", 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012922300000000001 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.598681+0000", 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001397564 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.598939+0000", 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025824600000000001 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.802687+0000", 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.203748596 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.802744+0000", 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6913999999999998e-05 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.802786+0000", 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1692e-05 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:34.991 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:15:34.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:35.013 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:35.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:35.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:35.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:35.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:35.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 49 16 62925 62909 80.2337 11.75 0.00261453 0.0122822 2024-09-02T10:15:35.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:35.259 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:35.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:35.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:35.378 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2958330226' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:15:35.378 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:35 smithi151 ceph-mon[16639]: osdmap e462: 12 total, 12 up, 10 in 2024-09-02T10:15:35.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:35.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:35.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:35.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:35.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:35.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2958330226' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:15:35.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:35 smithi187 ceph-mon[20072]: osdmap e462: 12 total, 12 up, 10 in 2024-09-02T10:15:35.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2958330226' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:15:35.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:35 smithi162 ceph-mon[20279]: osdmap e462: 12 total, 12 up, 10 in 2024-09-02T10:15:35.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:35.627 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:35.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:35.628 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:35.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 10 1 2024-09-02T10:15:35.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:15:35.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:35.751 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:35.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:35.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:35.859 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:35.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:35.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:35.967 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:35.978 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:36.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:36.001 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:36.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:36.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:36.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:36.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 50 16 62980 62964 78.6978 3.4375 0.00265394 0.0122881 2024-09-02T10:15:36.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:36.239 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:36 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:15:36.239 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:36 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:15:36.239 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:36 smithi151 ceph-mon[16639]: osdmap e463: 12 total, 12 up, 10 in 2024-09-02T10:15:36.239 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:36 smithi151 ceph-mon[16639]: pgmap v1081: 47 pgs: 47 active+clean; 3.7 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-02T10:15:36.239 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:36 smithi151 ceph-mon[16639]: osdmap e464: 12 total, 12 up, 10 in 2024-09-02T10:15:36.248 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:36.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:36.249 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:36.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:36.375 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:36.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:36.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:36.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:36 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:15:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:36 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:15:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:36 smithi187 ceph-mon[20072]: osdmap e463: 12 total, 12 up, 10 in 2024-09-02T10:15:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:36 smithi187 ceph-mon[20072]: pgmap v1081: 47 pgs: 47 active+clean; 3.7 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-02T10:15:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:36 smithi187 ceph-mon[20072]: osdmap e464: 12 total, 12 up, 10 in 2024-09-02T10:15:36.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:36.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:36.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:36 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:15:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:36 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:15:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:36 smithi162 ceph-mon[20279]: osdmap e463: 12 total, 12 up, 10 in 2024-09-02T10:15:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:36 smithi162 ceph-mon[20279]: pgmap v1081: 47 pgs: 47 active+clean; 3.7 GiB data, 15 GiB used, 880 GiB / 894 GiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-02T10:15:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:36 smithi162 ceph-mon[20279]: osdmap e464: 12 total, 12 up, 10 in 2024-09-02T10:15:36.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:36.636 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:36.637 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:36.637 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:36.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:36.764 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:36.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:36.772 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63427 5.d 5:b83f691a:::benchmark_data_smithi187_89405_object63426:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.759314+0000", 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.011161269, 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011193286, 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63427 2024-09-02T10:15:36.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.759314+0000", 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.759314+0000", 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:36.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.759315+0000", 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6670000000000001e-06 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.759341+0000", 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6112999999999999e-05 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.759358+0000", 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7073999999999999e-05 2024-09-02T10:15:36.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.759367+0000", 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5660000000000007e-06 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.759453+0000", 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5592000000000003e-05 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:36.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.759591+0000", 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013800100000000001 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.759916+0000", 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00032501600000000002 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.763053+0000", 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003137567 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.764251+0000", 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011977260000000001 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63440 5.d 5:b9c129a8:::benchmark_data_smithi187_89405_object63439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.768655+0000", 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0018201070000000001, 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001933412, 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:15:36.778 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63440 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.768655+0000", 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.768655+0000", 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.768657+0000", 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8589999999999999e-06 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.768681+0000", 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4298e-05 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.768697+0000", 2024-09-02T10:15:36.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5781e-05 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.768704+0000", 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3180000000000001e-06 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.768790+0000", 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6419e-05 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.768932+0000", 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000141617 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.769093+0000", 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016096200000000001 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 2 2024-09-02T10:15:36.782 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:15:36.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:36.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:36.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:36.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:37.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:37.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:37.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:15:37.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:37.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:37.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:37.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 51 16 63959 63943 78.3544 61.1875 0.0190085 0.0127559 2024-09-02T10:15:37.216 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:15:37.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:37.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:37.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:37.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:37.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:37.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:37.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:37.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:37 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:15:37.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:37 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:15:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3009745908' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:37.468 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:37 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:15:37.468 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:37 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:15:37.468 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3009745908' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:37.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:37.520 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:15:37.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:37.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:37.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:37 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:15:37.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:37 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:15:37.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3009745908' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:37.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:37.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:37.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:37.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:37.809 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:37.810 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:37.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:37.928 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:37.944 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:37.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:37.945 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:38.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:38.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:38.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:38.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 52 16 65005 64989 78.1047 65.375 0.0188174 0.012792 2024-09-02T10:15:38.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:38.200 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:38.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:38.231 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-09-02T10:15:38.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:38.332 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:38.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:38.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:38.355 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:38 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:15:38.355 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:38 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:15:38.356 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3009745908' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:15:38.356 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:38 smithi187 ceph-mon[20072]: osdmap e465: 12 total, 12 up, 10 in 2024-09-02T10:15:38.356 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:38 smithi187 ceph-mon[20072]: pgmap v1084: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 50 MiB/s wr, 804 op/s 2024-09-02T10:15:38.356 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/244925764' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:15:38.356 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:38 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:15:38.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:38.457 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:38.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:38.492 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:15:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:15:38.493 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:15:38.493 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:15:38.493 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:15:38.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:38.586 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:38.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:38.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:38 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:15:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:38 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:15:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3009745908' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:15:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:38 smithi162 ceph-mon[20279]: osdmap e465: 12 total, 12 up, 10 in 2024-09-02T10:15:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:38 smithi162 ceph-mon[20279]: pgmap v1084: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 50 MiB/s wr, 804 op/s 2024-09-02T10:15:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/244925764' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:15:38.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:38 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:15:38.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:38.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:38 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:15:38.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:38 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:15:38.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3009745908' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:15:38.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:38 smithi151 ceph-mon[16639]: osdmap e465: 12 total, 12 up, 10 in 2024-09-02T10:15:38.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:38 smithi151 ceph-mon[16639]: pgmap v1084: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 50 MiB/s wr, 804 op/s 2024-09-02T10:15:38.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/244925764' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:15:38.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:38 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:15:38.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:38.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:38.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:38.843 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:38.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:38.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:38.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:38.968 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:38.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:39.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:39.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:39.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:39.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:39.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 53 16 66119 66103 77.9446 69.625 0.00668622 0.0128229 2024-09-02T10:15:39.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:39.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:15:39.231 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:39.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:39.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:39.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:39.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:39.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:39.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:39.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:39.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:39.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:39 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:15:39.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:39 smithi151 ceph-mon[16639]: osdmap e466: 12 total, 12 up, 10 in 2024-09-02T10:15:39.478 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:39 smithi151 ceph-mon[16639]: osdmap e467: 12 total, 12 up, 10 in 2024-09-02T10:15:39.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:39.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:39 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:15:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:39 smithi187 ceph-mon[20072]: osdmap e466: 12 total, 12 up, 10 in 2024-09-02T10:15:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:39 smithi187 ceph-mon[20072]: osdmap e467: 12 total, 12 up, 10 in 2024-09-02T10:15:39.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:39.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:39 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:15:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:39 smithi162 ceph-mon[20279]: osdmap e466: 12 total, 12 up, 10 in 2024-09-02T10:15:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:39 smithi162 ceph-mon[20279]: osdmap e467: 12 total, 12 up, 10 in 2024-09-02T10:15:39.596 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:15:39.613 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:39.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:39.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:39.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:39.737 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:39.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:39.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:39.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:39.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:39.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:39.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:39.995 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:39.996 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:40.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:40.125 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:40.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:40.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 54 16 67335 67319 77.9084 76 0.00350192 0.0128292 2024-09-02T10:15:40.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:40.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:40.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:40.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:40.376 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:40.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24451.0:194200 2.a 2:5976d7b0:::benchmark_data_smithi187_45882_object96449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:40.797959+0000", 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 599.58138528799998, 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57870098699999994, 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.382 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24451", 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/2434042963", 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194200 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797959+0000", 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797959+0000", 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797961+0000", 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8670000000000001e-06 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797969+0000", 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5510000000000002e-06 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797987+0000", 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7691000000000001e-05 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.797995+0000", 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7749999999999998e-06 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.798058+0000", 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2309000000000003e-05 2024-09-02T10:15:40.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.798148+0000", 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0781999999999999e-05 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.798225+0000", 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6240000000000002e-05 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.801976+0000", 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003751009 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802761+0000", 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00078486799999999998 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376548+0000", 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57378783700000002 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376616+0000", 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7465000000000005e-05 2024-09-02T10:15:40.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376660+0000", 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4592999999999998e-05 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.24451.0:194200 2.a e216/92 ondisk, result = 0)", 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:05:40.802119+0000", 2024-09-02T10:15:40.388 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 599.57722576799995, 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57454802699999996, 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802119+0000", 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802119+0000", 2024-09-02T10:15:40.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802120+0000", 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1119999999999999e-06 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:40.802126+0000", 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9309999999999996e-06 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376473+0000", 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.57434650499999995 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376478+0000", 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8109999999999999e-06 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376533+0000", 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4347e-05 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376541+0000", 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0039999999999999e-06 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:05:41.376667+0000", 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000126317 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.392 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:11571 4.d e326/318 4:b88a9f8d:::benchmark_data_smithi187_76542_object11570:head v 326'724, mlcod=326'721)", 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:10.795584+0000", 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 269.58376045099999, 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.49112458599999997, 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795584+0000", 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795584+0000", 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795585+0000", 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7899999999999998e-07 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795608+0000", 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3408e-05 2024-09-02T10:15:40.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285722+0000", 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.490114102 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285750+0000", 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7104e-05 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285822+0000", 2024-09-02T10:15:40.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2354999999999994e-05 2024-09-02T10:15:40.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:40.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285855+0000", 2024-09-02T10:15:40.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2731000000000002e-05 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.286659+0000", 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00080397800000000001 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.286709+0000", 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0229000000000003e-05 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.397 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:126679 4.1 4:86205e4c:::benchmark_data_smithi187_76542_object34343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:15.467394+0000", 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 144.91195106000001, 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44570083700000002, 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:40.398 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126679 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467394+0000", 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467394+0000", 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467395+0000", 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1459999999999999e-06 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467400+0000", 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.451e-06 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467413+0000", 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2799000000000001e-05 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467421+0000", 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4830000000000002e-06 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467468+0000", 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7494999999999998e-05 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467552+0000", 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3823e-05 2024-09-02T10:15:40.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467631+0000", 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8720000000000005e-05 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.469349+0000", 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017183739999999999 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.470063+0000", 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000713731 2024-09-02T10:15:40.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.912992+0000", 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44292904900000002 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.913049+0000", 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7197999999999997e-05 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.913095+0000", 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5568000000000001e-05 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "rep_scrubmap(1.4 e124 from shard 8)", 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:03.322332+0000", 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 97.057012272999998, 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96974535299999998, 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.404 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322332+0000", 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322332+0000", 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322333+0000", 2024-09-02T10:15:40.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9000000000000002e-07 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322336+0000", 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.785e-06 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.291988+0000", 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96965220600000002 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.291994+0000", 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3520000000000003e-06 2024-09-02T10:15:40.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.292044+0000", 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9968e-05 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.292054+0000", 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0244000000000001e-05 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.292078+0000", 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3408e-05 2024-09-02T10:15:40.407 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62748 5.8 5.b553c768 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.205545+0000", 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 6.1737994790000004, 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99588105199999999, 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.408 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62748 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205545+0000", 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205545+0000", 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205546+0000", 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3200000000000003e-07 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205565+0000", 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8519e-05 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205574+0000", 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.448e-06 2024-09-02T10:15:40.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205579+0000", 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0980000000000001e-06 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205627+0000", 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7806000000000003e-05 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205634+0000", 2024-09-02T10:15:40.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5860000000000001e-06 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201410+0000", 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99577610400000005 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201426+0000", 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6558999999999998e-05 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62759 5.8 5.f9616988 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.288073+0000", 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 6.0912721730000001, 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91353117399999995, 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62759 2024-09-02T10:15:40.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288073+0000", 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288073+0000", 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288074+0000", 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5060000000000001e-06 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288099+0000", 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4862999999999999e-05 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288113+0000", 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4521e-05 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288121+0000", 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8150000000000006e-06 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288184+0000", 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2637000000000004e-05 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288192+0000", 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5189999999999992e-06 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201570+0000", 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91337743400000004 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201604+0000", 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3878999999999999e-05 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.417 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62760 5.8 5.be230818 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.288254+0000", 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 6.0910907219999997, 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91340091000000001, 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62760 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288254+0000", 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288254+0000", 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288255+0000", 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1200000000000002e-07 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288277+0000", 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2768e-05 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288292+0000", 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4171999999999999e-05 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288298+0000", 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5950000000000004e-06 2024-09-02T10:15:40.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288350+0000", 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1588999999999997e-05 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288356+0000", 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4450000000000001e-06 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201646+0000", 2024-09-02T10:15:40.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91328941100000005 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201655+0000", 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2180000000000005e-06 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62763 5.8 5.1d470d68 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.289624+0000", 2024-09-02T10:15:40.422 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 6.089720851, 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91237609600000003, 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62763 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289624+0000", 2024-09-02T10:15:40.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289624+0000", 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289626+0000", 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6500000000000001e-06 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289649+0000", 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3442e-05 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289664+0000", 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.526e-05 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289672+0000", 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7170000000000008e-06 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289731+0000", 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8931999999999998e-05 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289738+0000", 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1069999999999999e-06 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201987+0000", 2024-09-02T10:15:40.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91224905899999997 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202000+0000", 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2928999999999999e-05 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62767 5.8 5.9781c208 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.295609+0000", 2024-09-02T10:15:40.427 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 6.0837358669999997, 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90642393600000004, 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62767 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295609+0000", 2024-09-02T10:15:40.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295609+0000", 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295610+0000", 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.621e-06 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295647+0000", 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6955000000000001e-05 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295665+0000", 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7795000000000002e-05 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295674+0000", 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2210000000000003e-06 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295743+0000", 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8320999999999999e-05 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295755+0000", 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2145e-05 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202026+0000", 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90627074100000005 2024-09-02T10:15:40.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202033+0000", 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1369999999999996e-06 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62787 5.8 5.93d7ce28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.316402+0000", 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 6.0629431, 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88566921200000004, 2024-09-02T10:15:40.432 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62787 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316402+0000", 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316402+0000", 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316404+0000", 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1950000000000002e-06 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316427+0000", 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3611000000000001e-05 2024-09-02T10:15:40.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316443+0000", 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5903999999999999e-05 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316451+0000", 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4569999999999999e-06 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316511+0000", 2024-09-02T10:15:40.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0031999999999997e-05 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316520+0000", 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3470000000000005e-06 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202063+0000", 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88554330000000003 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202071+0000", 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3660000000000003e-06 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62828 5.8 5.ced072f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.387938+0000", 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.9914069220000004, 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81416300900000005, 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.437 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62828 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387938+0000", 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387938+0000", 2024-09-02T10:15:40.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387939+0000", 2024-09-02T10:15:40.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.22e-06 2024-09-02T10:15:40.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387960+0000", 2024-09-02T10:15:40.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0571e-05 2024-09-02T10:15:40.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387971+0000", 2024-09-02T10:15:40.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1659e-05 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387977+0000", 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2099999999999998e-06 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.388039+0000", 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1771000000000002e-05 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.388048+0000", 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9600000000000006e-06 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202094+0000", 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81404575899999998 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202101+0000", 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8589999999999999e-06 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62833 5.8 5.47f28d28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.391791+0000", 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.9875535549999999, 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81033902899999999, 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62833 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.443 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391791+0000", 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391791+0000", 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391793+0000", 2024-09-02T10:15:40.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7129999999999999e-06 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391815+0000", 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2514000000000002e-05 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391830+0000", 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4115e-05 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391837+0000", 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3869999999999997e-06 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391895+0000", 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7762e-05 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391901+0000", 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4910000000000002e-06 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202123+0000", 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81022218700000004 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202130+0000", 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8600000000000004e-06 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62834 5.8 5.af89fd28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.447 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.393180+0000", 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.9861649230000005, 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.80897919100000004, 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62834 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393180+0000", 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393180+0000", 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393181+0000", 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3900000000000004e-07 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393200+0000", 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8949000000000001e-05 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393210+0000", 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0550000000000001e-05 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393215+0000", 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2789999999999998e-06 2024-09-02T10:15:40.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393255+0000", 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9678000000000002e-05 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393260+0000", 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.989e-06 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202152+0000", 2024-09-02T10:15:40.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.80889200100000003 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202159+0000", 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9059999999999996e-06 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62856 5.8 5.39aa96f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.474887+0000", 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.9044580660000001, 2024-09-02T10:15:40.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72730094999999995, 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62856 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474887+0000", 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474887+0000", 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474888+0000", 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2640000000000001e-06 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474909+0000", 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1347000000000001e-05 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474922+0000", 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2448e-05 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474928+0000", 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1519999999999997e-06 2024-09-02T10:15:40.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474974+0000", 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6578e-05 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474980+0000", 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8880000000000002e-06 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202181+0000", 2024-09-02T10:15:40.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72720049399999998 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202188+0000", 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7789999999999998e-06 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62861 5.8 5.909d6bb8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.497010+0000", 2024-09-02T10:15:40.457 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.8823345409999996, 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.70520625100000001, 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62861 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497010+0000", 2024-09-02T10:15:40.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497010+0000", 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497012+0000", 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.012e-06 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497039+0000", 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6999e-05 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497060+0000", 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0590000000000001e-05 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497067+0000", 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9199999999999998e-06 2024-09-02T10:15:40.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497118+0000", 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1029999999999998e-05 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497125+0000", 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8210000000000004e-06 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202209+0000", 2024-09-02T10:15:40.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.70508442100000002 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202216+0000", 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4580000000000004e-06 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62865 5.8 5.cf598d28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.545057+0000", 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.8342879139999999, 2024-09-02T10:15:40.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65719031900000002, 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62865 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545057+0000", 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545057+0000", 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545058+0000", 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4950000000000001e-06 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545081+0000", 2024-09-02T10:15:40.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2793999999999999e-05 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545095+0000", 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3619e-05 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545101+0000", 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9710000000000001e-06 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545150+0000", 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8814999999999999e-05 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545157+0000", 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9870000000000002e-06 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202240+0000", 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65708368900000003 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202247+0000", 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9489999999999999e-06 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62879 5.8 5.80da3848 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.568069+0000", 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.8112762360000003, 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63420845199999998, 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.467 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62879 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568069+0000", 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568069+0000", 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568070+0000", 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.719e-06 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568097+0000", 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6455e-05 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568112+0000", 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4916000000000001e-05 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568121+0000", 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4690000000000003e-06 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568184+0000", 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2724000000000005e-05 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568193+0000", 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6840000000000008e-06 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202270+0000", 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63407665199999996 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202277+0000", 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8329999999999996e-06 2024-09-02T10:15:40.471 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62911 5.8 5.47f6e298 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.672293+0000", 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.7070513250000001, 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.53001293400000005, 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.472 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62911 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672293+0000", 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672293+0000", 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672295+0000", 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.626e-06 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672316+0000", 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1151000000000001e-05 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672328+0000", 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2269000000000001e-05 2024-09-02T10:15:40.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672335+0000", 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6649999999999997e-06 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672382+0000", 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6575e-05 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672389+0000", 2024-09-02T10:15:40.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8809999999999998e-06 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202300+0000", 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.52991091899999998 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202306+0000", 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8480000000000003e-06 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.476 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62929 5.d 5:b28584c9:::benchmark_data_smithi187_89405_object62928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:35.215954+0000", 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 5.1633908870000003, 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.45650502599999998, 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62929 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.477 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215954+0000", 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215954+0000", 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215955+0000", 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3540000000000001e-06 2024-09-02T10:15:40.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215975+0000", 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9556999999999999e-05 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215988+0000", 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3335e-05 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215994+0000", 2024-09-02T10:15:40.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2729999999999999e-06 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.216041+0000", 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7110999999999997e-05 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.216134+0000", 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3008000000000001e-05 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.216218+0000", 2024-09-02T10:15:40.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3017999999999997e-05 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.217972+0000", 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001754979 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.218530+0000", 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00055731300000000004 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.672383+0000", 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.45385332499999997 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.672426+0000", 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2540999999999998e-05 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.672459+0000", 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3212e-05 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.482 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:15:40.483 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:15:40.483 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:15:40.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:40 smithi187 ceph-mon[20072]: pgmap v1087: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 56 MiB/s wr, 901 op/s; 3870/192435 objects misplaced (2.011%); 1.1 MiB/s, 17 objects/s recovering 2024-09-02T10:15:40.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:40.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:40.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:40.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:40 smithi162 ceph-mon[20279]: pgmap v1087: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 56 MiB/s wr, 901 op/s; 3870/192435 objects misplaced (2.011%); 1.1 MiB/s, 17 objects/s recovering 2024-09-02T10:15:40.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:40.637 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:40.639 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:40 smithi151 ceph-mon[16639]: pgmap v1087: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 56 MiB/s wr, 901 op/s; 3870/192435 objects misplaced (2.011%); 1.1 MiB/s, 17 objects/s recovering 2024-09-02T10:15:40.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:40.959 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:40.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:40.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:41.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:41.089 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:41.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:41.090 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:41.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 55 16 68336 68320 77.6293 62.5625 0.00520932 0.0128749 2024-09-02T10:15:41.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:41.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:15:41.214 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:41.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:41.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:41.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:41.341 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:41.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:41.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:41.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:41.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:41.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:41.594 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:41.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:41.691 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:41.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:41.701 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-02T10:15:41.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:15:41.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:41.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:41.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:41.862 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:41.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:41.863 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:41.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:41.989 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:41.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:41.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:42.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:42.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:42.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:42.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 56 16 69354 69338 77.3792 63.625 0.011538 0.0129174 2024-09-02T10:15:42.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:42.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:42.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:42.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:42.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:15:42.331 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:15:42.331 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:15:42.331 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:15:42.338 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:42.340 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:15:42.340 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:15:42.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:42.372 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:15:42.372 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:15:42.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:42.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:42.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:42.393 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:42.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:15:42.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:15:42.427 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:15:42.428 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:15:42.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:15:42.454 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:15:42.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:42.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:15:42.481 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:15:42.508 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:42.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:42.511 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:15:42.512 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:15:42.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:42.621 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:42 smithi151 ceph-mon[16639]: pgmap v1088: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 48 MiB/s wr, 774 op/s; 3870/194187 objects misplaced (1.993%); 867 KiB/s, 13 objects/s recovering 2024-09-02T10:15:42.621 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/294396847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:42.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:42.642 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:42.737 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:15:42.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:15:42.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:42.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:42 smithi187 ceph-mon[20072]: pgmap v1088: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 48 MiB/s wr, 774 op/s; 3870/194187 objects misplaced (1.993%); 867 KiB/s, 13 objects/s recovering 2024-09-02T10:15:42.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/294396847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:42.764 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:42.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:42.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:15:42.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:15:42.789 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:15:42.790 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:15:42.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:42 smithi162 ceph-mon[20279]: pgmap v1088: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.0 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 48 MiB/s wr, 774 op/s; 3870/194187 objects misplaced (1.993%); 867 KiB/s, 13 objects/s recovering 2024-09-02T10:15:42.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/294396847' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:42.866 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:42.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:42.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:42.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:43.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:43.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:43.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:43.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:43.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:43.157 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:43.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 57 16 70481 70465 77.2573 70.4375 0.0294488 0.0129365 2024-09-02T10:15:43.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:43.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:15:43.286 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:43.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:43.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:43.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:43.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:43.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:43.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:43.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:43.567 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:43 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:15:43.567 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:43 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:15:43.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:43.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:43.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:43.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:43 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:15:43.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:43 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:15:43.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:43.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:43.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:43.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:43 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:15:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:43 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:15:43.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:43.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:43.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:44.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:44.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:44.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:44.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:44.172 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 58 16 71425 71409 76.9424 59 0.00679976 0.0129902 2024-09-02T10:15:44.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:44.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:44.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:44.309 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:44.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:44.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:44.436 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:44.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:44.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:44.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:44.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:44.570 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:44.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:44.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:44 smithi151 ceph-mon[16639]: pgmap v1089: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s; 3633/206265 objects misplaced (1.761%); 3.1 MiB/s, 50 objects/s recovering 2024-09-02T10:15:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:44 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:44.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:44 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:44.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:44.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:44.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:44 smithi187 ceph-mon[20072]: pgmap v1089: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s; 3633/206265 objects misplaced (1.761%); 3.1 MiB/s, 50 objects/s recovering 2024-09-02T10:15:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:44 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:44 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:44.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:44.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:44.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:44.826 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:15:44.827 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:15:44.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:44 smithi162 ceph-mon[20279]: pgmap v1089: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 64 MiB/s wr, 1.02k op/s; 3633/206265 objects misplaced (1.761%); 3.1 MiB/s, 50 objects/s recovering 2024-09-02T10:15:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:44 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:44 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:44.926 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:44.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:44.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:45.057 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:45.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:45.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:45.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:45.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 59 16 72648 72632 76.9337 76.4375 0.00702059 0.0129922 2024-09-02T10:15:45.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:45.207 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:45.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:45.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:45.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:45.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:45.355 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:45.355 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:45.443 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:45.472 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:45.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:45.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:45.597 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:45.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:45.668 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:45 smithi151 ceph-mon[16639]: from='client.16578 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:15:45.668 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:45 smithi151 ceph-mon[16639]: osdmap e468: 12 total, 12 up, 10 in 2024-09-02T10:15:45.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:45.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:45.728 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:45.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:45 smithi187 ceph-mon[20072]: from='client.16578 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:15:45.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:45 smithi187 ceph-mon[20072]: osdmap e468: 12 total, 12 up, 10 in 2024-09-02T10:15:45.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:45.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:45 smithi162 ceph-mon[20279]: from='client.16578 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:15:45.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:45 smithi162 ceph-mon[20279]: osdmap e468: 12 total, 12 up, 10 in 2024-09-02T10:15:45.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:45.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:45.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:45.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:45.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:45.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:46.083 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:46.106 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:46.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:46.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:15:46.172820+0000 min lat: 0.00106838 max lat: 2.22758 avg lat: 0.013007 2024-09-02T10:15:46.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:15:46.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 60 16 73790 73774 76.841 71.375 0.041594 0.013007 2024-09-02T10:15:46.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:46.231 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:46.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:46.333 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:46.360 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:46.361 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:46.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:46.486 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:46.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:46.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:15:46.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:46.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:46.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:46.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:46.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:46 smithi151 ceph-mon[16639]: pgmap v1091: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 53 MiB/s wr, 855 op/s; 3633/209250 objects misplaced (1.736%); 2.1 MiB/s, 33 objects/s recovering 2024-09-02T10:15:46.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:46.755 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:46.756 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:46 smithi187 ceph-mon[20072]: pgmap v1091: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 53 MiB/s wr, 855 op/s; 3633/209250 objects misplaced (1.736%); 2.1 MiB/s, 33 objects/s recovering 2024-09-02T10:15:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:46 smithi162 ceph-mon[20279]: pgmap v1091: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.3 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 53 MiB/s wr, 855 op/s; 3633/209250 objects misplaced (1.736%); 2.1 MiB/s, 33 objects/s recovering 2024-09-02T10:15:46.857 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:46.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:46.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:46.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:47.017 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:47.018 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:47.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:47.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:47.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:47.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 61 16 74731 74715 76.5454 58.8125 0.00649398 0.013055 2024-09-02T10:15:47.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:47.268 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:47.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:47.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:47.398 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:47.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:47.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:47.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:47.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:47.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:47.649 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:47.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:47.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:47.678 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1089,"stamp":"2024-09-02T10:15:43.291326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4506386976,"num_objects":68755,"num_object_clones":0,"num_object_copies":206265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3633,"num_objects_unfound":0,"num_objects_dirty":68755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68753,"num_write_kb":4400192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2078,"num_bytes_recovered":136717376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71605,"ondisk_log_size":71605,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":16816176,"kb_used_data":13398700,"kb_used_omap":15,"kb_used_meta":3417264,"kb_avail":920635344,"statfs":{"total":959950356480,"available":942730592256,"internally_reserved":0,"allocated":13720268800,"data_stored":13714351121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3499278769},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,5,2,1,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":405471232,"num_objects":6187,"num_object_clones":0,"num_object_copies":18561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3633,"num_objects_unfound":0,"num_objects_dirty":6187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6187,"num_write_kb":395968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":20054016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.077690"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":260,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.993032+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:15:38.993032+0000","last_peered":"2024-09-02T10:15:38.993032+0000","last_clean":"2024-09-02T10:15:38.993032+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:15:38.993032+0000","last_undegraded":"2024-09-02T10:15:38.993032+0000","last_fullsized":"2024-09-02T10:15:38.993032+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:56.714204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":59,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.234197+0000","last_change":"2024-09-02T10:15:35.207918+0000","last_active":"2024-09-02T10:15:38.234197+0000","last_peered":"2024-09-02T10:15:38.234197+0000","last_clean":"2024-09-02T10:15:38.234197+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:15:38.234197+0000","last_undegraded":"2024-09-02T10:15:38.234197+0000","last_fullsized":"2024-09-02T10:15:38.234197+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:33.238949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:33.238949+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":518,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.991301+0000","last_change":"2024-09-02T10:15:12.091472+0000","last_active":"2024-09-02T10:15:38.991301+0000","last_peered":"2024-09-02T10:15:38.991301+0000","last_clean":"2024-09-02T10:15:38.991301+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:15:38.991301+0000","last_undegraded":"2024-09-02T10:15:38.991301+0000","last_fullsized":"2024-09-02T10:15:38.991301+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.091410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.091410+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:27.631420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":518,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.991295+0000","last_change":"2024-09-02T10:15:19.065187+0000","last_active":"2024-09-02T10:15:38.991295+0000","last_peered":"2024-09-02T10:15:38.991295+0000","last_clean":"2024-09-02T10:15:38.991295+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:15:38.991295+0000","last_undegraded":"2024-09-02T10:15:38.991295+0000","last_fullsized":"2024-09-02T10:15:38.991295+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:19.065134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:19.065134+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:36.285460+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":439,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.995454+0000","last_change":"2024-09-02T10:15:35.202261+0000","last_active":"2024-09-02T10:15:38.995454+0000","last_peered":"2024-09-02T10:15:38.995454+0000","last_clean":"2024-09-02T10:15:38.995454+0000","last_became_active":"2024-09-02T10:15:35.201887+0000","last_became_peered":"2024-09-02T10:15:35.201887+0000","last_unstale":"2024-09-02T10:15:38.995454+0000","last_undegraded":"2024-09-02T10:15:38.995454+0000","last_fullsized":"2024-09-02T10:15:38.995454+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:04.994667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:04.994667+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:30.739058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":368,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.993367+0000","last_change":"2024-09-02T10:14:11.689851+0000","last_active":"2024-09-02T10:15:38.993367+0000","last_peered":"2024-09-02T10:15:38.993367+0000","last_clean":"2024-09-02T10:15:38.993367+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:15:38.993367+0000","last_undegraded":"2024-09-02T10:15:38.993367+0000","last_fullsized":"2024-09-02T10:15:38.993367+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.689767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":883,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.996101+0000","last_change":"2024-09-02T10:15:07.059223+0000","last_active":"2024-09-02T10:15:38.996101+0000","last_peered":"2024-09-02T10:15:38.996101+0000","last_clean":"2024-09-02T10:15:38.996101+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:15:38.996101+0000","last_undegraded":"2024-09-02T10:15:38.996101+0000","last_fullsized":"2024-09-02T10:15:38.996101+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:45.351862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:45.351862+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:57.344255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":525,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.991177+0000","last_change":"2024-09-02T10:15:05.094874+0000","last_active":"2024-09-02T10:15:38.991177+0000","last_peered":"2024-09-02T10:15:38.991177+0000","last_clean":"2024-09-02T10:15:38.991177+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:15:38.991177+0000","last_undegraded":"2024-09-02T10:15:38.991177+0000","last_fullsized":"2024-09-02T10:15:38.991177+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:05.094792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:05.094792+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:09.107562+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":530,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.234243+0000","last_change":"2024-09-02T10:15:34.791760+0000","last_active":"2024-09-02T10:15:38.234243+0000","last_peered":"2024-09-02T10:15:38.234243+0000","last_clean":"2024-09-02T10:15:38.234243+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:15:38.234243+0000","last_undegraded":"2024-09-02T10:15:38.234243+0000","last_fullsized":"2024-09-02T10:15:38.234243+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:34.791708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:34.791708+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.857252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":501,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.995891+0000","last_change":"2024-09-02T10:14:49.165646+0000","last_active":"2024-09-02T10:15:38.995891+0000","last_peered":"2024-09-02T10:15:38.995891+0000","last_clean":"2024-09-02T10:15:38.995891+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:15:38.995891+0000","last_undegraded":"2024-09-02T10:15:38.995891+0000","last_fullsized":"2024-09-02T10:15:38.995891+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:49.165586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:49.165586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.197404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":353,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.995810+0000","last_change":"2024-09-02T10:14:45.203339+0000","last_active":"2024-09-02T10:15:38.995810+0000","last_peered":"2024-09-02T10:15:38.995810+0000","last_clean":"2024-09-02T10:15:38.995810+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:15:38.995810+0000","last_undegraded":"2024-09-02T10:15:38.995810+0000","last_fullsized":"2024-09-02T10:15:38.995810+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:45.203278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:45.203278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:51.594287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":856,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.234051+0000","last_change":"2024-09-02T10:15:06.447417+0000","last_active":"2024-09-02T10:15:38.234051+0000","last_peered":"2024-09-02T10:15:38.234051+0000","last_clean":"2024-09-02T10:15:38.234051+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:15:38.234051+0000","last_undegraded":"2024-09-02T10:15:38.234051+0000","last_fullsized":"2024-09-02T10:15:38.234051+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:06.447372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:06.447372+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:13.751465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":793,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.996292+0000","last_change":"2024-09-02T10:15:07.059344+0000","last_active":"2024-09-02T10:15:38.996292+0000","last_peered":"2024-09-02T10:15:38.996292+0000","last_clean":"2024-09-02T10:15:38.996292+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:15:38.996292+0000","last_undegraded":"2024-09-02T10:15:38.996292+0000","last_fullsized":"2024-09-02T10:15:38.996292+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.252526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.252526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:51.953514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":780,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.996936+0000","last_change":"2024-09-02T10:15:29.170702+0000","last_active":"2024-09-02T10:15:38.996936+0000","last_peered":"2024-09-02T10:15:38.996936+0000","last_clean":"2024-09-02T10:15:38.996936+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:15:38.996936+0000","last_undegraded":"2024-09-02T10:15:38.996936+0000","last_fullsized":"2024-09-02T10:15:38.996936+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.665438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.665438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:04.476082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":771,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.996276+0000","last_change":"2024-09-02T10:14:34.217697+0000","last_active":"2024-09-02T10:15:38.996276+0000","last_peered":"2024-09-02T10:15:38.996276+0000","last_clean":"2024-09-02T10:15:38.996276+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:15:38.996276+0000","last_undegraded":"2024-09-02T10:15:38.996276+0000","last_fullsized":"2024-09-02T10:15:38.996276+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.217624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.217624+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:02.148154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"466'3984","reported_seq":7617,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.560424+0000","last_change":"2024-09-02T10:15:24.164550+0000","last_active":"2024-09-02T10:15:38.560424+0000","last_peered":"2024-09-02T10:15:38.560424+0000","last_clean":"2024-09-02T10:15:38.560424+0000","last_became_active":"2024-09-02T10:15:24.141368+0000","last_became_peered":"2024-09-02T10:15:24.141368+0000","last_unstale":"2024-09-02T10:15:38.560424+0000","last_undegraded":"2024-09-02T10:15:38.560424+0000","last_fullsized":"2024-09-02T10:15:38.560424+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:51.435130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261095424,"num_objects":3984,"num_object_clones":0,"num_object_copies":11952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3984,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3984,"num_write_kb":254976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":115933184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":858,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.996203+0000","last_change":"2024-09-02T10:14:41.306927+0000","last_active":"2024-09-02T10:15:38.996203+0000","last_peered":"2024-09-02T10:15:38.996203+0000","last_clean":"2024-09-02T10:15:38.996203+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:15:38.996203+0000","last_undegraded":"2024-09-02T10:15:38.996203+0000","last_fullsized":"2024-09-02T10:15:38.996203+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:41.306868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:41.306868+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:04.992960+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"467'4384","reported_seq":4467,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:42.049164+0000","last_change":"2024-09-02T10:14:39.040351+0000","last_active":"2024-09-02T10:15:42.049164+0000","last_peered":"2024-09-02T10:15:42.049164+0000","last_clean":"2024-09-02T10:15:42.049164+0000","last_became_active":"2024-09-02T10:14:39.040025+0000","last_became_peered":"2024-09-02T10:14:39.040025+0000","last_unstale":"2024-09-02T10:15:42.049164+0000","last_undegraded":"2024-09-02T10:15:42.049164+0000","last_fullsized":"2024-09-02T10:15:42.049164+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:54.413941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287309824,"num_objects":4384,"num_object_clones":0,"num_object_copies":13152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4384,"num_write_kb":280576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":973,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.995440+0000","last_change":"2024-09-02T10:14:36.143055+0000","last_active":"2024-09-02T10:15:38.995440+0000","last_peered":"2024-09-02T10:15:38.995440+0000","last_clean":"2024-09-02T10:15:38.995440+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:15:38.995440+0000","last_undegraded":"2024-09-02T10:15:38.995440+0000","last_fullsized":"2024-09-02T10:15:38.995440+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.142993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.142993+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.623989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"467'4331","reported_seq":4413,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:42.950148+0000","last_change":"2024-09-02T10:14:39.038506+0000","last_active":"2024-09-02T10:15:42.950148+0000","last_peered":"2024-09-02T10:15:42.950148+0000","last_clean":"2024-09-02T10:15:42.950148+0000","last_became_active":"2024-09-02T10:14:39.038164+0000","last_became_peered":"2024-09-02T10:14:39.038164+0000","last_unstale":"2024-09-02T10:15:42.950148+0000","last_undegraded":"2024-09-02T10:15:42.950148+0000","last_fullsized":"2024-09-02T10:15:42.950148+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:50.238502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283836416,"num_objects":4331,"num_object_clones":0,"num_object_copies":12993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4331,"num_write_kb":277184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":615,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:39.001833+0000","last_change":"2024-09-02T10:14:31.581643+0000","last_active":"2024-09-02T10:15:39.001833+0000","last_peered":"2024-09-02T10:15:39.001833+0000","last_clean":"2024-09-02T10:15:39.001833+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:15:39.001833+0000","last_undegraded":"2024-09-02T10:15:39.001833+0000","last_fullsized":"2024-09-02T10:15:39.001833+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.581597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.581597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.091930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"467'4403","reported_seq":4486,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:42.011598+0000","last_change":"2024-09-02T10:14:39.037573+0000","last_active":"2024-09-02T10:15:42.011598+0000","last_peered":"2024-09-02T10:15:42.011598+0000","last_clean":"2024-09-02T10:15:42.011598+0000","last_became_active":"2024-09-02T10:14:39.037121+0000","last_became_peered":"2024-09-02T10:14:39.037121+0000","last_unstale":"2024-09-02T10:15:42.011598+0000","last_undegraded":"2024-09-02T10:15:42.011598+0000","last_fullsized":"2024-09-02T10:15:42.011598+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4403,"log_dups_size":0,"ondisk_log_size":4403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:01.026599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288555008,"num_objects":4403,"num_object_clones":0,"num_object_copies":13209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4403,"num_write_kb":281792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":20,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.992415+0000","last_change":"2024-09-02T10:15:35.208246+0000","last_active":"2024-09-02T10:15:38.992415+0000","last_peered":"2024-09-02T10:15:38.992415+0000","last_clean":"2024-09-02T10:15:38.992415+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:15:38.992415+0000","last_undegraded":"2024-09-02T10:15:38.992415+0000","last_fullsized":"2024-09-02T10:15:38.992415+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.668602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.668602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:42.806393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"467'4227","reported_seq":4310,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:41.599608+0000","last_change":"2024-09-02T10:14:39.642395+0000","last_active":"2024-09-02T10:15:41.599608+0000","last_peered":"2024-09-02T10:15:41.599608+0000","last_clean":"2024-09-02T10:15:41.599608+0000","last_became_active":"2024-09-02T10:14:39.641995+0000","last_became_peered":"2024-09-02T10:14:39.641995+0000","last_unstale":"2024-09-02T10:15:41.599608+0000","last_undegraded":"2024-09-02T10:15:41.599608+0000","last_fullsized":"2024-09-02T10:15:41.599608+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4227,"log_dups_size":0,"ondisk_log_size":4227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:55.658135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277020672,"num_objects":4227,"num_object_clones":0,"num_object_copies":12681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4227,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4227,"num_write_kb":270528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":928,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.992932+0000","last_change":"2024-09-02T10:15:29.195341+0000","last_active":"2024-09-02T10:15:38.992932+0000","last_peered":"2024-09-02T10:15:38.992932+0000","last_clean":"2024-09-02T10:15:38.992932+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:15:38.992932+0000","last_undegraded":"2024-09-02T10:15:38.992932+0000","last_fullsized":"2024-09-02T10:15:38.992932+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.277119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.277119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:28.784954+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"466'4072","reported_seq":4153,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.855865+0000","last_change":"2024-09-02T10:14:39.034903+0000","last_active":"2024-09-02T10:15:38.855865+0000","last_peered":"2024-09-02T10:15:38.855865+0000","last_clean":"2024-09-02T10:15:38.855865+0000","last_became_active":"2024-09-02T10:14:39.034636+0000","last_became_peered":"2024-09-02T10:14:39.034636+0000","last_unstale":"2024-09-02T10:15:38.855865+0000","last_undegraded":"2024-09-02T10:15:38.855865+0000","last_fullsized":"2024-09-02T10:15:38.855865+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4072,"log_dups_size":0,"ondisk_log_size":4072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:07.235133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266862592,"num_objects":4072,"num_object_clones":0,"num_object_copies":12216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4072,"num_write_kb":260608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":396,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.993716+0000","last_change":"2024-09-02T10:15:35.205684+0000","last_active":"2024-09-02T10:15:38.993716+0000","last_peered":"2024-09-02T10:15:38.993716+0000","last_clean":"2024-09-02T10:15:38.993716+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:15:38.993716+0000","last_undegraded":"2024-09-02T10:15:38.993716+0000","last_fullsized":"2024-09-02T10:15:38.993716+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.701773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.701773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.566711+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"467'4273","reported_seq":4355,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:42.170743+0000","last_change":"2024-09-02T10:14:39.038158+0000","last_active":"2024-09-02T10:15:42.170743+0000","last_peered":"2024-09-02T10:15:42.170743+0000","last_clean":"2024-09-02T10:15:42.170743+0000","last_became_active":"2024-09-02T10:14:39.037822+0000","last_became_peered":"2024-09-02T10:14:39.037822+0000","last_unstale":"2024-09-02T10:15:42.170743+0000","last_undegraded":"2024-09-02T10:15:42.170743+0000","last_fullsized":"2024-09-02T10:15:42.170743+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4273,"log_dups_size":0,"ondisk_log_size":4273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:47.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280035328,"num_objects":4273,"num_object_clones":0,"num_object_copies":12819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4273,"num_write_kb":273472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":368,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.993885+0000","last_change":"2024-09-02T10:14:09.768298+0000","last_active":"2024-09-02T10:15:38.993885+0000","last_peered":"2024-09-02T10:15:38.993885+0000","last_clean":"2024-09-02T10:15:38.993885+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:15:38.993885+0000","last_undegraded":"2024-09-02T10:15:38.993885+0000","last_fullsized":"2024-09-02T10:15:38.993885+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:09.768061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:09.768061+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"467'4500","reported_seq":4583,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:42.697962+0000","last_change":"2024-09-02T10:14:39.035303+0000","last_active":"2024-09-02T10:15:42.697962+0000","last_peered":"2024-09-02T10:15:42.697962+0000","last_clean":"2024-09-02T10:15:42.697962+0000","last_became_active":"2024-09-02T10:14:39.035100+0000","last_became_peered":"2024-09-02T10:14:39.035100+0000","last_unstale":"2024-09-02T10:15:42.697962+0000","last_undegraded":"2024-09-02T10:15:42.697962+0000","last_fullsized":"2024-09-02T10:15:42.697962+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4500,"log_dups_size":0,"ondisk_log_size":4500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:42.948880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294912000,"num_objects":4500,"num_object_clones":0,"num_object_copies":13500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4500,"num_write_kb":288000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":119,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.995081+0000","last_change":"2024-09-02T10:15:36.299961+0000","last_active":"2024-09-02T10:15:38.995081+0000","last_peered":"2024-09-02T10:15:38.995081+0000","last_clean":"2024-09-02T10:15:38.995081+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:15:38.995081+0000","last_undegraded":"2024-09-02T10:15:38.995081+0000","last_fullsized":"2024-09-02T10:15:38.995081+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:36.299914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:36.299914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:01.870130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"467'4382","reported_seq":4465,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:42.706382+0000","last_change":"2024-09-02T10:14:39.641746+0000","last_active":"2024-09-02T10:15:42.706382+0000","last_peered":"2024-09-02T10:15:42.706382+0000","last_clean":"2024-09-02T10:15:42.706382+0000","last_became_active":"2024-09-02T10:14:39.641478+0000","last_became_peered":"2024-09-02T10:14:39.641478+0000","last_unstale":"2024-09-02T10:15:42.706382+0000","last_undegraded":"2024-09-02T10:15:42.706382+0000","last_fullsized":"2024-09-02T10:15:42.706382+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:56.685441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287178752,"num_objects":4382,"num_object_clones":0,"num_object_copies":13146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4382,"num_write_kb":280448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":369,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.993462+0000","last_change":"2024-09-02T10:14:36.623992+0000","last_active":"2024-09-02T10:15:38.993462+0000","last_peered":"2024-09-02T10:15:38.993462+0000","last_clean":"2024-09-02T10:15:38.993462+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:15:38.993462+0000","last_undegraded":"2024-09-02T10:15:38.993462+0000","last_fullsized":"2024-09-02T10:15:38.993462+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.623930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.623930+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.218010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"467'4241","reported_seq":4324,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:41.947143+0000","last_change":"2024-09-02T10:14:39.038216+0000","last_active":"2024-09-02T10:15:41.947143+0000","last_peered":"2024-09-02T10:15:41.947143+0000","last_clean":"2024-09-02T10:15:41.947143+0000","last_became_active":"2024-09-02T10:14:39.037882+0000","last_became_peered":"2024-09-02T10:14:39.037882+0000","last_unstale":"2024-09-02T10:15:41.947143+0000","last_undegraded":"2024-09-02T10:15:41.947143+0000","last_fullsized":"2024-09-02T10:15:41.947143+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:43.534011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277938176,"num_objects":4241,"num_object_clones":0,"num_object_copies":12723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":271424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":615,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.995550+0000","last_change":"2024-09-02T10:14:40.192356+0000","last_active":"2024-09-02T10:15:38.995550+0000","last_peered":"2024-09-02T10:15:38.995550+0000","last_clean":"2024-09-02T10:15:38.995550+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:15:38.995550+0000","last_undegraded":"2024-09-02T10:15:38.995550+0000","last_fullsized":"2024-09-02T10:15:38.995550+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.192297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.192297+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.932788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"467'4316","reported_seq":4398,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:42.002892+0000","last_change":"2024-09-02T10:14:39.642277+0000","last_active":"2024-09-02T10:15:42.002892+0000","last_peered":"2024-09-02T10:15:42.002892+0000","last_clean":"2024-09-02T10:15:42.002892+0000","last_became_active":"2024-09-02T10:14:39.641932+0000","last_became_peered":"2024-09-02T10:14:39.641932+0000","last_unstale":"2024-09-02T10:15:42.002892+0000","last_undegraded":"2024-09-02T10:15:42.002892+0000","last_fullsized":"2024-09-02T10:15:42.002892+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:55.249885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":282853376,"num_objects":4316,"num_object_clones":0,"num_object_copies":12948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4316,"num_write_kb":276224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":872,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.234142+0000","last_change":"2024-09-02T10:15:29.169027+0000","last_active":"2024-09-02T10:15:38.234142+0000","last_peered":"2024-09-02T10:15:38.234142+0000","last_clean":"2024-09-02T10:15:38.234142+0000","last_became_active":"2024-09-02T10:15:29.168692+0000","last_became_peered":"2024-09-02T10:15:29.168692+0000","last_unstale":"2024-09-02T10:15:38.234142+0000","last_undegraded":"2024-09-02T10:15:38.234142+0000","last_fullsized":"2024-09-02T10:15:38.234142+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.804014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.804014+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:54.886555+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"467'4388","reported_seq":4471,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:41.622778+0000","last_change":"2024-09-02T10:14:39.038104+0000","last_active":"2024-09-02T10:15:41.622778+0000","last_peered":"2024-09-02T10:15:41.622778+0000","last_clean":"2024-09-02T10:15:41.622778+0000","last_became_active":"2024-09-02T10:14:39.037755+0000","last_became_peered":"2024-09-02T10:14:39.037755+0000","last_unstale":"2024-09-02T10:15:41.622778+0000","last_undegraded":"2024-09-02T10:15:41.622778+0000","last_fullsized":"2024-09-02T10:15:41.622778+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4388,"log_dups_size":0,"ondisk_log_size":4388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:00.080239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287571968,"num_objects":4388,"num_object_clones":0,"num_object_copies":13164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4388,"num_write_kb":280832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":633,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.995200+0000","last_change":"2024-09-02T10:15:35.202850+0000","last_active":"2024-09-02T10:15:38.995200+0000","last_peered":"2024-09-02T10:15:38.995200+0000","last_clean":"2024-09-02T10:15:38.995200+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:15:38.995200+0000","last_undegraded":"2024-09-02T10:15:38.995200+0000","last_fullsized":"2024-09-02T10:15:38.995200+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:48.187420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:48.187420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:57.257578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"467'4321","reported_seq":4404,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:42.943929+0000","last_change":"2024-09-02T10:14:39.039399+0000","last_active":"2024-09-02T10:15:42.943929+0000","last_peered":"2024-09-02T10:15:42.943929+0000","last_clean":"2024-09-02T10:15:42.943929+0000","last_became_active":"2024-09-02T10:14:39.039060+0000","last_became_peered":"2024-09-02T10:14:39.039060+0000","last_unstale":"2024-09-02T10:15:42.943929+0000","last_undegraded":"2024-09-02T10:15:42.943929+0000","last_fullsized":"2024-09-02T10:15:42.943929+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:48.488192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283181056,"num_objects":4321,"num_object_clones":0,"num_object_copies":12963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":276544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":873,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.995186+0000","last_change":"2024-09-02T10:14:34.284618+0000","last_active":"2024-09-02T10:15:38.995186+0000","last_peered":"2024-09-02T10:15:38.995186+0000","last_clean":"2024-09-02T10:15:38.995186+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:15:38.995186+0000","last_undegraded":"2024-09-02T10:15:38.995186+0000","last_fullsized":"2024-09-02T10:15:38.995186+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.284563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.284563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:59.753379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"467'4170","reported_seq":4253,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:39.317531+0000","last_change":"2024-09-02T10:14:39.036854+0000","last_active":"2024-09-02T10:15:39.317531+0000","last_peered":"2024-09-02T10:15:39.317531+0000","last_clean":"2024-09-02T10:15:39.317531+0000","last_became_active":"2024-09-02T10:14:39.036453+0000","last_became_peered":"2024-09-02T10:14:39.036453+0000","last_unstale":"2024-09-02T10:15:39.317531+0000","last_undegraded":"2024-09-02T10:15:39.317531+0000","last_fullsized":"2024-09-02T10:15:39.317531+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:57.124824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273285120,"num_objects":4170,"num_object_clones":0,"num_object_copies":12510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4170,"num_write_kb":266880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":408,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.995887+0000","last_change":"2024-09-02T10:14:47.184482+0000","last_active":"2024-09-02T10:15:38.995887+0000","last_peered":"2024-09-02T10:15:38.995887+0000","last_clean":"2024-09-02T10:15:38.995887+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:15:38.995887+0000","last_undegraded":"2024-09-02T10:15:38.995887+0000","last_fullsized":"2024-09-02T10:15:38.995887+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:47.184420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:47.184420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.550061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"467'4350","reported_seq":4432,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:41.105658+0000","last_change":"2024-09-02T10:14:39.037960+0000","last_active":"2024-09-02T10:15:41.105658+0000","last_peered":"2024-09-02T10:15:41.105658+0000","last_clean":"2024-09-02T10:15:41.105658+0000","last_became_active":"2024-09-02T10:14:39.037567+0000","last_became_peered":"2024-09-02T10:14:39.037567+0000","last_unstale":"2024-09-02T10:15:41.105658+0000","last_undegraded":"2024-09-02T10:15:41.105658+0000","last_fullsized":"2024-09-02T10:15:41.105658+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.027618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285081600,"num_objects":4350,"num_object_clones":0,"num_object_copies":13050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4350,"num_write_kb":278400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":653,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.995795+0000","last_change":"2024-09-02T10:15:31.899734+0000","last_active":"2024-09-02T10:15:38.995795+0000","last_peered":"2024-09-02T10:15:38.995795+0000","last_clean":"2024-09-02T10:15:38.995795+0000","last_became_active":"2024-09-02T10:15:29.167553+0000","last_became_peered":"2024-09-02T10:15:29.167553+0000","last_unstale":"2024-09-02T10:15:38.995795+0000","last_undegraded":"2024-09-02T10:15:38.995795+0000","last_fullsized":"2024-09-02T10:15:38.995795+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:31.899662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:31.899662+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.920777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"467'4411","reported_seq":5108,"reported_epoch":467,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:15:42.718060+0000","last_change":"2024-09-02T10:15:36.347809+0000","last_active":"2024-09-02T10:15:42.718060+0000","last_peered":"2024-09-02T10:15:42.718060+0000","last_clean":"2024-09-02T10:15:34.173584+0000","last_became_active":"2024-09-02T10:15:36.297925+0000","last_became_peered":"2024-09-02T10:15:36.297925+0000","last_unstale":"2024-09-02T10:15:42.718060+0000","last_undegraded":"2024-09-02T10:15:42.718060+0000","last_fullsized":"2024-09-02T10:15:36.200774+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4411,"log_dups_size":0,"ondisk_log_size":4411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289079296,"num_objects":4411,"num_object_clones":0,"num_object_copies":13233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3633,"num_objects_unfound":0,"num_objects_dirty":4411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4411,"num_write_kb":282304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":20054016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[0,10],"avail_no_missing":["0","9","10"],"object_location_counts":[{"shards":"0,9,10","objects":3633},{"shards":"0,10","objects":778}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":50,"reported_epoch":466,"state":"active+clean","last_fresh":"2024-09-02T10:15:38.233870+0000","last_change":"2024-09-02T10:15:12.091321+0000","last_active":"2024-09-02T10:15:38.233870+0000","last_peered":"2024-09-02T10:15:38.233870+0000","last_clean":"2024-09-02T10:15:38.233870+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:15:38.233870+0000","last_undegraded":"2024-09-02T10:15:38.233870+0000","last_fullsized":"2024-09-02T10:15:38.233870+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.617088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.617088+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:47.906797+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4505796608,"num_objects":68753,"num_object_clones":0,"num_object_copies":206259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3633,"num_objects_unfound":0,"num_objects_dirty":68753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68753,"num_write_kb":4400192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2075,"num_bytes_recovered":135987200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13618380800,"data_stored":13618380800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68753,"ondisk_log_size":68753,"up":48,"acting":47,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383867,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1514172,"kb_used_data":1149804,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":92230980,"statfs":{"total":95995035648,"available":94444523520,"internally_reserved":0,"allocated":1177399296,"data_stored":1176807248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":373094852},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612800,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2091200,"kb_used_data":1911468,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91653952,"statfs":{"total":95995035648,"available":93853646848,"internally_reserved":0,"allocated":1957343232,"data_stored":1956751184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":184023488},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841733,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1761084,"kb_used_data":1350892,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91984068,"statfs":{"total":95995035648,"available":94191685632,"internally_reserved":0,"allocated":1383313408,"data_stored":1382721360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":420018624},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070666,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1006152,"kb_used_data":798636,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":92739000,"statfs":{"total":95995035648,"available":94964736000,"internally_reserved":0,"allocated":817803264,"data_stored":817211216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299599,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1573700,"kb_used_data":1173236,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92171452,"statfs":{"total":95995035648,"available":94383566848,"internally_reserved":0,"allocated":1201393664,"data_stored":1200798064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":410057162},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528531,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1663164,"kb_used_data":1393196,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92081988,"statfs":{"total":95995035648,"available":94291955712,"internally_reserved":0,"allocated":1426632704,"data_stored":1426040656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":276429255},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724760,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2308996,"kb_used_data":1926580,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":91436156,"statfs":{"total":95995035648,"available":93630623744,"internally_reserved":0,"allocated":1972817920,"data_stored":1972222320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953692,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149922,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":661884,"kb_used_data":261408,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93083268,"statfs":{"total":95995035648,"available":95317266432,"internally_reserved":0,"allocated":267681792,"data_stored":267094249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":410057160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411558,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575083,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1629044,"kb_used_data":1236892,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92116108,"statfs":{"total":95995035648,"available":94326894592,"internally_reserved":0,"allocated":1266577408,"data_stored":1265990760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738610,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2606780,"kb_used_data":2196588,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91138372,"statfs":{"total":95995035648,"available":93325692928,"internally_reserved":0,"allocated":2249306112,"data_stored":2248714064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2280194048,"data_stored":2280194048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1061224448,"data_stored":1061224448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":266862592,"data_stored":266862592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1996619776,"data_stored":1996619776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1424687104,"data_stored":1424687104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1201930240,"data_stored":1201930240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":819593216,"data_stored":819593216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1397817344,"data_stored":1397817344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1972568064,"data_stored":1972568064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1196883968,"data_stored":1196883968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:15:47.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:15:47.694 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:47.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:47.774 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:47.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:47.876 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:47.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:47.899 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:48.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:48.025 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:48.026 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:48.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:48.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:48.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:48.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 62 16 75535 75519 76.1212 50.25 0.00800405 0.0131315 2024-09-02T10:15:48.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:48.284 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:48.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:48.293 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:48.295 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:48.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:48.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:48.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:48.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:48.553 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:48.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:48.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:48 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:48.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:48 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:48.604 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:48 smithi151 ceph-mon[16639]: pgmap v1092: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 62 MiB/s wr, 994 op/s; 3633/217197 objects misplaced (1.673%); 1.8 MiB/s, 28 objects/s recovering 2024-09-02T10:15:48.605 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:48 smithi151 ceph-mon[16639]: osdmap e469: 12 total, 12 up, 10 in 2024-09-02T10:15:48.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:48.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:48.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:48 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:48 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:48 smithi187 ceph-mon[20072]: pgmap v1092: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 62 MiB/s wr, 994 op/s; 3633/217197 objects misplaced (1.673%); 1.8 MiB/s, 28 objects/s recovering 2024-09-02T10:15:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:48 smithi187 ceph-mon[20072]: osdmap e469: 12 total, 12 up, 10 in 2024-09-02T10:15:48.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:48.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:48.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:48.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:48 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:48 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:48 smithi162 ceph-mon[20279]: pgmap v1092: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 62 MiB/s wr, 994 op/s; 3633/217197 objects misplaced (1.673%); 1.8 MiB/s, 28 objects/s recovering 2024-09-02T10:15:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:48 smithi162 ceph-mon[20279]: osdmap e469: 12 total, 12 up, 10 in 2024-09-02T10:15:48.920 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:48.932 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:15:48.933 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:15:48.945 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:48.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:48.946 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:49.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:49.074 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:49.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:49.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:49.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:15:49.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 63 16 76539 76523 75.9088 62.75 0.00637027 0.0131673 2024-09-02T10:15:49.176 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:49.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:49.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:49.205 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:49.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:49.333 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:49.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:49.378 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:15:49.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:49 smithi151 ceph-mon[16639]: 5.2 scrub starts 2024-09-02T10:15:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:49 smithi151 ceph-mon[16639]: 5.6 scrub starts 2024-09-02T10:15:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/908360874' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:49 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:49 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:15:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:49 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:15:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:49 smithi151 ceph-mon[16639]: 5.9 scrub starts 2024-09-02T10:15:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:49.461 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:49.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:49.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:49.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:49.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:49.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:49.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:49.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:49 smithi187 ceph-mon[20072]: 5.2 scrub starts 2024-09-02T10:15:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:49 smithi187 ceph-mon[20072]: 5.6 scrub starts 2024-09-02T10:15:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/908360874' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:49 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:49 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:15:49.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:49 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:15:49.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:49 smithi187 ceph-mon[20072]: 5.9 scrub starts 2024-09-02T10:15:49.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:49.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:49 smithi162 ceph-mon[20279]: 5.2 scrub starts 2024-09-02T10:15:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:49 smithi162 ceph-mon[20279]: 5.6 scrub starts 2024-09-02T10:15:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/908360874' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:49 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:15:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:49 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:15:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:49 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:15:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:49 smithi162 ceph-mon[20279]: 5.9 scrub starts 2024-09-02T10:15:49.855 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:15:49.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:49.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:49.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:49.984 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:49.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:50.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:50.113 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:50.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:50.114 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:50.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 64 16 77618 77602 75.7764 67.4375 0.00606675 0.0131913 2024-09-02T10:15:50.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:50.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:50.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:50.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:50.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:50.366 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:50.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:50 smithi151 ceph-mon[16639]: 5.2 scrub ok 2024-09-02T10:15:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:50 smithi151 ceph-mon[16639]: pgmap v1094: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s; 3362/224157 objects misplaced (1.500%); 4.0 MiB/s, 63 objects/s recovering 2024-09-02T10:15:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:50 smithi151 ceph-mon[16639]: 5.6 scrub ok 2024-09-02T10:15:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:50 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:50 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:15:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:50 smithi151 ceph-mon[16639]: osdmap e470: 12 total, 12 up, 10 in 2024-09-02T10:15:50.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:50.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:50.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:50.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:50.622 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:50.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:50.631 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:50.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:50.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:50.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:50 smithi187 ceph-mon[20072]: 5.2 scrub ok 2024-09-02T10:15:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:50 smithi187 ceph-mon[20072]: pgmap v1094: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s; 3362/224157 objects misplaced (1.500%); 4.0 MiB/s, 63 objects/s recovering 2024-09-02T10:15:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:50 smithi187 ceph-mon[20072]: 5.6 scrub ok 2024-09-02T10:15:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:50 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:50 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:15:50.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:50 smithi187 ceph-mon[20072]: osdmap e470: 12 total, 12 up, 10 in 2024-09-02T10:15:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:50 smithi162 ceph-mon[20279]: 5.2 scrub ok 2024-09-02T10:15:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:50 smithi162 ceph-mon[20279]: pgmap v1094: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 78 MiB/s wr, 1.25k op/s; 3362/224157 objects misplaced (1.500%); 4.0 MiB/s, 63 objects/s recovering 2024-09-02T10:15:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:50 smithi162 ceph-mon[20279]: 5.6 scrub ok 2024-09-02T10:15:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:50 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:15:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:50 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:15:50.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:50 smithi162 ceph-mon[20279]: osdmap e470: 12 total, 12 up, 10 in 2024-09-02T10:15:50.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:50.880 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:50.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:50.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:51.011 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:51.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:51.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:51.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:51.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:51.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:51.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 65 16 78756 78740 75.7047 71.125 0.00997046 0.0132022 2024-09-02T10:15:51.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:51.262 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:51.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:51.364 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:51.387 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:51.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:51.388 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:51.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:51.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:15:51.516 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:15:51.516 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:51.516 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:51.516 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:51 smithi151 ceph-mon[16639]: osdmap e471: 12 total, 12 up, 10 in 2024-09-02T10:15:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:51.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:51.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:51.540 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:15:51.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:15:51.564 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:15:51.564 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:15:51.600 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:15:51.600 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:15:51.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:51.635 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:15:51.635 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:15:51.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:51.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:51.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:51.664 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:15:51.664 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:15:51.691 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:15:51.691 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:15:51.721 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:15:51.721 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:15:51.745 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:15:51.746 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:15:51.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:51 smithi187 ceph-mon[20072]: osdmap e471: 12 total, 12 up, 10 in 2024-09-02T10:15:51.772 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:15:51.772 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:15:51.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:51.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:51.798 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:15:51.799 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:15:51.824 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:15:51.825 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:15:51.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:51 smithi162 ceph-mon[20279]: osdmap e471: 12 total, 12 up, 10 in 2024-09-02T10:15:51.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:51.900 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:51.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:51.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:52.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:52.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:52.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:52.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:52.157 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:52.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:52.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:52.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 66 16 79765 79749 75.5131 63.0625 0.0482984 0.013237 2024-09-02T10:15:52.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:52.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:52.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:52.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:52.406 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:52.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:52.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:52.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:52.530 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:52.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:52.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:52.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:52.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:15:52.654 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:52.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:52 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:15:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:52 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:15:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:52 smithi187 ceph-mon[20072]: pgmap v1097: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 58 MiB/s wr, 928 op/s; 3362/225963 objects misplaced (1.488%); 2.8 MiB/s, 45 objects/s recovering 2024-09-02T10:15:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:52 smithi187 ceph-mon[20072]: osdmap e472: 12 total, 12 up, 10 in 2024-09-02T10:15:52.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/348550617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:52.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:52 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:15:52.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:52 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:15:52.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:52 smithi151 ceph-mon[16639]: pgmap v1097: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 58 MiB/s wr, 928 op/s; 3362/225963 objects misplaced (1.488%); 2.8 MiB/s, 45 objects/s recovering 2024-09-02T10:15:52.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:52 smithi151 ceph-mon[16639]: osdmap e472: 12 total, 12 up, 10 in 2024-09-02T10:15:52.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/348550617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:52.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:52 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:15:52.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:52 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:15:52.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:52 smithi162 ceph-mon[20279]: pgmap v1097: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.6 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 58 MiB/s wr, 928 op/s; 3362/225963 objects misplaced (1.488%); 2.8 MiB/s, 45 objects/s recovering 2024-09-02T10:15:52.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:52 smithi162 ceph-mon[20279]: osdmap e472: 12 total, 12 up, 10 in 2024-09-02T10:15:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/348550617' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:15:52.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:52.780 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:52.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:52.882 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:52.909 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:52.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:53.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:53.034 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:53.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:53.136 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:53.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:53.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:53.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:53.173 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 67 16 80649 80633 75.2106 55.25 0.0035545 0.01329 2024-09-02T10:15:53.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:53.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:53.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:53.356 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:15:53.357 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:15:53.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:53.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:53.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:53.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:53 smithi187 ceph-mon[20072]: from='client.16593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:15:53.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:53.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:53.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:53.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:53.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:53.677 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:53.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:53.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:53 smithi151 ceph-mon[16639]: from='client.16593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:15:53.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:53.806 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:53.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:53.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:53 smithi162 ceph-mon[20279]: from='client.16593 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:15:53.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:53.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:53.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:53.934 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:54.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:54.066 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:54.067 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:54.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:54.174 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 68 16 81751 81735 75.1171 68.875 0.0254834 0.0133068 2024-09-02T10:15:54.198 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:54.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:54.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:54.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:54.328 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:54.330 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:54.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:54.457 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:54.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:54.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:54.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:54.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:54.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:54.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:54.685 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:54.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:54 smithi151 ceph-mon[16639]: pgmap v1099: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 3137/237390 objects misplaced (1.321%); 5.2 MiB/s, 83 objects/s recovering 2024-09-02T10:15:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:54 smithi151 ceph-mon[16639]: osdmap e473: 12 total, 12 up, 10 in 2024-09-02T10:15:54.710 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:54.711 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:54 smithi187 ceph-mon[20072]: pgmap v1099: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 3137/237390 objects misplaced (1.321%); 5.2 MiB/s, 83 objects/s recovering 2024-09-02T10:15:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:54 smithi187 ceph-mon[20072]: osdmap e473: 12 total, 12 up, 10 in 2024-09-02T10:15:54.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:54.842 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:54.843 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:54.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:54 smithi162 ceph-mon[20279]: pgmap v1099: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 71 MiB/s wr, 1.14k op/s; 3137/237390 objects misplaced (1.321%); 5.2 MiB/s, 83 objects/s recovering 2024-09-02T10:15:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:54 smithi162 ceph-mon[20279]: osdmap e473: 12 total, 12 up, 10 in 2024-09-02T10:15:54.868 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:54.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:54.976 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:54.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:55.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:55.105 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:55.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:55.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:55.174 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 69 16 82890 82874 75.06 71.1875 0.00665521 0.0133172 2024-09-02T10:15:55.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:55.234 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:55.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:55.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:55.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:55.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:55.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:55.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:55.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:55.496 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:55.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:55.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:55.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:55.619 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:55.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:55.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:55.746 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:55.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:55.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:55.873 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:55.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:55.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:15:56.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:56.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:15:56.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:15:56.128 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:56.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:15:56.174 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 70 16 84126 84110 75.0912 77.25 0.00797513 0.013312 2024-09-02T10:15:56.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:56.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:56.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:56.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:56.376 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:56.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:56.432 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:15:56.432 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:15:56.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:15:56.432 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:15:56.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.432 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.24976.0:109490 3.b e263/226 3:d613b4ca:::benchmark_data_smithi187_59989_object109489:head v 263'6808, mlcod=263'6805)", 2024-09-02T10:15:56.432 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:32.978241+0000", 2024-09-02T10:15:56.432 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 503.45139470200002, 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.53835022099999996, 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978241+0000", 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978241+0000", 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978242+0000", 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.44e-07 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978264+0000", 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2433e-05 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.513667+0000", 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.53540241300000002 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.513682+0000", 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5447e-05 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.513769+0000", 2024-09-02T10:15:56.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7028999999999996e-05 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.513788+0000", 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.942e-05 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.516553+0000", 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027647320000000002 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.516591+0000", 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8203e-05 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:154979 3.1 3.edb17651 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.263594+0000", 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 452.16604205200002, 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99702418999999998, 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.437 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154979 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263594+0000", 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263594+0000", 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263596+0000", 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.782e-06 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263604+0000", 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1240000000000005e-06 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263619+0000", 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5542000000000001e-05 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263634+0000", 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4473e-05 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263716+0000", 2024-09-02T10:15:56.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1929999999999997e-05 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.263729+0000", 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3038e-05 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260590+0000", 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.996860996 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260618+0000", 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8305000000000001e-05 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:154986 3.e 3.aae52e7e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.287904+0000", 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 452.14173220499998, 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97255893000000004, 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154986 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287904+0000", 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287904+0000", 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287905+0000", 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.497e-06 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287912+0000", 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9999999999999999e-06 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287932+0000", 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9681e-05 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.287941+0000", 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4679999999999998e-06 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.288009+0000", 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7584999999999998e-05 2024-09-02T10:15:56.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.288020+0000", 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0696000000000001e-05 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260431+0000", 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97241165799999996 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260463+0000", 2024-09-02T10:15:56.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1344999999999999e-05 2024-09-02T10:15:56.447 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.447 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.447 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155002 3.1 3.ef033051 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.312679+0000", 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 452.11695651600002, 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.94808788799999999, 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155002 2024-09-02T10:15:56.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312679+0000", 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312679+0000", 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312681+0000", 2024-09-02T10:15:56.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.367e-06 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312686+0000", 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3460000000000001e-06 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312698+0000", 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1854e-05 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312717+0000", 2024-09-02T10:15:56.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8736000000000002e-05 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312773+0000", 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6093e-05 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.312781+0000", 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8900000000000007e-06 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260684+0000", 2024-09-02T10:15:56.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.94790293400000003 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260767+0000", 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3667999999999999e-05 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155008 3.e 3.a75576ae (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.318646+0000", 2024-09-02T10:15:56.452 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 452.11099012099999, 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.941864861, 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155008 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318646+0000", 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318646+0000", 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318647+0000", 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.232e-06 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318652+0000", 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1499999999999998e-06 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318668+0000", 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5542000000000001e-05 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318676+0000", 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4989999999999994e-06 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318753+0000", 2024-09-02T10:15:56.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7109999999999999e-05 2024-09-02T10:15:56.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:15:56.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.318764+0000", 2024-09-02T10:15:56.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0397000000000001e-05 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260492+0000", 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.94172796400000003 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260511+0000", 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8967e-05 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.457 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155020 3.1 3.bfce4141 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.337044+0000", 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 452.09259171799999, 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.92377231199999998, 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.458 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155020 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337044+0000", 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337044+0000", 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337045+0000", 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8999999999999995e-07 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337049+0000", 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5300000000000001e-06 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337057+0000", 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1610000000000003e-06 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337061+0000", 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5279999999999997e-06 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337104+0000", 2024-09-02T10:15:56.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2446e-05 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.337112+0000", 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9140000000000008e-06 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260807+0000", 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.92369535000000003 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260816+0000", 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4930000000000004e-06 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155022 3.1 3.961fe4e1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.345927+0000", 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 452.08370909799999, 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91494058099999998, 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.463 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155022 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345927+0000", 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345927+0000", 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345928+0000", 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.071e-06 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345932+0000", 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9079999999999997e-06 2024-09-02T10:15:56.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345941+0000", 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8319999999999995e-06 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345945+0000", 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0840000000000002e-06 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345990+0000", 2024-09-02T10:15:56.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5074999999999998e-05 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.345997+0000", 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6090000000000003e-06 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260857+0000", 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91485953099999995 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260867+0000", 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0471e-05 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155038 3.1 3.9395eab1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.393719+0000", 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 452.03591654399997, 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86722825800000003, 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.468 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155038 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393719+0000", 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393719+0000", 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393720+0000", 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9800000000000002e-07 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393724+0000", 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.873e-06 2024-09-02T10:15:56.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393733+0000", 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8689999999999993e-06 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393739+0000", 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4350000000000002e-06 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393784+0000", 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4614999999999999e-05 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.393792+0000", 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.537e-06 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260938+0000", 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.86714591900000004 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260948+0000", 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0012e-05 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155069 3.1 3.babc3b41 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.560041+0000", 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 451.86959442599999, 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70095849499999996, 2024-09-02T10:15:56.473 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155069 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560041+0000", 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560041+0000", 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560043+0000", 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.209e-06 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560047+0000", 2024-09-02T10:15:56.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8439999999999997e-06 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560059+0000", 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1732e-05 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560066+0000", 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7870000000000004e-06 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560147+0000", 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1073000000000001e-05 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.560159+0000", 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2398999999999999e-05 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260991+0000", 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70083152199999998 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.261000+0000", 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9290000000000004e-06 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155076 3.e 3.59c2747e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e282)", 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:24.799566+0000", 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 451.63006941999998, 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46109792199999999, 2024-09-02T10:15:56.478 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155076 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799566+0000", 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799566+0000", 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799568+0000", 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5990000000000001e-06 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799574+0000", 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7570000000000001e-06 2024-09-02T10:15:56.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799587+0000", 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3127e-05 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799594+0000", 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4030000000000001e-06 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799652+0000", 2024-09-02T10:15:56.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8124000000000001e-05 2024-09-02T10:15:56.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:15:56.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:24.799660+0000", 2024-09-02T10:15:56.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7090000000000002e-06 2024-09-02T10:15:56.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260593+0000", 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46093320900000001 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:25.260664+0000", 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0994000000000005e-05 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 285/283 [PushReplyOp(3:713d2e03:::benchmark_data_smithi187_59989_object51866:head)])", 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:38.615472+0000", 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 437.81416339200001, 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35497819400000002, 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615472+0000", 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615472+0000", 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615473+0000", 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2399999999999997e-07 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615480+0000", 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4350000000000002e-06 2024-09-02T10:15:56.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615509+0000", 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9081000000000002e-05 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.615516+0000", 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2119999999999997e-06 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.968094+0000", 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.352578265 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.970451+0000", 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023563770000000002 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.488 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 285/283 [PushReplyOp(3:713d2e03:::benchmark_data_smithi187_59989_object51866:head)])", 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:38.617571+0000", 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 437.81206455699999, 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37077970100000002, 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617571+0000", 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617571+0000", 2024-09-02T10:15:56.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617572+0000", 2024-09-02T10:15:56.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0999999999999997e-07 2024-09-02T10:15:56.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617579+0000", 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.37e-06 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617619+0000", 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.002e-05 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.617627+0000", 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9750000000000007e-06 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.981676+0000", 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.364049503 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:38.988351+0000", 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0066749230000000001 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(3.e 286/283 [PushReplyOp(3:79ed81d9:::benchmark_data_smithi187_59989_object55678:head)])", 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:45.857626+0000", 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 430.572009401, 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30484871099999999, 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857626+0000", 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857626+0000", 2024-09-02T10:15:56.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:56.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857627+0000", 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3600000000000003e-07 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857636+0000", 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7369999999999995e-06 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857687+0000", 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0785999999999998e-05 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:45.857695+0000", 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.337e-06 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:46.157534+0000", 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29983937500000002 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:46.162475+0000", 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049408400000000002 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:181220 3.1 3:87d3bc93:::benchmark_data_smithi187_59989_object71677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e291)", 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:09:15.130995+0000", 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 401.29864102200003, 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29205802199999997, 2024-09-02T10:15:56.498 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 181220 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.130995+0000", 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.130995+0000", 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.130997+0000", 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9240000000000001e-06 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.131003+0000", 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3060000000000003e-06 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.131016+0000", 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.278e-05 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.131022+0000", 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.319e-06 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.131280+0000", 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00025800900000000002 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.131462+0000", 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018176600000000001 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.131657+0000", 2024-09-02T10:15:56.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019551 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.145529+0000", 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013871424 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.145589+0000", 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9917000000000002e-05 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.421184+0000", 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27559520100000001 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.421289+0000", 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104539 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:15.423053+0000", 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001764327 2024-09-02T10:15:56.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.1b,RELEASE,e:293,from:8,reservation_nonce:0)", 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:09:22.169444+0000", 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 394.26019171299998, 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47728760399999998, 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.169444+0000", 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.169444+0000", 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.169444+0000", 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2100000000000001e-07 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.169447+0000", 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7999999999999999e-06 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.646631+0000", 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47718416299999999 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.646638+0000", 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9399999999999996e-06 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.646687+0000", 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8977000000000002e-05 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.646713+0000", 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5446999999999999e-05 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.646732+0000", 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9055999999999998e-05 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.508 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:219058 3.6 3:62e84818:::benchmark_data_smithi187_59989_object109515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:10:17.211328+0000", 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 339.21830767099999, 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38832308100000001, 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.509 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 219058 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211328+0000", 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211328+0000", 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211330+0000", 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.468e-06 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211335+0000", 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.631e-06 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211352+0000", 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6923e-05 2024-09-02T10:15:56.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211360+0000", 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9349999999999994e-06 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.218608+0000", 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072476090000000003 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:15:56.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.218737+0000", 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012934000000000001 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.218854+0000", 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117108 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.224834+0000", 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0059799290000000001 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.224873+0000", 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8544999999999998e-05 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.599414+0000", 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.374541611 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.599604+0000", 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018962800000000001 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.599651+0000", 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7354000000000001e-05 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:11574 4.3 e326/318 4:c9ca2732:::benchmark_data_smithi187_76542_object11573:head v 326'718, mlcod=326'716)", 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.852685+0000", 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 285.57695105099998, 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97050503600000004, 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.515 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.852685+0000", 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.852685+0000", 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.852690+0000", 2024-09-02T10:15:56.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.293e-06 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.852750+0000", 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9536000000000001e-05 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.822320+0000", 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.96957074300000001 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.822330+0000", 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1239999999999994e-06 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.822415+0000", 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5823999999999997e-05 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.822437+0000", 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1841999999999999e-05 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.823147+0000", 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00070947100000000004 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.823190+0000", 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3203e-05 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:42144 4.b 4:d62c2ece:::benchmark_data_smithi187_76542_object42143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-02T10:15:56.519 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:48.539483+0000", 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 247.89015244000001, 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.391093368, 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 42144 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539483+0000", 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539483+0000", 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539485+0000", 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3039999999999999e-06 2024-09-02T10:15:56.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539544+0000", 2024-09-02T10:15:56.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9614999999999998e-05 2024-09-02T10:15:56.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539557+0000", 2024-09-02T10:15:56.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2551000000000001e-05 2024-09-02T10:15:56.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539563+0000", 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4099999999999996e-06 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539618+0000", 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4221999999999997e-05 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:15:56.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539747+0000", 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000129806 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539861+0000", 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113622 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.541533+0000", 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001672394 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.547909+0000", 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006375932 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.930451+0000", 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38254196499999998 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.930525+0000", 2024-09-02T10:15:56.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3911000000000004e-05 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.930577+0000", 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1635999999999998e-05 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:92326 4.b 4:d3b75331:::benchmark_data_smithi187_76542_object92325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:33.492476+0000", 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 202.93715966299999, 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79601677199999998, 2024-09-02T10:15:56.526 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92326 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492476+0000", 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492476+0000", 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492478+0000", 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4219999999999999e-06 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492566+0000", 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8583000000000002e-05 2024-09-02T10:15:56.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492587+0000", 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1084e-05 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492596+0000", 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1880000000000008e-06 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492668+0000", 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1876000000000001e-05 2024-09-02T10:15:56.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492825+0000", 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015684100000000001 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492969+0000", 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000143783 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.495907+0000", 2024-09-02T10:15:56.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029384139999999999 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.496430+0000", 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052270400000000003 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288414+0000", 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79198343699999996 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288458+0000", 2024-09-02T10:15:56.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4030999999999999e-05 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288493+0000", 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5408999999999999e-05 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:152158 4.5 e408/343 4:a3442e1d:::benchmark_data_smithi187_76542_object59822:head v 408'5790, mlcod=408'5787)", 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:13:45.672293+0000", 2024-09-02T10:15:56.532 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 130.75734293299999, 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29650176900000003, 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:13:45.672293+0000", 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:13:45.672293+0000", 2024-09-02T10:15:56.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:13:45.672294+0000", 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.048e-06 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:13:45.672300+0000", 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8279999999999999e-06 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:15:56.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:13:45.966643+0000", 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.294343366 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:13:45.966650+0000", 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1949999999999997e-06 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:13:45.966700+0000", 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9829e-05 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:13:45.966730+0000", 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9663e-05 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:13:45.968762+0000", 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020318620000000002 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:13:45.968795+0000", 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2978000000000002e-05 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.536 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.537 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.537 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:15:56.537 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:15:56.537 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:15:56.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:56.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:56.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:56.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:56.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:56 smithi151 ceph-mon[16639]: pgmap v1101: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 53 MiB/s wr, 854 op/s; 3137/240258 objects misplaced (1.306%); 2.4 MiB/s, 38 objects/s recovering 2024-09-02T10:15:56.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:56.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:56 smithi187 ceph-mon[20072]: pgmap v1101: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 53 MiB/s wr, 854 op/s; 3137/240258 objects misplaced (1.306%); 2.4 MiB/s, 38 objects/s recovering 2024-09-02T10:15:56.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:15:56.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:56.819 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:15:56.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:56 smithi162 ceph-mon[20279]: pgmap v1101: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 53 MiB/s wr, 854 op/s; 3137/240258 objects misplaced (1.306%); 2.4 MiB/s, 38 objects/s recovering 2024-09-02T10:15:56.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:56.944 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:56.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:57.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:57.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:57.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:57.174 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 71 16 85011 84995 74.8126 55.3125 0.00770238 0.013361 2024-09-02T10:15:57.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:57.198 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:57.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:57.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:57.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:57.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:57.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:57.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:57.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:57.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:57.562 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1098,"stamp":"2024-09-02T10:15:51.391928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4936696352,"num_objects":75321,"num_object_clones":0,"num_object_copies":225963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3362,"num_objects_unfound":0,"num_objects_dirty":75321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75319,"num_write_kb":4820416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2349,"num_bytes_recovered":154477632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78171,"ondisk_log_size":78171,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":18091136,"kb_used_data":14666704,"kb_used_omap":15,"kb_used_meta":3424240,"kb_avail":919360384,"statfs":{"total":959950356480,"available":941425033216,"internally_reserved":0,"allocated":15018704896,"data_stored":15012702480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3506422193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":191496192,"num_objects":2922,"num_object_clones":0,"num_object_copies":8766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-271,"num_objects_unfound":0,"num_objects_dirty":2922,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2922,"num_write_kb":187008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":271,"num_bytes_recovered":17760256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.099703"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":262,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.355885+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:15:44.355885+0000","last_peered":"2024-09-02T10:15:44.355885+0000","last_clean":"2024-09-02T10:15:44.355885+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:15:44.355885+0000","last_undegraded":"2024-09-02T10:15:44.355885+0000","last_fullsized":"2024-09-02T10:15:44.355885+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:56.714204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":73,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.375753+0000","last_change":"2024-09-02T10:15:41.899476+0000","last_active":"2024-09-02T10:15:47.375753+0000","last_peered":"2024-09-02T10:15:47.375753+0000","last_clean":"2024-09-02T10:15:47.375753+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:15:47.375753+0000","last_undegraded":"2024-09-02T10:15:47.375753+0000","last_fullsized":"2024-09-02T10:15:47.375753+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:41.899397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:41.899397+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:51.002146+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":522,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.373314+0000","last_change":"2024-09-02T10:15:12.091472+0000","last_active":"2024-09-02T10:15:47.373314+0000","last_peered":"2024-09-02T10:15:47.373314+0000","last_clean":"2024-09-02T10:15:47.373314+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:15:47.373314+0000","last_undegraded":"2024-09-02T10:15:47.373314+0000","last_fullsized":"2024-09-02T10:15:47.373314+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.091410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.091410+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:27.631420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":522,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.372910+0000","last_change":"2024-09-02T10:15:19.065187+0000","last_active":"2024-09-02T10:15:47.372910+0000","last_peered":"2024-09-02T10:15:47.372910+0000","last_clean":"2024-09-02T10:15:47.372910+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:15:47.372910+0000","last_undegraded":"2024-09-02T10:15:47.372910+0000","last_fullsized":"2024-09-02T10:15:47.372910+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:19.065134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:19.065134+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:36.285460+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":447,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-02T10:15:50.389886+0000","last_change":"2024-09-02T10:15:35.202261+0000","last_active":"2024-09-02T10:15:50.389886+0000","last_peered":"2024-09-02T10:15:50.389886+0000","last_clean":"2024-09-02T10:15:50.389886+0000","last_became_active":"2024-09-02T10:15:35.201887+0000","last_became_peered":"2024-09-02T10:15:35.201887+0000","last_unstale":"2024-09-02T10:15:50.389886+0000","last_undegraded":"2024-09-02T10:15:50.389886+0000","last_fullsized":"2024-09-02T10:15:50.389886+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:04.994667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:04.994667+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:30.739058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":370,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.356329+0000","last_change":"2024-09-02T10:14:11.689851+0000","last_active":"2024-09-02T10:15:44.356329+0000","last_peered":"2024-09-02T10:15:44.356329+0000","last_clean":"2024-09-02T10:15:44.356329+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:15:44.356329+0000","last_undegraded":"2024-09-02T10:15:44.356329+0000","last_fullsized":"2024-09-02T10:15:44.356329+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.689767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":885,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.354348+0000","last_change":"2024-09-02T10:15:07.059223+0000","last_active":"2024-09-02T10:15:44.354348+0000","last_peered":"2024-09-02T10:15:44.354348+0000","last_clean":"2024-09-02T10:15:44.354348+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:15:44.354348+0000","last_undegraded":"2024-09-02T10:15:44.354348+0000","last_fullsized":"2024-09-02T10:15:44.354348+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:45.351862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:45.351862+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:57.344255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":529,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.373057+0000","last_change":"2024-09-02T10:15:05.094874+0000","last_active":"2024-09-02T10:15:47.373057+0000","last_peered":"2024-09-02T10:15:47.373057+0000","last_clean":"2024-09-02T10:15:47.373057+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:15:47.373057+0000","last_undegraded":"2024-09-02T10:15:47.373057+0000","last_fullsized":"2024-09-02T10:15:47.373057+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:05.094792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:05.094792+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:09.107562+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":536,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.375859+0000","last_change":"2024-09-02T10:15:34.791760+0000","last_active":"2024-09-02T10:15:47.375859+0000","last_peered":"2024-09-02T10:15:47.375859+0000","last_clean":"2024-09-02T10:15:47.375859+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:15:47.375859+0000","last_undegraded":"2024-09-02T10:15:47.375859+0000","last_fullsized":"2024-09-02T10:15:47.375859+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:34.791708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:34.791708+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.857252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":509,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-02T10:15:50.390309+0000","last_change":"2024-09-02T10:14:49.165646+0000","last_active":"2024-09-02T10:15:50.390309+0000","last_peered":"2024-09-02T10:15:50.390309+0000","last_clean":"2024-09-02T10:15:50.390309+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:15:50.390309+0000","last_undegraded":"2024-09-02T10:15:50.390309+0000","last_fullsized":"2024-09-02T10:15:50.390309+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:49.165586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:49.165586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.197404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":361,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-02T10:15:50.390220+0000","last_change":"2024-09-02T10:14:45.203339+0000","last_active":"2024-09-02T10:15:50.390220+0000","last_peered":"2024-09-02T10:15:50.390220+0000","last_clean":"2024-09-02T10:15:50.390220+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:15:50.390220+0000","last_undegraded":"2024-09-02T10:15:50.390220+0000","last_fullsized":"2024-09-02T10:15:50.390220+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:45.203278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:45.203278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:51.594287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":862,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.375692+0000","last_change":"2024-09-02T10:15:06.447417+0000","last_active":"2024-09-02T10:15:47.375692+0000","last_peered":"2024-09-02T10:15:47.375692+0000","last_clean":"2024-09-02T10:15:47.375692+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:15:47.375692+0000","last_undegraded":"2024-09-02T10:15:47.375692+0000","last_fullsized":"2024-09-02T10:15:47.375692+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:06.447372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:06.447372+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:13.751465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":795,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.354479+0000","last_change":"2024-09-02T10:15:07.059344+0000","last_active":"2024-09-02T10:15:44.354479+0000","last_peered":"2024-09-02T10:15:44.354479+0000","last_clean":"2024-09-02T10:15:44.354479+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:15:44.354479+0000","last_undegraded":"2024-09-02T10:15:44.354479+0000","last_fullsized":"2024-09-02T10:15:44.354479+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.252526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.252526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:51.953514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":784,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.371966+0000","last_change":"2024-09-02T10:15:29.170702+0000","last_active":"2024-09-02T10:15:47.371966+0000","last_peered":"2024-09-02T10:15:47.371966+0000","last_clean":"2024-09-02T10:15:47.371966+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:15:47.371966+0000","last_undegraded":"2024-09-02T10:15:47.371966+0000","last_fullsized":"2024-09-02T10:15:47.371966+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.665438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.665438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:04.476082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":773,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.354371+0000","last_change":"2024-09-02T10:14:34.217697+0000","last_active":"2024-09-02T10:15:44.354371+0000","last_peered":"2024-09-02T10:15:44.354371+0000","last_clean":"2024-09-02T10:15:44.354371+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:15:44.354371+0000","last_undegraded":"2024-09-02T10:15:44.354371+0000","last_fullsized":"2024-09-02T10:15:44.354371+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.217624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.217624+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:02.148154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"469'4661","reported_seq":8302,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:48.589023+0000","last_change":"2024-09-02T10:15:24.164550+0000","last_active":"2024-09-02T10:15:48.589023+0000","last_peered":"2024-09-02T10:15:48.589023+0000","last_clean":"2024-09-02T10:15:48.589023+0000","last_became_active":"2024-09-02T10:15:24.141368+0000","last_became_peered":"2024-09-02T10:15:24.141368+0000","last_unstale":"2024-09-02T10:15:48.589023+0000","last_undegraded":"2024-09-02T10:15:48.589023+0000","last_fullsized":"2024-09-02T10:15:48.589023+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:51.435130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305463296,"num_objects":4661,"num_object_clones":0,"num_object_copies":13983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4661,"num_write_kb":298304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":115933184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":860,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.354483+0000","last_change":"2024-09-02T10:14:41.306927+0000","last_active":"2024-09-02T10:15:44.354483+0000","last_peered":"2024-09-02T10:15:44.354483+0000","last_clean":"2024-09-02T10:15:44.354483+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:15:44.354483+0000","last_undegraded":"2024-09-02T10:15:44.354483+0000","last_fullsized":"2024-09-02T10:15:44.354483+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:41.306868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:41.306868+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:04.992960+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"468'4704","reported_seq":4789,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.067718+0000","last_change":"2024-09-02T10:14:39.040351+0000","last_active":"2024-09-02T10:15:47.067718+0000","last_peered":"2024-09-02T10:15:47.067718+0000","last_clean":"2024-09-02T10:15:47.067718+0000","last_became_active":"2024-09-02T10:14:39.040025+0000","last_became_peered":"2024-09-02T10:14:39.040025+0000","last_unstale":"2024-09-02T10:15:47.067718+0000","last_undegraded":"2024-09-02T10:15:47.067718+0000","last_fullsized":"2024-09-02T10:15:47.067718+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4704,"log_dups_size":0,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:54.413941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308281344,"num_objects":4704,"num_object_clones":0,"num_object_copies":14112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4704,"num_write_kb":301056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":981,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-02T10:15:50.391238+0000","last_change":"2024-09-02T10:14:36.143055+0000","last_active":"2024-09-02T10:15:50.391238+0000","last_peered":"2024-09-02T10:15:50.391238+0000","last_clean":"2024-09-02T10:15:50.391238+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:15:50.391238+0000","last_undegraded":"2024-09-02T10:15:50.391238+0000","last_fullsized":"2024-09-02T10:15:50.391238+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.142993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.142993+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.623989+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"469'4644","reported_seq":4731,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.903297+0000","last_change":"2024-09-02T10:14:39.038506+0000","last_active":"2024-09-02T10:15:47.903297+0000","last_peered":"2024-09-02T10:15:47.903297+0000","last_clean":"2024-09-02T10:15:47.903297+0000","last_became_active":"2024-09-02T10:14:39.038164+0000","last_became_peered":"2024-09-02T10:14:39.038164+0000","last_unstale":"2024-09-02T10:15:47.903297+0000","last_undegraded":"2024-09-02T10:15:47.903297+0000","last_fullsized":"2024-09-02T10:15:47.903297+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4644,"log_dups_size":0,"ondisk_log_size":4644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:50.238502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304349184,"num_objects":4644,"num_object_clones":0,"num_object_copies":13932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4644,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4644,"num_write_kb":297216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":619,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.375593+0000","last_change":"2024-09-02T10:14:31.581643+0000","last_active":"2024-09-02T10:15:47.375593+0000","last_peered":"2024-09-02T10:15:47.375593+0000","last_clean":"2024-09-02T10:15:47.375593+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:15:47.375593+0000","last_undegraded":"2024-09-02T10:15:47.375593+0000","last_fullsized":"2024-09-02T10:15:47.375593+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.581597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.581597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.091930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"468'4735","reported_seq":4820,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.014657+0000","last_change":"2024-09-02T10:14:39.037573+0000","last_active":"2024-09-02T10:15:47.014657+0000","last_peered":"2024-09-02T10:15:47.014657+0000","last_clean":"2024-09-02T10:15:47.014657+0000","last_became_active":"2024-09-02T10:14:39.037121+0000","last_became_peered":"2024-09-02T10:14:39.037121+0000","last_unstale":"2024-09-02T10:15:47.014657+0000","last_undegraded":"2024-09-02T10:15:47.014657+0000","last_fullsized":"2024-09-02T10:15:47.014657+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4735,"log_dups_size":0,"ondisk_log_size":4735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:01.026599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310312960,"num_objects":4735,"num_object_clones":0,"num_object_copies":14205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4735,"num_write_kb":303040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":22,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.355700+0000","last_change":"2024-09-02T10:15:35.208246+0000","last_active":"2024-09-02T10:15:44.355700+0000","last_peered":"2024-09-02T10:15:44.355700+0000","last_clean":"2024-09-02T10:15:44.355700+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:15:44.355700+0000","last_undegraded":"2024-09-02T10:15:44.355700+0000","last_fullsized":"2024-09-02T10:15:44.355700+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.668602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.668602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"468'4555","reported_seq":4639,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:46.614987+0000","last_change":"2024-09-02T10:14:39.642395+0000","last_active":"2024-09-02T10:15:46.614987+0000","last_peered":"2024-09-02T10:15:46.614987+0000","last_clean":"2024-09-02T10:15:46.614987+0000","last_became_active":"2024-09-02T10:14:39.641995+0000","last_became_peered":"2024-09-02T10:14:39.641995+0000","last_unstale":"2024-09-02T10:15:46.614987+0000","last_undegraded":"2024-09-02T10:15:46.614987+0000","last_fullsized":"2024-09-02T10:15:46.614987+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4555,"log_dups_size":0,"ondisk_log_size":4555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:55.658135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298516480,"num_objects":4555,"num_object_clones":0,"num_object_copies":13665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4555,"num_write_kb":291520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":930,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.355779+0000","last_change":"2024-09-02T10:15:29.195341+0000","last_active":"2024-09-02T10:15:44.355779+0000","last_peered":"2024-09-02T10:15:44.355779+0000","last_clean":"2024-09-02T10:15:44.355779+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:15:44.355779+0000","last_undegraded":"2024-09-02T10:15:44.355779+0000","last_fullsized":"2024-09-02T10:15:44.355779+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.277119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.277119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:28.784954+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"469'4737","reported_seq":4824,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:48.874918+0000","last_change":"2024-09-02T10:14:39.034903+0000","last_active":"2024-09-02T10:15:48.874918+0000","last_peered":"2024-09-02T10:15:48.874918+0000","last_clean":"2024-09-02T10:15:48.874918+0000","last_became_active":"2024-09-02T10:14:39.034636+0000","last_became_peered":"2024-09-02T10:14:39.034636+0000","last_unstale":"2024-09-02T10:15:48.874918+0000","last_undegraded":"2024-09-02T10:15:48.874918+0000","last_fullsized":"2024-09-02T10:15:48.874918+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4737,"log_dups_size":0,"ondisk_log_size":4737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:07.235133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310444032,"num_objects":4737,"num_object_clones":0,"num_object_copies":14211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":303168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":398,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.356173+0000","last_change":"2024-09-02T10:15:35.205684+0000","last_active":"2024-09-02T10:15:44.356173+0000","last_peered":"2024-09-02T10:15:44.356173+0000","last_clean":"2024-09-02T10:15:44.356173+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:15:44.356173+0000","last_undegraded":"2024-09-02T10:15:44.356173+0000","last_fullsized":"2024-09-02T10:15:44.356173+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.701773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.701773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.566711+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"468'4615","reported_seq":4700,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.182137+0000","last_change":"2024-09-02T10:14:39.038158+0000","last_active":"2024-09-02T10:15:47.182137+0000","last_peered":"2024-09-02T10:15:47.182137+0000","last_clean":"2024-09-02T10:15:47.182137+0000","last_became_active":"2024-09-02T10:14:39.037822+0000","last_became_peered":"2024-09-02T10:14:39.037822+0000","last_unstale":"2024-09-02T10:15:47.182137+0000","last_undegraded":"2024-09-02T10:15:47.182137+0000","last_fullsized":"2024-09-02T10:15:47.182137+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4615,"log_dups_size":0,"ondisk_log_size":4615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:47.907922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302448640,"num_objects":4615,"num_object_clones":0,"num_object_copies":13845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4615,"num_write_kb":295360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":370,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.356654+0000","last_change":"2024-09-02T10:14:09.768298+0000","last_active":"2024-09-02T10:15:44.356654+0000","last_peered":"2024-09-02T10:15:44.356654+0000","last_clean":"2024-09-02T10:15:44.356654+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:15:44.356654+0000","last_undegraded":"2024-09-02T10:15:44.356654+0000","last_fullsized":"2024-09-02T10:15:44.356654+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:09.768061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:09.768061+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"469'4839","reported_seq":4926,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.685784+0000","last_change":"2024-09-02T10:14:39.035303+0000","last_active":"2024-09-02T10:15:47.685784+0000","last_peered":"2024-09-02T10:15:47.685784+0000","last_clean":"2024-09-02T10:15:47.685784+0000","last_became_active":"2024-09-02T10:14:39.035100+0000","last_became_peered":"2024-09-02T10:14:39.035100+0000","last_unstale":"2024-09-02T10:15:47.685784+0000","last_undegraded":"2024-09-02T10:15:47.685784+0000","last_fullsized":"2024-09-02T10:15:47.685784+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317128704,"num_objects":4839,"num_object_clones":0,"num_object_copies":14517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4839,"num_write_kb":309696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":121,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.352257+0000","last_change":"2024-09-02T10:15:36.299961+0000","last_active":"2024-09-02T10:15:44.352257+0000","last_peered":"2024-09-02T10:15:44.352257+0000","last_clean":"2024-09-02T10:15:44.352257+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:15:44.352257+0000","last_undegraded":"2024-09-02T10:15:44.352257+0000","last_fullsized":"2024-09-02T10:15:44.352257+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:36.299914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:36.299914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:01.870130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"469'4720","reported_seq":4807,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.699548+0000","last_change":"2024-09-02T10:14:39.641746+0000","last_active":"2024-09-02T10:15:47.699548+0000","last_peered":"2024-09-02T10:15:47.699548+0000","last_clean":"2024-09-02T10:15:47.699548+0000","last_became_active":"2024-09-02T10:14:39.641478+0000","last_became_peered":"2024-09-02T10:14:39.641478+0000","last_unstale":"2024-09-02T10:15:47.699548+0000","last_undegraded":"2024-09-02T10:15:47.699548+0000","last_fullsized":"2024-09-02T10:15:47.699548+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4720,"log_dups_size":0,"ondisk_log_size":4720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:56.685441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309329920,"num_objects":4720,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4720,"num_write_kb":302080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":371,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.356240+0000","last_change":"2024-09-02T10:14:36.623992+0000","last_active":"2024-09-02T10:15:44.356240+0000","last_peered":"2024-09-02T10:15:44.356240+0000","last_clean":"2024-09-02T10:15:44.356240+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:15:44.356240+0000","last_undegraded":"2024-09-02T10:15:44.356240+0000","last_fullsized":"2024-09-02T10:15:44.356240+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.623930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.623930+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:49.218010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"468'4570","reported_seq":4654,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.029991+0000","last_change":"2024-09-02T10:14:39.038216+0000","last_active":"2024-09-02T10:15:47.029991+0000","last_peered":"2024-09-02T10:15:47.029991+0000","last_clean":"2024-09-02T10:15:47.029991+0000","last_became_active":"2024-09-02T10:14:39.037882+0000","last_became_peered":"2024-09-02T10:14:39.037882+0000","last_unstale":"2024-09-02T10:15:47.029991+0000","last_undegraded":"2024-09-02T10:15:47.029991+0000","last_fullsized":"2024-09-02T10:15:47.029991+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299499520,"num_objects":4570,"num_object_clones":0,"num_object_copies":13710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4570,"num_write_kb":292480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":623,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-02T10:15:50.390066+0000","last_change":"2024-09-02T10:14:40.192356+0000","last_active":"2024-09-02T10:15:50.390066+0000","last_peered":"2024-09-02T10:15:50.390066+0000","last_clean":"2024-09-02T10:15:50.390066+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:15:50.390066+0000","last_undegraded":"2024-09-02T10:15:50.390066+0000","last_fullsized":"2024-09-02T10:15:50.390066+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.192297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.192297+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.932788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"468'4652","reported_seq":4737,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.026874+0000","last_change":"2024-09-02T10:14:39.642277+0000","last_active":"2024-09-02T10:15:47.026874+0000","last_peered":"2024-09-02T10:15:47.026874+0000","last_clean":"2024-09-02T10:15:47.026874+0000","last_became_active":"2024-09-02T10:14:39.641932+0000","last_became_peered":"2024-09-02T10:14:39.641932+0000","last_unstale":"2024-09-02T10:15:47.026874+0000","last_undegraded":"2024-09-02T10:15:47.026874+0000","last_fullsized":"2024-09-02T10:15:47.026874+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:55.249885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304873472,"num_objects":4652,"num_object_clones":0,"num_object_copies":13956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4652,"num_write_kb":297728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":878,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.375826+0000","last_change":"2024-09-02T10:15:29.169027+0000","last_active":"2024-09-02T10:15:47.375826+0000","last_peered":"2024-09-02T10:15:47.375826+0000","last_clean":"2024-09-02T10:15:47.375826+0000","last_became_active":"2024-09-02T10:15:29.168692+0000","last_became_peered":"2024-09-02T10:15:29.168692+0000","last_unstale":"2024-09-02T10:15:47.375826+0000","last_undegraded":"2024-09-02T10:15:47.375826+0000","last_fullsized":"2024-09-02T10:15:47.375826+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.804014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.804014+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:54.886555+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"468'4719","reported_seq":4802,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:46.604972+0000","last_change":"2024-09-02T10:14:39.038104+0000","last_active":"2024-09-02T10:15:46.604972+0000","last_peered":"2024-09-02T10:15:46.604972+0000","last_clean":"2024-09-02T10:15:46.604972+0000","last_became_active":"2024-09-02T10:14:39.037755+0000","last_became_peered":"2024-09-02T10:14:39.037755+0000","last_unstale":"2024-09-02T10:15:46.604972+0000","last_undegraded":"2024-09-02T10:15:46.604972+0000","last_fullsized":"2024-09-02T10:15:46.604972+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:00.080239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309264384,"num_objects":4719,"num_object_clones":0,"num_object_copies":14157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4719,"num_write_kb":302016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":641,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-02T10:15:50.389797+0000","last_change":"2024-09-02T10:15:35.202850+0000","last_active":"2024-09-02T10:15:50.389797+0000","last_peered":"2024-09-02T10:15:50.389797+0000","last_clean":"2024-09-02T10:15:50.389797+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:15:50.389797+0000","last_undegraded":"2024-09-02T10:15:50.389797+0000","last_fullsized":"2024-09-02T10:15:50.389797+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:48.187420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:48.187420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:57.257578+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"469'4642","reported_seq":4728,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:47.938348+0000","last_change":"2024-09-02T10:14:39.039399+0000","last_active":"2024-09-02T10:15:47.938348+0000","last_peered":"2024-09-02T10:15:47.938348+0000","last_clean":"2024-09-02T10:15:47.938348+0000","last_became_active":"2024-09-02T10:14:39.039060+0000","last_became_peered":"2024-09-02T10:14:39.039060+0000","last_unstale":"2024-09-02T10:15:47.938348+0000","last_undegraded":"2024-09-02T10:15:47.938348+0000","last_fullsized":"2024-09-02T10:15:47.938348+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:48.488192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304218112,"num_objects":4642,"num_object_clones":0,"num_object_copies":13926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4642,"num_write_kb":297088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":875,"reported_epoch":468,"state":"active+clean","last_fresh":"2024-09-02T10:15:44.352361+0000","last_change":"2024-09-02T10:14:34.284618+0000","last_active":"2024-09-02T10:15:44.352361+0000","last_peered":"2024-09-02T10:15:44.352361+0000","last_clean":"2024-09-02T10:15:44.352361+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:15:44.352361+0000","last_undegraded":"2024-09-02T10:15:44.352361+0000","last_fullsized":"2024-09-02T10:15:44.352361+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.284563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.284563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:59.753379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"469'4799","reported_seq":4882,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:49.290208+0000","last_change":"2024-09-02T10:14:39.036854+0000","last_active":"2024-09-02T10:15:49.290208+0000","last_peered":"2024-09-02T10:15:49.290208+0000","last_clean":"2024-09-02T10:15:49.290208+0000","last_became_active":"2024-09-02T10:14:39.036453+0000","last_became_peered":"2024-09-02T10:14:39.036453+0000","last_unstale":"2024-09-02T10:15:49.290208+0000","last_undegraded":"2024-09-02T10:15:49.290208+0000","last_fullsized":"2024-09-02T10:15:49.290208+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:57.124824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314507264,"num_objects":4799,"num_object_clones":0,"num_object_copies":14397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4799,"num_write_kb":307136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":416,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-02T10:15:50.391514+0000","last_change":"2024-09-02T10:14:47.184482+0000","last_active":"2024-09-02T10:15:50.391514+0000","last_peered":"2024-09-02T10:15:50.391514+0000","last_clean":"2024-09-02T10:15:50.391514+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:15:50.391514+0000","last_undegraded":"2024-09-02T10:15:50.391514+0000","last_fullsized":"2024-09-02T10:15:50.391514+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:47.184420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:47.184420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:53.550061+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"471'4972","reported_seq":5083,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-02T10:15:51.031393+0000","last_change":"2024-09-02T10:15:49.395716+0000","last_active":"2024-09-02T10:15:51.031393+0000","last_peered":"2024-09-02T10:15:51.031393+0000","last_clean":"2024-09-02T10:15:51.031393+0000","last_became_active":"2024-09-02T10:14:39.037567+0000","last_became_peered":"2024-09-02T10:14:39.037567+0000","last_unstale":"2024-09-02T10:15:51.031393+0000","last_undegraded":"2024-09-02T10:15:51.031393+0000","last_fullsized":"2024-09-02T10:15:51.031393+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":1699,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:08.193439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325844992,"num_objects":4972,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4972,"num_write_kb":318208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":661,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-09-02T10:15:50.390402+0000","last_change":"2024-09-02T10:15:31.899734+0000","last_active":"2024-09-02T10:15:50.390402+0000","last_peered":"2024-09-02T10:15:50.390402+0000","last_clean":"2024-09-02T10:15:50.390402+0000","last_became_active":"2024-09-02T10:15:29.167553+0000","last_became_peered":"2024-09-02T10:15:29.167553+0000","last_unstale":"2024-09-02T10:15:50.390402+0000","last_undegraded":"2024-09-02T10:15:50.390402+0000","last_fullsized":"2024-09-02T10:15:50.390402+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:31.899662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:31.899662+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.920777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"469'4755","reported_seq":5998,"reported_epoch":469,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:15:47.711560+0000","last_change":"2024-09-02T10:15:36.347809+0000","last_active":"2024-09-02T10:15:47.711560+0000","last_peered":"2024-09-02T10:15:47.711560+0000","last_clean":"2024-09-02T10:15:34.173584+0000","last_became_active":"2024-09-02T10:15:36.297925+0000","last_became_peered":"2024-09-02T10:15:36.297925+0000","last_unstale":"2024-09-02T10:15:47.711560+0000","last_undegraded":"2024-09-02T10:15:47.711560+0000","last_fullsized":"2024-09-02T10:15:36.200774+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311623680,"num_objects":4755,"num_object_clones":0,"num_object_copies":14265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3362,"num_objects_unfound":0,"num_objects_dirty":4755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4755,"num_write_kb":304320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":577,"num_bytes_recovered":37814272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[0,10],"avail_no_missing":["0","9","10"],"object_location_counts":[{"shards":"0,9,10","objects":3362},{"shards":"0,10","objects":1393}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":64,"reported_epoch":469,"state":"active+clean","last_fresh":"2024-09-02T10:15:48.739408+0000","last_change":"2024-09-02T10:15:48.739408+0000","last_active":"2024-09-02T10:15:48.739408+0000","last_peered":"2024-09-02T10:15:48.739408+0000","last_clean":"2024-09-02T10:15:48.739408+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:15:48.739408+0000","last_undegraded":"2024-09-02T10:15:48.739408+0000","last_fullsized":"2024-09-02T10:15:48.739408+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:48.739340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:48.739340+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:02.899055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4936105984,"num_objects":75319,"num_object_clones":0,"num_object_copies":225957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3362,"num_objects_unfound":0,"num_objects_dirty":75319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":75319,"num_write_kb":4820416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2346,"num_bytes_recovered":153747456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14909243392,"data_stored":14909243392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75319,"ondisk_log_size":75319,"up":48,"acting":47,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383868,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1629384,"kb_used_data":1273336,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92115768,"statfs":{"total":95995035648,"available":94326546432,"internally_reserved":0,"allocated":1303896064,"data_stored":1303299561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":364575172},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612801,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2239816,"kb_used_data":2064248,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91505336,"statfs":{"total":95995035648,"available":93701464064,"internally_reserved":0,"allocated":2113789952,"data_stored":2113193449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":179763648},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841734,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1869576,"kb_used_data":1437176,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91875576,"statfs":{"total":95995035648,"available":94080589824,"internally_reserved":0,"allocated":1471668224,"data_stored":1471071721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070668,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137748,"kb_used_data":921924,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":92607404,"statfs":{"total":95995035648,"available":94829981696,"internally_reserved":0,"allocated":944050176,"data_stored":943449218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299601,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1730172,"kb_used_data":1319972,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92014980,"statfs":{"total":95995035648,"available":94223339520,"internally_reserved":0,"allocated":1351651328,"data_stored":1351037908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":420018634},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528532,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1763144,"kb_used_data":1501496,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":91982008,"statfs":{"total":95995035648,"available":94189576192,"internally_reserved":0,"allocated":1537531904,"data_stored":1536935401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":267909575},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724761,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2459028,"kb_used_data":2080768,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":91286124,"statfs":{"total":95995035648,"available":93476990976,"internally_reserved":0,"allocated":2130706432,"data_stored":2130106377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149924,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":692564,"kb_used_data":304580,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93052588,"statfs":{"total":95995035648,"available":95285850112,"internally_reserved":0,"allocated":311889920,"data_stored":311288962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":397277640},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411559,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575085,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1790752,"kb_used_data":1390284,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91954400,"statfs":{"total":95995035648,"available":94161305600,"internally_reserved":0,"allocated":1423650816,"data_stored":1423046306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738611,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2778952,"kb_used_data":2372920,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":90966200,"statfs":{"total":95995035648,"available":93149388800,"internally_reserved":0,"allocated":2429870080,"data_stored":2429273577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2454454272,"data_stored":2454454272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1235681280,"data_stored":1235681280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":310444032,"data_stored":310444032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2149515264,"data_stored":2149515264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1535574016,"data_stored":1535574016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1350500352,"data_stored":1350500352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":943456256,"data_stored":943456256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1481506816,"data_stored":1481506816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2125660160,"data_stored":2125660160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1322450944,"data_stored":1322450944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:15:57.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:15:57.586 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:57.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:57.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:57.714 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:57.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:57.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:57.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:57.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:57.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:57.971 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:57.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:58.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:15:58.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:58.086 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:58.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:58.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:58.146 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:58.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:15:58.174 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 72 16 86207 86191 74.8116 74.75 0.00502699 0.013362 2024-09-02T10:15:58.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:15:58.245 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:58.246 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:15:58.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:58.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:58.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:58.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:58.479 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:58.508 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:58.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:58.512 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:58 smithi151 ceph-mon[16639]: pgmap v1102: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s; 3137/248013 objects misplaced (1.265%); 2.0 MiB/s, 32 objects/s recovering 2024-09-02T10:15:58.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:15:58.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:58.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:58.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:15:58.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:58 smithi187 ceph-mon[20072]: pgmap v1102: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s; 3137/248013 objects misplaced (1.265%); 2.0 MiB/s, 32 objects/s recovering 2024-09-02T10:15:58.756 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:58.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:58.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:58 smithi162 ceph-mon[20279]: pgmap v1102: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 69 MiB/s wr, 1.11k op/s; 3137/248013 objects misplaced (1.265%); 2.0 MiB/s, 32 objects/s recovering 2024-09-02T10:15:58.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:15:58.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:58.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:58.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:15:58.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:15:59.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:15:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:59.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:15:59.115 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:15:59.142 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:59.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:15:59.174 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 73 16 87163 87147 74.6052 59.75 0.00977484 0.0133987 2024-09-02T10:15:59.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:15:59.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:15:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:59.275 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:15:59.360 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:59.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:59.391 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:15:59.398 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:59.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:59.408 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:15:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1665717425' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:59.419 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:15:59.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:59.526 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:59.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:15:59.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:15:59.631 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:15:59.631 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1231,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":474,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":10,"osd_in_since":1725272054,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":85052,"data_bytes":5574427168,"bytes_used":20485185536,"bytes_avail":939465170944,"bytes_total":959950356480,"misplaced_objects":2880,"misplaced_total":255156,"misplaced_ratio":0.011287212528805908,"recovering_objects_per_sec":60,"recovering_bytes_per_sec":3947745,"recovering_keys_per_sec":0,"num_objects_recovered":482,"num_bytes_recovered":31588352,"num_keys_recovered":0,"write_bytes_sec":79700237,"read_op_per_sec":0,"write_op_per_sec":1216},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":33,"modified":"2024-09-02T10:15:39.290520+0000","services":{}},"progress_events":{}} 2024-09-02T10:15:59.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:59.650 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:59.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:15:59.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:15:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:15:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1665717425' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:59.776 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:59.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:59.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:15:59.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:15:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1665717425' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:15:59.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:15:59.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:15:59.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:15:59.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:00.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:00.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:00.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:00.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:00.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:00.158 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:16:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:16:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.158 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:88159 5.c 5:36116681:::benchmark_data_smithi187_89405_object88158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e474)", 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:00.152174+0000", 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0035095679999999998, 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003549665, 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 88159 2024-09-02T10:16:00.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152174+0000", 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152174+0000", 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152175+0000", 2024-09-02T10:16:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.601e-06 2024-09-02T10:16:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152235+0000", 2024-09-02T10:16:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9688999999999997e-05 2024-09-02T10:16:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152252+0000", 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7198999999999999e-05 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152260+0000", 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1729999999999995e-06 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152573+0000", 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031311400000000002 2024-09-02T10:16:00.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152719+0000", 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145678 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152859+0000", 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000139538 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:00.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:88160 5.c 5:3c291d55:::benchmark_data_smithi187_89405_object88159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e474)", 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:00.152324+0000", 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0033596339999999998, 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034764689999999998, 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:16:00.165 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 88160 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152324+0000", 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152324+0000", 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152324+0000", 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0400000000000005e-07 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152392+0000", 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7309000000000002e-05 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152402+0000", 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0141e-05 2024-09-02T10:16:00.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.152410+0000", 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8329999999999994e-06 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155364+0000", 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029544810000000001 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155510+0000", 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014619599999999999 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155652+0000", 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014204699999999999 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25933.0:88159 5.c e474/455)", 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:00.155505+0000", 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00017829799999999999, 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034908399999999999, 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155505+0000", 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155505+0000", 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155506+0000", 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7400000000000002e-07 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155510+0000", 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6140000000000002e-06 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155538+0000", 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7331999999999999e-05 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155544+0000", 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8850000000000001e-06 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25933.0:88159 5.c e474/455)", 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:00.155754+0000", 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4294967295.9999294, 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013490900000000001, 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155754+0000", 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155754+0000", 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155754+0000", 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7000000000000005e-07 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155758+0000", 2024-09-02T10:16:00.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8099999999999999e-06 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155781+0000", 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3042999999999999e-05 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:00.155786+0000", 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7910000000000004e-06 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:00.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:16:00.176 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 4 2024-09-02T10:16:00.176 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:16:00.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:00.177 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:00.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:00.177 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 74 15 88186 88171 74.4619 64 0.00989974 0.013424 2024-09-02T10:16:00.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:00.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:00.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:00.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:00.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:00.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:00.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:00.566 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:00.567 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:00.668 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:00 smithi151 ceph-mon[16639]: pgmap v1103: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s; 2880/255156 objects misplaced (1.129%); 3.8 MiB/s, 60 objects/s recovering 2024-09-02T10:16:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:00 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1665717425' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:00 smithi151 ceph-mon[16639]: osdmap e474: 12 total, 12 up, 10 in 2024-09-02T10:16:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/528392290' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:16:00.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:00.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:00 smithi187 ceph-mon[20072]: pgmap v1103: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s; 2880/255156 objects misplaced (1.129%); 3.8 MiB/s, 60 objects/s recovering 2024-09-02T10:16:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:00 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1665717425' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:00 smithi187 ceph-mon[20072]: osdmap e474: 12 total, 12 up, 10 in 2024-09-02T10:16:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/528392290' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:16:00.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:00.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:00.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:00 smithi162 ceph-mon[20279]: pgmap v1103: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 76 MiB/s wr, 1.22k op/s; 2880/255156 objects misplaced (1.129%); 3.8 MiB/s, 60 objects/s recovering 2024-09-02T10:16:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:00 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1665717425' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:00 smithi162 ceph-mon[20279]: osdmap e474: 12 total, 12 up, 10 in 2024-09-02T10:16:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/528392290' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:16:00.911 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:16:00.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:16:00.926 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:00.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:00.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:01.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:01.091 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:01.092 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:01.174 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 75 16 89288 89272 74.3865 68.8125 0.00407565 0.0134372 2024-09-02T10:16:01.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:01.218 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:01.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:01.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:01.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:01.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:01.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:01.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:01.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:01.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:01.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:01.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:01.603 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:01.604 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:01.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:01.730 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:01.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:01.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:01.857 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:01.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:01.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:01.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:01.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:02.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:02.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:02.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:02.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:02.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 76 16 90238 90222 74.1889 59.375 0.0338035 0.0134737 2024-09-02T10:16:02.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:02.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:02.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:02.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:02.366 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:02.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:02.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:02.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:02.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:02.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:02.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:02.628 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:02.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:02.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:02.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:02 smithi151 ceph-mon[16639]: pgmap v1105: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 51 MiB/s wr, 822 op/s; 2880/257136 objects misplaced (1.120%); 2.0 MiB/s, 32 objects/s recovering 2024-09-02T10:16:02.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:02 smithi187 ceph-mon[20072]: pgmap v1105: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 51 MiB/s wr, 822 op/s; 2880/257136 objects misplaced (1.120%); 2.0 MiB/s, 32 objects/s recovering 2024-09-02T10:16:02.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:02.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:02 smithi162 ceph-mon[20279]: pgmap v1105: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 51 MiB/s wr, 822 op/s; 2880/257136 objects misplaced (1.120%); 2.0 MiB/s, 32 objects/s recovering 2024-09-02T10:16:02.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:02.886 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:02.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:02.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:02.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:03.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:03.020 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:03.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:03.146 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:03.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:03.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:03.174 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 77 16 91204 91188 74.0094 60.375 0.0127582 0.0135068 2024-09-02T10:16:03.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:03.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:03.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:03.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:03.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:03.399 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:03.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:03.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:03.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:03.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:03.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:03.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:03.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:03.751 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:03.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:03.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:03.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:03.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:03.904 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:03.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:04.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:04.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:04.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:04.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:04.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:04.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:04.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:04.174 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 78 16 92079 92063 73.7616 54.6875 0.00592885 0.0135521 2024-09-02T10:16:04.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:04.296 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:04.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:04.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:04.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:04.423 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:04.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:04.525 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:04.551 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:04.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:04.552 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:04.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:04.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:04.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:04.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:04.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:04 smithi151 ceph-mon[16639]: pgmap v1106: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 2606/269019 objects misplaced (0.969%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:16:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:04 smithi151 ceph-mon[16639]: osdmap e475: 12 total, 12 up, 10 in 2024-09-02T10:16:04.698 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:04 smithi187 ceph-mon[20072]: pgmap v1106: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 2606/269019 objects misplaced (0.969%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:16:04.698 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:04 smithi187 ceph-mon[20072]: osdmap e475: 12 total, 12 up, 10 in 2024-09-02T10:16:04.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:04.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:04.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:04 smithi162 ceph-mon[20279]: pgmap v1106: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 67 MiB/s wr, 1.07k op/s; 2606/269019 objects misplaced (0.969%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:16:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:04 smithi162 ceph-mon[20279]: osdmap e475: 12 total, 12 up, 10 in 2024-09-02T10:16:04.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:04.927 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:04.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:04.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:05.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:05.052 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:05.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:05.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:05.174 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 79 16 92849 92833 73.4371 48.125 0.00819074 0.0136117 2024-09-02T10:16:05.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:05.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:05.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:16:05.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:05.313 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:05.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:05.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:05.442 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:05.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:05.514 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:05.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:05.567 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:05.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:05.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:05.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:05.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:05.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:05.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:05.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:05.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:05.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:05.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:05.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:06.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:06.071 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:06.083 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:06.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:06.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:16:06.174773+0000 min lat: 0.00106838 max lat: 2.22758 avg lat: 0.0136336 2024-09-02T10:16:06.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:16:06.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 80 16 93878 93862 73.3229 64.3125 0.0145367 0.0136336 2024-09-02T10:16:06.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:06.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:06.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:06.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:16:06.307 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:16:06.321 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:06.336 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:16:06.336 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:16:06.347 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:16:06.350 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:16:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:06.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:06.351 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:06.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:16:06.367 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:16:06.401 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:16:06.402 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:16:06.431 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:16:06.431 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:16:06.452 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:06.463 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:16:06.463 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:16:06.475 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:06.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:06.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:06.488 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:16:06.488 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:16:06.512 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:16:06.512 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:16:06.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:16:06.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:06.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:06 smithi151 ceph-mon[16639]: pgmap v1108: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 62 MiB/s wr, 995 op/s; 2606/271905 objects misplaced (0.958%); 4.1 MiB/s, 66 objects/s recovering 2024-09-02T10:16:06.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:06 smithi151 ceph-mon[16639]: osdmap e476: 12 total, 12 up, 10 in 2024-09-02T10:16:06.513 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2089880274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:06.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:16:06.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:16:06.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:16:06.568 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:16:06.568 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:16:06.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:06.595 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:16:06.595 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:16:06.603 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:06.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:06.605 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:06.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:16:06.622 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:16:06.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:06.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:06.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:16:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:06 smithi187 ceph-mon[20072]: pgmap v1108: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 62 MiB/s wr, 995 op/s; 2606/271905 objects misplaced (0.958%); 4.1 MiB/s, 66 objects/s recovering 2024-09-02T10:16:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:06 smithi187 ceph-mon[20072]: osdmap e476: 12 total, 12 up, 10 in 2024-09-02T10:16:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2089880274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:06.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:16:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:06 smithi162 ceph-mon[20279]: pgmap v1108: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 62 MiB/s wr, 995 op/s; 2606/271905 objects misplaced (0.958%); 4.1 MiB/s, 66 objects/s recovering 2024-09-02T10:16:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:06 smithi162 ceph-mon[20279]: osdmap e476: 12 total, 12 up, 10 in 2024-09-02T10:16:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2089880274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:06.863 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:06.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:06.873 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:16:06.873 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:16:06.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:06.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:06.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:07.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:07.118 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:07.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:07.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:07.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 81 15 94980 94965 73.2687 68.9375 0.0177878 0.0136435 2024-09-02T10:16:07.221 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:07.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:07.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:07.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:07.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:07.373 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:07.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:07.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:07.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:07.500 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:07.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:07.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:07.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:07.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:07.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:07.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:07.686 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:07 smithi151 ceph-mon[16639]: from='client.26189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:07.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:07 smithi187 ceph-mon[20072]: from='client.26189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:07.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:07.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:07.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:07 smithi162 ceph-mon[20279]: from='client.26189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:07.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:07.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:07.920 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:08.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:08.047 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:08.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:08.048 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:08.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:08.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:08.175 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:08.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 82 16 95940 95924 73.1061 59.9375 0.00398763 0.0136694 2024-09-02T10:16:08.206 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1109,"stamp":"2024-09-02T10:16:05.420341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5940314656,"num_objects":90635,"num_object_clones":0,"num_object_copies":271905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2606,"num_objects_unfound":0,"num_objects_dirty":90635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90633,"num_write_kb":5800512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3105,"num_bytes_recovered":204022848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93485,"ondisk_log_size":93485,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":21087696,"kb_used_data":17542500,"kb_used_omap":15,"kb_used_meta":3545008,"kb_avail":916363824,"statfs":{"total":959950356480,"available":938356555776,"internally_reserved":0,"allocated":17963520000,"data_stored":17957271509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3630088625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,8,3,7,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":365887488,"num_objects":5583,"num_object_clones":0,"num_object_copies":16749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-274,"num_objects_unfound":0,"num_objects_dirty":5583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5583,"num_write_kb":357312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":274,"num_bytes_recovered":17956864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.125623"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":275,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420967+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:15:59.420967+0000","last_peered":"2024-09-02T10:15:59.420967+0000","last_clean":"2024-09-02T10:15:59.420967+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:15:59.420967+0000","last_undegraded":"2024-09-02T10:15:59.420967+0000","last_fullsized":"2024-09-02T10:15:59.420967+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":83,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420887+0000","last_change":"2024-09-02T10:15:41.899476+0000","last_active":"2024-09-02T10:15:59.420887+0000","last_peered":"2024-09-02T10:15:59.420887+0000","last_clean":"2024-09-02T10:15:59.420887+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:15:59.420887+0000","last_undegraded":"2024-09-02T10:15:59.420887+0000","last_fullsized":"2024-09-02T10:15:59.420887+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:41.899397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:41.899397+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:51.002146+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":534,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-02T10:16:04.002956+0000","last_change":"2024-09-02T10:15:12.091472+0000","last_active":"2024-09-02T10:16:04.002956+0000","last_peered":"2024-09-02T10:16:04.002956+0000","last_clean":"2024-09-02T10:16:04.002956+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:16:04.002956+0000","last_undegraded":"2024-09-02T10:16:04.002956+0000","last_fullsized":"2024-09-02T10:16:04.002956+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.091410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.091410+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:27.631420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":534,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-02T10:16:04.000793+0000","last_change":"2024-09-02T10:15:19.065187+0000","last_active":"2024-09-02T10:16:04.000793+0000","last_peered":"2024-09-02T10:16:04.000793+0000","last_clean":"2024-09-02T10:16:04.000793+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:16:04.000793+0000","last_undegraded":"2024-09-02T10:16:04.000793+0000","last_fullsized":"2024-09-02T10:16:04.000793+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:19.065134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:19.065134+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:36.285460+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":454,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.424822+0000","last_change":"2024-09-02T10:15:35.202261+0000","last_active":"2024-09-02T10:15:59.424822+0000","last_peered":"2024-09-02T10:15:59.424822+0000","last_clean":"2024-09-02T10:15:59.424822+0000","last_became_active":"2024-09-02T10:15:35.201887+0000","last_became_peered":"2024-09-02T10:15:35.201887+0000","last_unstale":"2024-09-02T10:15:59.424822+0000","last_undegraded":"2024-09-02T10:15:59.424822+0000","last_fullsized":"2024-09-02T10:15:59.424822+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:04.994667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:04.994667+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:30.739058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":382,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420453+0000","last_change":"2024-09-02T10:14:11.689851+0000","last_active":"2024-09-02T10:15:59.420453+0000","last_peered":"2024-09-02T10:15:59.420453+0000","last_clean":"2024-09-02T10:15:59.420453+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:15:59.420453+0000","last_undegraded":"2024-09-02T10:15:59.420453+0000","last_fullsized":"2024-09-02T10:15:59.420453+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.689767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":897,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.419965+0000","last_change":"2024-09-02T10:15:07.059223+0000","last_active":"2024-09-02T10:15:59.419965+0000","last_peered":"2024-09-02T10:15:59.419965+0000","last_clean":"2024-09-02T10:15:59.419965+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:15:59.419965+0000","last_undegraded":"2024-09-02T10:15:59.419965+0000","last_fullsized":"2024-09-02T10:15:59.419965+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:45.351862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:45.351862+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":541,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-02T10:16:04.001004+0000","last_change":"2024-09-02T10:15:05.094874+0000","last_active":"2024-09-02T10:16:04.001004+0000","last_peered":"2024-09-02T10:16:04.001004+0000","last_clean":"2024-09-02T10:16:04.001004+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:16:04.001004+0000","last_undegraded":"2024-09-02T10:16:04.001004+0000","last_fullsized":"2024-09-02T10:16:04.001004+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:05.094792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:05.094792+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:09.107562+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":546,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420471+0000","last_change":"2024-09-02T10:15:34.791760+0000","last_active":"2024-09-02T10:15:59.420471+0000","last_peered":"2024-09-02T10:15:59.420471+0000","last_clean":"2024-09-02T10:15:59.420471+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:15:59.420471+0000","last_undegraded":"2024-09-02T10:15:59.420471+0000","last_fullsized":"2024-09-02T10:15:59.420471+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:34.791708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:34.791708+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.857252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":515,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.424991+0000","last_change":"2024-09-02T10:14:49.165646+0000","last_active":"2024-09-02T10:15:59.424991+0000","last_peered":"2024-09-02T10:15:59.424991+0000","last_clean":"2024-09-02T10:15:59.424991+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:15:59.424991+0000","last_undegraded":"2024-09-02T10:15:59.424991+0000","last_fullsized":"2024-09-02T10:15:59.424991+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:49.165586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:49.165586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:01.041497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":367,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.424966+0000","last_change":"2024-09-02T10:14:45.203339+0000","last_active":"2024-09-02T10:15:59.424966+0000","last_peered":"2024-09-02T10:15:59.424966+0000","last_clean":"2024-09-02T10:15:59.424966+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:15:59.424966+0000","last_undegraded":"2024-09-02T10:15:59.424966+0000","last_fullsized":"2024-09-02T10:15:59.424966+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:45.203278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:45.203278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":872,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420614+0000","last_change":"2024-09-02T10:15:06.447417+0000","last_active":"2024-09-02T10:15:59.420614+0000","last_peered":"2024-09-02T10:15:59.420614+0000","last_clean":"2024-09-02T10:15:59.420614+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:15:59.420614+0000","last_undegraded":"2024-09-02T10:15:59.420614+0000","last_fullsized":"2024-09-02T10:15:59.420614+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:06.447372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:06.447372+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:13.751465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":807,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420007+0000","last_change":"2024-09-02T10:15:07.059344+0000","last_active":"2024-09-02T10:15:59.420007+0000","last_peered":"2024-09-02T10:15:59.420007+0000","last_clean":"2024-09-02T10:15:59.420007+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:15:59.420007+0000","last_undegraded":"2024-09-02T10:15:59.420007+0000","last_fullsized":"2024-09-02T10:15:59.420007+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.252526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.252526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:03.261308+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":794,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.425299+0000","last_change":"2024-09-02T10:15:29.170702+0000","last_active":"2024-09-02T10:15:59.425299+0000","last_peered":"2024-09-02T10:15:59.425299+0000","last_clean":"2024-09-02T10:15:59.425299+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:15:59.425299+0000","last_undegraded":"2024-09-02T10:15:59.425299+0000","last_fullsized":"2024-09-02T10:15:59.425299+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.665438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.665438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:04.476082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":785,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.419975+0000","last_change":"2024-09-02T10:14:34.217697+0000","last_active":"2024-09-02T10:15:59.419975+0000","last_peered":"2024-09-02T10:15:59.419975+0000","last_clean":"2024-09-02T10:15:59.419975+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:15:59.419975+0000","last_undegraded":"2024-09-02T10:15:59.419975+0000","last_fullsized":"2024-09-02T10:15:59.419975+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.217624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.217624+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:02.148154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"474'5615","reported_seq":9266,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:03.594241+0000","last_change":"2024-09-02T10:15:24.164550+0000","last_active":"2024-09-02T10:16:03.594241+0000","last_peered":"2024-09-02T10:16:03.594241+0000","last_clean":"2024-09-02T10:16:03.594241+0000","last_became_active":"2024-09-02T10:15:24.141368+0000","last_became_peered":"2024-09-02T10:15:24.141368+0000","last_unstale":"2024-09-02T10:16:03.594241+0000","last_undegraded":"2024-09-02T10:16:03.594241+0000","last_fullsized":"2024-09-02T10:16:03.594241+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5615,"log_dups_size":0,"ondisk_log_size":5615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:05.787299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367984640,"num_objects":5615,"num_object_clones":0,"num_object_copies":16845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5615,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5615,"num_write_kb":359360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":115933184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":872,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420068+0000","last_change":"2024-09-02T10:14:41.306927+0000","last_active":"2024-09-02T10:15:59.420068+0000","last_peered":"2024-09-02T10:15:59.420068+0000","last_clean":"2024-09-02T10:15:59.420068+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:15:59.420068+0000","last_undegraded":"2024-09-02T10:15:59.420068+0000","last_fullsized":"2024-09-02T10:15:59.420068+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:41.306868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:41.306868+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:04.992960+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"474'5653","reported_seq":5748,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:02.085155+0000","last_change":"2024-09-02T10:14:39.040351+0000","last_active":"2024-09-02T10:16:02.085155+0000","last_peered":"2024-09-02T10:16:02.085155+0000","last_clean":"2024-09-02T10:16:02.085155+0000","last_became_active":"2024-09-02T10:14:39.040025+0000","last_became_peered":"2024-09-02T10:14:39.040025+0000","last_unstale":"2024-09-02T10:16:02.085155+0000","last_undegraded":"2024-09-02T10:16:02.085155+0000","last_fullsized":"2024-09-02T10:16:02.085155+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5653,"log_dups_size":0,"ondisk_log_size":5653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370475008,"num_objects":5653,"num_object_clones":0,"num_object_copies":16959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5653,"num_write_kb":361792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":987,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.425026+0000","last_change":"2024-09-02T10:14:36.143055+0000","last_active":"2024-09-02T10:15:59.425026+0000","last_peered":"2024-09-02T10:15:59.425026+0000","last_clean":"2024-09-02T10:15:59.425026+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:15:59.425026+0000","last_undegraded":"2024-09-02T10:15:59.425026+0000","last_fullsized":"2024-09-02T10:15:59.425026+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.142993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.142993+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"474'5590","reported_seq":5687,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:02.909184+0000","last_change":"2024-09-02T10:14:39.038506+0000","last_active":"2024-09-02T10:16:02.909184+0000","last_peered":"2024-09-02T10:16:02.909184+0000","last_clean":"2024-09-02T10:16:02.909184+0000","last_became_active":"2024-09-02T10:14:39.038164+0000","last_became_peered":"2024-09-02T10:14:39.038164+0000","last_unstale":"2024-09-02T10:16:02.909184+0000","last_undegraded":"2024-09-02T10:16:02.909184+0000","last_fullsized":"2024-09-02T10:16:02.909184+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5590,"log_dups_size":0,"ondisk_log_size":5590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366346240,"num_objects":5590,"num_object_clones":0,"num_object_copies":16770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5590,"num_write_kb":357760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":630,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.422947+0000","last_change":"2024-09-02T10:14:31.581643+0000","last_active":"2024-09-02T10:15:59.422947+0000","last_peered":"2024-09-02T10:15:59.422947+0000","last_clean":"2024-09-02T10:15:59.422947+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:15:59.422947+0000","last_undegraded":"2024-09-02T10:15:59.422947+0000","last_fullsized":"2024-09-02T10:15:59.422947+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.581597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.581597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"474'5707","reported_seq":5802,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:02.005100+0000","last_change":"2024-09-02T10:14:39.037573+0000","last_active":"2024-09-02T10:16:02.005100+0000","last_peered":"2024-09-02T10:16:02.005100+0000","last_clean":"2024-09-02T10:16:02.005100+0000","last_became_active":"2024-09-02T10:14:39.037121+0000","last_became_peered":"2024-09-02T10:14:39.037121+0000","last_unstale":"2024-09-02T10:16:02.005100+0000","last_undegraded":"2024-09-02T10:16:02.005100+0000","last_fullsized":"2024-09-02T10:16:02.005100+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5707,"log_dups_size":0,"ondisk_log_size":5707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374013952,"num_objects":5707,"num_object_clones":0,"num_object_copies":17121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5707,"num_write_kb":365248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":34,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420622+0000","last_change":"2024-09-02T10:15:35.208246+0000","last_active":"2024-09-02T10:15:59.420622+0000","last_peered":"2024-09-02T10:15:59.420622+0000","last_clean":"2024-09-02T10:15:59.420622+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:15:59.420622+0000","last_undegraded":"2024-09-02T10:15:59.420622+0000","last_fullsized":"2024-09-02T10:15:59.420622+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.668602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.668602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"474'5590","reported_seq":5687,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:01.604278+0000","last_change":"2024-09-02T10:14:39.642395+0000","last_active":"2024-09-02T10:16:01.604278+0000","last_peered":"2024-09-02T10:16:01.604278+0000","last_clean":"2024-09-02T10:16:01.604278+0000","last_became_active":"2024-09-02T10:14:39.641995+0000","last_became_peered":"2024-09-02T10:14:39.641995+0000","last_unstale":"2024-09-02T10:16:01.604278+0000","last_undegraded":"2024-09-02T10:16:01.604278+0000","last_fullsized":"2024-09-02T10:16:01.604278+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5590,"log_dups_size":0,"ondisk_log_size":5590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366346240,"num_objects":5590,"num_object_clones":0,"num_object_copies":16770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5590,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5590,"num_write_kb":357760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":942,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420855+0000","last_change":"2024-09-02T10:15:29.195341+0000","last_active":"2024-09-02T10:15:59.420855+0000","last_peered":"2024-09-02T10:15:59.420855+0000","last_clean":"2024-09-02T10:15:59.420855+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:15:59.420855+0000","last_undegraded":"2024-09-02T10:15:59.420855+0000","last_fullsized":"2024-09-02T10:15:59.420855+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.277119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.277119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:28.784954+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"474'5693","reported_seq":5788,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:03.898804+0000","last_change":"2024-09-02T10:14:39.034903+0000","last_active":"2024-09-02T10:16:03.898804+0000","last_peered":"2024-09-02T10:16:03.898804+0000","last_clean":"2024-09-02T10:16:03.898804+0000","last_became_active":"2024-09-02T10:14:39.034636+0000","last_became_peered":"2024-09-02T10:14:39.034636+0000","last_unstale":"2024-09-02T10:16:03.898804+0000","last_undegraded":"2024-09-02T10:16:03.898804+0000","last_fullsized":"2024-09-02T10:16:03.898804+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5693,"log_dups_size":0,"ondisk_log_size":5693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:07.235133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373096448,"num_objects":5693,"num_object_clones":0,"num_object_copies":17079,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5693,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5693,"num_write_kb":364352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":410,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420695+0000","last_change":"2024-09-02T10:15:35.205684+0000","last_active":"2024-09-02T10:15:59.420695+0000","last_peered":"2024-09-02T10:15:59.420695+0000","last_clean":"2024-09-02T10:15:59.420695+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:15:59.420695+0000","last_undegraded":"2024-09-02T10:15:59.420695+0000","last_fullsized":"2024-09-02T10:15:59.420695+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.701773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.701773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"474'5580","reported_seq":5732,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:02.200481+0000","last_change":"2024-09-02T10:15:49.158612+0000","last_active":"2024-09-02T10:16:02.200481+0000","last_peered":"2024-09-02T10:16:02.200481+0000","last_clean":"2024-09-02T10:16:02.200481+0000","last_became_active":"2024-09-02T10:14:39.037822+0000","last_became_peered":"2024-09-02T10:14:39.037822+0000","last_unstale":"2024-09-02T10:16:02.200481+0000","last_undegraded":"2024-09-02T10:16:02.200481+0000","last_fullsized":"2024-09-02T10:16:02.200481+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4732","last_scrub_stamp":"2024-09-02T10:15:49.158487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.158487+0000","objects_scrubbed":4682,"log_size":5580,"log_dups_size":0,"ondisk_log_size":5580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:57.249982+0000","scrub_duration":968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365690880,"num_objects":5580,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5580,"num_write_kb":357120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":382,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.421007+0000","last_change":"2024-09-02T10:14:09.768298+0000","last_active":"2024-09-02T10:15:59.421007+0000","last_peered":"2024-09-02T10:15:59.421007+0000","last_clean":"2024-09-02T10:15:59.421007+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:15:59.421007+0000","last_undegraded":"2024-09-02T10:15:59.421007+0000","last_fullsized":"2024-09-02T10:15:59.421007+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:09.768061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:09.768061+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"474'5845","reported_seq":5942,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:02.703592+0000","last_change":"2024-09-02T10:14:39.035303+0000","last_active":"2024-09-02T10:16:02.703592+0000","last_peered":"2024-09-02T10:16:02.703592+0000","last_clean":"2024-09-02T10:16:02.703592+0000","last_became_active":"2024-09-02T10:14:39.035100+0000","last_became_peered":"2024-09-02T10:14:39.035100+0000","last_unstale":"2024-09-02T10:16:02.703592+0000","last_undegraded":"2024-09-02T10:16:02.703592+0000","last_fullsized":"2024-09-02T10:16:02.703592+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5845,"log_dups_size":0,"ondisk_log_size":5845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383057920,"num_objects":5845,"num_object_clones":0,"num_object_copies":17535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5845,"num_write_kb":374080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":133,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.425652+0000","last_change":"2024-09-02T10:15:36.299961+0000","last_active":"2024-09-02T10:15:59.425652+0000","last_peered":"2024-09-02T10:15:59.425652+0000","last_clean":"2024-09-02T10:15:59.425652+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:15:59.425652+0000","last_undegraded":"2024-09-02T10:15:59.425652+0000","last_fullsized":"2024-09-02T10:15:59.425652+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:36.299914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:36.299914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:01.870130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"474'5674","reported_seq":5771,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:02.696324+0000","last_change":"2024-09-02T10:14:39.641746+0000","last_active":"2024-09-02T10:16:02.696324+0000","last_peered":"2024-09-02T10:16:02.696324+0000","last_clean":"2024-09-02T10:16:02.696324+0000","last_became_active":"2024-09-02T10:14:39.641478+0000","last_became_peered":"2024-09-02T10:14:39.641478+0000","last_unstale":"2024-09-02T10:16:02.696324+0000","last_undegraded":"2024-09-02T10:16:02.696324+0000","last_fullsized":"2024-09-02T10:16:02.696324+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5674,"log_dups_size":0,"ondisk_log_size":5674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371851264,"num_objects":5674,"num_object_clones":0,"num_object_copies":17022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5674,"num_write_kb":363136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":383,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420758+0000","last_change":"2024-09-02T10:14:36.623992+0000","last_active":"2024-09-02T10:15:59.420758+0000","last_peered":"2024-09-02T10:15:59.420758+0000","last_clean":"2024-09-02T10:15:59.420758+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:15:59.420758+0000","last_undegraded":"2024-09-02T10:15:59.420758+0000","last_fullsized":"2024-09-02T10:15:59.420758+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.623930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.623930+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"474'5571","reported_seq":5724,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:02.015857+0000","last_change":"2024-09-02T10:15:49.363717+0000","last_active":"2024-09-02T10:16:02.015857+0000","last_peered":"2024-09-02T10:16:02.015857+0000","last_clean":"2024-09-02T10:16:02.015857+0000","last_became_active":"2024-09-02T10:14:39.037882+0000","last_became_peered":"2024-09-02T10:14:39.037882+0000","last_unstale":"2024-09-02T10:16:02.015857+0000","last_undegraded":"2024-09-02T10:16:02.015857+0000","last_fullsized":"2024-09-02T10:16:02.015857+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4720","last_scrub_stamp":"2024-09-02T10:15:49.363588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.363588+0000","objects_scrubbed":4682,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:06.930040+0000","scrub_duration":1106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365101056,"num_objects":5571,"num_object_clones":0,"num_object_copies":16713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5571,"num_write_kb":356544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":629,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.424880+0000","last_change":"2024-09-02T10:14:40.192356+0000","last_active":"2024-09-02T10:15:59.424880+0000","last_peered":"2024-09-02T10:15:59.424880+0000","last_clean":"2024-09-02T10:15:59.424880+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:15:59.424880+0000","last_undegraded":"2024-09-02T10:15:59.424880+0000","last_fullsized":"2024-09-02T10:15:59.424880+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.192297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.192297+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"474'5652","reported_seq":5749,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:02.008040+0000","last_change":"2024-09-02T10:14:39.642277+0000","last_active":"2024-09-02T10:16:02.008040+0000","last_peered":"2024-09-02T10:16:02.008040+0000","last_clean":"2024-09-02T10:16:02.008040+0000","last_became_active":"2024-09-02T10:14:39.641932+0000","last_became_peered":"2024-09-02T10:14:39.641932+0000","last_unstale":"2024-09-02T10:16:02.008040+0000","last_undegraded":"2024-09-02T10:16:02.008040+0000","last_fullsized":"2024-09-02T10:16:02.008040+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5652,"log_dups_size":0,"ondisk_log_size":5652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:05.284147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370409472,"num_objects":5652,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":361728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":888,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420778+0000","last_change":"2024-09-02T10:15:29.169027+0000","last_active":"2024-09-02T10:15:59.420778+0000","last_peered":"2024-09-02T10:15:59.420778+0000","last_clean":"2024-09-02T10:15:59.420778+0000","last_became_active":"2024-09-02T10:15:29.168692+0000","last_became_peered":"2024-09-02T10:15:29.168692+0000","last_unstale":"2024-09-02T10:15:59.420778+0000","last_undegraded":"2024-09-02T10:15:59.420778+0000","last_fullsized":"2024-09-02T10:15:59.420778+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.804014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.804014+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:00.801964+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"474'5703","reported_seq":5800,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:01.598618+0000","last_change":"2024-09-02T10:14:39.038104+0000","last_active":"2024-09-02T10:16:01.598618+0000","last_peered":"2024-09-02T10:16:01.598618+0000","last_clean":"2024-09-02T10:16:01.598618+0000","last_became_active":"2024-09-02T10:14:39.037755+0000","last_became_peered":"2024-09-02T10:14:39.037755+0000","last_unstale":"2024-09-02T10:16:01.598618+0000","last_undegraded":"2024-09-02T10:16:01.598618+0000","last_fullsized":"2024-09-02T10:16:01.598618+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5703,"log_dups_size":0,"ondisk_log_size":5703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":373751808,"num_objects":5703,"num_object_clones":0,"num_object_copies":17109,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5703,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5703,"num_write_kb":364992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":647,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.424746+0000","last_change":"2024-09-02T10:15:35.202850+0000","last_active":"2024-09-02T10:15:59.424746+0000","last_peered":"2024-09-02T10:15:59.424746+0000","last_clean":"2024-09-02T10:15:59.424746+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:15:59.424746+0000","last_undegraded":"2024-09-02T10:15:59.424746+0000","last_fullsized":"2024-09-02T10:15:59.424746+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:48.187420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:48.187420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"474'5663","reported_seq":5760,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:02.929485+0000","last_change":"2024-09-02T10:14:39.039399+0000","last_active":"2024-09-02T10:16:02.929485+0000","last_peered":"2024-09-02T10:16:02.929485+0000","last_clean":"2024-09-02T10:16:02.929485+0000","last_became_active":"2024-09-02T10:14:39.039060+0000","last_became_peered":"2024-09-02T10:14:39.039060+0000","last_unstale":"2024-09-02T10:16:02.929485+0000","last_undegraded":"2024-09-02T10:16:02.929485+0000","last_fullsized":"2024-09-02T10:16:02.929485+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:04.071159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371130368,"num_objects":5663,"num_object_clones":0,"num_object_copies":16989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5663,"num_write_kb":362432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":887,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.425748+0000","last_change":"2024-09-02T10:14:34.284618+0000","last_active":"2024-09-02T10:15:59.425748+0000","last_peered":"2024-09-02T10:15:59.425748+0000","last_clean":"2024-09-02T10:15:59.425748+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:15:59.425748+0000","last_undegraded":"2024-09-02T10:15:59.425748+0000","last_fullsized":"2024-09-02T10:15:59.425748+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.284563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.284563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:15:59.753379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"475'5795","reported_seq":5894,"reported_epoch":475,"state":"active+clean","last_fresh":"2024-09-02T10:16:04.321958+0000","last_change":"2024-09-02T10:14:39.036854+0000","last_active":"2024-09-02T10:16:04.321958+0000","last_peered":"2024-09-02T10:16:04.321958+0000","last_clean":"2024-09-02T10:16:04.321958+0000","last_became_active":"2024-09-02T10:14:39.036453+0000","last_became_peered":"2024-09-02T10:14:39.036453+0000","last_unstale":"2024-09-02T10:16:04.321958+0000","last_undegraded":"2024-09-02T10:16:04.321958+0000","last_fullsized":"2024-09-02T10:16:04.321958+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5795,"log_dups_size":0,"ondisk_log_size":5795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:06.974148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379781120,"num_objects":5795,"num_object_clones":0,"num_object_copies":17385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5795,"num_write_kb":370880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":422,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.426677+0000","last_change":"2024-09-02T10:14:47.184482+0000","last_active":"2024-09-02T10:15:59.426677+0000","last_peered":"2024-09-02T10:15:59.426677+0000","last_clean":"2024-09-02T10:15:59.426677+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:15:59.426677+0000","last_undegraded":"2024-09-02T10:15:59.426677+0000","last_fullsized":"2024-09-02T10:15:59.426677+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:47.184420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:47.184420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"474'5614","reported_seq":5731,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:16:01.115996+0000","last_change":"2024-09-02T10:15:49.395716+0000","last_active":"2024-09-02T10:16:01.115996+0000","last_peered":"2024-09-02T10:16:01.115996+0000","last_clean":"2024-09-02T10:16:01.115996+0000","last_became_active":"2024-09-02T10:14:39.037567+0000","last_became_peered":"2024-09-02T10:14:39.037567+0000","last_unstale":"2024-09-02T10:16:01.115996+0000","last_undegraded":"2024-09-02T10:16:01.115996+0000","last_fullsized":"2024-09-02T10:16:01.115996+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":1699,"log_size":5614,"log_dups_size":0,"ondisk_log_size":5614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:08.193439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367919104,"num_objects":5614,"num_object_clones":0,"num_object_copies":16842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5614,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5614,"num_write_kb":359296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":667,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.425035+0000","last_change":"2024-09-02T10:15:31.899734+0000","last_active":"2024-09-02T10:15:59.425035+0000","last_peered":"2024-09-02T10:15:59.425035+0000","last_clean":"2024-09-02T10:15:59.425035+0000","last_became_active":"2024-09-02T10:15:29.167553+0000","last_became_peered":"2024-09-02T10:15:29.167553+0000","last_unstale":"2024-09-02T10:15:59.425035+0000","last_undegraded":"2024-09-02T10:15:59.425035+0000","last_fullsized":"2024-09-02T10:15:59.425035+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:31.899662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:31.899662+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.920777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"474'5688","reported_seq":8454,"reported_epoch":474,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:16:02.730128+0000","last_change":"2024-09-02T10:15:36.347809+0000","last_active":"2024-09-02T10:16:02.730128+0000","last_peered":"2024-09-02T10:16:02.730128+0000","last_clean":"2024-09-02T10:15:34.173584+0000","last_became_active":"2024-09-02T10:15:36.297925+0000","last_became_peered":"2024-09-02T10:15:36.297925+0000","last_unstale":"2024-09-02T10:16:02.730128+0000","last_undegraded":"2024-09-02T10:16:02.730128+0000","last_fullsized":"2024-09-02T10:15:36.200774+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":5688,"log_dups_size":0,"ondisk_log_size":5688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372768768,"num_objects":5688,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2606,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5688,"num_write_kb":364032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1333,"num_bytes_recovered":87359488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[0,10],"avail_no_missing":["0","9","10"],"object_location_counts":[{"shards":"0,9,10","objects":2606},{"shards":"0,10","objects":3082}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":74,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-09-02T10:15:59.420039+0000","last_change":"2024-09-02T10:15:48.739408+0000","last_active":"2024-09-02T10:15:59.420039+0000","last_peered":"2024-09-02T10:15:59.420039+0000","last_clean":"2024-09-02T10:15:59.420039+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:15:59.420039+0000","last_undegraded":"2024-09-02T10:15:59.420039+0000","last_fullsized":"2024-09-02T10:15:59.420039+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:48.739340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:48.739340+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:02.899055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5939724288,"num_objects":90633,"num_object_clones":0,"num_object_copies":271899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2606,"num_objects_unfound":0,"num_objects_dirty":90633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90633,"num_write_kb":5800512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3102,"num_bytes_recovered":203292672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17917804544,"data_stored":17917804544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90633,"ondisk_log_size":90633,"up":48,"acting":47,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383871,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2001872,"kb_used_data":1623616,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":91743280,"statfs":{"total":95995035648,"available":93945118720,"internally_reserved":0,"allocated":1662582784,"data_stored":1661959451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":387316164},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612804,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2732240,"kb_used_data":2494208,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":91012912,"statfs":{"total":95995035648,"available":93197221888,"internally_reserved":0,"allocated":2554068992,"data_stored":2553445659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":243726784},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841737,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2117008,"kb_used_data":1684608,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91628144,"statfs":{"total":95995035648,"available":93827219456,"internally_reserved":0,"allocated":1725038592,"data_stored":1724415259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070671,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1316448,"kb_used_data":1104780,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92428704,"statfs":{"total":95995035648,"available":94646992896,"internally_reserved":0,"allocated":1131294720,"data_stored":1130666882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299603,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1870616,"kb_used_data":1464584,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91874536,"statfs":{"total":95995035648,"available":94079524864,"internally_reserved":0,"allocated":1499734016,"data_stored":1499107131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":415758794},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528535,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2054356,"kb_used_data":1792704,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":91690796,"statfs":{"total":95995035648,"available":93891375104,"internally_reserved":0,"allocated":1835728896,"data_stored":1835105563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":267909575},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724764,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2911644,"kb_used_data":2523656,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":90833508,"statfs":{"total":95995035648,"available":93013512192,"internally_reserved":0,"allocated":2584223744,"data_stored":2583596859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149927,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758288,"kb_used_data":366144,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92986864,"statfs":{"total":95995035648,"available":95218548736,"internally_reserved":0,"allocated":374931456,"data_stored":374308123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":401537480},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411562,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2013272,"kb_used_data":1612808,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":91731880,"statfs":{"total":95995035648,"available":93933445120,"internally_reserved":0,"allocated":1651515392,"data_stored":1650888507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738614,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3311952,"kb_used_data":2875392,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":90433200,"statfs":{"total":95995035648,"available":92603596800,"internally_reserved":0,"allocated":2944401408,"data_stored":2943778075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2965569536,"data_stored":2965569536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1492910080,"data_stored":1492910080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":373161984,"data_stored":373161984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2603286528,"data_stored":2603286528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1851260928,"data_stored":1851260928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1499267072,"data_stored":1499267072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1134821376,"data_stored":1134821376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1738211328,"data_stored":1738211328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2576089088,"data_stored":2576089088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1683226624,"data_stored":1683226624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:16:08.276 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:08.300 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:08.301 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:08.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:08.427 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:08.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:08.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:08 smithi151 ceph-mon[16639]: pgmap v1110: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 60 MiB/s wr, 956 op/s; 2606/278748 objects misplaced (0.935%); 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:16:08.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:16:08.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:08.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:08.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:08.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:08.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:08.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:08.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:08.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:08 smithi187 ceph-mon[20072]: pgmap v1110: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 60 MiB/s wr, 956 op/s; 2606/278748 objects misplaced (0.935%); 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:16:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:16:08.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:08.819 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:08.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:08 smithi162 ceph-mon[20279]: pgmap v1110: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 60 MiB/s wr, 956 op/s; 2606/278748 objects misplaced (0.935%); 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:16:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:16:08.920 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:08.946 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:08.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:09.008 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:09.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:16:09.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:09.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:09.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:09.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:09.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 83 16 96880 96864 72.9331 58.75 0.00533286 0.0137051 2024-09-02T10:16:09.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:09.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:09.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:09.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:09.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:09.375 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:09.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:09.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:09.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:09.503 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:09.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:09.633 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:09.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:09.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:09.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:09.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:16:09.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:09.760 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:09.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:09.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:09.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:09.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:09.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:09.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:09.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:09.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:09.923 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:09 smithi151 ceph-mon[16639]: osdmap e477: 12 total, 12 up, 10 in 2024-09-02T10:16:09.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:16:09.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:16:09.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:09 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:09.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:09 smithi187 ceph-mon[20072]: osdmap e477: 12 total, 12 up, 10 in 2024-09-02T10:16:10.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:16:10.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:16:10.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:09 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:10.021 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:10.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:10.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:09 smithi162 ceph-mon[20279]: osdmap e477: 12 total, 12 up, 10 in 2024-09-02T10:16:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:16:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:16:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:09 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:10.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:10.150 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:10.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:10.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 84 16 97744 97728 72.7076 54 0.0249029 0.0137487 2024-09-02T10:16:10.192 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:16:10.192 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:16:10.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:10.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:10.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:10.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:10.404 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:10.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:10.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:10.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:10.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:10.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:10.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:10.663 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:10.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:10.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:10.686 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:16:10.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:10.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:10.798 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:10.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:10.929 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:10.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:10.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:10 smithi151 ceph-mon[16639]: pgmap v1112: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 56 MiB/s wr, 895 op/s; 2345/285144 objects misplaced (0.822%); 2.7 MiB/s, 43 objects/s recovering 2024-09-02T10:16:10.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3892430865' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:16:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:10 smithi187 ceph-mon[20072]: pgmap v1112: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 56 MiB/s wr, 895 op/s; 2345/285144 objects misplaced (0.822%); 2.7 MiB/s, 43 objects/s recovering 2024-09-02T10:16:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3892430865' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:16:11.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:11.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:16:11.057 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:11.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:10 smithi162 ceph-mon[20279]: pgmap v1112: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 56 MiB/s wr, 895 op/s; 2345/285144 objects misplaced (0.822%); 2.7 MiB/s, 43 objects/s recovering 2024-09-02T10:16:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3892430865' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:16:11.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:11.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 85 16 98880 98864 72.6875 71 0.00832531 0.0137523 2024-09-02T10:16:11.189 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:11.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:11.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:11.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:16:11.279 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:11.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:11.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:11.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:11.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:11.520 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:11.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:11.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:11.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:11.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:11.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:11.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:11.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:11.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:11.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:11.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:11.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:11.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:11.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:11.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:11 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3892430865' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:16:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:11 smithi151 ceph-mon[16639]: osdmap e478: 12 total, 12 up, 10 in 2024-09-02T10:16:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:11 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:16:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:11 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:16:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:11 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:16:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:11 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:16:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:11 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:16:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:11 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:16:12.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:11 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3892430865' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:16:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:11 smithi187 ceph-mon[20072]: osdmap e478: 12 total, 12 up, 10 in 2024-09-02T10:16:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:11 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:16:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:11 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:16:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:11 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:16:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:11 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:16:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:11 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:16:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:11 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:16:12.034 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:12.035 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:11 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3892430865' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:16:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:11 smithi162 ceph-mon[20279]: osdmap e478: 12 total, 12 up, 10 in 2024-09-02T10:16:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:11 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:16:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:11 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:16:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:11 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:16:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:11 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:16:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:11 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:16:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:11 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:16:12.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:12.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:12.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:12.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 86 16 99910 99894 72.5908 64.375 0.0171239 0.0137706 2024-09-02T10:16:12.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:12.291 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:12.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:12.292 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:12.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:12.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:12.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:12.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:12.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:12.552 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:12.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:12.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:12.684 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:12.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:12.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:12.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:12 smithi151 ceph-mon[16639]: pgmap v1114: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 53 MiB/s wr, 841 op/s; 2345/287055 objects misplaced (0.817%); 2.7 MiB/s, 43 objects/s recovering 2024-09-02T10:16:12.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/70839630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:12.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:12.816 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:12.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:12.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:12.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:12.941 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:12.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:12.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:16:12.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:16:12.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:16:12.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:16:12.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:16:12.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:16:12.950 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:16:12.950 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:16:12.950 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:16:12.950 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:16:12.950 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:16:12.950 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:16:12.950 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:16:12.951 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:16:12.951 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:16:12.951 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:16:12.951 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:16:12.951 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:16:12.951 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:16:12.951 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:16:12.951 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:16:12.951 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:16:12.951 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:16:12.952 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:16:13.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:13.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:13.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:13.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:13.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:12 smithi162 ceph-mon[20279]: pgmap v1114: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 53 MiB/s wr, 841 op/s; 2345/287055 objects misplaced (0.817%); 2.7 MiB/s, 43 objects/s recovering 2024-09-02T10:16:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/70839630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:13.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:13.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 87 16 100718 100702 72.3368 50.5 0.0299976 0.0138183 2024-09-02T10:16:13.202 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:13.203 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:12 smithi187 ceph-mon[20072]: pgmap v1114: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.8 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 53 MiB/s wr, 841 op/s; 2345/287055 objects misplaced (0.817%); 2.7 MiB/s, 43 objects/s recovering 2024-09-02T10:16:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/70839630' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:13.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:13.331 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:13.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:13.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:13.457 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:13.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:13.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:13.598 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:13.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:13.599 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:13.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:13.725 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:13.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:13.822 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:13.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:13.846 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:13.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:13.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:13.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:13.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:13.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:13.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:13.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:14.015 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:13 smithi151 ceph-mon[16639]: 5.e scrub starts 2024-09-02T10:16:14.094 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:13 smithi162 ceph-mon[20279]: 5.e scrub starts 2024-09-02T10:16:14.131 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:14.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:14.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:14.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 88 16 101809 101793 72.2895 68.1875 0.00436713 0.013827 2024-09-02T10:16:14.233 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:13 smithi187 ceph-mon[20072]: 5.e scrub starts 2024-09-02T10:16:14.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:14.265 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:14.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:14.403 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:14.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:14.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:14.429 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:16:14.429 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:16:14.429 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:16:14.429 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:16:14.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:14.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:14.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:14.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:14.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:14.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:14.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:14.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:14.793 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:14.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:14.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:14.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:14.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:14.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:15.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:15.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:15.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:15.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:15.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:14 smithi162 ceph-mon[20279]: pgmap v1115: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 2056/297963 objects misplaced (0.690%); 4.4 MiB/s, 69 objects/s recovering 2024-09-02T10:16:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:14 smithi162 ceph-mon[20279]: osdmap e479: 12 total, 12 up, 10 in 2024-09-02T10:16:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:14 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:16:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:14 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:16:15.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:14 smithi151 ceph-mon[16639]: pgmap v1115: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 2056/297963 objects misplaced (0.690%); 4.4 MiB/s, 69 objects/s recovering 2024-09-02T10:16:15.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:14 smithi151 ceph-mon[16639]: osdmap e479: 12 total, 12 up, 10 in 2024-09-02T10:16:15.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:14 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:16:15.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:14 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:16:15.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:16:15.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:15.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 89 16 102799 102783 72.1725 61.875 0.0163933 0.0138501 2024-09-02T10:16:15.184 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:15.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:14 smithi187 ceph-mon[20072]: pgmap v1115: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 69 MiB/s wr, 1.10k op/s; 2056/297963 objects misplaced (0.690%); 4.4 MiB/s, 69 objects/s recovering 2024-09-02T10:16:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:14 smithi187 ceph-mon[20072]: osdmap e479: 12 total, 12 up, 10 in 2024-09-02T10:16:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:14 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:16:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:14 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:16:15.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:15.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:15.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:15.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:15.327 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:16:15.422 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:15.446 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:15.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:15.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:15.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:15.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:15.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:15.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:15.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:15.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:15.748 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:15.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:16:15.806 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:15.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:15.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:15.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:15.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:15 smithi151 ceph-mon[16639]: 5.c scrub starts 2024-09-02T10:16:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:15 smithi151 ceph-mon[16639]: 5.e scrub ok 2024-09-02T10:16:15.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:15.986 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:15.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:15.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:15.990 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:16:15.990 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:16:16.088 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:16.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:15 smithi162 ceph-mon[20279]: 5.c scrub starts 2024-09-02T10:16:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:15 smithi162 ceph-mon[20279]: 5.e scrub ok 2024-09-02T10:16:16.120 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:16.121 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:16.175 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 90 10 103602 103592 71.9323 50.5625 0.023431 0.0138965 2024-09-02T10:16:16.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:16.243 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:16.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:15 smithi187 ceph-mon[20072]: 5.c scrub starts 2024-09-02T10:16:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:15 smithi187 ceph-mon[20072]: 5.e scrub ok 2024-09-02T10:16:16.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:16.366 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:16.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:16.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:16.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:16.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:16.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:16.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:16.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:16.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:16.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:16.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:16.738 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:16.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:16.839 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:16.861 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:16.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:16.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:16.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:16.984 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:16.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:16.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:17.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:17.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:16 smithi162 ceph-mon[20279]: pgmap v1117: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 51 MiB/s wr, 821 op/s; 2056/300669 objects misplaced (0.684%); 2.9 MiB/s, 45 objects/s recovering 2024-09-02T10:16:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:16 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:16:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:16 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:16:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:16 smithi162 ceph-mon[20279]: 5.7 scrub starts 2024-09-02T10:16:17.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:17.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total time run: 90.5208 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total writes made: 103602 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Write size: 65536 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object size: 65536 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Bandwidth (MB/sec): 71.5319 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Bandwidth: 23.0907 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max bandwidth (MB/sec): 126.562 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min bandwidth (MB/sec): 0 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average IOPS: 1144 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev IOPS: 369.454 2024-09-02T10:16:17.176 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max IOPS: 2025 2024-09-02T10:16:17.177 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min IOPS: 0 2024-09-02T10:16:17.177 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average Latency(s): 0.013923 2024-09-02T10:16:17.177 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Latency(s): 0.0389981 2024-09-02T10:16:17.177 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max latency(s): 2.22758 2024-09-02T10:16:17.177 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min latency(s): 0.00106838 2024-09-02T10:16:17.180 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Cleaning up (deleting benchmark objects) 2024-09-02T10:16:17.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:16 smithi151 ceph-mon[16639]: pgmap v1117: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 51 MiB/s wr, 821 op/s; 2056/300669 objects misplaced (0.684%); 2.9 MiB/s, 45 objects/s recovering 2024-09-02T10:16:17.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:16 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:16:17.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:16 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:16:17.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:16 smithi151 ceph-mon[16639]: 5.7 scrub starts 2024-09-02T10:16:17.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:17.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:17.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:17.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:16 smithi187 ceph-mon[20072]: pgmap v1117: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 51 MiB/s wr, 821 op/s; 2056/300669 objects misplaced (0.684%); 2.9 MiB/s, 45 objects/s recovering 2024-09-02T10:16:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:16 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:16:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:16 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:16:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:16 smithi187 ceph-mon[20072]: 5.7 scrub starts 2024-09-02T10:16:17.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:17.376 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:17.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:17.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:17.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:17.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:17.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:17.627 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:17.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:17.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:17.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:17.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:17.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:17.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:17.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:17.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:17.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:17.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:18.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:18.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:17 smithi162 ceph-mon[20279]: from='client.16629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:17 smithi162 ceph-mon[20279]: 5.7 scrub ok 2024-09-02T10:16:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:17 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:16:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:17 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:16:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:17 smithi162 ceph-mon[20279]: 5.c scrub ok 2024-09-02T10:16:18.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:18.126 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:18.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:18.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:18.131 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:17 smithi151 ceph-mon[16639]: from='client.16629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:18.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:17 smithi151 ceph-mon[16639]: 5.7 scrub ok 2024-09-02T10:16:18.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:17 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:16:18.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:17 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:16:18.132 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:17 smithi151 ceph-mon[16639]: 5.c scrub ok 2024-09-02T10:16:18.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:18.250 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:18.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:18.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:17 smithi187 ceph-mon[20072]: from='client.16629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:17 smithi187 ceph-mon[20072]: 5.7 scrub ok 2024-09-02T10:16:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:17 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:16:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:17 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:16:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:17 smithi187 ceph-mon[20072]: 5.c scrub ok 2024-09-02T10:16:18.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:18.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:18.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:18.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:18.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:18.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:18.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:18.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:18.630 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:18.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:18.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:18.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:18.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:18.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:18.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:18.883 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:18.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:18.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:19.012 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:19.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:18 smithi162 ceph-mon[20279]: pgmap v1118: 47 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 44 active+clean; 6.2 GiB data, 23 GiB used, 872 GiB / 894 GiB avail; 56 MiB/s wr, 899 op/s; 2056/306729 objects misplaced (0.670%); 2.3 MiB/s, 36 objects/s recovering 2024-09-02T10:16:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:18 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:16:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:18 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:16:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:18 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:16:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:18 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:18 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:16:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:18 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:19.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:19.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:19.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:18 smithi151 ceph-mon[16639]: pgmap v1118: 47 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 44 active+clean; 6.2 GiB data, 23 GiB used, 872 GiB / 894 GiB avail; 56 MiB/s wr, 899 op/s; 2056/306729 objects misplaced (0.670%); 2.3 MiB/s, 36 objects/s recovering 2024-09-02T10:16:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:18 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:16:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:18 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:16:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:18 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:16:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:18 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:18 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:16:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:18 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:19.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:18 smithi187 ceph-mon[20072]: pgmap v1118: 47 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 44 active+clean; 6.2 GiB data, 23 GiB used, 872 GiB / 894 GiB avail; 56 MiB/s wr, 899 op/s; 2056/306729 objects misplaced (0.670%); 2.3 MiB/s, 36 objects/s recovering 2024-09-02T10:16:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:18 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:16:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:18 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:16:19.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:18 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:16:19.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:18 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:19.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:18 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:16:19.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:18 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:19.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:19.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:19.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:19.395 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:19.396 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:19.497 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:19.524 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:19.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:19.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:19.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:19.657 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:19.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:19.658 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:19.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:19.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:19.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:19.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:19.914 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:19.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:19.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:19 smithi151 ceph-mon[16639]: osdmap e480: 12 total, 12 up, 10 in 2024-09-02T10:16:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:19 smithi151 ceph-mon[16639]: 5.b scrub starts 2024-09-02T10:16:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:19 smithi151 ceph-mon[16639]: osdmap e481: 12 total, 12 up, 10 in 2024-09-02T10:16:20.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:20.039 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:20.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:20.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:19 smithi162 ceph-mon[20279]: osdmap e480: 12 total, 12 up, 10 in 2024-09-02T10:16:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:19 smithi162 ceph-mon[20279]: 5.b scrub starts 2024-09-02T10:16:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:19 smithi162 ceph-mon[20279]: osdmap e481: 12 total, 12 up, 10 in 2024-09-02T10:16:20.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:20.165 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:20.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:19 smithi187 ceph-mon[20072]: osdmap e480: 12 total, 12 up, 10 in 2024-09-02T10:16:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:19 smithi187 ceph-mon[20072]: 5.b scrub starts 2024-09-02T10:16:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:19 smithi187 ceph-mon[20072]: osdmap e481: 12 total, 12 up, 10 in 2024-09-02T10:16:20.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:20.291 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:20.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:20.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:20.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:20.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:20.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:20.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:20.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:20.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:20.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:20.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:20.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:20.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:20.806 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:20.807 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:20.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:20.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:20.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:21.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:21.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:21.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:21.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:20 smithi162 ceph-mon[20279]: pgmap v1121: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 44 MiB/s wr, 776 op/s; 1708/309204 objects misplaced (0.552%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:16:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:20 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:16:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:20 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:16:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:20 smithi162 ceph-mon[20279]: 5.1 scrub starts 2024-09-02T10:16:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:20 smithi162 ceph-mon[20279]: 5.a scrub starts 2024-09-02T10:16:21.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:21.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:20 smithi151 ceph-mon[16639]: pgmap v1121: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 44 MiB/s wr, 776 op/s; 1708/309204 objects misplaced (0.552%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:16:21.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:20 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:16:21.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:20 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:16:21.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:20 smithi151 ceph-mon[16639]: 5.1 scrub starts 2024-09-02T10:16:21.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:20 smithi151 ceph-mon[16639]: 5.a scrub starts 2024-09-02T10:16:21.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:21.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:20 smithi187 ceph-mon[20072]: pgmap v1121: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 44 MiB/s wr, 776 op/s; 1708/309204 objects misplaced (0.552%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:16:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:20 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:16:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:20 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:16:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:20 smithi187 ceph-mon[20072]: 5.1 scrub starts 2024-09-02T10:16:21.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:20 smithi187 ceph-mon[20072]: 5.a scrub starts 2024-09-02T10:16:21.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:21.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:21.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:21.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:21.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:21.451 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:21.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:21.491 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:16:21.491 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:16:21.491 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:16:21.491 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:16:21.492 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:16:21.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:21.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:21.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:21.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:21.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:21.712 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:21.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:21.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:21.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:21.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:21.942 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:21.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:21.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:22.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:22.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:21 smithi162 ceph-mon[20279]: 5.b scrub ok 2024-09-02T10:16:22.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:21 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:16:22.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:21 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:16:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:22.096 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:22.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:22.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:21 smithi187 ceph-mon[20072]: 5.b scrub ok 2024-09-02T10:16:22.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:21 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:16:22.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:21 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:16:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:21 smithi151 ceph-mon[16639]: 5.b scrub ok 2024-09-02T10:16:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:21 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:16:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:21 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:16:22.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:22.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:22.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:22.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:22.349 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:22.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:22.451 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:22.480 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:22.481 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:22.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:22.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:22.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:22.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:22.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:22.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:22.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:22.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:22.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:22.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:22.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:22.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:16:22.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:23.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:23.007 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:22 smithi162 ceph-mon[20279]: pgmap v1122: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 31 MiB/s wr, 616 op/s; 1708/308331 objects misplaced (0.554%); 3.0 MiB/s, 47 objects/s recovering 2024-09-02T10:16:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:22 smithi162 ceph-mon[20279]: 5.a scrub ok 2024-09-02T10:16:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:22 smithi162 ceph-mon[20279]: 5.1 scrub ok 2024-09-02T10:16:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:22 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:16:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:22 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:16:23.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:22 smithi151 ceph-mon[16639]: pgmap v1122: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 31 MiB/s wr, 616 op/s; 1708/308331 objects misplaced (0.554%); 3.0 MiB/s, 47 objects/s recovering 2024-09-02T10:16:23.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:22 smithi151 ceph-mon[16639]: 5.a scrub ok 2024-09-02T10:16:23.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:22 smithi151 ceph-mon[16639]: 5.1 scrub ok 2024-09-02T10:16:23.105 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:22 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:16:23.106 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:22 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:16:23.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:23.113 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:22 smithi187 ceph-mon[20072]: pgmap v1122: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 23 GiB used, 871 GiB / 894 GiB avail; 31 MiB/s wr, 616 op/s; 1708/308331 objects misplaced (0.554%); 3.0 MiB/s, 47 objects/s recovering 2024-09-02T10:16:23.113 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:22 smithi187 ceph-mon[20072]: 5.a scrub ok 2024-09-02T10:16:23.113 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:22 smithi187 ceph-mon[20072]: 5.1 scrub ok 2024-09-02T10:16:23.113 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:22 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:16:23.113 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:22 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:16:23.137 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:23.138 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:23.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:23.268 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:23.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:23.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:23.403 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:23.404 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:23.479 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1117,"stamp":"2024-09-02T10:16:15.298335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6568673824,"num_objects":100223,"num_object_clones":0,"num_object_copies":300669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2056,"num_objects_unfound":0,"num_objects_dirty":100223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100221,"num_write_kb":6414144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3655,"num_bytes_recovered":240067648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103073,"ondisk_log_size":103073,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":23052204,"kb_used_data":19422328,"kb_used_omap":15,"kb_used_meta":3629680,"kb_avail":914399316,"statfs":{"total":959950356480,"available":936344899584,"internally_reserved":0,"allocated":19888463872,"data_stored":19882040186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3716792753},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":339148800,"num_objects":5175,"num_object_clones":0,"num_object_copies":15525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-289,"num_objects_unfound":0,"num_objects_dirty":5175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5175,"num_write_kb":331200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":289,"num_bytes_recovered":18939904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.297941"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":283,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.690066+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:16:10.690066+0000","last_peered":"2024-09-02T10:16:10.690066+0000","last_clean":"2024-09-02T10:16:10.690066+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:16:10.690066+0000","last_undegraded":"2024-09-02T10:16:10.690066+0000","last_fullsized":"2024-09-02T10:16:10.690066+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":91,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.686593+0000","last_change":"2024-09-02T10:15:41.899476+0000","last_active":"2024-09-02T10:16:10.686593+0000","last_peered":"2024-09-02T10:16:10.686593+0000","last_clean":"2024-09-02T10:16:10.686593+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:16:10.686593+0000","last_undegraded":"2024-09-02T10:16:10.686593+0000","last_fullsized":"2024-09-02T10:16:10.686593+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:41.899397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:41.899397+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:51.002146+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":542,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-02T10:16:14.010166+0000","last_change":"2024-09-02T10:15:12.091472+0000","last_active":"2024-09-02T10:16:14.010166+0000","last_peered":"2024-09-02T10:16:14.010166+0000","last_clean":"2024-09-02T10:16:14.010166+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:16:14.010166+0000","last_undegraded":"2024-09-02T10:16:14.010166+0000","last_fullsized":"2024-09-02T10:16:14.010166+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.091410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.091410+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:27.631420+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":542,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-02T10:16:14.009331+0000","last_change":"2024-09-02T10:15:19.065187+0000","last_active":"2024-09-02T10:16:14.009331+0000","last_peered":"2024-09-02T10:16:14.009331+0000","last_clean":"2024-09-02T10:16:14.009331+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:16:14.009331+0000","last_undegraded":"2024-09-02T10:16:14.009331+0000","last_fullsized":"2024-09-02T10:16:14.009331+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:19.065134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:19.065134+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:36.285460+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":462,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.691540+0000","last_change":"2024-09-02T10:15:35.202261+0000","last_active":"2024-09-02T10:16:10.691540+0000","last_peered":"2024-09-02T10:16:10.691540+0000","last_clean":"2024-09-02T10:16:10.691540+0000","last_became_active":"2024-09-02T10:15:35.201887+0000","last_became_peered":"2024-09-02T10:15:35.201887+0000","last_unstale":"2024-09-02T10:16:10.691540+0000","last_undegraded":"2024-09-02T10:16:10.691540+0000","last_fullsized":"2024-09-02T10:16:10.691540+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:04.994667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:04.994667+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:30.739058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":390,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.693562+0000","last_change":"2024-09-02T10:14:11.689851+0000","last_active":"2024-09-02T10:16:10.693562+0000","last_peered":"2024-09-02T10:16:10.693562+0000","last_clean":"2024-09-02T10:16:10.693562+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:16:10.693562+0000","last_undegraded":"2024-09-02T10:16:10.693562+0000","last_fullsized":"2024-09-02T10:16:10.693562+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.689767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":905,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.687724+0000","last_change":"2024-09-02T10:15:07.059223+0000","last_active":"2024-09-02T10:16:10.687724+0000","last_peered":"2024-09-02T10:16:10.687724+0000","last_clean":"2024-09-02T10:16:10.687724+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:16:10.687724+0000","last_undegraded":"2024-09-02T10:16:10.687724+0000","last_fullsized":"2024-09-02T10:16:10.687724+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:45.351862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:45.351862+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":558,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-02T10:16:14.009520+0000","last_change":"2024-09-02T10:16:10.994908+0000","last_active":"2024-09-02T10:16:14.009520+0000","last_peered":"2024-09-02T10:16:14.009520+0000","last_clean":"2024-09-02T10:16:14.009520+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:16:14.009520+0000","last_undegraded":"2024-09-02T10:16:14.009520+0000","last_fullsized":"2024-09-02T10:16:14.009520+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.994824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.994824+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:17.261476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":554,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.686671+0000","last_change":"2024-09-02T10:15:34.791760+0000","last_active":"2024-09-02T10:16:10.686671+0000","last_peered":"2024-09-02T10:16:10.686671+0000","last_clean":"2024-09-02T10:16:10.686671+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:16:10.686671+0000","last_undegraded":"2024-09-02T10:16:10.686671+0000","last_fullsized":"2024-09-02T10:16:10.686671+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:34.791708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:34.791708+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.857252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":523,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.691599+0000","last_change":"2024-09-02T10:14:49.165646+0000","last_active":"2024-09-02T10:16:10.691599+0000","last_peered":"2024-09-02T10:16:10.691599+0000","last_clean":"2024-09-02T10:16:10.691599+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:16:10.691599+0000","last_undegraded":"2024-09-02T10:16:10.691599+0000","last_fullsized":"2024-09-02T10:16:10.691599+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:49.165586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:49.165586+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":375,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.691771+0000","last_change":"2024-09-02T10:14:45.203339+0000","last_active":"2024-09-02T10:16:10.691771+0000","last_peered":"2024-09-02T10:16:10.691771+0000","last_clean":"2024-09-02T10:16:10.691771+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:16:10.691771+0000","last_undegraded":"2024-09-02T10:16:10.691771+0000","last_fullsized":"2024-09-02T10:16:10.691771+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:45.203278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:45.203278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":880,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.686597+0000","last_change":"2024-09-02T10:15:06.447417+0000","last_active":"2024-09-02T10:16:10.686597+0000","last_peered":"2024-09-02T10:16:10.686597+0000","last_clean":"2024-09-02T10:16:10.686597+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:16:10.686597+0000","last_undegraded":"2024-09-02T10:16:10.686597+0000","last_fullsized":"2024-09-02T10:16:10.686597+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:06.447372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:06.447372+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:13.751465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":815,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.687912+0000","last_change":"2024-09-02T10:15:07.059344+0000","last_active":"2024-09-02T10:16:10.687912+0000","last_peered":"2024-09-02T10:16:10.687912+0000","last_clean":"2024-09-02T10:16:10.687912+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:16:10.687912+0000","last_undegraded":"2024-09-02T10:16:10.687912+0000","last_fullsized":"2024-09-02T10:16:10.687912+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.252526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.252526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:15.297389+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":802,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.691917+0000","last_change":"2024-09-02T10:15:29.170702+0000","last_active":"2024-09-02T10:16:10.691917+0000","last_peered":"2024-09-02T10:16:10.691917+0000","last_clean":"2024-09-02T10:16:10.691917+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:16:10.691917+0000","last_undegraded":"2024-09-02T10:16:10.691917+0000","last_fullsized":"2024-09-02T10:16:10.691917+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.665438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.665438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":793,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.687736+0000","last_change":"2024-09-02T10:14:34.217697+0000","last_active":"2024-09-02T10:16:10.687736+0000","last_peered":"2024-09-02T10:16:10.687736+0000","last_clean":"2024-09-02T10:16:10.687736+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:16:10.687736+0000","last_undegraded":"2024-09-02T10:16:10.687736+0000","last_fullsized":"2024-09-02T10:16:10.687736+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.217624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.217624+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"478'6242","reported_seq":9899,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:13.600823+0000","last_change":"2024-09-02T10:15:24.164550+0000","last_active":"2024-09-02T10:16:13.600823+0000","last_peered":"2024-09-02T10:16:13.600823+0000","last_clean":"2024-09-02T10:16:13.600823+0000","last_became_active":"2024-09-02T10:15:24.141368+0000","last_became_peered":"2024-09-02T10:15:24.141368+0000","last_unstale":"2024-09-02T10:16:13.600823+0000","last_undegraded":"2024-09-02T10:16:13.600823+0000","last_fullsized":"2024-09-02T10:16:13.600823+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6242,"log_dups_size":0,"ondisk_log_size":6242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409075712,"num_objects":6242,"num_object_clones":0,"num_object_copies":18726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6242,"num_write_kb":399488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":115933184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":880,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.687862+0000","last_change":"2024-09-02T10:14:41.306927+0000","last_active":"2024-09-02T10:16:10.687862+0000","last_peered":"2024-09-02T10:16:10.687862+0000","last_clean":"2024-09-02T10:16:10.687862+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:16:10.687862+0000","last_undegraded":"2024-09-02T10:16:10.687862+0000","last_fullsized":"2024-09-02T10:16:10.687862+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:41.306868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:41.306868+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"478'6205","reported_seq":6310,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:12.088166+0000","last_change":"2024-09-02T10:14:39.040351+0000","last_active":"2024-09-02T10:16:12.088166+0000","last_peered":"2024-09-02T10:16:12.088166+0000","last_clean":"2024-09-02T10:16:12.088166+0000","last_became_active":"2024-09-02T10:14:39.040025+0000","last_became_peered":"2024-09-02T10:14:39.040025+0000","last_unstale":"2024-09-02T10:16:12.088166+0000","last_undegraded":"2024-09-02T10:16:12.088166+0000","last_fullsized":"2024-09-02T10:16:12.088166+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6205,"log_dups_size":0,"ondisk_log_size":6205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406650880,"num_objects":6205,"num_object_clones":0,"num_object_copies":18615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6205,"num_write_kb":397120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":995,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.691761+0000","last_change":"2024-09-02T10:14:36.143055+0000","last_active":"2024-09-02T10:16:10.691761+0000","last_peered":"2024-09-02T10:16:10.691761+0000","last_clean":"2024-09-02T10:16:10.691761+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:16:10.691761+0000","last_undegraded":"2024-09-02T10:16:10.691761+0000","last_fullsized":"2024-09-02T10:16:10.691761+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.142993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.142993+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"478'6207","reported_seq":6324,"reported_epoch":478,"state":"active+clean+scrubbing","last_fresh":"2024-09-02T10:16:12.953869+0000","last_change":"2024-09-02T10:16:12.862856+0000","last_active":"2024-09-02T10:16:12.953869+0000","last_peered":"2024-09-02T10:16:12.953869+0000","last_clean":"2024-09-02T10:16:12.953869+0000","last_became_active":"2024-09-02T10:14:39.038164+0000","last_became_peered":"2024-09-02T10:14:39.038164+0000","last_unstale":"2024-09-02T10:16:12.953869+0000","last_undegraded":"2024-09-02T10:16:12.953869+0000","last_fullsized":"2024-09-02T10:16:12.953869+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":799,"log_size":6207,"log_dups_size":0,"ondisk_log_size":6207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406781952,"num_objects":6207,"num_object_clones":0,"num_object_copies":18621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6207,"num_write_kb":397248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":646,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.884551+0000","last_change":"2024-09-02T10:16:10.884551+0000","last_active":"2024-09-02T10:16:10.884551+0000","last_peered":"2024-09-02T10:16:10.884551+0000","last_clean":"2024-09-02T10:16:10.884551+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:16:10.884551+0000","last_undegraded":"2024-09-02T10:16:10.884551+0000","last_fullsized":"2024-09-02T10:16:10.884551+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.884481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.884481+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:22.734487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"478'6314","reported_seq":6419,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:12.033234+0000","last_change":"2024-09-02T10:14:39.037573+0000","last_active":"2024-09-02T10:16:12.033234+0000","last_peered":"2024-09-02T10:16:12.033234+0000","last_clean":"2024-09-02T10:16:12.033234+0000","last_became_active":"2024-09-02T10:14:39.037121+0000","last_became_peered":"2024-09-02T10:14:39.037121+0000","last_unstale":"2024-09-02T10:16:12.033234+0000","last_undegraded":"2024-09-02T10:16:12.033234+0000","last_fullsized":"2024-09-02T10:16:12.033234+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413794304,"num_objects":6314,"num_object_clones":0,"num_object_copies":18942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6314,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6314,"num_write_kb":404096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":42,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.689694+0000","last_change":"2024-09-02T10:15:35.208246+0000","last_active":"2024-09-02T10:16:10.689694+0000","last_peered":"2024-09-02T10:16:10.689694+0000","last_clean":"2024-09-02T10:16:10.689694+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:16:10.689694+0000","last_undegraded":"2024-09-02T10:16:10.689694+0000","last_fullsized":"2024-09-02T10:16:10.689694+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.668602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.668602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"478'6179","reported_seq":6284,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:11.609425+0000","last_change":"2024-09-02T10:14:39.642395+0000","last_active":"2024-09-02T10:16:11.609425+0000","last_peered":"2024-09-02T10:16:11.609425+0000","last_clean":"2024-09-02T10:16:11.609425+0000","last_became_active":"2024-09-02T10:14:39.641995+0000","last_became_peered":"2024-09-02T10:14:39.641995+0000","last_unstale":"2024-09-02T10:16:11.609425+0000","last_undegraded":"2024-09-02T10:16:11.609425+0000","last_fullsized":"2024-09-02T10:16:11.609425+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6179,"log_dups_size":0,"ondisk_log_size":6179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404946944,"num_objects":6179,"num_object_clones":0,"num_object_copies":18537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6179,"num_write_kb":395456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":950,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.689891+0000","last_change":"2024-09-02T10:15:29.195341+0000","last_active":"2024-09-02T10:16:10.689891+0000","last_peered":"2024-09-02T10:16:10.689891+0000","last_clean":"2024-09-02T10:16:10.689891+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:16:10.689891+0000","last_undegraded":"2024-09-02T10:16:10.689891+0000","last_fullsized":"2024-09-02T10:16:10.689891+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.277119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.277119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:28.784954+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"478'6312","reported_seq":6417,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:13.858214+0000","last_change":"2024-09-02T10:14:39.034903+0000","last_active":"2024-09-02T10:16:13.858214+0000","last_peered":"2024-09-02T10:16:13.858214+0000","last_clean":"2024-09-02T10:16:13.858214+0000","last_became_active":"2024-09-02T10:14:39.034636+0000","last_became_peered":"2024-09-02T10:14:39.034636+0000","last_unstale":"2024-09-02T10:16:13.858214+0000","last_undegraded":"2024-09-02T10:16:13.858214+0000","last_fullsized":"2024-09-02T10:16:13.858214+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6312,"log_dups_size":0,"ondisk_log_size":6312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413663232,"num_objects":6312,"num_object_clones":0,"num_object_copies":18936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6312,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6312,"num_write_kb":403968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":418,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.693295+0000","last_change":"2024-09-02T10:15:35.205684+0000","last_active":"2024-09-02T10:16:10.693295+0000","last_peered":"2024-09-02T10:16:10.693295+0000","last_clean":"2024-09-02T10:16:10.693295+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:16:10.693295+0000","last_undegraded":"2024-09-02T10:16:10.693295+0000","last_fullsized":"2024-09-02T10:16:10.693295+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.701773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.701773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"478'6150","reported_seq":6310,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:12.189320+0000","last_change":"2024-09-02T10:15:49.158612+0000","last_active":"2024-09-02T10:16:12.189320+0000","last_peered":"2024-09-02T10:16:12.189320+0000","last_clean":"2024-09-02T10:16:12.189320+0000","last_became_active":"2024-09-02T10:14:39.037822+0000","last_became_peered":"2024-09-02T10:14:39.037822+0000","last_unstale":"2024-09-02T10:16:12.189320+0000","last_undegraded":"2024-09-02T10:16:12.189320+0000","last_fullsized":"2024-09-02T10:16:12.189320+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4732","last_scrub_stamp":"2024-09-02T10:15:49.158487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.158487+0000","objects_scrubbed":4682,"log_size":6150,"log_dups_size":0,"ondisk_log_size":6150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:57.249982+0000","scrub_duration":968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403046400,"num_objects":6150,"num_object_clones":0,"num_object_copies":18450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6150,"num_write_kb":393600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":390,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.694078+0000","last_change":"2024-09-02T10:14:09.768298+0000","last_active":"2024-09-02T10:16:10.694078+0000","last_peered":"2024-09-02T10:16:10.694078+0000","last_clean":"2024-09-02T10:16:10.694078+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:16:10.694078+0000","last_undegraded":"2024-09-02T10:16:10.694078+0000","last_fullsized":"2024-09-02T10:16:10.694078+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:09.768061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:09.768061+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"478'6443","reported_seq":6548,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:12.686248+0000","last_change":"2024-09-02T10:14:39.035303+0000","last_active":"2024-09-02T10:16:12.686248+0000","last_peered":"2024-09-02T10:16:12.686248+0000","last_clean":"2024-09-02T10:16:12.686248+0000","last_became_active":"2024-09-02T10:14:39.035100+0000","last_became_peered":"2024-09-02T10:14:39.035100+0000","last_unstale":"2024-09-02T10:16:12.686248+0000","last_undegraded":"2024-09-02T10:16:12.686248+0000","last_fullsized":"2024-09-02T10:16:12.686248+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6443,"log_dups_size":0,"ondisk_log_size":6443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422248448,"num_objects":6443,"num_object_clones":0,"num_object_copies":19329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6443,"num_write_kb":412352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":141,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.690499+0000","last_change":"2024-09-02T10:15:36.299961+0000","last_active":"2024-09-02T10:16:10.690499+0000","last_peered":"2024-09-02T10:16:10.690499+0000","last_clean":"2024-09-02T10:16:10.690499+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:16:10.690499+0000","last_undegraded":"2024-09-02T10:16:10.690499+0000","last_fullsized":"2024-09-02T10:16:10.690499+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:36.299914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:36.299914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:01.870130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"478'6251","reported_seq":6356,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:12.715753+0000","last_change":"2024-09-02T10:14:39.641746+0000","last_active":"2024-09-02T10:16:12.715753+0000","last_peered":"2024-09-02T10:16:12.715753+0000","last_clean":"2024-09-02T10:16:12.715753+0000","last_became_active":"2024-09-02T10:14:39.641478+0000","last_became_peered":"2024-09-02T10:14:39.641478+0000","last_unstale":"2024-09-02T10:16:12.715753+0000","last_undegraded":"2024-09-02T10:16:12.715753+0000","last_fullsized":"2024-09-02T10:16:12.715753+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6251,"log_dups_size":0,"ondisk_log_size":6251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409665536,"num_objects":6251,"num_object_clones":0,"num_object_copies":18753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6251,"num_write_kb":400064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":391,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.693066+0000","last_change":"2024-09-02T10:14:36.623992+0000","last_active":"2024-09-02T10:16:10.693066+0000","last_peered":"2024-09-02T10:16:10.693066+0000","last_clean":"2024-09-02T10:16:10.693066+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:16:10.693066+0000","last_undegraded":"2024-09-02T10:16:10.693066+0000","last_fullsized":"2024-09-02T10:16:10.693066+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.623930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.623930+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"478'6191","reported_seq":6351,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:12.012768+0000","last_change":"2024-09-02T10:15:49.363717+0000","last_active":"2024-09-02T10:16:12.012768+0000","last_peered":"2024-09-02T10:16:12.012768+0000","last_clean":"2024-09-02T10:16:12.012768+0000","last_became_active":"2024-09-02T10:14:39.037882+0000","last_became_peered":"2024-09-02T10:14:39.037882+0000","last_unstale":"2024-09-02T10:16:12.012768+0000","last_undegraded":"2024-09-02T10:16:12.012768+0000","last_fullsized":"2024-09-02T10:16:12.012768+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4720","last_scrub_stamp":"2024-09-02T10:15:49.363588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.363588+0000","objects_scrubbed":4682,"log_size":6191,"log_dups_size":0,"ondisk_log_size":6191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:06.930040+0000","scrub_duration":1106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405733376,"num_objects":6191,"num_object_clones":0,"num_object_copies":18573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":637,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.691651+0000","last_change":"2024-09-02T10:14:40.192356+0000","last_active":"2024-09-02T10:16:10.691651+0000","last_peered":"2024-09-02T10:16:10.691651+0000","last_clean":"2024-09-02T10:16:10.691651+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:16:10.691651+0000","last_undegraded":"2024-09-02T10:16:10.691651+0000","last_fullsized":"2024-09-02T10:16:10.691651+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.192297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.192297+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"478'6229","reported_seq":6334,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:12.028132+0000","last_change":"2024-09-02T10:14:39.642277+0000","last_active":"2024-09-02T10:16:12.028132+0000","last_peered":"2024-09-02T10:16:12.028132+0000","last_clean":"2024-09-02T10:16:12.028132+0000","last_became_active":"2024-09-02T10:14:39.641932+0000","last_became_peered":"2024-09-02T10:14:39.641932+0000","last_unstale":"2024-09-02T10:16:12.028132+0000","last_undegraded":"2024-09-02T10:16:12.028132+0000","last_fullsized":"2024-09-02T10:16:12.028132+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6229,"log_dups_size":0,"ondisk_log_size":6229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:12.256099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408223744,"num_objects":6229,"num_object_clones":0,"num_object_copies":18687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6229,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6229,"num_write_kb":398656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":896,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.686768+0000","last_change":"2024-09-02T10:15:29.169027+0000","last_active":"2024-09-02T10:16:10.686768+0000","last_peered":"2024-09-02T10:16:10.686768+0000","last_clean":"2024-09-02T10:16:10.686768+0000","last_became_active":"2024-09-02T10:15:29.168692+0000","last_became_peered":"2024-09-02T10:15:29.168692+0000","last_unstale":"2024-09-02T10:16:10.686768+0000","last_undegraded":"2024-09-02T10:16:10.686768+0000","last_fullsized":"2024-09-02T10:16:10.686768+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:35.804014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:35.804014+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"478'6313","reported_seq":6418,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:11.626604+0000","last_change":"2024-09-02T10:14:39.038104+0000","last_active":"2024-09-02T10:16:11.626604+0000","last_peered":"2024-09-02T10:16:11.626604+0000","last_clean":"2024-09-02T10:16:11.626604+0000","last_became_active":"2024-09-02T10:14:39.037755+0000","last_became_peered":"2024-09-02T10:14:39.037755+0000","last_unstale":"2024-09-02T10:16:11.626604+0000","last_undegraded":"2024-09-02T10:16:11.626604+0000","last_fullsized":"2024-09-02T10:16:11.626604+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6313,"log_dups_size":0,"ondisk_log_size":6313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413728768,"num_objects":6313,"num_object_clones":0,"num_object_copies":18939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6313,"num_write_kb":404032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":655,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.691463+0000","last_change":"2024-09-02T10:15:35.202850+0000","last_active":"2024-09-02T10:16:10.691463+0000","last_peered":"2024-09-02T10:16:10.691463+0000","last_clean":"2024-09-02T10:16:10.691463+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:16:10.691463+0000","last_undegraded":"2024-09-02T10:16:10.691463+0000","last_fullsized":"2024-09-02T10:16:10.691463+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:48.187420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:48.187420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"478'6288","reported_seq":6393,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:12.843959+0000","last_change":"2024-09-02T10:14:39.039399+0000","last_active":"2024-09-02T10:16:12.843959+0000","last_peered":"2024-09-02T10:16:12.843959+0000","last_clean":"2024-09-02T10:16:12.843959+0000","last_became_active":"2024-09-02T10:14:39.039060+0000","last_became_peered":"2024-09-02T10:14:39.039060+0000","last_unstale":"2024-09-02T10:16:12.843959+0000","last_undegraded":"2024-09-02T10:16:12.843959+0000","last_fullsized":"2024-09-02T10:16:12.843959+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6288,"log_dups_size":0,"ondisk_log_size":6288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:14.893766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412090368,"num_objects":6288,"num_object_clones":0,"num_object_copies":18864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":402432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":896,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.690635+0000","last_change":"2024-09-02T10:14:34.284618+0000","last_active":"2024-09-02T10:16:10.690635+0000","last_peered":"2024-09-02T10:16:10.690635+0000","last_clean":"2024-09-02T10:16:10.690635+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:16:10.690635+0000","last_undegraded":"2024-09-02T10:16:10.690635+0000","last_fullsized":"2024-09-02T10:16:10.690635+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.284563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.284563+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:14.179292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"479'6402","reported_seq":6508,"reported_epoch":479,"state":"active+clean","last_fresh":"2024-09-02T10:16:14.308568+0000","last_change":"2024-09-02T10:14:39.036854+0000","last_active":"2024-09-02T10:16:14.308568+0000","last_peered":"2024-09-02T10:16:14.308568+0000","last_clean":"2024-09-02T10:16:14.308568+0000","last_became_active":"2024-09-02T10:14:39.036453+0000","last_became_peered":"2024-09-02T10:14:39.036453+0000","last_unstale":"2024-09-02T10:16:14.308568+0000","last_undegraded":"2024-09-02T10:16:14.308568+0000","last_fullsized":"2024-09-02T10:16:14.308568+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419561472,"num_objects":6402,"num_object_clones":0,"num_object_copies":19206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6402,"num_write_kb":409728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":430,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.693374+0000","last_change":"2024-09-02T10:14:47.184482+0000","last_active":"2024-09-02T10:16:10.693374+0000","last_peered":"2024-09-02T10:16:10.693374+0000","last_clean":"2024-09-02T10:16:10.693374+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:16:10.693374+0000","last_undegraded":"2024-09-02T10:16:10.693374+0000","last_fullsized":"2024-09-02T10:16:10.693374+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:47.184420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:47.184420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"478'6212","reported_seq":6337,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:11.133159+0000","last_change":"2024-09-02T10:15:49.395716+0000","last_active":"2024-09-02T10:16:11.133159+0000","last_peered":"2024-09-02T10:16:11.133159+0000","last_clean":"2024-09-02T10:16:11.133159+0000","last_became_active":"2024-09-02T10:14:39.037567+0000","last_became_peered":"2024-09-02T10:14:39.037567+0000","last_unstale":"2024-09-02T10:16:11.133159+0000","last_undegraded":"2024-09-02T10:16:11.133159+0000","last_fullsized":"2024-09-02T10:16:11.133159+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":1699,"log_size":6212,"log_dups_size":0,"ondisk_log_size":6212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407109632,"num_objects":6212,"num_object_clones":0,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6212,"num_write_kb":397568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":676,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.691607+0000","last_change":"2024-09-02T10:15:31.899734+0000","last_active":"2024-09-02T10:16:10.691607+0000","last_peered":"2024-09-02T10:16:10.691607+0000","last_clean":"2024-09-02T10:16:10.691607+0000","last_became_active":"2024-09-02T10:15:29.167553+0000","last_became_peered":"2024-09-02T10:15:29.167553+0000","last_unstale":"2024-09-02T10:16:10.691607+0000","last_undegraded":"2024-09-02T10:16:10.691607+0000","last_fullsized":"2024-09-02T10:16:10.691607+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:31.899662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:31.899662+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.920777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"478'6283","reported_seq":10156,"reported_epoch":478,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:16:12.744627+0000","last_change":"2024-09-02T10:15:36.347809+0000","last_active":"2024-09-02T10:16:12.744627+0000","last_peered":"2024-09-02T10:16:12.744627+0000","last_clean":"2024-09-02T10:15:34.173584+0000","last_became_active":"2024-09-02T10:15:36.297925+0000","last_became_peered":"2024-09-02T10:15:36.297925+0000","last_unstale":"2024-09-02T10:16:12.744627+0000","last_undegraded":"2024-09-02T10:16:12.744627+0000","last_fullsized":"2024-09-02T10:15:36.200774+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6283,"log_dups_size":0,"ondisk_log_size":6283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":411762688,"num_objects":6283,"num_object_clones":0,"num_object_copies":18849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2056,"num_objects_unfound":0,"num_objects_dirty":6283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6283,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1883,"num_bytes_recovered":123404288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[0,10],"avail_no_missing":["0","9","10"],"object_location_counts":[{"shards":"0,9,10","objects":2056},{"shards":"0,10","objects":4227}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":82,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-09-02T10:16:10.692780+0000","last_change":"2024-09-02T10:15:48.739408+0000","last_active":"2024-09-02T10:16:10.692780+0000","last_peered":"2024-09-02T10:16:10.692780+0000","last_clean":"2024-09-02T10:16:10.692780+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:16:10.692780+0000","last_undegraded":"2024-09-02T10:16:10.692780+0000","last_fullsized":"2024-09-02T10:16:10.692780+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:48.739340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:48.739340+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:02.899055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6568083456,"num_objects":100221,"num_object_clones":0,"num_object_copies":300663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2056,"num_objects_unfound":0,"num_objects_dirty":100221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100221,"num_write_kb":6414144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3652,"num_bytes_recovered":239337472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19833094144,"data_stored":19833094144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100221,"ondisk_log_size":100221,"up":48,"acting":47,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383873,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2266052,"kb_used_data":1869744,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91479100,"statfs":{"total":95995035648,"available":93674598400,"internally_reserved":0,"allocated":1914617856,"data_stored":1913976553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":405797316},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612806,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3017800,"kb_used_data":2761712,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":90727352,"statfs":{"total":95995035648,"available":92904808448,"internally_reserved":0,"allocated":2827993088,"data_stored":2827351785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":262207936},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841739,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2284224,"kb_used_data":1855984,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":91460928,"statfs":{"total":95995035648,"available":93655990272,"internally_reserved":0,"allocated":1900527616,"data_stored":1899886313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":438499776},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070673,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1463056,"kb_used_data":1225020,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92282096,"statfs":{"total":95995035648,"available":94496866304,"internally_reserved":0,"allocated":1254420480,"data_stored":1253774672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299605,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2000124,"kb_used_data":1585772,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":91745028,"statfs":{"total":95995035648,"available":93946908672,"internally_reserved":0,"allocated":1623830528,"data_stored":1623190178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528537,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2291904,"kb_used_data":1999728,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91453248,"statfs":{"total":95995035648,"available":93648125952,"internally_reserved":0,"allocated":2047721472,"data_stored":2047080169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724766,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3177292,"kb_used_data":2789304,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":90567860,"statfs":{"total":95995035648,"available":92741488640,"internally_reserved":0,"allocated":2856247296,"data_stored":2855602441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953699,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149929,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802240,"kb_used_data":405936,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92942912,"statfs":{"total":95995035648,"available":95173541888,"internally_reserved":0,"allocated":415678464,"data_stored":415037161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":405797320},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411564,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575090,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2141896,"kb_used_data":1745592,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91603256,"statfs":{"total":95995035648,"available":93801734144,"internally_reserved":0,"allocated":1787486208,"data_stored":1786841353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738616,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3607616,"kb_used_data":3183536,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":90137536,"statfs":{"total":95995035648,"available":92300836864,"internally_reserved":0,"allocated":3259940864,"data_stored":3259299561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3282239488,"data_stored":3282239488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1647902720,"data_stored":1647902720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":413663232,"data_stored":413663232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2878341120,"data_stored":2878341120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2046951424,"data_stored":2046951424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1638072320,"data_stored":1638072320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1257046016,"data_stored":1257046016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1899298816,"data_stored":1899298816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2854748160,"data_stored":2854748160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1914830848,"data_stored":1914830848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:16:23.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:16:23.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:23.531 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:23.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:23.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:23.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:23.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:23.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:23.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:23.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:23.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:23.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:23.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:23.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:23.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:24.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:24.042 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:24.042 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:24.044 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:24.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:24.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:24.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:23 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:16:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:23 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:16:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:23 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:16:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:23 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:16:24.128 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:23 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:16:24.128 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:23 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:16:24.128 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:23 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:16:24.128 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:23 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:16:24.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:24.172 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:23 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:16:24.172 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:23 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:16:24.172 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:23 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:16:24.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:23 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:16:24.179 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:24.181 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:24.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:24.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:24.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:24.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:24.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:24.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:24.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:24.567 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:24.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:24.650 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:16:24.650 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:16:24.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:24.682 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:16:24.682 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:16:24.694 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:24.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:24.714 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:16:24.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:16:24.745 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:16:24.745 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:16:24.772 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:16:24.772 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:16:24.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:24.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:16:24.796 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:16:24.820 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:16:24.820 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:16:24.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:24.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:24.848 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:16:24.848 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:16:24.877 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:16:24.877 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:16:24.907 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:16:24.908 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:16:24.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:24.937 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:16:24.937 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:16:24.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:24.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:24.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:24.971 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:16:24.971 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:16:25.012 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:16:25.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:24 smithi151 ceph-mon[16639]: pgmap v1123: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 127 B/s rd, 26 MiB/s wr, 950 op/s; 1388/298137 objects misplaced (0.466%); 5.2 MiB/s, 83 objects/s recovering 2024-09-02T10:16:25.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:24 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:16:25.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:24 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:25.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:24 smithi151 ceph-mon[16639]: osdmap e482: 12 total, 12 up, 10 in 2024-09-02T10:16:25.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:24 smithi151 ceph-mon[16639]: osdmap e483: 12 total, 12 up, 10 in 2024-09-02T10:16:25.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2782681757' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:25.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:24 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:25.032 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2599928471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:25.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:25.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:25.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:24 smithi162 ceph-mon[20279]: pgmap v1123: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 127 B/s rd, 26 MiB/s wr, 950 op/s; 1388/298137 objects misplaced (0.466%); 5.2 MiB/s, 83 objects/s recovering 2024-09-02T10:16:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:24 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:16:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:24 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:24 smithi162 ceph-mon[20279]: osdmap e482: 12 total, 12 up, 10 in 2024-09-02T10:16:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:24 smithi162 ceph-mon[20279]: osdmap e483: 12 total, 12 up, 10 in 2024-09-02T10:16:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2782681757' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:24 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2599928471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:25.187 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:25.213 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:25.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:24 smithi187 ceph-mon[20072]: pgmap v1123: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.1 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 127 B/s rd, 26 MiB/s wr, 950 op/s; 1388/298137 objects misplaced (0.466%); 5.2 MiB/s, 83 objects/s recovering 2024-09-02T10:16:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:24 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:16:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:24 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:24 smithi187 ceph-mon[20072]: osdmap e482: 12 total, 12 up, 10 in 2024-09-02T10:16:25.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:24 smithi187 ceph-mon[20072]: osdmap e483: 12 total, 12 up, 10 in 2024-09-02T10:16:25.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2782681757' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:25.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:24 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:25.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2599928471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:25.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:25.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:25.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:25.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:25.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:25.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:25.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:25.510 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:16:25.510 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1257,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":484,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":10,"osd_in_since":1725272054,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":99379,"data_bytes":6513295928,"bytes_used":23678398464,"bytes_avail":936271958016,"bytes_total":959950356480,"misplaced_objects":1388,"misplaced_total":298137,"misplaced_ratio":0.0046555778048346904,"recovering_objects_per_sec":83,"recovering_bytes_per_sec":5471094,"recovering_keys_per_sec":0,"num_objects_recovered":668,"num_bytes_recovered":43778048,"num_keys_recovered":0,"read_bytes_sec":127,"write_bytes_sec":27691403,"read_op_per_sec":0,"write_op_per_sec":950},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":33,"modified":"2024-09-02T10:15:39.290520+0000","services":{}},"progress_events":{}} 2024-09-02T10:16:25.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:25.604 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:25.605 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:25.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:25.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:25.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:25.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:25.861 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:25.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:25.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:25 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:16:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:25 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:16:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:25 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:25 smithi151 ceph-mon[16639]: osdmap e484: 12 total, 12 up, 10 in 2024-09-02T10:16:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:25 smithi151 ceph-mon[16639]: 5.f scrub starts 2024-09-02T10:16:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3980400753' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:16:25.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:25.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:25.994 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:25 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:16:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:25 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:16:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:25 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:25 smithi162 ceph-mon[20279]: osdmap e484: 12 total, 12 up, 10 in 2024-09-02T10:16:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:25 smithi162 ceph-mon[20279]: 5.f scrub starts 2024-09-02T10:16:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3980400753' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:16:26.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:26.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:26.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:26.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:26.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:26.254 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:25 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:16:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:25 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:16:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:25 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:25 smithi187 ceph-mon[20072]: osdmap e484: 12 total, 12 up, 10 in 2024-09-02T10:16:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:25 smithi187 ceph-mon[20072]: 5.f scrub starts 2024-09-02T10:16:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3980400753' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:16:26.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:26.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:26.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:26.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:26.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:26.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:26.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:26.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:26.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:26.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:26.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:26.760 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:26.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:26.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:26.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:26.892 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:26 smithi151 ceph-mon[16639]: pgmap v1127: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 170 B/s rd, 853 KiB/s wr, 784 op/s; 1388/295572 objects misplaced (0.470%); 3.3 MiB/s, 53 objects/s recovering 2024-09-02T10:16:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:26 smithi151 ceph-mon[16639]: 5.f scrub ok 2024-09-02T10:16:26.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:27.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:27.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:27.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:26 smithi162 ceph-mon[20279]: pgmap v1127: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 170 B/s rd, 853 KiB/s wr, 784 op/s; 1388/295572 objects misplaced (0.470%); 3.3 MiB/s, 53 objects/s recovering 2024-09-02T10:16:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:26 smithi162 ceph-mon[20279]: 5.f scrub ok 2024-09-02T10:16:27.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:27.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:27.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:27.240 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:16:27.241 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:16:27.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:26 smithi187 ceph-mon[20072]: pgmap v1127: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 6.0 GiB data, 22 GiB used, 872 GiB / 894 GiB avail; 170 B/s rd, 853 KiB/s wr, 784 op/s; 1388/295572 objects misplaced (0.470%); 3.3 MiB/s, 53 objects/s recovering 2024-09-02T10:16:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:26 smithi187 ceph-mon[20072]: 5.f scrub ok 2024-09-02T10:16:27.276 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:27.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:27.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:27.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:27.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:27.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:27.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:27.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:27.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:27.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:27.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:27.662 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:27.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:27.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:27.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:27.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:27.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:27.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:27 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:16:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:27 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:16:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:27 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:16:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:27 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:16:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:27 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:16:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:27 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:16:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:27 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:28.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:28.044 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:28.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:27 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:16:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:27 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:16:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:27 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:16:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:27 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:16:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:27 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:16:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:27 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:16:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:27 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:28.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:16:28.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:28.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:28.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:27 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:16:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:27 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:16:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:27 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:16:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:27 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:16:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:27 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:16:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:27 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:16:28.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:27 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:16:28.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:28.344 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:28.344 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:28.445 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:28.472 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:28.473 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:28.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:16:28.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:28.597 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:28.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:28.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:28.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:28.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:28.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:28.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:28.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:28.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:28.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:28.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:28.991 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:29.001 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:16:29.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:16:29.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:29.110 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:29.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:29.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:29.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:29.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:29.243 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:29.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:29.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:29.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:29.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:29.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:29.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:29.544 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:29.596 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:29.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:29.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:29.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:29.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:29.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:29.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:29.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:29.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:29.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:29.896 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:29.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:29.898 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:29.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:30.020 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:16:30.028 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:30.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:30.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:16:30.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:30.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:30.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:30.265 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:30 smithi151 ceph-mon[16639]: osdmap e485: 12 total, 12 up, 10 in 2024-09-02T10:16:30.265 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:30 smithi151 ceph-mon[16639]: pgmap v1130: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 940 op/s; 1082/281802 objects misplaced (0.384%); 3.5 MiB/s, 55 objects/s recovering 2024-09-02T10:16:30.265 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/192837815' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:16:30.266 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:30 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:16:30.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:30.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:30.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:30 smithi162 ceph-mon[20279]: osdmap e485: 12 total, 12 up, 10 in 2024-09-02T10:16:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:30 smithi162 ceph-mon[20279]: pgmap v1130: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 940 op/s; 1082/281802 objects misplaced (0.384%); 3.5 MiB/s, 55 objects/s recovering 2024-09-02T10:16:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/192837815' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:16:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:30 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:16:30.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:30.418 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:30.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:30.419 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:30 smithi187 ceph-mon[20072]: osdmap e485: 12 total, 12 up, 10 in 2024-09-02T10:16:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:30 smithi187 ceph-mon[20072]: pgmap v1130: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 940 op/s; 1082/281802 objects misplaced (0.384%); 3.5 MiB/s, 55 objects/s recovering 2024-09-02T10:16:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/192837815' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:16:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:30 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:16:30.521 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:30.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:30.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:30.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:30.685 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:30.686 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:30.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:30.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:30.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:30.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:30.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:30.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:31.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:31.070 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:31.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:31.171 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:31.199 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:31.201 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:31.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:31 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:31.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:31 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:16:31.287 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:31 smithi151 ceph-mon[16639]: osdmap e486: 12 total, 12 up, 10 in 2024-09-02T10:16:31.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:31.325 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:31.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:31.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:31 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:31.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:31 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:16:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:31 smithi162 ceph-mon[20279]: osdmap e486: 12 total, 12 up, 10 in 2024-09-02T10:16:31.396 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:16:31.396 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:16:31.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:31.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:31.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:31.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:31 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:31 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:16:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:31 smithi187 ceph-mon[20072]: osdmap e486: 12 total, 12 up, 10 in 2024-09-02T10:16:31.530 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:16:31.530 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:16:31.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:31.577 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:31.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:31.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:31.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:31.703 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:31.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:31.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:31.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:31.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:31.932 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:31.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:31.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:32.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:32.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:32.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:32.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:32.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:16:32.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:32.210 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:32.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:32.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:32.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:32.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:32.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:32 smithi162 ceph-mon[20279]: pgmap v1132: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 1082/279840 objects misplaced (0.387%); 3.0 MiB/s, 48 objects/s recovering 2024-09-02T10:16:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:32 smithi162 ceph-mon[20279]: from='client.26179 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:32 smithi151 ceph-mon[16639]: pgmap v1132: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 1082/279840 objects misplaced (0.387%); 3.0 MiB/s, 48 objects/s recovering 2024-09-02T10:16:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:32 smithi151 ceph-mon[16639]: from='client.26179 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:32.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:32.463 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:32.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:32 smithi187 ceph-mon[20072]: pgmap v1132: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.7 GiB data, 21 GiB used, 873 GiB / 894 GiB avail; 0 B/s wr, 833 op/s; 1082/279840 objects misplaced (0.387%); 3.0 MiB/s, 48 objects/s recovering 2024-09-02T10:16:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:32 smithi187 ceph-mon[20072]: from='client.26179 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:32.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:32.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:32.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:32.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:32.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:32.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:32.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:32.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:32.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:32.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:32.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:32.855 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1132,"stamp":"2024-09-02T10:16:31.301859+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6113591864,"num_objects":93280,"num_object_clones":0,"num_object_copies":279840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1082,"num_objects_unfound":0,"num_objects_dirty":93280,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113928,"num_write_kb":6630529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4629,"num_bytes_recovered":303899712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116780,"ondisk_log_size":116780,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":22001392,"kb_used_data":18335416,"kb_used_omap":15,"kb_used_meta":3665776,"kb_avail":915450128,"statfs":{"total":959950356480,"available":937420931072,"internally_reserved":0,"allocated":18775465984,"data_stored":18768747767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3753755057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-343670784,"num_objects":-5244,"num_object_clones":0,"num_object_copies":-15732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-306,"num_objects_unfound":0,"num_objects_dirty":-5244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5244,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":20054016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.291566"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":295,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.014902+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:16:25.014902+0000","last_peered":"2024-09-02T10:16:25.014902+0000","last_clean":"2024-09-02T10:16:25.014902+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:16:25.014902+0000","last_undegraded":"2024-09-02T10:16:25.014902+0000","last_fullsized":"2024-09-02T10:16:25.014902+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":103,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.011825+0000","last_change":"2024-09-02T10:15:41.899476+0000","last_active":"2024-09-02T10:16:25.011825+0000","last_peered":"2024-09-02T10:16:25.011825+0000","last_clean":"2024-09-02T10:16:25.011825+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:16:25.011825+0000","last_undegraded":"2024-09-02T10:16:25.011825+0000","last_fullsized":"2024-09-02T10:16:25.011825+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:41.899397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:41.899397+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:51.002146+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":554,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-02T10:16:29.009573+0000","last_change":"2024-09-02T10:15:12.091472+0000","last_active":"2024-09-02T10:16:29.009573+0000","last_peered":"2024-09-02T10:16:29.009573+0000","last_clean":"2024-09-02T10:16:29.009573+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:16:29.009573+0000","last_undegraded":"2024-09-02T10:16:29.009573+0000","last_fullsized":"2024-09-02T10:16:29.009573+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.091410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.091410+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":554,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-02T10:16:29.009739+0000","last_change":"2024-09-02T10:15:19.065187+0000","last_active":"2024-09-02T10:16:29.009739+0000","last_peered":"2024-09-02T10:16:29.009739+0000","last_clean":"2024-09-02T10:16:29.009739+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:16:29.009739+0000","last_undegraded":"2024-09-02T10:16:29.009739+0000","last_fullsized":"2024-09-02T10:16:29.009739+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:19.065134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:19.065134+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:36.285460+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":478,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.020261+0000","last_change":"2024-09-02T10:15:35.202261+0000","last_active":"2024-09-02T10:16:30.020261+0000","last_peered":"2024-09-02T10:16:30.020261+0000","last_clean":"2024-09-02T10:16:30.020261+0000","last_became_active":"2024-09-02T10:15:35.201887+0000","last_became_peered":"2024-09-02T10:15:35.201887+0000","last_unstale":"2024-09-02T10:16:30.020261+0000","last_undegraded":"2024-09-02T10:16:30.020261+0000","last_fullsized":"2024-09-02T10:16:30.020261+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:04.994667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:04.994667+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:30.739058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":402,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.013569+0000","last_change":"2024-09-02T10:14:11.689851+0000","last_active":"2024-09-02T10:16:25.013569+0000","last_peered":"2024-09-02T10:16:25.013569+0000","last_clean":"2024-09-02T10:16:25.013569+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:16:25.013569+0000","last_undegraded":"2024-09-02T10:16:25.013569+0000","last_fullsized":"2024-09-02T10:16:25.013569+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.689767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":926,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.011950+0000","last_change":"2024-09-02T10:16:23.306394+0000","last_active":"2024-09-02T10:16:25.011950+0000","last_peered":"2024-09-02T10:16:25.011950+0000","last_clean":"2024-09-02T10:16:25.011950+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:16:25.011950+0000","last_undegraded":"2024-09-02T10:16:25.011950+0000","last_fullsized":"2024-09-02T10:16:25.011950+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:23.306331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:23.306331+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:34.300255+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":570,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-02T10:16:29.009633+0000","last_change":"2024-09-02T10:16:10.994908+0000","last_active":"2024-09-02T10:16:29.009633+0000","last_peered":"2024-09-02T10:16:29.009633+0000","last_clean":"2024-09-02T10:16:29.009633+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:16:29.009633+0000","last_undegraded":"2024-09-02T10:16:29.009633+0000","last_fullsized":"2024-09-02T10:16:29.009633+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.994824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.994824+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:17.261476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":567,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.011685+0000","last_change":"2024-09-02T10:15:34.791760+0000","last_active":"2024-09-02T10:16:25.011685+0000","last_peered":"2024-09-02T10:16:25.011685+0000","last_clean":"2024-09-02T10:16:25.011685+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:16:25.011685+0000","last_undegraded":"2024-09-02T10:16:25.011685+0000","last_fullsized":"2024-09-02T10:16:25.011685+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:34.791708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:34.791708+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.857252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":548,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.020821+0000","last_change":"2024-09-02T10:16:14.103234+0000","last_active":"2024-09-02T10:16:30.020821+0000","last_peered":"2024-09-02T10:16:30.020821+0000","last_clean":"2024-09-02T10:16:30.020821+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:16:30.020821+0000","last_undegraded":"2024-09-02T10:16:30.020821+0000","last_fullsized":"2024-09-02T10:16:30.020821+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:14.103166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:14.103166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:33.443226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":399,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.020479+0000","last_change":"2024-09-02T10:16:11.152819+0000","last_active":"2024-09-02T10:16:30.020479+0000","last_peered":"2024-09-02T10:16:30.020479+0000","last_clean":"2024-09-02T10:16:30.020479+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:16:30.020479+0000","last_undegraded":"2024-09-02T10:16:30.020479+0000","last_fullsized":"2024-09-02T10:16:30.020479+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:11.152686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:11.152686+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:26.925270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":900,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.011183+0000","last_change":"2024-09-02T10:16:19.673642+0000","last_active":"2024-09-02T10:16:25.011183+0000","last_peered":"2024-09-02T10:16:25.011183+0000","last_clean":"2024-09-02T10:16:25.011183+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:16:25.011183+0000","last_undegraded":"2024-09-02T10:16:25.011183+0000","last_fullsized":"2024-09-02T10:16:25.011183+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:19.673572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:19.673572+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:33.436612+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":835,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.011993+0000","last_change":"2024-09-02T10:16:15.378306+0000","last_active":"2024-09-02T10:16:25.011993+0000","last_peered":"2024-09-02T10:16:25.011993+0000","last_clean":"2024-09-02T10:16:25.011993+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:16:25.011993+0000","last_undegraded":"2024-09-02T10:16:25.011993+0000","last_fullsized":"2024-09-02T10:16:25.011993+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:15.378230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:15.378230+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:18.040488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":814,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.012620+0000","last_change":"2024-09-02T10:15:29.170702+0000","last_active":"2024-09-02T10:16:25.012620+0000","last_peered":"2024-09-02T10:16:25.012620+0000","last_clean":"2024-09-02T10:16:25.012620+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:16:25.012620+0000","last_undegraded":"2024-09-02T10:16:25.012620+0000","last_fullsized":"2024-09-02T10:16:25.012620+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.665438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.665438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":805,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.011957+0000","last_change":"2024-09-02T10:14:34.217697+0000","last_active":"2024-09-02T10:16:25.011957+0000","last_peered":"2024-09-02T10:16:25.011957+0000","last_clean":"2024-09-02T10:16:25.011957+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:16:25.011957+0000","last_undegraded":"2024-09-02T10:16:25.011957+0000","last_fullsized":"2024-09-02T10:16:25.011957+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.217624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.217624+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"484'7048","reported_seq":10789,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:28.602310+0000","last_change":"2024-09-02T10:16:17.226985+0000","last_active":"2024-09-02T10:16:28.602310+0000","last_peered":"2024-09-02T10:16:28.602310+0000","last_clean":"2024-09-02T10:16:28.602310+0000","last_became_active":"2024-09-02T10:15:24.141368+0000","last_became_peered":"2024-09-02T10:15:24.141368+0000","last_unstale":"2024-09-02T10:16:28.602310+0000","last_undegraded":"2024-09-02T10:16:28.602310+0000","last_fullsized":"2024-09-02T10:16:28.602310+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6399","last_scrub_stamp":"2024-09-02T10:16:17.226765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.226765+0000","objects_scrubbed":6342,"log_size":7048,"log_dups_size":0,"ondisk_log_size":7048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:37.316444+0000","scrub_duration":2531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375652352,"num_objects":5732,"num_object_clones":0,"num_object_copies":17196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7048,"num_write_kb":408960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":115933184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":892,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.012082+0000","last_change":"2024-09-02T10:14:41.306927+0000","last_active":"2024-09-02T10:16:25.012082+0000","last_peered":"2024-09-02T10:16:25.012082+0000","last_clean":"2024-09-02T10:16:25.012082+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:16:25.012082+0000","last_undegraded":"2024-09-02T10:16:25.012082+0000","last_fullsized":"2024-09-02T10:16:25.012082+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:41.306868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:41.306868+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"484'7056","reported_seq":7173,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:27.104398+0000","last_change":"2024-09-02T10:14:39.040351+0000","last_active":"2024-09-02T10:16:27.104398+0000","last_peered":"2024-09-02T10:16:27.104398+0000","last_clean":"2024-09-02T10:16:27.104398+0000","last_became_active":"2024-09-02T10:14:39.040025+0000","last_became_peered":"2024-09-02T10:14:39.040025+0000","last_unstale":"2024-09-02T10:16:27.104398+0000","last_undegraded":"2024-09-02T10:16:27.104398+0000","last_fullsized":"2024-09-02T10:16:27.104398+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7056,"log_dups_size":0,"ondisk_log_size":7056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380698648,"num_objects":5810,"num_object_clones":0,"num_object_copies":17430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5810,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7056,"num_write_kb":411649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1019,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.020273+0000","last_change":"2024-09-02T10:16:22.269544+0000","last_active":"2024-09-02T10:16:30.020273+0000","last_peered":"2024-09-02T10:16:30.020273+0000","last_clean":"2024-09-02T10:16:30.020273+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:16:30.020273+0000","last_undegraded":"2024-09-02T10:16:30.020273+0000","last_fullsized":"2024-09-02T10:16:30.020273+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:22.269495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:22.269495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:47.553829+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"484'7071","reported_seq":7261,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:27.938941+0000","last_change":"2024-09-02T10:16:15.009837+0000","last_active":"2024-09-02T10:16:27.938941+0000","last_peered":"2024-09-02T10:16:27.938941+0000","last_clean":"2024-09-02T10:16:27.938941+0000","last_became_active":"2024-09-02T10:14:39.038164+0000","last_became_peered":"2024-09-02T10:14:39.038164+0000","last_unstale":"2024-09-02T10:16:27.938941+0000","last_undegraded":"2024-09-02T10:16:27.938941+0000","last_fullsized":"2024-09-02T10:16:27.938941+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6340","last_scrub_stamp":"2024-09-02T10:16:15.009487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:15.009487+0000","objects_scrubbed":6273,"log_size":7071,"log_dups_size":0,"ondisk_log_size":7071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:35.488395+0000","scrub_duration":2145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376504320,"num_objects":5745,"num_object_clones":0,"num_object_copies":17235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7071,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":658,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.010573+0000","last_change":"2024-09-02T10:16:10.884551+0000","last_active":"2024-09-02T10:16:25.010573+0000","last_peered":"2024-09-02T10:16:25.010573+0000","last_clean":"2024-09-02T10:16:25.010573+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:16:25.010573+0000","last_undegraded":"2024-09-02T10:16:25.010573+0000","last_fullsized":"2024-09-02T10:16:25.010573+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.884481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.884481+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:22.734487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"484'7151","reported_seq":7337,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:27.047516+0000","last_change":"2024-09-02T10:16:26.551606+0000","last_active":"2024-09-02T10:16:27.047516+0000","last_peered":"2024-09-02T10:16:27.047516+0000","last_clean":"2024-09-02T10:16:27.047516+0000","last_became_active":"2024-09-02T10:14:39.037121+0000","last_became_peered":"2024-09-02T10:14:39.037121+0000","last_unstale":"2024-09-02T10:16:27.047516+0000","last_undegraded":"2024-09-02T10:16:27.047516+0000","last_fullsized":"2024-09-02T10:16:27.047516+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'7110","last_scrub_stamp":"2024-09-02T10:16:26.551442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:26.551442+0000","objects_scrubbed":6016,"log_size":7151,"log_dups_size":0,"ondisk_log_size":7151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:27.466417+0000","scrub_duration":1258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":390135808,"num_objects":5953,"num_object_clones":0,"num_object_copies":17859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7151,"num_write_kb":419328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":54,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.014566+0000","last_change":"2024-09-02T10:15:35.208246+0000","last_active":"2024-09-02T10:16:25.014566+0000","last_peered":"2024-09-02T10:16:25.014566+0000","last_clean":"2024-09-02T10:16:25.014566+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:16:25.014566+0000","last_undegraded":"2024-09-02T10:16:25.014566+0000","last_fullsized":"2024-09-02T10:16:25.014566+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.668602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.668602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"484'6992","reported_seq":7108,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:26.594465+0000","last_change":"2024-09-02T10:14:39.642395+0000","last_active":"2024-09-02T10:16:26.594465+0000","last_peered":"2024-09-02T10:16:26.594465+0000","last_clean":"2024-09-02T10:16:26.594465+0000","last_became_active":"2024-09-02T10:14:39.641995+0000","last_became_peered":"2024-09-02T10:14:39.641995+0000","last_unstale":"2024-09-02T10:16:26.594465+0000","last_undegraded":"2024-09-02T10:16:26.594465+0000","last_fullsized":"2024-09-02T10:16:26.594465+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":6992,"log_dups_size":0,"ondisk_log_size":6992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386793472,"num_objects":5902,"num_object_clones":0,"num_object_copies":17706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6992,"num_write_kb":412608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":963,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.014706+0000","last_change":"2024-09-02T10:15:29.195341+0000","last_active":"2024-09-02T10:16:25.014706+0000","last_peered":"2024-09-02T10:16:25.014706+0000","last_clean":"2024-09-02T10:16:25.014706+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:16:25.014706+0000","last_undegraded":"2024-09-02T10:16:25.014706+0000","last_fullsized":"2024-09-02T10:16:25.014706+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.277119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.277119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:28.784954+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"484'7175","reported_seq":7362,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:28.875079+0000","last_change":"2024-09-02T10:16:22.056000+0000","last_active":"2024-09-02T10:16:28.875079+0000","last_peered":"2024-09-02T10:16:28.875079+0000","last_clean":"2024-09-02T10:16:28.875079+0000","last_became_active":"2024-09-02T10:14:39.034636+0000","last_became_peered":"2024-09-02T10:14:39.034636+0000","last_unstale":"2024-09-02T10:16:28.875079+0000","last_undegraded":"2024-09-02T10:16:28.875079+0000","last_fullsized":"2024-09-02T10:16:28.875079+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6741","last_scrub_stamp":"2024-09-02T10:16:22.055909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:22.055909+0000","objects_scrubbed":6225,"log_size":7175,"log_dups_size":0,"ondisk_log_size":7175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:24.765454+0000","scrub_duration":2093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374931456,"num_objects":5721,"num_object_clones":0,"num_object_copies":17163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7175,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":431,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.013736+0000","last_change":"2024-09-02T10:15:35.205684+0000","last_active":"2024-09-02T10:16:25.013736+0000","last_peered":"2024-09-02T10:16:25.013736+0000","last_clean":"2024-09-02T10:16:25.013736+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:16:25.013736+0000","last_undegraded":"2024-09-02T10:16:25.013736+0000","last_fullsized":"2024-09-02T10:16:25.013736+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.701773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.701773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"484'6929","reported_seq":7101,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:27.190464+0000","last_change":"2024-09-02T10:15:49.158612+0000","last_active":"2024-09-02T10:16:27.190464+0000","last_peered":"2024-09-02T10:16:27.190464+0000","last_clean":"2024-09-02T10:16:27.190464+0000","last_became_active":"2024-09-02T10:14:39.037822+0000","last_became_peered":"2024-09-02T10:14:39.037822+0000","last_unstale":"2024-09-02T10:16:27.190464+0000","last_undegraded":"2024-09-02T10:16:27.190464+0000","last_fullsized":"2024-09-02T10:16:27.190464+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4732","last_scrub_stamp":"2024-09-02T10:15:49.158487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.158487+0000","objects_scrubbed":4682,"log_size":6929,"log_dups_size":0,"ondisk_log_size":6929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:57.249982+0000","scrub_duration":968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381616128,"num_objects":5823,"num_object_clones":0,"num_object_copies":17469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6929,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":402,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.015767+0000","last_change":"2024-09-02T10:14:09.768298+0000","last_active":"2024-09-02T10:16:25.015767+0000","last_peered":"2024-09-02T10:16:25.015767+0000","last_clean":"2024-09-02T10:16:25.015767+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:16:25.015767+0000","last_undegraded":"2024-09-02T10:16:25.015767+0000","last_fullsized":"2024-09-02T10:16:25.015767+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:09.768061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:09.768061+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"484'7298","reported_seq":7415,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:27.705181+0000","last_change":"2024-09-02T10:14:39.035303+0000","last_active":"2024-09-02T10:16:27.705181+0000","last_peered":"2024-09-02T10:16:27.705181+0000","last_clean":"2024-09-02T10:16:27.705181+0000","last_became_active":"2024-09-02T10:14:39.035100+0000","last_became_peered":"2024-09-02T10:14:39.035100+0000","last_unstale":"2024-09-02T10:16:27.705181+0000","last_undegraded":"2024-09-02T10:16:27.705181+0000","last_fullsized":"2024-09-02T10:16:27.705181+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7298,"log_dups_size":0,"ondisk_log_size":7298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391774208,"num_objects":5978,"num_object_clones":0,"num_object_copies":17934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7298,"num_write_kb":424832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":154,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.014500+0000","last_change":"2024-09-02T10:15:36.299961+0000","last_active":"2024-09-02T10:16:25.014500+0000","last_peered":"2024-09-02T10:16:25.014500+0000","last_clean":"2024-09-02T10:16:25.014500+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:16:25.014500+0000","last_undegraded":"2024-09-02T10:16:25.014500+0000","last_fullsized":"2024-09-02T10:16:25.014500+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:36.299914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:36.299914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:01.870130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"484'7112","reported_seq":7229,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:27.771611+0000","last_change":"2024-09-02T10:14:39.641746+0000","last_active":"2024-09-02T10:16:27.771611+0000","last_peered":"2024-09-02T10:16:27.771611+0000","last_clean":"2024-09-02T10:16:27.771611+0000","last_became_active":"2024-09-02T10:14:39.641478+0000","last_became_peered":"2024-09-02T10:14:39.641478+0000","last_unstale":"2024-09-02T10:16:27.771611+0000","last_undegraded":"2024-09-02T10:16:27.771611+0000","last_fullsized":"2024-09-02T10:16:27.771611+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7112,"log_dups_size":0,"ondisk_log_size":7112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382992384,"num_objects":5844,"num_object_clones":0,"num_object_copies":17532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7112,"num_write_kb":414592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":403,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.013624+0000","last_change":"2024-09-02T10:14:36.623992+0000","last_active":"2024-09-02T10:16:25.013624+0000","last_peered":"2024-09-02T10:16:25.013624+0000","last_clean":"2024-09-02T10:16:25.013624+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:16:25.013624+0000","last_undegraded":"2024-09-02T10:16:25.013624+0000","last_fullsized":"2024-09-02T10:16:25.013624+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.623930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.623930+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"484'7029","reported_seq":7202,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:27.046988+0000","last_change":"2024-09-02T10:15:49.363717+0000","last_active":"2024-09-02T10:16:27.046988+0000","last_peered":"2024-09-02T10:16:27.046988+0000","last_clean":"2024-09-02T10:16:27.046988+0000","last_became_active":"2024-09-02T10:14:39.037882+0000","last_became_peered":"2024-09-02T10:14:39.037882+0000","last_unstale":"2024-09-02T10:16:27.046988+0000","last_undegraded":"2024-09-02T10:16:27.046988+0000","last_fullsized":"2024-09-02T10:16:27.046988+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4720","last_scrub_stamp":"2024-09-02T10:15:49.363588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.363588+0000","objects_scrubbed":4682,"log_size":7029,"log_dups_size":0,"ondisk_log_size":7029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:06.930040+0000","scrub_duration":1106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382795776,"num_objects":5841,"num_object_clones":0,"num_object_copies":17523,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5841,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7029,"num_write_kb":411840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":661,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.020368+0000","last_change":"2024-09-02T10:16:23.198280+0000","last_active":"2024-09-02T10:16:30.020368+0000","last_peered":"2024-09-02T10:16:30.020368+0000","last_clean":"2024-09-02T10:16:30.020368+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:16:30.020368+0000","last_undegraded":"2024-09-02T10:16:30.020368+0000","last_fullsized":"2024-09-02T10:16:30.020368+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:23.198227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:23.198227+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:52.413888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"484'7067","reported_seq":7257,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:27.040074+0000","last_change":"2024-09-02T10:16:17.087529+0000","last_active":"2024-09-02T10:16:27.040074+0000","last_peered":"2024-09-02T10:16:27.040074+0000","last_clean":"2024-09-02T10:16:27.040074+0000","last_became_active":"2024-09-02T10:14:39.641932+0000","last_became_peered":"2024-09-02T10:14:39.641932+0000","last_unstale":"2024-09-02T10:16:27.040074+0000","last_undegraded":"2024-09-02T10:16:27.040074+0000","last_fullsized":"2024-09-02T10:16:27.040074+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6467","last_scrub_stamp":"2024-09-02T10:16:17.087427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.087427+0000","objects_scrubbed":6467,"log_size":7067,"log_dups_size":0,"ondisk_log_size":7067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:47.064105+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384499712,"num_objects":5867,"num_object_clones":0,"num_object_copies":17601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7067,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":917,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.011421+0000","last_change":"2024-09-02T10:16:17.754936+0000","last_active":"2024-09-02T10:16:25.011421+0000","last_peered":"2024-09-02T10:16:25.011421+0000","last_clean":"2024-09-02T10:16:25.011421+0000","last_became_active":"2024-09-02T10:15:29.168692+0000","last_became_peered":"2024-09-02T10:15:29.168692+0000","last_unstale":"2024-09-02T10:16:25.011421+0000","last_undegraded":"2024-09-02T10:16:25.011421+0000","last_fullsized":"2024-09-02T10:16:25.011421+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:17.754866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.754866+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:39.447349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"484'7154","reported_seq":7270,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:26.584825+0000","last_change":"2024-09-02T10:14:39.038104+0000","last_active":"2024-09-02T10:16:26.584825+0000","last_peered":"2024-09-02T10:16:26.584825+0000","last_clean":"2024-09-02T10:16:26.584825+0000","last_became_active":"2024-09-02T10:14:39.037755+0000","last_became_peered":"2024-09-02T10:14:39.037755+0000","last_unstale":"2024-09-02T10:16:26.584825+0000","last_undegraded":"2024-09-02T10:16:26.584825+0000","last_fullsized":"2024-09-02T10:16:26.584825+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7154,"log_dups_size":0,"ondisk_log_size":7154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395182080,"num_objects":6030,"num_object_clones":0,"num_object_copies":18090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7154,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":671,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.019926+0000","last_change":"2024-09-02T10:15:35.202850+0000","last_active":"2024-09-02T10:16:30.019926+0000","last_peered":"2024-09-02T10:16:30.019926+0000","last_clean":"2024-09-02T10:16:30.019926+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:16:30.019926+0000","last_undegraded":"2024-09-02T10:16:30.019926+0000","last_fullsized":"2024-09-02T10:16:30.019926+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:48.187420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:48.187420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"484'7086","reported_seq":7274,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:27.933862+0000","last_change":"2024-09-02T10:16:20.568272+0000","last_active":"2024-09-02T10:16:27.933862+0000","last_peered":"2024-09-02T10:16:27.933862+0000","last_clean":"2024-09-02T10:16:27.933862+0000","last_became_active":"2024-09-02T10:14:39.039060+0000","last_became_peered":"2024-09-02T10:14:39.039060+0000","last_unstale":"2024-09-02T10:16:27.933862+0000","last_undegraded":"2024-09-02T10:16:27.933862+0000","last_fullsized":"2024-09-02T10:16:27.933862+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6670","last_scrub_stamp":"2024-09-02T10:16:20.568081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:20.568081+0000","objects_scrubbed":6316,"log_size":7086,"log_dups_size":0,"ondisk_log_size":7086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:27.515779+0000","scrub_duration":1605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382861312,"num_objects":5842,"num_object_clones":0,"num_object_copies":17526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7086,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":916,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.014402+0000","last_change":"2024-09-02T10:16:17.179894+0000","last_active":"2024-09-02T10:16:25.014402+0000","last_peered":"2024-09-02T10:16:25.014402+0000","last_clean":"2024-09-02T10:16:25.014402+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:16:25.014402+0000","last_undegraded":"2024-09-02T10:16:25.014402+0000","last_fullsized":"2024-09-02T10:16:25.014402+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:17.179849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.179849+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:24.924645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"485'7230","reported_seq":7420,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-09-02T10:16:29.318096+0000","last_change":"2024-09-02T10:16:21.735648+0000","last_active":"2024-09-02T10:16:29.318096+0000","last_peered":"2024-09-02T10:16:29.318096+0000","last_clean":"2024-09-02T10:16:29.318096+0000","last_became_active":"2024-09-02T10:14:39.036453+0000","last_became_peered":"2024-09-02T10:14:39.036453+0000","last_unstale":"2024-09-02T10:16:29.318096+0000","last_undegraded":"2024-09-02T10:16:29.318096+0000","last_fullsized":"2024-09-02T10:16:29.318096+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6768","last_scrub_stamp":"2024-09-02T10:16:21.735508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:21.735508+0000","objects_scrubbed":6243,"log_size":7230,"log_dups_size":0,"ondisk_log_size":7230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:26.135470+0000","scrub_duration":1736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375521280,"num_objects":5730,"num_object_clones":0,"num_object_copies":17190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7230,"num_write_kb":414720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":454,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.020515+0000","last_change":"2024-09-02T10:16:21.146441+0000","last_active":"2024-09-02T10:16:30.020515+0000","last_peered":"2024-09-02T10:16:30.020515+0000","last_clean":"2024-09-02T10:16:30.020515+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:16:30.020515+0000","last_undegraded":"2024-09-02T10:16:30.020515+0000","last_fullsized":"2024-09-02T10:16:30.020515+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:21.146366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:21.146366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:23.276250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"486'7391","reported_seq":7532,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:31.126076+0000","last_change":"2024-09-02T10:15:49.395716+0000","last_active":"2024-09-02T10:16:31.126076+0000","last_peered":"2024-09-02T10:16:31.126076+0000","last_clean":"2024-09-02T10:16:31.126076+0000","last_became_active":"2024-09-02T10:14:39.037567+0000","last_became_peered":"2024-09-02T10:14:39.037567+0000","last_unstale":"2024-09-02T10:16:31.126076+0000","last_undegraded":"2024-09-02T10:16:31.126076+0000","last_fullsized":"2024-09-02T10:16:31.126076+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":1699,"log_size":7391,"log_dups_size":0,"ondisk_log_size":7391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369688576,"num_objects":5641,"num_object_clones":0,"num_object_copies":16923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7391,"num_write_kb":417024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":692,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.020825+0000","last_change":"2024-09-02T10:15:31.899734+0000","last_active":"2024-09-02T10:16:30.020825+0000","last_peered":"2024-09-02T10:16:30.020825+0000","last_clean":"2024-09-02T10:16:30.020825+0000","last_became_active":"2024-09-02T10:15:29.167553+0000","last_became_peered":"2024-09-02T10:15:29.167553+0000","last_unstale":"2024-09-02T10:16:30.020825+0000","last_undegraded":"2024-09-02T10:16:30.020825+0000","last_fullsized":"2024-09-02T10:16:30.020825+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:31.899662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:31.899662+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.920777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"484'7139","reported_seq":12973,"reported_epoch":484,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:16:27.743092+0000","last_change":"2024-09-02T10:15:36.347809+0000","last_active":"2024-09-02T10:16:27.743092+0000","last_peered":"2024-09-02T10:16:27.743092+0000","last_clean":"2024-09-02T10:15:34.173584+0000","last_became_active":"2024-09-02T10:15:36.297925+0000","last_became_peered":"2024-09-02T10:15:36.297925+0000","last_unstale":"2024-09-02T10:16:27.743092+0000","last_undegraded":"2024-09-02T10:16:27.743092+0000","last_fullsized":"2024-09-02T10:15:36.200774+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7139,"log_dups_size":0,"ondisk_log_size":7139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381353984,"num_objects":5819,"num_object_clones":0,"num_object_copies":17457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1082,"num_objects_unfound":0,"num_objects_dirty":5819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":414656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":187236352,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[0,10],"avail_no_missing":["0","9","10"],"object_location_counts":[{"shards":"0,9,10","objects":1082},{"shards":"0,10","objects":4737}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":94,"reported_epoch":484,"state":"active+clean","last_fresh":"2024-09-02T10:16:25.011007+0000","last_change":"2024-09-02T10:15:48.739408+0000","last_active":"2024-09-02T10:16:25.011007+0000","last_peered":"2024-09-02T10:16:25.011007+0000","last_clean":"2024-09-02T10:16:25.011007+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:16:25.011007+0000","last_undegraded":"2024-09-02T10:16:25.011007+0000","last_fullsized":"2024-09-02T10:16:25.011007+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:48.739340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:48.739340+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:02.899055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":6113001496,"num_objects":93278,"num_object_clones":0,"num_object_copies":279834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1082,"num_objects_unfound":0,"num_objects_dirty":93278,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":113928,"num_write_kb":6630529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4626,"num_bytes_recovered":303169536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18525794304,"data_stored":18525782088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113928,"ondisk_log_size":113928,"up":48,"acting":47,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383876,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2222348,"kb_used_data":1826044,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91522804,"statfs":{"total":95995035648,"available":93719351296,"internally_reserved":0,"allocated":1869869056,"data_stored":1869196801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":405797316},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612809,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2918804,"kb_used_data":2652984,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":90826348,"statfs":{"total":95995035648,"available":93006180352,"internally_reserved":0,"allocated":2716655616,"data_stored":2715987433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":272169408},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841742,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2163144,"kb_used_data":1739064,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":91582008,"statfs":{"total":95995035648,"available":93779976192,"internally_reserved":0,"allocated":1780801536,"data_stored":1780133353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":434239936},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070676,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1340056,"kb_used_data":1110340,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92405096,"statfs":{"total":95995035648,"available":94622818304,"internally_reserved":0,"allocated":1136988160,"data_stored":1136315472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":235207109},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299609,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1868908,"kb_used_data":1450396,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":91876244,"statfs":{"total":95995035648,"available":94081273856,"internally_reserved":0,"allocated":1485205504,"data_stored":1484520687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":428538314},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528540,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2183624,"kb_used_data":1887288,"kb_used_omap":1,"kb_used_meta":296318,"kb_avail":91561528,"statfs":{"total":95995035648,"available":93759004672,"internally_reserved":0,"allocated":1932582912,"data_stored":1931914729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":303430087},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724769,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3084052,"kb_used_data":2669700,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":90661100,"statfs":{"total":95995035648,"available":92836966400,"internally_reserved":0,"allocated":2733772800,"data_stored":2733096993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953702,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149932,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768200,"kb_used_data":371896,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92976952,"statfs":{"total":95995035648,"available":95208398848,"internally_reserved":0,"allocated":380821504,"data_stored":380153321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":405797320},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411567,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575093,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2005200,"kb_used_data":1613056,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91739952,"statfs":{"total":95995035648,"available":93941710848,"internally_reserved":0,"allocated":1651769344,"data_stored":1651097609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738619,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3447056,"kb_used_data":3014648,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":90298096,"statfs":{"total":95995035648,"available":92465250304,"internally_reserved":0,"allocated":3086999552,"data_stored":3086331369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3058368512,"data_stored":3058368512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1510866944,"data_stored":1510866944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":374865920,"data_stored":374865920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2712014848,"data_stored":2712010776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1927217152,"data_stored":1927217152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1465454592,"data_stored":1465450520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1131479040,"data_stored":1131479040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1777336320,"data_stored":1777336320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2714107904,"data_stored":2714107904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1854083072,"data_stored":1854079000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:16:32.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:32.976 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:32.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:33.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:33.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:33.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:33.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:33.231 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:33.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:33.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:33.356 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:33.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:33.358 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:33.402 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:33.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:33.482 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:33.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:33.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:33.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:33.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:33.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:33.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:33.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:33.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:33.868 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:33.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:33.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:33.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:33.993 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:34.093 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:34.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:34.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:34.193 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:16:34.194 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:16:34.218 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:16:34.218 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:16:34.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:34.245 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:16:34.245 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:16:34.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:34.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:34.270 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:16:34.270 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:16:34.296 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:16:34.296 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:16:34.324 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:16:34.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:16:34.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:34.354 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:16:34.354 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:16:34.379 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:34.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:34.382 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:16:34.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:16:34.413 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:16:34.414 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:16:34.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:34 smithi151 ceph-mon[16639]: pgmap v1133: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s; 695/268827 objects misplaced (0.259%); 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:16:34.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:34 smithi151 ceph-mon[16639]: osdmap e487: 12 total, 12 up, 10 in 2024-09-02T10:16:34.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/900938714' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:34.448 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:16:34.448 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:16:34.448 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:16:34.448 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:16:34.448 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:16:34.451 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:16:34.451 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:16:34.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:34.487 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:16:34.487 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:16:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:34 smithi187 ceph-mon[20072]: pgmap v1133: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s; 695/268827 objects misplaced (0.259%); 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:16:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:34 smithi187 ceph-mon[20072]: osdmap e487: 12 total, 12 up, 10 in 2024-09-02T10:16:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/900938714' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:34.508 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:34.509 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:34.518 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:16:34.518 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:16:34.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:34.634 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:34.635 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:34.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:34.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:34.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:34.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:34 smithi162 ceph-mon[20279]: pgmap v1133: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.5 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 1.11k op/s; 695/268827 objects misplaced (0.259%); 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:16:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:34 smithi162 ceph-mon[20279]: osdmap e487: 12 total, 12 up, 10 in 2024-09-02T10:16:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/900938714' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:34.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:34.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:34.895 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:34.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:35.025 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:35.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:35.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:35.152 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:35.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:35.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:35.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:35.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:35.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:35.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:16:35.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:16:35.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:35.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:35.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:35.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:35.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:35.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:35.632 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:35 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:16:35.632 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:35 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:35.632 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:35 smithi151 ceph-mon[16639]: osdmap e488: 12 total, 12 up, 10 in 2024-09-02T10:16:35.639 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:35.665 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:35.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:35 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:16:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:35 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:35 smithi187 ceph-mon[20072]: osdmap e488: 12 total, 12 up, 10 in 2024-09-02T10:16:35.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:35.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:35.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:35 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:16:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:35 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:35 smithi162 ceph-mon[20279]: osdmap e488: 12 total, 12 up, 10 in 2024-09-02T10:16:35.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:16:35.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:35.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:35.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:36.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:36.055 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:36.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:36.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:36.180 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:36.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:36.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:36.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:36.308 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:36.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:36.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:36.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:36.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:36.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:36.558 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:36.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:36.604 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:36.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:36.682 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:36.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:36.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:36 smithi151 ceph-mon[16639]: pgmap v1136: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 856 op/s; 695/266376 objects misplaced (0.261%); 4.0 MiB/s, 64 objects/s recovering 2024-09-02T10:16:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:36 smithi187 ceph-mon[20072]: pgmap v1136: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 856 op/s; 695/266376 objects misplaced (0.261%); 4.0 MiB/s, 64 objects/s recovering 2024-09-02T10:16:36.765 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:36.766 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:36.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:36.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:36.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:36.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:36 smithi162 ceph-mon[20279]: pgmap v1136: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.4 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 856 op/s; 695/266376 objects misplaced (0.261%); 4.0 MiB/s, 64 objects/s recovering 2024-09-02T10:16:36.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:36.941 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:36.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:37.043 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:37.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:37.080 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:37.080 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:37.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:37.202 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:37.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:37.304 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:37.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:37.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:37.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:37.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:37.451 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:37.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:37.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:37.577 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:37.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:37.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:37.679 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:37.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:37 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-02T10:16:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:37.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:37.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:37.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:37 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-02T10:16:37.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:37.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:37.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:37 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-02T10:16:37.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:37.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:37.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:37.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:38.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:38.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:38.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:38.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:38.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:38.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:38.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:38.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:38.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:38.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:38.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:38.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:38.463 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:38.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:38.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:38.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:16:38.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:38.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:38.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:38.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:38 smithi151 ceph-mon[16639]: pgmap v1137: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 970 op/s; 695/259626 objects misplaced (0.268%); 3.3 MiB/s, 53 objects/s recovering 2024-09-02T10:16:38.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:38.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:38.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:38.751 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:38.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:38 smithi187 ceph-mon[20072]: pgmap v1137: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 970 op/s; 695/259626 objects misplaced (0.268%); 3.3 MiB/s, 53 objects/s recovering 2024-09-02T10:16:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:38 smithi162 ceph-mon[20279]: pgmap v1137: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.3 GiB data, 20 GiB used, 874 GiB / 894 GiB avail; 0 B/s wr, 970 op/s; 695/259626 objects misplaced (0.268%); 3.3 MiB/s, 53 objects/s recovering 2024-09-02T10:16:38.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:38.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:38.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:38.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:38.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:39.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:39.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:39.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:39.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:39.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:39.136 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:39.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:39.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:39.261 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:39.261 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:16:39.305 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:16:39.362 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:39.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:39.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:39.386 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:39.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:39.511 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:39.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:39.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:39.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:39.639 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:39.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:16:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:39 smithi151 ceph-mon[16639]: osdmap e489: 12 total, 12 up, 10 in 2024-09-02T10:16:39.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:39.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:16:39.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:39.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:39 smithi187 ceph-mon[20072]: osdmap e489: 12 total, 12 up, 10 in 2024-09-02T10:16:39.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:39.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:39.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:16:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:39 smithi162 ceph-mon[20279]: osdmap e489: 12 total, 12 up, 10 in 2024-09-02T10:16:39.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:39.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:39.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:39.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:39.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:40.020 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:40.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:40.077 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:16:40.077 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:16:40.077 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:16:40.077 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:16:40.077 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:109490 3.b 3:d613b4ca:::benchmark_data_smithi187_59989_object109489:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:32.977730+0000", 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 547.09664972600001, 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.53911734600000005, 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109490 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.078 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977730+0000", 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977730+0000", 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.079 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977732+0000", 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.497e-06 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977772+0000", 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9984000000000002e-05 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977788+0000", 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6146000000000001e-05 2024-09-02T10:16:40.080 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977796+0000", 2024-09-02T10:16:40.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5549999999999997e-06 2024-09-02T10:16:40.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.081 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.081 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.081 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.977869+0000", 2024-09-02T10:16:40.081 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3405999999999995e-05 2024-09-02T10:16:40.081 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978004+0000", 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000134912 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.978159+0000", 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000154635 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:16:40.082 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.981783+0000", 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036246579999999998 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:32.982194+0000", 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00041032900000000002 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.516775+0000", 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.53458172199999998 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.083 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:16:40.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.516815+0000", 2024-09-02T10:16:40.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9320000000000003e-05 2024-09-02T10:16:40.084 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.084 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.084 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.084 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.516848+0000", 2024-09-02T10:16:40.084 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3182000000000002e-05 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:109515 3.7 3:e47edbba:::benchmark_data_smithi187_59989_object109514:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:33.000140+0000", 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 547.07423993500004, 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.66181097899999997, 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.085 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109515 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000140+0000", 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000140+0000", 2024-09-02T10:16:40.086 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000142+0000", 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.652e-06 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000185+0000", 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2762000000000002e-05 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.087 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000199+0000", 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4448e-05 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000208+0000", 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5380000000000002e-06 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000277+0000", 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9122000000000007e-05 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.088 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000429+0000", 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015264900000000001 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.000612+0000", 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018222299999999999 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.007079+0000", 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0064675729999999999 2024-09-02T10:16:40.089 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409419+0000", 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40233940000000001 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661860+0000", 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25244123099999999 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661908+0000", 2024-09-02T10:16:40.090 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8597999999999998e-05 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661951+0000", 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2783e-05 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:109515 3.7 e263/226 ondisk, result = 0)", 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:33.004435+0000", 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 547.06994519, 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40498933300000001, 2024-09-02T10:16:40.091 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:16:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.092 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004435+0000", 2024-09-02T10:16:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004435+0000", 2024-09-02T10:16:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.093 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004436+0000", 2024-09-02T10:16:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0599999999999999e-07 2024-09-02T10:16:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.094 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004444+0000", 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0960000000000001e-06 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409298+0000", 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.404853835 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409307+0000", 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2180000000000005e-06 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.095 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409404+0000", 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6917999999999996e-05 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409413+0000", 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9299999999999992e-06 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.409424+0000", 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.153e-05 2024-09-02T10:16:40.096 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:109515 3.7 e263/226 ondisk, result = 0)", 2024-09-02T10:16:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:33.004543+0000", 2024-09-02T10:16:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 547.06983742299997, 2024-09-02T10:16:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.65741479000000003, 2024-09-02T10:16:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.097 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004543+0000", 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004543+0000", 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.098 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004543+0000", 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7199999999999999e-07 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.004549+0000", 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4430000000000002e-06 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661760+0000", 2024-09-02T10:16:40.099 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.65721097299999998 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661768+0000", 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3690000000000007e-06 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661842+0000", 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4200000000000001e-05 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.100 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661854+0000", 2024-09-02T10:16:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1608e-05 2024-09-02T10:16:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.661958+0000", 2024-09-02T10:16:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103625 2024-09-02T10:16:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.101 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:109529 3.2 3:4d5fc6c5:::benchmark_data_smithi187_59989_object109528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:33.012817+0000", 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 547.06156264499998, 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.676394685, 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:16:40.102 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109529 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012817+0000", 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012817+0000", 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.103 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012819+0000", 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6640000000000001e-06 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012865+0000", 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5345e-05 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012886+0000", 2024-09-02T10:16:40.104 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999999e-05 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012896+0000", 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4010000000000003e-06 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.012963+0000", 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7180000000000004e-05 2024-09-02T10:16:40.105 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.013114+0000", 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015087100000000001 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.013264+0000", 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015018599999999999 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.016396+0000", 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031319469999999999 2024-09-02T10:16:40.106 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.016510+0000", 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011406 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689111+0000", 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67260133600000005 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:16:40.107 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689165+0000", 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4035e-05 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689212+0000", 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6810000000000001e-05 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.24976.0:109529 3.2 e263/226 ondisk, result = 0)", 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:07:33.018049+0000", 2024-09-02T10:16:40.108 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 547.05633099099998, 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67117003600000003, 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.018049+0000", 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.109 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.018049+0000", 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.018050+0000", 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6199999999999997e-07 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.018055+0000", 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8790000000000002e-06 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.110 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689017+0000", 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67096189699999997 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689027+0000", 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0617e-05 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689094+0000", 2024-09-02T10:16:40.111 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6507000000000005e-05 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689105+0000", 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1164e-05 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:07:33.689219+0000", 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011430999999999999 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.112 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:154987 3.7 3:e9a05f93:::benchmark_data_smithi187_59989_object45444:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292679+0000", 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 493.78170082399998, 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31734095899999998, 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154987 2024-09-02T10:16:40.113 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292679+0000", 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292679+0000", 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.114 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292680+0000", 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7500000000000001e-07 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292682+0000", 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2840000000000001e-06 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292689+0000", 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6540000000000002e-06 2024-09-02T10:16:40.115 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292693+0000", 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5879999999999999e-06 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294669+0000", 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019755459999999999 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:16:40.116 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294678+0000", 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4849999999999998e-06 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.590957+0000", 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29627842999999998 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.591095+0000", 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013822899999999999 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.117 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.591239+0000", 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144518 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.606991+0000", 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015751363000000001 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.607059+0000", 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8637e-05 2024-09-02T10:16:40.118 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.607077+0000", 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7581999999999999e-05 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.607146+0000", 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8596000000000005e-05 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.610020+0000", 2024-09-02T10:16:40.119 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002874672 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155049 3.7 3:e346e101:::benchmark_data_smithi187_59989_object45506:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292699+0000", 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 493.78168069999998, 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33504108799999999, 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:40.120 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155049 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292699+0000", 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292699+0000", 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.121 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292700+0000", 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7700000000000001e-07 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292702+0000", 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9180000000000001e-06 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.122 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292719+0000", 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6994999999999999e-05 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292723+0000", 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6380000000000003e-06 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294733+0000", 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020092690000000002 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.123 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:16:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294740+0000", 2024-09-02T10:16:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4490000000000002e-06 2024-09-02T10:16:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.602174+0000", 2024-09-02T10:16:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30743415000000002 2024-09-02T10:16:40.124 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.602323+0000", 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014901700000000001 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.602441+0000", 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117873 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.125 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.621007+0000", 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018566426 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.621054+0000", 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6263000000000001e-05 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.126 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.624195+0000", 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031411410000000001 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.624316+0000", 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120695 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627741+0000", 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034249770000000001 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.127 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155068 3.7 3:e666e28e:::benchmark_data_smithi187_59989_object45525:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 493.78165019199997, 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33844332199999999, 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.128 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155068 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.129 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8599999999999999e-07 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292733+0000", 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4530000000000001e-06 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.130 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292739+0000", 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3090000000000001e-06 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292745+0000", 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8420000000000001e-06 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294766+0000", 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020206989999999999 2024-09-02T10:16:40.131 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294769+0000", 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8099999999999999e-06 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.615962+0000", 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32119220799999998 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.132 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.616132+0000", 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000170359 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.616296+0000", 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016439699999999999 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627817+0000", 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011520871 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627838+0000", 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0364999999999999e-05 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627892+0000", 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4667999999999999e-05 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627957+0000", 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4299000000000002e-05 2024-09-02T10:16:40.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631173+0000", 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003216656 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155089 3.7 3:e3752e01:::benchmark_data_smithi187_59989_object45546:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 493.78162952399998, 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35327147199999998, 2024-09-02T10:16:40.135 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155089 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0199999999999998e-07 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292753+0000", 2024-09-02T10:16:40.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.182e-06 2024-09-02T10:16:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292759+0000", 2024-09-02T10:16:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6450000000000002e-06 2024-09-02T10:16:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292763+0000", 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4530000000000004e-06 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294792+0000", 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020289150000000001 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294796+0000", 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5149999999999998e-06 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631221+0000", 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33642498300000001 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631396+0000", 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175866 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631567+0000", 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017052899999999999 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642330+0000", 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010763403 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642513+0000", 2024-09-02T10:16:40.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018293399999999999 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642560+0000", 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6344000000000002e-05 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642615+0000", 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5634000000000003e-05 2024-09-02T10:16:40.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.646022+0000", 2024-09-02T10:16:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034067670000000002 2024-09-02T10:16:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.144 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11536 4.c 4.7fa12efc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.463377+0000", 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 329.61100332500001, 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.000768433, 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11536 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463377+0000", 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463377+0000", 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463378+0000", 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.655e-06 2024-09-02T10:16:40.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463418+0000", 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9276000000000001e-05 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463428+0000", 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0074999999999999e-05 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463434+0000", 2024-09-02T10:16:40.147 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2940000000000002e-06 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465896+0000", 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024620409999999999 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465906+0000", 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8660000000000001e-06 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464105+0000", 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99819907500000005 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464145+0000", 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0151000000000002e-05 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11545 4.c 4.151c8eac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:16:40.149 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.464747+0000", 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 329.60963290900003, 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99968865100000004, 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11545 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464747+0000", 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464747+0000", 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464748+0000", 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.24e-06 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464802+0000", 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3669999999999999e-05 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464810+0000", 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8420000000000005e-06 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464817+0000", 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4209999999999998e-06 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465935+0000", 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011172109999999999 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465942+0000", 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1509999999999998e-06 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464426+0000", 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99848471999999999 2024-09-02T10:16:40.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464436+0000", 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3959999999999995e-06 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11547 4.c 4.fa8faa7c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.465762+0000", 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 329.60861854000001, 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99873187699999999, 2024-09-02T10:16:40.154 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11547 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465762+0000", 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465762+0000", 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465763+0000", 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.331e-06 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465793+0000", 2024-09-02T10:16:40.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0453000000000001e-05 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465805+0000", 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1175999999999999e-05 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465812+0000", 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1960000000000002e-06 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466125+0000", 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031365100000000002 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466134+0000", 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8340000000000005e-06 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464486+0000", 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99835216500000001 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464493+0000", 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11568 4.c 4.acfb412c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.727519+0000", 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 329.346860746, 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.73701982099999996, 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.159 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11568 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727519+0000", 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727519+0000", 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727521+0000", 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7859999999999999e-06 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727593+0000", 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1581000000000005e-05 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727607+0000", 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3916e-05 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727614+0000", 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8889999999999996e-06 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727669+0000", 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5364000000000001e-05 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727676+0000", 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5969999999999997e-06 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464533+0000", 2024-09-02T10:16:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.73685725400000002 2024-09-02T10:16:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464539+0000", 2024-09-02T10:16:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-06 2024-09-02T10:16:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:41914 4.11 e353/344 4:884e7a1d:::benchmark_data_smithi187_76542_object41913:head v 353'2322, mlcod=335'1550)", 2024-09-02T10:16:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.849702+0000", 2024-09-02T10:16:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 292.22467845699998, 2024-09-02T10:16:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44488412300000002, 2024-09-02T10:16:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.165 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849702+0000", 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849702+0000", 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849730+0000", 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8532999999999999e-05 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849763+0000", 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2478999999999999e-05 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293343+0000", 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44358014400000001 2024-09-02T10:16:40.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293349+0000", 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2489999999999998e-06 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293401+0000", 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1523999999999997e-05 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293425+0000", 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4606000000000001e-05 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294550+0000", 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011250959999999999 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294586+0000", 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5491999999999998e-05 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.169 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.11 353/344 [PushOp(4:8de9d133:::benchmark_data_smithi187_76542_object24111:head, version: 335'1553, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8de9d133:::benchmark_data_smithi187_76542_object24111:head@335'1553, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.853882+0000", 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 292.22049774499999, 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.486507095, 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853882+0000", 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853882+0000", 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853884+0000", 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5629999999999999e-06 2024-09-02T10:16:40.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853923+0000", 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8707000000000001e-05 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339685+0000", 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.485762156 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339697+0000", 2024-09-02T10:16:40.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2575000000000001e-05 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339773+0000", 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5802000000000004e-05 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339782+0000", 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6600000000000001e-06 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.340389+0000", 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060763199999999999 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.11 354/344 [PushOp(4:8d3f2f46:::benchmark_data_smithi187_76542_object24793:head, version: 335'1596, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8d3f2f46:::benchmark_data_smithi187_76542_object24793:head@335'1596, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:48.472557+0000", 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 291.60182330100002, 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31961658199999998, 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.174 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472557+0000", 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472557+0000", 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.175 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472558+0000", 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.029e-06 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472626+0000", 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8273000000000002e-05 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791323+0000", 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31869734700000002 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791336+0000", 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2418e-05 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791429+0000", 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3016999999999994e-05 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791444+0000", 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5532e-05 2024-09-02T10:16:40.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.792173+0000", 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00072896600000000005 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.11 354/344 [PushOp(4:8b8fcb96:::benchmark_data_smithi187_76542_object24835:head, version: 335'1598, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8b8fcb96:::benchmark_data_smithi187_76542_object24835:head@335'1598, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:48.476340+0000", 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 291.59804045999999, 2024-09-02T10:16:40.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31867774999999998, 2024-09-02T10:16:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:16:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476340+0000", 2024-09-02T10:16:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476340+0000", 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476346+0000", 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2840000000000003e-06 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476539+0000", 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019279599999999999 2024-09-02T10:16:40.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794378+0000", 2024-09-02T10:16:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31783950900000002 2024-09-02T10:16:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794396+0000", 2024-09-02T10:16:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7655999999999998e-05 2024-09-02T10:16:40.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794481+0000", 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4628000000000004e-05 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794491+0000", 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0114e-05 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.795017+0000", 2024-09-02T10:16:40.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052676299999999997 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:75251 4.11 4:8ba31329:::benchmark_data_smithi187_76542_object75250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 262.40943096699999, 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37234742100000001, 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:40.183 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75251 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:16:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664951+0000", 2024-09-02T10:16:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9599999999999999e-06 2024-09-02T10:16:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664999+0000", 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7738999999999998e-05 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665013+0000", 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4209999999999999e-05 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665021+0000", 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.481e-06 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.186 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665075+0000", 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4642999999999998e-05 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665173+0000", 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8022000000000004e-05 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665258+0000", 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4666000000000006e-05 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.666435+0000", 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011775080000000001 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.666722+0000", 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028679699999999999 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037199+0000", 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37047664200000002 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037256+0000", 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7605999999999997e-05 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037297+0000", 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0147e-05 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.189 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:50217 5.1 5:84aaa397:::benchmark_data_smithi187_89405_object50216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 76.282623001000005, 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.384074792, 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50217 2024-09-02T10:16:40.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:16:40.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791759+0000", 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.454e-06 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791785+0000", 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6838e-05 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791800+0000", 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4997999999999999e-05 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791808+0000", 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2370000000000003e-06 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791862+0000", 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4765000000000002e-05 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791966+0000", 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010337500000000001 2024-09-02T10:16:40.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:16:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.792051+0000", 2024-09-02T10:16:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4779999999999998e-05 2024-09-02T10:16:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:16:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.793473+0000", 2024-09-02T10:16:40.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014225819999999999 2024-09-02T10:16:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.793928+0000", 2024-09-02T10:16:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045518199999999999 2024-09-02T10:16:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175728+0000", 2024-09-02T10:16:40.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38180008999999998 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175787+0000", 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8498999999999997e-05 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175832+0000", 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4991999999999999e-05 2024-09-02T10:16:40.196 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:40.197 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:40.197 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:16:40.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:40.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:40.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:40.350 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:40.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:40.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:40.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:40.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:40.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:40.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:40.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:40.661 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1138,"stamp":"2024-09-02T10:16:38.411724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5584585272,"num_objects":85208,"num_object_clones":0,"num_object_copies":255624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":251,"num_objects_unfound":0,"num_objects_dirty":85208,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122000,"num_write_kb":6630529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5460,"num_bytes_recovered":358360128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124852,"ondisk_log_size":124852,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":20528944,"kb_used_data":16804664,"kb_used_omap":15,"kb_used_meta":3724080,"kb_avail":916922576,"statfs":{"total":959950356480,"available":938928717824,"internally_reserved":0,"allocated":17207975936,"data_stored":17201113909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3813458353},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,5,4,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-529006592,"num_objects":-8072,"num_object_clones":0,"num_object_copies":-24216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-831,"num_objects_unfound":0,"num_objects_dirty":-8072,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8072,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54460416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.109865"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":303,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.035315+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:16:35.035315+0000","last_peered":"2024-09-02T10:16:35.035315+0000","last_clean":"2024-09-02T10:16:35.035315+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:16:35.035315+0000","last_undegraded":"2024-09-02T10:16:35.035315+0000","last_fullsized":"2024-09-02T10:16:35.035315+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":107,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.024870+0000","last_change":"2024-09-02T10:15:41.899476+0000","last_active":"2024-09-02T10:16:30.024870+0000","last_peered":"2024-09-02T10:16:30.024870+0000","last_clean":"2024-09-02T10:16:30.024870+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:16:30.024870+0000","last_undegraded":"2024-09-02T10:16:30.024870+0000","last_fullsized":"2024-09-02T10:16:30.024870+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:41.899397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:41.899397+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:51.002146+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":559,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-02T10:16:34.028920+0000","last_change":"2024-09-02T10:15:12.091472+0000","last_active":"2024-09-02T10:16:34.028920+0000","last_peered":"2024-09-02T10:16:34.028920+0000","last_clean":"2024-09-02T10:16:34.028920+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:16:34.028920+0000","last_undegraded":"2024-09-02T10:16:34.028920+0000","last_fullsized":"2024-09-02T10:16:34.028920+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.091410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.091410+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:37.035788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":559,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-02T10:16:34.028796+0000","last_change":"2024-09-02T10:15:19.065187+0000","last_active":"2024-09-02T10:16:34.028796+0000","last_peered":"2024-09-02T10:16:34.028796+0000","last_clean":"2024-09-02T10:16:34.028796+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:16:34.028796+0000","last_undegraded":"2024-09-02T10:16:34.028796+0000","last_fullsized":"2024-09-02T10:16:34.028796+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:19.065134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:19.065134+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:36.285460+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":482,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.030792+0000","last_change":"2024-09-02T10:15:35.202261+0000","last_active":"2024-09-02T10:16:35.030792+0000","last_peered":"2024-09-02T10:16:35.030792+0000","last_clean":"2024-09-02T10:16:35.030792+0000","last_became_active":"2024-09-02T10:15:35.201887+0000","last_became_peered":"2024-09-02T10:15:35.201887+0000","last_unstale":"2024-09-02T10:16:35.030792+0000","last_undegraded":"2024-09-02T10:16:35.030792+0000","last_fullsized":"2024-09-02T10:16:35.030792+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:04.994667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:04.994667+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":410,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.032312+0000","last_change":"2024-09-02T10:14:11.689851+0000","last_active":"2024-09-02T10:16:35.032312+0000","last_peered":"2024-09-02T10:16:35.032312+0000","last_clean":"2024-09-02T10:16:35.032312+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:16:35.032312+0000","last_undegraded":"2024-09-02T10:16:35.032312+0000","last_fullsized":"2024-09-02T10:16:35.032312+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.689767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":934,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.034517+0000","last_change":"2024-09-02T10:16:23.306394+0000","last_active":"2024-09-02T10:16:35.034517+0000","last_peered":"2024-09-02T10:16:35.034517+0000","last_clean":"2024-09-02T10:16:35.034517+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:16:35.034517+0000","last_undegraded":"2024-09-02T10:16:35.034517+0000","last_fullsized":"2024-09-02T10:16:35.034517+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:23.306331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:23.306331+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:34.300255+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":574,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-02T10:16:34.029058+0000","last_change":"2024-09-02T10:16:10.994908+0000","last_active":"2024-09-02T10:16:34.029058+0000","last_peered":"2024-09-02T10:16:34.029058+0000","last_clean":"2024-09-02T10:16:34.029058+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:16:34.029058+0000","last_undegraded":"2024-09-02T10:16:34.029058+0000","last_fullsized":"2024-09-02T10:16:34.029058+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.994824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.994824+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:17.261476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":571,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.025169+0000","last_change":"2024-09-02T10:15:34.791760+0000","last_active":"2024-09-02T10:16:30.025169+0000","last_peered":"2024-09-02T10:16:30.025169+0000","last_clean":"2024-09-02T10:16:30.025169+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:16:30.025169+0000","last_undegraded":"2024-09-02T10:16:30.025169+0000","last_fullsized":"2024-09-02T10:16:30.025169+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:34.791708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:34.791708+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.857252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":552,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.033732+0000","last_change":"2024-09-02T10:16:14.103234+0000","last_active":"2024-09-02T10:16:35.033732+0000","last_peered":"2024-09-02T10:16:35.033732+0000","last_clean":"2024-09-02T10:16:35.033732+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:16:35.033732+0000","last_undegraded":"2024-09-02T10:16:35.033732+0000","last_fullsized":"2024-09-02T10:16:35.033732+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:14.103166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:14.103166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:33.443226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":403,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.030995+0000","last_change":"2024-09-02T10:16:11.152819+0000","last_active":"2024-09-02T10:16:35.030995+0000","last_peered":"2024-09-02T10:16:35.030995+0000","last_clean":"2024-09-02T10:16:35.030995+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:16:35.030995+0000","last_undegraded":"2024-09-02T10:16:35.030995+0000","last_fullsized":"2024-09-02T10:16:35.030995+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:11.152686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:11.152686+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:26.925270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":904,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.024883+0000","last_change":"2024-09-02T10:16:19.673642+0000","last_active":"2024-09-02T10:16:30.024883+0000","last_peered":"2024-09-02T10:16:30.024883+0000","last_clean":"2024-09-02T10:16:30.024883+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:16:30.024883+0000","last_undegraded":"2024-09-02T10:16:30.024883+0000","last_fullsized":"2024-09-02T10:16:30.024883+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:19.673572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:19.673572+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:33.436612+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":843,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.034400+0000","last_change":"2024-09-02T10:16:15.378306+0000","last_active":"2024-09-02T10:16:35.034400+0000","last_peered":"2024-09-02T10:16:35.034400+0000","last_clean":"2024-09-02T10:16:35.034400+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:16:35.034400+0000","last_undegraded":"2024-09-02T10:16:35.034400+0000","last_fullsized":"2024-09-02T10:16:35.034400+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:15.378230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:15.378230+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:18.040488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":822,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.030209+0000","last_change":"2024-09-02T10:15:29.170702+0000","last_active":"2024-09-02T10:16:35.030209+0000","last_peered":"2024-09-02T10:16:35.030209+0000","last_clean":"2024-09-02T10:16:35.030209+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:16:35.030209+0000","last_undegraded":"2024-09-02T10:16:35.030209+0000","last_fullsized":"2024-09-02T10:16:35.030209+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.665438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.665438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":813,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.034285+0000","last_change":"2024-09-02T10:14:34.217697+0000","last_active":"2024-09-02T10:16:35.034285+0000","last_peered":"2024-09-02T10:16:35.034285+0000","last_clean":"2024-09-02T10:16:35.034285+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:16:35.034285+0000","last_undegraded":"2024-09-02T10:16:35.034285+0000","last_fullsized":"2024-09-02T10:16:35.034285+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.217624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.217624+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:36.222846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"486'7323","reported_seq":11068,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:33.571751+0000","last_change":"2024-09-02T10:16:17.226985+0000","last_active":"2024-09-02T10:16:33.571751+0000","last_peered":"2024-09-02T10:16:33.571751+0000","last_clean":"2024-09-02T10:16:33.571751+0000","last_became_active":"2024-09-02T10:15:24.141368+0000","last_became_peered":"2024-09-02T10:15:24.141368+0000","last_unstale":"2024-09-02T10:16:33.571751+0000","last_undegraded":"2024-09-02T10:16:33.571751+0000","last_fullsized":"2024-09-02T10:16:33.571751+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6399","last_scrub_stamp":"2024-09-02T10:16:17.226765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.226765+0000","objects_scrubbed":6342,"log_size":7323,"log_dups_size":0,"ondisk_log_size":7323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:37.316444+0000","scrub_duration":2531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357629952,"num_objects":5457,"num_object_clones":0,"num_object_copies":16371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7323,"num_write_kb":408960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":115933184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":900,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.034384+0000","last_change":"2024-09-02T10:14:41.306927+0000","last_active":"2024-09-02T10:16:35.034384+0000","last_peered":"2024-09-02T10:16:35.034384+0000","last_clean":"2024-09-02T10:16:35.034384+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:16:35.034384+0000","last_undegraded":"2024-09-02T10:16:35.034384+0000","last_fullsized":"2024-09-02T10:16:35.034384+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:41.306868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:41.306868+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"488'7698","reported_seq":7824,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:37.113063+0000","last_change":"2024-09-02T10:14:39.040351+0000","last_active":"2024-09-02T10:16:37.113063+0000","last_peered":"2024-09-02T10:16:37.113063+0000","last_clean":"2024-09-02T10:16:37.113063+0000","last_became_active":"2024-09-02T10:14:39.040025+0000","last_became_peered":"2024-09-02T10:14:39.040025+0000","last_unstale":"2024-09-02T10:16:37.113063+0000","last_undegraded":"2024-09-02T10:16:37.113063+0000","last_fullsized":"2024-09-02T10:16:37.113063+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7698,"log_dups_size":0,"ondisk_log_size":7698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338624536,"num_objects":5168,"num_object_clones":0,"num_object_copies":15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5168,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7698,"num_write_kb":411649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1023,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.030724+0000","last_change":"2024-09-02T10:16:22.269544+0000","last_active":"2024-09-02T10:16:35.030724+0000","last_peered":"2024-09-02T10:16:35.030724+0000","last_clean":"2024-09-02T10:16:35.030724+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:16:35.030724+0000","last_undegraded":"2024-09-02T10:16:35.030724+0000","last_fullsized":"2024-09-02T10:16:35.030724+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:22.269495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:22.269495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:47.553829+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"488'7626","reported_seq":7823,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:37.943903+0000","last_change":"2024-09-02T10:16:15.009837+0000","last_active":"2024-09-02T10:16:37.943903+0000","last_peered":"2024-09-02T10:16:37.943903+0000","last_clean":"2024-09-02T10:16:37.943903+0000","last_became_active":"2024-09-02T10:14:39.038164+0000","last_became_peered":"2024-09-02T10:14:39.038164+0000","last_unstale":"2024-09-02T10:16:37.943903+0000","last_undegraded":"2024-09-02T10:16:37.943903+0000","last_fullsized":"2024-09-02T10:16:37.943903+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6340","last_scrub_stamp":"2024-09-02T10:16:15.009487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:15.009487+0000","objects_scrubbed":6273,"log_size":7626,"log_dups_size":0,"ondisk_log_size":7626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:35.488395+0000","scrub_duration":2145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340131840,"num_objects":5190,"num_object_clones":0,"num_object_copies":15570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7626,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":666,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.032039+0000","last_change":"2024-09-02T10:16:10.884551+0000","last_active":"2024-09-02T10:16:35.032039+0000","last_peered":"2024-09-02T10:16:35.032039+0000","last_clean":"2024-09-02T10:16:35.032039+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:16:35.032039+0000","last_undegraded":"2024-09-02T10:16:35.032039+0000","last_fullsized":"2024-09-02T10:16:35.032039+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.884481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.884481+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:22.734487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"488'7759","reported_seq":7954,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:37.031651+0000","last_change":"2024-09-02T10:16:26.551606+0000","last_active":"2024-09-02T10:16:37.031651+0000","last_peered":"2024-09-02T10:16:37.031651+0000","last_clean":"2024-09-02T10:16:37.031651+0000","last_became_active":"2024-09-02T10:14:39.037121+0000","last_became_peered":"2024-09-02T10:14:39.037121+0000","last_unstale":"2024-09-02T10:16:37.031651+0000","last_undegraded":"2024-09-02T10:16:37.031651+0000","last_fullsized":"2024-09-02T10:16:37.031651+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'7110","last_scrub_stamp":"2024-09-02T10:16:26.551442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:26.551442+0000","objects_scrubbed":6016,"log_size":7759,"log_dups_size":0,"ondisk_log_size":7759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:27.466417+0000","scrub_duration":1258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350289920,"num_objects":5345,"num_object_clones":0,"num_object_copies":16035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7759,"num_write_kb":419328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":62,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.035052+0000","last_change":"2024-09-02T10:15:35.208246+0000","last_active":"2024-09-02T10:16:35.035052+0000","last_peered":"2024-09-02T10:16:35.035052+0000","last_clean":"2024-09-02T10:16:35.035052+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:16:35.035052+0000","last_undegraded":"2024-09-02T10:16:35.035052+0000","last_fullsized":"2024-09-02T10:16:35.035052+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.668602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.668602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"488'7563","reported_seq":7688,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:36.564208+0000","last_change":"2024-09-02T10:14:39.642395+0000","last_active":"2024-09-02T10:16:36.564208+0000","last_peered":"2024-09-02T10:16:36.564208+0000","last_clean":"2024-09-02T10:16:36.564208+0000","last_became_active":"2024-09-02T10:14:39.641995+0000","last_became_peered":"2024-09-02T10:14:39.641995+0000","last_unstale":"2024-09-02T10:16:36.564208+0000","last_undegraded":"2024-09-02T10:16:36.564208+0000","last_fullsized":"2024-09-02T10:16:36.564208+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7563,"log_dups_size":0,"ondisk_log_size":7563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349372416,"num_objects":5331,"num_object_clones":0,"num_object_copies":15993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7563,"num_write_kb":412608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":971,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.035218+0000","last_change":"2024-09-02T10:15:29.195341+0000","last_active":"2024-09-02T10:16:35.035218+0000","last_peered":"2024-09-02T10:16:35.035218+0000","last_clean":"2024-09-02T10:16:35.035218+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:16:35.035218+0000","last_undegraded":"2024-09-02T10:16:35.035218+0000","last_fullsized":"2024-09-02T10:16:35.035218+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.277119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.277119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"486'7450","reported_seq":7641,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:33.870242+0000","last_change":"2024-09-02T10:16:22.056000+0000","last_active":"2024-09-02T10:16:33.870242+0000","last_peered":"2024-09-02T10:16:33.870242+0000","last_clean":"2024-09-02T10:16:33.870242+0000","last_became_active":"2024-09-02T10:14:39.034636+0000","last_became_peered":"2024-09-02T10:14:39.034636+0000","last_unstale":"2024-09-02T10:16:33.870242+0000","last_undegraded":"2024-09-02T10:16:33.870242+0000","last_fullsized":"2024-09-02T10:16:33.870242+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6741","last_scrub_stamp":"2024-09-02T10:16:22.055909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:22.055909+0000","objects_scrubbed":6225,"log_size":7450,"log_dups_size":0,"ondisk_log_size":7450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:24.765454+0000","scrub_duration":2093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356909056,"num_objects":5446,"num_object_clones":0,"num_object_copies":16338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7450,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":439,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.032276+0000","last_change":"2024-09-02T10:15:35.205684+0000","last_active":"2024-09-02T10:16:35.032276+0000","last_peered":"2024-09-02T10:16:35.032276+0000","last_clean":"2024-09-02T10:16:35.032276+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:16:35.032276+0000","last_undegraded":"2024-09-02T10:16:35.032276+0000","last_fullsized":"2024-09-02T10:16:35.032276+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.701773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.701773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"488'7520","reported_seq":7700,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:37.077252+0000","last_change":"2024-09-02T10:15:49.158612+0000","last_active":"2024-09-02T10:16:37.077252+0000","last_peered":"2024-09-02T10:16:37.077252+0000","last_clean":"2024-09-02T10:16:37.077252+0000","last_became_active":"2024-09-02T10:14:39.037822+0000","last_became_peered":"2024-09-02T10:14:39.037822+0000","last_unstale":"2024-09-02T10:16:37.077252+0000","last_undegraded":"2024-09-02T10:16:37.077252+0000","last_fullsized":"2024-09-02T10:16:37.077252+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4732","last_scrub_stamp":"2024-09-02T10:15:49.158487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.158487+0000","objects_scrubbed":4682,"log_size":7520,"log_dups_size":0,"ondisk_log_size":7520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:57.249982+0000","scrub_duration":968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342884352,"num_objects":5232,"num_object_clones":0,"num_object_copies":15696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":411,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.035168+0000","last_change":"2024-09-02T10:14:09.768298+0000","last_active":"2024-09-02T10:16:35.035168+0000","last_peered":"2024-09-02T10:16:35.035168+0000","last_clean":"2024-09-02T10:16:35.035168+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:16:35.035168+0000","last_undegraded":"2024-09-02T10:16:35.035168+0000","last_fullsized":"2024-09-02T10:16:35.035168+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:09.768061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:09.768061+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"488'7833","reported_seq":7958,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:37.776921+0000","last_change":"2024-09-02T10:14:39.035303+0000","last_active":"2024-09-02T10:16:37.776921+0000","last_peered":"2024-09-02T10:16:37.776921+0000","last_clean":"2024-09-02T10:16:37.776921+0000","last_became_active":"2024-09-02T10:14:39.035100+0000","last_became_peered":"2024-09-02T10:14:39.035100+0000","last_unstale":"2024-09-02T10:16:37.776921+0000","last_undegraded":"2024-09-02T10:16:37.776921+0000","last_fullsized":"2024-09-02T10:16:37.776921+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7833,"log_dups_size":0,"ondisk_log_size":7833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356712448,"num_objects":5443,"num_object_clones":0,"num_object_copies":16329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7833,"num_write_kb":424832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":162,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.032697+0000","last_change":"2024-09-02T10:15:36.299961+0000","last_active":"2024-09-02T10:16:35.032697+0000","last_peered":"2024-09-02T10:16:35.032697+0000","last_clean":"2024-09-02T10:16:35.032697+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:16:35.032697+0000","last_undegraded":"2024-09-02T10:16:35.032697+0000","last_fullsized":"2024-09-02T10:16:35.032697+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:36.299914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:36.299914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:01.870130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"488'7692","reported_seq":7817,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:37.762058+0000","last_change":"2024-09-02T10:14:39.641746+0000","last_active":"2024-09-02T10:16:37.762058+0000","last_peered":"2024-09-02T10:16:37.762058+0000","last_clean":"2024-09-02T10:16:37.762058+0000","last_became_active":"2024-09-02T10:14:39.641478+0000","last_became_peered":"2024-09-02T10:14:39.641478+0000","last_unstale":"2024-09-02T10:16:37.762058+0000","last_undegraded":"2024-09-02T10:16:37.762058+0000","last_fullsized":"2024-09-02T10:16:37.762058+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7692,"log_dups_size":0,"ondisk_log_size":7692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344981504,"num_objects":5264,"num_object_clones":0,"num_object_copies":15792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7692,"num_write_kb":414592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":411,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.032409+0000","last_change":"2024-09-02T10:14:36.623992+0000","last_active":"2024-09-02T10:16:35.032409+0000","last_peered":"2024-09-02T10:16:35.032409+0000","last_clean":"2024-09-02T10:16:35.032409+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:16:35.032409+0000","last_undegraded":"2024-09-02T10:16:35.032409+0000","last_fullsized":"2024-09-02T10:16:35.032409+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.623930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.623930+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"488'7582","reported_seq":7763,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:37.030095+0000","last_change":"2024-09-02T10:15:49.363717+0000","last_active":"2024-09-02T10:16:37.030095+0000","last_peered":"2024-09-02T10:16:37.030095+0000","last_clean":"2024-09-02T10:16:37.030095+0000","last_became_active":"2024-09-02T10:14:39.037882+0000","last_became_peered":"2024-09-02T10:14:39.037882+0000","last_unstale":"2024-09-02T10:16:37.030095+0000","last_undegraded":"2024-09-02T10:16:37.030095+0000","last_fullsized":"2024-09-02T10:16:37.030095+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4720","last_scrub_stamp":"2024-09-02T10:15:49.363588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.363588+0000","objects_scrubbed":4682,"log_size":7582,"log_dups_size":0,"ondisk_log_size":7582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:06.930040+0000","scrub_duration":1106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346554368,"num_objects":5288,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7582,"num_write_kb":411840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":665,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.030888+0000","last_change":"2024-09-02T10:16:23.198280+0000","last_active":"2024-09-02T10:16:35.030888+0000","last_peered":"2024-09-02T10:16:35.030888+0000","last_clean":"2024-09-02T10:16:35.030888+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:16:35.030888+0000","last_undegraded":"2024-09-02T10:16:35.030888+0000","last_fullsized":"2024-09-02T10:16:35.030888+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:23.198227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:23.198227+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:52.413888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"488'7644","reported_seq":7840,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:37.038850+0000","last_change":"2024-09-02T10:16:17.087529+0000","last_active":"2024-09-02T10:16:37.038850+0000","last_peered":"2024-09-02T10:16:37.038850+0000","last_clean":"2024-09-02T10:16:37.038850+0000","last_became_active":"2024-09-02T10:14:39.641932+0000","last_became_peered":"2024-09-02T10:14:39.641932+0000","last_unstale":"2024-09-02T10:16:37.038850+0000","last_undegraded":"2024-09-02T10:16:37.038850+0000","last_fullsized":"2024-09-02T10:16:37.038850+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6467","last_scrub_stamp":"2024-09-02T10:16:17.087427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.087427+0000","objects_scrubbed":6467,"log_size":7644,"log_dups_size":0,"ondisk_log_size":7644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:47.064105+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346685440,"num_objects":5290,"num_object_clones":0,"num_object_copies":15870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7644,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":921,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.025023+0000","last_change":"2024-09-02T10:16:17.754936+0000","last_active":"2024-09-02T10:16:30.025023+0000","last_peered":"2024-09-02T10:16:30.025023+0000","last_clean":"2024-09-02T10:16:30.025023+0000","last_became_active":"2024-09-02T10:15:29.168692+0000","last_became_peered":"2024-09-02T10:15:29.168692+0000","last_unstale":"2024-09-02T10:16:30.025023+0000","last_undegraded":"2024-09-02T10:16:30.025023+0000","last_fullsized":"2024-09-02T10:16:30.025023+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:17.754866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.754866+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:39.447349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"488'7759","reported_seq":7884,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:36.535974+0000","last_change":"2024-09-02T10:14:39.038104+0000","last_active":"2024-09-02T10:16:36.535974+0000","last_peered":"2024-09-02T10:16:36.535974+0000","last_clean":"2024-09-02T10:16:36.535974+0000","last_became_active":"2024-09-02T10:14:39.037755+0000","last_became_peered":"2024-09-02T10:14:39.037755+0000","last_unstale":"2024-09-02T10:16:36.535974+0000","last_undegraded":"2024-09-02T10:16:36.535974+0000","last_fullsized":"2024-09-02T10:16:36.535974+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7759,"log_dups_size":0,"ondisk_log_size":7759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355532800,"num_objects":5425,"num_object_clones":0,"num_object_copies":16275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5425,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7759,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":675,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.030556+0000","last_change":"2024-09-02T10:15:35.202850+0000","last_active":"2024-09-02T10:16:35.030556+0000","last_peered":"2024-09-02T10:16:35.030556+0000","last_clean":"2024-09-02T10:16:35.030556+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:16:35.030556+0000","last_undegraded":"2024-09-02T10:16:35.030556+0000","last_fullsized":"2024-09-02T10:16:35.030556+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:48.187420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:48.187420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:39.213341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"488'7640","reported_seq":7836,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:37.929619+0000","last_change":"2024-09-02T10:16:20.568272+0000","last_active":"2024-09-02T10:16:37.929619+0000","last_peered":"2024-09-02T10:16:37.929619+0000","last_clean":"2024-09-02T10:16:37.929619+0000","last_became_active":"2024-09-02T10:14:39.039060+0000","last_became_peered":"2024-09-02T10:14:39.039060+0000","last_unstale":"2024-09-02T10:16:37.929619+0000","last_undegraded":"2024-09-02T10:16:37.929619+0000","last_fullsized":"2024-09-02T10:16:37.929619+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6670","last_scrub_stamp":"2024-09-02T10:16:20.568081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:20.568081+0000","objects_scrubbed":6316,"log_size":7640,"log_dups_size":0,"ondisk_log_size":7640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:27.515779+0000","scrub_duration":1605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346554368,"num_objects":5288,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7640,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":924,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.032818+0000","last_change":"2024-09-02T10:16:17.179894+0000","last_active":"2024-09-02T10:16:35.032818+0000","last_peered":"2024-09-02T10:16:35.032818+0000","last_clean":"2024-09-02T10:16:35.032818+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:16:35.032818+0000","last_undegraded":"2024-09-02T10:16:35.032818+0000","last_fullsized":"2024-09-02T10:16:35.032818+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:17.179849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.179849+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:24.924645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"487'7497","reported_seq":7691,"reported_epoch":487,"state":"active+clean","last_fresh":"2024-09-02T10:16:34.313487+0000","last_change":"2024-09-02T10:16:21.735648+0000","last_active":"2024-09-02T10:16:34.313487+0000","last_peered":"2024-09-02T10:16:34.313487+0000","last_clean":"2024-09-02T10:16:34.313487+0000","last_became_active":"2024-09-02T10:14:39.036453+0000","last_became_peered":"2024-09-02T10:14:39.036453+0000","last_unstale":"2024-09-02T10:16:34.313487+0000","last_undegraded":"2024-09-02T10:16:34.313487+0000","last_fullsized":"2024-09-02T10:16:34.313487+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6768","last_scrub_stamp":"2024-09-02T10:16:21.735508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:21.735508+0000","objects_scrubbed":6243,"log_size":7497,"log_dups_size":0,"ondisk_log_size":7497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:26.135470+0000","scrub_duration":1736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358023168,"num_objects":5463,"num_object_clones":0,"num_object_copies":16389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7497,"num_write_kb":414720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":458,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.031612+0000","last_change":"2024-09-02T10:16:21.146441+0000","last_active":"2024-09-02T10:16:35.031612+0000","last_peered":"2024-09-02T10:16:35.031612+0000","last_clean":"2024-09-02T10:16:35.031612+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:16:35.031612+0000","last_undegraded":"2024-09-02T10:16:35.031612+0000","last_fullsized":"2024-09-02T10:16:35.031612+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:21.146366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:21.146366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:23.276250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"488'7687","reported_seq":7832,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:36.127650+0000","last_change":"2024-09-02T10:15:49.395716+0000","last_active":"2024-09-02T10:16:36.127650+0000","last_peered":"2024-09-02T10:16:36.127650+0000","last_clean":"2024-09-02T10:16:36.127650+0000","last_became_active":"2024-09-02T10:14:39.037567+0000","last_became_peered":"2024-09-02T10:14:39.037567+0000","last_unstale":"2024-09-02T10:16:36.127650+0000","last_undegraded":"2024-09-02T10:16:36.127650+0000","last_fullsized":"2024-09-02T10:16:36.127650+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":1699,"log_size":7687,"log_dups_size":0,"ondisk_log_size":7687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:40.257445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350289920,"num_objects":5345,"num_object_clones":0,"num_object_copies":16035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7687,"num_write_kb":417024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":696,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-09-02T10:16:35.033861+0000","last_change":"2024-09-02T10:15:31.899734+0000","last_active":"2024-09-02T10:16:35.033861+0000","last_peered":"2024-09-02T10:16:35.033861+0000","last_clean":"2024-09-02T10:16:35.033861+0000","last_became_active":"2024-09-02T10:15:29.167553+0000","last_became_peered":"2024-09-02T10:15:29.167553+0000","last_unstale":"2024-09-02T10:16:35.033861+0000","last_undegraded":"2024-09-02T10:16:35.033861+0000","last_fullsized":"2024-09-02T10:16:35.033861+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:31.899662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:31.899662+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.920777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"488'7727","reported_seq":15231,"reported_epoch":488,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:16:37.761673+0000","last_change":"2024-09-02T10:15:36.347809+0000","last_active":"2024-09-02T10:16:37.761673+0000","last_peered":"2024-09-02T10:16:37.761673+0000","last_clean":"2024-09-02T10:15:34.173584+0000","last_became_active":"2024-09-02T10:15:36.297925+0000","last_became_peered":"2024-09-02T10:15:36.297925+0000","last_unstale":"2024-09-02T10:16:37.761673+0000","last_undegraded":"2024-09-02T10:16:37.761673+0000","last_fullsized":"2024-09-02T10:15:36.200774+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7727,"log_dups_size":0,"ondisk_log_size":7727,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342818816,"num_objects":5231,"num_object_clones":0,"num_object_copies":15693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":251,"num_objects_unfound":0,"num_objects_dirty":5231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7727,"num_write_kb":414656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3688,"num_bytes_recovered":241696768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[0,10],"avail_no_missing":["0","9","10"],"object_location_counts":[{"shards":"0,9,10","objects":251},{"shards":"0,10","objects":4980}],"blocked_by":[],"up_primary":11,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":98,"reported_epoch":486,"state":"active+clean","last_fresh":"2024-09-02T10:16:30.026443+0000","last_change":"2024-09-02T10:15:48.739408+0000","last_active":"2024-09-02T10:16:30.026443+0000","last_peered":"2024-09-02T10:16:30.026443+0000","last_clean":"2024-09-02T10:16:30.026443+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:16:30.026443+0000","last_undegraded":"2024-09-02T10:16:30.026443+0000","last_fullsized":"2024-09-02T10:16:30.026443+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:48.739340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:48.739340+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:02.899055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5583994904,"num_objects":85206,"num_object_clones":0,"num_object_copies":255618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":251,"num_objects_unfound":0,"num_objects_dirty":85206,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":122000,"num_write_kb":6630529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5457,"num_bytes_recovered":357629952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16989958144,"data_stored":16989945928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122000,"ondisk_log_size":122000,"up":48,"acting":47,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383878,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2078144,"kb_used_data":1681836,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91667008,"statfs":{"total":95995035648,"available":93867016192,"internally_reserved":0,"allocated":1722200064,"data_stored":1721509839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":405797316},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612811,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2654840,"kb_used_data":2389032,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":91090312,"statfs":{"total":95995035648,"available":93276479488,"internally_reserved":0,"allocated":2446368768,"data_stored":2445682615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":272169408},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841744,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2030648,"kb_used_data":1598248,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":91714504,"statfs":{"total":95995035648,"available":93915652096,"internally_reserved":0,"allocated":1636605952,"data_stored":1635919799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":442759616},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070677,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1326064,"kb_used_data":1056092,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92419088,"statfs":{"total":95995035648,"available":94637146112,"internally_reserved":0,"allocated":1081438208,"data_stored":1080756510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299610,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1778700,"kb_used_data":1372660,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91966452,"statfs":{"total":95995035648,"available":94173646848,"internally_reserved":0,"allocated":1405603840,"data_stored":1404910063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":415758794},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528542,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1988476,"kb_used_data":1700456,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91756676,"statfs":{"total":95995035648,"available":93958836224,"internally_reserved":0,"allocated":1741266944,"data_stored":1740580791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294910407},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724771,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2822152,"kb_used_data":2407796,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":90923000,"statfs":{"total":95995035648,"available":93105152000,"internally_reserved":0,"allocated":2465583104,"data_stored":2464889327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149933,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":753504,"kb_used_data":353040,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92991648,"statfs":{"total":95995035648,"available":95223447552,"internally_reserved":0,"allocated":361512960,"data_stored":360835767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":410057160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411569,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575094,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1920680,"kb_used_data":1524376,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91824472,"statfs":{"total":95995035648,"available":94028259328,"internally_reserved":0,"allocated":1560961024,"data_stored":1560280279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738621,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3175736,"kb_used_data":2721128,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":90569416,"statfs":{"total":95995035648,"available":92743081984,"internally_reserved":0,"allocated":2786435072,"data_stored":2785748919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2763194368,"data_stored":2763194368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1437663232,"data_stored":1437663232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":356909056,"data_stored":356909056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2443776000,"data_stored":2443771928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1738080256,"data_stored":1738080256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1391136768,"data_stored":1391132696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1076101120,"data_stored":1076101120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1631059968,"data_stored":1631059968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2444296192,"data_stored":2444296192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1707741184,"data_stored":1707737112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:16:40.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:16:40.686 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:40 smithi151 ceph-mon[16639]: from='client.26240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:40.686 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:40 smithi151 ceph-mon[16639]: pgmap v1139: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 825 op/s; 251/253956 objects misplaced (0.099%); 4.6 MiB/s, 73 objects/s recovering 2024-09-02T10:16:40.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:40.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:40.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:40.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:40.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:40 smithi187 ceph-mon[20072]: from='client.26240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:40.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:40 smithi187 ceph-mon[20072]: pgmap v1139: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 825 op/s; 251/253956 objects misplaced (0.099%); 4.6 MiB/s, 73 objects/s recovering 2024-09-02T10:16:40.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:40 smithi162 ceph-mon[20279]: from='client.26240 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:40 smithi162 ceph-mon[20279]: pgmap v1139: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.2 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 825 op/s; 251/253956 objects misplaced (0.099%); 4.6 MiB/s, 73 objects/s recovering 2024-09-02T10:16:40.864 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:40.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:40.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:40.966 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:40.990 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:40.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:40.991 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:41.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:41.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:41.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:41.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:41.237 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:41.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:41.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:41.313 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:41.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:41.359 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:41.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:41.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:41.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:41.483 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:41.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:41.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:41.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:41.606 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:41.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:41.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:41.731 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:41.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:41.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:16:41.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:41.856 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:41.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:41.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:41.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:41.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:42.010 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:42.019 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:16:42.042 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:16:42.081 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:42.138 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:42 smithi162 ceph-mon[20279]: osdmap e490: 12 total, 12 up, 10 in 2024-09-02T10:16:42.139 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:42 smithi162 ceph-mon[20279]: pgmap v1141: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 748 op/s; 251/252285 objects misplaced (0.099%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:16:42.139 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3845784570' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:42.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:42.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:42.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:42.281 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:42 smithi151 ceph-mon[16639]: osdmap e490: 12 total, 12 up, 10 in 2024-09-02T10:16:42.281 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:42 smithi151 ceph-mon[16639]: pgmap v1141: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 748 op/s; 251/252285 objects misplaced (0.099%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:16:42.281 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3845784570' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:42.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:42.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:42.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:42.420 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:42.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:42 smithi187 ceph-mon[20072]: osdmap e490: 12 total, 12 up, 10 in 2024-09-02T10:16:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:42 smithi187 ceph-mon[20072]: pgmap v1141: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 5.1 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 748 op/s; 251/252285 objects misplaced (0.099%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:16:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3845784570' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:42.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:42.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:42.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:42.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:42.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:42.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:42.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:42.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:42.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:42.804 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:42.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:42.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:42.817 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:16:42.818 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:16:42.818 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:16:42.818 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:16:42.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:42.929 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:42.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:42.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:43.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:43.057 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:43.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:43 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3845784570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:43 smithi162 ceph-mon[20279]: osdmap e491: 12 total, 12 up, 10 in 2024-09-02T10:16:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1437317517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:43.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:43 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:43.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3845784570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:43.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:43 smithi151 ceph-mon[16639]: osdmap e491: 12 total, 12 up, 10 in 2024-09-02T10:16:43.136 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1437317517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:43.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:43.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:43.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:43.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:43.312 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:43.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:43.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:43.345 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:16:43.345 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1275,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":491,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":10,"osd_in_since":1725272054,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":84095,"data_bytes":5511643704,"bytes_used":20793925632,"bytes_avail":939156430848,"bytes_total":959950356480,"misplaced_objects":251,"misplaced_total":252285,"misplaced_ratio":0.00099490655409556657,"recovering_objects_per_sec":70,"recovering_bytes_per_sec":4635819,"recovering_keys_per_sec":0,"num_objects_recovered":444,"num_bytes_recovered":29097984,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":748},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":33,"modified":"2024-09-02T10:15:39.290520+0000","services":{}},"progress_events":{}} 2024-09-02T10:16:43.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:43.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:43.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:43 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3845784570' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:43.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:43 smithi187 ceph-mon[20072]: osdmap e491: 12 total, 12 up, 10 in 2024-09-02T10:16:43.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1437317517' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:43.542 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:43.567 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:16:43.567 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:16:43.569 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:43.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:43.591 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:16:43.591 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:16:43.618 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:16:43.618 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:16:43.641 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:16:43.641 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:16:43.666 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:16:43.666 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:16:43.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:43.695 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:16:43.696 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:16:43.699 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:43.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:43.722 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:16:43.722 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:16:43.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:16:43.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:43.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:43.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:43.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:43.849 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:16:43.849 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:16:43.871 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:16:43.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:16:43.896 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:16:43.897 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:16:43.920 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:16:43.921 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:16:43.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:43.944 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:16:43.944 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:16:43.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:43.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:44.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:44.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:44.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:44.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:44.129 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:16:44.176 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:44 smithi151 ceph-mon[16639]: pgmap v1143: 47 pgs: 47 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 883 op/s; 7.2 MiB/s, 115 objects/s recovering 2024-09-02T10:16:44.176 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/176061281' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:16:44.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:44.199 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:44.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:44.205 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:16:44.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:16:44.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:44.326 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:44.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:44.327 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:44 smithi162 ceph-mon[20279]: pgmap v1143: 47 pgs: 47 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 883 op/s; 7.2 MiB/s, 115 objects/s recovering 2024-09-02T10:16:44.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/176061281' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:16:44.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:44.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:44.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:44.505 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:44 smithi187 ceph-mon[20072]: pgmap v1143: 47 pgs: 47 active+clean; 5.0 GiB data, 19 GiB used, 875 GiB / 894 GiB avail; 0 B/s wr, 883 op/s; 7.2 MiB/s, 115 objects/s recovering 2024-09-02T10:16:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/176061281' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:16:44.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:44.583 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:44.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:44.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:44.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:44.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:44.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:44.836 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:44.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:44.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:44.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:44.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:45.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:45.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:45.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:45.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:45.210 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:45.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:45.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:45.211 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:45 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-02T10:16:45.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:45.326 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:45 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-02T10:16:45.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:45.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:45.338 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:45.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:45.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:45.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:45 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-02T10:16:45.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:16:45.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:45.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:45.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:45.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:45.662 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:16:45.662 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:16:45.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:45.697 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:16:45.698 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:16:45.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:45.727 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:45.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:45.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:45.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:45.955 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:45.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:45.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:46.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:46.114 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:46.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:46.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:46.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:46.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:46.330 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:46 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:16:46.330 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:46 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:46.330 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:46 smithi162 ceph-mon[20279]: osdmap e492: 12 total, 12 up, 10 in 2024-09-02T10:16:46.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:46 smithi162 ceph-mon[20279]: pgmap v1145: 47 pgs: 47 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 692 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-09-02T10:16:46.331 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:46 smithi162 ceph-mon[20279]: from='client.16692 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:46.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:46.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:16:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:46.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:46.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:46 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:16:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:46 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:46 smithi151 ceph-mon[16639]: osdmap e492: 12 total, 12 up, 10 in 2024-09-02T10:16:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:46 smithi151 ceph-mon[16639]: pgmap v1145: 47 pgs: 47 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 692 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-09-02T10:16:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:46 smithi151 ceph-mon[16639]: from='client.16692 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:46.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:46 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:16:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:46 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:46 smithi187 ceph-mon[20072]: osdmap e492: 12 total, 12 up, 10 in 2024-09-02T10:16:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:46 smithi187 ceph-mon[20072]: pgmap v1145: 47 pgs: 47 active+clean; 4.9 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 692 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-09-02T10:16:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:46 smithi187 ceph-mon[20072]: from='client.16692 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:46.514 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:46.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:46.587 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1145,"stamp":"2024-09-02T10:16:45.304686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5275976248,"num_objects":80499,"num_object_clones":0,"num_object_copies":241497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80499,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126709,"num_write_kb":6630529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5711,"num_bytes_recovered":374809664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129561,"ondisk_log_size":129561,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":19317428,"kb_used_data":15600508,"kb_used_omap":15,"kb_used_meta":3715760,"kb_avail":918134092,"statfs":{"total":959950356480,"available":940169310208,"internally_reserved":0,"allocated":15974920192,"data_stored":15968841367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3804938673},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,7,2,0,1,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-272171008,"num_objects":-4153,"num_object_clones":0,"num_object_copies":-12459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-251,"num_objects_unfound":0,"num_objects_dirty":-4153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4153,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":16449536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001190"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":309,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.027243+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:16:42.027243+0000","last_peered":"2024-09-02T10:16:42.027243+0000","last_clean":"2024-09-02T10:16:42.027243+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:16:42.027243+0000","last_undegraded":"2024-09-02T10:16:42.027243+0000","last_fullsized":"2024-09-02T10:16:42.027243+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":117,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.115360+0000","last_change":"2024-09-02T10:15:41.899476+0000","last_active":"2024-09-02T10:16:42.115360+0000","last_peered":"2024-09-02T10:16:42.115360+0000","last_clean":"2024-09-02T10:16:42.115360+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:16:42.115360+0000","last_undegraded":"2024-09-02T10:16:42.115360+0000","last_fullsized":"2024-09-02T10:16:42.115360+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:41.899397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:41.899397+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:51.002146+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":567,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.021038+0000","last_change":"2024-09-02T10:15:12.091472+0000","last_active":"2024-09-02T10:16:42.021038+0000","last_peered":"2024-09-02T10:16:42.021038+0000","last_clean":"2024-09-02T10:16:42.021038+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:16:42.021038+0000","last_undegraded":"2024-09-02T10:16:42.021038+0000","last_fullsized":"2024-09-02T10:16:42.021038+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.091410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.091410+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:43.009445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":567,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.020872+0000","last_change":"2024-09-02T10:15:19.065187+0000","last_active":"2024-09-02T10:16:42.020872+0000","last_peered":"2024-09-02T10:16:42.020872+0000","last_clean":"2024-09-02T10:16:42.020872+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:16:42.020872+0000","last_undegraded":"2024-09-02T10:16:42.020872+0000","last_fullsized":"2024-09-02T10:16:42.020872+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:19.065134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:19.065134+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:45.985760+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":486,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.026733+0000","last_change":"2024-09-02T10:15:35.202261+0000","last_active":"2024-09-02T10:16:41.026733+0000","last_peered":"2024-09-02T10:16:41.026733+0000","last_clean":"2024-09-02T10:16:41.026733+0000","last_became_active":"2024-09-02T10:15:35.201887+0000","last_became_peered":"2024-09-02T10:15:35.201887+0000","last_unstale":"2024-09-02T10:16:41.026733+0000","last_undegraded":"2024-09-02T10:16:41.026733+0000","last_fullsized":"2024-09-02T10:16:41.026733+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:04.994667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:04.994667+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:42.263661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":414,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.020964+0000","last_change":"2024-09-02T10:14:11.689851+0000","last_active":"2024-09-02T10:16:41.020964+0000","last_peered":"2024-09-02T10:16:41.020964+0000","last_clean":"2024-09-02T10:16:41.020964+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:16:41.020964+0000","last_undegraded":"2024-09-02T10:16:41.020964+0000","last_fullsized":"2024-09-02T10:16:41.020964+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.689767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":940,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.020976+0000","last_change":"2024-09-02T10:16:23.306394+0000","last_active":"2024-09-02T10:16:42.020976+0000","last_peered":"2024-09-02T10:16:42.020976+0000","last_clean":"2024-09-02T10:16:42.020976+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:16:42.020976+0000","last_undegraded":"2024-09-02T10:16:42.020976+0000","last_fullsized":"2024-09-02T10:16:42.020976+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:23.306331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:23.306331+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:34.300255+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":582,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.021023+0000","last_change":"2024-09-02T10:16:10.994908+0000","last_active":"2024-09-02T10:16:42.021023+0000","last_peered":"2024-09-02T10:16:42.021023+0000","last_clean":"2024-09-02T10:16:42.021023+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:16:42.021023+0000","last_undegraded":"2024-09-02T10:16:42.021023+0000","last_fullsized":"2024-09-02T10:16:42.021023+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.994824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.994824+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:17.261476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":581,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.115396+0000","last_change":"2024-09-02T10:15:34.791760+0000","last_active":"2024-09-02T10:16:42.115396+0000","last_peered":"2024-09-02T10:16:42.115396+0000","last_clean":"2024-09-02T10:16:42.115396+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:16:42.115396+0000","last_undegraded":"2024-09-02T10:16:42.115396+0000","last_fullsized":"2024-09-02T10:16:42.115396+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:34.791708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:34.791708+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.857252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":556,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.026870+0000","last_change":"2024-09-02T10:16:14.103234+0000","last_active":"2024-09-02T10:16:41.026870+0000","last_peered":"2024-09-02T10:16:41.026870+0000","last_clean":"2024-09-02T10:16:41.026870+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:16:41.026870+0000","last_undegraded":"2024-09-02T10:16:41.026870+0000","last_fullsized":"2024-09-02T10:16:41.026870+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:14.103166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:14.103166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:33.443226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":407,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.026798+0000","last_change":"2024-09-02T10:16:11.152819+0000","last_active":"2024-09-02T10:16:41.026798+0000","last_peered":"2024-09-02T10:16:41.026798+0000","last_clean":"2024-09-02T10:16:41.026798+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:16:41.026798+0000","last_undegraded":"2024-09-02T10:16:41.026798+0000","last_fullsized":"2024-09-02T10:16:41.026798+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:11.152686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:11.152686+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:26.925270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":915,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.115244+0000","last_change":"2024-09-02T10:16:19.673642+0000","last_active":"2024-09-02T10:16:42.115244+0000","last_peered":"2024-09-02T10:16:42.115244+0000","last_clean":"2024-09-02T10:16:42.115244+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:16:42.115244+0000","last_undegraded":"2024-09-02T10:16:42.115244+0000","last_fullsized":"2024-09-02T10:16:42.115244+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:19.673572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:19.673572+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:33.436612+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":849,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.020997+0000","last_change":"2024-09-02T10:16:15.378306+0000","last_active":"2024-09-02T10:16:42.020997+0000","last_peered":"2024-09-02T10:16:42.020997+0000","last_clean":"2024-09-02T10:16:42.020997+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:16:42.020997+0000","last_undegraded":"2024-09-02T10:16:42.020997+0000","last_fullsized":"2024-09-02T10:16:42.020997+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:15.378230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:15.378230+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:18.040488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":828,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.028798+0000","last_change":"2024-09-02T10:15:29.170702+0000","last_active":"2024-09-02T10:16:42.028798+0000","last_peered":"2024-09-02T10:16:42.028798+0000","last_clean":"2024-09-02T10:16:42.028798+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:16:42.028798+0000","last_undegraded":"2024-09-02T10:16:42.028798+0000","last_fullsized":"2024-09-02T10:16:42.028798+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.665438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.665438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":819,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.020986+0000","last_change":"2024-09-02T10:14:34.217697+0000","last_active":"2024-09-02T10:16:42.020986+0000","last_peered":"2024-09-02T10:16:42.020986+0000","last_clean":"2024-09-02T10:16:42.020986+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:16:42.020986+0000","last_undegraded":"2024-09-02T10:16:42.020986+0000","last_fullsized":"2024-09-02T10:16:42.020986+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.217624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.217624+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:42.227584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"491'7849","reported_seq":11604,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:43.597563+0000","last_change":"2024-09-02T10:16:17.226985+0000","last_active":"2024-09-02T10:16:43.597563+0000","last_peered":"2024-09-02T10:16:43.597563+0000","last_clean":"2024-09-02T10:16:43.597563+0000","last_became_active":"2024-09-02T10:15:24.141368+0000","last_became_peered":"2024-09-02T10:15:24.141368+0000","last_unstale":"2024-09-02T10:16:43.597563+0000","last_undegraded":"2024-09-02T10:16:43.597563+0000","last_fullsized":"2024-09-02T10:16:43.597563+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6399","last_scrub_stamp":"2024-09-02T10:16:17.226765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.226765+0000","objects_scrubbed":6342,"log_size":7849,"log_dups_size":0,"ondisk_log_size":7849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:37.316444+0000","scrub_duration":2531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323158016,"num_objects":4931,"num_object_clones":0,"num_object_copies":14793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":408960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":115933184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":906,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.021152+0000","last_change":"2024-09-02T10:14:41.306927+0000","last_active":"2024-09-02T10:16:42.021152+0000","last_peered":"2024-09-02T10:16:42.021152+0000","last_clean":"2024-09-02T10:16:42.021152+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:16:42.021152+0000","last_undegraded":"2024-09-02T10:16:42.021152+0000","last_fullsized":"2024-09-02T10:16:42.021152+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:41.306868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:41.306868+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"490'7922","reported_seq":8052,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.027229+0000","last_change":"2024-09-02T10:14:39.040351+0000","last_active":"2024-09-02T10:16:42.027229+0000","last_peered":"2024-09-02T10:16:42.027229+0000","last_clean":"2024-09-02T10:16:42.027229+0000","last_became_active":"2024-09-02T10:14:39.040025+0000","last_became_peered":"2024-09-02T10:14:39.040025+0000","last_unstale":"2024-09-02T10:16:42.027229+0000","last_undegraded":"2024-09-02T10:16:42.027229+0000","last_fullsized":"2024-09-02T10:16:42.027229+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7922,"log_dups_size":0,"ondisk_log_size":7922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323944472,"num_objects":4944,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4944,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7922,"num_write_kb":411649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1027,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.027105+0000","last_change":"2024-09-02T10:16:22.269544+0000","last_active":"2024-09-02T10:16:41.027105+0000","last_peered":"2024-09-02T10:16:41.027105+0000","last_clean":"2024-09-02T10:16:41.027105+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:16:41.027105+0000","last_undegraded":"2024-09-02T10:16:41.027105+0000","last_fullsized":"2024-09-02T10:16:41.027105+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:22.269495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:22.269495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:47.553829+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"491'7869","reported_seq":8073,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.909528+0000","last_change":"2024-09-02T10:16:15.009837+0000","last_active":"2024-09-02T10:16:42.909528+0000","last_peered":"2024-09-02T10:16:42.909528+0000","last_clean":"2024-09-02T10:16:42.909528+0000","last_became_active":"2024-09-02T10:14:39.038164+0000","last_became_peered":"2024-09-02T10:14:39.038164+0000","last_unstale":"2024-09-02T10:16:42.909528+0000","last_undegraded":"2024-09-02T10:16:42.909528+0000","last_fullsized":"2024-09-02T10:16:42.909528+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6340","last_scrub_stamp":"2024-09-02T10:16:15.009487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:15.009487+0000","objects_scrubbed":6273,"log_size":7869,"log_dups_size":0,"ondisk_log_size":7869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:35.488395+0000","scrub_duration":2145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324206592,"num_objects":4947,"num_object_clones":0,"num_object_copies":14841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4947,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7869,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":672,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.046502+0000","last_change":"2024-09-02T10:16:10.884551+0000","last_active":"2024-09-02T10:16:42.046502+0000","last_peered":"2024-09-02T10:16:42.046502+0000","last_clean":"2024-09-02T10:16:42.046502+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:16:42.046502+0000","last_undegraded":"2024-09-02T10:16:42.046502+0000","last_fullsized":"2024-09-02T10:16:42.046502+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.884481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.884481+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:22.734487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"491'8011","reported_seq":8212,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.032267+0000","last_change":"2024-09-02T10:16:26.551606+0000","last_active":"2024-09-02T10:16:42.032267+0000","last_peered":"2024-09-02T10:16:42.032267+0000","last_clean":"2024-09-02T10:16:42.032267+0000","last_became_active":"2024-09-02T10:14:39.037121+0000","last_became_peered":"2024-09-02T10:14:39.037121+0000","last_unstale":"2024-09-02T10:16:42.032267+0000","last_undegraded":"2024-09-02T10:16:42.032267+0000","last_fullsized":"2024-09-02T10:16:42.032267+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'7110","last_scrub_stamp":"2024-09-02T10:16:26.551442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:26.551442+0000","objects_scrubbed":6016,"log_size":8011,"log_dups_size":0,"ondisk_log_size":8011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:27.466417+0000","scrub_duration":1258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333774848,"num_objects":5093,"num_object_clones":0,"num_object_copies":15279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8011,"num_write_kb":419328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":68,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.027080+0000","last_change":"2024-09-02T10:15:35.208246+0000","last_active":"2024-09-02T10:16:42.027080+0000","last_peered":"2024-09-02T10:16:42.027080+0000","last_clean":"2024-09-02T10:16:42.027080+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:16:42.027080+0000","last_undegraded":"2024-09-02T10:16:42.027080+0000","last_fullsized":"2024-09-02T10:16:42.027080+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.668602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.668602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"490'7782","reported_seq":7912,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.660926+0000","last_change":"2024-09-02T10:14:39.642395+0000","last_active":"2024-09-02T10:16:41.660926+0000","last_peered":"2024-09-02T10:16:41.660926+0000","last_clean":"2024-09-02T10:16:41.660926+0000","last_became_active":"2024-09-02T10:14:39.641995+0000","last_became_peered":"2024-09-02T10:14:39.641995+0000","last_unstale":"2024-09-02T10:16:41.660926+0000","last_undegraded":"2024-09-02T10:16:41.660926+0000","last_fullsized":"2024-09-02T10:16:41.660926+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7782,"log_dups_size":0,"ondisk_log_size":7782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335020032,"num_objects":5112,"num_object_clones":0,"num_object_copies":15336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7782,"num_write_kb":412608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":977,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.027104+0000","last_change":"2024-09-02T10:15:29.195341+0000","last_active":"2024-09-02T10:16:42.027104+0000","last_peered":"2024-09-02T10:16:42.027104+0000","last_clean":"2024-09-02T10:16:42.027104+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:16:42.027104+0000","last_undegraded":"2024-09-02T10:16:42.027104+0000","last_fullsized":"2024-09-02T10:16:42.027104+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.277119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.277119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"491'7957","reported_seq":8159,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:43.909360+0000","last_change":"2024-09-02T10:16:22.056000+0000","last_active":"2024-09-02T10:16:43.909360+0000","last_peered":"2024-09-02T10:16:43.909360+0000","last_clean":"2024-09-02T10:16:43.909360+0000","last_became_active":"2024-09-02T10:14:39.034636+0000","last_became_peered":"2024-09-02T10:14:39.034636+0000","last_unstale":"2024-09-02T10:16:43.909360+0000","last_undegraded":"2024-09-02T10:16:43.909360+0000","last_fullsized":"2024-09-02T10:16:43.909360+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6741","last_scrub_stamp":"2024-09-02T10:16:22.055909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:22.055909+0000","objects_scrubbed":6225,"log_size":7957,"log_dups_size":0,"ondisk_log_size":7957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:24.765454+0000","scrub_duration":2093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323682304,"num_objects":4939,"num_object_clones":0,"num_object_copies":14817,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4939,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":443,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.020992+0000","last_change":"2024-09-02T10:15:35.205684+0000","last_active":"2024-09-02T10:16:41.020992+0000","last_peered":"2024-09-02T10:16:41.020992+0000","last_clean":"2024-09-02T10:16:41.020992+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:16:41.020992+0000","last_undegraded":"2024-09-02T10:16:41.020992+0000","last_fullsized":"2024-09-02T10:16:41.020992+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.701773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.701773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"490'7765","reported_seq":7951,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.098664+0000","last_change":"2024-09-02T10:15:49.158612+0000","last_active":"2024-09-02T10:16:42.098664+0000","last_peered":"2024-09-02T10:16:42.098664+0000","last_clean":"2024-09-02T10:16:42.098664+0000","last_became_active":"2024-09-02T10:14:39.037822+0000","last_became_peered":"2024-09-02T10:14:39.037822+0000","last_unstale":"2024-09-02T10:16:42.098664+0000","last_undegraded":"2024-09-02T10:16:42.098664+0000","last_fullsized":"2024-09-02T10:16:42.098664+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4732","last_scrub_stamp":"2024-09-02T10:15:49.158487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.158487+0000","objects_scrubbed":4682,"log_size":7765,"log_dups_size":0,"ondisk_log_size":7765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:57.249982+0000","scrub_duration":968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326828032,"num_objects":4987,"num_object_clones":0,"num_object_copies":14961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4987,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7765,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":415,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.022347+0000","last_change":"2024-09-02T10:14:09.768298+0000","last_active":"2024-09-02T10:16:41.022347+0000","last_peered":"2024-09-02T10:16:41.022347+0000","last_clean":"2024-09-02T10:16:41.022347+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:16:41.022347+0000","last_undegraded":"2024-09-02T10:16:41.022347+0000","last_fullsized":"2024-09-02T10:16:41.022347+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:09.768061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:09.768061+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"491'8090","reported_seq":8221,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.746467+0000","last_change":"2024-09-02T10:14:39.035303+0000","last_active":"2024-09-02T10:16:42.746467+0000","last_peered":"2024-09-02T10:16:42.746467+0000","last_clean":"2024-09-02T10:16:42.746467+0000","last_became_active":"2024-09-02T10:14:39.035100+0000","last_became_peered":"2024-09-02T10:14:39.035100+0000","last_unstale":"2024-09-02T10:16:42.746467+0000","last_undegraded":"2024-09-02T10:16:42.746467+0000","last_fullsized":"2024-09-02T10:16:42.746467+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":8090,"log_dups_size":0,"ondisk_log_size":8090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339869696,"num_objects":5186,"num_object_clones":0,"num_object_copies":15558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8090,"num_write_kb":424832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":168,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.098745+0000","last_change":"2024-09-02T10:15:36.299961+0000","last_active":"2024-09-02T10:16:42.098745+0000","last_peered":"2024-09-02T10:16:42.098745+0000","last_clean":"2024-09-02T10:16:42.098745+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:16:42.098745+0000","last_undegraded":"2024-09-02T10:16:42.098745+0000","last_fullsized":"2024-09-02T10:16:42.098745+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:36.299914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:36.299914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:01.870130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"491'7965","reported_seq":8094,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.779013+0000","last_change":"2024-09-02T10:14:39.641746+0000","last_active":"2024-09-02T10:16:42.779013+0000","last_peered":"2024-09-02T10:16:42.779013+0000","last_clean":"2024-09-02T10:16:42.779013+0000","last_became_active":"2024-09-02T10:14:39.641478+0000","last_became_peered":"2024-09-02T10:14:39.641478+0000","last_unstale":"2024-09-02T10:16:42.779013+0000","last_undegraded":"2024-09-02T10:16:42.779013+0000","last_fullsized":"2024-09-02T10:16:42.779013+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7965,"log_dups_size":0,"ondisk_log_size":7965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327090176,"num_objects":4991,"num_object_clones":0,"num_object_copies":14973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7965,"num_write_kb":414592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":416,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.021275+0000","last_change":"2024-09-02T10:14:36.623992+0000","last_active":"2024-09-02T10:16:41.021275+0000","last_peered":"2024-09-02T10:16:41.021275+0000","last_clean":"2024-09-02T10:16:41.021275+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:16:41.021275+0000","last_undegraded":"2024-09-02T10:16:41.021275+0000","last_fullsized":"2024-09-02T10:16:41.021275+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.623930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.623930+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"491'7818","reported_seq":8004,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.021138+0000","last_change":"2024-09-02T10:15:49.363717+0000","last_active":"2024-09-02T10:16:42.021138+0000","last_peered":"2024-09-02T10:16:42.021138+0000","last_clean":"2024-09-02T10:16:42.021138+0000","last_became_active":"2024-09-02T10:14:39.037882+0000","last_became_peered":"2024-09-02T10:14:39.037882+0000","last_unstale":"2024-09-02T10:16:42.021138+0000","last_undegraded":"2024-09-02T10:16:42.021138+0000","last_fullsized":"2024-09-02T10:16:42.021138+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4720","last_scrub_stamp":"2024-09-02T10:15:49.363588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.363588+0000","objects_scrubbed":4682,"log_size":7818,"log_dups_size":0,"ondisk_log_size":7818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:06.930040+0000","scrub_duration":1106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331087872,"num_objects":5052,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7818,"num_write_kb":411840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":669,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.026791+0000","last_change":"2024-09-02T10:16:23.198280+0000","last_active":"2024-09-02T10:16:41.026791+0000","last_peered":"2024-09-02T10:16:41.026791+0000","last_clean":"2024-09-02T10:16:41.026791+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:16:41.026791+0000","last_undegraded":"2024-09-02T10:16:41.026791+0000","last_fullsized":"2024-09-02T10:16:41.026791+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:23.198227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:23.198227+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:52.413888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"490'7877","reported_seq":8081,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.021306+0000","last_change":"2024-09-02T10:16:17.087529+0000","last_active":"2024-09-02T10:16:42.021306+0000","last_peered":"2024-09-02T10:16:42.021306+0000","last_clean":"2024-09-02T10:16:42.021306+0000","last_became_active":"2024-09-02T10:14:39.641932+0000","last_became_peered":"2024-09-02T10:14:39.641932+0000","last_unstale":"2024-09-02T10:16:42.021306+0000","last_undegraded":"2024-09-02T10:16:42.021306+0000","last_fullsized":"2024-09-02T10:16:42.021306+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6467","last_scrub_stamp":"2024-09-02T10:16:17.087427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.087427+0000","objects_scrubbed":6467,"log_size":7877,"log_dups_size":0,"ondisk_log_size":7877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:47.064105+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331415552,"num_objects":5057,"num_object_clones":0,"num_object_copies":15171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7877,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":931,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.115302+0000","last_change":"2024-09-02T10:16:17.754936+0000","last_active":"2024-09-02T10:16:42.115302+0000","last_peered":"2024-09-02T10:16:42.115302+0000","last_clean":"2024-09-02T10:16:42.115302+0000","last_became_active":"2024-09-02T10:15:29.168692+0000","last_became_peered":"2024-09-02T10:15:29.168692+0000","last_unstale":"2024-09-02T10:16:42.115302+0000","last_undegraded":"2024-09-02T10:16:42.115302+0000","last_fullsized":"2024-09-02T10:16:42.115302+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:17.754866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.754866+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:39.447349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"490'8014","reported_seq":8143,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.647604+0000","last_change":"2024-09-02T10:14:39.038104+0000","last_active":"2024-09-02T10:16:41.647604+0000","last_peered":"2024-09-02T10:16:41.647604+0000","last_clean":"2024-09-02T10:16:41.647604+0000","last_became_active":"2024-09-02T10:14:39.037755+0000","last_became_peered":"2024-09-02T10:14:39.037755+0000","last_unstale":"2024-09-02T10:16:41.647604+0000","last_undegraded":"2024-09-02T10:16:41.647604+0000","last_fullsized":"2024-09-02T10:16:41.647604+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":8014,"log_dups_size":0,"ondisk_log_size":8014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338821120,"num_objects":5170,"num_object_clones":0,"num_object_copies":15510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8014,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":679,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.026684+0000","last_change":"2024-09-02T10:15:35.202850+0000","last_active":"2024-09-02T10:16:41.026684+0000","last_peered":"2024-09-02T10:16:41.026684+0000","last_clean":"2024-09-02T10:16:41.026684+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:16:41.026684+0000","last_undegraded":"2024-09-02T10:16:41.026684+0000","last_fullsized":"2024-09-02T10:16:41.026684+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:48.187420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:48.187420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:44.317721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"491'7873","reported_seq":8074,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.940247+0000","last_change":"2024-09-02T10:16:20.568272+0000","last_active":"2024-09-02T10:16:42.940247+0000","last_peered":"2024-09-02T10:16:42.940247+0000","last_clean":"2024-09-02T10:16:42.940247+0000","last_became_active":"2024-09-02T10:14:39.039060+0000","last_became_peered":"2024-09-02T10:14:39.039060+0000","last_unstale":"2024-09-02T10:16:42.940247+0000","last_undegraded":"2024-09-02T10:16:42.940247+0000","last_fullsized":"2024-09-02T10:16:42.940247+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6670","last_scrub_stamp":"2024-09-02T10:16:20.568081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:20.568081+0000","objects_scrubbed":6316,"log_size":7873,"log_dups_size":0,"ondisk_log_size":7873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:27.515779+0000","scrub_duration":1605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331284480,"num_objects":5055,"num_object_clones":0,"num_object_copies":15165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7873,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":930,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.098862+0000","last_change":"2024-09-02T10:16:17.179894+0000","last_active":"2024-09-02T10:16:42.098862+0000","last_peered":"2024-09-02T10:16:42.098862+0000","last_clean":"2024-09-02T10:16:42.098862+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:16:42.098862+0000","last_undegraded":"2024-09-02T10:16:42.098862+0000","last_fullsized":"2024-09-02T10:16:42.098862+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:17.179849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.179849+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:24.924645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"491'8045","reported_seq":8247,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:44.334924+0000","last_change":"2024-09-02T10:16:21.735648+0000","last_active":"2024-09-02T10:16:44.334924+0000","last_peered":"2024-09-02T10:16:44.334924+0000","last_clean":"2024-09-02T10:16:44.334924+0000","last_became_active":"2024-09-02T10:14:39.036453+0000","last_became_peered":"2024-09-02T10:14:39.036453+0000","last_unstale":"2024-09-02T10:16:44.334924+0000","last_undegraded":"2024-09-02T10:16:44.334924+0000","last_fullsized":"2024-09-02T10:16:44.334924+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6768","last_scrub_stamp":"2024-09-02T10:16:21.735508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:21.735508+0000","objects_scrubbed":6243,"log_size":8045,"log_dups_size":0,"ondisk_log_size":8045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:26.135470+0000","scrub_duration":1736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322109440,"num_objects":4915,"num_object_clones":0,"num_object_copies":14745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8045,"num_write_kb":414720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":462,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.029591+0000","last_change":"2024-09-02T10:16:21.146441+0000","last_active":"2024-09-02T10:16:41.029591+0000","last_peered":"2024-09-02T10:16:41.029591+0000","last_clean":"2024-09-02T10:16:41.029591+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:16:41.029591+0000","last_undegraded":"2024-09-02T10:16:41.029591+0000","last_fullsized":"2024-09-02T10:16:41.029591+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:21.146366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:21.146366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:23.276250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"490'7961","reported_seq":8109,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.082110+0000","last_change":"2024-09-02T10:15:49.395716+0000","last_active":"2024-09-02T10:16:41.082110+0000","last_peered":"2024-09-02T10:16:41.082110+0000","last_clean":"2024-09-02T10:16:41.082110+0000","last_became_active":"2024-09-02T10:14:39.037567+0000","last_became_peered":"2024-09-02T10:14:39.037567+0000","last_unstale":"2024-09-02T10:16:41.082110+0000","last_undegraded":"2024-09-02T10:16:41.082110+0000","last_fullsized":"2024-09-02T10:16:41.082110+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":1699,"log_size":7961,"log_dups_size":0,"ondisk_log_size":7961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332333056,"num_objects":5071,"num_object_clones":0,"num_object_copies":15213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7961,"num_write_kb":417024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":700,"reported_epoch":490,"state":"active+clean","last_fresh":"2024-09-02T10:16:41.026958+0000","last_change":"2024-09-02T10:15:31.899734+0000","last_active":"2024-09-02T10:16:41.026958+0000","last_peered":"2024-09-02T10:16:41.026958+0000","last_clean":"2024-09-02T10:16:41.026958+0000","last_became_active":"2024-09-02T10:15:29.167553+0000","last_became_peered":"2024-09-02T10:15:29.167553+0000","last_unstale":"2024-09-02T10:16:41.026958+0000","last_undegraded":"2024-09-02T10:16:41.026958+0000","last_fullsized":"2024-09-02T10:16:41.026958+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:31.899662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:31.899662+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.920777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"489'7911","reported_seq":15933,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.039912+0000","last_change":"2024-09-02T10:16:42.039584+0000","last_active":"2024-09-02T10:16:42.039912+0000","last_peered":"2024-09-02T10:16:42.039912+0000","last_clean":"2024-09-02T10:16:42.039912+0000","last_became_active":"2024-09-02T10:16:42.029488+0000","last_became_peered":"2024-09-02T10:16:42.029488+0000","last_unstale":"2024-09-02T10:16:42.039912+0000","last_undegraded":"2024-09-02T10:16:42.039912+0000","last_fullsized":"2024-09-02T10:16:42.039912+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":7911,"log_dups_size":0,"ondisk_log_size":7911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330760192,"num_objects":5047,"num_object_clones":0,"num_object_copies":15141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7911,"num_write_kb":414656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3939,"num_bytes_recovered":258146304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":108,"reported_epoch":491,"state":"active+clean","last_fresh":"2024-09-02T10:16:42.119287+0000","last_change":"2024-09-02T10:15:48.739408+0000","last_active":"2024-09-02T10:16:42.119287+0000","last_peered":"2024-09-02T10:16:42.119287+0000","last_clean":"2024-09-02T10:16:42.119287+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:16:42.119287+0000","last_undegraded":"2024-09-02T10:16:42.119287+0000","last_fullsized":"2024-09-02T10:16:42.119287+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:48.739340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:48.739340+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:02.899055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":5275385880,"num_objects":80497,"num_object_clones":0,"num_object_copies":241491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80497,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":126709,"num_write_kb":6630529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5708,"num_bytes_recovered":374079488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15839277056,"data_stored":15839264840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":126709,"ondisk_log_size":126709,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383879,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2006360,"kb_used_data":1618128,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":91738792,"statfs":{"total":95995035648,"available":93940523008,"internally_reserved":0,"allocated":1656963072,"data_stored":1656328195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":397277636},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612812,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2546900,"kb_used_data":2276748,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91198252,"statfs":{"total":95995035648,"available":93387010048,"internally_reserved":0,"allocated":2331389952,"data_stored":2330759147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":276429248},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841745,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1719092,"kb_used_data":1282532,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92026060,"statfs":{"total":95995035648,"available":94234685440,"internally_reserved":0,"allocated":1313312768,"data_stored":1312744343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070679,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1228532,"kb_used_data":954404,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":92516620,"statfs":{"total":95995035648,"available":94737018880,"internally_reserved":0,"allocated":977309696,"data_stored":976741271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299611,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1713808,"kb_used_data":1303616,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92031344,"statfs":{"total":95995035648,"available":94240096256,"internally_reserved":0,"allocated":1334902784,"data_stored":1334204552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":420018634},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528543,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1919508,"kb_used_data":1622988,"kb_used_omap":1,"kb_used_meta":296318,"kb_avail":91825644,"statfs":{"total":95995035648,"available":94029459456,"internally_reserved":0,"allocated":1661939712,"data_stored":1661308907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":303430087},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724772,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2695072,"kb_used_data":2288856,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":91050080,"statfs":{"total":95995035648,"available":93235281920,"internally_reserved":0,"allocated":2343788544,"data_stored":2343150115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953705,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149935,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":718452,"kb_used_data":317988,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93026700,"statfs":{"total":95995035648,"available":95259340800,"internally_reserved":0,"allocated":325619712,"data_stored":325051287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":410057160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411570,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575096,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1747316,"kb_used_data":1359148,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":91997836,"statfs":{"total":95995035648,"available":94205784064,"internally_reserved":0,"allocated":1391767552,"data_stored":1391195575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738622,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3022388,"kb_used_data":2576100,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":90722764,"statfs":{"total":95995035648,"available":92900110336,"internally_reserved":0,"allocated":2637926400,"data_stored":2637357975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2635923456,"data_stored":2635923456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1300365312,"data_stored":1300365312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323682304,"data_stored":323682304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2335576064,"data_stored":2335571992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1659830272,"data_stored":1659830272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1321209856,"data_stored":1321205784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":971571200,"data_stored":971571200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1310851072,"data_stored":1310851072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2329018368,"data_stored":2329018368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1651249152,"data_stored":1651245080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:16:46.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:46.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:46.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:46.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:46.684 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:46.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:46.762 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:16:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:46.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:46.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:46.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:46.905 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:46.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:46.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:16:47.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:47.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:47.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:47.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:47.196 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:47.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:47.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:47.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:47.321 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:47.322 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:47.332 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:47.393 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:16:47.393 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:16:47.423 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:47.456 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:47.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:47.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:47.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:47.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:47.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:47.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:47.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:47.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:47.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:47.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:47.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:47.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:47.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:47.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:16:47.886 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:16:47.909 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:16:47.910 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:16:47.934 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:16:47.934 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:16:47.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:47.964 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:16:47.964 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:16:47.965 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:47.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:47.993 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:16:47.994 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:16:48.023 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:16:48.023 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:16:48.050 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:16:48.050 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:16:48.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:16:48.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:48.075 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:16:48.075 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:16:48.101 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:16:48.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:16:48.125 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:48.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:48.131 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:16:48.131 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:16:48.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:16:48.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:16:48.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:16:48.185 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:16:48.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:48.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:48.255 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:48.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:48.370 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:16:48.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:48.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:48.395 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:16:48.487 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:48.512 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:48.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:48.513 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:48.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:48.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:48.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:48.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:48 smithi162 ceph-mon[20279]: pgmap v1146: 47 pgs: 47 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 998 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-09-02T10:16:48.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1508283597' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:16:48.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:48 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:16:48.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3328831878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:48 smithi151 ceph-mon[16639]: pgmap v1146: 47 pgs: 47 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 998 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-09-02T10:16:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1508283597' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:16:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:48 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:16:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3328831878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:48.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:48 smithi187 ceph-mon[20072]: pgmap v1146: 47 pgs: 47 active+clean; 4.8 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 998 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-09-02T10:16:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1508283597' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:16:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:48 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:16:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3328831878' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:48.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:48.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:48.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:48.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:48.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:48.898 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:48.944 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:16:48.944 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:16:48.944 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:132314 5.b e493/431 5:db8cccf1:::benchmark_data_smithi187_89405_object28709:head v 493'8203, mlcod=493'8201)", 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:16:48.938482+0000", 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0043824800000000002, 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044070000000000003, 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.938482+0000", 2024-09-02T10:16:48.945 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.938482+0000", 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.938482+0000", 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0200000000000003e-07 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.946 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.938486+0000", 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9419999999999997e-06 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.938549+0000", 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2257999999999999e-05 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.947 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.938553+0000", 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5920000000000002e-06 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.938614+0000", 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0788000000000001e-05 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.938630+0000", 2024-09-02T10:16:48.948 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.628e-05 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:132317 5.7 5:eb6d1602:::benchmark_data_smithi187_89405_object28712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e493)", 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:16:48.940009+0000", 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.002855289, 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029373110000000002, 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:48.949 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 132317 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.940009+0000", 2024-09-02T10:16:48.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.940009+0000", 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.940011+0000", 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4219999999999999e-06 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.951 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:48.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.940016+0000", 2024-09-02T10:16:48.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0699999999999997e-06 2024-09-02T10:16:48.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:48.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.940025+0000", 2024-09-02T10:16:48.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9679999999999995e-06 2024-09-02T10:16:48.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.940030+0000", 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2290000000000002e-06 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.940075+0000", 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5120000000000002e-05 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:48.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.940159+0000", 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4325000000000003e-05 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:48.940218+0000", 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8802999999999999e-05 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:16:48.954 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-02T10:16:48.955 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:16:48.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:49.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:49.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:49.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:49.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:49.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:49.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:49.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:49.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:49.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:49.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:49.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:49.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:49.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:49.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:49.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:49.539 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:49.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:16:49.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:49 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:49 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:16:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:49 smithi162 ceph-mon[20279]: osdmap e493: 12 total, 12 up, 10 in 2024-09-02T10:16:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:49 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:49 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:16:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:49 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:16:49.612 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:49 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:49.612 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:49 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:16:49.612 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:49 smithi151 ceph-mon[16639]: osdmap e493: 12 total, 12 up, 10 in 2024-09-02T10:16:49.612 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:49 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:49.612 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:49 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:16:49.612 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:49 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:16:49.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:49.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:49.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:49 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:16:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:49 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:16:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:49 smithi187 ceph-mon[20072]: osdmap e493: 12 total, 12 up, 10 in 2024-09-02T10:16:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:49 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:49 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:16:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:49 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:16:49.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:49.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:49.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:49.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:49.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:49.915 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:49.926 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:49.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:49.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:50.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:50.053 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:50.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:50.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:50.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:50.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:50.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:50.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:50.311 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:50.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:50.312 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:50.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:16:50.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:50.456 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:50.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:50.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:50.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:50.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:50.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:50.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:50.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:50 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:16:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:50 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:16:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:50 smithi151 ceph-mon[16639]: pgmap v1148: 47 pgs: 47 active+clean; 4.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 979 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:16:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:50 smithi151 ceph-mon[16639]: osdmap e494: 12 total, 12 up, 10 in 2024-09-02T10:16:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:50 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:16:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:50 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:16:50.727 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:50.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:50.735 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:16:50.735 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:16:50.738 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:16:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:50 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:16:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:50 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:16:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:50 smithi187 ceph-mon[20072]: pgmap v1148: 47 pgs: 47 active+clean; 4.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 979 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:16:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:50 smithi187 ceph-mon[20072]: osdmap e494: 12 total, 12 up, 10 in 2024-09-02T10:16:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:50 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:16:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:50 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:16:50.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:50 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:16:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:50 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:16:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:50 smithi162 ceph-mon[20279]: pgmap v1148: 47 pgs: 47 active+clean; 4.7 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 979 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:16:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:50 smithi162 ceph-mon[20279]: osdmap e494: 12 total, 12 up, 10 in 2024-09-02T10:16:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:50 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:16:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:50 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:16:50.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:50.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:50.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:50.990 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:50.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:51.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:51.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:51.122 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:51.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:16:51.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:51.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:51.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:51.290 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:16:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:16:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:16:51.290 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:16:51.291 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:16:51.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:51.379 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:51.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:51.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:51.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:51.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:51.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:51.509 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:51 smithi151 ceph-mon[16639]: from='client.16710 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:51.509 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:51 smithi151 ceph-mon[16639]: 5.9 scrub starts 2024-09-02T10:16:51.510 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1149,"stamp":"2024-09-02T10:16:49.372564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4996792888,"num_objects":76239,"num_object_clones":0,"num_object_copies":228717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76239,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130969,"num_write_kb":6630529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5711,"num_bytes_recovered":374809664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":133821,"ondisk_log_size":133821,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":18542940,"kb_used_data":14829204,"kb_used_omap":15,"kb_used_meta":3713008,"kb_avail":918908580,"statfs":{"total":959950356480,"available":940962385920,"internally_reserved":0,"allocated":15185104896,"data_stored":15180085674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3802120625},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-327811072,"num_objects":-5002,"num_object_clones":0,"num_object_copies":-15006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5002,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.068318"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":311,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.032010+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:16:45.032010+0000","last_peered":"2024-09-02T10:16:45.032010+0000","last_clean":"2024-09-02T10:16:45.032010+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:16:45.032010+0000","last_undegraded":"2024-09-02T10:16:45.032010+0000","last_fullsized":"2024-09-02T10:16:45.032010+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":121,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:48.372229+0000","last_change":"2024-09-02T10:15:41.899476+0000","last_active":"2024-09-02T10:16:48.372229+0000","last_peered":"2024-09-02T10:16:48.372229+0000","last_clean":"2024-09-02T10:16:48.372229+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:16:48.372229+0000","last_undegraded":"2024-09-02T10:16:48.372229+0000","last_fullsized":"2024-09-02T10:16:48.372229+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:41.899397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:41.899397+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:51.002146+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":579,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:48.940363+0000","last_change":"2024-09-02T10:16:48.940363+0000","last_active":"2024-09-02T10:16:48.940363+0000","last_peered":"2024-09-02T10:16:48.940363+0000","last_clean":"2024-09-02T10:16:48.940363+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:16:48.940363+0000","last_undegraded":"2024-09-02T10:16:48.940363+0000","last_fullsized":"2024-09-02T10:16:48.940363+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:48.940301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:48.940301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:14.457035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":571,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:48.369793+0000","last_change":"2024-09-02T10:15:19.065187+0000","last_active":"2024-09-02T10:16:48.369793+0000","last_peered":"2024-09-02T10:16:48.369793+0000","last_clean":"2024-09-02T10:16:48.369793+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:16:48.369793+0000","last_undegraded":"2024-09-02T10:16:48.369793+0000","last_fullsized":"2024-09-02T10:16:48.369793+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:19.065134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:19.065134+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:51.939426+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":490,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.032341+0000","last_change":"2024-09-02T10:15:35.202261+0000","last_active":"2024-09-02T10:16:45.032341+0000","last_peered":"2024-09-02T10:16:45.032341+0000","last_clean":"2024-09-02T10:16:45.032341+0000","last_became_active":"2024-09-02T10:15:35.201887+0000","last_became_peered":"2024-09-02T10:15:35.201887+0000","last_unstale":"2024-09-02T10:16:45.032341+0000","last_undegraded":"2024-09-02T10:16:45.032341+0000","last_fullsized":"2024-09-02T10:16:45.032341+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:04.994667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:04.994667+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":418,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.033546+0000","last_change":"2024-09-02T10:14:11.689851+0000","last_active":"2024-09-02T10:16:45.033546+0000","last_peered":"2024-09-02T10:16:45.033546+0000","last_clean":"2024-09-02T10:16:45.033546+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:16:45.033546+0000","last_undegraded":"2024-09-02T10:16:45.033546+0000","last_fullsized":"2024-09-02T10:16:45.033546+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:11.689767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:11.689767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:49.512426+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":942,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.033228+0000","last_change":"2024-09-02T10:16:23.306394+0000","last_active":"2024-09-02T10:16:45.033228+0000","last_peered":"2024-09-02T10:16:45.033228+0000","last_clean":"2024-09-02T10:16:45.033228+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:16:45.033228+0000","last_undegraded":"2024-09-02T10:16:45.033228+0000","last_fullsized":"2024-09-02T10:16:45.033228+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:23.306331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:23.306331+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:34.300255+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":586,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:48.369974+0000","last_change":"2024-09-02T10:16:10.994908+0000","last_active":"2024-09-02T10:16:48.369974+0000","last_peered":"2024-09-02T10:16:48.369974+0000","last_clean":"2024-09-02T10:16:48.369974+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:16:48.369974+0000","last_undegraded":"2024-09-02T10:16:48.369974+0000","last_fullsized":"2024-09-02T10:16:48.369974+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.994824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.994824+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:17.261476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":585,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:48.372081+0000","last_change":"2024-09-02T10:15:34.791760+0000","last_active":"2024-09-02T10:16:48.372081+0000","last_peered":"2024-09-02T10:16:48.372081+0000","last_clean":"2024-09-02T10:16:48.372081+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:16:48.372081+0000","last_undegraded":"2024-09-02T10:16:48.372081+0000","last_fullsized":"2024-09-02T10:16:48.372081+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:34.791708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:34.791708+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.857252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":560,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.032384+0000","last_change":"2024-09-02T10:16:14.103234+0000","last_active":"2024-09-02T10:16:45.032384+0000","last_peered":"2024-09-02T10:16:45.032384+0000","last_clean":"2024-09-02T10:16:45.032384+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:16:45.032384+0000","last_undegraded":"2024-09-02T10:16:45.032384+0000","last_fullsized":"2024-09-02T10:16:45.032384+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:14.103166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:14.103166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:33.443226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":411,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.032289+0000","last_change":"2024-09-02T10:16:11.152819+0000","last_active":"2024-09-02T10:16:45.032289+0000","last_peered":"2024-09-02T10:16:45.032289+0000","last_clean":"2024-09-02T10:16:45.032289+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:16:45.032289+0000","last_undegraded":"2024-09-02T10:16:45.032289+0000","last_fullsized":"2024-09-02T10:16:45.032289+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:11.152686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:11.152686+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:26.925270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":919,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:48.372031+0000","last_change":"2024-09-02T10:16:19.673642+0000","last_active":"2024-09-02T10:16:48.372031+0000","last_peered":"2024-09-02T10:16:48.372031+0000","last_clean":"2024-09-02T10:16:48.372031+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:16:48.372031+0000","last_undegraded":"2024-09-02T10:16:48.372031+0000","last_fullsized":"2024-09-02T10:16:48.372031+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:19.673572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:19.673572+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:33.436612+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":852,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.033597+0000","last_change":"2024-09-02T10:16:15.378306+0000","last_active":"2024-09-02T10:16:45.033597+0000","last_peered":"2024-09-02T10:16:45.033597+0000","last_clean":"2024-09-02T10:16:45.033597+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:16:45.033597+0000","last_undegraded":"2024-09-02T10:16:45.033597+0000","last_fullsized":"2024-09-02T10:16:45.033597+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:15.378230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:15.378230+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:18.040488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":831,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.029716+0000","last_change":"2024-09-02T10:15:29.170702+0000","last_active":"2024-09-02T10:16:45.029716+0000","last_peered":"2024-09-02T10:16:45.029716+0000","last_clean":"2024-09-02T10:16:45.029716+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:16:45.029716+0000","last_undegraded":"2024-09-02T10:16:45.029716+0000","last_fullsized":"2024-09-02T10:16:45.029716+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.665438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.665438+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":822,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.033369+0000","last_change":"2024-09-02T10:14:34.217697+0000","last_active":"2024-09-02T10:16:45.033369+0000","last_peered":"2024-09-02T10:16:45.033369+0000","last_clean":"2024-09-02T10:16:45.033369+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:16:45.033369+0000","last_undegraded":"2024-09-02T10:16:45.033369+0000","last_fullsized":"2024-09-02T10:16:45.033369+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:34.217624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:34.217624+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:50.229417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"493'8097","reported_seq":11857,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:48.586832+0000","last_change":"2024-09-02T10:16:17.226985+0000","last_active":"2024-09-02T10:16:48.586832+0000","last_peered":"2024-09-02T10:16:48.586832+0000","last_clean":"2024-09-02T10:16:48.586832+0000","last_became_active":"2024-09-02T10:15:24.141368+0000","last_became_peered":"2024-09-02T10:15:24.141368+0000","last_unstale":"2024-09-02T10:16:48.586832+0000","last_undegraded":"2024-09-02T10:16:48.586832+0000","last_fullsized":"2024-09-02T10:16:48.586832+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6399","last_scrub_stamp":"2024-09-02T10:16:17.226765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.226765+0000","objects_scrubbed":6342,"log_size":8097,"log_dups_size":0,"ondisk_log_size":8097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:37.316444+0000","scrub_duration":2531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306905088,"num_objects":4683,"num_object_clones":0,"num_object_copies":14049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8097,"num_write_kb":408960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":115933184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":908,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.033360+0000","last_change":"2024-09-02T10:14:41.306927+0000","last_active":"2024-09-02T10:16:45.033360+0000","last_peered":"2024-09-02T10:16:45.033360+0000","last_clean":"2024-09-02T10:16:45.033360+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:16:45.033360+0000","last_undegraded":"2024-09-02T10:16:45.033360+0000","last_fullsized":"2024-09-02T10:16:45.033360+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:41.306868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:41.306868+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"492'8186","reported_seq":8320,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:47.125793+0000","last_change":"2024-09-02T10:14:39.040351+0000","last_active":"2024-09-02T10:16:47.125793+0000","last_peered":"2024-09-02T10:16:47.125793+0000","last_clean":"2024-09-02T10:16:47.125793+0000","last_became_active":"2024-09-02T10:14:39.040025+0000","last_became_peered":"2024-09-02T10:14:39.040025+0000","last_unstale":"2024-09-02T10:16:47.125793+0000","last_undegraded":"2024-09-02T10:16:47.125793+0000","last_fullsized":"2024-09-02T10:16:47.125793+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":8186,"log_dups_size":0,"ondisk_log_size":8186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:50.166483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306642968,"num_objects":4680,"num_object_clones":0,"num_object_copies":14040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4680,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8186,"num_write_kb":411649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1031,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.032298+0000","last_change":"2024-09-02T10:16:22.269544+0000","last_active":"2024-09-02T10:16:45.032298+0000","last_peered":"2024-09-02T10:16:45.032298+0000","last_clean":"2024-09-02T10:16:45.032298+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:16:45.032298+0000","last_undegraded":"2024-09-02T10:16:45.032298+0000","last_fullsized":"2024-09-02T10:16:45.032298+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:22.269495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:22.269495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:47.553829+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"492'8110","reported_seq":8315,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:47.964084+0000","last_change":"2024-09-02T10:16:15.009837+0000","last_active":"2024-09-02T10:16:47.964084+0000","last_peered":"2024-09-02T10:16:47.964084+0000","last_clean":"2024-09-02T10:16:47.964084+0000","last_became_active":"2024-09-02T10:14:39.038164+0000","last_became_peered":"2024-09-02T10:14:39.038164+0000","last_unstale":"2024-09-02T10:16:47.964084+0000","last_undegraded":"2024-09-02T10:16:47.964084+0000","last_fullsized":"2024-09-02T10:16:47.964084+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6340","last_scrub_stamp":"2024-09-02T10:16:15.009487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:15.009487+0000","objects_scrubbed":6273,"log_size":8110,"log_dups_size":0,"ondisk_log_size":8110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:35.488395+0000","scrub_duration":2145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308412416,"num_objects":4706,"num_object_clones":0,"num_object_copies":14118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4706,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8110,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":674,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.031624+0000","last_change":"2024-09-02T10:16:10.884551+0000","last_active":"2024-09-02T10:16:45.031624+0000","last_peered":"2024-09-02T10:16:45.031624+0000","last_clean":"2024-09-02T10:16:45.031624+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:16:45.031624+0000","last_undegraded":"2024-09-02T10:16:45.031624+0000","last_fullsized":"2024-09-02T10:16:45.031624+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.884481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.884481+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:22.734487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"492'8285","reported_seq":8488,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:47.061437+0000","last_change":"2024-09-02T10:16:26.551606+0000","last_active":"2024-09-02T10:16:47.061437+0000","last_peered":"2024-09-02T10:16:47.061437+0000","last_clean":"2024-09-02T10:16:47.061437+0000","last_became_active":"2024-09-02T10:14:39.037121+0000","last_became_peered":"2024-09-02T10:14:39.037121+0000","last_unstale":"2024-09-02T10:16:47.061437+0000","last_undegraded":"2024-09-02T10:16:47.061437+0000","last_fullsized":"2024-09-02T10:16:47.061437+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'7110","last_scrub_stamp":"2024-09-02T10:16:26.551442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:26.551442+0000","objects_scrubbed":6016,"log_size":8285,"log_dups_size":0,"ondisk_log_size":8285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:27.466417+0000","scrub_duration":1258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315817984,"num_objects":4819,"num_object_clones":0,"num_object_copies":14457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8285,"num_write_kb":419328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":70,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.031879+0000","last_change":"2024-09-02T10:15:35.208246+0000","last_active":"2024-09-02T10:16:45.031879+0000","last_peered":"2024-09-02T10:16:45.031879+0000","last_clean":"2024-09-02T10:16:45.031879+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:16:45.031879+0000","last_undegraded":"2024-09-02T10:16:45.031879+0000","last_fullsized":"2024-09-02T10:16:45.031879+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:31.668602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:31.668602+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:49.129751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"492'8026","reported_seq":8160,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:46.605161+0000","last_change":"2024-09-02T10:14:39.642395+0000","last_active":"2024-09-02T10:16:46.605161+0000","last_peered":"2024-09-02T10:16:46.605161+0000","last_clean":"2024-09-02T10:16:46.605161+0000","last_became_active":"2024-09-02T10:14:39.641995+0000","last_became_peered":"2024-09-02T10:14:39.641995+0000","last_unstale":"2024-09-02T10:16:46.605161+0000","last_undegraded":"2024-09-02T10:16:46.605161+0000","last_fullsized":"2024-09-02T10:16:46.605161+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":8026,"log_dups_size":0,"ondisk_log_size":8026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319029248,"num_objects":4868,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8026,"num_write_kb":412608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":979,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.032018+0000","last_change":"2024-09-02T10:15:29.195341+0000","last_active":"2024-09-02T10:16:45.032018+0000","last_peered":"2024-09-02T10:16:45.032018+0000","last_clean":"2024-09-02T10:16:45.032018+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:16:45.032018+0000","last_undegraded":"2024-09-02T10:16:45.032018+0000","last_fullsized":"2024-09-02T10:16:45.032018+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.277119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.277119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"493'8238","reported_seq":8443,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:48.904049+0000","last_change":"2024-09-02T10:16:22.056000+0000","last_active":"2024-09-02T10:16:48.904049+0000","last_peered":"2024-09-02T10:16:48.904049+0000","last_clean":"2024-09-02T10:16:48.904049+0000","last_became_active":"2024-09-02T10:14:39.034636+0000","last_became_peered":"2024-09-02T10:14:39.034636+0000","last_unstale":"2024-09-02T10:16:48.904049+0000","last_undegraded":"2024-09-02T10:16:48.904049+0000","last_fullsized":"2024-09-02T10:16:48.904049+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6741","last_scrub_stamp":"2024-09-02T10:16:22.055909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:22.055909+0000","objects_scrubbed":6225,"log_size":8238,"log_dups_size":0,"ondisk_log_size":8238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:24.765454+0000","scrub_duration":2093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305266688,"num_objects":4658,"num_object_clones":0,"num_object_copies":13974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8238,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":447,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.033513+0000","last_change":"2024-09-02T10:15:35.205684+0000","last_active":"2024-09-02T10:16:45.033513+0000","last_peered":"2024-09-02T10:16:45.033513+0000","last_clean":"2024-09-02T10:16:45.033513+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:16:45.033513+0000","last_undegraded":"2024-09-02T10:16:45.033513+0000","last_fullsized":"2024-09-02T10:16:45.033513+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.701773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.701773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"492'8038","reported_seq":8227,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:47.213669+0000","last_change":"2024-09-02T10:15:49.158612+0000","last_active":"2024-09-02T10:16:47.213669+0000","last_peered":"2024-09-02T10:16:47.213669+0000","last_clean":"2024-09-02T10:16:47.213669+0000","last_became_active":"2024-09-02T10:14:39.037822+0000","last_became_peered":"2024-09-02T10:14:39.037822+0000","last_unstale":"2024-09-02T10:16:47.213669+0000","last_undegraded":"2024-09-02T10:16:47.213669+0000","last_fullsized":"2024-09-02T10:16:47.213669+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4732","last_scrub_stamp":"2024-09-02T10:15:49.158487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.158487+0000","objects_scrubbed":4682,"log_size":8038,"log_dups_size":0,"ondisk_log_size":8038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:57.249982+0000","scrub_duration":968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308936704,"num_objects":4714,"num_object_clones":0,"num_object_copies":14142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8038,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":419,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.033896+0000","last_change":"2024-09-02T10:14:09.768298+0000","last_active":"2024-09-02T10:16:45.033896+0000","last_peered":"2024-09-02T10:16:45.033896+0000","last_clean":"2024-09-02T10:16:45.033896+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:16:45.033896+0000","last_undegraded":"2024-09-02T10:16:45.033896+0000","last_fullsized":"2024-09-02T10:16:45.033896+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:09.768061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:09.768061+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"492'8386","reported_seq":8518,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:47.776424+0000","last_change":"2024-09-02T10:14:39.035303+0000","last_active":"2024-09-02T10:16:47.776424+0000","last_peered":"2024-09-02T10:16:47.776424+0000","last_clean":"2024-09-02T10:16:47.776424+0000","last_became_active":"2024-09-02T10:14:39.035100+0000","last_became_peered":"2024-09-02T10:14:39.035100+0000","last_unstale":"2024-09-02T10:16:47.776424+0000","last_undegraded":"2024-09-02T10:16:47.776424+0000","last_fullsized":"2024-09-02T10:16:47.776424+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":8386,"log_dups_size":0,"ondisk_log_size":8386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:51.786988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320471040,"num_objects":4890,"num_object_clones":0,"num_object_copies":14670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8386,"num_write_kb":424832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":170,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.035031+0000","last_change":"2024-09-02T10:15:36.299961+0000","last_active":"2024-09-02T10:16:45.035031+0000","last_peered":"2024-09-02T10:16:45.035031+0000","last_clean":"2024-09-02T10:16:45.035031+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:16:45.035031+0000","last_undegraded":"2024-09-02T10:16:45.035031+0000","last_fullsized":"2024-09-02T10:16:45.035031+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:36.299914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:36.299914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:01.870130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"492'8235","reported_seq":8369,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:47.771269+0000","last_change":"2024-09-02T10:14:39.641746+0000","last_active":"2024-09-02T10:16:47.771269+0000","last_peered":"2024-09-02T10:16:47.771269+0000","last_clean":"2024-09-02T10:16:47.771269+0000","last_became_active":"2024-09-02T10:14:39.641478+0000","last_became_peered":"2024-09-02T10:14:39.641478+0000","last_unstale":"2024-09-02T10:16:47.771269+0000","last_undegraded":"2024-09-02T10:16:47.771269+0000","last_fullsized":"2024-09-02T10:16:47.771269+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":8235,"log_dups_size":0,"ondisk_log_size":8235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309395456,"num_objects":4721,"num_object_clones":0,"num_object_copies":14163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8235,"num_write_kb":414592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":420,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.033647+0000","last_change":"2024-09-02T10:14:36.623992+0000","last_active":"2024-09-02T10:16:45.033647+0000","last_peered":"2024-09-02T10:16:45.033647+0000","last_clean":"2024-09-02T10:16:45.033647+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:16:45.033647+0000","last_undegraded":"2024-09-02T10:16:45.033647+0000","last_fullsized":"2024-09-02T10:16:45.033647+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.623930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.623930+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"492'8094","reported_seq":8281,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:47.038734+0000","last_change":"2024-09-02T10:15:49.363717+0000","last_active":"2024-09-02T10:16:47.038734+0000","last_peered":"2024-09-02T10:16:47.038734+0000","last_clean":"2024-09-02T10:16:47.038734+0000","last_became_active":"2024-09-02T10:14:39.037882+0000","last_became_peered":"2024-09-02T10:14:39.037882+0000","last_unstale":"2024-09-02T10:16:47.038734+0000","last_undegraded":"2024-09-02T10:16:47.038734+0000","last_fullsized":"2024-09-02T10:16:47.038734+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4720","last_scrub_stamp":"2024-09-02T10:15:49.363588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.363588+0000","objects_scrubbed":4682,"log_size":8094,"log_dups_size":0,"ondisk_log_size":8094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:06.930040+0000","scrub_duration":1106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312999936,"num_objects":4776,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8094,"num_write_kb":411840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":673,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.032454+0000","last_change":"2024-09-02T10:16:23.198280+0000","last_active":"2024-09-02T10:16:45.032454+0000","last_peered":"2024-09-02T10:16:45.032454+0000","last_clean":"2024-09-02T10:16:45.032454+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:16:45.032454+0000","last_undegraded":"2024-09-02T10:16:45.032454+0000","last_fullsized":"2024-09-02T10:16:45.032454+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:23.198227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:23.198227+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:52.413888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"492'8152","reported_seq":8358,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:47.066535+0000","last_change":"2024-09-02T10:16:17.087529+0000","last_active":"2024-09-02T10:16:47.066535+0000","last_peered":"2024-09-02T10:16:47.066535+0000","last_clean":"2024-09-02T10:16:47.066535+0000","last_became_active":"2024-09-02T10:14:39.641932+0000","last_became_peered":"2024-09-02T10:14:39.641932+0000","last_unstale":"2024-09-02T10:16:47.066535+0000","last_undegraded":"2024-09-02T10:16:47.066535+0000","last_fullsized":"2024-09-02T10:16:47.066535+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6467","last_scrub_stamp":"2024-09-02T10:16:17.087427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.087427+0000","objects_scrubbed":6467,"log_size":8152,"log_dups_size":0,"ondisk_log_size":8152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:47.064105+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313393152,"num_objects":4782,"num_object_clones":0,"num_object_copies":14346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8152,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":935,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:48.372170+0000","last_change":"2024-09-02T10:16:17.754936+0000","last_active":"2024-09-02T10:16:48.372170+0000","last_peered":"2024-09-02T10:16:48.372170+0000","last_clean":"2024-09-02T10:16:48.372170+0000","last_became_active":"2024-09-02T10:15:29.168692+0000","last_became_peered":"2024-09-02T10:15:29.168692+0000","last_unstale":"2024-09-02T10:16:48.372170+0000","last_undegraded":"2024-09-02T10:16:48.372170+0000","last_fullsized":"2024-09-02T10:16:48.372170+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:17.754866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.754866+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:39.447349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"492'8283","reported_seq":8417,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:46.640000+0000","last_change":"2024-09-02T10:14:39.038104+0000","last_active":"2024-09-02T10:16:46.640000+0000","last_peered":"2024-09-02T10:16:46.640000+0000","last_clean":"2024-09-02T10:16:46.640000+0000","last_became_active":"2024-09-02T10:14:39.037755+0000","last_became_peered":"2024-09-02T10:14:39.037755+0000","last_unstale":"2024-09-02T10:16:46.640000+0000","last_undegraded":"2024-09-02T10:16:46.640000+0000","last_fullsized":"2024-09-02T10:16:46.640000+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":8283,"log_dups_size":0,"ondisk_log_size":8283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321191936,"num_objects":4901,"num_object_clones":0,"num_object_copies":14703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8283,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":683,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.032184+0000","last_change":"2024-09-02T10:15:35.202850+0000","last_active":"2024-09-02T10:16:45.032184+0000","last_peered":"2024-09-02T10:16:45.032184+0000","last_clean":"2024-09-02T10:16:45.032184+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:16:45.032184+0000","last_undegraded":"2024-09-02T10:16:45.032184+0000","last_fullsized":"2024-09-02T10:16:45.032184+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:48.187420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:48.187420+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"492'8141","reported_seq":8345,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:47.899191+0000","last_change":"2024-09-02T10:16:20.568272+0000","last_active":"2024-09-02T10:16:47.899191+0000","last_peered":"2024-09-02T10:16:47.899191+0000","last_clean":"2024-09-02T10:16:47.899191+0000","last_became_active":"2024-09-02T10:14:39.039060+0000","last_became_peered":"2024-09-02T10:14:39.039060+0000","last_unstale":"2024-09-02T10:16:47.899191+0000","last_undegraded":"2024-09-02T10:16:47.899191+0000","last_fullsized":"2024-09-02T10:16:47.899191+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6670","last_scrub_stamp":"2024-09-02T10:16:20.568081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:20.568081+0000","objects_scrubbed":6316,"log_size":8141,"log_dups_size":0,"ondisk_log_size":8141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:27.515779+0000","scrub_duration":1605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313720832,"num_objects":4787,"num_object_clones":0,"num_object_copies":14361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4787,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8141,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":932,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.035135+0000","last_change":"2024-09-02T10:16:17.179894+0000","last_active":"2024-09-02T10:16:45.035135+0000","last_peered":"2024-09-02T10:16:45.035135+0000","last_clean":"2024-09-02T10:16:45.035135+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:16:45.035135+0000","last_undegraded":"2024-09-02T10:16:45.035135+0000","last_fullsized":"2024-09-02T10:16:45.035135+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:17.179849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.179849+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:24.924645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"493'8290","reported_seq":8497,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:49.322594+0000","last_change":"2024-09-02T10:16:21.735648+0000","last_active":"2024-09-02T10:16:49.322594+0000","last_peered":"2024-09-02T10:16:49.322594+0000","last_clean":"2024-09-02T10:16:49.322594+0000","last_became_active":"2024-09-02T10:14:39.036453+0000","last_became_peered":"2024-09-02T10:14:39.036453+0000","last_unstale":"2024-09-02T10:16:49.322594+0000","last_undegraded":"2024-09-02T10:16:49.322594+0000","last_fullsized":"2024-09-02T10:16:49.322594+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6768","last_scrub_stamp":"2024-09-02T10:16:21.735508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:21.735508+0000","objects_scrubbed":6243,"log_size":8290,"log_dups_size":0,"ondisk_log_size":8290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:26.135470+0000","scrub_duration":1736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306053120,"num_objects":4670,"num_object_clones":0,"num_object_copies":14010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8290,"num_write_kb":414720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":466,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.032410+0000","last_change":"2024-09-02T10:16:21.146441+0000","last_active":"2024-09-02T10:16:45.032410+0000","last_peered":"2024-09-02T10:16:45.032410+0000","last_clean":"2024-09-02T10:16:45.032410+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:16:45.032410+0000","last_undegraded":"2024-09-02T10:16:45.032410+0000","last_fullsized":"2024-09-02T10:16:45.032410+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:21.146366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:21.146366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:23.276250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"492'8194","reported_seq":8347,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:46.047054+0000","last_change":"2024-09-02T10:15:49.395716+0000","last_active":"2024-09-02T10:16:46.047054+0000","last_peered":"2024-09-02T10:16:46.047054+0000","last_clean":"2024-09-02T10:16:46.047054+0000","last_became_active":"2024-09-02T10:14:39.037567+0000","last_became_peered":"2024-09-02T10:14:39.037567+0000","last_unstale":"2024-09-02T10:16:46.047054+0000","last_undegraded":"2024-09-02T10:16:46.047054+0000","last_fullsized":"2024-09-02T10:16:46.047054+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":1699,"log_size":8194,"log_dups_size":0,"ondisk_log_size":8194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317063168,"num_objects":4838,"num_object_clones":0,"num_object_copies":14514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8194,"num_write_kb":417024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":704,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:45.032481+0000","last_change":"2024-09-02T10:15:31.899734+0000","last_active":"2024-09-02T10:16:45.032481+0000","last_peered":"2024-09-02T10:16:45.032481+0000","last_clean":"2024-09-02T10:16:45.032481+0000","last_became_active":"2024-09-02T10:15:29.167553+0000","last_became_peered":"2024-09-02T10:15:29.167553+0000","last_unstale":"2024-09-02T10:16:45.032481+0000","last_undegraded":"2024-09-02T10:16:45.032481+0000","last_fullsized":"2024-09-02T10:16:45.032481+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:31.899662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:31.899662+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:16:56.920777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"492'8214","reported_seq":16238,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-02T10:16:47.070989+0000","last_change":"2024-09-02T10:16:42.039584+0000","last_active":"2024-09-02T10:16:47.070989+0000","last_peered":"2024-09-02T10:16:47.070989+0000","last_clean":"2024-09-02T10:16:47.070989+0000","last_became_active":"2024-09-02T10:16:42.029488+0000","last_became_peered":"2024-09-02T10:16:42.029488+0000","last_unstale":"2024-09-02T10:16:47.070989+0000","last_undegraded":"2024-09-02T10:16:47.070989+0000","last_fullsized":"2024-09-02T10:16:47.070989+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":8214,"log_dups_size":0,"ondisk_log_size":8214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310902784,"num_objects":4744,"num_object_clones":0,"num_object_copies":14232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8214,"num_write_kb":414656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3939,"num_bytes_recovered":258146304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":112,"reported_epoch":493,"state":"active+clean","last_fresh":"2024-09-02T10:16:48.375399+0000","last_change":"2024-09-02T10:15:48.739408+0000","last_active":"2024-09-02T10:16:48.375399+0000","last_peered":"2024-09-02T10:16:48.375399+0000","last_clean":"2024-09-02T10:16:48.375399+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:16:48.375399+0000","last_undegraded":"2024-09-02T10:16:48.375399+0000","last_fullsized":"2024-09-02T10:16:48.375399+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:48.739340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:48.739340+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:02.899055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4996202520,"num_objects":76237,"num_object_clones":0,"num_object_copies":228711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76237,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":130969,"num_write_kb":6630529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5708,"num_bytes_recovered":374079488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14998581248,"data_stored":14998569032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130969,"ondisk_log_size":130969,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383880,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1936456,"kb_used_data":1544128,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91808696,"statfs":{"total":95995035648,"available":94012104704,"internally_reserved":0,"allocated":1581187072,"data_stored":1580677482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":401537476},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612813,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2427844,"kb_used_data":2166012,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":91317308,"statfs":{"total":95995035648,"available":93508923392,"internally_reserved":0,"allocated":2217996288,"data_stored":2217490770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":267909568},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841746,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1638476,"kb_used_data":1214396,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92106676,"statfs":{"total":95995035648,"available":94317236224,"internally_reserved":0,"allocated":1243541504,"data_stored":1243035986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":434239936},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070680,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1166884,"kb_used_data":905236,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92578268,"statfs":{"total":95995035648,"available":94800146432,"internally_reserved":0,"allocated":926961664,"data_stored":926514976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299612,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1658072,"kb_used_data":1243720,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92087080,"statfs":{"total":95995035648,"available":94297169920,"internally_reserved":0,"allocated":1273569280,"data_stored":1273056138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528544,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1838156,"kb_used_data":1536252,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91906996,"statfs":{"total":95995035648,"available":94112763904,"internally_reserved":0,"allocated":1573122048,"data_stored":1572616530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724773,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2593104,"kb_used_data":2182728,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91152048,"statfs":{"total":95995035648,"available":93339697152,"internally_reserved":0,"allocated":2235113472,"data_stored":2234600330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953706,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149936,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":703884,"kb_used_data":303420,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93041268,"statfs":{"total":95995035648,"available":95274258432,"internally_reserved":0,"allocated":310702080,"data_stored":310196562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":410057160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411571,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575097,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689044,"kb_used_data":1296900,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92056108,"statfs":{"total":95995035648,"available":94265454592,"internally_reserved":0,"allocated":1328025600,"data_stored":1327516530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738623,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2891020,"kb_used_data":2436412,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":90854132,"statfs":{"total":95995035648,"available":93034631168,"internally_reserved":0,"allocated":2494885888,"data_stored":2494380370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":465500613},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2492465152,"data_stored":2492465152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1230503936,"data_stored":1230503936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":305266688,"data_stored":305266688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2209222656,"data_stored":2209218584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1567293440,"data_stored":1567293440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1261047808,"data_stored":1261043736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":920715264,"data_stored":920715264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1240465408,"data_stored":1240465408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":2212364288,"data_stored":2212364288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1559236608,"data_stored":1559232536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:16:51.512 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-02T10:16:51.512 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:16:51.513 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-09-02T10:16:51.513 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-02T10:16:51.513 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-02T10:16:51.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-09-02T10:16:51.541 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:51 smithi162 ceph-mon[20279]: from='client.16710 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:51 smithi162 ceph-mon[20279]: 5.9 scrub starts 2024-09-02T10:16:51.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:51.639 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:51.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:51.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:51.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:16:51.741 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:51 smithi187 ceph-mon[20072]: from='client.16710 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:16:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:51 smithi187 ceph-mon[20072]: 5.9 scrub starts 2024-09-02T10:16:51.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:51.772 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:51.869 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:16:51.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:51.898 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:51.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:51.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:51.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:16:52.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:52.036 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:52.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:52.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:52.166 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:52.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:52.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:52.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:52.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:52.316 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:16:52.317 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:52.326 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:16:52.326 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:16:52.360 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:16:52.360 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:16:52.391 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:16:52.392 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:16:52.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:52.422 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:16:52.422 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:16:52.447 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:52.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:52.449 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:16:52.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:16:52.450 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:16:52.450 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-02T10:16:52.450 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:16:52.470 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:16:52.470 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:16:52.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:16:52.500 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:16:52.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:52.576 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:52.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:52.577 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:52.608 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:52 smithi187 ceph-mon[20072]: pgmap v1150: 47 pgs: 47 active+clean; 4.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 729 op/s 2024-09-02T10:16:52.608 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:52 smithi187 ceph-mon[20072]: 5.5 scrub starts 2024-09-02T10:16:52.609 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3934839334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:52.609 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:52 smithi187 ceph-mon[20072]: 5.9 scrub ok 2024-09-02T10:16:52.620 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:52 smithi162 ceph-mon[20279]: pgmap v1150: 47 pgs: 47 active+clean; 4.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 729 op/s 2024-09-02T10:16:52.621 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:52 smithi162 ceph-mon[20279]: 5.5 scrub starts 2024-09-02T10:16:52.621 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3934839334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:52.621 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:52 smithi162 ceph-mon[20279]: 5.9 scrub ok 2024-09-02T10:16:52.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:52.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:52 smithi151 ceph-mon[16639]: pgmap v1150: 47 pgs: 47 active+clean; 4.6 GiB data, 18 GiB used, 876 GiB / 894 GiB avail; 0 B/s wr, 729 op/s 2024-09-02T10:16:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:52 smithi151 ceph-mon[16639]: 5.5 scrub starts 2024-09-02T10:16:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3934839334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:52 smithi151 ceph-mon[16639]: 5.9 scrub ok 2024-09-02T10:16:52.703 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:52.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:52.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:52.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:52.831 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:52.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:52.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:52.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:52.942 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:16:52.942 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:16:52.954 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:52.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:52.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:53.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:53.083 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:53.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:53.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:53.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:53.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:53.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:53.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:53.328 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:16:53.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:16:53.349 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:53.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:53.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:53.352 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:16:53.352 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:16:53.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:16:53.385 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:16:53.422 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:16:53.423 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:16:53.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:53 smithi151 ceph-mon[16639]: 5.3 scrub starts 2024-09-02T10:16:53.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:53.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:16:53.454 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:16:53.474 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:53.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:53 smithi187 ceph-mon[20072]: 5.3 scrub starts 2024-09-02T10:16:53.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:53.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:53 smithi162 ceph-mon[20279]: 5.3 scrub starts 2024-09-02T10:16:53.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:53.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:53.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:53.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:53.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:53.833 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:53.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:53.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:53.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:53.989 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:53.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:54.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:54.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:16:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:54.123 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:54.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:54.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:54.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:54.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:54.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:54.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:54.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:54.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:54.403 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:16:54.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:54.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:54.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:54.502 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:16:54.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:54.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:54.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:54.658 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:16:54.658 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:16:54.659 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: 5.5 scrub ok 2024-09-02T10:16:54.659 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:16:54.659 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:54.659 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: pgmap v1151: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-02T10:16:54.659 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: osdmap e495: 12 total, 12 up, 10 in 2024-09-02T10:16:54.660 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3973107904' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:54.660 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:54.660 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:16:54.660 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:54 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:16:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:16:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:16:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: 5.5 scrub ok 2024-09-02T10:16:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:16:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: pgmap v1151: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-02T10:16:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: osdmap e495: 12 total, 12 up, 10 in 2024-09-02T10:16:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3973107904' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:54.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:54.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:16:54.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:54 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:16:54.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:16:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:16:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: 5.5 scrub ok 2024-09-02T10:16:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:16:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: pgmap v1151: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 1.02k op/s 2024-09-02T10:16:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: osdmap e495: 12 total, 12 up, 10 in 2024-09-02T10:16:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3973107904' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:16:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:16:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:54 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:16:54.764 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:54.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:54.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:54.895 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:54.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:54.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:55.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:55.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:55.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:55.084 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:16:55.085 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:16:55.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:16:55.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:55.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:55.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:16:55.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:55.278 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:55.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:55.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:55.408 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:55.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:55.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:55.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:55.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:55.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:55.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:55.661 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:55 smithi162 ceph-mon[20279]: 5.3 scrub ok 2024-09-02T10:16:55.661 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:55 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:16:55.661 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:55 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:16:55.661 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:55 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:16:55.662 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:55 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:16:55.662 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:55 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:55.662 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:55 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:16:55.662 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:55 smithi162 ceph-mon[20279]: osdmap e496: 12 total, 12 up, 10 in 2024-09-02T10:16:55.662 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:55 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:16:55.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:55.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:55 smithi151 ceph-mon[16639]: 5.3 scrub ok 2024-09-02T10:16:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:55 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:16:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:55 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:16:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:55 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:16:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:55 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:16:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:55 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:55 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:16:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:55 smithi151 ceph-mon[16639]: osdmap e496: 12 total, 12 up, 10 in 2024-09-02T10:16:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:55 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:16:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:55 smithi187 ceph-mon[20072]: 5.3 scrub ok 2024-09-02T10:16:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:55 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:16:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:55 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:16:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:55 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:16:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:55 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:16:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:55 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:16:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:55 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:16:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:55 smithi187 ceph-mon[20072]: osdmap e496: 12 total, 12 up, 10 in 2024-09-02T10:16:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:55 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:16:55.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:55.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:16:55.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:55.789 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:55.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:55.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:55.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:55.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:56.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:56.048 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:56.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:56.111 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:16:56.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:56.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:56.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:56.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:56.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:16:56.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:56.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:56.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:56.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:56.455 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:56.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:56.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:56 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:16:56.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:56 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:16:56.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:56 smithi162 ceph-mon[20279]: pgmap v1154: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 836 op/s 2024-09-02T10:16:56.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:56 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:16:56.555 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:56 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:16:56.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:56.585 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:56.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:56.607 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:56.684 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:16:56.685 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:16:56.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:56.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:56 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:16:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:56 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:16:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:56 smithi151 ceph-mon[16639]: pgmap v1154: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 836 op/s 2024-09-02T10:16:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:56 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:16:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:56 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:16:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:56.717 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:56.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:56 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:16:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:56 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:16:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:56 smithi187 ceph-mon[20072]: pgmap v1154: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.4 GiB data, 17 GiB used, 877 GiB / 894 GiB avail; 0 B/s wr, 836 op/s 2024-09-02T10:16:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:56 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:16:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:56 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:16:56.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:56.842 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:56.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:56.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:56.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:56.970 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:56.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:57.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:16:57.096 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:57.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:57.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:16:57.160 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:16:57.160 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:16:57.176 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:16:57.177 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:16:57.198 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:57.199 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:16:57.199 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:16:57.227 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:16:57.228 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:16:57.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:57.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:57.254 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:16:57.254 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:16:57.281 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:16:57.281 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:16:57.308 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:16:57.308 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:16:57.331 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:16:57.331 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:16:57.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:57.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:16:57.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:16:57.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:16:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:57.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:57.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:57.380 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:16:57.381 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:16:57.405 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:16:57.406 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:16:57.432 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:16:57.432 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:16:57.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:16:57.461 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:57.485 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:57.485 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:57.586 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:57 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:16:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:57 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:16:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:57 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:16:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:57 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:57 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:16:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:57 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:16:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:57 smithi162 ceph-mon[20279]: osdmap e497: 12 total, 12 up, 10 in 2024-09-02T10:16:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2471185496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:57 smithi162 ceph-mon[20279]: 5.2 scrub starts 2024-09-02T10:16:57.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:57.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:57.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:57.671 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:16:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:57 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:16:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:57 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:16:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:57 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:16:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:57 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:57 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:16:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:57 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:16:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:57 smithi151 ceph-mon[16639]: osdmap e497: 12 total, 12 up, 10 in 2024-09-02T10:16:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2471185496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:57 smithi151 ceph-mon[16639]: 5.2 scrub starts 2024-09-02T10:16:57.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:57.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:57.746 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:57 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:16:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:57 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:16:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:57 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:16:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:57 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:57 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:16:57.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:57 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:16:57.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:57 smithi187 ceph-mon[20072]: osdmap e497: 12 total, 12 up, 10 in 2024-09-02T10:16:57.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2471185496' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:16:57.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:57 smithi187 ceph-mon[20072]: 5.2 scrub starts 2024-09-02T10:16:57.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:57.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:57.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:57.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:16:57.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:57.997 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:16:58.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:16:58.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:58.125 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:16:58.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:58.237 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:16:58.237 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:16:58.237 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-02T10:16:58.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:16:58.256 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:58.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:16:58.333 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.24976.0:75424 3.a 3:58480fd1:::benchmark_data_smithi187_59989_object75423:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 597.94021333000001, 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22582144000000001, 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.334 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.24976", 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75424 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.335 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359598+0000", 2024-09-02T10:16:58.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359600+0000", 2024-09-02T10:16:58.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3820000000000001e-06 2024-09-02T10:16:58.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359629+0000", 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8775000000000001e-05 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359640+0000", 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1151e-05 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359648+0000", 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0620000000000001e-06 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359688+0000", 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9882000000000002e-05 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359791+0000", 2024-09-02T10:16:58.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010327699999999999 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.359876+0000", 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5418000000000001e-05 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.361865+0000", 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019887289999999998 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362104+0000", 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023905199999999999 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585329+0000", 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223224432 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585374+0000", 2024-09-02T10:16:58.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4972999999999998e-05 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585420+0000", 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6307000000000002e-05 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.341 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.24976.0:75424 3.a e253/226 ondisk, result = 0)", 2024-09-02T10:16:58.342 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:16:58.342 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 597.93750779000004, 2024-09-02T10:16:58.342 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22312427800000001, 2024-09-02T10:16:58.342 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.342 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.342 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:16:58.342 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.342 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:16:58.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.343 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.343 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.343 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362304+0000", 2024-09-02T10:16:58.343 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.343 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362305+0000", 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9100000000000003e-07 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.362309+0000", 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2899999999999996e-06 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585259+0000", 2024-09-02T10:16:58.344 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22295003499999999 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585266+0000", 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6460000000000004e-06 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585311+0000", 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5802e-05 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.345 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585321+0000", 2024-09-02T10:16:58.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3090000000000001e-06 2024-09-02T10:16:58.346 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.346 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.346 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.346 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:07:00.585428+0000", 2024-09-02T10:16:58.346 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010750499999999999 2024-09-02T10:16:58.346 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.346 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.346 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:155062 3.2 e282/226 3:4fe18cc9:::benchmark_data_smithi187_59989_object45519:head v 282'9762, mlcod=282'9758)", 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 513.82330443299998, 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29158908, 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.347 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:16:58.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.348 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.348 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476507+0000", 2024-09-02T10:16:58.348 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.348 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.348 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476508+0000", 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1500000000000003e-07 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.476514+0000", 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6559999999999997e-06 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.763870+0000", 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28735592300000001 2024-09-02T10:16:58.349 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.763885+0000", 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4823e-05 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.764037+0000", 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015244400000000001 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.350 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.764077+0000", 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0371999999999999e-05 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.768047+0000", 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039701650000000003 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:08:24.768096+0000", 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8881999999999997e-05 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.351 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.352 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.352 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.352 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:181206 3.3 e291/226 3:c5ec1ffd:::benchmark_data_smithi187_59989_object71663:head v 291'11456, mlcod=291'11454)", 2024-09-02T10:16:58.352 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:16:58.352 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 463.29360874600002, 2024-09-02T10:16:58.352 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37650431099999998, 2024-09-02T10:16:58.352 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.352 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.352 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.352 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.353 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006204+0000", 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6000000000000002e-07 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006211+0000", 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7880000000000001e-06 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.354 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377595+0000", 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.371383927 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377601+0000", 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0769999999999996e-06 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377649+0000", 2024-09-02T10:16:58.355 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8368000000000002e-05 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377669+0000", 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0258000000000002e-05 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.382663+0000", 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049941880000000001 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.356 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.382707+0000", 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3844999999999998e-05 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3855 4.a 4:57b917f5:::benchmark_data_smithi187_76542_object3854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 352.39049502, 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22371366500000001, 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.357 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3855 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.358 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:16:58.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.359 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.359 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909327+0000", 2024-09-02T10:16:58.359 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9559999999999992e-06 2024-09-02T10:16:58.359 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.359 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909360+0000", 2024-09-02T10:16:58.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3556999999999997e-05 2024-09-02T10:16:58.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.360 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.360 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909377+0000", 2024-09-02T10:16:58.360 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6283e-05 2024-09-02T10:16:58.360 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.360 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909385+0000", 2024-09-02T10:16:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4030000000000007e-06 2024-09-02T10:16:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.361 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909444+0000", 2024-09-02T10:16:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9203000000000002e-05 2024-09-02T10:16:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.362 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909571+0000", 2024-09-02T10:16:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127324 2024-09-02T10:16:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:16:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909715+0000", 2024-09-02T10:16:58.363 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143888 2024-09-02T10:16:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:16:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.910145+0000", 2024-09-02T10:16:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042936599999999998 2024-09-02T10:16:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.364 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.917291+0000", 2024-09-02T10:16:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071467290000000001 2024-09-02T10:16:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132953+0000", 2024-09-02T10:16:58.365 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.215661517 2024-09-02T10:16:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132989+0000", 2024-09-02T10:16:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6123e-05 2024-09-02T10:16:58.366 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.133030+0000", 2024-09-02T10:16:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1316000000000002e-05 2024-09-02T10:16:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.367 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3871 4.a 4:530be978:::benchmark_data_smithi187_76542_object3870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:16:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:16:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 352.38371921300001, 2024-09-02T10:16:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21808112499999999, 2024-09-02T10:16:58.368 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3871 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.369 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1899999999999998e-07 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916113+0000", 2024-09-02T10:16:58.370 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9800999999999999e-05 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916118+0000", 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916121+0000", 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3330000000000001e-06 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.371 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916168+0000", 2024-09-02T10:16:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7147000000000001e-05 2024-09-02T10:16:58.372 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916291+0000", 2024-09-02T10:16:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122934 2024-09-02T10:16:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.373 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:16:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916427+0000", 2024-09-02T10:16:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135337 2024-09-02T10:16:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:16:58.374 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916951+0000", 2024-09-02T10:16:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052393700000000004 2024-09-02T10:16:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924167+0000", 2024-09-02T10:16:58.375 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072167580000000002 2024-09-02T10:16:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.376 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134113+0000", 2024-09-02T10:16:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20994539000000001 2024-09-02T10:16:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134147+0000", 2024-09-02T10:16:58.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4149000000000001e-05 2024-09-02T10:16:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134174+0000", 2024-09-02T10:16:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6798999999999999e-05 2024-09-02T10:16:58.378 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3887 4.a 4:5466a52d:::benchmark_data_smithi187_76542_object3886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:16:58.379 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:16:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 352.37806092300002, 2024-09-02T10:16:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21253008300000001, 2024-09-02T10:16:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.380 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:16:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:16:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3887 2024-09-02T10:16:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:16:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:16:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921752+0000", 2024-09-02T10:16:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4160000000000001e-06 2024-09-02T10:16:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921813+0000", 2024-09-02T10:16:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0420000000000001e-05 2024-09-02T10:16:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921822+0000", 2024-09-02T10:16:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4299999999999995e-06 2024-09-02T10:16:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921827+0000", 2024-09-02T10:16:58.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9819999999999999e-06 2024-09-02T10:16:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921867+0000", 2024-09-02T10:16:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0071000000000003e-05 2024-09-02T10:16:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921944+0000", 2024-09-02T10:16:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6533999999999996e-05 2024-09-02T10:16:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:16:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922025+0000", 2024-09-02T10:16:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1601999999999996e-05 2024-09-02T10:16:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:16:58.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922360+0000", 2024-09-02T10:16:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033462200000000002 2024-09-02T10:16:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924931+0000", 2024-09-02T10:16:58.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025713160000000001 2024-09-02T10:16:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134215+0000", 2024-09-02T10:16:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209284054 2024-09-02T10:16:58.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134247+0000", 2024-09-02T10:16:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2147999999999998e-05 2024-09-02T10:16:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134281+0000", 2024-09-02T10:16:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3488000000000002e-05 2024-09-02T10:16:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.393 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3898 4.a 4:55cc6e50:::benchmark_data_smithi187_76542_object3897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:16:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:16:58.394 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 352.37306499099998, 2024-09-02T10:16:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20761331199999999, 2024-09-02T10:16:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.395 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:16:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:16:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3898 2024-09-02T10:16:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.396 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:16:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:16:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926748+0000", 2024-09-02T10:16:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6599999999999994e-07 2024-09-02T10:16:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926786+0000", 2024-09-02T10:16:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8506e-05 2024-09-02T10:16:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.400 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926798+0000", 2024-09-02T10:16:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.221e-05 2024-09-02T10:16:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926804+0000", 2024-09-02T10:16:58.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3140000000000003e-06 2024-09-02T10:16:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926853+0000", 2024-09-02T10:16:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8804999999999998e-05 2024-09-02T10:16:58.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926965+0000", 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112256 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927110+0000", 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014520099999999999 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927488+0000", 2024-09-02T10:16:58.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000378332 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929483+0000", 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019947670000000002 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134309+0000", 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20482557300000001 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134337+0000", 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8334999999999999e-05 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134360+0000", 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3047000000000001e-05 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.405 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3901 4.a 4:53342710:::benchmark_data_smithi187_76542_object3900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 352.371236899, 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20590491499999999, 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3901 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928576+0000", 2024-09-02T10:16:58.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1699999999999997e-07 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928631+0000", 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5276999999999999e-05 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928642+0000", 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1195999999999999e-05 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928649+0000", 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.669e-06 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928691+0000", 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1868e-05 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928778+0000", 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7416999999999993e-05 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928866+0000", 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8471e-05 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929233+0000", 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036694499999999998 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.942995+0000", 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013761572999999999 2024-09-02T10:16:58.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134423+0000", 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19142793799999999 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134461+0000", 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7839000000000001e-05 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134480+0000", 2024-09-02T10:16:58.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9004999999999998e-05 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:6012 4.5 4:a5fa00f8:::benchmark_data_smithi187_76542_object6011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 351.03680389700003, 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209785581, 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.412 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6012 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:16:58.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263009+0000", 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0139999999999999e-06 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263120+0000", 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011099000000000001 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263134+0000", 2024-09-02T10:16:58.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4467999999999999e-05 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263141+0000", 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9820000000000002e-06 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263387+0000", 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024598300000000002 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263500+0000", 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113174 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263599+0000", 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8974999999999993e-05 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264127+0000", 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052785799999999997 2024-09-02T10:16:58.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264358+0000", 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023054600000000001 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472691+0000", 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20833321599999999 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472750+0000", 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8897000000000002e-05 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472793+0000", 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3477999999999999e-05 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:24533 4.c e335/333 4:3a0221c2:::benchmark_data_smithi187_76542_object24532:head v 335'1521, mlcod=335'1517)", 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 333.28877084499999, 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212837002, 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011042+0000", 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1650000000000001e-06 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011129+0000", 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7443999999999998e-05 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222953+0000", 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21182389400000001 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222960+0000", 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8569999999999997e-06 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223010+0000", 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0182000000000002e-05 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223032+0000", 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1699999999999999e-05 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223838+0000", 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080572200000000004 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223878+0000", 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0037999999999999e-05 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:25165 4.c e336/333 4:3c3f6316:::benchmark_data_smithi187_76542_object25164:head v 336'1563, mlcod=335'1560)", 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 332.654541295, 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204647308, 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645271+0000", 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9999999999999996e-07 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645443+0000", 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017179700000000001 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848893+0000", 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203449884 2024-09-02T10:16:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848900+0000", 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8360000000000003e-06 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848961+0000", 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1163999999999999e-05 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848982+0000", 2024-09-02T10:16:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1403000000000002e-05 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849888+0000", 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000905469 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849918+0000", 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9855000000000001e-05 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:52446 4.15 e362/353 4:acabd05d:::benchmark_data_smithi187_76542_object52445:head v 362'2522, mlcod=362'2519)", 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 299.60641193499998, 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.220391959, 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9599999999999999e-07 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693457+0000", 2024-09-02T10:16:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6582000000000001e-05 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912812+0000", 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21935476500000001 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912819+0000", 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4989999999999997e-06 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912871+0000", 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1388000000000003e-05 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912887+0000", 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5911999999999999e-05 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.913759+0000", 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087215899999999996 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.913792+0000", 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2957999999999999e-05 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:58518 4.1 e366/341 4:8490730e:::benchmark_data_smithi187_76542_object58517:head v 366'2807, mlcod=366'2804)", 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 293.93468370800002, 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209866793, 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365129+0000", 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7100000000000001e-07 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365181+0000", 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1816e-05 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573805+0000", 2024-09-02T10:16:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.2086249 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573812+0000", 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1349999999999997e-06 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573866+0000", 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4373000000000003e-05 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573888+0000", 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2008999999999999e-05 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.574955+0000", 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010673519999999999 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.574995+0000", 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9536999999999998e-05 2024-09-02T10:16:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:75251 4.11 e371/356 4:8ba31329:::benchmark_data_smithi187_76542_object75250:head v 371'3396, mlcod=371'3394)", 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 280.63435468799997, 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37157874099999999, 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665458+0000", 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9400000000000004e-07 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665521+0000", 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2685e-05 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035523+0000", 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37000295799999999 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035542+0000", 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8692999999999999e-05 2024-09-02T10:16:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035709+0000", 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016700999999999999 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035745+0000", 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5913e-05 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.037004+0000", 2024-09-02T10:16:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001258459 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.037036+0000", 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2228999999999999e-05 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:90191 4.15 e375/353 4:afce227e:::benchmark_data_smithi187_76542_object90190:head v 375'3680, mlcod=375'3677)", 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 267.20156514199999, 2024-09-02T10:16:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20444974699999999, 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9299999999999997e-07 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098273+0000", 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5378999999999999e-05 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301201+0000", 2024-09-02T10:16:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20292787800000001 2024-09-02T10:16:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301213+0000", 2024-09-02T10:16:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860000000000001e-05 2024-09-02T10:16:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:58.445 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301360+0000", 2024-09-02T10:16:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014643600000000001 2024-09-02T10:16:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301383+0000", 2024-09-02T10:16:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2782999999999999e-05 2024-09-02T10:16:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302662+0000", 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012796470000000001 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302696+0000", 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3970999999999997e-05 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126679 4.1 e392/341 4:86205e4c:::benchmark_data_smithi187_76542_object34343:head v 392'4951, mlcod=392'4948)", 2024-09-02T10:16:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 222.83200487100001, 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44507702900000001, 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467808+0000", 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.277e-06 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467816+0000", 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3690000000000007e-06 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910880+0000", 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44306368899999998 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910889+0000", 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0380000000000005e-06 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910948+0000", 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8498000000000001e-05 2024-09-02T10:16:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910970+0000", 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2580000000000001e-05 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912839+0000", 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018688120000000001 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912884+0000", 2024-09-02T10:16:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4765999999999998e-05 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126802 4.1 e392/341 4:81a050df:::benchmark_data_smithi187_76542_object34466:head v 392'4952, mlcod=392'4949)", 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 222.38896149499999, 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25345677799999999, 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910851+0000", 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0690000000000001e-06 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910859+0000", 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.233e-06 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162387+0000", 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25152825099999998 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162402+0000", 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4905999999999999e-05 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162538+0000", 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013591800000000001 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162581+0000", 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2991000000000002e-05 2024-09-02T10:16:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164270+0000", 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016893419999999999 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164307+0000", 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7067999999999998e-05 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:64011 5.9 e465/431 5:9035e690:::benchmark_data_smithi187_89405_object64010:head v 465'4071, mlcod=464'4068)", 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 81.082542818999997, 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.200897045, 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1100000000000002e-07 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217293+0000", 2024-09-02T10:16:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3326999999999999e-05 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416283+0000", 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19899007799999999 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416287+0000", 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3529999999999997e-06 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416337+0000", 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0272000000000003e-05 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416360+0000", 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3147e-05 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.418133+0000", 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001772605 2024-09-02T10:16:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.418166+0000", 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2951999999999999e-05 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:86619 5.9 e473/431 5:9bb5eff1:::benchmark_data_smithi187_89405_object86618:head v 473'5442, mlcod=473'5439)", 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 59.760071187000001, 2024-09-02T10:16:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21915551599999999, 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:16:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.09e-07 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539774+0000", 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2465999999999997e-05 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.755999+0000", 2024-09-02T10:16:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.216224731 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756004+0000", 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9009999999999999e-06 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756086+0000", 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1767000000000005e-05 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756100+0000", 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4285e-05 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.758849+0000", 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027488349999999998 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.758896+0000", 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6721999999999998e-05 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:16:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:16:58.466 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:16:58.490 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:58.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:58.533 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:58 smithi151 ceph-mon[16639]: pgmap v1156: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-09-02T10:16:58.533 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:58 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:16:58.533 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:58 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:16:58.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:16:58.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:58.623 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:58.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:16:58.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:58.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:58.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:58 smithi187 ceph-mon[20072]: pgmap v1156: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-09-02T10:16:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:58 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:16:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:58 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:16:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:58 smithi162 ceph-mon[20279]: pgmap v1156: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.2 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 1.17k op/s 2024-09-02T10:16:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:58 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:16:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:58 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:16:58.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:58.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:58.884 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:58.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:58.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:59.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:59.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:59.013 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:16:59.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:16:59.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:59.136 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:16:59.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:16:59.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:59.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:16:59.362 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:16:59.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:16:59.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:59.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:16:59.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:16:59.514 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:59.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:59.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:16:59.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:16:59.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:59.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:59.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:16:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:59 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:16:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:59 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:16:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:59 smithi151 ceph-mon[16639]: 5.2 scrub ok 2024-09-02T10:16:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:59 smithi151 ceph-mon[16639]: osdmap e498: 12 total, 12 up, 10 in 2024-09-02T10:16:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:16:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1255872886' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:16:59.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:16:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:59 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:16:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:59 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:16:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:59 smithi187 ceph-mon[20072]: 5.2 scrub ok 2024-09-02T10:16:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:59 smithi187 ceph-mon[20072]: osdmap e498: 12 total, 12 up, 10 in 2024-09-02T10:16:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:16:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1255872886' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:16:59.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:16:59.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:59.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:16:59.778 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:16:59.778 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:16:59.778 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 498, 2024-09-02T10:16:59.778 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:16:59.778 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:16:59.778 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:16:59.013611+0000", 2024-09-02T10:16:59.778 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:16:59.778 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:14:14.425747+0000", 2024-09-02T10:16:59.778 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5799936, 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:16:59.779 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 5, 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:16:59.780 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-09-02T10:16:59.781 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "264", 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "241", 2024-09-02T10:16:59.782 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:16:59.783 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:16:59.784 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:16:59.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 3.1099998950958252, 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 3.1099998950958252, 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.82999998331069946, 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.5799999237060547, 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.5799999237060547, 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 5, 2024-09-02T10:16:59.786 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_3", 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:14:37.792221+0000", 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:16:59.787 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.788 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "435", 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:16:59.789 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:16:59.790 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:16:59.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.2699999809265137, 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.2699999809265137, 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.82999998331069946, 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:16:59.792 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 489, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.793 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.794 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.795 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.795 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.795 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:16:59.795 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:16:59.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:16:59.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:16:59.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.798 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 462, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.799 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.800 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.801 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.802 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.803 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:16:59.804 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 349, 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.805 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.806 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.807 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.808 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:16:59.809 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:16:59.810 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 448, 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:16:59.812 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.813 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.814 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:16:59.815 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.816 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 378, 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.817 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.818 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.819 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.820 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.821 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 445, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.822 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.823 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.824 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:16:59.825 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:16:59.826 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.827 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 448, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.828 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.829 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.830 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.831 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.832 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.832 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.832 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.832 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:16:59.832 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:16:59.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.833 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.833 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:16:59.833 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:16:59.833 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.833 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.833 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.834 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 462, 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.835 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.836 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.837 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.838 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.839 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.840 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 431, 2024-09-02T10:16:59.841 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:16:59.842 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:16:59.843 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.844 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.845 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.846 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:16:59.847 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 431, 2024-09-02T10:16:59.848 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.848 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.848 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.848 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.848 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.848 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:16:59.848 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:16:59.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:16:59.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:16:59.850 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.851 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:16:59.852 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:16:59.853 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:16:59.854 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.854 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.854 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.854 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:16:59.854 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 462, 2024-09-02T10:16:59.854 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.854 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.854 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.854 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.855 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.856 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.857 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.857 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.857 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.857 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:16:59.857 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:16:59.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.858 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:16:59.859 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:16:59.860 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 490, 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:16:59.861 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.862 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:16:59.864 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.865 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:16:59.866 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:15:47.021645+0000", 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.867 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:05.044139+0000", 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 65536, 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:44.575682+0000", 2024-09-02T10:16:59.868 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:17.872826+0000", 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.869 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 10656, 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:55.820991+0000", 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.870 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:23.787835+0000", 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:16:59.871 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:53.300024+0000", 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:18.160682+0000", 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:38.383719+0000", 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:16:59.873 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:55.729087+0000", 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.874 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:34.312272+0000", 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:16:59.875 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:16:48.832346+0000", 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.1e", 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: 11, 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: 9 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.876 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [ 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.7", 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: "mappings": [ 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: "from": 10, 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: "to": 11 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [], 2024-09-02T10:16:59.877 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.878 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:16:59.879 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:16:59.880 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:16:59.880 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:16:59.880 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:16:59.880 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:16:59.880 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:16:59.880 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:16:59.880 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:16:59.880 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:16:59.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:16:59.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:59 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:16:59.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:59 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:16:59.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:59 smithi162 ceph-mon[20279]: 5.2 scrub ok 2024-09-02T10:16:59.881 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:59 smithi162 ceph-mon[20279]: osdmap e498: 12 total, 12 up, 10 in 2024-09-02T10:16:59.882 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:16:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1255872886' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:16:59.906 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:16:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:16:59.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:00.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:00.032 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:17:00.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:00.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:00.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:00.161 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:00.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:00.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:00.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:00.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:00.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:00.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:17:00.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:00.418 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:00.418 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:00.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:00.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:00.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:00.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:00.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:00.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:00 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:17:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:00 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:17:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:00 smithi151 ceph-mon[16639]: pgmap v1158: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 883 op/s 2024-09-02T10:17:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:00 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:17:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:00 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:17:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3806596377' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:17:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:00 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1255872886' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:17:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:00 smithi151 ceph-mon[16639]: osdmap e499: 12 total, 12 up, 10 in 2024-09-02T10:17:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:00 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:17:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:00 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:17:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:00 smithi187 ceph-mon[20072]: pgmap v1158: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 883 op/s 2024-09-02T10:17:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:00 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:17:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:00 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:17:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3806596377' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:17:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:00 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1255872886' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:17:00.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:00 smithi187 ceph-mon[20072]: osdmap e499: 12 total, 12 up, 10 in 2024-09-02T10:17:00.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:00.807 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:00.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:00.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:00 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:17:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:00 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:17:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:00 smithi162 ceph-mon[20279]: pgmap v1158: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 883 op/s 2024-09-02T10:17:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:00 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:17:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:00 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:17:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3806596377' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:17:00.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:00 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:00.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1255872886' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:17:00.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:00 smithi162 ceph-mon[20279]: osdmap e499: 12 total, 12 up, 10 in 2024-09-02T10:17:00.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:00.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:00.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:00.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:01.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:01.060 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:01.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:01.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:01.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:17:01.189 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:01.190 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:01.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:01.326 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:01.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:01.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:01.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:01.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:01.541 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:17:01.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:01.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:01.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:01.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:01.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:01.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:01.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:01.848 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:01.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:01.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:01.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:01.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:01.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:02.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:02.088 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:17:02.088 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:17:02.088 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:17:02.088 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:17:02.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:02.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:02.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:02.232 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:02.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:02.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:02.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:02.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:02.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:02.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:02.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:02.483 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:02.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:02.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:02.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:02.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:02 smithi151 ceph-mon[16639]: pgmap v1160: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 816 op/s 2024-09-02T10:17:02.708 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:02.711 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:02 smithi162 ceph-mon[20279]: pgmap v1160: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 816 op/s 2024-09-02T10:17:02.733 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:02.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:02.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:02 smithi187 ceph-mon[20072]: pgmap v1160: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.1 GiB data, 16 GiB used, 878 GiB / 894 GiB avail; 0 B/s wr, 816 op/s 2024-09-02T10:17:02.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:17:02.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:02.861 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:02.862 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:02.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:02.994 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:02.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:03.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:03.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:03.121 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:03.122 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:03.127 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:17:03.206 INFO:tasks.thrashosds.thrasher:j is {'epoch': 498, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:16:59.013611+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:14:14.425747+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 5, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.109999895095825, 'score_stable': 3.109999895095825, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5799999237060547, 'raw_score_stable': 2.5799999237060547, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-02T10:14:37.792221+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '435', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 489, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 462, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 349, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 448, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 378, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 445, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 448, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 462, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 431, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 431, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 462, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 490, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:15:47.021645+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:16:05.044139+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-02T10:16:44.575682+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:16:17.872826+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 10656, 'last_purged_snaps_scrub': '2024-09-02T10:16:55.820991+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:16:23.787835+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:16:53.300024+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:16:18.160682+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:16:38.383719+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:16:55.729087+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:16:34.312272+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:16:48.832346+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.1e', 'osds': [11, 10, 9]}], 'pg_upmap_items': [{'pgid': '1.7', 'mappings': [{'from': 10, 'to': 11}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:17:03.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:17:03.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:03.246 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:03.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:03.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:03.301 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:03.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:03.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:03.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:03.479 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:03.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:03.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:03.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:03.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:03.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:03.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:03.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:03.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:03.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:03.789 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:17:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:17:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:17:03.789 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:17:03.790 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:17:03.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:03.893 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:03.893 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:03.931 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:03.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:03.996 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:17:03.997 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:17:04.025 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:17:04.026 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:17:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:04.027 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:04.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:04.049 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:17:04.049 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:17:04.073 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:17:04.074 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:17:04.100 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:17:04.100 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:17:04.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:04.131 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:17:04.132 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:17:04.157 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:04.158 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:04.161 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:17:04.161 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:17:04.188 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:17:04.188 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:17:04.217 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:17:04.217 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:17:04.248 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:17:04.248 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:17:04.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:04.273 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:17:04.273 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:17:04.283 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:04.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:04.304 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:17:04.304 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:17:04.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:04.410 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:04.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:04.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:17:04.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:04.540 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:04.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:04.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:04.666 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:04.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:04.677 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:17:04.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:04 smithi151 ceph-mon[16639]: pgmap v1161: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-09-02T10:17:04.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:04 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2136734105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:04.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:04 smithi151 ceph-mon[16639]: osdmap e500: 12 total, 12 up, 10 in 2024-09-02T10:17:04.679 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:17:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:04 smithi187 ceph-mon[20072]: pgmap v1161: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-09-02T10:17:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:04 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2136734105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:04 smithi187 ceph-mon[20072]: osdmap e500: 12 total, 12 up, 10 in 2024-09-02T10:17:04.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:17:04.766 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:17:04.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:04.787 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:04.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:04.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:04 smithi162 ceph-mon[20279]: pgmap v1161: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.9 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.14k op/s 2024-09-02T10:17:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:04 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2136734105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:04 smithi162 ceph-mon[20279]: osdmap e500: 12 total, 12 up, 10 in 2024-09-02T10:17:04.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:04.914 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:04.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:04.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:05.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:05.046 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:05.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:05.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:05.172 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:05.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:05.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:05.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:05.303 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:05.322 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:17:05.322 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:17:05.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:05.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:05.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:05.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:05.556 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:05.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:05 smithi162 ceph-mon[20279]: from='client.16752 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:17:05.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:05.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:05 smithi151 ceph-mon[16639]: from='client.16752 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:17:05.692 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:05.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:05.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:05 smithi187 ceph-mon[20072]: from='client.16752 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:17:05.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:05.814 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1162,"stamp":"2024-09-02T10:17:04.017891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4120969784,"num_objects":62875,"num_object_clones":0,"num_object_copies":188625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62875,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144333,"num_write_kb":6630529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5711,"num_bytes_recovered":374809664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147185,"ondisk_log_size":147185,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":15981932,"kb_used_data":12220084,"kb_used_omap":15,"kb_used_meta":3761584,"kb_avail":921469588,"statfs":{"total":959950356480,"available":943584858112,"internally_reserved":0,"allocated":12513366016,"data_stored":12510120512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15951,"internal_metadata":3851862449},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,3,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-394788864,"num_objects":-6024,"num_object_clones":0,"num_object_copies":-18072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6024,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.710948"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":325,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.040970+0000","last_change":"2024-09-02T10:14:36.326372+0000","last_active":"2024-09-02T10:17:00.040970+0000","last_peered":"2024-09-02T10:17:00.040970+0000","last_clean":"2024-09-02T10:17:00.040970+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:17:00.040970+0000","last_undegraded":"2024-09-02T10:17:00.040970+0000","last_fullsized":"2024-09-02T10:17:00.040970+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:36.326315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:36.326315+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":141,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.042029+0000","last_change":"2024-09-02T10:16:54.645685+0000","last_active":"2024-09-02T10:17:00.042029+0000","last_peered":"2024-09-02T10:17:00.042029+0000","last_clean":"2024-09-02T10:17:00.042029+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:17:00.042029+0000","last_undegraded":"2024-09-02T10:17:00.042029+0000","last_fullsized":"2024-09-02T10:17:00.042029+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:54.645614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:54.645614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:18.879851+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":589,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-09-02T10:16:59.021933+0000","last_change":"2024-09-02T10:16:48.940363+0000","last_active":"2024-09-02T10:16:59.021933+0000","last_peered":"2024-09-02T10:16:59.021933+0000","last_clean":"2024-09-02T10:16:59.021933+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:16:59.021933+0000","last_undegraded":"2024-09-02T10:16:59.021933+0000","last_fullsized":"2024-09-02T10:16:59.021933+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:48.940301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:48.940301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:14.457035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":589,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-09-02T10:16:59.022031+0000","last_change":"2024-09-02T10:16:53.996217+0000","last_active":"2024-09-02T10:16:59.022031+0000","last_peered":"2024-09-02T10:16:59.022031+0000","last_clean":"2024-09-02T10:16:59.022031+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:16:59.022031+0000","last_undegraded":"2024-09-02T10:16:59.022031+0000","last_fullsized":"2024-09-02T10:16:59.022031+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:53.996155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:53.996155+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:15.426032+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":512,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.039850+0000","last_change":"2024-09-02T10:16:54.403148+0000","last_active":"2024-09-02T10:17:00.039850+0000","last_peered":"2024-09-02T10:17:00.039850+0000","last_clean":"2024-09-02T10:17:00.039850+0000","last_became_active":"2024-09-02T10:15:35.201887+0000","last_became_peered":"2024-09-02T10:15:35.201887+0000","last_unstale":"2024-09-02T10:17:00.039850+0000","last_undegraded":"2024-09-02T10:17:00.039850+0000","last_fullsized":"2024-09-02T10:17:00.039850+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:54.403086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:54.403086+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:06.479917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":440,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.042176+0000","last_change":"2024-09-02T10:16:52.570499+0000","last_active":"2024-09-02T10:17:00.042176+0000","last_peered":"2024-09-02T10:17:00.042176+0000","last_clean":"2024-09-02T10:17:00.042176+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:17:00.042176+0000","last_undegraded":"2024-09-02T10:17:00.042176+0000","last_fullsized":"2024-09-02T10:17:00.042176+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:52.570412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:52.570412+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:54.267832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":956,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.061023+0000","last_change":"2024-09-02T10:16:23.306394+0000","last_active":"2024-09-02T10:17:00.061023+0000","last_peered":"2024-09-02T10:17:00.061023+0000","last_clean":"2024-09-02T10:17:00.061023+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:17:00.061023+0000","last_undegraded":"2024-09-02T10:17:00.061023+0000","last_fullsized":"2024-09-02T10:17:00.061023+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:23.306331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:23.306331+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:34.300255+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":596,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-09-02T10:16:59.022191+0000","last_change":"2024-09-02T10:16:10.994908+0000","last_active":"2024-09-02T10:16:59.022191+0000","last_peered":"2024-09-02T10:16:59.022191+0000","last_clean":"2024-09-02T10:16:59.022191+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:16:59.022191+0000","last_undegraded":"2024-09-02T10:16:59.022191+0000","last_fullsized":"2024-09-02T10:16:59.022191+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.994824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.994824+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:17.261476+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":606,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.040747+0000","last_change":"2024-09-02T10:16:58.636430+0000","last_active":"2024-09-02T10:17:00.040747+0000","last_peered":"2024-09-02T10:17:00.040747+0000","last_clean":"2024-09-02T10:17:00.040747+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:17:00.040747+0000","last_undegraded":"2024-09-02T10:17:00.040747+0000","last_fullsized":"2024-09-02T10:17:00.040747+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:58.636360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:58.636360+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:10.738687+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":574,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.039983+0000","last_change":"2024-09-02T10:16:14.103234+0000","last_active":"2024-09-02T10:17:00.039983+0000","last_peered":"2024-09-02T10:17:00.039983+0000","last_clean":"2024-09-02T10:17:00.039983+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:17:00.039983+0000","last_undegraded":"2024-09-02T10:17:00.039983+0000","last_fullsized":"2024-09-02T10:17:00.039983+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:14.103166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:14.103166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:33.443226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":425,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.039873+0000","last_change":"2024-09-02T10:16:11.152819+0000","last_active":"2024-09-02T10:17:00.039873+0000","last_peered":"2024-09-02T10:17:00.039873+0000","last_clean":"2024-09-02T10:17:00.039873+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:17:00.039873+0000","last_undegraded":"2024-09-02T10:17:00.039873+0000","last_fullsized":"2024-09-02T10:17:00.039873+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:11.152686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:11.152686+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:26.925270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":931,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.040452+0000","last_change":"2024-09-02T10:16:19.673642+0000","last_active":"2024-09-02T10:17:00.040452+0000","last_peered":"2024-09-02T10:17:00.040452+0000","last_clean":"2024-09-02T10:17:00.040452+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:17:00.040452+0000","last_undegraded":"2024-09-02T10:17:00.040452+0000","last_fullsized":"2024-09-02T10:17:00.040452+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:19.673572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:19.673572+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:33.436612+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":866,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.061378+0000","last_change":"2024-09-02T10:16:15.378306+0000","last_active":"2024-09-02T10:17:00.061378+0000","last_peered":"2024-09-02T10:17:00.061378+0000","last_clean":"2024-09-02T10:17:00.061378+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:17:00.061378+0000","last_undegraded":"2024-09-02T10:17:00.061378+0000","last_fullsized":"2024-09-02T10:17:00.061378+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:15.378230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:15.378230+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:18.040488+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":853,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.038832+0000","last_change":"2024-09-02T10:16:55.583563+0000","last_active":"2024-09-02T10:17:00.038832+0000","last_peered":"2024-09-02T10:17:00.038832+0000","last_clean":"2024-09-02T10:17:00.038832+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:17:00.038832+0000","last_undegraded":"2024-09-02T10:17:00.038832+0000","last_fullsized":"2024-09-02T10:17:00.038832+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:55.583506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:55.583506+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:06.173422+0000","scrub_duration":824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":844,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.061132+0000","last_change":"2024-09-02T10:16:54.366876+0000","last_active":"2024-09-02T10:17:00.061132+0000","last_peered":"2024-09-02T10:17:00.061132+0000","last_clean":"2024-09-02T10:17:00.061132+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:17:00.061132+0000","last_undegraded":"2024-09-02T10:17:00.061132+0000","last_fullsized":"2024-09-02T10:17:00.061132+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:54.366801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:54.366801+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:24.281988+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"499'8935","reported_seq":12707,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:03.519447+0000","last_change":"2024-09-02T10:16:17.226985+0000","last_active":"2024-09-02T10:17:03.519447+0000","last_peered":"2024-09-02T10:17:03.519447+0000","last_clean":"2024-09-02T10:17:03.519447+0000","last_became_active":"2024-09-02T10:15:24.141368+0000","last_became_peered":"2024-09-02T10:15:24.141368+0000","last_unstale":"2024-09-02T10:17:03.519447+0000","last_undegraded":"2024-09-02T10:17:03.519447+0000","last_fullsized":"2024-09-02T10:17:03.519447+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6399","last_scrub_stamp":"2024-09-02T10:16:17.226765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.226765+0000","objects_scrubbed":6342,"log_size":8935,"log_dups_size":0,"ondisk_log_size":8935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:37.316444+0000","scrub_duration":2531,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251985920,"num_objects":3845,"num_object_clones":0,"num_object_copies":11535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8935,"num_write_kb":408960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1769,"num_bytes_recovered":115933184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":931,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.061252+0000","last_change":"2024-09-02T10:16:59.313962+0000","last_active":"2024-09-02T10:17:00.061252+0000","last_peered":"2024-09-02T10:17:00.061252+0000","last_clean":"2024-09-02T10:17:00.061252+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:17:00.061252+0000","last_undegraded":"2024-09-02T10:17:00.061252+0000","last_fullsized":"2024-09-02T10:17:00.061252+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:59.313897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:59.313897+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:11.684071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"499'9043","reported_seq":9192,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:02.117083+0000","last_change":"2024-09-02T10:14:39.040351+0000","last_active":"2024-09-02T10:17:02.117083+0000","last_peered":"2024-09-02T10:17:02.117083+0000","last_clean":"2024-09-02T10:17:02.117083+0000","last_became_active":"2024-09-02T10:14:39.040025+0000","last_became_peered":"2024-09-02T10:14:39.040025+0000","last_unstale":"2024-09-02T10:17:02.117083+0000","last_undegraded":"2024-09-02T10:17:02.117083+0000","last_fullsized":"2024-09-02T10:17:02.117083+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":9043,"log_dups_size":0,"ondisk_log_size":9043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:05.205018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250478616,"num_objects":3823,"num_object_clones":0,"num_object_copies":11469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9043,"num_write_kb":411649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1046,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.039869+0000","last_change":"2024-09-02T10:16:22.269544+0000","last_active":"2024-09-02T10:17:00.039869+0000","last_peered":"2024-09-02T10:17:00.039869+0000","last_clean":"2024-09-02T10:17:00.039869+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:17:00.039869+0000","last_undegraded":"2024-09-02T10:17:00.039869+0000","last_fullsized":"2024-09-02T10:17:00.039869+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:22.269495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:22.269495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:47.553829+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"499'8958","reported_seq":9178,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:02.966963+0000","last_change":"2024-09-02T10:16:15.009837+0000","last_active":"2024-09-02T10:17:02.966963+0000","last_peered":"2024-09-02T10:17:02.966963+0000","last_clean":"2024-09-02T10:17:02.966963+0000","last_became_active":"2024-09-02T10:14:39.038164+0000","last_became_peered":"2024-09-02T10:14:39.038164+0000","last_unstale":"2024-09-02T10:17:02.966963+0000","last_undegraded":"2024-09-02T10:17:02.966963+0000","last_fullsized":"2024-09-02T10:17:02.966963+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6340","last_scrub_stamp":"2024-09-02T10:16:15.009487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:15.009487+0000","objects_scrubbed":6273,"log_size":8958,"log_dups_size":0,"ondisk_log_size":8958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:35.488395+0000","scrub_duration":2145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252837888,"num_objects":3858,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8958,"num_write_kb":410112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,0],"acting":[9,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":688,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.041299+0000","last_change":"2024-09-02T10:16:10.884551+0000","last_active":"2024-09-02T10:17:00.041299+0000","last_peered":"2024-09-02T10:17:00.041299+0000","last_clean":"2024-09-02T10:17:00.041299+0000","last_became_active":"2024-09-02T10:11:11.472480+0000","last_became_peered":"2024-09-02T10:11:11.472480+0000","last_unstale":"2024-09-02T10:17:00.041299+0000","last_undegraded":"2024-09-02T10:17:00.041299+0000","last_fullsized":"2024-09-02T10:17:00.041299+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:10.884481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:10.884481+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:22.734487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,0],"acting":[9,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"5.f","version":"499'9138","reported_seq":9356,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:02.072668+0000","last_change":"2024-09-02T10:16:26.551606+0000","last_active":"2024-09-02T10:17:02.072668+0000","last_peered":"2024-09-02T10:17:02.072668+0000","last_clean":"2024-09-02T10:17:02.072668+0000","last_became_active":"2024-09-02T10:14:39.037121+0000","last_became_peered":"2024-09-02T10:14:39.037121+0000","last_unstale":"2024-09-02T10:17:02.072668+0000","last_undegraded":"2024-09-02T10:17:02.072668+0000","last_fullsized":"2024-09-02T10:17:02.072668+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"484'7110","last_scrub_stamp":"2024-09-02T10:16:26.551442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:26.551442+0000","objects_scrubbed":6016,"log_size":9138,"log_dups_size":0,"ondisk_log_size":9138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:27.466417+0000","scrub_duration":1258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259915776,"num_objects":3966,"num_object_clones":0,"num_object_copies":11898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3966,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9138,"num_write_kb":419328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,11,10],"acting":[5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":92,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.040912+0000","last_change":"2024-09-02T10:16:58.302959+0000","last_active":"2024-09-02T10:17:00.040912+0000","last_peered":"2024-09-02T10:17:00.040912+0000","last_clean":"2024-09-02T10:17:00.040912+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:17:00.040912+0000","last_undegraded":"2024-09-02T10:17:00.040912+0000","last_fullsized":"2024-09-02T10:17:00.040912+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:58.302913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:58.302913+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:14.521625+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.0","version":"499'8891","reported_seq":9039,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:01.654020+0000","last_change":"2024-09-02T10:14:39.642395+0000","last_active":"2024-09-02T10:17:01.654020+0000","last_peered":"2024-09-02T10:17:01.654020+0000","last_clean":"2024-09-02T10:17:01.654020+0000","last_became_active":"2024-09-02T10:14:39.641995+0000","last_became_peered":"2024-09-02T10:14:39.641995+0000","last_unstale":"2024-09-02T10:17:01.654020+0000","last_undegraded":"2024-09-02T10:17:01.654020+0000","last_fullsized":"2024-09-02T10:17:01.654020+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":8891,"log_dups_size":0,"ondisk_log_size":8891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:06.545298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262340608,"num_objects":4003,"num_object_clones":0,"num_object_copies":12009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8891,"num_write_kb":412608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":993,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.041001+0000","last_change":"2024-09-02T10:15:29.195341+0000","last_active":"2024-09-02T10:17:00.041001+0000","last_peered":"2024-09-02T10:17:00.041001+0000","last_clean":"2024-09-02T10:17:00.041001+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:17:00.041001+0000","last_undegraded":"2024-09-02T10:17:00.041001+0000","last_fullsized":"2024-09-02T10:17:00.041001+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:12.277119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:12.277119+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"5.1","version":"499'9080","reported_seq":9296,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:03.920517+0000","last_change":"2024-09-02T10:16:22.056000+0000","last_active":"2024-09-02T10:17:03.920517+0000","last_peered":"2024-09-02T10:17:03.920517+0000","last_clean":"2024-09-02T10:17:03.920517+0000","last_became_active":"2024-09-02T10:14:39.034636+0000","last_became_peered":"2024-09-02T10:14:39.034636+0000","last_unstale":"2024-09-02T10:17:03.920517+0000","last_undegraded":"2024-09-02T10:17:03.920517+0000","last_fullsized":"2024-09-02T10:17:03.920517+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6741","last_scrub_stamp":"2024-09-02T10:16:22.055909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:22.055909+0000","objects_scrubbed":6225,"log_size":9080,"log_dups_size":0,"ondisk_log_size":9080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:24.765454+0000","scrub_duration":2093,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250085376,"num_objects":3816,"num_object_clones":0,"num_object_copies":11448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9080,"num_write_kb":412672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,9],"acting":[3,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":461,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.042301+0000","last_change":"2024-09-02T10:15:35.205684+0000","last_active":"2024-09-02T10:17:00.042301+0000","last_peered":"2024-09-02T10:17:00.042301+0000","last_clean":"2024-09-02T10:17:00.042301+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:17:00.042301+0000","last_undegraded":"2024-09-02T10:17:00.042301+0000","last_fullsized":"2024-09-02T10:17:00.042301+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:14:40.701773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:14:40.701773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.2","version":"499'8894","reported_seq":9146,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:02.195993+0000","last_change":"2024-09-02T10:16:58.440826+0000","last_active":"2024-09-02T10:17:02.195993+0000","last_peered":"2024-09-02T10:17:02.195993+0000","last_clean":"2024-09-02T10:17:02.195993+0000","last_became_active":"2024-09-02T10:14:39.037822+0000","last_became_peered":"2024-09-02T10:14:39.037822+0000","last_unstale":"2024-09-02T10:17:02.195993+0000","last_undegraded":"2024-09-02T10:17:02.195993+0000","last_fullsized":"2024-09-02T10:17:02.195993+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'8695","last_scrub_stamp":"2024-09-02T10:16:58.440528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:58.440528+0000","objects_scrubbed":4086,"log_size":8894,"log_dups_size":0,"ondisk_log_size":8894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:13.396036+0000","scrub_duration":1179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252837888,"num_objects":3858,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8894,"num_write_kb":408064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,10],"acting":[6,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":441,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.042396+0000","last_change":"2024-09-02T10:16:48.593277+0000","last_active":"2024-09-02T10:17:00.042396+0000","last_peered":"2024-09-02T10:17:00.042396+0000","last_clean":"2024-09-02T10:17:00.042396+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:17:00.042396+0000","last_undegraded":"2024-09-02T10:17:00.042396+0000","last_fullsized":"2024-09-02T10:17:00.042396+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:48.593208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:48.593208+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:49.220591+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.3","version":"499'9275","reported_seq":9477,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:02.794600+0000","last_change":"2024-09-02T10:16:53.975895+0000","last_active":"2024-09-02T10:17:02.794600+0000","last_peered":"2024-09-02T10:17:02.794600+0000","last_clean":"2024-09-02T10:17:02.794600+0000","last_became_active":"2024-09-02T10:14:39.035100+0000","last_became_peered":"2024-09-02T10:14:39.035100+0000","last_unstale":"2024-09-02T10:17:02.794600+0000","last_undegraded":"2024-09-02T10:17:02.794600+0000","last_fullsized":"2024-09-02T10:17:02.794600+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"495'8753","last_scrub_stamp":"2024-09-02T10:16:53.975800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:53.975800+0000","objects_scrubbed":4548,"log_size":9275,"log_dups_size":0,"ondisk_log_size":9275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:55.753425+0000","scrub_duration":1224,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262209536,"num_objects":4001,"num_object_clones":0,"num_object_copies":12003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9275,"num_write_kb":424832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":184,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.033570+0000","last_change":"2024-09-02T10:15:36.299961+0000","last_active":"2024-09-02T10:17:00.033570+0000","last_peered":"2024-09-02T10:17:00.033570+0000","last_clean":"2024-09-02T10:17:00.033570+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:17:00.033570+0000","last_undegraded":"2024-09-02T10:17:00.033570+0000","last_fullsized":"2024-09-02T10:17:00.033570+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:36.299914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:36.299914+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:01.870130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"499'9070","reported_seq":9270,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:02.760944+0000","last_change":"2024-09-02T10:16:52.776720+0000","last_active":"2024-09-02T10:17:02.760944+0000","last_peered":"2024-09-02T10:17:02.760944+0000","last_clean":"2024-09-02T10:17:02.760944+0000","last_became_active":"2024-09-02T10:14:39.641478+0000","last_became_peered":"2024-09-02T10:14:39.641478+0000","last_unstale":"2024-09-02T10:17:02.760944+0000","last_undegraded":"2024-09-02T10:17:02.760944+0000","last_fullsized":"2024-09-02T10:17:02.760944+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8513","last_scrub_stamp":"2024-09-02T10:16:52.776607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:52.776607+0000","objects_scrubbed":4459,"log_size":9070,"log_dups_size":0,"ondisk_log_size":9070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:14.411098+0000","scrub_duration":1045,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254672896,"num_objects":3886,"num_object_clones":0,"num_object_copies":11658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9070,"num_write_kb":414592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":442,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.042211+0000","last_change":"2024-09-02T10:16:55.634615+0000","last_active":"2024-09-02T10:17:00.042211+0000","last_peered":"2024-09-02T10:17:00.042211+0000","last_clean":"2024-09-02T10:17:00.042211+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:17:00.042211+0000","last_undegraded":"2024-09-02T10:17:00.042211+0000","last_fullsized":"2024-09-02T10:17:00.042211+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:55.634548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:55.634548+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:01.409660+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"5.6","version":"499'8966","reported_seq":9170,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:02.039236+0000","last_change":"2024-09-02T10:15:49.363717+0000","last_active":"2024-09-02T10:17:02.039236+0000","last_peered":"2024-09-02T10:17:02.039236+0000","last_clean":"2024-09-02T10:17:02.039236+0000","last_became_active":"2024-09-02T10:14:39.037882+0000","last_became_peered":"2024-09-02T10:14:39.037882+0000","last_unstale":"2024-09-02T10:17:02.039236+0000","last_undegraded":"2024-09-02T10:17:02.039236+0000","last_fullsized":"2024-09-02T10:17:02.039236+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"469'4720","last_scrub_stamp":"2024-09-02T10:15:49.363588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:15:49.363588+0000","objects_scrubbed":4682,"log_size":8966,"log_dups_size":0,"ondisk_log_size":8966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:06.930040+0000","scrub_duration":1106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255852544,"num_objects":3904,"num_object_clones":0,"num_object_copies":11712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8966,"num_write_kb":411840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":687,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.039859+0000","last_change":"2024-09-02T10:16:23.198280+0000","last_active":"2024-09-02T10:17:00.039859+0000","last_peered":"2024-09-02T10:17:00.039859+0000","last_clean":"2024-09-02T10:17:00.039859+0000","last_became_active":"2024-09-02T10:08:25.263086+0000","last_became_peered":"2024-09-02T10:08:25.263086+0000","last_unstale":"2024-09-02T10:17:00.039859+0000","last_undegraded":"2024-09-02T10:17:00.039859+0000","last_fullsized":"2024-09-02T10:17:00.039859+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:23.198227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:23.198227+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:52.413888+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,8],"acting":[7,0,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"499'8979","reported_seq":9199,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:02.022738+0000","last_change":"2024-09-02T10:16:17.087529+0000","last_active":"2024-09-02T10:17:02.022738+0000","last_peered":"2024-09-02T10:17:02.022738+0000","last_clean":"2024-09-02T10:17:02.022738+0000","last_became_active":"2024-09-02T10:14:39.641932+0000","last_became_peered":"2024-09-02T10:14:39.641932+0000","last_unstale":"2024-09-02T10:17:02.022738+0000","last_undegraded":"2024-09-02T10:17:02.022738+0000","last_fullsized":"2024-09-02T10:17:02.022738+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'6467","last_scrub_stamp":"2024-09-02T10:16:17.087427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.087427+0000","objects_scrubbed":6467,"log_size":8979,"log_dups_size":0,"ondisk_log_size":8979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:47.064105+0000","scrub_duration":739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259194880,"num_objects":3955,"num_object_clones":0,"num_object_copies":11865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8979,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":947,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.040584+0000","last_change":"2024-09-02T10:16:17.754936+0000","last_active":"2024-09-02T10:17:00.040584+0000","last_peered":"2024-09-02T10:17:00.040584+0000","last_clean":"2024-09-02T10:17:00.040584+0000","last_became_active":"2024-09-02T10:15:29.168692+0000","last_became_peered":"2024-09-02T10:15:29.168692+0000","last_unstale":"2024-09-02T10:17:00.040584+0000","last_undegraded":"2024-09-02T10:17:00.040584+0000","last_fullsized":"2024-09-02T10:17:00.040584+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:17.754866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.754866+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:39.447349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,8],"acting":[1,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"499'9145","reported_seq":9293,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:01.642021+0000","last_change":"2024-09-02T10:14:39.038104+0000","last_active":"2024-09-02T10:17:01.642021+0000","last_peered":"2024-09-02T10:17:01.642021+0000","last_clean":"2024-09-02T10:17:01.642021+0000","last_became_active":"2024-09-02T10:14:39.037755+0000","last_became_peered":"2024-09-02T10:14:39.037755+0000","last_unstale":"2024-09-02T10:17:01.642021+0000","last_undegraded":"2024-09-02T10:17:01.642021+0000","last_fullsized":"2024-09-02T10:17:01.642021+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":9145,"log_dups_size":0,"ondisk_log_size":9145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264699904,"num_objects":4039,"num_object_clones":0,"num_object_copies":12117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9145,"num_write_kb":421888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,8],"acting":[10,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":706,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.039730+0000","last_change":"2024-09-02T10:16:50.290435+0000","last_active":"2024-09-02T10:17:00.039730+0000","last_peered":"2024-09-02T10:17:00.039730+0000","last_clean":"2024-09-02T10:17:00.039730+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:17:00.039730+0000","last_undegraded":"2024-09-02T10:17:00.039730+0000","last_fullsized":"2024-09-02T10:17:00.039730+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:50.290368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:50.290368+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:02.137283+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"499'9001","reported_seq":9219,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:02.942472+0000","last_change":"2024-09-02T10:16:20.568272+0000","last_active":"2024-09-02T10:17:02.942472+0000","last_peered":"2024-09-02T10:17:02.942472+0000","last_clean":"2024-09-02T10:17:02.942472+0000","last_became_active":"2024-09-02T10:14:39.039060+0000","last_became_peered":"2024-09-02T10:14:39.039060+0000","last_unstale":"2024-09-02T10:17:02.942472+0000","last_undegraded":"2024-09-02T10:17:02.942472+0000","last_fullsized":"2024-09-02T10:17:02.942472+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6670","last_scrub_stamp":"2024-09-02T10:16:20.568081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:20.568081+0000","objects_scrubbed":6316,"log_size":9001,"log_dups_size":0,"ondisk_log_size":9001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:27.515779+0000","scrub_duration":1605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257359872,"num_objects":3927,"num_object_clones":0,"num_object_copies":11781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9001,"num_write_kb":413696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,5],"acting":[9,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":946,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.033662+0000","last_change":"2024-09-02T10:16:17.179894+0000","last_active":"2024-09-02T10:17:00.033662+0000","last_peered":"2024-09-02T10:17:00.033662+0000","last_clean":"2024-09-02T10:17:00.033662+0000","last_became_active":"2024-09-02T10:11:11.467061+0000","last_became_peered":"2024-09-02T10:11:11.467061+0000","last_unstale":"2024-09-02T10:17:00.033662+0000","last_undegraded":"2024-09-02T10:17:00.033662+0000","last_fullsized":"2024-09-02T10:17:00.033662+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:17.179849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:17.179849+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:24.924645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.a","version":"498'8849","reported_seq":9067,"reported_epoch":498,"state":"active+clean","last_fresh":"2024-09-02T10:16:59.319704+0000","last_change":"2024-09-02T10:16:21.735648+0000","last_active":"2024-09-02T10:16:59.319704+0000","last_peered":"2024-09-02T10:16:59.319704+0000","last_clean":"2024-09-02T10:16:59.319704+0000","last_became_active":"2024-09-02T10:14:39.036453+0000","last_became_peered":"2024-09-02T10:14:39.036453+0000","last_unstale":"2024-09-02T10:16:59.319704+0000","last_undegraded":"2024-09-02T10:16:59.319704+0000","last_fullsized":"2024-09-02T10:16:59.319704+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'6768","last_scrub_stamp":"2024-09-02T10:16:21.735508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:21.735508+0000","objects_scrubbed":6243,"log_size":8849,"log_dups_size":0,"ondisk_log_size":8849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:26.135470+0000","scrub_duration":1736,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269418496,"num_objects":4111,"num_object_clones":0,"num_object_copies":12333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8849,"num_write_kb":414720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11],"acting":[8,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":480,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.040057+0000","last_change":"2024-09-02T10:16:21.146441+0000","last_active":"2024-09-02T10:17:00.040057+0000","last_peered":"2024-09-02T10:17:00.040057+0000","last_clean":"2024-09-02T10:17:00.040057+0000","last_became_active":"2024-09-02T10:08:25.263860+0000","last_became_peered":"2024-09-02T10:08:25.263860+0000","last_unstale":"2024-09-02T10:17:00.040057+0000","last_undegraded":"2024-09-02T10:17:00.040057+0000","last_fullsized":"2024-09-02T10:17:00.040057+0000","mapping_epoch":282,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":283,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:21.146366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:21.146366+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:23.276250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,8],"acting":[7,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"499'9065","reported_seq":9285,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:01.118137+0000","last_change":"2024-09-02T10:16:52.305281+0000","last_active":"2024-09-02T10:17:01.118137+0000","last_peered":"2024-09-02T10:17:01.118137+0000","last_clean":"2024-09-02T10:17:01.118137+0000","last_became_active":"2024-09-02T10:14:39.037567+0000","last_became_peered":"2024-09-02T10:14:39.037567+0000","last_unstale":"2024-09-02T10:17:01.118137+0000","last_undegraded":"2024-09-02T10:17:01.118137+0000","last_fullsized":"2024-09-02T10:17:01.118137+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'8567","last_scrub_stamp":"2024-09-02T10:16:52.305119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:16:52.305119+0000","objects_scrubbed":4488,"log_size":9065,"log_dups_size":0,"ondisk_log_size":9065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:59.273397+0000","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259981312,"num_objects":3967,"num_object_clones":0,"num_object_copies":11901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9065,"num_write_kb":417024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":726,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.039998+0000","last_change":"2024-09-02T10:16:57.391434+0000","last_active":"2024-09-02T10:17:00.039998+0000","last_peered":"2024-09-02T10:17:00.039998+0000","last_clean":"2024-09-02T10:17:00.039998+0000","last_became_active":"2024-09-02T10:15:29.167553+0000","last_became_peered":"2024-09-02T10:15:29.167553+0000","last_unstale":"2024-09-02T10:17:00.039998+0000","last_undegraded":"2024-09-02T10:17:00.039998+0000","last_fullsized":"2024-09-02T10:17:00.039998+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:16:57.391357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:16:57.391357+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:18:00.156075+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,8],"acting":[7,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"499'9044","reported_seq":17081,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:02.098532+0000","last_change":"2024-09-02T10:16:42.039584+0000","last_active":"2024-09-02T10:17:02.098532+0000","last_peered":"2024-09-02T10:17:02.098532+0000","last_clean":"2024-09-02T10:17:02.098532+0000","last_became_active":"2024-09-02T10:16:42.029488+0000","last_became_peered":"2024-09-02T10:16:42.029488+0000","last_unstale":"2024-09-02T10:17:02.098532+0000","last_undegraded":"2024-09-02T10:17:02.098532+0000","last_fullsized":"2024-09-02T10:17:02.098532+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":431,"last_epoch_clean":491,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:14:38.017507+0000","last_clean_scrub_stamp":"2024-09-02T10:14:38.017507+0000","objects_scrubbed":0,"log_size":9044,"log_dups_size":0,"ondisk_log_size":9044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256507904,"num_objects":3914,"num_object_clones":0,"num_object_copies":11742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":414656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3939,"num_bytes_recovered":258146304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,10],"acting":[11,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":124,"reported_epoch":499,"state":"active+clean","last_fresh":"2024-09-02T10:17:00.117185+0000","last_change":"2024-09-02T10:15:48.739408+0000","last_active":"2024-09-02T10:17:00.117185+0000","last_peered":"2024-09-02T10:17:00.117185+0000","last_clean":"2024-09-02T10:17:00.117185+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:17:00.117185+0000","last_undegraded":"2024-09-02T10:17:00.117185+0000","last_fullsized":"2024-09-02T10:17:00.117185+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:15:48.739340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:15:48.739340+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:17:02.899055+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":4120379416,"num_objects":62873,"num_object_clones":0,"num_object_copies":188619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62873,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144333,"num_write_kb":6630529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5708,"num_bytes_recovered":374079488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12370980864,"data_stored":12370968648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144333,"ondisk_log_size":144333,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383883,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1662688,"kb_used_data":1270536,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":92082464,"statfs":{"total":95995035648,"available":94292443136,"internally_reserved":0,"allocated":1301028864,"data_stored":1300701622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":401537476},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612816,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2050072,"kb_used_data":1780100,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91695080,"statfs":{"total":95995035648,"available":93895761920,"internally_reserved":0,"allocated":1822822400,"data_stored":1822499230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":276429248},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841749,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1434580,"kb_used_data":998020,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92310572,"statfs":{"total":95995035648,"available":94526025728,"internally_reserved":0,"allocated":1021972480,"data_stored":1021649310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":447019456},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070682,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1064508,"kb_used_data":794540,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92680644,"statfs":{"total":95995035648,"available":94904979456,"internally_reserved":0,"allocated":813608960,"data_stored":813294858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299615,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1432160,"kb_used_data":1021968,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92312992,"statfs":{"total":95995035648,"available":94528503808,"internally_reserved":0,"allocated":1046495232,"data_stored":1046164438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":420018634},"hb_peers":[0,1,2,3,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528547,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1558356,"kb_used_data":1256452,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92186796,"statfs":{"total":95995035648,"available":94399279104,"internally_reserved":0,"allocated":1286606848,"data_stored":1286283678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724776,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2212648,"kb_used_data":1794064,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":91532504,"statfs":{"total":95995035648,"available":93729284096,"internally_reserved":0,"allocated":1837121536,"data_stored":1836790742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[1,2,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149939,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636628,"kb_used_data":248644,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":93108524,"statfs":{"total":95995035648,"available":95343128576,"internally_reserved":0,"allocated":254611456,"data_stored":254288286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":397277640},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411574,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1459296,"kb_used_data":1053260,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92285856,"statfs":{"total":95995035648,"available":94500716544,"internally_reserved":0,"allocated":1078538240,"data_stored":1078211518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738626,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2470996,"kb_used_data":2002500,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91274156,"statfs":{"total":95995035648,"available":93464735744,"internally_reserved":0,"allocated":2050560000,"data_stored":2050236830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2046230528,"data_stored":2046230528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1012924416,"data_stored":1012924416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":250085376,"data_stored":250085376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1814237184,"data_stored":1814233112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1284964352,"data_stored":1284964352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1033244672,"data_stored":1033240600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":809369600,"data_stored":809369600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1019936768,"data_stored":1019936768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1818558464,"data_stored":1818558464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1281429504,"data_stored":1281425432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:17:05.817 INFO:tasks.thrashosds.thrasher:Setting 1.e to [4, 9, 2] 2024-09-02T10:17:05.817 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.e', '4', '9', '2'] 2024-09-02T10:17:05.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap 1.e 4 9 2 2024-09-02T10:17:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:05.818 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:05.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:05.898 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:05.919 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:05.950 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:05.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:05.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:06.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:06.076 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:06.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:06.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:06.208 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:17:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:06.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:06.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:06.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:06.334 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:06.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:06.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:06.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:06.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:06.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:06.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:06 smithi187 ceph-mon[20072]: pgmap v1163: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 721 op/s 2024-09-02T10:17:06.513 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:06.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:06.573 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:06.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:06.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:06.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:06 smithi151 ceph-mon[16639]: pgmap v1163: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 721 op/s 2024-09-02T10:17:06.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:06.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:06.725 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:06.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:06.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:06 smithi162 ceph-mon[20279]: pgmap v1163: 47 pgs: 47 active+clean; 3.8 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 721 op/s 2024-09-02T10:17:06.852 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:06.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:06.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:06.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:06.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:06.979 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:06.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:07.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:07.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:07.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:07.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:07.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:07.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:07.247 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:07.260 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:17:07.260 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:17:07.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:07.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:07.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:07.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:07.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:17:07.444 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:17:07.446 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.e pg_upmap mapping to [4,9,2] 2024-09-02T10:17:07.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:07.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:07.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:07.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:07.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:07.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1340402189' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2774610647' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.e", "id": [4, 9, 2]} : dispatch 2024-09-02T10:17:07.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1340402189' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2774610647' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.e", "id": [4, 9, 2]} : dispatch 2024-09-02T10:17:07.760 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:07.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1340402189' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2774610647' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.e", "id": [4, 9, 2]} : dispatch 2024-09-02T10:17:07.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:07.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:07.893 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:07.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:08.018 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:08.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:08.120 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:08.151 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:08.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:08.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:08.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:08.281 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:08.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:08.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:08.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:08.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:08.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:08.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:08.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:08.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:08.654 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:08.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:08.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:08.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:08 smithi187 ceph-mon[20072]: pgmap v1164: 47 pgs: 47 active+clean; 3.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 864 op/s 2024-09-02T10:17:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:08 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1340402189' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2774610647' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.e", "id": [4, 9, 2]}]': finished 2024-09-02T10:17:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:08 smithi187 ceph-mon[20072]: osdmap e501: 12 total, 12 up, 10 in 2024-09-02T10:17:08.785 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:08.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:08 smithi162 ceph-mon[20279]: pgmap v1164: 47 pgs: 47 active+clean; 3.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 864 op/s 2024-09-02T10:17:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:08 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1340402189' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2774610647' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.e", "id": [4, 9, 2]}]': finished 2024-09-02T10:17:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:08 smithi162 ceph-mon[20279]: osdmap e501: 12 total, 12 up, 10 in 2024-09-02T10:17:08.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:08.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:08 smithi151 ceph-mon[16639]: pgmap v1164: 47 pgs: 47 active+clean; 3.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 864 op/s 2024-09-02T10:17:08.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:08 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:08.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1340402189' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:08.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2774610647' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.e", "id": [4, 9, 2]}]': finished 2024-09-02T10:17:08.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:08 smithi151 ceph-mon[16639]: osdmap e501: 12 total, 12 up, 10 in 2024-09-02T10:17:08.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:08.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:09.000 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:09.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:09.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:17:09.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:09.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:09.051 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:09.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:09.174 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:09.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:09.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:09.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:09.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:09.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:09.407 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:09.411 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:09.446 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:09.448 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:09.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:09.557 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:17:09.557 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:17:09.578 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:09.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:09.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:17:09.584 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:17:09.611 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:17:09.612 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:17:09.635 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:17:09.635 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:17:09.657 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:17:09.657 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:17:09.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:09.682 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:17:09.683 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:17:09.701 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:09.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:09.710 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:17:09.710 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:17:09.733 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:17:09.734 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:17:09.764 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:17:09.765 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:17:09.791 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:17:09.792 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:17:09.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:09.816 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:17:09.816 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:17:09.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:09.833 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:09.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:17:09.842 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:17:09.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:09.959 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:09.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:10.032 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:17:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:17:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:17:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:17:10.032 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:17:10.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:10.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:10.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:10.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:10.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:10.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:10.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:10.339 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:10.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:10.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:10.468 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:10.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:10.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:10.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:10.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:10.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:10.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:10 smithi151 ceph-mon[16639]: pgmap v1166: 47 pgs: 47 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 988 op/s 2024-09-02T10:17:10.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1455495414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:10.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:10.717 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:10 smithi187 ceph-mon[20072]: pgmap v1166: 47 pgs: 47 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 988 op/s 2024-09-02T10:17:10.717 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1455495414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:10.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:10.723 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:10.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:10.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:17:10.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:10 smithi162 ceph-mon[20279]: pgmap v1166: 47 pgs: 47 active+clean; 3.6 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 988 op/s 2024-09-02T10:17:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1455495414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:10.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:10.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:10.866 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:10.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:17:10.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:10.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:10.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:11.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:11.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:11.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:11.136 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:11.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:11.245 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:11.250 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:11.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:11.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:11.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:11.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:11.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:11.509 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:11.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:11.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:11.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:11.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:11.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:11.652 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:11.726 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:17:11.726 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.24976.0:109536 3.9 3:9af67999:::benchmark_data_smithi187_59989_object109535:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:07:33.018363+0000", 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 578.70502762499996, 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.368789486, 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.727 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.24976", 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 109536 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018363+0000", 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018363+0000", 2024-09-02T10:17:11.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018364+0000", 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0750000000000001e-06 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018385+0000", 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1044e-05 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018396+0000", 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1812e-05 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.018403+0000", 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9580000000000001e-06 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021486+0000", 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030823600000000001 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021631+0000", 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014475600000000001 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,11", 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.021787+0000", 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015694600000000001 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.024639+0000", 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028514159999999998 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.025503+0000", 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086444899999999999 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.387070+0000", 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36156629699999998 2024-09-02T10:17:11.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.387107+0000", 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7620000000000002e-05 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:07:33.387152+0000", 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4753000000000003e-05 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:42314 4.0 e355/341 4:049d10b0:::benchmark_data_smithi187_76542_object42313:head v 355'2240, mlcod=355'2237)", 2024-09-02T10:17:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:49.052907+0000", 2024-09-02T10:17:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 322.67048361600001, 2024-09-02T10:17:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.227636956, 2024-09-02T10:17:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.052907+0000", 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.052907+0000", 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.052908+0000", 2024-09-02T10:17:11.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.088e-06 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.053042+0000", 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013480799999999999 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279523+0000", 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22648093799999999 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279535+0000", 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1678000000000001e-05 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279599+0000", 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4323000000000001e-05 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279617+0000", 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7971000000000001e-05 2024-09-02T10:17:11.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.280511+0000", 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00089334699999999996 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.280544+0000", 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2802999999999997e-05 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90191 4.15 4:afce227e:::benchmark_data_smithi187_76542_object90190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:31.097664+0000", 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 280.62572579599998, 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.205197566, 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90191 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097664+0000", 2024-09-02T10:17:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097664+0000", 2024-09-02T10:17:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097666+0000", 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5859999999999999e-06 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097685+0000", 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9361e-05 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097699+0000", 2024-09-02T10:17:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3359e-05 2024-09-02T10:17:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097704+0000", 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6849999999999998e-06 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097760+0000", 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5340000000000002e-05 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097871+0000", 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111147 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097958+0000", 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7638000000000004e-05 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.099226+0000", 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012679360000000001 2024-09-02T10:17:11.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.229945+0000", 2024-09-02T10:17:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.130718953 2024-09-02T10:17:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.302738+0000", 2024-09-02T10:17:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.072792361999999999 2024-09-02T10:17:11.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.302790+0000", 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1853e-05 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.302862+0000", 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2346000000000002e-05 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90787 4.15 4:afba0c08:::benchmark_data_smithi187_76542_object90786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:17:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:32.022436+0000", 2024-09-02T10:17:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 279.70095457500003, 2024-09-02T10:17:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27502997200000001, 2024-09-02T10:17:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.747 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90787 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022436+0000", 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022436+0000", 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022437+0000", 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5659999999999999e-06 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022462+0000", 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4437e-05 2024-09-02T10:17:11.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022480+0000", 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8493000000000001e-05 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022489+0000", 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6959999999999994e-06 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022548+0000", 2024-09-02T10:17:11.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9709e-05 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022678+0000", 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000129689 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022825+0000", 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014708899999999999 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.024159+0000", 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013338320000000001 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.024979+0000", 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00081976499999999997 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297377+0000", 2024-09-02T10:17:11.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.272397677 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297432+0000", 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5513000000000002e-05 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297466+0000", 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3506000000000001e-05 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90806 4.15 4:aae97575:::benchmark_data_smithi187_76542_object90805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:17:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:32.085698+0000", 2024-09-02T10:17:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 279.63769173700001, 2024-09-02T10:17:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21198275899999999, 2024-09-02T10:17:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.754 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:17:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:17:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90806 2024-09-02T10:17:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085698+0000", 2024-09-02T10:17:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085698+0000", 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085700+0000", 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.429e-06 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085722+0000", 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2178999999999999e-05 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085736+0000", 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3687e-05 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085742+0000", 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1879999999999999e-06 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085792+0000", 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0034999999999998e-05 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085909+0000", 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011707200000000001 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:17:11.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.086007+0000", 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7526000000000001e-05 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.087291+0000", 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001284246 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.087763+0000", 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000472466 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297588+0000", 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20982505200000001 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297648+0000", 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0167999999999997e-05 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297681+0000", 2024-09-02T10:17:11.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2710999999999999e-05 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90807 4.15 4:a985eaa0:::benchmark_data_smithi187_76542_object90806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:32.098906+0000", 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 279.62448423299998, 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20543761199999999, 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.761 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90807 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098906+0000", 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098906+0000", 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098908+0000", 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0729999999999999e-06 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098965+0000", 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6520000000000001e-05 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098975+0000", 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0797e-05 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098981+0000", 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3630000000000002e-06 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.099019+0000", 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8587000000000002e-05 2024-09-02T10:17:11.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.099101+0000", 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2199000000000004e-05 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.099187+0000", 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5387000000000001e-05 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.100008+0000", 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00082087200000000005 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.101112+0000", 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011045569999999999 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.304195+0000", 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20308306400000001 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.304295+0000", 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9996999999999998e-05 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.304344+0000", 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8195999999999997e-05 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.767 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPush(4.4 392/385 [PushOp(4:21e15029:::benchmark_data_smithi187_76542_object88778:head, version: 375'3759, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:21e15029:::benchmark_data_smithi187_76542_object88778:head@375'3759, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:17:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:15.639682+0000", 2024-09-02T10:17:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 236.08370828899999, 2024-09-02T10:17:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23632888699999999, 2024-09-02T10:17:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:17:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639682+0000", 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639682+0000", 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639683+0000", 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9500000000000002e-07 2024-09-02T10:17:11.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639708+0000", 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5466e-05 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.874909+0000", 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23520074799999999 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.874920+0000", 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0825999999999999e-05 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.874993+0000", 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3734999999999998e-05 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.875002+0000", 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0610000000000002e-06 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.876011+0000", 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001008356 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:180712 4.2 e422/341 4:40dddee0:::benchmark_data_smithi187_76542_object88376:head v 422'6632, mlcod=422'6630)", 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:18.534975+0000", 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 173.18841521100001, 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21512784800000001, 2024-09-02T10:17:11.772 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534975+0000", 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534975+0000", 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534976+0000", 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.257e-06 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534985+0000", 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3289999999999994e-06 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747701+0000", 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21271659000000001 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747709+0000", 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8609999999999998e-06 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747772+0000", 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2969e-05 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747788+0000", 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6201999999999998e-05 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.750057+0000", 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022690900000000001 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.750103+0000", 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5550000000000003e-05 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:17898 5.a 5:5b8e596f:::benchmark_data_smithi187_89405_object17897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.425237+0000", 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 134.29815346399999, 2024-09-02T10:17:11.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.209706585, 2024-09-02T10:17:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:11.778 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 17898 2024-09-02T10:17:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425237+0000", 2024-09-02T10:17:11.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425237+0000", 2024-09-02T10:17:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425238+0000", 2024-09-02T10:17:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.525e-06 2024-09-02T10:17:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425259+0000", 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0987000000000001e-05 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425273+0000", 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3309000000000001e-05 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425278+0000", 2024-09-02T10:17:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.891e-06 2024-09-02T10:17:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425329+0000", 2024-09-02T10:17:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0488999999999997e-05 2024-09-02T10:17:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425415+0000", 2024-09-02T10:17:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6077999999999998e-05 2024-09-02T10:17:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:17:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425520+0000", 2024-09-02T10:17:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010503800000000001 2024-09-02T10:17:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.426467+0000", 2024-09-02T10:17:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00094675800000000002 2024-09-02T10:17:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.426778+0000", 2024-09-02T10:17:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031167899999999998 2024-09-02T10:17:11.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634817+0000", 2024-09-02T10:17:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20803839900000001 2024-09-02T10:17:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634893+0000", 2024-09-02T10:17:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6501000000000005e-05 2024-09-02T10:17:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634943+0000", 2024-09-02T10:17:11.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9931e-05 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:17898 5.a e440/431 ondisk, result = 0)", 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.427025+0000", 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 134.29636519799999, 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.207926318, 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:17:11.788 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.427025+0000", 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.427025+0000", 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.427026+0000", 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1999999999999999e-07 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.427029+0000", 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5159999999999999e-06 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634717+0000", 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.207688189 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634725+0000", 2024-09-02T10:17:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5259999999999996e-06 2024-09-02T10:17:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634792+0000", 2024-09-02T10:17:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7148000000000002e-05 2024-09-02T10:17:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634810+0000", 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.806e-05 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634951+0000", 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014115899999999999 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.794 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:36410 5.4 e449/431 5:20a8e25f:::benchmark_data_smithi187_89405_object36409:head v 449'2296, mlcod=449'2292)", 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:12.597403+0000", 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 119.12598685, 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20486853799999999, 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.597403+0000", 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.597403+0000", 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.597404+0000", 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0410000000000001e-06 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.597429+0000", 2024-09-02T10:17:11.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4851999999999999e-05 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.800838+0000", 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20340900000000001 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.800854+0000", 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5631e-05 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.800914+0000", 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0421999999999998e-05 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.800935+0000", 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.037e-05 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.802220+0000", 2024-09-02T10:17:11.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012849630000000001 2024-09-02T10:17:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.802272+0000", 2024-09-02T10:17:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2259000000000002e-05 2024-09-02T10:17:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:36418 5.a 5:5b018ae7:::benchmark_data_smithi187_89405_object36417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-09-02T10:17:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:12.602739+0000", 2024-09-02T10:17:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 119.120651043, 2024-09-02T10:17:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19998452899999999, 2024-09-02T10:17:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:11.800 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 36418 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.602739+0000", 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.602739+0000", 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.602740+0000", 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.308e-06 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.602761+0000", 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0611999999999998e-05 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.602773+0000", 2024-09-02T10:17:11.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2422999999999999e-05 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.602779+0000", 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8449999999999999e-06 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.602833+0000", 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4020000000000001e-05 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.602937+0000", 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010398700000000001 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.603028+0000", 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.077e-05 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.604251+0000", 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001222725 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.604650+0000", 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039925200000000001 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.802617+0000", 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19796725500000001 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.802676+0000", 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8687000000000002e-05 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:12.802724+0000", 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7645000000000002e-05 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:62678 5.e e461/431 5:7900e0bc:::benchmark_data_smithi187_89405_object62677:head v 461'3875, mlcod=461'3872)", 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.057709+0000", 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 97.665681090000007, 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20245972100000001, 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.806 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.057709+0000", 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.057709+0000", 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.057710+0000", 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4099999999999997e-07 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.057735+0000", 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5389999999999999e-05 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.258252+0000", 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.200516363 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.258259+0000", 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9600000000000003e-06 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.258343+0000", 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4651e-05 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.258365+0000", 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1588999999999999e-05 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.260117+0000", 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001751568 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.260169+0000", 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2259000000000002e-05 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.810 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63248 5.a 5:5adbc538:::benchmark_data_smithi187_89405_object63247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 95.209230223999995, 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21653889600000001, 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:11.811 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63248 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514161+0000", 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.099e-06 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514181+0000", 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.999e-05 2024-09-02T10:17:11.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514212+0000", 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0648999999999997e-05 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514371+0000", 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015914799999999999 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514421+0000", 2024-09-02T10:17:11.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9804e-05 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514561+0000", 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000140285 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514666+0000", 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104963 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.516942+0000", 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022756780000000002 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521266+0000", 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043245380000000002 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730584+0000", 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20931819099999999 2024-09-02T10:17:11.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730647+0000", 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3040999999999995e-05 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730699+0000", 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.151e-05 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63255 5.a 5:57c5feec:::benchmark_data_smithi187_89405_object63254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 95.205420324000002, 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21287715600000001, 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63255 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517972+0000", 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.832e-06 2024-09-02T10:17:11.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517998+0000", 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6004e-05 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518024+0000", 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6290000000000001e-05 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518039+0000", 2024-09-02T10:17:11.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.484e-05 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518103+0000", 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4350000000000006e-05 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518230+0000", 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012653999999999999 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:17:11.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518365+0000", 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013489900000000001 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.520755+0000", 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023907780000000001 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.523888+0000", 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031328179999999999 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730745+0000", 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.206856442 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730800+0000", 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5055000000000001e-05 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730847+0000", 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7308000000000002e-05 2024-09-02T10:17:11.823 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63260 5.a 5:581436a1:::benchmark_data_smithi187_89405_object63259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 95.202115051999996, 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21179099900000001, 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.824 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63260 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521277+0000", 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.782e-06 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521301+0000", 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3893e-05 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521320+0000", 2024-09-02T10:17:11.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8879999999999999e-05 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521327+0000", 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6890000000000004e-06 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521399+0000", 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1439000000000005e-05 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521552+0000", 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000153268 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521685+0000", 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013246699999999999 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.524092+0000", 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024077679999999998 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.525479+0000", 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001386916 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.732968+0000", 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20748902399999999 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.733024+0000", 2024-09-02T10:17:11.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5269999999999998e-05 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.733066+0000", 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2602999999999999e-05 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:86595 5.a 5:52430871:::benchmark_data_smithi187_89405_object86594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 73.217305757999995, 2024-09-02T10:17:11.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.225552366, 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 86595 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506087+0000", 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2469999999999998e-06 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506114+0000", 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7141999999999999e-05 2024-09-02T10:17:11.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506133+0000", 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9137e-05 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506142+0000", 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8859999999999993e-06 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506210+0000", 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7992999999999998e-05 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506361+0000", 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015085899999999999 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506506+0000", 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014534300000000001 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.509602+0000", 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030956009999999999 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.597950+0000", 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088348135999999994 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731542+0000", 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13359195700000001 2024-09-02T10:17:11.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731597+0000", 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5368999999999998e-05 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731637+0000", 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9696000000000001e-05 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:92684 5.a 5:5c70a25d:::benchmark_data_smithi187_89405_object92683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:04.924611+0000", 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 66.798779069000005, 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20088767399999999, 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 92684 2024-09-02T10:17:11.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:04.924611+0000", 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:04.924611+0000", 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:04.924613+0000", 2024-09-02T10:17:11.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.64e-06 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:04.924634+0000", 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1375000000000001e-05 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:04.924648+0000", 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4096e-05 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:04.924655+0000", 2024-09-02T10:17:11.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4989999999999999e-06 2024-09-02T10:17:11.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:04.924704+0000", 2024-09-02T10:17:11.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9746000000000001e-05 2024-09-02T10:17:11.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:04.924789+0000", 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4606999999999998e-05 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:04.924883+0000", 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4003000000000001e-05 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:04.927411+0000", 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025277139999999999 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:05.084669+0000", 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15725841099999999 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:05.125386+0000", 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.040717259999999998 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:05.125456+0000", 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9807000000000004e-05 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:05.125499+0000", 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2515999999999997e-05 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:122098 5.e e488/431 5:783becf1:::benchmark_data_smithi187_89405_object18493:head v 488'7571, mlcod=488'7567)", 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 35.151710635999997, 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21171738700000001, 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.843 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:17:11.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1999999999999999e-07 2024-09-02T10:17:11.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571686+0000", 2024-09-02T10:17:11.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6010000000000002e-06 2024-09-02T10:17:11.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780330+0000", 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20864460000000001 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780339+0000", 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1869999999999997e-06 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780399+0000", 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9941000000000001e-05 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780424+0000", 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5751000000000001e-05 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.783348+0000", 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029234819999999998 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.783397+0000", 2024-09-02T10:17:11.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9104999999999998e-05 2024-09-02T10:17:11.848 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.848 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.848 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.848 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:126123 5.e e490/431 5:7800327a:::benchmark_data_smithi187_89405_object22518:head v 490'7825, mlcod=490'7821)", 2024-09-02T10:17:11.848 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:17:11.848 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 30.183365355999999, 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25454940500000001, 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540026+0000", 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1000000000000001e-06 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540033+0000", 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8009999999999998e-06 2024-09-02T10:17:11.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791598+0000", 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25156501399999998 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791605+0000", 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9569999999999996e-06 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791659+0000", 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4088000000000001e-05 2024-09-02T10:17:11.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791680+0000", 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1450999999999998e-05 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.794543+0000", 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028628120000000002 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:11.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.794574+0000", 2024-09-02T10:17:11.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1182000000000001e-05 2024-09-02T10:17:11.853 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.853 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.853 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.853 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:11.853 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:11.853 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:17:11.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:11.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:11.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:11.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:11.979 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:11.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:12.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:12.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:12.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:12.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:12.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:12.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:12.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:12.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:12.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:12.356 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:12.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:12.456 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:17:12.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:12.483 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:12.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:12.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:12.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:17:12.629 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:12.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:12.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:12.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:17:12.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:12.755 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:12.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:12.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:12 smithi187 ceph-mon[20072]: pgmap v1167: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 657 op/s 2024-09-02T10:17:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2264373233' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:17:12.831 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:12 smithi151 ceph-mon[16639]: pgmap v1167: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 657 op/s 2024-09-02T10:17:12.832 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2264373233' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:17:12.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:12 smithi162 ceph-mon[20279]: pgmap v1167: 47 pgs: 47 active+clean; 3.5 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 657 op/s 2024-09-02T10:17:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2264373233' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:17:12.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:12.880 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:12.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:12.980 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:12.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:13.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:13.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:13.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:13.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:13.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:13.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:13.273 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:13.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:13.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:13.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:13.400 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:13.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:13.401 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:13.491 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:17:13.491 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:17:13.491 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-02T10:17:13.491 INFO:tasks.thrashosds.thrasher:Setting osd 8 primary_affinity to 1.000000 2024-09-02T10:17:13.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 8 1 2024-09-02T10:17:13.501 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:13.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:13 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:13.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2264373233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:17:13.516 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:13 smithi151 ceph-mon[16639]: osdmap e502: 12 total, 12 up, 10 in 2024-09-02T10:17:13.525 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:13.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:13.553 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:17:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:17:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:154149 5.e 5:7ae09753:::benchmark_data_smithi187_89405_object50544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e503)", 2024-09-02T10:17:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:13.536910+0000", 2024-09-02T10:17:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.014825194999999999, 2024-09-02T10:17:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014848293, 2024-09-02T10:17:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:13.553 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 154149 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.536910+0000", 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.536910+0000", 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.536912+0000", 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2749999999999998e-06 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.536918+0000", 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3729999999999998e-06 2024-09-02T10:17:13.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.536937+0000", 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8151999999999998e-05 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.536946+0000", 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0699999999999996e-06 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:13.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.537028+0000", 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2602999999999995e-05 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.537146+0000", 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011758500000000001 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,8", 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.537264+0000", 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117868 2024-09-02T10:17:13.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.542305+0000", 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005041439 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:154160 5.7 e503/431 5:e880b362:::benchmark_data_smithi187_89405_object50555:head v 503'9610, mlcod=503'9607)", 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:13.548673+0000", 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0030621480000000002, 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031333540000000001, 2024-09-02T10:17:13.558 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.548673+0000", 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.548673+0000", 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.548674+0000", 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.085e-06 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.548682+0000", 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8429999999999993e-06 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:13.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.548705+0000", 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3518000000000001e-05 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.548714+0000", 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3580000000000003e-06 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.548772+0000", 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8258999999999999e-05 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:13.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:13.548793+0000", 2024-09-02T10:17:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1460000000000001e-05 2024-09-02T10:17:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:17:13.562 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 2 2024-09-02T10:17:13.562 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:17:13.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:13.656 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:13.657 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:13 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2264373233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:17:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:13 smithi187 ceph-mon[20072]: osdmap e502: 12 total, 12 up, 10 in 2024-09-02T10:17:13.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:13.782 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:13.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:13 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2264373233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:17:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:13 smithi162 ceph-mon[20279]: osdmap e502: 12 total, 12 up, 10 in 2024-09-02T10:17:13.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:13.911 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:13.912 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:14.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:14.043 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:14.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:14.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:14.046 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:14.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:14.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:14.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:14.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:17:14.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:14.301 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:14.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:14.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:14.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:14.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:14.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:14.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:14.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:14.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:14.608 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:14.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:14.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:14.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:14.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:14 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:17:14.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:14 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:17:14.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:14 smithi187 ceph-mon[20072]: pgmap v1169: 47 pgs: 47 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 980 op/s 2024-09-02T10:17:14.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:14 smithi187 ceph-mon[20072]: osdmap e503: 12 total, 12 up, 10 in 2024-09-02T10:17:14.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1267879525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:14.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:14.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:14.822 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:14 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:17:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:14 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:17:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:14 smithi162 ceph-mon[20279]: pgmap v1169: 47 pgs: 47 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 980 op/s 2024-09-02T10:17:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:14 smithi162 ceph-mon[20279]: osdmap e503: 12 total, 12 up, 10 in 2024-09-02T10:17:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1267879525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:14.856 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:17:14.857 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:17:14.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:14 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:17:14.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:14 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:17:14.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:14 smithi151 ceph-mon[16639]: pgmap v1169: 47 pgs: 47 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 980 op/s 2024-09-02T10:17:14.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:14 smithi151 ceph-mon[16639]: osdmap e503: 12 total, 12 up, 10 in 2024-09-02T10:17:14.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1267879525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:14.884 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:17:14.884 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:17:14.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:17:14.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:17:14.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:14.940 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:17:14.940 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:17:14.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:14.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:14.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:17:14.963 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:17:14.991 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:17:14.991 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:17:15.020 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:17:15.020 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:17:15.044 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:17:15.044 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:17:15.051 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:15.071 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:17:15.072 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:17:15.077 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:15.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:15.098 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:17:15.098 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:17:15.129 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:17:15.129 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:17:15.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:17:15.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:17:15.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:15.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:15.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:15.212 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:17:15.212 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:17:15.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:17:15.307 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:15.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:15.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:15.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:15.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:15.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-02T10:17:15.570 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:15.599 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:15.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:15.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:15.728 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:15.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3091598850' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:17:15.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:15.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3091598850' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:17:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:15.856 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:15.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3091598850' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:17:15.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:17:15.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:16.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:16.020 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:16.021 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:16.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:16.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:16.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:16.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:16.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:16.299 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:16.299 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:16.345 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:16.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:16.431 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:16.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:16.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:16.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:16.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:16.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:16.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:16.683 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:16.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:16.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:16.686 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:16 smithi151 ceph-mon[16639]: 5.d scrub starts 2024-09-02T10:17:16.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:16 smithi151 ceph-mon[16639]: pgmap v1171: 47 pgs: 47 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 799 op/s 2024-09-02T10:17:16.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3091598850' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:17:16.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:16 smithi151 ceph-mon[16639]: osdmap e504: 12 total, 12 up, 10 in 2024-09-02T10:17:16.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:16 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:17:16.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:16 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:17:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:16 smithi187 ceph-mon[20072]: 5.d scrub starts 2024-09-02T10:17:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:16 smithi187 ceph-mon[20072]: pgmap v1171: 47 pgs: 47 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 799 op/s 2024-09-02T10:17:16.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3091598850' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:17:16.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:16 smithi187 ceph-mon[20072]: osdmap e504: 12 total, 12 up, 10 in 2024-09-02T10:17:16.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:16 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:17:16.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:16 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:17:16.760 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:16.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:16.811 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:16.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:16 smithi162 ceph-mon[20279]: 5.d scrub starts 2024-09-02T10:17:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:16 smithi162 ceph-mon[20279]: pgmap v1171: 47 pgs: 47 active+clean; 3.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 799 op/s 2024-09-02T10:17:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3091598850' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:17:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:16 smithi162 ceph-mon[20279]: osdmap e504: 12 total, 12 up, 10 in 2024-09-02T10:17:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:16 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:17:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:16 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:17:16.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:16.930 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:17:16.930 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:17:16.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:16.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:16.950 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:17.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:17.094 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:17.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:17.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:17.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:17.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:17.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:17.332 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:17.358 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:17.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:17.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:17.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:17.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:17.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:17.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:17:17.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:17.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:17.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:17 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:17:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:17 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:17:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:17 smithi151 ceph-mon[16639]: 5.6 scrub starts 2024-09-02T10:17:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:17 smithi151 ceph-mon[16639]: 5.d scrub ok 2024-09-02T10:17:17.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:17.738 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:17.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:17.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:17 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:17:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:17 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:17:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:17 smithi187 ceph-mon[20072]: 5.6 scrub starts 2024-09-02T10:17:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:17 smithi187 ceph-mon[20072]: 5.d scrub ok 2024-09-02T10:17:17.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:17.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:17 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:17:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:17 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:17:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:17 smithi162 ceph-mon[20279]: 5.6 scrub starts 2024-09-02T10:17:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:17 smithi162 ceph-mon[20279]: 5.d scrub ok 2024-09-02T10:17:17.864 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:17.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:17.949 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:17.965 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:18.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:18.006 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:18.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:17:18.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:18.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:18.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:18.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:18.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:18.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:18.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:18.321 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:18.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:18.390 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:18.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:18.481 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:17:18.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:18.514 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:18.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:18.528 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:17:18.528 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:17:18.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:18.638 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:18.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:18.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:18.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:18 smithi187 ceph-mon[20072]: 5.0 scrub starts 2024-09-02T10:17:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:18 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:17:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:18 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:17:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:18 smithi187 ceph-mon[20072]: pgmap v1173: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-02T10:17:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:18 smithi187 ceph-mon[20072]: 5.6 scrub ok 2024-09-02T10:17:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3233705338' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:18.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:18 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:17:18.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:18 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:17:18.762 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:18.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:18 smithi162 ceph-mon[20279]: 5.0 scrub starts 2024-09-02T10:17:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:18 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:17:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:18 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:17:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:18 smithi162 ceph-mon[20279]: pgmap v1173: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-02T10:17:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:18 smithi162 ceph-mon[20279]: 5.6 scrub ok 2024-09-02T10:17:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3233705338' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:18 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:17:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:18 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:17:18.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:18.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:18.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:18.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:18.906 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:18 smithi151 ceph-mon[16639]: 5.0 scrub starts 2024-09-02T10:17:18.906 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:18 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:17:18.906 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:18 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:17:18.906 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:18 smithi151 ceph-mon[16639]: pgmap v1173: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-02T10:17:18.906 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:18 smithi151 ceph-mon[16639]: 5.6 scrub ok 2024-09-02T10:17:18.906 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3233705338' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:18.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:18 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:17:18.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:18 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:17:18.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:19.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:19.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:19.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:19.143 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:19.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:19.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:19.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:17:19.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:19.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:19.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:19.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:19.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:19.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:19.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:19.493 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:19 smithi187 ceph-mon[20072]: 5.0 scrub ok 2024-09-02T10:17:19.493 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:19 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:17:19.493 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:19 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:17:19.493 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3233705338' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:19.493 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:19 smithi187 ceph-mon[20072]: osdmap e505: 12 total, 12 up, 10 in 2024-09-02T10:17:19.497 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:19.537 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:19.537 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:19.576 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:19 smithi151 ceph-mon[16639]: 5.0 scrub ok 2024-09-02T10:17:19.576 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:19 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:17:19.576 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:19 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:17:19.576 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3233705338' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:19.576 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:19 smithi151 ceph-mon[16639]: osdmap e505: 12 total, 12 up, 10 in 2024-09-02T10:17:19.593 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:19.609 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:17:19.609 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:17:19.635 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:17:19.636 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:17:19.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:19.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:17:19.660 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:17:19.668 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:19.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:19.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:19.686 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:17:19.686 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:17:19.711 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:17:19.712 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:17:19.735 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:17:19.735 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:17:19.763 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:17:19.763 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:17:19.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:19.785 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:17:19.785 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:17:19.811 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:19.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:19.817 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:17:19.817 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:17:19.843 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:17:19.844 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:17:19.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:19 smithi162 ceph-mon[20279]: 5.0 scrub ok 2024-09-02T10:17:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:19 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:17:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:19 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:17:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3233705338' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:19 smithi162 ceph-mon[20279]: osdmap e505: 12 total, 12 up, 10 in 2024-09-02T10:17:19.869 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:17:19.870 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:17:19.899 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:17:19.899 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:17:19.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:19.938 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:19.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:20.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:20.063 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:20.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:20.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:20.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:20.189 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:20.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:20.216 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:17:20.216 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:17:20.216 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:17:20.216 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:17:20.216 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:17:20.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:20.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:20.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:20.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:20.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:20.441 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:20.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:20.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:20.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:20.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:20.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:20.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:20.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:20.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:20.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:20 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:17:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:20 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:17:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:20 smithi187 ceph-mon[20072]: pgmap v1175: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 793 op/s 2024-09-02T10:17:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2125058343' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:20.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:20.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:20.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:20.825 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:20 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:17:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:20 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:17:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:20 smithi162 ceph-mon[20279]: pgmap v1175: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 793 op/s 2024-09-02T10:17:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2125058343' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:20.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:20.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:20 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:17:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:20 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:17:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:20 smithi151 ceph-mon[16639]: pgmap v1175: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 3.1 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 793 op/s 2024-09-02T10:17:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2125058343' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:20.948 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:20.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:21.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:21.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:17:21.074 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:21.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:21.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:21.201 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:21.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:21.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:21.303 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:21.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:21.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:21.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:21.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:17:21.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:21.455 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:21.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:21.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:21.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:21.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:21.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:21 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:17:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:21 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:17:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:21 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:17:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:21 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:17:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:21 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:17:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:21 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:17:21.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:21.729 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:17:21.730 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:17:21.730 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-02T10:17:21.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-02T10:17:21.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:21.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:21.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:21 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:17:21.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:21 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:17:21.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:21 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:17:21.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:21 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:17:21.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:21 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:17:21.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:21 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:17:21.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:21.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:21 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:17:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:21 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:17:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:21 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:17:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:21 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:17:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:21 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:17:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:21 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:17:21.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:21.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:21.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:21.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:22.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:22.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:22.073 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:17:22.073 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:17:22.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:17:22.073 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:25387 4.a e336/333 4:51410dd3:::benchmark_data_smithi187_76542_object25386:head v 336'1582, mlcod=325'647)", 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 356.103713368, 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20680125099999999, 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964243+0000", 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964244+0000", 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5900000000000005e-07 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:25.964284+0000", 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0487e-05 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169804+0000", 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20551941600000001 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169809+0000", 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6319999999999996e-06 2024-09-02T10:17:22.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169867+0000", 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7938e-05 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.169895+0000", 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7430999999999999e-05 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.171002+0000", 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011072790000000001 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:26.171044+0000", 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2109e-05 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103692 4.19 4.40dd9139 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:17:22.078 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 275.97221297800002, 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.993593167, 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103692 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:17:22.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095744+0000", 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6899999999999996e-07 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095748+0000", 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4499999999999997e-06 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095758+0000", 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2140000000000002e-06 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095764+0000", 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5690000000000001e-06 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095808+0000", 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3588999999999999e-05 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095814+0000", 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2299999999999996e-06 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089305+0000", 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99349078599999996 2024-09-02T10:17:22.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089336+0000", 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.146e-05 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:103721 4.e e384/340 4:78ce6951:::benchmark_data_smithi187_76542_object11385:head v 384'6547, mlcod=384'6544)", 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 275.67616301100003, 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.265184527, 2024-09-02T10:17:22.083 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391794+0000", 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4000000000000001e-07 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391799+0000", 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2009999999999998e-06 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654392+0000", 2024-09-02T10:17:22.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26259298599999997 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654398+0000", 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7880000000000003e-06 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654446+0000", 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8183000000000003e-05 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654470+0000", 2024-09-02T10:17:22.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4093000000000001e-05 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.656943+0000", 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024726489999999999 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.656978+0000", 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4887e-05 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103725 4.19 4.a40a4399 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 275.631468332, 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65288606400000004, 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103725 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.088 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436489+0000", 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2750000000000001e-06 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436494+0000", 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4660000000000001e-06 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436506+0000", 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2123000000000001e-05 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436512+0000", 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6749999999999996e-06 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436566+0000", 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3285000000000002e-05 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436572+0000", 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6019999999999996e-06 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089367+0000", 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65279432500000001 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089374+0000", 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3130000000000001e-06 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103743 4.19 4.222eb379 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:17:22.092 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 275.39862852300001, 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42007966200000002, 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103743 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669329+0000", 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.221e-06 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669334+0000", 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1340000000000003e-06 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669346+0000", 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2155e-05 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669351+0000", 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9810000000000003e-06 2024-09-02T10:17:22.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669433+0000", 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1617000000000001e-05 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669445+0000", 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.207e-05 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089400+0000", 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41995552600000002 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089407+0000", 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9580000000000001e-06 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103767 4.19 4.bc594c79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:17:22.097 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 275.29817414600001, 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31965745400000001, 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103767 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769785+0000", 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5299999999999999e-06 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769796+0000", 2024-09-02T10:17:22.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.164e-05 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769823+0000", 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7055000000000001e-05 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769828+0000", 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0350000000000001e-06 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769891+0000", 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2973999999999997e-05 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769903+0000", 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1569999999999999e-05 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089432+0000", 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31952966900000002 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089439+0000", 2024-09-02T10:17:22.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9809999999999997e-06 2024-09-02T10:17:22.102 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.102 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:181097 4.7 4:e7273704:::benchmark_data_smithi187_76542_object88761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:19.485221+0000", 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 182.58273554199999, 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23570571100000001, 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:17:22.103 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 181097 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485221+0000", 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485221+0000", 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485222+0000", 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.155e-06 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485226+0000", 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2409999999999997e-06 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485239+0000", 2024-09-02T10:17:22.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2782000000000001e-05 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485245+0000", 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9830000000000001e-06 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485290+0000", 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5024999999999997e-05 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485380+0000", 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9802999999999994e-05 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.485451+0000", 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1474e-05 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.487590+0000", 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021388309999999999 2024-09-02T10:17:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.674878+0000", 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18728809299999999 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.720859+0000", 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.045981517999999999 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.720895+0000", 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5024999999999998e-05 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:19.720926+0000", 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1780999999999999e-05 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.109 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:183023 4.e 4:7b0884ad:::benchmark_data_smithi187_76542_object90687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:21.990649+0000", 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 180.07730666200001, 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.209310995, 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183023 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990649+0000", 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990649+0000", 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990651+0000", 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.083e-06 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990655+0000", 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6070000000000001e-06 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990666+0000", 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0400999999999999e-05 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990671+0000", 2024-09-02T10:17:22.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1800000000000004e-06 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990712+0000", 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1507000000000001e-05 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990793+0000", 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1162000000000007e-05 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.990876+0000", 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2533000000000004e-05 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.994433+0000", 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035574809999999999 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:21.995109+0000", 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00067595600000000002 2024-09-02T10:17:22.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:22.199869+0000", 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20475953599999999 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:22.199920+0000", 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1403999999999998e-05 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:22.199960+0000", 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0145000000000003e-05 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:18186 5.0 5:0af1a60d:::benchmark_data_smithi187_89405_object18185:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.734715+0000", 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 144.33324148, 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.211869428, 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.116 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 18186 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734715+0000", 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734715+0000", 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734716+0000", 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.012e-06 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734736+0000", 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0123999999999999e-05 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734751+0000", 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4955e-05 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734779+0000", 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8138000000000001e-05 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734850+0000", 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0975000000000004e-05 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.734970+0000", 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000119677 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735115+0000", 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014499000000000001 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.736868+0000", 2024-09-02T10:17:22.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001753065 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.737123+0000", 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000255732 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946471+0000", 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20934799300000001 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946538+0000", 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6598000000000002e-05 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946584+0000", 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6168999999999998e-05 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.122 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:18189 5.0 5:053524f9:::benchmark_data_smithi187_89405_object18188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.735686+0000", 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 144.332270163, 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.211005743, 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 18189 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.123 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735686+0000", 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735686+0000", 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735687+0000", 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0419999999999999e-06 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735713+0000", 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5760000000000001e-05 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735728+0000", 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5184000000000001e-05 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735736+0000", 2024-09-02T10:17:22.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5789999999999998e-06 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.735969+0000", 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00023298500000000001 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.736084+0000", 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115158 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.736174+0000", 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0140000000000001e-05 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.737138+0000", 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00096445099999999996 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.749747+0000", 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012608546 2024-09-02T10:17:22.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946627+0000", 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.196880686 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946661+0000", 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3976000000000001e-05 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.946692+0000", 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0236e-05 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:22168 5.8 e441/431 5:166a9b24:::benchmark_data_smithi187_89405_object22167:head v 441'1424, mlcod=441'1421)", 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:00.180685+0000", 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 141.88727162399999, 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.211436923, 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.180685+0000", 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.180685+0000", 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.180685+0000", 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4000000000000001e-07 2024-09-02T10:17:22.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.180785+0000", 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9597000000000002e-05 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.390284+0000", 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20949867599999999 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.390291+0000", 2024-09-02T10:17:22.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6190000000000002e-06 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.390379+0000", 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7923999999999994e-05 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.390399+0000", 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.965e-05 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.392072+0000", 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001673477 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:00.392121+0000", 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9240000000000003e-05 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62902 5.0 5:0a3860dd:::benchmark_data_smithi187_89405_object62901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:17:22.133 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 107.432889539, 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28493924999999998, 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62902 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635068+0000", 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.502e-06 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635089+0000", 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0916000000000001e-05 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635100+0000", 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1243000000000001e-05 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635107+0000", 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5359999999999998e-06 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635155+0000", 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7939000000000003e-05 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635250+0000", 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5571999999999999e-05 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635339+0000", 2024-09-02T10:17:22.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9158000000000003e-05 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.636945+0000", 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001605555 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.637559+0000", 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00061385400000000005 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.919922+0000", 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28236284 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.919962+0000", 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0227e-05 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.920006+0000", 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3908e-05 2024-09-02T10:17:22.139 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:86595 5.a e473/431 5:52430871:::benchmark_data_smithi187_89405_object86594:head v 473'5443, mlcod=473'5440)", 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:58.506650+0000", 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 83.561305696999995, 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22450877699999999, 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506650+0000", 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506650+0000", 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506651+0000", 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.017e-06 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506675+0000", 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3232000000000001e-05 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.727778+0000", 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22110306800000001 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.727784+0000", 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4570000000000002e-06 2024-09-02T10:17:22.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.727856+0000", 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1624000000000004e-05 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.727875+0000", 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9249000000000002e-05 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731117+0000", 2024-09-02T10:17:22.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032417129999999998 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731159+0000", 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2416999999999997e-05 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:86613 5.0 5:09f926a4:::benchmark_data_smithi187_89405_object86612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:58.523804+0000", 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 83.544152625999999, 2024-09-02T10:17:22.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.211303772, 2024-09-02T10:17:22.145 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.145 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:22.146 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.146 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.146 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:22.146 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:22.146 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 86613 2024-09-02T10:17:22.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.523804+0000", 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.523804+0000", 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.523804+0000", 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8599999999999998e-07 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.523853+0000", 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9271999999999999e-05 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.523862+0000", 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.636e-06 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.523866+0000", 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3150000000000002e-06 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.523902+0000", 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5379000000000001e-05 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.523981+0000", 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8804e-05 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.524064+0000", 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3562999999999999e-05 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.525765+0000", 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001700958 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.530227+0000", 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004462227 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.734920+0000", 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.204692556 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.735058+0000", 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013833699999999999 2024-09-02T10:17:22.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.735107+0000", 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9039000000000003e-05 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:86613 5.0 e473/431 ondisk, result = 0)", 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:58.525902+0000", 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 83.542054144999994, 2024-09-02T10:17:22.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20921139999999999, 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.525902+0000", 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.525902+0000", 2024-09-02T10:17:22.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.525902+0000", 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1399999999999997e-07 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.525906+0000", 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0410000000000002e-06 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.734404+0000", 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20849890900000001 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.734411+0000", 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7329999999999997e-06 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.734906+0000", 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00049483000000000003 2024-09-02T10:17:22.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.734914+0000", 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2640000000000008e-06 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.735113+0000", 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00019910899999999999 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:126223 5.4 5:2d883f96:::benchmark_data_smithi187_89405_object22618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 40.142028461999999, 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32322968800000001, 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126223 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.157 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925929+0000", 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2020000000000001e-06 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925933+0000", 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3479999999999997e-06 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925945+0000", 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1559999999999999e-05 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925952+0000", 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4120000000000004e-06 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925997+0000", 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4759999999999998e-05 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.926077+0000", 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0203000000000004e-05 2024-09-02T10:17:22.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.926145+0000", 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8311000000000004e-05 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929053+0000", 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029072289999999999 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.989494+0000", 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.060441307999999999 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249062+0000", 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25956764199999999 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249115+0000", 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2896000000000003e-05 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.162 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249157+0000", 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2817e-05 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:126223 5.4 e490/431 ondisk, result = 0)", 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 40.138684347999998, 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31989100399999998, 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.163 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2900000000000003e-07 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929276+0000", 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4989999999999999e-06 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.248945+0000", 2024-09-02T10:17:22.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31966908500000002 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.248950+0000", 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1749999999999996e-06 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249048+0000", 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7676000000000005e-05 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249056+0000", 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2439999999999993e-06 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249163+0000", 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106696 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:132469 5.0 5:007b3dda:::benchmark_data_smithi187_89405_object28864:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e493)", 2024-09-02T10:17:22.167 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:49.076763+0000", 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 32.991192916000003, 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22282516299999999, 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 132469 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.076763+0000", 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.076763+0000", 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.076764+0000", 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4300000000000002e-07 2024-09-02T10:17:22.169 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.076768+0000", 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.109e-06 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.076776+0000", 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8299999999999996e-06 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.170 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.076781+0000", 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5010000000000004e-06 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.080821+0000", 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004039328 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.080937+0000", 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011602 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:17:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.081037+0000", 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010009399999999999 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.094662+0000", 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013625396999999999 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.094705+0000", 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2688000000000002e-05 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.172 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.299521+0000", 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20481611599999999 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.299558+0000", 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7117999999999999e-05 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.299588+0000", 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0219e-05 2024-09-02T10:17:22.173 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:132478 5.0 5:024d4078:::benchmark_data_smithi187_89405_object28873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e493)", 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:49.084710+0000", 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 32.983246002999998, 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21845674200000001, 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.174 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 132478 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.084710+0000", 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.084710+0000", 2024-09-02T10:17:22.175 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.084712+0000", 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.765e-06 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.084718+0000", 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1489999999999999e-06 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.176 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.084734+0000", 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5588999999999999e-05 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.084741+0000", 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3250000000000002e-06 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.087135+0000", 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023937020000000002 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.177 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.087290+0000", 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015555000000000001 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.087396+0000", 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000105726 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.094724+0000", 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0073284719999999999 2024-09-02T10:17:22.178 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.133744+0000", 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.039019877000000001 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.303072+0000", 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16932782800000001 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.179 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.303124+0000", 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1678999999999998e-05 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:49.303167+0000", 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3080000000000001e-05 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:141659 5.0 5:0d7eb9e1:::benchmark_data_smithi187_89405_object38054:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:59.188482+0000", 2024-09-02T10:17:22.180 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 22.879474557000002, 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20682020000000001, 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 141659 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.188482+0000", 2024-09-02T10:17:22.181 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.188482+0000", 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.188483+0000", 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7850000000000001e-06 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.188488+0000", 2024-09-02T10:17:22.182 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1089999999999997e-06 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.188499+0000", 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0378e-05 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.188504+0000", 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9540000000000003e-06 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:22.183 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.188547+0000", 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3538999999999998e-05 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.188632+0000", 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5118999999999996e-05 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.188749+0000", 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116932 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.184 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.192427+0000", 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036778050000000001 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.193437+0000", 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001009883 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.395226+0000", 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.201788674 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.185 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.395259+0000", 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3148000000000002e-05 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:59.395302+0000", 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2874000000000003e-05 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:22.186 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:17:22.221 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:22.222 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:22.236 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:22.252 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:22.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:22.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:22.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:22.450 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:22.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:22.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:22.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:22.611 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:22.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:22.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:22.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:22.714 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:22 smithi187 ceph-mon[20072]: 5.4 scrub starts 2024-09-02T10:17:22.714 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:22 smithi187 ceph-mon[20072]: 5.8 scrub starts 2024-09-02T10:17:22.714 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:22 smithi187 ceph-mon[20072]: 5.4 scrub ok 2024-09-02T10:17:22.714 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:17:22.714 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:17:22.715 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:17:22.737 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:22.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:22.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:22.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:22 smithi162 ceph-mon[20279]: 5.4 scrub starts 2024-09-02T10:17:22.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:22 smithi162 ceph-mon[20279]: 5.8 scrub starts 2024-09-02T10:17:22.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:22 smithi162 ceph-mon[20279]: 5.4 scrub ok 2024-09-02T10:17:22.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:17:22.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:17:22.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:17:22.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:22 smithi151 ceph-mon[16639]: 5.4 scrub starts 2024-09-02T10:17:22.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:22 smithi151 ceph-mon[16639]: 5.8 scrub starts 2024-09-02T10:17:22.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:22 smithi151 ceph-mon[16639]: 5.4 scrub ok 2024-09-02T10:17:22.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:17:22.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:17:22.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:17:22.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:22.867 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:22.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:22.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:22.994 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:22.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:23.041 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:17:23.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:23.123 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:23.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:23.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:23.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:23.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:17:23.299 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:17:23.300 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:17:23.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:23.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:23.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:23.454 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:23.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:23.503 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:23.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:23.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:23.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:23 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:17:23.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:23 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:17:23.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:23 smithi151 ceph-mon[16639]: 5.8 scrub ok 2024-09-02T10:17:23.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:23 smithi151 ceph-mon[16639]: osdmap e506: 12 total, 12 up, 10 in 2024-09-02T10:17:23.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2746012691' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:17:23.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:23 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:17:23.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:23 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:17:23.636 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:23.637 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:23.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:23 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:17:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:23 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:17:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:23 smithi187 ceph-mon[20072]: 5.8 scrub ok 2024-09-02T10:17:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:23 smithi187 ceph-mon[20072]: osdmap e506: 12 total, 12 up, 10 in 2024-09-02T10:17:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2746012691' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:17:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:23 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:17:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:23 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:17:23.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:23.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:23.841 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:23 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:17:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:23 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:17:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:23 smithi162 ceph-mon[20279]: 5.8 scrub ok 2024-09-02T10:17:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:23 smithi162 ceph-mon[20279]: osdmap e506: 12 total, 12 up, 10 in 2024-09-02T10:17:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2746012691' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:17:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:23 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:17:23.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:23 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:17:23.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:23.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:23.896 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:17:23.961 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:17:23.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-02T10:17:23.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:23.999 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:17:24.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:24.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:24.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:24.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:24.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:24.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:24.159 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:24.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:17:24.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:24.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:24.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:24.288 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:24.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:24.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:24.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:24.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:24.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:24.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:24.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:24.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:24.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:24.663 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:24.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:24 smithi187 ceph-mon[20072]: pgmap v1178: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 939 op/s 2024-09-02T10:17:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:24 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2746012691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:17:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:24 smithi187 ceph-mon[20072]: osdmap e507: 12 total, 12 up, 10 in 2024-09-02T10:17:24.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:24.795 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:24.796 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:24.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:24 smithi151 ceph-mon[16639]: pgmap v1178: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 939 op/s 2024-09-02T10:17:24.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:24 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:24.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2746012691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:17:24.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:24 smithi151 ceph-mon[16639]: osdmap e507: 12 total, 12 up, 10 in 2024-09-02T10:17:24.841 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:24 smithi162 ceph-mon[20279]: pgmap v1178: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 939 op/s 2024-09-02T10:17:24.841 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:24 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:24.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2746012691' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:17:24.842 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:24 smithi162 ceph-mon[20279]: osdmap e507: 12 total, 12 up, 10 in 2024-09-02T10:17:24.875 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:24.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:24.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:24.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:25.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:25.064 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:25.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:25.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:25.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:25.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:25.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:25.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:25.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:25.321 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:25.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:25.422 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:25.426 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:25.447 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:25.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:17:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:17:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:17:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:17:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:17:25.455 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:17:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:17:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:17:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:17:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:17:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:17:25.456 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:17:25.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:25.573 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:25.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:25.675 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:25.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:25.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:25.802 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:25.839 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:25.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:25.889 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:17:25.889 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:17:25.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.889 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:163753 5.d 5:bbaa917b:::benchmark_data_smithi187_89405_object60148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:17:25.889 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:25.878470+0000", 2024-09-02T10:17:25.889 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0082718089999999998, 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0083028760000000007, 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 163753 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.878470+0000", 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:25.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.878470+0000", 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.878472+0000", 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2729999999999999e-06 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:25.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.878477+0000", 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8900000000000004e-06 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.878489+0000", 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1905e-05 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.878495+0000", 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5729999999999998e-06 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.878555+0000", 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0078000000000003e-05 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.878694+0000", 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000138801 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.878825+0000", 2024-09-02T10:17:25.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000130865 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.883610+0000", 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047848109999999999 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:163742 5.9 e507/431 5:98c58a0a:::benchmark_data_smithi187_89405_object60137:head v 507'10356, mlcod=507'10354)", 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:25.859465+0000", 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.027277084, 2024-09-02T10:17:25.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.027383431999999999, 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.859465+0000", 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.859465+0000", 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.859466+0000", 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1689999999999999e-06 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.859474+0000", 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4220000000000003e-06 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.883883+0000", 2024-09-02T10:17:25.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.024409040999999999 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.883889+0000", 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2820000000000002e-06 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.883949+0000", 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9802000000000001e-05 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:25.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.883967+0000", 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8597000000000002e-05 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:163729 5.d 5:be596b23:::benchmark_data_smithi187_89405_object60124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:25.837479+0000", 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.049263239, 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.049418931999999999, 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:17:25.898 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 163729 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.837479+0000", 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.837479+0000", 2024-09-02T10:17:25.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.837480+0000", 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.407e-06 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.837486+0000", 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2240000000000003e-06 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:17:25.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.837501+0000", 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5486e-05 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.837509+0000", 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4940000000000003e-06 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.837583+0000", 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3759999999999999e-05 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:17:25.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.837715+0000", 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013128599999999999 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.837828+0000", 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011306000000000001 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.843459+0000", 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0056313930000000002 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:17:25.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:17:25.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:25.859585+0000", 2024-09-02T10:17:25.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016126430000000001 2024-09-02T10:17:25.903 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:25.903 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:17:25.903 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:25.903 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:17:25.903 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:17:25.903 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 3 2024-09-02T10:17:25.903 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:17:25.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:25.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:25.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:26.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:26.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:26.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:26.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:26.219 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:26.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:26.272 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:17:26.272 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:17:26.285 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:17:26.285 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:17:26.309 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:17:26.310 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:17:26.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:26.336 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:17:26.337 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:17:26.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:26.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:26.360 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:17:26.360 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:17:26.386 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:17:26.387 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:17:26.409 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:17:26.409 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:17:26.433 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:17:26.433 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:17:26.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:26.462 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:17:26.462 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:17:26.471 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:26.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:26.489 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:17:26.489 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:17:26.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:17:26.516 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:17:26.543 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:17:26.543 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:17:26.563 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:26 smithi187 ceph-mon[20072]: pgmap v1180: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 675 op/s 2024-09-02T10:17:26.563 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3556636457' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:26.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:26.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:26.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:26.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:17:26.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:26 smithi151 ceph-mon[16639]: pgmap v1180: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 675 op/s 2024-09-02T10:17:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3556636457' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:26.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:26.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:26.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:26.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:17:26.730 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:17:26.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-02T10:17:26.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:26.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:26 smithi162 ceph-mon[20279]: pgmap v1180: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 2.8 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 675 op/s 2024-09-02T10:17:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3556636457' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:26.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:17:26.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:26.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:26.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:26.982 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:26.990 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:26.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:27.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:27.122 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:27.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:27.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:27.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:27.255 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:27.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:27.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:27.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:27.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:27.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:27.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:27.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:27.576 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:17:27.577 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:17:27.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:27.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:27.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:27.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:27.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:27.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:27.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:27.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:27.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:27.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:27.948 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:27.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:28.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:28.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:28.121 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:28.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:28.152 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:28.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:28.276 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:28.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:28.315 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:17:28.378 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:28.404 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:28.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:28.491 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:28.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:28.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:28.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:28.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:28.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:28.659 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:28 smithi151 ceph-mon[16639]: pgmap v1181: 47 pgs: 47 active+clean; 2.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 776 op/s 2024-09-02T10:17:28.659 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:28 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:17:28.659 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:28 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:17:28.659 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:28 smithi151 ceph-mon[16639]: osdmap e508: 12 total, 12 up, 10 in 2024-09-02T10:17:28.663 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:28.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:28.686 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:17:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:28 smithi187 ceph-mon[20072]: pgmap v1181: 47 pgs: 47 active+clean; 2.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 776 op/s 2024-09-02T10:17:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:28 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:17:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:28 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:17:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:28 smithi187 ceph-mon[20072]: osdmap e508: 12 total, 12 up, 10 in 2024-09-02T10:17:28.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:28.790 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:28.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:28.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:28 smithi162 ceph-mon[20279]: pgmap v1181: 47 pgs: 47 active+clean; 2.7 GiB data, 12 GiB used, 882 GiB / 894 GiB avail; 0 B/s wr, 776 op/s 2024-09-02T10:17:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:28 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:17:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:28 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:17:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:28 smithi162 ceph-mon[20279]: osdmap e508: 12 total, 12 up, 10 in 2024-09-02T10:17:28.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:28.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:28.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:29.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:29.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:29.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:29.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:29.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:29.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:29.182 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:29.263 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:17:29.263 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:17:29.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:29.311 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:29.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:29.354 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:17:29.354 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:17:29.354 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:17:29.355 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:17:29.355 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:17:29.355 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:17:29.355 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:17:29.355 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:17:29.355 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:17:29.355 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:17:29.355 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:17:29.355 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:17:29.413 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:29.446 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:29.447 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:29.545 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:17:29.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:29.575 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:29.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:29.576 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:29.677 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:29.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/152880313' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:29.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:29.703 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:29.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:29.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:29.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:29.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:29.842 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/152880313' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/152880313' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:29.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:17:29.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:29.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:29.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:30.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:30.099 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:30.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:30.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:30.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:30.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:30.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:30.327 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:30.356 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:30.357 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:30.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:30.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:30.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:30.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:30.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:30.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:30.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:30 smithi151 ceph-mon[16639]: pgmap v1183: 47 pgs: 47 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 711 op/s 2024-09-02T10:17:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:30 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/152880313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:30 smithi151 ceph-mon[16639]: osdmap e509: 12 total, 12 up, 10 in 2024-09-02T10:17:30.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:30.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:30.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:30.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:30.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:30 smithi162 ceph-mon[20279]: pgmap v1183: 47 pgs: 47 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 711 op/s 2024-09-02T10:17:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:30 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/152880313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:30 smithi162 ceph-mon[20279]: osdmap e509: 12 total, 12 up, 10 in 2024-09-02T10:17:30.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:30.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:30.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:30.916 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:17:30.916 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:17:30.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-02T10:17:30.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:31.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:31.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:30 smithi187 ceph-mon[20072]: pgmap v1183: 47 pgs: 47 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 711 op/s 2024-09-02T10:17:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:30 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/152880313' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:30 smithi187 ceph-mon[20072]: osdmap e509: 12 total, 12 up, 10 in 2024-09-02T10:17:31.041 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:31.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:31.126 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:31.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:31.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:31.228 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:31.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:17:31.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:31.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:31.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:31.390 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:31.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:31.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:31.516 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:31.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:31.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:31.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:31.654 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:17:31.655 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:17:31.675 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:31.684 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:31.687 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:17:31.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:17:31.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:17:31.719 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:17:31.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:31.745 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:17:31.746 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:17:31.771 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:31.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:31.773 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:17:31.773 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:17:31.802 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:17:31.802 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:17:31.828 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:17:31.829 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:17:31.856 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:17:31.856 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:17:31.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:31.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:17:31.885 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:17:31.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:31.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:31.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:31.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:17:31.913 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:17:31.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:17:31.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:17:31.979 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:17:31.979 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:17:32.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:32.026 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:32.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:32.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:32.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:32.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:32.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:32.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:32.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:32.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:32.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:32.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:17:32.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:32.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:32.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:32.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:32.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:32.559 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:32.559 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:32.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:32.683 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:32.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:32.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:32.731 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:32 smithi151 ceph-mon[16639]: pgmap v1185: 47 pgs: 47 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 592 op/s 2024-09-02T10:17:32.732 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3841041628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:32.784 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:32.820 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:32.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:32.821 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:32.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:32 smithi162 ceph-mon[20279]: pgmap v1185: 47 pgs: 47 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 592 op/s 2024-09-02T10:17:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3841041628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:32.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:32.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:32.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:32 smithi187 ceph-mon[20072]: pgmap v1185: 47 pgs: 47 active+clean; 2.6 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 592 op/s 2024-09-02T10:17:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3841041628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:33.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:33.078 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:33.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:33.079 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:33.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:33.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:33.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:33.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:33.385 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:33.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:33.487 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:33.489 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:17:33.489 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:17:33.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.489 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25933.0:169582 5.0 e509/431 5:065e2c8f:::benchmark_data_smithi187_89405_object65977:head v 509'10507, mlcod=509'10505)", 2024-09-02T10:17:33.489 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:17:32.728126+0000", 2024-09-02T10:17:33.489 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.002939938, 2024-09-02T10:17:33.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029667600000000001, 2024-09-02T10:17:33.489 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:17:33.489 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.728126+0000", 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.728126+0000", 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.728127+0000", 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6799999999999999e-07 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.728132+0000", 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9540000000000003e-06 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.728157+0000", 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5650999999999999e-05 2024-09-02T10:17:33.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.728164+0000", 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6649999999999997e-06 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.728210+0000", 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5574000000000001e-05 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:17:33.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.728225+0000", 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5089000000000001e-05 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25933.0:169585 5.2 e509/431 5:4836a0f7:::benchmark_data_smithi187_89405_object65980:head v 509'10427, mlcod=509'10425)", 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:17:32.730719+0000", 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00034722799999999999, 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00042759999999999999, 2024-09-02T10:17:33.493 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.730719+0000", 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.730719+0000", 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.730719+0000", 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599999999999997e-07 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.730723+0000", 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1799999999999998e-06 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.730741+0000", 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7659000000000002e-05 2024-09-02T10:17:33.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.730747+0000", 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4009999999999996e-06 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.730787+0000", 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0843000000000002e-05 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:17:33.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:32.730801+0000", 2024-09-02T10:17:33.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3526999999999999e-05 2024-09-02T10:17:33.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:33.497 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:17:33.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:33.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:33.497 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:17:33.497 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 2 2024-09-02T10:17:33.497 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:17:33.515 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:33.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:33.515 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:33.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:33.642 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:33.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:33.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:33.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:33.779 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:33.781 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:33.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:33.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:33.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:33.918 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:17:33.918 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:17:33.918 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:17:33.918 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:17:33.919 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:17:33.919 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:17:33.919 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:17:33.919 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:17:33.919 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:17:33.919 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:17:33.919 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:17:33.919 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:17:34.006 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:34.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:34.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:34.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:34.168 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:34.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:34.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:34.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:34.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:34.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:34.425 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:34.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:34.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:34.557 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:34.557 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:34.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:34.685 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:34.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:34.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:34.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:34.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:34.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:34.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:34.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:34 smithi162 ceph-mon[20279]: pgmap v1186: 47 pgs: 47 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 905 op/s 2024-09-02T10:17:34.902 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:34 smithi151 ceph-mon[16639]: pgmap v1186: 47 pgs: 47 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 905 op/s 2024-09-02T10:17:34.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:34.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:34.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:34.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:34 smithi187 ceph-mon[20072]: pgmap v1186: 47 pgs: 47 active+clean; 2.4 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 905 op/s 2024-09-02T10:17:35.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:35.070 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:35.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:35.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:35.197 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:35.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:35.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:35.321 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:35.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:35.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:35.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:35.451 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:35.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:35.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:17:35.577 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:17:35.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-02T10:17:35.579 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:35.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:35.680 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:35.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:35.708 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:35.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:35.840 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:35.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:35.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:35.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:35.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:36.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:36.091 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:36.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:36.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:17:36.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:36.216 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:36.219 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:36.220 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:36.256 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:36.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:36.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:36.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:36.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:36.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:36.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:36.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:36.626 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:36 smithi151 ceph-mon[16639]: pgmap v1187: 47 pgs: 47 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 747 op/s 2024-09-02T10:17:36.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:36.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:36.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:36.731 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:36.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:36.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:36.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:36 smithi162 ceph-mon[20279]: pgmap v1187: 47 pgs: 47 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 747 op/s 2024-09-02T10:17:36.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:36.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:36.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:36.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:17:36.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:36 smithi187 ceph-mon[20072]: pgmap v1187: 47 pgs: 47 active+clean; 2.3 GiB data, 11 GiB used, 883 GiB / 894 GiB avail; 0 B/s wr, 747 op/s 2024-09-02T10:17:37.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:37.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:37.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:37.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:37.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:37.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:37.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:37.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:37.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:37.368 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:17:37.368 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:17:37.368 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:17:37.368 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:17:37.368 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:17:37.369 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:17:37.369 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:17:37.369 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:17:37.369 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:17:37.369 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:17:37.369 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:17:37.369 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:17:37.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:37.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:37.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:37.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:37.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:37.530 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:17:37.536 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:37.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:37.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:37.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4234346236' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:17:37.544 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:37 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:17:37.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:37.664 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:37.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:37.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:37.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:37.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:37.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:37.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4234346236' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:17:37.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:37 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:17:37.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:37.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:37.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:37.923 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:38.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4234346236' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:17:38.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:37 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:17:38.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:38.046 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:38.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:38.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:38.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:38.172 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:38.172 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:38.273 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:38.307 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:38.307 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:38.404 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:38.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:38.434 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:38.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:38.451 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:38.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:38.564 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:38.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:38.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:38.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:38 smithi151 ceph-mon[16639]: pgmap v1188: 47 pgs: 47 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 853 op/s 2024-09-02T10:17:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:38 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:38 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:17:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:38 smithi151 ceph-mon[16639]: osdmap e510: 12 total, 12 up, 10 in 2024-09-02T10:17:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:38 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:17:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:38 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:17:38.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:38.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:38.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:38.832 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:38.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:38 smithi162 ceph-mon[20279]: pgmap v1188: 47 pgs: 47 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 853 op/s 2024-09-02T10:17:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:38 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:38 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:17:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:38 smithi162 ceph-mon[20279]: osdmap e510: 12 total, 12 up, 10 in 2024-09-02T10:17:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:38 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:17:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:38 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:17:38.933 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:38.963 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:38.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:38 smithi187 ceph-mon[20072]: pgmap v1188: 47 pgs: 47 active+clean; 2.2 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 853 op/s 2024-09-02T10:17:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:38 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:38 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:17:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:38 smithi187 ceph-mon[20072]: osdmap e510: 12 total, 12 up, 10 in 2024-09-02T10:17:39.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:38 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:17:39.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:38 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:17:39.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:39.097 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:39.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:39.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:39.226 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:39.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:39.239 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:17:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:17:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:17:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:17:39.239 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:17:39.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:39.353 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:39.354 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:39.455 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:39.482 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:39.483 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:39.584 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:39.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:39.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:39.711 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:39.738 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:39.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:39.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:39.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:39 smithi151 ceph-mon[16639]: 5.f scrub starts 2024-09-02T10:17:39.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:17:39.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:17:39.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:17:39.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:17:39.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:39 smithi151 ceph-mon[16639]: osdmap e511: 12 total, 12 up, 10 in 2024-09-02T10:17:39.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:39 smithi151 ceph-mon[16639]: 5.f scrub ok 2024-09-02T10:17:39.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:39 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:17:39.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:39 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:17:39.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1673775436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:39.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:39 smithi162 ceph-mon[20279]: 5.f scrub starts 2024-09-02T10:17:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:17:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:17:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:17:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:17:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:39 smithi162 ceph-mon[20279]: osdmap e511: 12 total, 12 up, 10 in 2024-09-02T10:17:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:39 smithi162 ceph-mon[20279]: 5.f scrub ok 2024-09-02T10:17:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:39 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:17:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:39 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:17:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1673775436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:39.867 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:39.867 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:39.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:39.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:39.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:39 smithi187 ceph-mon[20072]: 5.f scrub starts 2024-09-02T10:17:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:17:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:17:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:17:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:17:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:39 smithi187 ceph-mon[20072]: osdmap e511: 12 total, 12 up, 10 in 2024-09-02T10:17:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:39 smithi187 ceph-mon[20072]: 5.f scrub ok 2024-09-02T10:17:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:39 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:17:40.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:39 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:17:40.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1673775436' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:40.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:40.124 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:40.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:17:40.180 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:17:40.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-02T10:17:40.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:40.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:40.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:40.308 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:17:40.309 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:17:40.339 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:17:40.339 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:17:40.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:40.363 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:17:40.364 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:17:40.384 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:40.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:40.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:40.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:17:40.393 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:17:40.393 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:17:40.425 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:17:40.426 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:17:40.462 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:17:40.463 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:17:40.486 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:40.501 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:17:40.501 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:17:40.509 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:40.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:40.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:40.534 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:17:40.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:17:40.563 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:17:40.564 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:17:40.589 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:17:40.589 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:17:40.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:40.615 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:17:40.615 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:17:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:40.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:40.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:40.652 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:17:40.652 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:17:40.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:40 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:17:40.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:40 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:17:40.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:40 smithi151 ceph-mon[16639]: 5.c scrub starts 2024-09-02T10:17:40.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:40 smithi151 ceph-mon[16639]: 5.c scrub ok 2024-09-02T10:17:40.680 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:40 smithi151 ceph-mon[16639]: pgmap v1191: 47 pgs: 47 active+clean; 2.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 986 op/s 2024-09-02T10:17:40.680 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:40 smithi151 ceph-mon[16639]: osdmap e512: 12 total, 12 up, 10 in 2024-09-02T10:17:40.680 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:40 smithi151 ceph-mon[16639]: 5.a scrub starts 2024-09-02T10:17:40.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:17:40.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:40.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:40.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:40 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:17:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:40 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:17:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:40 smithi162 ceph-mon[20279]: 5.c scrub starts 2024-09-02T10:17:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:40 smithi162 ceph-mon[20279]: 5.c scrub ok 2024-09-02T10:17:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:40 smithi162 ceph-mon[20279]: pgmap v1191: 47 pgs: 47 active+clean; 2.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 986 op/s 2024-09-02T10:17:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:40 smithi162 ceph-mon[20279]: osdmap e512: 12 total, 12 up, 10 in 2024-09-02T10:17:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:40 smithi162 ceph-mon[20279]: 5.a scrub starts 2024-09-02T10:17:40.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:40.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:40.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:40 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:17:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:40 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:17:41.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:40 smithi187 ceph-mon[20072]: 5.c scrub starts 2024-09-02T10:17:41.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:40 smithi187 ceph-mon[20072]: 5.c scrub ok 2024-09-02T10:17:41.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:40 smithi187 ceph-mon[20072]: pgmap v1191: 47 pgs: 47 active+clean; 2.1 GiB data, 10 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 986 op/s 2024-09-02T10:17:41.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:40 smithi187 ceph-mon[20072]: osdmap e512: 12 total, 12 up, 10 in 2024-09-02T10:17:41.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:40 smithi187 ceph-mon[20072]: 5.a scrub starts 2024-09-02T10:17:41.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:41.036 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:41.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:41.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:41.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:41.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:41.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:41.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:41.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:41.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:41.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:41.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:41.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:41.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:41.544 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:41.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:41.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:41.618 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:41.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:41.714 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:41.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:41.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:41.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:41 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:17:41.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:41 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:17:41.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:41 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:17:41.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:41 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:17:41.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:41 smithi151 ceph-mon[16639]: 5.a scrub ok 2024-09-02T10:17:41.737 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:41 smithi151 ceph-mon[16639]: 5.b scrub starts 2024-09-02T10:17:41.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:41.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:41.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:41.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:41 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:17:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:41 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:17:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:41 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:17:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:41 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:17:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:41 smithi162 ceph-mon[20279]: 5.a scrub ok 2024-09-02T10:17:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:41 smithi162 ceph-mon[20279]: 5.b scrub starts 2024-09-02T10:17:41.856 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:41.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:41.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:41.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:41.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:41 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:17:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:41 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:17:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:41 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:17:42.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:41 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:17:42.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:41 smithi187 ceph-mon[20072]: 5.a scrub ok 2024-09-02T10:17:42.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:41 smithi187 ceph-mon[20072]: 5.b scrub starts 2024-09-02T10:17:42.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:42.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:42.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:42.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:42.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:42.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:42.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:42.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:42.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:42.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:42.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:42.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:42.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:42.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:42.489 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:42.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:42.615 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:42.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:42.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:42.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:42.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:42.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:42 smithi162 ceph-mon[20279]: pgmap v1193: 47 pgs: 47 active+clean; 2.0 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 731 op/s 2024-09-02T10:17:42.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:42 smithi162 ceph-mon[20279]: 5.b scrub ok 2024-09-02T10:17:42.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:42.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:42 smithi151 ceph-mon[16639]: pgmap v1193: 47 pgs: 47 active+clean; 2.0 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 731 op/s 2024-09-02T10:17:42.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:42 smithi151 ceph-mon[16639]: 5.b scrub ok 2024-09-02T10:17:42.876 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:42.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:42.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:42.956 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:17:42.956 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:17:42.956 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:17:42.956 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:17:42.956 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:17:42.956 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:17:42.956 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:17:42.956 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:17:42.956 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:17:42.957 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:17:42.957 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:17:42.957 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:17:42.977 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:43.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:42 smithi187 ceph-mon[20072]: pgmap v1193: 47 pgs: 47 active+clean; 2.0 GiB data, 9.9 GiB used, 884 GiB / 894 GiB avail; 0 B/s wr, 731 op/s 2024-09-02T10:17:43.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:42 smithi187 ceph-mon[20072]: 5.b scrub ok 2024-09-02T10:17:43.007 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:43.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:43.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:43.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:43.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:43.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:43.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:43.272 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:43.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:43.305 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:17:43.305 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:17:43.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:43.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:43.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:43.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:43.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:17:43.529 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:43.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:43.565 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:17:43.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:43.658 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:43.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:43.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:43.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:43.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:43.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:43 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:17:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:43 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:17:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/667689854' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:43 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:17:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:43 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:17:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:43 smithi162 ceph-mon[20279]: 5.1 scrub starts 2024-09-02T10:17:43.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:43.920 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:43.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:43 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:17:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:43 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:17:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/667689854' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:43 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:17:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:43 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:17:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:43 smithi151 ceph-mon[16639]: 5.1 scrub starts 2024-09-02T10:17:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:43 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:17:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:43 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:17:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/667689854' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:44.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:43 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:17:44.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:43 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:17:44.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:43 smithi187 ceph-mon[20072]: 5.1 scrub starts 2024-09-02T10:17:44.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:44.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:44.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:44.051 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:44.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:44.180 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:44.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:44.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:17:44.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:44.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:44.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:44.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:44.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:44.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:44.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:44.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:44.579 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:44.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:44.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:44.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:44.705 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:44.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:44.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:44.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:44 smithi151 ceph-mon[16639]: pgmap v1194: 47 pgs: 47 active+clean; 1.8 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 872 op/s 2024-09-02T10:17:44.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:44 smithi151 ceph-mon[16639]: 5.1 scrub ok 2024-09-02T10:17:44.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:44 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:17:44.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:44 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:17:44.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/667689854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:44.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:44 smithi151 ceph-mon[16639]: osdmap e513: 12 total, 12 up, 10 in 2024-09-02T10:17:44.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:44 smithi151 ceph-mon[16639]: 5.e scrub starts 2024-09-02T10:17:44.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:44 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:17:44.712 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:44 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:17:44.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:44.834 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:44.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:44.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:44 smithi162 ceph-mon[20279]: pgmap v1194: 47 pgs: 47 active+clean; 1.8 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 872 op/s 2024-09-02T10:17:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:44 smithi162 ceph-mon[20279]: 5.1 scrub ok 2024-09-02T10:17:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:44 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:17:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:44 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:17:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/667689854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:44 smithi162 ceph-mon[20279]: osdmap e513: 12 total, 12 up, 10 in 2024-09-02T10:17:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:44 smithi162 ceph-mon[20279]: 5.e scrub starts 2024-09-02T10:17:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:44 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:17:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:44 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:17:44.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:44.964 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:44.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:44.965 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:44 smithi187 ceph-mon[20072]: pgmap v1194: 47 pgs: 47 active+clean; 1.8 GiB data, 9.3 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 872 op/s 2024-09-02T10:17:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:44 smithi187 ceph-mon[20072]: 5.1 scrub ok 2024-09-02T10:17:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:44 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:17:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:44 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:17:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/667689854' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:44 smithi187 ceph-mon[20072]: osdmap e513: 12 total, 12 up, 10 in 2024-09-02T10:17:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:44 smithi187 ceph-mon[20072]: 5.e scrub starts 2024-09-02T10:17:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:44 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:17:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:44 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:17:45.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:45.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:45.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:45.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:45.218 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:45.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:45.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:45.350 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:45.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:45.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:45.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:45.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:45.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:45.577 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:45.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:17:45.578 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:17:45.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-09-02T10:17:45.650 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:45.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:45.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:45.776 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:45.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:45.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:45.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:45 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:17:45.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:45 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:17:45.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:45 smithi151 ceph-mon[16639]: 5.e scrub ok 2024-09-02T10:17:45.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:17:45.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:17:45.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:17:45.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:17:45.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:17:45.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:17:45.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:45 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:17:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:45 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:17:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:45 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:17:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:45 smithi162 ceph-mon[20279]: 5.e scrub ok 2024-09-02T10:17:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:17:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:17:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:17:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:17:45.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:17:45.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:17:45.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:45 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:17:45.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:45.903 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:45.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:46.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:45 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:17:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:45 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:17:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:45 smithi187 ceph-mon[20072]: 5.e scrub ok 2024-09-02T10:17:46.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:17:46.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:17:46.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:17:46.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:17:46.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:17:46.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:17:46.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:45 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:17:46.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:46.036 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:46.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:46.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:46.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:46.214 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:46.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:46.290 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:46.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:46.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:46.418 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:46.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:46.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:46.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:46.542 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:46.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:46.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:46.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1885354907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:46.675 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:46.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:46.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:46.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:46.791 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:17:46.791 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:17:46.791 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:17:46.791 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:17:46.791 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:17:46.792 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:17:46.792 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:17:46.792 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:17:46.792 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:17:46.792 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:17:46.792 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:17:46.792 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:17:46.796 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:46.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:46.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1885354907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:46.909 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:46.941 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:46.942 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1885354907' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:47.028 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:17:47.028 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:17:47.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:47.069 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:17:47.070 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:17:47.074 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:47.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:47.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:17:47.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:17:47.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:17:47.135 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:17:47.135 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:17:47.166 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:17:47.166 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:17:47.176 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:47.202 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:17:47.202 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:17:47.204 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:47.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:47.229 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:17:47.229 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:17:47.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:17:47.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:17:47.285 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:17:47.285 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:17:47.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:47.320 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:17:47.321 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:17:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:47.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:47.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:47.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:47.461 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:47.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:47.542 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:17:47.543 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:17:47.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:47.568 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:17:47.569 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:17:47.585 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:47.585 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:17:47.650 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:17:47.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-09-02T10:17:47.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:47.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:47.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:47.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:47.846 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:47.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:47.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:47.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:47.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:47.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:48.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:48.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:48.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:48.107 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:48.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:48.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:48.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:48.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:48.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:48.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:48.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:48.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:48.460 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:48.487 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:48.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:48.488 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:48.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:48.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:48.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:48 smithi151 ceph-mon[16639]: 5.7 scrub starts 2024-09-02T10:17:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:48 smithi151 ceph-mon[16639]: pgmap v1197: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.6 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 881 op/s 2024-09-02T10:17:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:48 smithi151 ceph-mon[16639]: 5.7 scrub ok 2024-09-02T10:17:48.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:48.737 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:48.755 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:48.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:48 smithi162 ceph-mon[20279]: 5.7 scrub starts 2024-09-02T10:17:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:48 smithi162 ceph-mon[20279]: pgmap v1197: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.6 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 881 op/s 2024-09-02T10:17:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:48 smithi162 ceph-mon[20279]: 5.7 scrub ok 2024-09-02T10:17:48.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:48.882 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:48.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:48.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:48 smithi187 ceph-mon[20072]: 5.7 scrub starts 2024-09-02T10:17:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:48 smithi187 ceph-mon[20072]: pgmap v1197: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.6 GiB data, 8.8 GiB used, 885 GiB / 894 GiB avail; 0 B/s wr, 881 op/s 2024-09-02T10:17:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:48 smithi187 ceph-mon[20072]: 5.7 scrub ok 2024-09-02T10:17:49.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:49.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:49.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:49.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:49.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:49.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:49.262 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:17:49.263 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:17:49.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.263 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:183618 5.c 5:3a3324e4:::benchmark_data_smithi187_89405_object80013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e513)", 2024-09-02T10:17:49.263 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:17:49.253771+0000", 2024-09-02T10:17:49.263 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0071982030000000002, 2024-09-02T10:17:49.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072196020000000003, 2024-09-02T10:17:49.263 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:17:49.263 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:17:49.263 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 183618 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.253771+0000", 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:49.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:17:49.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.253771+0000", 2024-09-02T10:17:49.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:49.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:17:49.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.253773+0000", 2024-09-02T10:17:49.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6330000000000001e-06 2024-09-02T10:17:49.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:17:49.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.253779+0000", 2024-09-02T10:17:49.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.066e-06 2024-09-02T10:17:49.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:17:49.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.253797+0000", 2024-09-02T10:17:49.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7949999999999999e-05 2024-09-02T10:17:49.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:17:49.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.253804+0000", 2024-09-02T10:17:49.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1729999999999998e-06 2024-09-02T10:17:49.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:17:49.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.253878+0000", 2024-09-02T10:17:49.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3978000000000002e-05 2024-09-02T10:17:49.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:17:49.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.254019+0000", 2024-09-02T10:17:49.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014126300000000001 2024-09-02T10:17:49.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:17:49.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.254160+0000", 2024-09-02T10:17:49.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014121000000000001 2024-09-02T10:17:49.268 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.269 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:17:49.269 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.269 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25933.0:183619 5.7 e513/431 5:e97e827c:::benchmark_data_smithi187_89405_object80014:head v 513'11472, mlcod=513'11469)", 2024-09-02T10:17:49.269 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:17:49.257628+0000", 2024-09-02T10:17:49.269 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0033418079999999999, 2024-09-02T10:17:49.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034064260000000002, 2024-09-02T10:17:49.270 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:17:49.270 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:17:49.270 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:17:49.270 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:17:49.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:17:49.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.257628+0000", 2024-09-02T10:17:49.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:49.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:17:49.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.257628+0000", 2024-09-02T10:17:49.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:49.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:17:49.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.257629+0000", 2024-09-02T10:17:49.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.125e-06 2024-09-02T10:17:49.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:17:49.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.257636+0000", 2024-09-02T10:17:49.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7559999999999999e-06 2024-09-02T10:17:49.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:17:49.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.257654+0000", 2024-09-02T10:17:49.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7649999999999999e-05 2024-09-02T10:17:49.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:17:49.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.257662+0000", 2024-09-02T10:17:49.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6210000000000004e-06 2024-09-02T10:17:49.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:17:49.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.257719+0000", 2024-09-02T10:17:49.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.736e-05 2024-09-02T10:17:49.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:17:49.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.257737+0000", 2024-09-02T10:17:49.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7689e-05 2024-09-02T10:17:49.274 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.274 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:17:49.274 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:183621 5.c 5:3c039c9c:::benchmark_data_smithi187_89405_object80016:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e513)", 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:17:49.258152+0000", 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0028169879999999999, 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029111160000000001, 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:17:49.275 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:17:49.276 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:17:49.276 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 183621 2024-09-02T10:17:49.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.276 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:17:49.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:17:49.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258152+0000", 2024-09-02T10:17:49.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:49.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:17:49.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258152+0000", 2024-09-02T10:17:49.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:49.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:17:49.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258154+0000", 2024-09-02T10:17:49.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.595e-06 2024-09-02T10:17:49.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:17:49.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258160+0000", 2024-09-02T10:17:49.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9460000000000003e-06 2024-09-02T10:17:49.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:17:49.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258174+0000", 2024-09-02T10:17:49.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3654e-05 2024-09-02T10:17:49.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258182+0000", 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0350000000000001e-06 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.259237+0000", 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001055074 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.259355+0000", 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011875300000000001 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.259465+0000", 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109585 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:17:49.280 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.281 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25933.0:183618 5.c e513/455)", 2024-09-02T10:17:49.281 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:17:49.258635+0000", 2024-09-02T10:17:49.281 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.002334184, 2024-09-02T10:17:49.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024609110000000001, 2024-09-02T10:17:49.281 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:17:49.281 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:17:49.281 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "queued for pg", 2024-09-02T10:17:49.282 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:17:49.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:17:49.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258635+0000", 2024-09-02T10:17:49.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:49.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:17:49.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258635+0000", 2024-09-02T10:17:49.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:49.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258636+0000", 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0299999999999998e-07 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258640+0000", 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.313e-06 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258656+0000", 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5464e-05 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258662+0000", 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8590000000000001e-06 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25933.0:183622 5.0 e513/431 5:07380ac4:::benchmark_data_smithi187_89405_object80017:head v 513'11385, mlcod=513'11383)", 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:17:49.258819+0000", 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0021508289999999999, 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002303495, 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:17:49.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258819+0000", 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258819+0000", 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258819+0000", 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.82e-07 2024-09-02T10:17:49.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258825+0000", 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3480000000000003e-06 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258836+0000", 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1491999999999999e-05 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258841+0000", 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9980000000000003e-06 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258894+0000", 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2948999999999997e-05 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.258911+0000", 2024-09-02T10:17:49.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7309999999999999e-05 2024-09-02T10:17:49.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.290 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:17:49.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.290 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:17:49.290 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:17:49.290 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 5 2024-09-02T10:17:49.290 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:17:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:49.291 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:49.291 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:49.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:49.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:49.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:49.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:49.552 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:49.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:49.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:49.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:49.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:49.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:49.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:49.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:49.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:49.943 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:49.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:49.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:50.044 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:50.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:50.077 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:50.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:50.206 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:50.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:50.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:17:50.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:50.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:50.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:50.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:50.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:17:50.469 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:50.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:50.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:50.571 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:50.600 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:17:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:50.605 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:50.606 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:50.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:50.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:50.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:50.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:50 smithi151 ceph-mon[16639]: pgmap v1198: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.5 GiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 859 op/s 2024-09-02T10:17:50.828 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4250556600' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:17:50.828 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:50 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:17:50.828 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:50 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:17:50.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:50.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:50 smithi162 ceph-mon[20279]: pgmap v1198: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.5 GiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 859 op/s 2024-09-02T10:17:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4250556600' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:17:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:50 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:17:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:50 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:17:50.866 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:50.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:50.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:50.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:50.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:50 smithi187 ceph-mon[20072]: pgmap v1198: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.5 GiB data, 8.5 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 859 op/s 2024-09-02T10:17:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4250556600' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:17:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:50 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:17:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:50 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:17:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:17:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:17:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:17:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:17:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:17:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:17:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:17:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:17:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:17:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:17:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:17:51.016 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:17:51.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:51.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:51.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:51.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:51.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:51.247 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:51.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:51.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:51.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:51.375 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:51.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:17:51.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:51.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:17:51.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:51.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:51.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:51.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:51.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:51 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:51.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4250556600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:17:51.665 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:51 smithi151 ceph-mon[16639]: osdmap e514: 12 total, 12 up, 10 in 2024-09-02T10:17:51.712 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:17:51.712 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:17:51.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-09-02T10:17:51.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:51.757 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:51.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:51 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4250556600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:17:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:51 smithi162 ceph-mon[20279]: osdmap e514: 12 total, 12 up, 10 in 2024-09-02T10:17:51.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:51.887 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:51.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:51.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:51 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4250556600' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:17:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:51 smithi187 ceph-mon[20072]: osdmap e514: 12 total, 12 up, 10 in 2024-09-02T10:17:52.021 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:52.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:52.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:52.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:52.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:52.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:52.166 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:52.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:52.278 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:52.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:52.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:52.403 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:52.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:52.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:52.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:52.528 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:52.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:52.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:52.657 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:52.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:52.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:52.766 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:52 smithi151 ceph-mon[16639]: pgmap v1200: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.5 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 683 op/s 2024-09-02T10:17:52.785 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:52.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:52.837 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:17:52.837 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:17:52.837 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:17:52.837 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:17:52.837 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:17:52.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:52.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:52 smithi187 ceph-mon[20072]: pgmap v1200: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.5 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 683 op/s 2024-09-02T10:17:52.914 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:52.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:53.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:53.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:53.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:53.086 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:53.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:52 smithi162 ceph-mon[20279]: pgmap v1200: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 1.5 GiB data, 8.4 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 683 op/s 2024-09-02T10:17:53.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:53.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:53.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:53.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:53.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:53.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:53.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:53.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:53.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:53.442 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:53.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:53.583 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:53.584 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:53.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:53.686 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:53.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:17:53.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:53.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:53.816 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:53.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:53.849 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:53.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:53.974 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:53.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:54.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:54.099 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:54.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:54.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:17:54.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:54.223 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:54.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:54.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:54.352 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:54.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:54.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:54.478 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:54.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:54.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:54.609 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:54.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:54.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:54.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:54.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:54.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:54.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:54.858 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:54 smithi151 ceph-mon[16639]: pgmap v1201: 47 pgs: 47 active+clean; 1.3 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 900 op/s 2024-09-02T10:17:54.862 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:54.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:54.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:54.993 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:54.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:54 smithi187 ceph-mon[20072]: pgmap v1201: 47 pgs: 47 active+clean; 1.3 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 900 op/s 2024-09-02T10:17:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:54 smithi162 ceph-mon[20279]: pgmap v1201: 47 pgs: 47 active+clean; 1.3 GiB data, 7.8 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 900 op/s 2024-09-02T10:17:55.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:55.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:55.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:55.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:55.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:55.249 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:55.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:55.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:55.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:55.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:55.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:55.511 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:55.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:55.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:55.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:55.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:55.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:55.767 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:55.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:55.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:55.891 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:55.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:55.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:55.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:56.019 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:56.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:56.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:56.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:17:56.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:17:56.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:17:56.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:17:56.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:17:56.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:17:56.133 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:17:56.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:17:56.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:17:56.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:17:56.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:17:56.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:17:56.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:17:56.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:17:56.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:17:56.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:17:56.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:17:56.135 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:17:56.135 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:17:56.135 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:17:56.135 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:17:56.135 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:17:56.135 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:17:56.135 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:17:56.135 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:17:56.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:56.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:56.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:17:56.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:56.283 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:56.283 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:56.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:56.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:56.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:56.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:56.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:56.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:56.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:56.668 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:56.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:56.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:56.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:56.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:56.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:56 smithi162 ceph-mon[20279]: pgmap v1202: 47 pgs: 47 active+clean; 1.3 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 871 op/s 2024-09-02T10:17:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1863113597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:56.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:56.922 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:56.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:56 smithi151 ceph-mon[16639]: pgmap v1202: 47 pgs: 47 active+clean; 1.3 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 871 op/s 2024-09-02T10:17:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1863113597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:56 smithi187 ceph-mon[20072]: pgmap v1202: 47 pgs: 47 active+clean; 1.3 GiB data, 7.7 GiB used, 886 GiB / 894 GiB avail; 0 B/s wr, 871 op/s 2024-09-02T10:17:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1863113597' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:17:57.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:57.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:57.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:57.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:57.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:17:57.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:57.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:17:57.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:57.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:57.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:57.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:17:57.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:57.426 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:57.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:57.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:57.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:57.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:57.557 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:57.603 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:17:57.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:57.682 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:57.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:57.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:57.812 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:57.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:57.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:57.937 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:57.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:57.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:58.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:58.066 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:58.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:58.168 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:17:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:58.193 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:58.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:17:58.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:58.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:58.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:58.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:58.402 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:17:58.402 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:17:58.402 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:17:58.402 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:17:58.402 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:17:58.402 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:17:58.402 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:17:58.402 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:17:58.402 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:17:58.403 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:17:58.403 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:17:58.403 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:17:58.422 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:58.445 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:58.446 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:58.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:17:58.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:58.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:17:58.613 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:17:58.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:58 smithi151 ceph-mon[16639]: pgmap v1203: 47 pgs: 47 active+clean; 1.1 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 846 op/s 2024-09-02T10:17:58.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/889461751' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:58.675 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:17:58.701 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:58.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:58.702 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:17:58.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:17:58.830 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:58.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:17:58.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:58.955 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:58.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:58.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:58 smithi187 ceph-mon[20072]: pgmap v1203: 47 pgs: 47 active+clean; 1.1 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 846 op/s 2024-09-02T10:17:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/889461751' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:59.051 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:17:59.052 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:17:59.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:59.086 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:59.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:59.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:59.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:58 smithi162 ceph-mon[20279]: pgmap v1203: 47 pgs: 47 active+clean; 1.1 GiB data, 7.2 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 846 op/s 2024-09-02T10:17:59.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/889461751' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:17:59.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:59.222 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:59.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:17:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:59.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:59.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:17:59.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:59.349 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:17:59.349 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:17:59.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:59.472 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:59.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:59.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:59.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:17:59.596 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:59.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:59.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:17:59.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:17:59.721 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:59.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:17:59.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:17:59.849 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:17:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:59.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:17:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:59 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:17:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/889461751' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:17:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:17:59 smithi151 ceph-mon[16639]: osdmap e515: 12 total, 12 up, 10 in 2024-09-02T10:17:59.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:17:59.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:17:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:17:59.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:59 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/889461751' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:17:59 smithi187 ceph-mon[20072]: osdmap e515: 12 total, 12 up, 10 in 2024-09-02T10:18:00.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:00.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:59 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/889461751' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:17:59 smithi162 ceph-mon[20279]: osdmap e515: 12 total, 12 up, 10 in 2024-09-02T10:18:00.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:00.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:00.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:18:00.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:00.233 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:00.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:00.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:00.361 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:00.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:00.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:00.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:00.494 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:00.495 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:00.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:00.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:00.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:00.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:00.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:00.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:00.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:00 smithi151 ceph-mon[16639]: pgmap v1205: 47 pgs: 47 active+clean; 1.0 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 924 op/s 2024-09-02T10:18:00.844 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:00.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:00.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:00.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:00.880 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:00.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:00 smithi187 ceph-mon[20072]: pgmap v1205: 47 pgs: 47 active+clean; 1.0 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 924 op/s 2024-09-02T10:18:01.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:01.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:01.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:00 smithi162 ceph-mon[20279]: pgmap v1205: 47 pgs: 47 active+clean; 1.0 GiB data, 7.0 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 924 op/s 2024-09-02T10:18:01.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:01.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:01.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:01.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:01.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:01.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:01.365 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:01.392 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:01.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:01.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:18:01.423 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:18:01.423 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-09-02T10:18:01.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:01.521 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:01.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:01.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:01.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:01.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:01.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:01.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:01.777 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:01.778 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:01.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:01.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:01.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:01.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:01.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:01 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:18:01.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:01 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:18:01.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:01 smithi151 ceph-mon[16639]: osdmap e516: 12 total, 12 up, 10 in 2024-09-02T10:18:01.945 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:01.990 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:18:01.991 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:18:01.995 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:01 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:18:02.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:01 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:18:02.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:01 smithi187 ceph-mon[20072]: osdmap e516: 12 total, 12 up, 10 in 2024-09-02T10:18:02.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:02.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:18:02.021 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:18:02.030 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:02.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:02.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:02.048 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:18:02.049 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:18:02.070 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:18:02.071 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:18:02.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:18:02.094 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:18:02.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:01 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:18:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:01 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:18:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:01 smithi162 ceph-mon[20279]: osdmap e516: 12 total, 12 up, 10 in 2024-09-02T10:18:02.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:18:02.116 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:18:02.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:02.138 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:18:02.138 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:18:02.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:02.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:02.160 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:18:02.161 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:18:02.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:18:02.187 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:18:02.213 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:18:02.213 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:18:02.245 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:18:02.246 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:18:02.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:02.280 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:18:02.280 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:18:02.283 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:02.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:02.385 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:02.411 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:02.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:02.411 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:02.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:02.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:02.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:02.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:02.665 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:02.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:02.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:02.790 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:02.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:02.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:18:02.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:02 smithi151 ceph-mon[16639]: pgmap v1207: 47 pgs: 47 active+clean; 1.0 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 622 op/s 2024-09-02T10:18:02.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/969951311' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:02.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:02.916 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:02.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:02 smithi187 ceph-mon[20072]: pgmap v1207: 47 pgs: 47 active+clean; 1.0 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 622 op/s 2024-09-02T10:18:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/969951311' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:03.012 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:18:03.012 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:18:03.013 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:18:03.013 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:18:03.013 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:18:03.013 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:18:03.013 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:18:03.013 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:18:03.013 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:18:03.013 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:18:03.013 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:18:03.013 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:18:03.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:03.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:03.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:03.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:02 smithi162 ceph-mon[20279]: pgmap v1207: 47 pgs: 47 active+clean; 1.0 GiB data, 6.9 GiB used, 887 GiB / 894 GiB avail; 0 B/s wr, 622 op/s 2024-09-02T10:18:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/969951311' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:03.144 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:03.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:03.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:03.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:03.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:03.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:03.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:03.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:03.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:03.431 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:03.431 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:03.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:03.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:03.562 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:03.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:03.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:03.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:03.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:03.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:03 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:18:03.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:03 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:18:03.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:03.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:03.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:03.818 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:03.919 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:03.946 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:03.947 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:03 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:18:04.126 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:03 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:18:04.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:04.128 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:03 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:18:04.128 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:03 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:18:04.152 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:04.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:04.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:04.282 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:04.283 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:04.358 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:18:04.358 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:18:04.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:04.408 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:04.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:04.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:04.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:04.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:04.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:04.648 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:18:04.648 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:18:04.648 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:18:04.648 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:18:04.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:04.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:04 smithi187 ceph-mon[20072]: pgmap v1208: 47 pgs: 47 active+clean; 817 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 940 op/s 2024-09-02T10:18:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:04 smithi187 ceph-mon[20072]: osdmap e517: 12 total, 12 up, 10 in 2024-09-02T10:18:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:04 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:18:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:04 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:18:04.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:04.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:04.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:04.893 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:04.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:04.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:04.919 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:04.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:04 smithi151 ceph-mon[16639]: pgmap v1208: 47 pgs: 47 active+clean; 817 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 940 op/s 2024-09-02T10:18:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:04 smithi151 ceph-mon[16639]: osdmap e517: 12 total, 12 up, 10 in 2024-09-02T10:18:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:04 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:18:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:04 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:18:05.019 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:05.058 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:05.059 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:05.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:04 smithi162 ceph-mon[20279]: pgmap v1208: 47 pgs: 47 active+clean; 817 MiB data, 6.4 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 940 op/s 2024-09-02T10:18:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:04 smithi162 ceph-mon[20279]: osdmap e517: 12 total, 12 up, 10 in 2024-09-02T10:18:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:04 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:18:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:04 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:18:05.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:05.196 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:05.197 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:05.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:05.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:05.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:05.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:18:05.423 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:05.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:05.455 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:05.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:18:05.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:05.583 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:05.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:05.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:05.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:05.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:05.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:05.845 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:05.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:05.886 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:05 smithi151 ceph-mon[16639]: osdmap e518: 12 total, 12 up, 10 in 2024-09-02T10:18:05.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:05.955 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:05.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:05.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:05.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:05.991 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:05 smithi187 ceph-mon[20072]: osdmap e518: 12 total, 12 up, 10 in 2024-09-02T10:18:06.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:06.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:05 smithi162 ceph-mon[20279]: osdmap e518: 12 total, 12 up, 10 in 2024-09-02T10:18:06.109 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:06.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:06.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:06.233 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:06.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:06.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:06.359 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:06.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:06.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:06.484 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:06.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:06.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:06.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:06.612 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:06.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:06.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:06.687 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:18:06.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:06.718 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:18:06.718 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:18:06.719 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:18:06.719 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:18:06.719 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:18:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:06.740 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:06.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:06.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:06.867 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:06.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:06 smithi151 ceph-mon[16639]: pgmap v1211: 47 pgs: 47 active+clean; 786 MiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 722 op/s 2024-09-02T10:18:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3932228706' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:18:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:18:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:18:06.969 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:06.995 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:06.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:06.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:06 smithi187 ceph-mon[20072]: pgmap v1211: 47 pgs: 47 active+clean; 786 MiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 722 op/s 2024-09-02T10:18:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3932228706' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:18:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:18:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:18:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:06 smithi162 ceph-mon[20279]: pgmap v1211: 47 pgs: 47 active+clean; 786 MiB data, 6.2 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 722 op/s 2024-09-02T10:18:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3932228706' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:18:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:18:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:18:07.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:07.128 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:07.129 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:07.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:07.253 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:07.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:07.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:07.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:07.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:07.480 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:07.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:07.511 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:07.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:07.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:07.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:07.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:07.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:07.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:07.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:07.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:07.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:18:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:07.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:07.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:07 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3932228706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:18:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:07 smithi151 ceph-mon[16639]: osdmap e519: 12 total, 12 up, 10 in 2024-09-02T10:18:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1138878789' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:07.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:07 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3932228706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:18:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:07 smithi187 ceph-mon[20072]: osdmap e519: 12 total, 12 up, 10 in 2024-09-02T10:18:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1138878789' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:08.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:08.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:07 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3932228706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:18:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:07 smithi162 ceph-mon[20279]: osdmap e519: 12 total, 12 up, 10 in 2024-09-02T10:18:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1138878789' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:08.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:08.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:08.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:08.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:08.269 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:08.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:08.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:08.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:08.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:08.505 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:08.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:08.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:08.585 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:18:08.585 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:18:08.585 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:18:08.585 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:18:08.585 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:18:08.585 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:18:08.585 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:18:08.585 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:18:08.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:18:08.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:18:08.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:18:08.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:18:08.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:18:08.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:08.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:08.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:08.757 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:08.785 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:08.786 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:08.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:08.915 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:08.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:08.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:08 smithi151 ceph-mon[16639]: pgmap v1213: 47 pgs: 47 active+clean; 653 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 995 op/s 2024-09-02T10:18:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:08 smithi187 ceph-mon[20072]: pgmap v1213: 47 pgs: 47 active+clean; 653 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 995 op/s 2024-09-02T10:18:09.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:09.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:09.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:09.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:08 smithi162 ceph-mon[20279]: pgmap v1213: 47 pgs: 47 active+clean; 653 MiB data, 5.8 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 995 op/s 2024-09-02T10:18:09.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:09.166 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:09.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:09.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:09.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:09.293 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:09.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:09.359 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10] out_osds: [2, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:18:09.359 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:18:09.359 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-02T10:18:09.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd in 4 2024-09-02T10:18:09.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:09.423 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:09.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:09.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:09.479 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:09.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:09.550 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:09.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:09.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:09.677 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:09.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:09.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:09.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:09.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:09.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:09.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:09.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:09.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:09.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:09.940 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:10.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:10.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:10.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:10.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:10.086 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:10.086 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:10.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:10.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:10.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:10.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:10.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:10.318 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:10 smithi162 ceph-mon[20279]: pgmap v1214: 47 pgs: 47 active+clean; 562 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 681 op/s 2024-09-02T10:18:10.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:18:10.369 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:10 smithi151 ceph-mon[16639]: pgmap v1214: 47 pgs: 47 active+clean; 562 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 681 op/s 2024-09-02T10:18:10.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:10.449 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:10.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:10.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:10 smithi187 ceph-mon[20072]: pgmap v1214: 47 pgs: 47 active+clean; 562 MiB data, 5.6 GiB used, 888 GiB / 894 GiB avail; 0 B/s wr, 681 op/s 2024-09-02T10:18:10.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:10.580 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:10.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:10.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:10.709 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:10.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:10.710 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:10.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:18:10.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:10.839 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:10.840 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:10.848 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:10.862 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:10.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:10.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:10.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:11.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:11.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:11.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:11.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:11.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:11.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:11.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:11 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:18:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:11 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:18:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:11 smithi162 ceph-mon[20279]: osdmap e520: 12 total, 12 up, 10 in 2024-09-02T10:18:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:11 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:18:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:11 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:18:11.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:11 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:18:11.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:11 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:18:11.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:11 smithi151 ceph-mon[16639]: osdmap e520: 12 total, 12 up, 10 in 2024-09-02T10:18:11.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:11 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:18:11.354 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:11 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:18:11.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:11.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:11.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:18:11.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:11.500 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:11.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:11 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:18:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:11 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:18:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:11 smithi187 ceph-mon[20072]: osdmap e520: 12 total, 12 up, 10 in 2024-09-02T10:18:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:11 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:18:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:11 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:18:11.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:11.628 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:11.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:11.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:11.754 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:11.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:11.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:11.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:11.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:11.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:12.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:12.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:12.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:12.047 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:11 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:18:11.757+0000 7fe3ffed4640 -1 mon.a@0(leader).osd e520 definitely_dead 0 2024-09-02T10:18:12.051 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.4. 2024-09-02T10:18:12.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:12.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:12.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:12.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:12.275 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:12.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:12 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:18:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:12 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:18:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:12 smithi162 ceph-mon[20279]: pgmap v1216: 47 pgs: 47 active+clean; 528 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 617 op/s 2024-09-02T10:18:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:12 smithi162 ceph-mon[20279]: 5.3 scrub starts 2024-09-02T10:18:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:12 smithi162 ceph-mon[20279]: 5.3 scrub ok 2024-09-02T10:18:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3499591366' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-02T10:18:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:12 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-02T10:18:12.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:12.398 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:12.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:12.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:12.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:12 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:18:12.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:12 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:18:12.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:12 smithi151 ceph-mon[16639]: pgmap v1216: 47 pgs: 47 active+clean; 528 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 617 op/s 2024-09-02T10:18:12.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:12 smithi151 ceph-mon[16639]: 5.3 scrub starts 2024-09-02T10:18:12.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:12 smithi151 ceph-mon[16639]: 5.3 scrub ok 2024-09-02T10:18:12.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3499591366' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-02T10:18:12.411 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:12 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-02T10:18:12.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:12 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:18:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:12 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:18:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:12 smithi187 ceph-mon[20072]: pgmap v1216: 47 pgs: 47 active+clean; 528 MiB data, 5.5 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 617 op/s 2024-09-02T10:18:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:12 smithi187 ceph-mon[20072]: 5.3 scrub starts 2024-09-02T10:18:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:12 smithi187 ceph-mon[20072]: 5.3 scrub ok 2024-09-02T10:18:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3499591366' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-02T10:18:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:12 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["4"]} : dispatch 2024-09-02T10:18:12.522 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:12.523 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:12.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:12.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:12.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:12.669 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:12.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:12.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:12.770 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:12.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:12.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:12.892 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:12.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:12.893 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:12.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:13.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:13.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:13.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:13.053 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:18:13.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:13.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:13.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:13.168 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-02T10:18:13.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:13.261 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:13.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:13 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:18:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:13 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:18:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:13 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-02T10:18:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:13 smithi162 ceph-mon[20279]: osdmap e521: 12 total, 12 up, 11 in 2024-09-02T10:18:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2304168364' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:13.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:13.385 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:13.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:13.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:13 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:18:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:13 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:18:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:13 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-02T10:18:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:13 smithi151 ceph-mon[16639]: osdmap e521: 12 total, 12 up, 11 in 2024-09-02T10:18:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2304168364' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:13.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:13 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:18:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:13 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:18:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:13 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-02T10:18:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:13 smithi187 ceph-mon[20072]: osdmap e521: 12 total, 12 up, 11 in 2024-09-02T10:18:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2304168364' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:13.510 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:13.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:13.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:13.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:13.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:13.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:13.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:13.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:13.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:13.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:13.864 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:13.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:13.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:13.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:14.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:14.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:14.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:14.129 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:14.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:14.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:14 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:18:14.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:14 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:18:14.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2304168364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:14.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:14 smithi151 ceph-mon[16639]: osdmap e522: 12 total, 12 up, 11 in 2024-09-02T10:18:14.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:14 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:18:14.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:14 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:18:14.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:14 smithi151 ceph-mon[16639]: pgmap v1219: 47 pgs: 47 active+clean; 320 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 886 op/s 2024-09-02T10:18:14.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:14.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:14.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:14.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:14 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:18:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:14 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:18:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2304168364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:14 smithi162 ceph-mon[20279]: osdmap e522: 12 total, 12 up, 11 in 2024-09-02T10:18:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:14 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:18:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:14 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:18:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:14 smithi162 ceph-mon[20279]: pgmap v1219: 47 pgs: 47 active+clean; 320 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 886 op/s 2024-09-02T10:18:14.358 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:14.385 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:14.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:14.450 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:14.450 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:14.486 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:14 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:18:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:14 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:18:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2304168364' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:14 smithi187 ceph-mon[20072]: osdmap e522: 12 total, 12 up, 11 in 2024-09-02T10:18:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:14 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:18:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:14 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:18:14.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:14 smithi187 ceph-mon[20072]: pgmap v1219: 47 pgs: 47 active+clean; 320 MiB data, 4.9 GiB used, 889 GiB / 894 GiB avail; 0 B/s wr, 886 op/s 2024-09-02T10:18:14.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:14.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:14.518 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:14.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:14.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:14.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:14.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:14.771 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:14.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:14.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:14.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:14.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:14.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:14.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:15.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:15.026 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:15.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:15.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:15.157 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:15.158 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:15.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:15.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:15.286 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:15 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:18:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:15 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:18:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:15 smithi162 ceph-mon[20279]: osdmap e523: 12 total, 12 up, 11 in 2024-09-02T10:18:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:15 smithi162 ceph-mon[20279]: 5.c starting backfill to osd.4 from (0'0,0'0] MAX to 520'12479 2024-09-02T10:18:15.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:15.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:15.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:15.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:15 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:18:15.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:15 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:18:15.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:15 smithi151 ceph-mon[16639]: osdmap e523: 12 total, 12 up, 11 in 2024-09-02T10:18:15.421 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:15 smithi151 ceph-mon[16639]: 5.c starting backfill to osd.4 from (0'0,0'0] MAX to 520'12479 2024-09-02T10:18:15.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:18:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:15 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:18:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:15 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:18:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:15 smithi187 ceph-mon[20072]: osdmap e523: 12 total, 12 up, 11 in 2024-09-02T10:18:15.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:15 smithi187 ceph-mon[20072]: 5.c starting backfill to osd.4 from (0'0,0'0] MAX to 520'12479 2024-09-02T10:18:15.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:15.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:15.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:15.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:15.677 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:15.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:15.722 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:18:15.723 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:18:15.751 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:18:15.751 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:18:15.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:15.780 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:18:15.781 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:18:15.812 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:18:15.812 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:18:15.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:15.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:15.816 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:15.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:18:15.847 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:18:15.847 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:18:15.881 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:18:15.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:18:15.910 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:18:15.910 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:18:15.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:15.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:18:15.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:18:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:15.953 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:15.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:15.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:18:15.973 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:18:16.008 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:18:16.008 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:18:16.039 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:18:16.040 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:18:16.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:16.075 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:18:16.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:18:16.079 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:16.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:16.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:16.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:16.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:16.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:16.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:16.344 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:16 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:18:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:16 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:18:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:16 smithi162 ceph-mon[20279]: pgmap v1221: 47 pgs: 1 remapped+peering, 46 active+clean; 298 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 747 op/s 2024-09-02T10:18:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3091014293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:16.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:16.346 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:16 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:18:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:16 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:18:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:16 smithi151 ceph-mon[16639]: pgmap v1221: 47 pgs: 1 remapped+peering, 46 active+clean; 298 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 747 op/s 2024-09-02T10:18:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3091014293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:16.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:16.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:16.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:16 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:18:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:16 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:18:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:16 smithi187 ceph-mon[20072]: pgmap v1221: 47 pgs: 1 remapped+peering, 46 active+clean; 298 MiB data, 5.0 GiB used, 978 GiB / 983 GiB avail; 0 B/s wr, 747 op/s 2024-09-02T10:18:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3091014293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:16.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:16.595 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:18:16.595 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:18:16.595 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:18:16.595 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:18:16.595 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:18:16.612 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:16.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:16.714 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:16.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:16.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:16.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:16.877 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:16.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:16.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:17.005 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:17.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:17.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:17.133 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:17.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:17.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:17.255 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:17.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:17 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:18:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:17 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:18:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:17 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:18:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:17 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:18:17.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:17.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:17.379 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:17 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:18:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:17 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:18:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:17 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:18:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:17 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:18:17.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:17.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:17.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:17.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:17 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:18:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:17 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:18:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:17 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:18:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:17 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:18:17.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:17.626 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:17.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:17.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:17.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:17.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:17.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:17.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:17.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:17.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:17.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:17.996 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:17.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:17.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:18.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:18.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:18.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:18.169 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:18:18.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-09-02T10:18:18.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:18.246 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:18.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:18 smithi162 ceph-mon[20279]: osdmap e524: 12 total, 12 up, 11 in 2024-09-02T10:18:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:18 smithi162 ceph-mon[20279]: pgmap v1223: 47 pgs: 1 remapped+peering, 46 active+clean; 202 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 889 op/s 2024-09-02T10:18:18.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:18.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:18.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:18 smithi151 ceph-mon[16639]: osdmap e524: 12 total, 12 up, 11 in 2024-09-02T10:18:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:18 smithi151 ceph-mon[16639]: pgmap v1223: 47 pgs: 1 remapped+peering, 46 active+clean; 202 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 889 op/s 2024-09-02T10:18:18.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:18 smithi187 ceph-mon[20072]: osdmap e524: 12 total, 12 up, 11 in 2024-09-02T10:18:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:18 smithi187 ceph-mon[20072]: pgmap v1223: 47 pgs: 1 remapped+peering, 46 active+clean; 202 MiB data, 4.7 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 889 op/s 2024-09-02T10:18:18.512 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:18.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:18.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:18.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:18.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:18.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:18.777 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:18.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:18.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:18.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:18.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:18.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:18.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:19.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:19.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:19.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:19.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:19.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:18:19.125 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:19.138 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:19.163 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:19.164 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:19.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:19.292 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:19.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:19 smithi162 ceph-mon[20279]: 5.2 scrub starts 2024-09-02T10:18:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:19 smithi162 ceph-mon[20279]: 5.2 scrub ok 2024-09-02T10:18:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:19 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:18:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:19 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:18:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:19 smithi162 ceph-mon[20279]: osdmap e525: 12 total, 12 up, 11 in 2024-09-02T10:18:19.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:19.430 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:19.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:19 smithi151 ceph-mon[16639]: 5.2 scrub starts 2024-09-02T10:18:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:19 smithi151 ceph-mon[16639]: 5.2 scrub ok 2024-09-02T10:18:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:19 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:18:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:19 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:18:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:19 smithi151 ceph-mon[16639]: osdmap e525: 12 total, 12 up, 11 in 2024-09-02T10:18:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:19 smithi187 ceph-mon[20072]: 5.2 scrub starts 2024-09-02T10:18:19.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:19 smithi187 ceph-mon[20072]: 5.2 scrub ok 2024-09-02T10:18:19.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:19 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:18:19.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:19 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:18:19.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:19 smithi187 ceph-mon[20072]: osdmap e525: 12 total, 12 up, 11 in 2024-09-02T10:18:19.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:19.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:19.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:19.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:19.690 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:19.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:19.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:19.820 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:19.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:19.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:19.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:19.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:19.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:19.948 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:20.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:20.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:20.078 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:20.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:20.201 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:20.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:20.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:20.219 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:20.219 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:20.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:20.324 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:20.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:20.340 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:20 smithi151 ceph-mon[16639]: pgmap v1225: 47 pgs: 47 active+clean; 133 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-09-02T10:18:20.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:20 smithi162 ceph-mon[20279]: pgmap v1225: 47 pgs: 47 active+clean; 133 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-09-02T10:18:20.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:20.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:20.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:20.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:20 smithi187 ceph-mon[20072]: pgmap v1225: 47 pgs: 47 active+clean; 133 MiB data, 4.5 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 499 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-09-02T10:18:20.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:20.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:20.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:20.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:20.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:20.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:20.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:20.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:20.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:20.858 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Removed 103602 objects 2024-09-02T10:18:20.858 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Clean up completed and total clean up time :123.678 2024-09-02T10:18:20.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:20.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:20.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:21.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:21.067 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:21.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:21.093 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:18:21.093 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-09-02T10:18:21.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:21.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:21 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:18:21.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:21 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:18:21.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3624684499' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:18:21.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3208055866' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:18:21.192 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:21.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:21.193 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:21.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:21.303 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:18:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:18:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:18:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:18:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155049 3.7 3:e346e101:::benchmark_data_smithi187_59989_object45506:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:18:21.303 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292699+0000", 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 595.00098877100004, 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33504108799999999, 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155049 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292699+0000", 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292699+0000", 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292700+0000", 2024-09-02T10:18:21.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7700000000000001e-07 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292702+0000", 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9180000000000001e-06 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292719+0000", 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6994999999999999e-05 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292723+0000", 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6380000000000003e-06 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294733+0000", 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020092690000000002 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294740+0000", 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4490000000000002e-06 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.602174+0000", 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30743415000000002 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.602323+0000", 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014901700000000001 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:18:21.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.602441+0000", 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117873 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.621007+0000", 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018566426 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.621054+0000", 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6263000000000001e-05 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.624195+0000", 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031411410000000001 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.624316+0000", 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000120695 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627741+0000", 2024-09-02T10:18:21.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034249770000000001 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155068 3.7 3:e666e28e:::benchmark_data_smithi187_59989_object45525:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 595.00095826300003, 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33844332199999999, 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.311 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155068 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292730+0000", 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8599999999999999e-07 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292733+0000", 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4530000000000001e-06 2024-09-02T10:18:21.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292739+0000", 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3090000000000001e-06 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292745+0000", 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8420000000000001e-06 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294766+0000", 2024-09-02T10:18:21.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020206989999999999 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294769+0000", 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8099999999999999e-06 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.615962+0000", 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32119220799999998 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.616132+0000", 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000170359 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.616296+0000", 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016439699999999999 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627817+0000", 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011520871 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627838+0000", 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0364999999999999e-05 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627892+0000", 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4667999999999999e-05 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.627957+0000", 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4299000000000002e-05 2024-09-02T10:18:21.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631173+0000", 2024-09-02T10:18:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003216656 2024-09-02T10:18:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.318 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:155089 3.7 3:e3752e01:::benchmark_data_smithi187_59989_object45546:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e283)", 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 595.00093759499998, 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35327147199999998, 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 155089 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292751+0000", 2024-09-02T10:18:21.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0199999999999998e-07 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292753+0000", 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.182e-06 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292759+0000", 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6450000000000002e-06 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.292763+0000", 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4530000000000004e-06 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294792+0000", 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020289150000000001 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.294796+0000", 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5149999999999998e-06 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631221+0000", 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33642498300000001 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631396+0000", 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175866 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,7", 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.631567+0000", 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017052899999999999 2024-09-02T10:18:21.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642330+0000", 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010763403 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642513+0000", 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018293399999999999 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642560+0000", 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6344000000000002e-05 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.642615+0000", 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5634000000000003e-05 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:08:26.646022+0000", 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034067670000000002 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.325 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11536 4.c 4.7fa12efc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.463377+0000", 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 430.83031139600001, 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.000768433, 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11536 2024-09-02T10:18:21.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463377+0000", 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463377+0000", 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463378+0000", 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.655e-06 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463418+0000", 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9276000000000001e-05 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463428+0000", 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0074999999999999e-05 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463434+0000", 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2940000000000002e-06 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465896+0000", 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024620409999999999 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465906+0000", 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8660000000000001e-06 2024-09-02T10:18:21.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464105+0000", 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99819907500000005 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464145+0000", 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0151000000000002e-05 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11545 4.c 4.151c8eac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.464747+0000", 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 430.82894098000003, 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99968865100000004, 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11545 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.331 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464747+0000", 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464747+0000", 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464748+0000", 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.24e-06 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464802+0000", 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3669999999999999e-05 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464810+0000", 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8420000000000005e-06 2024-09-02T10:18:21.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464817+0000", 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4209999999999998e-06 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465935+0000", 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011172109999999999 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:18:21.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465942+0000", 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1509999999999998e-06 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464426+0000", 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99848471999999999 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464436+0000", 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3959999999999995e-06 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11547 4.c 4.fa8faa7c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.465762+0000", 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 430.82792661100001, 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99873187699999999, 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:21.336 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11547 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465762+0000", 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465762+0000", 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465763+0000", 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.331e-06 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465793+0000", 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0453000000000001e-05 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465805+0000", 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1175999999999999e-05 2024-09-02T10:18:21.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465812+0000", 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1960000000000002e-06 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466125+0000", 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031365100000000002 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:18:21.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466134+0000", 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8340000000000005e-06 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464486+0000", 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99835216500000001 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464493+0000", 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.340 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11568 4.c 4.acfb412c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.727519+0000", 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 430.566168817, 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.73701982099999996, 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11568 2024-09-02T10:18:21.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727519+0000", 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727519+0000", 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.342 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727521+0000", 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7859999999999999e-06 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727593+0000", 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1581000000000005e-05 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727607+0000", 2024-09-02T10:18:21.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3916e-05 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727614+0000", 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8889999999999996e-06 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727669+0000", 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5364000000000001e-05 2024-09-02T10:18:21.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727676+0000", 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5969999999999997e-06 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464533+0000", 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.73685725400000002 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464539+0000", 2024-09-02T10:18:21.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-06 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:41914 4.11 e353/344 4:884e7a1d:::benchmark_data_smithi187_76542_object41913:head v 353'2322, mlcod=335'1550)", 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.849702+0000", 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 393.44398652799998, 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44488412300000002, 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849702+0000", 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849702+0000", 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849730+0000", 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8532999999999999e-05 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.347 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849763+0000", 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2478999999999999e-05 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293343+0000", 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44358014400000001 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293349+0000", 2024-09-02T10:18:21.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2489999999999998e-06 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293401+0000", 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1523999999999997e-05 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293425+0000", 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4606000000000001e-05 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294550+0000", 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011250959999999999 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294586+0000", 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5491999999999998e-05 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.11 353/344 [PushOp(4:8de9d133:::benchmark_data_smithi187_76542_object24111:head, version: 335'1553, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8de9d133:::benchmark_data_smithi187_76542_object24111:head@335'1553, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:18:21.350 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.853882+0000", 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 393.43980581599999, 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.486507095, 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853882+0000", 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853882+0000", 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853884+0000", 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5629999999999999e-06 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853923+0000", 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8707000000000001e-05 2024-09-02T10:18:21.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339685+0000", 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.485762156 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339697+0000", 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2575000000000001e-05 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339773+0000", 2024-09-02T10:18:21.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5802000000000004e-05 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339782+0000", 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6600000000000001e-06 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.340389+0000", 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060763199999999999 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.354 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.11 354/344 [PushOp(4:8d3f2f46:::benchmark_data_smithi187_76542_object24793:head, version: 335'1596, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8d3f2f46:::benchmark_data_smithi187_76542_object24793:head@335'1596, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:48.472557+0000", 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 392.82113137200002, 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31961658199999998, 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472557+0000", 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472557+0000", 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472558+0000", 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.029e-06 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472626+0000", 2024-09-02T10:18:21.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8273000000000002e-05 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791323+0000", 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31869734700000002 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791336+0000", 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2418e-05 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791429+0000", 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3016999999999994e-05 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791444+0000", 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5532e-05 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.792173+0000", 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00072896600000000005 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:21.360 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.11 354/344 [PushOp(4:8b8fcb96:::benchmark_data_smithi187_76542_object24835:head, version: 335'1598, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8b8fcb96:::benchmark_data_smithi187_76542_object24835:head@335'1598, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:48.476340+0000", 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 392.81734853099999, 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31867774999999998, 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476340+0000", 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476340+0000", 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476346+0000", 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2840000000000003e-06 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476539+0000", 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019279599999999999 2024-09-02T10:18:21.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794378+0000", 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31783950900000002 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794396+0000", 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7655999999999998e-05 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794481+0000", 2024-09-02T10:18:21.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4628000000000004e-05 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794491+0000", 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0114e-05 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.795017+0000", 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052676299999999997 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:75251 4.11 4:8ba31329:::benchmark_data_smithi187_76542_object75250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 363.62873903799999, 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37234742100000001, 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75251 2024-09-02T10:18:21.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664951+0000", 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9599999999999999e-06 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664999+0000", 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7738999999999998e-05 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665013+0000", 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4209999999999999e-05 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665021+0000", 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.481e-06 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665075+0000", 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4642999999999998e-05 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665173+0000", 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8022000000000004e-05 2024-09-02T10:18:21.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665258+0000", 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4666000000000006e-05 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.666435+0000", 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011775080000000001 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.369 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.666722+0000", 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028679699999999999 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037199+0000", 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37047664200000002 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037256+0000", 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7605999999999997e-05 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037297+0000", 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0147e-05 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:50217 5.1 5:84aaa397:::benchmark_data_smithi187_89405_object50216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 177.50193107199999, 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.384074792, 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.371 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50217 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791759+0000", 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.454e-06 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791785+0000", 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6838e-05 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791800+0000", 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4997999999999999e-05 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791808+0000", 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2370000000000003e-06 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791862+0000", 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4765000000000002e-05 2024-09-02T10:18:21.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791966+0000", 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010337500000000001 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.792051+0000", 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4779999999999998e-05 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.793473+0000", 2024-09-02T10:18:21.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014225819999999999 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.793928+0000", 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045518199999999999 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175728+0000", 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38180008999999998 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175787+0000", 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8498999999999997e-05 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175832+0000", 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4991999999999999e-05 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.16,RELEASE,e:497,from:1,reservation_nonce:0)", 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:18:21.377 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 82.656995929000004, 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58412980699999995, 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636693+0000", 2024-09-02T10:18:21.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6e-07 2024-09-02T10:18:21.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636695+0000", 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.841e-06 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220721+0000", 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58402521600000001 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220727+0000", 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4219999999999997e-06 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220798+0000", 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1131999999999996e-05 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220810+0000", 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1863e-05 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220822+0000", 2024-09-02T10:18:21.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2072999999999999e-05 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:182739 5.1 5:8cb35d44:::benchmark_data_smithi187_89405_object79134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e513)", 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:17:48.225340+0000", 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 33.068347793000001, 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16236800900000001, 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.382 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 182739 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.225340+0000", 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.225340+0000", 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.225343+0000", 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.435e-06 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.225351+0000", 2024-09-02T10:18:21.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1969999999999996e-06 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.225370+0000", 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8658e-05 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.225379+0000", 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1819999999999995e-06 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.225443+0000", 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4121999999999994e-05 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.225534+0000", 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1286999999999995e-05 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.225611+0000", 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7061e-05 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.229584+0000", 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003972293 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.229708+0000", 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012393199999999999 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.387605+0000", 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15789763500000001 2024-09-02T10:18:21.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.387661+0000", 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5804000000000003e-05 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.387708+0000", 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7403e-05 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.388 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:182758 5.1 5:897a7179:::benchmark_data_smithi187_89405_object79153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e513)", 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:17:48.247969+0000", 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 33.045718809999997, 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15983488500000001, 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 182758 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.389 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.247969+0000", 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.247969+0000", 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.247972+0000", 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2089999999999999e-06 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.247980+0000", 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2360000000000004e-06 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.248001+0000", 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1095999999999999e-05 2024-09-02T10:18:21.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.248010+0000", 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3070000000000008e-06 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.248082+0000", 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1868999999999999e-05 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.392 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.248222+0000", 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013960399999999999 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.248350+0000", 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128625 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.252202+0000", 2024-09-02T10:18:21.393 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038512989999999999 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.252401+0000", 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000199046 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.407736+0000", 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.15533492900000001 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.394 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.407772+0000", 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6554999999999998e-05 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:48.407804+0000", 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2110000000000003e-05 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:183776 5.1 5:8e7a2d8c:::benchmark_data_smithi187_89405_object80171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e513)", 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:17:49.425929+0000", 2024-09-02T10:18:21.395 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 31.867759200999998, 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.13569047400000001, 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 183776 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.425929+0000", 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.396 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.425929+0000", 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.425931+0000", 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3800000000000001e-06 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.425940+0000", 2024-09-02T10:18:21.397 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1419999999999993e-06 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.425961+0000", 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1367999999999999e-05 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.425971+0000", 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.058e-05 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.426045+0000", 2024-09-02T10:18:21.398 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3570000000000002e-05 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.426129+0000", 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4129999999999996e-05 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.426221+0000", 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2219000000000006e-05 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.430153+0000", 2024-09-02T10:18:21.399 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039320550000000003 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.558485+0000", 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.12833113800000001 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.561448+0000", 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029632289999999999 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.400 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.561565+0000", 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011742499999999999 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:17:49.561619+0000", 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4237999999999998e-05 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:205302 5.1 5:85fde85c:::benchmark_data_smithi187_89405_object101697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:18:21.401 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.0521677010000001, 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31845405399999999, 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 205302 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.402 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241523+0000", 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3429999999999998e-06 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.403 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241533+0000", 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0139999999999999e-05 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241562+0000", 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8600000000000001e-05 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.404 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241581+0000", 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9656e-05 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241772+0000", 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019121200000000001 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241906+0000", 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013322299999999999 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.405 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241975+0000", 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9129999999999997e-05 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.245690+0000", 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037148390000000002 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.246234+0000", 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00054468699999999997 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.406 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559894+0000", 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31365966099999998 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559928+0000", 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3813000000000003e-05 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559975+0000", 2024-09-02T10:18:21.407 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6749999999999998e-05 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:205324 5.1 5:85edc945:::benchmark_data_smithi187_89405_object101719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.6914299330000002, 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38627865099999997, 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.408 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 205324 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.409 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602260+0000", 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4300000000000001e-06 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602266+0000", 2024-09-02T10:18:21.410 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6599999999999998e-06 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602283+0000", 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6646e-05 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602307+0000", 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3853999999999999e-05 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.411 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602361+0000", 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4200000000000003e-05 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602447+0000", 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6116e-05 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602524+0000", 2024-09-02T10:18:21.412 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6596999999999999e-05 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.606604+0000", 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040804589999999998 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.606691+0000", 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6608999999999997e-05 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.413 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988466+0000", 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38177503699999998 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988502+0000", 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6217000000000002e-05 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988537+0000", 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4826000000000002e-05 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.414 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:205348 5.1 5:8f32706a:::benchmark_data_smithi187_89405_object101743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.983873+0000", 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.3098150579999999, 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.187727485, 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:18:21.415 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 205348 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983873+0000", 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983873+0000", 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.416 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983876+0000", 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3700000000000002e-06 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983884+0000", 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2400000000000007e-06 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983901+0000", 2024-09-02T10:18:21.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7669999999999999e-05 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983912+0000", 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0131e-05 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983993+0000", 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1901000000000001e-05 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.418 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.984091+0000", 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7608000000000005e-05 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.984172+0000", 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0489999999999997e-05 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988092+0000", 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039208810000000002 2024-09-02T10:18:21.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988434+0000", 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034132100000000001 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:18.170743+0000", 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18230938299999999 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:18.170789+0000", 2024-09-02T10:18:21.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5469999999999997e-05 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:18.171601+0000", 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00081201999999999999 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:18:21.421 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:21.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:21.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:21 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:18:21.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:21 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:18:21.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3624684499' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:18:21.422 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3208055866' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:18:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:21 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:18:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:21 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:18:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3624684499' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:18:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3208055866' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:18:21.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:21.547 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:21.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:21.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:21.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:21.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:18:21.673 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:21.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:21.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:21.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:21.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:21.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:21.923 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:21.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:22.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:22.051 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:22.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:22.153 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:22.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:22.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:22.278 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:22.304 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:22.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:22.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3624684499' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3208055866' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: osdmap e526: 12 total, 12 up, 11 in 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: pgmap v1227: 47 pgs: 47 active+clean; 109 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-09-02T10:18:22.344 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:22 smithi151 ceph-mon[16639]: 5.5 scrub starts 2024-09-02T10:18:22.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:18:22.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:18:22.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:22.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3624684499' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:18:22.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3208055866' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:18:22.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: osdmap e526: 12 total, 12 up, 11 in 2024-09-02T10:18:22.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:18:22.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:18:22.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:18:22.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:18:22.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: pgmap v1227: 47 pgs: 47 active+clean; 109 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-09-02T10:18:22.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:22 smithi162 ceph-mon[20279]: 5.5 scrub starts 2024-09-02T10:18:22.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:18:22.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:22.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:18:22.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:18:22.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:22.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3624684499' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:18:22.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3208055866' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:18:22.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: osdmap e526: 12 total, 12 up, 11 in 2024-09-02T10:18:22.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:18:22.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:18:22.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:18:22.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:18:22.409 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: pgmap v1227: 47 pgs: 47 active+clean; 109 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 503 op/s; 2.3 MiB/s, 36 objects/s recovering 2024-09-02T10:18:22.409 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:22 smithi187 ceph-mon[20072]: 5.5 scrub starts 2024-09-02T10:18:22.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:22.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:22.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:22.534 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:22.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:22.556 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:22.628 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:22.657 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:22.680 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:22.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:22.781 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:22.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:22.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:22.903 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:22.925 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:22.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:23.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:23.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:23.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:23.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:23.132 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:18:23.132 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:18:23.132 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:18:23.132 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:18:23.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:23.178 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:23.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:23.251 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:23 smithi151 ceph-mon[16639]: 5.0 scrub starts 2024-09-02T10:18:23.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:23.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:23.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:23.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:23 smithi162 ceph-mon[20279]: 5.0 scrub starts 2024-09-02T10:18:23.409 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:23.430 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:23.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:23.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:23 smithi187 ceph-mon[20072]: 5.0 scrub starts 2024-09-02T10:18:23.532 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:23.552 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:23.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:23.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:23.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:23.677 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:23.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:23.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-09-02T10:18:23.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:23.796 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:23.801 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:23.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:23.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:18:23.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:23.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:23.924 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:23.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:24.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:24.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:24.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:24.105 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:24.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:24.174 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:24.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:24.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:24.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:24 smithi151 ceph-mon[16639]: pgmap v1228: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.9 MiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 659 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:18:24.276 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:24.299 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:24.300 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:24.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:18:24.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:18:24.348 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:18:24.349 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:18:24.354 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:24.373 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:24 smithi162 ceph-mon[20279]: pgmap v1228: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.9 MiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 659 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:18:24.375 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:18:24.376 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:18:24.396 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:18:24.396 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:18:24.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:24.417 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:18:24.417 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:18:24.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:24.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:24.439 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:18:24.439 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:18:24.459 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:18:24.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:18:24.480 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:18:24.480 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:18:24.501 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:18:24.501 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:18:24.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:24 smithi187 ceph-mon[20072]: pgmap v1228: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 7.9 MiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 659 op/s; 2.2 MiB/s, 34 objects/s recovering 2024-09-02T10:18:24.522 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:18:24.523 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:18:24.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:24.544 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:18:24.544 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:18:24.547 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:24.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:24.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:24.565 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:18:24.565 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:18:24.599 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:24.599 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:24.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:24.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:24.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:24.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:24.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:24.800 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:24.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:24.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:24.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:24.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:24.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:25.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:25.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:25.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:25.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:25.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2186647512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:25.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:25.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:25.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:18:25.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:25.303 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2186647512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:25.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:18:25.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:25.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:25.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:25.451 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:25.477 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:25.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:25.479 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2186647512' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:25.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:25.585 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:25.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:25.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:25.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:25.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:25.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:25.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:25.790 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-02T10:18:25.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-02T10:18:25.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:25.853 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:25.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:25.853 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:25.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:25.977 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:25.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:26.074 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:26.074 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:26.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:26.104 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:26.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:26.108 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-02T10:18:26.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:26.227 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:26.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:26.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:26.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:26.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:26 smithi162 ceph-mon[20279]: pgmap v1229: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 403 op/s; 1.7 MiB/s, 27 objects/s recovering 2024-09-02T10:18:26.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/781780740' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:18:26.351 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:26.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:26.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:26 smithi151 ceph-mon[16639]: pgmap v1229: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 403 op/s; 1.7 MiB/s, 27 objects/s recovering 2024-09-02T10:18:26.356 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/781780740' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:18:26.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:26.479 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:26.480 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:26 smithi187 ceph-mon[20072]: pgmap v1229: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 403 op/s; 1.7 MiB/s, 27 objects/s recovering 2024-09-02T10:18:26.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/781780740' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:18:26.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:26.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:26.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:26.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:26.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:26.729 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:26.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:18:26.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:26.855 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:26.856 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:26.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:26.981 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:26.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:27.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-09-02T10:18:27.058 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:27.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:27.104 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:27.105 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:27.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:27.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:27.208 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:27.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:27.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:27.235 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:27.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/781780740' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:18:27.286 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:27 smithi151 ceph-mon[16639]: osdmap e527: 12 total, 12 up, 11 in 2024-09-02T10:18:27.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:27.358 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:27.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:27.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:27.424 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/781780740' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:18:27.424 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:27 smithi162 ceph-mon[20279]: osdmap e527: 12 total, 12 up, 11 in 2024-09-02T10:18:27.459 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:27.485 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:27.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/781780740' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:18:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:27 smithi187 ceph-mon[20072]: osdmap e527: 12 total, 12 up, 11 in 2024-09-02T10:18:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:27.523 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:27.586 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:27.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:27.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:27.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:27.737 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:27.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:27.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:27.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:27.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:27.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:27.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:27.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:27.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:28.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:28.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:28.115 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:28.148 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:28.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:18:28.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:28.238 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:28.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:28.340 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:28.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:28.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:28.368 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:28.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:18:28.413 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:28.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:28.493 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:28.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:28.494 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:28.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:28.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:28.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:28.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:28.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:28.725 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:18:28.725 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_3' removed 2024-09-02T10:18:28.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:28.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:28.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:28 smithi162 ceph-mon[20279]: pgmap v1231: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 264 op/s 2024-09-02T10:18:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:18:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:18:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3079618337' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:18:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:18:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:18:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:18:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2127446643' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:28.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:28.872 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:28.873 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:28.873 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:28.873 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:28 smithi151 ceph-mon[16639]: pgmap v1231: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 264 op/s 2024-09-02T10:18:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:18:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:18:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3079618337' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:18:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:18:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:18:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:18:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2127446643' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:28.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:28.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:28.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:28 smithi187 ceph-mon[20072]: pgmap v1231: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail; 0 B/s wr, 264 op/s 2024-09-02T10:18:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:18:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:18:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3079618337' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:18:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:18:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:18:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:18:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2127446643' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:29.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:29.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:29.123 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:29.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:29.249 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:29.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:29.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:29.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:29.378 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:29.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:29.502 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:29.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:29.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:18:29.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:29.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:29.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:29.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:29.733 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-02T10:18:29.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:29.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:29.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:29 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:18:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:29 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:18:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:29 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3079618337' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:18:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2127446643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:29 smithi162 ceph-mon[20279]: osdmap e528: 12 total, 12 up, 11 in 2024-09-02T10:18:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2917723282' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:18:29.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:29.855 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:29.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:29.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:29.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:29.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:29.996 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:29.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:29.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:29 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:18:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:29 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:18:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:29 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3079618337' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:18:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2127446643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:29 smithi187 ceph-mon[20072]: osdmap e528: 12 total, 12 up, 11 in 2024-09-02T10:18:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2917723282' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:18:30.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:30.098 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:29 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:18:30.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:29 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:18:30.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:29 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:30.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3079618337' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:18:30.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2127446643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:30.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:29 smithi151 ceph-mon[16639]: osdmap e528: 12 total, 12 up, 11 in 2024-09-02T10:18:30.099 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2917723282' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:18:30.119 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:30.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:30.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:30.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:30.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:30.248 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:30.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:30.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:30.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:30.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:30.422 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:30.422 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:30.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:30.497 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:30.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:30.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:30.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-09-02T10:18:30.546 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:30.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:30.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:30.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:30.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:30.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:30.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:30.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:30.869 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:30.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:30.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:30 smithi151 ceph-mon[16639]: pgmap v1233: 31 pgs: 31 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:30.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2917723282' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:18:30.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:30 smithi151 ceph-mon[16639]: osdmap e529: 12 total, 12 up, 11 in 2024-09-02T10:18:30.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:30 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:18:30.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:30 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:18:30.949 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-02T10:18:30.951 INFO:tasks.radosbench:Beginning radosbench... 2024-09-02T10:18:30.951 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-02T10:18:30.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-02T10:18:30.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:30 smithi187 ceph-mon[20072]: pgmap v1233: 31 pgs: 31 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2917723282' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:18:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:30 smithi187 ceph-mon[20072]: osdmap e529: 12 total, 12 up, 11 in 2024-09-02T10:18:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:30 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:18:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:30 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:18:31.007 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:31.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:31.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:30 smithi162 ceph-mon[20279]: pgmap v1233: 31 pgs: 31 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2917723282' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:18:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:30 smithi162 ceph-mon[20279]: osdmap e529: 12 total, 12 up, 11 in 2024-09-02T10:18:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:30 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:18:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:30 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:18:31.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:31.141 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:31.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:31.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:18:31.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:31.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:31.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:31.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:31.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:31.411 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:31.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:31.449 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:31.488 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:18:31.488 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:18:31.509 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:18:31.509 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:18:31.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:31.530 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:18:31.530 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:18:31.534 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:31.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:31.549 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:18:31.550 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:18:31.571 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:18:31.571 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:18:31.591 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:18:31.591 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:18:31.612 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:18:31.613 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:18:31.633 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:18:31.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:18:31.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:31.654 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:18:31.654 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:18:31.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:31.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:31.663 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:31.676 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:18:31.676 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:18:31.698 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:18:31.698 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:18:31.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:18:31.719 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:18:31.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:31.784 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:31.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:31.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:31 smithi151 ceph-mon[16639]: osdmap e530: 12 total, 12 up, 11 in 2024-09-02T10:18:31.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1743212694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:31.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:18:31.839 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:31 smithi162 ceph-mon[20279]: osdmap e530: 12 total, 12 up, 11 in 2024-09-02T10:18:31.839 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1743212694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:31.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:31.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:31.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:31.939 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:18:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:18:31.940 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:18:31.940 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:18:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:31 smithi187 ceph-mon[20072]: osdmap e530: 12 total, 12 up, 11 in 2024-09-02T10:18:32.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1743212694' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:32.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:32.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:32.039 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:32.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:32.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:32.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:32.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:32.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:32.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:32.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:32.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:32.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:32.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:32.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:32.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:32.541 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:32.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:32.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:32.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:18:32.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:32.705 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:32.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:32.756 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:32.768 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-02T10:18:32.779 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:32.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:32.830 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:32.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:32.882 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:32 smithi151 ceph-mon[16639]: pgmap v1236: 31 pgs: 31 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:32.882 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2870351557' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:18:32.889 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:32.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:32.953 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:32.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:32.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:33.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:32 smithi162 ceph-mon[20279]: pgmap v1236: 31 pgs: 31 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:33.010 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2870351557' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:18:33.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:33.077 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:33.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:33.119 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:32 smithi187 ceph-mon[20072]: pgmap v1236: 31 pgs: 31 active+clean; 577 KiB data, 4.0 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:33.119 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2870351557' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:18:33.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:33.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:33.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:33.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:33.326 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:33.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:33.339 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:18:33.339 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:18:33.339 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:18:33.339 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:18:33.339 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:18:33.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:33.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:33.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:33.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:33.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:33.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-09-02T10:18:33.574 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:33.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:33.676 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:33.698 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:33.699 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:33.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2870351557' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:18:33.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:33 smithi151 ceph-mon[16639]: osdmap e531: 12 total, 12 up, 11 in 2024-09-02T10:18:33.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:33.822 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:33.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:33.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:33.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:33.949 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2870351557' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:18:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:33 smithi187 ceph-mon[20072]: osdmap e531: 12 total, 12 up, 11 in 2024-09-02T10:18:34.019 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2870351557' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:18:34.019 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:33 smithi162 ceph-mon[20279]: osdmap e531: 12 total, 12 up, 11 in 2024-09-02T10:18:34.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:18:34.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:34.070 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:34.071 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:34.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:34.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:34.193 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:34.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:34.294 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:34.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:34.318 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:34.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:34.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:34.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:34.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:34.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:34.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:34.565 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:34.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:34.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:34.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:34.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:34.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:34.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:18:34.756 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:18:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:18:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:18:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:18:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.24976.0:181206 3.3 e291/226 3:c5ec1ffd:::benchmark_data_smithi187_59989_object71663:head v 291'11456, mlcod=291'11454)", 2024-09-02T10:18:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:18:34.756 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 559.74762616500004, 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37650431099999998, 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006203+0000", 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006204+0000", 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6000000000000002e-07 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.006211+0000", 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7880000000000001e-06 2024-09-02T10:18:34.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377595+0000", 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.371383927 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377601+0000", 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0769999999999996e-06 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377649+0000", 2024-09-02T10:18:34.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8368000000000002e-05 2024-09-02T10:18:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.377669+0000", 2024-09-02T10:18:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0258000000000002e-05 2024-09-02T10:18:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.382663+0000", 2024-09-02T10:18:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049941880000000001 2024-09-02T10:18:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:09:15.382707+0000", 2024-09-02T10:18:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3844999999999998e-05 2024-09-02T10:18:34.761 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3855 4.a 4:57b917f5:::benchmark_data_smithi187_76542_object3854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.84451243900003, 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22371366500000001, 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:18:34.762 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3855 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:18:34.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909327+0000", 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9559999999999992e-06 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909360+0000", 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3556999999999997e-05 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909377+0000", 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6283e-05 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909385+0000", 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4030000000000007e-06 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909444+0000", 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9203000000000002e-05 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909571+0000", 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127324 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909715+0000", 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143888 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.910145+0000", 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042936599999999998 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.917291+0000", 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071467290000000001 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132953+0000", 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.215661517 2024-09-02T10:18:34.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132989+0000", 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6123e-05 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.133030+0000", 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1316000000000002e-05 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.768 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3871 4.a 4:530be978:::benchmark_data_smithi187_76542_object3870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.83773663199997, 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21808112499999999, 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3871 2024-09-02T10:18:34.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1899999999999998e-07 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916113+0000", 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9800999999999999e-05 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916118+0000", 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.771 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916121+0000", 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3330000000000001e-06 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916168+0000", 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7147000000000001e-05 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916291+0000", 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122934 2024-09-02T10:18:34.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916427+0000", 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135337 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916951+0000", 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052393700000000004 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924167+0000", 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072167580000000002 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134113+0000", 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20994539000000001 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134147+0000", 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4149000000000001e-05 2024-09-02T10:18:34.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134174+0000", 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6798999999999999e-05 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3887 4.a 4:5466a52d:::benchmark_data_smithi187_76542_object3886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.83207834199999, 2024-09-02T10:18:34.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21253008300000001, 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3887 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:18:34.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921752+0000", 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4160000000000001e-06 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921813+0000", 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0420000000000001e-05 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921822+0000", 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4299999999999995e-06 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921827+0000", 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9819999999999999e-06 2024-09-02T10:18:34.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921867+0000", 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0071000000000003e-05 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921944+0000", 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6533999999999996e-05 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:18:34.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922025+0000", 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1601999999999996e-05 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922360+0000", 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033462200000000002 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924931+0000", 2024-09-02T10:18:34.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025713160000000001 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134215+0000", 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209284054 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134247+0000", 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2147999999999998e-05 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134281+0000", 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3488000000000002e-05 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3898 4.a 4:55cc6e50:::benchmark_data_smithi187_76542_object3897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.82708241, 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20761331199999999, 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.782 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3898 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926748+0000", 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6599999999999994e-07 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926786+0000", 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8506e-05 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926798+0000", 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.221e-05 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926804+0000", 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3140000000000003e-06 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926853+0000", 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8804999999999998e-05 2024-09-02T10:18:34.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926965+0000", 2024-09-02T10:18:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112256 2024-09-02T10:18:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:18:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927110+0000", 2024-09-02T10:18:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014520099999999999 2024-09-02T10:18:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927488+0000", 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000378332 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929483+0000", 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019947670000000002 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134309+0000", 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20482557300000001 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134337+0000", 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8334999999999999e-05 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134360+0000", 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3047000000000001e-05 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3901 4.a 4:53342710:::benchmark_data_smithi187_76542_object3900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.82525431800002, 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20590491499999999, 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3901 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.790 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928576+0000", 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1699999999999997e-07 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928631+0000", 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5276999999999999e-05 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928642+0000", 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1195999999999999e-05 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928649+0000", 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.669e-06 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928691+0000", 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1868e-05 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928778+0000", 2024-09-02T10:18:34.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7416999999999993e-05 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928866+0000", 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8471e-05 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929233+0000", 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036694499999999998 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.794 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.942995+0000", 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013761572999999999 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134423+0000", 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19142793799999999 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134461+0000", 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7839000000000001e-05 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134480+0000", 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9004999999999998e-05 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:6012 4.5 4:a5fa00f8:::benchmark_data_smithi187_76542_object6011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 447.49082131599999, 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209785581, 2024-09-02T10:18:34.796 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6012 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263009+0000", 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0139999999999999e-06 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263120+0000", 2024-09-02T10:18:34.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011099000000000001 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263134+0000", 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4467999999999999e-05 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263141+0000", 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9820000000000002e-06 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263387+0000", 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024598300000000002 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263500+0000", 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113174 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263599+0000", 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8974999999999993e-05 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264127+0000", 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052785799999999997 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264358+0000", 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023054600000000001 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472691+0000", 2024-09-02T10:18:34.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20833321599999999 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472750+0000", 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8897000000000002e-05 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472793+0000", 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3477999999999999e-05 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:24533 4.c e335/333 4:3a0221c2:::benchmark_data_smithi187_76542_object24532:head v 335'1521, mlcod=335'1517)", 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 429.74278826400001, 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212837002, 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011042+0000", 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1650000000000001e-06 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011129+0000", 2024-09-02T10:18:34.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7443999999999998e-05 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222953+0000", 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21182389400000001 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222960+0000", 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8569999999999997e-06 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223010+0000", 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0182000000000002e-05 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223032+0000", 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1699999999999999e-05 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223838+0000", 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080572200000000004 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223878+0000", 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0037999999999999e-05 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:25165 4.c e336/333 4:3c3f6316:::benchmark_data_smithi187_76542_object25164:head v 336'1563, mlcod=335'1560)", 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 429.10855871400003, 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204647308, 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.807 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645271+0000", 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9999999999999996e-07 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645443+0000", 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017179700000000001 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848893+0000", 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203449884 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848900+0000", 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8360000000000003e-06 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848961+0000", 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1163999999999999e-05 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848982+0000", 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1403000000000002e-05 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849888+0000", 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000905469 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849918+0000", 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9855000000000001e-05 2024-09-02T10:18:34.811 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:52446 4.15 e362/353 4:acabd05d:::benchmark_data_smithi187_76542_object52445:head v 362'2522, mlcod=362'2519)", 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 396.06042935400001, 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.220391959, 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9599999999999999e-07 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.813 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693457+0000", 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6582000000000001e-05 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912812+0000", 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21935476500000001 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912819+0000", 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4989999999999997e-06 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.814 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912871+0000", 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1388000000000003e-05 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912887+0000", 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5911999999999999e-05 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.913759+0000", 2024-09-02T10:18:34.815 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087215899999999996 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.913792+0000", 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2957999999999999e-05 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:58518 4.1 e366/341 4:8490730e:::benchmark_data_smithi187_76542_object58517:head v 366'2807, mlcod=366'2804)", 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 390.38870112699999, 2024-09-02T10:18:34.816 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209866793, 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.817 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365129+0000", 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7100000000000001e-07 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365181+0000", 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1816e-05 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573805+0000", 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.2086249 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.818 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573812+0000", 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1349999999999997e-06 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573866+0000", 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4373000000000003e-05 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573888+0000", 2024-09-02T10:18:34.819 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2008999999999999e-05 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.574955+0000", 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010673519999999999 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.574995+0000", 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9536999999999998e-05 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.820 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:75251 4.11 e371/356 4:8ba31329:::benchmark_data_smithi187_76542_object75250:head v 371'3396, mlcod=371'3394)", 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 377.088372107, 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37157874099999999, 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.821 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665458+0000", 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9400000000000004e-07 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.822 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665521+0000", 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2685e-05 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035523+0000", 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37000295799999999 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035542+0000", 2024-09-02T10:18:34.823 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8692999999999999e-05 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035709+0000", 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016700999999999999 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035745+0000", 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5913e-05 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.824 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.037004+0000", 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001258459 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.037036+0000", 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2228999999999999e-05 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.825 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:90191 4.15 e375/353 4:afce227e:::benchmark_data_smithi187_76542_object90190:head v 375'3680, mlcod=375'3677)", 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 363.65558256100002, 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20444974699999999, 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.826 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9299999999999997e-07 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098273+0000", 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5378999999999999e-05 2024-09-02T10:18:34.827 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301201+0000", 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20292787800000001 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301213+0000", 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860000000000001e-05 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.828 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301360+0000", 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014643600000000001 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301383+0000", 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2782999999999999e-05 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302662+0000", 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012796470000000001 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.829 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302696+0000", 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3970999999999997e-05 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126679 4.1 e392/341 4:86205e4c:::benchmark_data_smithi187_76542_object34343:head v 392'4951, mlcod=392'4948)", 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 319.28602229000001, 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44507702900000001, 2024-09-02T10:18:34.830 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.831 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467808+0000", 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.277e-06 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467816+0000", 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3690000000000007e-06 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910880+0000", 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44306368899999998 2024-09-02T10:18:34.832 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910889+0000", 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0380000000000005e-06 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910948+0000", 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8498000000000001e-05 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.833 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910970+0000", 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2580000000000001e-05 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912839+0000", 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018688120000000001 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912884+0000", 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4765999999999998e-05 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.834 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126802 4.1 e392/341 4:81a050df:::benchmark_data_smithi187_76542_object34466:head v 392'4952, mlcod=392'4949)", 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 318.84297891400001, 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25345677799999999, 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.835 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910851+0000", 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0690000000000001e-06 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.836 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910859+0000", 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.233e-06 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162387+0000", 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25152825099999998 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162402+0000", 2024-09-02T10:18:34.837 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4905999999999999e-05 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162538+0000", 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013591800000000001 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162581+0000", 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2991000000000002e-05 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.838 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164270+0000", 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016893419999999999 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164307+0000", 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7067999999999998e-05 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.839 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:64011 5.9 e465/431 5:9035e690:::benchmark_data_smithi187_89405_object64010:head v 465'4071, mlcod=464'4068)", 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 177.53656023799999, 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.200897045, 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.840 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1100000000000002e-07 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217293+0000", 2024-09-02T10:18:34.841 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3326999999999999e-05 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416283+0000", 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19899007799999999 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416287+0000", 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3529999999999997e-06 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.842 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416337+0000", 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0272000000000003e-05 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416360+0000", 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3147e-05 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.418133+0000", 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001772605 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.843 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.418166+0000", 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2951999999999999e-05 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:86619 5.9 e473/431 5:9bb5eff1:::benchmark_data_smithi187_89405_object86618:head v 473'5442, mlcod=473'5439)", 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 156.21408860599999, 2024-09-02T10:18:34.844 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21915551599999999, 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:18:34.845 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.09e-07 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539774+0000", 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2465999999999997e-05 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.846 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.755999+0000", 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.216224731 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756004+0000", 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9009999999999999e-06 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756086+0000", 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1767000000000005e-05 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.847 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756100+0000", 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4285e-05 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.758849+0000", 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027488349999999998 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.758896+0000", 2024-09-02T10:18:34.848 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6721999999999998e-05 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162692 5.2 5:4dd653e2:::benchmark_data_smithi187_89405_object59087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 70.400664436, 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25261044100000002, 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:18:34.849 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162692 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.850 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353166+0000", 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8200000000000008e-07 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353170+0000", 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3880000000000002e-06 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.851 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353180+0000", 2024-09-02T10:18:34.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0336999999999999e-05 2024-09-02T10:18:34.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.852 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.852 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353188+0000", 2024-09-02T10:18:34.852 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7060000000000004e-06 2024-09-02T10:18:34.852 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.852 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353233+0000", 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4453000000000003e-05 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353306+0000", 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3559999999999994e-05 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353375+0000", 2024-09-02T10:18:34.853 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9202999999999995e-05 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360656+0000", 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072804360000000004 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360676+0000", 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0075e-05 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.854 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605656+0000", 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24498062500000001 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605693+0000", 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6377000000000001e-05 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605775+0000", 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2299000000000006e-05 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.855 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162694 5.2 5:4549dcce:::benchmark_data_smithi187_89405_object59089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 70.400068249, 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25215807000000001, 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:18:34.856 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162694 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.857 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353762+0000", 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3600000000000003e-07 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353765+0000", 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6559999999999998e-06 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353776+0000", 2024-09-02T10:18:34.858 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0243e-05 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353781+0000", 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0540000000000002e-06 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.356988+0000", 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032069899999999998 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:18:34.859 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.357116+0000", 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012815 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.357235+0000", 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011919 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360716+0000", 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003481254 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.860 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.409925+0000", 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049208533999999998 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605820+0000", 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19589521600000001 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605878+0000", 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8149000000000002e-05 2024-09-02T10:18:34.861 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:18:34.862 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:18:34.862 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:18:34.862 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605919+0000", 2024-09-02T10:18:34.862 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0898e-05 2024-09-02T10:18:34.862 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.862 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.862 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.862 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:18:34.862 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:18:34.862 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:18:34.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:34.884 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:34.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:34.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:35.042 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-02T10:18:34.975+0000 7fd47f7fe640 0 --2- 172.21.15.151:0/2530439873 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd4801038e0 0x7fd480198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-02T10:18:35.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:34 smithi151 ceph-mon[16639]: pgmap v1238: 31 pgs: 31 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:35.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3291922482' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:18:35.043 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:18:35.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:35.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:35.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:34 smithi162 ceph-mon[20279]: pgmap v1238: 31 pgs: 31 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3291922482' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:18:35.175 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:35.201 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:35.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:34 smithi187 ceph-mon[20072]: pgmap v1238: 31 pgs: 31 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3291922482' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:18:35.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:35.324 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:35.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:35.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:35.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:35.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:35.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:35.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:35.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:35.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:35.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:35.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:35.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:35.726 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:35.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:35.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:35.821 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:35.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:35.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:35 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3291922482' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:18:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:35 smithi151 ceph-mon[16639]: osdmap e532: 12 total, 12 up, 11 in 2024-09-02T10:18:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:35 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:18:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:35 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:18:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1045799857' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-02T10:18:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2530439873' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:18:35.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:35.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:36.032 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:35 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:36.032 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3291922482' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:18:36.032 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:35 smithi162 ceph-mon[20279]: osdmap e532: 12 total, 12 up, 11 in 2024-09-02T10:18:36.032 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:35 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:18:36.032 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:35 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:18:36.032 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1045799857' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-02T10:18:36.032 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2530439873' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:18:36.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:18:36.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:36.053 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_4' created 2024-09-02T10:18:36.053 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-02T10:18:36.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:36.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:36.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:36.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:36.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:36.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:18:36.209 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:18:36.230 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:18:36.230 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:18:36.250 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:18:36.250 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:18:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:35 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3291922482' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:18:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:35 smithi187 ceph-mon[20072]: osdmap e532: 12 total, 12 up, 11 in 2024-09-02T10:18:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:35 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:18:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:35 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:18:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1045799857' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16} : dispatch 2024-09-02T10:18:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2530439873' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:18:36.270 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:18:36.271 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:18:36.281 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:36.291 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:18:36.291 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:18:36.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:36.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:18:36.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:18:36.322 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:36.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:36.332 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:18:36.333 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:18:36.353 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:18:36.354 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:18:36.375 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:18:36.375 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:18:36.395 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:18:36.396 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:18:36.417 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:18:36.418 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:18:36.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:36.441 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:18:36.441 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:18:36.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:36.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:36.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:36.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:36.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:36.671 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:36.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:36.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:36.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:36.708 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:36.708 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:36.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:36.816 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:36.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:36.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-02T10:18:36.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:36 smithi151 ceph-mon[16639]: pgmap v1240: 31 pgs: 31 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1045799857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-02T10:18:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2530439873' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:18:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:36 smithi151 ceph-mon[16639]: osdmap e533: 12 total, 12 up, 11 in 2024-09-02T10:18:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2912247696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:36.943 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:36.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:37.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:37.071 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:37.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:36 smithi162 ceph-mon[20279]: pgmap v1240: 31 pgs: 31 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1045799857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-02T10:18:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2530439873' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:18:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:36 smithi162 ceph-mon[20279]: osdmap e533: 12 total, 12 up, 11 in 2024-09-02T10:18:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2912247696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:37.173 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:37.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:37.201 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:37.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:36 smithi187 ceph-mon[20072]: pgmap v1240: 31 pgs: 31 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1045799857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-02T10:18:37.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2530439873' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:18:37.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:36 smithi187 ceph-mon[20072]: osdmap e533: 12 total, 12 up, 11 in 2024-09-02T10:18:37.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2912247696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:37.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:37.328 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:37.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:37.329 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:37.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:18:37.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:37.451 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:37.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:37.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:37.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:37.575 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:37.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:37.675 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:37.676 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:37.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:37.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:37.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:18:37.804 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:37.829 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:37.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:37.830 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:37.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:37.956 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:37.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:38.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:38.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:38.086 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:38.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-09-02T10:18:38.160 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:38.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:38.191 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:38.192 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:38.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:38.213 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:38.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:38.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:38 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:18:38.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:38 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:18:38.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:38 smithi151 ceph-mon[16639]: osdmap e534: 12 total, 12 up, 11 in 2024-09-02T10:18:38.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:38 smithi151 ceph-mon[16639]: pgmap v1243: 47 pgs: 2 creating+activating, 4 creating+peering, 9 unknown, 32 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:38.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:38.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:38.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:38.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:38 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:18:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:38 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:18:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:38 smithi162 ceph-mon[20279]: osdmap e534: 12 total, 12 up, 11 in 2024-09-02T10:18:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:38 smithi162 ceph-mon[20279]: pgmap v1243: 47 pgs: 2 creating+activating, 4 creating+peering, 9 unknown, 32 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:38.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:38.467 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:38.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:38.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:38 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:18:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:38 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:18:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:38 smithi187 ceph-mon[20072]: osdmap e534: 12 total, 12 up, 11 in 2024-09-02T10:18:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:38 smithi187 ceph-mon[20072]: pgmap v1243: 47 pgs: 2 creating+activating, 4 creating+peering, 9 unknown, 32 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:38.569 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:38.592 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:38.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:38.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:38.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:38.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:38.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:38.735 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:38.771 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:38.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:38.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:18:38.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:38.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:38.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:38.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:38.968 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:38.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:38.969 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:39.065 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-02T10:18:39.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:39.072 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:18:39.092 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:39.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:39.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:39.215 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:39.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:39.317 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:39 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:18:39.317 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:39 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:18:39.317 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:39 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:18:39.317 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:39 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:18:39.317 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:18:39.317 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:18:39.317 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:39 smithi162 ceph-mon[20279]: osdmap e535: 12 total, 12 up, 11 in 2024-09-02T10:18:39.317 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3902702705' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:18:39.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:39.340 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:39.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:39 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:18:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:39 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:18:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:39 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:18:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:39 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:18:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:18:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:18:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:39 smithi151 ceph-mon[16639]: osdmap e535: 12 total, 12 up, 11 in 2024-09-02T10:18:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3902702705' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:18:39.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:39.467 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:39.468 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:39 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:18:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:39 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:18:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:39 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:18:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:39 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:18:39.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:18:39.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:18:39.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:39 smithi187 ceph-mon[20072]: osdmap e535: 12 total, 12 up, 11 in 2024-09-02T10:18:39.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3902702705' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:18:39.541 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:39.541 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:39.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:39.595 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:39.597 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:39.698 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:39.720 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:39.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:39.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:18:39.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:39.845 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:39.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:39.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:39.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:39.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:40.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:40.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:40.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:40.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:40.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:18:40.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:40.215 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:40.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:40.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:40.339 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:40.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:40 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:18:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:40 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:18:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3902702705' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:18:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:40 smithi162 ceph-mon[20279]: osdmap e536: 12 total, 12 up, 11 in 2024-09-02T10:18:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:40 smithi162 ceph-mon[20279]: pgmap v1246: 47 pgs: 2 creating+activating, 4 creating+peering, 1 unknown, 40 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:40.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:40 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:18:40.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:40 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:18:40.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3902702705' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:18:40.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:40 smithi151 ceph-mon[16639]: osdmap e536: 12 total, 12 up, 11 in 2024-09-02T10:18:40.353 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:40 smithi151 ceph-mon[16639]: pgmap v1246: 47 pgs: 2 creating+activating, 4 creating+peering, 1 unknown, 40 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:40.392 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:40.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:40.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:40.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:40.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:40 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:18:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:40 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:18:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3902702705' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:18:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:40 smithi187 ceph-mon[20072]: osdmap e536: 12 total, 12 up, 11 in 2024-09-02T10:18:40.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:40 smithi187 ceph-mon[20072]: pgmap v1246: 47 pgs: 2 creating+activating, 4 creating+peering, 1 unknown, 40 active+clean; 577 KiB data, 4.1 GiB used, 979 GiB / 983 GiB avail 2024-09-02T10:18:40.543 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:40.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:40.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:40.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:40.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:40.713 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:40.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:40.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:40.838 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:40.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:40.839 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:40.866 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:40.866 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:40.888 DEBUG:teuthology.orchestra.run.smithi187:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-02T10:18:40.891 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-02T10:18:40.891 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-02T10:18:40.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:40.946 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:hints = 1 2024-09-02T10:18:40.946 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-02T10:18:40.946 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object prefix: benchmark_data_smithi187_102448 2024-09-02T10:18:40.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:18:40.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 0 0 0 0 0 0 - 0 2024-09-02T10:18:40.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:40.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:40.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:41.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:41.093 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:18:41.093 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-02T10:18:41.096 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:41.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:41.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:41.198 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:41.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:41.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:41.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:41.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:41 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:18:41.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:41 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:18:41.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1938530159' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:41.331 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2726354822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:18:41.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:41 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:18:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:41 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:18:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1938530159' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2726354822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:18:41.353 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:41.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:41.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:41.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:41.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:41 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:18:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:41 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:18:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1938530159' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2726354822' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:18:41.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:41.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:41.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:41.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:41.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:18:41.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:41.729 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:41.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:41.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:41.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:41.852 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:41.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:41.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 1 16 1792 1776 110.986 111 0.00531462 0.00876999 2024-09-02T10:18:41.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:41.959 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:41.968 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:18:41.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:18:41.979 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:41.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:41.996 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:18:41.996 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:18:42.025 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:18:42.025 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:18:42.049 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:18:42.049 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:18:42.070 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:18:42.070 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:18:42.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:42.091 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:18:42.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:18:42.103 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:42.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:42.115 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:18:42.115 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:18:42.151 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:18:42.151 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:18:42.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:42 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:18:42.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:42 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:18:42.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:42 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:18:42.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:42 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:18:42.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1938530159' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:42.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2726354822' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:18:42.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:42 smithi151 ceph-mon[16639]: osdmap e537: 12 total, 12 up, 11 in 2024-09-02T10:18:42.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1323139537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:42.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:42 smithi151 ceph-mon[16639]: pgmap v1248: 47 pgs: 2 creating+activating, 1 creating+peering, 44 active+clean; 9.0 MiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 2.0 MiB/s wr, 31 op/s 2024-09-02T10:18:42.174 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:18:42.174 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:18:42.199 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:18:42.200 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:18:42.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:42.230 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:18:42.230 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:18:42.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:42.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:42.235 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:42.252 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:18:42.252 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:18:42.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:42.338 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:42 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:18:42.338 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:42 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:18:42.338 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:42 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:18:42.338 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:42 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:18:42.338 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1938530159' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:42.339 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2726354822' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:18:42.339 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:42 smithi187 ceph-mon[20072]: osdmap e537: 12 total, 12 up, 11 in 2024-09-02T10:18:42.339 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1323139537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:42.339 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:42 smithi187 ceph-mon[20072]: pgmap v1248: 47 pgs: 2 creating+activating, 1 creating+peering, 44 active+clean; 9.0 MiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 2.0 MiB/s wr, 31 op/s 2024-09-02T10:18:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:42 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:18:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:42 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:18:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:42 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:18:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:42 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:18:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1938530159' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2726354822' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:18:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:42 smithi162 ceph-mon[20279]: osdmap e537: 12 total, 12 up, 11 in 2024-09-02T10:18:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1323139537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:42 smithi162 ceph-mon[20279]: pgmap v1248: 47 pgs: 2 creating+activating, 1 creating+peering, 44 active+clean; 9.0 MiB data, 4.1 GiB used, 979 GiB / 983 GiB avail; 2.0 MiB/s wr, 31 op/s 2024-09-02T10:18:42.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:42.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:42.453 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:18:42.453 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:18:42.453 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:18:42.453 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:18:42.453 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:18:42.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:42.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:42.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:42.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:42.615 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:42.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:42.616 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:42.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:42.746 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:42.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:42.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-09-02T10:18:42.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:42.874 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:42.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:42.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:42.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 2 16 3635 3619 113.082 115.188 0.00298815 0.00879006 2024-09-02T10:18:42.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:43.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:43.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:43.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:43.130 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:43.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:43.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:43.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:18:43.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:43.258 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:43.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:43.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:43.380 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:43.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:43.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:43 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:18:43.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:43 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:18:43.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:43.493 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:43 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:18:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:43 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:18:43.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:43.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:43.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:43.581 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:43.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:43 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:18:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:43 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:18:43.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:43.634 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:43.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:43.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:43.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:43.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:43.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:43.885 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:43.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:43.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 3 16 5045 5029 104.761 88.125 0.0029062 0.00940624 2024-09-02T10:18:43.980 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:18:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:18:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:42314 4.0 e355/341 4:049d10b0:::benchmark_data_smithi187_76542_object42313:head v 355'2240, mlcod=355'2237)", 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:49.052907+0000", 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 414.92472065599998, 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.227636956, 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.052907+0000", 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.052907+0000", 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.052908+0000", 2024-09-02T10:18:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.088e-06 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.053042+0000", 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013480799999999999 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279523+0000", 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22648093799999999 2024-09-02T10:18:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279535+0000", 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1678000000000001e-05 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279599+0000", 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4323000000000001e-05 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279617+0000", 2024-09-02T10:18:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7971000000000001e-05 2024-09-02T10:18:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.280511+0000", 2024-09-02T10:18:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00089334699999999996 2024-09-02T10:18:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.280544+0000", 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2802999999999997e-05 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90191 4.15 4:afce227e:::benchmark_data_smithi187_76542_object90190:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:31.097664+0000", 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 372.879962836, 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.205197566, 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90191 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097664+0000", 2024-09-02T10:18:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097664+0000", 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097666+0000", 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5859999999999999e-06 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097685+0000", 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9361e-05 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097699+0000", 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3359e-05 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097704+0000", 2024-09-02T10:18:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6849999999999998e-06 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097760+0000", 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5340000000000002e-05 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097871+0000", 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000111147 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.097958+0000", 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7638000000000004e-05 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.099226+0000", 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012679360000000001 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.229945+0000", 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.130718953 2024-09-02T10:18:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.302738+0000", 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.072792361999999999 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.302790+0000", 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1853e-05 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:31.302862+0000", 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2346000000000002e-05 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90787 4.15 4:afba0c08:::benchmark_data_smithi187_76542_object90786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:32.022436+0000", 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 371.95519161499999, 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27502997200000001, 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90787 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022436+0000", 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022436+0000", 2024-09-02T10:18:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022437+0000", 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5659999999999999e-06 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022462+0000", 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4437e-05 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022480+0000", 2024-09-02T10:18:43.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8493000000000001e-05 2024-09-02T10:18:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022489+0000", 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6959999999999994e-06 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022548+0000", 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9709e-05 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022678+0000", 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000129689 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022825+0000", 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014708899999999999 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.024159+0000", 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013338320000000001 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.024979+0000", 2024-09-02T10:18:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00081976499999999997 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297377+0000", 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.272397677 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297432+0000", 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5513000000000002e-05 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297466+0000", 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3506000000000001e-05 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90806 4.15 4:aae97575:::benchmark_data_smithi187_76542_object90805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:32.085698+0000", 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 371.89192877699998, 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21198275899999999, 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:18:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90806 2024-09-02T10:18:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085698+0000", 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085698+0000", 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085700+0000", 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.429e-06 2024-09-02T10:18:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085722+0000", 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2178999999999999e-05 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085736+0000", 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3687e-05 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085742+0000", 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1879999999999999e-06 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085792+0000", 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0034999999999998e-05 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085909+0000", 2024-09-02T10:18:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011707200000000001 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.086007+0000", 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7526000000000001e-05 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.087291+0000", 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001284246 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.087763+0000", 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000472466 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297588+0000", 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20982505200000001 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297648+0000", 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0167999999999997e-05 2024-09-02T10:18:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297681+0000", 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2710999999999999e-05 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90807 4.15 4:a985eaa0:::benchmark_data_smithi187_76542_object90806:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:32.098906+0000", 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 371.878721273, 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20543761199999999, 2024-09-02T10:18:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90807 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098906+0000", 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098906+0000", 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098908+0000", 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0729999999999999e-06 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098965+0000", 2024-09-02T10:18:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6520000000000001e-05 2024-09-02T10:18:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098975+0000", 2024-09-02T10:18:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0797e-05 2024-09-02T10:18:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.098981+0000", 2024-09-02T10:18:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3630000000000002e-06 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.099019+0000", 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8587000000000002e-05 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.099101+0000", 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2199000000000004e-05 2024-09-02T10:18:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.099187+0000", 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5387000000000001e-05 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.100008+0000", 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00082087200000000005 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.101112+0000", 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011045569999999999 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.304195+0000", 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20308306400000001 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.304295+0000", 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9996999999999998e-05 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.304344+0000", 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8195999999999997e-05 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPush(4.4 392/385 [PushOp(4:21e15029:::benchmark_data_smithi187_76542_object88778:head, version: 375'3759, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:21e15029:::benchmark_data_smithi187_76542_object88778:head@375'3759, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:15.639682+0000", 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 328.33794532899998, 2024-09-02T10:18:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23632888699999999, 2024-09-02T10:18:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:18:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639682+0000", 2024-09-02T10:18:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639682+0000", 2024-09-02T10:18:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639683+0000", 2024-09-02T10:18:44.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9500000000000002e-07 2024-09-02T10:18:44.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639708+0000", 2024-09-02T10:18:44.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5466e-05 2024-09-02T10:18:44.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.874909+0000", 2024-09-02T10:18:44.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23520074799999999 2024-09-02T10:18:44.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.874920+0000", 2024-09-02T10:18:44.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0825999999999999e-05 2024-09-02T10:18:44.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.874993+0000", 2024-09-02T10:18:44.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3734999999999998e-05 2024-09-02T10:18:44.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.875002+0000", 2024-09-02T10:18:44.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0610000000000002e-06 2024-09-02T10:18:44.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.876011+0000", 2024-09-02T10:18:44.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001008356 2024-09-02T10:18:44.023 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.023 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.024 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.024 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:180712 4.2 e422/341 4:40dddee0:::benchmark_data_smithi187_76542_object88376:head v 422'6632, mlcod=422'6630)", 2024-09-02T10:18:44.024 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:18.534975+0000", 2024-09-02T10:18:44.024 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 265.44265225100003, 2024-09-02T10:18:44.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21512784800000001, 2024-09-02T10:18:44.025 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.025 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.025 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.025 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534975+0000", 2024-09-02T10:18:44.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534975+0000", 2024-09-02T10:18:44.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534976+0000", 2024-09-02T10:18:44.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.257e-06 2024-09-02T10:18:44.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534985+0000", 2024-09-02T10:18:44.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3289999999999994e-06 2024-09-02T10:18:44.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747701+0000", 2024-09-02T10:18:44.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21271659000000001 2024-09-02T10:18:44.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747709+0000", 2024-09-02T10:18:44.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8609999999999998e-06 2024-09-02T10:18:44.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747772+0000", 2024-09-02T10:18:44.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2969e-05 2024-09-02T10:18:44.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747788+0000", 2024-09-02T10:18:44.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6201999999999998e-05 2024-09-02T10:18:44.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.750057+0000", 2024-09-02T10:18:44.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022690900000000001 2024-09-02T10:18:44.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.750103+0000", 2024-09-02T10:18:44.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5550000000000003e-05 2024-09-02T10:18:44.034 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.035 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.035 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.035 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:17898 5.a 5:5b8e596f:::benchmark_data_smithi187_89405_object17897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-02T10:18:44.035 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.425237+0000", 2024-09-02T10:18:44.035 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 226.55239050399999, 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.209706585, 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 17898 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425237+0000", 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425237+0000", 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425238+0000", 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.525e-06 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425259+0000", 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0987000000000001e-05 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425273+0000", 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3309000000000001e-05 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425278+0000", 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.891e-06 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425329+0000", 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0488999999999997e-05 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425415+0000", 2024-09-02T10:18:44.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6077999999999998e-05 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425520+0000", 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010503800000000001 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.426467+0000", 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00094675800000000002 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.426778+0000", 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031167899999999998 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634817+0000", 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20803839900000001 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634893+0000", 2024-09-02T10:18:44.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6501000000000005e-05 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634943+0000", 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9931e-05 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:17898 5.a e440/431 ondisk, result = 0)", 2024-09-02T10:18:44.042 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.427025+0000", 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 226.55060223800001, 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.207926318, 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.427025+0000", 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.427025+0000", 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.427026+0000", 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1999999999999999e-07 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.427029+0000", 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5159999999999999e-06 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634717+0000", 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.207688189 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634725+0000", 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5259999999999996e-06 2024-09-02T10:18:44.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634792+0000", 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7148000000000002e-05 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634810+0000", 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.806e-05 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634951+0000", 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014115899999999999 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63248 5.a 5:5adbc538:::benchmark_data_smithi187_89405_object63247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 187.463467264, 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21653889600000001, 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.047 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.048 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.048 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:44.048 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:44.048 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63248 2024-09-02T10:18:44.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.048 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:18:44.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514161+0000", 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.099e-06 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514181+0000", 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.999e-05 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514212+0000", 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0648999999999997e-05 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514371+0000", 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015914799999999999 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514421+0000", 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9804e-05 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514561+0000", 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000140285 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514666+0000", 2024-09-02T10:18:44.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104963 2024-09-02T10:18:44.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.516942+0000", 2024-09-02T10:18:44.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022756780000000002 2024-09-02T10:18:44.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521266+0000", 2024-09-02T10:18:44.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043245380000000002 2024-09-02T10:18:44.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730584+0000", 2024-09-02T10:18:44.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20931819099999999 2024-09-02T10:18:44.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730647+0000", 2024-09-02T10:18:44.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3040999999999995e-05 2024-09-02T10:18:44.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730699+0000", 2024-09-02T10:18:44.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.151e-05 2024-09-02T10:18:44.054 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.055 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.055 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.055 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63255 5.a 5:57c5feec:::benchmark_data_smithi187_89405_object63254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:18:44.055 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:18:44.055 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 187.45965736400001, 2024-09-02T10:18:44.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21287715600000001, 2024-09-02T10:18:44.056 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.056 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.056 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.056 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.056 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:44.056 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:44.056 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63255 2024-09-02T10:18:44.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.057 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:18:44.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517972+0000", 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.832e-06 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517998+0000", 2024-09-02T10:18:44.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6004e-05 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518024+0000", 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6290000000000001e-05 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518039+0000", 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.484e-05 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518103+0000", 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4350000000000006e-05 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518230+0000", 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012653999999999999 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:18:44.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518365+0000", 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013489900000000001 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.520755+0000", 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023907780000000001 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.523888+0000", 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031328179999999999 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730745+0000", 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.206856442 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730800+0000", 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5055000000000001e-05 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730847+0000", 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7308000000000002e-05 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63260 5.a 5:581436a1:::benchmark_data_smithi187_89405_object63259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 187.456352092, 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21179099900000001, 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.063 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63260 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521277+0000", 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.782e-06 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521301+0000", 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3893e-05 2024-09-02T10:18:44.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521320+0000", 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8879999999999999e-05 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521327+0000", 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6890000000000004e-06 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521399+0000", 2024-09-02T10:18:44.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1439000000000005e-05 2024-09-02T10:18:44.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521552+0000", 2024-09-02T10:18:44.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000153268 2024-09-02T10:18:44.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:18:44.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521685+0000", 2024-09-02T10:18:44.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013246699999999999 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.524092+0000", 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024077679999999998 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.525479+0000", 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001386916 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.732968+0000", 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20748902399999999 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.733024+0000", 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5269999999999998e-05 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.733066+0000", 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2602999999999999e-05 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.069 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.070 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.071 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:86595 5.a 5:52430871:::benchmark_data_smithi187_89405_object86594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-09-02T10:18:44.071 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:18:44.071 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 165.471542798, 2024-09-02T10:18:44.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.225552366, 2024-09-02T10:18:44.071 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 86595 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506087+0000", 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2469999999999998e-06 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506114+0000", 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7141999999999999e-05 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506133+0000", 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9137e-05 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506142+0000", 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8859999999999993e-06 2024-09-02T10:18:44.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506210+0000", 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7992999999999998e-05 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506361+0000", 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015085899999999999 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:18:44.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506506+0000", 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014534300000000001 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.509602+0000", 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030956009999999999 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.597950+0000", 2024-09-02T10:18:44.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088348135999999994 2024-09-02T10:18:44.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731542+0000", 2024-09-02T10:18:44.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13359195700000001 2024-09-02T10:18:44.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731597+0000", 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5368999999999998e-05 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731637+0000", 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9696000000000001e-05 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:122098 5.e e488/431 5:783becf1:::benchmark_data_smithi187_89405_object18493:head v 488'7571, mlcod=488'7567)", 2024-09-02T10:18:44.078 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 127.405947676, 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21171738700000001, 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1999999999999999e-07 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571686+0000", 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6010000000000002e-06 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780330+0000", 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20864460000000001 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780339+0000", 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1869999999999997e-06 2024-09-02T10:18:44.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780399+0000", 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9941000000000001e-05 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780424+0000", 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5751000000000001e-05 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.783348+0000", 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029234819999999998 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.783397+0000", 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9104999999999998e-05 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:126123 5.e e490/431 5:7800327a:::benchmark_data_smithi187_89405_object22518:head v 490'7825, mlcod=490'7821)", 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 122.437602396, 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25454940500000001, 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540026+0000", 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1000000000000001e-06 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540033+0000", 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8009999999999998e-06 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791598+0000", 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25156501399999998 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791605+0000", 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9569999999999996e-06 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791659+0000", 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4088000000000001e-05 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791680+0000", 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1450999999999998e-05 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.794543+0000", 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028628120000000002 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.794574+0000", 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1182000000000001e-05 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:194962 5.a 5:5b6b76d9:::benchmark_data_smithi187_89405_object91357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-02T10:18:44.088 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 41.018922525999997, 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22579173, 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194962 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958706+0000", 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.269e-06 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958711+0000", 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7010000000000004e-06 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958723+0000", 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2641e-05 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958731+0000", 2024-09-02T10:18:44.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9980000000000003e-06 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958775+0000", 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3399999999999998e-05 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958862+0000", 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7163999999999994e-05 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958935+0000", 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2804000000000003e-05 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.962637+0000", 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037025320000000001 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.963124+0000", 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048696900000000002 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184411+0000", 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.221286813 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184450+0000", 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9230999999999998e-05 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184496+0000", 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6208000000000002e-05 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:194969 5.a 5:503ca7c3:::benchmark_data_smithi187_89405_object91364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 41.008025387000004, 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21939375699999999, 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.095 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194969 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969603+0000", 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.068e-06 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969607+0000", 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3850000000000004e-06 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969617+0000", 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7750000000000004e-06 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969622+0000", 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2449999999999998e-06 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969678+0000", 2024-09-02T10:18:44.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5784e-05 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969792+0000", 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113993 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969919+0000", 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012692700000000001 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.975053+0000", 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051336569999999998 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.975381+0000", 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00032868799999999999 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188892+0000", 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21351113099999999 2024-09-02T10:18:44.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188951+0000", 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8777000000000003e-05 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188996+0000", 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4326999999999998e-05 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.102 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:194975 5.a 5:578d816e:::benchmark_data_smithi187_89405_object91370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-02T10:18:44.102 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:02.980192+0000", 2024-09-02T10:18:44.102 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 40.997434785000003, 2024-09-02T10:18:44.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20893245699999999, 2024-09-02T10:18:44.102 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.102 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194975 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.980192+0000", 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.980192+0000", 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.980194+0000", 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6950000000000001e-06 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.980200+0000", 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2169999999999999e-06 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.980219+0000", 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8853000000000001e-05 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.980228+0000", 2024-09-02T10:18:44.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3639999999999999e-06 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.980283+0000", 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5414999999999997e-05 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.980404+0000", 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121278 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.980519+0000", 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000114527 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.985021+0000", 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045018660000000002 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.985156+0000", 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013584799999999999 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.189048+0000", 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20389106100000001 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.189089+0000", 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1909999999999997e-05 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.189125+0000", 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5423000000000003e-05 2024-09-02T10:18:44.108 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207198 5.a 5:57bb5924:::benchmark_data_smithi187_89405_object103593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 23.861353391000002, 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44270453799999998, 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:44.109 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207198 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116275+0000", 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.393e-06 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116281+0000", 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.694e-06 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116296+0000", 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5234e-05 2024-09-02T10:18:44.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116305+0000", 2024-09-02T10:18:44.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5650000000000002e-06 2024-09-02T10:18:44.112 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116360+0000", 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5164999999999998e-05 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116445+0000", 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5158e-05 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116517+0000", 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2376e-05 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.121133+0000", 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046159479999999999 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.121221+0000", 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7831999999999995e-05 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:44.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558874+0000", 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43765253900000001 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558932+0000", 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7908000000000002e-05 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558978+0000", 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6726e-05 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:207204 5.4 e525/431 5:2473bd20:::benchmark_data_smithi187_89405_object103599:head v 525'13184, mlcod=525'13181)", 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 23.850198433999999, 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42712563199999998, 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:18:44.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127430+0000", 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.251e-06 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:44.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127438+0000", 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9039999999999992e-06 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550537+0000", 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42309940200000001 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550550+0000", 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2262999999999999e-05 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:44.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550695+0000", 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000145845 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550728+0000", 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2923000000000003e-05 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554509+0000", 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037808550000000001 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:44.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554554+0000", 2024-09-02T10:18:44.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5188999999999997e-05 2024-09-02T10:18:44.120 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.120 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.120 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.120 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:44.120 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:44.120 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:18:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:44.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:44.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:44.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:18:44.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:44.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:44.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:44.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:44 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:18:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:44 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:18:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:44 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:18:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:44 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:18:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:44 smithi187 ceph-mon[20072]: pgmap v1249: 47 pgs: 47 active+clean; 133 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 22 MiB/s wr, 353 op/s 2024-09-02T10:18:44.355 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:44.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:44.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:44 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:18:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:44 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:18:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:44 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:18:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:44 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:18:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:44 smithi151 ceph-mon[16639]: pgmap v1249: 47 pgs: 47 active+clean; 133 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 22 MiB/s wr, 353 op/s 2024-09-02T10:18:44.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:44.501 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:44.516 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:44.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:44.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:44 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:18:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:44 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:18:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:44 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:18:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:44 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:18:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:44 smithi162 ceph-mon[20279]: pgmap v1249: 47 pgs: 47 active+clean; 133 MiB data, 4.4 GiB used, 979 GiB / 983 GiB avail; 22 MiB/s wr, 353 op/s 2024-09-02T10:18:44.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:44.639 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:44.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:44.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:44.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:44.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:44.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:44.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:44.892 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:44.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:44.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 4 16 6397 6381 99.6945 84.5 0.00436678 0.0100121 2024-09-02T10:18:44.965 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:18:44.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:45.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:45.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:45.110 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-09-02T10:18:45.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:45.163 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:45.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:45 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:18:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:45 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:18:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1295038315' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:18:45.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:45.282 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:45.293 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:45.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:45.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:45.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:45.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:45.420 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:45.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:45 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:18:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:45 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:18:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1295038315' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:18:45.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:18:45.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:45.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:45.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:45 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:18:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:45 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:18:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1295038315' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:18:45.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:45.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:45.677 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:45.778 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:45.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:45.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:45.813 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:18:45.813 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:18:45.813 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:18:45.813 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:18:45.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-09-02T10:18:45.880 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:45.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:45.925 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:45.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:45.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:45.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 5 16 8078 8062 100.766 105.062 0.0505941 0.00984886 2024-09-02T10:18:46.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:46.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:46.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:46.050 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:46.118 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:18:46.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:46.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:46.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:46.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:46.303 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:46.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:46.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:46.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1295038315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:18:46.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:46 smithi187 ceph-mon[20072]: osdmap e538: 12 total, 12 up, 11 in 2024-09-02T10:18:46.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:46 smithi187 ceph-mon[20072]: pgmap v1251: 47 pgs: 47 active+clean; 190 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 30 MiB/s wr, 484 op/s 2024-09-02T10:18:46.427 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1465847099' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:18:46.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:46.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1295038315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:18:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:46 smithi151 ceph-mon[16639]: osdmap e538: 12 total, 12 up, 11 in 2024-09-02T10:18:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:46 smithi151 ceph-mon[16639]: pgmap v1251: 47 pgs: 47 active+clean; 190 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 30 MiB/s wr, 484 op/s 2024-09-02T10:18:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1465847099' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:18:46.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:46.553 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:18:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:46.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:46.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1295038315' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:18:46.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:46 smithi162 ceph-mon[20279]: osdmap e538: 12 total, 12 up, 11 in 2024-09-02T10:18:46.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:46 smithi162 ceph-mon[20279]: pgmap v1251: 47 pgs: 47 active+clean; 190 MiB data, 4.6 GiB used, 979 GiB / 983 GiB avail; 30 MiB/s wr, 484 op/s 2024-09-02T10:18:46.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1465847099' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:18:46.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:46.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:46.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:46.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:46.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:46.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:46.882 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:46.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:46.943 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:46.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:46.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 6 16 10031 10015 104.314 122.062 0.00164406 0.00953559 2024-09-02T10:18:47.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:47.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:47.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:47.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:47.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:47.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:47.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:47 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:47.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1465847099' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:18:47.231 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:47 smithi151 ceph-mon[16639]: osdmap e539: 12 total, 12 up, 11 in 2024-09-02T10:18:47.278 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:47 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:47.278 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1465847099' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:18:47.278 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:47 smithi187 ceph-mon[20072]: osdmap e539: 12 total, 12 up, 11 in 2024-09-02T10:18:47.284 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:47.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:47.308 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:18:47.309 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:18:47.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:47.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:47.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:47.345 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:18:47.345 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:18:47.376 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:18:47.376 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:18:47.403 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:18:47.404 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:18:47.413 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:18:47.413 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:18:47.413 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:18:47.414 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:18:47.414 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:18:47.432 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:18:47.432 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:18:47.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:47.462 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:18:47.462 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:18:47.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:47.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:47.489 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:18:47.489 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:18:47.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:47.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:47.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:47.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:47 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1465847099' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:18:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:47 smithi162 ceph-mon[20279]: osdmap e539: 12 total, 12 up, 11 in 2024-09-02T10:18:47.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:47.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:18:47.716 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:18:47.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:47.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:47.742 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:18:47.743 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:18:47.769 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:18:47.769 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:18:47.806 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:18:47.806 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:18:47.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:47.833 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:18:47.834 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:18:47.845 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:47.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:47.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:47.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 7 15 11537 11522 102.866 94.1875 0.00268416 0.00970837 2024-09-02T10:18:47.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:47.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:48.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:48.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:48.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:48.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:48.120 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-09-02T10:18:48.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:48.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:48.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:48.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:18:48.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:48.350 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:48.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3299835410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:48 smithi151 ceph-mon[16639]: pgmap v1253: 47 pgs: 47 active+clean; 407 MiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:18:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2306167356' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:18:48.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:48.474 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:48.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:48.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3299835410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:48 smithi187 ceph-mon[20072]: pgmap v1253: 47 pgs: 47 active+clean; 407 MiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:18:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2306167356' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:18:48.523 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:48.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:48.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3299835410' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:48 smithi162 ceph-mon[20279]: pgmap v1253: 47 pgs: 47 active+clean; 407 MiB data, 5.4 GiB used, 978 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:18:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2306167356' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:18:48.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:48.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:48.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:48.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:48.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:48.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:48.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:48.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:48.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 8 16 12692 12676 99.0229 72.125 0.00296527 0.0100883 2024-09-02T10:18:48.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:48.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:18:48.986 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:48.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:49.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:49.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:49.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:49.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:49.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:49.239 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:49.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:49.274 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:49.274 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:49.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:49.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:49.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2306167356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:18:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:49 smithi151 ceph-mon[16639]: osdmap e540: 12 total, 12 up, 11 in 2024-09-02T10:18:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:49 smithi151 ceph-mon[16639]: osdmap e541: 12 total, 12 up, 11 in 2024-09-02T10:18:49.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:49.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:49.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:49.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2306167356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:18:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:49 smithi187 ceph-mon[20072]: osdmap e540: 12 total, 12 up, 11 in 2024-09-02T10:18:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:49 smithi187 ceph-mon[20072]: osdmap e541: 12 total, 12 up, 11 in 2024-09-02T10:18:49.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2306167356' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:18:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:49 smithi162 ceph-mon[20279]: osdmap e540: 12 total, 12 up, 11 in 2024-09-02T10:18:49.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:49 smithi162 ceph-mon[20279]: osdmap e541: 12 total, 12 up, 11 in 2024-09-02T10:18:49.622 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:49.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:49.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:49.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:49.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:49.747 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:49.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:49.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:49.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:49.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 9 16 14208 14192 98.5474 94.75 0.00719757 0.0101316 2024-09-02T10:18:49.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:50.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:50.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:50.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:18:50.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:50.128 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:50.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:50.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:50.255 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:50.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:50.357 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:50.379 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:50.381 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:50.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:50.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:50.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:50 smithi151 ceph-mon[16639]: pgmap v1256: 47 pgs: 47 active+clean; 646 MiB data, 5.9 GiB used, 977 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-02T10:18:50.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-09-02T10:18:50.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:50.500 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:50 smithi187 ceph-mon[20072]: pgmap v1256: 47 pgs: 47 active+clean; 646 MiB data, 5.9 GiB used, 977 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-02T10:18:50.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:50.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:50.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:50.561 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:50.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:50 smithi162 ceph-mon[20279]: pgmap v1256: 47 pgs: 47 active+clean; 646 MiB data, 5.9 GiB used, 977 GiB / 983 GiB avail; 108 MiB/s wr, 1.73k op/s 2024-09-02T10:18:50.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:50.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:50.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:50.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:50.730 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:50.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:50.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:50.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:50.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:50.880 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:50.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:50.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 10 16 14946 14930 93.3048 46.125 0.0260247 0.0107029 2024-09-02T10:18:50.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:51.006 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:51.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:51.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:51.136 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:51.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:51.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:51.205 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:51.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:51.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:51.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:51.300 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:18:51.300 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:18:51.300 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:51 smithi187 ceph-mon[20072]: osdmap e542: 12 total, 12 up, 11 in 2024-09-02T10:18:51.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:18:51.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:18:51.332 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:51 smithi151 ceph-mon[16639]: osdmap e542: 12 total, 12 up, 11 in 2024-09-02T10:18:51.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:51.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:51.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:51.438 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:51.438 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:51.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:51.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:51.531 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:51.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:51.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:18:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:18:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:18:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:51 smithi162 ceph-mon[20279]: osdmap e542: 12 total, 12 up, 11 in 2024-09-02T10:18:51.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:51.657 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:51.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:51.658 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:51.759 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:51.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:51.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:51.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:51.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:51.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:51.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:51.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 11 16 16744 16728 95.0377 112.375 0.0038307 0.0105152 2024-09-02T10:18:51.983 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:52.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:52.040 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:52.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:52.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:52.153 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:18:52.155 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-09-02T10:18:52.168 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:52.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:52.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:52.208 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:18:52.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:52.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:52.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:52.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:52.421 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:52.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:52.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:52 smithi151 ceph-mon[16639]: pgmap v1258: 47 pgs: 47 active+clean; 762 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-02T10:18:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/437951995' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3703409046' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:18:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:52 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:18:52.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:52 smithi162 ceph-mon[20279]: pgmap v1258: 47 pgs: 47 active+clean; 762 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-02T10:18:52.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/437951995' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:52.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3703409046' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:18:52.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:52 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:18:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:52 smithi187 ceph-mon[20072]: pgmap v1258: 47 pgs: 47 active+clean; 762 MiB data, 6.2 GiB used, 977 GiB / 983 GiB avail; 110 MiB/s wr, 1.75k op/s 2024-09-02T10:18:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/437951995' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:18:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3703409046' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:18:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:52 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:18:52.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:52.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:52.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:52.558 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:52.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:52.677 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:52.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:52.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:52.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:52.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:52.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:52.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:52.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:52.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:52.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 12 16 18104 18088 94.2007 85 0.00322598 0.0106073 2024-09-02T10:18:53.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:53.063 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:53.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:53.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:53.072 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:18:53.072 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:18:53.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:18:53.148 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:18:53.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:53.177 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:18:53.177 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:18:53.194 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:53.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:53.210 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:18:53.211 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:18:53.236 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:18:53.236 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:18:53.236 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:53 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:53.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/437951995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:53.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:53 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:18:53.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:53 smithi151 ceph-mon[16639]: osdmap e543: 12 total, 12 up, 11 in 2024-09-02T10:18:53.237 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2546651672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:53.266 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:18:53.267 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:18:53.297 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:53.297 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:18:53.298 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:18:53.326 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:53.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:53.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:53.331 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:18:53.331 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:18:53.365 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:18:53.365 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:18:53.393 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:18:53.393 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:18:53.400 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:53 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:53.400 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/437951995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:53.400 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:53 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:18:53.400 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:53 smithi187 ceph-mon[20072]: osdmap e543: 12 total, 12 up, 11 in 2024-09-02T10:18:53.401 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2546651672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:53.423 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:18:53.423 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:18:53.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:53.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:18:53.453 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:18:53.457 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:53.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:53.483 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:18:53.484 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:18:53.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:53.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:53 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:53.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/437951995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:18:53.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:53 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:18:53.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:53 smithi162 ceph-mon[20279]: osdmap e543: 12 total, 12 up, 11 in 2024-09-02T10:18:53.564 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2546651672' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:18:53.590 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:53.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:53.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:53.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:53.718 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:53.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:53.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:53.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:53.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:18:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:53.842 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:53.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:53.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:53.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 13 16 19834 19818 95.271 108.125 0.00225576 0.0104839 2024-09-02T10:18:53.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:53.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:54.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:54.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:54.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:54.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:54.131 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:54.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:54 smithi151 ceph-mon[16639]: pgmap v1260: 47 pgs: 47 active+clean; 1.0 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-02T10:18:54.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:54 smithi151 ceph-mon[16639]: osdmap e544: 12 total, 12 up, 11 in 2024-09-02T10:18:54.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:54.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:54.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:54.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:54.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:54.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:54.416 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:54 smithi187 ceph-mon[20072]: pgmap v1260: 47 pgs: 47 active+clean; 1.0 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-02T10:18:54.417 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:54 smithi187 ceph-mon[20072]: osdmap e544: 12 total, 12 up, 11 in 2024-09-02T10:18:54.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:54.476 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:54.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:54.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:54 smithi162 ceph-mon[20279]: pgmap v1260: 47 pgs: 47 active+clean; 1.0 GiB data, 7.1 GiB used, 976 GiB / 983 GiB avail; 108 MiB/s wr, 1.72k op/s 2024-09-02T10:18:54.527 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:54 smithi162 ceph-mon[20279]: osdmap e544: 12 total, 12 up, 11 in 2024-09-02T10:18:54.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:54.606 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:54.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:54.607 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:54.665 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:18:54.665 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:18:54.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:18:54.665 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:18:54.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.665 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103692 4.19 4.40dd9139 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:18:54.665 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:18:54.665 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 368.565889056, 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.993593167, 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103692 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:18:54.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:18:54.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095744+0000", 2024-09-02T10:18:54.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6899999999999996e-07 2024-09-02T10:18:54.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095748+0000", 2024-09-02T10:18:54.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4499999999999997e-06 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095758+0000", 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2140000000000002e-06 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095764+0000", 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5690000000000001e-06 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095808+0000", 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3588999999999999e-05 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095814+0000", 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2299999999999996e-06 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089305+0000", 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99349078599999996 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089336+0000", 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.146e-05 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:103721 4.e e384/340 4:78ce6951:::benchmark_data_smithi187_76542_object11385:head v 384'6547, mlcod=384'6544)", 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 368.26983908900002, 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.265184527, 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.671 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391794+0000", 2024-09-02T10:18:54.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4000000000000001e-07 2024-09-02T10:18:54.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391799+0000", 2024-09-02T10:18:54.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2009999999999998e-06 2024-09-02T10:18:54.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654392+0000", 2024-09-02T10:18:54.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26259298599999997 2024-09-02T10:18:54.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654398+0000", 2024-09-02T10:18:54.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7880000000000003e-06 2024-09-02T10:18:54.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654446+0000", 2024-09-02T10:18:54.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8183000000000003e-05 2024-09-02T10:18:54.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654470+0000", 2024-09-02T10:18:54.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4093000000000001e-05 2024-09-02T10:18:54.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.656943+0000", 2024-09-02T10:18:54.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024726489999999999 2024-09-02T10:18:54.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.656978+0000", 2024-09-02T10:18:54.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4887e-05 2024-09-02T10:18:54.678 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.678 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.678 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.679 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103725 4.19 4.a40a4399 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:18:54.679 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:18:54.679 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 368.22514440999998, 2024-09-02T10:18:54.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65288606400000004, 2024-09-02T10:18:54.679 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.679 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.680 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:54.680 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.680 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:18:54.680 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:54.680 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103725 2024-09-02T10:18:54.680 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436489+0000", 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2750000000000001e-06 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436494+0000", 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4660000000000001e-06 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436506+0000", 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2123000000000001e-05 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436512+0000", 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6749999999999996e-06 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436566+0000", 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3285000000000002e-05 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436572+0000", 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6019999999999996e-06 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089367+0000", 2024-09-02T10:18:54.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65279432500000001 2024-09-02T10:18:54.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089374+0000", 2024-09-02T10:18:54.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3130000000000001e-06 2024-09-02T10:18:54.685 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.685 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.685 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.685 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103743 4.19 4.222eb379 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 367.992304601, 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42007966200000002, 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103743 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.686 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669329+0000", 2024-09-02T10:18:54.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.221e-06 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669334+0000", 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1340000000000003e-06 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669346+0000", 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2155e-05 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669351+0000", 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9810000000000003e-06 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669433+0000", 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1617000000000001e-05 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669445+0000", 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.207e-05 2024-09-02T10:18:54.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089400+0000", 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41995552600000002 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089407+0000", 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9580000000000001e-06 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103767 4.19 4.bc594c79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 367.891850224, 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31965745400000001, 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103767 2024-09-02T10:18:54.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769785+0000", 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5299999999999999e-06 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769796+0000", 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.164e-05 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769823+0000", 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7055000000000001e-05 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769828+0000", 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0350000000000001e-06 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769891+0000", 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2973999999999997e-05 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769903+0000", 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1569999999999999e-05 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089432+0000", 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31952966900000002 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089439+0000", 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9809999999999997e-06 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.695 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62902 5.0 5:0a3860dd:::benchmark_data_smithi187_89405_object62901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 200.02656561699999, 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28493924999999998, 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62902 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635068+0000", 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.502e-06 2024-09-02T10:18:54.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635089+0000", 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0916000000000001e-05 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635100+0000", 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1243000000000001e-05 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635107+0000", 2024-09-02T10:18:54.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5359999999999998e-06 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635155+0000", 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7939000000000003e-05 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635250+0000", 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5571999999999999e-05 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:18:54.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635339+0000", 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9158000000000003e-05 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.636945+0000", 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001605555 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.637559+0000", 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00061385400000000005 2024-09-02T10:18:54.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.919922+0000", 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28236284 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.919962+0000", 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0227e-05 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.920006+0000", 2024-09-02T10:18:54.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3908e-05 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:126223 5.4 5:2d883f96:::benchmark_data_smithi187_89405_object22618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 132.73570454, 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32322968800000001, 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126223 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925929+0000", 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2020000000000001e-06 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925933+0000", 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3479999999999997e-06 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925945+0000", 2024-09-02T10:18:54.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1559999999999999e-05 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925952+0000", 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4120000000000004e-06 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925997+0000", 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4759999999999998e-05 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.926077+0000", 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0203000000000004e-05 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.926145+0000", 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8311000000000004e-05 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929053+0000", 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029072289999999999 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.989494+0000", 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.060441307999999999 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249062+0000", 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25956764199999999 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249115+0000", 2024-09-02T10:18:54.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2896000000000003e-05 2024-09-02T10:18:54.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249157+0000", 2024-09-02T10:18:54.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2817e-05 2024-09-02T10:18:54.708 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.708 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.708 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:126223 5.4 e490/431 ondisk, result = 0)", 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 132.73236042600001, 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31989100399999998, 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:18:54.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2900000000000003e-07 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929276+0000", 2024-09-02T10:18:54.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4989999999999999e-06 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.248945+0000", 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31966908500000002 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.248950+0000", 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1749999999999996e-06 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249048+0000", 2024-09-02T10:18:54.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7676000000000005e-05 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249056+0000", 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2439999999999993e-06 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249163+0000", 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106696 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162650 5.0 5:0595c7e6:::benchmark_data_smithi187_89405_object59045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 90.358942537999994, 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29653194100000002, 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162650 2024-09-02T10:18:54.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302691+0000", 2024-09-02T10:18:54.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6139999999999999e-06 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302696+0000", 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1599999999999997e-06 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302706+0000", 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3959999999999995e-06 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302711+0000", 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2920000000000003e-06 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302770+0000", 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8545999999999998e-05 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302912+0000", 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014217599999999999 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.303032+0000", 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120482 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.307337+0000", 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004304993 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.307512+0000", 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017463799999999999 2024-09-02T10:18:54.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598774+0000", 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29126201499999999 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599166+0000", 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039243999999999998 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599222+0000", 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5189000000000003e-05 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:162688 5.8 e507/490 5:1af44aef:::benchmark_data_smithi187_89405_object59083:head v 507'10194, mlcod=507'10191)", 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 90.313728544, 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.262002719, 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.719 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8700000000000002e-07 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347908+0000", 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2030000000000002e-06 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347966+0000", 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7603999999999999e-05 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347970+0000", 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1160000000000001e-06 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.348622+0000", 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00065184499999999996 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.348639+0000", 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7731e-05 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609811+0000", 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26117189299999999 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609906+0000", 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5039999999999998e-05 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:162690 5.f e507/431 5:f963755e:::benchmark_data_smithi187_89405_object59085:head v 507'10319, mlcod=507'10317)", 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:18:54.723 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 90.310024615000003, 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26002222000000003, 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3699999999999999e-07 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351615+0000", 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1940000000000003e-06 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353238+0000", 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001623471 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353245+0000", 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4540000000000004e-06 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353298+0000", 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3186999999999997e-05 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353317+0000", 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9083999999999999e-05 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611594+0000", 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.258277337 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611630+0000", 2024-09-02T10:18:54.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5655999999999999e-05 2024-09-02T10:18:54.728 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.728 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.728 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.728 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162691 5.4 5:2922879e:::benchmark_data_smithi187_89405_object59086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:18:54.728 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:18:54.728 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 90.309564753999993, 2024-09-02T10:18:54.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25893540799999998, 2024-09-02T10:18:54.728 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162691 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352069+0000", 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6869999999999999e-06 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352075+0000", 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5940000000000001e-06 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352087+0000", 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.216e-05 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352093+0000", 2024-09-02T10:18:54.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5969999999999997e-06 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599399+0000", 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24730590099999999 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599528+0000", 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012868299999999999 2024-09-02T10:18:54.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599654+0000", 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000125959 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610381+0000", 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.01072701 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610393+0000", 2024-09-02T10:18:54.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1921e-05 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610740+0000", 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00034708900000000002 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610837+0000", 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7175000000000004e-05 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611003+0000", 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000165632 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162695 5.0 5:02845cd7:::benchmark_data_smithi187_89405_object59090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 90.307799110000005, 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.266211531, 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.735 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162695 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353834+0000", 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3900000000000004e-07 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353838+0000", 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3460000000000004e-06 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598349+0000", 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24451044699999999 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598367+0000", 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8236999999999998e-05 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609481+0000", 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011113576 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609606+0000", 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012559100000000001 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610188+0000", 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058149899999999997 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619903+0000", 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0097155599999999998 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619914+0000", 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1126e-05 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619947+0000", 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2432999999999999e-05 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.620001+0000", 2024-09-02T10:18:54.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4295000000000001e-05 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.620045+0000", 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3581999999999997e-05 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:173247 5.8 e509/490 5:173cf939:::benchmark_data_smithi187_89405_object69642:head v 509'10875, mlcod=509'10872)", 2024-09-02T10:18:54.741 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 77.519216104999998, 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25939572700000002, 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:18:54.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:54.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142417+0000", 2024-09-02T10:18:54.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6099999999999999e-07 2024-09-02T10:18:54.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142424+0000", 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9489999999999999e-06 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396116+0000", 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25369161699999998 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396123+0000", 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5059999999999998e-06 2024-09-02T10:18:54.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396178+0000", 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5271e-05 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396199+0000", 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1009000000000002e-05 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.401729+0000", 2024-09-02T10:18:54.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0055293260000000002 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.401812+0000", 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3088000000000001e-05 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207153 5.4 5:2d15d89f:::benchmark_data_smithi187_89405_object103548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 34.617115431999999, 2024-09-02T10:18:54.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.39443205100000001, 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207153 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044518+0000", 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1179999999999999e-06 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044523+0000", 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.113e-06 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044535+0000", 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1868999999999999e-05 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044540+0000", 2024-09-02T10:18:54.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9370000000000003e-06 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044599+0000", 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9589000000000002e-05 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044722+0000", 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122166 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044855+0000", 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133774 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.048607+0000", 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037520180000000002 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.075260+0000", 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.026652973 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438874+0000", 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.363613939 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438913+0000", 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8795999999999999e-05 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438949+0000", 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5759000000000001e-05 2024-09-02T10:18:54.753 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207153 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 34.573901075999999, 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35122426299999998, 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087732+0000", 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5300000000000004e-07 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087736+0000", 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0659999999999997e-06 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438760+0000", 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.351023794 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438766+0000", 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6499999999999999e-06 2024-09-02T10:18:54.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438864+0000", 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8116999999999996e-05 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438871+0000", 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1430000000000003e-06 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438955+0000", 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4839999999999994e-05 2024-09-02T10:18:54.757 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:207198 5.a e525/431 5:57bb5924:::benchmark_data_smithi187_89405_object103593:head v 525'12960, mlcod=525'12956)", 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 34.545045580999997, 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44214100000000001, 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1399999999999996e-07 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116593+0000", 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0079999999999999e-06 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554609+0000", 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43801511199999998 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554615+0000", 2024-09-02T10:18:54.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8349999999999998e-06 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554662+0000", 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7018000000000003e-05 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554678+0000", 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5116000000000001e-05 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558692+0000", 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040144350000000002 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558728+0000", 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5661999999999998e-05 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207204 5.4 5:2473bd20:::benchmark_data_smithi187_89405_object103599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:18:54.762 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 34.534513294, 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73061778899999996, 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207204 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127120+0000", 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2810000000000001e-06 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127126+0000", 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1210000000000003e-06 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127137+0000", 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0474e-05 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127142+0000", 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5419999999999996e-06 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127191+0000", 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8485000000000001e-05 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127278+0000", 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7492999999999997e-05 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127352+0000", 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3591999999999996e-05 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131106+0000", 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003754496 2024-09-02T10:18:54.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850393+0000", 2024-09-02T10:18:54.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.719286219 2024-09-02T10:18:54.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:54.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857585+0000", 2024-09-02T10:18:54.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071919810000000001 2024-09-02T10:18:54.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857692+0000", 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107038 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857737+0000", 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5067000000000001e-05 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207204 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 34.529986579000003, 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72609860400000004, 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131647+0000", 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2529999999999999e-06 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131653+0000", 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4119999999999998e-06 2024-09-02T10:18:54.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857394+0000", 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72574053699999996 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857411+0000", 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7283e-05 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857560+0000", 2024-09-02T10:18:54.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014918700000000001 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857574+0000", 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.329e-05 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857744+0000", 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017064200000000001 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207204 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 34.107045368999998, 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.295830757, 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554588+0000", 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8899999999999998e-07 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554593+0000", 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1039999999999998e-06 2024-09-02T10:18:54.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850206+0000", 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29561359100000001 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850223+0000", 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6730000000000001e-05 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:54.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850366+0000", 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014310500000000001 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850383+0000", 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6419999999999999e-05 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850418+0000", 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4918e-05 2024-09-02T10:18:54.777 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.778 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.778 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:54.778 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:54.778 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:18:54.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:54.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:54.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:54.923 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:54.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:54.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 14 16 20900 20884 93.2245 66.625 0.00614321 0.0106743 2024-09-02T10:18:55.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:55.054 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:55.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:55.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:55.181 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:55.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:55.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:55.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:55.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:55.309 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:55.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:55.434 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:55.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:55.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 10 1 2024-09-02T10:18:55.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:55.557 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:55.557 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:55.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:55.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:55.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:55.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:55.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:18:55.801 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:55.807 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:55.809 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:55.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:55.936 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:55.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:55.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:55.947 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 15 16 22096 22080 91.9925 74.75 0.0025388 0.0108583 2024-09-02T10:18:55.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:18:56.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:56.063 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:56.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:56.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:56.100 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:56.165 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:56.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:56.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:56.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:18:56.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:56.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:18:56.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:56.441 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:18:56.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:56.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:56.464 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:56 smithi187 ceph-mon[20072]: pgmap v1262: 47 pgs: 47 active+clean; 1.1 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-02T10:18:56.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:56.565 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:56 smithi151 ceph-mon[16639]: pgmap v1262: 47 pgs: 47 active+clean; 1.1 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-02T10:18:56.592 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:56.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:18:56.654 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:18:56.654 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:18:56.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:56.654 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:22840 6.2 e544/533 6:467ab248:::benchmark_data_smithi187_102448_object22839:head v 544'1458, mlcod=544'1453)", 2024-09-02T10:18:56.654 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:56.624888+0000", 2024-09-02T10:18:56.654 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.027837549, 2024-09-02T10:18:56.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.027863382999999999, 2024-09-02T10:18:56.654 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:56.624888+0000", 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:56.624888+0000", 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:56.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:56.624889+0000", 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7899999999999998e-07 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:56.624954+0000", 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5049999999999996e-05 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:56.652076+0000", 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.027121921 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:56.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:56.652083+0000", 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9800000000000001e-06 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:56.652144+0000", 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1216000000000005e-05 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:56.652171+0000", 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7484999999999999e-05 2024-09-02T10:18:56.657 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:56.658 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:56.658 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:56.658 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:56.658 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:18:56.658 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:18:56.658 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:18:56.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:18:56.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:56.721 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:56.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:56.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:56.844 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:56.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:56.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:56 smithi162 ceph-mon[20279]: pgmap v1262: 47 pgs: 47 active+clean; 1.1 GiB data, 7.3 GiB used, 976 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s 2024-09-02T10:18:56.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:56.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 16 15 23269 23254 90.8286 73.375 0.00350985 0.0109985 2024-09-02T10:18:56.972 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:56.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:57.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:18:57.098 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:57.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:18:57.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:57.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:57.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:57.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:57.353 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:57.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:57.426 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-09-02T10:18:57.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:18:57.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:57.481 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:57.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:57.584 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:57.610 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:57.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:57.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2960944936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:18:57.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:57.740 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:57.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2960944936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:18:57.756 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:57.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:18:57.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2960944936' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:18:57.864 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:57.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:18:57.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 17 16 24881 24865 91.4082 100.688 0.0413377 0.01093 2024-09-02T10:18:57.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:57.990 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:57.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:58.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:58.115 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:58.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:58.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:58.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:58.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:58.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:58.244 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:58.260 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:18:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:18:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:18:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:18:58.261 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:18:58.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:58.368 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:58.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:58.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:18:58.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:58.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:18:58.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:58.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:58.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:58 smithi151 ceph-mon[16639]: pgmap v1263: 47 pgs: 47 active+clean; 1.3 GiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-09-02T10:18:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2960944936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:18:58.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:58 smithi151 ceph-mon[16639]: osdmap e545: 12 total, 12 up, 11 in 2024-09-02T10:18:58.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:18:58.753 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:58.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:18:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:58 smithi187 ceph-mon[20072]: pgmap v1263: 47 pgs: 47 active+clean; 1.3 GiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-09-02T10:18:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2960944936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:18:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:58 smithi187 ceph-mon[20072]: osdmap e545: 12 total, 12 up, 11 in 2024-09-02T10:18:58.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:58 smithi162 ceph-mon[20279]: pgmap v1263: 47 pgs: 47 active+clean; 1.3 GiB data, 7.9 GiB used, 975 GiB / 983 GiB avail; 85 MiB/s wr, 1.37k op/s 2024-09-02T10:18:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2960944936' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:18:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:58 smithi162 ceph-mon[20279]: osdmap e545: 12 total, 12 up, 11 in 2024-09-02T10:18:58.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:58.881 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:58.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:58.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 18 16 26264 26248 91.1317 86.4375 0.012183 0.0109671 2024-09-02T10:18:58.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:18:58.983 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:18:59.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:59.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:18:59.048 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:18:59.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:18:59.164 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:59.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:18:59.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:59.297 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:59.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:59.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:59.321 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:18:59.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:18:59.421 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:59.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:18:59.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:18:59.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:18:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:59.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:18:59.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:18:59.669 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:59.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:18:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4061855799' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:18:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:59 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4061855799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:18:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:18:59 smithi151 ceph-mon[16639]: osdmap e546: 12 total, 12 up, 11 in 2024-09-02T10:18:59.750 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4061855799' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:18:59.750 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:59 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:59.750 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4061855799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:18:59.750 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:18:59 smithi187 ceph-mon[20072]: osdmap e546: 12 total, 12 up, 11 in 2024-09-02T10:18:59.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:18:59.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:18:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:59.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:18:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4061855799' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:18:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:59 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:18:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4061855799' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:18:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:18:59 smithi162 ceph-mon[20279]: osdmap e546: 12 total, 12 up, 11 in 2024-09-02T10:18:59.858 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:18:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:18:59.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:11571 4.d e326/318 4:b88a9f8d:::benchmark_data_smithi187_76542_object11570:head v 326'724, mlcod=326'721)", 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:10.795584+0000", 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 469.05977735099998, 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.49112458599999997, 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795584+0000", 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795584+0000", 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795585+0000", 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7899999999999998e-07 2024-09-02T10:18:59.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:10.795608+0000", 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3408e-05 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285722+0000", 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.490114102 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285750+0000", 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7104e-05 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285822+0000", 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2354999999999994e-05 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.285855+0000", 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2731000000000002e-05 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.286659+0000", 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00080397800000000001 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:11.286709+0000", 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0229000000000003e-05 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.863 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:126679 4.1 4:86205e4c:::benchmark_data_smithi187_76542_object34343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:15.467394+0000", 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 344.38796796000003, 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44570083700000002, 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126679 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.864 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467394+0000", 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467394+0000", 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467395+0000", 2024-09-02T10:18:59.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1459999999999999e-06 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467400+0000", 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.451e-06 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467413+0000", 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2799000000000001e-05 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467421+0000", 2024-09-02T10:18:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4830000000000002e-06 2024-09-02T10:18:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467468+0000", 2024-09-02T10:18:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7494999999999998e-05 2024-09-02T10:18:59.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467552+0000", 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3823e-05 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467631+0000", 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8720000000000005e-05 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:59.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.469349+0000", 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017183739999999999 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.470063+0000", 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000713731 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.912992+0000", 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44292904900000002 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:59.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.913049+0000", 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7197999999999997e-05 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.913095+0000", 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5568000000000001e-05 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "rep_scrubmap(1.4 e124 from shard 8)", 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:03.322332+0000", 2024-09-02T10:18:59.870 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 296.53302917299999, 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96974535299999998, 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322332+0000", 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322332+0000", 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322333+0000", 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9000000000000002e-07 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322336+0000", 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.785e-06 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.291988+0000", 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96965220600000002 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.291994+0000", 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3520000000000003e-06 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.292044+0000", 2024-09-02T10:18:59.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9968e-05 2024-09-02T10:18:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.292054+0000", 2024-09-02T10:18:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0244000000000001e-05 2024-09-02T10:18:59.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.292078+0000", 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3408e-05 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62748 5.8 5.b553c768 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.875 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.205545+0000", 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.64981637899999, 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99588105199999999, 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62748 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205545+0000", 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205545+0000", 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205546+0000", 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3200000000000003e-07 2024-09-02T10:18:59.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205565+0000", 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8519e-05 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205574+0000", 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.448e-06 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205579+0000", 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0980000000000001e-06 2024-09-02T10:18:59.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205627+0000", 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7806000000000003e-05 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205634+0000", 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5860000000000001e-06 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201410+0000", 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99577610400000005 2024-09-02T10:18:59.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201426+0000", 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6558999999999998e-05 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62759 5.8 5.f9616988 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.288073+0000", 2024-09-02T10:18:59.880 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.56728907300001, 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91353117399999995, 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62759 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288073+0000", 2024-09-02T10:18:59.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288073+0000", 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288074+0000", 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5060000000000001e-06 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288099+0000", 2024-09-02T10:18:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4862999999999999e-05 2024-09-02T10:18:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288113+0000", 2024-09-02T10:18:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4521e-05 2024-09-02T10:18:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288121+0000", 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8150000000000006e-06 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288184+0000", 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2637000000000004e-05 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288192+0000", 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5189999999999992e-06 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201570+0000", 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91337743400000004 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201604+0000", 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3878999999999999e-05 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62760 5.8 5.be230818 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.885 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.288254+0000", 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.56710762200001, 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91340091000000001, 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62760 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288254+0000", 2024-09-02T10:18:59.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288254+0000", 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288255+0000", 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1200000000000002e-07 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288277+0000", 2024-09-02T10:18:59.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2768e-05 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288292+0000", 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4171999999999999e-05 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288298+0000", 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5950000000000004e-06 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288350+0000", 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1588999999999997e-05 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288356+0000", 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4450000000000001e-06 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201646+0000", 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91328941100000005 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201655+0000", 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2180000000000005e-06 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62763 5.8 5.1d470d68 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.289624+0000", 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.565737751, 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91237609600000003, 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.890 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62763 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289624+0000", 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289624+0000", 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289626+0000", 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6500000000000001e-06 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289649+0000", 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3442e-05 2024-09-02T10:18:59.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289664+0000", 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.526e-05 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289672+0000", 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7170000000000008e-06 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289731+0000", 2024-09-02T10:18:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8931999999999998e-05 2024-09-02T10:18:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289738+0000", 2024-09-02T10:18:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1069999999999999e-06 2024-09-02T10:18:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201987+0000", 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91224905899999997 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202000+0000", 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2928999999999999e-05 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.896 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62767 5.8 5.9781c208 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.295609+0000", 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.55975276699999, 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90642393600000004, 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62767 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295609+0000", 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295609+0000", 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295610+0000", 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.621e-06 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295647+0000", 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6955000000000001e-05 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295665+0000", 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7795000000000002e-05 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295674+0000", 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2210000000000003e-06 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295743+0000", 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8320999999999999e-05 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295755+0000", 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2145e-05 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202026+0000", 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90627074100000005 2024-09-02T10:18:59.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202033+0000", 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1369999999999996e-06 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62787 5.8 5.93d7ce28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.316402+0000", 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.53896, 2024-09-02T10:18:59.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88566921200000004, 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62787 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316402+0000", 2024-09-02T10:18:59.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316402+0000", 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316404+0000", 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1950000000000002e-06 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316427+0000", 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3611000000000001e-05 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316443+0000", 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5903999999999999e-05 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316451+0000", 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4569999999999999e-06 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316511+0000", 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0031999999999997e-05 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316520+0000", 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3470000000000005e-06 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202063+0000", 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88554330000000003 2024-09-02T10:18:59.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202071+0000", 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3660000000000003e-06 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62828 5.8 5.ced072f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.387938+0000", 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.467423822, 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81416300900000005, 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.906 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62828 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387938+0000", 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387938+0000", 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387939+0000", 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.22e-06 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387960+0000", 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0571e-05 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387971+0000", 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1659e-05 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387977+0000", 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2099999999999998e-06 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.388039+0000", 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1771000000000002e-05 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.388048+0000", 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9600000000000006e-06 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202094+0000", 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81404575899999998 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202101+0000", 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8589999999999999e-06 2024-09-02T10:18:59.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62833 5.8 5.47f28d28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.391791+0000", 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.463570455, 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81033902899999999, 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62833 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391791+0000", 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391791+0000", 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391793+0000", 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7129999999999999e-06 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391815+0000", 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2514000000000002e-05 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391830+0000", 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4115e-05 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391837+0000", 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3869999999999997e-06 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391895+0000", 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7762e-05 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391901+0000", 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4910000000000002e-06 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202123+0000", 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81022218700000004 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202130+0000", 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8600000000000004e-06 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.915 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62834 5.8 5.af89fd28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.393180+0000", 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.46218182300001, 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.80897919100000004, 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62834 2024-09-02T10:18:59.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393180+0000", 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393180+0000", 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393181+0000", 2024-09-02T10:18:59.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3900000000000004e-07 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393200+0000", 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8949000000000001e-05 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393210+0000", 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0550000000000001e-05 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393215+0000", 2024-09-02T10:18:59.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:18:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2789999999999998e-06 2024-09-02T10:18:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393255+0000", 2024-09-02T10:18:59.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9678000000000002e-05 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393260+0000", 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.989e-06 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202152+0000", 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.80889200100000003 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202159+0000", 2024-09-02T10:18:59.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9059999999999996e-06 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62856 5.8 5.39aa96f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.474887+0000", 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.38047496600001, 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72730094999999995, 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.922 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62856 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474887+0000", 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474887+0000", 2024-09-02T10:18:59.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474888+0000", 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2640000000000001e-06 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474909+0000", 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1347000000000001e-05 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474922+0000", 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2448e-05 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474928+0000", 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1519999999999997e-06 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474974+0000", 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6578e-05 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474980+0000", 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8880000000000002e-06 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202181+0000", 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72720049399999998 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202188+0000", 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7789999999999998e-06 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.926 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62861 5.8 5.909d6bb8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.497010+0000", 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.358351441, 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.70520625100000001, 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.927 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62861 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497010+0000", 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497010+0000", 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497012+0000", 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.012e-06 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497039+0000", 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6999e-05 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497060+0000", 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0590000000000001e-05 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497067+0000", 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9199999999999998e-06 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497118+0000", 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1029999999999998e-05 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497125+0000", 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8210000000000004e-06 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202209+0000", 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.70508442100000002 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202216+0000", 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4580000000000004e-06 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62865 5.8 5.cf598d28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.931 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.545057+0000", 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.31030481400001, 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65719031900000002, 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62865 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545057+0000", 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545057+0000", 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545058+0000", 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4950000000000001e-06 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545081+0000", 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2793999999999999e-05 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545095+0000", 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3619e-05 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545101+0000", 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9710000000000001e-06 2024-09-02T10:18:59.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545150+0000", 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8814999999999999e-05 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545157+0000", 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9870000000000002e-06 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202240+0000", 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65708368900000003 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202247+0000", 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9489999999999999e-06 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62879 5.8 5.80da3848 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.568069+0000", 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.28729313599999, 2024-09-02T10:18:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63420845199999998, 2024-09-02T10:18:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.937 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62879 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568069+0000", 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568069+0000", 2024-09-02T10:18:59.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568070+0000", 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.719e-06 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568097+0000", 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6455e-05 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568112+0000", 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4916000000000001e-05 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568121+0000", 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4690000000000003e-06 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568184+0000", 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2724000000000005e-05 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568193+0000", 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6840000000000008e-06 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202270+0000", 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63407665199999996 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202277+0000", 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8329999999999996e-06 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.941 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62911 5.8 5.47f6e298 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.672293+0000", 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 205.183068225, 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.53001293400000005, 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.942 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62911 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672293+0000", 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672293+0000", 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672295+0000", 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.626e-06 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672316+0000", 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1151000000000001e-05 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672328+0000", 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2269000000000001e-05 2024-09-02T10:18:59.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672335+0000", 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6649999999999997e-06 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672382+0000", 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6575e-05 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672389+0000", 2024-09-02T10:18:59.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8809999999999998e-06 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202300+0000", 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.52991091899999998 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202306+0000", 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8480000000000003e-06 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62929 5.d 5:b28584c9:::benchmark_data_smithi187_89405_object62928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:35.215954+0000", 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 204.63940778700001, 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.45650502599999998, 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62929 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.947 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215954+0000", 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215954+0000", 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215955+0000", 2024-09-02T10:18:59.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3540000000000001e-06 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215975+0000", 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9556999999999999e-05 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215988+0000", 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3335e-05 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215994+0000", 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2729999999999999e-06 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.216041+0000", 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7110999999999997e-05 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.216134+0000", 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3008000000000001e-05 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.216218+0000", 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3017999999999997e-05 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.217972+0000", 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001754979 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.218530+0000", 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00055731300000000004 2024-09-02T10:18:59.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.672383+0000", 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.45385332499999997 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.672426+0000", 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2540999999999998e-05 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.672459+0000", 2024-09-02T10:18:59.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3212e-05 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:126219 5.d 5:b74c10c7:::benchmark_data_smithi187_89405_object22614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.906310+0000", 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 137.94905207100001, 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.330515067, 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:18:59.953 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126219 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906310+0000", 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906310+0000", 2024-09-02T10:18:59.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906311+0000", 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7199999999999997e-07 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906315+0000", 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0439999999999997e-06 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906325+0000", 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0905e-05 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906331+0000", 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6710000000000004e-06 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906372+0000", 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1084999999999997e-05 2024-09-02T10:18:59.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906446+0000", 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3543999999999999e-05 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906507+0000", 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1243999999999998e-05 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:18:59.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.909064+0000", 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025565589999999999 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.988945+0000", 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.079880947999999993 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236544+0000", 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24759898499999999 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236681+0000", 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013784499999999999 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236825+0000", 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000143265 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:126219 5.d e490/431 ondisk, result = 0)", 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.910662+0000", 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 137.94469995099999, 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32617011499999998, 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.910662+0000", 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.910662+0000", 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.910662+0000", 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7700000000000004e-07 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:18:59.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.910666+0000", 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.332e-06 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236345+0000", 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32567926400000002 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236357+0000", 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2213e-05 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236518+0000", 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016120800000000001 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236534+0000", 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5622999999999999e-05 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236832+0000", 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000297898 2024-09-02T10:18:59.963 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.964 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.964 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.964 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:18:59.964 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:18:59.964 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:18:59.965 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 19 15 27474 27459 90.3185 75.6875 0.00363918 0.0110663 2024-09-02T10:18:59.990 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:18:59.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:18:59.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:00.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:00.117 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:00.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:00.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:00.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:00.245 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:00.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:00.346 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:00.370 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:00.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:00.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:00.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:00.500 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:00.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:00.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:00.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:00.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:00.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:00.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:00.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:19:00.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:00 smithi151 ceph-mon[16639]: pgmap v1266: 47 pgs: 47 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-02T10:19:00.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:00 smithi187 ceph-mon[20072]: pgmap v1266: 47 pgs: 47 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-02T10:19:00.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:00.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:00.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:00 smithi162 ceph-mon[20279]: pgmap v1266: 47 pgs: 47 active+clean; 1.5 GiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-02T10:19:00.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:00.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:00.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:00.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:19:00.948083+0000 min lat: 0.00104401 max lat: 0.392567 avg lat: 0.010927 2024-09-02T10:19:00.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:19:00.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 20 16 29290 29274 91.4741 113.438 0.00410743 0.010927 2024-09-02T10:19:01.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:01.027 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:01.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:01.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:01.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:01.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:01.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:01.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:01.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:01.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-09-02T10:19:01.282 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:01.282 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:01.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:01.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:01.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:01.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:01.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:01.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:01.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:01.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:01.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:01.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:01.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:01 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:19:01.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:01 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:19:01.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:01 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:19:01.599 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:01 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:19:01.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:01 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:19:01.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:01 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:19:01.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:01.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:01.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:01 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:19:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:01 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:19:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:01 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:19:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:01 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:19:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:01 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:19:01.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:01 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:19:01.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:01.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:01.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:01.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:01 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:19:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:01 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:19:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:01 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:19:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:01 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:19:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:01 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:19:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:01 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:19:01.885 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:01.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:01.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:01.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:01.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:01.936 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:01.947 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:19:01.948 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:19:01.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 21 16 30856 30840 91.7784 97.875 0.00392757 0.0108891 2024-09-02T10:19:01.977 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:19:01.978 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:19:02.004 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:19:02.004 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:19:02.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:02.028 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:19:02.028 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:19:02.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:02.042 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:02.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:19:02.058 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:19:02.085 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:19:02.085 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:19:02.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:19:02.116 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:19:02.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:02.147 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:19:02.147 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:19:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:02.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:02.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:02.175 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:19:02.175 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:19:02.206 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:19:02.206 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:19:02.235 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:19:02.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:19:02.266 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:19:02.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:19:02.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:02.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:02.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:02.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:02.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:02.418 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:02.419 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:02.520 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:02.534 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:19:02.534 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:19:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:02.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:02.549 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:02.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:02.678 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:02.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:02 smithi187 ceph-mon[20072]: pgmap v1267: 47 pgs: 47 active+clean; 1.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-02T10:19:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:02 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:19:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:02 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:19:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/571210782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:02.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:02.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:02.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:02.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:19:02.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:02 smithi151 ceph-mon[16639]: pgmap v1267: 47 pgs: 47 active+clean; 1.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-02T10:19:02.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:02 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:19:02.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:02 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:19:02.844 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/571210782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:02.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:02 smithi162 ceph-mon[20279]: pgmap v1267: 47 pgs: 47 active+clean; 1.6 GiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s 2024-09-02T10:19:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:02 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:19:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:02 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:19:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/571210782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:02.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:02.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:02.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:02.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 22 16 32533 32517 92.3705 104.812 0.0111361 0.0108188 2024-09-02T10:19:03.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:03.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:03.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:03.155 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:03.178 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:03.180 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:03.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:03.307 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:03.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:03.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:03.409 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:03.433 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:03.434 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:03.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:03.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:03.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:03.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:03 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:19:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:03 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:19:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:03 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:19:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:03 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:19:03.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:03.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:03 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:19:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:03 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:19:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:03 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:19:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:03 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:19:03.794 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:03.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:03.819 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:03 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:19:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:03 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:19:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:03 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:19:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:03 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:19:03.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:03.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:03.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:03.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 23 16 33983 33967 92.2943 90.625 0.00711409 0.0108295 2024-09-02T10:19:04.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:04.051 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-09-02T10:19:04.075 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:04.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:04.177 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:04.199 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:04.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:04.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:04.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:04.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:04.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:04.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:04.447 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:04.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:04.570 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:04.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:04.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:04 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:19:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:04 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:19:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:04 smithi151 ceph-mon[16639]: pgmap v1268: 47 pgs: 47 active+clean; 1.9 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-02T10:19:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:04 smithi151 ceph-mon[16639]: osdmap e547: 12 total, 12 up, 11 in 2024-09-02T10:19:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:04 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/893147138' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:19:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:04 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/893147138' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:19:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:04 smithi151 ceph-mon[16639]: osdmap e548: 12 total, 12 up, 11 in 2024-09-02T10:19:04.693 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:04.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:04 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:19:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:04 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:19:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:04 smithi187 ceph-mon[20072]: pgmap v1268: 47 pgs: 47 active+clean; 1.9 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-02T10:19:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:04 smithi187 ceph-mon[20072]: osdmap e547: 12 total, 12 up, 11 in 2024-09-02T10:19:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:04 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/893147138' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:19:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:04 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/893147138' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:19:04.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:04 smithi187 ceph-mon[20072]: osdmap e548: 12 total, 12 up, 11 in 2024-09-02T10:19:04.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:04.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:04.818 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:04.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:04 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:19:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:04 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:19:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:04 smithi162 ceph-mon[20279]: pgmap v1268: 47 pgs: 47 active+clean; 1.9 GiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 99 MiB/s wr, 1.58k op/s 2024-09-02T10:19:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:04 smithi162 ceph-mon[20279]: osdmap e547: 12 total, 12 up, 11 in 2024-09-02T10:19:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:04 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/893147138' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:19:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:04 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/893147138' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:19:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:04 smithi162 ceph-mon[20279]: osdmap e548: 12 total, 12 up, 11 in 2024-09-02T10:19:04.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:19:04.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:04.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:04.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:04.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 24 16 34182 34166 88.9669 12.4375 0.236105 0.0109514 2024-09-02T10:19:05.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:05.068 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:05.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:05.170 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:05.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:05.192 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:05.192 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:05.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:05.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:05.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:05.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:05.437 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:05.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:05.539 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:05.562 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:05.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:05.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:05 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:19:05.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:05 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:19:05.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:05 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:19:05.584 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:05 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:19:05.585 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:05 smithi151 ceph-mon[16639]: osdmap e549: 12 total, 12 up, 11 in 2024-09-02T10:19:05.613 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-02T10:19:05.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:19:05.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:05.752 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:05.753 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:05 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:19:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:05 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:19:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:05 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:19:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:05 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:19:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:05 smithi187 ceph-mon[20072]: osdmap e549: 12 total, 12 up, 11 in 2024-09-02T10:19:05.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:05.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:05 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:19:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:05 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:19:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:05 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:19:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:05 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:19:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:05 smithi162 ceph-mon[20279]: osdmap e549: 12 total, 12 up, 11 in 2024-09-02T10:19:05.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:05.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:05.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:05.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:05.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 25 16 34191 34175 85.4307 0.5625 0.00223635 0.0109491 2024-09-02T10:19:05.999 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:06.024 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:06.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:06.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:06.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:06.149 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:06.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:06.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:06.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:06.274 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:06.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:06.376 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:06.399 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:06.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:06.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:06.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:19:06.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:06.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:06.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:06.529 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:06 smithi151 ceph-mon[16639]: pgmap v1272: 47 pgs: 2 unknown, 2 peering, 43 active+clean; 1.7 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 39 MiB/s wr, 626 op/s 2024-09-02T10:19:06.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:06 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:19:06.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:06 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:19:06.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:06 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:19:06.530 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:06 smithi151 ceph-mon[16639]: osdmap e550: 12 total, 12 up, 11 in 2024-09-02T10:19:06.536 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:19:06.536 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:19:06.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:06.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:06.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:06.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:06.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:06 smithi187 ceph-mon[20072]: pgmap v1272: 47 pgs: 2 unknown, 2 peering, 43 active+clean; 1.7 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 39 MiB/s wr, 626 op/s 2024-09-02T10:19:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:06 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:19:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:06 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:19:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:06 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:19:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:06 smithi187 ceph-mon[20072]: osdmap e550: 12 total, 12 up, 11 in 2024-09-02T10:19:06.776 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:06.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:06.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:06 smithi162 ceph-mon[20279]: pgmap v1272: 47 pgs: 2 unknown, 2 peering, 43 active+clean; 1.7 GiB data, 9.9 GiB used, 973 GiB / 983 GiB avail; 39 MiB/s wr, 626 op/s 2024-09-02T10:19:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:06 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:19:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:06 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:19:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:06 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:19:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:06 smithi162 ceph-mon[20279]: osdmap e550: 12 total, 12 up, 11 in 2024-09-02T10:19:06.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:06.912 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:06.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:06.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 26 16 34311 34295 82.4333 7.5 0.0043645 0.0121255 2024-09-02T10:19:07.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:07.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:07.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:07.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:07.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:07.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:07.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:07.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:07.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:07.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:19:07.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:07.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:07.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:07.427 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:07.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:07.428 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:07.447 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:07.470 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:07.472 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:19:07.495 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/285367653' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:07.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:07.557 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:07.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:07.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:07.681 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:07.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:07.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/285367653' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:07.784 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:07.807 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:07.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:07.808 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/285367653' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:07.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:07.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:07.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:07.948 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 27 16 35606 35590 82.3777 80.9375 0.00255025 0.0121328 2024-09-02T10:19:08.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:08.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:08.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:08.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:08.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:08.197 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:08.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:08.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:08.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:08.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:08.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:08.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:08.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:08.461 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:08.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:08.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:08.590 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:08.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:08.692 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:08.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:08.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:08.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:08 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:19:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:08 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:19:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:08 smithi187 ceph-mon[20072]: pgmap v1274: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 unknown, 2 peering, 42 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 2118/88557 objects degraded (2.392%) 2024-09-02T10:19:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:08 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 2118/88557 objects degraded (2.392%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:19:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:08 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:19:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/285367653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:19:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:08 smithi187 ceph-mon[20072]: osdmap e551: 12 total, 12 up, 11 in 2024-09-02T10:19:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:08 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:19:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:08 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:19:08.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:08 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:19:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:08 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:19:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:08 smithi162 ceph-mon[20279]: pgmap v1274: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 unknown, 2 peering, 42 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 2118/88557 objects degraded (2.392%) 2024-09-02T10:19:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:08 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 2118/88557 objects degraded (2.392%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:19:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:08 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:19:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/285367653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:19:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:08 smithi162 ceph-mon[20279]: osdmap e551: 12 total, 12 up, 11 in 2024-09-02T10:19:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:08 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:19:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:08 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:19:08.846 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:08.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:08.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:08 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:19:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:08 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:19:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:08 smithi151 ceph-mon[16639]: pgmap v1274: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 unknown, 2 peering, 42 active+clean; 1.8 GiB data, 10 GiB used, 973 GiB / 983 GiB avail; 2118/88557 objects degraded (2.392%) 2024-09-02T10:19:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:08 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 2118/88557 objects degraded (2.392%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:19:08.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:08 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:19:08.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/285367653' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:19:08.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:08 smithi151 ceph-mon[16639]: osdmap e551: 12 total, 12 up, 11 in 2024-09-02T10:19:08.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:08 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:19:08.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:08 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:19:08.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:08.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 28 16 36784 36768 82.0649 73.625 0.185579 0.0121343 2024-09-02T10:19:08.979 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:08.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:08.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:09.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:09.111 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:09.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:09.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:09.213 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:09.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:09.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:09.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:09.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:09.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:09.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:09.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:09.504 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:09.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:09.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:09.627 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:09.628 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:09.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:09 smithi187 ceph-mon[20072]: osdmap e552: 12 total, 12 up, 11 in 2024-09-02T10:19:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:09 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:19:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:09 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:19:09.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:09.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:09.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:09 smithi151 ceph-mon[16639]: osdmap e552: 12 total, 12 up, 11 in 2024-09-02T10:19:09.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:09 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:19:09.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:09 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:19:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:09 smithi162 ceph-mon[20279]: osdmap e552: 12 total, 12 up, 11 in 2024-09-02T10:19:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:09 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:19:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:09 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:19:09.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:09.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:09.885 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:09.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 29 16 38120 38104 82.1141 83.5 0.011021 0.0121719 2024-09-02T10:19:09.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:10.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:10.011 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:10.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:10.142 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:10.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:10.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:10.277 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:10.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:10.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:10.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:10.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:10.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:10.534 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:10.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:10.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:10.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:10 smithi151 ceph-mon[16639]: pgmap v1277: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s; 2119/105630 objects degraded (2.006%); 6116/105630 objects misplaced (5.790%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:19:10.556 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:10 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:19:10.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:10.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:10.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:10 smithi187 ceph-mon[20072]: pgmap v1277: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s; 2119/105630 objects degraded (2.006%); 6116/105630 objects misplaced (5.790%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:19:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:10 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:19:10.768 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:10.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:10.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:10 smithi162 ceph-mon[20279]: pgmap v1277: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 109 MiB/s wr, 1.74k op/s; 2119/105630 objects degraded (2.006%); 6116/105630 objects misplaced (5.790%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:19:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:10 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:19:10.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:10.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:10.915 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:10.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 30 16 39399 39383 82.0414 79.9375 0.066695 0.012177 2024-09-02T10:19:11.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:11.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:11.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:11.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:11.161 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:11.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:11.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:19:11.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:19:11.223 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:19:11.224 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:19:11.224 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:19:11.224 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:19:11.224 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:19:11.224 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:19:11.224 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:19:11.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:11.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:11.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:11.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:11.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:11.419 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:11.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:11.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:11.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:11.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:11.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:11.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:11.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:11.674 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:11.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:11 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:19:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:11 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:19:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:11 smithi187 ceph-mon[20072]: osdmap e553: 12 total, 12 up, 11 in 2024-09-02T10:19:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3940538873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:11.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:11.803 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:11.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:11 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:19:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:11 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:19:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:11 smithi162 ceph-mon[20279]: osdmap e553: 12 total, 12 up, 11 in 2024-09-02T10:19:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3940538873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:11.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:11.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:11.928 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:11.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:11 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:19:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:11 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:19:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:11 smithi151 ceph-mon[16639]: osdmap e553: 12 total, 12 up, 11 in 2024-09-02T10:19:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3940538873' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:11.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 31 16 40313 40297 81.2375 57.125 0.00940884 0.0123034 2024-09-02T10:19:12.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:12.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:12.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:12.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:12.183 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:12.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:12.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:12.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:19:12.312 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:12.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:12.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:12.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:12.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:12.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:12.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:12.578 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:12.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:12.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:12.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:12 smithi151 ceph-mon[16639]: pgmap v1279: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 43 active+clean; 2.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s; 2119/109377 objects degraded (1.937%); 6116/109377 objects misplaced (5.592%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:19:12.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:12.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:12 smithi187 ceph-mon[20072]: pgmap v1279: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 43 active+clean; 2.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s; 2119/109377 objects degraded (1.937%); 6116/109377 objects misplaced (5.592%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:19:12.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:12.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:12.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:12 smithi162 ceph-mon[20279]: pgmap v1279: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 43 active+clean; 2.2 GiB data, 11 GiB used, 973 GiB / 983 GiB avail; 103 MiB/s wr, 1.65k op/s; 2119/109377 objects degraded (1.937%); 6116/109377 objects misplaced (5.592%); 3.6 MiB/s, 57 objects/s recovering 2024-09-02T10:19:12.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:12.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 32 16 41593 41577 81.1986 80 0.0307041 0.0123079 2024-09-02T10:19:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:12.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:13.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:13.089 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:13.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:13.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:13.216 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:13.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:13.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:13.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:13.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:13.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:13.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:13.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:13.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:13.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:13.587 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:13.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:13.689 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:13.711 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:13.712 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:13.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:13 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2119/120384 objects degraded (1.760%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:13.845 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:13.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:13.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:13 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2119/120384 objects degraded (1.760%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:13.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:13.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 33 16 42600 42584 80.6451 62.9375 0.0642582 0.0123735 2024-09-02T10:19:13.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:13.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:13 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2119/120384 objects degraded (1.760%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:14.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:14.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:14.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:14.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:14.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:14.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:14.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:14.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:14.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:14.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:14.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:14.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:14.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:14.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:14.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:14.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:14.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:14.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:14.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:14.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:14.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:14.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:14 smithi162 ceph-mon[20279]: pgmap v1280: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 110 MiB/s wr, 1.77k op/s; 2119/120384 objects degraded (1.760%); 5497/120384 objects misplaced (4.566%); 9.6 MiB/s, 153 objects/s recovering 2024-09-02T10:19:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:14 smithi162 ceph-mon[20279]: osdmap e554: 12 total, 12 up, 11 in 2024-09-02T10:19:14.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:14.873 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:14.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:14 smithi151 ceph-mon[16639]: pgmap v1280: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 110 MiB/s wr, 1.77k op/s; 2119/120384 objects degraded (1.760%); 5497/120384 objects misplaced (4.566%); 9.6 MiB/s, 153 objects/s recovering 2024-09-02T10:19:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:14 smithi151 ceph-mon[16639]: osdmap e554: 12 total, 12 up, 11 in 2024-09-02T10:19:14.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 34 16 43549 43533 80.0175 59.3125 0.00490499 0.0124858 2024-09-02T10:19:14.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:14.997 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:14.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:14 smithi187 ceph-mon[20072]: pgmap v1280: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 110 MiB/s wr, 1.77k op/s; 2119/120384 objects degraded (1.760%); 5497/120384 objects misplaced (4.566%); 9.6 MiB/s, 153 objects/s recovering 2024-09-02T10:19:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:14 smithi187 ceph-mon[20072]: osdmap e554: 12 total, 12 up, 11 in 2024-09-02T10:19:15.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:15.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:15.123 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:15.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:15.253 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:15.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:15.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:15.385 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:15.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:15.487 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:15.511 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:15.512 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:15.613 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:15.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:15.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:15.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:15.768 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:15.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:15.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:15.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:15.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:15.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:15.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 35 16 44746 44730 79.8686 74.8125 0.0216089 0.012515 2024-09-02T10:19:16.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:16.033 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:16.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:16.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:16.166 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:16.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:16.221 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:19:16.222 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:19:16.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:16.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:16.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:16.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:16.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:16.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:16.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:16.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:16.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:16.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:16.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:16.679 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:16.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:16.780 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:16.811 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:16.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:16.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:16 smithi162 ceph-mon[20279]: pgmap v1282: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 2119/123564 objects degraded (1.715%); 5139/123564 objects misplaced (4.159%); 10 MiB/s, 160 objects/s recovering 2024-09-02T10:19:16.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:16.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:16.935 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:16 smithi151 ceph-mon[16639]: pgmap v1282: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 2119/123564 objects degraded (1.715%); 5139/123564 objects misplaced (4.159%); 10 MiB/s, 160 objects/s recovering 2024-09-02T10:19:16.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 36 16 45790 45774 79.4624 65.25 0.0138353 0.012566 2024-09-02T10:19:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:16 smithi187 ceph-mon[20072]: pgmap v1282: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.5 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 2119/123564 objects degraded (1.715%); 5139/123564 objects misplaced (4.159%); 10 MiB/s, 160 objects/s recovering 2024-09-02T10:19:17.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:17.058 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:17.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:17.060 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:17.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:17.185 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:17.186 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:17.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:17.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:17.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:17.419 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:17.449 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:17.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:17.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:17.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:17.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:17.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:17.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:17.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:17.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:17.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:17.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:17.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:17.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:17 smithi162 ceph-mon[20279]: from='client.17151 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:17.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:17 smithi151 ceph-mon[16639]: from='client.17151 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:17.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:17.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 37 16 47013 46997 79.3805 76.4375 0.024152 0.0125823 2024-09-02T10:19:17.962 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:17.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:17 smithi187 ceph-mon[20072]: from='client.17151 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:18.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:18.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:18.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:18.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:18.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:18.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:18.316 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:18.339 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:18.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:18.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:18.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:18.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:18.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:18.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:18.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:18.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:18.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:18.720 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:18.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:18.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:18.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:18.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:18 smithi162 ceph-mon[20279]: pgmap v1283: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 2119/130512 objects degraded (1.624%); 5139/130512 objects misplaced (3.938%); 7.6 MiB/s, 122 objects/s recovering 2024-09-02T10:19:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:18.845 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:18 smithi151 ceph-mon[16639]: pgmap v1283: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 2119/130512 objects degraded (1.624%); 5139/130512 objects misplaced (3.938%); 7.6 MiB/s, 122 objects/s recovering 2024-09-02T10:19:18.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:18.949 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 38 16 48022 48006 78.9509 63.0625 0.0107396 0.0126595 2024-09-02T10:19:18.978 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:18.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:18 smithi187 ceph-mon[20072]: pgmap v1283: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 2119/130512 objects degraded (1.624%); 5139/130512 objects misplaced (3.938%); 7.6 MiB/s, 122 objects/s recovering 2024-09-02T10:19:19.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:19.109 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:19.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:19.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:19.236 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:19.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:19.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:19.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:19.364 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:19.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:19.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:19.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:19.489 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:19.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:19.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:19.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:19.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:19.745 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:19.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2119/130512 objects degraded (1.624%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:19.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:19.871 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2119/130512 objects degraded (1.624%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:19.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:19.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:19.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:19.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 39 16 49366 49350 79.0802 84 0.0545127 0.012636 2024-09-02T10:19:19.977 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:19.999 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:20.000 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2119/130512 objects degraded (1.624%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:20.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:20.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:20.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:20.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:20.260 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:20.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:20.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:20.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:20.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:20.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:20.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:20.518 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:20.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:20.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:20.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:20.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:20.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:20.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:20.675 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:20.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:20.778 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:20.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:20.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:20 smithi151 ceph-mon[16639]: pgmap v1284: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 77 MiB/s wr, 1.22k op/s; 2119/138771 objects degraded (1.527%); 3892/138771 objects misplaced (2.805%); 16 MiB/s, 251 objects/s recovering 2024-09-02T10:19:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:20 smithi162 ceph-mon[20279]: pgmap v1284: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 77 MiB/s wr, 1.22k op/s; 2119/138771 objects degraded (1.527%); 3892/138771 objects misplaced (2.805%); 16 MiB/s, 251 objects/s recovering 2024-09-02T10:19:20.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:20.908 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:20.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:20.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:19:20.949723+0000 min lat: 0.00104401 max lat: 2.84044 avg lat: 0.0126948 2024-09-02T10:19:20.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:19:20.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 40 16 50397 50381 78.714 64.4375 0.00227588 0.0126948 2024-09-02T10:19:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:20 smithi187 ceph-mon[20072]: pgmap v1284: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.8 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 77 MiB/s wr, 1.22k op/s; 2119/138771 objects degraded (1.527%); 3892/138771 objects misplaced (2.805%); 16 MiB/s, 251 objects/s recovering 2024-09-02T10:19:21.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:21.032 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:21.033 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:21.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:21.165 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:21.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:21.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:21.297 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:21.298 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:21.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:21.344 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1282,"stamp":"2024-09-02T10:19:15.336046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2699756064,"num_objects":41188,"num_object_clones":0,"num_object_copies":123564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":5139,"num_objects_unfound":0,"num_objects_dirty":41188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41186,"num_write_kb":2635904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1283,"num_bytes_recovered":84616256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44038,"ondisk_log_size":44038,"up":141,"acting":137,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":12408668,"kb_used_data":7947932,"kb_used_omap":17,"kb_used_meta":4460526,"kb_avail":1018788004,"statfs":{"total":1055945392128,"available":1043238916096,"internally_reserved":0,"allocated":8138682368,"data_stored":8134555798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":4567579510},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,3,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":554762240,"num_objects":8465,"num_object_clones":0,"num_object_copies":25395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1009,"num_objects_unfound":0,"num_objects_dirty":8465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8465,"num_write_kb":541760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1102,"num_bytes_recovered":72220672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.866068"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":450,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.484648+0000","last_change":"2024-09-02T10:18:43.049145+0000","last_active":"2024-09-02T10:19:10.484648+0000","last_peered":"2024-09-02T10:19:10.484648+0000","last_clean":"2024-09-02T10:19:10.484648+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:19:10.484648+0000","last_undegraded":"2024-09-02T10:19:10.484648+0000","last_fullsized":"2024-09-02T10:19:10.484648+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:43.049077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:43.049077+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:07.603384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":258,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.483179+0000","last_change":"2024-09-02T10:18:19.826521+0000","last_active":"2024-09-02T10:19:10.483179+0000","last_peered":"2024-09-02T10:19:10.483179+0000","last_clean":"2024-09-02T10:19:10.483179+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:19:10.483179+0000","last_undegraded":"2024-09-02T10:19:10.483179+0000","last_fullsized":"2024-09-02T10:19:10.483179+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:19.826468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:19.826468+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:21.451618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":710,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.053365+0000","last_change":"2024-09-02T10:18:16.145420+0000","last_active":"2024-09-02T10:19:14.053365+0000","last_peered":"2024-09-02T10:19:14.053365+0000","last_clean":"2024-09-02T10:19:14.053365+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:19:14.053365+0000","last_undegraded":"2024-09-02T10:19:14.053365+0000","last_fullsized":"2024-09-02T10:19:14.053365+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:16.145346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:16.145346+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:31.421793+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":710,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.053223+0000","last_change":"2024-09-02T10:18:39.045140+0000","last_active":"2024-09-02T10:19:14.053223+0000","last_peered":"2024-09-02T10:19:14.053223+0000","last_clean":"2024-09-02T10:19:14.053223+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:19:14.053223+0000","last_undegraded":"2024-09-02T10:19:14.053223+0000","last_fullsized":"2024-09-02T10:19:14.053223+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:39.045083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:39.045083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:03.744946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":22,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.069415+0000","last_change":"2024-09-02T10:19:05.058923+0000","last_active":"2024-09-02T10:19:14.069415+0000","last_peered":"2024-09-02T10:19:14.069415+0000","last_clean":"2024-09-02T10:19:14.069415+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:19:14.069415+0000","last_undegraded":"2024-09-02T10:19:14.069415+0000","last_fullsized":"2024-09-02T10:19:14.069415+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:01.810089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:01.810089+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:19.405935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":556,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.508511+0000","last_change":"2024-09-02T10:18:17.772780+0000","last_active":"2024-09-02T10:19:10.508511+0000","last_peered":"2024-09-02T10:19:10.508511+0000","last_clean":"2024-09-02T10:19:10.508511+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:19:10.508511+0000","last_undegraded":"2024-09-02T10:19:10.508511+0000","last_fullsized":"2024-09-02T10:19:10.508511+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:17.772723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:17.772723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:28.508722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1081,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.485448+0000","last_change":"2024-09-02T10:19:03.880037+0000","last_active":"2024-09-02T10:19:10.485448+0000","last_peered":"2024-09-02T10:19:10.485448+0000","last_clean":"2024-09-02T10:19:10.485448+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:19:10.485448+0000","last_undegraded":"2024-09-02T10:19:10.485448+0000","last_fullsized":"2024-09-02T10:19:10.485448+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:03.879968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:03.879968+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.678816+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":725,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.053496+0000","last_change":"2024-09-02T10:18:35.059181+0000","last_active":"2024-09-02T10:19:14.053496+0000","last_peered":"2024-09-02T10:19:14.053496+0000","last_clean":"2024-09-02T10:19:14.053496+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:19:14.053496+0000","last_undegraded":"2024-09-02T10:19:14.053496+0000","last_fullsized":"2024-09-02T10:19:14.053496+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:35.059123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:35.059123+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:59.080089+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":722,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.483664+0000","last_change":"2024-09-02T10:18:10.904183+0000","last_active":"2024-09-02T10:19:10.483664+0000","last_peered":"2024-09-02T10:19:10.483664+0000","last_clean":"2024-09-02T10:19:10.483664+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:19:10.483664+0000","last_undegraded":"2024-09-02T10:19:10.483664+0000","last_fullsized":"2024-09-02T10:19:10.483664+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:10.904126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:10.904126+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:14.821462+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":699,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.485969+0000","last_change":"2024-09-02T10:19:05.910750+0000","last_active":"2024-09-02T10:19:10.485969+0000","last_peered":"2024-09-02T10:19:10.485969+0000","last_clean":"2024-09-02T10:19:10.485969+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:19:10.485969+0000","last_undegraded":"2024-09-02T10:19:10.485969+0000","last_fullsized":"2024-09-02T10:19:10.485969+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:05.910691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:05.910691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:20.353104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":542,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.485783+0000","last_change":"2024-09-02T10:17:44.275503+0000","last_active":"2024-09-02T10:19:10.485783+0000","last_peered":"2024-09-02T10:19:10.485783+0000","last_clean":"2024-09-02T10:19:10.485783+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:19:10.485783+0000","last_undegraded":"2024-09-02T10:19:10.485783+0000","last_fullsized":"2024-09-02T10:19:10.485783+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:17:44.275434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:17:44.275434+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1056,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.482996+0000","last_change":"2024-09-02T10:19:00.052912+0000","last_active":"2024-09-02T10:19:10.482996+0000","last_peered":"2024-09-02T10:19:10.482996+0000","last_clean":"2024-09-02T10:19:10.482996+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:19:10.482996+0000","last_undegraded":"2024-09-02T10:19:10.482996+0000","last_fullsized":"2024-09-02T10:19:10.482996+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.052857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.052857+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:29.321752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":991,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.485835+0000","last_change":"2024-09-02T10:18:41.060635+0000","last_active":"2024-09-02T10:19:10.485835+0000","last_peered":"2024-09-02T10:19:10.485835+0000","last_clean":"2024-09-02T10:19:10.485835+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:19:10.485835+0000","last_undegraded":"2024-09-02T10:19:10.485835+0000","last_fullsized":"2024-09-02T10:19:10.485835+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:41.060566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:41.060566+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:47.963172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":970,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.495158+0000","last_change":"2024-09-02T10:18:16.400250+0000","last_active":"2024-09-02T10:19:10.495158+0000","last_peered":"2024-09-02T10:19:10.495158+0000","last_clean":"2024-09-02T10:19:10.495158+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:19:10.495158+0000","last_undegraded":"2024-09-02T10:19:10.495158+0000","last_fullsized":"2024-09-02T10:19:10.495158+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:16.400175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:16.400175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:25.500090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":961,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.485511+0000","last_change":"2024-09-02T10:18:40.069068+0000","last_active":"2024-09-02T10:19:10.485511+0000","last_peered":"2024-09-02T10:19:10.485511+0000","last_clean":"2024-09-02T10:19:10.485511+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:19:10.485511+0000","last_undegraded":"2024-09-02T10:19:10.485511+0000","last_fullsized":"2024-09-02T10:19:10.485511+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:40.069013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:40.069013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:45.996592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"553'2529","reported_seq":2577,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:11.741687+0000","last_change":"2024-09-02T10:18:37.599555+0000","last_active":"2024-09-02T10:19:11.741687+0000","last_peered":"2024-09-02T10:19:11.741687+0000","last_clean":"2024-09-02T10:19:11.741687+0000","last_became_active":"2024-09-02T10:18:37.599064+0000","last_became_peered":"2024-09-02T10:18:37.599064+0000","last_unstale":"2024-09-02T10:19:11.741687+0000","last_undegraded":"2024-09-02T10:19:11.741687+0000","last_fullsized":"2024-09-02T10:19:11.741687+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2529,"log_dups_size":0,"ondisk_log_size":2529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:50.285721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165740544,"num_objects":2529,"num_object_clones":0,"num_object_copies":7587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2529,"num_write_kb":161856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1048,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.485590+0000","last_change":"2024-09-02T10:18:13.075935+0000","last_active":"2024-09-02T10:19:10.485590+0000","last_peered":"2024-09-02T10:19:10.485590+0000","last_clean":"2024-09-02T10:19:10.485590+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:19:10.485590+0000","last_undegraded":"2024-09-02T10:19:10.485590+0000","last_fullsized":"2024-09-02T10:19:10.485590+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:13.075888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:13.075888+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:39.573123+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"553'2666","reported_seq":3684,"reported_epoch":553,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:19:13.671169+0000","last_change":"2024-09-02T10:19:06.888078+0000","last_active":"2024-09-02T10:19:13.671169+0000","last_peered":"2024-09-02T10:19:13.671169+0000","last_clean":"2024-09-02T10:19:04.053778+0000","last_became_active":"2024-09-02T10:19:06.870452+0000","last_became_peered":"2024-09-02T10:19:06.870452+0000","last_unstale":"2024-09-02T10:19:13.671169+0000","last_undegraded":"2024-09-02T10:19:13.671169+0000","last_fullsized":"2024-09-02T10:19:06.061691+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2666,"log_dups_size":0,"ondisk_log_size":2666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174718976,"num_objects":2666,"num_object_clones":0,"num_object_copies":7998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1628,"num_objects_unfound":0,"num_objects_dirty":2666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2666,"num_write_kb":170624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":483,"num_bytes_recovered":31653888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[1,5],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5","objects":1038},{"shards":"1,5,7","objects":1628}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1163,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.485570+0000","last_change":"2024-09-02T10:17:50.300169+0000","last_active":"2024-09-02T10:19:10.485570+0000","last_peered":"2024-09-02T10:19:10.485570+0000","last_clean":"2024-09-02T10:19:10.485570+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:19:10.485570+0000","last_undegraded":"2024-09-02T10:19:10.485570+0000","last_fullsized":"2024-09-02T10:19:10.485570+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:17:50.300100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:17:50.300100+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"553'2608","reported_seq":2658,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:12.150060+0000","last_change":"2024-09-02T10:18:37.062429+0000","last_active":"2024-09-02T10:19:12.150060+0000","last_peered":"2024-09-02T10:19:12.150060+0000","last_clean":"2024-09-02T10:19:12.150060+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:19:12.150060+0000","last_undegraded":"2024-09-02T10:19:12.150060+0000","last_fullsized":"2024-09-02T10:19:12.150060+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2608,"log_dups_size":0,"ondisk_log_size":2608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:01.072199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170917888,"num_objects":2608,"num_object_clones":0,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2608,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2608,"num_write_kb":166912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":820,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.496787+0000","last_change":"2024-09-02T10:19:05.058003+0000","last_active":"2024-09-02T10:19:10.496787+0000","last_peered":"2024-09-02T10:19:10.496787+0000","last_clean":"2024-09-02T10:19:10.496787+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:19:10.496787+0000","last_undegraded":"2024-09-02T10:19:10.496787+0000","last_fullsized":"2024-09-02T10:19:10.496787+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.206480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.206480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.672102+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"553'2652","reported_seq":2701,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:13.183331+0000","last_change":"2024-09-02T10:18:37.848522+0000","last_active":"2024-09-02T10:19:13.183331+0000","last_peered":"2024-09-02T10:19:13.183331+0000","last_clean":"2024-09-02T10:19:13.183331+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:19:13.183331+0000","last_undegraded":"2024-09-02T10:19:13.183331+0000","last_fullsized":"2024-09-02T10:19:13.183331+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2652,"log_dups_size":0,"ondisk_log_size":2652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:56.425124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173801472,"num_objects":2652,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2652,"num_write_kb":169728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":209,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.484516+0000","last_change":"2024-09-02T10:18:15.017709+0000","last_active":"2024-09-02T10:19:10.484516+0000","last_peered":"2024-09-02T10:19:10.484516+0000","last_clean":"2024-09-02T10:19:10.484516+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:19:10.484516+0000","last_undegraded":"2024-09-02T10:19:10.484516+0000","last_fullsized":"2024-09-02T10:19:10.484516+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:15.017640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:15.017640+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:27.366269+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"553'2506","reported_seq":2556,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:11.202513+0000","last_change":"2024-09-02T10:18:37.850883+0000","last_active":"2024-09-02T10:19:11.202513+0000","last_peered":"2024-09-02T10:19:11.202513+0000","last_clean":"2024-09-02T10:19:11.202513+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:19:11.202513+0000","last_undegraded":"2024-09-02T10:19:11.202513+0000","last_fullsized":"2024-09-02T10:19:11.202513+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2506,"log_dups_size":0,"ondisk_log_size":2506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:05.087393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164233216,"num_objects":2506,"num_object_clones":0,"num_object_copies":7518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2506,"num_write_kb":160384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1118,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.484596+0000","last_change":"2024-09-02T10:18:44.023062+0000","last_active":"2024-09-02T10:19:10.484596+0000","last_peered":"2024-09-02T10:19:10.484596+0000","last_clean":"2024-09-02T10:19:10.484596+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:19:10.484596+0000","last_undegraded":"2024-09-02T10:19:10.484596+0000","last_fullsized":"2024-09-02T10:19:10.484596+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:44.023008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:44.023008+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:49.645929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"553'2604","reported_seq":3494,"reported_epoch":553,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:19:13.148130+0000","last_change":"2024-09-02T10:19:06.924994+0000","last_active":"2024-09-02T10:19:13.148130+0000","last_peered":"2024-09-02T10:19:13.148130+0000","last_clean":"2024-09-02T10:19:04.052018+0000","last_became_active":"2024-09-02T10:19:06.875023+0000","last_became_peered":"2024-09-02T10:19:06.875023+0000","last_unstale":"2024-09-02T10:19:13.148130+0000","last_undegraded":"2024-09-02T10:19:13.148130+0000","last_fullsized":"2024-09-02T10:19:06.059430+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2604,"log_dups_size":0,"ondisk_log_size":2604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170655744,"num_objects":2604,"num_object_clones":0,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1716,"num_objects_unfound":0,"num_objects_dirty":2604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2604,"num_write_kb":166656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":419,"num_bytes_recovered":27459584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[3,11],"avail_no_missing":["3","8","11"],"object_location_counts":[{"shards":"3,8,11","objects":1716},{"shards":"3,11","objects":888}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":585,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.508538+0000","last_change":"2024-09-02T10:18:41.761050+0000","last_active":"2024-09-02T10:19:10.508538+0000","last_peered":"2024-09-02T10:19:10.508538+0000","last_clean":"2024-09-02T10:19:10.508538+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:19:10.508538+0000","last_undegraded":"2024-09-02T10:19:10.508538+0000","last_fullsized":"2024-09-02T10:19:10.508538+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:41.760998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:41.760998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:57.499320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"554'2634","reported_seq":2686,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.356141+0000","last_change":"2024-09-02T10:18:37.600687+0000","last_active":"2024-09-02T10:19:14.356141+0000","last_peered":"2024-09-02T10:19:14.356141+0000","last_clean":"2024-09-02T10:19:14.356141+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:19:14.356141+0000","last_undegraded":"2024-09-02T10:19:14.356141+0000","last_fullsized":"2024-09-02T10:19:14.356141+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:59.230189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172621824,"num_objects":2634,"num_object_clones":0,"num_object_copies":7902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2634,"num_write_kb":168576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":557,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.508604+0000","last_change":"2024-09-02T10:18:11.764437+0000","last_active":"2024-09-02T10:19:10.508604+0000","last_peered":"2024-09-02T10:19:10.508604+0000","last_clean":"2024-09-02T10:19:10.508604+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:19:10.508604+0000","last_undegraded":"2024-09-02T10:19:10.508604+0000","last_fullsized":"2024-09-02T10:19:10.508604+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:11.764362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:11.764362+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:19.805086+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"553'2569","reported_seq":2618,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:12.893182+0000","last_change":"2024-09-02T10:18:37.065153+0000","last_active":"2024-09-02T10:19:12.893182+0000","last_peered":"2024-09-02T10:19:12.893182+0000","last_clean":"2024-09-02T10:19:12.893182+0000","last_became_active":"2024-09-02T10:18:37.064844+0000","last_became_peered":"2024-09-02T10:18:37.064844+0000","last_unstale":"2024-09-02T10:19:12.893182+0000","last_undegraded":"2024-09-02T10:19:12.893182+0000","last_fullsized":"2024-09-02T10:19:12.893182+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2569,"log_dups_size":0,"ondisk_log_size":2569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:42.124650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168361984,"num_objects":2569,"num_object_clones":0,"num_object_copies":7707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2569,"num_write_kb":164416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":309,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.482599+0000","last_change":"2024-09-02T10:18:37.859095+0000","last_active":"2024-09-02T10:19:10.482599+0000","last_peered":"2024-09-02T10:19:10.482599+0000","last_clean":"2024-09-02T10:19:10.482599+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:19:10.482599+0000","last_undegraded":"2024-09-02T10:19:10.482599+0000","last_fullsized":"2024-09-02T10:19:10.482599+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:37.859053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:37.859053+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:57.896931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"553'2640","reported_seq":3447,"reported_epoch":553,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:19:13.138184+0000","last_change":"2024-09-02T10:19:06.888347+0000","last_active":"2024-09-02T10:19:13.138184+0000","last_peered":"2024-09-02T10:19:13.138184+0000","last_clean":"2024-09-02T10:19:04.040745+0000","last_became_active":"2024-09-02T10:19:06.870623+0000","last_became_peered":"2024-09-02T10:19:06.870623+0000","last_unstale":"2024-09-02T10:19:13.138184+0000","last_undegraded":"2024-09-02T10:19:06.071144+0000","last_fullsized":"2024-09-02T10:19:06.059552+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2640,"log_dups_size":0,"ondisk_log_size":2640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173015040,"num_objects":2640,"num_object_clones":0,"num_object_copies":7920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1795,"num_objects_unfound":0,"num_objects_dirty":2640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2640,"num_write_kb":168960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":24772608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,7],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":1795},{"shards":"3,7","objects":844}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":558,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.508567+0000","last_change":"2024-09-02T10:18:20.714703+0000","last_active":"2024-09-02T10:19:10.508567+0000","last_peered":"2024-09-02T10:19:10.508567+0000","last_clean":"2024-09-02T10:19:10.508567+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:19:10.508567+0000","last_undegraded":"2024-09-02T10:19:10.508567+0000","last_fullsized":"2024-09-02T10:19:10.508567+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:20.714649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:20.714649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:35.390231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"553'2452","reported_seq":2502,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:11.102532+0000","last_change":"2024-09-02T10:18:37.848300+0000","last_active":"2024-09-02T10:19:11.102532+0000","last_peered":"2024-09-02T10:19:11.102532+0000","last_clean":"2024-09-02T10:19:11.102532+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:19:11.102532+0000","last_undegraded":"2024-09-02T10:19:11.102532+0000","last_fullsized":"2024-09-02T10:19:11.102532+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2452,"log_dups_size":0,"ondisk_log_size":2452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:01.700433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160694272,"num_objects":2452,"num_object_clones":0,"num_object_copies":7356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2452,"num_write_kb":156928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":811,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.485671+0000","last_change":"2024-09-02T10:19:05.058925+0000","last_active":"2024-09-02T10:19:10.485671+0000","last_peered":"2024-09-02T10:19:10.485671+0000","last_clean":"2024-09-02T10:19:10.485671+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:19:10.485671+0000","last_undegraded":"2024-09-02T10:19:10.485671+0000","last_fullsized":"2024-09-02T10:19:10.485671+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.789947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.789947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:10.614688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"553'2659","reported_seq":2709,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:13.596992+0000","last_change":"2024-09-02T10:18:37.064957+0000","last_active":"2024-09-02T10:19:13.596992+0000","last_peered":"2024-09-02T10:19:13.596992+0000","last_clean":"2024-09-02T10:19:13.596992+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:19:13.596992+0000","last_undegraded":"2024-09-02T10:19:13.596992+0000","last_fullsized":"2024-09-02T10:19:13.596992+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2659,"log_dups_size":0,"ondisk_log_size":2659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:49.603479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174260224,"num_objects":2659,"num_object_clones":0,"num_object_copies":7977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2659,"num_write_kb":170176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1079,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.483034+0000","last_change":"2024-09-02T10:19:05.058662+0000","last_active":"2024-09-02T10:19:10.483034+0000","last_peered":"2024-09-02T10:19:10.483034+0000","last_clean":"2024-09-02T10:19:10.483034+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:19:10.483034+0000","last_undegraded":"2024-09-02T10:19:10.483034+0000","last_fullsized":"2024-09-02T10:19:10.483034+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:42.912082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:42.912082+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:51.861067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"553'2508","reported_seq":2561,"reported_epoch":553,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:19:12.128276+0000","last_change":"2024-09-02T10:19:06.881012+0000","last_active":"2024-09-02T10:19:12.128276+0000","last_peered":"2024-09-02T10:19:12.128276+0000","last_clean":"2024-09-02T10:19:04.021218+0000","last_became_active":"2024-09-02T10:19:06.874629+0000","last_became_peered":"2024-09-02T10:19:06.874629+0000","last_unstale":"2024-09-02T10:19:12.128276+0000","last_undegraded":"2024-09-02T10:19:06.069315+0000","last_fullsized":"2024-09-02T10:19:06.060341+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2508,"log_dups_size":0,"ondisk_log_size":2508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164364288,"num_objects":2508,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2508,"num_write_kb":160512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":2118},{"shards":"3,5","objects":390}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":823,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.485960+0000","last_change":"2024-09-02T10:18:37.852037+0000","last_active":"2024-09-02T10:19:10.485960+0000","last_peered":"2024-09-02T10:19:10.485960+0000","last_clean":"2024-09-02T10:19:10.485960+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:19:10.485960+0000","last_undegraded":"2024-09-02T10:19:10.485960+0000","last_fullsized":"2024-09-02T10:19:10.485960+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:37.851984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:37.851984+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:46.708384+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"553'2450","reported_seq":2499,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:11.140023+0000","last_change":"2024-09-02T10:18:37.850899+0000","last_active":"2024-09-02T10:19:11.140023+0000","last_peered":"2024-09-02T10:19:11.140023+0000","last_clean":"2024-09-02T10:19:11.140023+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:19:11.140023+0000","last_undegraded":"2024-09-02T10:19:11.140023+0000","last_fullsized":"2024-09-02T10:19:11.140023+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2450,"log_dups_size":0,"ondisk_log_size":2450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:02.178728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160563200,"num_objects":2450,"num_object_clones":0,"num_object_copies":7350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2450,"num_write_kb":156800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1078,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.482685+0000","last_change":"2024-09-02T10:19:08.999797+0000","last_active":"2024-09-02T10:19:10.482685+0000","last_peered":"2024-09-02T10:19:10.482685+0000","last_clean":"2024-09-02T10:19:10.482685+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:19:10.482685+0000","last_undegraded":"2024-09-02T10:19:10.482685+0000","last_fullsized":"2024-09-02T10:19:10.482685+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:08.999706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:08.999706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:13.510899+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"553'2545","reported_seq":2595,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:12.889031+0000","last_change":"2024-09-02T10:18:37.600768+0000","last_active":"2024-09-02T10:19:12.889031+0000","last_peered":"2024-09-02T10:19:12.889031+0000","last_clean":"2024-09-02T10:19:12.889031+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:19:12.889031+0000","last_undegraded":"2024-09-02T10:19:12.889031+0000","last_fullsized":"2024-09-02T10:19:12.889031+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2545,"log_dups_size":0,"ondisk_log_size":2545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:54.523794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166789120,"num_objects":2545,"num_object_clones":0,"num_object_copies":7635,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2545,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2545,"num_write_kb":162880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":30,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.069416+0000","last_change":"2024-09-02T10:19:06.947029+0000","last_active":"2024-09-02T10:19:14.069416+0000","last_peered":"2024-09-02T10:19:14.069416+0000","last_clean":"2024-09-02T10:19:14.069416+0000","last_became_active":"2024-09-02T10:19:06.049121+0000","last_became_peered":"2024-09-02T10:19:06.049121+0000","last_unstale":"2024-09-02T10:19:14.069416+0000","last_undegraded":"2024-09-02T10:19:14.069416+0000","last_fullsized":"2024-09-02T10:19:14.069416+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:06.946957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:06.946957+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:24.543000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"553'2511","reported_seq":2560,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:12.188602+0000","last_change":"2024-09-02T10:18:37.848530+0000","last_active":"2024-09-02T10:19:12.188602+0000","last_peered":"2024-09-02T10:19:12.188602+0000","last_clean":"2024-09-02T10:19:12.188602+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:19:12.188602+0000","last_undegraded":"2024-09-02T10:19:12.188602+0000","last_fullsized":"2024-09-02T10:19:12.188602+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2511,"log_dups_size":0,"ondisk_log_size":2511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:58.128911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164560896,"num_objects":2511,"num_object_clones":0,"num_object_copies":7533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2511,"num_write_kb":160704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":850,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.486032+0000","last_change":"2024-09-02T10:19:05.059514+0000","last_active":"2024-09-02T10:19:10.486032+0000","last_peered":"2024-09-02T10:19:10.486032+0000","last_clean":"2024-09-02T10:19:10.486032+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:19:10.486032+0000","last_undegraded":"2024-09-02T10:19:10.486032+0000","last_fullsized":"2024-09-02T10:19:10.486032+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:10.104220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:10.104220+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:37.010094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"553'2653","reported_seq":2703,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:13.169851+0000","last_change":"2024-09-02T10:18:37.066015+0000","last_active":"2024-09-02T10:19:13.169851+0000","last_peered":"2024-09-02T10:19:13.169851+0000","last_clean":"2024-09-02T10:19:13.169851+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:19:13.169851+0000","last_undegraded":"2024-09-02T10:19:13.169851+0000","last_fullsized":"2024-09-02T10:19:13.169851+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2653,"log_dups_size":0,"ondisk_log_size":2653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:02.695338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173867008,"num_objects":2653,"num_object_clones":0,"num_object_copies":7959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2653,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2653,"num_write_kb":169792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":249,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-02T10:19:10.485896+0000","last_change":"2024-09-02T10:19:02.956010+0000","last_active":"2024-09-02T10:19:10.485896+0000","last_peered":"2024-09-02T10:19:10.485896+0000","last_clean":"2024-09-02T10:19:10.485896+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:19:10.485896+0000","last_undegraded":"2024-09-02T10:19:10.485896+0000","last_fullsized":"2024-09-02T10:19:10.485896+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:02.955947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:02.955947+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:26.242539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":2699165696,"num_objects":41186,"num_object_clones":0,"num_object_copies":123558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":5139,"num_objects_unfound":0,"num_objects_dirty":41186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41186,"num_write_kb":2635904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1280,"num_bytes_recovered":83886080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8208646144,"data_stored":8208646144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41186,"ondisk_log_size":41186,"up":48,"acting":44,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383909,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":923932,"kb_used_data":483212,"kb_used_omap":1,"kb_used_meta":440702,"kb_avail":92821220,"statfs":{"total":95995035648,"available":95048929280,"internally_reserved":0,"allocated":494809088,"data_stored":494434914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":451279300},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612842,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799776,"kb_used_data":483980,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92945376,"statfs":{"total":95995035648,"available":95176065024,"internally_reserved":0,"allocated":495595520,"data_stored":495221346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":323353024},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841775,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1101984,"kb_used_data":641804,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92643168,"statfs":{"total":95995035648,"available":94866604032,"internally_reserved":0,"allocated":657207296,"data_stored":656833122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":471202240},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070709,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":773148,"kb_used_data":471244,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92972004,"statfs":{"total":95995035648,"available":95203332096,"internally_reserved":0,"allocated":482553856,"data_stored":482179682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299641,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1397800,"kb_used_data":925140,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92347352,"statfs":{"total":95995035648,"available":94563688448,"internally_reserved":0,"allocated":947343360,"data_stored":946965634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":483981770},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528573,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1100252,"kb_used_data":784460,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92644900,"statfs":{"total":95995035648,"available":94868377600,"internally_reserved":0,"allocated":803287040,"data_stored":802912866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":323353031},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724802,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1783524,"kb_used_data":1296980,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91961628,"statfs":{"total":95995035648,"available":94168707072,"internally_reserved":0,"allocated":1328107520,"data_stored":1327729794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":498203077},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953736,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":467548,"kb_used_data":229516,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93277604,"statfs":{"total":95995035648,"available":95516266496,"internally_reserved":0,"allocated":235024384,"data_stored":234650210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149965,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1775260,"kb_used_data":1333132,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":91969892,"statfs":{"total":95995035648,"available":94177169408,"internally_reserved":0,"allocated":1365127168,"data_stored":1364752994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":452721096},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411600,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575126,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1253348,"kb_used_data":811220,"kb_used_omap":1,"kb_used_meta":442110,"kb_avail":92491804,"statfs":{"total":95995035648,"available":94711607296,"internally_reserved":0,"allocated":830689280,"data_stored":830311554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":452721093},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738652,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1032096,"kb_used_data":487244,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92713056,"statfs":{"total":95995035648,"available":94938169344,"internally_reserved":0,"allocated":498937856,"data_stored":498563682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":503971840,"data_stored":503971840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":840957952,"data_stored":840957952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1373831168,"data_stored":1373831168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":238747648,"data_stored":238747648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1333133312,"data_stored":1333133312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":811073536,"data_stored":811073536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":950861824,"data_stored":950861824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":485687296,"data_stored":485687296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":676069376,"data_stored":676069376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":495910912,"data_stored":495910912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":498401280,"data_stored":498401280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:19:21.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:19:21.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:21.425 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:21.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:21.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:21.553 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:21.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:21.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:21.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:21.681 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:21.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:21.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:21.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:21.817 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:21.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:21.939 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:19:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:19:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:19:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:19:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.1b,RELEASE,e:293,from:8,reservation_nonce:0)", 2024-09-02T10:19:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:09:22.169444+0000", 2024-09-02T10:19:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 599.76654003500005, 2024-09-02T10:19:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47728760399999998, 2024-09-02T10:19:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.940 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.169444+0000", 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.169444+0000", 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.169444+0000", 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2100000000000001e-07 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.169447+0000", 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7999999999999999e-06 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.646631+0000", 2024-09-02T10:19:21.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47718416299999999 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.646638+0000", 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9399999999999996e-06 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.646687+0000", 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8977000000000002e-05 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.646713+0000", 2024-09-02T10:19:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5446999999999999e-05 2024-09-02T10:19:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:09:22.646732+0000", 2024-09-02T10:19:21.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9055999999999998e-05 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.24976.0:219058 3.6 3:62e84818:::benchmark_data_smithi187_59989_object109515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e304)", 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:10:17.211328+0000", 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 544.72465599300006, 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38832308100000001, 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.945 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.24976", 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1265874209", 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 219058 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211328+0000", 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211328+0000", 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211330+0000", 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.468e-06 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211335+0000", 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.631e-06 2024-09-02T10:19:21.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211352+0000", 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6923e-05 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.211360+0000", 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9349999999999994e-06 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:21.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.218608+0000", 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072476090000000003 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.218737+0000", 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012934000000000001 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.218854+0000", 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117108 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.224834+0000", 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0059799290000000001 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.224873+0000", 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8544999999999998e-05 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.599414+0000", 2024-09-02T10:19:21.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.374541611 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.599604+0000", 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018962800000000001 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:10:17.599651+0000", 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7354000000000001e-05 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:11574 4.3 e326/318 4:c9ca2732:::benchmark_data_smithi187_76542_object11573:head v 326'718, mlcod=326'716)", 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.852685+0000", 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 491.08329937299999, 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97050503600000004, 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.852685+0000", 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.852685+0000", 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.852690+0000", 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.293e-06 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.852750+0000", 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9536000000000001e-05 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.822320+0000", 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.96957074300000001 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.822330+0000", 2024-09-02T10:19:21.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1239999999999994e-06 2024-09-02T10:19:21.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:21.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.822415+0000", 2024-09-02T10:19:21.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5823999999999997e-05 2024-09-02T10:19:21.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:21.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.822437+0000", 2024-09-02T10:19:21.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1841999999999999e-05 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.823147+0000", 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00070947100000000004 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.823190+0000", 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3203e-05 2024-09-02T10:19:21.957 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:42144 4.b 4:d62c2ece:::benchmark_data_smithi187_76542_object42143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:48.539483+0000", 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 453.39650076200002, 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.391093368, 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:21.958 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 42144 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539483+0000", 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539483+0000", 2024-09-02T10:19:21.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539485+0000", 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3039999999999999e-06 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539544+0000", 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9614999999999998e-05 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539557+0000", 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2551000000000001e-05 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539563+0000", 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4099999999999996e-06 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539618+0000", 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4221999999999997e-05 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539747+0000", 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000129806 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539861+0000", 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113622 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.541533+0000", 2024-09-02T10:19:21.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001672394 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.547909+0000", 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006375932 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.930451+0000", 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38254196499999998 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.930525+0000", 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3911000000000004e-05 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.930577+0000", 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1635999999999998e-05 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.964 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:92326 4.b 4:d3b75331:::benchmark_data_smithi187_76542_object92325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-02T10:19:21.965 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:33.492476+0000", 2024-09-02T10:19:21.965 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 408.443507985, 2024-09-02T10:19:21.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79601677199999998, 2024-09-02T10:19:21.965 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.965 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.965 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:21.965 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:21.965 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:21.965 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:21.965 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92326 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492476+0000", 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492476+0000", 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492478+0000", 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4219999999999999e-06 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492566+0000", 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8583000000000002e-05 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492587+0000", 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1084e-05 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492596+0000", 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1880000000000008e-06 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492668+0000", 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1876000000000001e-05 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492825+0000", 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015684100000000001 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492969+0000", 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000143783 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.495907+0000", 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029384139999999999 2024-09-02T10:19:21.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.496430+0000", 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052270400000000003 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288414+0000", 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79198343699999996 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288458+0000", 2024-09-02T10:19:21.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4030999999999999e-05 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288493+0000", 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5408999999999999e-05 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(5.8 479/463 [PushReplyOp(5:1936829f:::benchmark_data_smithi187_89405_object32750:head)])", 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:19:21.971 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 185.757479499, 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36677579799999999, 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9100000000000004e-07 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178510+0000", 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.51e-06 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.543260+0000", 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36475006799999998 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.543276+0000", 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5837000000000001e-05 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.543392+0000", 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115862 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.545280+0000", 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00188893 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25933.0:205325 5.e e524/431 5:726c3a45:::benchmark_data_smithi187_89405_object101720:head v 524'12690, mlcod=524'12688)", 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.618253+0000", 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 64.317731136999996, 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38099444199999999, 2024-09-02T10:19:21.975 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618253+0000", 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618253+0000", 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618254+0000", 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1600000000000001e-07 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618258+0000", 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3320000000000002e-06 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995375+0000", 2024-09-02T10:19:21.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37711745899999999 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995380+0000", 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7330000000000003e-06 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995450+0000", 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9474000000000006e-05 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:21.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995475+0000", 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5579999999999999e-05 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.999213+0000", 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003737476 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.999247+0000", 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4771999999999998e-05 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.979 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:207192 5.3 5:c539d868:::benchmark_data_smithi187_89405_object103587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:20.104200+0000", 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.831784442, 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52651428300000003, 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:19:21.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 207192 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104200+0000", 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104200+0000", 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104201+0000", 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104205+0000", 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.98e-06 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104213+0000", 2024-09-02T10:19:21.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7560000000000005e-06 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104218+0000", 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104263+0000", 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4708e-05 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:21.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104392+0000", 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128987 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104522+0000", 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012982799999999999 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.109243+0000", 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047210799999999999 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.109411+0000", 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016770899999999999 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630621+0000", 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52120988899999998 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630666+0000", 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4883999999999999e-05 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630714+0000", 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8497e-05 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25933.0:207192 5.3 e525/431 ondisk, result = 0)", 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:20.108691+0000", 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 61.827292739000001, 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52202865700000001, 2024-09-02T10:19:21.986 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108691+0000", 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108691+0000", 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108692+0000", 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9100000000000004e-07 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108695+0000", 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.529e-06 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630413+0000", 2024-09-02T10:19:21.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52171714700000005 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630428+0000", 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5843999999999999e-05 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630579+0000", 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000150252 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:21.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630598+0000", 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8865999999999999e-05 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630720+0000", 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122528 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:19723 6.9 6:94e8ee31:::benchmark_data_smithi187_102448_object19722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:19:21.990 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:53.705822+0000", 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.230161858999999, 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21665701200000001, 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 19723 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705822+0000", 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705822+0000", 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705823+0000", 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.082e-06 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705916+0000", 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2105e-05 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705929+0000", 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.397e-05 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705937+0000", 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7330000000000003e-06 2024-09-02T10:19:21.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705981+0000", 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3949999999999998e-05 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.706119+0000", 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013788900000000001 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.706208+0000", 2024-09-02T10:19:21.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9197000000000007e-05 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.707009+0000", 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080078799999999998 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.714056+0000", 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070474170000000003 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:21.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922382+0000", 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20832547500000001 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922434+0000", 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2473999999999999e-05 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922479+0000", 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4932000000000003e-05 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:21.996 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:19732 6.9 6:9f276e57:::benchmark_data_smithi187_102448_object19731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:53.712327+0000", 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.223657505999999, 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.216236916, 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 19732 2024-09-02T10:19:21.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712327+0000", 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712327+0000", 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:21.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712328+0000", 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0449999999999999e-06 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712379+0000", 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0779999999999998e-05 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712391+0000", 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2085e-05 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:21.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:22.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:22.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:22.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:22.001 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 41 15 51653 51638 78.7101 78.5625 0.00521595 0.0127008 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712398+0000", 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7540000000000006e-06 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712448+0000", 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9880999999999999e-05 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712542+0000", 2024-09-02T10:19:22.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3450999999999997e-05 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712634+0000", 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2317000000000004e-05 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713395+0000", 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00076103200000000003 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.787692+0000", 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.074297236000000003 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.928492+0000", 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14080002899999999 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.928534+0000", 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1220000000000002e-05 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.928564+0000", 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0086e-05 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26524.0:19723 6.9 e543/533 ondisk, result = 0)", 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:53.713901+0000", 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 28.222083654999999, 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20865452900000001, 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:22.005 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713901+0000", 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713901+0000", 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713901+0000", 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713904+0000", 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8420000000000001e-06 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922307+0000", 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.208403057 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922312+0000", 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0849999999999996e-06 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922372+0000", 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9664000000000003e-05 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922378+0000", 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9989999999999997e-06 2024-09-02T10:19:22.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922555+0000", 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017727500000000001 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:39567 6.0 6:0ea24a11:::benchmark_data_smithi187_102448_object39566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:11.107391+0000", 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.828592846999999, 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26867963700000003, 2024-09-02T10:19:22.009 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 39567 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107391+0000", 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107391+0000", 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107393+0000", 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.341e-06 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107430+0000", 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7738999999999999e-05 2024-09-02T10:19:22.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107447+0000", 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6472000000000001e-05 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107456+0000", 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6279999999999994e-06 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:22.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107520+0000", 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4442000000000004e-05 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107677+0000", 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015673999999999999 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107865+0000", 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018834400000000001 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.109307+0000", 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014422720000000001 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.109639+0000", 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033185300000000001 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.375963+0000", 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.266324318 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.376022+0000", 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8971999999999997e-05 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.376071+0000", 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8516000000000001e-05 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.015 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:39596 6.0 6:0a53121f:::benchmark_data_smithi187_102448_object39595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:11.195799+0000", 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 10.740184750999999, 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19974002900000001, 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 39596 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.195799+0000", 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.195799+0000", 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.195806+0000", 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4949999999999996e-06 2024-09-02T10:19:22.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.195848+0000", 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2286000000000001e-05 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.195870+0000", 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1849999999999999e-05 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.195880+0000", 2024-09-02T10:19:22.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0234000000000001e-05 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.195947+0000", 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6735000000000004e-05 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.196125+0000", 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017833200000000001 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:19:22.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.196235+0000", 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109324 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.197858+0000", 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016232340000000001 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.197873+0000", 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5319e-05 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.395423+0000", 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19754987399999999 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.395488+0000", 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4912999999999993e-05 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.395539+0000", 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1433e-05 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.021 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:41986 6.0 6:05acd652:::benchmark_data_smithi187_102448_object41985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:13.276491+0000", 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.6594934660000007, 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19555461699999999, 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:19:22.022 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 41986 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.276491+0000", 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.276491+0000", 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.276492+0000", 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1100000000000004e-07 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.276623+0000", 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013094499999999999 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.276631+0000", 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6740000000000003e-06 2024-09-02T10:19:22.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.276639+0000", 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5209999999999997e-06 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.278031+0000", 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013918509999999999 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.278147+0000", 2024-09-02T10:19:22.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116612 2024-09-02T10:19:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:19:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.278271+0000", 2024-09-02T10:19:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012339800000000001 2024-09-02T10:19:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:22.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.279758+0000", 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014868710000000001 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.283355+0000", 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035976609999999998 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.471946+0000", 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18859115600000001 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.472002+0000", 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5717000000000002e-05 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.472045+0000", 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3300000000000002e-05 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.028 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:41991 6.0 6:0cfbcc2e:::benchmark_data_smithi187_102448_object41990:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:13.279980+0000", 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.6560037760000004, 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19425189600000001, 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 41991 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:22.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.279980+0000", 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.279980+0000", 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.279982+0000", 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1039999999999999e-06 2024-09-02T10:19:22.030 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.280016+0000", 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4628000000000001e-05 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.280026+0000", 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0256e-05 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.280033+0000", 2024-09-02T10:19:22.031 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8940000000000004e-06 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.280071+0000", 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7388000000000002e-05 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.280202+0000", 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013087299999999999 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.032 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.280338+0000", 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013673599999999999 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.281806+0000", 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014672260000000001 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.472091+0000", 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.190285226 2024-09-02T10:19:22.033 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.474128+0000", 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002037184 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.474189+0000", 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0711999999999997e-05 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:22.034 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.474232+0000", 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3668999999999998e-05 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:41993 6.0 6:0c875c6a:::benchmark_data_smithi187_102448_object41992:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:13.280954+0000", 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.6550305109999996, 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.199994056, 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:22.035 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 41993 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.280954+0000", 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:22.036 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.280954+0000", 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.280955+0000", 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9400000000000004e-07 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.281023+0000", 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8003000000000006e-05 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.037 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.281035+0000", 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2706e-05 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.281042+0000", 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9e-06 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.281832+0000", 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00078961099999999996 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.038 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.281965+0000", 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000133609 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.282097+0000", 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013171400000000001 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.472122+0000", 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.190024835 2024-09-02T10:19:22.039 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.477868+0000", 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0057463130000000003 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.480849+0000", 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029803210000000002 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.480904+0000", 2024-09-02T10:19:22.040 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5492000000000003e-05 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.480948+0000", 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3658000000000001e-05 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26524.0:41991 6.0 e553/533 ondisk, result = 0)", 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:13.281814+0000", 2024-09-02T10:19:22.041 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.6541704119999991, 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.192426862, 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.281814+0000", 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.281814+0000", 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.042 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.281814+0000", 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5899999999999996e-07 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.281818+0000", 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8399999999999997e-06 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.472649+0000", 2024-09-02T10:19:22.043 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19083066100000001 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.472654+0000", 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6879999999999998e-06 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.474112+0000", 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014589749999999999 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.044 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.474122+0000", 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1460000000000002e-06 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.474241+0000", 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000118993 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:41996 6.9 6:9f275ad5:::benchmark_data_smithi187_102448_object41995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-09-02T10:19:22.045 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:13.282192+0000", 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.6537921139999998, 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19361078300000001, 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 41996 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.046 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.282192+0000", 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.282192+0000", 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.282193+0000", 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2699999999999998e-07 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.047 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.282259+0000", 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5939999999999995e-05 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.282267+0000", 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5760000000000006e-06 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.282272+0000", 2024-09-02T10:19:22.048 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9830000000000004e-06 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.282320+0000", 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7302000000000002e-05 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.282440+0000", 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012079299999999999 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.049 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.282571+0000", 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013025499999999999 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.471961+0000", 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.189390691 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.472110+0000", 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014897900000000001 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.050 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.475701+0000", 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003590181 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.475761+0000", 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0281000000000001e-05 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.475803+0000", 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1974999999999997e-05 2024-09-02T10:19:22.051 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26524.0:41993 6.0 e553/533 ondisk, result = 0)", 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:13.283438+0000", 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 8.6525458499999992, 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.197515414, 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:22.052 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.283438+0000", 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.283438+0000", 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.283439+0000", 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2900000000000003e-07 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.053 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.283443+0000", 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6899999999999998e-06 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.478013+0000", 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19457029000000001 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.478019+0000", 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1489999999999999e-06 2024-09-02T10:19:22.054 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.480836+0000", 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028166810000000001 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.480843+0000", 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.272e-06 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:13.480954+0000", 2024-09-02T10:19:22.055 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000110603 2024-09-02T10:19:22.056 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.056 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.056 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.056 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:22.056 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:22.056 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:19:22.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:22.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:22.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:22.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:22.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:22.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:22.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:22.398 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:22.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:22.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:22.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:22.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:22.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:22.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:22.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:22.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:22.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:19:22.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:22.785 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:22.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:22.823 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:22 smithi187 ceph-mon[20072]: pgmap v1285: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 2119/143157 objects degraded (1.480%); 3892/143157 objects misplaced (2.719%); 14 MiB/s, 222 objects/s recovering 2024-09-02T10:19:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:22 smithi162 ceph-mon[20279]: pgmap v1285: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 2119/143157 objects degraded (1.480%); 3892/143157 objects misplaced (2.719%); 14 MiB/s, 222 objects/s recovering 2024-09-02T10:19:22.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:22.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:22 smithi151 ceph-mon[16639]: pgmap v1285: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 2.9 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 2119/143157 objects degraded (1.480%); 3892/143157 objects misplaced (2.719%); 14 MiB/s, 222 objects/s recovering 2024-09-02T10:19:22.914 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:22.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:22.923 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:22.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 42 16 52972 52956 78.7973 82.375 0.0169281 0.0126823 2024-09-02T10:19:23.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:23.044 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:23.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:23.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:23.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:23.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:23.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:23.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:23.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:23.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:23.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:23.436 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:23.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:23.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:19:23.567 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:23.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:23.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:23.572 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:19:23.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:23.694 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:23.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:23.695 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:23.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/416186010' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:19:23.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:23.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:23.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:23.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/416186010' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:19:23.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:23.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 43 16 54152 54136 78.6798 73.75 0.0417956 0.0127047 2024-09-02T10:19:23.958 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:23.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/416186010' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:19:24.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:24.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:24.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:24.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:24.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:24.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:24.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:24.335 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:24.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:24.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:24.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:24.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:24.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:24.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:24.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:24.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:24.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:24.726 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:24.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:24.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:24.798 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:24.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:24 smithi162 ceph-mon[20279]: pgmap v1286: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s; 2119/154254 objects degraded (1.374%); 2999/154254 objects misplaced (1.944%); 16 MiB/s, 249 objects/s recovering 2024-09-02T10:19:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:24 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:19:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/416186010' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:19:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:24 smithi162 ceph-mon[20279]: osdmap e555: 12 total, 12 up, 11 in 2024-09-02T10:19:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2119/154254 objects degraded (1.374%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:24 smithi162 ceph-mon[20279]: osdmap e556: 12 total, 12 up, 11 in 2024-09-02T10:19:24.864 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:24.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:24.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:24 smithi151 ceph-mon[16639]: pgmap v1286: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s; 2119/154254 objects degraded (1.374%); 2999/154254 objects misplaced (1.944%); 16 MiB/s, 249 objects/s recovering 2024-09-02T10:19:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:24 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:19:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/416186010' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:19:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:24 smithi151 ceph-mon[16639]: osdmap e555: 12 total, 12 up, 11 in 2024-09-02T10:19:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2119/154254 objects degraded (1.374%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:24 smithi151 ceph-mon[16639]: osdmap e556: 12 total, 12 up, 11 in 2024-09-02T10:19:24.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 44 16 55058 55042 78.1784 56.625 0.0165687 0.0127858 2024-09-02T10:19:24.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:24.995 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:24.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:24 smithi187 ceph-mon[20072]: pgmap v1286: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.1 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s; 2119/154254 objects degraded (1.374%); 2999/154254 objects misplaced (1.944%); 16 MiB/s, 249 objects/s recovering 2024-09-02T10:19:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:24 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:19:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/416186010' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:19:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:24 smithi187 ceph-mon[20072]: osdmap e555: 12 total, 12 up, 11 in 2024-09-02T10:19:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2119/154254 objects degraded (1.374%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:24 smithi187 ceph-mon[20072]: osdmap e556: 12 total, 12 up, 11 in 2024-09-02T10:19:25.049 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:19:25.049 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:19:25.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:25.130 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:25.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:25.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:25.259 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:25.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:25.361 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:25.386 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:25.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:25.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:25.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:25.518 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:25.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:25.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:25.649 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:25.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:25.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:25.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:25.774 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:25.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:25.775 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:25.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:25.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:25.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:25.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:25.923 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1288,"stamp":"2024-09-02T10:19:24.054887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3419734560,"num_objects":52174,"num_object_clones":0,"num_object_copies":156522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":52174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52172,"num_write_kb":3339008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3841,"num_bytes_recovered":252257344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55024,"ondisk_log_size":55024,"up":141,"acting":137,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":14683344,"kb_used_data":10076896,"kb_used_omap":17,"kb_used_meta":4606254,"kb_avail":1016513328,"statfs":{"total":1055945392128,"available":1040909647872,"internally_reserved":0,"allocated":10318741504,"data_stored":10314567227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":4716804982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,4,5,3,6,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":719978496,"num_objects":10986,"num_object_clones":0,"num_object_copies":32958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2558,"num_objects_unfound":0,"num_objects_dirty":10986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10986,"num_write_kb":703104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2558,"num_bytes_recovered":167641088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.718841"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":452,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.058887+0000","last_change":"2024-09-02T10:18:43.049145+0000","last_active":"2024-09-02T10:19:14.058887+0000","last_peered":"2024-09-02T10:19:14.058887+0000","last_clean":"2024-09-02T10:19:14.058887+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:19:14.058887+0000","last_undegraded":"2024-09-02T10:19:14.058887+0000","last_fullsized":"2024-09-02T10:19:14.058887+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:43.049077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:43.049077+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:07.603384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":262,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-09-02T10:19:23.587747+0000","last_change":"2024-09-02T10:18:19.826521+0000","last_active":"2024-09-02T10:19:23.587747+0000","last_peered":"2024-09-02T10:19:23.587747+0000","last_clean":"2024-09-02T10:19:23.587747+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:19:23.587747+0000","last_undegraded":"2024-09-02T10:19:23.587747+0000","last_fullsized":"2024-09-02T10:19:23.587747+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:19.826468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:19.826468+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":710,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.053365+0000","last_change":"2024-09-02T10:18:16.145420+0000","last_active":"2024-09-02T10:19:14.053365+0000","last_peered":"2024-09-02T10:19:14.053365+0000","last_clean":"2024-09-02T10:19:14.053365+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:19:14.053365+0000","last_undegraded":"2024-09-02T10:19:14.053365+0000","last_fullsized":"2024-09-02T10:19:14.053365+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:16.145346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:16.145346+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:31.421793+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":710,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.053223+0000","last_change":"2024-09-02T10:18:39.045140+0000","last_active":"2024-09-02T10:19:14.053223+0000","last_peered":"2024-09-02T10:19:14.053223+0000","last_clean":"2024-09-02T10:19:14.053223+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:19:14.053223+0000","last_undegraded":"2024-09-02T10:19:14.053223+0000","last_fullsized":"2024-09-02T10:19:14.053223+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:39.045083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:39.045083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:03.744946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":22,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.069415+0000","last_change":"2024-09-02T10:19:05.058923+0000","last_active":"2024-09-02T10:19:14.069415+0000","last_peered":"2024-09-02T10:19:14.069415+0000","last_clean":"2024-09-02T10:19:14.069415+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:19:14.069415+0000","last_undegraded":"2024-09-02T10:19:14.069415+0000","last_fullsized":"2024-09-02T10:19:14.069415+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:01.810089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:01.810089+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:19.405935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":558,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.066080+0000","last_change":"2024-09-02T10:18:17.772780+0000","last_active":"2024-09-02T10:19:14.066080+0000","last_peered":"2024-09-02T10:19:14.066080+0000","last_clean":"2024-09-02T10:19:14.066080+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:19:14.066080+0000","last_undegraded":"2024-09-02T10:19:14.066080+0000","last_fullsized":"2024-09-02T10:19:14.066080+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:17.772723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:17.772723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:28.508722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1083,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.059398+0000","last_change":"2024-09-02T10:19:03.880037+0000","last_active":"2024-09-02T10:19:14.059398+0000","last_peered":"2024-09-02T10:19:14.059398+0000","last_clean":"2024-09-02T10:19:14.059398+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:19:14.059398+0000","last_undegraded":"2024-09-02T10:19:14.059398+0000","last_fullsized":"2024-09-02T10:19:14.059398+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:03.879968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:03.879968+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.678816+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":725,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.053496+0000","last_change":"2024-09-02T10:18:35.059181+0000","last_active":"2024-09-02T10:19:14.053496+0000","last_peered":"2024-09-02T10:19:14.053496+0000","last_clean":"2024-09-02T10:19:14.053496+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:19:14.053496+0000","last_undegraded":"2024-09-02T10:19:14.053496+0000","last_fullsized":"2024-09-02T10:19:14.053496+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:35.059123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:35.059123+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:59.080089+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":727,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-09-02T10:19:23.588637+0000","last_change":"2024-09-02T10:18:10.904183+0000","last_active":"2024-09-02T10:19:23.588637+0000","last_peered":"2024-09-02T10:19:23.588637+0000","last_clean":"2024-09-02T10:19:23.588637+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:19:23.588637+0000","last_undegraded":"2024-09-02T10:19:23.588637+0000","last_fullsized":"2024-09-02T10:19:23.588637+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:10.904126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:10.904126+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":701,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.056681+0000","last_change":"2024-09-02T10:19:05.910750+0000","last_active":"2024-09-02T10:19:14.056681+0000","last_peered":"2024-09-02T10:19:14.056681+0000","last_clean":"2024-09-02T10:19:14.056681+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:19:14.056681+0000","last_undegraded":"2024-09-02T10:19:14.056681+0000","last_fullsized":"2024-09-02T10:19:14.056681+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:05.910691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:05.910691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:20.353104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":544,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.056637+0000","last_change":"2024-09-02T10:17:44.275503+0000","last_active":"2024-09-02T10:19:14.056637+0000","last_peered":"2024-09-02T10:19:14.056637+0000","last_clean":"2024-09-02T10:19:14.056637+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:19:14.056637+0000","last_undegraded":"2024-09-02T10:19:14.056637+0000","last_fullsized":"2024-09-02T10:19:14.056637+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:17:44.275434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:17:44.275434+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1060,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-09-02T10:19:23.579833+0000","last_change":"2024-09-02T10:19:00.052912+0000","last_active":"2024-09-02T10:19:23.579833+0000","last_peered":"2024-09-02T10:19:23.579833+0000","last_clean":"2024-09-02T10:19:23.579833+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:19:23.579833+0000","last_undegraded":"2024-09-02T10:19:23.579833+0000","last_fullsized":"2024-09-02T10:19:23.579833+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.052857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.052857+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:29.321752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":993,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.059981+0000","last_change":"2024-09-02T10:18:41.060635+0000","last_active":"2024-09-02T10:19:14.059981+0000","last_peered":"2024-09-02T10:19:14.059981+0000","last_clean":"2024-09-02T10:19:14.059981+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:19:14.059981+0000","last_undegraded":"2024-09-02T10:19:14.059981+0000","last_fullsized":"2024-09-02T10:19:14.059981+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:41.060566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:41.060566+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:47.963172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":972,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.055366+0000","last_change":"2024-09-02T10:18:16.400250+0000","last_active":"2024-09-02T10:19:14.055366+0000","last_peered":"2024-09-02T10:19:14.055366+0000","last_clean":"2024-09-02T10:19:14.055366+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:19:14.055366+0000","last_undegraded":"2024-09-02T10:19:14.055366+0000","last_fullsized":"2024-09-02T10:19:14.055366+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:16.400175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:16.400175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:25.500090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":963,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.059611+0000","last_change":"2024-09-02T10:18:40.069068+0000","last_active":"2024-09-02T10:19:14.059611+0000","last_peered":"2024-09-02T10:19:14.059611+0000","last_clean":"2024-09-02T10:19:14.059611+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:19:14.059611+0000","last_undegraded":"2024-09-02T10:19:14.059611+0000","last_fullsized":"2024-09-02T10:19:14.059611+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:40.069013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:40.069013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:45.996592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"554'3208","reported_seq":3259,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:21.737709+0000","last_change":"2024-09-02T10:18:37.599555+0000","last_active":"2024-09-02T10:19:21.737709+0000","last_peered":"2024-09-02T10:19:21.737709+0000","last_clean":"2024-09-02T10:19:21.737709+0000","last_became_active":"2024-09-02T10:18:37.599064+0000","last_became_peered":"2024-09-02T10:18:37.599064+0000","last_unstale":"2024-09-02T10:19:21.737709+0000","last_undegraded":"2024-09-02T10:19:21.737709+0000","last_fullsized":"2024-09-02T10:19:21.737709+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3208,"log_dups_size":0,"ondisk_log_size":3208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:50.285721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210239488,"num_objects":3208,"num_object_clones":0,"num_object_copies":9624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3208,"num_write_kb":205312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1050,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.059727+0000","last_change":"2024-09-02T10:18:13.075935+0000","last_active":"2024-09-02T10:19:14.059727+0000","last_peered":"2024-09-02T10:19:14.059727+0000","last_clean":"2024-09-02T10:19:14.059727+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:19:14.059727+0000","last_undegraded":"2024-09-02T10:19:14.059727+0000","last_fullsized":"2024-09-02T10:19:14.059727+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:13.075888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:13.075888+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:39.573123+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"555'3411","reported_seq":6087,"reported_epoch":555,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:19:23.668117+0000","last_change":"2024-09-02T10:19:06.888078+0000","last_active":"2024-09-02T10:19:23.668117+0000","last_peered":"2024-09-02T10:19:23.668117+0000","last_clean":"2024-09-02T10:19:04.053778+0000","last_became_active":"2024-09-02T10:19:06.870452+0000","last_became_peered":"2024-09-02T10:19:06.870452+0000","last_unstale":"2024-09-02T10:19:23.668117+0000","last_undegraded":"2024-09-02T10:19:23.668117+0000","last_fullsized":"2024-09-02T10:19:06.061691+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223543296,"num_objects":3411,"num_object_clones":0,"num_object_copies":10233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":801,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3411,"num_write_kb":218304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1310,"num_bytes_recovered":85852160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[1,5],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5","objects":2610},{"shards":"1,5,7","objects":801}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1165,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.056570+0000","last_change":"2024-09-02T10:17:50.300169+0000","last_active":"2024-09-02T10:19:14.056570+0000","last_peered":"2024-09-02T10:19:14.056570+0000","last_clean":"2024-09-02T10:19:14.056570+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:19:14.056570+0000","last_undegraded":"2024-09-02T10:19:14.056570+0000","last_fullsized":"2024-09-02T10:19:14.056570+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:17:50.300100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:17:50.300100+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"554'3296","reported_seq":3348,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:22.161361+0000","last_change":"2024-09-02T10:18:37.062429+0000","last_active":"2024-09-02T10:19:22.161361+0000","last_peered":"2024-09-02T10:19:22.161361+0000","last_clean":"2024-09-02T10:19:22.161361+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:19:22.161361+0000","last_undegraded":"2024-09-02T10:19:22.161361+0000","last_fullsized":"2024-09-02T10:19:22.161361+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3296,"log_dups_size":0,"ondisk_log_size":3296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:01.072199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216006656,"num_objects":3296,"num_object_clones":0,"num_object_copies":9888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3296,"num_write_kb":210944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":822,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.058539+0000","last_change":"2024-09-02T10:19:05.058003+0000","last_active":"2024-09-02T10:19:14.058539+0000","last_peered":"2024-09-02T10:19:14.058539+0000","last_clean":"2024-09-02T10:19:14.058539+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:19:14.058539+0000","last_undegraded":"2024-09-02T10:19:14.058539+0000","last_fullsized":"2024-09-02T10:19:14.058539+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.206480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.206480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.672102+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"554'3338","reported_seq":3389,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:23.192733+0000","last_change":"2024-09-02T10:18:37.848522+0000","last_active":"2024-09-02T10:19:23.192733+0000","last_peered":"2024-09-02T10:19:23.192733+0000","last_clean":"2024-09-02T10:19:23.192733+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:19:23.192733+0000","last_undegraded":"2024-09-02T10:19:23.192733+0000","last_fullsized":"2024-09-02T10:19:23.192733+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3338,"log_dups_size":0,"ondisk_log_size":3338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:56.425124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218759168,"num_objects":3338,"num_object_clones":0,"num_object_copies":10014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3338,"num_write_kb":213632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":211,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.058094+0000","last_change":"2024-09-02T10:18:15.017709+0000","last_active":"2024-09-02T10:19:14.058094+0000","last_peered":"2024-09-02T10:19:14.058094+0000","last_clean":"2024-09-02T10:19:14.058094+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:19:14.058094+0000","last_undegraded":"2024-09-02T10:19:14.058094+0000","last_fullsized":"2024-09-02T10:19:14.058094+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:15.017640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:15.017640+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:27.366269+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"554'3210","reported_seq":3262,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:21.206220+0000","last_change":"2024-09-02T10:18:37.850883+0000","last_active":"2024-09-02T10:19:21.206220+0000","last_peered":"2024-09-02T10:19:21.206220+0000","last_clean":"2024-09-02T10:19:21.206220+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:19:21.206220+0000","last_undegraded":"2024-09-02T10:19:21.206220+0000","last_fullsized":"2024-09-02T10:19:21.206220+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3210,"log_dups_size":0,"ondisk_log_size":3210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:05.087393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210370560,"num_objects":3210,"num_object_clones":0,"num_object_copies":9630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3210,"num_write_kb":205440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1120,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.058433+0000","last_change":"2024-09-02T10:18:44.023062+0000","last_active":"2024-09-02T10:19:14.058433+0000","last_peered":"2024-09-02T10:19:14.058433+0000","last_clean":"2024-09-02T10:19:14.058433+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:19:14.058433+0000","last_undegraded":"2024-09-02T10:19:14.058433+0000","last_fullsized":"2024-09-02T10:19:14.058433+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:44.023008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:44.023008+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:49.645929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"554'3345","reported_seq":6041,"reported_epoch":554,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:19:23.205074+0000","last_change":"2024-09-02T10:19:06.924994+0000","last_active":"2024-09-02T10:19:23.205074+0000","last_peered":"2024-09-02T10:19:23.205074+0000","last_clean":"2024-09-02T10:19:04.052018+0000","last_became_active":"2024-09-02T10:19:06.875023+0000","last_became_peered":"2024-09-02T10:19:06.875023+0000","last_unstale":"2024-09-02T10:19:23.205074+0000","last_undegraded":"2024-09-02T10:19:23.205074+0000","last_fullsized":"2024-09-02T10:19:06.059430+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3345,"log_dups_size":0,"ondisk_log_size":3345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219217920,"num_objects":3345,"num_object_clones":0,"num_object_copies":10035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":814,"num_objects_unfound":0,"num_objects_dirty":3345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3345,"num_write_kb":214080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1321,"num_bytes_recovered":86573056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[3,11],"avail_no_missing":["3","8","11"],"object_location_counts":[{"shards":"3,8,11","objects":814},{"shards":"3,11","objects":2531}],"blocked_by":[],"up_primary":4,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":587,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.066687+0000","last_change":"2024-09-02T10:18:41.761050+0000","last_active":"2024-09-02T10:19:14.066687+0000","last_peered":"2024-09-02T10:19:14.066687+0000","last_clean":"2024-09-02T10:19:14.066687+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:19:14.066687+0000","last_undegraded":"2024-09-02T10:19:14.066687+0000","last_fullsized":"2024-09-02T10:19:14.066687+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:41.760998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:41.760998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:57.499320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"554'2960","reported_seq":3012,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:19.360072+0000","last_change":"2024-09-02T10:18:37.600687+0000","last_active":"2024-09-02T10:19:19.360072+0000","last_peered":"2024-09-02T10:19:19.360072+0000","last_clean":"2024-09-02T10:19:19.360072+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:19:19.360072+0000","last_undegraded":"2024-09-02T10:19:19.360072+0000","last_fullsized":"2024-09-02T10:19:19.360072+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":2960,"log_dups_size":0,"ondisk_log_size":2960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:59.230189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193986560,"num_objects":2960,"num_object_clones":0,"num_object_copies":8880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2960,"num_write_kb":189440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":559,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.066178+0000","last_change":"2024-09-02T10:18:11.764437+0000","last_active":"2024-09-02T10:19:14.066178+0000","last_peered":"2024-09-02T10:19:14.066178+0000","last_clean":"2024-09-02T10:19:14.066178+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:19:14.066178+0000","last_undegraded":"2024-09-02T10:19:14.066178+0000","last_fullsized":"2024-09-02T10:19:14.066178+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:11.764362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:11.764362+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:19.805086+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"554'3313","reported_seq":3364,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:22.880087+0000","last_change":"2024-09-02T10:18:37.065153+0000","last_active":"2024-09-02T10:19:22.880087+0000","last_peered":"2024-09-02T10:19:22.880087+0000","last_clean":"2024-09-02T10:19:22.880087+0000","last_became_active":"2024-09-02T10:18:37.064844+0000","last_became_peered":"2024-09-02T10:18:37.064844+0000","last_unstale":"2024-09-02T10:19:22.880087+0000","last_undegraded":"2024-09-02T10:19:22.880087+0000","last_fullsized":"2024-09-02T10:19:22.880087+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3313,"log_dups_size":0,"ondisk_log_size":3313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:42.124650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217120768,"num_objects":3313,"num_object_clones":0,"num_object_copies":9939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3313,"num_write_kb":212032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":311,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.053735+0000","last_change":"2024-09-02T10:18:37.859095+0000","last_active":"2024-09-02T10:19:14.053735+0000","last_peered":"2024-09-02T10:19:14.053735+0000","last_clean":"2024-09-02T10:19:14.053735+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:19:14.053735+0000","last_undegraded":"2024-09-02T10:19:14.053735+0000","last_fullsized":"2024-09-02T10:19:14.053735+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:37.859053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:37.859053+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:57.896931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"554'3389","reported_seq":5857,"reported_epoch":554,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:19:23.209878+0000","last_change":"2024-09-02T10:19:06.888347+0000","last_active":"2024-09-02T10:19:23.209878+0000","last_peered":"2024-09-02T10:19:23.209878+0000","last_clean":"2024-09-02T10:19:04.040745+0000","last_became_active":"2024-09-02T10:19:06.870623+0000","last_became_peered":"2024-09-02T10:19:06.870623+0000","last_unstale":"2024-09-02T10:19:23.209878+0000","last_undegraded":"2024-09-02T10:19:06.071144+0000","last_fullsized":"2024-09-02T10:19:06.059552+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3389,"log_dups_size":0,"ondisk_log_size":3389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222101504,"num_objects":3389,"num_object_clones":0,"num_object_copies":10167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":966,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3389,"num_write_kb":216896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1207,"num_bytes_recovered":79101952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,7],"avail_no_missing":["3","6","7"],"object_location_counts":[{"shards":"3,6,7","objects":966},{"shards":"3,7","objects":2422}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":560,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.066133+0000","last_change":"2024-09-02T10:18:20.714703+0000","last_active":"2024-09-02T10:19:14.066133+0000","last_peered":"2024-09-02T10:19:14.066133+0000","last_clean":"2024-09-02T10:19:14.066133+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:19:14.066133+0000","last_undegraded":"2024-09-02T10:19:14.066133+0000","last_fullsized":"2024-09-02T10:19:14.066133+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:20.714649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:20.714649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:35.390231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"554'3149","reported_seq":3200,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:21.214041+0000","last_change":"2024-09-02T10:18:37.848300+0000","last_active":"2024-09-02T10:19:21.214041+0000","last_peered":"2024-09-02T10:19:21.214041+0000","last_clean":"2024-09-02T10:19:21.214041+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:19:21.214041+0000","last_undegraded":"2024-09-02T10:19:21.214041+0000","last_fullsized":"2024-09-02T10:19:21.214041+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3149,"log_dups_size":0,"ondisk_log_size":3149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:01.700433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206372864,"num_objects":3149,"num_object_clones":0,"num_object_copies":9447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3149,"num_write_kb":201536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":813,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.056782+0000","last_change":"2024-09-02T10:19:05.058925+0000","last_active":"2024-09-02T10:19:14.056782+0000","last_peered":"2024-09-02T10:19:14.056782+0000","last_clean":"2024-09-02T10:19:14.056782+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:19:14.056782+0000","last_undegraded":"2024-09-02T10:19:14.056782+0000","last_fullsized":"2024-09-02T10:19:14.056782+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.789947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.789947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:10.614688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"555'3378","reported_seq":3430,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-09-02T10:19:23.647187+0000","last_change":"2024-09-02T10:18:37.064957+0000","last_active":"2024-09-02T10:19:23.647187+0000","last_peered":"2024-09-02T10:19:23.647187+0000","last_clean":"2024-09-02T10:19:23.647187+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:19:23.647187+0000","last_undegraded":"2024-09-02T10:19:23.647187+0000","last_fullsized":"2024-09-02T10:19:23.647187+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3378,"log_dups_size":0,"ondisk_log_size":3378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:49.603479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221380608,"num_objects":3378,"num_object_clones":0,"num_object_copies":10134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3378,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3378,"num_write_kb":216192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1083,"reported_epoch":555,"state":"active+clean","last_fresh":"2024-09-02T10:19:23.587603+0000","last_change":"2024-09-02T10:19:05.058662+0000","last_active":"2024-09-02T10:19:23.587603+0000","last_peered":"2024-09-02T10:19:23.587603+0000","last_clean":"2024-09-02T10:19:23.587603+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:19:23.587603+0000","last_undegraded":"2024-09-02T10:19:23.587603+0000","last_fullsized":"2024-09-02T10:19:23.587603+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:42.912082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:42.912082+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:51.861067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"554'3240","reported_seq":3295,"reported_epoch":554,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:19:22.200230+0000","last_change":"2024-09-02T10:19:06.881012+0000","last_active":"2024-09-02T10:19:22.200230+0000","last_peered":"2024-09-02T10:19:22.200230+0000","last_clean":"2024-09-02T10:19:04.021218+0000","last_became_active":"2024-09-02T10:19:06.874629+0000","last_became_peered":"2024-09-02T10:19:06.874629+0000","last_unstale":"2024-09-02T10:19:22.200230+0000","last_undegraded":"2024-09-02T10:19:06.069315+0000","last_fullsized":"2024-09-02T10:19:06.060341+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3240,"log_dups_size":0,"ondisk_log_size":3240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212336640,"num_objects":3240,"num_object_clones":0,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3240,"num_write_kb":207360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":2118},{"shards":"3,5","objects":1122}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":825,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.056666+0000","last_change":"2024-09-02T10:18:37.852037+0000","last_active":"2024-09-02T10:19:14.056666+0000","last_peered":"2024-09-02T10:19:14.056666+0000","last_clean":"2024-09-02T10:19:14.056666+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:19:14.056666+0000","last_undegraded":"2024-09-02T10:19:14.056666+0000","last_fullsized":"2024-09-02T10:19:14.056666+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:37.851984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:37.851984+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:46.708384+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"554'3175","reported_seq":3226,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:21.209143+0000","last_change":"2024-09-02T10:18:37.850899+0000","last_active":"2024-09-02T10:19:21.209143+0000","last_peered":"2024-09-02T10:19:21.209143+0000","last_clean":"2024-09-02T10:19:21.209143+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:19:21.209143+0000","last_undegraded":"2024-09-02T10:19:21.209143+0000","last_fullsized":"2024-09-02T10:19:21.209143+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3175,"log_dups_size":0,"ondisk_log_size":3175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:02.178728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208076800,"num_objects":3175,"num_object_clones":0,"num_object_copies":9525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3175,"num_write_kb":203200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1080,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.053866+0000","last_change":"2024-09-02T10:19:08.999797+0000","last_active":"2024-09-02T10:19:14.053866+0000","last_peered":"2024-09-02T10:19:14.053866+0000","last_clean":"2024-09-02T10:19:14.053866+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:19:14.053866+0000","last_undegraded":"2024-09-02T10:19:14.053866+0000","last_fullsized":"2024-09-02T10:19:14.053866+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:08.999706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:08.999706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:13.510899+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"554'3237","reported_seq":3289,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:22.857548+0000","last_change":"2024-09-02T10:18:37.600768+0000","last_active":"2024-09-02T10:19:22.857548+0000","last_peered":"2024-09-02T10:19:22.857548+0000","last_clean":"2024-09-02T10:19:22.857548+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:19:22.857548+0000","last_undegraded":"2024-09-02T10:19:22.857548+0000","last_fullsized":"2024-09-02T10:19:22.857548+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3237,"log_dups_size":0,"ondisk_log_size":3237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:54.523794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212140032,"num_objects":3237,"num_object_clones":0,"num_object_copies":9711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3237,"num_write_kb":207168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":30,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.069416+0000","last_change":"2024-09-02T10:19:06.947029+0000","last_active":"2024-09-02T10:19:14.069416+0000","last_peered":"2024-09-02T10:19:14.069416+0000","last_clean":"2024-09-02T10:19:14.069416+0000","last_became_active":"2024-09-02T10:19:06.049121+0000","last_became_peered":"2024-09-02T10:19:06.049121+0000","last_unstale":"2024-09-02T10:19:14.069416+0000","last_undegraded":"2024-09-02T10:19:14.069416+0000","last_fullsized":"2024-09-02T10:19:14.069416+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:06.946957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:06.946957+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:24.543000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"554'3205","reported_seq":3257,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:22.176236+0000","last_change":"2024-09-02T10:18:37.848530+0000","last_active":"2024-09-02T10:19:22.176236+0000","last_peered":"2024-09-02T10:19:22.176236+0000","last_clean":"2024-09-02T10:19:22.176236+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:19:22.176236+0000","last_undegraded":"2024-09-02T10:19:22.176236+0000","last_fullsized":"2024-09-02T10:19:22.176236+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3205,"log_dups_size":0,"ondisk_log_size":3205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:58.128911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210042880,"num_objects":3205,"num_object_clones":0,"num_object_copies":9615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3205,"num_write_kb":205120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":852,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.056852+0000","last_change":"2024-09-02T10:19:05.059514+0000","last_active":"2024-09-02T10:19:14.056852+0000","last_peered":"2024-09-02T10:19:14.056852+0000","last_clean":"2024-09-02T10:19:14.056852+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:19:14.056852+0000","last_undegraded":"2024-09-02T10:19:14.056852+0000","last_fullsized":"2024-09-02T10:19:14.056852+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:10.104220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:10.104220+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:37.010094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"554'3318","reported_seq":3370,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:23.172888+0000","last_change":"2024-09-02T10:18:37.066015+0000","last_active":"2024-09-02T10:19:23.172888+0000","last_peered":"2024-09-02T10:19:23.172888+0000","last_clean":"2024-09-02T10:19:23.172888+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:19:23.172888+0000","last_undegraded":"2024-09-02T10:19:23.172888+0000","last_fullsized":"2024-09-02T10:19:23.172888+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3318,"log_dups_size":0,"ondisk_log_size":3318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:02.695338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217448448,"num_objects":3318,"num_object_clones":0,"num_object_copies":9954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3318,"num_write_kb":212352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":251,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-09-02T10:19:14.057557+0000","last_change":"2024-09-02T10:19:02.956010+0000","last_active":"2024-09-02T10:19:14.057557+0000","last_peered":"2024-09-02T10:19:14.057557+0000","last_clean":"2024-09-02T10:19:14.057557+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:19:14.057557+0000","last_undegraded":"2024-09-02T10:19:14.057557+0000","last_fullsized":"2024-09-02T10:19:14.057557+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:02.955947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:02.955947+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:26.242539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":3419144192,"num_objects":52172,"num_object_clones":0,"num_object_copies":156516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":2581,"num_objects_unfound":0,"num_objects_dirty":52172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52172,"num_write_kb":3339008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3838,"num_bytes_recovered":251527168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10384048128,"data_stored":10384048128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52172,"ondisk_log_size":52172,"up":48,"acting":44,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383911,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1074024,"kb_used_data":623576,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92671128,"statfs":{"total":95995035648,"available":94895235072,"internally_reserved":0,"allocated":638541824,"data_stored":638163313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":461240772},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612844,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":935976,"kb_used_data":611864,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92809176,"statfs":{"total":95995035648,"available":95036596224,"internally_reserved":0,"allocated":626548736,"data_stored":626170225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841777,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1309672,"kb_used_data":837016,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92435480,"statfs":{"total":95995035648,"available":94653931520,"internally_reserved":0,"allocated":857104384,"data_stored":856725873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":483981760},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070710,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":819624,"kb_used_data":513560,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92925528,"statfs":{"total":95995035648,"available":95155740672,"internally_reserved":0,"allocated":525885440,"data_stored":525506929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299643,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1617072,"kb_used_data":1148576,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92128080,"statfs":{"total":95995035648,"available":94339153920,"internally_reserved":0,"allocated":1176141824,"data_stored":1175759761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":479721930},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528575,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1290088,"kb_used_data":961816,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92455064,"statfs":{"total":95995035648,"available":94673985536,"internally_reserved":0,"allocated":984899584,"data_stored":984521073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":336132551},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724804,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2166704,"kb_used_data":1657952,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":91578448,"statfs":{"total":95995035648,"available":93776330752,"internally_reserved":0,"allocated":1697742848,"data_stored":1697360785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953737,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":691560,"kb_used_data":403544,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93053592,"statfs":{"total":95995035648,"available":95286878208,"internally_reserved":0,"allocated":413229056,"data_stored":412850545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149967,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2149672,"kb_used_data":1699224,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91595480,"statfs":{"total":95995035648,"available":93793771520,"internally_reserved":0,"allocated":1740005376,"data_stored":1739626865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":461240776},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411602,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575128,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1465584,"kb_used_data":992928,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92279568,"statfs":{"total":95995035648,"available":94494277632,"internally_reserved":0,"allocated":1016758272,"data_stored":1016376209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738654,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1163368,"kb_used_data":626840,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92581784,"statfs":{"total":95995035648,"available":94803746816,"internally_reserved":0,"allocated":641884160,"data_stored":641505649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":642711552,"data_stored":642711552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1027276800,"data_stored":1027276800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1749417984,"data_stored":1749417984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":419823616,"data_stored":419823616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1709768704,"data_stored":1709768704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":996212736,"data_stored":996212736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1182793728,"data_stored":1182793728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":529006592,"data_stored":529006592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":856555520,"data_stored":856555520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":628097024,"data_stored":628097024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":642383872,"data_stored":642383872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:19:25.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:19:25.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 45 15 56282 56267 78.1424 76.5625 0.00543709 0.0127876 2024-09-02T10:19:26.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:26.037 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:26.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:26.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:26.140 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:26.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:26.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:26.166 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:19:26.166 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:19:26.196 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:19:26.196 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:19:26.226 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:19:26.226 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:19:26.255 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:19:26.255 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:19:26.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:26.283 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:19:26.283 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:19:26.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:26.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:26.315 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:19:26.315 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:19:26.343 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:19:26.344 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:19:26.371 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:19:26.371 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:19:26.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:26.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:19:26.400 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:19:26.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:26.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:26.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:26.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:19:26.434 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:19:26.466 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:19:26.467 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:19:26.494 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:19:26.495 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:19:26.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:26.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:19:26.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:26.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:26.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:26.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:26.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:26 smithi151 ceph-mon[16639]: from='client.17163 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:26 smithi151 ceph-mon[16639]: pgmap v1289: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s; 2119/157593 objects degraded (1.345%); 2581/157593 objects misplaced (1.638%); 20 MiB/s, 319 objects/s recovering 2024-09-02T10:19:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/247193941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:26.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:26.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:26.692 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:26.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:26.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:26.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:26.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:26.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:26 smithi162 ceph-mon[20279]: from='client.17163 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:26 smithi162 ceph-mon[20279]: pgmap v1289: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s; 2119/157593 objects degraded (1.345%); 2581/157593 objects misplaced (1.638%); 20 MiB/s, 319 objects/s recovering 2024-09-02T10:19:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/247193941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:26.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:26.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 46 16 57356 57340 77.9014 67.0625 0.0113459 0.012829 2024-09-02T10:19:26.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:26.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:26.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:26 smithi187 ceph-mon[20072]: from='client.17163 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:26 smithi187 ceph-mon[20072]: pgmap v1289: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.2 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 71 MiB/s wr, 1.13k op/s; 2119/157593 objects degraded (1.345%); 2581/157593 objects misplaced (1.638%); 20 MiB/s, 319 objects/s recovering 2024-09-02T10:19:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/247193941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:27.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:27.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:27.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:27.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:27.155 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:19:27.155 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:19:27.181 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:27.202 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:27.203 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:27.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:27.332 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:27.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:27.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:27.463 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:27.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:27.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:27.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:27.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:27.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:27.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:27.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:27.719 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:27.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:27.821 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:27.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:27.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:27.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:27.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 47 16 58231 58215 77.4074 54.6875 0.0194443 0.0129129 2024-09-02T10:19:27.952 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:27.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:27.978 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:28.079 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:28.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:28.104 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:28.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:28.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:28.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:28.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:19:28.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:28.350 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:28.351 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:28.452 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:28.477 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:28.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:28.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:28.615 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:28.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:28.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:28.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:28.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:28 smithi162 ceph-mon[20279]: pgmap v1290: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 2119/165141 objects degraded (1.283%); 2581/165141 objects misplaced (1.563%); 10 MiB/s, 163 objects/s recovering 2024-09-02T10:19:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:28 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:19:28.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:28.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:28.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:28 smithi151 ceph-mon[16639]: pgmap v1290: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 2119/165141 objects degraded (1.283%); 2581/165141 objects misplaced (1.563%); 10 MiB/s, 163 objects/s recovering 2024-09-02T10:19:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:28 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:19:28.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 48 16 59252 59236 77.1241 63.8125 0.00904022 0.0129614 2024-09-02T10:19:28.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:29.004 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:29.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:29.005 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:28 smithi187 ceph-mon[20072]: pgmap v1290: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.4 GiB data, 15 GiB used, 969 GiB / 983 GiB avail; 69 MiB/s wr, 1.10k op/s; 2119/165141 objects degraded (1.283%); 2581/165141 objects misplaced (1.563%); 10 MiB/s, 163 objects/s recovering 2024-09-02T10:19:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:28 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:19:29.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:29.136 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:29.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:29.216 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:29.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:29.267 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:29.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:29.348 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:29.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:29.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:29.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:29.504 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:29.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:29.526 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:29.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:29.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:29.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:29.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:29.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:29.783 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:29 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2119/165141 objects degraded (1.283%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:29.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:29.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:29.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:29.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:29 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2119/165141 objects degraded (1.283%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:29.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 49 16 60206 60190 76.7668 59.625 0.00270604 0.0130087 2024-09-02T10:19:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:29 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2119/165141 objects degraded (1.283%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:19:30.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:30.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:30.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:30.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:30.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:30.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:30.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:30.157 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:30.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:30.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:30.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:30.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:30.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:30.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:30.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:30.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:30.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:30.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:30.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:30.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:30.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:30.679 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:30.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:30.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:30 smithi151 ceph-mon[16639]: pgmap v1291: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s; 2119/172869 objects degraded (1.226%); 961/172869 objects misplaced (0.556%); 23 MiB/s, 366 objects/s recovering 2024-09-02T10:19:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:19:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:30 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:19:30.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:30.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:30.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:30 smithi162 ceph-mon[20279]: pgmap v1291: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s; 2119/172869 objects degraded (1.226%); 961/172869 objects misplaced (0.556%); 23 MiB/s, 366 objects/s recovering 2024-09-02T10:19:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:19:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:30 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:19:30.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:30.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:30.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:30.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:30.950 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 50 16 60953 60937 76.1651 46.6875 0.00544 0.0130821 2024-09-02T10:19:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:30 smithi187 ceph-mon[20072]: pgmap v1291: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.5 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 77 MiB/s wr, 1.24k op/s; 2119/172869 objects degraded (1.226%); 961/172869 objects misplaced (0.556%); 23 MiB/s, 366 objects/s recovering 2024-09-02T10:19:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:19:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:30 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:19:31.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:31.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:31.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:31.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:31.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:31.173 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:31.174 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:31.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:19:31.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:31.296 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:31.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:31.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:31.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:31.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:31.422 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:31.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:31.523 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:31.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:31.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:31.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:31.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:31.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:31.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:31.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:31.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:31.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:31.803 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:31 smithi162 ceph-mon[20279]: osdmap e557: 12 total, 12 up, 11 in 2024-09-02T10:19:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:19:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:19:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:19:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:31 smithi162 ceph-mon[20279]: osdmap e558: 12 total, 12 up, 11 in 2024-09-02T10:19:31.904 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:31.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:31 smithi151 ceph-mon[16639]: osdmap e557: 12 total, 12 up, 11 in 2024-09-02T10:19:31.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:19:31.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:19:31.911 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:19:31.911 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:31 smithi151 ceph-mon[16639]: osdmap e558: 12 total, 12 up, 11 in 2024-09-02T10:19:31.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:31.926 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:31.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:31.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 51 16 61202 61186 74.9768 15.5625 0.00312352 0.013271 2024-09-02T10:19:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:31 smithi187 ceph-mon[20072]: osdmap e557: 12 total, 12 up, 11 in 2024-09-02T10:19:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:19:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:19:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:19:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:31 smithi187 ceph-mon[20072]: osdmap e558: 12 total, 12 up, 11 in 2024-09-02T10:19:32.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:32.039 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:32.040 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:32.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:32.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:32.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:32.152 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:19:32.152 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2119/176319 objects degraded (1.202%), 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1444,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":558,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":11,"osd_in_since":1725272292,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":43},{"state_name":"active+recovering+undersized+remapped","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":58773,"data_bytes":3852206624,"bytes_used":16571994112,"bytes_avail":1039373398016,"bytes_total":1055945392128,"degraded_objects":2119,"degraded_total":176319,"degraded_ratio":0.012017990120179902,"misplaced_objects":961,"misplaced_total":176319,"misplaced_ratio":0.0054503485160419477,"recovering_objects_per_sec":262,"recovering_bytes_per_sec":17183150,"recovering_keys_per_sec":0,"num_objects_recovered":2038,"num_bytes_recovered":133562368,"num_keys_recovered":0,"write_bytes_sec":62012792,"read_op_per_sec":0,"write_op_per_sec":946},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":35,"modified":"2024-09-02T10:18:59.332463+0000","services":{}},"progress_events":{}} 2024-09-02T10:19:32.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:32.173 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:32.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:32.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:32.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:32.296 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:32.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:32.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:32.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:32.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:32.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:32.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:32.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:32.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:32.545 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:32.585 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:19:32.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:32.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:32.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:32.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:32.772 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:32.795 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:32.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:32.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:32 smithi151 ceph-mon[16639]: pgmap v1293: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 59 MiB/s wr, 946 op/s; 2119/176319 objects degraded (1.202%); 961/176319 objects misplaced (0.545%); 16 MiB/s, 262 objects/s recovering 2024-09-02T10:19:32.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4173112335' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:32.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2450622580' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:19:32.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:32 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:19:32.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4173112335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:19:32.815 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:32 smithi151 ceph-mon[16639]: osdmap e559: 12 total, 12 up, 11 in 2024-09-02T10:19:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:32 smithi162 ceph-mon[20279]: pgmap v1293: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 59 MiB/s wr, 946 op/s; 2119/176319 objects degraded (1.202%); 961/176319 objects misplaced (0.545%); 16 MiB/s, 262 objects/s recovering 2024-09-02T10:19:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4173112335' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2450622580' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:19:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:32 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:19:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4173112335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:19:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:32 smithi162 ceph-mon[20279]: osdmap e559: 12 total, 12 up, 11 in 2024-09-02T10:19:32.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:32.918 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:32.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:32.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 52 16 61415 61399 73.7908 13.3125 0.00246094 0.0134866 2024-09-02T10:19:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:32 smithi187 ceph-mon[20072]: pgmap v1293: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 43 active+clean; 3.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 59 MiB/s wr, 946 op/s; 2119/176319 objects degraded (1.202%); 961/176319 objects misplaced (0.545%); 16 MiB/s, 262 objects/s recovering 2024-09-02T10:19:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4173112335' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2450622580' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:19:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:32 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:19:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4173112335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:19:33.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:32 smithi187 ceph-mon[20072]: osdmap e559: 12 total, 12 up, 11 in 2024-09-02T10:19:33.019 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:33.040 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:33.041 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:33.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:33.163 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:33.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:33.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:33.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:33.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:33.390 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:33.412 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:33.413 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:33.507 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:19:33.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:19:33.514 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:33.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:33.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:33.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:33.663 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:33.664 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:33.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:33.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:33.791 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:33.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:33 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:19:33.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:33 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:19:33.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:33 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:19:33.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:33 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:19:33.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:33 smithi151 ceph-mon[16639]: osdmap e560: 12 total, 12 up, 11 in 2024-09-02T10:19:33.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:33.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:33.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:33.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 53 16 61980 61964 73.0648 35.3125 0.0107607 0.0136807 2024-09-02T10:19:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:33 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:19:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:33 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:19:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:33 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:19:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:33 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:19:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:33 smithi187 ceph-mon[20072]: osdmap e560: 12 total, 12 up, 11 in 2024-09-02T10:19:34.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:34.043 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:34.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:33 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:19:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:33 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:19:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:33 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:19:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:33 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:19:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:33 smithi162 ceph-mon[20279]: osdmap e560: 12 total, 12 up, 11 in 2024-09-02T10:19:34.144 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:34.168 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:34.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:34.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:34.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:34.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:34.312 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:19:34.312 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:19:34.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:34.425 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:34.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:34.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:34.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:34.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:34.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:34.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:34.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:34.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:34.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:34.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:34.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:34.890 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:19:34.891 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:19:34.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:34 smithi151 ceph-mon[16639]: pgmap v1296: 47 pgs: 1 peering, 3 active+recovering+undersized+remapped, 43 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 927 op/s; 2415/181860 objects misplaced (1.328%); 24 MiB/s, 380 objects/s recovering 2024-09-02T10:19:34.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:34 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2119/176319 objects degraded (1.202%), 2 pgs degraded) 2024-09-02T10:19:34.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2005277729' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:34.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:34.917 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:19:34.917 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:19:34.930 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:34.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:34.943 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:19:34.944 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:19:34.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 54 16 62967 62951 72.854 61.6875 0.00636032 0.0137179 2024-09-02T10:19:34.969 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:19:34.969 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:19:34.992 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:19:34.992 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:19:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:34 smithi187 ceph-mon[20072]: pgmap v1296: 47 pgs: 1 peering, 3 active+recovering+undersized+remapped, 43 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 927 op/s; 2415/181860 objects misplaced (1.328%); 24 MiB/s, 380 objects/s recovering 2024-09-02T10:19:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:34 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2119/176319 objects degraded (1.202%), 2 pgs degraded) 2024-09-02T10:19:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2005277729' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:35.019 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:19:35.020 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:19:35.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:35.047 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:19:35.047 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:19:35.058 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:35.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:34 smithi162 ceph-mon[20279]: pgmap v1296: 47 pgs: 1 peering, 3 active+recovering+undersized+remapped, 43 active+clean; 3.7 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 927 op/s; 2415/181860 objects misplaced (1.328%); 24 MiB/s, 380 objects/s recovering 2024-09-02T10:19:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:34 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2119/176319 objects degraded (1.202%), 2 pgs degraded) 2024-09-02T10:19:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2005277729' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:35.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:35.168 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:19:35.169 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:19:35.190 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:35.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:35.194 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:19:35.194 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:19:35.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:19:35.222 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:19:35.254 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:19:35.254 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:19:35.277 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:19:35.278 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:19:35.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:35.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:35.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:35.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:35.439 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:35.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:35.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:35.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:35.563 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:35.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:35.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:35.687 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:35.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:35.789 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:35.811 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:35.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:35.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:35.935 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:35.935 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:35.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 55 16 63472 63456 72.1032 31.5625 0.00313053 0.0137461 2024-09-02T10:19:36.036 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:36.062 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:36.063 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:36.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:36.187 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:36.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:36.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:36.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:36.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:36.314 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:36.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:36.486 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:36.487 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:36.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:36.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:36.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:36.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:36.745 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:36.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:36.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:36.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:36.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:36 smithi151 ceph-mon[16639]: pgmap v1298: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 46 MiB/s wr, 735 op/s; 1880/184401 objects misplaced (1.020%); 16 MiB/s, 251 objects/s recovering 2024-09-02T10:19:36.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 56 16 64483 64467 71.9439 63.1875 0.0236497 0.013891 2024-09-02T10:19:36.973 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:36.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:36.995 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:36 smithi187 ceph-mon[20072]: pgmap v1298: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 46 MiB/s wr, 735 op/s; 1880/184401 objects misplaced (1.020%); 16 MiB/s, 251 objects/s recovering 2024-09-02T10:19:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:36 smithi162 ceph-mon[20279]: pgmap v1298: 47 pgs: 1 peering, 1 active+recovering+undersized+remapped, 45 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 46 MiB/s wr, 735 op/s; 1880/184401 objects misplaced (1.020%); 16 MiB/s, 251 objects/s recovering 2024-09-02T10:19:37.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:37.118 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:37.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:37.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:37.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:37.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:37.346 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:37.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:37.371 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:37.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:37.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:37.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:37.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:37.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:37.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:37.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:37.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:37.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:37.752 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:37.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:37.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:37.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:37.883 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:37.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:37.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 57 16 65200 65184 71.4678 44.8125 0.0198186 0.0139742 2024-09-02T10:19:37.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:38.006 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:38.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:38.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:38.108 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:38.132 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:38.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:38.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:38.267 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:38.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:38.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:38.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:38.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:38.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:38.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:38.532 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:38.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:38.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:38.655 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:38.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:38.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:38.786 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:38.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:38.889 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:38.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:38.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:38.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:38 smithi151 ceph-mon[16639]: pgmap v1299: 47 pgs: 1 active+recovering+undersized+remapped, 1 peering, 45 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 42 MiB/s wr, 674 op/s; 1436/188454 objects misplaced (0.762%); 17 MiB/s, 273 objects/s recovering 2024-09-02T10:19:38.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 58 16 66308 66292 71.4295 69.25 0.0219576 0.0139929 2024-09-02T10:19:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:38 smithi187 ceph-mon[20072]: pgmap v1299: 47 pgs: 1 active+recovering+undersized+remapped, 1 peering, 45 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 42 MiB/s wr, 674 op/s; 1436/188454 objects misplaced (0.762%); 17 MiB/s, 273 objects/s recovering 2024-09-02T10:19:39.013 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:39.039 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:39.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:38 smithi162 ceph-mon[20279]: pgmap v1299: 47 pgs: 1 active+recovering+undersized+remapped, 1 peering, 45 active+clean; 3.8 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 42 MiB/s wr, 674 op/s; 1436/188454 objects misplaced (0.762%); 17 MiB/s, 273 objects/s recovering 2024-09-02T10:19:39.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:39.172 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:39.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:39.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:39.213 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:39.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:39.305 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:39.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:39.407 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:39.430 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:39.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:39.431 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:39.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:39.564 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:39.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:39.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:39.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:39.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:39.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:39.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:39.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:39.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:39.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:39.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:39.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 59 16 67643 67627 71.6329 83.4375 0.0371975 0.0139539 2024-09-02T10:19:40.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:40.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:40.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:40.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:40.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:40.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:40.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:40.325 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:40.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:40.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:40 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:19:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:40 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:19:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:40 smithi162 ceph-mon[20279]: pgmap v1300: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 49 MiB/s wr, 786 op/s; 1436/194643 objects misplaced (0.738%); 13 MiB/s, 211 objects/s recovering 2024-09-02T10:19:40.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:40.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:40 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:19:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:40 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:19:40.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:40 smithi151 ceph-mon[16639]: pgmap v1300: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 49 MiB/s wr, 786 op/s; 1436/194643 objects misplaced (0.738%); 13 MiB/s, 211 objects/s recovering 2024-09-02T10:19:40.453 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:40.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:40 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:19:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:40 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:19:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:40 smithi187 ceph-mon[20072]: pgmap v1300: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 49 MiB/s wr, 786 op/s; 1436/194643 objects misplaced (0.738%); 13 MiB/s, 211 objects/s recovering 2024-09-02T10:19:40.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:40.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:40.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:40.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:40.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:19:40.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:40.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:40.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:40.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:40.851 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:40.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:40.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:19:40.951411+0000 min lat: 0.00104401 max lat: 2.84044 avg lat: 0.0139489 2024-09-02T10:19:40.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:19:40.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 60 16 68822 68806 71.6671 73.6875 0.00354766 0.0139489 2024-09-02T10:19:40.953 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:40.978 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:40.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:40.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:41.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:41.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:41 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:19:41.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:41 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:19:41.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:41 smithi151 ceph-mon[16639]: osdmap e561: 12 total, 12 up, 11 in 2024-09-02T10:19:41.105 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:41.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:41.106 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:41.204 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:41.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:41.230 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:41.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:41.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:19:41.332 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:41.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:41 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:19:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:41 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:19:41.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:41 smithi162 ceph-mon[20279]: osdmap e561: 12 total, 12 up, 11 in 2024-09-02T10:19:41.347 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:19:41.348 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:19:41.360 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:41.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:41.462 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:41.485 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:41.486 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:41 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:19:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:41 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:19:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:41 smithi187 ceph-mon[20072]: osdmap e561: 12 total, 12 up, 11 in 2024-09-02T10:19:41.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:41.613 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:41.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:41.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:41.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:41.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:41.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:19:41.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:41.870 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:41.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:41.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 61 16 69814 69798 71.5085 62 0.00961376 0.0139798 2024-09-02T10:19:41.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:41.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:41.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:41.997 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:42.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:42.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:42.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:42.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:42 smithi151 ceph-mon[16639]: from='client.26714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:42.169 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:42 smithi151 ceph-mon[16639]: pgmap v1302: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 44 MiB/s wr, 696 op/s; 1436/198585 objects misplaced (0.723%); 7.6 MiB/s, 122 objects/s recovering 2024-09-02T10:19:42.213 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1301,"stamp":"2024-09-02T10:19:40.064416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4271047200,"num_objects":65164,"num_object_clones":0,"num_object_copies":195492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1436,"num_objects_unfound":0,"num_objects_dirty":65164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65162,"num_write_kb":4170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7105,"num_bytes_recovered":466166848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68014,"ondisk_log_size":68014,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":17454312,"kb_used_data":12763128,"kb_used_omap":17,"kb_used_meta":4690990,"kb_avail":1013742360,"statfs":{"total":1055945392128,"available":1038072176640,"internally_reserved":0,"allocated":13069443072,"data_stored":13064982003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":4803574646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,3,2,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":357761024,"num_objects":5459,"num_object_clones":0,"num_object_copies":16377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":-1405,"num_objects_unfound":0,"num_objects_dirty":5459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5459,"num_write_kb":349376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1406,"num_bytes_recovered":92143616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.480247"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":464,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.602969+0000","last_change":"2024-09-02T10:18:43.049145+0000","last_active":"2024-09-02T10:19:33.602969+0000","last_peered":"2024-09-02T10:19:33.602969+0000","last_clean":"2024-09-02T10:19:33.602969+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:19:33.602969+0000","last_undegraded":"2024-09-02T10:19:33.602969+0000","last_fullsized":"2024-09-02T10:19:33.602969+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:43.049077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:43.049077+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:07.603384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":272,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.610010+0000","last_change":"2024-09-02T10:18:19.826521+0000","last_active":"2024-09-02T10:19:33.610010+0000","last_peered":"2024-09-02T10:19:33.610010+0000","last_clean":"2024-09-02T10:19:33.610010+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:19:33.610010+0000","last_undegraded":"2024-09-02T10:19:33.610010+0000","last_fullsized":"2024-09-02T10:19:33.610010+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:19.826468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:19.826468+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":722,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.600778+0000","last_change":"2024-09-02T10:18:16.145420+0000","last_active":"2024-09-02T10:19:33.600778+0000","last_peered":"2024-09-02T10:19:33.600778+0000","last_clean":"2024-09-02T10:19:33.600778+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:19:33.600778+0000","last_undegraded":"2024-09-02T10:19:33.600778+0000","last_fullsized":"2024-09-02T10:19:33.600778+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:16.145346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:16.145346+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:37.101102+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":722,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.600739+0000","last_change":"2024-09-02T10:18:39.045140+0000","last_active":"2024-09-02T10:19:33.600739+0000","last_peered":"2024-09-02T10:19:33.600739+0000","last_clean":"2024-09-02T10:19:33.600739+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:19:33.600739+0000","last_undegraded":"2024-09-02T10:19:33.600739+0000","last_fullsized":"2024-09-02T10:19:33.600739+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:39.045083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:39.045083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:03.744946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":34,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.589283+0000","last_change":"2024-09-02T10:19:05.058923+0000","last_active":"2024-09-02T10:19:33.589283+0000","last_peered":"2024-09-02T10:19:33.589283+0000","last_clean":"2024-09-02T10:19:33.589283+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:19:33.589283+0000","last_undegraded":"2024-09-02T10:19:33.589283+0000","last_fullsized":"2024-09-02T10:19:33.589283+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:01.810089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:01.810089+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:19.405935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":570,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.586720+0000","last_change":"2024-09-02T10:18:17.772780+0000","last_active":"2024-09-02T10:19:33.586720+0000","last_peered":"2024-09-02T10:19:33.586720+0000","last_clean":"2024-09-02T10:19:33.586720+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:19:33.586720+0000","last_undegraded":"2024-09-02T10:19:33.586720+0000","last_fullsized":"2024-09-02T10:19:33.586720+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:17.772723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:17.772723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:37.792033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1095,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.606081+0000","last_change":"2024-09-02T10:19:03.880037+0000","last_active":"2024-09-02T10:19:33.606081+0000","last_peered":"2024-09-02T10:19:33.606081+0000","last_clean":"2024-09-02T10:19:33.606081+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:19:33.606081+0000","last_undegraded":"2024-09-02T10:19:33.606081+0000","last_fullsized":"2024-09-02T10:19:33.606081+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:03.879968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:03.879968+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.678816+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":737,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.600813+0000","last_change":"2024-09-02T10:18:35.059181+0000","last_active":"2024-09-02T10:19:33.600813+0000","last_peered":"2024-09-02T10:19:33.600813+0000","last_clean":"2024-09-02T10:19:33.600813+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:19:33.600813+0000","last_undegraded":"2024-09-02T10:19:33.600813+0000","last_fullsized":"2024-09-02T10:19:33.600813+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:35.059123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:35.059123+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:59.080089+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":737,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.609792+0000","last_change":"2024-09-02T10:18:10.904183+0000","last_active":"2024-09-02T10:19:33.609792+0000","last_peered":"2024-09-02T10:19:33.609792+0000","last_clean":"2024-09-02T10:19:33.609792+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:19:33.609792+0000","last_undegraded":"2024-09-02T10:19:33.609792+0000","last_fullsized":"2024-09-02T10:19:33.609792+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:10.904126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:10.904126+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:36.952028+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":713,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.589445+0000","last_change":"2024-09-02T10:19:05.910750+0000","last_active":"2024-09-02T10:19:33.589445+0000","last_peered":"2024-09-02T10:19:33.589445+0000","last_clean":"2024-09-02T10:19:33.589445+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:19:33.589445+0000","last_undegraded":"2024-09-02T10:19:33.589445+0000","last_fullsized":"2024-09-02T10:19:33.589445+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:05.910691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:05.910691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:20.353104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":556,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.589582+0000","last_change":"2024-09-02T10:17:44.275503+0000","last_active":"2024-09-02T10:19:33.589582+0000","last_peered":"2024-09-02T10:19:33.589582+0000","last_clean":"2024-09-02T10:19:33.589582+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:19:33.589582+0000","last_undegraded":"2024-09-02T10:19:33.589582+0000","last_fullsized":"2024-09-02T10:19:33.589582+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:17:44.275434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:17:44.275434+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1070,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.608155+0000","last_change":"2024-09-02T10:19:00.052912+0000","last_active":"2024-09-02T10:19:33.608155+0000","last_peered":"2024-09-02T10:19:33.608155+0000","last_clean":"2024-09-02T10:19:33.608155+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:19:33.608155+0000","last_undegraded":"2024-09-02T10:19:33.608155+0000","last_fullsized":"2024-09-02T10:19:33.608155+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.052857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.052857+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:29.321752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1005,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.604925+0000","last_change":"2024-09-02T10:18:41.060635+0000","last_active":"2024-09-02T10:19:33.604925+0000","last_peered":"2024-09-02T10:19:33.604925+0000","last_clean":"2024-09-02T10:19:33.604925+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:19:33.604925+0000","last_undegraded":"2024-09-02T10:19:33.604925+0000","last_fullsized":"2024-09-02T10:19:33.604925+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:41.060566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:41.060566+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:47.963172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":984,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.605827+0000","last_change":"2024-09-02T10:18:16.400250+0000","last_active":"2024-09-02T10:19:33.605827+0000","last_peered":"2024-09-02T10:19:33.605827+0000","last_clean":"2024-09-02T10:19:33.605827+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:19:33.605827+0000","last_undegraded":"2024-09-02T10:19:33.605827+0000","last_fullsized":"2024-09-02T10:19:33.605827+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:16.400175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:16.400175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":975,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.606171+0000","last_change":"2024-09-02T10:18:40.069068+0000","last_active":"2024-09-02T10:19:33.606171+0000","last_peered":"2024-09-02T10:19:33.606171+0000","last_clean":"2024-09-02T10:19:33.606171+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:19:33.606171+0000","last_undegraded":"2024-09-02T10:19:33.606171+0000","last_fullsized":"2024-09-02T10:19:33.606171+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:40.069013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:40.069013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:45.996592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"560'4018","reported_seq":4081,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:36.763102+0000","last_change":"2024-09-02T10:18:37.599555+0000","last_active":"2024-09-02T10:19:36.763102+0000","last_peered":"2024-09-02T10:19:36.763102+0000","last_clean":"2024-09-02T10:19:36.763102+0000","last_became_active":"2024-09-02T10:18:37.599064+0000","last_became_peered":"2024-09-02T10:18:37.599064+0000","last_unstale":"2024-09-02T10:19:36.763102+0000","last_undegraded":"2024-09-02T10:19:36.763102+0000","last_fullsized":"2024-09-02T10:19:36.763102+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:50.285721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263323648,"num_objects":4018,"num_object_clones":0,"num_object_copies":12054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4018,"num_write_kb":257152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1062,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.606184+0000","last_change":"2024-09-02T10:18:13.075935+0000","last_active":"2024-09-02T10:19:33.606184+0000","last_peered":"2024-09-02T10:19:33.606184+0000","last_clean":"2024-09-02T10:19:33.606184+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:19:33.606184+0000","last_undegraded":"2024-09-02T10:19:33.606184+0000","last_fullsized":"2024-09-02T10:19:33.606184+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:13.075888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:13.075888+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:39.573123+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"560'4206","reported_seq":8502,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:39.153156+0000","last_change":"2024-09-02T10:19:32.601261+0000","last_active":"2024-09-02T10:19:39.153156+0000","last_peered":"2024-09-02T10:19:39.153156+0000","last_clean":"2024-09-02T10:19:39.153156+0000","last_became_active":"2024-09-02T10:19:32.596751+0000","last_became_peered":"2024-09-02T10:19:32.596751+0000","last_unstale":"2024-09-02T10:19:39.153156+0000","last_undegraded":"2024-09-02T10:19:39.153156+0000","last_fullsized":"2024-09-02T10:19:39.153156+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4206,"log_dups_size":0,"ondisk_log_size":4206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:50.137401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275644416,"num_objects":4206,"num_object_clones":0,"num_object_copies":12618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4206,"num_write_kb":269184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":138346496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1177,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.589261+0000","last_change":"2024-09-02T10:17:50.300169+0000","last_active":"2024-09-02T10:19:33.589261+0000","last_peered":"2024-09-02T10:19:33.589261+0000","last_clean":"2024-09-02T10:19:33.589261+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:19:33.589261+0000","last_undegraded":"2024-09-02T10:19:33.589261+0000","last_fullsized":"2024-09-02T10:19:33.589261+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:17:50.300100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:17:50.300100+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:37.849900+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"560'4018","reported_seq":4082,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:37.161688+0000","last_change":"2024-09-02T10:18:37.062429+0000","last_active":"2024-09-02T10:19:37.161688+0000","last_peered":"2024-09-02T10:19:37.161688+0000","last_clean":"2024-09-02T10:19:37.161688+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:19:37.161688+0000","last_undegraded":"2024-09-02T10:19:37.161688+0000","last_fullsized":"2024-09-02T10:19:37.161688+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:01.072199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263323648,"num_objects":4018,"num_object_clones":0,"num_object_copies":12054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4018,"num_write_kb":257152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":834,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.604754+0000","last_change":"2024-09-02T10:19:05.058003+0000","last_active":"2024-09-02T10:19:33.604754+0000","last_peered":"2024-09-02T10:19:33.604754+0000","last_clean":"2024-09-02T10:19:33.604754+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:19:33.604754+0000","last_undegraded":"2024-09-02T10:19:33.604754+0000","last_fullsized":"2024-09-02T10:19:33.604754+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.206480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.206480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.672102+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"560'4035","reported_seq":4099,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:38.206077+0000","last_change":"2024-09-02T10:18:37.848522+0000","last_active":"2024-09-02T10:19:38.206077+0000","last_peered":"2024-09-02T10:19:38.206077+0000","last_clean":"2024-09-02T10:19:38.206077+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:19:38.206077+0000","last_undegraded":"2024-09-02T10:19:38.206077+0000","last_fullsized":"2024-09-02T10:19:38.206077+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4035,"log_dups_size":0,"ondisk_log_size":4035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:56.425124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264437760,"num_objects":4035,"num_object_clones":0,"num_object_copies":12105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4035,"num_write_kb":258240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":223,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.602804+0000","last_change":"2024-09-02T10:18:15.017709+0000","last_active":"2024-09-02T10:19:33.602804+0000","last_peered":"2024-09-02T10:19:33.602804+0000","last_clean":"2024-09-02T10:19:33.602804+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:19:33.602804+0000","last_undegraded":"2024-09-02T10:19:33.602804+0000","last_fullsized":"2024-09-02T10:19:33.602804+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:15.017640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:15.017640+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:35.958303+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"560'4008","reported_seq":4072,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:36.202380+0000","last_change":"2024-09-02T10:18:37.850883+0000","last_active":"2024-09-02T10:19:36.202380+0000","last_peered":"2024-09-02T10:19:36.202380+0000","last_clean":"2024-09-02T10:19:36.202380+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:19:36.202380+0000","last_undegraded":"2024-09-02T10:19:36.202380+0000","last_fullsized":"2024-09-02T10:19:36.202380+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4008,"log_dups_size":0,"ondisk_log_size":4008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:05.087393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262668288,"num_objects":4008,"num_object_clones":0,"num_object_copies":12024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4008,"num_write_kb":256512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1132,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.602775+0000","last_change":"2024-09-02T10:18:44.023062+0000","last_active":"2024-09-02T10:19:33.602775+0000","last_peered":"2024-09-02T10:19:33.602775+0000","last_clean":"2024-09-02T10:19:33.602775+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:19:33.602775+0000","last_undegraded":"2024-09-02T10:19:33.602775+0000","last_fullsized":"2024-09-02T10:19:33.602775+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:44.023008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:44.023008+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:49.645929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"560'4172","reported_seq":8518,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:39.179796+0000","last_change":"2024-09-02T10:19:31.591657+0000","last_active":"2024-09-02T10:19:39.179796+0000","last_peered":"2024-09-02T10:19:39.179796+0000","last_clean":"2024-09-02T10:19:39.179796+0000","last_became_active":"2024-09-02T10:19:31.588084+0000","last_became_peered":"2024-09-02T10:19:31.588084+0000","last_unstale":"2024-09-02T10:19:39.179796+0000","last_undegraded":"2024-09-02T10:19:39.179796+0000","last_fullsized":"2024-09-02T10:19:39.179796+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:48.891698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273416192,"num_objects":4172,"num_object_clones":0,"num_object_copies":12516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4172,"num_write_kb":267008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2135,"num_bytes_recovered":139919360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":599,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.586692+0000","last_change":"2024-09-02T10:18:41.761050+0000","last_active":"2024-09-02T10:19:33.586692+0000","last_peered":"2024-09-02T10:19:33.586692+0000","last_clean":"2024-09-02T10:19:33.586692+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:19:33.586692+0000","last_undegraded":"2024-09-02T10:19:33.586692+0000","last_fullsized":"2024-09-02T10:19:33.586692+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:41.760998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:41.760998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:57.499320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"560'4105","reported_seq":4169,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:39.396715+0000","last_change":"2024-09-02T10:18:37.600687+0000","last_active":"2024-09-02T10:19:39.396715+0000","last_peered":"2024-09-02T10:19:39.396715+0000","last_clean":"2024-09-02T10:19:39.396715+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:19:39.396715+0000","last_undegraded":"2024-09-02T10:19:39.396715+0000","last_fullsized":"2024-09-02T10:19:39.396715+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4105,"log_dups_size":0,"ondisk_log_size":4105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:59.230189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269025280,"num_objects":4105,"num_object_clones":0,"num_object_copies":12315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4105,"num_write_kb":262720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":571,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.586710+0000","last_change":"2024-09-02T10:18:11.764437+0000","last_active":"2024-09-02T10:19:33.586710+0000","last_peered":"2024-09-02T10:19:33.586710+0000","last_clean":"2024-09-02T10:19:33.586710+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:19:33.586710+0000","last_undegraded":"2024-09-02T10:19:33.586710+0000","last_fullsized":"2024-09-02T10:19:33.586710+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:11.764362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:11.764362+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:36.776026+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"560'4107","reported_seq":4171,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:37.893093+0000","last_change":"2024-09-02T10:18:37.065153+0000","last_active":"2024-09-02T10:19:37.893093+0000","last_peered":"2024-09-02T10:19:37.893093+0000","last_clean":"2024-09-02T10:19:37.893093+0000","last_became_active":"2024-09-02T10:18:37.064844+0000","last_became_peered":"2024-09-02T10:18:37.064844+0000","last_unstale":"2024-09-02T10:19:37.893093+0000","last_undegraded":"2024-09-02T10:19:37.893093+0000","last_fullsized":"2024-09-02T10:19:37.893093+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4107,"log_dups_size":0,"ondisk_log_size":4107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:42.124650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269156352,"num_objects":4107,"num_object_clones":0,"num_object_copies":12321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4107,"num_write_kb":262848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":323,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.586851+0000","last_change":"2024-09-02T10:18:37.859095+0000","last_active":"2024-09-02T10:19:33.586851+0000","last_peered":"2024-09-02T10:19:33.586851+0000","last_clean":"2024-09-02T10:19:33.586851+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:19:33.586851+0000","last_undegraded":"2024-09-02T10:19:33.586851+0000","last_fullsized":"2024-09-02T10:19:33.586851+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:37.859053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:37.859053+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:57.896931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"560'4119","reported_seq":8540,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:38.197022+0000","last_change":"2024-09-02T10:19:33.596264+0000","last_active":"2024-09-02T10:19:38.197022+0000","last_peered":"2024-09-02T10:19:38.197022+0000","last_clean":"2024-09-02T10:19:38.197022+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:19:38.197022+0000","last_undegraded":"2024-09-02T10:19:38.197022+0000","last_fullsized":"2024-09-02T10:19:38.197022+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4119,"log_dups_size":0,"ondisk_log_size":4119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:55.430237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269942784,"num_objects":4119,"num_object_clones":0,"num_object_copies":12357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4119,"num_write_kb":263616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":572,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.586717+0000","last_change":"2024-09-02T10:18:20.714703+0000","last_active":"2024-09-02T10:19:33.586717+0000","last_peered":"2024-09-02T10:19:33.586717+0000","last_clean":"2024-09-02T10:19:33.586717+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:19:33.586717+0000","last_undegraded":"2024-09-02T10:19:33.586717+0000","last_fullsized":"2024-09-02T10:19:33.586717+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:20.714649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:20.714649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:35.390231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"560'3961","reported_seq":4025,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:36.198418+0000","last_change":"2024-09-02T10:18:37.848300+0000","last_active":"2024-09-02T10:19:36.198418+0000","last_peered":"2024-09-02T10:19:36.198418+0000","last_clean":"2024-09-02T10:19:36.198418+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:19:36.198418+0000","last_undegraded":"2024-09-02T10:19:36.198418+0000","last_fullsized":"2024-09-02T10:19:36.198418+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:01.700433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259588096,"num_objects":3961,"num_object_clones":0,"num_object_copies":11883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3961,"num_write_kb":253504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":826,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.589467+0000","last_change":"2024-09-02T10:19:05.058925+0000","last_active":"2024-09-02T10:19:33.589467+0000","last_peered":"2024-09-02T10:19:33.589467+0000","last_clean":"2024-09-02T10:19:33.589467+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:19:33.589467+0000","last_undegraded":"2024-09-02T10:19:33.589467+0000","last_fullsized":"2024-09-02T10:19:33.589467+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.789947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.789947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:10.614688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"560'4222","reported_seq":4285,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:38.621143+0000","last_change":"2024-09-02T10:18:37.064957+0000","last_active":"2024-09-02T10:19:38.621143+0000","last_peered":"2024-09-02T10:19:38.621143+0000","last_clean":"2024-09-02T10:19:38.621143+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:19:38.621143+0000","last_undegraded":"2024-09-02T10:19:38.621143+0000","last_fullsized":"2024-09-02T10:19:38.621143+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:49.603479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276692992,"num_objects":4222,"num_object_clones":0,"num_object_copies":12666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4222,"num_write_kb":270208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1093,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.608244+0000","last_change":"2024-09-02T10:19:05.058662+0000","last_active":"2024-09-02T10:19:33.608244+0000","last_peered":"2024-09-02T10:19:33.608244+0000","last_clean":"2024-09-02T10:19:33.608244+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:19:33.608244+0000","last_undegraded":"2024-09-02T10:19:33.608244+0000","last_fullsized":"2024-09-02T10:19:33.608244+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:42.912082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:42.912082+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:51.861067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"560'4056","reported_seq":5488,"reported_epoch":560,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:19:37.203160+0000","last_change":"2024-09-02T10:19:29.964931+0000","last_active":"2024-09-02T10:19:37.203160+0000","last_peered":"2024-09-02T10:19:37.203160+0000","last_clean":"2024-09-02T10:19:04.021218+0000","last_became_active":"2024-09-02T10:19:06.874629+0000","last_became_peered":"2024-09-02T10:19:06.874629+0000","last_unstale":"2024-09-02T10:19:37.203160+0000","last_undegraded":"2024-09-02T10:19:37.203160+0000","last_fullsized":"2024-09-02T10:19:06.060341+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4056,"log_dups_size":0,"ondisk_log_size":4056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265814016,"num_objects":4056,"num_object_clones":0,"num_object_copies":12168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1436,"num_objects_unfound":0,"num_objects_dirty":4056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4056,"num_write_kb":259584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":682,"num_bytes_recovered":44695552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":1436},{"shards":"3,5","objects":2620}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":837,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.589335+0000","last_change":"2024-09-02T10:18:37.852037+0000","last_active":"2024-09-02T10:19:33.589335+0000","last_peered":"2024-09-02T10:19:33.589335+0000","last_clean":"2024-09-02T10:19:33.589335+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:19:33.589335+0000","last_undegraded":"2024-09-02T10:19:33.589335+0000","last_fullsized":"2024-09-02T10:19:33.589335+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:37.851984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:37.851984+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:46.708384+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"560'3992","reported_seq":4056,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:36.214456+0000","last_change":"2024-09-02T10:18:37.850899+0000","last_active":"2024-09-02T10:19:36.214456+0000","last_peered":"2024-09-02T10:19:36.214456+0000","last_clean":"2024-09-02T10:19:36.214456+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:19:36.214456+0000","last_undegraded":"2024-09-02T10:19:36.214456+0000","last_fullsized":"2024-09-02T10:19:36.214456+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:02.178728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261619712,"num_objects":3992,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3992,"num_write_kb":255488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1092,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.586888+0000","last_change":"2024-09-02T10:19:08.999797+0000","last_active":"2024-09-02T10:19:33.586888+0000","last_peered":"2024-09-02T10:19:33.586888+0000","last_clean":"2024-09-02T10:19:33.586888+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:19:33.586888+0000","last_undegraded":"2024-09-02T10:19:33.586888+0000","last_fullsized":"2024-09-02T10:19:33.586888+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:08.999706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:08.999706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:13.510899+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"560'4029","reported_seq":4092,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:37.872555+0000","last_change":"2024-09-02T10:18:37.600768+0000","last_active":"2024-09-02T10:19:37.872555+0000","last_peered":"2024-09-02T10:19:37.872555+0000","last_clean":"2024-09-02T10:19:37.872555+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:19:37.872555+0000","last_undegraded":"2024-09-02T10:19:37.872555+0000","last_fullsized":"2024-09-02T10:19:37.872555+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4029,"log_dups_size":0,"ondisk_log_size":4029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:54.523794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264044544,"num_objects":4029,"num_object_clones":0,"num_object_copies":12087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4029,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4029,"num_write_kb":257856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":42,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.589429+0000","last_change":"2024-09-02T10:19:06.947029+0000","last_active":"2024-09-02T10:19:33.589429+0000","last_peered":"2024-09-02T10:19:33.589429+0000","last_clean":"2024-09-02T10:19:33.589429+0000","last_became_active":"2024-09-02T10:19:06.049121+0000","last_became_peered":"2024-09-02T10:19:06.049121+0000","last_unstale":"2024-09-02T10:19:33.589429+0000","last_undegraded":"2024-09-02T10:19:33.589429+0000","last_fullsized":"2024-09-02T10:19:33.589429+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:06.946957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:06.946957+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:24.543000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"560'4040","reported_seq":4104,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:37.196750+0000","last_change":"2024-09-02T10:18:37.848530+0000","last_active":"2024-09-02T10:19:37.196750+0000","last_peered":"2024-09-02T10:19:37.196750+0000","last_clean":"2024-09-02T10:19:37.196750+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:19:37.196750+0000","last_undegraded":"2024-09-02T10:19:37.196750+0000","last_fullsized":"2024-09-02T10:19:37.196750+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4040,"log_dups_size":0,"ondisk_log_size":4040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:58.128911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264765440,"num_objects":4040,"num_object_clones":0,"num_object_copies":12120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4040,"num_write_kb":258560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":864,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.589552+0000","last_change":"2024-09-02T10:19:05.059514+0000","last_active":"2024-09-02T10:19:33.589552+0000","last_peered":"2024-09-02T10:19:33.589552+0000","last_clean":"2024-09-02T10:19:33.589552+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:19:33.589552+0000","last_undegraded":"2024-09-02T10:19:33.589552+0000","last_fullsized":"2024-09-02T10:19:33.589552+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:10.104220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:10.104220+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:37.010094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"560'4074","reported_seq":4138,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:38.162374+0000","last_change":"2024-09-02T10:18:37.066015+0000","last_active":"2024-09-02T10:19:38.162374+0000","last_peered":"2024-09-02T10:19:38.162374+0000","last_clean":"2024-09-02T10:19:38.162374+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:19:38.162374+0000","last_undegraded":"2024-09-02T10:19:38.162374+0000","last_fullsized":"2024-09-02T10:19:38.162374+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:02.695338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266993664,"num_objects":4074,"num_object_clones":0,"num_object_copies":12222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":260736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":263,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-02T10:19:33.588077+0000","last_change":"2024-09-02T10:19:02.956010+0000","last_active":"2024-09-02T10:19:33.588077+0000","last_peered":"2024-09-02T10:19:33.588077+0000","last_clean":"2024-09-02T10:19:33.588077+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:19:33.588077+0000","last_undegraded":"2024-09-02T10:19:33.588077+0000","last_fullsized":"2024-09-02T10:19:33.588077+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:02.955947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:02.955947+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:26.242539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4270456832,"num_objects":65162,"num_object_clones":0,"num_object_copies":195486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1436,"num_objects_unfound":0,"num_objects_dirty":65162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":65162,"num_write_kb":4170368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7102,"num_bytes_recovered":465436672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13114867712,"data_stored":13114867712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65162,"ondisk_log_size":65162,"up":48,"acting":47,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383914,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1235312,"kb_used_data":775136,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92509840,"statfs":{"total":95995035648,"available":94730076160,"internally_reserved":0,"allocated":793739264,"data_stored":793334681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":471202244},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612847,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1080944,"kb_used_data":765152,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92664208,"statfs":{"total":95995035648,"available":94888148992,"internally_reserved":0,"allocated":783515648,"data_stored":783111065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":323353024},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841780,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1509808,"kb_used_data":1023264,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92235344,"statfs":{"total":95995035648,"available":94448992256,"internally_reserved":0,"allocated":1047822336,"data_stored":1047417753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":498203072},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070714,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":944176,"kb_used_data":646432,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92800976,"statfs":{"total":95995035648,"available":95028199424,"internally_reserved":0,"allocated":661946368,"data_stored":661541785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299646,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1885368,"kb_used_data":1398824,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91859784,"statfs":{"total":95995035648,"available":94064418816,"internally_reserved":0,"allocated":1432395776,"data_stored":1431987641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528578,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1369712,"kb_used_data":1027552,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":92375440,"statfs":{"total":95995035648,"available":94592450560,"internally_reserved":0,"allocated":1052213248,"data_stored":1051808665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":350353863},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724807,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2574904,"kb_used_data":2074472,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91170248,"statfs":{"total":95995035648,"available":93358333952,"internally_reserved":0,"allocated":2124259328,"data_stored":2123851193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953741,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302256,"kb_used_data":992032,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92442896,"statfs":{"total":95995035648,"available":94661525504,"internally_reserved":0,"allocated":1015840768,"data_stored":1015436185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149970,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2567472,"kb_used_data":2094816,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":91177680,"statfs":{"total":95995035648,"available":93365944320,"internally_reserved":0,"allocated":2145091584,"data_stored":2144687001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":483981768},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411605,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575131,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1655416,"kb_used_data":1186920,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92089736,"statfs":{"total":95995035648,"available":94299889664,"internally_reserved":0,"allocated":1215406080,"data_stored":1214997945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738657,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1328944,"kb_used_data":778528,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92416208,"statfs":{"total":95995035648,"available":94634196992,"internally_reserved":0,"allocated":797212672,"data_stored":796808089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":800391168,"data_stored":800391168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1226833920,"data_stored":1226833920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2151284736,"data_stored":2151284736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1015283712,"data_stored":1015283712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2125529088,"data_stored":2125529088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1056047104,"data_stored":1056047104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1438777344,"data_stored":1438777344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":663814144,"data_stored":663814144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1057685504,"data_stored":1057685504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":783286272,"data_stored":783286272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":795934720,"data_stored":795934720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:19:42.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:42.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:42.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:42 smithi162 ceph-mon[20279]: from='client.26714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:42 smithi162 ceph-mon[20279]: pgmap v1302: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 44 MiB/s wr, 696 op/s; 1436/198585 objects misplaced (0.723%); 7.6 MiB/s, 122 objects/s recovering 2024-09-02T10:19:42.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:42.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:42.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:42.431 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:19:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:19:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:19:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:19:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11549 4.f 4:fdcceb31:::benchmark_data_smithi187_76542_object11548:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:19:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.466618+0000", 2024-09-02T10:19:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 511.96122303999999, 2024-09-02T10:19:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25702618900000002, 2024-09-02T10:19:42.431 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11549 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466618+0000", 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466618+0000", 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466619+0000", 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6099999999999999e-07 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466662+0000", 2024-09-02T10:19:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2951000000000003e-05 2024-09-02T10:19:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466671+0000", 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9600000000000006e-06 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466676+0000", 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0019999999999997e-06 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466738+0000", 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1978999999999997e-05 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466884+0000", 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014551599999999999 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.467035+0000", 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015158899999999999 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.468034+0000", 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00099856899999999993 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.468192+0000", 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000157918 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.723541+0000", 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25534909900000002 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.723597+0000", 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6008999999999999e-05 2024-09-02T10:19:42.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.723645+0000", 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7636000000000003e-05 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11564 4.f 4:f03605e0:::benchmark_data_smithi187_76542_object11563:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.523090+0000", 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 511.90475133799998, 2024-09-02T10:19:42.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26754456500000001, 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11564 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523090+0000", 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523090+0000", 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523091+0000", 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3069999999999999e-06 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523141+0000", 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9268000000000003e-05 2024-09-02T10:19:42.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523152+0000", 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0915e-05 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523157+0000", 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7450000000000001e-06 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523203+0000", 2024-09-02T10:19:42.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5537000000000001e-05 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523312+0000", 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010900000000000001 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.523417+0000", 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105166 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.524047+0000", 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00063040700000000004 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.524435+0000", 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00038750699999999999 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.790567+0000", 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.266131952 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.790609+0000", 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1706999999999999e-05 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.790635+0000", 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6054000000000002e-05 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:41912 4.f 4:f03c9d73:::benchmark_data_smithi187_76542_object41911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-02T10:19:42.446 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.767259+0000", 2024-09-02T10:19:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 474.660582197, 2024-09-02T10:19:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52722899400000001, 2024-09-02T10:19:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.447 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 41912 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767259+0000", 2024-09-02T10:19:42.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767259+0000", 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767260+0000", 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2999999999999999e-07 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767309+0000", 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8523999999999998e-05 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767321+0000", 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767331+0000", 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.01e-05 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767384+0000", 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3012e-05 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767516+0000", 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131166 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767648+0000", 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000132106 2024-09-02T10:19:42.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.768976+0000", 2024-09-02T10:19:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013279559999999999 2024-09-02T10:19:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.774494+0000", 2024-09-02T10:19:42.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055181090000000002 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294421+0000", 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51992769000000005 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294459+0000", 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7832e-05 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294488+0000", 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8980999999999999e-05 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.18 353/344 [PushOp(4:1c2dd2e1:::benchmark_data_smithi187_76542_object24960:head, version: 335'1638, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1c2dd2e1:::benchmark_data_smithi187_76542_object24960:head@335'1638, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 474.65094846199997, 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29217855399999998, 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.454 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8299999999999997e-07 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.777026+0000", 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013260400000000001 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068430+0000", 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29140371700000001 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068439+0000", 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2909999999999996e-06 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068496+0000", 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6597e-05 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068504+0000", 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2199999999999992e-06 2024-09-02T10:19:42.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.069072+0000", 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056764200000000004 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:44135 4.f 4:fa3c3350:::benchmark_data_smithi187_76542_object44134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:50.893955+0000", 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 471.533886188, 2024-09-02T10:19:42.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32054305900000002, 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 44135 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.893955+0000", 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.459 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.893955+0000", 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.893957+0000", 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.604e-06 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894004+0000", 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6853e-05 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894015+0000", 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.129e-05 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894021+0000", 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6049999999999997e-06 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.461 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894061+0000", 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0114999999999998e-05 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894142+0000", 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1724000000000006e-05 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894228+0000", 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5729000000000005e-05 2024-09-02T10:19:42.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.895543+0000", 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001314747 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900651+0000", 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051081329999999999 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.214312+0000", 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31366078800000002 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.214448+0000", 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013631299999999999 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.214498+0000", 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0158000000000003e-05 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.464 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(4.16 356/345 [PushReplyOp(4:6d00321c:::benchmark_data_smithi187_76542_object7271:head)])", 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:50.900564+0000", 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 471.52727734699999, 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31795428799999997, 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900564+0000", 2024-09-02T10:19:42.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900564+0000", 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900565+0000", 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900569+0000", 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7049999999999998e-06 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.217556+0000", 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31698648099999999 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.217564+0000", 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.943e-06 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.217625+0000", 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0806e-05 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.218518+0000", 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00089372300000000002 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90780 4.f 4:f9a425ff:::benchmark_data_smithi187_76542_object90779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.016735+0000", 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 430.411106222, 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28756859099999998, 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90780 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016735+0000", 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016735+0000", 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016736+0000", 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.540000000000001e-07 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016790+0000", 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3933e-05 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016802+0000", 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1605e-05 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016807+0000", 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0869999999999997e-06 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016846+0000", 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9502000000000002e-05 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016939+0000", 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2520000000000002e-05 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.017047+0000", 2024-09-02T10:19:42.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010813400000000001 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.020060+0000", 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030132090000000002 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.031359+0000", 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011299346 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.473 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304201+0000", 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.272841522 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304259+0000", 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7732000000000002e-05 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304304+0000", 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5046999999999997e-05 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.474 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90797 4.f 4:f65eb26e:::benchmark_data_smithi187_76542_object90796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.028192+0000", 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 430.39964960999998, 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.276265435, 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.475 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90797 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028192+0000", 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028192+0000", 2024-09-02T10:19:42.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028193+0000", 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3740000000000001e-06 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028251+0000", 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8097000000000003e-05 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028264+0000", 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2223e-05 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028270+0000", 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6290000000000004e-06 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028310+0000", 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9527000000000003e-05 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028410+0000", 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9959999999999998e-05 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028501+0000", 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0976000000000004e-05 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.030870+0000", 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002369491 2024-09-02T10:19:42.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034987+0000", 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041169179999999998 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304357+0000", 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26936986000000002 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304419+0000", 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1709000000000002e-05 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304457+0000", 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8671000000000003e-05 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.481 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90800 4.f 4:f0d21d3f:::benchmark_data_smithi187_76542_object90799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.032169+0000", 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 430.39567272300002, 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27243329199999999, 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90800 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032169+0000", 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032169+0000", 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032170+0000", 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7699999999999992e-07 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032221+0000", 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1365e-05 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032233+0000", 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2102e-05 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032240+0000", 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6669999999999999e-06 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032277+0000", 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7024000000000003e-05 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032359+0000", 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2163999999999995e-05 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032440+0000", 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0870000000000003e-05 2024-09-02T10:19:42.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034544+0000", 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021037590000000002 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300042+0000", 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26549787899999999 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304502+0000", 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044609840000000003 2024-09-02T10:19:42.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304561+0000", 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8257999999999997e-05 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304602+0000", 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1242999999999998e-05 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.487 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:90800 4.f e376/318 ondisk, result = 0)", 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 430.39291164700001, 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26511866899999997, 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8500000000000002e-07 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.489 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:42.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034933+0000", 2024-09-02T10:19:42.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2229999999999999e-06 2024-09-02T10:19:42.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.298510+0000", 2024-09-02T10:19:42.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26357608700000001 2024-09-02T10:19:42.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.298516+0000", 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.657e-06 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300029+0000", 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015128190000000001 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300036+0000", 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9280000000000004e-06 2024-09-02T10:19:42.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300048+0000", 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2469999999999999e-05 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:92326 4.b e377/337 4:d3b75331:::benchmark_data_smithi187_76542_object92325:head v 377'5809, mlcod=377'5807)", 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:33.493173+0000", 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 428.93466845199998, 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79507954800000002, 2024-09-02T10:19:42.493 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.493173+0000", 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.493173+0000", 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.493174+0000", 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6499999999999998e-07 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.493198+0000", 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4283000000000001e-05 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.285696+0000", 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79249790399999998 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.285708+0000", 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2272e-05 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.285766+0000", 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7948000000000002e-05 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.285786+0000", 2024-09-02T10:19:42.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9712e-05 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288220+0000", 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024338620000000002 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288253+0000", 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2702e-05 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:103703 4.9 4:93bfdaf4:::benchmark_data_smithi187_76542_object11367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:46.146088+0000", 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 416.28175300200002, 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29017199100000002, 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 103703 2024-09-02T10:19:42.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146088+0000", 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146088+0000", 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146090+0000", 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1170000000000001e-06 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146094+0000", 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146104+0000", 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0169999999999999e-05 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146110+0000", 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5199999999999997e-06 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146157+0000", 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7511e-05 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146239+0000", 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1837999999999998e-05 2024-09-02T10:19:42.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146304+0000", 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4709000000000007e-05 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.147684+0000", 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013800970000000001 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.148414+0000", 2024-09-02T10:19:42.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073035299999999999 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.436148+0000", 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28773412900000001 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.436211+0000", 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2335000000000005e-05 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.436260+0000", 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9598999999999997e-05 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:103765 4.9 4:90c7bfbd:::benchmark_data_smithi187_76542_object11429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:46.738804+0000", 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 415.689037896, 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26836812300000001, 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.504 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 103765 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738804+0000", 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738804+0000", 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738805+0000", 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.579e-06 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738810+0000", 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5310000000000003e-06 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738820+0000", 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0132000000000001e-05 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738826+0000", 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4529999999999999e-06 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738883+0000", 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6373999999999999e-05 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.739018+0000", 2024-09-02T10:19:42.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013532799999999999 2024-09-02T10:19:42.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:19:42.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.739147+0000", 2024-09-02T10:19:42.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012891899999999999 2024-09-02T10:19:42.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.740688+0000", 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015409 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.740758+0000", 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0320000000000004e-05 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:47.007109+0000", 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26635045299999999 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:47.007142+0000", 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3821e-05 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:47.007172+0000", 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9312999999999999e-05 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:166621 4.f 4:f82e681d:::benchmark_data_smithi187_76542_object74285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:01.187965+0000", 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 341.23987629099997, 2024-09-02T10:19:42.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26233720799999999, 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 166621 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187965+0000", 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187965+0000", 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187966+0000", 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2589999999999999e-06 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187970+0000", 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4230000000000001e-06 2024-09-02T10:19:42.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187979+0000", 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2830000000000007e-06 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187984+0000", 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9060000000000001e-06 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.188024+0000", 2024-09-02T10:19:42.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9814999999999997e-05 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.188095+0000", 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1250999999999999e-05 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.188161+0000", 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6068000000000005e-05 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.192572+0000", 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044111189999999998 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.201485+0000", 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089132410000000006 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450212+0000", 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24872681999999999 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450263+0000", 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0427999999999999e-05 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450302+0000", 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9595000000000003e-05 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:166625 4.f 4:fff404f6:::benchmark_data_smithi187_76542_object74289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:01.194131+0000", 2024-09-02T10:19:42.516 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 341.23371077500002, 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.256297687, 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 166625 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194131+0000", 2024-09-02T10:19:42.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194131+0000", 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194132+0000", 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1769999999999999e-06 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194143+0000", 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-05 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194154+0000", 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1124999999999999e-05 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194162+0000", 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7400000000000004e-06 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194203+0000", 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1207e-05 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194272+0000", 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9036999999999996e-05 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194347+0000", 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4309999999999998e-05 2024-09-02T10:19:42.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.198684+0000", 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043371069999999998 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.425961+0000", 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22727677499999999 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450340+0000", 2024-09-02T10:19:42.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024379764000000002 2024-09-02T10:19:42.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450388+0000", 2024-09-02T10:19:42.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7160000000000002e-05 2024-09-02T10:19:42.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450428+0000", 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0735000000000002e-05 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25933.0:62902 5.0 e462/431 5:0a3860dd:::benchmark_data_smithi187_89405_object62901:head v 462'3865, mlcod=462'3862)", 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:15:34.635547+0000", 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 247.79229490700001, 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28423237499999998, 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.523 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.635547+0000", 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.635547+0000", 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.635547+0000", 2024-09-02T10:19:42.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.635575+0000", 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7557999999999999e-05 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.917385+0000", 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28181001999999999 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.917393+0000", 2024-09-02T10:19:42.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.571e-06 2024-09-02T10:19:42.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.917450+0000", 2024-09-02T10:19:42.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7225000000000002e-05 2024-09-02T10:19:42.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.917471+0000", 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1362999999999999e-05 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.919731+0000", 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002259629 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.919779+0000", 2024-09-02T10:19:42.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8151e-05 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:103601 5.c 5:3323d659:::benchmark_data_smithi187_89405_object103600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:16.164978+0000", 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 206.262863337, 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52291455399999998, 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.528 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 103601 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.164978+0000", 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.164978+0000", 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.164980+0000", 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.556e-06 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165044+0000", 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4633000000000003e-05 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165057+0000", 2024-09-02T10:19:42.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2872e-05 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165066+0000", 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6139999999999992e-06 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165137+0000", 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1643000000000005e-05 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165270+0000", 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013258200000000001 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165426+0000", 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015579 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.175521+0000", 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010095550999999999 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.175557+0000", 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5237999999999997e-05 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687808+0000", 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51225187800000005 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687852+0000", 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3550000000000001e-05 2024-09-02T10:19:42.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687893+0000", 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0646999999999999e-05 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25933.0:103601 5.c e479/455 ondisk, result = 0)", 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:16.178712+0000", 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 206.249129385, 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50918673599999997, 2024-09-02T10:19:42.534 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178712+0000", 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178712+0000", 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178713+0000", 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178716+0000", 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.822e-06 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687751+0000", 2024-09-02T10:19:42.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50903458599999996 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687756+0000", 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3990000000000003e-06 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687799+0000", 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2830999999999997e-05 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687805+0000", 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.558e-06 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687899+0000", 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4003000000000001e-05 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.538 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "rep_scrubmap(5.c e455 from shard 0)", 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:16.358849+0000", 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 206.06899279699999, 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.73020200300000004, 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.358849+0000", 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.358849+0000", 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.358849+0000", 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9999999999999997e-07 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.358868+0000", 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8343999999999999e-05 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:17.088882+0000", 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.73001465499999996 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:17.088886+0000", 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0319999999999997e-06 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:17.088925+0000", 2024-09-02T10:19:42.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8992000000000002e-05 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:17.088933+0000", 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4569999999999999e-06 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:17.089051+0000", 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117823 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:207205 5.c 5:3323d659:::benchmark_data_smithi187_89405_object103600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:20.127850+0000", 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 82.299991168999995, 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21962476, 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 207205 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:42.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127850+0000", 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127850+0000", 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127853+0000", 2024-09-02T10:19:42.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127863+0000", 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0377000000000001e-05 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127884+0000", 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1361000000000001e-05 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:42.545 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127896+0000", 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1559999999999999e-05 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127960+0000", 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3800000000000006e-05 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.128066+0000", 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106196 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.128142+0000", 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6427999999999994e-05 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.131929+0000", 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003786392 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.132050+0000", 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121067 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.347347+0000", 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21529720299999999 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.347417+0000", 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0129999999999994e-05 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.347475+0000", 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7861e-05 2024-09-02T10:19:42.548 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:42.549 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:42.549 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:19:42.550 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:42 smithi187 ceph-mon[20072]: from='client.26714 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:42.550 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:42 smithi187 ceph-mon[20072]: pgmap v1302: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.0 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 44 MiB/s wr, 696 op/s; 1436/198585 objects misplaced (0.723%); 7.6 MiB/s, 122 objects/s recovering 2024-09-02T10:19:42.550 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:42.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:42.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:42.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:42.678 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:42.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:42.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:42.809 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:42.810 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:42.833 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:42.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:42.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:42.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:42.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 62 16 71134 71118 71.6857 82.5 0.0137667 0.0139451 2024-09-02T10:19:43.039 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:43.064 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:43.065 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:43.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:43.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:43.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:43.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:43.319 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:43.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:43.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:43.452 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:43.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:43.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:19:43.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:43.580 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:43.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:43.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:43.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:43.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:43.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:43.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:43.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:43.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:43.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:43.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:43.951 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 63 16 72366 72350 71.7699 77 0.0198642 0.0139276 2024-09-02T10:19:43.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:43.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:44.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:44.092 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:44.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:44.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:44.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:44.229 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:44.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:44.234 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:19:44.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:19:44.267 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:19:44.267 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:19:44.291 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:19:44.292 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:19:44.317 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:19:44.317 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:19:44.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:44.350 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:19:44.350 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:19:44.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:44.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:44.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:44.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:19:44.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:19:44.418 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:19:44.418 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:19:44.455 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:19:44.455 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:19:44.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:44.486 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:44.487 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:44.488 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:19:44.488 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:19:44.523 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:19:44.523 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:19:44.531 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:44.561 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:19:44.562 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:19:44.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:44.595 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:19:44.595 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:19:44.612 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:44 smithi151 ceph-mon[16639]: pgmap v1303: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 56 MiB/s wr, 900 op/s; 1065/208224 objects misplaced (0.511%); 8.6 MiB/s, 138 objects/s recovering 2024-09-02T10:19:44.612 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:44 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:19:44.612 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:44 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:19:44.612 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:44 smithi151 ceph-mon[16639]: osdmap e562: 12 total, 12 up, 11 in 2024-09-02T10:19:44.613 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/506266977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:44.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:44.614 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:44.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:44.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:44.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:44 smithi187 ceph-mon[20072]: pgmap v1303: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 56 MiB/s wr, 900 op/s; 1065/208224 objects misplaced (0.511%); 8.6 MiB/s, 138 objects/s recovering 2024-09-02T10:19:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:44 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:19:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:44 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:19:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:44 smithi187 ceph-mon[20072]: osdmap e562: 12 total, 12 up, 11 in 2024-09-02T10:19:44.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/506266977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:44.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:44 smithi162 ceph-mon[20279]: pgmap v1303: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.2 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 56 MiB/s wr, 900 op/s; 1065/208224 objects misplaced (0.511%); 8.6 MiB/s, 138 objects/s recovering 2024-09-02T10:19:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:44 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:19:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:44 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:19:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:44 smithi162 ceph-mon[20279]: osdmap e562: 12 total, 12 up, 11 in 2024-09-02T10:19:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/506266977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:44.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:44.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:44.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 64 16 73638 73622 71.8906 79.5 0.025112 0.0139037 2024-09-02T10:19:44.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:44.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:44.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:44.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:45.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:45.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:45.120 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:45.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:19:45.221 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:45.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:45.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:45.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:45.373 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:45.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:45.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:45.409 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:19:45.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:45.491 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/599720137' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:19:45.491 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:45 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:19:45.505 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:45.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:45.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:45.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:45.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:45.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:45.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:45.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/599720137' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:19:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:45 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:19:45.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:45.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:45.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/599720137' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:19:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:45 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:19:45.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:45.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:45.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:45.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:45.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 65 16 74729 74713 71.8336 68.1875 0.0114038 0.013916 2024-09-02T10:19:45.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:46.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:46.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:46.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:46.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:46.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:46.141 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:46.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:46.266 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:46.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:46.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:46.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:46.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:46.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:46.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:46.519 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:46.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:46.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:46.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:46.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:46.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:46.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:46 smithi151 ceph-mon[16639]: pgmap v1305: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 1065/212748 objects misplaced (0.501%); 2.9 MiB/s, 46 objects/s recovering 2024-09-02T10:19:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:46 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:19:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:46 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:19:46.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:46 smithi151 ceph-mon[16639]: osdmap e563: 12 total, 12 up, 11 in 2024-09-02T10:19:46.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:46 smithi187 ceph-mon[20072]: pgmap v1305: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 1065/212748 objects misplaced (0.501%); 2.9 MiB/s, 46 objects/s recovering 2024-09-02T10:19:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:46 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:19:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:46 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:19:46.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:46 smithi187 ceph-mon[20072]: osdmap e563: 12 total, 12 up, 11 in 2024-09-02T10:19:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:46.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:46.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:46 smithi162 ceph-mon[20279]: pgmap v1305: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 63 MiB/s wr, 1.01k op/s; 1065/212748 objects misplaced (0.501%); 2.9 MiB/s, 46 objects/s recovering 2024-09-02T10:19:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:46 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:19:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:46 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:19:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:46 smithi162 ceph-mon[20279]: osdmap e563: 12 total, 12 up, 11 in 2024-09-02T10:19:46.876 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:46.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:46.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:46.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 66 16 75930 75914 71.8824 75.0625 0.00666853 0.0139066 2024-09-02T10:19:47.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:47.032 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:47.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:47.100 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:19:47.101 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:19:47.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:47.168 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:47.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:47.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:47.305 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:47.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:47.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:47.444 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:47.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:47.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:47.547 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:47.578 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:47.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:47.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:47.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:47 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:19:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:47 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:19:47.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:47.717 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:47.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:47.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:47 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:19:47.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:47 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:19:47.814 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:47.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:47 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:19:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:47 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:19:47.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:47.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:47.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:47.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:19:47.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 67 16 77251 77235 72.0417 82.5625 0.010861 0.013875 2024-09-02T10:19:47.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:47.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:47.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:47.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:48.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:48.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:48.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:48.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:48.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:48.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:48.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:19:48.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:48.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:48.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:48.382 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:48.459 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:48.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:48.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:48.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:48.506 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:48.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:48.619 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:48 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:19:48.619 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:48 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:19:48.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:48 smithi151 ceph-mon[16639]: pgmap v1307: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s; 584/220710 objects misplaced (0.265%); 7.3 MiB/s, 116 objects/s recovering 2024-09-02T10:19:48.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:48 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:19:48.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:48 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:19:48.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:48 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:19:48.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:48 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:19:48.635 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:19:48.635 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:19:48.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:48.638 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:48.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:48.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:48 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:19:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:48 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:19:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:48 smithi187 ceph-mon[20072]: pgmap v1307: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s; 584/220710 objects misplaced (0.265%); 7.3 MiB/s, 116 objects/s recovering 2024-09-02T10:19:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:48 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:19:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:48 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:19:48.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:48 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:19:48.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:48 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:19:48.772 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:48.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:48 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:19:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:48 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:19:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:48 smithi162 ceph-mon[20279]: pgmap v1307: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 72 MiB/s wr, 1.15k op/s; 584/220710 objects misplaced (0.265%); 7.3 MiB/s, 116 objects/s recovering 2024-09-02T10:19:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:48 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:19:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:48 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:19:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:48 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:19:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:48 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:19:48.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:48.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:19:48.905 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:48.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:48.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 68 16 78215 78199 71.8683 60.25 0.00907465 0.0139099 2024-09-02T10:19:49.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:49.037 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:49.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:49.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:49.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:49.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:49.216 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:19:49.217 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:19:49.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:49.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:49.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:49.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:49.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:49 smithi151 ceph-mon[16639]: from='client.17208 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:49 smithi151 ceph-mon[16639]: osdmap e564: 12 total, 12 up, 11 in 2024-09-02T10:19:49.438 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:49.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:49.464 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:49.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:49.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:49.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:49.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:49.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:49.698 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:49.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:49 smithi187 ceph-mon[20072]: from='client.17208 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:49 smithi187 ceph-mon[20072]: osdmap e564: 12 total, 12 up, 11 in 2024-09-02T10:19:49.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:49.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:49.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:49 smithi162 ceph-mon[20279]: from='client.17208 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:19:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:49 smithi162 ceph-mon[20279]: osdmap e564: 12 total, 12 up, 11 in 2024-09-02T10:19:49.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:49.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:49.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:49.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 69 16 79432 79416 71.929 76.0625 0.0119924 0.0138977 2024-09-02T10:19:50.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:50.083 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:50.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:50.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:50.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:50.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:50.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:50.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:50.348 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:50.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:50.419 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1307,"stamp":"2024-09-02T10:19:47.348986+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4821942816,"num_objects":73570,"num_object_clones":0,"num_object_copies":220710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":584,"num_objects_unfound":0,"num_objects_dirty":73570,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73568,"num_write_kb":4708352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7957,"num_bytes_recovered":522003520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76420,"ondisk_log_size":76420,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":19130964,"kb_used_data":14413324,"kb_used_omap":17,"kb_used_meta":4717358,"kb_avail":1012065708,"statfs":{"total":1055945392128,"available":1036355284992,"internally_reserved":0,"allocated":14759243776,"data_stored":14754682405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":4830575478},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,3,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":550895616,"num_objects":8406,"num_object_clones":0,"num_object_copies":25218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-852,"num_objects_unfound":0,"num_objects_dirty":8406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8406,"num_write_kb":537984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":852,"num_bytes_recovered":55836672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.284570"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":470,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.410861+0000","last_change":"2024-09-02T10:18:43.049145+0000","last_active":"2024-09-02T10:19:45.410861+0000","last_peered":"2024-09-02T10:19:45.410861+0000","last_clean":"2024-09-02T10:19:45.410861+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:19:45.410861+0000","last_undegraded":"2024-09-02T10:19:45.410861+0000","last_fullsized":"2024-09-02T10:19:45.410861+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:43.049077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:43.049077+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:07.603384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":274,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:40.070701+0000","last_change":"2024-09-02T10:18:19.826521+0000","last_active":"2024-09-02T10:19:40.070701+0000","last_peered":"2024-09-02T10:19:40.070701+0000","last_clean":"2024-09-02T10:19:40.070701+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:19:40.070701+0000","last_undegraded":"2024-09-02T10:19:40.070701+0000","last_fullsized":"2024-09-02T10:19:40.070701+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:19.826468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:19.826468+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:44.959309+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":726,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-02T10:19:44.067076+0000","last_change":"2024-09-02T10:18:16.145420+0000","last_active":"2024-09-02T10:19:44.067076+0000","last_peered":"2024-09-02T10:19:44.067076+0000","last_clean":"2024-09-02T10:19:44.067076+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:19:44.067076+0000","last_undegraded":"2024-09-02T10:19:44.067076+0000","last_fullsized":"2024-09-02T10:19:44.067076+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:16.145346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:16.145346+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:44.195211+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":726,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-02T10:19:44.066984+0000","last_change":"2024-09-02T10:18:39.045140+0000","last_active":"2024-09-02T10:19:44.066984+0000","last_peered":"2024-09-02T10:19:44.066984+0000","last_clean":"2024-09-02T10:19:44.066984+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:19:44.066984+0000","last_undegraded":"2024-09-02T10:19:44.066984+0000","last_fullsized":"2024-09-02T10:19:44.066984+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:39.045083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:39.045083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:03.744946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":38,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-02T10:19:44.068839+0000","last_change":"2024-09-02T10:19:05.058923+0000","last_active":"2024-09-02T10:19:44.068839+0000","last_peered":"2024-09-02T10:19:44.068839+0000","last_clean":"2024-09-02T10:19:44.068839+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:19:44.068839+0000","last_undegraded":"2024-09-02T10:19:44.068839+0000","last_fullsized":"2024-09-02T10:19:44.068839+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:01.810089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:01.810089+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:19.405935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":585,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.629126+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:19:45.629126+0000","last_peered":"2024-09-02T10:19:45.629126+0000","last_clean":"2024-09-02T10:19:45.629126+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:19:45.629126+0000","last_undegraded":"2024-09-02T10:19:45.629126+0000","last_fullsized":"2024-09-02T10:19:45.629126+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:08.487285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1101,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.407537+0000","last_change":"2024-09-02T10:19:03.880037+0000","last_active":"2024-09-02T10:19:45.407537+0000","last_peered":"2024-09-02T10:19:45.407537+0000","last_clean":"2024-09-02T10:19:45.407537+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:19:45.407537+0000","last_undegraded":"2024-09-02T10:19:45.407537+0000","last_fullsized":"2024-09-02T10:19:45.407537+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:03.879968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:03.879968+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.678816+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":741,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-02T10:19:44.067200+0000","last_change":"2024-09-02T10:18:35.059181+0000","last_active":"2024-09-02T10:19:44.067200+0000","last_peered":"2024-09-02T10:19:44.067200+0000","last_clean":"2024-09-02T10:19:44.067200+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:19:44.067200+0000","last_undegraded":"2024-09-02T10:19:44.067200+0000","last_fullsized":"2024-09-02T10:19:44.067200+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:35.059123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:35.059123+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:59.080089+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":739,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:40.070724+0000","last_change":"2024-09-02T10:18:10.904183+0000","last_active":"2024-09-02T10:19:40.070724+0000","last_peered":"2024-09-02T10:19:40.070724+0000","last_clean":"2024-09-02T10:19:40.070724+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:19:40.070724+0000","last_undegraded":"2024-09-02T10:19:40.070724+0000","last_fullsized":"2024-09-02T10:19:40.070724+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:10.904126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:10.904126+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:42.968651+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":719,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.409821+0000","last_change":"2024-09-02T10:19:05.910750+0000","last_active":"2024-09-02T10:19:45.409821+0000","last_peered":"2024-09-02T10:19:45.409821+0000","last_clean":"2024-09-02T10:19:45.409821+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:19:45.409821+0000","last_undegraded":"2024-09-02T10:19:45.409821+0000","last_fullsized":"2024-09-02T10:19:45.409821+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:05.910691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:05.910691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:20.353104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":562,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.409631+0000","last_change":"2024-09-02T10:17:44.275503+0000","last_active":"2024-09-02T10:19:45.409631+0000","last_peered":"2024-09-02T10:19:45.409631+0000","last_clean":"2024-09-02T10:19:45.409631+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:19:45.409631+0000","last_undegraded":"2024-09-02T10:19:45.409631+0000","last_fullsized":"2024-09-02T10:19:45.409631+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:17:44.275434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:17:44.275434+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:45.831669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1072,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:40.070515+0000","last_change":"2024-09-02T10:19:00.052912+0000","last_active":"2024-09-02T10:19:40.070515+0000","last_peered":"2024-09-02T10:19:40.070515+0000","last_clean":"2024-09-02T10:19:40.070515+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:19:40.070515+0000","last_undegraded":"2024-09-02T10:19:40.070515+0000","last_fullsized":"2024-09-02T10:19:40.070515+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.052857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.052857+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:29.321752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1011,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.407666+0000","last_change":"2024-09-02T10:18:41.060635+0000","last_active":"2024-09-02T10:19:45.407666+0000","last_peered":"2024-09-02T10:19:45.407666+0000","last_clean":"2024-09-02T10:19:45.407666+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:19:45.407666+0000","last_undegraded":"2024-09-02T10:19:45.407666+0000","last_fullsized":"2024-09-02T10:19:45.407666+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:41.060566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:41.060566+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:47.963172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":986,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:40.075092+0000","last_change":"2024-09-02T10:18:16.400250+0000","last_active":"2024-09-02T10:19:40.075092+0000","last_peered":"2024-09-02T10:19:40.075092+0000","last_clean":"2024-09-02T10:19:40.075092+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:19:40.075092+0000","last_undegraded":"2024-09-02T10:19:40.075092+0000","last_fullsized":"2024-09-02T10:19:40.075092+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:16.400175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:16.400175+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:41.543272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":981,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.407556+0000","last_change":"2024-09-02T10:18:40.069068+0000","last_active":"2024-09-02T10:19:45.407556+0000","last_peered":"2024-09-02T10:19:45.407556+0000","last_clean":"2024-09-02T10:19:45.407556+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:19:45.407556+0000","last_undegraded":"2024-09-02T10:19:45.407556+0000","last_fullsized":"2024-09-02T10:19:45.407556+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:40.069013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:40.069013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:45.996592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"563'4725","reported_seq":4794,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:46.759245+0000","last_change":"2024-09-02T10:18:37.599555+0000","last_active":"2024-09-02T10:19:46.759245+0000","last_peered":"2024-09-02T10:19:46.759245+0000","last_clean":"2024-09-02T10:19:46.759245+0000","last_became_active":"2024-09-02T10:18:37.599064+0000","last_became_peered":"2024-09-02T10:18:37.599064+0000","last_unstale":"2024-09-02T10:19:46.759245+0000","last_undegraded":"2024-09-02T10:19:46.759245+0000","last_fullsized":"2024-09-02T10:19:46.759245+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4725,"log_dups_size":0,"ondisk_log_size":4725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:50.285721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309657600,"num_objects":4725,"num_object_clones":0,"num_object_copies":14175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4725,"num_write_kb":302400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1068,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.407649+0000","last_change":"2024-09-02T10:18:13.075935+0000","last_active":"2024-09-02T10:19:45.407649+0000","last_peered":"2024-09-02T10:19:45.407649+0000","last_clean":"2024-09-02T10:19:45.407649+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:19:45.407649+0000","last_undegraded":"2024-09-02T10:19:45.407649+0000","last_fullsized":"2024-09-02T10:19:45.407649+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:13.075888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:13.075888+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"562'4574","reported_seq":8875,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-02T10:19:44.166835+0000","last_change":"2024-09-02T10:19:32.601261+0000","last_active":"2024-09-02T10:19:44.166835+0000","last_peered":"2024-09-02T10:19:44.166835+0000","last_clean":"2024-09-02T10:19:44.166835+0000","last_became_active":"2024-09-02T10:19:32.596751+0000","last_became_peered":"2024-09-02T10:19:32.596751+0000","last_unstale":"2024-09-02T10:19:44.166835+0000","last_undegraded":"2024-09-02T10:19:44.166835+0000","last_fullsized":"2024-09-02T10:19:44.166835+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4574,"log_dups_size":0,"ondisk_log_size":4574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:50.137401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299761664,"num_objects":4574,"num_object_clones":0,"num_object_copies":13722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4574,"num_write_kb":292736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":138346496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1183,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.409571+0000","last_change":"2024-09-02T10:17:50.300169+0000","last_active":"2024-09-02T10:19:45.409571+0000","last_peered":"2024-09-02T10:19:45.409571+0000","last_clean":"2024-09-02T10:19:45.409571+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:19:45.409571+0000","last_undegraded":"2024-09-02T10:19:45.409571+0000","last_fullsized":"2024-09-02T10:19:45.409571+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:17:50.300100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:17:50.300100+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"563'4702","reported_seq":4771,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:47.165540+0000","last_change":"2024-09-02T10:18:37.062429+0000","last_active":"2024-09-02T10:19:47.165540+0000","last_peered":"2024-09-02T10:19:47.165540+0000","last_clean":"2024-09-02T10:19:47.165540+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:19:47.165540+0000","last_undegraded":"2024-09-02T10:19:47.165540+0000","last_fullsized":"2024-09-02T10:19:47.165540+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:01.072199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308150272,"num_objects":4702,"num_object_clones":0,"num_object_copies":14106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4702,"num_write_kb":300928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":836,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:40.075765+0000","last_change":"2024-09-02T10:19:05.058003+0000","last_active":"2024-09-02T10:19:40.075765+0000","last_peered":"2024-09-02T10:19:40.075765+0000","last_clean":"2024-09-02T10:19:40.075765+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:19:40.075765+0000","last_undegraded":"2024-09-02T10:19:40.075765+0000","last_fullsized":"2024-09-02T10:19:40.075765+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.206480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.206480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.672102+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"561'4410","reported_seq":4475,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:43.174564+0000","last_change":"2024-09-02T10:18:37.848522+0000","last_active":"2024-09-02T10:19:43.174564+0000","last_peered":"2024-09-02T10:19:43.174564+0000","last_clean":"2024-09-02T10:19:43.174564+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:19:43.174564+0000","last_undegraded":"2024-09-02T10:19:43.174564+0000","last_fullsized":"2024-09-02T10:19:43.174564+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4410,"log_dups_size":0,"ondisk_log_size":4410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:56.425124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289013760,"num_objects":4410,"num_object_clones":0,"num_object_copies":13230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4410,"num_write_kb":282240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":229,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.410756+0000","last_change":"2024-09-02T10:18:15.017709+0000","last_active":"2024-09-02T10:19:45.410756+0000","last_peered":"2024-09-02T10:19:45.410756+0000","last_clean":"2024-09-02T10:19:45.410756+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:19:45.410756+0000","last_undegraded":"2024-09-02T10:19:45.410756+0000","last_fullsized":"2024-09-02T10:19:45.410756+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:15.017640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:15.017640+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:47.839480+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"563'4736","reported_seq":4804,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:46.213873+0000","last_change":"2024-09-02T10:18:37.850883+0000","last_active":"2024-09-02T10:19:46.213873+0000","last_peered":"2024-09-02T10:19:46.213873+0000","last_clean":"2024-09-02T10:19:46.213873+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:19:46.213873+0000","last_undegraded":"2024-09-02T10:19:46.213873+0000","last_fullsized":"2024-09-02T10:19:46.213873+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:05.087393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":4736,"num_object_clones":0,"num_object_copies":14208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1138,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.410786+0000","last_change":"2024-09-02T10:18:44.023062+0000","last_active":"2024-09-02T10:19:45.410786+0000","last_peered":"2024-09-02T10:19:45.410786+0000","last_clean":"2024-09-02T10:19:45.410786+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:19:45.410786+0000","last_undegraded":"2024-09-02T10:19:45.410786+0000","last_fullsized":"2024-09-02T10:19:45.410786+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:44.023008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:44.023008+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:49.645929+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"562'4532","reported_seq":8881,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-02T10:19:44.196903+0000","last_change":"2024-09-02T10:19:31.591657+0000","last_active":"2024-09-02T10:19:44.196903+0000","last_peered":"2024-09-02T10:19:44.196903+0000","last_clean":"2024-09-02T10:19:44.196903+0000","last_became_active":"2024-09-02T10:19:31.588084+0000","last_became_peered":"2024-09-02T10:19:31.588084+0000","last_unstale":"2024-09-02T10:19:44.196903+0000","last_undegraded":"2024-09-02T10:19:44.196903+0000","last_fullsized":"2024-09-02T10:19:44.196903+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4532,"log_dups_size":0,"ondisk_log_size":4532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:48.891698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297009152,"num_objects":4532,"num_object_clones":0,"num_object_copies":13596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4532,"num_write_kb":290048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2135,"num_bytes_recovered":139919360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":605,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.408116+0000","last_change":"2024-09-02T10:18:41.761050+0000","last_active":"2024-09-02T10:19:45.408116+0000","last_peered":"2024-09-02T10:19:45.408116+0000","last_clean":"2024-09-02T10:19:45.408116+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:19:45.408116+0000","last_undegraded":"2024-09-02T10:19:45.408116+0000","last_fullsized":"2024-09-02T10:19:45.408116+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:41.760998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:41.760998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:57.499320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"562'4503","reported_seq":4571,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-02T10:19:44.389712+0000","last_change":"2024-09-02T10:18:37.600687+0000","last_active":"2024-09-02T10:19:44.389712+0000","last_peered":"2024-09-02T10:19:44.389712+0000","last_clean":"2024-09-02T10:19:44.389712+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:19:44.389712+0000","last_undegraded":"2024-09-02T10:19:44.389712+0000","last_fullsized":"2024-09-02T10:19:44.389712+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:59.230189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295108608,"num_objects":4503,"num_object_clones":0,"num_object_copies":13509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":288192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":577,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.408207+0000","last_change":"2024-09-02T10:18:11.764437+0000","last_active":"2024-09-02T10:19:45.408207+0000","last_peered":"2024-09-02T10:19:45.408207+0000","last_clean":"2024-09-02T10:19:45.408207+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:19:45.408207+0000","last_undegraded":"2024-09-02T10:19:45.408207+0000","last_fullsized":"2024-09-02T10:19:45.408207+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:11.764362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:11.764362+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:49.613289+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"561'4456","reported_seq":4521,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:42.873363+0000","last_change":"2024-09-02T10:18:37.065153+0000","last_active":"2024-09-02T10:19:42.873363+0000","last_peered":"2024-09-02T10:19:42.873363+0000","last_clean":"2024-09-02T10:19:42.873363+0000","last_became_active":"2024-09-02T10:18:37.064844+0000","last_became_peered":"2024-09-02T10:18:37.064844+0000","last_unstale":"2024-09-02T10:19:42.873363+0000","last_undegraded":"2024-09-02T10:19:42.873363+0000","last_fullsized":"2024-09-02T10:19:42.873363+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4456,"log_dups_size":0,"ondisk_log_size":4456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292028416,"num_objects":4456,"num_object_clones":0,"num_object_copies":13368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4456,"num_write_kb":285184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":329,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.406827+0000","last_change":"2024-09-02T10:18:37.859095+0000","last_active":"2024-09-02T10:19:45.406827+0000","last_peered":"2024-09-02T10:19:45.406827+0000","last_clean":"2024-09-02T10:19:45.406827+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:19:45.406827+0000","last_undegraded":"2024-09-02T10:19:45.406827+0000","last_fullsized":"2024-09-02T10:19:45.406827+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:37.859053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:37.859053+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:57.896931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"561'4501","reported_seq":8925,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:43.182252+0000","last_change":"2024-09-02T10:19:33.596264+0000","last_active":"2024-09-02T10:19:43.182252+0000","last_peered":"2024-09-02T10:19:43.182252+0000","last_clean":"2024-09-02T10:19:43.182252+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:19:43.182252+0000","last_undegraded":"2024-09-02T10:19:43.182252+0000","last_fullsized":"2024-09-02T10:19:43.182252+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:55.430237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294977536,"num_objects":4501,"num_object_clones":0,"num_object_copies":13503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4501,"num_write_kb":288064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":578,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.409408+0000","last_change":"2024-09-02T10:18:20.714703+0000","last_active":"2024-09-02T10:19:45.409408+0000","last_peered":"2024-09-02T10:19:45.409408+0000","last_clean":"2024-09-02T10:19:45.409408+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:19:45.409408+0000","last_undegraded":"2024-09-02T10:19:45.409408+0000","last_fullsized":"2024-09-02T10:19:45.409408+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:20.714649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:20.714649+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:47.653799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"563'4659","reported_seq":4728,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:46.221615+0000","last_change":"2024-09-02T10:18:37.848300+0000","last_active":"2024-09-02T10:19:46.221615+0000","last_peered":"2024-09-02T10:19:46.221615+0000","last_clean":"2024-09-02T10:19:46.221615+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:19:46.221615+0000","last_undegraded":"2024-09-02T10:19:46.221615+0000","last_fullsized":"2024-09-02T10:19:46.221615+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4659,"log_dups_size":0,"ondisk_log_size":4659,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:01.700433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305332224,"num_objects":4659,"num_object_clones":0,"num_object_copies":13977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4659,"num_write_kb":298176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":832,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.409628+0000","last_change":"2024-09-02T10:19:05.058925+0000","last_active":"2024-09-02T10:19:45.409628+0000","last_peered":"2024-09-02T10:19:45.409628+0000","last_clean":"2024-09-02T10:19:45.409628+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:19:45.409628+0000","last_undegraded":"2024-09-02T10:19:45.409628+0000","last_fullsized":"2024-09-02T10:19:45.409628+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.789947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.789947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:10.614688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"561'4604","reported_seq":4669,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:43.642443+0000","last_change":"2024-09-02T10:18:37.064957+0000","last_active":"2024-09-02T10:19:43.642443+0000","last_peered":"2024-09-02T10:19:43.642443+0000","last_clean":"2024-09-02T10:19:43.642443+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:19:43.642443+0000","last_undegraded":"2024-09-02T10:19:43.642443+0000","last_fullsized":"2024-09-02T10:19:43.642443+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:49.603479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301727744,"num_objects":4604,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4604,"num_write_kb":294656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1095,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:40.070524+0000","last_change":"2024-09-02T10:19:05.058662+0000","last_active":"2024-09-02T10:19:40.070524+0000","last_peered":"2024-09-02T10:19:40.070524+0000","last_clean":"2024-09-02T10:19:40.070524+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:19:40.070524+0000","last_undegraded":"2024-09-02T10:19:40.070524+0000","last_fullsized":"2024-09-02T10:19:40.070524+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:42.912082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:42.912082+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:51.861067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"563'4806","reported_seq":7948,"reported_epoch":563,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:19:47.198908+0000","last_change":"2024-09-02T10:19:29.964931+0000","last_active":"2024-09-02T10:19:47.198908+0000","last_peered":"2024-09-02T10:19:47.198908+0000","last_clean":"2024-09-02T10:19:04.021218+0000","last_became_active":"2024-09-02T10:19:06.874629+0000","last_became_peered":"2024-09-02T10:19:06.874629+0000","last_unstale":"2024-09-02T10:19:47.198908+0000","last_undegraded":"2024-09-02T10:19:47.198908+0000","last_fullsized":"2024-09-02T10:19:06.060341+0000","mapping_epoch":549,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4806,"log_dups_size":0,"ondisk_log_size":4806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314966016,"num_objects":4806,"num_object_clones":0,"num_object_copies":14418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":584,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4806,"num_write_kb":307584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1534,"num_bytes_recovered":100532224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":584},{"shards":"3,5","objects":4222}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":843,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.409574+0000","last_change":"2024-09-02T10:18:37.852037+0000","last_active":"2024-09-02T10:19:45.409574+0000","last_peered":"2024-09-02T10:19:45.409574+0000","last_clean":"2024-09-02T10:19:45.409574+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:19:45.409574+0000","last_undegraded":"2024-09-02T10:19:45.409574+0000","last_fullsized":"2024-09-02T10:19:45.409574+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:37.851984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:37.851984+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:46.708384+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"563'4759","reported_seq":4829,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:46.216425+0000","last_change":"2024-09-02T10:18:37.850899+0000","last_active":"2024-09-02T10:19:46.216425+0000","last_peered":"2024-09-02T10:19:46.216425+0000","last_clean":"2024-09-02T10:19:46.216425+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:19:46.216425+0000","last_undegraded":"2024-09-02T10:19:46.216425+0000","last_fullsized":"2024-09-02T10:19:46.216425+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4759,"log_dups_size":0,"ondisk_log_size":4759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:02.178728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311885824,"num_objects":4759,"num_object_clones":0,"num_object_copies":14277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4759,"num_write_kb":304576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1098,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.406923+0000","last_change":"2024-09-02T10:19:08.999797+0000","last_active":"2024-09-02T10:19:45.406923+0000","last_peered":"2024-09-02T10:19:45.406923+0000","last_clean":"2024-09-02T10:19:45.406923+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:19:45.406923+0000","last_undegraded":"2024-09-02T10:19:45.406923+0000","last_fullsized":"2024-09-02T10:19:45.406923+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:08.999706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:08.999706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:13.510899+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"561'4407","reported_seq":4473,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:42.889642+0000","last_change":"2024-09-02T10:18:37.600768+0000","last_active":"2024-09-02T10:19:42.889642+0000","last_peered":"2024-09-02T10:19:42.889642+0000","last_clean":"2024-09-02T10:19:42.889642+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:19:42.889642+0000","last_undegraded":"2024-09-02T10:19:42.889642+0000","last_fullsized":"2024-09-02T10:19:42.889642+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:54.523794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288817152,"num_objects":4407,"num_object_clones":0,"num_object_copies":13221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4407,"num_write_kb":282048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":46,"reported_epoch":562,"state":"active+clean","last_fresh":"2024-09-02T10:19:44.066581+0000","last_change":"2024-09-02T10:19:06.947029+0000","last_active":"2024-09-02T10:19:44.066581+0000","last_peered":"2024-09-02T10:19:44.066581+0000","last_clean":"2024-09-02T10:19:44.066581+0000","last_became_active":"2024-09-02T10:19:06.049121+0000","last_became_peered":"2024-09-02T10:19:06.049121+0000","last_unstale":"2024-09-02T10:19:44.066581+0000","last_undegraded":"2024-09-02T10:19:44.066581+0000","last_fullsized":"2024-09-02T10:19:44.066581+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:06.946957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:06.946957+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:24.543000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"563'4737","reported_seq":4807,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:47.180734+0000","last_change":"2024-09-02T10:18:37.848530+0000","last_active":"2024-09-02T10:19:47.180734+0000","last_peered":"2024-09-02T10:19:47.180734+0000","last_clean":"2024-09-02T10:19:47.180734+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:19:47.180734+0000","last_undegraded":"2024-09-02T10:19:47.180734+0000","last_fullsized":"2024-09-02T10:19:47.180734+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4737,"log_dups_size":0,"ondisk_log_size":4737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:19:58.128911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310444032,"num_objects":4737,"num_object_clones":0,"num_object_copies":14211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4737,"num_write_kb":303168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":870,"reported_epoch":563,"state":"active+clean","last_fresh":"2024-09-02T10:19:45.409836+0000","last_change":"2024-09-02T10:19:05.059514+0000","last_active":"2024-09-02T10:19:45.409836+0000","last_peered":"2024-09-02T10:19:45.409836+0000","last_clean":"2024-09-02T10:19:45.409836+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:19:45.409836+0000","last_undegraded":"2024-09-02T10:19:45.409836+0000","last_fullsized":"2024-09-02T10:19:45.409836+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:10.104220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:10.104220+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"561'4457","reported_seq":4522,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:43.206567+0000","last_change":"2024-09-02T10:18:37.066015+0000","last_active":"2024-09-02T10:19:43.206567+0000","last_peered":"2024-09-02T10:19:43.206567+0000","last_clean":"2024-09-02T10:19:43.206567+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:19:43.206567+0000","last_undegraded":"2024-09-02T10:19:43.206567+0000","last_fullsized":"2024-09-02T10:19:43.206567+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":4457,"log_dups_size":0,"ondisk_log_size":4457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:02.695338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292093952,"num_objects":4457,"num_object_clones":0,"num_object_copies":13371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4457,"num_write_kb":285248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":266,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-02T10:19:40.067258+0000","last_change":"2024-09-02T10:19:02.956010+0000","last_active":"2024-09-02T10:19:40.067258+0000","last_peered":"2024-09-02T10:19:40.067258+0000","last_clean":"2024-09-02T10:19:40.067258+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:19:40.067258+0000","last_undegraded":"2024-09-02T10:19:40.067258+0000","last_fullsized":"2024-09-02T10:19:40.067258+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:02.955947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:02.955947+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:26.242539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":4821352448,"num_objects":73568,"num_object_clones":0,"num_object_copies":220704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":584,"num_objects_unfound":0,"num_objects_dirty":73568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73568,"num_write_kb":4708352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7954,"num_bytes_recovered":521273344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14814543872,"data_stored":14814543872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73568,"ondisk_log_size":73568,"up":48,"acting":47,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383916,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1383260,"kb_used_data":914756,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92361892,"statfs":{"total":95995035648,"available":94578577408,"internally_reserved":0,"allocated":936710144,"data_stored":936292439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":479721924},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612849,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1223188,"kb_used_data":899076,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92521964,"statfs":{"total":95995035648,"available":94742491136,"internally_reserved":0,"allocated":920653824,"data_stored":920236119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":331872704},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841781,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1609020,"kb_used_data":1130796,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92136132,"statfs":{"total":95995035648,"available":94347399168,"internally_reserved":0,"allocated":1157935104,"data_stored":1157526197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":489683392},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070715,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":990156,"kb_used_data":688248,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92754996,"statfs":{"total":95995035648,"available":94981115904,"internally_reserved":0,"allocated":704765952,"data_stored":704352671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299648,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2092060,"kb_used_data":1605452,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91653092,"statfs":{"total":95995035648,"available":93852766208,"internally_reserved":0,"allocated":1643982848,"data_stored":1643561591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528580,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1549400,"kb_used_data":1211396,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92195752,"statfs":{"total":95995035648,"available":94408450048,"internally_reserved":0,"allocated":1240469504,"data_stored":1240051799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":346094023},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724809,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2966364,"kb_used_data":2439564,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":90778788,"statfs":{"total":95995035648,"available":92957478912,"internally_reserved":0,"allocated":2498113536,"data_stored":2497692279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953742,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1407308,"kb_used_data":1109560,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92337844,"statfs":{"total":95995035648,"available":94553952256,"internally_reserved":0,"allocated":1136189440,"data_stored":1135776159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149971,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2754812,"kb_used_data":2286316,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":90990340,"statfs":{"total":95995035648,"available":93174108160,"internally_reserved":0,"allocated":2341187584,"data_stored":2340778677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":479721928},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411607,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575132,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1755012,"kb_used_data":1282356,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":91990140,"statfs":{"total":95995035648,"available":94197903360,"internally_reserved":0,"allocated":1313132544,"data_stored":1312720085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738658,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1400384,"kb_used_data":845804,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92344768,"statfs":{"total":95995035648,"available":94561042432,"internally_reserved":0,"allocated":866103296,"data_stored":865694389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":872022016,"data_stored":872022016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1325203456,"data_stored":1325203456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2351759360,"data_stored":2351759360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1136328704,"data_stored":1136328704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2500067328,"data_stored":2500067328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1247215616,"data_stored":1247215616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1653997568,"data_stored":1653997568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":706084864,"data_stored":706084864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1157890048,"data_stored":1157890048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":921632768,"data_stored":921632768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":942342144,"data_stored":942342144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:19:50.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:19:50.426 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:50.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:50.473 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:50.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:50.575 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:50.598 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:50.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:50.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:50 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:19:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:50 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:19:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:50 smithi151 ceph-mon[16639]: pgmap v1309: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 584/229530 objects misplaced (0.254%); 5.0 MiB/s, 80 objects/s recovering 2024-09-02T10:19:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:50 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:19:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:50 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:19:50.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:50.722 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:50.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:50.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:50 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:19:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:50 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:19:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:50 smithi187 ceph-mon[20072]: pgmap v1309: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 584/229530 objects misplaced (0.254%); 5.0 MiB/s, 80 objects/s recovering 2024-09-02T10:19:50.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:50 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:19:50.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:50 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:19:50.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:50 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:19:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:50 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:19:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:50 smithi162 ceph-mon[20279]: pgmap v1309: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 74 MiB/s wr, 1.18k op/s; 584/229530 objects misplaced (0.254%); 5.0 MiB/s, 80 objects/s recovering 2024-09-02T10:19:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:50 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:19:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:50 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:19:50.848 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:50.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:50.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:50.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 70 16 80148 80132 71.5407 44.75 0.0596855 0.0139735 2024-09-02T10:19:50.985 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:50.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:51.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:51.110 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:51.111 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:51.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:51.241 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:51.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:51.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:51.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:51.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:51.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:51.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:51.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:51.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:51.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:51.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:51.624 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:51.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:51 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:19:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:51 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:19:51.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:51 smithi151 ceph-mon[16639]: 6.0 scrub starts 2024-09-02T10:19:51.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:51.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:51.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:51.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:51 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:19:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:51 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:19:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:51 smithi187 ceph-mon[20072]: 6.0 scrub starts 2024-09-02T10:19:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:51 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:19:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:51 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:19:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:51 smithi162 ceph-mon[20279]: 6.0 scrub starts 2024-09-02T10:19:51.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:51.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:51.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:51.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:51.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 71 16 81280 81264 71.5295 70.75 0.00808853 0.0139745 2024-09-02T10:19:51.983 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:52.008 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:52.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:52.009 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:52.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:19:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:52.133 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:52.134 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:19:52.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:52.258 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:52.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:52.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:52.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:52.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:52.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:52.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:52.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:52.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:52.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:52.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:52.636 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:52.637 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:52 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:19:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:52 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:19:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:52 smithi151 ceph-mon[16639]: pgmap v1310: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 584/233805 objects misplaced (0.250%); 4.1 MiB/s, 66 objects/s recovering 2024-09-02T10:19:52.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:52 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:19:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:52 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:19:52.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:52 smithi162 ceph-mon[20279]: pgmap v1310: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 584/233805 objects misplaced (0.250%); 4.1 MiB/s, 66 objects/s recovering 2024-09-02T10:19:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:52 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:19:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:52 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:19:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:52 smithi187 ceph-mon[20072]: pgmap v1310: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 70 MiB/s wr, 1.12k op/s; 584/233805 objects misplaced (0.250%); 4.1 MiB/s, 66 objects/s recovering 2024-09-02T10:19:52.758 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:52.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:52.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:52.883 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:52.883 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:52.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:19:52.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 72 16 82003 81987 71.1635 45.1875 0.0194614 0.0140071 2024-09-02T10:19:52.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:53.007 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:53.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:53.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:53.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:53.131 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:53.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:53.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:53.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:53.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:53.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:53.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:53.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:53.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:53.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:53.505 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:53.506 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:19:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:19:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:19:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:19:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: 6.0 scrub ok 2024-09-02T10:19:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:19:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:19:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: 6.4 scrub starts 2024-09-02T10:19:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: osdmap e565: 12 total, 12 up, 11 in 2024-09-02T10:19:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:19:53.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:53 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:19:53.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:53.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:53.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:53.682 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:19:53.682 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:19:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:19:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:19:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:19:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:19:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: 6.0 scrub ok 2024-09-02T10:19:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:19:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:19:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: 6.4 scrub starts 2024-09-02T10:19:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: osdmap e565: 12 total, 12 up, 11 in 2024-09-02T10:19:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:19:53.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:53 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:19:53.710 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:19:53.711 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:19:53.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:19:53.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:19:53.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: 6.0 scrub ok 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: 6.4 scrub starts 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: osdmap e565: 12 total, 12 up, 11 in 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:19:53.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:53 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:19:53.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:53.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:53.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:19:53.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:19:53.795 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:19:53.795 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:19:53.825 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:19:53.826 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:19:53.851 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:19:53.851 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:19:53.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:53.885 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:19:53.885 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:19:53.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:53.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:53.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:53.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:19:53.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:19:53.941 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:19:53.942 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:19:53.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 73 16 82574 82558 70.6775 35.6875 0.00569842 0.0141441 2024-09-02T10:19:53.968 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:19:53.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:19:53.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:54.000 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:19:54.000 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:19:54.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:54.023 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:54.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:54.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:54.149 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:54.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:54.283 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:54.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:54.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:54.410 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:54.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:54.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:54.534 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:54.535 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:54.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:54.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:54.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:54.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: 6.2 scrub starts 2024-09-02T10:19:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:19:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:19:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: pgmap v1312: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 4.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 8.4 MiB/s, 134 objects/s recovering 2024-09-02T10:19:54.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:19:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:19:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: osdmap e566: 12 total, 12 up, 11 in 2024-09-02T10:19:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: 6.4 scrub ok 2024-09-02T10:19:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/976200525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3643663350' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:54.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:54 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:54.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:54.793 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:54.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: 6.2 scrub starts 2024-09-02T10:19:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:19:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:19:54.802 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: pgmap v1312: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 4.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 8.4 MiB/s, 134 objects/s recovering 2024-09-02T10:19:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:19:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:19:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: osdmap e566: 12 total, 12 up, 11 in 2024-09-02T10:19:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: 6.4 scrub ok 2024-09-02T10:19:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/976200525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3643663350' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:54.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:54 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: 6.2 scrub starts 2024-09-02T10:19:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:19:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:19:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: pgmap v1312: 47 pgs: 1 active+undersized+remapped, 46 active+clean; 4.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 77 MiB/s wr, 1.23k op/s; 8.4 MiB/s, 134 objects/s recovering 2024-09-02T10:19:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:19:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:19:54.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: osdmap e566: 12 total, 12 up, 11 in 2024-09-02T10:19:54.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: 6.4 scrub ok 2024-09-02T10:19:54.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/976200525' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:19:54.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3643663350' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:54.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:54 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:19:54.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:54.918 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:54.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:54.919 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:54.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 74 16 83516 83500 70.5179 58.875 0.00674422 0.0141761 2024-09-02T10:19:55.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:55.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:55.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:55.069 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:19:55.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:55.177 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:55.178 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:55.279 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:55.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:55.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:55.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:55.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:55.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:55.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:55.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:55.472 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:55.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:55.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:55.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:55.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:55.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:55.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:55.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:55 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:19:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:55 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:19:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:55 smithi187 ceph-mon[20072]: 6.e scrub starts 2024-09-02T10:19:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:55 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:19:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:55 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:19:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:55 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:19:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:55 smithi187 ceph-mon[20072]: osdmap e567: 12 total, 12 up, 11 in 2024-09-02T10:19:55.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:55.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:55.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:55 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:19:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:55 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:19:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:55 smithi162 ceph-mon[20279]: 6.e scrub starts 2024-09-02T10:19:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:55 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:19:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:55 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:19:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:55 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:19:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:55 smithi162 ceph-mon[20279]: osdmap e567: 12 total, 12 up, 11 in 2024-09-02T10:19:55.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:55.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:55 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:19:55.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:55 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:19:55.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:55 smithi151 ceph-mon[16639]: 6.e scrub starts 2024-09-02T10:19:55.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:55 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:19:55.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:55 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:19:55.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:55 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:19:55.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:55 smithi151 ceph-mon[16639]: osdmap e567: 12 total, 12 up, 11 in 2024-09-02T10:19:55.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:55.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:55.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:55.936 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:55.952 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 75 16 84334 84318 70.2593 51.125 0.018811 0.0142265 2024-09-02T10:19:56.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:56.056 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:56.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:56.057 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:56.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:56.189 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:56.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:56.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:56.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:56.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:56.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:56.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:56.445 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:56.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:56.446 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:56.547 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:19:56.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:56.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:56.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:19:56.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:56 smithi151 ceph-mon[16639]: 6.2 scrub ok 2024-09-02T10:19:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:56 smithi151 ceph-mon[16639]: 6.7 scrub starts 2024-09-02T10:19:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:56 smithi151 ceph-mon[16639]: pgmap v1315: 47 pgs: 2 active+clean+scrubbing, 1 active+undersized+remapped, 44 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 55 MiB/s wr, 876 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:19:56.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:56.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:56.697 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:56 smithi187 ceph-mon[20072]: 6.2 scrub ok 2024-09-02T10:19:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:56 smithi187 ceph-mon[20072]: 6.7 scrub starts 2024-09-02T10:19:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:56 smithi187 ceph-mon[20072]: pgmap v1315: 47 pgs: 2 active+clean+scrubbing, 1 active+undersized+remapped, 44 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 55 MiB/s wr, 876 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:19:56.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:56.816 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:19:56.816 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1468,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":567,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":11,"osd_in_since":1725272292,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":44},{"state_name":"active+clean+scrubbing","count":2},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":81772,"data_bytes":5359469088,"bytes_used":21292371968,"bytes_avail":1034653020160,"bytes_total":1055945392128,"recovering_objects_per_sec":97,"recovering_bytes_per_sec":6377426,"recovering_keys_per_sec":0,"num_objects_recovered":584,"num_bytes_recovered":38273024,"num_keys_recovered":0,"write_bytes_sec":57462358,"read_op_per_sec":0,"write_op_per_sec":876},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":35,"modified":"2024-09-02T10:18:59.332463+0000","services":{}},"progress_events":{}} 2024-09-02T10:19:56.824 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:56.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:56 smithi162 ceph-mon[20279]: 6.2 scrub ok 2024-09-02T10:19:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:56 smithi162 ceph-mon[20279]: 6.7 scrub starts 2024-09-02T10:19:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:56 smithi162 ceph-mon[20279]: pgmap v1315: 47 pgs: 2 active+clean+scrubbing, 1 active+undersized+remapped, 44 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 55 MiB/s wr, 876 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:19:56.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:56.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 76 16 85409 85393 70.2188 67.1875 0.0169317 0.0142359 2024-09-02T10:19:56.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:56.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:56.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:57.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:57.087 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:57.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:57.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:57.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:57.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:57.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:57.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:19:57.263 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:19:57.263 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:19:57.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:41902 4.14 e353/351 4:2ffda43f:::benchmark_data_smithi187_76542_object41901:head v 353'2264, mlcod=353'2262)", 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.743167+0000", 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 489.22979842799998, 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50111127099999997, 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743167+0000", 2024-09-02T10:19:57.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743167+0000", 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743168+0000", 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8599999999999996e-07 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743268+0000", 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9846000000000006e-05 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243322+0000", 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50005331600000003 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243327+0000", 2024-09-02T10:19:57.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.057e-06 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243388+0000", 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0948e-05 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243407+0000", 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9040000000000001e-05 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.244241+0000", 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00083470799999999998 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.244279+0000", 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7370000000000003e-05 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.268 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90150 4.4 4:22e9747b:::benchmark_data_smithi187_76542_object90149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.040493+0000", 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 445.93247269300002, 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.181265486, 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90150 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:57.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040493+0000", 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040493+0000", 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040495+0000", 2024-09-02T10:19:57.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4130000000000001e-06 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040588+0000", 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3113000000000001e-05 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040600+0000", 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1834000000000001e-05 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:57.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040605+0000", 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7049999999999996e-06 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040652+0000", 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6643e-05 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040741+0000", 2024-09-02T10:19:57.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8797999999999999e-05 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040826+0000", 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5191000000000005e-05 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042413+0000", 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001587605 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:57.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042531+0000", 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117157 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221691+0000", 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17916051699999999 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221731+0000", 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9650999999999997e-05 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221759+0000", 2024-09-02T10:19:57.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7858999999999999e-05 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:90150 4.4 e375/341 ondisk, result = 0)", 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.042430+0000", 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 445.930535857, 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17933455100000001, 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:57.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042430+0000", 2024-09-02T10:19:57.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:57.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042430+0000", 2024-09-02T10:19:57.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:57.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042431+0000", 2024-09-02T10:19:57.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9999999999999996e-07 2024-09-02T10:19:57.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:57.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042439+0000", 2024-09-02T10:19:57.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4810000000000006e-06 2024-09-02T10:19:57.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:57.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221645+0000", 2024-09-02T10:19:57.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17920517899999999 2024-09-02T10:19:57.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:57.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221649+0000", 2024-09-02T10:19:57.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.865e-06 2024-09-02T10:19:57.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221683+0000", 2024-09-02T10:19:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3324000000000001e-05 2024-09-02T10:19:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:57.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221688+0000", 2024-09-02T10:19:57.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.887e-06 2024-09-02T10:19:57.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:57.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221765+0000", 2024-09-02T10:19:57.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6915000000000005e-05 2024-09-02T10:19:57.282 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.282 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:57.282 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90160 4.4 4:2097a6ca:::benchmark_data_smithi187_76542_object90159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.048271+0000", 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 445.92469456399999, 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17505789499999999, 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90160 2024-09-02T10:19:57.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048271+0000", 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048271+0000", 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048276+0000", 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.938e-06 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048319+0000", 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2793999999999997e-05 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048330+0000", 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.118e-05 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048335+0000", 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2660000000000001e-06 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048380+0000", 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4733e-05 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048474+0000", 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3494999999999998e-05 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:19:57.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048562+0000", 2024-09-02T10:19:57.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8528000000000003e-05 2024-09-02T10:19:57.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:19:57.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.049898+0000", 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001336159 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050229+0000", 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033104400000000001 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223275+0000", 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.173045588 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:57.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223306+0000", 2024-09-02T10:19:57.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0621999999999999e-05 2024-09-02T10:19:57.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:57.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223329+0000", 2024-09-02T10:19:57.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3547999999999999e-05 2024-09-02T10:19:57.289 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.289 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:57.289 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.290 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:90160 4.4 e375/341 ondisk, result = 0)", 2024-09-02T10:19:57.290 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:19:57.290 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 445.92261004199997, 2024-09-02T10:19:57.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.172980419, 2024-09-02T10:19:57.290 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:57.290 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:57.291 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:19:57.291 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:57.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:57.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:19:57.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:57.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:19:57.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:57.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:19:57.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4600000000000005e-07 2024-09-02T10:19:57.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:57.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050360+0000", 2024-09-02T10:19:57.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.253e-06 2024-09-02T10:19:57.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:57.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223231+0000", 2024-09-02T10:19:57.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.172871466 2024-09-02T10:19:57.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:57.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223235+0000", 2024-09-02T10:19:57.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7309999999999999e-06 2024-09-02T10:19:57.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:57.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223266+0000", 2024-09-02T10:19:57.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1443999999999999e-05 2024-09-02T10:19:57.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:57.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223272+0000", 2024-09-02T10:19:57.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3249999999999998e-06 2024-09-02T10:19:57.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:57.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223336+0000", 2024-09-02T10:19:57.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4653999999999995e-05 2024-09-02T10:19:57.296 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.296 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:57.296 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.297 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:90175 4.19 e375/354 4:988493fd:::benchmark_data_smithi187_76542_object90174:head v 375'3736, mlcod=375'3734)", 2024-09-02T10:19:57.297 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.066480+0000", 2024-09-02T10:19:57.297 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 445.90648555299998, 2024-09-02T10:19:57.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.188910198, 2024-09-02T10:19:57.297 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:19:57.297 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:19:57.297 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:19:57.297 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:19:57.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:19:57.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066480+0000", 2024-09-02T10:19:57.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:19:57.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066480+0000", 2024-09-02T10:19:57.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:19:57.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:19:57.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066482+0000", 2024-09-02T10:19:57.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3939999999999999e-06 2024-09-02T10:19:57.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:19:57.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066535+0000", 2024-09-02T10:19:57.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3510000000000001e-05 2024-09-02T10:19:57.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:19:57.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253612+0000", 2024-09-02T10:19:57.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18707629000000001 2024-09-02T10:19:57.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:19:57.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253619+0000", 2024-09-02T10:19:57.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9719999999999992e-06 2024-09-02T10:19:57.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:19:57.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253697+0000", 2024-09-02T10:19:57.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7881999999999997e-05 2024-09-02T10:19:57.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:19:57.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253728+0000", 2024-09-02T10:19:57.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1118000000000003e-05 2024-09-02T10:19:57.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:19:57.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.255346+0000", 2024-09-02T10:19:57.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016170609999999999 2024-09-02T10:19:57.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:19:57.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:19:57.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:19:57.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.255391+0000", 2024-09-02T10:19:57.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4971e-05 2024-09-02T10:19:57.303 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.303 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:57.303 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.304 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:19:57.304 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:19:57.304 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:19:57.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:19:57.347 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:57.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:19:57.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:57.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:57.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:57.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:57.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:57 smithi151 ceph-mon[16639]: 6.e scrub ok 2024-09-02T10:19:57.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:57 smithi151 ceph-mon[16639]: 6.7 scrub ok 2024-09-02T10:19:57.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2124641636' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:19:57.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:57 smithi151 ceph-mon[16639]: 6.6 scrub starts 2024-09-02T10:19:57.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:57.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:57.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:19:57.737 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:57.738 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:19:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:57 smithi187 ceph-mon[20072]: 6.e scrub ok 2024-09-02T10:19:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:57 smithi187 ceph-mon[20072]: 6.7 scrub ok 2024-09-02T10:19:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2124641636' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:19:57.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:57 smithi187 ceph-mon[20072]: 6.6 scrub starts 2024-09-02T10:19:57.839 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:57 smithi162 ceph-mon[20279]: 6.e scrub ok 2024-09-02T10:19:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:57 smithi162 ceph-mon[20279]: 6.7 scrub ok 2024-09-02T10:19:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2124641636' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:19:57.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:57 smithi162 ceph-mon[20279]: 6.6 scrub starts 2024-09-02T10:19:57.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:57.864 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:57.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 77 16 86571 86555 70.25 72.625 0.00434433 0.0142286 2024-09-02T10:19:57.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:19:57.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:57.987 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:57.988 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:58.078 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:19:58.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:58.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:58.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:58.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:58.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:19:58.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:58.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:19:58.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:58.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:58.416 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:58.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:58.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:58.540 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:19:58.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:19:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:58.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:58.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:58.621 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:58.635 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:58.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:58.677 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:19:58.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:58.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:58 smithi187 ceph-mon[20072]: pgmap v1316: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 59 MiB/s wr, 938 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:19:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:58 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:19:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:19:58 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:19:58.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:58 smithi151 ceph-mon[16639]: pgmap v1316: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 59 MiB/s wr, 938 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:19:58.765 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:58 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:19:58.766 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:19:58 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:19:58.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:58.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:58.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:58 smithi162 ceph-mon[20279]: pgmap v1316: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.1 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 59 MiB/s wr, 938 op/s; 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:19:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:58 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:19:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:19:58 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:19:58.941 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:19:58.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 78 15 87591 87576 70.1674 63.8125 0.00720449 0.0142467 2024-09-02T10:19:58.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:58.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:58.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:19:59.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:19:59.113 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:59.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:19:59.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:19:59.246 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:59.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:19:59.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:19:59.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:19:59.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:59.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:19:59.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:59.510 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:59.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:59.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:19:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:59.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:59.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:19:59.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:19:59.763 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:19:59.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:59.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:19:59.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:19:59.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:19:59.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:19:59.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:19:59.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 79 16 88714 88698 70.1668 70.125 0.0102328 0.0142472 2024-09-02T10:19:59.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:00.025 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:00.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:00.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:00.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:00.157 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:00.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:00.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:00.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:00.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:00.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:00.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:00 smithi151 ceph-mon[16639]: 6.6 scrub ok 2024-09-02T10:20:00.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:00 smithi151 ceph-mon[16639]: 6.9 scrub starts 2024-09-02T10:20:00.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:00 smithi151 ceph-mon[16639]: 6.f scrub starts 2024-09-02T10:20:00.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:00 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:20:00.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:00 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:20:00.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:00 smithi151 ceph-mon[16639]: osdmap e568: 12 total, 12 up, 11 in 2024-09-02T10:20:00.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:00 smithi151 ceph-mon[16639]: pgmap v1318: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-02T10:20:00.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:00 smithi151 ceph-mon[16639]: overall HEALTH_OK 2024-09-02T10:20:00.318 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:20:00.318 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:20:00.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.318 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:89098 6.b 6:dc638e63:::benchmark_data_smithi187_102448_object89097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e568)", 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:00.300875+0000", 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.015893424, 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015928222999999998, 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 89098 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.300875+0000", 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.300875+0000", 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.300879+0000", 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1459999999999998e-06 2024-09-02T10:20:00.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.300919+0000", 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.968e-05 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.300931+0000", 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2553e-05 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:00.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.300938+0000", 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8889999999999996e-06 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.300988+0000", 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9713000000000003e-05 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.301114+0000", 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126243 2024-09-02T10:20:00.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.301246+0000", 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131225 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.303723+0000", 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024773989999999999 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:89109 6.6 6:694b3384:::benchmark_data_smithi187_102448_object89108:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e568)", 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:00.311684+0000", 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0050846090000000004, 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0052104960000000002, 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 89109 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.324 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.311684+0000", 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.311684+0000", 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.311685+0000", 2024-09-02T10:20:00.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9999999999999997e-07 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.311733+0000", 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8390000000000003e-05 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.311743+0000", 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0059e-05 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:00.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.311753+0000", 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9590000000000007e-06 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.311810+0000", 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6749999999999997e-05 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.311931+0000", 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012105299999999999 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.312098+0000", 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000167527 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.314915+0000", 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028162690000000002 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.315480+0000", 2024-09-02T10:20:00.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056498699999999998 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:89112 6.7 e568/565 6:e7aedea7:::benchmark_data_smithi187_102448_object89111:head v 568'5614, mlcod=568'5612)", 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:00.316150+0000", 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.00061851600000000005, 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00079206000000000003, 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.316150+0000", 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.316150+0000", 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.316151+0000", 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8100000000000002e-07 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.316210+0000", 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9511e-05 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.316229+0000", 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8278999999999999e-05 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.316235+0000", 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8469999999999998e-06 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.316279+0000", 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3569000000000002e-05 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:00.316292+0000", 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3022e-05 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:00.332 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:20:00.333 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 3 2024-09-02T10:20:00.333 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:20:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:00 smithi162 ceph-mon[20279]: 6.6 scrub ok 2024-09-02T10:20:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:00 smithi162 ceph-mon[20279]: 6.9 scrub starts 2024-09-02T10:20:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:00 smithi162 ceph-mon[20279]: 6.f scrub starts 2024-09-02T10:20:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:00 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:20:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:00 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:20:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:00 smithi162 ceph-mon[20279]: osdmap e568: 12 total, 12 up, 11 in 2024-09-02T10:20:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:00 smithi162 ceph-mon[20279]: pgmap v1318: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-02T10:20:00.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:00 smithi162 ceph-mon[20279]: overall HEALTH_OK 2024-09-02T10:20:00.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:00.417 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:00.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:00.479 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:00 smithi187 ceph-mon[20072]: 6.6 scrub ok 2024-09-02T10:20:00.479 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:00 smithi187 ceph-mon[20072]: 6.9 scrub starts 2024-09-02T10:20:00.479 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:00 smithi187 ceph-mon[20072]: 6.f scrub starts 2024-09-02T10:20:00.479 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:00 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:20:00.479 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:00 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:20:00.479 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:00 smithi187 ceph-mon[20072]: osdmap e568: 12 total, 12 up, 11 in 2024-09-02T10:20:00.480 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:00 smithi187 ceph-mon[20072]: pgmap v1318: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 56 MiB/s wr, 889 op/s 2024-09-02T10:20:00.480 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:00 smithi187 ceph-mon[20072]: overall HEALTH_OK 2024-09-02T10:20:00.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:00.552 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:00.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:00.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:00.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:00.686 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:00.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:00.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:00.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:00.811 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:00.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:00.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:00.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:00.937 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:00.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:20:00.952935+0000 min lat: 0.00104401 max lat: 2.84044 avg lat: 0.0142533 2024-09-02T10:20:00.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:20:00.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 80 16 89797 89781 70.1358 67.6875 0.0175217 0.0142533 2024-09-02T10:20:01.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:01.067 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:01.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:01.085 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:20:01.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:01.196 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:01.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:01.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:01.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:01.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:01.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:01.342 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:01 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:20:01.342 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:01 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:20:01.342 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:01 smithi162 ceph-mon[20279]: 6.f scrub ok 2024-09-02T10:20:01.342 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:01 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:20:01.342 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:01 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:20:01.343 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:01 smithi162 ceph-mon[20279]: 6.9 scrub ok 2024-09-02T10:20:01.343 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:01 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:20:01.343 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:01 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:01.343 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2770610569' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:20:01.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:01 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:20:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:01 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:20:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:01 smithi151 ceph-mon[16639]: 6.f scrub ok 2024-09-02T10:20:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:01 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:20:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:01 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:20:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:01 smithi151 ceph-mon[16639]: 6.9 scrub ok 2024-09-02T10:20:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:01 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:20:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:01 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2770610569' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:20:01.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:01.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:01 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:20:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:01 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:20:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:01 smithi187 ceph-mon[20072]: 6.f scrub ok 2024-09-02T10:20:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:01 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:20:01.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:01 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:20:01.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:01 smithi187 ceph-mon[20072]: 6.9 scrub ok 2024-09-02T10:20:01.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:01 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:20:01.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:01 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:01.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2770610569' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:20:01.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:01.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:01.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:01.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:01.712 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:01.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:01.814 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:01.837 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:01.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:01.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:01.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:20:01.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:01.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 81 15 90857 90842 70.0885 66.3125 0.0225329 0.0142619 2024-09-02T10:20:02.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:02.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:02.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:02.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:02.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:02.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:02 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:20:02.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:02 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:20:02.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:02 smithi151 ceph-mon[16639]: 6.c scrub starts 2024-09-02T10:20:02.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:02 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:02.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2770610569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:20:02.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:02 smithi151 ceph-mon[16639]: osdmap e569: 12 total, 12 up, 11 in 2024-09-02T10:20:02.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:02 smithi151 ceph-mon[16639]: 6.a scrub starts 2024-09-02T10:20:02.180 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:02 smithi151 ceph-mon[16639]: pgmap v1320: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-02T10:20:02.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:02.263 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:02.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:02 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:20:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:02 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:20:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:02 smithi162 ceph-mon[20279]: 6.c scrub starts 2024-09-02T10:20:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:02 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2770610569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:20:02.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:02 smithi162 ceph-mon[20279]: osdmap e569: 12 total, 12 up, 11 in 2024-09-02T10:20:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:02 smithi162 ceph-mon[20279]: 6.a scrub starts 2024-09-02T10:20:02.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:02 smithi162 ceph-mon[20279]: pgmap v1320: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-02T10:20:02.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:02.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:02.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:02.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:02 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:20:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:02 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:20:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:02 smithi187 ceph-mon[20072]: 6.c scrub starts 2024-09-02T10:20:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:02 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2770610569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:20:02.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:02 smithi187 ceph-mon[20072]: osdmap e569: 12 total, 12 up, 11 in 2024-09-02T10:20:02.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:02 smithi187 ceph-mon[20072]: 6.a scrub starts 2024-09-02T10:20:02.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:02 smithi187 ceph-mon[20072]: pgmap v1320: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 56 MiB/s wr, 895 op/s 2024-09-02T10:20:02.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:02.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:02.566 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:02.566 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:02.566 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:02.566 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:02.566 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:02.566 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:02.566 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:02.567 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:02.567 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:02.567 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:02.567 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:02.567 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:02.567 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:02.567 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:02.567 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:02.567 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:02.567 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:02.568 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:02.568 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:02.568 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:02.568 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:02.568 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:02.568 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:02.568 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:02.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:02.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:02.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:02.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:02.787 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:02.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:02.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:02.919 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:02.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:02.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 82 16 92009 91993 70.111 71.9375 0.0138358 0.0142579 2024-09-02T10:20:03.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:03.045 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:03.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:03.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:03.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/461147287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:03.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:03.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:03.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:03.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:03.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:03.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/461147287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:03.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:03.433 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:03.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/461147287' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:03.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:03.562 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:03.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:03.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:03.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:03.694 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:03.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:03.825 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:03.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:03.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:03.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 83 16 93151 93135 70.1261 71.375 0.0520707 0.0142551 2024-09-02T10:20:03.958 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:03.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:03.967 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:03.974 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:04.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:04.091 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:04.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:04.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:04 smithi151 ceph-mon[16639]: pgmap v1321: 47 pgs: 47 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:20:04.173 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:04 smithi151 ceph-mon[16639]: osdmap e570: 12 total, 12 up, 11 in 2024-09-02T10:20:04.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:04.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:04.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:04.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:04 smithi162 ceph-mon[20279]: pgmap v1321: 47 pgs: 47 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:20:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:04 smithi162 ceph-mon[20279]: osdmap e570: 12 total, 12 up, 11 in 2024-09-02T10:20:04.356 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:04.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:04.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:04.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:04.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:04.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:04.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:04 smithi187 ceph-mon[20072]: pgmap v1321: 47 pgs: 47 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:20:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:04 smithi187 ceph-mon[20072]: osdmap e570: 12 total, 12 up, 11 in 2024-09-02T10:20:04.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:04.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:04.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:04.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:04.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:04.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:04.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:04.874 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:04.875 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:04.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:04.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:04.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 84 16 94281 94265 70.132 70.625 0.0152186 0.0142546 2024-09-02T10:20:04.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:05.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:05.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:05.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:05.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:05.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:05.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:05.266 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:05.266 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:05.368 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:05.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:05.395 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:05.396 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:20:05.397 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:20:05.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:20:05.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:05.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:05.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:05.629 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:05.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:05.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:05.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:05.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:05.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:05.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:05.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:05.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:05.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 85 16 95301 95285 70.0569 63.75 0.00712009 0.0142676 2024-09-02T10:20:05.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:20:06.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:06.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:06.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:06.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:06.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:06.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:06.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:06.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:06.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:20:06.273 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:06.298 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:06.298 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:06.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:06.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:06.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:06.424 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:06.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:06.554 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:06.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:06.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:06.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:06.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:06.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:06.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:06 smithi151 ceph-mon[16639]: pgmap v1323: 47 pgs: 47 active+clean; 5.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-02T10:20:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:06 smithi151 ceph-mon[16639]: from='client.26689 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:06 smithi187 ceph-mon[20072]: pgmap v1323: 47 pgs: 47 active+clean; 5.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-02T10:20:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:06 smithi187 ceph-mon[20072]: from='client.26689 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:06.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:06.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:06.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:06.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:06 smithi162 ceph-mon[20279]: pgmap v1323: 47 pgs: 47 active+clean; 5.6 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 64 MiB/s wr, 1.03k op/s 2024-09-02T10:20:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:06 smithi162 ceph-mon[20279]: from='client.26689 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:06.854 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1323,"stamp":"2024-09-02T10:20:05.352695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6017581600,"num_objects":91814,"num_object_clones":0,"num_object_copies":275442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91812,"num_write_kb":5875968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8541,"num_bytes_recovered":560276544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94664,"ondisk_log_size":94664,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":22608760,"kb_used_data":17817500,"kb_used_omap":17,"kb_used_meta":4790958,"kb_avail":1008587912,"statfs":{"total":1055945392128,"available":1032794021888,"internally_reserved":0,"allocated":18245120000,"data_stored":18240221917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":4905941878},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,5,2,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":424214528,"num_objects":6473,"num_object_clones":0,"num_object_copies":19419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6473,"num_write_kb":414272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.287070"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":483,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.085619+0000","last_change":"2024-09-02T10:18:43.049145+0000","last_active":"2024-09-02T10:20:01.085619+0000","last_peered":"2024-09-02T10:20:01.085619+0000","last_clean":"2024-09-02T10:20:01.085619+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:20:01.085619+0000","last_undegraded":"2024-09-02T10:20:01.085619+0000","last_fullsized":"2024-09-02T10:20:01.085619+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:43.049077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:43.049077+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:07.603384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":298,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.096378+0000","last_change":"2024-09-02T10:19:47.121714+0000","last_active":"2024-09-02T10:20:01.096378+0000","last_peered":"2024-09-02T10:20:01.096378+0000","last_clean":"2024-09-02T10:20:01.096378+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:20:01.096378+0000","last_undegraded":"2024-09-02T10:20:01.096378+0000","last_fullsized":"2024-09-02T10:20:01.096378+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:04.209863+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":750,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-09-02T10:20:04.068641+0000","last_change":"2024-09-02T10:19:48.146767+0000","last_active":"2024-09-02T10:20:04.068641+0000","last_peered":"2024-09-02T10:20:04.068641+0000","last_clean":"2024-09-02T10:20:04.068641+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:20:04.068641+0000","last_undegraded":"2024-09-02T10:20:04.068641+0000","last_fullsized":"2024-09-02T10:20:04.068641+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:48.146721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:48.146721+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:55.994378+0000","scrub_duration":995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":742,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-09-02T10:20:04.068499+0000","last_change":"2024-09-02T10:18:39.045140+0000","last_active":"2024-09-02T10:20:04.068499+0000","last_peered":"2024-09-02T10:20:04.068499+0000","last_clean":"2024-09-02T10:20:04.068499+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:20:04.068499+0000","last_undegraded":"2024-09-02T10:20:04.068499+0000","last_fullsized":"2024-09-02T10:20:04.068499+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:39.045083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:39.045083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":54,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-09-02T10:20:04.071874+0000","last_change":"2024-09-02T10:19:05.058923+0000","last_active":"2024-09-02T10:20:04.071874+0000","last_peered":"2024-09-02T10:20:04.071874+0000","last_clean":"2024-09-02T10:20:04.071874+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:20:04.071874+0000","last_undegraded":"2024-09-02T10:20:04.071874+0000","last_fullsized":"2024-09-02T10:20:04.071874+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:01.810089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:01.810089+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:19.405935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":597,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.093675+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:20:01.093675+0000","last_peered":"2024-09-02T10:20:01.093675+0000","last_clean":"2024-09-02T10:20:01.093675+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:20:01.093675+0000","last_undegraded":"2024-09-02T10:20:01.093675+0000","last_fullsized":"2024-09-02T10:20:01.093675+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:08.487285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1113,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.092927+0000","last_change":"2024-09-02T10:19:03.880037+0000","last_active":"2024-09-02T10:20:01.092927+0000","last_peered":"2024-09-02T10:20:01.092927+0000","last_clean":"2024-09-02T10:20:01.092927+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:20:01.092927+0000","last_undegraded":"2024-09-02T10:20:01.092927+0000","last_fullsized":"2024-09-02T10:20:01.092927+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:03.879968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:03.879968+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.678816+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":765,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-09-02T10:20:04.068770+0000","last_change":"2024-09-02T10:20:00.175101+0000","last_active":"2024-09-02T10:20:04.068770+0000","last_peered":"2024-09-02T10:20:04.068770+0000","last_clean":"2024-09-02T10:20:04.068770+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:20:04.068770+0000","last_undegraded":"2024-09-02T10:20:04.068770+0000","last_fullsized":"2024-09-02T10:20:04.068770+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.175032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.175032+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:28.457490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":763,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.093882+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:20:01.093882+0000","last_peered":"2024-09-02T10:20:01.093882+0000","last_clean":"2024-09-02T10:20:01.093882+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:20:01.093882+0000","last_undegraded":"2024-09-02T10:20:01.093882+0000","last_fullsized":"2024-09-02T10:20:01.093882+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:52.877235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":731,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.090163+0000","last_change":"2024-09-02T10:19:05.910750+0000","last_active":"2024-09-02T10:20:01.090163+0000","last_peered":"2024-09-02T10:20:01.090163+0000","last_clean":"2024-09-02T10:20:01.090163+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:20:01.090163+0000","last_undegraded":"2024-09-02T10:20:01.090163+0000","last_fullsized":"2024-09-02T10:20:01.090163+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:05.910691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:05.910691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:20.353104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":583,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.090030+0000","last_change":"2024-09-02T10:19:52.963372+0000","last_active":"2024-09-02T10:20:01.090030+0000","last_peered":"2024-09-02T10:20:01.090030+0000","last_clean":"2024-09-02T10:20:01.090030+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:20:01.090030+0000","last_undegraded":"2024-09-02T10:20:01.090030+0000","last_fullsized":"2024-09-02T10:20:01.090030+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.963289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.963289+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:10.821299+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1088,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.094646+0000","last_change":"2024-09-02T10:19:00.052912+0000","last_active":"2024-09-02T10:20:01.094646+0000","last_peered":"2024-09-02T10:20:01.094646+0000","last_clean":"2024-09-02T10:20:01.094646+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:20:01.094646+0000","last_undegraded":"2024-09-02T10:20:01.094646+0000","last_fullsized":"2024-09-02T10:20:01.094646+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.052857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.052857+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:29.321752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1031,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.092603+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:20:01.092603+0000","last_peered":"2024-09-02T10:20:01.092603+0000","last_clean":"2024-09-02T10:20:01.092603+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:20:01.092603+0000","last_undegraded":"2024-09-02T10:20:01.092603+0000","last_fullsized":"2024-09-02T10:20:01.092603+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:57.041883+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1010,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.088678+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:20:01.088678+0000","last_peered":"2024-09-02T10:20:01.088678+0000","last_clean":"2024-09-02T10:20:01.088678+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:20:01.088678+0000","last_undegraded":"2024-09-02T10:20:01.088678+0000","last_fullsized":"2024-09-02T10:20:01.088678+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:01.846422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1001,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.092971+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:20:01.092971+0000","last_peered":"2024-09-02T10:20:01.092971+0000","last_clean":"2024-09-02T10:20:01.092971+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:20:01.092971+0000","last_undegraded":"2024-09-02T10:20:01.092971+0000","last_fullsized":"2024-09-02T10:20:01.092971+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.850900+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"569'5604","reported_seq":5746,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.700829+0000","last_change":"2024-09-02T10:19:59.986749+0000","last_active":"2024-09-02T10:20:01.700829+0000","last_peered":"2024-09-02T10:20:01.700829+0000","last_clean":"2024-09-02T10:20:01.700829+0000","last_became_active":"2024-09-02T10:18:37.599064+0000","last_became_peered":"2024-09-02T10:18:37.599064+0000","last_unstale":"2024-09-02T10:20:01.700829+0000","last_undegraded":"2024-09-02T10:20:01.700829+0000","last_fullsized":"2024-09-02T10:20:01.700829+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":5604,"log_dups_size":0,"ondisk_log_size":5604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:01.232192+0000","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367263744,"num_objects":5604,"num_object_clones":0,"num_object_copies":16812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5604,"num_write_kb":358656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1088,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.093058+0000","last_change":"2024-09-02T10:19:52.109223+0000","last_active":"2024-09-02T10:20:01.093058+0000","last_peered":"2024-09-02T10:20:01.093058+0000","last_clean":"2024-09-02T10:20:01.093058+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:20:01.093058+0000","last_undegraded":"2024-09-02T10:20:01.093058+0000","last_fullsized":"2024-09-02T10:20:01.093058+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:04.388339+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"570'5852","reported_seq":10231,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-09-02T10:20:04.190007+0000","last_change":"2024-09-02T10:19:55.677256+0000","last_active":"2024-09-02T10:20:04.190007+0000","last_peered":"2024-09-02T10:20:04.190007+0000","last_clean":"2024-09-02T10:20:04.190007+0000","last_became_active":"2024-09-02T10:19:32.596751+0000","last_became_peered":"2024-09-02T10:19:32.596751+0000","last_unstale":"2024-09-02T10:20:04.190007+0000","last_undegraded":"2024-09-02T10:20:04.190007+0000","last_fullsized":"2024-09-02T10:20:04.190007+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":5852,"log_dups_size":0,"ondisk_log_size":5852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:09.732905+0000","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383516672,"num_objects":5852,"num_object_clones":0,"num_object_copies":17556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5852,"num_write_kb":374528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":138346496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1203,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.089858+0000","last_change":"2024-09-02T10:19:49.817426+0000","last_active":"2024-09-02T10:20:01.089858+0000","last_peered":"2024-09-02T10:20:01.089858+0000","last_clean":"2024-09-02T10:20:01.089858+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:20:01.089858+0000","last_undegraded":"2024-09-02T10:20:01.089858+0000","last_fullsized":"2024-09-02T10:20:01.089858+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.817347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.817347+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:54.901580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"569'5624","reported_seq":5706,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:02.192041+0000","last_change":"2024-09-02T10:18:37.062429+0000","last_active":"2024-09-02T10:20:02.192041+0000","last_peered":"2024-09-02T10:20:02.192041+0000","last_clean":"2024-09-02T10:20:02.192041+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:20:02.192041+0000","last_undegraded":"2024-09-02T10:20:02.192041+0000","last_fullsized":"2024-09-02T10:20:02.192041+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":5624,"log_dups_size":0,"ondisk_log_size":5624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:07.018410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368574464,"num_objects":5624,"num_object_clones":0,"num_object_copies":16872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5624,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5624,"num_write_kb":359936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":852,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.090267+0000","last_change":"2024-09-02T10:19:05.058003+0000","last_active":"2024-09-02T10:20:01.090267+0000","last_peered":"2024-09-02T10:20:01.090267+0000","last_clean":"2024-09-02T10:20:01.090267+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:20:01.090267+0000","last_undegraded":"2024-09-02T10:20:01.090267+0000","last_fullsized":"2024-09-02T10:20:01.090267+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.206480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.206480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.672102+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"569'5716","reported_seq":5810,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:03.210076+0000","last_change":"2024-09-02T10:20:01.109999+0000","last_active":"2024-09-02T10:20:03.210076+0000","last_peered":"2024-09-02T10:20:03.210076+0000","last_clean":"2024-09-02T10:20:03.210076+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:20:03.210076+0000","last_undegraded":"2024-09-02T10:20:03.210076+0000","last_fullsized":"2024-09-02T10:20:03.210076+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":899,"log_size":5716,"log_dups_size":0,"ondisk_log_size":5716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:06.104769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374603776,"num_objects":5716,"num_object_clones":0,"num_object_copies":17148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5716,"num_write_kb":365824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":249,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.085724+0000","last_change":"2024-09-02T10:19:50.392195+0000","last_active":"2024-09-02T10:20:01.085724+0000","last_peered":"2024-09-02T10:20:01.085724+0000","last_clean":"2024-09-02T10:20:01.085724+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:20:01.085724+0000","last_undegraded":"2024-09-02T10:20:01.085724+0000","last_fullsized":"2024-09-02T10:20:01.085724+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.392146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.392146+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.406921+0000","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"569'5685","reported_seq":5766,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.221475+0000","last_change":"2024-09-02T10:18:37.850883+0000","last_active":"2024-09-02T10:20:01.221475+0000","last_peered":"2024-09-02T10:20:01.221475+0000","last_clean":"2024-09-02T10:20:01.221475+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:20:01.221475+0000","last_undegraded":"2024-09-02T10:20:01.221475+0000","last_fullsized":"2024-09-02T10:20:01.221475+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:05.087393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372572160,"num_objects":5685,"num_object_clones":0,"num_object_copies":17055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5685,"num_write_kb":363840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1158,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.085519+0000","last_change":"2024-09-02T10:19:51.832888+0000","last_active":"2024-09-02T10:20:01.085519+0000","last_peered":"2024-09-02T10:20:01.085519+0000","last_clean":"2024-09-02T10:20:01.085519+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:20:01.085519+0000","last_undegraded":"2024-09-02T10:20:01.085519+0000","last_fullsized":"2024-09-02T10:20:01.085519+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.832822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.832822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.770876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"570'5864","reported_seq":10290,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-09-02T10:20:04.192279+0000","last_change":"2024-09-02T10:19:54.548123+0000","last_active":"2024-09-02T10:20:04.192279+0000","last_peered":"2024-09-02T10:20:04.192279+0000","last_clean":"2024-09-02T10:20:04.192279+0000","last_became_active":"2024-09-02T10:19:31.588084+0000","last_became_peered":"2024-09-02T10:19:31.588084+0000","last_unstale":"2024-09-02T10:20:04.192279+0000","last_undegraded":"2024-09-02T10:20:04.192279+0000","last_fullsized":"2024-09-02T10:20:04.192279+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":5864,"log_dups_size":0,"ondisk_log_size":5864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:12.643132+0000","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384303104,"num_objects":5864,"num_object_clones":0,"num_object_copies":17592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5864,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5864,"num_write_kb":375296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2135,"num_bytes_recovered":139919360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":625,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.095226+0000","last_change":"2024-09-02T10:20:00.749882+0000","last_active":"2024-09-02T10:20:01.095226+0000","last_peered":"2024-09-02T10:20:01.095226+0000","last_clean":"2024-09-02T10:20:01.095226+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:20:01.095226+0000","last_undegraded":"2024-09-02T10:20:01.095226+0000","last_fullsized":"2024-09-02T10:20:01.095226+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.876682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"570'5775","reported_seq":5858,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-09-02T10:20:04.373546+0000","last_change":"2024-09-02T10:18:37.600687+0000","last_active":"2024-09-02T10:20:04.373546+0000","last_peered":"2024-09-02T10:20:04.373546+0000","last_clean":"2024-09-02T10:20:04.373546+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:20:04.373546+0000","last_undegraded":"2024-09-02T10:20:04.373546+0000","last_fullsized":"2024-09-02T10:20:04.373546+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":5775,"log_dups_size":0,"ondisk_log_size":5775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378470400,"num_objects":5775,"num_object_clones":0,"num_object_copies":17325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5775,"num_write_kb":369600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":597,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.093741+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:20:01.093741+0000","last_peered":"2024-09-02T10:20:01.093741+0000","last_clean":"2024-09-02T10:20:01.093741+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:20:01.093741+0000","last_undegraded":"2024-09-02T10:20:01.093741+0000","last_fullsized":"2024-09-02T10:20:01.093741+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:51.265964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"569'5794","reported_seq":5933,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:02.874356+0000","last_change":"2024-09-02T10:19:52.017169+0000","last_active":"2024-09-02T10:20:02.874356+0000","last_peered":"2024-09-02T10:20:02.874356+0000","last_clean":"2024-09-02T10:20:02.874356+0000","last_became_active":"2024-09-02T10:18:37.064844+0000","last_became_peered":"2024-09-02T10:18:37.064844+0000","last_unstale":"2024-09-02T10:20:02.874356+0000","last_undegraded":"2024-09-02T10:20:02.874356+0000","last_fullsized":"2024-09-02T10:20:02.874356+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:00.776952+0000","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379715584,"num_objects":5794,"num_object_clones":0,"num_object_copies":17382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5794,"num_write_kb":370816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":349,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.091493+0000","last_change":"2024-09-02T10:19:58.949821+0000","last_active":"2024-09-02T10:20:01.091493+0000","last_peered":"2024-09-02T10:20:01.091493+0000","last_clean":"2024-09-02T10:20:01.091493+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:20:01.091493+0000","last_undegraded":"2024-09-02T10:20:01.091493+0000","last_fullsized":"2024-09-02T10:20:01.091493+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:58.949745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.949745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.256455+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"569'5793","reported_seq":10296,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:03.220734+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:20:03.220734+0000","last_peered":"2024-09-02T10:20:03.220734+0000","last_clean":"2024-09-02T10:20:03.220734+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:20:03.220734+0000","last_undegraded":"2024-09-02T10:20:03.220734+0000","last_fullsized":"2024-09-02T10:20:03.220734+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":5793,"log_dups_size":0,"ondisk_log_size":5793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:22.534088+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379650048,"num_objects":5793,"num_object_clones":0,"num_object_copies":17379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5793,"num_write_kb":370752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":598,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.093754+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:20:01.093754+0000","last_peered":"2024-09-02T10:20:01.093754+0000","last_clean":"2024-09-02T10:20:01.093754+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:20:01.093754+0000","last_undegraded":"2024-09-02T10:20:01.093754+0000","last_fullsized":"2024-09-02T10:20:01.093754+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:02.710854+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"569'5652","reported_seq":5734,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.224736+0000","last_change":"2024-09-02T10:18:37.848300+0000","last_active":"2024-09-02T10:20:01.224736+0000","last_peered":"2024-09-02T10:20:01.224736+0000","last_clean":"2024-09-02T10:20:01.224736+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:20:01.224736+0000","last_undegraded":"2024-09-02T10:20:01.224736+0000","last_fullsized":"2024-09-02T10:20:01.224736+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":5652,"log_dups_size":0,"ondisk_log_size":5652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:01.700433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370409472,"num_objects":5652,"num_object_clones":0,"num_object_copies":16956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5652,"num_write_kb":361728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":844,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.089990+0000","last_change":"2024-09-02T10:19:05.058925+0000","last_active":"2024-09-02T10:20:01.089990+0000","last_peered":"2024-09-02T10:20:01.089990+0000","last_clean":"2024-09-02T10:20:01.089990+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:20:01.089990+0000","last_undegraded":"2024-09-02T10:20:01.089990+0000","last_fullsized":"2024-09-02T10:20:01.089990+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.789947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.789947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:10.614688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"569'5849","reported_seq":5991,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:03.666525+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:20:03.666525+0000","last_peered":"2024-09-02T10:20:03.666525+0000","last_clean":"2024-09-02T10:20:03.666525+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:20:03.666525+0000","last_undegraded":"2024-09-02T10:20:03.666525+0000","last_fullsized":"2024-09-02T10:20:03.666525+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":5849,"log_dups_size":0,"ondisk_log_size":5849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.641560+0000","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383320064,"num_objects":5849,"num_object_clones":0,"num_object_copies":17547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5849,"num_write_kb":374336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1119,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.094679+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:20:01.094679+0000","last_peered":"2024-09-02T10:20:01.094679+0000","last_clean":"2024-09-02T10:20:01.094679+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:20:01.094679+0000","last_undegraded":"2024-09-02T10:20:01.094679+0000","last_fullsized":"2024-09-02T10:20:01.094679+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:59.441621+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"569'5743","reported_seq":10134,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:02.208389+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:20:02.208389+0000","last_peered":"2024-09-02T10:20:02.208389+0000","last_clean":"2024-09-02T10:20:02.208389+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:20:02.208389+0000","last_undegraded":"2024-09-02T10:20:02.208389+0000","last_fullsized":"2024-09-02T10:20:02.208389+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":5743,"log_dups_size":0,"ondisk_log_size":5743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:23.125229+0000","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376373248,"num_objects":5743,"num_object_clones":0,"num_object_copies":17229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5743,"num_write_kb":367552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":863,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.089875+0000","last_change":"2024-09-02T10:19:59.731786+0000","last_active":"2024-09-02T10:20:01.089875+0000","last_peered":"2024-09-02T10:20:01.089875+0000","last_clean":"2024-09-02T10:20:01.089875+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:20:01.089875+0000","last_undegraded":"2024-09-02T10:20:01.089875+0000","last_fullsized":"2024-09-02T10:20:01.089875+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:59.731706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.731706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:16.407855+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"569'5687","reported_seq":5769,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.172363+0000","last_change":"2024-09-02T10:18:37.850899+0000","last_active":"2024-09-02T10:20:01.172363+0000","last_peered":"2024-09-02T10:20:01.172363+0000","last_clean":"2024-09-02T10:20:01.172363+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:20:01.172363+0000","last_undegraded":"2024-09-02T10:20:01.172363+0000","last_fullsized":"2024-09-02T10:20:01.172363+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":5687,"log_dups_size":0,"ondisk_log_size":5687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:02.178728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372703232,"num_objects":5687,"num_object_clones":0,"num_object_copies":17061,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5687,"num_write_kb":363968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1110,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.091522+0000","last_change":"2024-09-02T10:19:08.999797+0000","last_active":"2024-09-02T10:20:01.091522+0000","last_peered":"2024-09-02T10:20:01.091522+0000","last_clean":"2024-09-02T10:20:01.091522+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:20:01.091522+0000","last_undegraded":"2024-09-02T10:20:01.091522+0000","last_fullsized":"2024-09-02T10:20:01.091522+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:08.999706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:08.999706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:13.510899+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"569'5714","reported_seq":5860,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:02.870623+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:20:02.870623+0000","last_peered":"2024-09-02T10:20:02.870623+0000","last_clean":"2024-09-02T10:20:02.870623+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:20:02.870623+0000","last_undegraded":"2024-09-02T10:20:02.870623+0000","last_fullsized":"2024-09-02T10:20:02.870623+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":5714,"log_dups_size":0,"ondisk_log_size":5714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.389208+0000","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374472704,"num_objects":5714,"num_object_clones":0,"num_object_copies":17142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5714,"num_write_kb":365696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":62,"reported_epoch":570,"state":"active+clean","last_fresh":"2024-09-02T10:20:04.072136+0000","last_change":"2024-09-02T10:19:06.947029+0000","last_active":"2024-09-02T10:20:04.072136+0000","last_peered":"2024-09-02T10:20:04.072136+0000","last_clean":"2024-09-02T10:20:04.072136+0000","last_became_active":"2024-09-02T10:19:06.049121+0000","last_became_peered":"2024-09-02T10:19:06.049121+0000","last_unstale":"2024-09-02T10:20:04.072136+0000","last_undegraded":"2024-09-02T10:20:04.072136+0000","last_fullsized":"2024-09-02T10:20:04.072136+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:06.946957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:06.946957+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:24.543000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"569'5688","reported_seq":5772,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:02.209784+0000","last_change":"2024-09-02T10:20:01.132876+0000","last_active":"2024-09-02T10:20:02.209784+0000","last_peered":"2024-09-02T10:20:02.209784+0000","last_clean":"2024-09-02T10:20:02.209784+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:20:02.209784+0000","last_undegraded":"2024-09-02T10:20:02.209784+0000","last_fullsized":"2024-09-02T10:20:02.209784+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":5688,"log_dups_size":0,"ondisk_log_size":5688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:06.108619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372768768,"num_objects":5688,"num_object_clones":0,"num_object_copies":17064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5688,"num_write_kb":364032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":890,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.090220+0000","last_change":"2024-09-02T10:19:57.798578+0000","last_active":"2024-09-02T10:20:01.090220+0000","last_peered":"2024-09-02T10:20:01.090220+0000","last_clean":"2024-09-02T10:20:01.090220+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:20:01.090220+0000","last_undegraded":"2024-09-02T10:20:01.090220+0000","last_fullsized":"2024-09-02T10:20:01.090220+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:57.798501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:57.798501+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:05.189778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"569'5772","reported_seq":5854,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:03.170010+0000","last_change":"2024-09-02T10:18:37.066015+0000","last_active":"2024-09-02T10:20:03.170010+0000","last_peered":"2024-09-02T10:20:03.170010+0000","last_clean":"2024-09-02T10:20:03.170010+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:20:03.170010+0000","last_undegraded":"2024-09-02T10:20:03.170010+0000","last_fullsized":"2024-09-02T10:20:03.170010+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":5772,"log_dups_size":0,"ondisk_log_size":5772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378273792,"num_objects":5772,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5772,"num_write_kb":369408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":282,"reported_epoch":569,"state":"active+clean","last_fresh":"2024-09-02T10:20:01.101249+0000","last_change":"2024-09-02T10:19:02.956010+0000","last_active":"2024-09-02T10:20:01.101249+0000","last_peered":"2024-09-02T10:20:01.101249+0000","last_clean":"2024-09-02T10:20:01.101249+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:20:01.101249+0000","last_undegraded":"2024-09-02T10:20:01.101249+0000","last_fullsized":"2024-09-02T10:20:01.101249+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:02.955947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:02.955947+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:26.242539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":6016991232,"num_objects":91812,"num_object_clones":0,"num_object_copies":275436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":91812,"num_write_kb":5875968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8538,"num_bytes_recovered":559546368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18315280384,"data_stored":18315280384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91812,"ondisk_log_size":91812,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383919,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1564836,"kb_used_data":1100492,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92180316,"statfs":{"total":95995035648,"available":94392643584,"internally_reserved":0,"allocated":1126903808,"data_stored":1126459501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":475462084},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612852,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1395552,"kb_used_data":1075596,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92349600,"statfs":{"total":95995035648,"available":94565990400,"internally_reserved":0,"allocated":1101410304,"data_stored":1100965997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":327612864},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841785,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1962016,"kb_used_data":1461580,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91783136,"statfs":{"total":95995035648,"available":93985931264,"internally_reserved":0,"allocated":1496657920,"data_stored":1496213613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":512424384},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070719,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1142060,"kb_used_data":813784,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92603092,"statfs":{"total":95995035648,"available":94825566208,"internally_reserved":0,"allocated":833314816,"data_stored":832866056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299651,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2365916,"kb_used_data":1879368,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91379236,"statfs":{"total":95995035648,"available":93572337664,"internally_reserved":0,"allocated":1924472832,"data_stored":1924029376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528583,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1790368,"kb_used_data":1456524,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91954784,"statfs":{"total":95995035648,"available":94161698816,"internally_reserved":0,"allocated":1491480576,"data_stored":1491036269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":341834183},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724812,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3429352,"kb_used_data":2906708,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":90315800,"statfs":{"total":95995035648,"available":92483379200,"internally_reserved":0,"allocated":2976468992,"data_stored":2976021133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953746,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1791196,"kb_used_data":1485132,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":91953956,"statfs":{"total":95995035648,"available":94160850944,"internally_reserved":0,"allocated":1520775168,"data_stored":1520330861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149975,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3470560,"kb_used_data":2956236,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":90274592,"statfs":{"total":95995035648,"available":92441182208,"internally_reserved":0,"allocated":3027185664,"data_stored":3026741357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":526645704},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411610,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575136,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2051816,"kb_used_data":1591636,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91693336,"statfs":{"total":95995035648,"available":93893976064,"internally_reserved":0,"allocated":1629835264,"data_stored":1629387405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738662,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1645088,"kb_used_data":1090444,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92100064,"statfs":{"total":95995035648,"available":94310465536,"internally_reserved":0,"allocated":1116614656,"data_stored":1116170349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1130233856,"data_stored":1130233856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1636171776,"data_stored":1636171776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3037134848,"data_stored":3037134848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1537605632,"data_stored":1537605632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2981429248,"data_stored":2981429248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1494745088,"data_stored":1494745088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1934622720,"data_stored":1934622720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":835190784,"data_stored":835190784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1498808320,"data_stored":1498808320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1100349440,"data_stored":1100349440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1128988672,"data_stored":1128988672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:20:06.918 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:06.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:06.934 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:06.941 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:06.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:06.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:06.953 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 86 16 96383 96367 70.0285 67.625 0.00481922 0.0142752 2024-09-02T10:20:06.978 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:07.043 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:07.067 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:07.068 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:07.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:07.195 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:07.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:07.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:07.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:07.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:07.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:07.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:07.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:07.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:07.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:07.583 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:07.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:07.685 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:07.718 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:07.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:07.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:07.848 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:07.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:07.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:07.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:07.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 87 16 97487 97471 70.0166 69 0.00354301 0.0142766 2024-09-02T10:20:07.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:07.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:08.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:08.110 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:08.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:08.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:08.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:08.240 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:08.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:08.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:08.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:08.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:08.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:08.497 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:08.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:08.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:08.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:08.622 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:08.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:08.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:08 smithi151 ceph-mon[16639]: pgmap v1324: 47 pgs: 47 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-02T10:20:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:08 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:08.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:08.745 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:08.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:08 smithi187 ceph-mon[20072]: pgmap v1324: 47 pgs: 47 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-02T10:20:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:08 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:08 smithi162 ceph-mon[20279]: pgmap v1324: 47 pgs: 47 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-02T10:20:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:08 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:08.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:08.876 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:08.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:08.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 88 16 98418 98402 69.8822 58.1875 0.00409391 0.014304 2024-09-02T10:20:08.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:09.002 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:09.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:09.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:09.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:09.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:09.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:09.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:09.256 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:09.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:09.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:09.380 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:09.381 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:09.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:09.509 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:09.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:09.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:09.633 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:09.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:09.634 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:09 smithi151 ceph-mon[16639]: osdmap e571: 12 total, 12 up, 11 in 2024-09-02T10:20:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:09 smithi151 ceph-mon[16639]: osdmap e572: 12 total, 12 up, 11 in 2024-09-02T10:20:09.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:09 smithi187 ceph-mon[20072]: osdmap e571: 12 total, 12 up, 11 in 2024-09-02T10:20:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:09 smithi187 ceph-mon[20072]: osdmap e572: 12 total, 12 up, 11 in 2024-09-02T10:20:09.760 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:09.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:09 smithi162 ceph-mon[20279]: osdmap e571: 12 total, 12 up, 11 in 2024-09-02T10:20:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:09 smithi162 ceph-mon[20279]: osdmap e572: 12 total, 12 up, 11 in 2024-09-02T10:20:09.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:20:09.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:09.888 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:09.889 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:09.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 89 16 99308 99292 69.7219 55.625 0.018479 0.0143381 2024-09-02T10:20:09.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:10.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:10.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:10.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:10.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:10.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:10.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:10.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:10.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:10.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:10.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:10.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:10.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:10.401 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:10.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:10.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:10.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:10.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:10.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:10.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:10.448 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:10.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:10.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:10.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:10.475 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:10.476 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:10.502 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:10.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:10 smithi151 ceph-mon[16639]: pgmap v1327: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-02T10:20:10.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/620530154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:10.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:10.517 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:10.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:10.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:10.550 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:10.550 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:10.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:10.657 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:10.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:10.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:10 smithi187 ceph-mon[20072]: pgmap v1327: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-02T10:20:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/620530154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:10.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:10.780 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:10.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:10.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:10 smithi162 ceph-mon[20279]: pgmap v1327: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-09-02T10:20:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/620530154' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:10.879 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:10.880 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:10.882 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:10.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:10.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:10.905 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:10.906 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:10.918 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:20:10.918 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:20:10.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:20:10.918 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:20:10.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.918 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11536 4.c 4.7fa12efc (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:20:10.918 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.463377+0000", 2024-09-02T10:20:10.918 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 539.99536784500003, 2024-09-02T10:20:10.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.000768433, 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11536 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463377+0000", 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463377+0000", 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463378+0000", 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.655e-06 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463418+0000", 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9276000000000001e-05 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463428+0000", 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0074999999999999e-05 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.463434+0000", 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2940000000000002e-06 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.921 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465896+0000", 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024620409999999999 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465906+0000", 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8660000000000001e-06 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464105+0000", 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99819907500000005 2024-09-02T10:20:10.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464145+0000", 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0151000000000002e-05 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11545 4.c 4.151c8eac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.464747+0000", 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 539.99399742900005, 2024-09-02T10:20:10.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99968865100000004, 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11545 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464747+0000", 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464747+0000", 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464748+0000", 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.24e-06 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464802+0000", 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3669999999999999e-05 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464810+0000", 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8420000000000005e-06 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.464817+0000", 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4209999999999998e-06 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.926 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465935+0000", 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011172109999999999 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465942+0000", 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1509999999999998e-06 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464426+0000", 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99848471999999999 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464436+0000", 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3959999999999995e-06 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11547 4.c 4.fa8faa7c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.465762+0000", 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 539.99298306000003, 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99873187699999999, 2024-09-02T10:20:10.928 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11547 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465762+0000", 2024-09-02T10:20:10.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465762+0000", 2024-09-02T10:20:10.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.930 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.930 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465763+0000", 2024-09-02T10:20:10.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.331e-06 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465793+0000", 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0453000000000001e-05 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465805+0000", 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1175999999999999e-05 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.465812+0000", 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1960000000000002e-06 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466125+0000", 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031365100000000002 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.466134+0000", 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8340000000000005e-06 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464486+0000", 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99835216500000001 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464493+0000", 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0709999999999997e-06 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.933 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:11568 4.c 4.acfb412c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:10.727519+0000", 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 539.73122526600002, 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.73701982099999996, 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 11568 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727519+0000", 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727519+0000", 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727521+0000", 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7859999999999999e-06 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727593+0000", 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1581000000000005e-05 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727607+0000", 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3916e-05 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727614+0000", 2024-09-02T10:20:10.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8889999999999996e-06 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727669+0000", 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5364000000000001e-05 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:10.727676+0000", 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5969999999999997e-06 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464533+0000", 2024-09-02T10:20:10.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.73685725400000002 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:11.464539+0000", 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-06 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:41914 4.11 e353/344 4:884e7a1d:::benchmark_data_smithi187_76542_object41913:head v 353'2322, mlcod=335'1550)", 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.849702+0000", 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 502.609042977, 2024-09-02T10:20:10.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44488412300000002, 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849702+0000", 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849702+0000", 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849730+0000", 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8532999999999999e-05 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.849763+0000", 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2478999999999999e-05 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293343+0000", 2024-09-02T10:20:10.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44358014400000001 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293349+0000", 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2489999999999998e-06 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293401+0000", 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1523999999999997e-05 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.293425+0000", 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4606000000000001e-05 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294550+0000", 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011250959999999999 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294586+0000", 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5491999999999998e-05 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.942 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.11 353/344 [PushOp(4:8de9d133:::benchmark_data_smithi187_76542_object24111:head, version: 335'1553, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8de9d133:::benchmark_data_smithi187_76542_object24111:head@335'1553, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.853882+0000", 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 502.60486226500001, 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.486507095, 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853882+0000", 2024-09-02T10:20:10.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853882+0000", 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853884+0000", 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5629999999999999e-06 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.853923+0000", 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8707000000000001e-05 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339685+0000", 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.485762156 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339697+0000", 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2575000000000001e-05 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339773+0000", 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5802000000000004e-05 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.339782+0000", 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6600000000000001e-06 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.340389+0000", 2024-09-02T10:20:10.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060763199999999999 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.11 354/344 [PushOp(4:8d3f2f46:::benchmark_data_smithi187_76542_object24793:head, version: 335'1596, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8d3f2f46:::benchmark_data_smithi187_76542_object24793:head@335'1596, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:48.472557+0000", 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 501.98618782099999, 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31961658199999998, 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472557+0000", 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472557+0000", 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472558+0000", 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.029e-06 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.472626+0000", 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8273000000000002e-05 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791323+0000", 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31869734700000002 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791336+0000", 2024-09-02T10:20:10.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2418e-05 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791429+0000", 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3016999999999994e-05 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.791444+0000", 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5532e-05 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.950 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.792173+0000", 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00072896600000000005 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.11 354/344 [PushOp(4:8b8fcb96:::benchmark_data_smithi187_76542_object24835:head, version: 335'1598, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:8b8fcb96:::benchmark_data_smithi187_76542_object24835:head@335'1598, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:48.476340+0000", 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 501.98240498000001, 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31867774999999998, 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:20:10.951 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476340+0000", 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476340+0000", 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476346+0000", 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2840000000000003e-06 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.476539+0000", 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019279599999999999 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794378+0000", 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31783950900000002 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794396+0000", 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7655999999999998e-05 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794481+0000", 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4628000000000004e-05 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.794491+0000", 2024-09-02T10:20:10.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0114e-05 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.795017+0000", 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052676299999999997 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:75251 4.11 4:8ba31329:::benchmark_data_smithi187_76542_object75250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 472.79379548700001, 2024-09-02T10:20:10.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37234742100000001, 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75251 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:20:10.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664951+0000", 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9599999999999999e-06 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664999+0000", 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7738999999999998e-05 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665013+0000", 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4209999999999999e-05 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665021+0000", 2024-09-02T10:20:10.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.481e-06 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665075+0000", 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4642999999999998e-05 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665173+0000", 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8022000000000004e-05 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665258+0000", 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4666000000000006e-05 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.666435+0000", 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011775080000000001 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.666722+0000", 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028679699999999999 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037199+0000", 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37047664200000002 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037256+0000", 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7605999999999997e-05 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037297+0000", 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0147e-05 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:50217 5.1 5:84aaa397:::benchmark_data_smithi187_89405_object50216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:20:10.962 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 286.66698752100001, 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.384074792, 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50217 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.963 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791759+0000", 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.454e-06 2024-09-02T10:20:10.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791785+0000", 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6838e-05 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791800+0000", 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4997999999999999e-05 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791808+0000", 2024-09-02T10:20:10.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2370000000000003e-06 2024-09-02T10:20:10.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791862+0000", 2024-09-02T10:20:10.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4765000000000002e-05 2024-09-02T10:20:10.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791966+0000", 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010337500000000001 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.792051+0000", 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4779999999999998e-05 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.793473+0000", 2024-09-02T10:20:10.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014225819999999999 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.793928+0000", 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045518199999999999 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175728+0000", 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38180008999999998 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175787+0000", 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8498999999999997e-05 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175832+0000", 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4991999999999999e-05 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.16,RELEASE,e:497,from:1,reservation_nonce:0)", 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 191.822052378, 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58412980699999995, 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636693+0000", 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6e-07 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636695+0000", 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.841e-06 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220721+0000", 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58402521600000001 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220727+0000", 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4219999999999997e-06 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220798+0000", 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1131999999999996e-05 2024-09-02T10:20:10.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220810+0000", 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1863e-05 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220822+0000", 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2072999999999999e-05 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:205302 5.1 5:85fde85c:::benchmark_data_smithi187_89405_object101697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 113.21722415000001, 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31845405399999999, 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 205302 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241523+0000", 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3429999999999998e-06 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241533+0000", 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0139999999999999e-05 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241562+0000", 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8600000000000001e-05 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241581+0000", 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9656e-05 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241772+0000", 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019121200000000001 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241906+0000", 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013322299999999999 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241975+0000", 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9129999999999997e-05 2024-09-02T10:20:10.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.245690+0000", 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037148390000000002 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.246234+0000", 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00054468699999999997 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559894+0000", 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31365966099999998 2024-09-02T10:20:10.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559928+0000", 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3813000000000003e-05 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559975+0000", 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6749999999999998e-05 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:205324 5.1 5:85edc945:::benchmark_data_smithi187_89405_object101719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 112.856486382, 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38627865099999997, 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 205324 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.980 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602260+0000", 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4300000000000001e-06 2024-09-02T10:20:10.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602266+0000", 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6599999999999998e-06 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602283+0000", 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6646e-05 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602307+0000", 2024-09-02T10:20:10.982 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3853999999999999e-05 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602361+0000", 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4200000000000003e-05 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602447+0000", 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6116e-05 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.983 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602524+0000", 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6596999999999999e-05 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.606604+0000", 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040804589999999998 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.606691+0000", 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6608999999999997e-05 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.984 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988466+0000", 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38177503699999998 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988502+0000", 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6217000000000002e-05 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988537+0000", 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4826000000000002e-05 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.985 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:205348 5.1 5:8f32706a:::benchmark_data_smithi187_89405_object101743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.983873+0000", 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 112.474871507, 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.187727485, 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:10.986 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 205348 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983873+0000", 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983873+0000", 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983876+0000", 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3700000000000002e-06 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983884+0000", 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2400000000000007e-06 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983901+0000", 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7669999999999999e-05 2024-09-02T10:20:10.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983912+0000", 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0131e-05 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.983993+0000", 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1901000000000001e-05 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.984091+0000", 2024-09-02T10:20:10.989 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7608000000000005e-05 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.984172+0000", 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0489999999999997e-05 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988092+0000", 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039208810000000002 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988434+0000", 2024-09-02T10:20:10.990 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034132100000000001 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:18.170743+0000", 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18230938299999999 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:18.170789+0000", 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5469999999999997e-05 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.991 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:18.171601+0000", 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00081201999999999999 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:22146 6.2 6:45876345:::benchmark_data_smithi187_102448_object22145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e544)", 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:55.988493+0000", 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.470251513999997, 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18607343000000001, 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.992 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 22146 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.988493+0000", 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.988493+0000", 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.988495+0000", 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4100000000000001e-06 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.988541+0000", 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6693000000000002e-05 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.988553+0000", 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2139e-05 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.988561+0000", 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3350000000000001e-06 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.988640+0000", 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9511999999999994e-05 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.988743+0000", 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102447 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,11", 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.988827+0000", 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4478e-05 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.989654+0000", 2024-09-02T10:20:10.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00082675700000000001 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:55.990197+0000", 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00054321600000000001 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.174486+0000", 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18428876199999999 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.174520+0000", 2024-09-02T10:20:10.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4362000000000001e-05 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.174567+0000", 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6319000000000002e-05 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:22174 6.2 6:4463cf06:::benchmark_data_smithi187_102448_object22173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e544)", 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:56.006609+0000", 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.452136014000004, 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17970319500000001, 2024-09-02T10:20:10.998 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 22174 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.006609+0000", 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:10.999 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.006609+0000", 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.006610+0000", 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.009e-06 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.006656+0000", 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6458000000000001e-05 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.000 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.006674+0000", 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7880000000000002e-05 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.006681+0000", 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1740000000000003e-06 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.006745+0000", 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4115999999999994e-05 2024-09-02T10:20:11.001 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.006868+0000", 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012223900000000001 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,11", 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.006992+0000", 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012436700000000001 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.007966+0000", 2024-09-02T10:20:11.002 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00097366100000000001 2024-09-02T10:20:11.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:11.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.008628+0000", 2024-09-02T10:20:11.003 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00066266399999999998 2024-09-02T10:20:11.003 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.003 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.003 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:11.003 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.186188+0000", 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17755948899999999 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.186250+0000", 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2702999999999999e-05 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.186312+0000", 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1434999999999997e-05 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:11.004 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:22408 6.2 6:4d4f7482:::benchmark_data_smithi187_102448_object22407:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e544)", 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:56.225197+0000", 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 74.233547506999997, 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19448252099999999, 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:11.005 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 22408 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.225197+0000", 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.225197+0000", 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.006 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.225198+0000", 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4099999999999999e-07 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.225257+0000", 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9555000000000002e-05 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.225265+0000", 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5639999999999999e-06 2024-09-02T10:20:11.007 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.225269+0000", 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0709999999999996e-06 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.225302+0000", 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3049999999999997e-05 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.225375+0000", 2024-09-02T10:20:11.008 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3535000000000007e-05 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,11", 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.225458+0000", 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2050000000000002e-05 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.226198+0000", 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000740214 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.009 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.226694+0000", 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049635300000000005 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.419615+0000", 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19292129699999999 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.419648+0000", 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2271999999999999e-05 2024-09-02T10:20:11.010 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:56.419680+0000", 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2019e-05 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:43141 6.c 6:30fc4bbf:::benchmark_data_smithi187_102448_object43140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:14.433321+0000", 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 56.025423553000003, 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22430181399999999, 2024-09-02T10:20:11.011 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 43141 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433321+0000", 2024-09-02T10:20:11.012 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433321+0000", 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433322+0000", 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8599999999999996e-07 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.013 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433400+0000", 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8250000000000005e-05 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433406+0000", 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8390000000000003e-06 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:11.014 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433410+0000", 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9419999999999997e-06 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433461+0000", 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0822999999999998e-05 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433600+0000", 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013943699999999999 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.015 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433763+0000", 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016227400000000001 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.435509+0000", 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017465650000000001 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.435671+0000", 2024-09-02T10:20:11.016 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016191799999999999 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.657514+0000", 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22184321900000001 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.657582+0000", 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7825999999999997e-05 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.017 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.657623+0000", 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0735000000000002e-05 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:54374 6.6 6:66e3d897:::benchmark_data_smithi187_102448_object54373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:24.167366+0000", 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.291378823000002, 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18088959600000001, 2024-09-02T10:20:11.018 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 54374 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.167366+0000", 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.019 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.167366+0000", 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.167367+0000", 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6299999999999993e-07 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.167425+0000", 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8168999999999998e-05 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.020 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.167435+0000", 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6770000000000007e-06 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.167439+0000", 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6500000000000004e-06 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.167479+0000", 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9254000000000001e-05 2024-09-02T10:20:11.021 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.167569+0000", 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0315999999999994e-05 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.167678+0000", 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109097 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.169157+0000", 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001479029 2024-09-02T10:20:11.022 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.172442+0000", 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032850240000000001 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.347503+0000", 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17506091500000001 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.347550+0000", 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7098000000000002e-05 2024-09-02T10:20:11.023 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.348255+0000", 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00070540399999999997 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:65120 6.c 6:34a903cd:::benchmark_data_smithi187_102448_object65119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:37.696338+0000", 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 32.762406597999998, 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18130332199999999, 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:20:11.024 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 65120 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.696338+0000", 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.025 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.696338+0000", 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.696339+0000", 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2799999999999996e-07 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.696397+0000", 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8514999999999998e-05 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.026 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.696405+0000", 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6279999999999996e-06 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.696409+0000", 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8419999999999998e-06 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.696463+0000", 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4459000000000002e-05 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.027 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.696572+0000", 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109246 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.696724+0000", 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000151893 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.697833+0000", 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011086640000000001 2024-09-02T10:20:11.028 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.779080+0000", 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.081246674000000005 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.877566+0000", 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.098486039999999997 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.877614+0000", 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8850999999999997e-05 2024-09-02T10:20:11.029 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:20:11.030 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:20:11.030 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:20:11.030 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:37.877641+0000", 2024-09-02T10:20:11.030 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6982e-05 2024-09-02T10:20:11.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.030 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:11.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.030 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:20:11.030 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:20:11.030 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:20:11.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:11.032 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 90 10 100269 100259 69.6187 60.4375 0.0141621 0.0143586 2024-09-02T10:20:11.032 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:11.032 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:11.033 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:11.033 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:11.033 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:11.033 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:11.052 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:11.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:11.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:11.154 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:11.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:11.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:11.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:11.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:11.299 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:11.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:11.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:11.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:11.423 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:11.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:11.425 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:20:11.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:11.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:11.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:11.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:11.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:11.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:11.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:11.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4153707380' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4153707380' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:11.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:11.799 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:11.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:11.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:11.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:20:11.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4153707380' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:11.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:11.923 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:11.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:11.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:11.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total time run: 90.5985 2024-09-02T10:20:11.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total writes made: 100269 2024-09-02T10:20:11.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Write size: 65536 2024-09-02T10:20:11.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object size: 65536 2024-09-02T10:20:11.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Bandwidth (MB/sec): 69.1713 2024-09-02T10:20:11.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Bandwidth: 23.1315 2024-09-02T10:20:11.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max bandwidth (MB/sec): 122.062 2024-09-02T10:20:11.954 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min bandwidth (MB/sec): 0.5625 2024-09-02T10:20:11.955 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average IOPS: 1106 2024-09-02T10:20:11.955 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev IOPS: 370.104 2024-09-02T10:20:11.955 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max IOPS: 1953 2024-09-02T10:20:11.955 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min IOPS: 9 2024-09-02T10:20:11.955 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average Latency(s): 0.0143852 2024-09-02T10:20:11.955 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Latency(s): 0.0425433 2024-09-02T10:20:11.955 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max latency(s): 2.84044 2024-09-02T10:20:11.955 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min latency(s): 0.00104401 2024-09-02T10:20:11.960 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Cleaning up (deleting benchmark objects) 2024-09-02T10:20:12.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:12.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:12.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:12.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:12.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:12.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:12.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:12.187 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:12.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:12.313 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:12.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:12.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:12.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:12.450 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:12.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:12.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:12.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:12 smithi151 ceph-mon[16639]: pgmap v1328: 47 pgs: 47 active+clean; 6.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-02T10:20:12.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:12 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:12.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4153707380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:12.505 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:12 smithi151 ceph-mon[16639]: osdmap e573: 12 total, 12 up, 11 in 2024-09-02T10:20:12.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:12.579 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:12.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:12.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:12 smithi162 ceph-mon[20279]: pgmap v1328: 47 pgs: 47 active+clean; 6.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-02T10:20:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:12 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4153707380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:12 smithi162 ceph-mon[20279]: osdmap e573: 12 total, 12 up, 11 in 2024-09-02T10:20:12.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:12.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:12.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:12 smithi187 ceph-mon[20072]: pgmap v1328: 47 pgs: 47 active+clean; 6.0 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 63 MiB/s wr, 1.00k op/s 2024-09-02T10:20:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:12 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4153707380' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:12 smithi187 ceph-mon[20072]: osdmap e573: 12 total, 12 up, 11 in 2024-09-02T10:20:12.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:12.830 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:12.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:12.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:12.877 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:20:12.877 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:20:12.877 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:20:12.877 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:20:12.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:12.956 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:12.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:12.957 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:13.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:13.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:13.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:13.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:13.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:13.213 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:13.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:13.234 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:13.315 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:13.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:13.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:13.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:13.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:13.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:13.572 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:13.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:13.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:13.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:13.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:13.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:13.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:13.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:20:13.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:13.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:13.852 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:13.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:13.990 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:13.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:14.016 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:20:14.016 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:20:14.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:14.125 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:14.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:14.145 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:14.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:14.248 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:14.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:14.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:14.376 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:14.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:14.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:14.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:14.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:14.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:20:14.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:14.632 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:14.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:14 smithi151 ceph-mon[16639]: pgmap v1330: 47 pgs: 47 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 170 B/s rd, 59 MiB/s wr, 1.02k op/s 2024-09-02T10:20:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:14 smithi151 ceph-mon[16639]: osdmap e574: 12 total, 12 up, 11 in 2024-09-02T10:20:14.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:14 smithi187 ceph-mon[20072]: pgmap v1330: 47 pgs: 47 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 170 B/s rd, 59 MiB/s wr, 1.02k op/s 2024-09-02T10:20:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:14 smithi187 ceph-mon[20072]: osdmap e574: 12 total, 12 up, 11 in 2024-09-02T10:20:14.762 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:14.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:14.776 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1330,"stamp":"2024-09-02T10:20:13.354311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6512968248,"num_objects":99374,"num_object_clones":0,"num_object_copies":298122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99374,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100312,"num_write_kb":6389825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8541,"num_bytes_recovered":560276544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103164,"ondisk_log_size":103164,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":24205888,"kb_used_data":19337004,"kb_used_omap":17,"kb_used_meta":4868654,"kb_avail":1006990784,"statfs":{"total":1055945392128,"available":1031158562816,"internally_reserved":0,"allocated":19801092096,"data_stored":19796043492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":4985502582},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,0,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":338034712,"num_objects":5159,"num_object_clones":0,"num_object_copies":15477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6099,"num_write_kb":360193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001163"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":491,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.962918+0000","last_change":"2024-09-02T10:18:43.049145+0000","last_active":"2024-09-02T10:20:11.962918+0000","last_peered":"2024-09-02T10:20:11.962918+0000","last_clean":"2024-09-02T10:20:11.962918+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:20:11.962918+0000","last_undegraded":"2024-09-02T10:20:11.962918+0000","last_fullsized":"2024-09-02T10:20:11.962918+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:43.049077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:43.049077+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:16.819608+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":303,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-09-02T10:20:08.419981+0000","last_change":"2024-09-02T10:19:47.121714+0000","last_active":"2024-09-02T10:20:08.419981+0000","last_peered":"2024-09-02T10:20:08.419981+0000","last_clean":"2024-09-02T10:20:08.419981+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:20:08.419981+0000","last_undegraded":"2024-09-02T10:20:08.419981+0000","last_fullsized":"2024-09-02T10:20:08.419981+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:04.209863+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":754,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.070312+0000","last_change":"2024-09-02T10:19:48.146767+0000","last_active":"2024-09-02T10:20:09.070312+0000","last_peered":"2024-09-02T10:20:09.070312+0000","last_clean":"2024-09-02T10:20:09.070312+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:20:09.070312+0000","last_undegraded":"2024-09-02T10:20:09.070312+0000","last_fullsized":"2024-09-02T10:20:09.070312+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:48.146721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:48.146721+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:55.994378+0000","scrub_duration":995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":746,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.070252+0000","last_change":"2024-09-02T10:18:39.045140+0000","last_active":"2024-09-02T10:20:09.070252+0000","last_peered":"2024-09-02T10:20:09.070252+0000","last_clean":"2024-09-02T10:20:09.070252+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:20:09.070252+0000","last_undegraded":"2024-09-02T10:20:09.070252+0000","last_fullsized":"2024-09-02T10:20:09.070252+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:39.045083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:39.045083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:13.077725+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":58,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.075386+0000","last_change":"2024-09-02T10:19:05.058923+0000","last_active":"2024-09-02T10:20:09.075386+0000","last_peered":"2024-09-02T10:20:09.075386+0000","last_clean":"2024-09-02T10:20:09.075386+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:20:09.075386+0000","last_undegraded":"2024-09-02T10:20:09.075386+0000","last_fullsized":"2024-09-02T10:20:09.075386+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:01.810089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:01.810089+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:19.405935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":603,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.073109+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:20:09.073109+0000","last_peered":"2024-09-02T10:20:09.073109+0000","last_clean":"2024-09-02T10:20:09.073109+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:20:09.073109+0000","last_undegraded":"2024-09-02T10:20:09.073109+0000","last_fullsized":"2024-09-02T10:20:09.073109+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:08.487285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1121,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.427783+0000","last_change":"2024-09-02T10:19:03.880037+0000","last_active":"2024-09-02T10:20:11.427783+0000","last_peered":"2024-09-02T10:20:11.427783+0000","last_clean":"2024-09-02T10:20:11.427783+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:20:11.427783+0000","last_undegraded":"2024-09-02T10:20:11.427783+0000","last_fullsized":"2024-09-02T10:20:11.427783+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:03.879968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:03.879968+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.678816+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":769,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.070292+0000","last_change":"2024-09-02T10:20:00.175101+0000","last_active":"2024-09-02T10:20:09.070292+0000","last_peered":"2024-09-02T10:20:09.070292+0000","last_clean":"2024-09-02T10:20:09.070292+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:20:09.070292+0000","last_undegraded":"2024-09-02T10:20:09.070292+0000","last_fullsized":"2024-09-02T10:20:09.070292+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.175032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.175032+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:28.457490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":767,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-09-02T10:20:08.417936+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:20:08.417936+0000","last_peered":"2024-09-02T10:20:08.417936+0000","last_clean":"2024-09-02T10:20:08.417936+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:20:08.417936+0000","last_undegraded":"2024-09-02T10:20:08.417936+0000","last_fullsized":"2024-09-02T10:20:08.417936+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:52.877235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":737,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.074447+0000","last_change":"2024-09-02T10:19:05.910750+0000","last_active":"2024-09-02T10:20:09.074447+0000","last_peered":"2024-09-02T10:20:09.074447+0000","last_clean":"2024-09-02T10:20:09.074447+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:20:09.074447+0000","last_undegraded":"2024-09-02T10:20:09.074447+0000","last_fullsized":"2024-09-02T10:20:09.074447+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:05.910691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:05.910691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:20.353104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":589,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.074334+0000","last_change":"2024-09-02T10:19:52.963372+0000","last_active":"2024-09-02T10:20:09.074334+0000","last_peered":"2024-09-02T10:20:09.074334+0000","last_clean":"2024-09-02T10:20:09.074334+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:20:09.074334+0000","last_undegraded":"2024-09-02T10:20:09.074334+0000","last_fullsized":"2024-09-02T10:20:09.074334+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.963289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.963289+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:10.821299+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1092,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-09-02T10:20:08.417749+0000","last_change":"2024-09-02T10:19:00.052912+0000","last_active":"2024-09-02T10:20:08.417749+0000","last_peered":"2024-09-02T10:20:08.417749+0000","last_clean":"2024-09-02T10:20:08.417749+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:20:08.417749+0000","last_undegraded":"2024-09-02T10:20:08.417749+0000","last_fullsized":"2024-09-02T10:20:08.417749+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.052857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.052857+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:29.321752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1039,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.427866+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:20:11.427866+0000","last_peered":"2024-09-02T10:20:11.427866+0000","last_clean":"2024-09-02T10:20:11.427866+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:20:11.427866+0000","last_undegraded":"2024-09-02T10:20:11.427866+0000","last_fullsized":"2024-09-02T10:20:11.427866+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:57.041883+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1018,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.961832+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:20:11.961832+0000","last_peered":"2024-09-02T10:20:11.961832+0000","last_clean":"2024-09-02T10:20:11.961832+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:20:11.961832+0000","last_undegraded":"2024-09-02T10:20:11.961832+0000","last_fullsized":"2024-09-02T10:20:11.961832+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:01.846422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1009,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.427848+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:20:11.427848+0000","last_peered":"2024-09-02T10:20:11.427848+0000","last_clean":"2024-09-02T10:20:11.427848+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:20:11.427848+0000","last_undegraded":"2024-09-02T10:20:11.427848+0000","last_fullsized":"2024-09-02T10:20:11.427848+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.850900+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"572'6197","reported_seq":6346,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:10.902672+0000","last_change":"2024-09-02T10:19:59.986749+0000","last_active":"2024-09-02T10:20:10.902672+0000","last_peered":"2024-09-02T10:20:10.902672+0000","last_clean":"2024-09-02T10:20:10.902672+0000","last_became_active":"2024-09-02T10:18:37.599064+0000","last_became_peered":"2024-09-02T10:18:37.599064+0000","last_unstale":"2024-09-02T10:20:10.902672+0000","last_undegraded":"2024-09-02T10:20:10.902672+0000","last_fullsized":"2024-09-02T10:20:10.902672+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":6197,"log_dups_size":0,"ondisk_log_size":6197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:01.232192+0000","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406126592,"num_objects":6197,"num_object_clones":0,"num_object_copies":18591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6197,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1096,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.427898+0000","last_change":"2024-09-02T10:19:52.109223+0000","last_active":"2024-09-02T10:20:11.427898+0000","last_peered":"2024-09-02T10:20:11.427898+0000","last_clean":"2024-09-02T10:20:11.427898+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:20:11.427898+0000","last_undegraded":"2024-09-02T10:20:11.427898+0000","last_fullsized":"2024-09-02T10:20:11.427898+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:04.388339+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"572'6191","reported_seq":10574,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.206156+0000","last_change":"2024-09-02T10:19:55.677256+0000","last_active":"2024-09-02T10:20:09.206156+0000","last_peered":"2024-09-02T10:20:09.206156+0000","last_clean":"2024-09-02T10:20:09.206156+0000","last_became_active":"2024-09-02T10:19:32.596751+0000","last_became_peered":"2024-09-02T10:19:32.596751+0000","last_unstale":"2024-09-02T10:20:09.206156+0000","last_undegraded":"2024-09-02T10:20:09.206156+0000","last_fullsized":"2024-09-02T10:20:09.206156+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":6191,"log_dups_size":0,"ondisk_log_size":6191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:09.732905+0000","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405733376,"num_objects":6191,"num_object_clones":0,"num_object_copies":18573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6191,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":138346496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1209,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.074053+0000","last_change":"2024-09-02T10:19:49.817426+0000","last_active":"2024-09-02T10:20:09.074053+0000","last_peered":"2024-09-02T10:20:09.074053+0000","last_clean":"2024-09-02T10:20:09.074053+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:20:09.074053+0000","last_undegraded":"2024-09-02T10:20:09.074053+0000","last_fullsized":"2024-09-02T10:20:09.074053+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.817347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.817347+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:54.901580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"573'6193","reported_seq":6284,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:12.199797+0000","last_change":"2024-09-02T10:18:37.062429+0000","last_active":"2024-09-02T10:20:12.199797+0000","last_peered":"2024-09-02T10:20:12.199797+0000","last_clean":"2024-09-02T10:20:12.199797+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:20:12.199797+0000","last_undegraded":"2024-09-02T10:20:12.199797+0000","last_fullsized":"2024-09-02T10:20:12.199797+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6193,"log_dups_size":0,"ondisk_log_size":6193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:15.038875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403570712,"num_objects":6159,"num_object_clones":0,"num_object_copies":18477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6159,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6193,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":861,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.545600+0000","last_change":"2024-09-02T10:19:05.058003+0000","last_active":"2024-09-02T10:20:11.545600+0000","last_peered":"2024-09-02T10:20:11.545600+0000","last_clean":"2024-09-02T10:20:11.545600+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:20:11.545600+0000","last_undegraded":"2024-09-02T10:20:11.545600+0000","last_fullsized":"2024-09-02T10:20:11.545600+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.206480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.206480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"573'6287","reported_seq":6388,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:13.215108+0000","last_change":"2024-09-02T10:20:01.109999+0000","last_active":"2024-09-02T10:20:13.215108+0000","last_peered":"2024-09-02T10:20:13.215108+0000","last_clean":"2024-09-02T10:20:13.215108+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:20:13.215108+0000","last_undegraded":"2024-09-02T10:20:13.215108+0000","last_fullsized":"2024-09-02T10:20:13.215108+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":899,"log_size":6287,"log_dups_size":0,"ondisk_log_size":6287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:17.039049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399441920,"num_objects":6095,"num_object_clones":0,"num_object_copies":18285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":258,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.963057+0000","last_change":"2024-09-02T10:19:50.392195+0000","last_active":"2024-09-02T10:20:11.963057+0000","last_peered":"2024-09-02T10:20:11.963057+0000","last_clean":"2024-09-02T10:20:11.963057+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:20:11.963057+0000","last_undegraded":"2024-09-02T10:20:11.963057+0000","last_fullsized":"2024-09-02T10:20:11.963057+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.392146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.392146+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.406921+0000","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"572'6303","reported_seq":6391,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:10.923720+0000","last_change":"2024-09-02T10:18:37.850883+0000","last_active":"2024-09-02T10:20:10.923720+0000","last_peered":"2024-09-02T10:20:10.923720+0000","last_clean":"2024-09-02T10:20:10.923720+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:20:10.923720+0000","last_undegraded":"2024-09-02T10:20:10.923720+0000","last_fullsized":"2024-09-02T10:20:10.923720+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413073408,"num_objects":6303,"num_object_clones":0,"num_object_copies":18909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6303,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1166,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.962772+0000","last_change":"2024-09-02T10:19:51.832888+0000","last_active":"2024-09-02T10:20:11.962772+0000","last_peered":"2024-09-02T10:20:11.962772+0000","last_clean":"2024-09-02T10:20:11.962772+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:20:11.962772+0000","last_undegraded":"2024-09-02T10:20:11.962772+0000","last_fullsized":"2024-09-02T10:20:11.962772+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.832822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.832822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.770876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"572'6188","reported_seq":10618,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.166607+0000","last_change":"2024-09-02T10:19:54.548123+0000","last_active":"2024-09-02T10:20:09.166607+0000","last_peered":"2024-09-02T10:20:09.166607+0000","last_clean":"2024-09-02T10:20:09.166607+0000","last_became_active":"2024-09-02T10:19:31.588084+0000","last_became_peered":"2024-09-02T10:19:31.588084+0000","last_unstale":"2024-09-02T10:20:09.166607+0000","last_undegraded":"2024-09-02T10:20:09.166607+0000","last_fullsized":"2024-09-02T10:20:09.166607+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":6188,"log_dups_size":0,"ondisk_log_size":6188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:12.643132+0000","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405536768,"num_objects":6188,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6188,"num_write_kb":396032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2135,"num_bytes_recovered":139919360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":631,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.073201+0000","last_change":"2024-09-02T10:20:00.749882+0000","last_active":"2024-09-02T10:20:09.073201+0000","last_peered":"2024-09-02T10:20:09.073201+0000","last_clean":"2024-09-02T10:20:09.073201+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:20:09.073201+0000","last_undegraded":"2024-09-02T10:20:09.073201+0000","last_fullsized":"2024-09-02T10:20:09.073201+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.876682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"572'6093","reported_seq":6180,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.403048+0000","last_change":"2024-09-02T10:18:37.600687+0000","last_active":"2024-09-02T10:20:09.403048+0000","last_peered":"2024-09-02T10:20:09.403048+0000","last_clean":"2024-09-02T10:20:09.403048+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:20:09.403048+0000","last_undegraded":"2024-09-02T10:20:09.403048+0000","last_fullsized":"2024-09-02T10:20:09.403048+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6093,"log_dups_size":0,"ondisk_log_size":6093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:11.104419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399310848,"num_objects":6093,"num_object_clones":0,"num_object_copies":18279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6093,"num_write_kb":389952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":603,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.073273+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:20:09.073273+0000","last_peered":"2024-09-02T10:20:09.073273+0000","last_clean":"2024-09-02T10:20:09.073273+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:20:09.073273+0000","last_undegraded":"2024-09-02T10:20:09.073273+0000","last_fullsized":"2024-09-02T10:20:09.073273+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:51.265964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"573'6413","reported_seq":6560,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:12.877152+0000","last_change":"2024-09-02T10:19:52.017169+0000","last_active":"2024-09-02T10:20:12.877152+0000","last_peered":"2024-09-02T10:20:12.877152+0000","last_clean":"2024-09-02T10:20:12.877152+0000","last_became_active":"2024-09-02T10:18:37.064844+0000","last_became_peered":"2024-09-02T10:18:37.064844+0000","last_unstale":"2024-09-02T10:20:12.877152+0000","last_undegraded":"2024-09-02T10:20:12.877152+0000","last_fullsized":"2024-09-02T10:20:12.877152+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":6413,"log_dups_size":0,"ondisk_log_size":6413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:00.776952+0000","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410583040,"num_objects":6265,"num_object_clones":0,"num_object_copies":18795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":357,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.426131+0000","last_change":"2024-09-02T10:19:58.949821+0000","last_active":"2024-09-02T10:20:11.426131+0000","last_peered":"2024-09-02T10:20:11.426131+0000","last_clean":"2024-09-02T10:20:11.426131+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:20:11.426131+0000","last_undegraded":"2024-09-02T10:20:11.426131+0000","last_fullsized":"2024-09-02T10:20:11.426131+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:58.949745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.949745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.256455+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"573'6360","reported_seq":10871,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:13.214978+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:20:13.214978+0000","last_peered":"2024-09-02T10:20:13.214978+0000","last_clean":"2024-09-02T10:20:13.214978+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:20:13.214978+0000","last_undegraded":"2024-09-02T10:20:13.214978+0000","last_fullsized":"2024-09-02T10:20:13.214978+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":6360,"log_dups_size":0,"ondisk_log_size":6360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:22.534088+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405143552,"num_objects":6182,"num_object_clones":0,"num_object_copies":18546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6360,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":604,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.073216+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:20:09.073216+0000","last_peered":"2024-09-02T10:20:09.073216+0000","last_clean":"2024-09-02T10:20:09.073216+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:20:09.073216+0000","last_undegraded":"2024-09-02T10:20:09.073216+0000","last_fullsized":"2024-09-02T10:20:09.073216+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:02.710854+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"572'6316","reported_seq":6404,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:10.936501+0000","last_change":"2024-09-02T10:18:37.848300+0000","last_active":"2024-09-02T10:20:10.936501+0000","last_peered":"2024-09-02T10:20:10.936501+0000","last_clean":"2024-09-02T10:20:10.936501+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:20:10.936501+0000","last_undegraded":"2024-09-02T10:20:10.936501+0000","last_fullsized":"2024-09-02T10:20:10.936501+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6316,"log_dups_size":0,"ondisk_log_size":6316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:14.769647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413925376,"num_objects":6316,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6316,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":850,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.074136+0000","last_change":"2024-09-02T10:19:05.058925+0000","last_active":"2024-09-02T10:20:09.074136+0000","last_peered":"2024-09-02T10:20:09.074136+0000","last_clean":"2024-09-02T10:20:09.074136+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:20:09.074136+0000","last_undegraded":"2024-09-02T10:20:09.074136+0000","last_fullsized":"2024-09-02T10:20:09.074136+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.789947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.789947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:10.614688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"571'6145","reported_seq":6291,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-09-02T10:20:08.636761+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:20:08.636761+0000","last_peered":"2024-09-02T10:20:08.636761+0000","last_clean":"2024-09-02T10:20:08.636761+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:20:08.636761+0000","last_undegraded":"2024-09-02T10:20:08.636761+0000","last_fullsized":"2024-09-02T10:20:08.636761+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":6145,"log_dups_size":0,"ondisk_log_size":6145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.641560+0000","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402718720,"num_objects":6145,"num_object_clones":0,"num_object_copies":18435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6145,"num_write_kb":393280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1123,"reported_epoch":571,"state":"active+clean","last_fresh":"2024-09-02T10:20:08.417775+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:20:08.417775+0000","last_peered":"2024-09-02T10:20:08.417775+0000","last_clean":"2024-09-02T10:20:08.417775+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:20:08.417775+0000","last_undegraded":"2024-09-02T10:20:08.417775+0000","last_fullsized":"2024-09-02T10:20:08.417775+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:59.441621+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"573'6334","reported_seq":10732,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:12.212767+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:20:12.212767+0000","last_peered":"2024-09-02T10:20:12.212767+0000","last_clean":"2024-09-02T10:20:12.212767+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:20:12.212767+0000","last_undegraded":"2024-09-02T10:20:12.212767+0000","last_fullsized":"2024-09-02T10:20:12.212767+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":6334,"log_dups_size":0,"ondisk_log_size":6334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:23.125229+0000","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412745728,"num_objects":6298,"num_object_clones":0,"num_object_copies":18894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6334,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":869,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.074714+0000","last_change":"2024-09-02T10:19:59.731786+0000","last_active":"2024-09-02T10:20:09.074714+0000","last_peered":"2024-09-02T10:20:09.074714+0000","last_clean":"2024-09-02T10:20:09.074714+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:20:09.074714+0000","last_undegraded":"2024-09-02T10:20:09.074714+0000","last_fullsized":"2024-09-02T10:20:09.074714+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:59.731706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.731706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:16.407855+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"572'6325","reported_seq":6413,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:10.917437+0000","last_change":"2024-09-02T10:18:37.850899+0000","last_active":"2024-09-02T10:20:10.917437+0000","last_peered":"2024-09-02T10:20:10.917437+0000","last_clean":"2024-09-02T10:20:10.917437+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:20:10.917437+0000","last_undegraded":"2024-09-02T10:20:10.917437+0000","last_fullsized":"2024-09-02T10:20:10.917437+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6325,"log_dups_size":0,"ondisk_log_size":6325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414515200,"num_objects":6325,"num_object_clones":0,"num_object_copies":18975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6325,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1118,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.426203+0000","last_change":"2024-09-02T10:19:08.999797+0000","last_active":"2024-09-02T10:20:11.426203+0000","last_peered":"2024-09-02T10:20:11.426203+0000","last_clean":"2024-09-02T10:20:11.426203+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:20:11.426203+0000","last_undegraded":"2024-09-02T10:20:11.426203+0000","last_fullsized":"2024-09-02T10:20:11.426203+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:08.999706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:08.999706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:13.510899+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"573'6308","reported_seq":6462,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:12.907619+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:20:12.907619+0000","last_peered":"2024-09-02T10:20:12.907619+0000","last_clean":"2024-09-02T10:20:12.907619+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:20:12.907619+0000","last_undegraded":"2024-09-02T10:20:12.907619+0000","last_fullsized":"2024-09-02T10:20:12.907619+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.389208+0000","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405536768,"num_objects":6188,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6308,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":66,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.078326+0000","last_change":"2024-09-02T10:19:06.947029+0000","last_active":"2024-09-02T10:20:09.078326+0000","last_peered":"2024-09-02T10:20:09.078326+0000","last_clean":"2024-09-02T10:20:09.078326+0000","last_became_active":"2024-09-02T10:19:06.049121+0000","last_became_peered":"2024-09-02T10:19:06.049121+0000","last_unstale":"2024-09-02T10:20:09.078326+0000","last_undegraded":"2024-09-02T10:20:09.078326+0000","last_fullsized":"2024-09-02T10:20:09.078326+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:06.946957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:06.946957+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:24.543000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"573'6282","reported_seq":6375,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:12.188082+0000","last_change":"2024-09-02T10:20:01.132876+0000","last_active":"2024-09-02T10:20:12.188082+0000","last_peered":"2024-09-02T10:20:12.188082+0000","last_clean":"2024-09-02T10:20:12.188082+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:20:12.188082+0000","last_undegraded":"2024-09-02T10:20:12.188082+0000","last_fullsized":"2024-09-02T10:20:12.188082+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6282,"log_dups_size":0,"ondisk_log_size":6282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408813568,"num_objects":6238,"num_object_clones":0,"num_object_copies":18714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6282,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":896,"reported_epoch":572,"state":"active+clean","last_fresh":"2024-09-02T10:20:09.074538+0000","last_change":"2024-09-02T10:19:57.798578+0000","last_active":"2024-09-02T10:20:09.074538+0000","last_peered":"2024-09-02T10:20:09.074538+0000","last_clean":"2024-09-02T10:20:09.074538+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:20:09.074538+0000","last_undegraded":"2024-09-02T10:20:09.074538+0000","last_fullsized":"2024-09-02T10:20:09.074538+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:57.798501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:57.798501+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:05.189778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"573'6377","reported_seq":6467,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:13.206540+0000","last_change":"2024-09-02T10:18:37.066015+0000","last_active":"2024-09-02T10:20:13.206540+0000","last_peered":"2024-09-02T10:20:13.206540+0000","last_clean":"2024-09-02T10:20:13.206540+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:20:13.206540+0000","last_undegraded":"2024-09-02T10:20:13.206540+0000","last_fullsized":"2024-09-02T10:20:13.206540+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6377,"log_dups_size":0,"ondisk_log_size":6377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:18.014019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405602304,"num_objects":6189,"num_object_clones":0,"num_object_copies":18567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":290,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.966810+0000","last_change":"2024-09-02T10:19:02.956010+0000","last_active":"2024-09-02T10:20:11.966810+0000","last_peered":"2024-09-02T10:20:11.966810+0000","last_clean":"2024-09-02T10:20:11.966810+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:20:11.966810+0000","last_undegraded":"2024-09-02T10:20:11.966810+0000","last_fullsized":"2024-09-02T10:20:11.966810+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:02.955947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:02.955947+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:26.242539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":6512377880,"num_objects":99372,"num_object_clones":0,"num_object_copies":298116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":100312,"num_write_kb":6389825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8538,"num_bytes_recovered":559546368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19770642432,"data_stored":19770638360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100312,"ondisk_log_size":100312,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383921,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1683464,"kb_used_data":1210800,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92061688,"statfs":{"total":95995035648,"available":94271168512,"internally_reserved":0,"allocated":1239859200,"data_stored":1239401584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":483981764},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1516608,"kb_used_data":1188336,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92228544,"statfs":{"total":95995035648,"available":94442029056,"internally_reserved":0,"allocated":1216856064,"data_stored":1216398448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":336132544},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841787,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2092304,"kb_used_data":1583548,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":91652848,"statfs":{"total":95995035648,"available":93852516352,"internally_reserved":0,"allocated":1621553152,"data_stored":1621091083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":520944064},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070720,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1160696,"kb_used_data":844900,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92584456,"statfs":{"total":95995035648,"available":94806482944,"internally_reserved":0,"allocated":865177600,"data_stored":864724437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299653,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2557768,"kb_used_data":2075384,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":91187384,"statfs":{"total":95995035648,"available":93375881216,"internally_reserved":0,"allocated":2125193216,"data_stored":2124732048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":493943242},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528585,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1952340,"kb_used_data":1606012,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":91792812,"statfs":{"total":95995035648,"available":93995839488,"internally_reserved":0,"allocated":1644556288,"data_stored":1644094219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":354613703},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724814,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3756444,"kb_used_data":3197704,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":89988708,"statfs":{"total":95995035648,"available":92148436992,"internally_reserved":0,"allocated":3274448896,"data_stored":3273979203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953747,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1881832,"kb_used_data":1571608,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91863320,"statfs":{"total":95995035648,"available":94068039680,"internally_reserved":0,"allocated":1609326592,"data_stored":1608877832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149977,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3701008,"kb_used_data":3182524,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":90044144,"statfs":{"total":95995035648,"available":92205203456,"internally_reserved":0,"allocated":3258904576,"data_stored":3258442507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":530905544},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411612,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575137,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2137328,"kb_used_data":1668832,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91607824,"statfs":{"total":95995035648,"available":93806411776,"internally_reserved":0,"allocated":1708883968,"data_stored":1708431656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738664,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1766096,"kb_used_data":1207356,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91979056,"statfs":{"total":95995035648,"available":94186553344,"internally_reserved":0,"allocated":1236332544,"data_stored":1235870475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1221066752,"data_stored":1221066752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1714028544,"data_stored":1714028544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3248291840,"data_stored":3248291840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1622081536,"data_stored":1622081536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3269070848,"data_stored":3269066776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1636237312,"data_stored":1636237312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2125987840,"data_stored":2125987840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":866844672,"data_stored":866844672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1616052224,"data_stored":1616052224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1216479232,"data_stored":1216479232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1234501632,"data_stored":1234501632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:20:14.779 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-02T10:20:14.779 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:20:14.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:20:14.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:14 smithi162 ceph-mon[20279]: pgmap v1330: 47 pgs: 47 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 170 B/s rd, 59 MiB/s wr, 1.02k op/s 2024-09-02T10:20:14.799 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:14 smithi162 ceph-mon[20279]: osdmap e574: 12 total, 12 up, 11 in 2024-09-02T10:20:14.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:14.866 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:14.891 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:14.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:14.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:14.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:15.029 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:15.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:15.096 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:20:15.096 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:20:15.096 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:20:15.096 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:20:15.096 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:20:15.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:15.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:15.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:15.155 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:15.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:20:15.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:15.277 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:15.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:15.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:15.409 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:15.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:15.470 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:15.470 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:15.495 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:15.508 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:15.508 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:15.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:15.545 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:15.545 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:15.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:15 smithi151 ceph-mon[16639]: from='client.17259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:15.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:15.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:15.582 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:15.582 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:15.607 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:15.607 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:15.632 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:15.632 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:15.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:15.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:15.659 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:15.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:15.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:15.688 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:15.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:15.711 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:15.711 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:15.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:15.732 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:15 smithi187 ceph-mon[20072]: from='client.17259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:15.758 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:15.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:15.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:15.788 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:15.788 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:15.807 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:15.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:15.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:15 smithi162 ceph-mon[20279]: from='client.17259 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:15.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:15.932 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:15.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:16.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:16.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:16.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:16.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:16.188 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:16.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:16.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:16.318 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:16.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:16.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:16.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:16.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:16.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:16.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:16.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:16.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:16.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:16.651 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:16 smithi151 ceph-mon[16639]: pgmap v1332: 47 pgs: 47 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 162 B/s rd, 40 MiB/s wr, 809 op/s 2024-09-02T10:20:16.651 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2259464262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:16.675 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:16.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:16 smithi162 ceph-mon[20279]: pgmap v1332: 47 pgs: 47 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 162 B/s rd, 40 MiB/s wr, 809 op/s 2024-09-02T10:20:16.697 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2259464262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:16.698 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:16.699 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:16 smithi187 ceph-mon[20072]: pgmap v1332: 47 pgs: 47 active+clean; 6.1 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 162 B/s rd, 40 MiB/s wr, 809 op/s 2024-09-02T10:20:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2259464262' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:16.799 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:16.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:16.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:16.823 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:16.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:16.950 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:16.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:16.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:17.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:17.085 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:17.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:17.134 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:17.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:20:17.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:17.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:17.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:17.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:17.342 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:17.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:17.380 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:20:17.381 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:20:17.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:17.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:17.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:17.483 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:17.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:17.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:17.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:17 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:20:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:17 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:17 smithi151 ceph-mon[16639]: osdmap e575: 12 total, 12 up, 11 in 2024-09-02T10:20:17.695 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:17.718 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:17.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:17 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:20:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:17 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:17 smithi187 ceph-mon[20072]: osdmap e575: 12 total, 12 up, 11 in 2024-09-02T10:20:17.772 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:17 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:20:17.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:17 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:17.773 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:17 smithi162 ceph-mon[20279]: osdmap e575: 12 total, 12 up, 11 in 2024-09-02T10:20:17.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:17.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:17.843 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:17.844 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:17.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:17.976 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:17.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:18.066 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:18.067 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:18.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:18.111 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:18.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:18.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:18.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:18.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:18.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:18.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:18.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:18.436 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1334,"stamp":"2024-09-02T10:20:17.355213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6361973304,"num_objects":97070,"num_object_clones":0,"num_object_copies":291210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97070,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103472,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8541,"num_bytes_recovered":560276544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106324,"ondisk_log_size":106324,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":23773548,"kb_used_data":18915672,"kb_used_omap":17,"kb_used_meta":4857582,"kb_avail":1007423124,"statfs":{"total":1055945392128,"available":1031601278976,"internally_reserved":0,"allocated":19369648128,"data_stored":19364507023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":4974164854},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,6,2,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-66912232,"num_objects":-1020,"num_object_clones":0,"num_object_copies":-3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1020,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5384,"num_write_kb":139585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001251"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":495,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.433705+0000","last_change":"2024-09-02T10:18:43.049145+0000","last_active":"2024-09-02T10:20:16.433705+0000","last_peered":"2024-09-02T10:20:16.433705+0000","last_clean":"2024-09-02T10:20:16.433705+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:20:16.433705+0000","last_undegraded":"2024-09-02T10:20:16.433705+0000","last_fullsized":"2024-09-02T10:20:16.433705+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:43.049077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:43.049077+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:16.819608+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":307,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.957830+0000","last_change":"2024-09-02T10:19:47.121714+0000","last_active":"2024-09-02T10:20:11.957830+0000","last_peered":"2024-09-02T10:20:11.957830+0000","last_clean":"2024-09-02T10:20:11.957830+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:20:11.957830+0000","last_undegraded":"2024-09-02T10:20:11.957830+0000","last_fullsized":"2024-09-02T10:20:11.957830+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:04.209863+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":758,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.075955+0000","last_change":"2024-09-02T10:19:48.146767+0000","last_active":"2024-09-02T10:20:14.075955+0000","last_peered":"2024-09-02T10:20:14.075955+0000","last_clean":"2024-09-02T10:20:14.075955+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:20:14.075955+0000","last_undegraded":"2024-09-02T10:20:14.075955+0000","last_fullsized":"2024-09-02T10:20:14.075955+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:48.146721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:48.146721+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:55.994378+0000","scrub_duration":995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":750,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.075928+0000","last_change":"2024-09-02T10:18:39.045140+0000","last_active":"2024-09-02T10:20:14.075928+0000","last_peered":"2024-09-02T10:20:14.075928+0000","last_clean":"2024-09-02T10:20:14.075928+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:20:14.075928+0000","last_undegraded":"2024-09-02T10:20:14.075928+0000","last_fullsized":"2024-09-02T10:20:14.075928+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:39.045083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:39.045083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":62,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.077056+0000","last_change":"2024-09-02T10:19:05.058923+0000","last_active":"2024-09-02T10:20:14.077056+0000","last_peered":"2024-09-02T10:20:14.077056+0000","last_clean":"2024-09-02T10:20:14.077056+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:20:14.077056+0000","last_undegraded":"2024-09-02T10:20:14.077056+0000","last_fullsized":"2024-09-02T10:20:14.077056+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:01.810089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:01.810089+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:19.405935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":609,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.439200+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:20:16.439200+0000","last_peered":"2024-09-02T10:20:16.439200+0000","last_clean":"2024-09-02T10:20:16.439200+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:20:16.439200+0000","last_undegraded":"2024-09-02T10:20:16.439200+0000","last_fullsized":"2024-09-02T10:20:16.439200+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:08.487285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1125,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.436241+0000","last_change":"2024-09-02T10:19:03.880037+0000","last_active":"2024-09-02T10:20:16.436241+0000","last_peered":"2024-09-02T10:20:16.436241+0000","last_clean":"2024-09-02T10:20:16.436241+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:20:16.436241+0000","last_undegraded":"2024-09-02T10:20:16.436241+0000","last_fullsized":"2024-09-02T10:20:16.436241+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:03.879968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:03.879968+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":773,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.076003+0000","last_change":"2024-09-02T10:20:00.175101+0000","last_active":"2024-09-02T10:20:14.076003+0000","last_peered":"2024-09-02T10:20:14.076003+0000","last_clean":"2024-09-02T10:20:14.076003+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:20:14.076003+0000","last_undegraded":"2024-09-02T10:20:14.076003+0000","last_fullsized":"2024-09-02T10:20:14.076003+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.175032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.175032+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:28.457490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":771,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.956155+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:20:11.956155+0000","last_peered":"2024-09-02T10:20:11.956155+0000","last_clean":"2024-09-02T10:20:11.956155+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:20:11.956155+0000","last_undegraded":"2024-09-02T10:20:11.956155+0000","last_fullsized":"2024-09-02T10:20:11.956155+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:52.877235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":741,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.077911+0000","last_change":"2024-09-02T10:19:05.910750+0000","last_active":"2024-09-02T10:20:14.077911+0000","last_peered":"2024-09-02T10:20:14.077911+0000","last_clean":"2024-09-02T10:20:14.077911+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:20:14.077911+0000","last_undegraded":"2024-09-02T10:20:14.077911+0000","last_fullsized":"2024-09-02T10:20:14.077911+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:05.910691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:05.910691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:20.353104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":593,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.077083+0000","last_change":"2024-09-02T10:19:52.963372+0000","last_active":"2024-09-02T10:20:14.077083+0000","last_peered":"2024-09-02T10:20:14.077083+0000","last_clean":"2024-09-02T10:20:14.077083+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:20:14.077083+0000","last_undegraded":"2024-09-02T10:20:14.077083+0000","last_fullsized":"2024-09-02T10:20:14.077083+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.963289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.963289+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:10.821299+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1096,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.956038+0000","last_change":"2024-09-02T10:19:00.052912+0000","last_active":"2024-09-02T10:20:11.956038+0000","last_peered":"2024-09-02T10:20:11.956038+0000","last_clean":"2024-09-02T10:20:11.956038+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:20:11.956038+0000","last_undegraded":"2024-09-02T10:20:11.956038+0000","last_fullsized":"2024-09-02T10:20:11.956038+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.052857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.052857+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:29.321752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1043,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.438166+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:20:16.438166+0000","last_peered":"2024-09-02T10:20:16.438166+0000","last_clean":"2024-09-02T10:20:16.438166+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:20:16.438166+0000","last_undegraded":"2024-09-02T10:20:16.438166+0000","last_fullsized":"2024-09-02T10:20:16.438166+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:57.041883+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1018,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.961832+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:20:11.961832+0000","last_peered":"2024-09-02T10:20:11.961832+0000","last_clean":"2024-09-02T10:20:11.961832+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:20:11.961832+0000","last_undegraded":"2024-09-02T10:20:11.961832+0000","last_fullsized":"2024-09-02T10:20:11.961832+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:01.846422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1013,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.436303+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:20:16.436303+0000","last_peered":"2024-09-02T10:20:16.436303+0000","last_clean":"2024-09-02T10:20:16.436303+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:20:16.436303+0000","last_undegraded":"2024-09-02T10:20:16.436303+0000","last_fullsized":"2024-09-02T10:20:16.436303+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.850900+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"575'6519","reported_seq":6674,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.743183+0000","last_change":"2024-09-02T10:19:59.986749+0000","last_active":"2024-09-02T10:20:16.743183+0000","last_peered":"2024-09-02T10:20:16.743183+0000","last_clean":"2024-09-02T10:20:16.743183+0000","last_became_active":"2024-09-02T10:18:37.599064+0000","last_became_peered":"2024-09-02T10:18:37.599064+0000","last_unstale":"2024-09-02T10:20:16.743183+0000","last_undegraded":"2024-09-02T10:20:16.743183+0000","last_fullsized":"2024-09-02T10:20:16.743183+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":6519,"log_dups_size":0,"ondisk_log_size":6519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:01.232192+0000","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385024000,"num_objects":5875,"num_object_clones":0,"num_object_copies":17625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5875,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6519,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1100,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.436341+0000","last_change":"2024-09-02T10:19:52.109223+0000","last_active":"2024-09-02T10:20:16.436341+0000","last_peered":"2024-09-02T10:20:16.436341+0000","last_clean":"2024-09-02T10:20:16.436341+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:20:16.436341+0000","last_undegraded":"2024-09-02T10:20:16.436341+0000","last_fullsized":"2024-09-02T10:20:16.436341+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:04.388339+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"574'6440","reported_seq":10827,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.176873+0000","last_change":"2024-09-02T10:19:55.677256+0000","last_active":"2024-09-02T10:20:14.176873+0000","last_peered":"2024-09-02T10:20:14.176873+0000","last_clean":"2024-09-02T10:20:14.176873+0000","last_became_active":"2024-09-02T10:19:32.596751+0000","last_became_peered":"2024-09-02T10:19:32.596751+0000","last_unstale":"2024-09-02T10:20:14.176873+0000","last_undegraded":"2024-09-02T10:20:14.176873+0000","last_fullsized":"2024-09-02T10:20:14.176873+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":6440,"log_dups_size":0,"ondisk_log_size":6440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:09.732905+0000","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402391040,"num_objects":6140,"num_object_clones":0,"num_object_copies":18420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6440,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":138346496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1213,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.076810+0000","last_change":"2024-09-02T10:19:49.817426+0000","last_active":"2024-09-02T10:20:14.076810+0000","last_peered":"2024-09-02T10:20:14.076810+0000","last_clean":"2024-09-02T10:20:14.076810+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:20:14.076810+0000","last_undegraded":"2024-09-02T10:20:14.076810+0000","last_fullsized":"2024-09-02T10:20:14.076810+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.817347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.817347+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:54.901580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"575'6579","reported_seq":6674,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:17.212407+0000","last_change":"2024-09-02T10:18:37.062429+0000","last_active":"2024-09-02T10:20:17.212407+0000","last_peered":"2024-09-02T10:20:17.212407+0000","last_clean":"2024-09-02T10:20:17.212407+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:20:17.212407+0000","last_undegraded":"2024-09-02T10:20:17.212407+0000","last_fullsized":"2024-09-02T10:20:17.212407+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6579,"log_dups_size":0,"ondisk_log_size":6579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378273816,"num_objects":5773,"num_object_clones":0,"num_object_copies":17319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6579,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":861,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.545600+0000","last_change":"2024-09-02T10:19:05.058003+0000","last_active":"2024-09-02T10:20:11.545600+0000","last_peered":"2024-09-02T10:20:11.545600+0000","last_clean":"2024-09-02T10:20:11.545600+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:20:11.545600+0000","last_undegraded":"2024-09-02T10:20:11.545600+0000","last_fullsized":"2024-09-02T10:20:11.545600+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.206480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.206480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"573'6287","reported_seq":6388,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:13.215108+0000","last_change":"2024-09-02T10:20:01.109999+0000","last_active":"2024-09-02T10:20:13.215108+0000","last_peered":"2024-09-02T10:20:13.215108+0000","last_clean":"2024-09-02T10:20:13.215108+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:20:13.215108+0000","last_undegraded":"2024-09-02T10:20:13.215108+0000","last_fullsized":"2024-09-02T10:20:13.215108+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":899,"log_size":6287,"log_dups_size":0,"ondisk_log_size":6287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:17.039049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399441920,"num_objects":6095,"num_object_clones":0,"num_object_copies":18285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":262,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.433663+0000","last_change":"2024-09-02T10:19:50.392195+0000","last_active":"2024-09-02T10:20:16.433663+0000","last_peered":"2024-09-02T10:20:16.433663+0000","last_clean":"2024-09-02T10:20:16.433663+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:20:16.433663+0000","last_undegraded":"2024-09-02T10:20:16.433663+0000","last_fullsized":"2024-09-02T10:20:16.433663+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.392146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.392146+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.406921+0000","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"574'6590","reported_seq":6682,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.209397+0000","last_change":"2024-09-02T10:18:37.850883+0000","last_active":"2024-09-02T10:20:16.209397+0000","last_peered":"2024-09-02T10:20:16.209397+0000","last_clean":"2024-09-02T10:20:16.209397+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:20:16.209397+0000","last_undegraded":"2024-09-02T10:20:16.209397+0000","last_fullsized":"2024-09-02T10:20:16.209397+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6590,"log_dups_size":0,"ondisk_log_size":6590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:18.765786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394264576,"num_objects":6016,"num_object_clones":0,"num_object_copies":18048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6590,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1170,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.433605+0000","last_change":"2024-09-02T10:19:51.832888+0000","last_active":"2024-09-02T10:20:16.433605+0000","last_peered":"2024-09-02T10:20:16.433605+0000","last_clean":"2024-09-02T10:20:16.433605+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:20:16.433605+0000","last_undegraded":"2024-09-02T10:20:16.433605+0000","last_fullsized":"2024-09-02T10:20:16.433605+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.832822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.832822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.770876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"574'6407","reported_seq":10841,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.176325+0000","last_change":"2024-09-02T10:19:54.548123+0000","last_active":"2024-09-02T10:20:14.176325+0000","last_peered":"2024-09-02T10:20:14.176325+0000","last_clean":"2024-09-02T10:20:14.176325+0000","last_became_active":"2024-09-02T10:19:31.588084+0000","last_became_peered":"2024-09-02T10:19:31.588084+0000","last_unstale":"2024-09-02T10:20:14.176325+0000","last_undegraded":"2024-09-02T10:20:14.176325+0000","last_fullsized":"2024-09-02T10:20:14.176325+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":6407,"log_dups_size":0,"ondisk_log_size":6407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:12.643132+0000","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402194432,"num_objects":6137,"num_object_clones":0,"num_object_copies":18411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6407,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2135,"num_bytes_recovered":139919360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":637,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.441122+0000","last_change":"2024-09-02T10:20:00.749882+0000","last_active":"2024-09-02T10:20:16.441122+0000","last_peered":"2024-09-02T10:20:16.441122+0000","last_clean":"2024-09-02T10:20:16.441122+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:20:16.441122+0000","last_undegraded":"2024-09-02T10:20:16.441122+0000","last_fullsized":"2024-09-02T10:20:16.441122+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.876682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"574'6344","reported_seq":6434,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.415871+0000","last_change":"2024-09-02T10:18:37.600687+0000","last_active":"2024-09-02T10:20:14.415871+0000","last_peered":"2024-09-02T10:20:14.415871+0000","last_clean":"2024-09-02T10:20:14.415871+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:20:14.415871+0000","last_undegraded":"2024-09-02T10:20:14.415871+0000","last_fullsized":"2024-09-02T10:20:14.415871+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6344,"log_dups_size":0,"ondisk_log_size":6344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395575296,"num_objects":6036,"num_object_clones":0,"num_object_copies":18108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6344,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":609,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.439289+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:20:16.439289+0000","last_peered":"2024-09-02T10:20:16.439289+0000","last_clean":"2024-09-02T10:20:16.439289+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:20:16.439289+0000","last_undegraded":"2024-09-02T10:20:16.439289+0000","last_fullsized":"2024-09-02T10:20:16.439289+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:51.265964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"573'6413","reported_seq":6560,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:12.877152+0000","last_change":"2024-09-02T10:19:52.017169+0000","last_active":"2024-09-02T10:20:12.877152+0000","last_peered":"2024-09-02T10:20:12.877152+0000","last_clean":"2024-09-02T10:20:12.877152+0000","last_became_active":"2024-09-02T10:18:37.064844+0000","last_became_peered":"2024-09-02T10:18:37.064844+0000","last_unstale":"2024-09-02T10:20:12.877152+0000","last_undegraded":"2024-09-02T10:20:12.877152+0000","last_fullsized":"2024-09-02T10:20:12.877152+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":6413,"log_dups_size":0,"ondisk_log_size":6413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:00.776952+0000","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410583040,"num_objects":6265,"num_object_clones":0,"num_object_copies":18795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6413,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":361,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.434824+0000","last_change":"2024-09-02T10:19:58.949821+0000","last_active":"2024-09-02T10:20:16.434824+0000","last_peered":"2024-09-02T10:20:16.434824+0000","last_clean":"2024-09-02T10:20:16.434824+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:20:16.434824+0000","last_undegraded":"2024-09-02T10:20:16.434824+0000","last_fullsized":"2024-09-02T10:20:16.434824+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:58.949745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.949745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.256455+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"573'6360","reported_seq":10871,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:13.214978+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:20:13.214978+0000","last_peered":"2024-09-02T10:20:13.214978+0000","last_clean":"2024-09-02T10:20:13.214978+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:20:13.214978+0000","last_undegraded":"2024-09-02T10:20:13.214978+0000","last_fullsized":"2024-09-02T10:20:13.214978+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":6360,"log_dups_size":0,"ondisk_log_size":6360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:22.534088+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405143552,"num_objects":6182,"num_object_clones":0,"num_object_copies":18546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6360,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":610,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.439348+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:20:16.439348+0000","last_peered":"2024-09-02T10:20:16.439348+0000","last_clean":"2024-09-02T10:20:16.439348+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:20:16.439348+0000","last_undegraded":"2024-09-02T10:20:16.439348+0000","last_fullsized":"2024-09-02T10:20:16.439348+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:02.710854+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"574'6584","reported_seq":6676,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.218072+0000","last_change":"2024-09-02T10:18:37.848300+0000","last_active":"2024-09-02T10:20:16.218072+0000","last_peered":"2024-09-02T10:20:16.218072+0000","last_clean":"2024-09-02T10:20:16.218072+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:20:16.218072+0000","last_undegraded":"2024-09-02T10:20:16.218072+0000","last_fullsized":"2024-09-02T10:20:16.218072+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6584,"log_dups_size":0,"ondisk_log_size":6584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396361728,"num_objects":6048,"num_object_clones":0,"num_object_copies":18144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6584,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":854,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.076977+0000","last_change":"2024-09-02T10:19:05.058925+0000","last_active":"2024-09-02T10:20:14.076977+0000","last_peered":"2024-09-02T10:20:14.076977+0000","last_clean":"2024-09-02T10:20:14.076977+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:20:14.076977+0000","last_undegraded":"2024-09-02T10:20:14.076977+0000","last_fullsized":"2024-09-02T10:20:14.076977+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.789947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.789947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"573'6423","reported_seq":6573,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:13.668702+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:20:13.668702+0000","last_peered":"2024-09-02T10:20:13.668702+0000","last_clean":"2024-09-02T10:20:13.668702+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:20:13.668702+0000","last_undegraded":"2024-09-02T10:20:13.668702+0000","last_fullsized":"2024-09-02T10:20:13.668702+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":6423,"log_dups_size":0,"ondisk_log_size":6423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.641560+0000","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403898368,"num_objects":6163,"num_object_clones":0,"num_object_copies":18489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6423,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1127,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.956114+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:20:11.956114+0000","last_peered":"2024-09-02T10:20:11.956114+0000","last_clean":"2024-09-02T10:20:11.956114+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:20:11.956114+0000","last_undegraded":"2024-09-02T10:20:11.956114+0000","last_fullsized":"2024-09-02T10:20:11.956114+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:59.441621+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"575'6641","reported_seq":11043,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:17.213488+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:20:17.213488+0000","last_peered":"2024-09-02T10:20:17.213488+0000","last_clean":"2024-09-02T10:20:17.213488+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:20:17.213488+0000","last_undegraded":"2024-09-02T10:20:17.213488+0000","last_fullsized":"2024-09-02T10:20:17.213488+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:23.125229+0000","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392626176,"num_objects":5991,"num_object_clones":0,"num_object_copies":17973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6641,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":873,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.077328+0000","last_change":"2024-09-02T10:19:59.731786+0000","last_active":"2024-09-02T10:20:14.077328+0000","last_peered":"2024-09-02T10:20:14.077328+0000","last_clean":"2024-09-02T10:20:14.077328+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:20:14.077328+0000","last_undegraded":"2024-09-02T10:20:14.077328+0000","last_fullsized":"2024-09-02T10:20:14.077328+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:59.731706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.731706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:16.407855+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"574'6596","reported_seq":6688,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.229045+0000","last_change":"2024-09-02T10:18:37.850899+0000","last_active":"2024-09-02T10:20:16.229045+0000","last_peered":"2024-09-02T10:20:16.229045+0000","last_clean":"2024-09-02T10:20:16.229045+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:20:16.229045+0000","last_undegraded":"2024-09-02T10:20:16.229045+0000","last_fullsized":"2024-09-02T10:20:16.229045+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6596,"log_dups_size":0,"ondisk_log_size":6596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:16.783871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396754944,"num_objects":6054,"num_object_clones":0,"num_object_copies":18162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6596,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1122,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:16.434924+0000","last_change":"2024-09-02T10:19:08.999797+0000","last_active":"2024-09-02T10:20:16.434924+0000","last_peered":"2024-09-02T10:20:16.434924+0000","last_clean":"2024-09-02T10:20:16.434924+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:20:16.434924+0000","last_undegraded":"2024-09-02T10:20:16.434924+0000","last_fullsized":"2024-09-02T10:20:16.434924+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:08.999706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:08.999706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"573'6308","reported_seq":6462,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:12.907619+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:20:12.907619+0000","last_peered":"2024-09-02T10:20:12.907619+0000","last_clean":"2024-09-02T10:20:12.907619+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:20:12.907619+0000","last_undegraded":"2024-09-02T10:20:12.907619+0000","last_fullsized":"2024-09-02T10:20:12.907619+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.389208+0000","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405536768,"num_objects":6188,"num_object_clones":0,"num_object_copies":18564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6188,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6308,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":70,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.079434+0000","last_change":"2024-09-02T10:19:06.947029+0000","last_active":"2024-09-02T10:20:14.079434+0000","last_peered":"2024-09-02T10:20:14.079434+0000","last_clean":"2024-09-02T10:20:14.079434+0000","last_became_active":"2024-09-02T10:19:06.049121+0000","last_became_peered":"2024-09-02T10:19:06.049121+0000","last_unstale":"2024-09-02T10:20:14.079434+0000","last_undegraded":"2024-09-02T10:20:14.079434+0000","last_fullsized":"2024-09-02T10:20:14.079434+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:06.946957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:06.946957+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:24.543000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"575'6604","reported_seq":6700,"reported_epoch":575,"state":"active+clean","last_fresh":"2024-09-02T10:20:17.195157+0000","last_change":"2024-09-02T10:20:01.132876+0000","last_active":"2024-09-02T10:20:17.195157+0000","last_peered":"2024-09-02T10:20:17.195157+0000","last_clean":"2024-09-02T10:20:17.195157+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:20:17.195157+0000","last_undegraded":"2024-09-02T10:20:17.195157+0000","last_fullsized":"2024-09-02T10:20:17.195157+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6604,"log_dups_size":0,"ondisk_log_size":6604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:20.023586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387710976,"num_objects":5916,"num_object_clones":0,"num_object_copies":17748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6604,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":900,"reported_epoch":574,"state":"active+clean","last_fresh":"2024-09-02T10:20:14.077989+0000","last_change":"2024-09-02T10:19:57.798578+0000","last_active":"2024-09-02T10:20:14.077989+0000","last_peered":"2024-09-02T10:20:14.077989+0000","last_clean":"2024-09-02T10:20:14.077989+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:20:14.077989+0000","last_undegraded":"2024-09-02T10:20:14.077989+0000","last_fullsized":"2024-09-02T10:20:14.077989+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:57.798501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:57.798501+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:05.189778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"573'6377","reported_seq":6467,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:13.206540+0000","last_change":"2024-09-02T10:18:37.066015+0000","last_active":"2024-09-02T10:20:13.206540+0000","last_peered":"2024-09-02T10:20:13.206540+0000","last_clean":"2024-09-02T10:20:13.206540+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:20:13.206540+0000","last_undegraded":"2024-09-02T10:20:13.206540+0000","last_fullsized":"2024-09-02T10:20:13.206540+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6377,"log_dups_size":0,"ondisk_log_size":6377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:18.014019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405602304,"num_objects":6189,"num_object_clones":0,"num_object_copies":18567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6377,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":290,"reported_epoch":573,"state":"active+clean","last_fresh":"2024-09-02T10:20:11.966810+0000","last_change":"2024-09-02T10:19:02.956010+0000","last_active":"2024-09-02T10:20:11.966810+0000","last_peered":"2024-09-02T10:20:11.966810+0000","last_clean":"2024-09-02T10:20:11.966810+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:20:11.966810+0000","last_undegraded":"2024-09-02T10:20:11.966810+0000","last_fullsized":"2024-09-02T10:20:11.966810+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:02.955947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:02.955947+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:26.242539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":6361382936,"num_objects":97068,"num_object_clones":0,"num_object_copies":291204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97068,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":103472,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8538,"num_bytes_recovered":559546368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19254489088,"data_stored":19254476872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103472,"ondisk_log_size":103472,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383922,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1611244,"kb_used_data":1151060,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":92133908,"statfs":{"total":95995035648,"available":94345121792,"internally_reserved":0,"allocated":1178685440,"data_stored":1178214515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":471202244},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612855,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1482524,"kb_used_data":1136204,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92262628,"statfs":{"total":95995035648,"available":94476931072,"internally_reserved":0,"allocated":1163472896,"data_stored":1163002302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841787,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2092304,"kb_used_data":1583548,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":91652848,"statfs":{"total":95995035648,"available":93852516352,"internally_reserved":0,"allocated":1621553152,"data_stored":1621091083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":520944064},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070721,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1154060,"kb_used_data":834108,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92591092,"statfs":{"total":95995035648,"available":94813278208,"internally_reserved":0,"allocated":854126592,"data_stored":853664523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299654,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2478436,"kb_used_data":1991888,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91266716,"statfs":{"total":95995035648,"available":93457117184,"internally_reserved":0,"allocated":2039693312,"data_stored":2039223238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528586,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1858664,"kb_used_data":1524820,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91886488,"statfs":{"total":95995035648,"available":94091763712,"internally_reserved":0,"allocated":1561415680,"data_stored":1560944755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":341834183},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724815,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3590452,"kb_used_data":3035872,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":90154700,"statfs":{"total":95995035648,"available":92318412800,"internally_reserved":0,"allocated":3108732928,"data_stored":3108254379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953748,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1885584,"kb_used_data":1587836,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":91859568,"statfs":{"total":95995035648,"available":94064197632,"internally_reserved":0,"allocated":1625944064,"data_stored":1625481995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149977,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3701008,"kb_used_data":3182524,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":90044144,"statfs":{"total":95995035648,"available":92205203456,"internally_reserved":0,"allocated":3258904576,"data_stored":3258442507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":530905544},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411613,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575138,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2153176,"kb_used_data":1680456,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":91591976,"statfs":{"total":95995035648,"available":93790183424,"internally_reserved":0,"allocated":1720786944,"data_stored":1720317251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738664,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1766096,"kb_used_data":1207356,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91979056,"statfs":{"total":95995035648,"available":94186553344,"internally_reserved":0,"allocated":1236332544,"data_stored":1235870475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1221066752,"data_stored":1221066752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1704595456,"data_stored":1704591384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3248291840,"data_stored":3248291840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1608318976,"data_stored":1608318976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3103395840,"data_stored":3103391768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1553661952,"data_stored":1553661952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2026373120,"data_stored":2026373120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":849412096,"data_stored":849412096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1616052224,"data_stored":1616052224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1151995904,"data_stored":1151991832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1171324928,"data_stored":1171324928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:20:18.452 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:20:18.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:18.496 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:18.497 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:18.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:18.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:18.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:18.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:20:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:18 smithi151 ceph-mon[16639]: pgmap v1334: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 170 B/s rd, 23 MiB/s wr, 897 op/s 2024-09-02T10:20:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:18 smithi151 ceph-mon[16639]: from='client.17271 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/481466847' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:20:18.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:18 smithi187 ceph-mon[20072]: pgmap v1334: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 170 B/s rd, 23 MiB/s wr, 897 op/s 2024-09-02T10:20:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:18 smithi187 ceph-mon[20072]: from='client.17271 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/481466847' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:20:18.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:18.758 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:18.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:18.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:18 smithi162 ceph-mon[20279]: pgmap v1334: 47 pgs: 47 active+clean; 5.9 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 170 B/s rd, 23 MiB/s wr, 897 op/s 2024-09-02T10:20:18.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:18 smithi162 ceph-mon[20279]: from='client.17271 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:18.810 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/481466847' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:20:18.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:20:18.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:18.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:18.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:18.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:18.983 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:19.008 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:19.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:19.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:19.133 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:19.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:19.225 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:19.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:19.256 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:19.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:19.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:19.327 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:19.359 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:19.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:19.386 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:19.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:19.525 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:19.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:19.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:19.654 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:19.654 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:19.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:19 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:20:19.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:19 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:19.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:19 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:19.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/481466847' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:20:19.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:19 smithi162 ceph-mon[20279]: osdmap e576: 12 total, 12 up, 11 in 2024-09-02T10:20:19.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:19 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:20:19.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:19 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:20:19.656 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:19 smithi162 ceph-mon[20279]: 6.c scrub starts 2024-09-02T10:20:19.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:19 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:20:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:19 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:19 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/481466847' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:20:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:19 smithi187 ceph-mon[20072]: osdmap e576: 12 total, 12 up, 11 in 2024-09-02T10:20:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:19 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:20:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:19 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:20:19.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:19 smithi187 ceph-mon[20072]: 6.c scrub starts 2024-09-02T10:20:19.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:19.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:19.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:19.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:19 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:20:19.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:19 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:19.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:19 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:19.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/481466847' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:20:19.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:19 smithi151 ceph-mon[16639]: osdmap e576: 12 total, 12 up, 11 in 2024-09-02T10:20:19.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:19 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:20:19.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:19 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:20:19.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:19 smithi151 ceph-mon[16639]: 6.c scrub starts 2024-09-02T10:20:19.795 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:19.795 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:19.879 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:19.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:19.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:19.933 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:19.933 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:19.965 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:19.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:19.996 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:19.996 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:20.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:20.025 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:20.025 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:20.041 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:20.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:20.051 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:20.051 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:20.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:20.077 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:20.105 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:20.105 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:20.127 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:20.128 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:20.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:20.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:20.149 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:20.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:20.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:20.166 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:20.178 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:20.178 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:20.200 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:20.200 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:20.235 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:20.236 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:20.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:20.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:20.302 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:20.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:20.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:20.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:20.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:20.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:20.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:20.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:20.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:20.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:20.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:20.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:20.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:20 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:20:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:20 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:20:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:20 smithi187 ceph-mon[20072]: pgmap v1336: 47 pgs: 47 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 4.5 MiB/s wr, 923 op/s 2024-09-02T10:20:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:20 smithi187 ceph-mon[20072]: 6.b scrub starts 2024-09-02T10:20:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/505211897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:20.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:20.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:20.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:20.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:20 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:20:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:20 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:20:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:20 smithi162 ceph-mon[20279]: pgmap v1336: 47 pgs: 47 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 4.5 MiB/s wr, 923 op/s 2024-09-02T10:20:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:20 smithi162 ceph-mon[20279]: 6.b scrub starts 2024-09-02T10:20:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/505211897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:20.915 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:20 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:20:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:20 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:20:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:20 smithi151 ceph-mon[16639]: pgmap v1336: 47 pgs: 47 active+clean; 5.8 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 4.5 MiB/s wr, 923 op/s 2024-09-02T10:20:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:20 smithi151 ceph-mon[16639]: 6.b scrub starts 2024-09-02T10:20:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/505211897' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:20.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:20.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:21.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:21.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:21.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:21.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:21.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:21.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:21.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:21.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:21.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:20:21.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:21.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:21.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:21.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:21.466 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:21.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:21.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:20:21.533 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:21.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:21.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:21.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:21.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:21.722 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:21.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:21.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:21 smithi187 ceph-mon[20072]: 6.a scrub starts 2024-09-02T10:20:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:21 smithi187 ceph-mon[20072]: 6.c scrub ok 2024-09-02T10:20:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:21 smithi187 ceph-mon[20072]: 6.b scrub ok 2024-09-02T10:20:21.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:21 smithi151 ceph-mon[16639]: 6.a scrub starts 2024-09-02T10:20:21.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:21 smithi151 ceph-mon[16639]: 6.c scrub ok 2024-09-02T10:20:21.808 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:21 smithi151 ceph-mon[16639]: 6.b scrub ok 2024-09-02T10:20:21.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:21.845 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:21.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:21 smithi162 ceph-mon[20279]: 6.a scrub starts 2024-09-02T10:20:21.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:21 smithi162 ceph-mon[20279]: 6.c scrub ok 2024-09-02T10:20:21.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:21 smithi162 ceph-mon[20279]: 6.b scrub ok 2024-09-02T10:20:21.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:21.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:21.885 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:21.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:21.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:21.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:22.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:22.104 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:22.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:22.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:22.237 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:22.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:22.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:22.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:22.361 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:22.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:22.458 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:20:22.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:22.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:22.488 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:22.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:22.621 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:22.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:22.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:22.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:22.751 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:22 smithi187 ceph-mon[20072]: pgmap v1337: 47 pgs: 47 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 2.4 MiB/s wr, 886 op/s 2024-09-02T10:20:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:22 smithi187 ceph-mon[20072]: 6.5 scrub starts 2024-09-02T10:20:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4065878367' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:22 smithi162 ceph-mon[20279]: pgmap v1337: 47 pgs: 47 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 2.4 MiB/s wr, 886 op/s 2024-09-02T10:20:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:22 smithi162 ceph-mon[20279]: 6.5 scrub starts 2024-09-02T10:20:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4065878367' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:22.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:22.879 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:22.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:22.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:22 smithi151 ceph-mon[16639]: pgmap v1337: 47 pgs: 47 active+clean; 5.7 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 2.4 MiB/s wr, 886 op/s 2024-09-02T10:20:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:22 smithi151 ceph-mon[16639]: 6.5 scrub starts 2024-09-02T10:20:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4065878367' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:22.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:23.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:23.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:23.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:20:23.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:23.130 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:23.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:23.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:23.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:23.258 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:23.259 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:23.304 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:20:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:20:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:20:23.304 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:20:23.305 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:20:23.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:23.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:23.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:23.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:23.439 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:20:23.439 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:20:23.439 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-02T10:20:23.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:20:23.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:23.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:23.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:23.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:23.649 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:23.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:23.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:20:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: 6.a scrub ok 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4065878367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: osdmap e577: 12 total, 12 up, 11 in 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: 6.8 scrub starts 2024-09-02T10:20:23.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:23 smithi187 ceph-mon[20072]: 6.1 scrub starts 2024-09-02T10:20:23.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:23.777 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:23.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:20:23.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:23.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: 6.a scrub ok 2024-09-02T10:20:23.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:20:23.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:23.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:20:23.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:20:23.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4065878367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:23.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: osdmap e577: 12 total, 12 up, 11 in 2024-09-02T10:20:23.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:20:23.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:20:23.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: 6.8 scrub starts 2024-09-02T10:20:23.806 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:23 smithi151 ceph-mon[16639]: 6.1 scrub starts 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: 6.a scrub ok 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4065878367' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: osdmap e577: 12 total, 12 up, 11 in 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:20:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:20:23.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: 6.8 scrub starts 2024-09-02T10:20:23.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:23 smithi162 ceph-mon[20279]: 6.1 scrub starts 2024-09-02T10:20:23.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:23.895 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:23.900 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:23.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:23.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:23.917 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:24.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:24.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:24.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:24.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:20:24.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:24.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:24.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:24.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:24.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:24.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:24.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:24.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:24.402 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:24.403 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:24.421 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:24.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:24.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:24.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:24.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:24 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:20:24.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:24 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:20:24.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:24 smithi151 ceph-mon[16639]: pgmap v1339: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-02T10:20:24.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:24 smithi151 ceph-mon[16639]: 6.5 scrub ok 2024-09-02T10:20:24.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:24 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:20:24.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:24 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:20:24.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:24 smithi151 ceph-mon[16639]: 6.8 scrub ok 2024-09-02T10:20:24.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:24.662 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:24.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:24.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:24 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:20:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:24 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:20:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:24 smithi187 ceph-mon[20072]: pgmap v1339: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-02T10:20:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:24 smithi187 ceph-mon[20072]: 6.5 scrub ok 2024-09-02T10:20:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:24 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:20:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:24 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:20:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:24 smithi187 ceph-mon[20072]: 6.8 scrub ok 2024-09-02T10:20:24.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:24.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:24.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:24.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:24.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:24 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:20:24.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:24 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:20:24.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:24 smithi162 ceph-mon[20279]: pgmap v1339: 47 pgs: 47 active+clean; 5.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.09k op/s 2024-09-02T10:20:24.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:24 smithi162 ceph-mon[20279]: 6.5 scrub ok 2024-09-02T10:20:24.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:24 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:20:24.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:24 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:20:24.828 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:24 smithi162 ceph-mon[20279]: 6.8 scrub ok 2024-09-02T10:20:24.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:24.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:24.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:24.973 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:20:24.974 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:20:24.986 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:20:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:20:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:20:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:20:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:41914 4.11 4:884e7a1d:::benchmark_data_smithi187_76542_object41913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-02T10:20:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:47.849121+0000", 2024-09-02T10:20:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 517.13362150800003, 2024-09-02T10:20:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44599071800000001, 2024-09-02T10:20:24.986 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 41914 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.849121+0000", 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.849121+0000", 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.849123+0000", 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.528e-06 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.849166+0000", 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3135e-05 2024-09-02T10:20:24.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.849182+0000", 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5696e-05 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.849188+0000", 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.849253+0000", 2024-09-02T10:20:24.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4993000000000006e-05 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.849411+0000", 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000157269 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6,7", 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.849604+0000", 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000193573 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.850727+0000", 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011227170000000001 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.850964+0000", 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000236846 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:47.851008+0000", 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3692000000000001e-05 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:48.294783+0000", 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44377499999999998 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:48.294861+0000", 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8298000000000001e-05 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:48.295112+0000", 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025112700000000002 2024-09-02T10:20:24.992 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:44070 4.1 e356/341 4:85d0390b:::benchmark_data_smithi187_76542_object44069:head v 356'2337, mlcod=356'2334)", 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:50.499824+0000", 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 514.48291829200002, 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.346744469, 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.499824+0000", 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.499824+0000", 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.499825+0000", 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5199999999999996e-07 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.499916+0000", 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1275999999999998e-05 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.845553+0000", 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.34563696999999999 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.845566+0000", 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2277e-05 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.845698+0000", 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013243099999999999 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.845772+0000", 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3582000000000001e-05 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.846538+0000", 2024-09-02T10:20:24.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076622599999999995 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.846569+0000", 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1055e-05 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:44116 4.1 e356/341 4:81bb743d:::benchmark_data_smithi187_76542_object44115:head v 356'2338, mlcod=356'2335)", 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:50.845459+0000", 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 514.13728382800002, 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.371120214, 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:24.997 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.845459+0000", 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.845459+0000", 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.845461+0000", 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7099999999999999e-06 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:50.845519+0000", 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7855000000000001e-05 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:51.215507+0000", 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36998828099999997 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:24.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:51.215529+0000", 2024-09-02T10:20:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1778999999999999e-05 2024-09-02T10:20:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:51.215586+0000", 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7098000000000001e-05 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:51.215611+0000", 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4981e-05 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:51.216543+0000", 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00093186600000000005 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:51.216579+0000", 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6643999999999997e-05 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(4.17 385/381 [PushOp(4:ed4d2584:::benchmark_data_smithi187_76542_object22418:head, version: 334'1397, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:ed4d2584:::benchmark_data_smithi187_76542_object22418:head@334'1397, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:47.476872+0000", 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 457.505871222, 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64767617899999996, 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.002 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:47.476872+0000", 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:47.476872+0000", 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:47.476873+0000", 2024-09-02T10:20:25.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.082e-06 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:47.476898+0000", 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.512e-05 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:48.123425+0000", 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.64652752099999999 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:48.123447+0000", 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1304000000000001e-05 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:48.123542+0000", 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5079000000000002e-05 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:48.123552+0000", 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0543000000000001e-05 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:48.124548+0000", 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0009955299999999999 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(4.17 391/381 [PushOp(4:eca9b54d:::benchmark_data_smithi187_76542_object83526:head, version: 374'3505, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:eca9b54d:::benchmark_data_smithi187_76542_object83526:head@374'3505, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:11.182314+0000", 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 433.80042900299998, 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24430173099999999, 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.006 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:11.182314+0000", 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:11.182314+0000", 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:11.182315+0000", 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3900000000000003e-07 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:11.182346+0000", 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0812000000000002e-05 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:11.425137+0000", 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24279136200000001 2024-09-02T10:20:25.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:11.425154+0000", 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7028999999999999e-05 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:11.425250+0000", 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6135e-05 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:11.425264+0000", 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4388e-05 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:11.426616+0000", 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013510659999999999 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(4.19 392/385 [PushOp(4:9c803950:::benchmark_data_smithi187_76542_object35380:head, version: 348'2036, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:9c803950:::benchmark_data_smithi187_76542_object35380:head@348'2036, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.654719+0000", 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 429.32802381400001, 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24770604299999999, 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.010 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.654719+0000", 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.654719+0000", 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.654720+0000", 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4300000000000001e-06 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.654802+0000", 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1889000000000002e-05 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.900918+0000", 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24611580899999999 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.900927+0000", 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0990000000000005e-06 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.900990+0000", 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2521999999999995e-05 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.901003+0000", 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3064999999999999e-05 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.902425+0000", 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014222289999999999 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(4.b 392/381 [PushOp(4:d855f3df:::benchmark_data_smithi187_76542_object34609:head, version: 348'2174, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:d855f3df:::benchmark_data_smithi187_76542_object34609:head@348'2174, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:20:25.014 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:16.007523+0000", 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 428.97521990799999, 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.249960137, 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.007523+0000", 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.007523+0000", 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.007524+0000", 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3e-07 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.007583+0000", 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9558000000000001e-05 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.254833+0000", 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24725012099999999 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.254861+0000", 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8238999999999998e-05 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.254953+0000", 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1519000000000002e-05 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.254961+0000", 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5560000000000008e-06 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.257483+0000", 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025215139999999999 2024-09-02T10:20:25.018 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:152642 4.13 e409/354 4:cbf2c277:::benchmark_data_smithi187_76542_object60306:head v 409'5733, mlcod=409'5731)", 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:46.445065+0000", 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 398.53767813399998, 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25489055500000002, 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:46.445065+0000", 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:46.445065+0000", 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:46.445065+0000", 2024-09-02T10:20:25.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7100000000000001e-07 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:46.445070+0000", 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0769999999999998e-06 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:46.695921+0000", 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.250850246 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:46.695927+0000", 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1619999999999996e-06 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:46.696002+0000", 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4795999999999993e-05 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:46.696017+0000", 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5359e-05 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:46.699889+0000", 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038721440000000001 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:46.699955+0000", 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6099999999999994e-05 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:7228 5.3 e439/431 5:c2c93d2d:::benchmark_data_smithi187_89405_object7227:head v 439'482, mlcod=439'478)", 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:14:50.636109+0000", 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 334.34663423299997, 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.268947771, 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:50.636109+0000", 2024-09-02T10:20:25.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:50.636109+0000", 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:50.636109+0000", 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5499999999999998e-07 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:50.636209+0000", 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9609000000000001e-05 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:50.904371+0000", 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.268162383 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:50.904377+0000", 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3239999999999999e-06 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:50.904465+0000", 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7699000000000002e-05 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:50.904484+0000", 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8482e-05 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:50.905015+0000", 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00053093200000000002 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:14:50.905056+0000", 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1786999999999998e-05 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:62898 5.6 5:6eddce46:::benchmark_data_smithi187_89405_object62897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:34.623865+0000", 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 290.35887745299999, 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26170413300000001, 2024-09-02T10:20:25.028 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 62898 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.623865+0000", 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.623865+0000", 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.623867+0000", 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9800000000000001e-06 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.623975+0000", 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107248 2024-09-02T10:20:25.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.624003+0000", 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8606e-05 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.624017+0000", 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4100000000000001e-05 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.624129+0000", 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111481 2024-09-02T10:20:25.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.624243+0000", 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011379600000000001 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.624328+0000", 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5736999999999996e-05 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:25.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.626114+0000", 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001786057 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.626273+0000", 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000158777 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.885454+0000", 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25918118699999998 2024-09-02T10:20:25.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.885518+0000", 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4125e-05 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:34.885569+0000", 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1038999999999997e-05 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:121085 5.6 5:6b940c5b:::benchmark_data_smithi187_89405_object17480:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e488)", 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:16:35.407274+0000", 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 229.575468785, 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24649469199999999, 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 121085 2024-09-02T10:20:25.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.407274+0000", 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.407274+0000", 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.407275+0000", 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7400000000000002e-07 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.407279+0000", 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0439999999999997e-06 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.407295+0000", 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5772000000000001e-05 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.407302+0000", 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.537e-06 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.407347+0000", 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5079e-05 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.407424+0000", 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6376000000000002e-05 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:20:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.407489+0000", 2024-09-02T10:20:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5281000000000001e-05 2024-09-02T10:20:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.039 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.410471+0000", 2024-09-02T10:20:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029822519999999999 2024-09-02T10:20:25.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.414604+0000", 2024-09-02T10:20:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041324229999999997 2024-09-02T10:20:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.653640+0000", 2024-09-02T10:20:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23903595799999999 2024-09-02T10:20:25.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.653696+0000", 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6085999999999998e-05 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:16:35.653769+0000", 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3009999999999994e-05 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.042 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162675 5.7 5:e5906b20:::benchmark_data_smithi187_89405_object59070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.329566+0000", 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.653176425, 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.303775562, 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:20:25.043 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162675 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.329566+0000", 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.329566+0000", 2024-09-02T10:20:25.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.329567+0000", 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8800000000000001e-07 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.329571+0000", 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2509999999999996e-06 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.329581+0000", 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6290000000000005e-06 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.329588+0000", 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6939999999999998e-06 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.329648+0000", 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0186000000000003e-05 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.329793+0000", 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000144931 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.329926+0000", 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133499 2024-09-02T10:20:25.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.335006+0000", 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050793019999999999 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.335112+0000", 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010613899999999999 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.633246+0000", 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29813445300000002 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.633297+0000", 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0600999999999999e-05 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.633342+0000", 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4988999999999999e-05 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.049 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.25933.0:162675 5.7 e507/431 ondisk, result = 0)", 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.341095+0000", 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.64164791100001, 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29225268599999998, 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.341095+0000", 2024-09-02T10:20:25.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.341095+0000", 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.341095+0000", 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8199999999999999e-07 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.341096+0000", 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4109999999999999e-06 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.631872+0000", 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29077597399999999 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.631878+0000", 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8510000000000004e-06 2024-09-02T10:20:25.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.633233+0000", 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013545149999999999 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.633240+0000", 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5449999999999998e-06 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.633348+0000", 2024-09-02T10:20:25.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010720799999999999 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162690 5.f 5:f963755e:::benchmark_data_smithi187_89405_object59085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.351192+0000", 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.631550726, 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26079149800000001, 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:20:25.054 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162690 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.351192+0000", 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.351192+0000", 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.351193+0000", 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.207e-06 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.351198+0000", 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6229999999999996e-06 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.351209+0000", 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1531e-05 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.351215+0000", 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6160000000000001e-06 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.351280+0000", 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5298999999999999e-05 2024-09-02T10:20:25.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.351416+0000", 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013573400000000001 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.351547+0000", 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013054200000000001 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:25.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.356740+0000", 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051936129999999997 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.604982+0000", 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24824215199999999 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.611872+0000", 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0068895909999999996 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.611936+0000", 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4070000000000002e-05 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.611984+0000", 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7519999999999999e-05 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162693 5.6 5:61b070a9:::benchmark_data_smithi187_89405_object59088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.353616+0000", 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.629127023, 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25610818800000001, 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162693 2024-09-02T10:20:25.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353616+0000", 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353616+0000", 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353617+0000", 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8299999999999996e-07 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353620+0000", 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9389999999999999e-06 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353631+0000", 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0664e-05 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353636+0000", 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1769999999999997e-06 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353687+0000", 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0639000000000001e-05 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353770+0000", 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2736000000000002e-05 2024-09-02T10:20:25.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:20:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353859+0000", 2024-09-02T10:20:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8979000000000003e-05 2024-09-02T10:20:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.357773+0000", 2024-09-02T10:20:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039143399999999997 2024-09-02T10:20:25.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.358099+0000", 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032554900000000001 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.609631+0000", 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25153225200000001 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:25.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.609682+0000", 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1143999999999997e-05 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.609724+0000", 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1986000000000001e-05 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162699 5.7 5:ead0368d:::benchmark_data_smithi187_89405_object59094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.381131+0000", 2024-09-02T10:20:25.067 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 180.60161195699999, 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25235912300000002, 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162699 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.381131+0000", 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.381131+0000", 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.381132+0000", 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.263e-06 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.381139+0000", 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4760000000000003e-06 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.381153+0000", 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4887e-05 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.381160+0000", 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6159999999999998e-06 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.381223+0000", 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2675000000000006e-05 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.381355+0000", 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131977 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:20:25.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.381500+0000", 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000145277 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.385630+0000", 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00413015 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.386718+0000", 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010879329999999999 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.633395+0000", 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24667682599999999 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.633447+0000", 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2182000000000003e-05 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.633490+0000", 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2861000000000002e-05 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(6.7 560/549 [PushReplyOp(6:ef7e0a83:::benchmark_data_smithi187_102448_object8727:head)])", 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:35.551478+0000", 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 49.431264628999998, 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.225953931, 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.551478+0000", 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.551478+0000", 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.551479+0000", 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1500000000000003e-07 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.551484+0000", 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6800000000000001e-06 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.775764+0000", 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22427994300000001 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.775772+0000", 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1110000000000008e-06 2024-09-02T10:20:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.775865+0000", 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3246999999999997e-05 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.777432+0000", 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015671350000000001 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.26524.0:108513 6.7 e576/565 ondisk, result = 0)", 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:20.007908+0000", 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.9748347429999997, 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.109838687, 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.007908+0000", 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.007908+0000", 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.007909+0000", 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1900000000000002e-07 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.007913+0000", 2024-09-02T10:20:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4120000000000003e-06 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.117636+0000", 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.10972272700000001 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.117642+0000", 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.139e-06 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.117677+0000", 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4625999999999997e-05 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.117681+0000", 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1409999999999998e-06 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.117747+0000", 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6023000000000001e-05 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.26524.0:108529 6.7 e576/565 ondisk, result = 0)", 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:20.030074+0000", 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.9526688859999997, 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15446654600000001, 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.030074+0000", 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.030074+0000", 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.030075+0000", 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8999999999999996e-07 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.030087+0000", 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2113000000000001e-05 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.184442+0000", 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15435480700000001 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.184446+0000", 2024-09-02T10:20:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1740000000000002e-06 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.184475+0000", 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8932999999999999e-05 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.184479+0000", 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3930000000000001e-06 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.184540+0000", 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1435999999999999e-05 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:108580 6.7 6:eb55b752:::benchmark_data_smithi187_102448_object8308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e576)", 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:20.113968+0000", 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.8687744630000003, 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.103087761, 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 108580 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.113968+0000", 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.113968+0000", 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.113969+0000", 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4499999999999995e-07 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.113973+0000", 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7869999999999999e-06 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.113982+0000", 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2e-06 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.113987+0000", 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0209999999999999e-06 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.114030+0000", 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2895999999999997e-05 2024-09-02T10:20:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.114110+0000", 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0102e-05 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,4", 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.114178+0000", 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7984999999999994e-05 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.117555+0000", 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003376932 2024-09-02T10:20:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.117762+0000", 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000207052 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.216952+0000", 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099189814000000001 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.217011+0000", 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9256999999999999e-05 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:20.217056+0000", 2024-09-02T10:20:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.477e-05 2024-09-02T10:20:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:25.092 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:20:25.093 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:25.093 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:25.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:25.094 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:25.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:25.099 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:25.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:25.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:25.124 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:25.124 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:25.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:25.156 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:25.191 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:25.191 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:25.216 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:25.216 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:25.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:25.240 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:25.240 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:25.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:25.242 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:25.266 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:25.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:25.292 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:25.292 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:25.321 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:25.321 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:25.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:25.347 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:25.347 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:25.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:25.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:25.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:25.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:25.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:25.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:25.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:25.616 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:25.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:25.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:25.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:25.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:25.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:25 smithi187 ceph-mon[20072]: 6.1 scrub ok 2024-09-02T10:20:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:25 smithi187 ceph-mon[20072]: 6.3 scrub starts 2024-09-02T10:20:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:25 smithi187 ceph-mon[20072]: from='client.26740 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3639885981' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:25.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:25 smithi151 ceph-mon[16639]: 6.1 scrub ok 2024-09-02T10:20:25.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:25 smithi151 ceph-mon[16639]: 6.3 scrub starts 2024-09-02T10:20:25.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:25 smithi151 ceph-mon[16639]: from='client.26740 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:25.779 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3639885981' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:25.814 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1339,"stamp":"2024-09-02T10:20:23.356578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5953684024,"num_objects":90840,"num_object_clones":0,"num_object_copies":272520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90840,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109702,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8541,"num_bytes_recovered":560276544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112554,"ondisk_log_size":112554,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":22545924,"kb_used_data":17680796,"kb_used_omap":17,"kb_used_meta":4864558,"kb_avail":1008650748,"statfs":{"total":1055945392128,"available":1032858365952,"internally_reserved":0,"allocated":18105135104,"data_stored":18100018274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":4981308278},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-495910912,"num_objects":-7567,"num_object_clones":0,"num_object_copies":-22701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7567,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.924173"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":497,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.458236+0000","last_change":"2024-09-02T10:18:43.049145+0000","last_active":"2024-09-02T10:20:18.458236+0000","last_peered":"2024-09-02T10:20:18.458236+0000","last_clean":"2024-09-02T10:20:18.458236+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:20:18.458236+0000","last_undegraded":"2024-09-02T10:20:18.458236+0000","last_fullsized":"2024-09-02T10:20:18.458236+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:43.049077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:43.049077+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":313,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.459007+0000","last_change":"2024-09-02T10:19:47.121714+0000","last_active":"2024-09-02T10:20:18.459007+0000","last_peered":"2024-09-02T10:20:18.459007+0000","last_clean":"2024-09-02T10:20:18.459007+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:20:18.459007+0000","last_undegraded":"2024-09-02T10:20:18.459007+0000","last_fullsized":"2024-09-02T10:20:18.459007+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:04.209863+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":762,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.453049+0000","last_change":"2024-09-02T10:19:48.146767+0000","last_active":"2024-09-02T10:20:18.453049+0000","last_peered":"2024-09-02T10:20:18.453049+0000","last_clean":"2024-09-02T10:20:18.453049+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:20:18.453049+0000","last_undegraded":"2024-09-02T10:20:18.453049+0000","last_fullsized":"2024-09-02T10:20:18.453049+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:48.146721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:48.146721+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:55.994378+0000","scrub_duration":995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":754,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.453002+0000","last_change":"2024-09-02T10:18:39.045140+0000","last_active":"2024-09-02T10:20:18.453002+0000","last_peered":"2024-09-02T10:20:18.453002+0000","last_clean":"2024-09-02T10:20:18.453002+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:20:18.453002+0000","last_undegraded":"2024-09-02T10:20:18.453002+0000","last_fullsized":"2024-09-02T10:20:18.453002+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:18:39.045083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:18:39.045083+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":66,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.457197+0000","last_change":"2024-09-02T10:19:05.058923+0000","last_active":"2024-09-02T10:20:18.457197+0000","last_peered":"2024-09-02T10:20:18.457197+0000","last_clean":"2024-09-02T10:20:18.457197+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:20:18.457197+0000","last_undegraded":"2024-09-02T10:20:18.457197+0000","last_fullsized":"2024-09-02T10:20:18.457197+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:01.810089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:01.810089+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:19.405935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":611,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.472227+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:20:18.472227+0000","last_peered":"2024-09-02T10:20:18.472227+0000","last_clean":"2024-09-02T10:20:18.472227+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:20:18.472227+0000","last_undegraded":"2024-09-02T10:20:18.472227+0000","last_fullsized":"2024-09-02T10:20:18.472227+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:08.487285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1127,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.461779+0000","last_change":"2024-09-02T10:19:03.880037+0000","last_active":"2024-09-02T10:20:18.461779+0000","last_peered":"2024-09-02T10:20:18.461779+0000","last_clean":"2024-09-02T10:20:18.461779+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:20:18.461779+0000","last_undegraded":"2024-09-02T10:20:18.461779+0000","last_fullsized":"2024-09-02T10:20:18.461779+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:03.879968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:03.879968+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":777,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.453041+0000","last_change":"2024-09-02T10:20:00.175101+0000","last_active":"2024-09-02T10:20:18.453041+0000","last_peered":"2024-09-02T10:20:18.453041+0000","last_clean":"2024-09-02T10:20:18.453041+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:20:18.453041+0000","last_undegraded":"2024-09-02T10:20:18.453041+0000","last_fullsized":"2024-09-02T10:20:18.453041+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.175032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.175032+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:28.457490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":777,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.459005+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:20:18.459005+0000","last_peered":"2024-09-02T10:20:18.459005+0000","last_clean":"2024-09-02T10:20:18.459005+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:20:18.459005+0000","last_undegraded":"2024-09-02T10:20:18.459005+0000","last_fullsized":"2024-09-02T10:20:18.459005+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:52.877235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":745,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.458811+0000","last_change":"2024-09-02T10:19:05.910750+0000","last_active":"2024-09-02T10:20:18.458811+0000","last_peered":"2024-09-02T10:20:18.458811+0000","last_clean":"2024-09-02T10:20:18.458811+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:20:18.458811+0000","last_undegraded":"2024-09-02T10:20:18.458811+0000","last_fullsized":"2024-09-02T10:20:18.458811+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:05.910691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:05.910691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:20.353104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":597,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.458712+0000","last_change":"2024-09-02T10:19:52.963372+0000","last_active":"2024-09-02T10:20:18.458712+0000","last_peered":"2024-09-02T10:20:18.458712+0000","last_clean":"2024-09-02T10:20:18.458712+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:20:18.458712+0000","last_undegraded":"2024-09-02T10:20:18.458712+0000","last_fullsized":"2024-09-02T10:20:18.458712+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.963289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.963289+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:10.821299+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1102,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.458602+0000","last_change":"2024-09-02T10:19:00.052912+0000","last_active":"2024-09-02T10:20:18.458602+0000","last_peered":"2024-09-02T10:20:18.458602+0000","last_clean":"2024-09-02T10:20:18.458602+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:20:18.458602+0000","last_undegraded":"2024-09-02T10:20:18.458602+0000","last_fullsized":"2024-09-02T10:20:18.458602+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.052857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.052857+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:29.321752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1045,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.461975+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:20:18.461975+0000","last_peered":"2024-09-02T10:20:18.461975+0000","last_clean":"2024-09-02T10:20:18.461975+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:20:18.461975+0000","last_undegraded":"2024-09-02T10:20:18.461975+0000","last_fullsized":"2024-09-02T10:20:18.461975+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:57.041883+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1026,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-09-02T10:20:22.462751+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:20:22.462751+0000","last_peered":"2024-09-02T10:20:22.462751+0000","last_clean":"2024-09-02T10:20:22.462751+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:20:22.462751+0000","last_undegraded":"2024-09-02T10:20:22.462751+0000","last_fullsized":"2024-09-02T10:20:22.462751+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:01.846422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1015,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.461818+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:20:18.461818+0000","last_peered":"2024-09-02T10:20:18.461818+0000","last_clean":"2024-09-02T10:20:18.461818+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:20:18.461818+0000","last_undegraded":"2024-09-02T10:20:18.461818+0000","last_fullsized":"2024-09-02T10:20:18.461818+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.850900+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"576'6844","reported_seq":7001,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:21.723661+0000","last_change":"2024-09-02T10:19:59.986749+0000","last_active":"2024-09-02T10:20:21.723661+0000","last_peered":"2024-09-02T10:20:21.723661+0000","last_clean":"2024-09-02T10:20:21.723661+0000","last_became_active":"2024-09-02T10:18:37.599064+0000","last_became_peered":"2024-09-02T10:18:37.599064+0000","last_unstale":"2024-09-02T10:20:21.723661+0000","last_undegraded":"2024-09-02T10:20:21.723661+0000","last_fullsized":"2024-09-02T10:20:21.723661+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":6844,"log_dups_size":0,"ondisk_log_size":6844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:01.232192+0000","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363724800,"num_objects":5550,"num_object_clones":0,"num_object_copies":16650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6844,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1102,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.461881+0000","last_change":"2024-09-02T10:19:52.109223+0000","last_active":"2024-09-02T10:20:18.461881+0000","last_peered":"2024-09-02T10:20:18.461881+0000","last_clean":"2024-09-02T10:20:18.461881+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:20:18.461881+0000","last_undegraded":"2024-09-02T10:20:18.461881+0000","last_fullsized":"2024-09-02T10:20:18.461881+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:04.388339+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"576'6735","reported_seq":11125,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:19.212200+0000","last_change":"2024-09-02T10:19:55.677256+0000","last_active":"2024-09-02T10:20:19.212200+0000","last_peered":"2024-09-02T10:20:19.212200+0000","last_clean":"2024-09-02T10:20:19.212200+0000","last_became_active":"2024-09-02T10:19:32.596751+0000","last_became_peered":"2024-09-02T10:19:32.596751+0000","last_unstale":"2024-09-02T10:20:19.212200+0000","last_undegraded":"2024-09-02T10:20:19.212200+0000","last_fullsized":"2024-09-02T10:20:19.212200+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":6735,"log_dups_size":0,"ondisk_log_size":6735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:09.732905+0000","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383057920,"num_objects":5845,"num_object_clones":0,"num_object_copies":17535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6735,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":138346496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1217,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.458582+0000","last_change":"2024-09-02T10:19:49.817426+0000","last_active":"2024-09-02T10:20:18.458582+0000","last_peered":"2024-09-02T10:20:18.458582+0000","last_clean":"2024-09-02T10:20:18.458582+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:20:18.458582+0000","last_undegraded":"2024-09-02T10:20:18.458582+0000","last_fullsized":"2024-09-02T10:20:18.458582+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.817347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.817347+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:54.901580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"576'6858","reported_seq":6955,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:22.142045+0000","last_change":"2024-09-02T10:18:37.062429+0000","last_active":"2024-09-02T10:20:22.142045+0000","last_peered":"2024-09-02T10:20:22.142045+0000","last_clean":"2024-09-02T10:20:22.142045+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:20:22.142045+0000","last_undegraded":"2024-09-02T10:20:22.142045+0000","last_fullsized":"2024-09-02T10:20:22.142045+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6858,"log_dups_size":0,"ondisk_log_size":6858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359989272,"num_objects":5494,"num_object_clones":0,"num_object_copies":16482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5494,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6858,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":877,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-09-02T10:20:22.463735+0000","last_change":"2024-09-02T10:20:18.812280+0000","last_active":"2024-09-02T10:20:22.463735+0000","last_peered":"2024-09-02T10:20:22.463735+0000","last_clean":"2024-09-02T10:20:22.463735+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:20:22.463735+0000","last_undegraded":"2024-09-02T10:20:22.463735+0000","last_fullsized":"2024-09-02T10:20:22.463735+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.812212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.812212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:47.719780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"577'6865","reported_seq":7040,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-09-02T10:20:23.151971+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:20:23.151971+0000","last_peered":"2024-09-02T10:20:23.151971+0000","last_clean":"2024-09-02T10:20:23.151971+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:20:23.151971+0000","last_undegraded":"2024-09-02T10:20:23.151971+0000","last_fullsized":"2024-09-02T10:20:23.151971+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":6865,"log_dups_size":0,"ondisk_log_size":6865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:37.995088+0000","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361562112,"num_objects":5517,"num_object_clones":0,"num_object_copies":16551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5517,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6865,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":264,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.458112+0000","last_change":"2024-09-02T10:19:50.392195+0000","last_active":"2024-09-02T10:20:18.458112+0000","last_peered":"2024-09-02T10:20:18.458112+0000","last_clean":"2024-09-02T10:20:18.458112+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:20:18.458112+0000","last_undegraded":"2024-09-02T10:20:18.458112+0000","last_fullsized":"2024-09-02T10:20:18.458112+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.392146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.392146+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.406921+0000","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"576'6888","reported_seq":6984,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:21.267929+0000","last_change":"2024-09-02T10:18:37.850883+0000","last_active":"2024-09-02T10:20:21.267929+0000","last_peered":"2024-09-02T10:20:21.267929+0000","last_clean":"2024-09-02T10:20:21.267929+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:20:21.267929+0000","last_undegraded":"2024-09-02T10:20:21.267929+0000","last_fullsized":"2024-09-02T10:20:21.267929+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6888,"log_dups_size":0,"ondisk_log_size":6888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374734848,"num_objects":5718,"num_object_clones":0,"num_object_copies":17154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6888,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1172,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.458131+0000","last_change":"2024-09-02T10:19:51.832888+0000","last_active":"2024-09-02T10:20:18.458131+0000","last_peered":"2024-09-02T10:20:18.458131+0000","last_clean":"2024-09-02T10:20:18.458131+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:20:18.458131+0000","last_undegraded":"2024-09-02T10:20:18.458131+0000","last_fullsized":"2024-09-02T10:20:18.458131+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.832822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.832822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.770876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"576'6752","reported_seq":11190,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:19.201237+0000","last_change":"2024-09-02T10:19:54.548123+0000","last_active":"2024-09-02T10:20:19.201237+0000","last_peered":"2024-09-02T10:20:19.201237+0000","last_clean":"2024-09-02T10:20:19.201237+0000","last_became_active":"2024-09-02T10:19:31.588084+0000","last_became_peered":"2024-09-02T10:19:31.588084+0000","last_unstale":"2024-09-02T10:20:19.201237+0000","last_undegraded":"2024-09-02T10:20:19.201237+0000","last_fullsized":"2024-09-02T10:20:19.201237+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":6752,"log_dups_size":0,"ondisk_log_size":6752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:12.643132+0000","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379584512,"num_objects":5792,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6752,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2135,"num_bytes_recovered":139919360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":639,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.474172+0000","last_change":"2024-09-02T10:20:00.749882+0000","last_active":"2024-09-02T10:20:18.474172+0000","last_peered":"2024-09-02T10:20:18.474172+0000","last_clean":"2024-09-02T10:20:18.474172+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:20:18.474172+0000","last_undegraded":"2024-09-02T10:20:18.474172+0000","last_fullsized":"2024-09-02T10:20:18.474172+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.876682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"576'6641","reported_seq":6737,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:19.389582+0000","last_change":"2024-09-02T10:18:37.600687+0000","last_active":"2024-09-02T10:20:19.389582+0000","last_peered":"2024-09-02T10:20:19.389582+0000","last_clean":"2024-09-02T10:20:19.389582+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:20:19.389582+0000","last_undegraded":"2024-09-02T10:20:19.389582+0000","last_fullsized":"2024-09-02T10:20:19.389582+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6641,"log_dups_size":0,"ondisk_log_size":6641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:21.954579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376111104,"num_objects":5739,"num_object_clones":0,"num_object_copies":17217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6641,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":611,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.472304+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:20:18.472304+0000","last_peered":"2024-09-02T10:20:18.472304+0000","last_clean":"2024-09-02T10:20:18.472304+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:20:18.472304+0000","last_undegraded":"2024-09-02T10:20:18.472304+0000","last_fullsized":"2024-09-02T10:20:18.472304+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:51.265964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"577'6999","reported_seq":7154,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-09-02T10:20:22.890915+0000","last_change":"2024-09-02T10:19:52.017169+0000","last_active":"2024-09-02T10:20:22.890915+0000","last_peered":"2024-09-02T10:20:22.890915+0000","last_clean":"2024-09-02T10:20:22.890915+0000","last_became_active":"2024-09-02T10:18:37.064844+0000","last_became_peered":"2024-09-02T10:18:37.064844+0000","last_unstale":"2024-09-02T10:20:22.890915+0000","last_undegraded":"2024-09-02T10:20:22.890915+0000","last_fullsized":"2024-09-02T10:20:22.890915+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":6999,"log_dups_size":0,"ondisk_log_size":6999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:00.776952+0000","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372178944,"num_objects":5679,"num_object_clones":0,"num_object_copies":17037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5679,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6999,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":363,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.456335+0000","last_change":"2024-09-02T10:19:58.949821+0000","last_active":"2024-09-02T10:20:18.456335+0000","last_peered":"2024-09-02T10:20:18.456335+0000","last_clean":"2024-09-02T10:20:18.456335+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:20:18.456335+0000","last_undegraded":"2024-09-02T10:20:18.456335+0000","last_fullsized":"2024-09-02T10:20:18.456335+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:58.949745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.949745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.256455+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"577'6962","reported_seq":11481,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-09-02T10:20:23.212239+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:20:23.212239+0000","last_peered":"2024-09-02T10:20:23.212239+0000","last_clean":"2024-09-02T10:20:23.212239+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:20:23.212239+0000","last_undegraded":"2024-09-02T10:20:23.212239+0000","last_fullsized":"2024-09-02T10:20:23.212239+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":6962,"log_dups_size":0,"ondisk_log_size":6962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:22.534088+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365690880,"num_objects":5580,"num_object_clones":0,"num_object_copies":16740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":612,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.472268+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:20:18.472268+0000","last_peered":"2024-09-02T10:20:18.472268+0000","last_clean":"2024-09-02T10:20:18.472268+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:20:18.472268+0000","last_undegraded":"2024-09-02T10:20:18.472268+0000","last_fullsized":"2024-09-02T10:20:18.472268+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:02.710854+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"576'6885","reported_seq":6981,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:21.267168+0000","last_change":"2024-09-02T10:18:37.848300+0000","last_active":"2024-09-02T10:20:21.267168+0000","last_peered":"2024-09-02T10:20:21.267168+0000","last_clean":"2024-09-02T10:20:21.267168+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:20:21.267168+0000","last_undegraded":"2024-09-02T10:20:21.267168+0000","last_fullsized":"2024-09-02T10:20:21.267168+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6885,"log_dups_size":0,"ondisk_log_size":6885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376635392,"num_objects":5747,"num_object_clones":0,"num_object_copies":17241,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5747,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6885,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":858,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.458671+0000","last_change":"2024-09-02T10:19:05.058925+0000","last_active":"2024-09-02T10:20:18.458671+0000","last_peered":"2024-09-02T10:20:18.458671+0000","last_clean":"2024-09-02T10:20:18.458671+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:20:18.458671+0000","last_undegraded":"2024-09-02T10:20:18.458671+0000","last_fullsized":"2024-09-02T10:20:18.458671+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.789947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.789947+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"576'6712","reported_seq":6868,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.664192+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:20:18.664192+0000","last_peered":"2024-09-02T10:20:18.664192+0000","last_clean":"2024-09-02T10:20:18.664192+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:20:18.664192+0000","last_undegraded":"2024-09-02T10:20:18.664192+0000","last_fullsized":"2024-09-02T10:20:18.664192+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":6712,"log_dups_size":0,"ondisk_log_size":6712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.641560+0000","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384958464,"num_objects":5874,"num_object_clones":0,"num_object_copies":17622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6712,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1133,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.460953+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:20:18.460953+0000","last_peered":"2024-09-02T10:20:18.460953+0000","last_clean":"2024-09-02T10:20:18.460953+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:20:18.460953+0000","last_undegraded":"2024-09-02T10:20:18.460953+0000","last_fullsized":"2024-09-02T10:20:18.460953+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:59.441621+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"576'6956","reported_seq":11361,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:22.205702+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:20:22.205702+0000","last_peered":"2024-09-02T10:20:22.205702+0000","last_clean":"2024-09-02T10:20:22.205702+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:20:22.205702+0000","last_undegraded":"2024-09-02T10:20:22.205702+0000","last_fullsized":"2024-09-02T10:20:22.205702+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":6956,"log_dups_size":0,"ondisk_log_size":6956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:23.125229+0000","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371982336,"num_objects":5676,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6956,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":877,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.458615+0000","last_change":"2024-09-02T10:19:59.731786+0000","last_active":"2024-09-02T10:20:18.458615+0000","last_peered":"2024-09-02T10:20:18.458615+0000","last_clean":"2024-09-02T10:20:18.458615+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:20:18.458615+0000","last_undegraded":"2024-09-02T10:20:18.458615+0000","last_fullsized":"2024-09-02T10:20:18.458615+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:59.731706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.731706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:16.407855+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"576'6896","reported_seq":6992,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:21.258227+0000","last_change":"2024-09-02T10:18:37.850899+0000","last_active":"2024-09-02T10:20:21.258227+0000","last_peered":"2024-09-02T10:20:21.258227+0000","last_clean":"2024-09-02T10:20:21.258227+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:20:21.258227+0000","last_undegraded":"2024-09-02T10:20:21.258227+0000","last_fullsized":"2024-09-02T10:20:21.258227+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:18:36.041320+0000","objects_scrubbed":0,"log_size":6896,"log_dups_size":0,"ondisk_log_size":6896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377094144,"num_objects":5754,"num_object_clones":0,"num_object_copies":17262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6896,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1132,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.886910+0000","last_change":"2024-09-02T10:20:18.886910+0000","last_active":"2024-09-02T10:20:18.886910+0000","last_peered":"2024-09-02T10:20:18.886910+0000","last_clean":"2024-09-02T10:20:18.886910+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:20:18.886910+0000","last_undegraded":"2024-09-02T10:20:18.886910+0000","last_fullsized":"2024-09-02T10:20:18.886910+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.886832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.886832+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.398760+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"577'6898","reported_seq":7061,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-09-02T10:20:22.872232+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:20:22.872232+0000","last_peered":"2024-09-02T10:20:22.872232+0000","last_clean":"2024-09-02T10:20:22.872232+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:20:22.872232+0000","last_undegraded":"2024-09-02T10:20:22.872232+0000","last_fullsized":"2024-09-02T10:20:22.872232+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":6898,"log_dups_size":0,"ondisk_log_size":6898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.389208+0000","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":366870528,"num_objects":5598,"num_object_clones":0,"num_object_copies":16794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5598,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6898,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":74,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.457233+0000","last_change":"2024-09-02T10:19:06.947029+0000","last_active":"2024-09-02T10:20:18.457233+0000","last_peered":"2024-09-02T10:20:18.457233+0000","last_clean":"2024-09-02T10:20:18.457233+0000","last_became_active":"2024-09-02T10:19:06.049121+0000","last_became_peered":"2024-09-02T10:19:06.049121+0000","last_unstale":"2024-09-02T10:20:18.457233+0000","last_undegraded":"2024-09-02T10:20:18.457233+0000","last_fullsized":"2024-09-02T10:20:18.457233+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:06.946957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:06.946957+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:24.543000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"576'6873","reported_seq":7037,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:22.114912+0000","last_change":"2024-09-02T10:20:21.937271+0000","last_active":"2024-09-02T10:20:22.114912+0000","last_peered":"2024-09-02T10:20:22.114912+0000","last_clean":"2024-09-02T10:20:22.114912+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:20:22.114912+0000","last_undegraded":"2024-09-02T10:20:22.114912+0000","last_fullsized":"2024-09-02T10:20:22.114912+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6863","last_scrub_stamp":"2024-09-02T10:20:21.937140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.937140+0000","objects_scrubbed":5699,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:39.939497+0000","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370081792,"num_objects":5647,"num_object_clones":0,"num_object_copies":16941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6873,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":904,"reported_epoch":576,"state":"active+clean","last_fresh":"2024-09-02T10:20:18.458838+0000","last_change":"2024-09-02T10:19:57.798578+0000","last_active":"2024-09-02T10:20:18.458838+0000","last_peered":"2024-09-02T10:20:18.458838+0000","last_clean":"2024-09-02T10:20:18.458838+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:20:18.458838+0000","last_undegraded":"2024-09-02T10:20:18.458838+0000","last_fullsized":"2024-09-02T10:20:18.458838+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:57.798501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:57.798501+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:05.189778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"577'6938","reported_seq":7101,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-09-02T10:20:23.187353+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:20:23.187353+0000","last_peered":"2024-09-02T10:20:23.187353+0000","last_clean":"2024-09-02T10:20:23.187353+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:20:23.187353+0000","last_undegraded":"2024-09-02T10:20:23.187353+0000","last_fullsized":"2024-09-02T10:20:23.187353+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":6938,"log_dups_size":0,"ondisk_log_size":6938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:39.633585+0000","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368836608,"num_objects":5628,"num_object_clones":0,"num_object_copies":16884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6938,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":298,"reported_epoch":577,"state":"active+clean","last_fresh":"2024-09-02T10:20:22.464381+0000","last_change":"2024-09-02T10:19:02.956010+0000","last_active":"2024-09-02T10:20:22.464381+0000","last_peered":"2024-09-02T10:20:22.464381+0000","last_clean":"2024-09-02T10:20:22.464381+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:20:22.464381+0000","last_undegraded":"2024-09-02T10:20:22.464381+0000","last_fullsized":"2024-09-02T10:20:22.464381+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:02.955947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:02.955947+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:26.242539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5953093656,"num_objects":90838,"num_object_clones":0,"num_object_copies":272514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90838,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":109702,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8538,"num_bytes_recovered":559546368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17999540224,"data_stored":17999528008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109702,"ondisk_log_size":109702,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383923,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1558512,"kb_used_data":1094176,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92186640,"statfs":{"total":95995035648,"available":94399119360,"internally_reserved":0,"allocated":1120436224,"data_stored":1119960896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":475462084},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1410868,"kb_used_data":1077028,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92334284,"statfs":{"total":95995035648,"available":94550306816,"internally_reserved":0,"allocated":1102876672,"data_stored":1102397272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":341834176},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841789,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1932864,"kb_used_data":1432248,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":91812288,"statfs":{"total":95995035648,"available":94015782912,"internally_reserved":0,"allocated":1466621952,"data_stored":1466209546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":512424384},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070722,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1106228,"kb_used_data":786272,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92638924,"statfs":{"total":95995035648,"available":94862258176,"internally_reserved":0,"allocated":805142528,"data_stored":804667200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":327612869},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299655,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2363768,"kb_used_data":1885544,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":91381384,"statfs":{"total":95995035648,"available":93574537216,"internally_reserved":0,"allocated":1930797056,"data_stored":1930318176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":489683402},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528587,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1807856,"kb_used_data":1451808,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":91937296,"statfs":{"total":95995035648,"available":94143791104,"internally_reserved":0,"allocated":1486651392,"data_stored":1486176064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724816,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3442748,"kb_used_data":2879852,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":90302404,"statfs":{"total":95995035648,"available":92469661696,"internally_reserved":0,"allocated":2948968448,"data_stored":2948485496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953749,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1811684,"kb_used_data":1509780,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91933468,"statfs":{"total":95995035648,"available":94139871232,"internally_reserved":0,"allocated":1546014720,"data_stored":1545543795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149979,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3418240,"kb_used_data":2881528,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":90326912,"statfs":{"total":95995035648,"available":92494757888,"internally_reserved":0,"allocated":2950684672,"data_stored":2950272266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":549386696},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411614,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575139,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2051252,"kb_used_data":1591072,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91693900,"statfs":{"total":95995035648,"available":93894553600,"internally_reserved":0,"allocated":1629257728,"data_stored":1628779179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1641904,"kb_used_data":1091488,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":92103248,"statfs":{"total":95995035648,"available":94313725952,"internally_reserved":0,"allocated":1117683712,"data_stored":1117208384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1107361792,"data_stored":1107361792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1613631488,"data_stored":1613627416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2939158528,"data_stored":2939158528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1527119872,"data_stored":1527119872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2946371584,"data_stored":2946367512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1479540736,"data_stored":1479540736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1918369792,"data_stored":1918369792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":799670272,"data_stored":799670272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1460338688,"data_stored":1460338688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1093210112,"data_stored":1093206040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1114767360,"data_stored":1114767360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:20:25.816 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-02T10:20:25.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pg_num 61 2024-09-02T10:20:25.834 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:25 smithi162 ceph-mon[20279]: 6.1 scrub ok 2024-09-02T10:20:25.834 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:25 smithi162 ceph-mon[20279]: 6.3 scrub starts 2024-09-02T10:20:25.834 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:25 smithi162 ceph-mon[20279]: from='client.26740 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:25.834 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3639885981' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:25.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:25.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:20:25.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:20:25.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:25.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:25.987 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:26.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:26.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:26.012 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:26.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:26.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:26.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:26.219 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:26.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:26.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:26.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:26.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:26.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:26.389 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:26.390 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:26.491 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:26.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:26.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:26.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:26.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:26 smithi162 ceph-mon[20279]: pgmap v1340: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-09-02T10:20:26.632 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:26 smithi162 ceph-mon[20279]: 6.3 scrub ok 2024-09-02T10:20:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:26.645 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:26.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:26.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:26 smithi151 ceph-mon[16639]: pgmap v1340: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-09-02T10:20:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:26 smithi151 ceph-mon[16639]: 6.3 scrub ok 2024-09-02T10:20:26.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:26 smithi187 ceph-mon[20072]: pgmap v1340: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 917 op/s 2024-09-02T10:20:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:26 smithi187 ceph-mon[20072]: 6.3 scrub ok 2024-09-02T10:20:26.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:26.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:26.781 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:26.781 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:26.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:26.893 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:26.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:26.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:27.032 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:27.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:27.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:27.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:27.161 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:27.162 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:27.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:27.286 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:27.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:27.287 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:27.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:27.410 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:27.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:27.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:27.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:27.539 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:27.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:27.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:27.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:27 smithi187 ceph-mon[20072]: 6.d scrub starts 2024-09-02T10:20:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:27 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:20:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:27 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:20:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:27 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:20:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:27 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:20:27.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:27.790 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:27.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:27 smithi162 ceph-mon[20279]: 6.d scrub starts 2024-09-02T10:20:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:27 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:20:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:27 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:20:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:27 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:20:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:27 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:20:27.892 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:27.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:27.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:27.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:27 smithi151 ceph-mon[16639]: 6.d scrub starts 2024-09-02T10:20:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:27 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:20:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:27 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:20:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:27 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:20:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:27 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:20:28.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:28.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:28.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:28.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:28.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:28.167 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:28.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:28.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:28.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:28.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:28.421 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:28.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:28.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:28.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:28.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:28.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:28.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:28.674 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:28.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: pgmap v1341: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-09-02T10:20:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: 6.d scrub ok 2024-09-02T10:20:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:20:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:20:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=1333506646,config=2353167246,health=861809967,kv=4281324046}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=2537089532,logm=1299112001}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3422563265}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394561994}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235351720}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339135379}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1742893620}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3194280671}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976199122}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3675823093}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3589594566}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1900594196}) 2024-09-02T10:20:28.758 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2511487125}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2511757863}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1812285412}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=764930229}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1229281346}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069663686}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4142785937}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175002317}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3720584235}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825906324}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2393624600}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1350740152}) 2024-09-02T10:20:28.759 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4097647588}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4183953198}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383791129}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4164848907}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878929884}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3964847026}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374452880}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38750898}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782824332}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807011271}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=121055713}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28996956}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307011806}) 2024-09-02T10:20:28.760 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356291506}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4171280873}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2011838619}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039158286}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012341232}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=429329950}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986816243}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2910651837}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3561983478}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196167028}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110008513}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912963387}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238412201}) 2024-09-02T10:20:28.761 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426725054}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2761914502}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620775143}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180482029}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=675484366}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2751051210}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=79,mdsmap=3,mgr=18} crc {logm=1493671237,mdsmap=1083786272,mgr=3566298445}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=84} crc {mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=1447224602,mon_config_key=2243116194}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1073253487}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=638139881}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=2591077247,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=285419537,osdmap=326993014}) 2024-09-02T10:20:28.762 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=32} crc {osdmap=2173811700}) 2024-09-02T10:20:28.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:28.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:28.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: pgmap v1341: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-09-02T10:20:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: 6.d scrub ok 2024-09-02T10:20:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:20:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:20:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=1333506646,config=2353167246,health=861809967,kv=4281324046}) 2024-09-02T10:20:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=2537089532,logm=1299112001}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3422563265}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394561994}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235351720}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339135379}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1742893620}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3194280671}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976199122}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3675823093}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3589594566}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1900594196}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2511487125}) 2024-09-02T10:20:28.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2511757863}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1812285412}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=764930229}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1229281346}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069663686}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4142785937}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175002317}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3720584235}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825906324}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2393624600}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1350740152}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4097647588}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4183953198}) 2024-09-02T10:20:28.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383791129}) 2024-09-02T10:20:28.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4164848907}) 2024-09-02T10:20:28.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878929884}) 2024-09-02T10:20:28.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3964847026}) 2024-09-02T10:20:28.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374452880}) 2024-09-02T10:20:28.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38750898}) 2024-09-02T10:20:28.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782824332}) 2024-09-02T10:20:28.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807011271}) 2024-09-02T10:20:28.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=121055713}) 2024-09-02T10:20:28.848 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28996956}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307011806}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356291506}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4171280873}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2011838619}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039158286}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012341232}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=429329950}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986816243}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2910651837}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3561983478}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196167028}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110008513}) 2024-09-02T10:20:28.849 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912963387}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238412201}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426725054}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2761914502}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620775143}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180482029}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=675484366}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2751051210}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=79,mdsmap=3,mgr=18} crc {logm=1493671237,mdsmap=1083786272,mgr=3566298445}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=84} crc {mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=1447224602,mon_config_key=2243116194}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1073253487}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=638139881}) 2024-09-02T10:20:28.850 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=2591077247,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=285419537,osdmap=326993014}) 2024-09-02T10:20:28.851 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=32} crc {osdmap=2173811700}) 2024-09-02T10:20:28.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:28.926 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:28.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: pgmap v1341: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 879 op/s 2024-09-02T10:20:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: 6.d scrub ok 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=1333506646,config=2353167246,health=861809967,kv=4281324046}) 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {kv=8,logm=92} crc {kv=2537089532,logm=1299112001}) 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3422563265}) 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=394561994}) 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=235351720}) 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2339135379}) 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1742893620}) 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3194280671}) 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976199122}) 2024-09-02T10:20:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3675823093}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3589594566}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1900594196}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2511487125}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2511757863}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1812285412}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=764930229}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1229281346}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3069663686}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4142785937}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=175002317}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3720584235}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2825906324}) 2024-09-02T10:20:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2393624600}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1350740152}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4097647588}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4183953198}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383791129}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4164848907}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878929884}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3964847026}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374452880}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38750898}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782824332}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807011271}) 2024-09-02T10:20:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=121055713}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28996956}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307011806}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356291506}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4171280873}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2011838619}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039158286}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012341232}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=429329950}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986816243}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2910651837}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3561983478}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196167028}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110008513}) 2024-09-02T10:20:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912963387}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238412201}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426725054}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2761914502}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620775143}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180482029}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=675484366}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2751051210}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=79,mdsmap=3,mgr=18} crc {logm=1493671237,mdsmap=1083786272,mgr=3566298445}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=84} crc {mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=1447224602,mon_config_key=2243116194}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1073253487}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=638139881}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=2591077247,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=285419537,osdmap=326993014}) 2024-09-02T10:20:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=32} crc {osdmap=2173811700}) 2024-09-02T10:20:29.028 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:29.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:29.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:29.054 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:29.155 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:29.187 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:29.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:29.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:29.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:29.318 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:29.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:29.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:29.326 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:29.333 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:29.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:29.446 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:29.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:29.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:29.570 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:29.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:29.571 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:29.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:29.694 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:29.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:29.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:29.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:29.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:29.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:29.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:29.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:29.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:29.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:30.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:30.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:30.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:30.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:30.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:30.210 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:30.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:30.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:30.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:20:30.341 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:30.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:30.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:30.475 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:30.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:30.496 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:20:30.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:30 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:20:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:30 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:20:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:30 smithi162 ceph-mon[20279]: pgmap v1342: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 903 op/s 2024-09-02T10:20:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/88492335' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:20:30.602 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:30.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:30.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:30 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:20:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:30 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:20:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:30 smithi151 ceph-mon[16639]: pgmap v1342: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 903 op/s 2024-09-02T10:20:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/88492335' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:20:30.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:30.714 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pg_num 61 < pg_num_max 32 2024-09-02T10:20:30.741 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:30.749 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:30.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:30 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:20:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:30 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:20:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:30 smithi187 ceph-mon[20072]: pgmap v1342: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.2 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 903 op/s 2024-09-02T10:20:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/88492335' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:20:30.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:30.889 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:30.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:30.991 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:31.015 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:31.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:31.016 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:31.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:31.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:31.140 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:31.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:20:31.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:31.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:31.264 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:31.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:31.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:31.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:31.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:31.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:31.514 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:31.514 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:20:31.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:31.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:31.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:31.762 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:31.762 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:31.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:31.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:31.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:31 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/88492335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:20:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:31 smithi162 ceph-mon[20279]: osdmap e578: 12 total, 12 up, 11 in 2024-09-02T10:20:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3989096067' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-02T10:20:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:31 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:20:31.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:31.894 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:31.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:31 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/88492335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:20:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:31 smithi151 ceph-mon[16639]: osdmap e578: 12 total, 12 up, 11 in 2024-09-02T10:20:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3989096067' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-02T10:20:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:31 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:20:31.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:31 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/88492335' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:20:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:31 smithi187 ceph-mon[20072]: osdmap e578: 12 total, 12 up, 11 in 2024-09-02T10:20:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3989096067' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "61"} : dispatch 2024-09-02T10:20:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:31 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:20:32.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:32.026 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:32.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:32.043 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:32.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:32.153 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:32.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:32.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:32.289 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:32.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:32.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:32.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:32.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:32.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:32.566 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:32.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:32.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:20:32.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:32.700 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:32.700 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:32.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:32.731 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:32.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:32.732 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:32.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:32.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:32.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:32.790 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:32.790 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:32.790 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:32 smithi151 ceph-mon[16639]: pgmap v1344: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 881 op/s 2024-09-02T10:20:32.825 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:32.825 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:32.833 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:32.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:32 smithi162 ceph-mon[20279]: pgmap v1344: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 881 op/s 2024-09-02T10:20:32.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:32.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:32.858 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:32.869 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:32.869 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:32.906 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:32.907 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:32.942 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:32.942 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:32.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:32.970 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:32.971 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:32.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:32.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:32.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:33.002 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:33.002 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:32 smithi187 ceph-mon[20072]: pgmap v1344: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 881 op/s 2024-09-02T10:20:33.019 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:33.038 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:33.039 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:33.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:33.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:33.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:33.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:33.243 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:33.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:33.278 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:33.278 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:33.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:33.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:33.371 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:33.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:33.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:33.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:33.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:33.564 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:20:33.565 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:20:33.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:20:33.565 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:20:33.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.565 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3855 4.a 4:57b917f5:::benchmark_data_smithi187_76542_object3854:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:20:33.565 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:20:33.565 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 567.65130009899997, 2024-09-02T10:20:33.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22371366500000001, 2024-09-02T10:20:33.565 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3855 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:20:33.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909317+0000", 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909327+0000", 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9559999999999992e-06 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909360+0000", 2024-09-02T10:20:33.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3556999999999997e-05 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909377+0000", 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6283e-05 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909385+0000", 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4030000000000007e-06 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909444+0000", 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9203000000000002e-05 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909571+0000", 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127324 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.909715+0000", 2024-09-02T10:20:33.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143888 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.910145+0000", 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00042936599999999998 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.917291+0000", 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071467290000000001 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132953+0000", 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.215661517 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.132989+0000", 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6123e-05 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.133030+0000", 2024-09-02T10:20:33.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1316000000000002e-05 2024-09-02T10:20:33.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.576 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.576 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.576 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3871 4.a 4:530be978:::benchmark_data_smithi187_76542_object3870:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:20:33.576 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:20:33.576 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 567.64452429200003, 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21808112499999999, 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3871 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916093+0000", 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1899999999999998e-07 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916113+0000", 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9800999999999999e-05 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916118+0000", 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916121+0000", 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3330000000000001e-06 2024-09-02T10:20:33.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916168+0000", 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7147000000000001e-05 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916291+0000", 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122934 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:20:33.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916427+0000", 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135337 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.916951+0000", 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052393700000000004 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924167+0000", 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072167580000000002 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134113+0000", 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20994539000000001 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134147+0000", 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4149000000000001e-05 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134174+0000", 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6798999999999999e-05 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.582 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3887 4.a 4:5466a52d:::benchmark_data_smithi187_76542_object3886:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 567.63886600199999, 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21253008300000001, 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:33.583 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3887 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921751+0000", 2024-09-02T10:20:33.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921752+0000", 2024-09-02T10:20:33.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4160000000000001e-06 2024-09-02T10:20:33.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921813+0000", 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0420000000000001e-05 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921822+0000", 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4299999999999995e-06 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921827+0000", 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9819999999999999e-06 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921867+0000", 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0071000000000003e-05 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.921944+0000", 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6533999999999996e-05 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922025+0000", 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1601999999999996e-05 2024-09-02T10:20:33.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.922360+0000", 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00033462200000000002 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.924931+0000", 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025713160000000001 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134215+0000", 2024-09-02T10:20:33.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209284054 2024-09-02T10:20:33.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134247+0000", 2024-09-02T10:20:33.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2147999999999998e-05 2024-09-02T10:20:33.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134281+0000", 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3488000000000002e-05 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3898 4.a 4:55cc6e50:::benchmark_data_smithi187_76542_object3897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 567.63387006999994, 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20761331199999999, 2024-09-02T10:20:33.590 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3898 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926747+0000", 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926748+0000", 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6599999999999994e-07 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926786+0000", 2024-09-02T10:20:33.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8506e-05 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926798+0000", 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.221e-05 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926804+0000", 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3140000000000003e-06 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926853+0000", 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8804999999999998e-05 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.926965+0000", 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000112256 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927110+0000", 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014520099999999999 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:33.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.927488+0000", 2024-09-02T10:20:33.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000378332 2024-09-02T10:20:33.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929483+0000", 2024-09-02T10:20:33.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0019947670000000002 2024-09-02T10:20:33.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134309+0000", 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20482557300000001 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134337+0000", 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8334999999999999e-05 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134360+0000", 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3047000000000001e-05 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:3901 4.a 4:53342710:::benchmark_data_smithi187_76542_object3900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 567.63204197799996, 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20590491499999999, 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.597 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3901 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928575+0000", 2024-09-02T10:20:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928576+0000", 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1699999999999997e-07 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928631+0000", 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5276999999999999e-05 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928642+0000", 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1195999999999999e-05 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928649+0000", 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.669e-06 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928691+0000", 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1868e-05 2024-09-02T10:20:33.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928778+0000", 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7416999999999993e-05 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.928866+0000", 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8471e-05 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.929233+0000", 2024-09-02T10:20:33.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036694499999999998 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:05.942995+0000", 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013761572999999999 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134423+0000", 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19142793799999999 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134461+0000", 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7839000000000001e-05 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:06.134480+0000", 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9004999999999998e-05 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:6012 4.5 4:a5fa00f8:::benchmark_data_smithi187_76542_object6011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-09-02T10:20:33.604 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 566.29760897599999, 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209785581, 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6012 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263008+0000", 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263009+0000", 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0139999999999999e-06 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263120+0000", 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011099000000000001 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263134+0000", 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4467999999999999e-05 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263141+0000", 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9820000000000002e-06 2024-09-02T10:20:33.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263387+0000", 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024598300000000002 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263500+0000", 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113174 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:20:33.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.263599+0000", 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8974999999999993e-05 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264127+0000", 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00052785799999999997 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.264358+0000", 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00023054600000000001 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472691+0000", 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20833321599999999 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472750+0000", 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8897000000000002e-05 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:07.472793+0000", 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3477999999999999e-05 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.610 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:24533 4.c e335/333 4:3a0221c2:::benchmark_data_smithi187_76542_object24532:head v 335'1521, mlcod=335'1517)", 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 548.54957592400001, 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.212837002, 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011041+0000", 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011042+0000", 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1650000000000001e-06 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.011129+0000", 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7443999999999998e-05 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222953+0000", 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21182389400000001 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.222960+0000", 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8569999999999997e-06 2024-09-02T10:20:33.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223010+0000", 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0182000000000002e-05 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223032+0000", 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1699999999999999e-05 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223838+0000", 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080572200000000004 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.223878+0000", 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0037999999999999e-05 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:25165 4.c e336/333 4:3c3f6316:::benchmark_data_smithi187_76542_object25164:head v 336'1563, mlcod=335'1560)", 2024-09-02T10:20:33.615 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 547.91534637400002, 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204647308, 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645270+0000", 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645271+0000", 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9999999999999996e-07 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.645443+0000", 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017179700000000001 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848893+0000", 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203449884 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848900+0000", 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8360000000000003e-06 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848961+0000", 2024-09-02T10:20:33.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1163999999999999e-05 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.848982+0000", 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1403000000000002e-05 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849888+0000", 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000905469 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:25.849918+0000", 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9855000000000001e-05 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:52446 4.15 e362/353 4:acabd05d:::benchmark_data_smithi187_76542_object52445:head v 362'2522, mlcod=362'2519)", 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 514.86721701399995, 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.220391959, 2024-09-02T10:20:33.620 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693400+0000", 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9599999999999999e-07 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.693457+0000", 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6582000000000001e-05 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912812+0000", 2024-09-02T10:20:33.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21935476500000001 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912819+0000", 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4989999999999997e-06 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912871+0000", 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1388000000000003e-05 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.912887+0000", 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5911999999999999e-05 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.913759+0000", 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00087215899999999996 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:11:58.913792+0000", 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2957999999999999e-05 2024-09-02T10:20:33.624 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:58518 4.1 e366/341 4:8490730e:::benchmark_data_smithi187_76542_object58517:head v 366'2807, mlcod=366'2804)", 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 509.19548878699999, 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209866793, 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.625 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365128+0000", 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365129+0000", 2024-09-02T10:20:33.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7100000000000001e-07 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.365181+0000", 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1816e-05 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573805+0000", 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.2086249 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573812+0000", 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1349999999999997e-06 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573866+0000", 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4373000000000003e-05 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.573888+0000", 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2008999999999999e-05 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.574955+0000", 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010673519999999999 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:04.574995+0000", 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9536999999999998e-05 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:75251 4.11 e371/356 4:8ba31329:::benchmark_data_smithi187_76542_object75250:head v 371'3396, mlcod=371'3394)", 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 495.895159767, 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37157874099999999, 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:20:33.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665457+0000", 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665458+0000", 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9400000000000004e-07 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.665521+0000", 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2685e-05 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035523+0000", 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37000295799999999 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035542+0000", 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8692999999999999e-05 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035709+0000", 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016700999999999999 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.035745+0000", 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5913e-05 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.037004+0000", 2024-09-02T10:20:33.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001258459 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.037036+0000", 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2228999999999999e-05 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:90191 4.15 e375/353 4:afce227e:::benchmark_data_smithi187_76542_object90190:head v 375'3680, mlcod=375'3677)", 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:20:33.634 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 482.46237022100001, 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20444974699999999, 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9299999999999997e-07 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098273+0000", 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5378999999999999e-05 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301201+0000", 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20292787800000001 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301213+0000", 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860000000000001e-05 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301360+0000", 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014643600000000001 2024-09-02T10:20:33.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301383+0000", 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2782999999999999e-05 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302662+0000", 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012796470000000001 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302696+0000", 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3970999999999997e-05 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126679 4.1 e392/341 4:86205e4c:::benchmark_data_smithi187_76542_object34343:head v 392'4951, mlcod=392'4948)", 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 438.09280995, 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44507702900000001, 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.639 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467808+0000", 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.277e-06 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467816+0000", 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3690000000000007e-06 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910880+0000", 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44306368899999998 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910889+0000", 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0380000000000005e-06 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910948+0000", 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8498000000000001e-05 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910970+0000", 2024-09-02T10:20:33.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2580000000000001e-05 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912839+0000", 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018688120000000001 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912884+0000", 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4765999999999998e-05 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126802 4.1 e392/341 4:81a050df:::benchmark_data_smithi187_76542_object34466:head v 392'4952, mlcod=392'4949)", 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 437.64976657400001, 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25345677799999999, 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910851+0000", 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0690000000000001e-06 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910859+0000", 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.233e-06 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162387+0000", 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25152825099999998 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162402+0000", 2024-09-02T10:20:33.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4905999999999999e-05 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162538+0000", 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013591800000000001 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162581+0000", 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2991000000000002e-05 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164270+0000", 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016893419999999999 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164307+0000", 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7067999999999998e-05 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.648 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:64011 5.9 e465/431 5:9035e690:::benchmark_data_smithi187_89405_object64010:head v 465'4071, mlcod=464'4068)", 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 296.34334789799999, 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.200897045, 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1100000000000002e-07 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217293+0000", 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3326999999999999e-05 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416283+0000", 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19899007799999999 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416287+0000", 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3529999999999997e-06 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416337+0000", 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0272000000000003e-05 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416360+0000", 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3147e-05 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.418133+0000", 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001772605 2024-09-02T10:20:33.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.418166+0000", 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2951999999999999e-05 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:86619 5.9 e473/431 5:9bb5eff1:::benchmark_data_smithi187_89405_object86618:head v 473'5442, mlcod=473'5439)", 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 275.02087626600002, 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21915551599999999, 2024-09-02T10:20:33.653 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:20:33.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.09e-07 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539774+0000", 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2465999999999997e-05 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.755999+0000", 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.216224731 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756004+0000", 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9009999999999999e-06 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756086+0000", 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1767000000000005e-05 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756100+0000", 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4285e-05 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.758849+0000", 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027488349999999998 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.758896+0000", 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6721999999999998e-05 2024-09-02T10:20:33.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:33.659 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162692 5.2 5:4dd653e2:::benchmark_data_smithi187_89405_object59087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:20:33.659 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:20:33.659 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.207452096, 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25261044100000002, 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162692 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353166+0000", 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8200000000000008e-07 2024-09-02T10:20:33.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353170+0000", 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3880000000000002e-06 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353180+0000", 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0336999999999999e-05 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353188+0000", 2024-09-02T10:20:33.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7060000000000004e-06 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353233+0000", 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4453000000000003e-05 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353306+0000", 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3559999999999994e-05 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353375+0000", 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9202999999999995e-05 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360656+0000", 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072804360000000004 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360676+0000", 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0075e-05 2024-09-02T10:20:33.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605656+0000", 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24498062500000001 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605693+0000", 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6377000000000001e-05 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605775+0000", 2024-09-02T10:20:33.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2299000000000006e-05 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162694 5.2 5:4549dcce:::benchmark_data_smithi187_89405_object59089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.20685590900001, 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25215807000000001, 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.666 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162694 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353762+0000", 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3600000000000003e-07 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353765+0000", 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6559999999999998e-06 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353776+0000", 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0243e-05 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353781+0000", 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0540000000000002e-06 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.356988+0000", 2024-09-02T10:20:33.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032069899999999998 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.357116+0000", 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012815 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.357235+0000", 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011919 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360716+0000", 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003481254 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.409925+0000", 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049208533999999998 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605820+0000", 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19589521600000001 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605878+0000", 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8149000000000002e-05 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605919+0000", 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0898e-05 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:43141 6.c e554/533 6:30fc4bbf:::benchmark_data_smithi187_102448_object43140:head v 554'2733, mlcod=554'2729)", 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:14.433920+0000", 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 79.126696695000007, 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223318874, 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.433920+0000", 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.433920+0000", 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.433921+0000", 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7899999999999997e-07 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.433978+0000", 2024-09-02T10:20:33.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7401999999999997e-05 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.655660+0000", 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.221681669 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.655668+0000", 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6410000000000002e-06 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.655723+0000", 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5186000000000003e-05 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.655737+0000", 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4226000000000001e-05 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.657205+0000", 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014677030000000001 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.657239+0000", 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4168000000000002e-05 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:63922 6.a e560/533 6:57bf15bd:::benchmark_data_smithi187_102448_object63921:head v 560'3998, mlcod=560'3994)", 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:36.274019+0000", 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 57.286597649000001, 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18662683599999999, 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:33.677 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:36.274019+0000", 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:36.274019+0000", 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:33.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:36.274020+0000", 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4399999999999999e-07 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:36.274043+0000", 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3008999999999999e-05 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:36.458779+0000", 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18473577699999999 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:36.458787+0000", 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.225e-06 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:36.458841+0000", 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3594999999999998e-05 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:36.458857+0000", 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6898e-05 2024-09-02T10:20:33.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:36.460610+0000", 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001752205 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:36.460646+0000", 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6383e-05 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:33.681 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:33.682 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:20:33.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:33.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:33.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:33.810 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:33.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:33 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:20:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:33 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2419247902' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:20:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:20:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:20:33.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:33.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:33.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:33.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:33 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:20:33.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:33 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:33.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2419247902' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:33.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:20:33.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:20:33.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:20:33.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:20:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:33 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:20:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:33 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2419247902' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:20:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:20:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:20:34.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:34.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:34.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:34.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:34.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:34.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:34.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:34.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:34.318 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:34.319 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:34.325 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:34.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:34.443 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:34.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:34.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:20:34.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:34.574 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:34.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:34.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:34.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:34.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:34.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:34.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:34 smithi151 ceph-mon[16639]: pgmap v1345: 47 pgs: 47 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 902 op/s 2024-09-02T10:20:34.756 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:34 smithi151 ceph-mon[16639]: osdmap e579: 12 total, 12 up, 11 in 2024-09-02T10:20:34.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:34.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:34.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:34.825 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:34.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:34 smithi162 ceph-mon[20279]: pgmap v1345: 47 pgs: 47 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 902 op/s 2024-09-02T10:20:34.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:34 smithi162 ceph-mon[20279]: osdmap e579: 12 total, 12 up, 11 in 2024-09-02T10:20:34.848 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:34.926 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:34.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:34.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:34 smithi187 ceph-mon[20072]: pgmap v1345: 47 pgs: 47 active+clean; 5.0 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 902 op/s 2024-09-02T10:20:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:34 smithi187 ceph-mon[20072]: osdmap e579: 12 total, 12 up, 11 in 2024-09-02T10:20:35.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:35.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:35.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:35.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:35.214 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:35.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:35.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:35.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:35.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:35.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:35.415 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:122074 6.3 6:c1d32375:::benchmark_data_smithi187_102448_object21802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e579)", 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:35.405571+0000", 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0076992470000000002, 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077273740000000004, 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:35.416 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 122074 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.405571+0000", 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.405571+0000", 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:35.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.405572+0000", 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860000000000001e-06 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.405578+0000", 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0569999999999998e-06 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:35.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.405592+0000", 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4039000000000001e-05 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.405600+0000", 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8730000000000007e-06 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.406003+0000", 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040325199999999999 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.406131+0000", 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012756900000000001 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.406257+0000", 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000125821 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410260+0000", 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040029699999999998 2024-09-02T10:20:35.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410431+0000", 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017162899999999999 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:122075 6.3 6:ce2b2f22:::benchmark_data_smithi187_102448_object21803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e579)", 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:35.406764+0000", 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.006506464, 2024-09-02T10:20:35.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0066102080000000002, 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 122075 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.406764+0000", 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:35.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.406764+0000", 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.406765+0000", 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2200000000000002e-07 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.406769+0000", 2024-09-02T10:20:35.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4290000000000003e-06 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.406823+0000", 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3743999999999999e-05 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.406832+0000", 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2539999999999998e-06 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:35.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410471+0000", 2024-09-02T10:20:35.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036394119999999999 2024-09-02T10:20:35.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:35.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410606+0000", 2024-09-02T10:20:35.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013489900000000001 2024-09-02T10:20:35.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410717+0000", 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110325 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:122077 6.b e579/533 6:d98bb638:::benchmark_data_smithi187_102448_object21805:head v 579'7632, mlcod=579'7630)", 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:35.407916+0000", 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0053544769999999998, 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0055176490000000003, 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:35.426 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:20:35.427 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:35.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:35.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.407916+0000", 2024-09-02T10:20:35.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:35.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.407916+0000", 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.407916+0000", 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8699999999999995e-07 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.407921+0000", 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2750000000000003e-06 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.407947+0000", 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5296e-05 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.407953+0000", 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0519999999999998e-06 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410293+0000", 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023405119999999999 2024-09-02T10:20:35.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410311+0000", 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7645000000000001e-05 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:122079 6.5 6:aa70816b:::benchmark_data_smithi187_102448_object21807:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e579)", 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:35.410330+0000", 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0029404219999999998, 2024-09-02T10:20:35.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031518079999999999, 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 122079 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410330+0000", 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:35.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410330+0000", 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410331+0000", 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0580000000000001e-06 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410335+0000", 2024-09-02T10:20:35.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9339999999999999e-06 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410345+0000", 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0134e-05 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410352+0000", 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4760000000000001e-06 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410411+0000", 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8443999999999998e-05 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410548+0000", 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013696100000000001 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.410664+0000", 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116785 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:35.434 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:122060 6.5 6:a682a3b1:::benchmark_data_smithi187_102448_object21788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e579)", 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:35.389739+0000", 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.023530714000000001, 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023801065, 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 122060 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.435 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.389739+0000", 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.389739+0000", 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.389741+0000", 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.781e-06 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.389748+0000", 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.404e-06 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.389775+0000", 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7696e-05 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.389782+0000", 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8900000000000001e-06 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.389862+0000", 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9509000000000001e-05 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.389992+0000", 2024-09-02T10:20:35.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013033700000000001 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.390103+0000", 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110518 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.393326+0000", 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032235419999999998 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:20:35.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:35.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:35.397224+0000", 2024-09-02T10:20:35.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00389829 2024-09-02T10:20:35.440 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.440 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:20:35.440 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.440 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:20:35.440 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:20:35.440 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 5 2024-09-02T10:20:35.440 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:20:35.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:35.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:35.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:35.537 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:20:35.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:35.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:35.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:35.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:35.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:35.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:35.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:35.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2825898740' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:35.859 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2825898740' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:35.861 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:35.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:35.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:35.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:35.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2825898740' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:36.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:36.127 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:36.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:36.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:20:36.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:20:36.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:36.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:36.277 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:36.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:36.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:36.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:36.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:36.467 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:36.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:36.558 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:36.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:36.559 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:20:36.559 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:20:36.559 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-02T10:20:36.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:20:36.603 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:36.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:36.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:36.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:36.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:36.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:36 smithi151 ceph-mon[16639]: pgmap v1347: 47 pgs: 47 active+clean; 4.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 892 op/s 2024-09-02T10:20:36.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:36 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:36.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2825898740' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:36.785 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:36 smithi151 ceph-mon[16639]: osdmap e580: 12 total, 12 up, 11 in 2024-09-02T10:20:36.805 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:36.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:36.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:36 smithi162 ceph-mon[20279]: pgmap v1347: 47 pgs: 47 active+clean; 4.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 892 op/s 2024-09-02T10:20:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:36 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2825898740' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:36 smithi162 ceph-mon[20279]: osdmap e580: 12 total, 12 up, 11 in 2024-09-02T10:20:36.907 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:36.928 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:36.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:36.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:36 smithi187 ceph-mon[20072]: pgmap v1347: 47 pgs: 47 active+clean; 4.9 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 892 op/s 2024-09-02T10:20:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:36 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2825898740' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:36 smithi187 ceph-mon[20072]: osdmap e580: 12 total, 12 up, 11 in 2024-09-02T10:20:37.030 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:37.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:37.053 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:37.070 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:37.070 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:37.073 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:37.096 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:37.096 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:37.122 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:37.122 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:37.145 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:37.145 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:37.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:37.164 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:20:37.164 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:20:37.164 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:20:37.164 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:20:37.164 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:20:37.170 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:37.170 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:37.182 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:37.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:37.194 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:37.194 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:37.218 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:37.218 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:37.242 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:37.243 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:37.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:37.271 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:37.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:37.293 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:37.294 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:37.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:37.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:37.321 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:37.321 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:37.349 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:37.349 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:37.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:37.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:37.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:37.537 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:37.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:37.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:37.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:37.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:37.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:37.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:37.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/434804941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:37.788 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:37.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:37.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:37.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/434804941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:37.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:37.952 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:37.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:38.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/434804941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:38.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:38.084 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:38.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:38.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:38.115 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:20:38.116 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:20:38.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:38.215 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:38.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:38.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:38.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:38.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:38.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:38.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:20:38.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:38.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:38.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:38.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:38.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:20:38.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:38.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:38.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:38.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:38 smithi151 ceph-mon[16639]: pgmap v1349: 47 pgs: 47 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 946 op/s 2024-09-02T10:20:38.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:38.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:38.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:38.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:38 smithi162 ceph-mon[20279]: pgmap v1349: 47 pgs: 47 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 946 op/s 2024-09-02T10:20:38.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:38.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:38.885 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:20:38.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:38.978 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:38.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:38 smithi187 ceph-mon[20072]: pgmap v1349: 47 pgs: 47 active+clean; 4.8 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 946 op/s 2024-09-02T10:20:39.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:39.104 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1349,"stamp":"2024-09-02T10:20:37.359372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5168562744,"num_objects":78860,"num_object_clones":0,"num_object_copies":236580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78860,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121682,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8541,"num_bytes_recovered":560276544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124534,"ondisk_log_size":124534,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":20241480,"kb_used_data":15285816,"kb_used_omap":17,"kb_used_meta":4954734,"kb_avail":1010955192,"statfs":{"total":1055945392128,"available":1035218116608,"internally_reserved":0,"allocated":15652675584,"data_stored":15649290557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17546,"internal_metadata":5073648502},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-425918464,"num_objects":-6499,"num_object_clones":0,"num_object_copies":-19497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6499,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.862889"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":513,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.549377+0000","last_change":"2024-09-02T10:20:22.673265+0000","last_active":"2024-09-02T10:20:35.549377+0000","last_peered":"2024-09-02T10:20:35.549377+0000","last_clean":"2024-09-02T10:20:35.549377+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:20:35.549377+0000","last_undegraded":"2024-09-02T10:20:35.549377+0000","last_fullsized":"2024-09-02T10:20:35.549377+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:22.673202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:22.673202+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:50.511460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":319,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.519678+0000","last_change":"2024-09-02T10:19:47.121714+0000","last_active":"2024-09-02T10:20:33.519678+0000","last_peered":"2024-09-02T10:20:33.519678+0000","last_clean":"2024-09-02T10:20:33.519678+0000","last_became_active":"2024-09-02T10:15:35.207658+0000","last_became_peered":"2024-09-02T10:15:35.207658+0000","last_unstale":"2024-09-02T10:20:33.519678+0000","last_undegraded":"2024-09-02T10:20:33.519678+0000","last_fullsized":"2024-09-02T10:20:33.519678+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:04.209863+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,5],"acting":[1,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":768,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.517175+0000","last_change":"2024-09-02T10:19:48.146767+0000","last_active":"2024-09-02T10:20:33.517175+0000","last_peered":"2024-09-02T10:20:33.517175+0000","last_clean":"2024-09-02T10:20:33.517175+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:20:33.517175+0000","last_undegraded":"2024-09-02T10:20:33.517175+0000","last_fullsized":"2024-09-02T10:20:33.517175+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:48.146721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:48.146721+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:55.994378+0000","scrub_duration":995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":768,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.517077+0000","last_change":"2024-09-02T10:20:23.955274+0000","last_active":"2024-09-02T10:20:33.517077+0000","last_peered":"2024-09-02T10:20:33.517077+0000","last_clean":"2024-09-02T10:20:33.517077+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:20:33.517077+0000","last_undegraded":"2024-09-02T10:20:33.517077+0000","last_fullsized":"2024-09-02T10:20:33.517077+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.955207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.955207+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:44.626949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":80,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.525019+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:20:33.525019+0000","last_peered":"2024-09-02T10:20:33.525019+0000","last_clean":"2024-09-02T10:20:33.525019+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:20:33.525019+0000","last_undegraded":"2024-09-02T10:20:33.525019+0000","last_fullsized":"2024-09-02T10:20:33.525019+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:46.185150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":619,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.540773+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:20:35.540773+0000","last_peered":"2024-09-02T10:20:35.540773+0000","last_clean":"2024-09-02T10:20:35.540773+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:20:35.540773+0000","last_undegraded":"2024-09-02T10:20:35.540773+0000","last_fullsized":"2024-09-02T10:20:35.540773+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:08.487285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1143,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.538817+0000","last_change":"2024-09-02T10:20:29.189672+0000","last_active":"2024-09-02T10:20:35.538817+0000","last_peered":"2024-09-02T10:20:35.538817+0000","last_clean":"2024-09-02T10:20:35.538817+0000","last_became_active":"2024-09-02T10:15:07.056232+0000","last_became_peered":"2024-09-02T10:15:07.056232+0000","last_unstale":"2024-09-02T10:20:35.538817+0000","last_undegraded":"2024-09-02T10:20:35.538817+0000","last_fullsized":"2024-09-02T10:20:35.538817+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:29.189593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:29.189593+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:29.322170+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":402496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":783,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.517141+0000","last_change":"2024-09-02T10:20:00.175101+0000","last_active":"2024-09-02T10:20:33.517141+0000","last_peered":"2024-09-02T10:20:33.517141+0000","last_clean":"2024-09-02T10:20:33.517141+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:20:33.517141+0000","last_undegraded":"2024-09-02T10:20:33.517141+0000","last_fullsized":"2024-09-02T10:20:33.517141+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.175032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.175032+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:28.457490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":783,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.519759+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:20:33.519759+0000","last_peered":"2024-09-02T10:20:33.519759+0000","last_clean":"2024-09-02T10:20:33.519759+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:20:33.519759+0000","last_undegraded":"2024-09-02T10:20:33.519759+0000","last_fullsized":"2024-09-02T10:20:33.519759+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:52.877235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":753,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.542698+0000","last_change":"2024-09-02T10:19:05.910750+0000","last_active":"2024-09-02T10:20:35.542698+0000","last_peered":"2024-09-02T10:20:35.542698+0000","last_clean":"2024-09-02T10:20:35.542698+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:20:35.542698+0000","last_undegraded":"2024-09-02T10:20:35.542698+0000","last_fullsized":"2024-09-02T10:20:35.542698+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:05.910691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:05.910691+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:35.938716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":605,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.542722+0000","last_change":"2024-09-02T10:19:52.963372+0000","last_active":"2024-09-02T10:20:35.542722+0000","last_peered":"2024-09-02T10:20:35.542722+0000","last_clean":"2024-09-02T10:20:35.542722+0000","last_became_active":"2024-09-02T10:12:47.091327+0000","last_became_peered":"2024-09-02T10:12:47.091327+0000","last_unstale":"2024-09-02T10:20:35.542722+0000","last_undegraded":"2024-09-02T10:20:35.542722+0000","last_fullsized":"2024-09-02T10:20:35.542722+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.963289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.963289+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:10.821299+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,1],"acting":[7,9,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1108,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.519639+0000","last_change":"2024-09-02T10:19:00.052912+0000","last_active":"2024-09-02T10:20:33.519639+0000","last_peered":"2024-09-02T10:20:33.519639+0000","last_clean":"2024-09-02T10:20:33.519639+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:20:33.519639+0000","last_undegraded":"2024-09-02T10:20:33.519639+0000","last_fullsized":"2024-09-02T10:20:33.519639+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:00.052857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:00.052857+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:37.984640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1053,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.540453+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:20:35.540453+0000","last_peered":"2024-09-02T10:20:35.540453+0000","last_clean":"2024-09-02T10:20:35.540453+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:20:35.540453+0000","last_undegraded":"2024-09-02T10:20:35.540453+0000","last_fullsized":"2024-09-02T10:20:35.540453+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:57.041883+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1028,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-09-02T10:20:30.500306+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:20:30.500306+0000","last_peered":"2024-09-02T10:20:30.500306+0000","last_clean":"2024-09-02T10:20:30.500306+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:20:30.500306+0000","last_undegraded":"2024-09-02T10:20:30.500306+0000","last_fullsized":"2024-09-02T10:20:30.500306+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:01.846422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1023,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.538919+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:20:35.538919+0000","last_peered":"2024-09-02T10:20:35.538919+0000","last_clean":"2024-09-02T10:20:35.538919+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:20:35.538919+0000","last_undegraded":"2024-09-02T10:20:35.538919+0000","last_fullsized":"2024-09-02T10:20:35.538919+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.850900+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"580'7583","reported_seq":7748,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:36.596339+0000","last_change":"2024-09-02T10:19:59.986749+0000","last_active":"2024-09-02T10:20:36.596339+0000","last_peered":"2024-09-02T10:20:36.596339+0000","last_clean":"2024-09-02T10:20:36.596339+0000","last_became_active":"2024-09-02T10:18:37.599064+0000","last_became_peered":"2024-09-02T10:18:37.599064+0000","last_unstale":"2024-09-02T10:20:36.596339+0000","last_undegraded":"2024-09-02T10:20:36.596339+0000","last_fullsized":"2024-09-02T10:20:36.596339+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":7583,"log_dups_size":0,"ondisk_log_size":7583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:01.232192+0000","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315293696,"num_objects":4811,"num_object_clones":0,"num_object_copies":14433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7583,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,5],"acting":[10,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1110,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.539001+0000","last_change":"2024-09-02T10:19:52.109223+0000","last_active":"2024-09-02T10:20:35.539001+0000","last_peered":"2024-09-02T10:20:35.539001+0000","last_clean":"2024-09-02T10:20:35.539001+0000","last_became_active":"2024-09-02T10:12:47.092367+0000","last_became_peered":"2024-09-02T10:12:47.092367+0000","last_unstale":"2024-09-02T10:20:35.539001+0000","last_undegraded":"2024-09-02T10:20:35.539001+0000","last_fullsized":"2024-09-02T10:20:35.539001+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:04.388339+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,8],"acting":[5,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"579'7572","reported_seq":11968,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:34.208592+0000","last_change":"2024-09-02T10:19:55.677256+0000","last_active":"2024-09-02T10:20:34.208592+0000","last_peered":"2024-09-02T10:20:34.208592+0000","last_clean":"2024-09-02T10:20:34.208592+0000","last_became_active":"2024-09-02T10:19:32.596751+0000","last_became_peered":"2024-09-02T10:19:32.596751+0000","last_unstale":"2024-09-02T10:20:34.208592+0000","last_undegraded":"2024-09-02T10:20:34.208592+0000","last_fullsized":"2024-09-02T10:20:34.208592+0000","mapping_epoch":558,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":7572,"log_dups_size":0,"ondisk_log_size":7572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:09.732905+0000","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328204288,"num_objects":5008,"num_object_clones":0,"num_object_copies":15024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7572,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2111,"num_bytes_recovered":138346496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1225,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.542475+0000","last_change":"2024-09-02T10:19:49.817426+0000","last_active":"2024-09-02T10:20:35.542475+0000","last_peered":"2024-09-02T10:20:35.542475+0000","last_clean":"2024-09-02T10:20:35.542475+0000","last_became_active":"2024-09-02T10:12:47.091305+0000","last_became_peered":"2024-09-02T10:12:47.091305+0000","last_unstale":"2024-09-02T10:20:35.542475+0000","last_undegraded":"2024-09-02T10:20:35.542475+0000","last_fullsized":"2024-09-02T10:20:35.542475+0000","mapping_epoch":384,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":385,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.817347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.817347+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:54.901580+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"580'7717","reported_seq":7883,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:37.198823+0000","last_change":"2024-09-02T10:20:27.715367+0000","last_active":"2024-09-02T10:20:37.198823+0000","last_peered":"2024-09-02T10:20:37.198823+0000","last_clean":"2024-09-02T10:20:37.198823+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:20:37.198823+0000","last_undegraded":"2024-09-02T10:20:37.198823+0000","last_fullsized":"2024-09-02T10:20:37.198823+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7181","last_scrub_stamp":"2024-09-02T10:20:27.715126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:27.715126+0000","objects_scrubbed":5211,"log_size":7717,"log_dups_size":0,"ondisk_log_size":7717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:27.902373+0000","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303693848,"num_objects":4635,"num_object_clones":0,"num_object_copies":13905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4635,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7717,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":879,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-09-02T10:20:30.495665+0000","last_change":"2024-09-02T10:20:18.812280+0000","last_active":"2024-09-02T10:20:30.495665+0000","last_peered":"2024-09-02T10:20:30.495665+0000","last_clean":"2024-09-02T10:20:30.495665+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:20:30.495665+0000","last_undegraded":"2024-09-02T10:20:30.495665+0000","last_fullsized":"2024-09-02T10:20:30.495665+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.812212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.812212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:47.719780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"578'7472","reported_seq":7647,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.199148+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:20:33.199148+0000","last_peered":"2024-09-02T10:20:33.199148+0000","last_clean":"2024-09-02T10:20:33.199148+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:20:33.199148+0000","last_undegraded":"2024-09-02T10:20:33.199148+0000","last_fullsized":"2024-09-02T10:20:33.199148+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":7472,"log_dups_size":0,"ondisk_log_size":7472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:37.995088+0000","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321781760,"num_objects":4910,"num_object_clones":0,"num_object_copies":14730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7472,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":272,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.549264+0000","last_change":"2024-09-02T10:19:50.392195+0000","last_active":"2024-09-02T10:20:35.549264+0000","last_peered":"2024-09-02T10:20:35.549264+0000","last_clean":"2024-09-02T10:20:35.549264+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:20:35.549264+0000","last_undegraded":"2024-09-02T10:20:35.549264+0000","last_fullsized":"2024-09-02T10:20:35.549264+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.392146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.392146+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.406921+0000","scrub_duration":560,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"580'7747","reported_seq":7912,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:36.282386+0000","last_change":"2024-09-02T10:20:25.985232+0000","last_active":"2024-09-02T10:20:36.282386+0000","last_peered":"2024-09-02T10:20:36.282386+0000","last_clean":"2024-09-02T10:20:36.282386+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:20:36.282386+0000","last_undegraded":"2024-09-02T10:20:36.282386+0000","last_fullsized":"2024-09-02T10:20:36.282386+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7149","last_scrub_stamp":"2024-09-02T10:20:25.984961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:25.984961+0000","objects_scrubbed":5493,"log_size":7747,"log_dups_size":0,"ondisk_log_size":7747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:35.881813+0000","scrub_duration":1180,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318439424,"num_objects":4859,"num_object_clones":0,"num_object_copies":14577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7747,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1181,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.549322+0000","last_change":"2024-09-02T10:19:51.832888+0000","last_active":"2024-09-02T10:20:35.549322+0000","last_peered":"2024-09-02T10:20:35.549322+0000","last_clean":"2024-09-02T10:20:35.549322+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:20:35.549322+0000","last_undegraded":"2024-09-02T10:20:35.549322+0000","last_fullsized":"2024-09-02T10:20:35.549322+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.832822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.832822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.770876+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"579'7611","reported_seq":12055,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:34.159423+0000","last_change":"2024-09-02T10:19:54.548123+0000","last_active":"2024-09-02T10:20:34.159423+0000","last_peered":"2024-09-02T10:20:34.159423+0000","last_clean":"2024-09-02T10:20:34.159423+0000","last_became_active":"2024-09-02T10:19:31.588084+0000","last_became_peered":"2024-09-02T10:19:31.588084+0000","last_unstale":"2024-09-02T10:20:34.159423+0000","last_undegraded":"2024-09-02T10:20:34.159423+0000","last_fullsized":"2024-09-02T10:20:34.159423+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":7611,"log_dups_size":0,"ondisk_log_size":7611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:12.643132+0000","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323289088,"num_objects":4933,"num_object_clones":0,"num_object_copies":14799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7611,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2135,"num_bytes_recovered":139919360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,11],"acting":[4,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":648,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.540897+0000","last_change":"2024-09-02T10:20:00.749882+0000","last_active":"2024-09-02T10:20:35.540897+0000","last_peered":"2024-09-02T10:20:35.540897+0000","last_clean":"2024-09-02T10:20:35.540897+0000","last_became_active":"2024-09-02T10:15:35.205440+0000","last_became_peered":"2024-09-02T10:15:35.205440+0000","last_unstale":"2024-09-02T10:20:35.540897+0000","last_undegraded":"2024-09-02T10:20:35.540897+0000","last_fullsized":"2024-09-02T10:20:35.540897+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.876682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,11],"acting":[10,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"579'7442","reported_seq":7608,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:34.364800+0000","last_change":"2024-09-02T10:20:24.623694+0000","last_active":"2024-09-02T10:20:34.364800+0000","last_peered":"2024-09-02T10:20:34.364800+0000","last_clean":"2024-09-02T10:20:34.364800+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:20:34.364800+0000","last_undegraded":"2024-09-02T10:20:34.364800+0000","last_fullsized":"2024-09-02T10:20:34.364800+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'6910","last_scrub_stamp":"2024-09-02T10:20:24.623633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:24.623633+0000","objects_scrubbed":5509,"log_size":7442,"log_dups_size":0,"ondisk_log_size":7442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:26.955962+0000","scrub_duration":1654,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323616768,"num_objects":4938,"num_object_clones":0,"num_object_copies":14814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7442,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":619,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.540974+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:20:35.540974+0000","last_peered":"2024-09-02T10:20:35.540974+0000","last_clean":"2024-09-02T10:20:35.540974+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:20:35.540974+0000","last_undegraded":"2024-09-02T10:20:35.540974+0000","last_fullsized":"2024-09-02T10:20:35.540974+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:51.265964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"578'7557","reported_seq":7715,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-09-02T10:20:32.891048+0000","last_change":"2024-09-02T10:19:52.017169+0000","last_active":"2024-09-02T10:20:32.891048+0000","last_peered":"2024-09-02T10:20:32.891048+0000","last_clean":"2024-09-02T10:20:32.891048+0000","last_became_active":"2024-09-02T10:18:37.064844+0000","last_became_peered":"2024-09-02T10:18:37.064844+0000","last_unstale":"2024-09-02T10:20:32.891048+0000","last_undegraded":"2024-09-02T10:20:32.891048+0000","last_fullsized":"2024-09-02T10:20:32.891048+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":7557,"log_dups_size":0,"ondisk_log_size":7557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:00.776952+0000","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335609856,"num_objects":5121,"num_object_clones":0,"num_object_copies":15363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7557,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":371,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.541045+0000","last_change":"2024-09-02T10:19:58.949821+0000","last_active":"2024-09-02T10:20:35.541045+0000","last_peered":"2024-09-02T10:20:35.541045+0000","last_clean":"2024-09-02T10:20:35.541045+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:20:35.541045+0000","last_undegraded":"2024-09-02T10:20:35.541045+0000","last_fullsized":"2024-09-02T10:20:35.541045+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:58.949745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.949745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.256455+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"578'7546","reported_seq":12067,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.242214+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:20:33.242214+0000","last_peered":"2024-09-02T10:20:33.242214+0000","last_clean":"2024-09-02T10:20:33.242214+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:20:33.242214+0000","last_undegraded":"2024-09-02T10:20:33.242214+0000","last_fullsized":"2024-09-02T10:20:33.242214+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":7546,"log_dups_size":0,"ondisk_log_size":7546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:22.534088+0000","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327417856,"num_objects":4996,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7546,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":620,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.540837+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:20:35.540837+0000","last_peered":"2024-09-02T10:20:35.540837+0000","last_clean":"2024-09-02T10:20:35.540837+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:20:35.540837+0000","last_undegraded":"2024-09-02T10:20:35.540837+0000","last_fullsized":"2024-09-02T10:20:35.540837+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:02.710854+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"580'7729","reported_seq":7899,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:36.244459+0000","last_change":"2024-09-02T10:20:23.889528+0000","last_active":"2024-09-02T10:20:36.244459+0000","last_peered":"2024-09-02T10:20:36.244459+0000","last_clean":"2024-09-02T10:20:36.244459+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:20:36.244459+0000","last_undegraded":"2024-09-02T10:20:36.244459+0000","last_fullsized":"2024-09-02T10:20:36.244459+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7040","last_scrub_stamp":"2024-09-02T10:20:23.889443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.889443+0000","objects_scrubbed":5664,"log_size":7729,"log_dups_size":0,"ondisk_log_size":7729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:24.393565+0000","scrub_duration":2079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321323008,"num_objects":4903,"num_object_clones":0,"num_object_copies":14709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7729,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":874,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.542607+0000","last_change":"2024-09-02T10:20:26.878564+0000","last_active":"2024-09-02T10:20:35.542607+0000","last_peered":"2024-09-02T10:20:35.542607+0000","last_clean":"2024-09-02T10:20:35.542607+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:20:35.542607+0000","last_undegraded":"2024-09-02T10:20:35.542607+0000","last_fullsized":"2024-09-02T10:20:35.542607+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.878519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.878519+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:33.891746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"579'7554","reported_seq":7716,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.692074+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:20:33.692074+0000","last_peered":"2024-09-02T10:20:33.692074+0000","last_clean":"2024-09-02T10:20:33.692074+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:20:33.692074+0000","last_undegraded":"2024-09-02T10:20:33.692074+0000","last_fullsized":"2024-09-02T10:20:33.692074+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":7554,"log_dups_size":0,"ondisk_log_size":7554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.641560+0000","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329777152,"num_objects":5032,"num_object_clones":0,"num_object_copies":15096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7554,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1139,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.519754+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:20:33.519754+0000","last_peered":"2024-09-02T10:20:33.519754+0000","last_clean":"2024-09-02T10:20:33.519754+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:20:33.519754+0000","last_undegraded":"2024-09-02T10:20:33.519754+0000","last_fullsized":"2024-09-02T10:20:33.519754+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:20:59.441621+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"580'7762","reported_seq":12175,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:37.244515+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:20:37.244515+0000","last_peered":"2024-09-02T10:20:37.244515+0000","last_clean":"2024-09-02T10:20:37.244515+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:20:37.244515+0000","last_undegraded":"2024-09-02T10:20:37.244515+0000","last_fullsized":"2024-09-02T10:20:37.244515+0000","mapping_epoch":565,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:23.125229+0000","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319160320,"num_objects":4870,"num_object_clones":0,"num_object_copies":14610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":885,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.542598+0000","last_change":"2024-09-02T10:19:59.731786+0000","last_active":"2024-09-02T10:20:35.542598+0000","last_peered":"2024-09-02T10:20:35.542598+0000","last_clean":"2024-09-02T10:20:35.542598+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:20:35.542598+0000","last_undegraded":"2024-09-02T10:20:35.542598+0000","last_fullsized":"2024-09-02T10:20:35.542598+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:59.731706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.731706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:16.407855+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"580'7701","reported_seq":7870,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:36.284127+0000","last_change":"2024-09-02T10:20:24.259375+0000","last_active":"2024-09-02T10:20:36.284127+0000","last_peered":"2024-09-02T10:20:36.284127+0000","last_clean":"2024-09-02T10:20:36.284127+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:20:36.284127+0000","last_undegraded":"2024-09-02T10:20:36.284127+0000","last_fullsized":"2024-09-02T10:20:36.284127+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7041","last_scrub_stamp":"2024-09-02T10:20:24.259063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:24.259063+0000","objects_scrubbed":5644,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:31.570636+0000","scrub_duration":1490,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324337664,"num_objects":4949,"num_object_clones":0,"num_object_copies":14847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7701,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1140,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.541102+0000","last_change":"2024-09-02T10:20:18.886910+0000","last_active":"2024-09-02T10:20:35.541102+0000","last_peered":"2024-09-02T10:20:35.541102+0000","last_clean":"2024-09-02T10:20:35.541102+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:20:35.541102+0000","last_undegraded":"2024-09-02T10:20:35.541102+0000","last_fullsized":"2024-09-02T10:20:35.541102+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.886832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.886832+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:20.398760+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"578'7469","reported_seq":7633,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-09-02T10:20:32.900258+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:20:32.900258+0000","last_peered":"2024-09-02T10:20:32.900258+0000","last_clean":"2024-09-02T10:20:32.900258+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:20:32.900258+0000","last_undegraded":"2024-09-02T10:20:32.900258+0000","last_fullsized":"2024-09-02T10:20:32.900258+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":7469,"log_dups_size":0,"ondisk_log_size":7469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:03.389208+0000","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329449472,"num_objects":5027,"num_object_clones":0,"num_object_copies":15081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7469,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":88,"reported_epoch":579,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.524980+0000","last_change":"2024-09-02T10:20:26.313678+0000","last_active":"2024-09-02T10:20:33.524980+0000","last_peered":"2024-09-02T10:20:33.524980+0000","last_clean":"2024-09-02T10:20:33.524980+0000","last_became_active":"2024-09-02T10:19:06.049121+0000","last_became_peered":"2024-09-02T10:19:06.049121+0000","last_unstale":"2024-09-02T10:20:33.524980+0000","last_undegraded":"2024-09-02T10:20:33.524980+0000","last_fullsized":"2024-09-02T10:20:33.524980+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:51.434613+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"580'7701","reported_seq":7873,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:37.245345+0000","last_change":"2024-09-02T10:20:21.937271+0000","last_active":"2024-09-02T10:20:37.245345+0000","last_peered":"2024-09-02T10:20:37.245345+0000","last_clean":"2024-09-02T10:20:37.245345+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:20:37.245345+0000","last_undegraded":"2024-09-02T10:20:37.245345+0000","last_fullsized":"2024-09-02T10:20:37.245345+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6863","last_scrub_stamp":"2024-09-02T10:20:21.937140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.937140+0000","objects_scrubbed":5699,"log_size":7701,"log_dups_size":0,"ondisk_log_size":7701,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:39.939497+0000","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315817984,"num_objects":4819,"num_object_clones":0,"num_object_copies":14457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7701,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":913,"reported_epoch":580,"state":"active+clean","last_fresh":"2024-09-02T10:20:35.542838+0000","last_change":"2024-09-02T10:19:57.798578+0000","last_active":"2024-09-02T10:20:35.542838+0000","last_peered":"2024-09-02T10:20:35.542838+0000","last_clean":"2024-09-02T10:20:35.542838+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:20:35.542838+0000","last_undegraded":"2024-09-02T10:20:35.542838+0000","last_fullsized":"2024-09-02T10:20:35.542838+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:57.798501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:57.798501+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:05.189778+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"578'7519","reported_seq":7684,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-09-02T10:20:33.218211+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:20:33.218211+0000","last_peered":"2024-09-02T10:20:33.218211+0000","last_clean":"2024-09-02T10:20:33.218211+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:20:33.218211+0000","last_undegraded":"2024-09-02T10:20:33.218211+0000","last_fullsized":"2024-09-02T10:20:33.218211+0000","mapping_epoch":533,"log_start":"0'0","ondisk_log_start":"0'0","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":7519,"log_dups_size":0,"ondisk_log_size":7519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:39.633585+0000","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330760192,"num_objects":5047,"num_object_clones":0,"num_object_copies":15141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7519,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":308,"reported_epoch":578,"state":"active+clean","last_fresh":"2024-09-02T10:20:30.504154+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:20:30.504154+0000","last_peered":"2024-09-02T10:20:30.504154+0000","last_clean":"2024-09-02T10:20:30.504154+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:20:30.504154+0000","last_undegraded":"2024-09-02T10:20:30.504154+0000","last_fullsized":"2024-09-02T10:20:30.504154+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:21:53.802058+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":5167972376,"num_objects":78858,"num_object_clones":0,"num_object_copies":236574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78858,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121682,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8538,"num_bytes_recovered":559546368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15555964928,"data_stored":15555952712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121682,"ondisk_log_size":121682,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":730176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1802240,"data_stored":1788032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383926,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1407752,"kb_used_data":935020,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92337400,"statfs":{"total":95995035648,"available":94553497600,"internally_reserved":0,"allocated":957460480,"data_stored":957194949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":483981764},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612859,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266956,"kb_used_data":920560,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92478196,"statfs":{"total":95995035648,"available":94697672704,"internally_reserved":0,"allocated":942653440,"data_stored":942383837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":354613696},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841791,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1804000,"kb_used_data":1291088,"kb_used_omap":1,"kb_used_meta":512894,"kb_avail":91941152,"statfs":{"total":95995035648,"available":94147739648,"internally_reserved":0,"allocated":1322074112,"data_stored":1321724705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":525203904},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070725,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":980272,"kb_used_data":651816,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92764880,"statfs":{"total":95995035648,"available":94991237120,"internally_reserved":0,"allocated":667459584,"data_stored":667172119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299658,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2078736,"kb_used_data":1582324,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":91666416,"statfs":{"total":95995035648,"available":93866409984,"internally_reserved":0,"allocated":1620299776,"data_stored":1620030693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":508164554},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528590,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1588796,"kb_used_data":1236908,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92156356,"statfs":{"total":95995035648,"available":94368108544,"internally_reserved":0,"allocated":1266593792,"data_stored":1266328261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":360315335},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724819,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3034644,"kb_used_data":2461944,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":90710508,"statfs":{"total":95995035648,"available":92887560192,"internally_reserved":0,"allocated":2521030656,"data_stored":2520757501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953752,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1589472,"kb_used_data":1287568,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92155680,"statfs":{"total":95995035648,"available":94367416320,"internally_reserved":0,"allocated":1318469632,"data_stored":1318120225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149981,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3130400,"kb_used_data":2589712,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":90614752,"statfs":{"total":95995035648,"available":92789506048,"internally_reserved":0,"allocated":2651865088,"data_stored":2651515681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":553646536},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":77309411617,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575142,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1816484,"kb_used_data":1343644,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":91928668,"statfs":{"total":95995035648,"available":94134956032,"internally_reserved":0,"allocated":1375891456,"data_stored":1375534425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738668,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1543968,"kb_used_data":985232,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92201184,"statfs":{"total":95995035648,"available":94414012416,"internally_reserved":0,"allocated":1008877568,"data_stored":1008528161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":995295232,"data_stored":995295232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1363415040,"data_stored":1363410968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2638413824,"data_stored":2638413824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1304625152,"data_stored":1304625152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2515079168,"data_stored":2515075096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1258684416,"data_stored":1258684416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1613168640,"data_stored":1613168640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":664403968,"data_stored":664403968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1314390016,"data_stored":1314390016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":936120320,"data_stored":936116248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":952369152,"data_stored":952369152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:20:39.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 61 2024-09-02T10:20:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:39.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:39.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:39.178 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:39.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:39.234 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:39.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:39.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:39.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:39.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:39.453 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:39.453 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:39.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:39.500 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:39.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:39.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:39 smithi162 ceph-mon[20279]: from='client.26779 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:39.602 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:39.628 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:39 smithi151 ceph-mon[16639]: from='client.26779 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:39.633 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:39.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:39.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:39.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:39.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:39.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:39.801 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:39.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:39.880 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:39.882 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:39.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:39 smithi187 ceph-mon[20072]: from='client.26779 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:20:40.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:40.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:40.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:40.133 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:40.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:40.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:20:40.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:40.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:40.282 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:40.383 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:40.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:20:40.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:40.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:40.505 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:40.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:40.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:40.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:40.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:40.553 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:20:40.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:40.678 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:40.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:40.680 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pgp_num 61 > pg_num 31 2024-09-02T10:20:40.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:40.807 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:40.807 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:40.808 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:40 smithi187 ceph-mon[20072]: pgmap v1350: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 965 op/s 2024-09-02T10:20:40.808 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1036710134' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:20:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:40 smithi162 ceph-mon[20279]: pgmap v1350: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 965 op/s 2024-09-02T10:20:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1036710134' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:20:40.907 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:40.921 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:40 smithi151 ceph-mon[16639]: pgmap v1350: 47 pgs: 47 active+clean; 4.7 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 965 op/s 2024-09-02T10:20:40.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1036710134' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:20:40.934 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:40.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:40.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:41.037 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:41.056 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:41.056 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:41.061 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:41.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:41.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:41.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:41.191 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:41.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:41.293 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:41.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:41.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:41.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:41.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:41.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:41.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:41.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:41.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:41.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:41.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:41 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:41.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1036710134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:20:41.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:41 smithi187 ceph-mon[20072]: osdmap e581: 12 total, 12 up, 11 in 2024-09-02T10:20:41.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2512536853' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:20:41.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:41 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:20:41.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:41.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:41 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:41.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1036710134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:20:41.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:41 smithi151 ceph-mon[16639]: osdmap e581: 12 total, 12 up, 11 in 2024-09-02T10:20:41.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2512536853' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:20:41.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:41 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:20:41.710 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:41.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:41.812 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:41.841 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:41.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:41 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1036710134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:20:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:41 smithi162 ceph-mon[20279]: osdmap e581: 12 total, 12 up, 11 in 2024-09-02T10:20:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2512536853' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:20:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:41 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:20:41.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:41.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:41.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:41.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:42.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:42.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:42.104 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:42.165 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:20:42.190 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:20:42.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:42.215 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:42.238 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:42.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:42.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:42.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:42.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:42.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:42.492 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:42.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:42.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:42.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:42.625 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:42.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:42.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:42.752 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:42.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:42.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:42 smithi151 ceph-mon[16639]: pgmap v1352: 47 pgs: 47 active+clean; 4.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 779 op/s 2024-09-02T10:20:42.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:42 smithi162 ceph-mon[20279]: pgmap v1352: 47 pgs: 47 active+clean; 4.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 779 op/s 2024-09-02T10:20:42.854 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:42.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:42.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:42.901 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:42 smithi187 ceph-mon[20072]: pgmap v1352: 47 pgs: 47 active+clean; 4.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 779 op/s 2024-09-02T10:20:42.949 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:42.950 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:42.978 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:42.978 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:42.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:43.002 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:43.002 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:43.012 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:43.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:43.022 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:43.023 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:43.028 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:43.028 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:43.058 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:43.058 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:43.093 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:43.093 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:43.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:43.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:43.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:43.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:43.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:43.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:43.149 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:43.149 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:43.178 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:43.178 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:43.203 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:43.204 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:43.235 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:43.235 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:43.239 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:43.270 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:43.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:43.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:43.271 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:43.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:43.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:20:43.404 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:43.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:43.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:43.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:43.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:43.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:43.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:43.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:43.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1051019778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:43.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:20:43.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:43.831 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:43.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:43.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1051019778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:43.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:43.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:43.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1051019778' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:44.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:44.067 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:44.083 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:44.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:44.084 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:44.132 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:44.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:44.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:44.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:44.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:44.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:44.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:44.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:44.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:44.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:44.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:44.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:44.595 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:44.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:44.663 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:20:44.663 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:20:44.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:20:44.663 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:20:44.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.663 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:42314 4.0 e355/341 4:049d10b0:::benchmark_data_smithi187_76542_object42313:head v 355'2240, mlcod=355'2237)", 2024-09-02T10:20:44.663 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:11:49.052907+0000", 2024-09-02T10:20:44.663 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 535.53747367999995, 2024-09-02T10:20:44.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.227636956, 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.052907+0000", 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.052907+0000", 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.052908+0000", 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.088e-06 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.053042+0000", 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013480799999999999 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279523+0000", 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22648093799999999 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279535+0000", 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1678000000000001e-05 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279599+0000", 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4323000000000001e-05 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.279617+0000", 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7971000000000001e-05 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.280511+0000", 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00089334699999999996 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:11:49.280544+0000", 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2802999999999997e-05 2024-09-02T10:20:44.667 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90787 4.15 4:afba0c08:::benchmark_data_smithi187_76542_object90786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:32.022436+0000", 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 492.56794463900002, 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27502997200000001, 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:20:44.668 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90787 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022436+0000", 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022436+0000", 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022437+0000", 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5659999999999999e-06 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022462+0000", 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4437e-05 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022480+0000", 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8493000000000001e-05 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022489+0000", 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6959999999999994e-06 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022548+0000", 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9709e-05 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022678+0000", 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000129689 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.022825+0000", 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014708899999999999 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.024159+0000", 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013338320000000001 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.024979+0000", 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00081976499999999997 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.672 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297377+0000", 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.272397677 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297432+0000", 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5513000000000002e-05 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297466+0000", 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3506000000000001e-05 2024-09-02T10:20:44.673 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:90806 4.15 4:aae97575:::benchmark_data_smithi187_76542_object90805:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:32.085698+0000", 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 492.504681801, 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21198275899999999, 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:20:44.674 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 90806 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085698+0000", 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085698+0000", 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085700+0000", 2024-09-02T10:20:44.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.429e-06 2024-09-02T10:20:44.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085722+0000", 2024-09-02T10:20:44.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2178999999999999e-05 2024-09-02T10:20:44.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085736+0000", 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3687e-05 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085742+0000", 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1879999999999999e-06 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085792+0000", 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0034999999999998e-05 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.085909+0000", 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011707200000000001 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.086007+0000", 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7526000000000001e-05 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.087291+0000", 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001284246 2024-09-02T10:20:44.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.087763+0000", 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000472466 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297588+0000", 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20982505200000001 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297648+0000", 2024-09-02T10:20:44.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0167999999999997e-05 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:32.297681+0000", 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2710999999999999e-05 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPush(4.4 392/385 [PushOp(4:21e15029:::benchmark_data_smithi187_76542_object88778:head, version: 375'3759, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:21e15029:::benchmark_data_smithi187_76542_object88778:head@375'3759, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:15.639682+0000", 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 448.95069835300001, 2024-09-02T10:20:44.680 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23632888699999999, 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639682+0000", 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639682+0000", 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639683+0000", 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9500000000000002e-07 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.639708+0000", 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5466e-05 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.874909+0000", 2024-09-02T10:20:44.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23520074799999999 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.874920+0000", 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0825999999999999e-05 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.874993+0000", 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3734999999999998e-05 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.875002+0000", 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0610000000000002e-06 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.876011+0000", 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001008356 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:180712 4.2 e422/341 4:40dddee0:::benchmark_data_smithi187_76542_object88376:head v 422'6632, mlcod=422'6630)", 2024-09-02T10:20:44.684 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:18.534975+0000", 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 386.055405275, 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21512784800000001, 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534975+0000", 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.685 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534975+0000", 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534976+0000", 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.257e-06 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.534985+0000", 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3289999999999994e-06 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747701+0000", 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21271659000000001 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747709+0000", 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8609999999999998e-06 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747772+0000", 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2969e-05 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.747788+0000", 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6201999999999998e-05 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.750057+0000", 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022690900000000001 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:18.750103+0000", 2024-09-02T10:20:44.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5550000000000003e-05 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:17898 5.a 5:5b8e596f:::benchmark_data_smithi187_89405_object17897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e440)", 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:57.425237+0000", 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 347.16514352799999, 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.209706585, 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.689 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 17898 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425237+0000", 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425237+0000", 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425238+0000", 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.525e-06 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425259+0000", 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0987000000000001e-05 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425273+0000", 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3309000000000001e-05 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425278+0000", 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.891e-06 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425329+0000", 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0488999999999997e-05 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425415+0000", 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6077999999999998e-05 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.425520+0000", 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010503800000000001 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.426467+0000", 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00094675800000000002 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.693 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.426778+0000", 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031167899999999998 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634817+0000", 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20803839900000001 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634893+0000", 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6501000000000005e-05 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:57.634943+0000", 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9931e-05 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63248 5.a 5:5adbc538:::benchmark_data_smithi187_89405_object63247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 308.076220288, 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21653889600000001, 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.695 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63248 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514161+0000", 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.099e-06 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514181+0000", 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.999e-05 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514212+0000", 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0648999999999997e-05 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514371+0000", 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015914799999999999 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514421+0000", 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9804e-05 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514561+0000", 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000140285 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514666+0000", 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104963 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.516942+0000", 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022756780000000002 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521266+0000", 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043245380000000002 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730584+0000", 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20931819099999999 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730647+0000", 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3040999999999995e-05 2024-09-02T10:20:44.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730699+0000", 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.151e-05 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63255 5.a 5:57c5feec:::benchmark_data_smithi187_89405_object63254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 308.07241038799998, 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21287715600000001, 2024-09-02T10:20:44.701 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63255 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517972+0000", 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.832e-06 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517998+0000", 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6004e-05 2024-09-02T10:20:44.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518024+0000", 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6290000000000001e-05 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518039+0000", 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.484e-05 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518103+0000", 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4350000000000006e-05 2024-09-02T10:20:44.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518230+0000", 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012653999999999999 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518365+0000", 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013489900000000001 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.520755+0000", 2024-09-02T10:20:44.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023907780000000001 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.523888+0000", 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031328179999999999 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730745+0000", 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.206856442 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730800+0000", 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5055000000000001e-05 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730847+0000", 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7308000000000002e-05 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63260 5.a 5:581436a1:::benchmark_data_smithi187_89405_object63259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:20:44.707 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 308.069105116, 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21179099900000001, 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63260 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521277+0000", 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.782e-06 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.709 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521301+0000", 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3893e-05 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521320+0000", 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8879999999999999e-05 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521327+0000", 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6890000000000004e-06 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521399+0000", 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1439000000000005e-05 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521552+0000", 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000153268 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521685+0000", 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013246699999999999 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.524092+0000", 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024077679999999998 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.525479+0000", 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001386916 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.732968+0000", 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20748902399999999 2024-09-02T10:20:44.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.733024+0000", 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5269999999999998e-05 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.733066+0000", 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2602999999999999e-05 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:86595 5.a 5:52430871:::benchmark_data_smithi187_89405_object86594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 286.084295822, 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.225552366, 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 86595 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506087+0000", 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2469999999999998e-06 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506114+0000", 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7141999999999999e-05 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506133+0000", 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9137e-05 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506142+0000", 2024-09-02T10:20:44.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8859999999999993e-06 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506210+0000", 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7992999999999998e-05 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506361+0000", 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015085899999999999 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506506+0000", 2024-09-02T10:20:44.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014534300000000001 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.509602+0000", 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030956009999999999 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.597950+0000", 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088348135999999994 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731542+0000", 2024-09-02T10:20:44.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13359195700000001 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731597+0000", 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5368999999999998e-05 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731637+0000", 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9696000000000001e-05 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:122098 5.e e488/431 5:783becf1:::benchmark_data_smithi187_89405_object18493:head v 488'7571, mlcod=488'7567)", 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 248.01870070000001, 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21171738700000001, 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:20:44.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1999999999999999e-07 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571686+0000", 2024-09-02T10:20:44.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6010000000000002e-06 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780330+0000", 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20864460000000001 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780339+0000", 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1869999999999997e-06 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780399+0000", 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9941000000000001e-05 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780424+0000", 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5751000000000001e-05 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.783348+0000", 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029234819999999998 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.723 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.783397+0000", 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9104999999999998e-05 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:126123 5.e e490/431 5:7800327a:::benchmark_data_smithi187_89405_object22518:head v 490'7825, mlcod=490'7821)", 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 243.05035541999999, 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25454940500000001, 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.724 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540026+0000", 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1000000000000001e-06 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540033+0000", 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8009999999999998e-06 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791598+0000", 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25156501399999998 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791605+0000", 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9569999999999996e-06 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791659+0000", 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4088000000000001e-05 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791680+0000", 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1450999999999998e-05 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.794543+0000", 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028628120000000002 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.794574+0000", 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1182000000000001e-05 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:194962 5.a 5:5b6b76d9:::benchmark_data_smithi187_89405_object91357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:20:44.728 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 161.63167555000001, 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22579173, 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194962 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:20:44.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958706+0000", 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.269e-06 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.730 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958711+0000", 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7010000000000004e-06 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958723+0000", 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2641e-05 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958731+0000", 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9980000000000003e-06 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958775+0000", 2024-09-02T10:20:44.731 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3399999999999998e-05 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958862+0000", 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7163999999999994e-05 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958935+0000", 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2804000000000003e-05 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.732 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.962637+0000", 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037025320000000001 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.963124+0000", 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048696900000000002 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184411+0000", 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.221286813 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184450+0000", 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9230999999999998e-05 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184496+0000", 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6208000000000002e-05 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:194969 5.a 5:503ca7c3:::benchmark_data_smithi187_89405_object91364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:20:44.734 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 161.620778411, 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21939375699999999, 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194969 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969603+0000", 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.068e-06 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969607+0000", 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3850000000000004e-06 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969617+0000", 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7750000000000004e-06 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969622+0000", 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2449999999999998e-06 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969678+0000", 2024-09-02T10:20:44.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5784e-05 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969792+0000", 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113993 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969919+0000", 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012692700000000001 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.975053+0000", 2024-09-02T10:20:44.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051336569999999998 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.975381+0000", 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00032868799999999999 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188892+0000", 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21351113099999999 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.739 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188951+0000", 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8777000000000003e-05 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188996+0000", 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4326999999999998e-05 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207198 5.a 5:57bb5924:::benchmark_data_smithi187_89405_object103593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:20:44.740 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 144.47410641499999, 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44270453799999998, 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207198 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:20:44.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116275+0000", 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.393e-06 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116281+0000", 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.694e-06 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116296+0000", 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5234e-05 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116305+0000", 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5650000000000002e-06 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116360+0000", 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5164999999999998e-05 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116445+0000", 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5158e-05 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116517+0000", 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2376e-05 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.121133+0000", 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046159479999999999 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.121221+0000", 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7831999999999995e-05 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558874+0000", 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43765253900000001 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558932+0000", 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7908000000000002e-05 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558978+0000", 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6726e-05 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:207204 5.4 e525/431 5:2473bd20:::benchmark_data_smithi187_89405_object103599:head v 525'13184, mlcod=525'13181)", 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:20:44.746 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 144.46295145799999, 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42712563199999998, 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127430+0000", 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.251e-06 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127438+0000", 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9039999999999992e-06 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550537+0000", 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42309940200000001 2024-09-02T10:20:44.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550550+0000", 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2262999999999999e-05 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550695+0000", 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000145845 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550728+0000", 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2923000000000003e-05 2024-09-02T10:20:44.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554509+0000", 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037808550000000001 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554554+0000", 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5188999999999997e-05 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:19698 6.1 6:8f6a7646:::benchmark_data_smithi187_102448_object19697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:53.693434+0000", 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 110.896946457, 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22523997900000001, 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 19698 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.751 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693434+0000", 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693434+0000", 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.752 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693436+0000", 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7400000000000001e-06 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693457+0000", 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1824999999999999e-05 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693471+0000", 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3200999999999999e-05 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693478+0000", 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5820000000000004e-06 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693538+0000", 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9824999999999997e-05 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693698+0000", 2024-09-02T10:20:44.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015993299999999999 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693861+0000", 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016271300000000001 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.694864+0000", 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010034600000000001 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.695808+0000", 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00094396200000000001 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.918543+0000", 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22273512600000001 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.918616+0000", 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2878000000000003e-05 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.918674+0000", 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7734e-05 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:19714 6.1 6:8a5fdd45:::benchmark_data_smithi187_102448_object19713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:53.701207+0000", 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 110.88917277500001, 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21946518000000001, 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.757 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 19714 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701207+0000", 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701207+0000", 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701209+0000", 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2839999999999999e-06 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701234+0000", 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5704999999999999e-05 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701250+0000", 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5739999999999998e-05 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701259+0000", 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3310000000000004e-06 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701320+0000", 2024-09-02T10:20:44.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1755999999999996e-05 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701450+0000", 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000129794 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701592+0000", 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014243400000000001 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.703694+0000", 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002101416 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.703718+0000", 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3785999999999999e-05 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.920572+0000", 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21685423000000001 2024-09-02T10:20:44.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.920630+0000", 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8134000000000003e-05 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.920673+0000", 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2570000000000001e-05 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:19717 6.1 6:8971b800:::benchmark_data_smithi187_102448_object19716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:53.702495+0000", 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 110.887884739, 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.219151597, 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 19717 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.764 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702495+0000", 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702495+0000", 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702497+0000", 2024-09-02T10:20:44.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.567e-06 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702518+0000", 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0837999999999999e-05 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702531+0000", 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3467e-05 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702539+0000", 2024-09-02T10:20:44.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7419999999999997e-06 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702609+0000", 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9890999999999999e-05 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702775+0000", 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000165575 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:20:44.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702939+0000", 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016468799999999999 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.703801+0000", 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086205600000000004 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.704004+0000", 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000203177 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.921562+0000", 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21755740500000001 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.921611+0000", 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9421e-05 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.921647+0000", 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5769999999999998e-05 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:100226 6.1 6:80d0f12d:::benchmark_data_smithi187_102448_object100225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:20:10.905652+0000", 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 33.684727776999999, 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63928361700000003, 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 100226 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.770 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905652+0000", 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905652+0000", 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905654+0000", 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.593e-06 2024-09-02T10:20:44.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905727+0000", 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3153999999999998e-05 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905744+0000", 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6371e-05 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905753+0000", 2024-09-02T10:20:44.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2299999999999997e-06 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905816+0000", 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3464000000000001e-05 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905925+0000", 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010829599999999999 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906047+0000", 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012230699999999999 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.908790+0000", 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027432860000000002 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.909351+0000", 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00056067900000000002 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544856+0000", 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63550532299999996 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544894+0000", 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8143999999999999e-05 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544936+0000", 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1770000000000002e-05 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.775 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.776 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.776 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:44.776 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:44.776 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:20:44.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:44.777 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:44 smithi151 ceph-mon[16639]: pgmap v1353: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 939 op/s 2024-09-02T10:20:44.778 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:44 smithi187 ceph-mon[20072]: pgmap v1353: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 939 op/s 2024-09-02T10:20:44.802 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:44.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:44 smithi162 ceph-mon[20279]: pgmap v1353: 47 pgs: 47 active+clean; 4.5 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 939 op/s 2024-09-02T10:20:44.905 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:44.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:44.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:45.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:45.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:45.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:45.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:45.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:45.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:45.183 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:45.183 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:45.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:45.311 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:45.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:45.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:45.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:45.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:45.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:45.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:45.567 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:45.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:45.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:45.581 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:20:45.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:45.694 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:45.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:45.695 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:45 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:20:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:45 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:20:45.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2254870994' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:45.796 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:45.830 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:45.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:45 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:20:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:45 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:20:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2254870994' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:45.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:45.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:45 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:20:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:45 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:20:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2254870994' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:45.954 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:45.955 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:46.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:46.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:46.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:46.082 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:46.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:20:46.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:46.207 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:46.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:46.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:46.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:20:46.309 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:46.335 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:46.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:46.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:46.437 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:46.466 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:46.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:46.467 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:46.474 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:46.564 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:46.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:46.602 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:46.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:46.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: pgmap v1354: 47 pgs: 47 active+clean; 4.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 883 op/s 2024-09-02T10:20:46.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:20:46.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:20:46.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2254870994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:46.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: osdmap e582: 12 total, 12 up, 11 in 2024-09-02T10:20:46.642 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:20:46.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:46.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:20:46.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:20:46.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:20:46.643 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:46 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:46.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:46.734 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:46.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:46.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:46.835 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: pgmap v1354: 47 pgs: 47 active+clean; 4.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 883 op/s 2024-09-02T10:20:46.835 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:20:46.835 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:20:46.835 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2254870994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:46.835 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: osdmap e582: 12 total, 12 up, 11 in 2024-09-02T10:20:46.835 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:20:46.835 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:46.835 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:20:46.835 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:20:46.835 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:20:46.836 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:46 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:46.836 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: pgmap v1354: 47 pgs: 47 active+clean; 4.4 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 0 B/s wr, 883 op/s 2024-09-02T10:20:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:20:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:20:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2254870994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: osdmap e582: 12 total, 12 up, 11 in 2024-09-02T10:20:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:20:46.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:46.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:20:46.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:20:46.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:20:46.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:46 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:46.863 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:46.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:46.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:46.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:46.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:46.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:46.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:47.093 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:20:47.093 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:20:47.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:47.093 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:132589 6.9 e583/533 6:90d93490:::benchmark_data_smithi187_102448_object32317:head v 583'8203, mlcod=583'8201)", 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:20:47.090858+0000", 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00086658999999999998, 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00088471199999999996, 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:47.090858+0000", 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:47.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:47.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:47.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:47.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:47.090858+0000", 2024-09-02T10:20:47.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:47.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:47.090859+0000", 2024-09-02T10:20:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1200000000000001e-06 2024-09-02T10:20:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:47.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:47.090870+0000", 2024-09-02T10:20:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0366999999999999e-05 2024-09-02T10:20:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:47.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:47.090902+0000", 2024-09-02T10:20:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2540999999999999e-05 2024-09-02T10:20:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:47.090910+0000", 2024-09-02T10:20:47.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6450000000000005e-06 2024-09-02T10:20:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:47.090968+0000", 2024-09-02T10:20:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7865999999999998e-05 2024-09-02T10:20:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:47.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:47.090991+0000", 2024-09-02T10:20:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2986999999999999e-05 2024-09-02T10:20:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:47.100 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:20:47.101 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:20:47.101 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:20:47.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:47.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:47.103 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:47.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:47.122 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:47.127 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:47.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:47.147 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:47.148 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:47.170 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:47.171 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:47.191 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:20:47.191 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:20:47.191 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-02T10:20:47.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd in 2 2024-09-02T10:20:47.198 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:47.198 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:47.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:47.231 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:47.231 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:47.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:47.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:47.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:47.264 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:47.265 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:47.295 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:47.296 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:47.327 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:47.327 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:47.354 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:47.354 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:47.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:47.390 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:47.390 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:47.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:47.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:47.423 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:47.423 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:47.502 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:47.526 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:47.527 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:47.628 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:47.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:47.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:47.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:47.787 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:47.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:47.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:20:47.864 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:47 smithi187 ceph-mon[20072]: osdmap e583: 12 total, 12 up, 11 in 2024-09-02T10:20:47.864 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3407125440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:47.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:47.917 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:47.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:47.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:47 smithi151 ceph-mon[16639]: osdmap e583: 12 total, 12 up, 11 in 2024-09-02T10:20:47.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3407125440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:48.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:48.047 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:48.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:47 smithi162 ceph-mon[20279]: osdmap e583: 12 total, 12 up, 11 in 2024-09-02T10:20:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3407125440' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:48.145 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:48.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:48.175 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:48.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:48.276 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:48.298 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:48.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:48.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:48.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:48.425 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:48.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:48.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:20:48.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:48.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:48.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:48.643 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:48.643 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:48.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:48.678 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:48.679 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:48.739 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:48 smithi151 ceph-mon[16639]: pgmap v1357: 47 pgs: 47 active+clean; 4.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 959 op/s 2024-09-02T10:20:48.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:48.793 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:48.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:48.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:48.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:48 smithi162 ceph-mon[20279]: pgmap v1357: 47 pgs: 47 active+clean; 4.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 959 op/s 2024-09-02T10:20:48.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:48.929 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:48.930 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:48 smithi187 ceph-mon[20072]: pgmap v1357: 47 pgs: 47 active+clean; 4.3 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 959 op/s 2024-09-02T10:20:49.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:49.056 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:49.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:49.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:49.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:49.186 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:49.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:49.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:49.312 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:49.314 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:49.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:20:49.361 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:49.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:49.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:49.453 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:49.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:49.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:49.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:49.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:49.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:49.709 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:49.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:49.710 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:49.711 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:49.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:49.839 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:49.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:49.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:49.966 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:49.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:49.967 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:50.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:50.090 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:20:50.094 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:50.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:50.128 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:50 smithi187 ceph-mon[20072]: pgmap v1358: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-02T10:20:50.128 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3982291070' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:20:50.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:50.221 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:50.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:50.222 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:50.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:50.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:50 smithi162 ceph-mon[20279]: pgmap v1358: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-02T10:20:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3982291070' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:20:50.349 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:50.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:50 smithi151 ceph-mon[16639]: pgmap v1358: 47 pgs: 47 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s 2024-09-02T10:20:50.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3982291070' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:20:50.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:50.470 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:50.470 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:50.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:50.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:50.577 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:50.598 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:50.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:50.700 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:50.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:50.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:50.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:20:50.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:50 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:20:50.472+0000 7fe3ffed4640 -1 mon.a@0(leader).osd e584 definitely_dead 0 2024-09-02T10:20:50.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:50.853 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:50.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:50.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:50.978 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:50.979 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:51.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:51.093 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.2. 2024-09-02T10:20:51.109 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:51.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:51.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:51.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:51.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:20:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:51 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3982291070' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:20:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:51 smithi187 ceph-mon[20072]: osdmap e584: 12 total, 12 up, 11 in 2024-09-02T10:20:51.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2709808653' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-02T10:20:51.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:20:51.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:51.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:20:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:51 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3982291070' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:20:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:51 smithi162 ceph-mon[20279]: osdmap e584: 12 total, 12 up, 11 in 2024-09-02T10:20:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2709808653' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-02T10:20:51.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:51.357 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:51.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:51.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:20:51.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:51.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:51 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:51.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3982291070' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:20:51.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:51 smithi151 ceph-mon[16639]: osdmap e584: 12 total, 12 up, 11 in 2024-09-02T10:20:51.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2709808653' entity='client.admin' cmd={"prefix": "osd in", "ids": ["2"]} : dispatch 2024-09-02T10:20:51.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:51.480 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:51.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:51.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:51.579 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:51.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:51.605 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:51.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:51.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:51.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:51.729 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:51.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:51.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:51.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:51.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:51.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:51.977 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:51.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:52.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:52.088 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:52.088 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:52.106 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:52.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:52.208 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:52.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:52.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:52.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2709808653' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-02T10:20:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:52 smithi187 ceph-mon[20072]: osdmap e585: 12 total, 12 up, 12 in 2024-09-02T10:20:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:52 smithi187 ceph-mon[20072]: pgmap v1361: 47 pgs: 1 peering, 46 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 931 op/s 2024-09-02T10:20:52.257 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2709808653' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-02T10:20:52.257 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:52 smithi151 ceph-mon[16639]: osdmap e585: 12 total, 12 up, 12 in 2024-09-02T10:20:52.257 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:52 smithi151 ceph-mon[16639]: pgmap v1361: 47 pgs: 1 peering, 46 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 931 op/s 2024-09-02T10:20:52.284 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-02T10:20:52.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:52.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:52.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:52.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:52.368 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:52.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:52.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:52.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:52.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2709808653' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-02T10:20:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:52 smithi162 ceph-mon[20279]: osdmap e585: 12 total, 12 up, 12 in 2024-09-02T10:20:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:52 smithi162 ceph-mon[20279]: pgmap v1361: 47 pgs: 1 peering, 46 active+clean; 4.1 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 0 B/s wr, 931 op/s 2024-09-02T10:20:52.603 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:52.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:52.705 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:52.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:20:52.727 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:52.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:52.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:52.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:52.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:52.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:52.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:52.869 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:52.870 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:52.890 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:52.890 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:52.910 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:52.911 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:52.930 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:52.931 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:52.951 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:52.951 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:52.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:52.958 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:52.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:52.973 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:52.976 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:52.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:52.992 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:52.993 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:53.014 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:53.014 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:53.034 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:53.034 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:53.054 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:53.054 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:53.075 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:53.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:53.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:53.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:53.097 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:53.102 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:53.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:53.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:53.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:53.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:53.331 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:53 smithi187 ceph-mon[20072]: osdmap e586: 12 total, 12 up, 12 in 2024-09-02T10:20:53.331 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:53 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:20:53.331 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:53 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:20:53.331 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:53 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:20:53.331 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2524348565' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:53.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:53.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:53 smithi162 ceph-mon[20279]: osdmap e586: 12 total, 12 up, 12 in 2024-09-02T10:20:53.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:53 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:20:53.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:53 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:20:53.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:53 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:20:53.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2524348565' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:53.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:53.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:53.429 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:20:53.430 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:20:53.430 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:20:53.430 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:20:53.430 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:20:53.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:53 smithi151 ceph-mon[16639]: osdmap e586: 12 total, 12 up, 12 in 2024-09-02T10:20:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:53 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:20:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:53 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:20:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:53 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:20:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2524348565' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:53.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:53.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:53.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:53.492 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:53.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:53.629 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:53.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:53.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:20:53.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:53.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:53.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:53.866 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:53.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:53.889 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:53.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:54.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:54.027 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:54.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:54.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:20:54.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:54.167 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:54.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:54.168 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:54.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:54.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:54.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:54 smithi162 ceph-mon[20279]: osdmap e587: 12 total, 12 up, 12 in 2024-09-02T10:20:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:54 smithi162 ceph-mon[20279]: pgmap v1364: 47 pgs: 2 remapped+peering, 3 peering, 42 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 685 op/s; 61 KiB/s, 0 objects/s recovering 2024-09-02T10:20:54.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:54.404 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:54.416 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:54 smithi187 ceph-mon[20072]: osdmap e587: 12 total, 12 up, 12 in 2024-09-02T10:20:54.416 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:54 smithi187 ceph-mon[20072]: pgmap v1364: 47 pgs: 2 remapped+peering, 3 peering, 42 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 685 op/s; 61 KiB/s, 0 objects/s recovering 2024-09-02T10:20:54.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:54.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:54.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:54 smithi151 ceph-mon[16639]: osdmap e587: 12 total, 12 up, 12 in 2024-09-02T10:20:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:54 smithi151 ceph-mon[16639]: pgmap v1364: 47 pgs: 2 remapped+peering, 3 peering, 42 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 16 KiB/s wr, 685 op/s; 61 KiB/s, 0 objects/s recovering 2024-09-02T10:20:54.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:54.555 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:54.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:54.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:54.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:54.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:54.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:54.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:54.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:54.824 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:54.925 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:54.953 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:54.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:54.985 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:20:54.985 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103692 4.19 4.40dd9139 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 488.88717186700001, 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.993593167, 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:20:54.986 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103692 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095743+0000", 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:54.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095744+0000", 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6899999999999996e-07 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095748+0000", 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4499999999999997e-06 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095758+0000", 2024-09-02T10:20:54.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2140000000000002e-06 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095764+0000", 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5690000000000001e-06 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095808+0000", 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3588999999999999e-05 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:20:54.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.095814+0000", 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2299999999999996e-06 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089305+0000", 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99349078599999996 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089336+0000", 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.146e-05 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:54.990 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25507.0:103721 4.e e384/340 4:78ce6951:::benchmark_data_smithi187_76542_object11385:head v 384'6547, mlcod=384'6544)", 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 488.59112190000002, 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.265184527, 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:20:54.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391793+0000", 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391794+0000", 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4000000000000001e-07 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.391799+0000", 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2009999999999998e-06 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654392+0000", 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26259298599999997 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654398+0000", 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7880000000000003e-06 2024-09-02T10:20:54.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654446+0000", 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8183000000000003e-05 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.654470+0000", 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4093000000000001e-05 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.656943+0000", 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024726489999999999 2024-09-02T10:20:54.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.656978+0000", 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4887e-05 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103725 4.19 4.a40a4399 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 488.54642722099999, 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65288606400000004, 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:54.995 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103725 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436488+0000", 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436489+0000", 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2750000000000001e-06 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436494+0000", 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4660000000000001e-06 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436506+0000", 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2123000000000001e-05 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436512+0000", 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6749999999999996e-06 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436566+0000", 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3285000000000002e-05 2024-09-02T10:20:54.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.436572+0000", 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6019999999999996e-06 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089367+0000", 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65279432500000001 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089374+0000", 2024-09-02T10:20:54.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3130000000000001e-06 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103743 4.19 4.222eb379 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 488.313587412, 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42007966200000002, 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:20:55.000 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103743 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669328+0000", 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669329+0000", 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.221e-06 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669334+0000", 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1340000000000003e-06 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669346+0000", 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2155e-05 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669351+0000", 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9810000000000003e-06 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669433+0000", 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1617000000000001e-05 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.669445+0000", 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.207e-05 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089400+0000", 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41995552600000002 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089407+0000", 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9580000000000001e-06 2024-09-02T10:20:55.004 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:103767 4.19 4.bc594c79 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 488.213133035, 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31965745400000001, 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:20:55.005 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 103767 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769782+0000", 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769785+0000", 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5299999999999999e-06 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769796+0000", 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.164e-05 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769823+0000", 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7055000000000001e-05 2024-09-02T10:20:55.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769828+0000", 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0350000000000001e-06 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769891+0000", 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2973999999999997e-05 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:46.769903+0000", 2024-09-02T10:20:55.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1569999999999999e-05 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089432+0000", 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31952966900000002 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:12:47.089439+0000", 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9809999999999997e-06 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62902 5.0 5:0a3860dd:::benchmark_data_smithi187_89405_object62901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 320.34784842800002, 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28493924999999998, 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62902 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.010 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635068+0000", 2024-09-02T10:20:55.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.502e-06 2024-09-02T10:20:55.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635089+0000", 2024-09-02T10:20:55.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0916000000000001e-05 2024-09-02T10:20:55.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635100+0000", 2024-09-02T10:20:55.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1243000000000001e-05 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635107+0000", 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5359999999999998e-06 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635155+0000", 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7939000000000003e-05 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635250+0000", 2024-09-02T10:20:55.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5571999999999999e-05 2024-09-02T10:20:55.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:20:55.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635339+0000", 2024-09-02T10:20:55.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9158000000000003e-05 2024-09-02T10:20:55.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.636945+0000", 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001605555 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.637559+0000", 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00061385400000000005 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.919922+0000", 2024-09-02T10:20:55.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28236284 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.919962+0000", 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0227e-05 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.920006+0000", 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3908e-05 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.017 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:126223 5.4 5:2d883f96:::benchmark_data_smithi187_89405_object22618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 253.056987351, 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32322968800000001, 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126223 2024-09-02T10:20:55.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925929+0000", 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2020000000000001e-06 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925933+0000", 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3479999999999997e-06 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925945+0000", 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1559999999999999e-05 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925952+0000", 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4120000000000004e-06 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925997+0000", 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4759999999999998e-05 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.926077+0000", 2024-09-02T10:20:55.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0203000000000004e-05 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.926145+0000", 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8311000000000004e-05 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929053+0000", 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029072289999999999 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.989494+0000", 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.060441307999999999 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249062+0000", 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25956764199999999 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249115+0000", 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2896000000000003e-05 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249157+0000", 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2817e-05 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:126223 5.4 e490/431 ondisk, result = 0)", 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 253.05364323699999, 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31989100399999998, 2024-09-02T10:20:55.024 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2900000000000003e-07 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929276+0000", 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4989999999999999e-06 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.248945+0000", 2024-09-02T10:20:55.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31966908500000002 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.248950+0000", 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1749999999999996e-06 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249048+0000", 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7676000000000005e-05 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249056+0000", 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2439999999999993e-06 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249163+0000", 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106696 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162650 5.0 5:0595c7e6:::benchmark_data_smithi187_89405_object59045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:20:55.028 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 210.68022534900001, 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29653194100000002, 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162650 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302691+0000", 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6139999999999999e-06 2024-09-02T10:20:55.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302696+0000", 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1599999999999997e-06 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302706+0000", 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3959999999999995e-06 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302711+0000", 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2920000000000003e-06 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302770+0000", 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8545999999999998e-05 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302912+0000", 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014217599999999999 2024-09-02T10:20:55.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.303032+0000", 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120482 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.307337+0000", 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004304993 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.307512+0000", 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017463799999999999 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598774+0000", 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29126201499999999 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599166+0000", 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039243999999999998 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599222+0000", 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5189000000000003e-05 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:162688 5.8 e507/490 5:1af44aef:::benchmark_data_smithi187_89405_object59083:head v 507'10194, mlcod=507'10191)", 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 210.63501135499999, 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.262002719, 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.035 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:20:55.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8700000000000002e-07 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347908+0000", 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2030000000000002e-06 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347966+0000", 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7603999999999999e-05 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347970+0000", 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1160000000000001e-06 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.348622+0000", 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00065184499999999996 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.348639+0000", 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7731e-05 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609811+0000", 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26117189299999999 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609906+0000", 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5039999999999998e-05 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:162690 5.f e507/431 5:f963755e:::benchmark_data_smithi187_89405_object59085:head v 507'10319, mlcod=507'10317)", 2024-09-02T10:20:55.039 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 210.63130742600001, 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26002222000000003, 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3699999999999999e-07 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351615+0000", 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1940000000000003e-06 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353238+0000", 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001623471 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353245+0000", 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4540000000000004e-06 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353298+0000", 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3186999999999997e-05 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353317+0000", 2024-09-02T10:20:55.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9083999999999999e-05 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611594+0000", 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.258277337 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611630+0000", 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5655999999999999e-05 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162691 5.4 5:2922879e:::benchmark_data_smithi187_89405_object59086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 210.63084756500001, 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25893540799999998, 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162691 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.044 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352069+0000", 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6869999999999999e-06 2024-09-02T10:20:55.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352075+0000", 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5940000000000001e-06 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352087+0000", 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.216e-05 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352093+0000", 2024-09-02T10:20:55.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5969999999999997e-06 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599399+0000", 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24730590099999999 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599528+0000", 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012868299999999999 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:20:55.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599654+0000", 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000125959 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610381+0000", 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.01072701 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610393+0000", 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1921e-05 2024-09-02T10:20:55.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610740+0000", 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00034708900000000002 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610837+0000", 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7175000000000004e-05 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611003+0000", 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000165632 2024-09-02T10:20:55.049 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162695 5.0 5:02845cd7:::benchmark_data_smithi187_89405_object59090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 210.62908192099999, 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.266211531, 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:55.050 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162695 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353834+0000", 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3900000000000004e-07 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353838+0000", 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3460000000000004e-06 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598349+0000", 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24451044699999999 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598367+0000", 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8236999999999998e-05 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609481+0000", 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011113576 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609606+0000", 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012559100000000001 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610188+0000", 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058149899999999997 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619903+0000", 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0097155599999999998 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619914+0000", 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1126e-05 2024-09-02T10:20:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619947+0000", 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2432999999999999e-05 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.620001+0000", 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4295000000000001e-05 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.620045+0000", 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3581999999999997e-05 2024-09-02T10:20:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:173247 5.8 e509/490 5:173cf939:::benchmark_data_smithi187_89405_object69642:head v 509'10875, mlcod=509'10872)", 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 197.840498916, 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25939572700000002, 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142417+0000", 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6099999999999999e-07 2024-09-02T10:20:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142424+0000", 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9489999999999999e-06 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396116+0000", 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25369161699999998 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396123+0000", 2024-09-02T10:20:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5059999999999998e-06 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396178+0000", 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5271e-05 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396199+0000", 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1009000000000002e-05 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.401729+0000", 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0055293260000000002 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.401812+0000", 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3088000000000001e-05 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.060 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207153 5.4 5:2d15d89f:::benchmark_data_smithi187_89405_object103548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 154.93839824299999, 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.39443205100000001, 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207153 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044518+0000", 2024-09-02T10:20:55.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1179999999999999e-06 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044523+0000", 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.113e-06 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044535+0000", 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1868999999999999e-05 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044540+0000", 2024-09-02T10:20:55.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9370000000000003e-06 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044599+0000", 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9589000000000002e-05 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044722+0000", 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122166 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:20:55.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044855+0000", 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133774 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.048607+0000", 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037520180000000002 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.075260+0000", 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.026652973 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438874+0000", 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.363613939 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438913+0000", 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8795999999999999e-05 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438949+0000", 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5759000000000001e-05 2024-09-02T10:20:55.066 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207153 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 154.895183887, 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35122426299999998, 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087732+0000", 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5300000000000004e-07 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087736+0000", 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0659999999999997e-06 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438760+0000", 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.351023794 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438766+0000", 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6499999999999999e-06 2024-09-02T10:20:55.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438864+0000", 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8116999999999996e-05 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438871+0000", 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1430000000000003e-06 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438955+0000", 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4839999999999994e-05 2024-09-02T10:20:55.070 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:207198 5.a e525/431 5:57bb5924:::benchmark_data_smithi187_89405_object103593:head v 525'12960, mlcod=525'12956)", 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 154.86632839200001, 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44214100000000001, 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.071 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1399999999999996e-07 2024-09-02T10:20:55.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116593+0000", 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0079999999999999e-06 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554609+0000", 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43801511199999998 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554615+0000", 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8349999999999998e-06 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554662+0000", 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7018000000000003e-05 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554678+0000", 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5116000000000001e-05 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558692+0000", 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040144350000000002 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558728+0000", 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5661999999999998e-05 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207204 5.4 5:2473bd20:::benchmark_data_smithi187_89405_object103599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 154.855796105, 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73061778899999996, 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207204 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127120+0000", 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2810000000000001e-06 2024-09-02T10:20:55.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127126+0000", 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1210000000000003e-06 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127137+0000", 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0474e-05 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127142+0000", 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5419999999999996e-06 2024-09-02T10:20:55.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127191+0000", 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8485000000000001e-05 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127278+0000", 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7492999999999997e-05 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127352+0000", 2024-09-02T10:20:55.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3591999999999996e-05 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131106+0000", 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003754496 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850393+0000", 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.719286219 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857585+0000", 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071919810000000001 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857692+0000", 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107038 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857737+0000", 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5067000000000001e-05 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.081 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207204 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 154.85126939, 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72609860400000004, 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131647+0000", 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2529999999999999e-06 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131653+0000", 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4119999999999998e-06 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857394+0000", 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72574053699999996 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857411+0000", 2024-09-02T10:20:55.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7283e-05 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857560+0000", 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014918700000000001 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857574+0000", 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.329e-05 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857744+0000", 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017064200000000001 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207204 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 154.42832817999999, 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.295830757, 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:20:55.086 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:20:55.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554588+0000", 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8899999999999998e-07 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554593+0000", 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1039999999999998e-06 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850206+0000", 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29561359100000001 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850223+0000", 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6730000000000001e-05 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850366+0000", 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014310500000000001 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850383+0000", 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6419999999999999e-05 2024-09-02T10:20:55.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:20:55.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:20:55.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:20:55.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850418+0000", 2024-09-02T10:20:55.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4918e-05 2024-09-02T10:20:55.090 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.090 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.090 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.090 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:20:55.090 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:20:55.090 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:20:55.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:55.147 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:55.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:55.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:55 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:20:55.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:55.280 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:55 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:20:55.293 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:55.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:55.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:55.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:55.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:55.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:20:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:55.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:55.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:20:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:55 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:20:55.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:55.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:55.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:55.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:55.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:55.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:55.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:55.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:55.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:55.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:56.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:56.098 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:56.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:56.133 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:20:56.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:56.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:56.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:56.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:20:56.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:56.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:56.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:56.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:56 smithi151 ceph-mon[16639]: pgmap v1365: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 3 peering, 40 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 388 B/s rd, 12 KiB/s wr, 566 op/s; 47/193155 objects degraded (0.024%); 8054/193155 objects misplaced (4.170%); 447 KiB/s, 6 objects/s recovering 2024-09-02T10:20:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1609657133' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:56 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:56.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:20:56.486 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:56.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:20:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:56 smithi187 ceph-mon[20072]: pgmap v1365: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 3 peering, 40 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 388 B/s rd, 12 KiB/s wr, 566 op/s; 47/193155 objects degraded (0.024%); 8054/193155 objects misplaced (4.170%); 447 KiB/s, 6 objects/s recovering 2024-09-02T10:20:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1609657133' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:56 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:56.549 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:56.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:56.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:56 smithi162 ceph-mon[20279]: pgmap v1365: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 remapped+peering, 3 peering, 40 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 388 B/s rd, 12 KiB/s wr, 566 op/s; 47/193155 objects degraded (0.024%); 8054/193155 objects misplaced (4.170%); 447 KiB/s, 6 objects/s recovering 2024-09-02T10:20:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1609657133' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:56 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:20:56.628 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:56.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:56.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:56.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:56.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:56.753 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:56.854 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:56.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:56.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:56.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:57.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:57.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:57.007 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:57.089 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:20:57.089 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:20:57.089 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:20:57.089 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:20:57.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:20:57.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:57.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:57.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:57.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:57 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 47/193155 objects degraded (0.024%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:20:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:57 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:57 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:57 smithi187 ceph-mon[20072]: osdmap e588: 12 total, 12 up, 12 in 2024-09-02T10:20:57.266 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:57.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:57.267 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:57.285 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:20:57.285 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:20:57.285 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-09-02T10:20:57.285 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-02T10:20:57.285 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-02T10:20:57.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-09-02T10:20:57.368 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:57.391 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:57.392 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:57 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 47/193155 objects degraded (0.024%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:20:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:57 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:57.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:57 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:57.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:57 smithi151 ceph-mon[16639]: osdmap e588: 12 total, 12 up, 12 in 2024-09-02T10:20:57.450 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:20:57.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:57.520 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:57.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:57.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:57 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 47/193155 objects degraded (0.024%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:20:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:57 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:20:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:57 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:20:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:57 smithi162 ceph-mon[20279]: osdmap e588: 12 total, 12 up, 12 in 2024-09-02T10:20:57.623 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:20:57.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:57.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:20:57.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:20:57.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:57.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:57.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:20:57.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:20:57.884 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:57.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:57.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:57.913 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:57.927 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:58.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:20:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:58.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:58.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:20:58.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:58.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:58.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:58.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:20:58.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:58 smithi151 ceph-mon[16639]: pgmap v1367: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 43 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 533 op/s; 47/190155 objects degraded (0.025%); 11867/190155 objects misplaced (6.241%); 1.7 MiB/s, 27 objects/s recovering 2024-09-02T10:20:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3863373404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:58.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:20:58.299 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:58.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:20:58.346 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:58 smithi187 ceph-mon[20072]: pgmap v1367: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 43 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 533 op/s; 47/190155 objects degraded (0.025%); 11867/190155 objects misplaced (6.241%); 1.7 MiB/s, 27 objects/s recovering 2024-09-02T10:20:58.346 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3863373404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:58.401 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:58.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:58.424 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:58.463 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:20:58.463 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "" 2024-09-02T10:20:58.463 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:20:58.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:20:58.557 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:58.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:20:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:58 smithi162 ceph-mon[20279]: pgmap v1367: 47 pgs: 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 1 remapped+peering, 43 active+clean; 3.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 11 KiB/s wr, 533 op/s; 47/190155 objects degraded (0.025%); 11867/190155 objects misplaced (6.241%); 1.7 MiB/s, 27 objects/s recovering 2024-09-02T10:20:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3863373404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:20:58.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:58.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:58.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:58.723 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:20:58.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:58.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:58.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:58.905 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:58.918 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:20:58.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:20:58.930 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:58.931 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:58.951 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:20:58.951 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:20:58.981 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:20:58.981 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:20:58.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:20:59.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:20:59.016 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:20:59.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:59.048 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:20:59.049 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:20:59.060 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:59.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:59.085 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:20:59.085 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:20:59.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:20:59.122 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:20:59.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:20:59.159 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:20:59.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:59.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:59 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:20:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:20:59 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:20:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:59.190 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:59.193 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:20:59.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:20:59.227 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:20:59.227 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:20:59.261 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:20:59.262 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:20:59.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:20:59.293 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:20:59.293 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:20:59.317 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:20:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:59.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:20:59.419 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:20:59.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:59 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:20:59.429 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:20:59 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:20:59.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:59.444 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:20:59.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:20:59.568 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:59.569 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:20:59.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:59 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:20:59.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:20:59 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:20:59.661 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:20:59.661 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:20:59.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:59.694 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:59.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:59.695 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:59.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:20:59.819 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:20:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:59.820 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:20:59.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:20:59.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:20:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:20:59.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:00.048 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:00.070 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:00.071 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:00.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:00.201 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:00.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:00.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:00.304 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:00.313 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:00 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:21:00.313 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:00 smithi187 ceph-mon[20072]: osdmap e589: 12 total, 12 up, 12 in 2024-09-02T10:21:00.313 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:00 smithi187 ceph-mon[20072]: pgmap v1369: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 6505/185412 objects degraded (3.508%); 11416/185412 objects misplaced (6.157%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:00.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:00.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:00.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:21:00.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:00 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:21:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:00 smithi151 ceph-mon[16639]: osdmap e589: 12 total, 12 up, 12 in 2024-09-02T10:21:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:00 smithi151 ceph-mon[16639]: pgmap v1369: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 6505/185412 objects degraded (3.508%); 11416/185412 objects misplaced (6.157%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:00.465 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:00.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:00.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:00.594 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:00.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:00 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:21:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:00 smithi162 ceph-mon[20279]: osdmap e589: 12 total, 12 up, 12 in 2024-09-02T10:21:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:00 smithi162 ceph-mon[20279]: pgmap v1369: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 6505/185412 objects degraded (3.508%); 11416/185412 objects misplaced (6.157%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:00.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:00.665 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:21:00.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:00.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:00.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:00.745 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:00.821 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:00.850 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:00.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:00.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:00.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:00.979 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:00.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:00.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:01.081 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:01.102 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:01.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:01.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:01.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:01.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:01.233 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:01.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:01.251 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:21:01.251 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:21:01.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:01.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:01.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:01.359 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:01.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:01.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:01.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:01.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:01.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:01.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:01.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:01.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:01 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6505/182886 objects degraded (3.557%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:01.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:01.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:01.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:01.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:01 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6505/182886 objects degraded (3.557%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:01 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6505/182886 objects degraded (3.557%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:01.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:01.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:01.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:01.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:21:01.981 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:02.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:02.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:02.004 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:02.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:02.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:02.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:02.127 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:02.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:02.258 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:02.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:02.263 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:02.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:02.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:02.385 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:02.483 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:21:02.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:02.513 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:02.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:02.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:02.615 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:02.637 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:02.638 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:02 smithi151 ceph-mon[16639]: pgmap v1370: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s; 6505/182886 objects degraded (3.557%); 11416/182886 objects misplaced (6.242%); 3.8 MiB/s, 61 objects/s recovering 2024-09-02T10:21:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2772344142' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:21:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:02 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:21:02.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:02 smithi187 ceph-mon[20072]: pgmap v1370: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s; 6505/182886 objects degraded (3.557%); 11416/182886 objects misplaced (6.242%); 3.8 MiB/s, 61 objects/s recovering 2024-09-02T10:21:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2772344142' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:21:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:02 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:21:02.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:21:02.769 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:02.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:02.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:02 smithi162 ceph-mon[20279]: pgmap v1370: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s; 6505/182886 objects degraded (3.557%); 11416/182886 objects misplaced (6.242%); 3.8 MiB/s, 61 objects/s recovering 2024-09-02T10:21:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2772344142' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:21:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:02 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:21:02.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:02.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:02.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:03.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:03.047 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:03.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:03.118 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:21:03.118 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:21:03.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:03.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:03.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:03.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:03.291 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:03.318 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:03.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:03.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:03.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:03.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:03.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:03.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:03.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:03.576 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:03.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:03.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:03 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:03 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:21:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:03 smithi162 ceph-mon[20279]: osdmap e590: 12 total, 12 up, 12 in 2024-09-02T10:21:03.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:03 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:03 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:21:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:03 smithi151 ceph-mon[16639]: osdmap e590: 12 total, 12 up, 12 in 2024-09-02T10:21:03.708 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:03.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:03 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:03 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:21:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:03 smithi187 ceph-mon[20072]: osdmap e590: 12 total, 12 up, 12 in 2024-09-02T10:21:03.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:03.837 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:03.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:03.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:03.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:03.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:03.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:04.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:04.089 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:04.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:04.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:04.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:04.209 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:21:04.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-09-02T10:21:04.217 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:04.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:04.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:04.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:04.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:04.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:04.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:04.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:04.512 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:21:04.543 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:04.583 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:04.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:04.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:04.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:04.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:04.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:04 smithi187 ceph-mon[20072]: pgmap v1372: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s; 6504/177837 objects degraded (3.657%); 11274/177837 objects misplaced (6.340%); 5.2 MiB/s, 83 objects/s recovering 2024-09-02T10:21:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:04 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:21:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:04 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:21:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:04 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:21:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:04 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:21:04.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:04.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:04.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:04.777 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:04 smithi151 ceph-mon[16639]: pgmap v1372: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s; 6504/177837 objects degraded (3.657%); 11274/177837 objects misplaced (6.340%); 5.2 MiB/s, 83 objects/s recovering 2024-09-02T10:21:04.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:04 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:21:04.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:04 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:21:04.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:04 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:21:04.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:04 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:21:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:04 smithi162 ceph-mon[20279]: pgmap v1372: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 705 op/s; 6504/177837 objects degraded (3.657%); 11274/177837 objects misplaced (6.340%); 5.2 MiB/s, 83 objects/s recovering 2024-09-02T10:21:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:04 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:21:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:04 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:21:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:04 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:21:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:04 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:21:04.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:04.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:04.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:05.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:05.033 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:05.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:05.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:05.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:05.158 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:05.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:05.269 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:21:05.270 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:21:05.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:05.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:05.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:05.300 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:21:05.300 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:21:05.327 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:21:05.327 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:21:05.353 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:21:05.353 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:21:05.380 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:21:05.380 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:21:05.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:05.414 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:21:05.414 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:21:05.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:05.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:05.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:05.446 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:21:05.446 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:21:05.485 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:21:05.485 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:21:05.519 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:21:05.519 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:21:05.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:05.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:05.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:05.553 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:21:05.553 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:21:05.586 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:21:05.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:21:05.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:05 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:21:05.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:05 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:21:05.587 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3091925074' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:05.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:21:05.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:21:05.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:05.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:05.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:05 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:21:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:05 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:21:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3091925074' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:05.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:21:05.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:05 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:21:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:05 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:21:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3091925074' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:05.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:05.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:05.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:05.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:05.981 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:05.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:05.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:06.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:06.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:06.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:06.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:06.248 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:06.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:06.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:06.373 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:06.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:06.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:06.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:06.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:06.616 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:06.648 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:06.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:06.750 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:06 smithi187 ceph-mon[20072]: pgmap v1373: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s; 6549/175698 objects degraded (3.727%); 10877/175698 objects misplaced (6.191%); 6.4 MiB/s, 103 objects/s recovering 2024-09-02T10:21:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:06 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:21:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:06 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:21:06.773 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:06.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:06.780 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:06 smithi151 ceph-mon[16639]: pgmap v1373: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s; 6549/175698 objects degraded (3.727%); 10877/175698 objects misplaced (6.191%); 6.4 MiB/s, 103 objects/s recovering 2024-09-02T10:21:06.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:06 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:21:06.781 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:06 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:21:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:06 smithi162 ceph-mon[20279]: pgmap v1373: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 602 op/s; 6549/175698 objects degraded (3.727%); 10877/175698 objects misplaced (6.191%); 6.4 MiB/s, 103 objects/s recovering 2024-09-02T10:21:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:06 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:21:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:06 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:21:06.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:06.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:06.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:06.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:07.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:07.001 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:07.019 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:07.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:07.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:07.023 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:07.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:07.153 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:07.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:07.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:07.282 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:07.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:07.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:07.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:07.412 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:07.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:07.500 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-09-02T10:21:07.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:07.536 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:07.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:07.611 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:21:07.612 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:21:07.612 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:21:07.612 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:21:07.612 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:21:07.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:07.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:07.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:07.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2799525846' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:21:07.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:07.801 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:07.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2799525846' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:21:07.884 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2799525846' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:21:07.903 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:07.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:07.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:07.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:08.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:08.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:08.062 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:08.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:08.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:08.201 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:08.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:08.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:08.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:08.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:08.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:08.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:08.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:08.449 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:08.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:08 smithi151 ceph-mon[16639]: pgmap v1374: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 6549/172230 objects degraded (3.802%); 10738/172230 objects misplaced (6.235%); 7.3 MiB/s, 117 objects/s recovering 2024-09-02T10:21:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:08 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:21:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:08 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:21:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:08 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6549/172230 objects degraded (3.802%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:08.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2799525846' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:21:08.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:08 smithi151 ceph-mon[16639]: osdmap e591: 12 total, 12 up, 12 in 2024-09-02T10:21:08.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:08 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:21:08.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:08 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:21:08.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:08.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:08 smithi187 ceph-mon[20072]: pgmap v1374: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 6549/172230 objects degraded (3.802%); 10738/172230 objects misplaced (6.235%); 7.3 MiB/s, 117 objects/s recovering 2024-09-02T10:21:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:08 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:21:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:08 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:21:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:08 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6549/172230 objects degraded (3.802%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2799525846' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:21:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:08 smithi187 ceph-mon[20072]: osdmap e591: 12 total, 12 up, 12 in 2024-09-02T10:21:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:08 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:21:08.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:08 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:21:08.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:08.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:08.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:08 smithi162 ceph-mon[20279]: pgmap v1374: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 586 op/s; 6549/172230 objects degraded (3.802%); 10738/172230 objects misplaced (6.235%); 7.3 MiB/s, 117 objects/s recovering 2024-09-02T10:21:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:08 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:21:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:08 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:21:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:08 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6549/172230 objects degraded (3.802%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2799525846' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:21:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:08 smithi162 ceph-mon[20279]: osdmap e591: 12 total, 12 up, 12 in 2024-09-02T10:21:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:08 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:21:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:08 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:21:08.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:08.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:08.968 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:08.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:21:09.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:09.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:09.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:09.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:09.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:09.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:09.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:09.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:09.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:21:09.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:09.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:09.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:09.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:09.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2209417657' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:09.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:09 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:09.516 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:21:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:09.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:09.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:09.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:09.676 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:09.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2209417657' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:09 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:09.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:09.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:09.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:09.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2209417657' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:09 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:09.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:09.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:09.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:09.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-09-02T10:21:10.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:10.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:10.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:10.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:10.204 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:10.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:10.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:10.328 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:10.329 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:10.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:10.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:10.460 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:10.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:10.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:10.589 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:10.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:10 smithi151 ceph-mon[16639]: pgmap v1376: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 6595/167715 objects degraded (3.932%); 10366/167715 objects misplaced (6.181%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:21:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:10 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:21:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:10 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:10 smithi151 ceph-mon[16639]: osdmap e592: 12 total, 12 up, 12 in 2024-09-02T10:21:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:10 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:21:10.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:10.714 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:10.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:10.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:10 smithi187 ceph-mon[20072]: pgmap v1376: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 6595/167715 objects degraded (3.932%); 10366/167715 objects misplaced (6.181%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:21:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:10 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:21:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:10 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:10 smithi187 ceph-mon[20072]: osdmap e592: 12 total, 12 up, 12 in 2024-09-02T10:21:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:10 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:21:10.772 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:10.789 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:10.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:10 smithi162 ceph-mon[20279]: pgmap v1376: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 6595/167715 objects degraded (3.932%); 10366/167715 objects misplaced (6.181%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:21:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:10 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:21:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:10 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:10 smithi162 ceph-mon[20279]: osdmap e592: 12 total, 12 up, 12 in 2024-09-02T10:21:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:10 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:21:10.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:10.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:10.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:10.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:10.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:11.077 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:11.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:11.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:11.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:11.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:11.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:11.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:11.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:11.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:11.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:11.392 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:11.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:11.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:11.523 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:11.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:11.600 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:11 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:21:11.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:11.647 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:11.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:11.648 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:11.679 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:21:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:21:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:21:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:21:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:42144 4.b 4:d62c2ece:::benchmark_data_smithi187_76542_object42143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e354)", 2024-09-02T10:21:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:48.539483+0000", 2024-09-02T10:21:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 563.05819550800004, 2024-09-02T10:21:11.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.391093368, 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 42144 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539483+0000", 2024-09-02T10:21:11.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539483+0000", 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539485+0000", 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3039999999999999e-06 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539544+0000", 2024-09-02T10:21:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9614999999999998e-05 2024-09-02T10:21:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539557+0000", 2024-09-02T10:21:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2551000000000001e-05 2024-09-02T10:21:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539563+0000", 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4099999999999996e-06 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539618+0000", 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4221999999999997e-05 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539747+0000", 2024-09-02T10:21:11.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000129806 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.539861+0000", 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113622 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.541533+0000", 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001672394 2024-09-02T10:21:11.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.547909+0000", 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006375932 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.930451+0000", 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38254196499999998 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.930525+0000", 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3911000000000004e-05 2024-09-02T10:21:11.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.930577+0000", 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1635999999999998e-05 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:92326 4.b 4:d3b75331:::benchmark_data_smithi187_76542_object92325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:33.492476+0000", 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 518.10520273099996, 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79601677199999998, 2024-09-02T10:21:11.686 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 92326 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492476+0000", 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492476+0000", 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492478+0000", 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4219999999999999e-06 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492566+0000", 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8583000000000002e-05 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492587+0000", 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1084e-05 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492596+0000", 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1880000000000008e-06 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492668+0000", 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1876000000000001e-05 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492825+0000", 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015684100000000001 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.492969+0000", 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000143783 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.495907+0000", 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029384139999999999 2024-09-02T10:21:11.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.496430+0000", 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00052270400000000003 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288414+0000", 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79198343699999996 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288458+0000", 2024-09-02T10:21:11.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4030999999999999e-05 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288493+0000", 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5408999999999999e-05 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(5.8 479/463 [PushReplyOp(5:1936829f:::benchmark_data_smithi187_89405_object32750:head)])", 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 295.41917424500002, 2024-09-02T10:21:11.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36677579799999999, 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9100000000000004e-07 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178510+0000", 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.51e-06 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.543260+0000", 2024-09-02T10:21:11.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36475006799999998 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.543276+0000", 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5837000000000001e-05 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.543392+0000", 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115862 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.545280+0000", 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00188893 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25933.0:205325 5.e e524/431 5:726c3a45:::benchmark_data_smithi187_89405_object101720:head v 524'12690, mlcod=524'12688)", 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.618253+0000", 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 173.979425883, 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38099444199999999, 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.696 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618253+0000", 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618253+0000", 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618254+0000", 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1600000000000001e-07 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618258+0000", 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3320000000000002e-06 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995375+0000", 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37711745899999999 2024-09-02T10:21:11.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995380+0000", 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7330000000000003e-06 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995450+0000", 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9474000000000006e-05 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995475+0000", 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5579999999999999e-05 2024-09-02T10:21:11.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.999213+0000", 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003737476 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.999247+0000", 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4771999999999998e-05 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:207192 5.3 5:c539d868:::benchmark_data_smithi187_89405_object103587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:20.104200+0000", 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 171.49347918800001, 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52651428300000003, 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 207192 2024-09-02T10:21:11.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104200+0000", 2024-09-02T10:21:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104200+0000", 2024-09-02T10:21:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104201+0000", 2024-09-02T10:21:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-02T10:21:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104205+0000", 2024-09-02T10:21:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.98e-06 2024-09-02T10:21:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104213+0000", 2024-09-02T10:21:11.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7560000000000005e-06 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104218+0000", 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104263+0000", 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4708e-05 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104392+0000", 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128987 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104522+0000", 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012982799999999999 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.109243+0000", 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047210799999999999 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.109411+0000", 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016770899999999999 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630621+0000", 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52120988899999998 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630666+0000", 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4883999999999999e-05 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630714+0000", 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8497e-05 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25933.0:207192 5.3 e525/431 ondisk, result = 0)", 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:20.108691+0000", 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 171.488987485, 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52202865700000001, 2024-09-02T10:21:11.708 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108691+0000", 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108691+0000", 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108692+0000", 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9100000000000004e-07 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108695+0000", 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.529e-06 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630413+0000", 2024-09-02T10:21:11.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52171714700000005 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630428+0000", 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5843999999999999e-05 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630579+0000", 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000150252 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630598+0000", 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8865999999999999e-05 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630720+0000", 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122528 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:19723 6.9 6:94e8ee31:::benchmark_data_smithi187_102448_object19722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:53.705822+0000", 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 137.89185660499999, 2024-09-02T10:21:11.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21665701200000001, 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 19723 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705822+0000", 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705822+0000", 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705823+0000", 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.082e-06 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705916+0000", 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2105e-05 2024-09-02T10:21:11.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705929+0000", 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.397e-05 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705937+0000", 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7330000000000003e-06 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705981+0000", 2024-09-02T10:21:11.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3949999999999998e-05 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.706119+0000", 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013788900000000001 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.706208+0000", 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9197000000000007e-05 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.707009+0000", 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080078799999999998 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.714056+0000", 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070474170000000003 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922382+0000", 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20832547500000001 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922434+0000", 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2473999999999999e-05 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922479+0000", 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4932000000000003e-05 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:19732 6.9 6:9f276e57:::benchmark_data_smithi187_102448_object19731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:53.712327+0000", 2024-09-02T10:21:11.718 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 137.88535225199999, 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.216236916, 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 19732 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712327+0000", 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712327+0000", 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712328+0000", 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0449999999999999e-06 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712379+0000", 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0779999999999998e-05 2024-09-02T10:21:11.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712391+0000", 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2085e-05 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712398+0000", 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7540000000000006e-06 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712448+0000", 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9880999999999999e-05 2024-09-02T10:21:11.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712542+0000", 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3450999999999997e-05 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712634+0000", 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2317000000000004e-05 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713395+0000", 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00076103200000000003 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.787692+0000", 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.074297236000000003 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.928492+0000", 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14080002899999999 2024-09-02T10:21:11.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.928534+0000", 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1220000000000002e-05 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.928564+0000", 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0086e-05 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26524.0:19723 6.9 e543/533 ondisk, result = 0)", 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:53.713901+0000", 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 137.883778401, 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20865452900000001, 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713901+0000", 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713901+0000", 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713901+0000", 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0699999999999997e-07 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713904+0000", 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8420000000000001e-06 2024-09-02T10:21:11.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922307+0000", 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.208403057 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922312+0000", 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0849999999999996e-06 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922372+0000", 2024-09-02T10:21:11.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9664000000000003e-05 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922378+0000", 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9989999999999997e-06 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922555+0000", 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017727500000000001 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.728 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:39567 6.0 6:0ea24a11:::benchmark_data_smithi187_102448_object39566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:11.107391+0000", 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 120.49028759300001, 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26867963700000003, 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:11.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 39567 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107391+0000", 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107391+0000", 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107393+0000", 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.341e-06 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107430+0000", 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7738999999999999e-05 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107447+0000", 2024-09-02T10:21:11.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6472000000000001e-05 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107456+0000", 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6279999999999994e-06 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107520+0000", 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4442000000000004e-05 2024-09-02T10:21:11.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107677+0000", 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015673999999999999 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107865+0000", 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018834400000000001 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.109307+0000", 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014422720000000001 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.109639+0000", 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033185300000000001 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.375963+0000", 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.266324318 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.376022+0000", 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8971999999999997e-05 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.376071+0000", 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8516000000000001e-05 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.735 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:54769 6.9 6:9e64a741:::benchmark_data_smithi187_102448_object54768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-09-02T10:21:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:24.601031+0000", 2024-09-02T10:21:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 106.99664788299999, 2024-09-02T10:21:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21469951200000001, 2024-09-02T10:21:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:11.736 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 54769 2024-09-02T10:21:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601031+0000", 2024-09-02T10:21:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601031+0000", 2024-09-02T10:21:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601032+0000", 2024-09-02T10:21:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3960000000000001e-06 2024-09-02T10:21:11.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601100+0000", 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7464000000000003e-05 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601111+0000", 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1240000000000001e-05 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601119+0000", 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4739999999999999e-06 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601187+0000", 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8659999999999997e-05 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601349+0000", 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000161991 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601506+0000", 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000157009 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.605407+0000", 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039008810000000001 2024-09-02T10:21:11.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.605419+0000", 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1984000000000001e-05 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815640+0000", 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21022085700000001 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815682+0000", 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2066e-05 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815731+0000", 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8489999999999998e-05 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.743 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:54772 6.9 6:97f3a42a:::benchmark_data_smithi187_102448_object54771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:24.601836+0000", 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 106.995842806, 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21406335900000001, 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 54772 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601836+0000", 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601836+0000", 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601837+0000", 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6300000000000004e-07 2024-09-02T10:21:11.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601910+0000", 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2541999999999998e-05 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601924+0000", 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4063000000000001e-05 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601931+0000", 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0509999999999999e-06 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.603340+0000", 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001409039 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.603478+0000", 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013832499999999999 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.603655+0000", 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017739000000000001 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.605744+0000", 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020888360000000002 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.606900+0000", 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001155339 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815803+0000", 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20890336700000001 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815861+0000", 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8230999999999999e-05 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815899+0000", 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8312999999999997e-05 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:135986 6.0 6:053ae704:::benchmark_data_smithi187_102448_object35714:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:21:11.750 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135293+0000", 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.462385861000001, 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32009341200000002, 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135986 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135293+0000", 2024-09-02T10:21:11.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135293+0000", 2024-09-02T10:21:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135295+0000", 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.894e-06 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135302+0000", 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8249999999999999e-06 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135316+0000", 2024-09-02T10:21:11.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4620000000000001e-05 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135326+0000", 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5759999999999995e-06 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136831+0000", 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015045989999999999 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136843+0000", 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2804e-05 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.435574+0000", 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29873042100000002 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.435695+0000", 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012073 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.435805+0000", 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011039099999999999 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.442308+0000", 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065034020000000001 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.447422+0000", 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051138219999999996 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.452276+0000", 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048543129999999999 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.452357+0000", 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0165000000000002e-05 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455386+0000", 2024-09-02T10:21:11.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030298500000000002 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:135994 6.0 6:0fdb7607:::benchmark_data_smithi187_102448_object35722:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135349+0000", 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.462329585999999, 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34087867300000002, 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.758 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135994 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135349+0000", 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135349+0000", 2024-09-02T10:21:11.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135350+0000", 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0400000000000002e-07 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135352+0000", 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3489999999999999e-06 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135357+0000", 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135361+0000", 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3759999999999999e-06 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136882+0000", 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015210169999999999 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136890+0000", 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.225e-06 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.442387+0000", 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30549738100000001 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.442561+0000", 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017320399999999999 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.442754+0000", 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019324399999999999 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455410+0000", 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012656499 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455455+0000", 2024-09-02T10:21:11.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4912e-05 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.472639+0000", 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017184003999999999 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.472712+0000", 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3100000000000001e-05 2024-09-02T10:21:11.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.476228+0000", 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035155809999999998 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:136020 6.0 6:06e73c6f:::benchmark_data_smithi187_102448_object35748:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135365+0000", 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.462313498, 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34402223500000001, 2024-09-02T10:21:11.765 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136020 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135365+0000", 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135365+0000", 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135366+0000", 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3999999999999998e-07 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135367+0000", 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.367e-06 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135372+0000", 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5120000000000002e-06 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135375+0000", 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9809999999999999e-06 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136925+0000", 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015500749999999999 2024-09-02T10:21:11.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:21:11.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136931+0000", 2024-09-02T10:21:11.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2260000000000002e-06 2024-09-02T10:21:11.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.447464+0000", 2024-09-02T10:21:11.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31053328699999999 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.447637+0000", 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017254299999999999 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.447810+0000", 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000173623 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455501+0000", 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0076903800000000001 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455514+0000", 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3022e-05 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.476298+0000", 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020784764000000001 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.476368+0000", 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9745000000000004e-05 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.479388+0000", 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030194699999999998 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.772 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:136029 6.0 6:0a2545ae:::benchmark_data_smithi187_102448_object35757:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.462300791000001, 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34708892600000002, 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136029 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:21:11.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1500000000000001e-07 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135380+0000", 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2359999999999999e-06 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135383+0000", 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.641e-06 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135386+0000", 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7920000000000001e-06 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136954+0000", 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001567492 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136959+0000", 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.49e-06 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455542+0000", 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31858257099999998 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455661+0000", 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119422 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455779+0000", 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011832699999999999 2024-09-02T10:21:11.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.464952+0000", 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091722350000000008 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.465554+0000", 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060224400000000004 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.479463+0000", 2024-09-02T10:21:11.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013909425 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.479532+0000", 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8818000000000004e-05 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.482467+0000", 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029350180000000002 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:136038 6.0 6:0e293183:::benchmark_data_smithi187_102448_object35766:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135389+0000", 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 18.462289553000002, 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34942438399999998, 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:11.780 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136038 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135389+0000", 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135389+0000", 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135390+0000", 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1299999999999999e-07 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135391+0000", 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.249e-06 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135395+0000", 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2150000000000003e-06 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135399+0000", 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4649999999999999e-06 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.137534+0000", 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021359560000000001 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.137545+0000", 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0312999999999999e-05 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.460231+0000", 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32268618199999999 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.460387+0000", 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015636900000000001 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:21:11.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.460536+0000", 2024-09-02T10:21:11.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014892599999999999 2024-09-02T10:21:11.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.465596+0000", 2024-09-02T10:21:11.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050595570000000001 2024-09-02T10:21:11.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.465617+0000", 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1362e-05 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.482532+0000", 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016914565999999999 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.482602+0000", 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9809999999999997e-05 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.484814+0000", 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002212201 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:144155 6.a e590/533 6:53bb1869:::benchmark_data_smithi187_102448_object43883:head v 590'8984, mlcod=590'8979)", 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:06.775838+0000", 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8218407399999998, 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.214603619, 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.787 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.775838+0000", 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.775838+0000", 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.775839+0000", 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2500000000000004e-07 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.775844+0000", 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2599999999999996e-06 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.981434+0000", 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20558944700000001 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.981442+0000", 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3999999999999992e-06 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.981528+0000", 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5721000000000001e-05 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.981551+0000", 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3544999999999999e-05 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.990342+0000", 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087902310000000008 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.990442+0000", 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010019 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:144161 6.a e590/533 6:52a794b3:::benchmark_data_smithi187_102448_object43889:head v 590'8985, mlcod=590'8980)", 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:06.783206+0000", 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.8144729589999997, 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.212957436, 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.783206+0000", 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.783206+0000", 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.783207+0000", 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7199999999999998e-07 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.783212+0000", 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4829999999999999e-06 2024-09-02T10:21:11.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.991155+0000", 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20794227500000001 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.991163+0000", 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3289999999999994e-06 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.991223+0000", 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0111000000000001e-05 2024-09-02T10:21:11.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.991244+0000", 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1484999999999998e-05 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.996076+0000", 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048314170000000002 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.996163+0000", 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7564000000000004e-05 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:147157 6.a e592/533 6:5353ac10:::benchmark_data_smithi187_102448_object46885:head v 592'9126, mlcod=592'9124)", 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:11.520293+0000", 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.077385627999999998, 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037332486999999998, 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:11.796 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:11.520293+0000", 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:11.520293+0000", 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:11.520294+0000", 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.133e-06 2024-09-02T10:21:11.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:11.520304+0000", 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3519999999999996e-06 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:11.550973+0000", 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.030669035000000001 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:11.550982+0000", 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3190000000000001e-06 2024-09-02T10:21:11.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:11.551045+0000", 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3242000000000002e-05 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:11.551068+0000", 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2385999999999999e-05 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:11.557571+0000", 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006502992 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:11.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:11.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:11.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:11.557626+0000", 2024-09-02T10:21:11.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5028000000000002e-05 2024-09-02T10:21:11.800 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.800 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.800 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.800 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:11.800 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:11.800 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:21:11.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:11.831 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:11.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:11.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:11 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:21:11.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:11.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:11.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:11.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:12.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:11 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:21:12.056 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:12.082 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:12.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:12.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:12.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:12.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:12.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:12.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:12.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:12.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:12.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:12.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:12.580 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:12.603 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:12.604 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:12.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:12.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:12.728 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:12.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:21:12.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:12 smithi151 ceph-mon[16639]: pgmap v1378: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 6595/165348 objects degraded (3.989%); 10366/165348 objects misplaced (6.269%); 6.3 MiB/s, 102 objects/s recovering 2024-09-02T10:21:12.794 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4130427996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:12.828 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:12 smithi162 ceph-mon[20279]: pgmap v1378: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 6595/165348 objects degraded (3.989%); 10366/165348 objects misplaced (6.269%); 6.3 MiB/s, 102 objects/s recovering 2024-09-02T10:21:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4130427996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:12.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:12.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:12.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:12.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:21:12.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:12.988 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:12.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:12.988 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:21:12.988 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:21:12.988 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:21:12.988 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:21:12.989 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:21:12.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:21:12.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:21:12.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:21:12.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:21:12.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:21:12.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:21:12.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:21:12.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:21:12.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:21:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:12 smithi187 ceph-mon[20072]: pgmap v1378: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 519 op/s; 6595/165348 objects degraded (3.989%); 10366/165348 objects misplaced (6.269%); 6.3 MiB/s, 102 objects/s recovering 2024-09-02T10:21:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4130427996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:13.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:13.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:13.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:13.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:13.249 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:13.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:13.250 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:13.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:13.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:13.373 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:13.474 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:13.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:13.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:13.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:13.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:13.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:13.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:13.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:13 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6594/160020 objects degraded (4.121%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:13.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:13.753 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:13.765 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:13.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:13.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:13 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6594/160020 objects degraded (4.121%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:13.888 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:13.916 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:13.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:13 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6594/160020 objects degraded (4.121%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:14.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:14.041 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:14.041 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:14.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:14.175 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:14.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:14.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:14.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:14.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:14.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:14.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:14.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:14.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:14.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:14.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:14.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:14.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:14.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:14.656 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:14.684 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:14.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:14.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:14.816 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:14.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:14.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:14 smithi162 ceph-mon[20279]: pgmap v1379: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s; 6594/160020 objects degraded (4.121%); 10193/160020 objects misplaced (6.370%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:14.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:14.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:14 smithi151 ceph-mon[16639]: pgmap v1379: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s; 6594/160020 objects degraded (4.121%); 10193/160020 objects misplaced (6.370%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:14.941 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:14.942 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:14 smithi187 ceph-mon[20072]: pgmap v1379: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s; 6594/160020 objects degraded (4.121%); 10193/160020 objects misplaced (6.370%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:15.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:15.069 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:15.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:15.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:15.171 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:15.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:15.194 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:15.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:15.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:15.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:15.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:15.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:15.452 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:15.452 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:15.468 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:21:15.468 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:21:15.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:15.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:15.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:15.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:15.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:15.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:15.703 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:15.703 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:15.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:21:15.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:15.830 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:15.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:15.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:15.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:15.967 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:15.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:16.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:16.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:16.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:16.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:16.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:16.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:16.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:16.352 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:16.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:16.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:16.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:16.476 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:16.570 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-02T10:21:16.577 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:16.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:16.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:16.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:16.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:16.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:16.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:16 smithi162 ceph-mon[20279]: pgmap v1380: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s; 6686/157671 objects degraded (4.240%); 9786/157671 objects misplaced (6.207%); 6.3 MiB/s, 101 objects/s recovering 2024-09-02T10:21:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:16 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:21:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:16 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:21:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2448350659' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:21:16.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:16.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:16.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:16 smithi151 ceph-mon[16639]: pgmap v1380: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s; 6686/157671 objects degraded (4.240%); 9786/157671 objects misplaced (6.207%); 6.3 MiB/s, 101 objects/s recovering 2024-09-02T10:21:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:16 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:21:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:16 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:21:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2448350659' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:21:16.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:17.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:17.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:16 smithi187 ceph-mon[20072]: pgmap v1380: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 606 op/s; 6686/157671 objects degraded (4.240%); 9786/157671 objects misplaced (6.207%); 6.3 MiB/s, 101 objects/s recovering 2024-09-02T10:21:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:16 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:21:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:16 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:21:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2448350659' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:21:17.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:21:17.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:17.131 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:17.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:17.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:17.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:17.262 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:17.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:17.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:17.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:17.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:17.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:17.522 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:17.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:17.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:17.593 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:21:17.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:17.645 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:17.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:17.646 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:17.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:17.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:17.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:17.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2448350659' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:21:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:17 smithi162 ceph-mon[20279]: osdmap e593: 12 total, 12 up, 12 in 2024-09-02T10:21:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/452340208' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:17 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:17.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:17.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2448350659' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:21:17.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:17 smithi151 ceph-mon[16639]: osdmap e593: 12 total, 12 up, 12 in 2024-09-02T10:21:17.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/452340208' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:17.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:17 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:17.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:17.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:17.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:17.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-09-02T10:21:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2448350659' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:21:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:17 smithi187 ceph-mon[20072]: osdmap e593: 12 total, 12 up, 12 in 2024-09-02T10:21:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/452340208' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:17 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:18.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:18.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:18.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:18.036 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:18.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:18.163 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:18.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:18.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:18.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:18.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:18.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:18.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:18.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:18.440 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:18.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:18.589 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:18.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:18.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:18.718 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:18.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:18.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:18 smithi162 ceph-mon[20279]: pgmap v1382: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 6686/153477 objects degraded (4.356%); 9640/153477 objects misplaced (6.281%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:18 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:18 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:21:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:18 smithi162 ceph-mon[20279]: osdmap e594: 12 total, 12 up, 12 in 2024-09-02T10:21:18.846 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:18.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:18.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:18.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:18 smithi151 ceph-mon[16639]: pgmap v1382: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 6686/153477 objects degraded (4.356%); 9640/153477 objects misplaced (6.281%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:18.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:18 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:18.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:18 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:21:18.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:18 smithi151 ceph-mon[16639]: osdmap e594: 12 total, 12 up, 12 in 2024-09-02T10:21:18.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:18.978 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:18.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:18.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:18 smithi187 ceph-mon[20072]: pgmap v1382: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 593 op/s; 6686/153477 objects degraded (4.356%); 9640/153477 objects misplaced (6.281%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:18 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:18 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:21:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:18 smithi187 ceph-mon[20072]: osdmap e594: 12 total, 12 up, 12 in 2024-09-02T10:21:19.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:19.113 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:19.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:19.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:19.215 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:19.216 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:19.217 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:19.241 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:19.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:19.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:19.283 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:21:19.283 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:21:19.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:21:19.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:21:19.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:19.349 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:21:19.349 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:21:19.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:19.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:19.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:19.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:21:19.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:21:19.415 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:21:19.415 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:21:19.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:19.496 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:19.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:19.598 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:19.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:19.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:19.651 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:21:19.652 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:21:19.688 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:21:19.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:21:19.720 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:21:19.720 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:21:19.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:19.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:19.746 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:19.756 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:21:19.756 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:21:19.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:21:19.786 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:21:19.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6686/153477 objects degraded (4.356%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:19.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/463908987' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:19.825 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:21:19.825 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:21:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6686/153477 objects degraded (4.356%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/463908987' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:19.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:19.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:21:19.862 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:21:19.881 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:19.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:19.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6686/153477 objects degraded (4.356%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/463908987' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:20.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:20.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:20.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:20.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:20.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:20.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:20.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:20.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:20.291 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:20.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:20.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:20.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:20.424 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:20.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:20.526 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:20.547 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:21:20.547 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:21:20.548 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:21:20.548 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:21:20.548 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:21:20.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:20.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:20.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:20.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:20.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:21:20.679 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:20.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:20.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:20.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:20 smithi151 ceph-mon[16639]: pgmap v1384: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s; 6685/150087 objects degraded (4.454%); 9640/150087 objects misplaced (6.423%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1763657201' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:21:20.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:20.813 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:20.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:20 smithi162 ceph-mon[20279]: pgmap v1384: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s; 6685/150087 objects degraded (4.454%); 9640/150087 objects misplaced (6.423%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1763657201' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:21:20.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:20.946 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:20.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:20.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:20 smithi187 ceph-mon[20072]: pgmap v1384: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 635 op/s; 6685/150087 objects degraded (4.454%); 9640/150087 objects misplaced (6.423%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:21:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1763657201' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:21:21.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:21.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:21.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:21.113 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-02T10:21:21.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:21.210 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:21.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:21.312 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:21.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:21.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:21.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:21.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:21.466 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:21.466 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:21.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:21.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:21.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:21.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:21.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:21.729 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:21.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:21.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:21.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:21.855 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:21.856 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:21.957 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:21.989 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:21.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:22.092 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:22.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:21:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:22.125 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:22.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:22.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:22.248 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:22.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:22.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:22.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1763657201' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:21:22.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:22 smithi151 ceph-mon[16639]: osdmap e595: 12 total, 12 up, 12 in 2024-09-02T10:21:22.312 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:22 smithi151 ceph-mon[16639]: pgmap v1386: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 6744/146493 objects degraded (4.604%); 9249/146493 objects misplaced (6.314%); 5.0 MiB/s, 79 objects/s recovering 2024-09-02T10:21:22.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:22.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:22.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:22.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:22.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:22.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:22.499 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1763657201' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:21:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:22 smithi187 ceph-mon[20072]: osdmap e595: 12 total, 12 up, 12 in 2024-09-02T10:21:22.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:22 smithi187 ceph-mon[20072]: pgmap v1386: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 6744/146493 objects degraded (4.604%); 9249/146493 objects misplaced (6.314%); 5.0 MiB/s, 79 objects/s recovering 2024-09-02T10:21:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1763657201' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:21:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:22 smithi162 ceph-mon[20279]: osdmap e595: 12 total, 12 up, 12 in 2024-09-02T10:21:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:22 smithi162 ceph-mon[20279]: pgmap v1386: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 620 op/s; 6744/146493 objects degraded (4.604%); 9249/146493 objects misplaced (6.314%); 5.0 MiB/s, 79 objects/s recovering 2024-09-02T10:21:22.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:22.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:22.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:22.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:22.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:22.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:22.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:22.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-09-02T10:21:22.871 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:22.903 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:22.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:23.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:23.031 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:23.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:23.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:23.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:23.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:23.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:23.157 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:23.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:23.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:23.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:23.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:23.346 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:23.348 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:23.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:23.413 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:23.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:23.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:23.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:23.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:23.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:23.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:23.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:23.713 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:23.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:23.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:23.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:23.951 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:23.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:23.974 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:24.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:24.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:24.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:24.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:24.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:24.236 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:24.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:24.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:24.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:24.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:24.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:24.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:24.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:24.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:24.595 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:21:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:21:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:21:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:21:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:41912 4.f 4:f03c9d73:::benchmark_data_smithi187_76542_object41911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e353)", 2024-09-02T10:21:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.767259+0000", 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 576.67577052900003, 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52722899400000001, 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 41912 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767259+0000", 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767259+0000", 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767260+0000", 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2999999999999999e-07 2024-09-02T10:21:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767309+0000", 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8523999999999998e-05 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767321+0000", 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2687999999999999e-05 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767331+0000", 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.01e-05 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767384+0000", 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3012e-05 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767516+0000", 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000131166 2024-09-02T10:21:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:21:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.767648+0000", 2024-09-02T10:21:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000132106 2024-09-02T10:21:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.768976+0000", 2024-09-02T10:21:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013279559999999999 2024-09-02T10:21:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.774494+0000", 2024-09-02T10:21:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0055181090000000002 2024-09-02T10:21:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294421+0000", 2024-09-02T10:21:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51992769000000005 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294459+0000", 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7832e-05 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.294488+0000", 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8980999999999999e-05 2024-09-02T10:21:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(4.18 353/344 [PushOp(4:1c2dd2e1:::benchmark_data_smithi187_76542_object24960:head, version: 335'1638, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(4:1c2dd2e1:::benchmark_data_smithi187_76542_object24960:head@335'1638, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 576.66613679399995, 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29217855399999998, 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.776893+0000", 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8299999999999997e-07 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.777026+0000", 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013260400000000001 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068430+0000", 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29140371700000001 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068439+0000", 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2909999999999996e-06 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068496+0000", 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6597e-05 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.068504+0000", 2024-09-02T10:21:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2199999999999992e-06 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.069072+0000", 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056764200000000004 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:44135 4.f 4:fa3c3350:::benchmark_data_smithi187_76542_object44134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e356)", 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:50.893955+0000", 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 573.54907451999998, 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32054305900000002, 2024-09-02T10:21:24.607 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 44135 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.893955+0000", 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.893955+0000", 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.893957+0000", 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.604e-06 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894004+0000", 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6853e-05 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894015+0000", 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.129e-05 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894021+0000", 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6049999999999997e-06 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894061+0000", 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0114999999999998e-05 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894142+0000", 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1724000000000006e-05 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.894228+0000", 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5729000000000005e-05 2024-09-02T10:21:24.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.895543+0000", 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001314747 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900651+0000", 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051081329999999999 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.214312+0000", 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31366078800000002 2024-09-02T10:21:24.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.214448+0000", 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013631299999999999 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.214498+0000", 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0158000000000003e-05 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(4.16 356/345 [PushReplyOp(4:6d00321c:::benchmark_data_smithi187_76542_object7271:head)])", 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:50.900564+0000", 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 573.54246567899997, 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31795428799999997, 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900564+0000", 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900564+0000", 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900565+0000", 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3e-07 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:50.900569+0000", 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7049999999999998e-06 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.217556+0000", 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31698648099999999 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.217564+0000", 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.943e-06 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.217625+0000", 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0806e-05 2024-09-02T10:21:24.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:51.218518+0000", 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00089372300000000002 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90780 4.f 4:f9a425ff:::benchmark_data_smithi187_76542_object90779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.016735+0000", 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 532.42629455400004, 2024-09-02T10:21:24.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28756859099999998, 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90780 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016735+0000", 2024-09-02T10:21:24.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016735+0000", 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016736+0000", 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.540000000000001e-07 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016790+0000", 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3933e-05 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016802+0000", 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1605e-05 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016807+0000", 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0869999999999997e-06 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016846+0000", 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9502000000000002e-05 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.016939+0000", 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2520000000000002e-05 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.017047+0000", 2024-09-02T10:21:24.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010813400000000001 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.020060+0000", 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030132090000000002 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.031359+0000", 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011299346 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304201+0000", 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.272841522 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304259+0000", 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7732000000000002e-05 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304304+0000", 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5046999999999997e-05 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.623 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90797 4.f 4:f65eb26e:::benchmark_data_smithi187_76542_object90796:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.028192+0000", 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 532.41483794199996, 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.276265435, 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:24.624 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90797 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028192+0000", 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028192+0000", 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028193+0000", 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3740000000000001e-06 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028251+0000", 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8097000000000003e-05 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028264+0000", 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2223e-05 2024-09-02T10:21:24.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028270+0000", 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6290000000000004e-06 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028310+0000", 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9527000000000003e-05 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028410+0000", 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9959999999999998e-05 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.028501+0000", 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0976000000000004e-05 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.030870+0000", 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002369491 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034987+0000", 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041169179999999998 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304357+0000", 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26936986000000002 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304419+0000", 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1709000000000002e-05 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304457+0000", 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8671000000000003e-05 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90800 4.f 4:f0d21d3f:::benchmark_data_smithi187_76542_object90799:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e376)", 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.032169+0000", 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 532.41086105500005, 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27243329199999999, 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90800 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032169+0000", 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032169+0000", 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032170+0000", 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7699999999999992e-07 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032221+0000", 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1365e-05 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032233+0000", 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2102e-05 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032240+0000", 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6669999999999999e-06 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032277+0000", 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7024000000000003e-05 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032359+0000", 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2163999999999995e-05 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.032440+0000", 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0870000000000003e-05 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034544+0000", 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021037590000000002 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300042+0000", 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26549787899999999 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304502+0000", 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044609840000000003 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304561+0000", 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8257999999999997e-05 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.304602+0000", 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1242999999999998e-05 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:90800 4.f e376/318 ondisk, result = 0)", 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 532.40809997899999, 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26511866899999997, 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034930+0000", 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8500000000000002e-07 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.034933+0000", 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2229999999999999e-06 2024-09-02T10:21:24.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.298510+0000", 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26357608700000001 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.298516+0000", 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.657e-06 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300029+0000", 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015128190000000001 2024-09-02T10:21:24.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300036+0000", 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9280000000000004e-06 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:32.300048+0000", 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2469999999999999e-05 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:92326 4.b e377/337 4:d3b75331:::benchmark_data_smithi187_76542_object92325:head v 377'5809, mlcod=377'5807)", 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:33.493173+0000", 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 530.94985678399996, 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79507954800000002, 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.493173+0000", 2024-09-02T10:21:24.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.493173+0000", 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.493174+0000", 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6499999999999998e-07 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:33.493198+0000", 2024-09-02T10:21:24.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4283000000000001e-05 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.285696+0000", 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.79249790399999998 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.285708+0000", 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2272e-05 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.285766+0000", 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7948000000000002e-05 2024-09-02T10:21:24.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.285786+0000", 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9712e-05 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288220+0000", 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024338620000000002 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:34.288253+0000", 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2702e-05 2024-09-02T10:21:24.643 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:103703 4.9 4:93bfdaf4:::benchmark_data_smithi187_76542_object11367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:46.146088+0000", 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 518.29694133400005, 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29017199100000002, 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:24.644 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 103703 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146088+0000", 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146088+0000", 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146090+0000", 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1170000000000001e-06 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146094+0000", 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6129999999999997e-06 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146104+0000", 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0169999999999999e-05 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146110+0000", 2024-09-02T10:21:24.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5199999999999997e-06 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146157+0000", 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7511e-05 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146239+0000", 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1837999999999998e-05 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.146304+0000", 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4709000000000007e-05 2024-09-02T10:21:24.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.147684+0000", 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013800970000000001 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.148414+0000", 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00073035299999999999 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.436148+0000", 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28773412900000001 2024-09-02T10:21:24.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.436211+0000", 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2335000000000005e-05 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.436260+0000", 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9598999999999997e-05 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:103765 4.9 4:90c7bfbd:::benchmark_data_smithi187_76542_object11429:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:46.738804+0000", 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 517.70422622800004, 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26836812300000001, 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 103765 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738804+0000", 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738804+0000", 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738805+0000", 2024-09-02T10:21:24.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.579e-06 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738810+0000", 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5310000000000003e-06 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738820+0000", 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0132000000000001e-05 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738826+0000", 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4529999999999999e-06 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.738883+0000", 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6373999999999999e-05 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.739018+0000", 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013532799999999999 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.739147+0000", 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012891899999999999 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.740688+0000", 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015409 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:46.740758+0000", 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0320000000000004e-05 2024-09-02T10:21:24.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:47.007109+0000", 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26635045299999999 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:47.007142+0000", 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3821e-05 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:47.007172+0000", 2024-09-02T10:21:24.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9312999999999999e-05 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:166621 4.f 4:f82e681d:::benchmark_data_smithi187_76542_object74285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:01.187965+0000", 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 443.25506462300001, 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26233720799999999, 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 166621 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187965+0000", 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187965+0000", 2024-09-02T10:21:24.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187966+0000", 2024-09-02T10:21:24.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2589999999999999e-06 2024-09-02T10:21:24.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187970+0000", 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4230000000000001e-06 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187979+0000", 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2830000000000007e-06 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.187984+0000", 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9060000000000001e-06 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.188024+0000", 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9814999999999997e-05 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.188095+0000", 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1250999999999999e-05 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.188161+0000", 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6068000000000005e-05 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.192572+0000", 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044111189999999998 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.201485+0000", 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089132410000000006 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450212+0000", 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24872681999999999 2024-09-02T10:21:24.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450263+0000", 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0427999999999999e-05 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450302+0000", 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9595000000000003e-05 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.664 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:166625 4.f 4:fff404f6:::benchmark_data_smithi187_76542_object74289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:14:01.194131+0000", 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 443.248899107, 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.256297687, 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 166625 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194131+0000", 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194131+0000", 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194132+0000", 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1769999999999999e-06 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194143+0000", 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.155e-05 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194154+0000", 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1124999999999999e-05 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194162+0000", 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7400000000000004e-06 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194203+0000", 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1207e-05 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194272+0000", 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9036999999999996e-05 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.194347+0000", 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4309999999999998e-05 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.198684+0000", 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043371069999999998 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.425961+0000", 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22727677499999999 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450340+0000", 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.024379764000000002 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450388+0000", 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7160000000000002e-05 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:14:01.450428+0000", 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0735000000000002e-05 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.670 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25933.0:62902 5.0 e462/431 5:0a3860dd:::benchmark_data_smithi187_89405_object62901:head v 462'3865, mlcod=462'3862)", 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:15:34.635547+0000", 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 349.80748323900002, 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28423237499999998, 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.635547+0000", 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.635547+0000", 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.635547+0000", 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5799999999999998e-07 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.635575+0000", 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7557999999999999e-05 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.917385+0000", 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28181001999999999 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.917393+0000", 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.571e-06 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.917450+0000", 2024-09-02T10:21:24.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7225000000000002e-05 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.917471+0000", 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1362999999999999e-05 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.919731+0000", 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002259629 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:34.919779+0000", 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8151e-05 2024-09-02T10:21:24.674 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:103601 5.c 5:3323d659:::benchmark_data_smithi187_89405_object103600:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e479)", 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:16.164978+0000", 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 308.27805166899998, 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52291455399999998, 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:21:24.675 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 103601 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.164978+0000", 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.164978+0000", 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.164980+0000", 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.556e-06 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165044+0000", 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4633000000000003e-05 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165057+0000", 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2872e-05 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165066+0000", 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6139999999999992e-06 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165137+0000", 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1643000000000005e-05 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165270+0000", 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013258200000000001 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:21:24.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.165426+0000", 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015579 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.175521+0000", 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010095550999999999 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.175557+0000", 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5237999999999997e-05 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687808+0000", 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51225187800000005 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687852+0000", 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3550000000000001e-05 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687893+0000", 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0646999999999999e-05 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25933.0:103601 5.c e479/455 ondisk, result = 0)", 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:16.178712+0000", 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 308.26431771699998, 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50918673599999997, 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178712+0000", 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178712+0000", 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178713+0000", 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.37e-07 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178716+0000", 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.822e-06 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687751+0000", 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50903458599999996 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687756+0000", 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3990000000000003e-06 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687799+0000", 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2830999999999997e-05 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687805+0000", 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.558e-06 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.687899+0000", 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4003000000000001e-05 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "rep_scrubmap(5.c e455 from shard 0)", 2024-09-02T10:21:24.684 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:16.358849+0000", 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 308.084181129, 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.73020200300000004, 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.358849+0000", 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.358849+0000", 2024-09-02T10:21:24.685 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.358849+0000", 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9999999999999997e-07 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.358868+0000", 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8343999999999999e-05 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:17.088882+0000", 2024-09-02T10:21:24.686 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.73001465499999996 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:17.088886+0000", 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0319999999999997e-06 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:17.088925+0000", 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8992000000000002e-05 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:17.088933+0000", 2024-09-02T10:21:24.687 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4569999999999999e-06 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:17.089051+0000", 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117823 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:207205 5.c 5:3323d659:::benchmark_data_smithi187_89405_object103600:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:20.127850+0000", 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 184.31517950099999, 2024-09-02T10:21:24.688 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21962476, 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 207205 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127850+0000", 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.689 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127850+0000", 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127853+0000", 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.385e-06 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127863+0000", 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0377000000000001e-05 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127884+0000", 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1361000000000001e-05 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127896+0000", 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1559999999999999e-05 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.127960+0000", 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3800000000000006e-05 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.691 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.128066+0000", 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106196 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.128142+0000", 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6427999999999994e-05 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.131929+0000", 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003786392 2024-09-02T10:21:24.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.132050+0000", 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121067 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.347347+0000", 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21529720299999999 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.347417+0000", 2024-09-02T10:21:24.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0129999999999994e-05 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.347475+0000", 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7861e-05 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:146643 6.4 6:2b5d94bb:::benchmark_data_smithi187_102448_object46371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e592)", 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:10.666230+0000", 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.776800242, 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.11855310199999999, 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.694 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 146643 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.666230+0000", 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.666230+0000", 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.666231+0000", 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.841e-06 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.666237+0000", 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7250000000000002e-06 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.666249+0000", 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1758e-05 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.666255+0000", 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9320000000000001e-06 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.666311+0000", 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6338999999999997e-05 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.666426+0000", 2024-09-02T10:21:24.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114654 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.666525+0000", 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9728999999999993e-05 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.671042+0000", 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045164840000000003 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.671316+0000", 2024-09-02T10:21:24.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00027396799999999998 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.784436+0000", 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.113120342 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.784624+0000", 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000187942 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.784783+0000", 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000158388 2024-09-02T10:21:24.699 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:146648 6.d e592/533 6:b82c0657:::benchmark_data_smithi187_102448_object46376:head v 592'9125, mlcod=592'9122)", 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:10.671489+0000", 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.771540311000001, 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.097960064999999999, 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:24.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.671489+0000", 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.671489+0000", 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.671490+0000", 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4700000000000002e-07 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.671494+0000", 2024-09-02T10:21:24.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8260000000000003e-06 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.764116+0000", 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.092622653999999999 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.764125+0000", 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0100000000000001e-06 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.764204+0000", 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8300000000000006e-05 2024-09-02T10:21:24.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.764231+0000", 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7517000000000001e-05 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.769382+0000", 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051508320000000002 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.769450+0000", 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7478999999999995e-05 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.703 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.704 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.704 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:24.704 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:24.704 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:21:24.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:24.705 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:24 smithi151 ceph-mon[16639]: pgmap v1387: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 718 op/s; 6742/141153 objects degraded (4.776%); 9066/141153 objects misplaced (6.423%); 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:21:24.705 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:24 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:21:24.705 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:24 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:21:24.705 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6742/141153 objects degraded (4.776%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:24.735 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:24.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:24.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:24 smithi187 ceph-mon[20072]: pgmap v1387: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 718 op/s; 6742/141153 objects degraded (4.776%); 9066/141153 objects misplaced (6.423%); 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:21:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:24 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:21:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:24 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:21:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6742/141153 objects degraded (4.776%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:24.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:24 smithi162 ceph-mon[20279]: pgmap v1387: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 718 op/s; 6742/141153 objects degraded (4.776%); 9066/141153 objects misplaced (6.423%); 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:21:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:24 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:21:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:24 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:21:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6742/141153 objects degraded (4.776%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:24.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:24.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:24.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:24.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:25.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:25.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:25.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:25.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:25.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:25.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:25.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:25.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:25.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:25.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:21:25.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:25.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:25.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:25.454 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:21:25.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:25.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:25.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:25.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:25.659 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:25.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:25.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:25 smithi151 ceph-mon[16639]: osdmap e596: 12 total, 12 up, 12 in 2024-09-02T10:21:25.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4037336728' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:25.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:25 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:25.711 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:21:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:25 smithi187 ceph-mon[20072]: osdmap e596: 12 total, 12 up, 12 in 2024-09-02T10:21:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4037336728' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:25 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:25.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:25.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:25.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:25 smithi162 ceph-mon[20279]: osdmap e596: 12 total, 12 up, 12 in 2024-09-02T10:21:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4037336728' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:25 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:25.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:25.922 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:25.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:26.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:26.047 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:26.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:26.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:26.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:26.186 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:26.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:26.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:26.310 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:26.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:26.311 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:26.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:26.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:26.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:26.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:26.574 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:26.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:26.676 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:26.700 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:26.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:26.701 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:26 smithi187 ceph-mon[20072]: pgmap v1389: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 6809/138786 objects degraded (4.906%); 8664/138786 objects misplaced (6.243%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:21:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:26 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:26 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:26 smithi187 ceph-mon[20072]: osdmap e597: 12 total, 12 up, 12 in 2024-09-02T10:21:26.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:26.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:26.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:26.840 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:26 smithi151 ceph-mon[16639]: pgmap v1389: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 6809/138786 objects degraded (4.906%); 8664/138786 objects misplaced (6.243%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:21:26.840 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:26 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:26.840 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:26 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:26.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:26 smithi151 ceph-mon[16639]: osdmap e597: 12 total, 12 up, 12 in 2024-09-02T10:21:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:26 smithi162 ceph-mon[20279]: pgmap v1389: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 629 op/s; 6809/138786 objects degraded (4.906%); 8664/138786 objects misplaced (6.243%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:21:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:26 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:26 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:26 smithi162 ceph-mon[20279]: osdmap e597: 12 total, 12 up, 12 in 2024-09-02T10:21:26.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:26.963 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:26.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:27.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:27.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:27.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:27.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:27.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:27.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:27.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:27.356 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:27.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:27.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:27.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:27.481 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:27.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:27.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:27.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:27.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:27.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:27.745 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:27.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/989439214' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:21:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/989439214' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:21:27.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:27.871 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:27.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/989439214' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:21:27.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:27.998 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:27.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:28.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:28.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:28.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:28.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:28.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:28.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:28.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:28.248 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-02T10:21:28.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:28.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:28.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:28.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:28.500 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:28.501 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:28.539 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:28.539 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:28.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:28.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:28.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:28.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:28.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:28.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:28.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:28 smithi187 ceph-mon[20072]: pgmap v1391: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 6809/134613 objects degraded (5.058%); 8512/134613 objects misplaced (6.323%); 6.6 MiB/s, 107 objects/s recovering 2024-09-02T10:21:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/989439214' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:21:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:28 smithi187 ceph-mon[20072]: osdmap e598: 12 total, 12 up, 12 in 2024-09-02T10:21:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:28 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:21:28.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:28 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:21:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:28 smithi162 ceph-mon[20279]: pgmap v1391: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 6809/134613 objects degraded (5.058%); 8512/134613 objects misplaced (6.323%); 6.6 MiB/s, 107 objects/s recovering 2024-09-02T10:21:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/989439214' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:21:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:28 smithi162 ceph-mon[20279]: osdmap e598: 12 total, 12 up, 12 in 2024-09-02T10:21:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:28 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:21:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:28 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:21:28.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:28.884 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:28.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:28 smithi151 ceph-mon[16639]: pgmap v1391: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 729 op/s; 6809/134613 objects degraded (5.058%); 8512/134613 objects misplaced (6.323%); 6.6 MiB/s, 107 objects/s recovering 2024-09-02T10:21:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/989439214' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:21:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:28 smithi151 ceph-mon[16639]: osdmap e598: 12 total, 12 up, 12 in 2024-09-02T10:21:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:28 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:21:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:28 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:21:28.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:29.014 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:29.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:29.116 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:29.142 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:29.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:29.144 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:29.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-09-02T10:21:29.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:29.267 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:29.268 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:29.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:29.391 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:29.392 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:29.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:29.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:29.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:29.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:29.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:29 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:21:29.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:29 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:21:29.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:29 smithi151 ceph-mon[16639]: osdmap e599: 12 total, 12 up, 12 in 2024-09-02T10:21:29.634 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:29 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6809/134613 objects degraded (5.058%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:29.648 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:29.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:29.750 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:29 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:21:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:29 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:21:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:29 smithi187 ceph-mon[20072]: osdmap e599: 12 total, 12 up, 12 in 2024-09-02T10:21:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:29 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6809/134613 objects degraded (5.058%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:29.773 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:29.774 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:29 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:21:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:29 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:21:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:29 smithi162 ceph-mon[20279]: osdmap e599: 12 total, 12 up, 12 in 2024-09-02T10:21:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:29 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6809/134613 objects degraded (5.058%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:29.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:29.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:29.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:29.985 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:21:29.985 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:21:29.985 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:158111 6.0 e599/586 6:07c0a539:::benchmark_data_smithi187_102448_object57839:head v 599'9963, mlcod=556'3536)", 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:29.919799+0000", 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.015428008, 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.01544719, 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.919799+0000", 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.986 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.919799+0000", 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.919801+0000", 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.308e-06 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.919808+0000", 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5800000000000003e-06 2024-09-02T10:21:29.987 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:29.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.919849+0000", 2024-09-02T10:21:29.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0567e-05 2024-09-02T10:21:29.988 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.988 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.988 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:29.988 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.919857+0000", 2024-09-02T10:21:29.988 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8399999999999995e-06 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.919908+0000", 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1425999999999999e-05 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.919934+0000", 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6123e-05 2024-09-02T10:21:29.992 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.935203+0000", 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015269034000000001 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:158115 6.0 e599/586 6:0f094a40:::benchmark_data_smithi187_102448_object57843:head v 599'9964, mlcod=556'3536)", 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:29.925252+0000", 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.0099757790000000006, 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010032034, 2024-09-02T10:21:29.993 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.925252+0000", 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.925252+0000", 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.994 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.925253+0000", 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8400000000000002e-07 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.925260+0000", 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7489999999999998e-06 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.925306+0000", 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5507000000000003e-05 2024-09-02T10:21:29.995 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.925313+0000", 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1049999999999997e-06 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.928892+0000", 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035792530000000001 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.996 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.928910+0000", 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8187000000000001e-05 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.935283+0000", 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063729809999999998 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:21:29.997 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 2 2024-09-02T10:21:29.998 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:21:30.010 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:30.032 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:30.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:30.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:21:30.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:30.155 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:30.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:30.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:30.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:30.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:30.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:30.322 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:21:30.322 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:21:30.322 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:21:30.322 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:21:30.322 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:21:30.322 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:21:30.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:21:30.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:21:30.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:21:30.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:21:30.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:21:30.323 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:21:30.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:21:30.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:21:30.324 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:21:30.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:21:30.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:21:30.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:21:30.337 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:21:30.337 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:21:30.372 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:21:30.373 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:21:30.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:30.403 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:21:30.404 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:21:30.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:30.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:30.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:30.424 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:30.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:30.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:30.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:30.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:30.643 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:30.664 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:30.665 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:30 smithi151 ceph-mon[16639]: pgmap v1394: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 6806/131538 objects degraded (5.174%); 8512/131538 objects misplaced (6.471%); 6.1 MiB/s, 99 objects/s recovering 2024-09-02T10:21:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2135026872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:30 smithi187 ceph-mon[20072]: pgmap v1394: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 6806/131538 objects degraded (5.174%); 8512/131538 objects misplaced (6.471%); 6.1 MiB/s, 99 objects/s recovering 2024-09-02T10:21:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2135026872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:30.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:30.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:30.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:30 smithi162 ceph-mon[20279]: pgmap v1394: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.7 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 570 op/s; 6806/131538 objects degraded (5.174%); 8512/131538 objects misplaced (6.471%); 6.1 MiB/s, 99 objects/s recovering 2024-09-02T10:21:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2135026872' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:30.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:30.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:30.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:31.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:31.065 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:31.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:31.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:31.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:31.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:31.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:31.323 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:31.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:31.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:31.461 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:31.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:31.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:31.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:31.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:31.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:31.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:31.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:31.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:31.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:31.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:31 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:21:31.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:31 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:21:31.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:31 smithi151 ceph-mon[16639]: osdmap e600: 12 total, 12 up, 12 in 2024-09-02T10:21:31.812 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:31.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:31.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:31 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:21:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:31 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:21:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:31 smithi162 ceph-mon[20279]: osdmap e600: 12 total, 12 up, 12 in 2024-09-02T10:21:31.845 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:31.846 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:31.852 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:21:31.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:31.974 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:31.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:31.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:31 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:21:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:31 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:21:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:31 smithi187 ceph-mon[20072]: osdmap e600: 12 total, 12 up, 12 in 2024-09-02T10:21:32.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:32.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:32.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:32.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:32.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:32.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:32.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:32.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:32.371 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:32.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:32.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:32.501 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:32.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:32.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:32.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:32.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:32.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:32 smithi151 ceph-mon[16639]: pgmap v1396: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 567 op/s; 6830/128706 objects degraded (5.307%); 8072/128706 objects misplaced (6.272%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:21:32.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:32.760 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:32.760 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:32 smithi162 ceph-mon[20279]: pgmap v1396: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 567 op/s; 6830/128706 objects degraded (5.307%); 8072/128706 objects misplaced (6.272%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:21:32.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:32.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:32.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:32.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:32.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:32 smithi187 ceph-mon[20072]: pgmap v1396: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 567 op/s; 6830/128706 objects degraded (5.307%); 8072/128706 objects misplaced (6.272%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:21:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:33.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:33.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:33.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:33.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:33.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:33.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:33.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:21:33.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:33.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:33.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:33.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:33.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:33.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:33.523 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:21:33.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:33.595 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1757762242' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:21:33.596 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:33 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:21:33.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1757762242' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:21:33.602 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:33 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:21:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:33.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:33.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:33.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:33.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:33.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1757762242' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:21:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:33 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:21:33.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:33.894 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:33.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:33.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:34.025 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:34.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:34.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:34.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:34.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:34.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:34.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:34.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:34.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:34.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:34.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:34.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:34.489 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:34.531 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-02T10:21:34.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:34.604 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:34.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:34.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:34.619 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:34 smithi151 ceph-mon[16639]: pgmap v1397: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 6828/124377 objects degraded (5.490%); 7848/124377 objects misplaced (6.310%); 6.7 MiB/s, 107 objects/s recovering 2024-09-02T10:21:34.619 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:34 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:34.619 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1757762242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:21:34.619 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:34 smithi151 ceph-mon[16639]: osdmap e601: 12 total, 12 up, 12 in 2024-09-02T10:21:34.619 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/74366748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:21:34.619 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:34 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:21:34.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:34 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6828/124377 objects degraded (5.490%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:34.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:34 smithi151 ceph-mon[16639]: 6.8 scrub starts 2024-09-02T10:21:34.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:34 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:21:34.620 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:34 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:21:34.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:34.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:34.730 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:34.830 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:34 smithi162 ceph-mon[20279]: pgmap v1397: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 6828/124377 objects degraded (5.490%); 7848/124377 objects misplaced (6.310%); 6.7 MiB/s, 107 objects/s recovering 2024-09-02T10:21:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:34 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1757762242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:21:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:34 smithi162 ceph-mon[20279]: osdmap e601: 12 total, 12 up, 12 in 2024-09-02T10:21:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/74366748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:21:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:34 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:21:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:34 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6828/124377 objects degraded (5.490%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:34 smithi162 ceph-mon[20279]: 6.8 scrub starts 2024-09-02T10:21:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:34 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:21:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:34 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:21:34.854 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:34.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:34.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:34.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:34.979 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:34.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:34.981 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:34 smithi187 ceph-mon[20072]: pgmap v1397: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 568 op/s; 6828/124377 objects degraded (5.490%); 7848/124377 objects misplaced (6.310%); 6.7 MiB/s, 107 objects/s recovering 2024-09-02T10:21:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:34 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1757762242' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:21:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:34 smithi187 ceph-mon[20072]: osdmap e601: 12 total, 12 up, 12 in 2024-09-02T10:21:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/74366748' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:21:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:34 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:21:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:34 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6828/124377 objects degraded (5.490%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:34 smithi187 ceph-mon[20072]: 6.8 scrub starts 2024-09-02T10:21:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:34 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:21:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:34 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:21:35.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:35.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:35.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:35.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:35.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:35.238 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:35.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:35.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:35.368 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:35.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:35.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:35.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:35.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:35.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:35.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:35.640 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:35.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:35.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:21:35.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:21:35.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:21:35.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: osdmap e602: 12 total, 12 up, 12 in 2024-09-02T10:21:35.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:21:35.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:21:35.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:21:35.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:21:35.670 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:21:35.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:21:35.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:21:35.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:21:35.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:21:35.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:21:35.671 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:35 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:21:35.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-09-02T10:21:35.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:35.776 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:35.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:21:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:21:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:21:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: osdmap e602: 12 total, 12 up, 12 in 2024-09-02T10:21:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:21:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:21:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:21:35.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:21:35.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:21:35.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:21:35.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:21:35.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:21:35.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:21:35.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:21:35.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:35 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:21:35.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:35.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:35.903 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:35.921 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:35.996 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:21:35.997 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:21:36.004 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:21:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:21:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: osdmap e602: 12 total, 12 up, 12 in 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:21:36.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:35 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:21:36.035 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:21:36.035 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:21:36.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:36.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:36.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:21:36.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:21:36.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:21:36.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:21:36.147 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:36.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:36.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:36.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:36.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:36.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:36.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:36.353 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:21:36.353 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:21:36.395 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:21:36.395 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:21:36.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:36.427 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:21:36.427 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:21:36.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:21:36.467 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:21:36.468 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:21:36.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:36.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:36.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:21:36.515 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:21:36.567 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:21:36.567 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:21:36.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:36.594 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:21:36.594 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:21:36.594 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:21:36.594 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:21:36.594 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:21:36.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:36.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:36.597 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:36.656 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:21:36.657 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:21:36.695 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:21:36.695 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:21:36.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:36.728 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:36.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:36.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:36.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:36 smithi162 ceph-mon[20279]: pgmap v1400: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 6828/122172 objects degraded (5.589%); 7493/122172 objects misplaced (6.133%); 10 MiB/s, 166 objects/s recovering 2024-09-02T10:21:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:36 smithi162 ceph-mon[20279]: 6.8 scrub ok 2024-09-02T10:21:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:36 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:21:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:36 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:21:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/646040314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:36.862 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:36.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:36 smithi151 ceph-mon[16639]: pgmap v1400: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 6828/122172 objects degraded (5.589%); 7493/122172 objects misplaced (6.133%); 10 MiB/s, 166 objects/s recovering 2024-09-02T10:21:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:36 smithi151 ceph-mon[16639]: 6.8 scrub ok 2024-09-02T10:21:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:36 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:21:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:36 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:21:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/646040314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:36.963 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:36.985 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:36.986 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:36 smithi187 ceph-mon[20072]: pgmap v1400: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 6828/122172 objects degraded (5.589%); 7493/122172 objects misplaced (6.133%); 10 MiB/s, 166 objects/s recovering 2024-09-02T10:21:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:36 smithi187 ceph-mon[20072]: 6.8 scrub ok 2024-09-02T10:21:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:36 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:21:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:36 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:21:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/646040314' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:37.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:37.115 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:37.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:37.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:37.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:37.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:37.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:37.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:37.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:37.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:37.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:37.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:37.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:37.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:37.640 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:37.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:37.742 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:37.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:37.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:37.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:37.893 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:37.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:37.894 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:37.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:38.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:38.017 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:38.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:38.147 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:38.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:38.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:38.270 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:38.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:38.271 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:38.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:38.399 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:38.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:38.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:38.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:21:38.532 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:38.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:38.567 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-02T10:21:38.634 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:38.655 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:38.656 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:38.757 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:38.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:38.782 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:38 smithi162 ceph-mon[20279]: pgmap v1401: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 463 op/s; 6828/120513 objects degraded (5.666%); 7493/120513 objects misplaced (6.218%); 5.3 MiB/s, 84 objects/s recovering 2024-09-02T10:21:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:38 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:21:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:38 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:21:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3010492116' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:21:38.883 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:38.913 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:38.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:38.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:38 smithi151 ceph-mon[16639]: pgmap v1401: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 463 op/s; 6828/120513 objects degraded (5.666%); 7493/120513 objects misplaced (6.218%); 5.3 MiB/s, 84 objects/s recovering 2024-09-02T10:21:38.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:38 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:21:38.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:38 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:21:38.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3010492116' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:21:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:38 smithi187 ceph-mon[20072]: pgmap v1401: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 463 op/s; 6828/120513 objects degraded (5.666%); 7493/120513 objects misplaced (6.218%); 5.3 MiB/s, 84 objects/s recovering 2024-09-02T10:21:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:38 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:21:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:38 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:21:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3010492116' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:21:39.015 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:39.036 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:39.037 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:39.138 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:39.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:39.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:39.161 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:39.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:39.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:39.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:39.286 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:39.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:39.415 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:39.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:39.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-09-02T10:21:39.516 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:39.545 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:39.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:39.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3010492116' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:21:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:39 smithi151 ceph-mon[16639]: osdmap e603: 12 total, 12 up, 12 in 2024-09-02T10:21:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:21:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:21:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:39 smithi151 ceph-mon[16639]: 6.1 scrub starts 2024-09-02T10:21:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6828/120513 objects degraded (5.666%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:39 smithi151 ceph-mon[16639]: 6.5 scrub starts 2024-09-02T10:21:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:39.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:39.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:39.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:39.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:39.816 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:39.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3010492116' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:21:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:39 smithi162 ceph-mon[20279]: osdmap e603: 12 total, 12 up, 12 in 2024-09-02T10:21:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:21:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:21:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:39 smithi162 ceph-mon[20279]: 6.1 scrub starts 2024-09-02T10:21:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6828/120513 objects degraded (5.666%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:39 smithi162 ceph-mon[20279]: 6.5 scrub starts 2024-09-02T10:21:39.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:39.944 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:39.950 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:39.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:39.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:39.955 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3010492116' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:21:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:39 smithi187 ceph-mon[20072]: osdmap e603: 12 total, 12 up, 12 in 2024-09-02T10:21:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:21:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:21:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:39 smithi187 ceph-mon[20072]: 6.1 scrub starts 2024-09-02T10:21:40.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6828/120513 objects degraded (5.666%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:40.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:39 smithi187 ceph-mon[20072]: 6.5 scrub starts 2024-09-02T10:21:40.052 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:40.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:40.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:40.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:40.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:40.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:40.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:40.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:40.339 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:40.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:40.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:40.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:40.465 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:40.540 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:40.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:40.585 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:21:40.585 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:21:40.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:21:40.585 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:41902 4.14 e353/351 4:2ffda43f:::benchmark_data_smithi187_76542_object41901:head v 353'2264, mlcod=353'2262)", 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:11:47.743167+0000", 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 592.83775497099998, 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50111127099999997, 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743167+0000", 2024-09-02T10:21:40.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743167+0000", 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743168+0000", 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8599999999999996e-07 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:47.743268+0000", 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9846000000000006e-05 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243322+0000", 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50005331600000003 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243327+0000", 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.057e-06 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243388+0000", 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0948e-05 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.243407+0000", 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9040000000000001e-05 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.244241+0000", 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00083470799999999998 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:11:48.244279+0000", 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7370000000000003e-05 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90150 4.4 4:22e9747b:::benchmark_data_smithi187_76542_object90149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.040493+0000", 2024-09-02T10:21:40.590 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 549.54042923600002, 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.181265486, 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90150 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040493+0000", 2024-09-02T10:21:40.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040493+0000", 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040495+0000", 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4130000000000001e-06 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040588+0000", 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3113000000000001e-05 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040600+0000", 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1834000000000001e-05 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040605+0000", 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7049999999999996e-06 2024-09-02T10:21:40.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040652+0000", 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6643e-05 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040741+0000", 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8797999999999999e-05 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.040826+0000", 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5191000000000005e-05 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042413+0000", 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001587605 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042531+0000", 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117157 2024-09-02T10:21:40.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221691+0000", 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17916051699999999 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221731+0000", 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9650999999999997e-05 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221759+0000", 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7858999999999999e-05 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:90150 4.4 e375/341 ondisk, result = 0)", 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.042430+0000", 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 549.5384924, 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17933455100000001, 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:40.597 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042430+0000", 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042430+0000", 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042431+0000", 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9999999999999996e-07 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.042439+0000", 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4810000000000006e-06 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221645+0000", 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17920517899999999 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221649+0000", 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.865e-06 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221683+0000", 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3324000000000001e-05 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221688+0000", 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.887e-06 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.221765+0000", 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6915000000000005e-05 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:90160 4.4 4:2097a6ca:::benchmark_data_smithi187_76542_object90159:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.048271+0000", 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 549.53265110699999, 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.17505789499999999, 2024-09-02T10:21:40.601 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 90160 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048271+0000", 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048271+0000", 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048276+0000", 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.938e-06 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048319+0000", 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2793999999999997e-05 2024-09-02T10:21:40.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048330+0000", 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.118e-05 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048335+0000", 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2660000000000001e-06 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048380+0000", 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4733e-05 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048474+0000", 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3494999999999998e-05 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.048562+0000", 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8528000000000003e-05 2024-09-02T10:21:40.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.049898+0000", 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001336159 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050229+0000", 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033104400000000001 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223275+0000", 2024-09-02T10:21:40.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.173045588 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223306+0000", 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0621999999999999e-05 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223329+0000", 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3547999999999999e-05 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.607 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25507.0:90160 4.4 e375/341 ondisk, result = 0)", 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 549.53056658499997, 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.172980419, 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050356+0000", 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4600000000000005e-07 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.050360+0000", 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.253e-06 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223231+0000", 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.172871466 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223235+0000", 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7309999999999999e-06 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223266+0000", 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1443999999999999e-05 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223272+0000", 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3249999999999998e-06 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.223336+0000", 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4653999999999995e-05 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25507.0:90175 4.19 e375/354 4:988493fd:::benchmark_data_smithi187_76542_object90174:head v 375'3736, mlcod=375'3734)", 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:31.066480+0000", 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 549.51444209600004, 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.188910198, 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.612 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066480+0000", 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066480+0000", 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066482+0000", 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3939999999999999e-06 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.066535+0000", 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3510000000000001e-05 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253612+0000", 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18707629000000001 2024-09-02T10:21:40.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253619+0000", 2024-09-02T10:21:40.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9719999999999992e-06 2024-09-02T10:21:40.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253697+0000", 2024-09-02T10:21:40.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7881999999999997e-05 2024-09-02T10:21:40.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.253728+0000", 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1118000000000003e-05 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.255346+0000", 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016170609999999999 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:31.255391+0000", 2024-09-02T10:21:40.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4971e-05 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:136184 6.2 e587/586 6:40cc0bb5:::benchmark_data_smithi187_102448_object35912:head v 587'8505, mlcod=551'2232)", 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.641761+0000", 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.939161542999997, 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.088366534999999996, 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.618 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.641761+0000", 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.641761+0000", 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.641762+0000", 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4300000000000003e-07 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.641768+0000", 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9830000000000001e-06 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.646339+0000", 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045714450000000004 2024-09-02T10:21:40.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.646346+0000", 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.691e-06 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.649719+0000", 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033730539999999999 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.649748+0000", 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9241e-05 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.729924+0000", 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.080176199000000004 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.730127+0000", 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000203279 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.622 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:136187 6.2 e587/586 6:44ff5bfd:::benchmark_data_smithi187_102448_object35915:head v 587'8506, mlcod=551'2232)", 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.646312+0000", 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.934610274999997, 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.083849362999999996, 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.646312+0000", 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.646312+0000", 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.646313+0000", 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3900000000000004e-07 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.646319+0000", 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9580000000000004e-06 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.652371+0000", 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0060524359999999996 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.652378+0000", 2024-09-02T10:21:40.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5529999999999998e-06 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.657437+0000", 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050588609999999996 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.657456+0000", 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8960000000000001e-05 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.730132+0000", 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.072676456 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.730162+0000", 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.94e-05 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:136180 6.2 e587/586 6:469fba95:::benchmark_data_smithi187_102448_object35908:head v 587'8507, mlcod=551'2232)", 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.657035+0000", 2024-09-02T10:21:40.627 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.923887418, 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.083466563999999993, 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.657035+0000", 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.657035+0000", 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.657037+0000", 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5999999999999999e-06 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.657045+0000", 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8089999999999999e-06 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.664832+0000", 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0077862720000000003 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.664842+0000", 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0603e-05 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.677165+0000", 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012322843999999999 2024-09-02T10:21:40.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.677191+0000", 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.548e-05 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.740188+0000", 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.062997656999999999 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.740502+0000", 2024-09-02T10:21:40.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00031329900000000001 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:136202 6.2 e587/586 6:4d505a65:::benchmark_data_smithi187_102448_object35930:head v 587'8508, mlcod=551'2233)", 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.664800+0000", 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.916122735000002, 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.075898621999999999, 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.632 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.664800+0000", 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.664800+0000", 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.664801+0000", 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.37e-06 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.664810+0000", 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1900000000000001e-06 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.673790+0000", 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089795610000000005 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.673798+0000", 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4130000000000006e-06 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.688937+0000", 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015138804 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.688971+0000", 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4260000000000001e-05 2024-09-02T10:21:40.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.740509+0000", 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.051537843 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.740698+0000", 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018918099999999999 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:136256 6.0 e587/586 6:060ad347:::benchmark_data_smithi187_102448_object35984:head v 587'8561, mlcod=551'2205)", 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.748184+0000", 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.832738231999997, 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.068802826999999997, 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.748184+0000", 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.748184+0000", 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.748185+0000", 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.198e-06 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.748195+0000", 2024-09-02T10:21:40.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0084e-05 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.748270+0000", 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4945999999999997e-05 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.748280+0000", 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3789999999999995e-06 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.787818+0000", 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.039537803000000003 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.787847+0000", 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9516999999999999e-05 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.816937+0000", 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.029089851 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.816987+0000", 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0049000000000001e-05 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:136269 6.e e587/586 6:78695e59:::benchmark_data_smithi187_102448_object35997:head v 587'8513, mlcod=551'2219)", 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.766714+0000", 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 46.814208886000003, 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.068290319000000002, 2024-09-02T10:21:40.641 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.766714+0000", 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.766714+0000", 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.766714+0000", 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6199999999999996e-07 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.766722+0000", 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.52e-06 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.774762+0000", 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080402830000000005 2024-09-02T10:21:40.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.774769+0000", 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7399999999999998e-06 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.797119+0000", 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022350355999999998 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.797145+0000", 2024-09-02T10:21:40.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5452e-05 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.834955+0000", 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037809817000000003 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.835004+0000", 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9289000000000002e-05 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.645 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.f 588/586 [PushOp(6:f44133b5:::benchmark_data_smithi187_102448_object39129:head, version: 553'2467, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:f44133b5:::benchmark_data_smithi187_102448_object39129:head@553'2467, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:58.199960+0000", 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 42.380962275999998, 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.072058043000000002, 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:58.199960+0000", 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:58.199960+0000", 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:58.199961+0000", 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2410000000000001e-06 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:58.200028+0000", 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6379999999999998e-05 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:58.269002+0000", 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.068974670000000002 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:58.269023+0000", 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0831000000000001e-05 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:58.269126+0000", 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102898 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:58.269140+0000", 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3501e-05 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:58.272018+0000", 2024-09-02T10:21:40.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028785220000000001 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.f 592/586 [PushOp(6:f7a18317:::benchmark_data_smithi187_102448_object46081:head, version: 554'2891, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:f7a18317:::benchmark_data_smithi187_102448_object46081:head@554'2891, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:10.172390+0000", 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 30.408532298000001, 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.080211394000000005, 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:40.650 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.172390+0000", 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.172390+0000", 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.172391+0000", 2024-09-02T10:21:40.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1999999999999999e-06 2024-09-02T10:21:40.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.172471+0000", 2024-09-02T10:21:40.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9427999999999999e-05 2024-09-02T10:21:40.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.248755+0000", 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.076284555000000004 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.248774+0000", 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8819999999999999e-05 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.248874+0000", 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9976999999999995e-05 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.248888+0000", 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3783e-05 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:10.252602+0000", 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037136309999999998 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:152109 6.0 e594/586 6:075b7b12:::benchmark_data_smithi187_102448_object51837:head v 594'9593, mlcod=554'3159)", 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:19.387902+0000", 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 21.193020881999999, 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.10815807099999999, 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.387902+0000", 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.387902+0000", 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.387903+0000", 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1489999999999999e-06 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.656 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.387910+0000", 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.216e-06 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.478610+0000", 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.090699851999999997 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.478617+0000", 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4789999999999999e-06 2024-09-02T10:21:40.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.478683+0000", 2024-09-02T10:21:40.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:40.659 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:40.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:40.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5371999999999997e-05 2024-09-02T10:21:40.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.478701+0000", 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7989999999999999e-05 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.496025+0000", 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017323970000000001 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.496060+0000", 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5043000000000003e-05 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:152211 6.0 e594/586 6:0168d4b1:::benchmark_data_smithi187_102448_object51939:head v 594'9596, mlcod=554'3159)", 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:19.538203+0000", 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 21.042719185999999, 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.084772286000000002, 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.661 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.538203+0000", 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.538203+0000", 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.538204+0000", 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0640000000000001e-06 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.538211+0000", 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8789999999999997e-06 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.604714+0000", 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.066503256999999996 2024-09-02T10:21:40.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.604723+0000", 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1389999999999995e-06 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.604832+0000", 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109743 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.604857+0000", 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4899999999999999e-05 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.622934+0000", 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018076313 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:19.622976+0000", 2024-09-02T10:21:40.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1990999999999998e-05 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.f 595/586 [PushOp(6:fa761538:::benchmark_data_smithi187_102448_object54905:head, version: 556'3430, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:fa761538:::benchmark_data_smithi187_102448_object54905:head@556'3430, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:24.231041+0000", 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.349881014000001, 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.079941625000000002, 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:40.666 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.231041+0000", 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.231041+0000", 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.231043+0000", 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.381e-06 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.231085+0000", 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2048999999999997e-05 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.305777+0000", 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.074692193000000004 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.305793+0000", 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6226e-05 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.305894+0000", 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100312 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.305906+0000", 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2354000000000001e-05 2024-09-02T10:21:40.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.310983+0000", 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050771100000000001 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:155539 6.2 e596/586 6:44b212d5:::benchmark_data_smithi187_102448_object55267:head v 596'9729, mlcod=556'3469)", 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:24.837187+0000", 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 15.74373508, 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.071355821999999999, 2024-09-02T10:21:40.670 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.837187+0000", 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.837187+0000", 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.837188+0000", 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5699999999999998e-07 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.837193+0000", 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8450000000000002e-06 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.837202+0000", 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5489999999999995e-06 2024-09-02T10:21:40.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.837206+0000", 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0339999999999998e-06 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.851288+0000", 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014081549 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.851340+0000", 2024-09-02T10:21:40.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2442999999999999e-05 2024-09-02T10:21:40.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:40.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.908508+0000", 2024-09-02T10:21:40.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.057167629999999997 2024-09-02T10:21:40.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:24.908543+0000", 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5315000000000003e-05 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.f 599/586 [PushOp(6:f73e1ed4:::benchmark_data_smithi187_102448_object58703:head, version: 556'3680, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:f73e1ed4:::benchmark_data_smithi187_102448_object58703:head@556'3680, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:29.406808+0000", 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 11.174114154, 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.11780626299999999, 2024-09-02T10:21:40.675 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.406808+0000", 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.406808+0000", 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.406810+0000", 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3200000000000001e-06 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.406872+0000", 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1989000000000006e-05 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.521892+0000", 2024-09-02T10:21:40.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.115020343 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.521904+0000", 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2498e-05 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.521983+0000", 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8541999999999994e-05 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.521995+0000", 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1637999999999999e-05 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.524615+0000", 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026199330000000001 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(6.f 603/586 [PushOp(6:fdf6954f:::benchmark_data_smithi187_102448_object64801:head, version: 560'4047, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:fdf6954f:::benchmark_data_smithi187_102448_object64801:head@560'4047, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:21:40.679 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:39.340693+0000", 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.2402297090000001, 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.079674411000000001, 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:39.340693+0000", 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:39.340693+0000", 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:39.340694+0000", 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.063e-06 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:39.340748+0000", 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4536000000000001e-05 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:39.416961+0000", 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.076212774999999996 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:39.416981+0000", 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9613999999999999e-05 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:39.417121+0000", 2024-09-02T10:21:40.682 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014022199999999999 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:39.417164+0000", 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2880000000000003e-05 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:39.420367+0000", 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032033209999999999 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.683 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:40.684 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:40.684 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:21:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:40 smithi151 ceph-mon[16639]: pgmap v1403: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 6827/114924 objects degraded (5.940%); 7323/114924 objects misplaced (6.372%); 5.5 MiB/s, 87 objects/s recovering 2024-09-02T10:21:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:40 smithi151 ceph-mon[16639]: 6.1 scrub ok 2024-09-02T10:21:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:40 smithi151 ceph-mon[16639]: osdmap e604: 12 total, 12 up, 12 in 2024-09-02T10:21:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:40 smithi151 ceph-mon[16639]: 6.5 scrub ok 2024-09-02T10:21:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:40 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:21:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:40 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:21:40.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:40.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:40.785 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:40 smithi162 ceph-mon[20279]: pgmap v1403: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 6827/114924 objects degraded (5.940%); 7323/114924 objects misplaced (6.372%); 5.5 MiB/s, 87 objects/s recovering 2024-09-02T10:21:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:40 smithi162 ceph-mon[20279]: 6.1 scrub ok 2024-09-02T10:21:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:40 smithi162 ceph-mon[20279]: osdmap e604: 12 total, 12 up, 12 in 2024-09-02T10:21:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:40 smithi162 ceph-mon[20279]: 6.5 scrub ok 2024-09-02T10:21:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:40 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:21:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:40 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:21:40.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:40.908 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:40.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:40.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:40 smithi187 ceph-mon[20072]: pgmap v1403: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 525 op/s; 6827/114924 objects degraded (5.940%); 7323/114924 objects misplaced (6.372%); 5.5 MiB/s, 87 objects/s recovering 2024-09-02T10:21:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:40 smithi187 ceph-mon[20072]: 6.1 scrub ok 2024-09-02T10:21:41.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:40 smithi187 ceph-mon[20072]: osdmap e604: 12 total, 12 up, 12 in 2024-09-02T10:21:41.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:40 smithi187 ceph-mon[20072]: 6.5 scrub ok 2024-09-02T10:21:41.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:40 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:21:41.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:40 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:21:41.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:41.043 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:41.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:41.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:41.175 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:41.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:41.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:41.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:41.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:41.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:41.302 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:41.403 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:41.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:41.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:41.426 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:41.527 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:41.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:41.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:41.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:41.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:41.681 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:41.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:41.682 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:41.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:41.805 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:41.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:41.806 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:41 smithi162 ceph-mon[20279]: 6.3 scrub starts 2024-09-02T10:21:41.878 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:41 smithi151 ceph-mon[16639]: 6.3 scrub starts 2024-09-02T10:21:41.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:41.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:21:41.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:41.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:41.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:41 smithi187 ceph-mon[20072]: 6.3 scrub starts 2024-09-02T10:21:42.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:42.061 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:42.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:42.062 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:42.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:42.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:42.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:42.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:42.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:42.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:42.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:42.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:42.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:42.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:42.546 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:42.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:42.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:42.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:42.711 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:42.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:42.813 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:42.830 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:42 smithi151 ceph-mon[16639]: pgmap v1405: 47 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 6861/111387 objects degraded (6.160%); 6977/111387 objects misplaced (6.264%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:21:42.830 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:42 smithi151 ceph-mon[16639]: 6.3 scrub ok 2024-09-02T10:21:42.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:42.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:42.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:42.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:42 smithi162 ceph-mon[20279]: pgmap v1405: 47 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 6861/111387 objects degraded (6.160%); 6977/111387 objects misplaced (6.264%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:21:42.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:42 smithi162 ceph-mon[20279]: 6.3 scrub ok 2024-09-02T10:21:42.937 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:42.959 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:42.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:42.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:42 smithi187 ceph-mon[20072]: pgmap v1405: 47 pgs: 1 active+recovering+remapped, 2 active+clean+scrubbing, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 524 op/s; 6861/111387 objects degraded (6.160%); 6977/111387 objects misplaced (6.264%); 4.4 MiB/s, 70 objects/s recovering 2024-09-02T10:21:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:42 smithi187 ceph-mon[20072]: 6.3 scrub ok 2024-09-02T10:21:43.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:43.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:43.085 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:43.086 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:43.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:43.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:43.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:43.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:21:43.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:43.344 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:43.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:43.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:43.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:43.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:43.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:43.587 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:21:43.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2878461615' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:43.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:43.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:43.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:43.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:43.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:43.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2878461615' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:43.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:43.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:43.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:44.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:44.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2878461615' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:44.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:44.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:44.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:44.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:44.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:44.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:44.300 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:44.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:44.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:44.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:44.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:44.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:44.570 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:44.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:44.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:44.704 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:44.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:44.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:44.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:44 smithi151 ceph-mon[16639]: pgmap v1406: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 6861/105978 objects degraded (6.474%); 6799/105978 objects misplaced (6.415%); 5.1 MiB/s, 82 objects/s recovering 2024-09-02T10:21:44.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:44 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:21:44.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2878461615' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:44 smithi151 ceph-mon[16639]: osdmap e605: 12 total, 12 up, 12 in 2024-09-02T10:21:44.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:44 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6861/105978 objects degraded (6.474%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:44.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:44.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:44.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:44.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:44 smithi162 ceph-mon[20279]: pgmap v1406: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 6861/105978 objects degraded (6.474%); 6799/105978 objects misplaced (6.415%); 5.1 MiB/s, 82 objects/s recovering 2024-09-02T10:21:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:44 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:21:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2878461615' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:44 smithi162 ceph-mon[20279]: osdmap e605: 12 total, 12 up, 12 in 2024-09-02T10:21:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:44 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6861/105978 objects degraded (6.474%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:44.869 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:44.925 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:44.939 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:44.960 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:44.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:44 smithi187 ceph-mon[20072]: pgmap v1406: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 41 active+clean; 2.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s; 6861/105978 objects degraded (6.474%); 6799/105978 objects misplaced (6.415%); 5.1 MiB/s, 82 objects/s recovering 2024-09-02T10:21:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:44 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:21:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2878461615' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:44 smithi187 ceph-mon[20072]: osdmap e605: 12 total, 12 up, 12 in 2024-09-02T10:21:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:44 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6861/105978 objects degraded (6.474%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:45.061 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:45.091 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:45.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:45.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:45.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:45.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:45.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:45.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:45.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:45.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:45.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:45.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:45.526 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:21:45.526 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:21:45.530 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:45.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:45.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:45.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:21:45.558 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:21:45.583 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:21:45.584 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:21:45.584 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-09-02T10:21:45.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:21:45.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:21:45.632 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:45.660 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:21:45.660 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:21:45.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:45.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:45.688 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:21:45.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:21:45.728 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:21:45.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:21:45.764 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:21:45.764 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:21:45.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:45.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:45.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:45.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:45.799 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:21:45.799 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:21:45.828 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:21:45.828 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:21:45.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3677613091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:21:45.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:45 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:21:45.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/70197998' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3677613091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:21:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:45 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:21:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/70197998' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:45.853 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:21:45.853 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:21:45.884 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:21:45.884 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:21:45.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:45.912 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:45.913 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3677613091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:21:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:45 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:21:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/70197998' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:46.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:46.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:46.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:46.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:46.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:46.181 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:46.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:46.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:46.283 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:46.322 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:46.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:46.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:46.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:46.443 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:21:46.443 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:21:46.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:46.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:46.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:21:46.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:46.587 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:46.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:46.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:46.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:46.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:46.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:46.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:46 smithi162 ceph-mon[20279]: pgmap v1408: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s; 6862/103512 objects degraded (6.629%); 6450/103512 objects misplaced (6.231%); 7.7 MiB/s, 122 objects/s recovering 2024-09-02T10:21:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:46 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:21:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:46 smithi162 ceph-mon[20279]: osdmap e606: 12 total, 12 up, 12 in 2024-09-02T10:21:46.850 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:46.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:46 smithi151 ceph-mon[16639]: pgmap v1408: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s; 6862/103512 objects degraded (6.629%); 6450/103512 objects misplaced (6.231%); 7.7 MiB/s, 122 objects/s recovering 2024-09-02T10:21:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:46 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:21:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:46 smithi151 ceph-mon[16639]: osdmap e606: 12 total, 12 up, 12 in 2024-09-02T10:21:46.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:46.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:46.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:46.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:46 smithi187 ceph-mon[20072]: pgmap v1408: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s; 6862/103512 objects degraded (6.629%); 6450/103512 objects misplaced (6.231%); 7.7 MiB/s, 122 objects/s recovering 2024-09-02T10:21:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:46 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:21:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:46 smithi187 ceph-mon[20072]: osdmap e606: 12 total, 12 up, 12 in 2024-09-02T10:21:47.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:47.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:47.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:47.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:47.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:47.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:47.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:47.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:21:47.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:47.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:47.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:47.485 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:47.508 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:47.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:47.509 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:47.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:47.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:47.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:47.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:47.738 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:47.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:47.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:47.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:47.868 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:47.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:47.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:47.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:47.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:48.032 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:48.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:48.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:48.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:48.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:48.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:48.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:48.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:48.409 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:48.430 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:48.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:48.431 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:48.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-09-02T10:21:48.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:48.559 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:48.583 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:48.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:48.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:48.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:48.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:48.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:48.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:48 smithi151 ceph-mon[16639]: pgmap v1410: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 6862/101499 objects degraded (6.761%); 6450/101499 objects misplaced (6.355%); 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:21:48.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:48 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:21:48.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:48 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:21:48.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:48.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:48.841 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:48.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:48.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:48.973 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:48.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:48.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:48 smithi187 ceph-mon[20072]: pgmap v1410: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 6862/101499 objects degraded (6.761%); 6450/101499 objects misplaced (6.355%); 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:21:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:48 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:21:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:48 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:21:49.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:48 smithi162 ceph-mon[20279]: pgmap v1410: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 491 op/s; 6862/101499 objects degraded (6.761%); 6450/101499 objects misplaced (6.355%); 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:21:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:48 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:21:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:48 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:21:49.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:49.098 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:49.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:49.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:49.233 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:49.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:49.295 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:21:49.296 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:21:49.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:49.367 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:49.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:49.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:49.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:49.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:49.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:49.619 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:49.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:49.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:49 smithi151 ceph-mon[16639]: osdmap e607: 12 total, 12 up, 12 in 2024-09-02T10:21:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:49 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6862/101499 objects degraded (6.761%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:49.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:49.743 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:49.745 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:49.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:21:49.846 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:49.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:49.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:49.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:49.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:49.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:49.993 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:49 smithi187 ceph-mon[20072]: osdmap e607: 12 total, 12 up, 12 in 2024-09-02T10:21:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:49 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6862/101499 objects degraded (6.761%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:50.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:49 smithi162 ceph-mon[20279]: osdmap e607: 12 total, 12 up, 12 in 2024-09-02T10:21:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:49 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6862/101499 objects degraded (6.761%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:21:50.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:50.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:50.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:50.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:50.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:50.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:50.253 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:50.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:21:50.354 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:50.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:50.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:50.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:50.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:50.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:50.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:50.544 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:50.616 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:50.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:50.666 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:50.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:50.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:50.798 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:50.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:50.846 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:50 smithi151 ceph-mon[16639]: pgmap v1412: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 6861/95379 objects degraded (7.193%); 6294/95379 objects misplaced (6.599%); 5.2 MiB/s, 84 objects/s recovering 2024-09-02T10:21:50.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:50.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:50.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:50 smithi187 ceph-mon[20072]: pgmap v1412: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 6861/95379 objects degraded (7.193%); 6294/95379 objects misplaced (6.599%); 5.2 MiB/s, 84 objects/s recovering 2024-09-02T10:21:51.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:51.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:51.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:51.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:50 smithi162 ceph-mon[20279]: pgmap v1412: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 6861/95379 objects degraded (7.193%); 6294/95379 objects misplaced (6.599%); 5.2 MiB/s, 84 objects/s recovering 2024-09-02T10:21:51.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:51.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:51.199 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:51.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:51.300 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:51.325 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:51.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:51.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:51.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:51.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:51.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:51.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:51.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:51.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:51.638 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:21:51.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2549769552' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:51.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:51 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:51.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:51 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:21:51.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:51 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:21:51.718 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:21:51.718 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:21:51.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:51.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:51.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:51.755 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:21:51.755 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:21:51.788 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:21:51.788 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:21:51.819 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:21:51.820 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:21:51.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:51.856 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:21:51.857 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:21:51.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:51.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:51.887 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:21:51.887 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:21:51.893 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:51.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:21:51.927 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:21:51.960 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:21:51.961 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:21:51.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:51.993 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:21:51.993 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:21:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2549769552' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:51 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:51 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:21:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:51 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:21:52.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:52.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:52.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:52.023 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:21:52.024 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:21:52.060 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:21:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:21:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2549769552' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:51 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:21:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:51 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:21:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:51 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:21:52.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:52.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:52.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:52.217 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:21:52.217 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:21:52.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:52.277 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:52.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:52.278 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:52.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:52.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:52.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:52.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:52.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:52.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:52.634 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-09-02T10:21:52.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:52.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:52.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:52.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:52.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:52.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:52.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:52.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:52 smithi187 ceph-mon[20072]: pgmap v1413: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 533 op/s; 6934/93498 objects degraded (7.416%); 5889/93498 objects misplaced (6.299%); 6.7 MiB/s, 107 objects/s recovering 2024-09-02T10:21:52.908 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:52 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:52.908 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1972824003' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:21:52.908 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:52 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:21:52.908 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:52 smithi187 ceph-mon[20072]: osdmap e608: 12 total, 12 up, 12 in 2024-09-02T10:21:52.908 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:52 smithi187 ceph-mon[20072]: from='client.26956 172.21.15.151:0/1342625623' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:52.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:52 smithi151 ceph-mon[16639]: pgmap v1413: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 533 op/s; 6934/93498 objects degraded (7.416%); 5889/93498 objects misplaced (6.299%); 6.7 MiB/s, 107 objects/s recovering 2024-09-02T10:21:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:52 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1972824003' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:21:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:52 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:21:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:52 smithi151 ceph-mon[16639]: osdmap e608: 12 total, 12 up, 12 in 2024-09-02T10:21:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:52 smithi151 ceph-mon[16639]: from='client.26956 172.21.15.151:0/1342625623' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:52.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:52.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:52.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:52.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:52 smithi162 ceph-mon[20279]: pgmap v1413: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.9 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 533 op/s; 6934/93498 objects degraded (7.416%); 5889/93498 objects misplaced (6.299%); 6.7 MiB/s, 107 objects/s recovering 2024-09-02T10:21:52.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:52 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:52.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1972824003' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:21:52.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:52 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:21:52.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:52 smithi162 ceph-mon[20279]: osdmap e608: 12 total, 12 up, 12 in 2024-09-02T10:21:52.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:52 smithi162 ceph-mon[20279]: from='client.26956 172.21.15.151:0/1342625623' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:21:53.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:53.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:53.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:53.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:53.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:53.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:53.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:53.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:53.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:53.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:53.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:53.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:53.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:53.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:53.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:53.553 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:21:53.553 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:21:53.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:21:53.553 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:21:53.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.553 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25507.0:75251 4.11 4:8ba31329:::benchmark_data_smithi187_76542_object75250:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 575.88343931099996, 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37234742100000001, 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25507", 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 75251 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664949+0000", 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664951+0000", 2024-09-02T10:21:53.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9599999999999999e-06 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.664999+0000", 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7738999999999998e-05 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665013+0000", 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4209999999999999e-05 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665021+0000", 2024-09-02T10:21:53.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.481e-06 2024-09-02T10:21:53.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665075+0000", 2024-09-02T10:21:53.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4642999999999998e-05 2024-09-02T10:21:53.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665173+0000", 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8022000000000004e-05 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,7", 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.665258+0000", 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4666000000000006e-05 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.666435+0000", 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011775080000000001 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:17.666722+0000", 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00028679699999999999 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037199+0000", 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37047664200000002 2024-09-02T10:21:53.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037256+0000", 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7605999999999997e-05 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:12:18.037297+0000", 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0147e-05 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:50217 5.1 5:84aaa397:::benchmark_data_smithi187_89405_object50216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e455)", 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 389.75663134500002, 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.384074792, 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 50217 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.561 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791757+0000", 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791759+0000", 2024-09-02T10:21:53.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.454e-06 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791785+0000", 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6838e-05 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791800+0000", 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4997999999999999e-05 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791808+0000", 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2370000000000003e-06 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791862+0000", 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4765000000000002e-05 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.791966+0000", 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010337500000000001 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.792051+0000", 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4779999999999998e-05 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.793473+0000", 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014225819999999999 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:23.793928+0000", 2024-09-02T10:21:53.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045518199999999999 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175728+0000", 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38180008999999998 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175787+0000", 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8498999999999997e-05 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:15:24.175832+0000", 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4991999999999999e-05 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.16,RELEASE,e:497,from:1,reservation_nonce:0)", 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 294.91169620199997, 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58412980699999995, 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.567 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636693+0000", 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6e-07 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636695+0000", 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.841e-06 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220721+0000", 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58402521600000001 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220727+0000", 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4219999999999997e-06 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220798+0000", 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1131999999999996e-05 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220810+0000", 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1863e-05 2024-09-02T10:21:53.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220822+0000", 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2072999999999999e-05 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:205302 5.1 5:85fde85c:::benchmark_data_smithi187_89405_object101697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 216.306867974, 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31845405399999999, 2024-09-02T10:21:53.571 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 205302 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241523+0000", 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3429999999999998e-06 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241533+0000", 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0139999999999999e-05 2024-09-02T10:21:53.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241562+0000", 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8600000000000001e-05 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241581+0000", 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9656e-05 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241772+0000", 2024-09-02T10:21:53.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019121200000000001 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241906+0000", 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013322299999999999 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241975+0000", 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9129999999999997e-05 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.245690+0000", 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037148390000000002 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.246234+0000", 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00054468699999999997 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559894+0000", 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31365966099999998 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559928+0000", 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3813000000000003e-05 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559975+0000", 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6749999999999998e-05 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.577 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:205324 5.1 5:85edc945:::benchmark_data_smithi187_89405_object101719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 215.94613020599999, 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38627865099999997, 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 205324 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602260+0000", 2024-09-02T10:21:53.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4300000000000001e-06 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602266+0000", 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6599999999999998e-06 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602283+0000", 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6646e-05 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602307+0000", 2024-09-02T10:21:53.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3853999999999999e-05 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602361+0000", 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4200000000000003e-05 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602447+0000", 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6116e-05 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:21:53.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602524+0000", 2024-09-02T10:21:53.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6596999999999999e-05 2024-09-02T10:21:53.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.582 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:53.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.606604+0000", 2024-09-02T10:21:53.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040804589999999998 2024-09-02T10:21:53.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.606691+0000", 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6608999999999997e-05 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988466+0000", 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38177503699999998 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988502+0000", 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6217000000000002e-05 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988537+0000", 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4826000000000002e-05 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:43141 6.c 6:30fc4bbf:::benchmark_data_smithi187_102448_object43140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e554)", 2024-09-02T10:21:53.585 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:14.433321+0000", 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 159.115067377, 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22430181399999999, 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 43141 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433321+0000", 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433321+0000", 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433322+0000", 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8599999999999996e-07 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433400+0000", 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8250000000000005e-05 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433406+0000", 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8390000000000003e-06 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433410+0000", 2024-09-02T10:21:53.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9419999999999997e-06 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433461+0000", 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0822999999999998e-05 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433600+0000", 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013943699999999999 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:21:53.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.433763+0000", 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016227400000000001 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.435509+0000", 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017465650000000001 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.435671+0000", 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016191799999999999 2024-09-02T10:21:53.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.657514+0000", 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22184321900000001 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.657582+0000", 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7825999999999997e-05 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.591 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:14.657623+0000", 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0735000000000002e-05 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:100267 6.b 6:dbd5fb01:::benchmark_data_smithi187_102448_object100266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:10.945411+0000", 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 102.60297719499999, 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54857257299999995, 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.592 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 100267 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945411+0000", 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945411+0000", 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945413+0000", 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3850000000000001e-06 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945444+0000", 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0843999999999998e-05 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945455+0000", 2024-09-02T10:21:53.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1107000000000001e-05 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945462+0000", 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9029999999999998e-06 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945541+0000", 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9658000000000002e-05 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945679+0000", 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013753800000000001 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945835+0000", 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015657799999999999 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949089+0000", 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032539589999999998 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949384+0000", 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029429199999999998 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493901+0000", 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54451749699999996 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493945+0000", 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3469e-05 2024-09-02T10:21:53.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493984+0000", 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9342999999999999e-05 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26524.0:100267 6.b e572/533 ondisk, result = 0)", 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:10.949464+0000", 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 102.598924602, 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54452646699999996, 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.598 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949464+0000", 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949464+0000", 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949465+0000", 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5099999999999999e-07 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949468+0000", 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3440000000000001e-06 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493766+0000", 2024-09-02T10:21:53.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54429846599999998 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493783+0000", 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7013000000000001e-05 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493889+0000", 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105537 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493897+0000", 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1009999999999992e-06 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493990+0000", 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3355000000000003e-05 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.602 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:121249 6.c 6:31dbf052:::benchmark_data_smithi187_102448_object20977:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e579)", 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:34.368226+0000", 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 79.180162773999996, 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23468649699999999, 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 121249 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.368226+0000", 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.368226+0000", 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.368227+0000", 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9800000000000003e-07 2024-09-02T10:21:53.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.368230+0000", 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9129999999999996e-06 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.368239+0000", 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.986e-06 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.368244+0000", 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0669999999999999e-06 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.371154+0000", 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029091030000000001 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.371260+0000", 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106112 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.371360+0000", 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000100445 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.378963+0000", 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0076025449999999996 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.378980+0000", 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7051999999999999e-05 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.602815+0000", 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22383554 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.602861+0000", 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5408999999999998e-05 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:34.602912+0000", 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1527000000000003e-05 2024-09-02T10:21:53.608 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157268 6.b 6:db1c5c67:::benchmark_data_smithi187_102448_object56996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.723542323, 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62417882300000005, 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.609 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157268 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824847+0000", 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824850+0000", 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.76e-06 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824860+0000", 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2690000000000005e-06 2024-09-02T10:21:53.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824876+0000", 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5882000000000001e-05 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824934+0000", 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8780000000000003e-05 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.825044+0000", 2024-09-02T10:21:53.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010938900000000001 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.825142+0000", 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8009000000000003e-05 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830102+0000", 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049605530000000004 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.838185+0000", 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080826430000000005 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.448954+0000", 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61076896000000003 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.448992+0000", 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7732999999999999e-05 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449025+0000", 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3315999999999997e-05 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157271 6.6 6:69d4aafc:::benchmark_data_smithi187_102448_object56999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.717987684000001, 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62368362300000002, 2024-09-02T10:21:53.615 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157271 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830403+0000", 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7179999999999999e-06 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830409+0000", 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.703e-06 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830424+0000", 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4961e-05 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830432+0000", 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4880000000000001e-06 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830495+0000", 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3758999999999997e-05 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830625+0000", 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012918000000000001 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830756+0000", 2024-09-02T10:21:53.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013138200000000001 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.835882+0000", 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051260960000000001 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.836280+0000", 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039814100000000002 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454013+0000", 2024-09-02T10:21:53.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61773251699999998 2024-09-02T10:21:53.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454048+0000", 2024-09-02T10:21:53.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5559999999999998e-05 2024-09-02T10:21:53.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454084+0000", 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6118000000000002e-05 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157274 6.6 6:6387cbb8:::benchmark_data_smithi187_102448_object57002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.71095953, 2024-09-02T10:21:53.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61674791900000003, 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157274 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837431+0000", 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7010000000000001e-06 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837437+0000", 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0379999999999996e-06 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837448+0000", 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1557e-05 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837456+0000", 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.396e-06 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837510+0000", 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4666000000000001e-05 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.625 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.626 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837636+0000", 2024-09-02T10:21:53.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125313 2024-09-02T10:21:53.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:21:53.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837757+0000", 2024-09-02T10:21:53.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121212 2024-09-02T10:21:53.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.845343+0000", 2024-09-02T10:21:53.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075863880000000003 2024-09-02T10:21:53.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.847219+0000", 2024-09-02T10:21:53.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00187546 2024-09-02T10:21:53.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454127+0000", 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60690782600000004 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454156+0000", 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9281999999999998e-05 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454177+0000", 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1080000000000001e-05 2024-09-02T10:21:53.629 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157277 6.6 6:67446ce0:::benchmark_data_smithi187_102448_object57005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.710427920000001, 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61628705500000003, 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.630 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157277 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0699999999999996e-07 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837964+0000", 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8569999999999999e-06 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837973+0000", 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0159999999999997e-06 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837978+0000", 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7770000000000002e-06 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842046+0000", 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040683289999999999 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842130+0000", 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3751000000000005e-05 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842199+0000", 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8411999999999995e-05 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.845512+0000", 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003313795 2024-09-02T10:21:53.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.847365+0000", 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018529009999999999 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454203+0000", 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60683791600000003 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454229+0000", 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5609000000000002e-05 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454248+0000", 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8885e-05 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157282 6.b 6:d751a93a:::benchmark_data_smithi187_102448_object57010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:21:53.636 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.674457463, 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57523664399999996, 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157282 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873933+0000", 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.08e-06 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.638 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873941+0000", 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0919999999999998e-06 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873961+0000", 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0231e-05 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873971+0000", 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2969999999999992e-06 2024-09-02T10:21:53.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874034+0000", 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3257999999999996e-05 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874177+0000", 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142712 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874303+0000", 2024-09-02T10:21:53.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126248 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.878010+0000", 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037068610000000001 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.946302+0000", 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.068292383999999998 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449070+0000", 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50276810199999999 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449124+0000", 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3712999999999999e-05 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449168+0000", 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3665999999999998e-05 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.642 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157307 6.6 6:675a1a62:::benchmark_data_smithi187_102448_object57035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.530368782, 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43718100199999999, 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.643 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157307 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018021+0000", 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6559999999999999e-06 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018029+0000", 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6310000000000003e-06 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018046+0000", 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6779999999999999e-05 2024-09-02T10:21:53.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018054+0000", 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3180000000000007e-06 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018619+0000", 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056491400000000002 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018723+0000", 2024-09-02T10:21:53.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103619 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018800+0000", 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7859000000000001e-05 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.022686+0000", 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038855399999999998 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031392+0000", 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087064819999999998 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455118+0000", 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42372546999999999 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455167+0000", 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8773000000000002e-05 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455201+0000", 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.396e-05 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.649 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157310 6.b 6:d937f28e:::benchmark_data_smithi187_102448_object57038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 25.517264355999998, 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42301055599999998, 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157310 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031125+0000", 2024-09-02T10:21:53.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8599999999999997e-07 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031130+0000", 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7949999999999998e-06 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031141+0000", 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0735e-05 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031148+0000", 2024-09-02T10:21:53.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4189999999999996e-06 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031196+0000", 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7571000000000002e-05 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031278+0000", 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2262000000000006e-05 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031363+0000", 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5279999999999997e-05 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.035027+0000", 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036643140000000001 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.290856+0000", 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25582839899999998 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454040+0000", 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.163183774 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454092+0000", 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2224e-05 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454135+0000", 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2896999999999999e-05 2024-09-02T10:21:53.655 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:171979 6.6 6:6c2351d7:::benchmark_data_smithi187_102448_object71707:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:52.102920+0000", 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.4454687850000001, 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044263341999999997, 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 171979 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.102920+0000", 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.102920+0000", 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.102921+0000", 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3549999999999999e-06 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.102927+0000", 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1750000000000002e-06 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.102940+0000", 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3064999999999999e-05 2024-09-02T10:21:53.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.102948+0000", 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7880000000000007e-06 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.102997+0000", 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8915999999999997e-05 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.103128+0000", 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013124299999999999 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.103258+0000", 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000129326 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.107230+0000", 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039723859999999996 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.108193+0000", 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00096295899999999999 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.147080+0000", 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.038886878999999999 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.147137+0000", 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7222999999999997e-05 2024-09-02T10:21:53.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.147183+0000", 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6026999999999998e-05 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:172164 6.b 6:d1cf9d8e:::benchmark_data_smithi187_102448_object71892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:52.375585+0000", 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.172803904, 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044181316999999998, 2024-09-02T10:21:53.662 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172164 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.375585+0000", 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.375585+0000", 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.375585+0000", 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8100000000000002e-07 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.375588+0000", 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0630000000000002e-06 2024-09-02T10:21:53.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.375605+0000", 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6226999999999999e-05 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.375611+0000", 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8199999999999999e-06 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.375659+0000", 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7380999999999999e-05 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.375772+0000", 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000113307 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.375883+0000", 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011055700000000001 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.380762+0000", 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00487895 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.382090+0000", 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013283069999999999 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.419618+0000", 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037528463999999997 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.419717+0000", 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8715000000000006e-05 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.419766+0000", 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8745000000000002e-05 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:172263 6.b 6:d479ab62:::benchmark_data_smithi187_102448_object71991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e608)", 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:52.534973+0000", 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.0134150850000001, 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.044831513000000003, 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172263 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.534973+0000", 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.534973+0000", 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.534974+0000", 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2099999999999996e-07 2024-09-02T10:21:53.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.534978+0000", 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8179999999999997e-06 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.534987+0000", 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3919999999999992e-06 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.534994+0000", 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3749999999999999e-06 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.539261+0000", 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042671949999999997 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.539339+0000", 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7632999999999993e-05 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.539413+0000", 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4616000000000005e-05 2024-09-02T10:21:53.672 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.543576+0000", 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0041627180000000001 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.561504+0000", 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017928522999999998 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.673 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.579715+0000", 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.018210924 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.579760+0000", 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4393e-05 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:52.579805+0000", 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5204999999999998e-05 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.674 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:172827 6.b 6:dceedf50:::benchmark_data_smithi187_102448_object72555:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e609)", 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:53.383033+0000", 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.165355853, 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.042485088999999997, 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 172827 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.675 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.383033+0000", 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.383033+0000", 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.676 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.383034+0000", 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1090000000000001e-06 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.383038+0000", 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5700000000000003e-06 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.383055+0000", 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7028e-05 2024-09-02T10:21:53.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.383062+0000", 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1010000000000003e-06 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.383136+0000", 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3814999999999997e-05 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.383243+0000", 2024-09-02T10:21:53.678 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106653 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.383346+0000", 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103179 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.392789+0000", 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0094430989999999999 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.392841+0000", 2024-09-02T10:21:53.679 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1461000000000001e-05 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.425392+0000", 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.032551575999999999 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.425463+0000", 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1019999999999994e-05 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:21:53.680 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:21:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:53.425518+0000", 2024-09-02T10:21:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4478000000000003e-05 2024-09-02T10:21:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:21:53.681 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:21:53.681 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:21:53.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:53.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:53.683 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:53.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:53.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1972824003' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:21:53.803 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:53 smithi151 ceph-mon[16639]: osdmap e609: 12 total, 12 up, 12 in 2024-09-02T10:21:53.809 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:53.811 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:53.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:53.916 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1972824003' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:21:53.917 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:53 smithi187 ceph-mon[20072]: osdmap e609: 12 total, 12 up, 12 in 2024-09-02T10:21:53.939 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:53.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:54.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:54.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:54.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:54.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1972824003' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:21:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:53 smithi162 ceph-mon[20279]: osdmap e609: 12 total, 12 up, 12 in 2024-09-02T10:21:54.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:54.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:54.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:54.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:54.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:54.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:54.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:54.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:54.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:54.570 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:54.596 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:54.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:54.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:54 smithi151 ceph-mon[16639]: pgmap v1416: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 6931/86403 objects degraded (8.022%); 5703/86403 objects misplaced (6.600%); 7.0 MiB/s, 112 objects/s recovering 2024-09-02T10:21:54.720 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:54 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6931/86403 objects degraded (8.022%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:21:54.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:54.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:54.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:54.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:54.894 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:54.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:54.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:54 smithi187 ceph-mon[20072]: pgmap v1416: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 6931/86403 objects degraded (8.022%); 5703/86403 objects misplaced (6.600%); 7.0 MiB/s, 112 objects/s recovering 2024-09-02T10:21:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:54 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6931/86403 objects degraded (8.022%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:21:55.018 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:55.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:55.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:21:55.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:54 smithi162 ceph-mon[20279]: pgmap v1416: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 838 op/s; 6931/86403 objects degraded (8.022%); 5703/86403 objects misplaced (6.600%); 7.0 MiB/s, 112 objects/s recovering 2024-09-02T10:21:55.048 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:54 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6931/86403 objects degraded (8.022%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:21:55.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:55.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:55.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:55.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:55.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:55.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:55.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:55.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:21:55.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:55.396 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:55.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:55.397 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:55.400 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:21:55.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:21:55.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:55.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:21:55.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:55.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:55.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:55.662 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:55.762 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:55.786 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:55.788 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:55.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:55.919 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:55.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:55.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:56.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:56.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-09-02T10:21:56.042 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:56.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:56.086 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:56.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:56.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:56.169 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:56.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:56.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:56.292 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:56.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:56.293 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:56.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:56.439 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:56.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:56.464 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:21:56.464 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:174748 6.6 e609/559 6:6ae531ba:::benchmark_data_smithi187_102448_object74476:head v 609'10962, mlcod=609'10959)", 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:56.411164+0000", 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.050715033, 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.050742699000000002, 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.411164+0000", 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.411164+0000", 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:21:56.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.411165+0000", 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4900000000000005e-07 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.411172+0000", 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.52e-06 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.431893+0000", 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.020721045 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.431900+0000", 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0439999999999998e-06 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.442379+0000", 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010479110999999999 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.442415+0000", 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5655000000000003e-05 2024-09-02T10:21:56.468 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:174760 6.6 e609/559 6:65f3a416:::benchmark_data_smithi187_102448_object74488:head v 609'10963, mlcod=609'10959)", 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:56.431783+0000", 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.030095332999999998, 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.030185986000000001, 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:21:56.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.431783+0000", 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.431783+0000", 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.431784+0000", 2024-09-02T10:21:56.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0699999999999996e-07 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.431791+0000", 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.277e-06 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.444392+0000", 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012600323 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:21:56.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.444399+0000", 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5170000000000002e-06 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.447898+0000", 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034984880000000001 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.447969+0000", 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1316999999999995e-05 2024-09-02T10:21:56.472 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:174765 6.6 e609/559 6:62abd88a:::benchmark_data_smithi187_102448_object74493:head v 609'10964, mlcod=609'10959)", 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:56.444366+0000", 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.017513177000000001, 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0176592, 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:21:56.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.444366+0000", 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.444366+0000", 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.444366+0000", 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3099999999999997e-07 2024-09-02T10:21:56.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.444372+0000", 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6189999999999999e-06 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.450450+0000", 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0060781990000000003 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:21:56.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.450457+0000", 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4170000000000003e-06 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.453658+0000", 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032009529999999999 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.453705+0000", 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6554000000000002e-05 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:21:56.476 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:21:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:174771 6.7 e609/565)", 2024-09-02T10:21:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:56.461993+0000", 2024-09-02T10:21:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4294967295.999886, 2024-09-02T10:21:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0778000000000005e-05, 2024-09-02T10:21:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:21:56.477 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:21:56.478 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-02T10:21:56.478 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:21:56.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:21:56.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.461993+0000", 2024-09-02T10:21:56.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.461993+0000", 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.461994+0000", 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0499999999999999e-06 2024-09-02T10:21:56.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.462002+0000", 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0539999999999995e-06 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.462027+0000", 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5548e-05 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:56.462041+0000", 2024-09-02T10:21:56.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3730999999999999e-05 2024-09-02T10:21:56.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:21:56.481 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:21:56.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:21:56.481 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:21:56.481 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:21:56.481 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 4 2024-09-02T10:21:56.481 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:21:56.541 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:56.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:56.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:56.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:56.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:56.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:56.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:56.737 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:56.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:56.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:56.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:56.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:56 smithi162 ceph-mon[20279]: pgmap v1417: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s; 6947/84096 objects degraded (8.261%); 5314/84096 objects misplaced (6.319%); 8.2 MiB/s, 132 objects/s recovering 2024-09-02T10:21:56.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:56 smithi151 ceph-mon[16639]: pgmap v1417: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s; 6947/84096 objects degraded (8.261%); 5314/84096 objects misplaced (6.319%); 8.2 MiB/s, 132 objects/s recovering 2024-09-02T10:21:56.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:56.954 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:56.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:56 smithi187 ceph-mon[20072]: pgmap v1417: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 585 op/s; 6947/84096 objects degraded (8.261%); 5314/84096 objects misplaced (6.319%); 8.2 MiB/s, 132 objects/s recovering 2024-09-02T10:21:57.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:57.081 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:57.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:57.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:57.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:57.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:57.315 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:57.337 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:57.338 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:57.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:57.470 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:57.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:57.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:21:57.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:21:57.595 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:57.596 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:21:57.664 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:21:57.673 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:21:57.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:21:57.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:57.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:57.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:21:57.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:57.895 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:57.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/593044165' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:57.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:21:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/593044165' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:58.034 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:58.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:58.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:21:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/593044165' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:21:58.115 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:21:58.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:58.172 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:58.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:58.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:58.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:58.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:58.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:21:58.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:58.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:21:58.539 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:58.561 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:58.562 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:58.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:58.671 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-09-02T10:21:58.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:58.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:58.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:58.835 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:58.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:58.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:58 smithi151 ceph-mon[16639]: pgmap v1418: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 548 op/s; 6947/82212 objects degraded (8.450%); 5314/82212 objects misplaced (6.464%); 7.0 MiB/s, 111 objects/s recovering 2024-09-02T10:21:58.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:58 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:58.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/593044165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:58.903 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:58 smithi151 ceph-mon[16639]: osdmap e610: 12 total, 12 up, 12 in 2024-09-02T10:21:58.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1580770487' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:21:58.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:58 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:21:58.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:58 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:21:58.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:58 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:21:58.937 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:58.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:58.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:58 smithi187 ceph-mon[20072]: pgmap v1418: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 548 op/s; 6947/82212 objects degraded (8.450%); 5314/82212 objects misplaced (6.464%); 7.0 MiB/s, 111 objects/s recovering 2024-09-02T10:21:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:58 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/593044165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:58 smithi187 ceph-mon[20072]: osdmap e610: 12 total, 12 up, 12 in 2024-09-02T10:21:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1580770487' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:21:59.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:58 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:21:59.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:58 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:21:59.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:58 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:21:59.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:21:59.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:59.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:21:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:58 smithi162 ceph-mon[20279]: pgmap v1418: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 548 op/s; 6947/82212 objects degraded (8.450%); 5314/82212 objects misplaced (6.464%); 7.0 MiB/s, 111 objects/s recovering 2024-09-02T10:21:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:58 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:21:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/593044165' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:21:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:58 smithi162 ceph-mon[20279]: osdmap e610: 12 total, 12 up, 12 in 2024-09-02T10:21:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1580770487' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:21:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:58 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:21:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:58 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:21:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:58 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:21:59.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:21:59.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:59.217 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:59.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:21:59.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:21:59.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:21:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:59.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:21:59.446 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:21:59.468 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:59.469 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:21:59.570 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:21:59.599 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:59.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:59.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:21:59.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:59.729 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:59.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:59.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:21:59.739 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:21:59.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 10 1 2024-09-02T10:21:59.831 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:21:59.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:21:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:59.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:21:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:59 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:21:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:59 smithi151 ceph-mon[16639]: osdmap e611: 12 total, 12 up, 12 in 2024-09-02T10:21:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:21:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6947/82212 objects degraded (8.450%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:21:59.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:21:59.985 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:21:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:21:59.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:59 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:22:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:59 smithi187 ceph-mon[20072]: osdmap e611: 12 total, 12 up, 12 in 2024-09-02T10:22:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:21:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6947/82212 objects degraded (8.450%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:00.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:00.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:59 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:22:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:59 smithi162 ceph-mon[20279]: osdmap e611: 12 total, 12 up, 12 in 2024-09-02T10:22:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:21:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6947/82212 objects degraded (8.450%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:00.115 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:00.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:00.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:00.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:00.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:00.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:00.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:00.342 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:00.373 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:00.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:00.395 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:22:00.395 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:22:00.397 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:00.426 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:22:00.426 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:22:00.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:22:00.448 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:22:00.469 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:22:00.469 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:22:00.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:00.497 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:22:00.497 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:22:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:00.504 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:00.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:00.525 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:22:00.525 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:22:00.559 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:22:00.559 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:22:00.597 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:22:00.597 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:22:00.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:00.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:22:00.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:22:00.635 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:00.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:00.652 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:22:00.652 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:22:00.680 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:22:00.680 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:22:00.709 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:22:00.709 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:22:00.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:00.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:00.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:00.875 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:00.896 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:00.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:00.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:00 smithi151 ceph-mon[16639]: pgmap v1421: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 637 op/s; 6947/76500 objects degraded (9.081%); 5140/76500 objects misplaced (6.719%); 5.1 MiB/s, 81 objects/s recovering 2024-09-02T10:22:00.928 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3491218470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:00.969 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:22:00.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:00 smithi187 ceph-mon[20072]: pgmap v1421: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 637 op/s; 6947/76500 objects degraded (9.081%); 5140/76500 objects misplaced (6.719%); 5.1 MiB/s, 81 objects/s recovering 2024-09-02T10:22:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3491218470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:01.019 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:01.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:01.020 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:00 smithi162 ceph-mon[20279]: pgmap v1421: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.6 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 637 op/s; 6947/76500 objects degraded (9.081%); 5140/76500 objects misplaced (6.719%); 5.1 MiB/s, 81 objects/s recovering 2024-09-02T10:22:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3491218470' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:01.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:01.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:01.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:01.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:01.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:01.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:01.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:01.399 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:01.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:01.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:01.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:01.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:01.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:01.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:01.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:01.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:01.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:01.690 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-09-02T10:22:01.760 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:01.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:01.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:01.886 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:01.908 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:01.909 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/853615628' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:22:02.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:02.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:02.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/853615628' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:22:02.113 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/853615628' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:22:02.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:02.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:02.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:02.219 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:22:02.219 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:22:02.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:02.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:02.305 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:02.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:02.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:02.436 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:02.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:02.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-09-02T10:22:02.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:02.561 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:02.562 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:02.595 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:02.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:02.692 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:02.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:02.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:02 smithi151 ceph-mon[16639]: pgmap v1422: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 484 op/s; 6989/74763 objects degraded (9.348%); 4737/74763 objects misplaced (6.336%); 7.1 MiB/s, 113 objects/s recovering 2024-09-02T10:22:02.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/853615628' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:22:02.735 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:02 smithi151 ceph-mon[16639]: osdmap e612: 12 total, 12 up, 12 in 2024-09-02T10:22:02.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:02.824 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:02.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:02.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:02.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:02.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:02.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:02 smithi187 ceph-mon[20072]: pgmap v1422: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 484 op/s; 6989/74763 objects degraded (9.348%); 4737/74763 objects misplaced (6.336%); 7.1 MiB/s, 113 objects/s recovering 2024-09-02T10:22:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/853615628' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:22:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:02 smithi187 ceph-mon[20072]: osdmap e612: 12 total, 12 up, 12 in 2024-09-02T10:22:03.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:03.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:03.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:02 smithi162 ceph-mon[20279]: pgmap v1422: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 484 op/s; 6989/74763 objects degraded (9.348%); 4737/74763 objects misplaced (6.336%); 7.1 MiB/s, 113 objects/s recovering 2024-09-02T10:22:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/853615628' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:22:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:02 smithi162 ceph-mon[20279]: osdmap e612: 12 total, 12 up, 12 in 2024-09-02T10:22:03.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:03.229 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:03.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:03.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:03.344 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:03.360 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:03.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:03.463 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:03.488 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:03.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:03.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:03.611 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:03.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:03.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:03.705 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:22:03.706 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/233974969' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:03.706 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:03 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:03.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:03.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:03.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:03.773 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:22:03.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:03.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:03.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:03.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:03.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:04.004 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:04.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:04.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/233974969' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:03 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/233974969' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:03 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:04.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:04.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:04.128 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:04.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:04.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:04.261 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:04.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:04.362 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:04.386 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:04.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:04.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:04.520 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:04.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:04.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:04.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:04.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:04.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:04.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:04.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:04.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:04.870 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:04.893 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:04.894 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:04 smithi151 ceph-mon[16639]: pgmap v1424: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 6989/69579 objects degraded (10.045%); 4532/69579 objects misplaced (6.513%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:22:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:04 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:04 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:22:04.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:04 smithi151 ceph-mon[16639]: osdmap e613: 12 total, 12 up, 12 in 2024-09-02T10:22:04.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:04 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:04.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:04 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:04.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6989/69579 objects degraded (10.045%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:04.995 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:04 smithi187 ceph-mon[20072]: pgmap v1424: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 6989/69579 objects degraded (10.045%); 4532/69579 objects misplaced (6.513%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:22:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:04 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:04 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:22:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:04 smithi187 ceph-mon[20072]: osdmap e613: 12 total, 12 up, 12 in 2024-09-02T10:22:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:04 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:04 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6989/69579 objects degraded (10.045%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:05.022 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:05.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:05.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:04 smithi162 ceph-mon[20279]: pgmap v1424: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 701 op/s; 6989/69579 objects degraded (10.045%); 4532/69579 objects misplaced (6.513%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:22:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:04 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:04 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:22:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:04 smithi162 ceph-mon[20279]: osdmap e613: 12 total, 12 up, 12 in 2024-09-02T10:22:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:04 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:04 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6989/69579 objects degraded (10.045%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:05.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:05.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:05.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:05.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:05.285 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:05.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:05.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:05.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:05.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:05.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:05.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:05.551 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:05.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:05.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:05.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:05.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:05.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:05.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:05.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:05.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:05.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:05.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:22:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:05.963 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:05.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:05 smithi187 ceph-mon[20072]: osdmap e614: 12 total, 12 up, 12 in 2024-09-02T10:22:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:05 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:22:06.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:05 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:22:06.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:05 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:06.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:05 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:06.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:06.086 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:06.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:06.087 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:06.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:05 smithi162 ceph-mon[20279]: osdmap e614: 12 total, 12 up, 12 in 2024-09-02T10:22:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:05 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:22:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:05 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:22:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:05 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:05 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:06.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:05 smithi151 ceph-mon[16639]: osdmap e614: 12 total, 12 up, 12 in 2024-09-02T10:22:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:05 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:22:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:05 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:22:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:05 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:06.108 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:05 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:06.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:06.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:06.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:06.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:06.313 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:06.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:06.341 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:22:06.358 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:06.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:06.460 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:06.497 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:06.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:06.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:06.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:06.620 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:06.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:06.621 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:06.722 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:06.754 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:06.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:06.777 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-09-02T10:22:06.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:06.879 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:06.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:06.981 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:07.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:07.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:07.004 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:07.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:06 smithi151 ceph-mon[16639]: pgmap v1427: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 6989/66051 objects degraded (10.581%); 4146/66051 objects misplaced (6.277%); 9.9 MiB/s, 158 objects/s recovering 2024-09-02T10:22:07.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:06 smithi151 ceph-mon[16639]: osdmap e615: 12 total, 12 up, 12 in 2024-09-02T10:22:07.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3691351063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:22:07.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:06 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:22:07.060 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:22:07.060 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:22:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:06 smithi162 ceph-mon[20279]: pgmap v1427: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 6989/66051 objects degraded (10.581%); 4146/66051 objects misplaced (6.277%); 9.9 MiB/s, 158 objects/s recovering 2024-09-02T10:22:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:06 smithi162 ceph-mon[20279]: osdmap e615: 12 total, 12 up, 12 in 2024-09-02T10:22:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3691351063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:22:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:06 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:22:07.100 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:22:07.100 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:22:07.104 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:07.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:22:07.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:22:07.134 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:07.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:07.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:07.163 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:22:07.163 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:22:07.204 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:22:07.204 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:22:07.236 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:06 smithi187 ceph-mon[20072]: pgmap v1427: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 580 op/s; 6989/66051 objects degraded (10.581%); 4146/66051 objects misplaced (6.277%); 9.9 MiB/s, 158 objects/s recovering 2024-09-02T10:22:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:06 smithi187 ceph-mon[20072]: osdmap e615: 12 total, 12 up, 12 in 2024-09-02T10:22:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3691351063' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:22:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:06 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:22:07.257 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:07.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:07.258 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:07.359 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:07.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:07.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:07.383 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:07.441 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:22:07.441 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:22:07.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:22:07.479 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:22:07.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:07.504 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:07.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:07.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:07.611 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:22:07.611 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:22:07.636 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:22:07.636 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:22:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:07.638 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:07.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:07.667 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:22:07.667 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:22:07.704 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:22:07.704 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:22:07.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:07.740 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:22:07.741 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:22:07.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:07.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:07.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:07 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:22:07.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:07 smithi151 ceph-mon[16639]: osdmap e616: 12 total, 12 up, 12 in 2024-09-02T10:22:07.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:07 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:07.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:07 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:07.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3729773839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:07.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:07.905 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:07.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:07.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 2 1 2024-09-02T10:22:08.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:08.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:08.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:08.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:08.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:07 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:22:08.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:07 smithi162 ceph-mon[20279]: osdmap e616: 12 total, 12 up, 12 in 2024-09-02T10:22:08.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:07 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:08.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:07 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:08.057 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3729773839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:08.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:08.163 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:22:08.163 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:22:08.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:22:08.163 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:22:08.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25507.0:75261 4.e 4:7d9bdb56:::benchmark_data_smithi187_76542_object75260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:17.679207+0000", 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 590.48069300300006, 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.39759054999999999, 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25507", 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 75261 2024-09-02T10:22:08.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.165 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679207+0000", 2024-09-02T10:22:08.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679207+0000", 2024-09-02T10:22:08.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679209+0000", 2024-09-02T10:22:08.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.722e-06 2024-09-02T10:22:08.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679269+0000", 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0007000000000003e-05 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679280+0000", 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1413999999999999e-05 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679295+0000", 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4572999999999999e-05 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.167 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679367+0000", 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1791000000000004e-05 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679550+0000", 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000183019 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,10", 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.679705+0000", 2024-09-02T10:22:08.168 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015513500000000001 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.681783+0000", 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020785370000000001 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.682235+0000", 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045192700000000003 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.169 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076685+0000", 2024-09-02T10:22:08.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.39444963199999999 2024-09-02T10:22:08.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.170 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076746+0000", 2024-09-02T10:22:08.170 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1613999999999997e-05 2024-09-02T10:22:08.170 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.170 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.170 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076798+0000", 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1178999999999999e-05 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.25507.0:75261 4.e e371/340 ondisk, result = 0)", 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:17.682016+0000", 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 590.47788426299996, 2024-09-02T10:22:08.171 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.394789379, 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.682016+0000", 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.682016+0000", 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.172 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.682016+0000", 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.99e-07 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:17.682020+0000", 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1829999999999998e-06 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.173 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076598+0000", 2024-09-02T10:22:08.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.394578759 2024-09-02T10:22:08.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.174 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.174 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.174 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076606+0000", 2024-09-02T10:22:08.174 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3089999999999998e-06 2024-09-02T10:22:08.174 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.175 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076669+0000", 2024-09-02T10:22:08.175 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3819999999999995e-05 2024-09-02T10:22:08.175 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.175 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.175 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076679+0000", 2024-09-02T10:22:08.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5279999999999992e-06 2024-09-02T10:22:08.176 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.176 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.176 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.176 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:18.076805+0000", 2024-09-02T10:22:08.176 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126181 2024-09-02T10:22:08.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.176 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.176 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.177 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.177 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.177 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:54374 6.6 e556/549 6:66e3d897:::benchmark_data_smithi187_102448_object54373:head v 556'3449, mlcod=544'1288)", 2024-09-02T10:22:08.177 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:24.167782+0000", 2024-09-02T10:22:08.177 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 163.99211818200001, 2024-09-02T10:22:08.177 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.175206898, 2024-09-02T10:22:08.177 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.177 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.177 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.177 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:24.167782+0000", 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:24.167782+0000", 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:24.167783+0000", 2024-09-02T10:22:08.178 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0200000000000001e-07 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:24.167890+0000", 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107079 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:24.339474+0000", 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17158436099999999 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.179 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:24.339484+0000", 2024-09-02T10:22:08.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8570000000000007e-06 2024-09-02T10:22:08.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.180 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:24.339585+0000", 2024-09-02T10:22:08.180 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100998 2024-09-02T10:22:08.180 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.180 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.180 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.181 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:24.339658+0000", 2024-09-02T10:22:08.181 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2596999999999996e-05 2024-09-02T10:22:08.181 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.181 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.181 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:24.342935+0000", 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003277308 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:24.342989+0000", 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3896e-05 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.182 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:63394 6.2 6:45958ca0:::benchmark_data_smithi187_102448_object63393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:35.329719+0000", 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 152.830180689, 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.245115466, 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 63394 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.183 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329719+0000", 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329719+0000", 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.184 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329720+0000", 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0600000000000001e-07 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329791+0000", 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0882000000000003e-05 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329799+0000", 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9799999999999998e-06 2024-09-02T10:22:08.185 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329803+0000", 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.459e-06 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.331811+0000", 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020077670000000001 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.331940+0000", 2024-09-02T10:22:08.186 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129333 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.332083+0000", 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014232000000000001 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.333709+0000", 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016269209999999999 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.187 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.334159+0000", 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000449591 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.574771+0000", 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24061200899999999 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.574805+0000", 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4241e-05 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.188 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.574835+0000", 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9556999999999998e-05 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(6.7 560/549 [PushOp(6:e91a0f22:::benchmark_data_smithi187_102448_object8663:head, version: 539'550, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(6:e91a0f22:::benchmark_data_smithi187_102448_object8663:head@539'550, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:35.344530+0000", 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 152.81537011099999, 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19135517099999999, 2024-09-02T10:22:08.189 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.344530+0000", 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.344530+0000", 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.190 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.344531+0000", 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9800000000000002e-07 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.344558+0000", 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6965e-05 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.534293+0000", 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18973551399999999 2024-09-02T10:22:08.191 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.534323+0000", 2024-09-02T10:22:08.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9853e-05 2024-09-02T10:22:08.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.192 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.192 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.192 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.534463+0000", 2024-09-02T10:22:08.192 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139727 2024-09-02T10:22:08.192 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.534472+0000", 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3789999999999995e-06 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.535885+0000", 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014128350000000001 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.193 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:63431 6.e 6:7d443e59:::benchmark_data_smithi187_102448_object63430:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:35.356450+0000", 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 152.80344994399999, 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.181822611, 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 63431 2024-09-02T10:22:08.194 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.356450+0000", 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.356450+0000", 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.195 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.356454+0000", 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0319999999999997e-06 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.356490+0000", 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5578999999999999e-05 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.356506+0000", 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6129000000000001e-05 2024-09-02T10:22:08.196 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.356513+0000", 2024-09-02T10:22:08.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0160000000000003e-06 2024-09-02T10:22:08.197 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.197 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.197 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.197 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.356565+0000", 2024-09-02T10:22:08.197 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1867999999999999e-05 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.356690+0000", 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0001249 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.356806+0000", 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011656599999999999 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.198 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.358857+0000", 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020511779999999999 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.359031+0000", 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000173275 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.538206+0000", 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.179175801 2024-09-02T10:22:08.199 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.538244+0000", 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7836000000000002e-05 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.538273+0000", 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8430999999999999e-05 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.200 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:63461 6.2 6:457e4280:::benchmark_data_smithi187_102448_object63460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:35.571810+0000", 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 152.58808964299999, 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24504642700000001, 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.201 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 63461 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571810+0000", 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571810+0000", 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.202 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571817+0000", 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.64e-06 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571864+0000", 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7219000000000003e-05 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571897+0000", 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2789000000000001e-05 2024-09-02T10:22:08.203 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571913+0000", 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5503999999999999e-05 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.572024+0000", 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111838 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.204 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.572167+0000", 2024-09-02T10:22:08.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000142762 2024-09-02T10:22:08.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:22:08.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.572280+0000", 2024-09-02T10:22:08.205 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113335 2024-09-02T10:22:08.205 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.205 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.205 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.205 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.573933+0000", 2024-09-02T10:22:08.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001652803 2024-09-02T10:22:08.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.574669+0000", 2024-09-02T10:22:08.206 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073586099999999998 2024-09-02T10:22:08.206 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.206 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.206 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.206 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.816773+0000", 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24210336299999999 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.816825+0000", 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.236e-05 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.816857+0000", 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1952999999999997e-05 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.207 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:110262 6.2 6:40592b50:::benchmark_data_smithi187_102448_object9990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e576)", 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:22.062831+0000", 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 106.097069303, 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21039696099999999, 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.208 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 110262 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062831+0000", 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062831+0000", 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.209 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062832+0000", 2024-09-02T10:22:08.210 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6190000000000001e-06 2024-09-02T10:22:08.210 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.210 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.210 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.210 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062838+0000", 2024-09-02T10:22:08.211 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8159999999999999e-06 2024-09-02T10:22:08.211 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062863+0000", 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4516999999999999e-05 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062870+0000", 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7870000000000002e-06 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.212 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062928+0000", 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7073000000000001e-05 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.063033+0000", 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010552600000000001 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.063137+0000", 2024-09-02T10:22:08.213 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104378 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.065961+0000", 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002823365 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.067130+0000", 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001169074 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.214 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.273125+0000", 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20599530199999999 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.273182+0000", 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7024000000000002e-05 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.273228+0000", 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5479999999999998e-05 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.215 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:110281 6.2 6:466107a3:::benchmark_data_smithi187_102448_object10009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e576)", 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:22.085562+0000", 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 106.07433792099999, 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20897753599999999, 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.216 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.217 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 110281 2024-09-02T10:22:08.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.217 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.217 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.217 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.217 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085562+0000", 2024-09-02T10:22:08.217 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.217 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.218 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.218 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.218 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085562+0000", 2024-09-02T10:22:08.218 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.219 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.219 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.219 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085564+0000", 2024-09-02T10:22:08.219 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0999999999999998e-06 2024-09-02T10:22:08.219 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.220 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.220 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.220 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085572+0000", 2024-09-02T10:22:08.220 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2500000000000006e-06 2024-09-02T10:22:08.220 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.221 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085596+0000", 2024-09-02T10:22:08.221 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3176e-05 2024-09-02T10:22:08.221 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.221 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.221 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085605+0000", 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5400000000000001e-06 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085661+0000", 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5930999999999997e-05 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085806+0000", 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000144988 2024-09-02T10:22:08.222 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085916+0000", 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110084 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.089195+0000", 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032792860000000002 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.089560+0000", 2024-09-02T10:22:08.223 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036478200000000002 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.294468+0000", 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204907331 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.294502+0000", 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4137999999999997e-05 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.224 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.294540+0000", 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7929999999999998e-05 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:121269 6.7 e579/565 6:e666b9e5:::benchmark_data_smithi187_102448_object20997:head v 579'7625, mlcod=579'7622)", 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:34.395659+0000", 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 93.764241100000007, 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20071814499999999, 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.225 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.395659+0000", 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.395659+0000", 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.226 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.395660+0000", 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0299999999999999e-07 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.395664+0000", 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8409999999999999e-06 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.584140+0000", 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18847551700000001 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.227 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.584147+0000", 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6470000000000001e-06 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.584221+0000", 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4035000000000005e-05 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.584246+0000", 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5167000000000001e-05 2024-09-02T10:22:08.228 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.229 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.229 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:08.230 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:08.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:08.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.596164+0000", 2024-09-02T10:22:08.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011918529000000001 2024-09-02T10:22:08.231 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.231 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.231 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.231 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.596377+0000", 2024-09-02T10:22:08.231 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021280600000000001 2024-09-02T10:22:08.231 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.231 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:121271 6.e 6:75ac4dfb:::benchmark_data_smithi187_102448_object20999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e579)", 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:34.398988+0000", 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 93.760912106999996, 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223294623, 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.232 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 121271 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.398988+0000", 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.398988+0000", 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.233 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.398989+0000", 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.085e-06 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.398994+0000", 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.399006+0000", 2024-09-02T10:22:08.234 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1814000000000001e-05 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.399013+0000", 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2799999999999998e-06 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.399066+0000", 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.287e-05 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.235 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.399186+0000", 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011956299999999999 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.399298+0000", 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111878 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.402719+0000", 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034216699999999999 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.236 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.402824+0000", 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010517400000000001 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.622131+0000", 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21930701699999999 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.622232+0000", 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100552 2024-09-02T10:22:08.237 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.622283+0000", 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0599000000000001e-05 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:123123 6.e 6:7f8b2075:::benchmark_data_smithi187_102448_object22851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e580)", 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:36.604069+0000", 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 91.555831124999997, 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.178728315, 2024-09-02T10:22:08.238 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 123123 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.604069+0000", 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.239 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.604069+0000", 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.604070+0000", 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1659999999999999e-06 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.604075+0000", 2024-09-02T10:22:08.240 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8489999999999997e-06 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.604085+0000", 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7699999999999996e-06 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.604090+0000", 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4979999999999997e-06 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.241 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.604124+0000", 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4183999999999997e-05 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.604223+0000", 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8515000000000001e-05 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.604296+0000", 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3578999999999995e-05 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.242 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.607355+0000", 2024-09-02T10:22:08.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030584330000000002 2024-09-02T10:22:08.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.607730+0000", 2024-09-02T10:22:08.243 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00037504599999999997 2024-09-02T10:22:08.243 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.243 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.243 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.243 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.782695+0000", 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17496463800000001 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.782750+0000", 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4917999999999998e-05 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:36.782797+0000", 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7719000000000002e-05 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.244 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:135976 6.e 6:7520f53c:::benchmark_data_smithi187_102448_object35704:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.137894+0000", 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.022006365999999, 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.248802408, 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.245 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135976 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137894+0000", 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137894+0000", 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.246 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137896+0000", 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0899999999999999e-06 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137902+0000", 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1789999999999996e-06 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137918+0000", 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5664000000000001e-05 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.247 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137929+0000", 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1711e-05 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139544+0000", 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001614905 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139556+0000", 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1411e-05 2024-09-02T10:22:08.248 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.386584+0000", 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.247028571 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.386696+0000", 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111877 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.249 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:135977 6.e 6:7fb0371e:::benchmark_data_smithi187_102448_object35705:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.137955+0000", 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.021945438000003, 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.250164689, 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135977 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.250 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137955+0000", 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137955+0000", 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137955+0000", 2024-09-02T10:22:08.251 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6899999999999998e-07 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137958+0000", 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0620000000000001e-06 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137969+0000", 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0808000000000001e-05 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.252 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137974+0000", 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1869999999999996e-06 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139586+0000", 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001612112 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139593+0000", 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2400000000000001e-06 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.253 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.388023+0000", 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.248429916 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.388119+0000", 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5895000000000003e-05 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.254 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:135999 6.e 6:73eba362:::benchmark_data_smithi187_102448_object35727:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.137984+0000", 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.021915578000005, 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28469357299999998, 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135999 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.255 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137984+0000", 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137984+0000", 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.256 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137985+0000", 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9900000000000001e-07 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137987+0000", 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.227e-06 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137993+0000", 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8490000000000002e-06 2024-09-02T10:22:08.257 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137997+0000", 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2250000000000002e-06 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139619+0000", 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016216850000000001 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:22:08.258 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139625+0000", 2024-09-02T10:22:08.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4880000000000004e-06 2024-09-02T10:22:08.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.388162+0000", 2024-09-02T10:22:08.259 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24853692999999999 2024-09-02T10:22:08.259 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.259 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.259 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.259 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.388319+0000", 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015642299999999999 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.388478+0000", 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015945999999999999 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395778+0000", 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072996850000000002 2024-09-02T10:22:08.260 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395788+0000", 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0217999999999999e-05 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.420479+0000", 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024690614999999999 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.261 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.420548+0000", 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9344000000000006e-05 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.422678+0000", 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021300249999999998 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:136003 6.e 6:75c4b550:::benchmark_data_smithi187_102448_object35731:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:22:08.262 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.138003+0000", 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.021897233000004, 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.289552264, 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136003 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.263 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138003+0000", 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138003+0000", 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138003+0000", 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8500000000000002e-07 2024-09-02T10:22:08.264 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138005+0000", 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8449999999999999e-06 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138010+0000", 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3510000000000001e-06 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.265 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138014+0000", 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1520000000000002e-06 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139659+0000", 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016441649999999999 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139666+0000", 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1810000000000004e-06 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.266 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.392451+0000", 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25278470600000003 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.392588+0000", 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013727600000000001 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.392676+0000", 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8603000000000005e-05 2024-09-02T10:22:08.267 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.398948+0000", 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062717700000000003 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.400446+0000", 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001498044 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.425036+0000", 2024-09-02T10:22:08.268 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024589508 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.425131+0000", 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5035000000000001e-05 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.427555+0000", 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024243429999999998 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:136017 6.e 6:7f8ba5a7:::benchmark_data_smithi187_102448_object35745:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.138020+0000", 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.021880374000006, 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29218707199999999, 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136017 2024-09-02T10:22:08.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138020+0000", 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138020+0000", 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138020+0000", 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0199999999999998e-07 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138022+0000", 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1280000000000002e-06 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138027+0000", 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3159999999999996e-06 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138032+0000", 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2849999999999996e-06 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139738+0000", 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017061380000000001 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139745+0000", 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1409999999999999e-06 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395833+0000", 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25608827899999997 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395931+0000", 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7783999999999998e-05 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.396015+0000", 2024-09-02T10:22:08.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4227999999999994e-05 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.399079+0000", 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030640709999999998 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.400485+0000", 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001405917 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.427645+0000", 2024-09-02T10:22:08.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027160153999999999 2024-09-02T10:22:08.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.427728+0000", 2024-09-02T10:22:08.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2545000000000003e-05 2024-09-02T10:22:08.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.430207+0000", 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024787839999999999 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:136022 6.2 6:49ff13c6:::benchmark_data_smithi187_102448_object35750:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.138646+0000", 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 75.021254373999994, 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28079392800000003, 2024-09-02T10:22:08.277 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136022 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138646+0000", 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138646+0000", 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138646+0000", 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9100000000000003e-07 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138650+0000", 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8430000000000003e-06 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138662+0000", 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1304999999999999e-05 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138668+0000", 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.438e-06 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.140143+0000", 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001474593 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.140150+0000", 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.391e-06 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395737+0000", 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25558662700000001 2024-09-02T10:22:08.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395868+0000", 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013133400000000001 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395955+0000", 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6730999999999993e-05 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.400421+0000", 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044657969999999996 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.400835+0000", 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041487400000000003 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.416898+0000", 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016062238 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.416971+0000", 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3213000000000005e-05 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.419440+0000", 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024687530000000002 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.284 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:164074 6.e 6:761ca277:::benchmark_data_smithi187_102448_object63802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e604)", 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:39.984974+0000", 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 28.174926366000001, 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19538498500000001, 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 164074 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.984974+0000", 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.984974+0000", 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.984975+0000", 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.488e-06 2024-09-02T10:22:08.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.984982+0000", 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.844e-06 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.984995+0000", 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2689e-05 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.985003+0000", 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.259e-06 2024-09-02T10:22:08.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.988198+0000", 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031951369999999998 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.988294+0000", 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5889000000000003e-05 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:22:08.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.988426+0000", 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000132335 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.993270+0000", 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048435570000000001 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:40.016288+0000", 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023017854000000001 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.289 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:40.178207+0000", 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16191962400000001 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:40.178308+0000", 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100739 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:40.180359+0000", 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020505699999999998 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.290 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:176781 6.e 6:7a7398b5:::benchmark_data_smithi187_102448_object76509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e611)", 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:59.662826+0000", 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 8.4970737720000002, 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.177332395, 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:08.291 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 176781 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.662826+0000", 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.662826+0000", 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:08.292 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.662828+0000", 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.548e-06 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.662832+0000", 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6269999999999999e-06 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.662847+0000", 2024-09-02T10:22:08.293 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4583000000000001e-05 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.662854+0000", 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4930000000000003e-06 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.663659+0000", 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00080498699999999996 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:08.294 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.663779+0000", 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012072499999999999 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.663937+0000", 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000158071 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.667960+0000", 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040223200000000002 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.678264+0000", 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010304663 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.839088+0000", 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.160823519 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.839167+0000", 2024-09-02T10:22:08.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9426999999999997e-05 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.840159+0000", 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00099143199999999999 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:08.297 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:22:08.298 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:22:08.299 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:07 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:22:08.299 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:07 smithi187 ceph-mon[20072]: osdmap e616: 12 total, 12 up, 12 in 2024-09-02T10:22:08.299 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:07 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:08.299 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:07 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:08.299 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3729773839' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:08.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:08.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:08.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:08.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:08.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:08.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:08.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:08.598 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:08.623 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:08.623 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:08.662 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:08.716 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:08.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:08.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:08.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:08.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:08 smithi151 ceph-mon[16639]: pgmap v1430: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 431 op/s; 6989/64401 objects degraded (10.852%); 4146/64401 objects misplaced (6.438%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:22:08.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:08 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:22:08.834 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:08 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:22:08.834 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:08 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:22:08.834 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:08 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:22:08.834 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:08 smithi151 ceph-mon[16639]: osdmap e617: 12 total, 12 up, 12 in 2024-09-02T10:22:08.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:08.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:08.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:08.997 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:09.018 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:09.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:09.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:09.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:08 smithi162 ceph-mon[20279]: pgmap v1430: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 431 op/s; 6989/64401 objects degraded (10.852%); 4146/64401 objects misplaced (6.438%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:22:09.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:08 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:22:09.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:08 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:22:09.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:08 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:22:09.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:08 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:22:09.030 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:08 smithi162 ceph-mon[20279]: osdmap e617: 12 total, 12 up, 12 in 2024-09-02T10:22:09.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:22:09.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:09.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:09.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:09.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:08 smithi187 ceph-mon[20072]: pgmap v1430: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.3 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 431 op/s; 6989/64401 objects degraded (10.852%); 4146/64401 objects misplaced (6.438%); 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:22:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:08 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:22:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:08 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:22:09.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:08 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:22:09.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:08 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:22:09.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:08 smithi187 ceph-mon[20072]: osdmap e617: 12 total, 12 up, 12 in 2024-09-02T10:22:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:09.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:09.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:09.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:09.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:09.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:09.442 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:22:09.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:09.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:09.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:09.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:09.658 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:09.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:09.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:09.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:09.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6989/64401 objects degraded (10.852%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/267201540' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:09 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:09.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:09.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:09.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:09.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:09.983 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:22:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:22:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:182528 6.a 6:5d15155c:::benchmark_data_smithi187_102448_object82256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-09-02T10:22:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:09.979203+0000", 2024-09-02T10:22:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0014852610000000001, 2024-09-02T10:22:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015131459999999999, 2024-09-02T10:22:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:09.983 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 182528 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979203+0000", 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979203+0000", 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979204+0000", 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.122e-06 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979209+0000", 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1579999999999998e-06 2024-09-02T10:22:09.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979219+0000", 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0633e-05 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979226+0000", 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5690000000000001e-06 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979277+0000", 2024-09-02T10:22:09.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1267999999999998e-05 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979389+0000", 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011213 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979504+0000", 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114857 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:09.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:182529 6.c e617/533 6:3528c762:::benchmark_data_smithi187_102448_object82257:head v 617'11284, mlcod=617'11282)", 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:09.979888+0000", 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00080050399999999995, 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00091764600000000004, 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979888+0000", 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:09.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979888+0000", 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979889+0000", 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4199999999999996e-07 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979894+0000", 2024-09-02T10:22:09.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4859999999999997e-06 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979922+0000", 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8306e-05 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979927+0000", 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7160000000000002e-06 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:09.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979968+0000", 2024-09-02T10:22:09.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1146999999999998e-05 2024-09-02T10:22:09.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:09.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.979983+0000", 2024-09-02T10:22:09.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5135e-05 2024-09-02T10:22:09.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:09.991 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:09.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:182531 6.a 6.b620074a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e617)", 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:09.980781+0000", 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4294967295.9999075, 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4859999999999998e-05, 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:09.992 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 182531 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.980781+0000", 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.980781+0000", 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.980782+0000", 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8100000000000002e-07 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.980787+0000", 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0420000000000002e-06 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.980798+0000", 2024-09-02T10:22:09.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0715e-05 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.980805+0000", 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3889999999999999e-06 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 3 2024-09-02T10:22:09.995 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:22:10.019 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:10.039 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:10.040 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:10.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6989/64401 objects degraded (10.852%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:10.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/267201540' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:10.059 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:09 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:10.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:10.143 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:22:10.173 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:10.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6989/64401 objects degraded (10.852%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/267201540' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:09 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:10.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:10.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:10.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:10.417 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:10.439 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:10.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:10.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:10.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:10.570 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:10.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:10.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:22:10.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:10.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:10.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:10.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:10.843 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:10.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:10.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:10 smithi162 ceph-mon[20279]: pgmap v1432: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s; 6985/59250 objects degraded (11.789%); 3985/59250 objects misplaced (6.726%); 2.2 MiB/s, 35 objects/s recovering 2024-09-02T10:22:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:10 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:22:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:10 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:10 smithi162 ceph-mon[20279]: osdmap e618: 12 total, 12 up, 12 in 2024-09-02T10:22:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3159235621' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:22:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:10 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:22:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:10 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:22:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:10 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:22:10.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:10.966 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:10.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:11.034 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:11.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:11.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:11.104 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:11.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:11.146 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-09-02T10:22:11.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:10 smithi151 ceph-mon[16639]: pgmap v1432: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s; 6985/59250 objects degraded (11.789%); 3985/59250 objects misplaced (6.726%); 2.2 MiB/s, 35 objects/s recovering 2024-09-02T10:22:11.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:10 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:22:11.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:10 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:11.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:10 smithi151 ceph-mon[16639]: osdmap e618: 12 total, 12 up, 12 in 2024-09-02T10:22:11.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3159235621' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:22:11.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:10 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:22:11.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:10 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:22:11.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:10 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:22:11.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:11.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:11.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:11.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:10 smithi187 ceph-mon[20072]: pgmap v1432: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 488 op/s; 6985/59250 objects degraded (11.789%); 3985/59250 objects misplaced (6.726%); 2.2 MiB/s, 35 objects/s recovering 2024-09-02T10:22:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:10 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:22:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:10 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:11.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:10 smithi187 ceph-mon[20072]: osdmap e618: 12 total, 12 up, 12 in 2024-09-02T10:22:11.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3159235621' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:22:11.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:10 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:22:11.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:10 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:22:11.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:10 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:22:11.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:11.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:11.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:11.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:11.489 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:11.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:11.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:11.591 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:11.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:11.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:11.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:11.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:11.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:11.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:11.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:11.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:11.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:11.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:12.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:12.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:12.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:12.059 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-02T10:22:12.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:22:12.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:12.146 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:12.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:12.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:12.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:12.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:12.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:12 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:22:12.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:12 smithi151 ceph-mon[16639]: osdmap e619: 12 total, 12 up, 12 in 2024-09-02T10:22:12.303 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:12 smithi151 ceph-mon[16639]: pgmap v1435: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s; 6985/57558 objects degraded (12.136%); 3593/57558 objects misplaced (6.242%); 7.5 MiB/s, 120 objects/s recovering 2024-09-02T10:22:12.367 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:22:12.368 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:22:12.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:12.402 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:12.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:12.403 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:12.411 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:22:12.412 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:22:12.445 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:12.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:22:12.458 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:22:12.501 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:22:12.501 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:22:12.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:12 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:22:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:12 smithi187 ceph-mon[20072]: osdmap e619: 12 total, 12 up, 12 in 2024-09-02T10:22:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:12 smithi187 ceph-mon[20072]: pgmap v1435: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s; 6985/57558 objects degraded (12.136%); 3593/57558 objects misplaced (6.242%); 7.5 MiB/s, 120 objects/s recovering 2024-09-02T10:22:12.541 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:22:12.541 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:22:12.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:12.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:12.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:12.574 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:22:12.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:22:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:12 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:22:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:12 smithi162 ceph-mon[20279]: osdmap e619: 12 total, 12 up, 12 in 2024-09-02T10:22:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:12 smithi162 ceph-mon[20279]: pgmap v1435: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.2 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 494 op/s; 6985/57558 objects degraded (12.136%); 3593/57558 objects misplaced (6.242%); 7.5 MiB/s, 120 objects/s recovering 2024-09-02T10:22:12.611 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:22:12.611 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:22:12.648 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:22:12.648 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:22:12.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:12.677 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:22:12.678 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:22:12.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:22:12.692 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:12.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:12.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:12.711 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:22:12.711 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:22:12.741 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:22:12.742 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:22:12.776 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:22:12.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:22:12.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:12.828 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:12.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:12.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:12.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:12.952 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:12.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:12.953 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:12.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:22:13.041 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:22:13.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:13.086 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:13.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:13.187 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:13.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:13.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:13.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:13.336 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:13.337 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3767259499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:13.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:13.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:13.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3767259499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:13.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:13.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3767259499' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:13.600 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:13.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:13.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:13.702 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:13.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:13.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:13.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:13.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:13.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:13.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:13.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:13.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:13.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:13.992 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:14.070 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:14.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:14.125 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:14.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:14.171 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:22:14.171 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:22:14.219 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:14.220 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:22:14.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:14.249 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:14.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:14 smithi162 ceph-mon[20279]: pgmap v1436: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s; 6982/52647 objects degraded (13.262%); 3399/52647 objects misplaced (6.456%); 7.8 MiB/s, 125 objects/s recovering 2024-09-02T10:22:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6985/57558 objects degraded (12.136%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:14.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:14.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:14.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:14.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:14 smithi151 ceph-mon[16639]: pgmap v1436: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s; 6982/52647 objects degraded (13.262%); 3399/52647 objects misplaced (6.456%); 7.8 MiB/s, 125 objects/s recovering 2024-09-02T10:22:14.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6985/57558 objects degraded (12.136%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:14.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:14 smithi187 ceph-mon[20072]: pgmap v1436: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1.1 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 652 op/s; 6982/52647 objects degraded (13.262%); 3399/52647 objects misplaced (6.456%); 7.8 MiB/s, 125 objects/s recovering 2024-09-02T10:22:14.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6985/57558 objects degraded (12.136%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:14.513 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:14.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:14.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:14.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:14.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:14.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:14.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:14.772 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:14.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:14.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:14.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:22:14.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:14.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:14.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:14.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:14.927 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1436,"stamp":"2024-09-02T10:22:13.387501+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1150485048,"num_objects":17549,"num_object_clones":0,"num_object_copies":52647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6982,"num_objects_misplaced":3399,"num_objects_unfound":0,"num_objects_dirty":17549,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182993,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16578,"num_bytes_recovered":1083586144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163745,"ondisk_log_size":163745,"up":141,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9435052,"kb_used_data":3979508,"kb_used_omap":18,"kb_used_meta":5455149,"kb_avail":1115506772,"statfs":{"total":1151940427776,"available":1142278934528,"internally_reserved":0,"allocated":4075016192,"data_stored":4071574808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5586072891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,3,14,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-256770048,"num_objects":-3918,"num_object_clones":0,"num_object_copies":-11754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-7,"num_objects_misplaced":-747,"num_objects_unfound":0,"num_objects_dirty":-3918,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3918,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":48955392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001441"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":600,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.155327+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:22:11.155327+0000","last_peered":"2024-09-02T10:22:11.155327+0000","last_clean":"2024-09-02T10:22:11.155327+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:22:11.155327+0000","last_undegraded":"2024-09-02T10:22:11.155327+0000","last_fullsized":"2024-09-02T10:22:11.155327+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":402,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:07.785311+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:22:07.785311+0000","last_peered":"2024-09-02T10:22:07.785311+0000","last_clean":"2024-09-02T10:22:07.785311+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:22:07.785311+0000","last_undegraded":"2024-09-02T10:22:07.785311+0000","last_fullsized":"2024-09-02T10:22:07.785311+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":853,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:07.784289+0000","last_change":"2024-09-02T10:21:03.860247+0000","last_active":"2024-09-02T10:22:07.784289+0000","last_peered":"2024-09-02T10:22:07.784289+0000","last_clean":"2024-09-02T10:22:07.784289+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:22:07.784289+0000","last_undegraded":"2024-09-02T10:22:07.784289+0000","last_fullsized":"2024-09-02T10:22:07.784289+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.860203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.860203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:32.006065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":853,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:07.784226+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:22:07.784226+0000","last_peered":"2024-09-02T10:22:07.784226+0000","last_clean":"2024-09-02T10:22:07.784226+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:22:07.784226+0000","last_undegraded":"2024-09-02T10:22:07.784226+0000","last_fullsized":"2024-09-02T10:22:07.784226+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.838669+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":157,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:07.801696+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:22:07.801696+0000","last_peered":"2024-09-02T10:22:07.801696+0000","last_clean":"2024-09-02T10:22:07.801696+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:22:07.801696+0000","last_undegraded":"2024-09-02T10:22:07.801696+0000","last_fullsized":"2024-09-02T10:22:07.801696+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":698,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.144295+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:22:11.144295+0000","last_peered":"2024-09-02T10:22:11.144295+0000","last_clean":"2024-09-02T10:22:11.144295+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:22:11.144295+0000","last_undegraded":"2024-09-02T10:22:11.144295+0000","last_fullsized":"2024-09-02T10:22:11.144295+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1233,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.153480+0000","last_change":"2024-09-02T10:20:52.113290+0000","last_active":"2024-09-02T10:22:11.153480+0000","last_peered":"2024-09-02T10:22:11.153480+0000","last_clean":"2024-09-02T10:22:11.153480+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:22:11.153480+0000","last_undegraded":"2024-09-02T10:22:11.153480+0000","last_fullsized":"2024-09-02T10:22:11.153480+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:29.189593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:29.189593+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":868,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:07.784282+0000","last_change":"2024-09-02T10:21:51.020121+0000","last_active":"2024-09-02T10:22:07.784282+0000","last_peered":"2024-09-02T10:22:07.784282+0000","last_clean":"2024-09-02T10:22:07.784282+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:22:07.784282+0000","last_undegraded":"2024-09-02T10:22:07.784282+0000","last_fullsized":"2024-09-02T10:22:07.784282+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:51.020075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:51.020075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.425292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":860,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:07.785772+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:22:07.785772+0000","last_peered":"2024-09-02T10:22:07.785772+0000","last_clean":"2024-09-02T10:22:07.785772+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:22:07.785772+0000","last_undegraded":"2024-09-02T10:22:07.785772+0000","last_fullsized":"2024-09-02T10:22:07.785772+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":848,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145351+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:22:11.145351+0000","last_peered":"2024-09-02T10:22:11.145351+0000","last_clean":"2024-09-02T10:22:11.145351+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:22:11.145351+0000","last_undegraded":"2024-09-02T10:22:11.145351+0000","last_fullsized":"2024-09-02T10:22:11.145351+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.685318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":92,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.164126+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:22:11.164126+0000","last_peered":"2024-09-02T10:22:11.164126+0000","last_clean":"2024-09-02T10:22:11.164126+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:22:11.164126+0000","last_undegraded":"2024-09-02T10:22:11.164126+0000","last_fullsized":"2024-09-02T10:22:11.164126+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1193,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:07.785175+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:22:07.785175+0000","last_peered":"2024-09-02T10:22:07.785175+0000","last_clean":"2024-09-02T10:22:07.785175+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:22:07.785175+0000","last_undegraded":"2024-09-02T10:22:07.785175+0000","last_fullsized":"2024-09-02T10:22:07.785175+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1132,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.153558+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:22:11.153558+0000","last_peered":"2024-09-02T10:22:11.153558+0000","last_clean":"2024-09-02T10:22:11.153558+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:22:11.153558+0000","last_undegraded":"2024-09-02T10:22:11.153558+0000","last_fullsized":"2024-09-02T10:22:11.153558+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1111,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.168221+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:22:11.168221+0000","last_peered":"2024-09-02T10:22:11.168221+0000","last_clean":"2024-09-02T10:22:11.168221+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:22:11.168221+0000","last_undegraded":"2024-09-02T10:22:11.168221+0000","last_fullsized":"2024-09-02T10:22:11.168221+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1102,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.153579+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:22:11.153579+0000","last_peered":"2024-09-02T10:22:11.153579+0000","last_clean":"2024-09-02T10:22:11.153579+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:22:11.153579+0000","last_undegraded":"2024-09-02T10:22:11.153579+0000","last_fullsized":"2024-09-02T10:22:11.153579+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"619'11421","reported_seq":17090,"reported_epoch":619,"state":"active+recovering+remapped","last_fresh":"2024-09-02T10:22:12.448145+0000","last_change":"2024-09-02T10:20:53.492211+0000","last_active":"2024-09-02T10:22:12.448145+0000","last_peered":"2024-09-02T10:22:12.448145+0000","last_clean":"2024-09-02T10:20:51.027001+0000","last_became_active":"2024-09-02T10:20:53.416507+0000","last_became_peered":"2024-09-02T10:20:53.416507+0000","last_unstale":"2024-09-02T10:22:12.448145+0000","last_undegraded":"2024-09-02T10:22:12.448145+0000","last_fullsized":"2024-09-02T10:22:12.448145+0000","mapping_epoch":586,"log_start":"544'1400","ondisk_log_start":"544'1400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63766528,"num_objects":973,"num_object_clones":0,"num_object_copies":2919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1230,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11421,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2712,"num_bytes_recovered":176619520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[5,10,9],"avail_no_missing":["5","9","10"],"object_location_counts":[{"shards":"5,9,10","objects":-257}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1196,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.153670+0000","last_change":"2024-09-02T10:20:52.111563+0000","last_active":"2024-09-02T10:22:11.153670+0000","last_peered":"2024-09-02T10:22:11.153670+0000","last_clean":"2024-09-02T10:22:11.153670+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:22:11.153670+0000","last_undegraded":"2024-09-02T10:22:11.153670+0000","last_fullsized":"2024-09-02T10:22:11.153670+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"617'11439","reported_seq":21003,"reported_epoch":617,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:09.499454+0000","last_change":"2024-09-02T10:20:53.422779+0000","last_active":"2024-09-02T10:22:09.499454+0000","last_peered":"2024-09-02T10:22:09.499454+0000","last_clean":"2024-09-02T10:20:51.098224+0000","last_became_active":"2024-09-02T10:20:53.370526+0000","last_became_peered":"2024-09-02T10:20:53.370526+0000","last_unstale":"2024-09-02T10:22:09.499454+0000","last_undegraded":"2024-09-02T10:20:53.137669+0000","last_fullsized":"2024-09-02T10:20:53.103100+0000","mapping_epoch":586,"log_start":"544'1400","ondisk_log_start":"544'1400","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74776576,"num_objects":1141,"num_object_clones":0,"num_object_copies":3423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":475,"num_objects_misplaced":1067,"num_objects_unfound":0,"num_objects_dirty":1141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11439,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4656,"num_bytes_recovered":304611328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-401},{"shards":"1,4,5","objects":1067}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1319,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145183+0000","last_change":"2024-09-02T10:21:11.035967+0000","last_active":"2024-09-02T10:22:11.145183+0000","last_peered":"2024-09-02T10:22:11.145183+0000","last_clean":"2024-09-02T10:22:11.145183+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:22:11.145183+0000","last_undegraded":"2024-09-02T10:22:11.145183+0000","last_fullsized":"2024-09-02T10:22:11.145183+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:11.035872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:11.035872+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:33.695004+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"619'11350","reported_seq":11594,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:12.352217+0000","last_change":"2024-09-02T10:20:27.715367+0000","last_active":"2024-09-02T10:22:12.352217+0000","last_peered":"2024-09-02T10:22:12.352217+0000","last_clean":"2024-09-02T10:22:12.352217+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:22:12.352217+0000","last_undegraded":"2024-09-02T10:22:12.352217+0000","last_fullsized":"2024-09-02T10:22:12.352217+0000","mapping_epoch":533,"log_start":"543'1300","ondisk_log_start":"543'1300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7181","last_scrub_stamp":"2024-09-02T10:20:27.715126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:27.715126+0000","objects_scrubbed":5211,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65601560,"num_objects":1002,"num_object_clones":0,"num_object_copies":3006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11350,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":961,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.161078+0000","last_change":"2024-09-02T10:20:18.812280+0000","last_active":"2024-09-02T10:22:11.161078+0000","last_peered":"2024-09-02T10:22:11.161078+0000","last_clean":"2024-09-02T10:22:11.161078+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:22:11.161078+0000","last_undegraded":"2024-09-02T10:22:11.161078+0000","last_fullsized":"2024-09-02T10:22:11.161078+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.812212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.812212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"617'11217","reported_seq":11472,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:08.351065+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:22:08.351065+0000","last_peered":"2024-09-02T10:22:08.351065+0000","last_clean":"2024-09-02T10:22:08.351065+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:22:08.351065+0000","last_undegraded":"2024-09-02T10:22:08.351065+0000","last_fullsized":"2024-09-02T10:22:08.351065+0000","mapping_epoch":533,"log_start":"543'1200","ondisk_log_start":"543'1200","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76349440,"num_objects":1165,"num_object_clones":0,"num_object_copies":3495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11217,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":358,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.155294+0000","last_change":"2024-09-02T10:21:04.496825+0000","last_active":"2024-09-02T10:22:11.155294+0000","last_peered":"2024-09-02T10:22:11.155294+0000","last_clean":"2024-09-02T10:22:11.155294+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:22:11.155294+0000","last_undegraded":"2024-09-02T10:22:11.155294+0000","last_fullsized":"2024-09-02T10:22:11.155294+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:04.496767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:04.496767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:12.959412+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"619'11540","reported_seq":11816,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.337900+0000","last_change":"2024-09-02T10:21:41.827351+0000","last_active":"2024-09-02T10:22:11.337900+0000","last_peered":"2024-09-02T10:22:11.337900+0000","last_clean":"2024-09-02T10:22:11.337900+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:22:11.337900+0000","last_undegraded":"2024-09-02T10:22:11.337900+0000","last_fullsized":"2024-09-02T10:22:11.337900+0000","mapping_epoch":533,"log_start":"544'1500","ondisk_log_start":"544'1500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10385","last_scrub_stamp":"2024-09-02T10:21:41.827172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:41.827172+0000","objects_scrubbed":2236,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:45.086417+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69861376,"num_objects":1066,"num_object_clones":0,"num_object_copies":3198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1267,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.155304+0000","last_change":"2024-09-02T10:21:07.446804+0000","last_active":"2024-09-02T10:22:11.155304+0000","last_peered":"2024-09-02T10:22:11.155304+0000","last_clean":"2024-09-02T10:22:11.155304+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:22:11.155304+0000","last_undegraded":"2024-09-02T10:22:11.155304+0000","last_fullsized":"2024-09-02T10:22:11.155304+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:07.446745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:07.446745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:31.909805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"617'11369","reported_seq":21398,"reported_epoch":617,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:09.493168+0000","last_change":"2024-09-02T10:20:53.407005+0000","last_active":"2024-09-02T10:22:09.493168+0000","last_peered":"2024-09-02T10:22:09.493168+0000","last_clean":"2024-09-02T10:20:51.097394+0000","last_became_active":"2024-09-02T10:20:53.379154+0000","last_became_peered":"2024-09-02T10:20:53.379154+0000","last_unstale":"2024-09-02T10:22:09.493168+0000","last_undegraded":"2024-09-02T10:20:53.138449+0000","last_fullsized":"2024-09-02T10:20:53.102891+0000","mapping_epoch":586,"log_start":"544'1300","ondisk_log_start":"544'1300","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77004800,"num_objects":1175,"num_object_clones":0,"num_object_copies":3525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":1102,"num_objects_unfound":0,"num_objects_dirty":1175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11369,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4887,"num_bytes_recovered":320143360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3],"avail_no_missing":["4","3","11"],"object_location_counts":[{"shards":"3,4","objects":-122},{"shards":"3,4,11","objects":1102}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":733,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.144170+0000","last_change":"2024-09-02T10:20:52.104138+0000","last_active":"2024-09-02T10:22:11.144170+0000","last_peered":"2024-09-02T10:22:11.144170+0000","last_clean":"2024-09-02T10:22:11.144170+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:22:11.144170+0000","last_undegraded":"2024-09-02T10:22:11.144170+0000","last_fullsized":"2024-09-02T10:22:11.144170+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"617'11249","reported_seq":11523,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:09.396475+0000","last_change":"2024-09-02T10:21:39.544819+0000","last_active":"2024-09-02T10:22:09.396475+0000","last_peered":"2024-09-02T10:22:09.396475+0000","last_clean":"2024-09-02T10:22:09.396475+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:22:09.396475+0000","last_undegraded":"2024-09-02T10:22:09.396475+0000","last_fullsized":"2024-09-02T10:22:09.396475+0000","mapping_epoch":533,"log_start":"543'1200","ondisk_log_start":"543'1200","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"603'10078","last_scrub_stamp":"2024-09-02T10:21:39.544630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.544630+0000","objects_scrubbed":2309,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.692572+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74121216,"num_objects":1131,"num_object_clones":0,"num_object_copies":3393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11249,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":698,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.144231+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:22:11.144231+0000","last_peered":"2024-09-02T10:22:11.144231+0000","last_clean":"2024-09-02T10:22:11.144231+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:22:11.144231+0000","last_undegraded":"2024-09-02T10:22:11.144231+0000","last_fullsized":"2024-09-02T10:22:11.144231+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"619'11585","reported_seq":11882,"reported_epoch":619,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:12.956973+0000","last_change":"2024-09-02T10:20:53.439960+0000","last_active":"2024-09-02T10:22:12.956973+0000","last_peered":"2024-09-02T10:22:12.956973+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:22:12.956973+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"544'1500","ondisk_log_start":"544'1500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71630848,"num_objects":1093,"num_object_clones":0,"num_object_copies":3279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11585,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-3014},{"shards":"0,3,5","objects":1093}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":466,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.148074+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:22:11.148074+0000","last_peered":"2024-09-02T10:22:11.148074+0000","last_clean":"2024-09-02T10:22:11.148074+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:22:11.148074+0000","last_undegraded":"2024-09-02T10:22:11.148074+0000","last_fullsized":"2024-09-02T10:22:11.148074+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:16.363558+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"617'11382","reported_seq":15981,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:08.347053+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:22:08.347053+0000","last_peered":"2024-09-02T10:22:08.347053+0000","last_clean":"2024-09-02T10:22:08.347053+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:22:08.347053+0000","last_undegraded":"2024-09-02T10:22:08.347053+0000","last_fullsized":"2024-09-02T10:22:08.347053+0000","mapping_epoch":559,"log_start":"544'1300","ondisk_log_start":"544'1300","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76021760,"num_objects":1160,"num_object_clones":0,"num_object_copies":3480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11382,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":699,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.144171+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:22:11.144171+0000","last_peered":"2024-09-02T10:22:11.144171+0000","last_clean":"2024-09-02T10:22:11.144171+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:22:11.144171+0000","last_undegraded":"2024-09-02T10:22:11.144171+0000","last_fullsized":"2024-09-02T10:22:11.144171+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"619'11522","reported_seq":11802,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.331667+0000","last_change":"2024-09-02T10:21:39.988693+0000","last_active":"2024-09-02T10:22:11.331667+0000","last_peered":"2024-09-02T10:22:11.331667+0000","last_clean":"2024-09-02T10:22:11.331667+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:22:11.331667+0000","last_undegraded":"2024-09-02T10:22:11.331667+0000","last_fullsized":"2024-09-02T10:22:11.331667+0000","mapping_epoch":533,"log_start":"545'1500","ondisk_log_start":"545'1500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10274","last_scrub_stamp":"2024-09-02T10:21:39.988364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.988364+0000","objects_scrubbed":2383,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:43.749337+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72744960,"num_objects":1110,"num_object_clones":0,"num_object_copies":3330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":961,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145354+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:22:11.145354+0000","last_peered":"2024-09-02T10:22:11.145354+0000","last_clean":"2024-09-02T10:22:11.145354+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:22:11.145354+0000","last_undegraded":"2024-09-02T10:22:11.145354+0000","last_fullsized":"2024-09-02T10:22:11.145354+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:54.988677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"617'11467","reported_seq":11706,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:08.738724+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:22:08.738724+0000","last_peered":"2024-09-02T10:22:08.738724+0000","last_clean":"2024-09-02T10:22:08.738724+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:22:08.738724+0000","last_undegraded":"2024-09-02T10:22:08.738724+0000","last_fullsized":"2024-09-02T10:22:08.738724+0000","mapping_epoch":533,"log_start":"544'1400","ondisk_log_start":"544'1400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73334784,"num_objects":1119,"num_object_clones":0,"num_object_copies":3357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11467,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1216,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:07.785275+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:22:07.785275+0000","last_peered":"2024-09-02T10:22:07.785275+0000","last_clean":"2024-09-02T10:22:07.785275+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:22:07.785275+0000","last_undegraded":"2024-09-02T10:22:07.785275+0000","last_fullsized":"2024-09-02T10:22:07.785275+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"619'11577","reported_seq":16068,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:12.337291+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:22:12.337291+0000","last_peered":"2024-09-02T10:22:12.337291+0000","last_clean":"2024-09-02T10:22:12.337291+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:22:12.337291+0000","last_undegraded":"2024-09-02T10:22:12.337291+0000","last_fullsized":"2024-09-02T10:22:12.337291+0000","mapping_epoch":565,"log_start":"545'1500","ondisk_log_start":"545'1500","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69140480,"num_objects":1055,"num_object_clones":0,"num_object_copies":3165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11577,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":972,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145164+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:22:11.145164+0000","last_peered":"2024-09-02T10:22:11.145164+0000","last_clean":"2024-09-02T10:22:11.145164+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:22:11.145164+0000","last_undegraded":"2024-09-02T10:22:11.145164+0000","last_fullsized":"2024-09-02T10:22:11.145164+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.846770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"619'11575","reported_seq":11858,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.363957+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:22:11.363957+0000","last_peered":"2024-09-02T10:22:11.363957+0000","last_clean":"2024-09-02T10:22:11.363957+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:22:11.363957+0000","last_undegraded":"2024-09-02T10:22:11.363957+0000","last_fullsized":"2024-09-02T10:22:11.363957+0000","mapping_epoch":533,"log_start":"545'1500","ondisk_log_start":"545'1500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.913566+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70451200,"num_objects":1075,"num_object_clones":0,"num_object_copies":3225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1227,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.148177+0000","last_change":"2024-09-02T10:21:35.696117+0000","last_active":"2024-09-02T10:22:11.148177+0000","last_peered":"2024-09-02T10:22:11.148177+0000","last_clean":"2024-09-02T10:22:11.148177+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:22:11.148177+0000","last_undegraded":"2024-09-02T10:22:11.148177+0000","last_fullsized":"2024-09-02T10:22:11.148177+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:35.696044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.696044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:46.753483+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"619'11470","reported_seq":11717,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:12.967681+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:22:12.967681+0000","last_peered":"2024-09-02T10:22:12.967681+0000","last_clean":"2024-09-02T10:22:12.967681+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:22:12.967681+0000","last_undegraded":"2024-09-02T10:22:12.967681+0000","last_fullsized":"2024-09-02T10:22:12.967681+0000","mapping_epoch":533,"log_start":"544'1400","ondisk_log_start":"544'1400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67239936,"num_objects":1026,"num_object_clones":0,"num_object_copies":3078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11470,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":172,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:07.801096+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:22:07.801096+0000","last_peered":"2024-09-02T10:22:07.801096+0000","last_clean":"2024-09-02T10:22:07.801096+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:22:07.801096+0000","last_undegraded":"2024-09-02T10:22:07.801096+0000","last_fullsized":"2024-09-02T10:22:07.801096+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"619'11480","reported_seq":11731,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:12.448146+0000","last_change":"2024-09-02T10:20:21.937271+0000","last_active":"2024-09-02T10:22:12.448146+0000","last_peered":"2024-09-02T10:22:12.448146+0000","last_clean":"2024-09-02T10:22:12.448146+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:22:12.448146+0000","last_undegraded":"2024-09-02T10:22:12.448146+0000","last_fullsized":"2024-09-02T10:22:12.448146+0000","mapping_epoch":533,"log_start":"544'1400","ondisk_log_start":"544'1400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6863","last_scrub_stamp":"2024-09-02T10:20:21.937140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.937140+0000","objects_scrubbed":5699,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11480,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":999,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145383+0000","last_change":"2024-09-02T10:21:08.213475+0000","last_active":"2024-09-02T10:22:11.145383+0000","last_peered":"2024-09-02T10:22:11.145383+0000","last_clean":"2024-09-02T10:22:11.145383+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:22:11.145383+0000","last_undegraded":"2024-09-02T10:22:11.145383+0000","last_fullsized":"2024-09-02T10:22:11.145383+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:08.213424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:08.213424+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:19.272314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"617'11350","reported_seq":11594,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:08.252272+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:22:08.252272+0000","last_peered":"2024-09-02T10:22:08.252272+0000","last_clean":"2024-09-02T10:22:08.252272+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:22:08.252272+0000","last_undegraded":"2024-09-02T10:22:08.252272+0000","last_fullsized":"2024-09-02T10:22:08.252272+0000","mapping_epoch":533,"log_start":"544'1300","ondisk_log_start":"544'1300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":1216,"num_object_clones":0,"num_object_copies":3648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11350,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":387,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-02T10:22:07.788573+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:22:07.788573+0000","last_peered":"2024-09-02T10:22:07.788573+0000","last_clean":"2024-09-02T10:22:07.788573+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:22:07.788573+0000","last_undegraded":"2024-09-02T10:22:07.788573+0000","last_fullsized":"2024-09-02T10:22:07.788573+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1149894680,"num_objects":17547,"num_object_clones":0,"num_object_copies":52641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6982,"num_objects_misplaced":3399,"num_objects_unfound":0,"num_objects_dirty":17547,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":182993,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16574,"num_bytes_recovered":1082654720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4028641280,"data_stored":4028629064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160893,"ondisk_log_size":160893,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383945,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":869124,"kb_used_data":396468,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92876028,"statfs":{"total":95995035648,"available":95105052672,"internally_reserved":0,"allocated":405983232,"data_stored":405695470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":483981764},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612878,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560656,"kb_used_data":200440,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93184496,"statfs":{"total":95995035648,"available":95420923904,"internally_reserved":0,"allocated":205250560,"data_stored":204958726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":368835008},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841811,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":786124,"kb_used_data":259316,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92959028,"statfs":{"total":95995035648,"available":95190044672,"internally_reserved":0,"allocated":265539584,"data_stored":265251822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":539425216},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070744,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491952,"kb_used_data":149788,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93253200,"statfs":{"total":95995035648,"available":95491276800,"internally_reserved":0,"allocated":153382912,"data_stored":153103808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299678,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870480,"kb_used_data":347836,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92874672,"statfs":{"total":95995035648,"available":95103664128,"internally_reserved":0,"allocated":356184064,"data_stored":355892750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":535165386},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528609,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639884,"kb_used_data":275508,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93105268,"statfs":{"total":95995035648,"available":95339794432,"internally_reserved":0,"allocated":282120192,"data_stored":281832430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1505296,"kb_used_data":918784,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92239856,"statfs":{"total":95995035648,"available":94453612544,"internally_reserved":0,"allocated":940834816,"data_stored":940539430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953771,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":631092,"kb_used_data":297244,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93114060,"statfs":{"total":95995035648,"available":95348797440,"internally_reserved":0,"allocated":304377856,"data_stored":304098752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150000,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1197100,"kb_used_data":616028,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92548052,"statfs":{"total":95995035648,"available":94769205248,"internally_reserved":0,"allocated":630812672,"data_stored":630533568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":594868680},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411636,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":295124,"kb_used_data":11260,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93450028,"statfs":{"total":95995035648,"available":95692828672,"internally_reserved":0,"allocated":11530240,"data_stored":11238926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575161,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":809012,"kb_used_data":304416,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92936140,"statfs":{"total":95995035648,"available":95166607360,"internally_reserved":0,"allocated":311721984,"data_stored":311438808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738688,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":779208,"kb_used_data":202420,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92965944,"statfs":{"total":95995035648,"available":95197126656,"internally_reserved":0,"allocated":207278080,"data_stored":206990318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":205717504,"data_stored":205717504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":301993984,"data_stored":301989912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11993088,"data_stored":11993088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":624623616,"data_stored":624623616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":301334528,"data_stored":301334528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":937365504,"data_stored":937361432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":273678336,"data_stored":273678336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":354287616,"data_stored":354287616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":149618688,"data_stored":149618688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":264437760,"data_stored":264437760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":201527296,"data_stored":201523224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":402063360,"data_stored":402063360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:22:14.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:22:15.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:15.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:15.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:15.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:15.180 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:15.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:15.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:15.239 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:22:15.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:15.318 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:15.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:15.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:15.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:15.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:15 smithi151 ceph-mon[16639]: from='client.17571 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:15.466 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:15.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:15.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:15.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:15.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:15 smithi187 ceph-mon[20072]: from='client.17571 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:15.512 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:15 smithi162 ceph-mon[20279]: from='client.17571 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:15.575 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:15.616 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:15.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:15.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:22:15.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:15.749 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:15.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:15.795 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:22:15.795 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:22:15.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:15.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:15.872 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:15.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:16.001 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:16.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:16.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:16.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:16.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:16.181 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:22:16.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:16.278 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:16.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:16.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:16.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:16.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:16.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:16.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:16 smithi151 ceph-mon[16639]: pgmap v1437: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1018 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 595 op/s; 6982/48831 objects degraded (14.298%); 2989/48831 objects misplaced (6.121%); 8.2 MiB/s, 131 objects/s recovering 2024-09-02T10:22:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:16 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:22:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:16 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:22:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4147994067' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:22:16.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:16 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:22:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:16 smithi187 ceph-mon[20072]: pgmap v1437: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1018 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 595 op/s; 6982/48831 objects degraded (14.298%); 2989/48831 objects misplaced (6.121%); 8.2 MiB/s, 131 objects/s recovering 2024-09-02T10:22:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:16 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:22:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:16 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:22:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4147994067' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:22:16.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:16 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:22:16.530 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:16.552 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:16.553 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:16 smithi162 ceph-mon[20279]: pgmap v1437: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 1018 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 595 op/s; 6982/48831 objects degraded (14.298%); 2989/48831 objects misplaced (6.121%); 8.2 MiB/s, 131 objects/s recovering 2024-09-02T10:22:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:16 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:22:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:16 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:22:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4147994067' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:22:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:16 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:22:16.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:16.655 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:16.656 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:22:16.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:16.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:16.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:16.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:16.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:16.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:16.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:16.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:16.939 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:16.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:17.041 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:17.063 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:17.064 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:17.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:17.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:17.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:17.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:17.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:17.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:17.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:17.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:17.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:17 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:17 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:22:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:17 smithi151 ceph-mon[16639]: osdmap e620: 12 total, 12 up, 12 in 2024-09-02T10:22:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:17 smithi151 ceph-mon[16639]: from='client.17577 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:17.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:17.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:17.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:17 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:17 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:22:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:17 smithi187 ceph-mon[20072]: osdmap e620: 12 total, 12 up, 12 in 2024-09-02T10:22:17.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:17 smithi187 ceph-mon[20072]: from='client.17577 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:17.537 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1438,"stamp":"2024-09-02T10:22:16.178171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1067123256,"num_objects":16277,"num_object_clones":0,"num_object_copies":48831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6982,"num_objects_misplaced":2989,"num_objects_unfound":0,"num_objects_dirty":16277,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184265,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16988,"num_bytes_recovered":1110455904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163717,"ondisk_log_size":163717,"up":141,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9221380,"kb_used_data":3765972,"kb_used_omap":18,"kb_used_meta":5455149,"kb_avail":1115720444,"statfs":{"total":1151940427776,"available":1142497734656,"internally_reserved":0,"allocated":3856355328,"data_stored":3852879312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5586072891},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,6,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-227606528,"num_objects":-3473,"num_object_clones":0,"num_object_copies":-10419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3,"num_objects_misplaced":-996,"num_objects_unfound":0,"num_objects_dirty":-3473,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3473,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":999,"num_bytes_recovered":65273856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.791631"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":600,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.155327+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:22:11.155327+0000","last_peered":"2024-09-02T10:22:11.155327+0000","last_clean":"2024-09-02T10:22:11.155327+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:22:11.155327+0000","last_undegraded":"2024-09-02T10:22:11.155327+0000","last_fullsized":"2024-09-02T10:22:11.155327+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":406,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145743+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:22:11.145743+0000","last_peered":"2024-09-02T10:22:11.145743+0000","last_clean":"2024-09-02T10:22:11.145743+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:22:11.145743+0000","last_undegraded":"2024-09-02T10:22:11.145743+0000","last_fullsized":"2024-09-02T10:22:11.145743+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":857,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.160660+0000","last_change":"2024-09-02T10:21:03.860247+0000","last_active":"2024-09-02T10:22:11.160660+0000","last_peered":"2024-09-02T10:22:11.160660+0000","last_clean":"2024-09-02T10:22:11.160660+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:22:11.160660+0000","last_undegraded":"2024-09-02T10:22:11.160660+0000","last_fullsized":"2024-09-02T10:22:11.160660+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.860203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.860203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:32.006065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":857,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.160622+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:22:11.160622+0000","last_peered":"2024-09-02T10:22:11.160622+0000","last_clean":"2024-09-02T10:22:11.160622+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:22:11.160622+0000","last_undegraded":"2024-09-02T10:22:11.160622+0000","last_fullsized":"2024-09-02T10:22:11.160622+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.838669+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":161,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.157638+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:22:11.157638+0000","last_peered":"2024-09-02T10:22:11.157638+0000","last_clean":"2024-09-02T10:22:11.157638+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:22:11.157638+0000","last_undegraded":"2024-09-02T10:22:11.157638+0000","last_fullsized":"2024-09-02T10:22:11.157638+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":698,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.144295+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:22:11.144295+0000","last_peered":"2024-09-02T10:22:11.144295+0000","last_clean":"2024-09-02T10:22:11.144295+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:22:11.144295+0000","last_undegraded":"2024-09-02T10:22:11.144295+0000","last_fullsized":"2024-09-02T10:22:11.144295+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1233,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.153480+0000","last_change":"2024-09-02T10:20:52.113290+0000","last_active":"2024-09-02T10:22:11.153480+0000","last_peered":"2024-09-02T10:22:11.153480+0000","last_clean":"2024-09-02T10:22:11.153480+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:22:11.153480+0000","last_undegraded":"2024-09-02T10:22:11.153480+0000","last_fullsized":"2024-09-02T10:22:11.153480+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:29.189593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:29.189593+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":872,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.160863+0000","last_change":"2024-09-02T10:21:51.020121+0000","last_active":"2024-09-02T10:22:11.160863+0000","last_peered":"2024-09-02T10:22:11.160863+0000","last_clean":"2024-09-02T10:22:11.160863+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:22:11.160863+0000","last_undegraded":"2024-09-02T10:22:11.160863+0000","last_fullsized":"2024-09-02T10:22:11.160863+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:51.020075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:51.020075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.425292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":864,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145753+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:22:11.145753+0000","last_peered":"2024-09-02T10:22:11.145753+0000","last_clean":"2024-09-02T10:22:11.145753+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:22:11.145753+0000","last_undegraded":"2024-09-02T10:22:11.145753+0000","last_fullsized":"2024-09-02T10:22:11.145753+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":848,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145351+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:22:11.145351+0000","last_peered":"2024-09-02T10:22:11.145351+0000","last_clean":"2024-09-02T10:22:11.145351+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:22:11.145351+0000","last_undegraded":"2024-09-02T10:22:11.145351+0000","last_fullsized":"2024-09-02T10:22:11.145351+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.685318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":92,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.164126+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:22:11.164126+0000","last_peered":"2024-09-02T10:22:11.164126+0000","last_clean":"2024-09-02T10:22:11.164126+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:22:11.164126+0000","last_undegraded":"2024-09-02T10:22:11.164126+0000","last_fullsized":"2024-09-02T10:22:11.164126+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1197,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.146025+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:22:11.146025+0000","last_peered":"2024-09-02T10:22:11.146025+0000","last_clean":"2024-09-02T10:22:11.146025+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:22:11.146025+0000","last_undegraded":"2024-09-02T10:22:11.146025+0000","last_fullsized":"2024-09-02T10:22:11.146025+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1132,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.153558+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:22:11.153558+0000","last_peered":"2024-09-02T10:22:11.153558+0000","last_clean":"2024-09-02T10:22:11.153558+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:22:11.153558+0000","last_undegraded":"2024-09-02T10:22:11.153558+0000","last_fullsized":"2024-09-02T10:22:11.153558+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1111,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.168221+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:22:11.168221+0000","last_peered":"2024-09-02T10:22:11.168221+0000","last_clean":"2024-09-02T10:22:11.168221+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:22:11.168221+0000","last_undegraded":"2024-09-02T10:22:11.168221+0000","last_fullsized":"2024-09-02T10:22:11.168221+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1102,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.153579+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:22:11.153579+0000","last_peered":"2024-09-02T10:22:11.153579+0000","last_clean":"2024-09-02T10:22:11.153579+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:22:11.153579+0000","last_undegraded":"2024-09-02T10:22:11.153579+0000","last_fullsized":"2024-09-02T10:22:11.153579+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"619'11421","reported_seq":17090,"reported_epoch":619,"state":"active+recovering+remapped","last_fresh":"2024-09-02T10:22:12.448145+0000","last_change":"2024-09-02T10:20:53.492211+0000","last_active":"2024-09-02T10:22:12.448145+0000","last_peered":"2024-09-02T10:22:12.448145+0000","last_clean":"2024-09-02T10:20:51.027001+0000","last_became_active":"2024-09-02T10:20:53.416507+0000","last_became_peered":"2024-09-02T10:20:53.416507+0000","last_unstale":"2024-09-02T10:22:12.448145+0000","last_undegraded":"2024-09-02T10:22:12.448145+0000","last_fullsized":"2024-09-02T10:22:12.448145+0000","mapping_epoch":586,"log_start":"544'1400","ondisk_log_start":"544'1400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63766528,"num_objects":973,"num_object_clones":0,"num_object_copies":2919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1230,"num_objects_unfound":0,"num_objects_dirty":973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11421,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2712,"num_bytes_recovered":176619520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[5,10,9],"avail_no_missing":["5","9","10"],"object_location_counts":[{"shards":"5,9,10","objects":-257}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1196,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.153670+0000","last_change":"2024-09-02T10:20:52.111563+0000","last_active":"2024-09-02T10:22:11.153670+0000","last_peered":"2024-09-02T10:22:11.153670+0000","last_clean":"2024-09-02T10:22:11.153670+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:22:11.153670+0000","last_undegraded":"2024-09-02T10:22:11.153670+0000","last_fullsized":"2024-09-02T10:22:11.153670+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"619'11640","reported_seq":21596,"reported_epoch":619,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:14.503660+0000","last_change":"2024-09-02T10:20:53.422779+0000","last_active":"2024-09-02T10:22:14.503660+0000","last_peered":"2024-09-02T10:22:14.503660+0000","last_clean":"2024-09-02T10:20:51.098224+0000","last_became_active":"2024-09-02T10:20:53.370526+0000","last_became_peered":"2024-09-02T10:20:53.370526+0000","last_unstale":"2024-09-02T10:22:14.503660+0000","last_undegraded":"2024-09-02T10:20:53.137669+0000","last_fullsized":"2024-09-02T10:20:53.103100+0000","mapping_epoch":586,"log_start":"545'1600","ondisk_log_start":"545'1600","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61603840,"num_objects":940,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":475,"num_objects_misplaced":874,"num_objects_unfound":0,"num_objects_dirty":940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11640,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4849,"num_bytes_recovered":317259776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-409},{"shards":"1,4,5","objects":874}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1319,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145183+0000","last_change":"2024-09-02T10:21:11.035967+0000","last_active":"2024-09-02T10:22:11.145183+0000","last_peered":"2024-09-02T10:22:11.145183+0000","last_clean":"2024-09-02T10:22:11.145183+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:22:11.145183+0000","last_undegraded":"2024-09-02T10:22:11.145183+0000","last_fullsized":"2024-09-02T10:22:11.145183+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:11.035872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:11.035872+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:33.695004+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"619'11350","reported_seq":11594,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:12.352217+0000","last_change":"2024-09-02T10:20:27.715367+0000","last_active":"2024-09-02T10:22:12.352217+0000","last_peered":"2024-09-02T10:22:12.352217+0000","last_clean":"2024-09-02T10:22:12.352217+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:22:12.352217+0000","last_undegraded":"2024-09-02T10:22:12.352217+0000","last_fullsized":"2024-09-02T10:22:12.352217+0000","mapping_epoch":533,"log_start":"543'1300","ondisk_log_start":"543'1300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7181","last_scrub_stamp":"2024-09-02T10:20:27.715126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:27.715126+0000","objects_scrubbed":5211,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65601560,"num_objects":1002,"num_object_clones":0,"num_object_copies":3006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11350,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":961,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.161078+0000","last_change":"2024-09-02T10:20:18.812280+0000","last_active":"2024-09-02T10:22:11.161078+0000","last_peered":"2024-09-02T10:22:11.161078+0000","last_clean":"2024-09-02T10:22:11.161078+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:22:11.161078+0000","last_undegraded":"2024-09-02T10:22:11.161078+0000","last_fullsized":"2024-09-02T10:22:11.161078+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.812212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.812212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"619'11393","reported_seq":11653,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:13.369454+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:22:13.369454+0000","last_peered":"2024-09-02T10:22:13.369454+0000","last_clean":"2024-09-02T10:22:13.369454+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:22:13.369454+0000","last_undegraded":"2024-09-02T10:22:13.369454+0000","last_fullsized":"2024-09-02T10:22:13.369454+0000","mapping_epoch":533,"log_start":"544'1300","ondisk_log_start":"544'1300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64815104,"num_objects":989,"num_object_clones":0,"num_object_copies":2967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11393,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":358,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.155294+0000","last_change":"2024-09-02T10:21:04.496825+0000","last_active":"2024-09-02T10:22:11.155294+0000","last_peered":"2024-09-02T10:22:11.155294+0000","last_clean":"2024-09-02T10:22:11.155294+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:22:11.155294+0000","last_undegraded":"2024-09-02T10:22:11.155294+0000","last_fullsized":"2024-09-02T10:22:11.155294+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:04.496767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:04.496767+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:12.959412+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"619'11540","reported_seq":11816,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.337900+0000","last_change":"2024-09-02T10:21:41.827351+0000","last_active":"2024-09-02T10:22:11.337900+0000","last_peered":"2024-09-02T10:22:11.337900+0000","last_clean":"2024-09-02T10:22:11.337900+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:22:11.337900+0000","last_undegraded":"2024-09-02T10:22:11.337900+0000","last_fullsized":"2024-09-02T10:22:11.337900+0000","mapping_epoch":533,"log_start":"544'1500","ondisk_log_start":"544'1500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10385","last_scrub_stamp":"2024-09-02T10:21:41.827172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:41.827172+0000","objects_scrubbed":2236,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:45.086417+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69861376,"num_objects":1066,"num_object_clones":0,"num_object_copies":3198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11540,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1267,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.155304+0000","last_change":"2024-09-02T10:21:07.446804+0000","last_active":"2024-09-02T10:22:11.155304+0000","last_peered":"2024-09-02T10:22:11.155304+0000","last_clean":"2024-09-02T10:22:11.155304+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:22:11.155304+0000","last_undegraded":"2024-09-02T10:22:11.155304+0000","last_fullsized":"2024-09-02T10:22:11.155304+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:07.446745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:07.446745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:31.909805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"619'11586","reported_seq":22053,"reported_epoch":619,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:14.504737+0000","last_change":"2024-09-02T10:20:53.407005+0000","last_active":"2024-09-02T10:22:14.504737+0000","last_peered":"2024-09-02T10:22:14.504737+0000","last_clean":"2024-09-02T10:20:51.097394+0000","last_became_active":"2024-09-02T10:20:53.379154+0000","last_became_peered":"2024-09-02T10:20:53.379154+0000","last_unstale":"2024-09-02T10:22:14.504737+0000","last_undegraded":"2024-09-02T10:20:53.138449+0000","last_fullsized":"2024-09-02T10:20:53.102891+0000","mapping_epoch":586,"log_start":"544'1500","ondisk_log_start":"544'1500","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62783488,"num_objects":958,"num_object_clones":0,"num_object_copies":2874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":885,"num_objects_unfound":0,"num_objects_dirty":958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11586,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5104,"num_bytes_recovered":334364672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3],"avail_no_missing":["4","3","11"],"object_location_counts":[{"shards":"3,4","objects":-122},{"shards":"3,4,11","objects":885}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":733,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.144170+0000","last_change":"2024-09-02T10:20:52.104138+0000","last_active":"2024-09-02T10:22:11.144170+0000","last_peered":"2024-09-02T10:22:11.144170+0000","last_clean":"2024-09-02T10:22:11.144170+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:22:11.144170+0000","last_undegraded":"2024-09-02T10:22:11.144170+0000","last_fullsized":"2024-09-02T10:22:11.144170+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"619'11413","reported_seq":11690,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:14.348622+0000","last_change":"2024-09-02T10:21:39.544819+0000","last_active":"2024-09-02T10:22:14.348622+0000","last_peered":"2024-09-02T10:22:14.348622+0000","last_clean":"2024-09-02T10:22:14.348622+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:22:14.348622+0000","last_undegraded":"2024-09-02T10:22:14.348622+0000","last_fullsized":"2024-09-02T10:22:14.348622+0000","mapping_epoch":533,"log_start":"544'1400","ondisk_log_start":"544'1400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"603'10078","last_scrub_stamp":"2024-09-02T10:21:39.544630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.544630+0000","objects_scrubbed":2309,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.692572+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63373312,"num_objects":967,"num_object_clones":0,"num_object_copies":2901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11413,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":698,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.144231+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:22:11.144231+0000","last_peered":"2024-09-02T10:22:11.144231+0000","last_clean":"2024-09-02T10:22:11.144231+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:22:11.144231+0000","last_undegraded":"2024-09-02T10:22:11.144231+0000","last_fullsized":"2024-09-02T10:22:11.144231+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"619'11585","reported_seq":11882,"reported_epoch":619,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:12.956973+0000","last_change":"2024-09-02T10:20:53.439960+0000","last_active":"2024-09-02T10:22:12.956973+0000","last_peered":"2024-09-02T10:22:12.956973+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:22:12.956973+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"544'1500","ondisk_log_start":"544'1500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71630848,"num_objects":1093,"num_object_clones":0,"num_object_copies":3279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11585,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-3014},{"shards":"0,3,5","objects":1093}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":466,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.148074+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:22:11.148074+0000","last_peered":"2024-09-02T10:22:11.148074+0000","last_clean":"2024-09-02T10:22:11.148074+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:22:11.148074+0000","last_undegraded":"2024-09-02T10:22:11.148074+0000","last_fullsized":"2024-09-02T10:22:11.148074+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:16.363558+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"619'11543","reported_seq":16147,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:13.280837+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:22:13.280837+0000","last_peered":"2024-09-02T10:22:13.280837+0000","last_clean":"2024-09-02T10:22:13.280837+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:22:13.280837+0000","last_undegraded":"2024-09-02T10:22:13.280837+0000","last_fullsized":"2024-09-02T10:22:13.280837+0000","mapping_epoch":559,"log_start":"544'1500","ondisk_log_start":"544'1500","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65470464,"num_objects":999,"num_object_clones":0,"num_object_copies":2997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11543,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":699,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.144171+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:22:11.144171+0000","last_peered":"2024-09-02T10:22:11.144171+0000","last_clean":"2024-09-02T10:22:11.144171+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:22:11.144171+0000","last_undegraded":"2024-09-02T10:22:11.144171+0000","last_fullsized":"2024-09-02T10:22:11.144171+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"619'11522","reported_seq":11802,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.331667+0000","last_change":"2024-09-02T10:21:39.988693+0000","last_active":"2024-09-02T10:22:11.331667+0000","last_peered":"2024-09-02T10:22:11.331667+0000","last_clean":"2024-09-02T10:22:11.331667+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:22:11.331667+0000","last_undegraded":"2024-09-02T10:22:11.331667+0000","last_fullsized":"2024-09-02T10:22:11.331667+0000","mapping_epoch":533,"log_start":"545'1500","ondisk_log_start":"545'1500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10274","last_scrub_stamp":"2024-09-02T10:21:39.988364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.988364+0000","objects_scrubbed":2383,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:43.749337+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72744960,"num_objects":1110,"num_object_clones":0,"num_object_copies":3330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11522,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":961,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145354+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:22:11.145354+0000","last_peered":"2024-09-02T10:22:11.145354+0000","last_clean":"2024-09-02T10:22:11.145354+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:22:11.145354+0000","last_undegraded":"2024-09-02T10:22:11.145354+0000","last_fullsized":"2024-09-02T10:22:11.145354+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:54.988677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"619'11631","reported_seq":11874,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:13.742150+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:22:13.742150+0000","last_peered":"2024-09-02T10:22:13.742150+0000","last_clean":"2024-09-02T10:22:13.742150+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:22:13.742150+0000","last_undegraded":"2024-09-02T10:22:13.742150+0000","last_fullsized":"2024-09-02T10:22:13.742150+0000","mapping_epoch":533,"log_start":"545'1600","ondisk_log_start":"545'1600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62586880,"num_objects":955,"num_object_clones":0,"num_object_copies":2865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11631,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1220,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.146125+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:22:11.146125+0000","last_peered":"2024-09-02T10:22:11.146125+0000","last_clean":"2024-09-02T10:22:11.146125+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:22:11.146125+0000","last_undegraded":"2024-09-02T10:22:11.146125+0000","last_fullsized":"2024-09-02T10:22:11.146125+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"619'11577","reported_seq":16068,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:12.337291+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:22:12.337291+0000","last_peered":"2024-09-02T10:22:12.337291+0000","last_clean":"2024-09-02T10:22:12.337291+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:22:12.337291+0000","last_undegraded":"2024-09-02T10:22:12.337291+0000","last_fullsized":"2024-09-02T10:22:12.337291+0000","mapping_epoch":565,"log_start":"545'1500","ondisk_log_start":"545'1500","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69140480,"num_objects":1055,"num_object_clones":0,"num_object_copies":3165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11577,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":972,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145164+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:22:11.145164+0000","last_peered":"2024-09-02T10:22:11.145164+0000","last_clean":"2024-09-02T10:22:11.145164+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:22:11.145164+0000","last_undegraded":"2024-09-02T10:22:11.145164+0000","last_fullsized":"2024-09-02T10:22:11.145164+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.846770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"619'11575","reported_seq":11858,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.363957+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:22:11.363957+0000","last_peered":"2024-09-02T10:22:11.363957+0000","last_clean":"2024-09-02T10:22:11.363957+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:22:11.363957+0000","last_undegraded":"2024-09-02T10:22:11.363957+0000","last_fullsized":"2024-09-02T10:22:11.363957+0000","mapping_epoch":533,"log_start":"545'1500","ondisk_log_start":"545'1500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.913566+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70451200,"num_objects":1075,"num_object_clones":0,"num_object_copies":3225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11575,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1227,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.148177+0000","last_change":"2024-09-02T10:21:35.696117+0000","last_active":"2024-09-02T10:22:11.148177+0000","last_peered":"2024-09-02T10:22:11.148177+0000","last_clean":"2024-09-02T10:22:11.148177+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:22:11.148177+0000","last_undegraded":"2024-09-02T10:22:11.148177+0000","last_fullsized":"2024-09-02T10:22:11.148177+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:35.696044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.696044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:46.753483+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"619'11470","reported_seq":11717,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:12.967681+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:22:12.967681+0000","last_peered":"2024-09-02T10:22:12.967681+0000","last_clean":"2024-09-02T10:22:12.967681+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:22:12.967681+0000","last_undegraded":"2024-09-02T10:22:12.967681+0000","last_fullsized":"2024-09-02T10:22:12.967681+0000","mapping_epoch":533,"log_start":"544'1400","ondisk_log_start":"544'1400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67239936,"num_objects":1026,"num_object_clones":0,"num_object_copies":3078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11470,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":176,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.163563+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:22:11.163563+0000","last_peered":"2024-09-02T10:22:11.163563+0000","last_clean":"2024-09-02T10:22:11.163563+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:22:11.163563+0000","last_undegraded":"2024-09-02T10:22:11.163563+0000","last_fullsized":"2024-09-02T10:22:11.163563+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"619'11480","reported_seq":11731,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:12.448146+0000","last_change":"2024-09-02T10:20:21.937271+0000","last_active":"2024-09-02T10:22:12.448146+0000","last_peered":"2024-09-02T10:22:12.448146+0000","last_clean":"2024-09-02T10:22:12.448146+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:22:12.448146+0000","last_undegraded":"2024-09-02T10:22:12.448146+0000","last_fullsized":"2024-09-02T10:22:12.448146+0000","mapping_epoch":533,"log_start":"544'1400","ondisk_log_start":"544'1400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6863","last_scrub_stamp":"2024-09-02T10:20:21.937140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.937140+0000","objects_scrubbed":5699,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11480,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":999,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.145383+0000","last_change":"2024-09-02T10:21:08.213475+0000","last_active":"2024-09-02T10:22:11.145383+0000","last_peered":"2024-09-02T10:22:11.145383+0000","last_clean":"2024-09-02T10:22:11.145383+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:22:11.145383+0000","last_undegraded":"2024-09-02T10:22:11.145383+0000","last_fullsized":"2024-09-02T10:22:11.145383+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:08.213424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:08.213424+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:19.272314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"619'11539","reported_seq":11787,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:13.391264+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:22:13.391264+0000","last_peered":"2024-09-02T10:22:13.391264+0000","last_clean":"2024-09-02T10:22:13.391264+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:22:13.391264+0000","last_undegraded":"2024-09-02T10:22:13.391264+0000","last_fullsized":"2024-09-02T10:22:13.391264+0000","mapping_epoch":533,"log_start":"544'1500","ondisk_log_start":"544'1500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67305472,"num_objects":1027,"num_object_clones":0,"num_object_copies":3081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11539,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":391,"reported_epoch":619,"state":"active+clean","last_fresh":"2024-09-02T10:22:11.154498+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:22:11.154498+0000","last_peered":"2024-09-02T10:22:11.154498+0000","last_clean":"2024-09-02T10:22:11.154498+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:22:11.154498+0000","last_undegraded":"2024-09-02T10:22:11.154498+0000","last_fullsized":"2024-09-02T10:22:11.154498+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":1066532888,"num_objects":16275,"num_object_clones":0,"num_object_copies":48825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6982,"num_objects_misplaced":2989,"num_objects_unfound":0,"num_objects_dirty":16275,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184265,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16984,"num_bytes_recovered":1109524480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3814010880,"data_stored":3813998664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160865,"ondisk_log_size":160865,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383945,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":869124,"kb_used_data":396468,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92876028,"statfs":{"total":95995035648,"available":95105052672,"internally_reserved":0,"allocated":405983232,"data_stored":405695470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":483981764},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612878,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":560656,"kb_used_data":200440,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93184496,"statfs":{"total":95995035648,"available":95420923904,"internally_reserved":0,"allocated":205250560,"data_stored":204958726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":368835008},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841811,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":786124,"kb_used_data":259316,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92959028,"statfs":{"total":95995035648,"available":95190044672,"internally_reserved":0,"allocated":265539584,"data_stored":265251822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":539425216},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070745,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473800,"kb_used_data":127476,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93271352,"statfs":{"total":95995035648,"available":95509864448,"internally_reserved":0,"allocated":130535424,"data_stored":130247662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299678,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":870480,"kb_used_data":347836,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":92874672,"statfs":{"total":95995035648,"available":95103664128,"internally_reserved":0,"allocated":356184064,"data_stored":355892750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":535165386},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528609,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639884,"kb_used_data":275508,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93105268,"statfs":{"total":95995035648,"available":95339794432,"internally_reserved":0,"allocated":282120192,"data_stored":281832430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724838,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1505296,"kb_used_data":918784,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92239856,"statfs":{"total":95995035648,"available":94453612544,"internally_reserved":0,"allocated":940834816,"data_stored":940539430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953772,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":588228,"kb_used_data":250228,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93156924,"statfs":{"total":95995035648,"available":95392690176,"internally_reserved":0,"allocated":256233472,"data_stored":255945710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150001,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1094344,"kb_used_data":525876,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92650808,"statfs":{"total":95995035648,"available":94874427392,"internally_reserved":0,"allocated":538497024,"data_stored":538209262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411636,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":295124,"kb_used_data":11260,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93450028,"statfs":{"total":95995035648,"available":95692828672,"internally_reserved":0,"allocated":11530240,"data_stored":11238926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575162,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":759112,"kb_used_data":250360,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92986040,"statfs":{"total":95995035648,"available":95217704960,"internally_reserved":0,"allocated":256368640,"data_stored":256076806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738688,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":779208,"kb_used_data":202420,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":92965944,"statfs":{"total":95995035648,"available":95197126656,"internally_reserved":0,"allocated":207278080,"data_stored":206990318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":205717504,"data_stored":205717504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":254545920,"data_stored":254541848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":11993088,"data_stored":11993088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":530513920,"data_stored":530513920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":250544128,"data_stored":250544128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":937365504,"data_stored":937361432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":273678336,"data_stored":273678336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":354287616,"data_stored":354287616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":127336448,"data_stored":127336448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":264437760,"data_stored":264437760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":201527296,"data_stored":201523224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":402063360,"data_stored":402063360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:22:17.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:22:17.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:17.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:17 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:17.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:17 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:22:17.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:17 smithi162 ceph-mon[20279]: osdmap e620: 12 total, 12 up, 12 in 2024-09-02T10:22:17.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:17 smithi162 ceph-mon[20279]: from='client.17577 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:17.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:17.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:17.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:22:17.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:17.769 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:17.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:17.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:17.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:17.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:17.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:17.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:18.018 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:18.019 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:18.051 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:22:18.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:18.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:18.163 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:18.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:18.264 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:18.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:18.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:18.289 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:18.302 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:18.318 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:18.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:18.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:18.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:18.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:18 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:18.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:18 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:18.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:18 smithi151 ceph-mon[16639]: osdmap e621: 12 total, 12 up, 12 in 2024-09-02T10:22:18.431 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:18 smithi151 ceph-mon[16639]: pgmap v1440: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 984 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 6982/47190 objects degraded (14.796%); 2989/47190 objects misplaced (6.334%); 6.0 MiB/s, 97 objects/s recovering 2024-09-02T10:22:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:18 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:18 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:18 smithi187 ceph-mon[20072]: osdmap e621: 12 total, 12 up, 12 in 2024-09-02T10:22:18.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:18 smithi187 ceph-mon[20072]: pgmap v1440: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 984 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 6982/47190 objects degraded (14.796%); 2989/47190 objects misplaced (6.334%); 6.0 MiB/s, 97 objects/s recovering 2024-09-02T10:22:18.518 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:18.542 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:18.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:18.543 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:18 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:18 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:18 smithi162 ceph-mon[20279]: osdmap e621: 12 total, 12 up, 12 in 2024-09-02T10:22:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:18 smithi162 ceph-mon[20279]: pgmap v1440: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 984 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 553 op/s; 6982/47190 objects degraded (14.796%); 2989/47190 objects misplaced (6.334%); 6.0 MiB/s, 97 objects/s recovering 2024-09-02T10:22:18.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:18.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:18.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:18.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:18.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:18.816 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:18.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:18.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:18.918 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:18.943 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:18.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:19.045 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:19.078 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:19.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:19.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:19.201 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:19.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:19.202 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:19.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:19.329 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:19.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:19.384 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6982/47190 objects degraded (14.796%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:19.385 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/797783033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:19.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:19.461 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:19.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:19.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6982/47190 objects degraded (14.796%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:19.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/797783033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:19.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:19.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:19.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:19.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6982/47190 objects degraded (14.796%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/797783033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:19.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:19.723 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:19.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:19.773 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:19.773 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6982/47190 objects degraded (14.796%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1611,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":621,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":12,"osd_in_since":1725272451,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":43},{"state_name":"active+recovering+undersized+degraded+remapped","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":15730,"data_bytes":1031275064,"bytes_used":9317834752,"bytes_avail":1142622593024,"bytes_total":1151940427776,"degraded_objects":6982,"degraded_total":47190,"degraded_ratio":0.14795507522780249,"misplaced_objects":2989,"misplaced_total":47190,"misplaced_ratio":0.063339690612417884,"recovering_objects_per_sec":97,"recovering_bytes_per_sec":6338412,"recovering_keys_per_sec":0,"num_objects_recovered":607,"num_bytes_recovered":39583744,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":553},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":37,"modified":"2024-09-02T10:21:41.376228+0000","services":{}},"progress_events":{}} 2024-09-02T10:22:19.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:19.860 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:19.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:19.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:19.970 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:22:19.970 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:22:19.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:19.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:20.047 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:22:20.048 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:22:20.082 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:22:20.082 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:22:20.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:20.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:22:20.121 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:22:20.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:20.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:20.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:20.151 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:22:20.151 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:22:20.189 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:22:20.189 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:22:20.222 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:22:20.222 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:22:20.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:22:20.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:20.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:20 smithi162 ceph-mon[20279]: pgmap v1441: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 864 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s; 6976/41451 objects degraded (16.830%); 2810/41451 objects misplaced (6.779%); 6.1 MiB/s, 98 objects/s recovering 2024-09-02T10:22:20.299 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/37529789' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:22:20.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:20.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:20.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:20.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:20 smithi151 ceph-mon[16639]: pgmap v1441: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 864 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s; 6976/41451 objects degraded (16.830%); 2810/41451 objects misplaced (6.779%); 6.1 MiB/s, 98 objects/s recovering 2024-09-02T10:22:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/37529789' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:22:20.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:20.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:20 smithi187 ceph-mon[20072]: pgmap v1441: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 864 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 670 op/s; 6976/41451 objects degraded (16.830%); 2810/41451 objects misplaced (6.779%); 6.1 MiB/s, 98 objects/s recovering 2024-09-02T10:22:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/37529789' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:22:20.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:20.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:20.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:20.643 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:22:20.643 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:22:20.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:22:20.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:20.701 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:20.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:20.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:20.713 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:20.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:20.855 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:20.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:20.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:20.958 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:20.979 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:20.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:20.980 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:21.012 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:22:21.062 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:22:21.063 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:22:21.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:21.099 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:22:21.100 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:22:21.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:21.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:21.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:21.130 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:22:21.130 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:22:21.159 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:22:21.159 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:22:21.190 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:22:21.191 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:22:21.215 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:21.224 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:22:21.224 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:22:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:21.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:21.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:21.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:21.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:21.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:21.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:21.526 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:21.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:21.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:21.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:21.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:21.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:21.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:21.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:21.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:21.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:21.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:21.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:21.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:22.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:22.070 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:22.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:22.155 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:22:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:22:22.155 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:22:22.156 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:22:22.156 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:22:22.171 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:22.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:22.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:22.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:22.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:22.318 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:22.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:22.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:22.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:22.441 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:22.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:22.442 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:22.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:22.563 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:22:22.577 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:22.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:22.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:22.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:22.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:22.729 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:22 smithi162 ceph-mon[20279]: pgmap v1442: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 827 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 6976/39699 objects degraded (17.572%); 2446/39699 objects misplaced (6.161%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:22:22.730 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1162106768' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:22.730 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:22 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:22 smithi187 ceph-mon[20072]: pgmap v1442: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 827 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 6976/39699 objects degraded (17.572%); 2446/39699 objects misplaced (6.161%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:22:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1162106768' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:22 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:22.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:22.828 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:22.829 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:22.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:22 smithi151 ceph-mon[16639]: pgmap v1442: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 827 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 6976/39699 objects degraded (17.572%); 2446/39699 objects misplaced (6.161%); 7.4 MiB/s, 119 objects/s recovering 2024-09-02T10:22:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1162106768' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:22 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:22.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:22:22.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:22.969 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:22.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:23.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:23.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:23.096 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:23.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:23.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:23.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:23.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:23.348 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:23.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:23.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:23.364 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:22:23.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:23.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:23.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:23.474 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:23.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:23.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:23.598 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:23 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:23 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:23 smithi151 ceph-mon[16639]: osdmap e622: 12 total, 12 up, 12 in 2024-09-02T10:22:23.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:23.728 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:23 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:23.729 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:23 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:23.729 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:23 smithi162 ceph-mon[20279]: osdmap e622: 12 total, 12 up, 12 in 2024-09-02T10:22:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:23.736 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:23.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:23 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:23 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:23 smithi187 ceph-mon[20072]: osdmap e622: 12 total, 12 up, 12 in 2024-09-02T10:22:23.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:23.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:23.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:23.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:23.989 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:23.990 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:24.091 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:24.112 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:24.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:24.113 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:24.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:24.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:24.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:24.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:24.340 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:24.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:24.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:24.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:24.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:24.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:24.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:24.625 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:24.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:24.652 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:24.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:24.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:24.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:24 smithi187 ceph-mon[20072]: pgmap v1444: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 720 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 6975/34524 objects degraded (20.203%); 2223/34524 objects misplaced (6.439%); 6.6 MiB/s, 107 objects/s recovering 2024-09-02T10:22:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6975/34524 objects degraded (20.203%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:24.785 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:24 smithi151 ceph-mon[16639]: pgmap v1444: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 720 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 6975/34524 objects degraded (20.203%); 2223/34524 objects misplaced (6.439%); 6.6 MiB/s, 107 objects/s recovering 2024-09-02T10:22:24.785 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6975/34524 objects degraded (20.203%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:24 smithi162 ceph-mon[20279]: pgmap v1444: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 720 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 661 op/s; 6975/34524 objects degraded (20.203%); 2223/34524 objects misplaced (6.439%); 6.6 MiB/s, 107 objects/s recovering 2024-09-02T10:22:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6975/34524 objects degraded (20.203%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:24.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:24.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:24.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:24.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:24.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:25.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:25.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:25.073 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:22:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:22:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:22:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:22:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:90191 4.15 e375/353 4:afce227e:::benchmark_data_smithi187_76542_object90190:head v 375'3680, mlcod=375'3677)", 2024-09-02T10:22:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:22:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 592.928766032, 2024-09-02T10:22:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20444974699999999, 2024-09-02T10:22:25.073 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098247+0000", 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9299999999999997e-07 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.098273+0000", 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5378999999999999e-05 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301201+0000", 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20292787800000001 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301213+0000", 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860000000000001e-05 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301360+0000", 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014643600000000001 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.301383+0000", 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2782999999999999e-05 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302662+0000", 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012796470000000001 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:12:31.302696+0000", 2024-09-02T10:22:25.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3970999999999997e-05 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126679 4.1 e392/341 4:86205e4c:::benchmark_data_smithi187_76542_object34343:head v 392'4951, mlcod=392'4948)", 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 548.55920576100004, 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44507702900000001, 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467807+0000", 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467808+0000", 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.277e-06 2024-09-02T10:22:25.079 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.467816+0000", 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3690000000000007e-06 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910880+0000", 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.44306368899999998 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910889+0000", 2024-09-02T10:22:25.080 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0380000000000005e-06 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910948+0000", 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8498000000000001e-05 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910970+0000", 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2580000000000001e-05 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.081 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912839+0000", 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018688120000000001 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.912884+0000", 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4765999999999998e-05 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25507.0:126802 4.1 e392/341 4:81a050df:::benchmark_data_smithi187_76542_object34466:head v 392'4952, mlcod=392'4949)", 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:22:25.082 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 548.11616238500005, 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25345677799999999, 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910850+0000", 2024-09-02T10:22:25.083 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910851+0000", 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0690000000000001e-06 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:15.910859+0000", 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.233e-06 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.084 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162387+0000", 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25152825099999998 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162402+0000", 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4905999999999999e-05 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162538+0000", 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013591800000000001 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.085 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.162581+0000", 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2991000000000002e-05 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164270+0000", 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016893419999999999 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:13:16.164307+0000", 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7067999999999998e-05 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.086 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:64011 5.9 e465/431 5:9035e690:::benchmark_data_smithi187_89405_object64010:head v 465'4071, mlcod=464'4068)", 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 406.80974370899997, 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.200897045, 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.087 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217269+0000", 2024-09-02T10:22:25.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1100000000000002e-07 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.217293+0000", 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3326999999999999e-05 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416283+0000", 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19899007799999999 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416287+0000", 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3529999999999997e-06 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416337+0000", 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0272000000000003e-05 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.416360+0000", 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3147e-05 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.418133+0000", 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001772605 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:37.418166+0000", 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2951999999999999e-05 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.091 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:86619 5.9 e473/431 5:9bb5eff1:::benchmark_data_smithi187_89405_object86618:head v 473'5442, mlcod=473'5439)", 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 385.487272077, 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21915551599999999, 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539741+0000", 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.09e-07 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.539774+0000", 2024-09-02T10:22:25.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2465999999999997e-05 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.755999+0000", 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.216224731 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756004+0000", 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9009999999999999e-06 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756086+0000", 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1767000000000005e-05 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.756100+0000", 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4285e-05 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.758849+0000", 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027488349999999998 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:15:58.758896+0000", 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6721999999999998e-05 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162692 5.2 5:4dd653e2:::benchmark_data_smithi187_89405_object59087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 299.67384790699998, 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25261044100000002, 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.096 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162692 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353166+0000", 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8200000000000008e-07 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353170+0000", 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3880000000000002e-06 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353180+0000", 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0336999999999999e-05 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353188+0000", 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7060000000000004e-06 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353233+0000", 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4453000000000003e-05 2024-09-02T10:22:25.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353306+0000", 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3559999999999994e-05 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353375+0000", 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9202999999999995e-05 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:25.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360656+0000", 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072804360000000004 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360676+0000", 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0075e-05 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605656+0000", 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24498062500000001 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605693+0000", 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6377000000000001e-05 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605775+0000", 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2299000000000006e-05 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162694 5.2 5:4549dcce:::benchmark_data_smithi187_89405_object59089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:22:25.102 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 299.67325172, 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25215807000000001, 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162694 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353762+0000", 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3600000000000003e-07 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353765+0000", 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6559999999999998e-06 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353776+0000", 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0243e-05 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353781+0000", 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0540000000000002e-06 2024-09-02T10:22:25.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.356988+0000", 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032069899999999998 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.357116+0000", 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012815 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:22:25.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.357235+0000", 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011919 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360716+0000", 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003481254 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.409925+0000", 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049208533999999998 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605820+0000", 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19589521600000001 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605878+0000", 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8149000000000002e-05 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605919+0000", 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0898e-05 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.108 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:43141 6.c e554/533 6:30fc4bbf:::benchmark_data_smithi187_102448_object43140:head v 554'2733, mlcod=554'2729)", 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:14.433920+0000", 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 189.593092506, 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223318874, 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.433920+0000", 2024-09-02T10:22:25.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.433920+0000", 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.433921+0000", 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7899999999999997e-07 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.433978+0000", 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7401999999999997e-05 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.655660+0000", 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.221681669 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.655668+0000", 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6410000000000002e-06 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.655723+0000", 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5186000000000003e-05 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.655737+0000", 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4226000000000001e-05 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.657205+0000", 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014677030000000001 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:14.657239+0000", 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4168000000000002e-05 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:121249 6.c e579/533 6:31dbf052:::benchmark_data_smithi187_102448_object20977:head v 579'7531, mlcod=579'7526)", 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:34.371591+0000", 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 109.655421906, 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.231073748, 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.113 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.371591+0000", 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.371591+0000", 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.371592+0000", 2024-09-02T10:22:25.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8000000000000004e-07 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.371598+0000", 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4790000000000001e-06 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.583533+0000", 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21193474500000001 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.583542+0000", 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8589999999999994e-06 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.583622+0000", 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9844000000000003e-05 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.583646+0000", 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4394999999999999e-05 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.602623+0000", 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018977099000000001 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.602664+0000", 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1446999999999998e-05 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:154513 6.c e595/533 6:323c7ae3:::benchmark_data_smithi187_102448_object54241:head v 595'9598, mlcod=595'9594)", 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:23.205471+0000", 2024-09-02T10:22:25.117 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.821541260000004, 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15966799500000001, 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.205471+0000", 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.205471+0000", 2024-09-02T10:22:25.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.205473+0000", 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.339e-06 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.205482+0000", 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8810000000000002e-06 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.361234+0000", 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15575192700000001 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.361241+0000", 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4399999999999999e-06 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.361288+0000", 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7293000000000003e-05 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.361303+0000", 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5167999999999999e-05 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.365105+0000", 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0038012129999999999 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.365139+0000", 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4734000000000003e-05 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.121 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:154619 6.5 e595/533 6:a2811ac1:::benchmark_data_smithi187_102448_object54347:head v 595'9686, mlcod=595'9684)", 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:23.384330+0000", 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.642682686999997, 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.082216311, 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.384330+0000", 2024-09-02T10:22:25.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.123 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.384330+0000", 2024-09-02T10:22:25.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.123 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.123 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.123 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.384331+0000", 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.42e-06 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.384341+0000", 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1889999999999996e-06 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.461541+0000", 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.077200956000000001 2024-09-02T10:22:25.124 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.461552+0000", 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0372e-05 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.461616+0000", 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3967999999999995e-05 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.461640+0000", 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4151999999999998e-05 2024-09-02T10:22:25.125 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.466501+0000", 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048608059999999996 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:23.466546+0000", 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5448000000000003e-05 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.126 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:157279 6.3 e598/533 6:c0eed291:::benchmark_data_smithi187_102448_object57007:head v 598'9930, mlcod=598'9927)", 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:27.860802+0000", 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.166210444999997, 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.094146578999999994, 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.860802+0000", 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.127 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.860802+0000", 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.860803+0000", 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.328e-06 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.860813+0000", 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7100000000000002e-06 2024-09-02T10:22:25.128 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.950005+0000", 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089192283999999997 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.950014+0000", 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0669999999999992e-06 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.950108+0000", 2024-09-02T10:22:25.129 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4905e-05 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.950129+0000", 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0361e-05 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.954889+0000", 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047605019999999998 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.130 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.954949+0000", 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9422000000000001e-05 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:162586 6.3 e602/533 6:c4adcd8e:::benchmark_data_smithi187_102448_object62314:head v 602'10235, mlcod=602'10232)", 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:37.514012+0000", 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 46.513000404000003, 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.103508585, 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.131 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:37.514012+0000", 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:37.514012+0000", 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:37.514013+0000", 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9800000000000002e-07 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:37.514020+0000", 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2819999999999999e-06 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:37.612243+0000", 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.098222689000000002 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:37.612251+0000", 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6580000000000002e-06 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:37.612320+0000", 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9160999999999997e-05 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:37.612339+0000", 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9222999999999999e-05 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:37.617469+0000", 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051295969999999996 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:37.617521+0000", 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1977000000000001e-05 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:163966 6.a e604/533 6:5fd30218:::benchmark_data_smithi187_102448_object63694:head v 604'10245, mlcod=604'10241)", 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:39.792316+0000", 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.234696970999998, 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15507043400000001, 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.792316+0000", 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.792316+0000", 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.792317+0000", 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4e-07 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.792324+0000", 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.131e-06 2024-09-02T10:22:25.137 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.942329+0000", 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15000506999999999 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.942336+0000", 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2470000000000002e-06 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.942421+0000", 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4987000000000005e-05 2024-09-02T10:22:25.138 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.942442+0000", 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0978000000000001e-05 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.947289+0000", 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048474110000000003 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:39.947386+0000", 2024-09-02T10:22:25.139 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6669999999999994e-05 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:176916 6.a e611/533 6:5b1bfdce:::benchmark_data_smithi187_102448_object76644:head v 611'11026, mlcod=611'11022)", 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:59.877084+0000", 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 24.149928286000002, 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089606509000000001, 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.140 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.877084+0000", 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.877084+0000", 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.877085+0000", 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5899999999999995e-07 2024-09-02T10:22:25.141 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.877093+0000", 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6699999999999994e-06 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.961429+0000", 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.084336374000000006 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.142 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.961439+0000", 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4010000000000003e-06 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.961505+0000", 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6835999999999995e-05 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.961528+0000", 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2880000000000001e-05 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.143 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.966593+0000", 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050647920000000003 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:59.966691+0000", 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7696999999999997e-05 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.144 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:182083 6.a e617/533 6:57383c6b:::benchmark_data_smithi187_102448_object81811:head v 617'11347, mlcod=617'11342)", 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:09.178771+0000", 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.848241825000001, 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.085982120999999995, 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.178771+0000", 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.145 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.178771+0000", 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.178772+0000", 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9299999999999996e-07 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.178781+0000", 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7100000000000002e-06 2024-09-02T10:22:25.146 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.258491+0000", 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.079709862000000006 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.258499+0000", 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6620000000000005e-06 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.259138+0000", 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00063940799999999995 2024-09-02T10:22:25.147 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.259156+0000", 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7308000000000001e-05 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.264564+0000", 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054080480000000004 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.148 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.264753+0000", 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018923 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:182253 6.a e617/533 6:5f3b34b6:::benchmark_data_smithi187_102448_object81981:head v 617'11361, mlcod=617'11358)", 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:09.488945+0000", 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.538068103000001, 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11687536799999999, 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.149 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.488945+0000", 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.488945+0000", 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.488946+0000", 2024-09-02T10:22:25.150 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6959999999999999e-06 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.488956+0000", 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8300000000000008e-06 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.599522+0000", 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11056561099999999 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.599529+0000", 2024-09-02T10:22:25.151 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0419999999999997e-06 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.599581+0000", 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2646999999999999e-05 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.599600+0000", 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8941e-05 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.605780+0000", 2024-09-02T10:22:25.152 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061801169999999997 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:09.605820+0000", 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9483999999999997e-05 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:190823 6.c e622/533 6:30003c98:::benchmark_data_smithi187_102448_object90551:head v 622'11800, mlcod=622'11795)", 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:23.653727+0000", 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.373285388, 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.089007368000000003, 2024-09-02T10:22:25.153 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.653727+0000", 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.653727+0000", 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.154 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.653728+0000", 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0500000000000002e-07 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.653735+0000", 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8689999999999998e-06 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.737377+0000", 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.083641848000000005 2024-09-02T10:22:25.155 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.737386+0000", 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2890000000000003e-06 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.737447+0000", 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1412000000000001e-05 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.737482+0000", 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4814999999999998e-05 2024-09-02T10:22:25.156 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.742648+0000", 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051656860000000001 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.742735+0000", 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6644000000000006e-05 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.157 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:190826 6.c e622/533 6:302c2484:::benchmark_data_smithi187_102448_object90554:head v 622'11801, mlcod=622'11795)", 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:23.658879+0000", 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.36813369400000001, 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16388957300000001, 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.658879+0000", 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.158 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.658879+0000", 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.658880+0000", 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6599999999999994e-07 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.658889+0000", 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6649999999999992e-06 2024-09-02T10:22:25.159 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.817689+0000", 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.158800199 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.817695+0000", 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5089999999999999e-06 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.817738+0000", 2024-09-02T10:22:25.160 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3126e-05 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.817753+0000", 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4963000000000001e-05 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.822707+0000", 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049539060000000001 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.161 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.822768+0000", 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1239000000000001e-05 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:190879 6.c e622/533 6:31db8940:::benchmark_data_smithi187_102448_object90607:head v 622'11802, mlcod=622'11797)", 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:23.737340+0000", 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.289672437, 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.108160593, 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:25.162 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.737340+0000", 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.737340+0000", 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.737341+0000", 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.093e-06 2024-09-02T10:22:25.163 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.737350+0000", 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8210000000000008e-06 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.830202+0000", 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.092851975000000003 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.830210+0000", 2024-09-02T10:22:25.164 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6399999999999997e-06 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.830312+0000", 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101826 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.830336+0000", 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4111e-05 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:22:25.165 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.845334+0000", 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014998770999999999 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:23.845501+0000", 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016635599999999999 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:25.166 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:22:25.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:25.206 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:25.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:25.207 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:25.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:25.334 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:25.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:25.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:25.467 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:25.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:25.554 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:25.555 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:22:25.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:25.602 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:25.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:25.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:25.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:25.730 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:25.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:25.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:25.842 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:22:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:25.862 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:25.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:25.963 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:25.984 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:25.985 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:26.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:26.118 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:26.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:26.218 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:26.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:26.244 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:26.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:26.375 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:26.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:26.459 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1445,"stamp":"2024-09-02T10:22:25.390888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":667419192,"num_objects":10178,"num_object_clones":0,"num_object_copies":30534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6975,"num_objects_misplaced":1786,"num_objects_unfound":0,"num_objects_dirty":10178,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190364,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18198,"num_bytes_recovered":1189295712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163616,"ondisk_log_size":163616,"up":141,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8176700,"kb_used_data":2675376,"kb_used_omap":18,"kb_used_meta":5500973,"kb_avail":1116765124,"statfs":{"total":1151940427776,"available":1143567486976,"internally_reserved":0,"allocated":2739585024,"data_stored":2735983951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5632996667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,2,6,4,4,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-363855872,"num_objects":-5552,"num_object_clones":0,"num_object_copies":-16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-7,"num_objects_misplaced":-1203,"num_objects_unfound":0,"num_objects_dirty":-5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5552,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1210,"num_bytes_recovered":78839808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.193564"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":604,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.207389+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:22:17.207389+0000","last_peered":"2024-09-02T10:22:17.207389+0000","last_clean":"2024-09-02T10:22:17.207389+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:22:17.207389+0000","last_undegraded":"2024-09-02T10:22:17.207389+0000","last_fullsized":"2024-09-02T10:22:17.207389+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":412,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.469544+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:22:22.469544+0000","last_peered":"2024-09-02T10:22:22.469544+0000","last_clean":"2024-09-02T10:22:22.469544+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:22:22.469544+0000","last_undegraded":"2024-09-02T10:22:22.469544+0000","last_fullsized":"2024-09-02T10:22:22.469544+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":863,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.463120+0000","last_change":"2024-09-02T10:21:03.860247+0000","last_active":"2024-09-02T10:22:22.463120+0000","last_peered":"2024-09-02T10:22:22.463120+0000","last_clean":"2024-09-02T10:22:22.463120+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:22:22.463120+0000","last_undegraded":"2024-09-02T10:22:22.463120+0000","last_fullsized":"2024-09-02T10:22:22.463120+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.860203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.860203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:32.006065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":863,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.463046+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:22:22.463046+0000","last_peered":"2024-09-02T10:22:22.463046+0000","last_clean":"2024-09-02T10:22:22.463046+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:22:22.463046+0000","last_undegraded":"2024-09-02T10:22:22.463046+0000","last_fullsized":"2024-09-02T10:22:22.463046+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.838669+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":167,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.473854+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:22:22.473854+0000","last_peered":"2024-09-02T10:22:22.473854+0000","last_clean":"2024-09-02T10:22:22.473854+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:22:22.473854+0000","last_undegraded":"2024-09-02T10:22:22.473854+0000","last_fullsized":"2024-09-02T10:22:22.473854+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":702,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.205045+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:22:17.205045+0000","last_peered":"2024-09-02T10:22:17.205045+0000","last_clean":"2024-09-02T10:22:17.205045+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:22:17.205045+0000","last_undegraded":"2024-09-02T10:22:17.205045+0000","last_fullsized":"2024-09-02T10:22:17.205045+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1239,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466314+0000","last_change":"2024-09-02T10:20:52.113290+0000","last_active":"2024-09-02T10:22:22.466314+0000","last_peered":"2024-09-02T10:22:22.466314+0000","last_clean":"2024-09-02T10:22:22.466314+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:22:22.466314+0000","last_undegraded":"2024-09-02T10:22:22.466314+0000","last_fullsized":"2024-09-02T10:22:22.466314+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:29.189593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:29.189593+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":878,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.463200+0000","last_change":"2024-09-02T10:21:51.020121+0000","last_active":"2024-09-02T10:22:22.463200+0000","last_peered":"2024-09-02T10:22:22.463200+0000","last_clean":"2024-09-02T10:22:22.463200+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:22:22.463200+0000","last_undegraded":"2024-09-02T10:22:22.463200+0000","last_fullsized":"2024-09-02T10:22:22.463200+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:51.020075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:51.020075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.425292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":870,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.469799+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:22:22.469799+0000","last_peered":"2024-09-02T10:22:22.469799+0000","last_clean":"2024-09-02T10:22:22.469799+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:22:22.469799+0000","last_undegraded":"2024-09-02T10:22:22.469799+0000","last_fullsized":"2024-09-02T10:22:22.469799+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":852,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.200836+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:22:17.200836+0000","last_peered":"2024-09-02T10:22:17.200836+0000","last_clean":"2024-09-02T10:22:17.200836+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:22:17.200836+0000","last_undegraded":"2024-09-02T10:22:17.200836+0000","last_fullsized":"2024-09-02T10:22:17.200836+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.685318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":96,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.210650+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:22:17.210650+0000","last_peered":"2024-09-02T10:22:17.210650+0000","last_clean":"2024-09-02T10:22:17.210650+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:22:17.210650+0000","last_undegraded":"2024-09-02T10:22:17.210650+0000","last_fullsized":"2024-09-02T10:22:17.210650+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1203,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.468957+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:22:22.468957+0000","last_peered":"2024-09-02T10:22:22.468957+0000","last_clean":"2024-09-02T10:22:22.468957+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:22:22.468957+0000","last_undegraded":"2024-09-02T10:22:22.468957+0000","last_fullsized":"2024-09-02T10:22:22.468957+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1138,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466509+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:22:22.466509+0000","last_peered":"2024-09-02T10:22:22.466509+0000","last_clean":"2024-09-02T10:22:22.466509+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:22:22.466509+0000","last_undegraded":"2024-09-02T10:22:22.466509+0000","last_fullsized":"2024-09-02T10:22:22.466509+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1117,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.462650+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:22:22.462650+0000","last_peered":"2024-09-02T10:22:22.462650+0000","last_clean":"2024-09-02T10:22:22.462650+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:22:22.462650+0000","last_undegraded":"2024-09-02T10:22:22.462650+0000","last_fullsized":"2024-09-02T10:22:22.462650+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1108,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466392+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:22:22.466392+0000","last_peered":"2024-09-02T10:22:22.466392+0000","last_clean":"2024-09-02T10:22:22.466392+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:22:22.466392+0000","last_undegraded":"2024-09-02T10:22:22.466392+0000","last_fullsized":"2024-09-02T10:22:22.466392+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"621'11759","reported_seq":18238,"reported_epoch":622,"state":"active+recovering+remapped","last_fresh":"2024-09-02T10:22:22.480577+0000","last_change":"2024-09-02T10:20:53.492211+0000","last_active":"2024-09-02T10:22:22.480577+0000","last_peered":"2024-09-02T10:22:22.480577+0000","last_clean":"2024-09-02T10:20:51.027001+0000","last_became_active":"2024-09-02T10:20:53.416507+0000","last_became_peered":"2024-09-02T10:20:53.416507+0000","last_unstale":"2024-09-02T10:22:22.480577+0000","last_undegraded":"2024-09-02T10:22:22.480577+0000","last_fullsized":"2024-09-02T10:22:22.480577+0000","mapping_epoch":586,"log_start":"546'1700","ondisk_log_start":"546'1700","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41615360,"num_objects":635,"num_object_clones":0,"num_object_copies":1905,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":828,"num_objects_unfound":0,"num_objects_dirty":635,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11759,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3114,"num_bytes_recovered":202964992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[5,10,9],"avail_no_missing":["5","9","10"],"object_location_counts":[{"shards":"5,9,10","objects":-193}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1202,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466415+0000","last_change":"2024-09-02T10:20:52.111563+0000","last_active":"2024-09-02T10:22:22.466415+0000","last_peered":"2024-09-02T10:22:22.466415+0000","last_clean":"2024-09-02T10:22:22.466415+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:22:22.466415+0000","last_undegraded":"2024-09-02T10:22:22.466415+0000","last_fullsized":"2024-09-02T10:22:22.466415+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"622'11978","reported_seq":22724,"reported_epoch":622,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:24.515244+0000","last_change":"2024-09-02T10:20:53.422779+0000","last_active":"2024-09-02T10:22:24.515244+0000","last_peered":"2024-09-02T10:22:24.515244+0000","last_clean":"2024-09-02T10:20:51.098224+0000","last_became_active":"2024-09-02T10:20:53.370526+0000","last_became_peered":"2024-09-02T10:20:53.370526+0000","last_unstale":"2024-09-02T10:22:24.515244+0000","last_undegraded":"2024-09-02T10:20:53.137669+0000","last_fullsized":"2024-09-02T10:20:53.103100+0000","mapping_epoch":586,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39452672,"num_objects":602,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":475,"num_objects_misplaced":482,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5241,"num_bytes_recovered":342949888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-355},{"shards":"1,4,5","objects":482}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1323,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.200475+0000","last_change":"2024-09-02T10:21:11.035967+0000","last_active":"2024-09-02T10:22:17.200475+0000","last_peered":"2024-09-02T10:22:17.200475+0000","last_clean":"2024-09-02T10:22:17.200475+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:22:17.200475+0000","last_undegraded":"2024-09-02T10:22:17.200475+0000","last_fullsized":"2024-09-02T10:22:17.200475+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:11.035872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:11.035872+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:33.695004+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"621'11719","reported_seq":11969,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466408+0000","last_change":"2024-09-02T10:20:27.715367+0000","last_active":"2024-09-02T10:22:22.466408+0000","last_peered":"2024-09-02T10:22:22.466408+0000","last_clean":"2024-09-02T10:22:22.466408+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:22:22.466408+0000","last_undegraded":"2024-09-02T10:22:22.466408+0000","last_fullsized":"2024-09-02T10:22:22.466408+0000","mapping_epoch":533,"log_start":"545'1700","ondisk_log_start":"545'1700","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7181","last_scrub_stamp":"2024-09-02T10:20:27.715126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:27.715126+0000","objects_scrubbed":5211,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41418776,"num_objects":633,"num_object_clones":0,"num_object_copies":1899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11719,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":968,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.465211+0000","last_change":"2024-09-02T10:20:18.812280+0000","last_active":"2024-09-02T10:22:22.465211+0000","last_peered":"2024-09-02T10:22:22.465211+0000","last_clean":"2024-09-02T10:22:22.465211+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:22:22.465211+0000","last_undegraded":"2024-09-02T10:22:22.465211+0000","last_fullsized":"2024-09-02T10:22:22.465211+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.812212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.812212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"622'11786","reported_seq":12050,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:23.352985+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:22:23.352985+0000","last_peered":"2024-09-02T10:22:23.352985+0000","last_clean":"2024-09-02T10:22:23.352985+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:22:23.352985+0000","last_undegraded":"2024-09-02T10:22:23.352985+0000","last_fullsized":"2024-09-02T10:22:23.352985+0000","mapping_epoch":533,"log_start":"546'1700","ondisk_log_start":"546'1700","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39059456,"num_objects":596,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11786,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":371,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.207532+0000","last_change":"2024-09-02T10:22:15.563077+0000","last_active":"2024-09-02T10:22:17.207532+0000","last_peered":"2024-09-02T10:22:17.207532+0000","last_clean":"2024-09-02T10:22:17.207532+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:22:17.207532+0000","last_undegraded":"2024-09-02T10:22:17.207532+0000","last_fullsized":"2024-09-02T10:22:17.207532+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:15.563022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:15.563022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:20.809417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"621'11918","reported_seq":12197,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:21.358371+0000","last_change":"2024-09-02T10:21:41.827351+0000","last_active":"2024-09-02T10:22:21.358371+0000","last_peered":"2024-09-02T10:22:21.358371+0000","last_clean":"2024-09-02T10:22:21.358371+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:22:21.358371+0000","last_undegraded":"2024-09-02T10:22:21.358371+0000","last_fullsized":"2024-09-02T10:22:21.358371+0000","mapping_epoch":533,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10385","last_scrub_stamp":"2024-09-02T10:21:41.827172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:41.827172+0000","objects_scrubbed":2236,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:45.086417+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45088768,"num_objects":688,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1271,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.207325+0000","last_change":"2024-09-02T10:21:07.446804+0000","last_active":"2024-09-02T10:22:17.207325+0000","last_peered":"2024-09-02T10:22:17.207325+0000","last_clean":"2024-09-02T10:22:17.207325+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:22:17.207325+0000","last_undegraded":"2024-09-02T10:22:17.207325+0000","last_fullsized":"2024-09-02T10:22:17.207325+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:07.446745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:07.446745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:31.909805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"622'11996","reported_seq":23287,"reported_epoch":622,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:24.527804+0000","last_change":"2024-09-02T10:20:53.407005+0000","last_active":"2024-09-02T10:22:24.527804+0000","last_peered":"2024-09-02T10:22:24.527804+0000","last_clean":"2024-09-02T10:20:51.097394+0000","last_became_active":"2024-09-02T10:20:53.379154+0000","last_became_peered":"2024-09-02T10:20:53.379154+0000","last_unstale":"2024-09-02T10:22:24.527804+0000","last_undegraded":"2024-09-02T10:20:53.138449+0000","last_fullsized":"2024-09-02T10:20:53.102891+0000","mapping_epoch":586,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":548,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":476,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11996,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5513,"num_bytes_recovered":361168896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3],"avail_no_missing":["4","3","11"],"object_location_counts":[{"shards":"3,4","objects":-123},{"shards":"3,4,11","objects":476}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":737,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.204518+0000","last_change":"2024-09-02T10:20:52.104138+0000","last_active":"2024-09-02T10:22:17.204518+0000","last_peered":"2024-09-02T10:22:17.204518+0000","last_clean":"2024-09-02T10:22:17.204518+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:22:17.204518+0000","last_undegraded":"2024-09-02T10:22:17.204518+0000","last_fullsized":"2024-09-02T10:22:17.204518+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"622'11795","reported_seq":12078,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:24.402972+0000","last_change":"2024-09-02T10:21:39.544819+0000","last_active":"2024-09-02T10:22:24.402972+0000","last_peered":"2024-09-02T10:22:24.402972+0000","last_clean":"2024-09-02T10:22:24.402972+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:22:24.402972+0000","last_undegraded":"2024-09-02T10:22:24.402972+0000","last_fullsized":"2024-09-02T10:22:24.402972+0000","mapping_epoch":533,"log_start":"546'1700","ondisk_log_start":"546'1700","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"603'10078","last_scrub_stamp":"2024-09-02T10:21:39.544630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.544630+0000","objects_scrubbed":2309,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.692572+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38338560,"num_objects":585,"num_object_clones":0,"num_object_copies":1755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11795,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":702,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.201557+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:22:17.201557+0000","last_peered":"2024-09-02T10:22:17.201557+0000","last_clean":"2024-09-02T10:22:17.201557+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:22:17.201557+0000","last_undegraded":"2024-09-02T10:22:17.201557+0000","last_fullsized":"2024-09-02T10:22:17.201557+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"622'12004","reported_seq":12321,"reported_epoch":622,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:22.907180+0000","last_change":"2024-09-02T10:20:53.439960+0000","last_active":"2024-09-02T10:22:22.907180+0000","last_peered":"2024-09-02T10:22:22.907180+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:22:22.907180+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"546'2000","ondisk_log_start":"546'2000","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44171264,"num_objects":674,"num_object_clones":0,"num_object_copies":2022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6305,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12004,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-3426},{"shards":"0,3,5","objects":674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":470,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.198244+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:22:17.198244+0000","last_peered":"2024-09-02T10:22:17.198244+0000","last_clean":"2024-09-02T10:22:17.198244+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:22:17.198244+0000","last_undegraded":"2024-09-02T10:22:17.198244+0000","last_fullsized":"2024-09-02T10:22:17.198244+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:16.363558+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"622'11932","reported_seq":16542,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:23.406804+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:22:23.406804+0000","last_peered":"2024-09-02T10:22:23.406804+0000","last_clean":"2024-09-02T10:22:23.406804+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:22:23.406804+0000","last_undegraded":"2024-09-02T10:22:23.406804+0000","last_fullsized":"2024-09-02T10:22:23.406804+0000","mapping_epoch":559,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39976960,"num_objects":610,"num_object_clones":0,"num_object_copies":1830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":610,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11932,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":703,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.205146+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:22:17.205146+0000","last_peered":"2024-09-02T10:22:17.205146+0000","last_clean":"2024-09-02T10:22:17.205146+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:22:17.205146+0000","last_undegraded":"2024-09-02T10:22:17.205146+0000","last_fullsized":"2024-09-02T10:22:17.205146+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"621'11904","reported_seq":12188,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:21.395028+0000","last_change":"2024-09-02T10:21:39.988693+0000","last_active":"2024-09-02T10:22:21.395028+0000","last_peered":"2024-09-02T10:22:21.395028+0000","last_clean":"2024-09-02T10:22:21.395028+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:22:21.395028+0000","last_undegraded":"2024-09-02T10:22:21.395028+0000","last_fullsized":"2024-09-02T10:22:21.395028+0000","mapping_epoch":533,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10274","last_scrub_stamp":"2024-09-02T10:21:39.988364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.988364+0000","objects_scrubbed":2383,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:43.749337+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":728,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11904,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":965,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.200618+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:22:17.200618+0000","last_peered":"2024-09-02T10:22:17.200618+0000","last_clean":"2024-09-02T10:22:17.200618+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:22:17.200618+0000","last_undegraded":"2024-09-02T10:22:17.200618+0000","last_fullsized":"2024-09-02T10:22:17.200618+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:54.988677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"622'12001","reported_seq":12249,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:23.711345+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:22:23.711345+0000","last_peered":"2024-09-02T10:22:23.711345+0000","last_clean":"2024-09-02T10:22:23.711345+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:22:23.711345+0000","last_undegraded":"2024-09-02T10:22:23.711345+0000","last_fullsized":"2024-09-02T10:22:23.711345+0000","mapping_epoch":533,"log_start":"546'2000","ondisk_log_start":"546'2000","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38338560,"num_objects":585,"num_object_clones":0,"num_object_copies":1755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1226,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.468985+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:22:22.468985+0000","last_peered":"2024-09-02T10:22:22.468985+0000","last_clean":"2024-09-02T10:22:22.468985+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:22:22.468985+0000","last_undegraded":"2024-09-02T10:22:22.468985+0000","last_fullsized":"2024-09-02T10:22:22.468985+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"621'11981","reported_seq":16478,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466652+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:22:22.466652+0000","last_peered":"2024-09-02T10:22:22.466652+0000","last_clean":"2024-09-02T10:22:22.466652+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:22:22.466652+0000","last_undegraded":"2024-09-02T10:22:22.466652+0000","last_fullsized":"2024-09-02T10:22:22.466652+0000","mapping_epoch":565,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42663936,"num_objects":651,"num_object_clones":0,"num_object_copies":1953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11981,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":976,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.206221+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:22:17.206221+0000","last_peered":"2024-09-02T10:22:17.206221+0000","last_clean":"2024-09-02T10:22:17.206221+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:22:17.206221+0000","last_undegraded":"2024-09-02T10:22:17.206221+0000","last_fullsized":"2024-09-02T10:22:17.206221+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.846770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"621'11969","reported_seq":12257,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:21.394118+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:22:21.394118+0000","last_peered":"2024-09-02T10:22:21.394118+0000","last_clean":"2024-09-02T10:22:21.394118+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:22:21.394118+0000","last_undegraded":"2024-09-02T10:22:21.394118+0000","last_fullsized":"2024-09-02T10:22:21.394118+0000","mapping_epoch":533,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.913566+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44630016,"num_objects":681,"num_object_clones":0,"num_object_copies":2043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11969,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1231,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.198356+0000","last_change":"2024-09-02T10:21:35.696117+0000","last_active":"2024-09-02T10:22:17.198356+0000","last_peered":"2024-09-02T10:22:17.198356+0000","last_clean":"2024-09-02T10:22:17.198356+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:22:17.198356+0000","last_undegraded":"2024-09-02T10:22:17.198356+0000","last_fullsized":"2024-09-02T10:22:17.198356+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:35.696044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.696044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:46.753483+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"622'11855","reported_seq":12109,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.966342+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:22:22.966342+0000","last_peered":"2024-09-02T10:22:22.966342+0000","last_clean":"2024-09-02T10:22:22.966342+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:22:22.966342+0000","last_undegraded":"2024-09-02T10:22:22.966342+0000","last_fullsized":"2024-09-02T10:22:22.966342+0000","mapping_epoch":533,"log_start":"546'1800","ondisk_log_start":"546'1800","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42008576,"num_objects":641,"num_object_clones":0,"num_object_copies":1923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11855,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":182,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.473936+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:22:22.473936+0000","last_peered":"2024-09-02T10:22:22.473936+0000","last_clean":"2024-09-02T10:22:22.473936+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:22:22.473936+0000","last_undegraded":"2024-09-02T10:22:22.473936+0000","last_fullsized":"2024-09-02T10:22:22.473936+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"621'11849","reported_seq":12106,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466397+0000","last_change":"2024-09-02T10:20:21.937271+0000","last_active":"2024-09-02T10:22:22.466397+0000","last_peered":"2024-09-02T10:22:22.466397+0000","last_clean":"2024-09-02T10:22:22.466397+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:22:22.466397+0000","last_undegraded":"2024-09-02T10:22:22.466397+0000","last_fullsized":"2024-09-02T10:22:22.466397+0000","mapping_epoch":533,"log_start":"546'1800","ondisk_log_start":"546'1800","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6863","last_scrub_stamp":"2024-09-02T10:20:21.937140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.937140+0000","objects_scrubbed":5699,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43974656,"num_objects":671,"num_object_clones":0,"num_object_copies":2013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11849,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1003,"reported_epoch":621,"state":"active+clean","last_fresh":"2024-09-02T10:22:17.200889+0000","last_change":"2024-09-02T10:21:08.213475+0000","last_active":"2024-09-02T10:22:17.200889+0000","last_peered":"2024-09-02T10:22:17.200889+0000","last_clean":"2024-09-02T10:22:17.200889+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:22:17.200889+0000","last_undegraded":"2024-09-02T10:22:17.200889+0000","last_fullsized":"2024-09-02T10:22:17.200889+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:08.213424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:08.213424+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:19.272314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"622'11918","reported_seq":12172,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:23.316132+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:22:23.316132+0000","last_peered":"2024-09-02T10:22:23.316132+0000","last_clean":"2024-09-02T10:22:23.316132+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:22:23.316132+0000","last_undegraded":"2024-09-02T10:22:23.316132+0000","last_fullsized":"2024-09-02T10:22:23.316132+0000","mapping_epoch":533,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42467328,"num_objects":648,"num_object_clones":0,"num_object_copies":1944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11918,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":397,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.467974+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:22:22.467974+0000","last_peered":"2024-09-02T10:22:22.467974+0000","last_clean":"2024-09-02T10:22:22.467974+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:22:22.467974+0000","last_undegraded":"2024-09-02T10:22:22.467974+0000","last_fullsized":"2024-09-02T10:22:22.467974+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":666828824,"num_objects":10176,"num_object_clones":0,"num_object_copies":30528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6975,"num_objects_misplaced":1786,"num_objects_unfound":0,"num_objects_dirty":10176,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":190364,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18194,"num_bytes_recovered":1188364288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2694459392,"data_stored":2694447176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160764,"ondisk_log_size":160764,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383947,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":824476,"kb_used_data":346252,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92920676,"statfs":{"total":95995035648,"available":95150772224,"internally_reserved":0,"allocated":354562048,"data_stored":354265670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":489683396},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612880,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":496160,"kb_used_data":131792,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93248992,"statfs":{"total":95995035648,"available":95486967808,"internally_reserved":0,"allocated":134955008,"data_stored":134654558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":373094848},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841813,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":709212,"kb_used_data":172684,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93035940,"statfs":{"total":95995035648,"available":95268802560,"internally_reserved":0,"allocated":176828416,"data_stored":176532038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":549386688},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070747,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":424488,"kb_used_data":78168,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93320664,"statfs":{"total":95995035648,"available":95560359936,"internally_reserved":0,"allocated":80044032,"data_stored":79743321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299680,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":761188,"kb_used_data":228820,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92983964,"statfs":{"total":95995035648,"available":95215579136,"internally_reserved":0,"allocated":234311680,"data_stored":234011750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":545126858},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528611,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":526044,"kb_used_data":169996,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93219108,"statfs":{"total":95995035648,"available":95456366592,"internally_reserved":0,"allocated":174075904,"data_stored":173779526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724840,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1370856,"kb_used_data":775896,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92374296,"statfs":{"total":95995035648,"available":94591279104,"internally_reserved":0,"allocated":794517504,"data_stored":794213502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,1,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953774,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":492588,"kb_used_data":150424,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93252564,"statfs":{"total":95995035648,"available":95490625536,"internally_reserved":0,"allocated":154034176,"data_stored":153733465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150003,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903724,"kb_used_data":322776,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92841428,"statfs":{"total":95995035648,"available":95069622272,"internally_reserved":0,"allocated":330522624,"data_stored":330221913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":594868680},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411638,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":305828,"kb_used_data":13652,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93439324,"statfs":{"total":95995035648,"available":95681867776,"internally_reserved":0,"allocated":13979648,"data_stored":13679718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575164,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":653872,"kb_used_data":157596,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":93091280,"statfs":{"total":95995035648,"available":95325470720,"internally_reserved":0,"allocated":161378304,"data_stored":161073521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738690,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":708264,"kb_used_data":127320,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93036888,"statfs":{"total":95995035648,"available":95269773312,"internally_reserved":0,"allocated":130375680,"data_stored":130074969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128647168,"data_stored":128647168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":158404608,"data_stored":158400536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":14417920,"data_stored":14417920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":323616768,"data_stored":323616768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":151257088,"data_stored":151257088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":791482368,"data_stored":791478296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":172883968,"data_stored":172883968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":230359040,"data_stored":230359040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":77463552,"data_stored":77463552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":165412864,"data_stored":165412864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":129568768,"data_stored":129564696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":350945280,"data_stored":350945280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:22:26.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:26.510 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:26.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:26.570 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:26.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:26.634 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:26.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:26.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:22:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:26 smithi151 ceph-mon[16639]: pgmap v1445: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 637 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 6975/30534 objects degraded (22.843%); 1786/30534 objects misplaced (5.849%); 9.2 MiB/s, 147 objects/s recovering 2024-09-02T10:22:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:26 smithi151 ceph-mon[16639]: from='client.27074 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:26.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:26 smithi187 ceph-mon[20072]: pgmap v1445: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 637 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 6975/30534 objects degraded (22.843%); 1786/30534 objects misplaced (5.849%); 9.2 MiB/s, 147 objects/s recovering 2024-09-02T10:22:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:26 smithi187 ceph-mon[20072]: from='client.27074 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:26.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:26.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:26 smithi162 ceph-mon[20279]: pgmap v1445: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 637 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 677 op/s; 6975/30534 objects degraded (22.843%); 1786/30534 objects misplaced (5.849%); 9.2 MiB/s, 147 objects/s recovering 2024-09-02T10:22:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:26 smithi162 ceph-mon[20279]: from='client.27074 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:26.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:26.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:26.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:27.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:27.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:27.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:27.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:27.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:27.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:27.182 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:27.280 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:27.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:27.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:27.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:27.307 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:27.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:27.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:27.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:27.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:27.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:27.569 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:27.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:27.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:27.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:27.700 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:27.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:27.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:27.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:27.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:27.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:27.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:27.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:27.958 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:28.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:28.102 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:28.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:28.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:28.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:28.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:28.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:28.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:28.367 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:28.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:28.469 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:28.482 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:22:28.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:28.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:28.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:28.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:28.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:28.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:28.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:28.753 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:28.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:28 smithi187 ceph-mon[20072]: pgmap v1446: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 603 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 6975/28932 objects degraded (24.108%); 1786/28932 objects misplaced (6.173%); 7.5 MiB/s, 120 objects/s recovering 2024-09-02T10:22:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2851160846' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:28 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/254545415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:28.817 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:22:28.818 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:22:28.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:28 smithi151 ceph-mon[16639]: pgmap v1446: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 603 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 6975/28932 objects degraded (24.108%); 1786/28932 objects misplaced (6.173%); 7.5 MiB/s, 120 objects/s recovering 2024-09-02T10:22:28.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2851160846' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:28.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:28 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:28.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/254545415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:28 smithi162 ceph-mon[20279]: pgmap v1446: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 603 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 608 op/s; 6975/28932 objects degraded (24.108%); 1786/28932 objects misplaced (6.173%); 7.5 MiB/s, 120 objects/s recovering 2024-09-02T10:22:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2851160846' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:28 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/254545415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:28.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:28.855 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:22:28.856 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:22:28.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:28.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:28.889 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:22:28.890 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:22:28.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:22:28.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:22:28.957 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:22:28.957 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:22:28.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:28.986 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:22:28.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:22:29.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:29.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:29.022 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:22:29.022 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:22:29.049 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:22:29.069 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:22:29.070 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:22:29.091 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:22:29.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:22:29.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:29.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:22:29.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:22:29.132 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:29.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:29.133 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:29.144 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:22:29.144 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:22:29.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:29.256 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:29.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:29.357 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:29.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:29.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:29.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:22:29.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:29.518 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:29.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:29.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:29.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:29.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:29.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:29.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:29.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:29 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:29 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:22:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:29 smithi187 ceph-mon[20072]: osdmap e623: 12 total, 12 up, 12 in 2024-09-02T10:22:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:29 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6975/28932 objects degraded (24.108%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:29.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:29.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:29.830 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:29 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:29.830 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:29 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:22:29.830 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:29 smithi162 ceph-mon[20279]: osdmap e623: 12 total, 12 up, 12 in 2024-09-02T10:22:29.830 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:29 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6975/28932 objects degraded (24.108%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:29.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:29.902 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:29.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:29 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:29 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:22:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:29 smithi151 ceph-mon[16639]: osdmap e623: 12 total, 12 up, 12 in 2024-09-02T10:22:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:29 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6975/28932 objects degraded (24.108%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:30.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:30.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:30.028 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:30.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:30.143 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:30.160 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:30.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:30.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:30.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:30.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:30.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:30.392 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:30.419 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:30.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:30.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:30.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:30.551 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:30.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:30.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:30 smithi151 ceph-mon[16639]: pgmap v1448: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 482 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 6973/23130 objects degraded (30.147%); 1570/23130 objects misplaced (6.788%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:22:30.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:30 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:22:30.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:30 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:22:30.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:30.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:22:30.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:30.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:30.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:30 smithi187 ceph-mon[20072]: pgmap v1448: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 482 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 6973/23130 objects degraded (30.147%); 1570/23130 objects misplaced (6.788%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:22:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:30 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:22:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:30 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:22:30.786 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:30.813 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:30.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:30 smithi162 ceph-mon[20279]: pgmap v1448: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 482 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 690 op/s; 6973/23130 objects degraded (30.147%); 1570/23130 objects misplaced (6.788%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:22:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:30 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:22:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:30 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:22:30.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:30.940 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:30.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:31.032 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:31.033 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:22:31.035 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:22:31.042 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:31.081 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:31.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:31.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:31.206 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:31.207 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:31.269 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:22:31.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:31.332 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:31.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:31.434 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:31.459 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:31.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:31.561 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:31.589 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:31.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:31.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:31.714 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:31.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:31.801 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1448,"stamp":"2024-09-02T10:22:29.391936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":505676344,"num_objects":7710,"num_object_clones":0,"num_object_copies":23130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6973,"num_objects_misplaced":1570,"num_objects_unfound":0,"num_objects_dirty":7710,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192832,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18416,"num_bytes_recovered":1203451488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163784,"ondisk_log_size":163784,"up":141,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":7769288,"kb_used_data":2213892,"kb_used_omap":18,"kb_used_meta":5555117,"kb_avail":1117172536,"statfs":{"total":1151940427776,"available":1143984676864,"internally_reserved":0,"allocated":2267025408,"data_stored":2263394004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5688440123},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,6,5,10,10,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-361955328,"num_objects":-5523,"num_object_clones":0,"num_object_copies":-16569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3,"num_objects_misplaced":-876,"num_objects_unfound":0,"num_objects_dirty":-5523,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5523,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":879,"num_bytes_recovered":57409536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002067"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":606,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.467378+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:22:22.467378+0000","last_peered":"2024-09-02T10:22:22.467378+0000","last_clean":"2024-09-02T10:22:22.467378+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:22:22.467378+0000","last_undegraded":"2024-09-02T10:22:22.467378+0000","last_fullsized":"2024-09-02T10:22:22.467378+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":414,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.479056+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:22:28.479056+0000","last_peered":"2024-09-02T10:22:28.479056+0000","last_clean":"2024-09-02T10:22:28.479056+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:22:28.479056+0000","last_undegraded":"2024-09-02T10:22:28.479056+0000","last_fullsized":"2024-09-02T10:22:28.479056+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":863,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.463120+0000","last_change":"2024-09-02T10:21:03.860247+0000","last_active":"2024-09-02T10:22:22.463120+0000","last_peered":"2024-09-02T10:22:22.463120+0000","last_clean":"2024-09-02T10:22:22.463120+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:22:22.463120+0000","last_undegraded":"2024-09-02T10:22:22.463120+0000","last_fullsized":"2024-09-02T10:22:22.463120+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.860203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.860203+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:32.006065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":863,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.463046+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:22:22.463046+0000","last_peered":"2024-09-02T10:22:22.463046+0000","last_clean":"2024-09-02T10:22:22.463046+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:22:22.463046+0000","last_undegraded":"2024-09-02T10:22:22.463046+0000","last_fullsized":"2024-09-02T10:22:22.463046+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.838669+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":167,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.473854+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:22:22.473854+0000","last_peered":"2024-09-02T10:22:22.473854+0000","last_clean":"2024-09-02T10:22:22.473854+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:22:22.473854+0000","last_undegraded":"2024-09-02T10:22:22.473854+0000","last_fullsized":"2024-09-02T10:22:22.473854+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":704,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.468632+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:22:22.468632+0000","last_peered":"2024-09-02T10:22:22.468632+0000","last_clean":"2024-09-02T10:22:22.468632+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:22:22.468632+0000","last_undegraded":"2024-09-02T10:22:22.468632+0000","last_fullsized":"2024-09-02T10:22:22.468632+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1239,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466314+0000","last_change":"2024-09-02T10:20:52.113290+0000","last_active":"2024-09-02T10:22:22.466314+0000","last_peered":"2024-09-02T10:22:22.466314+0000","last_clean":"2024-09-02T10:22:22.466314+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:22:22.466314+0000","last_undegraded":"2024-09-02T10:22:22.466314+0000","last_fullsized":"2024-09-02T10:22:22.466314+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:29.189593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:29.189593+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":878,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.463200+0000","last_change":"2024-09-02T10:21:51.020121+0000","last_active":"2024-09-02T10:22:22.463200+0000","last_peered":"2024-09-02T10:22:22.463200+0000","last_clean":"2024-09-02T10:22:22.463200+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:22:22.463200+0000","last_undegraded":"2024-09-02T10:22:22.463200+0000","last_fullsized":"2024-09-02T10:22:22.463200+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:51.020075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:51.020075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.425292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":872,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.479032+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:22:28.479032+0000","last_peered":"2024-09-02T10:22:28.479032+0000","last_clean":"2024-09-02T10:22:28.479032+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:22:28.479032+0000","last_undegraded":"2024-09-02T10:22:28.479032+0000","last_fullsized":"2024-09-02T10:22:28.479032+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":854,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466000+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:22:22.466000+0000","last_peered":"2024-09-02T10:22:22.466000+0000","last_clean":"2024-09-02T10:22:22.466000+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:22:22.466000+0000","last_undegraded":"2024-09-02T10:22:22.466000+0000","last_fullsized":"2024-09-02T10:22:22.466000+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.685318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":98,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.476103+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:22:22.476103+0000","last_peered":"2024-09-02T10:22:22.476103+0000","last_clean":"2024-09-02T10:22:22.476103+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:22:22.476103+0000","last_undegraded":"2024-09-02T10:22:22.476103+0000","last_fullsized":"2024-09-02T10:22:22.476103+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1205,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.478876+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:22:28.478876+0000","last_peered":"2024-09-02T10:22:28.478876+0000","last_clean":"2024-09-02T10:22:28.478876+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:22:28.478876+0000","last_undegraded":"2024-09-02T10:22:28.478876+0000","last_fullsized":"2024-09-02T10:22:28.478876+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1138,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466509+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:22:22.466509+0000","last_peered":"2024-09-02T10:22:22.466509+0000","last_clean":"2024-09-02T10:22:22.466509+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:22:22.466509+0000","last_undegraded":"2024-09-02T10:22:22.466509+0000","last_fullsized":"2024-09-02T10:22:22.466509+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1117,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.462650+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:22:22.462650+0000","last_peered":"2024-09-02T10:22:22.462650+0000","last_clean":"2024-09-02T10:22:22.462650+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:22:22.462650+0000","last_undegraded":"2024-09-02T10:22:22.462650+0000","last_fullsized":"2024-09-02T10:22:22.462650+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1108,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466392+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:22:22.466392+0000","last_peered":"2024-09-02T10:22:22.466392+0000","last_clean":"2024-09-02T10:22:22.466392+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:22:22.466392+0000","last_undegraded":"2024-09-02T10:22:22.466392+0000","last_fullsized":"2024-09-02T10:22:22.466392+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"622'11930","reported_seq":18840,"reported_epoch":622,"state":"active+recovering+remapped","last_fresh":"2024-09-02T10:22:27.484546+0000","last_change":"2024-09-02T10:20:53.492211+0000","last_active":"2024-09-02T10:22:27.484546+0000","last_peered":"2024-09-02T10:22:27.484546+0000","last_clean":"2024-09-02T10:20:51.027001+0000","last_became_active":"2024-09-02T10:20:53.416507+0000","last_became_peered":"2024-09-02T10:20:53.416507+0000","last_unstale":"2024-09-02T10:22:27.484546+0000","last_undegraded":"2024-09-02T10:22:27.484546+0000","last_fullsized":"2024-09-02T10:22:27.484546+0000","mapping_epoch":586,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30408704,"num_objects":464,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":612,"num_objects_unfound":0,"num_objects_dirty":464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11930,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3330,"num_bytes_recovered":217120768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[5,10,9],"avail_no_missing":["5","9","10"],"object_location_counts":[{"shards":"5,9,10","objects":-148}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1202,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466415+0000","last_change":"2024-09-02T10:20:52.111563+0000","last_active":"2024-09-02T10:22:22.466415+0000","last_peered":"2024-09-02T10:22:22.466415+0000","last_clean":"2024-09-02T10:22:22.466415+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:22:22.466415+0000","last_undegraded":"2024-09-02T10:22:22.466415+0000","last_fullsized":"2024-09-02T10:22:22.466415+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"622'11978","reported_seq":22724,"reported_epoch":622,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:24.515244+0000","last_change":"2024-09-02T10:20:53.422779+0000","last_active":"2024-09-02T10:22:24.515244+0000","last_peered":"2024-09-02T10:22:24.515244+0000","last_clean":"2024-09-02T10:20:51.098224+0000","last_became_active":"2024-09-02T10:20:53.370526+0000","last_became_peered":"2024-09-02T10:20:53.370526+0000","last_unstale":"2024-09-02T10:22:24.515244+0000","last_undegraded":"2024-09-02T10:20:53.137669+0000","last_fullsized":"2024-09-02T10:20:53.103100+0000","mapping_epoch":586,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39452672,"num_objects":602,"num_object_clones":0,"num_object_copies":1806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":475,"num_objects_misplaced":482,"num_objects_unfound":0,"num_objects_dirty":602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5241,"num_bytes_recovered":342949888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-355},{"shards":"1,4,5","objects":482}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1325,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.465837+0000","last_change":"2024-09-02T10:21:11.035967+0000","last_active":"2024-09-02T10:22:22.465837+0000","last_peered":"2024-09-02T10:22:22.465837+0000","last_clean":"2024-09-02T10:22:22.465837+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:22:22.465837+0000","last_undegraded":"2024-09-02T10:22:22.465837+0000","last_fullsized":"2024-09-02T10:22:22.465837+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:11.035872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:11.035872+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:33.695004+0000","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"622'11894","reported_seq":12144,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:27.393813+0000","last_change":"2024-09-02T10:20:27.715367+0000","last_active":"2024-09-02T10:22:27.393813+0000","last_peered":"2024-09-02T10:22:27.393813+0000","last_clean":"2024-09-02T10:22:27.393813+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:22:27.393813+0000","last_undegraded":"2024-09-02T10:22:27.393813+0000","last_fullsized":"2024-09-02T10:22:27.393813+0000","mapping_epoch":533,"log_start":"546'1800","ondisk_log_start":"546'1800","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7181","last_scrub_stamp":"2024-09-02T10:20:27.715126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:27.715126+0000","objects_scrubbed":5211,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29949976,"num_objects":458,"num_object_clones":0,"num_object_copies":1374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11894,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":968,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.465211+0000","last_change":"2024-09-02T10:20:18.812280+0000","last_active":"2024-09-02T10:22:22.465211+0000","last_peered":"2024-09-02T10:22:22.465211+0000","last_clean":"2024-09-02T10:22:22.465211+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:22:22.465211+0000","last_undegraded":"2024-09-02T10:22:22.465211+0000","last_fullsized":"2024-09-02T10:22:22.465211+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.812212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.812212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"622'11979","reported_seq":12245,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.279107+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:22:28.279107+0000","last_peered":"2024-09-02T10:22:28.279107+0000","last_clean":"2024-09-02T10:22:28.279107+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:22:28.279107+0000","last_undegraded":"2024-09-02T10:22:28.279107+0000","last_fullsized":"2024-09-02T10:22:28.279107+0000","mapping_epoch":533,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11979,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":373,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.467483+0000","last_change":"2024-09-02T10:22:15.563077+0000","last_active":"2024-09-02T10:22:22.467483+0000","last_peered":"2024-09-02T10:22:22.467483+0000","last_clean":"2024-09-02T10:22:22.467483+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:22:22.467483+0000","last_undegraded":"2024-09-02T10:22:22.467483+0000","last_fullsized":"2024-09-02T10:22:22.467483+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:15.563022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:15.563022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:20.809417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"622'12110","reported_seq":12392,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:26.180976+0000","last_change":"2024-09-02T10:21:41.827351+0000","last_active":"2024-09-02T10:22:26.180976+0000","last_peered":"2024-09-02T10:22:26.180976+0000","last_clean":"2024-09-02T10:22:26.180976+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:22:26.180976+0000","last_undegraded":"2024-09-02T10:22:26.180976+0000","last_fullsized":"2024-09-02T10:22:26.180976+0000","mapping_epoch":533,"log_start":"547'2100","ondisk_log_start":"547'2100","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10385","last_scrub_stamp":"2024-09-02T10:21:41.827172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:41.827172+0000","objects_scrubbed":2236,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:45.086417+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32505856,"num_objects":496,"num_object_clones":0,"num_object_copies":1488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12110,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1273,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.467244+0000","last_change":"2024-09-02T10:21:07.446804+0000","last_active":"2024-09-02T10:22:22.467244+0000","last_peered":"2024-09-02T10:22:22.467244+0000","last_clean":"2024-09-02T10:22:22.467244+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:22:22.467244+0000","last_undegraded":"2024-09-02T10:22:22.467244+0000","last_fullsized":"2024-09-02T10:22:22.467244+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:07.446745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:07.446745+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:31.909805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"622'11996","reported_seq":23287,"reported_epoch":622,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:24.527804+0000","last_change":"2024-09-02T10:20:53.407005+0000","last_active":"2024-09-02T10:22:24.527804+0000","last_peered":"2024-09-02T10:22:24.527804+0000","last_clean":"2024-09-02T10:20:51.097394+0000","last_became_active":"2024-09-02T10:20:53.379154+0000","last_became_peered":"2024-09-02T10:20:53.379154+0000","last_unstale":"2024-09-02T10:22:24.527804+0000","last_undegraded":"2024-09-02T10:20:53.138449+0000","last_fullsized":"2024-09-02T10:20:53.102891+0000","mapping_epoch":586,"log_start":"546'1900","ondisk_log_start":"546'1900","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":548,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":476,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11996,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5513,"num_bytes_recovered":361168896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3],"avail_no_missing":["4","3","11"],"object_location_counts":[{"shards":"3,4","objects":-123},{"shards":"3,4,11","objects":476}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":739,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466152+0000","last_change":"2024-09-02T10:20:52.104138+0000","last_active":"2024-09-02T10:22:22.466152+0000","last_peered":"2024-09-02T10:22:22.466152+0000","last_clean":"2024-09-02T10:22:22.466152+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:22:22.466152+0000","last_undegraded":"2024-09-02T10:22:22.466152+0000","last_fullsized":"2024-09-02T10:22:22.466152+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"622'11795","reported_seq":12078,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:24.402972+0000","last_change":"2024-09-02T10:21:39.544819+0000","last_active":"2024-09-02T10:22:24.402972+0000","last_peered":"2024-09-02T10:22:24.402972+0000","last_clean":"2024-09-02T10:22:24.402972+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:22:24.402972+0000","last_undegraded":"2024-09-02T10:22:24.402972+0000","last_fullsized":"2024-09-02T10:22:24.402972+0000","mapping_epoch":533,"log_start":"546'1700","ondisk_log_start":"546'1700","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"603'10078","last_scrub_stamp":"2024-09-02T10:21:39.544630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.544630+0000","objects_scrubbed":2309,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.692572+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38338560,"num_objects":585,"num_object_clones":0,"num_object_copies":1755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11795,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":704,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466136+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:22:22.466136+0000","last_peered":"2024-09-02T10:22:22.466136+0000","last_clean":"2024-09-02T10:22:22.466136+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:22:22.466136+0000","last_undegraded":"2024-09-02T10:22:22.466136+0000","last_fullsized":"2024-09-02T10:22:22.466136+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"622'12228","reported_seq":12550,"reported_epoch":622,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:27.964805+0000","last_change":"2024-09-02T10:20:53.439960+0000","last_active":"2024-09-02T10:22:27.964805+0000","last_peered":"2024-09-02T10:22:27.964805+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:22:27.964805+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"551'2200","ondisk_log_start":"551'2200","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29491200,"num_objects":450,"num_object_clones":0,"num_object_copies":1350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12228,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-3648},{"shards":"0,3,5","objects":450}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":472,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.460287+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:22:22.460287+0000","last_peered":"2024-09-02T10:22:22.460287+0000","last_clean":"2024-09-02T10:22:22.460287+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:22:22.460287+0000","last_undegraded":"2024-09-02T10:22:22.460287+0000","last_fullsized":"2024-09-02T10:22:22.460287+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:16.363558+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"622'12130","reported_seq":16739,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.338766+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:22:28.338766+0000","last_peered":"2024-09-02T10:22:28.338766+0000","last_clean":"2024-09-02T10:22:28.338766+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:22:28.338766+0000","last_undegraded":"2024-09-02T10:22:28.338766+0000","last_fullsized":"2024-09-02T10:22:28.338766+0000","mapping_epoch":559,"log_start":"546'2100","ondisk_log_start":"546'2100","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27000832,"num_objects":412,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12130,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":705,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.468715+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:22:22.468715+0000","last_peered":"2024-09-02T10:22:22.468715+0000","last_clean":"2024-09-02T10:22:22.468715+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:22:22.468715+0000","last_undegraded":"2024-09-02T10:22:22.468715+0000","last_fullsized":"2024-09-02T10:22:22.468715+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"622'12098","reported_seq":12384,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:26.386493+0000","last_change":"2024-09-02T10:21:39.988693+0000","last_active":"2024-09-02T10:22:26.386493+0000","last_peered":"2024-09-02T10:22:26.386493+0000","last_clean":"2024-09-02T10:22:26.386493+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:22:26.386493+0000","last_undegraded":"2024-09-02T10:22:26.386493+0000","last_fullsized":"2024-09-02T10:22:26.386493+0000","mapping_epoch":533,"log_start":"546'2000","ondisk_log_start":"546'2000","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10274","last_scrub_stamp":"2024-09-02T10:21:39.988364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.988364+0000","objects_scrubbed":2383,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:43.749337+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34996224,"num_objects":534,"num_object_clones":0,"num_object_copies":1602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":534,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12098,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":967,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.465876+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:22:22.465876+0000","last_peered":"2024-09-02T10:22:22.465876+0000","last_clean":"2024-09-02T10:22:22.465876+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:22:22.465876+0000","last_undegraded":"2024-09-02T10:22:22.465876+0000","last_fullsized":"2024-09-02T10:22:22.465876+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:54.988677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"623'12181","reported_seq":12432,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.762180+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:22:28.762180+0000","last_peered":"2024-09-02T10:22:28.762180+0000","last_clean":"2024-09-02T10:22:28.762180+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:22:28.762180+0000","last_undegraded":"2024-09-02T10:22:28.762180+0000","last_fullsized":"2024-09-02T10:22:28.762180+0000","mapping_epoch":533,"log_start":"547'2100","ondisk_log_start":"547'2100","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26542080,"num_objects":405,"num_object_clones":0,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12181,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1228,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.478945+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:22:28.478945+0000","last_peered":"2024-09-02T10:22:28.478945+0000","last_clean":"2024-09-02T10:22:28.478945+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:22:28.478945+0000","last_undegraded":"2024-09-02T10:22:28.478945+0000","last_fullsized":"2024-09-02T10:22:28.478945+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"622'12171","reported_seq":16669,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:27.436558+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:22:27.436558+0000","last_peered":"2024-09-02T10:22:27.436558+0000","last_clean":"2024-09-02T10:22:27.436558+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:22:27.436558+0000","last_undegraded":"2024-09-02T10:22:27.436558+0000","last_fullsized":"2024-09-02T10:22:27.436558+0000","mapping_epoch":565,"log_start":"547'2100","ondisk_log_start":"547'2100","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12171,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":978,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.465872+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:22:22.465872+0000","last_peered":"2024-09-02T10:22:22.465872+0000","last_clean":"2024-09-02T10:22:22.465872+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:22:22.465872+0000","last_undegraded":"2024-09-02T10:22:22.465872+0000","last_fullsized":"2024-09-02T10:22:22.465872+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.846770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"622'12117","reported_seq":12407,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:26.386835+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:22:26.386835+0000","last_peered":"2024-09-02T10:22:26.386835+0000","last_clean":"2024-09-02T10:22:26.386835+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:22:26.386835+0000","last_undegraded":"2024-09-02T10:22:26.386835+0000","last_fullsized":"2024-09-02T10:22:26.386835+0000","mapping_epoch":533,"log_start":"547'2100","ondisk_log_start":"547'2100","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.913566+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34930688,"num_objects":533,"num_object_clones":0,"num_object_copies":1599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12117,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1233,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.460390+0000","last_change":"2024-09-02T10:21:35.696117+0000","last_active":"2024-09-02T10:22:22.460390+0000","last_peered":"2024-09-02T10:22:22.460390+0000","last_clean":"2024-09-02T10:22:22.460390+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:22:22.460390+0000","last_undegraded":"2024-09-02T10:22:22.460390+0000","last_fullsized":"2024-09-02T10:22:22.460390+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:35.696044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.696044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:46.753483+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"622'12042","reported_seq":12295,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:27.927408+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:22:27.927408+0000","last_peered":"2024-09-02T10:22:27.927408+0000","last_clean":"2024-09-02T10:22:27.927408+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:22:27.927408+0000","last_undegraded":"2024-09-02T10:22:27.927408+0000","last_fullsized":"2024-09-02T10:22:27.927408+0000","mapping_epoch":533,"log_start":"546'2000","ondisk_log_start":"546'2000","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29753344,"num_objects":454,"num_object_clones":0,"num_object_copies":1362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12042,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":182,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.473936+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:22:22.473936+0000","last_peered":"2024-09-02T10:22:22.473936+0000","last_clean":"2024-09-02T10:22:22.473936+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:22:22.473936+0000","last_undegraded":"2024-09-02T10:22:22.473936+0000","last_fullsized":"2024-09-02T10:22:22.473936+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"622'12059","reported_seq":12316,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:27.469372+0000","last_change":"2024-09-02T10:20:21.937271+0000","last_active":"2024-09-02T10:22:27.469372+0000","last_peered":"2024-09-02T10:22:27.469372+0000","last_clean":"2024-09-02T10:22:27.469372+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:22:27.469372+0000","last_undegraded":"2024-09-02T10:22:27.469372+0000","last_fullsized":"2024-09-02T10:22:27.469372+0000","mapping_epoch":533,"log_start":"546'2000","ondisk_log_start":"546'2000","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6863","last_scrub_stamp":"2024-09-02T10:20:21.937140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.937140+0000","objects_scrubbed":5699,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12059,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1005,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.466096+0000","last_change":"2024-09-02T10:21:08.213475+0000","last_active":"2024-09-02T10:22:22.466096+0000","last_peered":"2024-09-02T10:22:22.466096+0000","last_clean":"2024-09-02T10:22:22.466096+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:22:22.466096+0000","last_undegraded":"2024-09-02T10:22:22.466096+0000","last_fullsized":"2024-09-02T10:22:22.466096+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:08.213424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:08.213424+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"622'12124","reported_seq":12378,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.326033+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:22:28.326033+0000","last_peered":"2024-09-02T10:22:28.326033+0000","last_clean":"2024-09-02T10:22:28.326033+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:22:28.326033+0000","last_undegraded":"2024-09-02T10:22:28.326033+0000","last_fullsized":"2024-09-02T10:22:28.326033+0000","mapping_epoch":533,"log_start":"547'2100","ondisk_log_start":"547'2100","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28966912,"num_objects":442,"num_object_clones":0,"num_object_copies":1326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12124,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":397,"reported_epoch":622,"state":"active+clean","last_fresh":"2024-09-02T10:22:22.467974+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:22:22.467974+0000","last_peered":"2024-09-02T10:22:22.467974+0000","last_clean":"2024-09-02T10:22:22.467974+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:22:22.467974+0000","last_undegraded":"2024-09-02T10:22:22.467974+0000","last_fullsized":"2024-09-02T10:22:22.467974+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":505085976,"num_objects":7708,"num_object_clones":0,"num_object_copies":23124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6973,"num_objects_misplaced":1570,"num_objects_unfound":0,"num_objects_dirty":7708,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192832,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18412,"num_bytes_recovered":1202520064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2224435200,"data_stored":2224422984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160932,"ondisk_log_size":160932,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383948,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802920,"kb_used_data":324696,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":92942232,"statfs":{"total":95995035648,"available":95172845568,"internally_reserved":0,"allocated":332488704,"data_stored":332187993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":489683396},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612881,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":463276,"kb_used_data":98908,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93281876,"statfs":{"total":95995035648,"available":95520641024,"internally_reserved":0,"allocated":101281792,"data_stored":100977009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":373094848},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841814,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":665960,"kb_used_data":125272,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93079192,"statfs":{"total":95995035648,"available":95313092608,"internally_reserved":0,"allocated":128278528,"data_stored":127977817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":553646528},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070747,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":424488,"kb_used_data":78168,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93320664,"statfs":{"total":95995035648,"available":95560359936,"internally_reserved":0,"allocated":80044032,"data_stored":79743321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299681,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":738416,"kb_used_data":165792,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":93006736,"statfs":{"total":95995035648,"available":95238897664,"internally_reserved":0,"allocated":169771008,"data_stored":169466745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":586349002},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,3,5,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528612,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":476968,"kb_used_data":120920,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93268184,"statfs":{"total":95995035648,"available":95506620416,"internally_reserved":0,"allocated":123822080,"data_stored":123521369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":364575175},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724841,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1319668,"kb_used_data":702564,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":92425484,"statfs":{"total":95995035648,"available":94643695616,"internally_reserved":0,"allocated":719425536,"data_stored":719117201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953774,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":492588,"kb_used_data":150424,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93252564,"statfs":{"total":95995035648,"available":95490625536,"internally_reserved":0,"allocated":154034176,"data_stored":153733465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150004,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":797864,"kb_used_data":225240,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92947288,"statfs":{"total":95995035648,"available":95178022912,"internally_reserved":0,"allocated":230645760,"data_stored":230345049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":586349000},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411639,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307952,"kb_used_data":24096,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93437200,"statfs":{"total":95995035648,"available":95679692800,"internally_reserved":0,"allocated":24674304,"data_stored":24370041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575165,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":611500,"kb_used_data":111068,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":93133652,"statfs":{"total":95995035648,"available":95368859648,"internally_reserved":0,"allocated":113733632,"data_stored":113428849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738691,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":667688,"kb_used_data":86744,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93077464,"statfs":{"total":95995035648,"available":95311323136,"internally_reserved":0,"allocated":88825856,"data_stored":88525145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":594868677},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":87556096,"data_stored":87556096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110956544,"data_stored":110952472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24707072,"data_stored":24707072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":223019008,"data_stored":223019008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":151257088,"data_stored":151257088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":717492224,"data_stored":717488152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":122224640,"data_stored":122224640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":167903232,"data_stored":167903232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":77463552,"data_stored":77463552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":117637120,"data_stored":117637120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":95948800,"data_stored":95944728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":328269824,"data_stored":328269824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:22:31.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:22:31.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:31.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:31.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:31.955 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:31.977 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:32.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:32.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:32.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:22:32.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:32.147 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:32.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:32.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:32.275 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:32.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:32.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:32.398 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:32.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:32.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:32.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:32.532 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:32.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:32.634 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:32.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:32.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:32.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:32 smithi187 ceph-mon[20072]: from='client.17613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:32 smithi187 ceph-mon[20072]: pgmap v1449: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 448 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 6973/21477 objects degraded (32.467%); 1082/21477 objects misplaced (5.038%); 9.5 MiB/s, 153 objects/s recovering 2024-09-02T10:22:32.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:32.767 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:32.772 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:32.786 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:32.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:32.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:32 smithi162 ceph-mon[20279]: from='client.17613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:32 smithi162 ceph-mon[20279]: pgmap v1449: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 448 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 6973/21477 objects degraded (32.467%); 1082/21477 objects misplaced (5.038%); 9.5 MiB/s, 153 objects/s recovering 2024-09-02T10:22:32.888 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:32.894 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:32 smithi151 ceph-mon[16639]: from='client.17613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:32.894 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:32 smithi151 ceph-mon[16639]: pgmap v1449: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 448 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 612 op/s; 6973/21477 objects degraded (32.467%); 1082/21477 objects misplaced (5.038%); 9.5 MiB/s, 153 objects/s recovering 2024-09-02T10:22:32.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:32.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:32.911 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:33.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:33.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:33.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:33.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:33.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:33.166 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:33.267 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:33.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:33.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:33.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:33.421 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:33.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:33.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:33.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:33.533 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:22:33.545 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:33.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:33.546 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:33.594 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:33 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:22:33.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:33 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:22:33.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/279959082' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:33.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:33 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:33.595 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:33 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:33.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:33.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:33.671 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:33.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:33.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:33 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:22:33.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:33 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:22:33.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/279959082' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:33.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:33 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:33.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:33 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:33.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:33.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:33.798 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:33.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:33.926 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:33.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:33 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:22:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:33 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:22:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/279959082' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:33 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:33 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:34.028 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:34.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:34.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:34.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:34.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:34.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:34.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:34.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:34.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:34.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:34.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:34.455 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:34.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:34.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:22:34.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:22:34.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:22:34.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:22:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:22:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:22:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:22:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:22:34.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:22:34.516 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:22:34.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:34.567 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:22:34.568 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:22:34.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:34 smithi151 ceph-mon[16639]: pgmap v1450: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 352 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 6969/16866 objects degraded (41.320%); 785/16866 objects misplaced (4.654%); 9.0 MiB/s, 144 objects/s recovering 2024-09-02T10:22:34.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:34 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:22:34.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/279959082' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:34.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:34 smithi151 ceph-mon[16639]: osdmap e624: 12 total, 12 up, 12 in 2024-09-02T10:22:34.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:34 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6969/16866 objects degraded (41.320%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:34.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:34 smithi151 ceph-mon[16639]: osdmap e625: 12 total, 12 up, 12 in 2024-09-02T10:22:34.568 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3843653396' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:34.595 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:34.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:34.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:34.620 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:22:34.620 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:22:34.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:34.713 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:22:34.714 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:22:34.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:34.725 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:34.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:34.741 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:22:34.741 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:22:34.773 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:22:34.773 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:22:34.812 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:22:34.813 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:22:34.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:34.844 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:22:34.845 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:22:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:34 smithi162 ceph-mon[20279]: pgmap v1450: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 352 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 6969/16866 objects degraded (41.320%); 785/16866 objects misplaced (4.654%); 9.0 MiB/s, 144 objects/s recovering 2024-09-02T10:22:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:34 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:22:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/279959082' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:34 smithi162 ceph-mon[20279]: osdmap e624: 12 total, 12 up, 12 in 2024-09-02T10:22:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:34 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6969/16866 objects degraded (41.320%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:34.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:34 smithi162 ceph-mon[20279]: osdmap e625: 12 total, 12 up, 12 in 2024-09-02T10:22:34.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3843653396' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:34.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:34.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:34.869 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:34.869 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6969/16866 objects degraded (41.320%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1626,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":625,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":12,"osd_in_since":1725272451,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":43},{"state_name":"active+recovering+undersized+degraded+remapped","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":5622,"data_bytes":368837176,"bytes_used":7565021184,"bytes_avail":1144375406592,"bytes_total":1151940427776,"degraded_objects":6969,"degraded_total":16866,"degraded_ratio":0.41319815012451078,"misplaced_objects":785,"misplaced_total":16866,"misplaced_ratio":0.046543341634056684,"recovering_objects_per_sec":144,"recovering_bytes_per_sec":9420718,"recovering_keys_per_sec":0,"num_objects_recovered":1444,"num_bytes_recovered":94240768,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":588},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":37,"modified":"2024-09-02T10:21:41.376228+0000","services":{}},"progress_events":{}} 2024-09-02T10:22:34.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:34.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:34.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:34.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:34 smithi187 ceph-mon[20072]: pgmap v1450: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 352 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 588 op/s; 6969/16866 objects degraded (41.320%); 785/16866 objects misplaced (4.654%); 9.0 MiB/s, 144 objects/s recovering 2024-09-02T10:22:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:34 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:22:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/279959082' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:34 smithi187 ceph-mon[20072]: osdmap e624: 12 total, 12 up, 12 in 2024-09-02T10:22:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:34 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6969/16866 objects degraded (41.320%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:34 smithi187 ceph-mon[20072]: osdmap e625: 12 total, 12 up, 12 in 2024-09-02T10:22:35.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3843653396' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:35.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:35.109 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:35.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:35.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:35.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:35.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:35.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:35.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:35.295 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:22:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:22:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:197080 6.5 6:aba340d5:::benchmark_data_smithi187_102448_object96808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e625)", 2024-09-02T10:22:35.295 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:35.015286+0000", 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.001766802, 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017928429999999999, 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 197080 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:35.296 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015286+0000", 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015286+0000", 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015287+0000", 2024-09-02T10:22:35.297 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5710000000000001e-06 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015292+0000", 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.549e-06 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015305+0000", 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2886e-05 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015312+0000", 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6369999999999999e-06 2024-09-02T10:22:35.298 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015355+0000", 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2837000000000003e-05 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015436+0000", 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0477999999999998e-05 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015511+0000", 2024-09-02T10:22:35.299 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5331000000000001e-05 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:197077 6.b e625/533 6:deb9e46c:::benchmark_data_smithi187_102448_object96805:head v 625'12343, mlcod=625'12340)", 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:35.015796+0000", 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0012568779999999999, 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001341346, 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.300 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015796+0000", 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015796+0000", 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015796+0000", 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5000000000000002e-07 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.301 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015802+0000", 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6779999999999997e-06 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015832+0000", 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0528999999999999e-05 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.015839+0000", 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8210000000000004e-06 2024-09-02T10:22:35.302 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.016353+0000", 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00051406399999999995 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.016397+0000", 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3664000000000001e-05 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:22:35.303 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 2 2024-09-02T10:22:35.304 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:22:35.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:35.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:35.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:35.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:35.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:35.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:35.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:35 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:22:35.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:35 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:22:35.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4263893095' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:22:35.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:35 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:22:35.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:35.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:35.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:35.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:35 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:22:35.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:35 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:22:35.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4263893095' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:22:35.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:35 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:22:35.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:35.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:35.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:35.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:35.857 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:35.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:35.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:35.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:35.981 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:36.003 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:36.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:36.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:35 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:22:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:35 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:22:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4263893095' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:22:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:35 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:22:36.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:36.133 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:36.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:36.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:36.257 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:36.258 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:36.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:36.388 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:36.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:36.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:36.523 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:36.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:36.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:36.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:36.635 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:22:36.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:22:36.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:36.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:36.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:36.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:36.770 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:36 smithi162 ceph-mon[20279]: pgmap v1453: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 284 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 485 op/s; 6969/13596 objects degraded (51.258%); 315/13596 objects misplaced (2.317%); 11 MiB/s, 197 objects/s recovering 2024-09-02T10:22:36.870 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:36.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:36.892 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:36 smithi151 ceph-mon[16639]: pgmap v1453: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 284 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 485 op/s; 6969/13596 objects degraded (51.258%); 315/13596 objects misplaced (2.317%); 11 MiB/s, 197 objects/s recovering 2024-09-02T10:22:36.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:36 smithi187 ceph-mon[20072]: pgmap v1453: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 284 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 485 op/s; 6969/13596 objects degraded (51.258%); 315/13596 objects misplaced (2.317%); 11 MiB/s, 197 objects/s recovering 2024-09-02T10:22:37.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:37.023 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:37.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:37.034 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:37.125 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:37.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:37.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:37.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:22:37.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:37.286 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:37.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:37.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:37.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:37.413 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:37.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:37.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:37.538 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:37.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:37.639 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:37.674 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:37.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:37.675 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:37.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:37.792 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:37 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:22:37.793 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:37 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:22:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:37.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:37.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:37 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:22:37.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:37 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:22:37.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:37.949 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:37.949 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:22:37.955 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:37.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:37.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:38.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:37 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:22:38.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:37 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:22:38.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:38.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:38.084 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:38.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:38.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:22:38.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:38.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:38.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:38.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:38.356 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:38.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:38.357 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:38.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:38.485 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:38.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:38.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:38.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:38.610 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:38.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:38.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:38.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:38.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:38.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:38.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:38 smithi162 ceph-mon[20279]: pgmap v1454: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 251 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 461 op/s; 6969/12045 objects degraded (57.858%); 315/12045 objects misplaced (2.615%); 9.8 MiB/s, 171 objects/s recovering 2024-09-02T10:22:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:22:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:22:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:38 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:22:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:38 smithi162 ceph-mon[20279]: from='client.27101 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:38.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:38.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:38 smithi151 ceph-mon[16639]: pgmap v1454: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 251 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 461 op/s; 6969/12045 objects degraded (57.858%); 315/12045 objects misplaced (2.615%); 9.8 MiB/s, 171 objects/s recovering 2024-09-02T10:22:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:22:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:22:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:38 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:22:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:38 smithi151 ceph-mon[16639]: from='client.27101 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:38.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:38.982 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:38.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:38 smithi187 ceph-mon[20072]: pgmap v1454: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 251 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 461 op/s; 6969/12045 objects degraded (57.858%); 315/12045 objects misplaced (2.615%); 9.8 MiB/s, 171 objects/s recovering 2024-09-02T10:22:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:22:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:22:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:38 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:22:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:38 smithi187 ceph-mon[20072]: from='client.27101 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:39.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:39.119 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:39.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:39.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:39.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:39.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:39.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:39.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:39.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:39.376 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:39.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:39.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:39.507 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:39.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:39.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:39.639 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:39.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6969/12045 objects degraded (57.858%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:39.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:39 smithi151 ceph-mon[16639]: osdmap e626: 12 total, 12 up, 12 in 2024-09-02T10:22:39.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:39.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:39.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:39.828 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1454,"stamp":"2024-09-02T10:22:37.395009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":263520824,"num_objects":4015,"num_object_clones":0,"num_object_copies":12045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6969,"num_objects_misplaced":315,"num_objects_unfound":0,"num_objects_dirty":4015,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196527,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19785,"num_bytes_recovered":1285699168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163579,"ondisk_log_size":163579,"up":141,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":7056832,"kb_used_data":1522288,"kb_used_omap":18,"kb_used_meta":5534317,"kb_avail":1117884992,"statfs":{"total":1151940427776,"available":1144714231808,"internally_reserved":0,"allocated":1558822912,"data_stored":1555083533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5667140923},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,2,0,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-242155520,"num_objects":-3695,"num_object_clones":0,"num_object_copies":-11085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":-1255,"num_objects_unfound":0,"num_objects_dirty":-3695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3695,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1369,"num_bytes_recovered":82247680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003073"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":612,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.178264+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:22:34.178264+0000","last_peered":"2024-09-02T10:22:34.178264+0000","last_clean":"2024-09-02T10:22:34.178264+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:22:34.178264+0000","last_undegraded":"2024-09-02T10:22:34.178264+0000","last_fullsized":"2024-09-02T10:22:34.178264+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":416,"reported_epoch":624,"state":"active+clean","last_fresh":"2024-09-02T10:22:33.547635+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:22:33.547635+0000","last_peered":"2024-09-02T10:22:33.547635+0000","last_clean":"2024-09-02T10:22:33.547635+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:22:33.547635+0000","last_undegraded":"2024-09-02T10:22:33.547635+0000","last_fullsized":"2024-09-02T10:22:33.547635+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":877,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.190176+0000","last_change":"2024-09-02T10:22:32.860224+0000","last_active":"2024-09-02T10:22:34.190176+0000","last_peered":"2024-09-02T10:22:34.190176+0000","last_clean":"2024-09-02T10:22:34.190176+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:22:34.190176+0000","last_undegraded":"2024-09-02T10:22:34.190176+0000","last_fullsized":"2024-09-02T10:22:34.190176+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:32.860180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:32.860180+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.373890+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":869,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.190169+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:22:34.190169+0000","last_peered":"2024-09-02T10:22:34.190169+0000","last_clean":"2024-09-02T10:22:34.190169+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:22:34.190169+0000","last_undegraded":"2024-09-02T10:22:34.190169+0000","last_fullsized":"2024-09-02T10:22:34.190169+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.838669+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":173,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.165006+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:22:34.165006+0000","last_peered":"2024-09-02T10:22:34.165006+0000","last_clean":"2024-09-02T10:22:34.165006+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:22:34.165006+0000","last_undegraded":"2024-09-02T10:22:34.165006+0000","last_fullsized":"2024-09-02T10:22:34.165006+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":710,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.152673+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:22:34.152673+0000","last_peered":"2024-09-02T10:22:34.152673+0000","last_clean":"2024-09-02T10:22:34.152673+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:22:34.152673+0000","last_undegraded":"2024-09-02T10:22:34.152673+0000","last_fullsized":"2024-09-02T10:22:34.152673+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1241,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.590478+0000","last_change":"2024-09-02T10:20:52.113290+0000","last_active":"2024-09-02T10:22:28.590478+0000","last_peered":"2024-09-02T10:22:28.590478+0000","last_clean":"2024-09-02T10:22:28.590478+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:22:28.590478+0000","last_undegraded":"2024-09-02T10:22:28.590478+0000","last_fullsized":"2024-09-02T10:22:28.590478+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:29.189593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:29.189593+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":884,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.190223+0000","last_change":"2024-09-02T10:21:51.020121+0000","last_active":"2024-09-02T10:22:34.190223+0000","last_peered":"2024-09-02T10:22:34.190223+0000","last_clean":"2024-09-02T10:22:34.190223+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:22:34.190223+0000","last_undegraded":"2024-09-02T10:22:34.190223+0000","last_fullsized":"2024-09-02T10:22:34.190223+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:51.020075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:51.020075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.425292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":874,"reported_epoch":624,"state":"active+clean","last_fresh":"2024-09-02T10:22:33.547765+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:22:33.547765+0000","last_peered":"2024-09-02T10:22:33.547765+0000","last_clean":"2024-09-02T10:22:33.547765+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:22:33.547765+0000","last_undegraded":"2024-09-02T10:22:33.547765+0000","last_fullsized":"2024-09-02T10:22:33.547765+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":860,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.159066+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:22:34.159066+0000","last_peered":"2024-09-02T10:22:34.159066+0000","last_clean":"2024-09-02T10:22:34.159066+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:22:34.159066+0000","last_undegraded":"2024-09-02T10:22:34.159066+0000","last_fullsized":"2024-09-02T10:22:34.159066+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.685318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":104,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.167463+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:22:34.167463+0000","last_peered":"2024-09-02T10:22:34.167463+0000","last_clean":"2024-09-02T10:22:34.167463+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:22:34.167463+0000","last_undegraded":"2024-09-02T10:22:34.167463+0000","last_fullsized":"2024-09-02T10:22:34.167463+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1207,"reported_epoch":624,"state":"active+clean","last_fresh":"2024-09-02T10:22:33.547596+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:22:33.547596+0000","last_peered":"2024-09-02T10:22:33.547596+0000","last_clean":"2024-09-02T10:22:33.547596+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:22:33.547596+0000","last_undegraded":"2024-09-02T10:22:33.547596+0000","last_fullsized":"2024-09-02T10:22:33.547596+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1140,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.589988+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:22:28.589988+0000","last_peered":"2024-09-02T10:22:28.589988+0000","last_clean":"2024-09-02T10:22:28.589988+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:22:28.589988+0000","last_undegraded":"2024-09-02T10:22:28.589988+0000","last_fullsized":"2024-09-02T10:22:28.589988+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1119,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.506999+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:22:28.506999+0000","last_peered":"2024-09-02T10:22:28.506999+0000","last_clean":"2024-09-02T10:22:28.506999+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:22:28.506999+0000","last_undegraded":"2024-09-02T10:22:28.506999+0000","last_fullsized":"2024-09-02T10:22:28.506999+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1110,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.590526+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:22:28.590526+0000","last_peered":"2024-09-02T10:22:28.590526+0000","last_clean":"2024-09-02T10:22:28.590526+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:22:28.590526+0000","last_undegraded":"2024-09-02T10:22:28.590526+0000","last_fullsized":"2024-09-02T10:22:28.590526+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"623'12106","reported_seq":19613,"reported_epoch":623,"state":"active+recovering+remapped","last_fresh":"2024-09-02T10:22:32.521715+0000","last_change":"2024-09-02T10:20:53.492211+0000","last_active":"2024-09-02T10:22:32.521715+0000","last_peered":"2024-09-02T10:22:32.521715+0000","last_clean":"2024-09-02T10:20:51.027001+0000","last_became_active":"2024-09-02T10:20:53.416507+0000","last_became_peered":"2024-09-02T10:20:53.416507+0000","last_unstale":"2024-09-02T10:22:32.521715+0000","last_undegraded":"2024-09-02T10:22:32.521715+0000","last_fullsized":"2024-09-02T10:22:32.521715+0000","mapping_epoch":586,"log_start":"547'2100","ondisk_log_start":"547'2100","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18874368,"num_objects":288,"num_object_clones":0,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":315,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12106,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3627,"num_bytes_recovered":236584960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[5,10,9],"avail_no_missing":["5","9","10"],"object_location_counts":[{"shards":"5,9,10","objects":-27}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1204,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.590594+0000","last_change":"2024-09-02T10:20:52.111563+0000","last_active":"2024-09-02T10:22:28.590594+0000","last_peered":"2024-09-02T10:22:28.590594+0000","last_clean":"2024-09-02T10:22:28.590594+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:22:28.590594+0000","last_undegraded":"2024-09-02T10:22:28.590594+0000","last_fullsized":"2024-09-02T10:22:28.590594+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"625'12342","reported_seq":24098,"reported_epoch":625,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:34.535223+0000","last_change":"2024-09-02T10:20:53.422779+0000","last_active":"2024-09-02T10:22:34.535223+0000","last_peered":"2024-09-02T10:22:34.535223+0000","last_clean":"2024-09-02T10:20:51.098224+0000","last_became_active":"2024-09-02T10:20:53.370526+0000","last_became_peered":"2024-09-02T10:20:53.370526+0000","last_unstale":"2024-09-02T10:22:34.535223+0000","last_undegraded":"2024-09-02T10:20:53.137669+0000","last_fullsized":"2024-09-02T10:20:53.103100+0000","mapping_epoch":586,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15597568,"num_objects":238,"num_object_clones":0,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12342,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5743,"num_bytes_recovered":374538240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-237}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1331,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.158792+0000","last_change":"2024-09-02T10:21:11.035967+0000","last_active":"2024-09-02T10:22:34.158792+0000","last_peered":"2024-09-02T10:22:34.158792+0000","last_clean":"2024-09-02T10:22:34.158792+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:22:34.158792+0000","last_undegraded":"2024-09-02T10:22:34.158792+0000","last_fullsized":"2024-09-02T10:22:34.158792+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:11.035872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:11.035872+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"623'12068","reported_seq":12319,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:32.459130+0000","last_change":"2024-09-02T10:20:27.715367+0000","last_active":"2024-09-02T10:22:32.459130+0000","last_peered":"2024-09-02T10:22:32.459130+0000","last_clean":"2024-09-02T10:22:32.459130+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:22:32.459130+0000","last_undegraded":"2024-09-02T10:22:32.459130+0000","last_fullsized":"2024-09-02T10:22:32.459130+0000","mapping_epoch":533,"log_start":"546'2000","ondisk_log_start":"546'2000","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7181","last_scrub_stamp":"2024-09-02T10:20:27.715126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:27.715126+0000","objects_scrubbed":5211,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18546712,"num_objects":284,"num_object_clones":0,"num_object_copies":852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12068,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":970,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.541524+0000","last_change":"2024-09-02T10:20:18.812280+0000","last_active":"2024-09-02T10:22:28.541524+0000","last_peered":"2024-09-02T10:22:28.541524+0000","last_clean":"2024-09-02T10:22:28.541524+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:22:28.541524+0000","last_undegraded":"2024-09-02T10:22:28.541524+0000","last_fullsized":"2024-09-02T10:22:28.541524+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.812212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.812212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"623'12134","reported_seq":12402,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:33.392685+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:22:33.392685+0000","last_peered":"2024-09-02T10:22:33.392685+0000","last_clean":"2024-09-02T10:22:33.392685+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:22:33.392685+0000","last_undegraded":"2024-09-02T10:22:33.392685+0000","last_fullsized":"2024-09-02T10:22:33.392685+0000","mapping_epoch":533,"log_start":"547'2100","ondisk_log_start":"547'2100","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16252928,"num_objects":248,"num_object_clones":0,"num_object_copies":744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12134,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":379,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.178275+0000","last_change":"2024-09-02T10:22:15.563077+0000","last_active":"2024-09-02T10:22:34.178275+0000","last_peered":"2024-09-02T10:22:34.178275+0000","last_clean":"2024-09-02T10:22:34.178275+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:22:34.178275+0000","last_undegraded":"2024-09-02T10:22:34.178275+0000","last_fullsized":"2024-09-02T10:22:34.178275+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:15.563022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:15.563022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:20.809417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"625'12453","reported_seq":12741,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:36.384411+0000","last_change":"2024-09-02T10:21:41.827351+0000","last_active":"2024-09-02T10:22:36.384411+0000","last_peered":"2024-09-02T10:22:36.384411+0000","last_clean":"2024-09-02T10:22:36.384411+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:22:36.384411+0000","last_undegraded":"2024-09-02T10:22:36.384411+0000","last_fullsized":"2024-09-02T10:22:36.384411+0000","mapping_epoch":533,"log_start":"552'2400","ondisk_log_start":"552'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10385","last_scrub_stamp":"2024-09-02T10:21:41.827172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:41.827172+0000","objects_scrubbed":2236,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:45.086417+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":153,"num_object_clones":0,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12453,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1287,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.647196+0000","last_change":"2024-09-02T10:22:34.647196+0000","last_active":"2024-09-02T10:22:34.647196+0000","last_peered":"2024-09-02T10:22:34.647196+0000","last_clean":"2024-09-02T10:22:34.647196+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:22:34.647196+0000","last_undegraded":"2024-09-02T10:22:34.647196+0000","last_fullsized":"2024-09-02T10:22:34.647196+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:34.647144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:34.647144+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:02.024298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"625'12330","reported_seq":24759,"reported_epoch":625,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:34.536519+0000","last_change":"2024-09-02T10:20:53.407005+0000","last_active":"2024-09-02T10:22:34.536519+0000","last_peered":"2024-09-02T10:22:34.536519+0000","last_clean":"2024-09-02T10:20:51.097394+0000","last_became_active":"2024-09-02T10:20:53.379154+0000","last_became_peered":"2024-09-02T10:20:53.379154+0000","last_unstale":"2024-09-02T10:22:34.536519+0000","last_undegraded":"2024-09-02T10:20:53.138449+0000","last_fullsized":"2024-09-02T10:20:53.102891+0000","mapping_epoch":586,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14024704,"num_objects":214,"num_object_clones":0,"num_object_copies":642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12330,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6079,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3],"avail_no_missing":["4","3","11"],"object_location_counts":[{"shards":"3,4","objects":19}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":746,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.152790+0000","last_change":"2024-09-02T10:20:52.104138+0000","last_active":"2024-09-02T10:22:34.152790+0000","last_peered":"2024-09-02T10:22:34.152790+0000","last_clean":"2024-09-02T10:22:34.152790+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:22:34.152790+0000","last_undegraded":"2024-09-02T10:22:34.152790+0000","last_fullsized":"2024-09-02T10:22:34.152790+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"625'12127","reported_seq":12417,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.394652+0000","last_change":"2024-09-02T10:21:39.544819+0000","last_active":"2024-09-02T10:22:34.394652+0000","last_peered":"2024-09-02T10:22:34.394652+0000","last_clean":"2024-09-02T10:22:34.394652+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:22:34.394652+0000","last_undegraded":"2024-09-02T10:22:34.394652+0000","last_fullsized":"2024-09-02T10:22:34.394652+0000","mapping_epoch":533,"log_start":"547'2100","ondisk_log_start":"547'2100","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"603'10078","last_scrub_stamp":"2024-09-02T10:21:39.544630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.544630+0000","objects_scrubbed":2309,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.692572+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16580608,"num_objects":253,"num_object_clones":0,"num_object_copies":759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12127,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":710,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.152824+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:22:34.152824+0000","last_peered":"2024-09-02T10:22:34.152824+0000","last_clean":"2024-09-02T10:22:34.152824+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:22:34.152824+0000","last_undegraded":"2024-09-02T10:22:34.152824+0000","last_fullsized":"2024-09-02T10:22:34.152824+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"623'12388","reported_seq":12719,"reported_epoch":623,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:32.626246+0000","last_change":"2024-09-02T10:20:53.439960+0000","last_active":"2024-09-02T10:22:32.626246+0000","last_peered":"2024-09-02T10:22:32.626246+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:22:32.626246+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19005440,"num_objects":290,"num_object_clones":0,"num_object_copies":870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6299,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12388,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-3804},{"shards":"0,3,5","objects":290}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":478,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.157389+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:22:34.157389+0000","last_peered":"2024-09-02T10:22:34.157389+0000","last_clean":"2024-09-02T10:22:34.157389+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:22:34.157389+0000","last_undegraded":"2024-09-02T10:22:34.157389+0000","last_fullsized":"2024-09-02T10:22:34.157389+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:16.363558+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"623'12276","reported_seq":16888,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:33.380122+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:22:33.380122+0000","last_peered":"2024-09-02T10:22:33.380122+0000","last_clean":"2024-09-02T10:22:33.380122+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:22:33.380122+0000","last_undegraded":"2024-09-02T10:22:33.380122+0000","last_fullsized":"2024-09-02T10:22:33.380122+0000","mapping_epoch":559,"log_start":"550'2200","ondisk_log_start":"550'2200","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17432576,"num_objects":266,"num_object_clones":0,"num_object_copies":798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12276,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":711,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.152728+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:22:34.152728+0000","last_peered":"2024-09-02T10:22:34.152728+0000","last_clean":"2024-09-02T10:22:34.152728+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:22:34.152728+0000","last_undegraded":"2024-09-02T10:22:34.152728+0000","last_fullsized":"2024-09-02T10:22:34.152728+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"625'12472","reported_seq":12763,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:36.408373+0000","last_change":"2024-09-02T10:21:39.988693+0000","last_active":"2024-09-02T10:22:36.408373+0000","last_peered":"2024-09-02T10:22:36.408373+0000","last_clean":"2024-09-02T10:22:36.408373+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:22:36.408373+0000","last_undegraded":"2024-09-02T10:22:36.408373+0000","last_fullsized":"2024-09-02T10:22:36.408373+0000","mapping_epoch":533,"log_start":"552'2400","ondisk_log_start":"552'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10274","last_scrub_stamp":"2024-09-02T10:21:39.988364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.988364+0000","objects_scrubbed":2383,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:43.749337+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":160,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12472,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":973,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.158870+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:22:34.158870+0000","last_peered":"2024-09-02T10:22:34.158870+0000","last_clean":"2024-09-02T10:22:34.158870+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:22:34.158870+0000","last_undegraded":"2024-09-02T10:22:34.158870+0000","last_fullsized":"2024-09-02T10:22:34.158870+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:54.988677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"624'12344","reported_seq":12597,"reported_epoch":624,"state":"active+clean","last_fresh":"2024-09-02T10:22:33.746556+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:22:33.746556+0000","last_peered":"2024-09-02T10:22:33.746556+0000","last_clean":"2024-09-02T10:22:33.746556+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:22:33.746556+0000","last_undegraded":"2024-09-02T10:22:33.746556+0000","last_fullsized":"2024-09-02T10:22:33.746556+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12344,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1230,"reported_epoch":624,"state":"active+clean","last_fresh":"2024-09-02T10:22:33.547870+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:22:33.547870+0000","last_peered":"2024-09-02T10:22:33.547870+0000","last_clean":"2024-09-02T10:22:33.547870+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:22:33.547870+0000","last_undegraded":"2024-09-02T10:22:33.547870+0000","last_fullsized":"2024-09-02T10:22:33.547870+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"623'12317","reported_seq":16817,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:32.515989+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:22:32.515989+0000","last_peered":"2024-09-02T10:22:32.515989+0000","last_clean":"2024-09-02T10:22:32.515989+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:22:32.515989+0000","last_undegraded":"2024-09-02T10:22:32.515989+0000","last_fullsized":"2024-09-02T10:22:32.515989+0000","mapping_epoch":565,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20643840,"num_objects":315,"num_object_clones":0,"num_object_copies":945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12317,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":984,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.158795+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:22:34.158795+0000","last_peered":"2024-09-02T10:22:34.158795+0000","last_clean":"2024-09-02T10:22:34.158795+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:22:34.158795+0000","last_undegraded":"2024-09-02T10:22:34.158795+0000","last_fullsized":"2024-09-02T10:22:34.158795+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.846770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"625'12461","reported_seq":12757,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:36.338189+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:22:36.338189+0000","last_peered":"2024-09-02T10:22:36.338189+0000","last_clean":"2024-09-02T10:22:36.338189+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:22:36.338189+0000","last_undegraded":"2024-09-02T10:22:36.338189+0000","last_fullsized":"2024-09-02T10:22:36.338189+0000","mapping_epoch":533,"log_start":"553'2400","ondisk_log_start":"553'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.913566+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12386304,"num_objects":189,"num_object_clones":0,"num_object_copies":567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12461,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1239,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.157288+0000","last_change":"2024-09-02T10:21:35.696117+0000","last_active":"2024-09-02T10:22:34.157288+0000","last_peered":"2024-09-02T10:22:34.157288+0000","last_clean":"2024-09-02T10:22:34.157288+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:22:34.157288+0000","last_undegraded":"2024-09-02T10:22:34.157288+0000","last_fullsized":"2024-09-02T10:22:34.157288+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:35.696044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.696044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:46.753483+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"623'12209","reported_seq":12465,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:32.901498+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:22:32.901498+0000","last_peered":"2024-09-02T10:22:32.901498+0000","last_clean":"2024-09-02T10:22:32.901498+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:22:32.901498+0000","last_undegraded":"2024-09-02T10:22:32.901498+0000","last_fullsized":"2024-09-02T10:22:32.901498+0000","mapping_epoch":533,"log_start":"551'2200","ondisk_log_start":"551'2200","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18808832,"num_objects":287,"num_object_clones":0,"num_object_copies":861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12209,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":188,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.165957+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:22:34.165957+0000","last_peered":"2024-09-02T10:22:34.165957+0000","last_clean":"2024-09-02T10:22:34.165957+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:22:34.165957+0000","last_undegraded":"2024-09-02T10:22:34.165957+0000","last_fullsized":"2024-09-02T10:22:34.165957+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"623'12229","reported_seq":12487,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:32.497780+0000","last_change":"2024-09-02T10:20:21.937271+0000","last_active":"2024-09-02T10:22:32.497780+0000","last_peered":"2024-09-02T10:22:32.497780+0000","last_clean":"2024-09-02T10:22:32.497780+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:22:32.497780+0000","last_undegraded":"2024-09-02T10:22:32.497780+0000","last_fullsized":"2024-09-02T10:22:32.497780+0000","mapping_epoch":533,"log_start":"551'2200","ondisk_log_start":"551'2200","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6863","last_scrub_stamp":"2024-09-02T10:20:21.937140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.937140+0000","objects_scrubbed":5699,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":291,"num_object_clones":0,"num_object_copies":873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12229,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1019,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.159096+0000","last_change":"2024-09-02T10:22:30.358040+0000","last_active":"2024-09-02T10:22:34.159096+0000","last_peered":"2024-09-02T10:22:34.159096+0000","last_clean":"2024-09-02T10:22:34.159096+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:22:34.159096+0000","last_undegraded":"2024-09-02T10:22:34.159096+0000","last_fullsized":"2024-09-02T10:22:34.159096+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:30.357970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:30.357970+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.221263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"623'12271","reported_seq":12527,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:33.314882+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:22:33.314882+0000","last_peered":"2024-09-02T10:22:33.314882+0000","last_clean":"2024-09-02T10:22:33.314882+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:22:33.314882+0000","last_undegraded":"2024-09-02T10:22:33.314882+0000","last_fullsized":"2024-09-02T10:22:33.314882+0000","mapping_epoch":533,"log_start":"550'2200","ondisk_log_start":"550'2200","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19333120,"num_objects":295,"num_object_clones":0,"num_object_copies":885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12271,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":399,"reported_epoch":623,"state":"active+clean","last_fresh":"2024-09-02T10:22:28.577611+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:22:28.577611+0000","last_peered":"2024-09-02T10:22:28.577611+0000","last_clean":"2024-09-02T10:22:28.577611+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:22:28.577611+0000","last_undegraded":"2024-09-02T10:22:28.577611+0000","last_fullsized":"2024-09-02T10:22:28.577611+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":262930456,"num_objects":4013,"num_object_clones":0,"num_object_copies":12039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6969,"num_objects_misplaced":315,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":196527,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19781,"num_bytes_recovered":1284767744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1509306368,"data_stored":1509294152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160727,"ondisk_log_size":160727,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383950,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":763020,"kb_used_data":280636,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":92982132,"statfs":{"total":95995035648,"available":95213703168,"internally_reserved":0,"allocated":287371264,"data_stored":287057604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":493943236},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612883,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":386772,"kb_used_data":34880,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":93358380,"statfs":{"total":95995035648,"available":95598981120,"internally_reserved":0,"allocated":35717120,"data_stored":35399388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":360315328},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841815,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614324,"kb_used_data":73636,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":93130828,"statfs":{"total":95995035648,"available":95365967872,"internally_reserved":0,"allocated":75403264,"data_stored":75098220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":553646528},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070749,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":369412,"kb_used_data":35568,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93375740,"statfs":{"total":95995035648,"available":95616757760,"internally_reserved":0,"allocated":36421632,"data_stored":36112305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299683,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636244,"kb_used_data":55300,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93108908,"statfs":{"total":95995035648,"available":95343521792,"internally_reserved":0,"allocated":56627200,"data_stored":56309988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528614,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":396364,"kb_used_data":31996,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93348788,"statfs":{"total":95995035648,"available":95589158912,"internally_reserved":0,"allocated":32763904,"data_stored":32450244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724842,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1243584,"kb_used_data":639024,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92501568,"statfs":{"total":95995035648,"available":94721605632,"internally_reserved":0,"allocated":654360576,"data_stored":654047908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953776,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":407632,"kb_used_data":61308,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":93337520,"statfs":{"total":95995035648,"available":95577620480,"internally_reserved":0,"allocated":62779392,"data_stored":62465732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150005,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722932,"kb_used_data":146148,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93022220,"statfs":{"total":95995035648,"available":95254753280,"internally_reserved":0,"allocated":149655552,"data_stored":149350508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":590608840},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411641,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312920,"kb_used_data":33220,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":93432232,"statfs":{"total":95995035648,"available":95674605568,"internally_reserved":0,"allocated":34017280,"data_stored":33700068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575166,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":574072,"kb_used_data":69480,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93171080,"statfs":{"total":95995035648,"available":95407185920,"internally_reserved":0,"allocated":71147520,"data_stored":70838404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738692,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":629556,"kb_used_data":61092,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93115596,"statfs":{"total":95995035648,"available":95350370304,"internally_reserved":0,"allocated":62558208,"data_stored":62253164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":55705600,"data_stored":55705600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":67768320,"data_stored":67764248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25231360,"data_stored":25231360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":143196160,"data_stored":143196160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61210624,"data_stored":61210624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":651431936,"data_stored":651427864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":31260672,"data_stored":31260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":54657024,"data_stored":54657024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32243712,"data_stored":32243712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":74383360,"data_stored":74383360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":29298688,"data_stored":29294616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":282918912,"data_stored":282918912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:22:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6969/12045 objects degraded (57.858%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:39 smithi162 ceph-mon[20279]: osdmap e626: 12 total, 12 up, 12 in 2024-09-02T10:22:39.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:39.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:39.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6969/12045 objects degraded (57.858%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:39 smithi187 ceph-mon[20072]: osdmap e626: 12 total, 12 up, 12 in 2024-09-02T10:22:40.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:40.043 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:40.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:40.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:40.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:40.151 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:40.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:40.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:40.295 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:40.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:40.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:40.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:40.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:40.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:40.456 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:40.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:40.581 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:40.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:40.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:40.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:40 smithi151 ceph-mon[16639]: pgmap v1456: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 144 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 6964/6873 objects degraded (101.324%); 294/6873 objects misplaced (4.278%); 5.1 MiB/s, 139 objects/s recovering 2024-09-02T10:22:40.606 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3971395555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:22:40.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:40.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:22:40.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:40.723 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:40.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:40.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:40 smithi162 ceph-mon[20279]: pgmap v1456: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 144 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 6964/6873 objects degraded (101.324%); 294/6873 objects misplaced (4.278%); 5.1 MiB/s, 139 objects/s recovering 2024-09-02T10:22:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3971395555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:22:40.846 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:40.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:40.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:40.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:40.970 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:40.971 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:40 smithi187 ceph-mon[20072]: pgmap v1456: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 144 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 6964/6873 objects degraded (101.324%); 294/6873 objects misplaced (4.278%); 5.1 MiB/s, 139 objects/s recovering 2024-09-02T10:22:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3971395555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:22:41.032 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:22:41.072 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:41.093 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:41.094 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:41.165 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:22:41.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:41.218 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:41.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:41.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:41.343 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:41.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:41.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:41.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:41.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:41.570 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:41.593 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:41.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:41.594 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:41.605 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:22:41.605 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:22:41.629 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:22:41.629 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:22:41.651 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:22:41.652 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:22:41.674 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:22:41.674 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:22:41.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:41.696 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:22:41.696 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:22:41.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:41.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:41.718 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:41.718 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:22:41.718 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:22:41.740 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:22:41.740 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:22:41.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:22:41.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:22:41.785 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:22:41.785 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:22:41.796 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:41 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:41.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:41 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:41.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2181934317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:41.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:41 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:41.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3971395555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:22:41.797 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:41 smithi162 ceph-mon[20279]: osdmap e627: 12 total, 12 up, 12 in 2024-09-02T10:22:41.806 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:22:41.807 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:22:41.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:41 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:41.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:41 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:41.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2181934317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:41.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:41 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:41.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3971395555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:22:41.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:41 smithi151 ceph-mon[16639]: osdmap e627: 12 total, 12 up, 12 in 2024-09-02T10:22:41.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:41.828 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:22:41.828 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:22:41.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:41.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:41.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:41.850 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:22:41.850 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:22:41.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:41.967 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:41.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:41.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:41 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:41 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2181934317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:41 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3971395555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:22:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:41 smithi187 ceph-mon[20072]: osdmap e627: 12 total, 12 up, 12 in 2024-09-02T10:22:42.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:42.090 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:42.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:42.099 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:22:42.099 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:22:42.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:42.214 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:42.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:42.316 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:42.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:42.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:42.343 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:42.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:42.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:42.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:42.571 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:42.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:42.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:42.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:42 smithi151 ceph-mon[16639]: pgmap v1458: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 109 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 451 op/s; 6964/5211 objects degraded (133.640%); 294/5211 objects misplaced (5.642%); 4.2 MiB/s, 189 objects/s recovering 2024-09-02T10:22:42.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:42.721 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:42.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:42.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:22:42.798 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:42 smithi162 ceph-mon[20279]: pgmap v1458: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 109 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 451 op/s; 6964/5211 objects degraded (133.640%); 294/5211 objects misplaced (5.642%); 4.2 MiB/s, 189 objects/s recovering 2024-09-02T10:22:42.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:42.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:22:42.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:42.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:42.849 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:42.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:42.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:42.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:42.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:42 smithi187 ceph-mon[20072]: pgmap v1458: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 109 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 451 op/s; 6964/5211 objects degraded (133.640%); 294/5211 objects misplaced (5.642%); 4.2 MiB/s, 189 objects/s recovering 2024-09-02T10:22:43.029 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:22:43.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:43.087 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:43.100 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:43.101 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:43.201 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:43.227 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:43.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:43.228 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:43.329 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:43.354 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:43.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:43.355 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:43.456 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:43.481 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:43.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:43.482 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:43.569 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:22:43.569 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:22:43.583 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:43.608 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:43.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:43.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:43.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:43.733 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:43.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:43.827 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:43.828 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:22:43.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:43.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:43.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:43.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:43.983 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:43.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:44.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:22:44.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:44.106 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:44.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:44.208 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:44.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:44.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:44.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:44.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:44.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:44.363 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:44.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:44.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:44.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:44.491 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:44.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:44.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:44.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:44.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:44.690 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1459,"stamp":"2024-09-02T10:22:43.396304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":33882680,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6964,"num_objects_misplaced":294,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200031,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20913,"num_bytes_recovered":1287075424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163583,"ondisk_log_size":163583,"up":141,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":6481948,"kb_used_data":900212,"kb_used_omap":18,"kb_used_meta":5581485,"kb_avail":1118459876,"statfs":{"total":1151940427776,"available":1145302913024,"internally_reserved":0,"allocated":921817088,"data_stored":917961318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5715440955},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,2,4,8],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-263520256,"num_objects":-4021,"num_object_clones":0,"num_object_copies":-12063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5,"num_objects_misplaced":-21,"num_objects_unfound":0,"num_objects_dirty":-4021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4021,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1128,"num_bytes_recovered":1376256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001698"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":616,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252327+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:22:41.252327+0000","last_peered":"2024-09-02T10:22:41.252327+0000","last_clean":"2024-09-02T10:22:41.252327+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:22:41.252327+0000","last_undegraded":"2024-09-02T10:22:41.252327+0000","last_fullsized":"2024-09-02T10:22:41.252327+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":419,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.158363+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:22:34.158363+0000","last_peered":"2024-09-02T10:22:34.158363+0000","last_clean":"2024-09-02T10:22:34.158363+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:22:34.158363+0000","last_undegraded":"2024-09-02T10:22:34.158363+0000","last_fullsized":"2024-09-02T10:22:34.158363+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":879,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-09-02T10:22:39.179997+0000","last_change":"2024-09-02T10:22:32.860224+0000","last_active":"2024-09-02T10:22:39.179997+0000","last_peered":"2024-09-02T10:22:39.179997+0000","last_clean":"2024-09-02T10:22:39.179997+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:22:39.179997+0000","last_undegraded":"2024-09-02T10:22:39.179997+0000","last_fullsized":"2024-09-02T10:22:39.179997+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:32.860180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:32.860180+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.373890+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":871,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-09-02T10:22:39.179905+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:22:39.179905+0000","last_peered":"2024-09-02T10:22:39.179905+0000","last_clean":"2024-09-02T10:22:39.179905+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:22:39.179905+0000","last_undegraded":"2024-09-02T10:22:39.179905+0000","last_fullsized":"2024-09-02T10:22:39.179905+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.838669+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":175,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-09-02T10:22:39.158933+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:22:39.158933+0000","last_peered":"2024-09-02T10:22:39.158933+0000","last_clean":"2024-09-02T10:22:39.158933+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:22:39.158933+0000","last_undegraded":"2024-09-02T10:22:39.158933+0000","last_fullsized":"2024-09-02T10:22:39.158933+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":714,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.172975+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:22:41.172975+0000","last_peered":"2024-09-02T10:22:41.172975+0000","last_clean":"2024-09-02T10:22:41.172975+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:22:41.172975+0000","last_undegraded":"2024-09-02T10:22:41.172975+0000","last_fullsized":"2024-09-02T10:22:41.172975+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1249,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252781+0000","last_change":"2024-09-02T10:20:52.113290+0000","last_active":"2024-09-02T10:22:41.252781+0000","last_peered":"2024-09-02T10:22:41.252781+0000","last_clean":"2024-09-02T10:22:41.252781+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:22:41.252781+0000","last_undegraded":"2024-09-02T10:22:41.252781+0000","last_fullsized":"2024-09-02T10:22:41.252781+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:29.189593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:29.189593+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":886,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-09-02T10:22:39.180033+0000","last_change":"2024-09-02T10:21:51.020121+0000","last_active":"2024-09-02T10:22:39.180033+0000","last_peered":"2024-09-02T10:22:39.180033+0000","last_clean":"2024-09-02T10:22:39.180033+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:22:39.180033+0000","last_undegraded":"2024-09-02T10:22:39.180033+0000","last_fullsized":"2024-09-02T10:22:39.180033+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:51.020075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:51.020075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.425292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":876,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.158257+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:22:34.158257+0000","last_peered":"2024-09-02T10:22:34.158257+0000","last_clean":"2024-09-02T10:22:34.158257+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:22:34.158257+0000","last_undegraded":"2024-09-02T10:22:34.158257+0000","last_fullsized":"2024-09-02T10:22:34.158257+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":864,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252839+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:22:41.252839+0000","last_peered":"2024-09-02T10:22:41.252839+0000","last_clean":"2024-09-02T10:22:41.252839+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:22:41.252839+0000","last_undegraded":"2024-09-02T10:22:41.252839+0000","last_fullsized":"2024-09-02T10:22:41.252839+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.685318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":108,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252626+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:22:41.252626+0000","last_peered":"2024-09-02T10:22:41.252626+0000","last_clean":"2024-09-02T10:22:41.252626+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:22:41.252626+0000","last_undegraded":"2024-09-02T10:22:41.252626+0000","last_fullsized":"2024-09-02T10:22:41.252626+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1209,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.158130+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:22:34.158130+0000","last_peered":"2024-09-02T10:22:34.158130+0000","last_clean":"2024-09-02T10:22:34.158130+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:22:34.158130+0000","last_undegraded":"2024-09-02T10:22:34.158130+0000","last_fullsized":"2024-09-02T10:22:34.158130+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1148,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252620+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:22:41.252620+0000","last_peered":"2024-09-02T10:22:41.252620+0000","last_clean":"2024-09-02T10:22:41.252620+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:22:41.252620+0000","last_undegraded":"2024-09-02T10:22:41.252620+0000","last_fullsized":"2024-09-02T10:22:41.252620+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1127,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.168895+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:22:41.168895+0000","last_peered":"2024-09-02T10:22:41.168895+0000","last_clean":"2024-09-02T10:22:41.168895+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:22:41.168895+0000","last_undegraded":"2024-09-02T10:22:41.168895+0000","last_fullsized":"2024-09-02T10:22:41.168895+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1118,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252994+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:22:41.252994+0000","last_peered":"2024-09-02T10:22:41.252994+0000","last_clean":"2024-09-02T10:22:41.252994+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:22:41.252994+0000","last_undegraded":"2024-09-02T10:22:41.252994+0000","last_fullsized":"2024-09-02T10:22:41.252994+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"626'12394","reported_seq":21083,"reported_epoch":627,"state":"active+recovering+remapped","last_fresh":"2024-09-02T10:22:42.525554+0000","last_change":"2024-09-02T10:20:53.492211+0000","last_active":"2024-09-02T10:22:42.525554+0000","last_peered":"2024-09-02T10:22:42.525554+0000","last_clean":"2024-09-02T10:20:51.027001+0000","last_became_active":"2024-09-02T10:20:53.416507+0000","last_became_peered":"2024-09-02T10:20:53.416507+0000","last_unstale":"2024-09-02T10:22:42.525554+0000","last_undegraded":"2024-09-02T10:22:42.525554+0000","last_fullsized":"2024-09-02T10:22:42.525554+0000","mapping_epoch":586,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":294,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12394,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4214,"num_bytes_recovered":237961216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[5,10,9],"avail_no_missing":["5","9","10"],"object_location_counts":[{"shards":"5,9,10","objects":-294}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1212,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.253095+0000","last_change":"2024-09-02T10:20:52.111563+0000","last_active":"2024-09-02T10:22:41.253095+0000","last_peered":"2024-09-02T10:22:41.253095+0000","last_clean":"2024-09-02T10:22:41.253095+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:22:41.253095+0000","last_undegraded":"2024-09-02T10:22:41.253095+0000","last_fullsized":"2024-09-02T10:22:41.253095+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"626'12535","reported_seq":24809,"reported_epoch":626,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:39.532800+0000","last_change":"2024-09-02T10:20:53.422779+0000","last_active":"2024-09-02T10:22:39.532800+0000","last_peered":"2024-09-02T10:22:39.532800+0000","last_clean":"2024-09-02T10:20:51.098224+0000","last_became_active":"2024-09-02T10:20:53.370526+0000","last_became_peered":"2024-09-02T10:20:53.370526+0000","last_unstale":"2024-09-02T10:22:39.532800+0000","last_undegraded":"2024-09-02T10:20:53.137669+0000","last_fullsized":"2024-09-02T10:20:53.103100+0000","mapping_epoch":586,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12535,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6002,"num_bytes_recovered":374538240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-430}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1343,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252475+0000","last_change":"2024-09-02T10:22:37.332903+0000","last_active":"2024-09-02T10:22:41.252475+0000","last_peered":"2024-09-02T10:22:41.252475+0000","last_clean":"2024-09-02T10:22:41.252475+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:22:41.252475+0000","last_undegraded":"2024-09-02T10:22:41.252475+0000","last_fullsized":"2024-09-02T10:22:41.252475+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:37.332859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:37.332859+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.675027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"626'12351","reported_seq":12612,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252989+0000","last_change":"2024-09-02T10:20:27.715367+0000","last_active":"2024-09-02T10:22:41.252989+0000","last_peered":"2024-09-02T10:22:41.252989+0000","last_clean":"2024-09-02T10:22:41.252989+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:22:41.252989+0000","last_undegraded":"2024-09-02T10:22:41.252989+0000","last_fullsized":"2024-09-02T10:22:41.252989+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7181","last_scrub_stamp":"2024-09-02T10:20:27.715126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:27.715126+0000","objects_scrubbed":5211,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12351,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":978,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252400+0000","last_change":"2024-09-02T10:20:18.812280+0000","last_active":"2024-09-02T10:22:41.252400+0000","last_peered":"2024-09-02T10:22:41.252400+0000","last_clean":"2024-09-02T10:22:41.252400+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:22:41.252400+0000","last_undegraded":"2024-09-02T10:22:41.252400+0000","last_fullsized":"2024-09-02T10:22:41.252400+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.812212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.812212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"625'12295","reported_seq":12567,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:38.321224+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:22:38.321224+0000","last_peered":"2024-09-02T10:22:38.321224+0000","last_clean":"2024-09-02T10:22:38.321224+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:22:38.321224+0000","last_undegraded":"2024-09-02T10:22:38.321224+0000","last_fullsized":"2024-09-02T10:22:38.321224+0000","mapping_epoch":533,"log_start":"550'2200","ondisk_log_start":"550'2200","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12295,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":383,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252255+0000","last_change":"2024-09-02T10:22:15.563077+0000","last_active":"2024-09-02T10:22:41.252255+0000","last_peered":"2024-09-02T10:22:41.252255+0000","last_clean":"2024-09-02T10:22:41.252255+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:22:41.252255+0000","last_undegraded":"2024-09-02T10:22:41.252255+0000","last_fullsized":"2024-09-02T10:22:41.252255+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:15.563022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:15.563022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:20.809417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"626'12606","reported_seq":12898,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252502+0000","last_change":"2024-09-02T10:21:41.827351+0000","last_active":"2024-09-02T10:22:41.252502+0000","last_peered":"2024-09-02T10:22:41.252502+0000","last_clean":"2024-09-02T10:22:41.252502+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:22:41.252502+0000","last_undegraded":"2024-09-02T10:22:41.252502+0000","last_fullsized":"2024-09-02T10:22:41.252502+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10385","last_scrub_stamp":"2024-09-02T10:21:41.827172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:41.827172+0000","objects_scrubbed":2236,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:45.086417+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1292,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252370+0000","last_change":"2024-09-02T10:22:34.647196+0000","last_active":"2024-09-02T10:22:41.252370+0000","last_peered":"2024-09-02T10:22:41.252370+0000","last_clean":"2024-09-02T10:22:41.252370+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:22:41.252370+0000","last_undegraded":"2024-09-02T10:22:41.252370+0000","last_fullsized":"2024-09-02T10:22:41.252370+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:34.647144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:34.647144+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:02.024298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"626'12502","reported_seq":25487,"reported_epoch":626,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:39.531333+0000","last_change":"2024-09-02T10:20:53.407005+0000","last_active":"2024-09-02T10:22:39.531333+0000","last_peered":"2024-09-02T10:22:39.531333+0000","last_clean":"2024-09-02T10:20:51.097394+0000","last_became_active":"2024-09-02T10:20:53.379154+0000","last_became_peered":"2024-09-02T10:20:53.379154+0000","last_unstale":"2024-09-02T10:22:39.531333+0000","last_undegraded":"2024-09-02T10:20:53.138449+0000","last_fullsized":"2024-09-02T10:20:53.102891+0000","mapping_epoch":586,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12502,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6356,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3],"avail_no_missing":["4","3","11"],"object_location_counts":[{"shards":"3,4","objects":-153}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":750,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.172966+0000","last_change":"2024-09-02T10:20:52.104138+0000","last_active":"2024-09-02T10:22:41.172966+0000","last_peered":"2024-09-02T10:22:41.172966+0000","last_clean":"2024-09-02T10:22:41.172966+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:22:41.172966+0000","last_undegraded":"2024-09-02T10:22:41.172966+0000","last_fullsized":"2024-09-02T10:22:41.172966+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"626'12316","reported_seq":12608,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-09-02T10:22:39.315817+0000","last_change":"2024-09-02T10:21:39.544819+0000","last_active":"2024-09-02T10:22:39.315817+0000","last_peered":"2024-09-02T10:22:39.315817+0000","last_clean":"2024-09-02T10:22:39.315817+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:22:39.315817+0000","last_undegraded":"2024-09-02T10:22:39.315817+0000","last_fullsized":"2024-09-02T10:22:39.315817+0000","mapping_epoch":533,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"603'10078","last_scrub_stamp":"2024-09-02T10:21:39.544630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.544630+0000","objects_scrubbed":2309,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.692572+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12316,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":714,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.172942+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:22:41.172942+0000","last_peered":"2024-09-02T10:22:41.172942+0000","last_clean":"2024-09-02T10:22:41.172942+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:22:41.172942+0000","last_undegraded":"2024-09-02T10:22:41.172942+0000","last_fullsized":"2024-09-02T10:22:41.172942+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"626'12678","reported_seq":13029,"reported_epoch":627,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:41.167770+0000","last_change":"2024-09-02T10:20:53.439960+0000","last_active":"2024-09-02T10:22:41.167770+0000","last_peered":"2024-09-02T10:22:41.167770+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:22:41.167770+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-4089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":482,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.166419+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:22:41.166419+0000","last_peered":"2024-09-02T10:22:41.166419+0000","last_clean":"2024-09-02T10:22:41.166419+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:22:41.166419+0000","last_undegraded":"2024-09-02T10:22:41.166419+0000","last_fullsized":"2024-09-02T10:22:41.166419+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:16.363558+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"625'12444","reported_seq":17060,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:38.417607+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:22:38.417607+0000","last_peered":"2024-09-02T10:22:38.417607+0000","last_clean":"2024-09-02T10:22:38.417607+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:22:38.417607+0000","last_undegraded":"2024-09-02T10:22:38.417607+0000","last_fullsized":"2024-09-02T10:22:38.417607+0000","mapping_epoch":559,"log_start":"552'2400","ondisk_log_start":"552'2400","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6422528,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12444,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":715,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.173055+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:22:41.173055+0000","last_peered":"2024-09-02T10:22:41.173055+0000","last_clean":"2024-09-02T10:22:41.173055+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:22:41.173055+0000","last_undegraded":"2024-09-02T10:22:41.173055+0000","last_fullsized":"2024-09-02T10:22:41.173055+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"626'12632","reported_seq":12928,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252639+0000","last_change":"2024-09-02T10:21:39.988693+0000","last_active":"2024-09-02T10:22:41.252639+0000","last_peered":"2024-09-02T10:22:41.252639+0000","last_clean":"2024-09-02T10:22:41.252639+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:22:41.252639+0000","last_undegraded":"2024-09-02T10:22:41.252639+0000","last_fullsized":"2024-09-02T10:22:41.252639+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10274","last_scrub_stamp":"2024-09-02T10:21:39.988364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.988364+0000","objects_scrubbed":2383,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:43.749337+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":977,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252813+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:22:41.252813+0000","last_peered":"2024-09-02T10:22:41.252813+0000","last_clean":"2024-09-02T10:22:41.252813+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:22:41.252813+0000","last_undegraded":"2024-09-02T10:22:41.252813+0000","last_fullsized":"2024-09-02T10:22:41.252813+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:54.988677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"625'12506","reported_seq":12761,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:38.734315+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:22:38.734315+0000","last_peered":"2024-09-02T10:22:38.734315+0000","last_clean":"2024-09-02T10:22:38.734315+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:22:38.734315+0000","last_undegraded":"2024-09-02T10:22:38.734315+0000","last_fullsized":"2024-09-02T10:22:38.734315+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12506,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1232,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.158236+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:22:34.158236+0000","last_peered":"2024-09-02T10:22:34.158236+0000","last_clean":"2024-09-02T10:22:34.158236+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:22:34.158236+0000","last_undegraded":"2024-09-02T10:22:34.158236+0000","last_fullsized":"2024-09-02T10:22:34.158236+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"626'12632","reported_seq":17140,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.255988+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:22:41.255988+0000","last_peered":"2024-09-02T10:22:41.255988+0000","last_clean":"2024-09-02T10:22:41.255988+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:22:41.255988+0000","last_undegraded":"2024-09-02T10:22:41.255988+0000","last_fullsized":"2024-09-02T10:22:41.255988+0000","mapping_epoch":565,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":988,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252926+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:22:41.252926+0000","last_peered":"2024-09-02T10:22:41.252926+0000","last_clean":"2024-09-02T10:22:41.252926+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:22:41.252926+0000","last_undegraded":"2024-09-02T10:22:41.252926+0000","last_fullsized":"2024-09-02T10:22:41.252926+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.846770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"626'12650","reported_seq":12950,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.253019+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:22:41.253019+0000","last_peered":"2024-09-02T10:22:41.253019+0000","last_clean":"2024-09-02T10:22:41.253019+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:22:41.253019+0000","last_undegraded":"2024-09-02T10:22:41.253019+0000","last_fullsized":"2024-09-02T10:22:41.253019+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.913566+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12650,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1243,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.166513+0000","last_change":"2024-09-02T10:21:35.696117+0000","last_active":"2024-09-02T10:22:41.166513+0000","last_peered":"2024-09-02T10:22:41.166513+0000","last_clean":"2024-09-02T10:22:41.166513+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:22:41.166513+0000","last_undegraded":"2024-09-02T10:22:41.166513+0000","last_fullsized":"2024-09-02T10:22:41.166513+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:35.696044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.696044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:46.753483+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"626'12496","reported_seq":12760,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.167772+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:22:41.167772+0000","last_peered":"2024-09-02T10:22:41.167772+0000","last_clean":"2024-09-02T10:22:41.167772+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:22:41.167772+0000","last_undegraded":"2024-09-02T10:22:41.167772+0000","last_fullsized":"2024-09-02T10:22:41.167772+0000","mapping_epoch":533,"log_start":"553'2400","ondisk_log_start":"553'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":190,"reported_epoch":626,"state":"active+clean","last_fresh":"2024-09-02T10:22:39.159047+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:22:39.159047+0000","last_peered":"2024-09-02T10:22:39.159047+0000","last_clean":"2024-09-02T10:22:39.159047+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:22:39.159047+0000","last_undegraded":"2024-09-02T10:22:39.159047+0000","last_fullsized":"2024-09-02T10:22:39.159047+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"626'12520","reported_seq":12787,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252583+0000","last_change":"2024-09-02T10:20:21.937271+0000","last_active":"2024-09-02T10:22:41.252583+0000","last_peered":"2024-09-02T10:22:41.252583+0000","last_clean":"2024-09-02T10:22:41.252583+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:22:41.252583+0000","last_undegraded":"2024-09-02T10:22:41.252583+0000","last_fullsized":"2024-09-02T10:22:41.252583+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6863","last_scrub_stamp":"2024-09-02T10:20:21.937140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.937140+0000","objects_scrubbed":5699,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12520,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1023,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252739+0000","last_change":"2024-09-02T10:22:30.358040+0000","last_active":"2024-09-02T10:22:41.252739+0000","last_peered":"2024-09-02T10:22:41.252739+0000","last_clean":"2024-09-02T10:22:41.252739+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:22:41.252739+0000","last_undegraded":"2024-09-02T10:22:41.252739+0000","last_fullsized":"2024-09-02T10:22:41.252739+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:30.357970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:30.357970+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.221263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"625'12474","reported_seq":12734,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:38.333866+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:22:38.333866+0000","last_peered":"2024-09-02T10:22:38.333866+0000","last_clean":"2024-09-02T10:22:38.333866+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:22:38.333866+0000","last_undegraded":"2024-09-02T10:22:38.333866+0000","last_fullsized":"2024-09-02T10:22:38.333866+0000","mapping_epoch":533,"log_start":"552'2400","ondisk_log_start":"552'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6029312,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12474,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":403,"reported_epoch":625,"state":"active+clean","last_fresh":"2024-09-02T10:22:34.164507+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:22:34.164507+0000","last_peered":"2024-09-02T10:22:34.164507+0000","last_clean":"2024-09-02T10:22:34.164507+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:22:34.164507+0000","last_undegraded":"2024-09-02T10:22:34.164507+0000","last_fullsized":"2024-09-02T10:22:34.164507+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":33292312,"num_objects":509,"num_object_clones":0,"num_object_copies":1527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6964,"num_objects_misplaced":294,"num_objects_unfound":0,"num_objects_dirty":509,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200031,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20909,"num_bytes_recovered":1286144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":893071360,"data_stored":893059144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160731,"ondisk_log_size":160731,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383951,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742380,"kb_used_data":259988,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93002772,"statfs":{"total":95995035648,"available":95234838528,"internally_reserved":0,"allocated":266227712,"data_stored":265905436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":493943236},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612884,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":356892,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93388260,"statfs":{"total":95995035648,"available":95629578240,"internally_reserved":0,"allocated":864256,"data_stored":542191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841817,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545704,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199448,"statfs":{"total":95995035648,"available":95436234752,"internally_reserved":0,"allocated":872448,"data_stored":550172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070750,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":349900,"kb_used_data":11900,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93395252,"statfs":{"total":95995035648,"available":95636738048,"internally_reserved":0,"allocated":12185600,"data_stored":11871940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299684,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582000,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93163152,"statfs":{"total":95995035648,"available":95399067648,"internally_reserved":0,"allocated":1077248,"data_stored":751420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528615,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365548,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379604,"statfs":{"total":95995035648,"available":95620714496,"internally_reserved":0,"allocated":1200128,"data_stored":877852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724844,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1139956,"kb_used_data":527072,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92605196,"statfs":{"total":95995035648,"available":94827720704,"internally_reserved":0,"allocated":539721728,"data_stored":539391828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953777,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347420,"kb_used_data":13576,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93397732,"statfs":{"total":95995035648,"available":95639277568,"internally_reserved":0,"allocated":13901824,"data_stored":13583831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150006,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":639500,"kb_used_data":58556,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93105652,"statfs":{"total":95995035648,"available":95340187648,"internally_reserved":0,"allocated":59961344,"data_stored":59647684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":594868680},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411642,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":293232,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":93451920,"statfs":{"total":95995035648,"available":95694766080,"internally_reserved":0,"allocated":1077248,"data_stored":751420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":299170245},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575167,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":527568,"kb_used_data":22976,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93217584,"statfs":{"total":95995035648,"available":95454806016,"internally_reserved":0,"allocated":23527424,"data_stored":23209692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738694,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":591848,"kb_used_data":1172,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93153304,"statfs":{"total":95995035648,"available":95388983296,"internally_reserved":0,"allocated":1200128,"data_stored":877852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19533824,"data_stored":19529752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49414144,"data_stored":49414144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11534336,"data_stored":11534336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":538644480,"data_stored":538640408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8585216,"data_stored":8585216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":265355264,"data_stored":265355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:22:44.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:22:44.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:44.739 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:44.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:44.789 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:44.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:22:44.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:44 smithi162 ceph-mon[20279]: pgmap v1459: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 32 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 502 op/s; 6964/1533 objects degraded (454.273%); 294/1533 objects misplaced (19.178%); 168 KiB/s, 140 objects/s recovering 2024-09-02T10:22:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:44 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:44 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:44 smithi162 ceph-mon[20279]: from='client.17649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:44 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6964/1533 objects degraded (454.273%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:44 smithi162 ceph-mon[20279]: osdmap e628: 12 total, 12 up, 12 in 2024-09-02T10:22:44.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:44 smithi151 ceph-mon[16639]: pgmap v1459: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 32 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 502 op/s; 6964/1533 objects degraded (454.273%); 294/1533 objects misplaced (19.178%); 168 KiB/s, 140 objects/s recovering 2024-09-02T10:22:44.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:44 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:44.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:44 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:44.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:44 smithi151 ceph-mon[16639]: from='client.17649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:44.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:44 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6964/1533 objects degraded (454.273%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:44.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:44 smithi151 ceph-mon[16639]: osdmap e628: 12 total, 12 up, 12 in 2024-09-02T10:22:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:44.864 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:44.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:44.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:44.986 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:44.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:44 smithi187 ceph-mon[20072]: pgmap v1459: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 32 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 502 op/s; 6964/1533 objects degraded (454.273%); 294/1533 objects misplaced (19.178%); 168 KiB/s, 140 objects/s recovering 2024-09-02T10:22:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:44 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:44 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:44 smithi187 ceph-mon[20072]: from='client.17649 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:44 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6964/1533 objects degraded (454.273%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:44 smithi187 ceph-mon[20072]: osdmap e628: 12 total, 12 up, 12 in 2024-09-02T10:22:45.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:45.115 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:45.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:45.217 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:45.240 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:45.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:45.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:45.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:45.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:45.389 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:45.411 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:45.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:45.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:45.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:45.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:45.625 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:45.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:45.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:45.750 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:45.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:45.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:45.874 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:45.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:45.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:22:45.940 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:22:45.961 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:22:45.961 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:22:45.976 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:45.984 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:22:45.985 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:22:45.997 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:45.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:46.005 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:22:46.006 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:22:46.027 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:22:46.028 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:22:46.049 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:22:46.049 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:22:46.070 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:22:46.070 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:22:46.091 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:22:46.092 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:22:46.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:46.113 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:22:46.113 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:22:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:46.126 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:46.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:46.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:22:46.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:22:46.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:22:46.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:22:46.177 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:22:46.177 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:22:46.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:46.250 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:46.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:46.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:46.379 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:46.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:46.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:46.501 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:46.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:46.562 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:22:46.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:46.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:46.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:46.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:46.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:46.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:46.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:46 smithi162 ceph-mon[20279]: pgmap v1461: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 366 op/s; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 282 objects/s recovering 2024-09-02T10:22:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/506358117' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:46 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1599988569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:46.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:46.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:46.886 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:46.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:46 smithi151 ceph-mon[16639]: pgmap v1461: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 366 op/s; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 282 objects/s recovering 2024-09-02T10:22:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/506358117' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:46 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1599988569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:46.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:46 smithi187 ceph-mon[20072]: pgmap v1461: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 366 op/s; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 282 objects/s recovering 2024-09-02T10:22:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/506358117' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:46 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:22:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1599988569' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:47.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:47.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:47.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:47.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:47.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:47.137 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:47.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:47.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:47.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:47.263 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:47.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:47.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:47.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:47.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:47.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:47.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:47.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:47.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:47.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:47.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:47.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:47.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:47.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:47.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:47.789 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:47.789 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6964/9 objects degraded (77377.778%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1639,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":629,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":12,"osd_in_since":1725272451,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":43},{"state_name":"active+recovering+undersized+degraded+remapped","count":2},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":3,"data_bytes":590392,"bytes_used":6519078912,"bytes_avail":1145421348864,"bytes_total":1151940427776,"degraded_objects":6964,"degraded_total":9,"degraded_ratio":773.77777777777783,"misplaced_objects":294,"misplaced_total":9,"misplaced_ratio":32.666666666666664,"recovering_objects_per_sec":197,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1228,"num_bytes_recovered":0,"num_keys_recovered":0,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":278},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":37,"modified":"2024-09-02T10:21:41.376228+0000","services":{}},"progress_events":{}} 2024-09-02T10:22:47.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:47 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:47.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:47 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:47.789 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:47 smithi151 ceph-mon[16639]: osdmap e629: 12 total, 12 up, 12 in 2024-09-02T10:22:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:47 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:47 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:47 smithi162 ceph-mon[20279]: osdmap e629: 12 total, 12 up, 12 in 2024-09-02T10:22:47.858 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:47.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:47.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:47.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:47.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:47 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:47 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:22:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:47 smithi187 ceph-mon[20072]: osdmap e629: 12 total, 12 up, 12 in 2024-09-02T10:22:48.007 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:48.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:48.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:48.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:48.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:48.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:48.136 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:48.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:48.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:48.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:48.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:48.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:48.385 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:48.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:48.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:48.511 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:48.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:48.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:48.635 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:48.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:48.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:48.758 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:48.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:48.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:48.830 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:48 smithi151 ceph-mon[16639]: pgmap v1463: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 197 objects/s recovering 2024-09-02T10:22:48.830 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4048363008' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:22:48.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:48 smithi162 ceph-mon[20279]: pgmap v1463: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 197 objects/s recovering 2024-09-02T10:22:48.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4048363008' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:22:48.859 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:22:48.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:22:48.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:48.882 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:48.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:48.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:48 smithi187 ceph-mon[20072]: pgmap v1463: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 278 op/s; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 197 objects/s recovering 2024-09-02T10:22:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4048363008' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:22:49.009 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:49.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:49.010 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:49.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:22:49.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:49.125 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:49.134 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:49.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:49.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:49.258 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:49.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:49.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:49.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:22:49.337 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:22:49.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:49.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:49.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:49.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:49.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:49.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:49.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:49.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:49.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:49.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:49.739 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:49 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 6964/9 objects degraded (77377.778%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:49.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:49 smithi187 ceph-mon[20072]: osdmap e630: 12 total, 12 up, 12 in 2024-09-02T10:22:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:49.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:49.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:49.764 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:49.788 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:22:49.788 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:22:49.789 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:22:49.789 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:22:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:49 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 6964/9 objects degraded (77377.778%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:49.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:49 smithi162 ceph-mon[20279]: osdmap e630: 12 total, 12 up, 12 in 2024-09-02T10:22:49.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:49.869 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:49.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:49 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 6964/9 objects degraded (77377.778%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:22:49.869 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:49 smithi151 ceph-mon[16639]: osdmap e630: 12 total, 12 up, 12 in 2024-09-02T10:22:49.871 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:22:49.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:49.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:49.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:49.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:50.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:50.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:50.012 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:50.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:50.134 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:50.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:50.135 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:50.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:22:50.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:50.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:50.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:50.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:50.382 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:50.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:50.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:50.506 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:50.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:50.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:22:50.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:50.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:50.629 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:50.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:50.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:50.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:50.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:50.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:50.792 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:22:50.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:50 smithi151 ceph-mon[16639]: pgmap v1465: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 84 op/s; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 245 objects/s recovering 2024-09-02T10:22:50.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:50.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:50.837 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:50 smithi151 ceph-mon[16639]: from='client.17673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:50.838 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1465,"stamp":"2024-09-02T10:22:49.397767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6964,"num_objects_misplaced":294,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22387,"num_bytes_recovered":1287075424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163791,"ondisk_log_size":163791,"up":141,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":6366408,"kb_used_data":797140,"kb_used_omap":18,"kb_used_meta":5569005,"kb_avail":1118575416,"statfs":{"total":1151940427776,"available":1145421225984,"internally_reserved":0,"allocated":816271360,"data_stored":812312348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5702661435},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-33292288,"num_objects":-508,"num_object_clones":0,"num_object_copies":-1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":508,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1474,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001462"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":620,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868569+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:22:46.868569+0000","last_peered":"2024-09-02T10:22:46.868569+0000","last_clean":"2024-09-02T10:22:46.868569+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:22:46.868569+0000","last_undegraded":"2024-09-02T10:22:46.868569+0000","last_fullsized":"2024-09-02T10:22:46.868569+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":427,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868541+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:22:46.868541+0000","last_peered":"2024-09-02T10:22:46.868541+0000","last_clean":"2024-09-02T10:22:46.868541+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:22:46.868541+0000","last_undegraded":"2024-09-02T10:22:46.868541+0000","last_fullsized":"2024-09-02T10:22:46.868541+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":881,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.279477+0000","last_change":"2024-09-02T10:22:32.860224+0000","last_active":"2024-09-02T10:22:41.279477+0000","last_peered":"2024-09-02T10:22:41.279477+0000","last_clean":"2024-09-02T10:22:41.279477+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:22:41.279477+0000","last_undegraded":"2024-09-02T10:22:41.279477+0000","last_fullsized":"2024-09-02T10:22:41.279477+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:32.860180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:32.860180+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.373890+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":873,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.279480+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:22:41.279480+0000","last_peered":"2024-09-02T10:22:41.279480+0000","last_clean":"2024-09-02T10:22:41.279480+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:22:41.279480+0000","last_undegraded":"2024-09-02T10:22:41.279480+0000","last_fullsized":"2024-09-02T10:22:41.279480+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.838669+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":177,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252231+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:22:41.252231+0000","last_peered":"2024-09-02T10:22:41.252231+0000","last_clean":"2024-09-02T10:22:41.252231+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:22:41.252231+0000","last_undegraded":"2024-09-02T10:22:41.252231+0000","last_fullsized":"2024-09-02T10:22:41.252231+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":716,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.508811+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:22:44.508811+0000","last_peered":"2024-09-02T10:22:44.508811+0000","last_clean":"2024-09-02T10:22:44.508811+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:22:44.508811+0000","last_undegraded":"2024-09-02T10:22:44.508811+0000","last_fullsized":"2024-09-02T10:22:44.508811+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1253,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.786787+0000","last_change":"2024-09-02T10:20:52.113290+0000","last_active":"2024-09-02T10:22:46.786787+0000","last_peered":"2024-09-02T10:22:46.786787+0000","last_clean":"2024-09-02T10:22:46.786787+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:22:46.786787+0000","last_undegraded":"2024-09-02T10:22:46.786787+0000","last_fullsized":"2024-09-02T10:22:46.786787+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:29.189593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:29.189593+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":888,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.279536+0000","last_change":"2024-09-02T10:21:51.020121+0000","last_active":"2024-09-02T10:22:41.279536+0000","last_peered":"2024-09-02T10:22:41.279536+0000","last_clean":"2024-09-02T10:22:41.279536+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:22:41.279536+0000","last_undegraded":"2024-09-02T10:22:41.279536+0000","last_fullsized":"2024-09-02T10:22:41.279536+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:51.020075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:51.020075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.425292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":884,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868187+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:22:46.868187+0000","last_peered":"2024-09-02T10:22:46.868187+0000","last_clean":"2024-09-02T10:22:46.868187+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:22:46.868187+0000","last_undegraded":"2024-09-02T10:22:46.868187+0000","last_fullsized":"2024-09-02T10:22:46.868187+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":866,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.382013+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:22:44.382013+0000","last_peered":"2024-09-02T10:22:44.382013+0000","last_clean":"2024-09-02T10:22:44.382013+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:22:44.382013+0000","last_undegraded":"2024-09-02T10:22:44.382013+0000","last_fullsized":"2024-09-02T10:22:44.382013+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.685318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":112,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.584895+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:22:46.584895+0000","last_peered":"2024-09-02T10:22:46.584895+0000","last_clean":"2024-09-02T10:22:46.584895+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:22:46.584895+0000","last_undegraded":"2024-09-02T10:22:46.584895+0000","last_fullsized":"2024-09-02T10:22:46.584895+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1217,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868249+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:22:46.868249+0000","last_peered":"2024-09-02T10:22:46.868249+0000","last_clean":"2024-09-02T10:22:46.868249+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:22:46.868249+0000","last_undegraded":"2024-09-02T10:22:46.868249+0000","last_fullsized":"2024-09-02T10:22:46.868249+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1152,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.786850+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:22:46.786850+0000","last_peered":"2024-09-02T10:22:46.786850+0000","last_clean":"2024-09-02T10:22:46.786850+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:22:46.786850+0000","last_undegraded":"2024-09-02T10:22:46.786850+0000","last_fullsized":"2024-09-02T10:22:46.786850+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1131,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868562+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:22:46.868562+0000","last_peered":"2024-09-02T10:22:46.868562+0000","last_clean":"2024-09-02T10:22:46.868562+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:22:46.868562+0000","last_undegraded":"2024-09-02T10:22:46.868562+0000","last_fullsized":"2024-09-02T10:22:46.868562+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1122,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.786854+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:22:46.786854+0000","last_peered":"2024-09-02T10:22:46.786854+0000","last_clean":"2024-09-02T10:22:46.786854+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:22:46.786854+0000","last_undegraded":"2024-09-02T10:22:46.786854+0000","last_fullsized":"2024-09-02T10:22:46.786854+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"626'12394","reported_seq":22253,"reported_epoch":629,"state":"active+recovering+remapped","last_fresh":"2024-09-02T10:22:47.430223+0000","last_change":"2024-09-02T10:20:53.492211+0000","last_active":"2024-09-02T10:22:47.430223+0000","last_peered":"2024-09-02T10:22:47.430223+0000","last_clean":"2024-09-02T10:20:51.027001+0000","last_became_active":"2024-09-02T10:20:53.416507+0000","last_became_peered":"2024-09-02T10:20:53.416507+0000","last_unstale":"2024-09-02T10:22:47.430223+0000","last_undegraded":"2024-09-02T10:22:47.430223+0000","last_fullsized":"2024-09-02T10:22:47.430223+0000","mapping_epoch":586,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":294,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12394,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4797,"num_bytes_recovered":237961216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[5,10,9],"avail_no_missing":["5","9","10"],"object_location_counts":[{"shards":"5,9,10","objects":-294}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1216,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.786907+0000","last_change":"2024-09-02T10:20:52.111563+0000","last_active":"2024-09-02T10:22:46.786907+0000","last_peered":"2024-09-02T10:22:46.786907+0000","last_clean":"2024-09-02T10:22:46.786907+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:22:46.786907+0000","last_undegraded":"2024-09-02T10:22:46.786907+0000","last_fullsized":"2024-09-02T10:22:46.786907+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"626'12580","reported_seq":25640,"reported_epoch":627,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:44.541324+0000","last_change":"2024-09-02T10:20:53.422779+0000","last_active":"2024-09-02T10:22:44.541324+0000","last_peered":"2024-09-02T10:22:44.541324+0000","last_clean":"2024-09-02T10:20:51.098224+0000","last_became_active":"2024-09-02T10:20:53.370526+0000","last_became_peered":"2024-09-02T10:20:53.370526+0000","last_unstale":"2024-09-02T10:22:44.541324+0000","last_undegraded":"2024-09-02T10:20:53.137669+0000","last_fullsized":"2024-09-02T10:20:53.103100+0000","mapping_epoch":586,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6393,"num_bytes_recovered":374538240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-475}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1345,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.381687+0000","last_change":"2024-09-02T10:22:37.332903+0000","last_active":"2024-09-02T10:22:44.381687+0000","last_peered":"2024-09-02T10:22:44.381687+0000","last_clean":"2024-09-02T10:22:44.381687+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:22:44.381687+0000","last_undegraded":"2024-09-02T10:22:44.381687+0000","last_fullsized":"2024-09-02T10:22:44.381687+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:37.332859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:37.332859+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.675027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"626'12351","reported_seq":12616,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.786970+0000","last_change":"2024-09-02T10:20:27.715367+0000","last_active":"2024-09-02T10:22:46.786970+0000","last_peered":"2024-09-02T10:22:46.786970+0000","last_clean":"2024-09-02T10:22:46.786970+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:22:46.786970+0000","last_undegraded":"2024-09-02T10:22:46.786970+0000","last_fullsized":"2024-09-02T10:22:46.786970+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7181","last_scrub_stamp":"2024-09-02T10:20:27.715126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:27.715126+0000","objects_scrubbed":5211,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12351,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":982,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868557+0000","last_change":"2024-09-02T10:20:18.812280+0000","last_active":"2024-09-02T10:22:46.868557+0000","last_peered":"2024-09-02T10:22:46.868557+0000","last_clean":"2024-09-02T10:22:46.868557+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:22:46.868557+0000","last_undegraded":"2024-09-02T10:22:46.868557+0000","last_fullsized":"2024-09-02T10:22:46.868557+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.812212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.812212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"626'12382","reported_seq":12662,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.556373+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:22:46.556373+0000","last_peered":"2024-09-02T10:22:46.556373+0000","last_clean":"2024-09-02T10:22:46.556373+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:22:46.556373+0000","last_undegraded":"2024-09-02T10:22:46.556373+0000","last_fullsized":"2024-09-02T10:22:46.556373+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":387,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868729+0000","last_change":"2024-09-02T10:22:15.563077+0000","last_active":"2024-09-02T10:22:46.868729+0000","last_peered":"2024-09-02T10:22:46.868729+0000","last_clean":"2024-09-02T10:22:46.868729+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:22:46.868729+0000","last_undegraded":"2024-09-02T10:22:46.868729+0000","last_fullsized":"2024-09-02T10:22:46.868729+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:15.563022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:15.563022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:20.809417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"626'12606","reported_seq":12900,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.381868+0000","last_change":"2024-09-02T10:21:41.827351+0000","last_active":"2024-09-02T10:22:44.381868+0000","last_peered":"2024-09-02T10:22:44.381868+0000","last_clean":"2024-09-02T10:22:44.381868+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:22:44.381868+0000","last_undegraded":"2024-09-02T10:22:44.381868+0000","last_fullsized":"2024-09-02T10:22:44.381868+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10385","last_scrub_stamp":"2024-09-02T10:21:41.827172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:41.827172+0000","objects_scrubbed":2236,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:45.086417+0000","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1296,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868512+0000","last_change":"2024-09-02T10:22:34.647196+0000","last_active":"2024-09-02T10:22:46.868512+0000","last_peered":"2024-09-02T10:22:46.868512+0000","last_clean":"2024-09-02T10:22:46.868512+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:22:46.868512+0000","last_undegraded":"2024-09-02T10:22:46.868512+0000","last_fullsized":"2024-09-02T10:22:46.868512+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:34.647144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:34.647144+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:02.024298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"626'12544","reported_seq":26531,"reported_epoch":627,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:44.536046+0000","last_change":"2024-09-02T10:20:53.407005+0000","last_active":"2024-09-02T10:22:44.536046+0000","last_peered":"2024-09-02T10:22:44.536046+0000","last_clean":"2024-09-02T10:20:51.097394+0000","last_became_active":"2024-09-02T10:20:53.379154+0000","last_became_peered":"2024-09-02T10:20:53.379154+0000","last_unstale":"2024-09-02T10:22:44.536046+0000","last_undegraded":"2024-09-02T10:20:53.138449+0000","last_fullsized":"2024-09-02T10:20:53.102891+0000","mapping_epoch":586,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6856,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3],"avail_no_missing":["4","3","11"],"object_location_counts":[{"shards":"3,4","objects":-195}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":752,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.508910+0000","last_change":"2024-09-02T10:20:52.104138+0000","last_active":"2024-09-02T10:22:44.508910+0000","last_peered":"2024-09-02T10:22:44.508910+0000","last_clean":"2024-09-02T10:22:44.508910+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:22:44.508910+0000","last_undegraded":"2024-09-02T10:22:44.508910+0000","last_fullsized":"2024-09-02T10:22:44.508910+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"626'12380","reported_seq":12674,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.279477+0000","last_change":"2024-09-02T10:21:39.544819+0000","last_active":"2024-09-02T10:22:41.279477+0000","last_peered":"2024-09-02T10:22:41.279477+0000","last_clean":"2024-09-02T10:22:41.279477+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:22:41.279477+0000","last_undegraded":"2024-09-02T10:22:41.279477+0000","last_fullsized":"2024-09-02T10:22:41.279477+0000","mapping_epoch":533,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"603'10078","last_scrub_stamp":"2024-09-02T10:21:39.544630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.544630+0000","objects_scrubbed":2309,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.692572+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12380,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":716,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.508956+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:22:44.508956+0000","last_peered":"2024-09-02T10:22:44.508956+0000","last_clean":"2024-09-02T10:22:44.508956+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:22:44.508956+0000","last_undegraded":"2024-09-02T10:22:44.508956+0000","last_fullsized":"2024-09-02T10:22:44.508956+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"626'12678","reported_seq":13033,"reported_epoch":629,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:46.868048+0000","last_change":"2024-09-02T10:20:53.439960+0000","last_active":"2024-09-02T10:22:46.868048+0000","last_peered":"2024-09-02T10:22:46.868048+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:22:46.868048+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-4089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":486,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868256+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:22:46.868256+0000","last_peered":"2024-09-02T10:22:46.868256+0000","last_clean":"2024-09-02T10:22:46.868256+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:22:46.868256+0000","last_undegraded":"2024-09-02T10:22:46.868256+0000","last_fullsized":"2024-09-02T10:22:46.868256+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:16.363558+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"626'12542","reported_seq":17166,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.556278+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:22:46.556278+0000","last_peered":"2024-09-02T10:22:46.556278+0000","last_clean":"2024-09-02T10:22:46.556278+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:22:46.556278+0000","last_undegraded":"2024-09-02T10:22:46.556278+0000","last_fullsized":"2024-09-02T10:22:46.556278+0000","mapping_epoch":559,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12542,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":717,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.508903+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:22:44.508903+0000","last_peered":"2024-09-02T10:22:44.508903+0000","last_clean":"2024-09-02T10:22:44.508903+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:22:44.508903+0000","last_undegraded":"2024-09-02T10:22:44.508903+0000","last_fullsized":"2024-09-02T10:22:44.508903+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"626'12632","reported_seq":12931,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.381982+0000","last_change":"2024-09-02T10:21:39.988693+0000","last_active":"2024-09-02T10:22:44.381982+0000","last_peered":"2024-09-02T10:22:44.381982+0000","last_clean":"2024-09-02T10:22:44.381982+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:22:44.381982+0000","last_undegraded":"2024-09-02T10:22:44.381982+0000","last_fullsized":"2024-09-02T10:22:44.381982+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10274","last_scrub_stamp":"2024-09-02T10:21:39.988364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.988364+0000","objects_scrubbed":2383,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:50.357883+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":979,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.381834+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:22:44.381834+0000","last_peered":"2024-09-02T10:22:44.381834+0000","last_clean":"2024-09-02T10:22:44.381834+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:22:44.381834+0000","last_undegraded":"2024-09-02T10:22:44.381834+0000","last_fullsized":"2024-09-02T10:22:44.381834+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:54.988677+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"626'12586","reported_seq":12850,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868353+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:22:46.868353+0000","last_peered":"2024-09-02T10:22:46.868353+0000","last_clean":"2024-09-02T10:22:46.868353+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:22:46.868353+0000","last_undegraded":"2024-09-02T10:22:46.868353+0000","last_fullsized":"2024-09-02T10:22:46.868353+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12586,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1241,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868373+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:22:46.868373+0000","last_peered":"2024-09-02T10:22:46.868373+0000","last_clean":"2024-09-02T10:22:46.868373+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:22:46.868373+0000","last_undegraded":"2024-09-02T10:22:46.868373+0000","last_fullsized":"2024-09-02T10:22:46.868373+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"626'12632","reported_seq":17144,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.787522+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:22:46.787522+0000","last_peered":"2024-09-02T10:22:46.787522+0000","last_clean":"2024-09-02T10:22:46.787522+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:22:46.787522+0000","last_undegraded":"2024-09-02T10:22:46.787522+0000","last_fullsized":"2024-09-02T10:22:46.787522+0000","mapping_epoch":565,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":990,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.381797+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:22:44.381797+0000","last_peered":"2024-09-02T10:22:44.381797+0000","last_clean":"2024-09-02T10:22:44.381797+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:22:44.381797+0000","last_undegraded":"2024-09-02T10:22:44.381797+0000","last_fullsized":"2024-09-02T10:22:44.381797+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.846770+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"626'12650","reported_seq":12952,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.381749+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:22:44.381749+0000","last_peered":"2024-09-02T10:22:44.381749+0000","last_clean":"2024-09-02T10:22:44.381749+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:22:44.381749+0000","last_undegraded":"2024-09-02T10:22:44.381749+0000","last_fullsized":"2024-09-02T10:22:44.381749+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:47.913566+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12650,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1247,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868377+0000","last_change":"2024-09-02T10:21:35.696117+0000","last_active":"2024-09-02T10:22:46.868377+0000","last_peered":"2024-09-02T10:22:46.868377+0000","last_clean":"2024-09-02T10:22:46.868377+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:22:46.868377+0000","last_undegraded":"2024-09-02T10:22:46.868377+0000","last_fullsized":"2024-09-02T10:22:46.868377+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:35.696044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.696044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"626'12496","reported_seq":12764,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.868460+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:22:46.868460+0000","last_peered":"2024-09-02T10:22:46.868460+0000","last_clean":"2024-09-02T10:22:46.868460+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:22:46.868460+0000","last_undegraded":"2024-09-02T10:22:46.868460+0000","last_fullsized":"2024-09-02T10:22:46.868460+0000","mapping_epoch":533,"log_start":"553'2400","ondisk_log_start":"553'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":192,"reported_epoch":627,"state":"active+clean","last_fresh":"2024-09-02T10:22:41.252387+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:22:41.252387+0000","last_peered":"2024-09-02T10:22:41.252387+0000","last_clean":"2024-09-02T10:22:41.252387+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:22:41.252387+0000","last_undegraded":"2024-09-02T10:22:41.252387+0000","last_fullsized":"2024-09-02T10:22:41.252387+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"626'12520","reported_seq":12791,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.786838+0000","last_change":"2024-09-02T10:20:21.937271+0000","last_active":"2024-09-02T10:22:46.786838+0000","last_peered":"2024-09-02T10:22:46.786838+0000","last_clean":"2024-09-02T10:22:46.786838+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:22:46.786838+0000","last_undegraded":"2024-09-02T10:22:46.786838+0000","last_fullsized":"2024-09-02T10:22:46.786838+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6863","last_scrub_stamp":"2024-09-02T10:20:21.937140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.937140+0000","objects_scrubbed":5699,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12520,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1025,"reported_epoch":628,"state":"active+clean","last_fresh":"2024-09-02T10:22:44.382036+0000","last_change":"2024-09-02T10:22:30.358040+0000","last_active":"2024-09-02T10:22:44.382036+0000","last_peered":"2024-09-02T10:22:44.382036+0000","last_clean":"2024-09-02T10:22:44.382036+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:22:44.382036+0000","last_undegraded":"2024-09-02T10:22:44.382036+0000","last_fullsized":"2024-09-02T10:22:44.382036+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:30.357970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:30.357970+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.221263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"626'12566","reported_seq":12834,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.556246+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:22:46.556246+0000","last_peered":"2024-09-02T10:22:46.556246+0000","last_clean":"2024-09-02T10:22:46.556246+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:22:46.556246+0000","last_undegraded":"2024-09-02T10:22:46.556246+0000","last_fullsized":"2024-09-02T10:22:46.556246+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":411,"reported_epoch":629,"state":"active+clean","last_fresh":"2024-09-02T10:22:46.556457+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:22:46.556457+0000","last_peered":"2024-09-02T10:22:46.556457+0000","last_clean":"2024-09-02T10:22:46.556457+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:22:46.556457+0000","last_undegraded":"2024-09-02T10:22:46.556457+0000","last_fullsized":"2024-09-02T10:22:46.556457+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6964,"num_objects_misplaced":294,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22383,"num_bytes_recovered":1286144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":804007936,"data_stored":803995720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383952,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742388,"kb_used_data":260000,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93002764,"statfs":{"total":95995035648,"available":95234830336,"internally_reserved":0,"allocated":266240000,"data_stored":265913441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":493943236},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612885,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":356920,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93388232,"statfs":{"total":95995035648,"available":95629549568,"internally_reserved":0,"allocated":888832,"data_stored":558201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841818,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545728,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199424,"statfs":{"total":95995035648,"available":95436210176,"internally_reserved":0,"allocated":897024,"data_stored":566132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070751,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338856,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93406296,"statfs":{"total":95995035648,"available":95648047104,"internally_reserved":0,"allocated":872448,"data_stored":550172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299685,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582012,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93163140,"statfs":{"total":95995035648,"available":95399055360,"internally_reserved":0,"allocated":1089536,"data_stored":759425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528616,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365556,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379596,"statfs":{"total":95995035648,"available":95620706304,"internally_reserved":0,"allocated":1212416,"data_stored":885857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724845,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1139980,"kb_used_data":527096,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92605172,"statfs":{"total":95995035648,"available":94827696128,"internally_reserved":0,"allocated":539746304,"data_stored":539407788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953778,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334696,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410456,"statfs":{"total":95995035648,"available":95652306944,"internally_reserved":0,"allocated":872448,"data_stored":550172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150008,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569668,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175484,"statfs":{"total":95995035648,"available":95411695616,"internally_reserved":0,"allocated":1224704,"data_stored":893812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411643,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":289096,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":93456056,"statfs":{"total":95995035648,"available":95699001344,"internally_reserved":0,"allocated":1101824,"data_stored":767380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575169,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509636,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235516,"statfs":{"total":95995035648,"available":95473168384,"internally_reserved":0,"allocated":901120,"data_stored":566156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738695,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":591872,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93153280,"statfs":{"total":95995035648,"available":95388958720,"internally_reserved":0,"allocated":1224704,"data_stored":893812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":538644480,"data_stored":538640408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":265355264,"data_stored":265355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:22:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:50 smithi162 ceph-mon[20279]: pgmap v1465: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 84 op/s; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 245 objects/s recovering 2024-09-02T10:22:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:50 smithi162 ceph-mon[20279]: from='client.17673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:50.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:50.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:50.877 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:50.902 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:50 smithi187 ceph-mon[20072]: pgmap v1465: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 84 op/s; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 245 objects/s recovering 2024-09-02T10:22:50.902 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:50.902 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:50.902 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:50 smithi187 ceph-mon[20072]: from='client.17673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:50.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:50.999 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:51.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:51.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:51.051 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:22:51.051 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:22:51.073 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:22:51.073 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:22:51.095 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:22:51.096 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:22:51.101 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:51.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:22:51.121 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:22:51.122 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:51.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:51.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:51.149 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:22:51.150 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:22:51.172 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:22:51.172 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:22:51.193 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:22:51.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:22:51.219 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:22:51.220 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:22:51.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:51.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:22:51.243 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:22:51.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:51.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:51.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:51.266 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:22:51.267 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:22:51.272 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:22:51.273 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:22:51.289 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:22:51.290 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:22:51.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:22:51.313 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:22:51.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:51.368 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:51.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:51.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:51.493 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:51.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:51.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:51.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:51.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:51 smithi151 ceph-mon[16639]: osdmap e631: 12 total, 12 up, 12 in 2024-09-02T10:22:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/159249903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:51.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:51.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:51.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:51.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:51.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:51 smithi162 ceph-mon[20279]: osdmap e631: 12 total, 12 up, 12 in 2024-09-02T10:22:51.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/159249903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:51.867 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:51.868 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:51 smithi187 ceph-mon[20072]: osdmap e631: 12 total, 12 up, 12 in 2024-09-02T10:22:51.868 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/159249903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:51.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:51.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:51.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:22:51.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:51.991 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:51.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:51.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:52.093 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:52.116 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:52.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:52.134 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:22:52.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:52.238 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:52.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:52.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:52.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:52.364 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:52.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:52.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:52.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:52.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:52.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:52.570 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:22:52.570 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:22:52.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:52.612 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:52.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:52.613 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:52.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:52.736 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:52.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:52.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:52.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:52.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:52 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:52.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:52 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:52.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:52 smithi162 ceph-mon[20279]: pgmap v1467: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 316 objects/s recovering 2024-09-02T10:22:52.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:52 smithi162 ceph-mon[20279]: osdmap e632: 12 total, 12 up, 12 in 2024-09-02T10:22:52.860 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:52.861 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:52 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:52 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:52 smithi151 ceph-mon[16639]: pgmap v1467: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 316 objects/s recovering 2024-09-02T10:22:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:52 smithi151 ceph-mon[16639]: osdmap e632: 12 total, 12 up, 12 in 2024-09-02T10:22:52.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:52.975 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:52 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:52.975 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:52 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:52.975 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:52 smithi187 ceph-mon[20072]: pgmap v1467: 47 pgs: 1 active+recovering+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 316 objects/s recovering 2024-09-02T10:22:52.975 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:52 smithi187 ceph-mon[20072]: osdmap e632: 12 total, 12 up, 12 in 2024-09-02T10:22:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:52.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:52.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:53.087 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:53.109 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:53.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:53.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:53.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:53.234 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:53.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:53.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:22:53.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:53.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:53.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:53.366 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:53.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:53.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:53.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:53.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:53.548 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:22:53.583 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:22:53.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:53.614 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:53.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:53.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:53.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:53.738 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:53.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:53.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:53.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:22:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:53 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:53 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:53 smithi162 ceph-mon[20279]: osdmap e633: 12 total, 12 up, 12 in 2024-09-02T10:22:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1974367339' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:53.861 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:53.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:53.862 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:53.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:53 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:53.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:53 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:53.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:53 smithi151 ceph-mon[16639]: osdmap e633: 12 total, 12 up, 12 in 2024-09-02T10:22:53.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1974367339' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:53.885 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:53 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:53.885 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:53 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:53.886 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:53 smithi187 ceph-mon[20072]: osdmap e633: 12 total, 12 up, 12 in 2024-09-02T10:22:53.886 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1974367339' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:22:53.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:53.985 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:53.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:53.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:54.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:54.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:54.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:54.112 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:22:54.112 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:22:54.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:22:54.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:54.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:54.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:54.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:54.373 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:54.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:54.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:54.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:54.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:54.499 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:54.600 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:54.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:54.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:54.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:54.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:54.745 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:54.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:54.749 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:54 smithi187 ceph-mon[20072]: pgmap v1470: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 508 objects/s recovering 2024-09-02T10:22:54.749 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:54 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:54.749 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1974367339' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:22:54.750 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:54 smithi187 ceph-mon[20072]: osdmap e634: 12 total, 12 up, 12 in 2024-09-02T10:22:54.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:22:54.799 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:54 smithi162 ceph-mon[20279]: pgmap v1470: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 508 objects/s recovering 2024-09-02T10:22:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:54 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1974367339' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:22:54.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:54 smithi162 ceph-mon[20279]: osdmap e634: 12 total, 12 up, 12 in 2024-09-02T10:22:54.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:54.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:54.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:54.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:54 smithi151 ceph-mon[16639]: pgmap v1470: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 508 objects/s recovering 2024-09-02T10:22:54.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:54 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:22:54.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1974367339' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:22:54.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:54 smithi151 ceph-mon[16639]: osdmap e634: 12 total, 12 up, 12 in 2024-09-02T10:22:54.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:54.995 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:54.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:55.019 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:22:55.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:55.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:55.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:55.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:55.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:55.244 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:55.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:55.346 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:55.368 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:55.369 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:55.402 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:55.470 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:22:55.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:55.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:22:55.511 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:22:55.511 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:22:55.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:55.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:55.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:55.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:55.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:55.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:55 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:22:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:55 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:22:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:55 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:22:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:55 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:22:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:55 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3740870716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:55 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:22:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:55 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:22:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:55 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:22:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:55 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:22:55.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:55 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3740870716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:55.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:22:55.870 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:55.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:55.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:22:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:55 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:22:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:55 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:22:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:55 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:22:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:55 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:22:55.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:55 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3740870716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:22:55.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:55.994 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:55.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:56.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:56.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:56.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:56.221 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:56.240 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:22:56.240 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:22:56.243 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:56.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:56.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:56.272 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:22:56.272 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:22:56.285 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:22:56.298 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:22:56.298 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:22:56.320 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:22:56.347 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:22:56.347 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:22:56.347 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:22:56.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:56.366 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:22:56.366 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:22:56.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:56.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:56.390 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:22:56.390 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:22:56.412 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:22:56.412 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:22:56.436 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:22:56.436 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:22:56.459 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:22:56.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:22:56.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:56.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:22:56.482 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:22:56.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:56.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:56.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:56.510 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:22:56.510 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:22:56.595 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:22:56.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:56.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:56.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:56.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:56.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:56.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:22:56.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:56.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:56.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:56 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:22:56.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:56 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:22:56.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:56 smithi151 ceph-mon[16639]: pgmap v1472: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 372 objects/s recovering 2024-09-02T10:22:56.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:56 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:22:56.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:56 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:22:56.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:56 smithi151 ceph-mon[16639]: osdmap e635: 12 total, 12 up, 12 in 2024-09-02T10:22:56.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:56.849 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:56 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:22:56.849 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:56 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:22:56.849 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:56 smithi187 ceph-mon[20072]: pgmap v1472: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 372 objects/s recovering 2024-09-02T10:22:56.849 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:56 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:22:56.849 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:56 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:22:56.849 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:56 smithi187 ceph-mon[20072]: osdmap e635: 12 total, 12 up, 12 in 2024-09-02T10:22:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:56.867 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:56.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:56.969 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:56.987 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:22:56.988 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:22:56.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:56.993 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:57.072 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:22:57.072 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:22:57.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:22:57.072 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:22:57.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.072 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGRecoveryDeleteReply(4.6 e393,385 4:606eb1b1:::benchmark_data_smithi187_76542_object7486:head,380'3976)", 2024-09-02T10:22:57.072 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:20.323853+0000", 2024-09-02T10:22:57.073 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 576.745804658, 2024-09-02T10:22:57.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33973209100000001, 2024-09-02T10:22:57.073 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.073 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.073 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:22:57.073 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.323853+0000", 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.323853+0000", 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.323854+0000", 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9500000000000002e-07 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.323859+0000", 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2279999999999998e-06 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.663347+0000", 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33948819099999999 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.663356+0000", 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6240000000000008e-06 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.663408+0000", 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2234999999999997e-05 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:20.663585+0000", 2024-09-02T10:22:57.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017721800000000001 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:50217 5.1 e455/431 5:84aaa397:::benchmark_data_smithi187_89405_object50216:head v 455'3114, mlcod=455'3111)", 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:23.792228+0000", 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 453.27742931500001, 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.382032398, 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.077 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:23.792228+0000", 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:23.792228+0000", 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:23.792230+0000", 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.393e-06 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:23.792299+0000", 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8938999999999998e-05 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.172069+0000", 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.37976990700000002 2024-09-02T10:22:57.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.172075+0000", 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1090000000000003e-06 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.172418+0000", 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00034368799999999998 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.172439+0000", 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0704e-05 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.174214+0000", 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00177505 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.174261+0000", 2024-09-02T10:22:57.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6607999999999998e-05 2024-09-02T10:22:57.082 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.082 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.082 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.082 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:126036 5.e 5:76dd882f:::benchmark_data_smithi187_89405_object22431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-02T10:22:57.083 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.216999+0000", 2024-09-02T10:22:57.083 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 375.85265877500001, 2024-09-02T10:22:57.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32619644800000003, 2024-09-02T10:22:57.083 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.083 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.083 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.083 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:22:57.083 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:22:57.083 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:22:57.083 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126036 2024-09-02T10:22:57.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.084 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.216999+0000", 2024-09-02T10:22:57.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.216999+0000", 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.217000+0000", 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1299999999999998e-07 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.217004+0000", 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3189999999999996e-06 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.217017+0000", 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3086e-05 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.217024+0000", 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8889999999999996e-06 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.218835+0000", 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0018111360000000001 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.218937+0000", 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101393 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,8", 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.219003+0000", 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6853999999999994e-05 2024-09-02T10:22:57.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.221568+0000", 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002564481 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.221904+0000", 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00033607100000000002 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.543122+0000", 2024-09-02T10:22:57.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32121784399999997 2024-09-02T10:22:57.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.543155+0000", 2024-09-02T10:22:57.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3068999999999998e-05 2024-09-02T10:22:57.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.543195+0000", 2024-09-02T10:22:57.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0392999999999998e-05 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:205325 5.e 5:726c3a45:::benchmark_data_smithi187_89405_object101720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:17.617930+0000", 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 279.451727285, 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.38179982800000001, 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:22:57.091 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 205325 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617930+0000", 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617930+0000", 2024-09-02T10:22:57.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617931+0000", 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1999999999999998e-07 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617936+0000", 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3499999999999999e-06 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617946+0000", 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0223e-05 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617951+0000", 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3159999999999996e-06 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617992+0000", 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0435000000000002e-05 2024-09-02T10:22:57.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.618066+0000", 2024-09-02T10:22:57.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4764000000000004e-05 2024-09-02T10:22:57.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,8", 2024-09-02T10:22:57.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.618133+0000", 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6779999999999994e-05 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.622877+0000", 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047439980000000001 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.623239+0000", 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036236100000000001 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.999558+0000", 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.37631872100000002 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.999642+0000", 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4165000000000005e-05 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.999730+0000", 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7794999999999995e-05 2024-09-02T10:22:57.097 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:100226 6.1 e572/533 6:80d0f12d:::benchmark_data_smithi187_102448_object100225:head v 572'6190, mlcod=572'6187)", 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 166.16353598800001, 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63855390000000001, 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906123+0000", 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.269e-06 2024-09-02T10:22:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906147+0000", 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3620000000000001e-05 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.541864+0000", 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.635717054 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.541879+0000", 2024-09-02T10:22:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5546e-05 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.542040+0000", 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000161022 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.542059+0000", 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8697000000000001e-05 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544638+0000", 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025792850000000002 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544676+0000", 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7407000000000003e-05 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:157058 6.5 e597/533 6:a63ec91b:::benchmark_data_smithi187_102448_object56786:head v 597'9839, mlcod=597'9836)", 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:21:27.330318+0000", 2024-09-02T10:22:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 89.739339883, 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19032256, 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.330318+0000", 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.330318+0000", 2024-09-02T10:22:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.330318+0000", 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1999999999999999e-07 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.330324+0000", 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1030000000000001e-06 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.517156+0000", 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18683244900000001 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.517162+0000", 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.297e-06 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.517232+0000", 2024-09-02T10:22:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9531999999999998e-05 2024-09-02T10:22:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.517246+0000", 2024-09-02T10:22:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3764000000000001e-05 2024-09-02T10:22:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.520611+0000", 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033651459999999998 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.520640+0000", 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9549000000000001e-05 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:157286 6.5 e598/533 6:aeee93d8:::benchmark_data_smithi187_102448_object57014:head v 598'9853, mlcod=598'9850)", 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 89.119643081999996, 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29959694599999998, 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2900000000000002e-07 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950022+0000", 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8739999999999997e-06 2024-09-02T10:22:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245512+0000", 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29548945300000001 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245524+0000", 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2347000000000001e-05 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245666+0000", 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014145800000000001 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245702+0000", 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6853000000000001e-05 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.249576+0000", 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003873385 2024-09-02T10:22:57.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.249611+0000", 2024-09-02T10:22:57.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5646999999999999e-05 2024-09-02T10:22:57.112 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.112 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.112 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.112 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.112 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:174779 6.1 e609/533 6:81b1e1cd:::benchmark_data_smithi187_102448_object74507:head v 609'10785, mlcod=609'10782)", 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:21:56.468798+0000", 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 60.600859751999998, 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14462064999999999, 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.468798+0000", 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.468798+0000", 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.468799+0000", 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4900000000000005e-07 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.468804+0000", 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.818e-06 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.608450+0000", 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.139645936 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.608455+0000", 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6639999999999997e-06 2024-09-02T10:22:57.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.608527+0000", 2024-09-02T10:22:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1967999999999999e-05 2024-09-02T10:22:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.608546+0000", 2024-09-02T10:22:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8525e-05 2024-09-02T10:22:57.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.613381+0000", 2024-09-02T10:22:57.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048350789999999999 2024-09-02T10:22:57.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.613419+0000", 2024-09-02T10:22:57.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7910999999999997e-05 2024-09-02T10:22:57.118 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.118 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.118 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.119 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:183270 6.1 e619/533 6:867e70f2:::benchmark_data_smithi187_102448_object82998:head v 619'11309, mlcod=619'11306)", 2024-09-02T10:22:57.119 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:11.229360+0000", 2024-09-02T10:22:57.119 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 45.840297667999998, 2024-09-02T10:22:57.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.142139139, 2024-09-02T10:22:57.119 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.119 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.229360+0000", 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.229360+0000", 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.229361+0000", 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7199999999999997e-07 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.229367+0000", 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.241e-06 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.367448+0000", 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.138081221 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.367455+0000", 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7089999999999996e-06 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.367513+0000", 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8069000000000003e-05 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.367539+0000", 2024-09-02T10:22:57.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.546e-05 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.371452+0000", 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039135130000000004 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.371499+0000", 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7054e-05 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:183295 6.1 e619/533 6:84afaa1d:::benchmark_data_smithi187_102448_object83023:head v 619'11310, mlcod=619'11307)", 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:11.278713+0000", 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 45.790944291000002, 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12982100699999999, 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.278713+0000", 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.278713+0000", 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.278714+0000", 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4600000000000004e-07 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.278720+0000", 2024-09-02T10:22:57.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7749999999999998e-06 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.404683+0000", 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12596357499999999 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.404689+0000", 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2170000000000002e-06 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.404737+0000", 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8763000000000001e-05 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.404758+0000", 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0159000000000001e-05 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.408500+0000", 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037429170000000001 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.408534+0000", 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3855e-05 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:183759 6.5 e619/533 6:a7e733ea:::benchmark_data_smithi187_102448_object83487:head v 619'11546, mlcod=619'11543)", 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:12.122288+0000", 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 44.947369631999997, 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13124562200000001, 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.128 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.122288+0000", 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.122288+0000", 2024-09-02T10:22:57.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.122289+0000", 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3499999999999995e-07 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.122294+0000", 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7139999999999998e-06 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.249802+0000", 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.127507173 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.249807+0000", 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8719999999999999e-06 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.249858+0000", 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0392000000000001e-05 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.249873+0000", 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4717e-05 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.253510+0000", 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036375639999999998 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.253534+0000", 2024-09-02T10:22:57.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3455000000000002e-05 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:183760 6.5 e619/533 6:a105bd12:::benchmark_data_smithi187_102448_object83488:head v 619'11547, mlcod=619'11543)", 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:12.127198+0000", 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 44.942459329000002, 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13583553700000001, 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.127198+0000", 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.127198+0000", 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.127200+0000", 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5e-06 2024-09-02T10:22:57.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.127210+0000", 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0349999999999999e-05 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.259243+0000", 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13203271499999999 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.259248+0000", 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4750000000000001e-06 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.259297+0000", 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9109e-05 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.259320+0000", 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2501999999999999e-05 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.262999+0000", 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036790249999999998 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.263034+0000", 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4860999999999997e-05 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.137 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:192975 6.9 e622/533 6:97bbf3ca:::benchmark_data_smithi187_102448_object92703:head v 622'12018, mlcod=622'12015)", 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:27.213777+0000", 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 29.855880963000001, 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.097081619999999993, 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.213777+0000", 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.213777+0000", 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.213777+0000", 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.06e-07 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.213782+0000", 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8520000000000003e-06 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.306074+0000", 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.092291746999999993 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.306083+0000", 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7690000000000002e-06 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.306136+0000", 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2976999999999998e-05 2024-09-02T10:22:57.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.306151+0000", 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5177999999999999e-05 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.310805+0000", 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046540729999999999 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.310858+0000", 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3318e-05 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:193188 6.9 e622/533 6:953b0875:::benchmark_data_smithi187_102448_object92916:head v 622'12029, mlcod=622'12025)", 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:27.568665+0000", 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 29.500992154999999, 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.114332931, 2024-09-02T10:22:57.142 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.568665+0000", 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.568665+0000", 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.568666+0000", 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4399999999999999e-07 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.568671+0000", 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7169999999999999e-06 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.679184+0000", 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.110513373 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.679191+0000", 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8850000000000001e-06 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.679244+0000", 2024-09-02T10:22:57.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3043e-05 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.679265+0000", 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0835e-05 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.682965+0000", 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037002509999999999 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:27.682998+0000", 2024-09-02T10:22:57.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3083000000000002e-05 2024-09-02T10:22:57.147 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.147 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.147 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.147 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:197068 6.1 e625/533 6:8571fbf9:::benchmark_data_smithi187_102448_object96796:head v 625'12147, mlcod=625'12144)", 2024-09-02T10:22:57.147 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:34.998371+0000", 2024-09-02T10:22:57.147 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 22.071286398000002, 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.066250399000000001, 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:34.998371+0000", 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:34.998371+0000", 2024-09-02T10:22:57.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:34.998372+0000", 2024-09-02T10:22:57.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4900000000000005e-07 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:34.998377+0000", 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2909999999999998e-06 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.059004+0000", 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.060626449999999998 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.059011+0000", 2024-09-02T10:22:57.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2810000000000003e-06 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.059061+0000", 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9727e-05 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.059080+0000", 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.912e-05 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.151 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.064589+0000", 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.005509234 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.064622+0000", 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2447000000000003e-05 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.152 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:197167 6.1 e625/533 6:8aee9ed8:::benchmark_data_smithi187_102448_object96895:head v 625'12152, mlcod=625'12149)", 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:35.137547+0000", 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 21.932110145999999, 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.084897563999999995, 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.137547+0000", 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.137547+0000", 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.137548+0000", 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.06e-07 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.137554+0000", 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6500000000000001e-06 2024-09-02T10:22:57.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.213256+0000", 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.075702581000000005 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.213266+0000", 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5040000000000008e-06 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.213333+0000", 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7257999999999999e-05 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.213357+0000", 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3337e-05 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.222352+0000", 2024-09-02T10:22:57.156 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0089955950000000003 2024-09-02T10:22:57.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.157 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.157 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:35.222445+0000", 2024-09-02T10:22:57.157 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2932999999999999e-05 2024-09-02T10:22:57.157 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.157 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.157 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.157 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.157 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:199541 6.9 e625/533 6:99e7e980:::benchmark_data_smithi187_102448_object99269:head v 625'12423, mlcod=625'12420)", 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:39.132919+0000", 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 17.936738221999999, 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.133414848, 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.158 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.132919+0000", 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.132919+0000", 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.132920+0000", 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5700000000000001e-07 2024-09-02T10:22:57.159 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.132926+0000", 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3310000000000003e-06 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.261661+0000", 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12873506200000001 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.160 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.261668+0000", 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5580000000000003e-06 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.261767+0000", 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9121000000000002e-05 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.261791+0000", 2024-09-02T10:22:57.161 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3553999999999998e-05 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.266292+0000", 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045013040000000002 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.266334+0000", 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2061000000000003e-05 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.162 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:199661 6.9 e626/533 6:9e7d8302:::benchmark_data_smithi187_102448_object99389:head v 626'12429, mlcod=626'12426)", 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:39.328057+0000", 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 17.741600629000001, 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.087408983999999995, 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.328057+0000", 2024-09-02T10:22:57.163 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.328057+0000", 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.328058+0000", 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9800000000000002e-07 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.164 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:22:57.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.328063+0000", 2024-09-02T10:22:57.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0930000000000002e-06 2024-09-02T10:22:57.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.165 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.165 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:22:57.165 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.411781+0000", 2024-09-02T10:22:57.165 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083717937000000006 2024-09-02T10:22:57.165 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.166 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.166 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:22:57.166 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.411786+0000", 2024-09-02T10:22:57.166 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8380000000000001e-06 2024-09-02T10:22:57.166 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:22:57.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.411869+0000", 2024-09-02T10:22:57.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2968999999999998e-05 2024-09-02T10:22:57.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.167 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.167 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:22:57.167 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.411884+0000", 2024-09-02T10:22:57.167 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5478999999999999e-05 2024-09-02T10:22:57.167 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.415437+0000", 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035524329999999998 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.415466+0000", 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9337000000000001e-05 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.168 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:22:57.169 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:22:57.169 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:22:57.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:22:57.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:56 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:22:57.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:56 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:22:57.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:56 smithi162 ceph-mon[20279]: pgmap v1472: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+remapped, 2 active+recovering+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 294/9 objects misplaced (3266.667%); 0 B/s, 372 objects/s recovering 2024-09-02T10:22:57.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:56 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:22:57.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:56 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:22:57.170 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:56 smithi162 ceph-mon[20279]: osdmap e635: 12 total, 12 up, 12 in 2024-09-02T10:22:57.190 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:57.191 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:22:57.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:57.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:57.315 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:57.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:57.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:57.439 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:57.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:57.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:22:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:57.562 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:57.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:22:57.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:57.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:57.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:57.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:22:57.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:57.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:57.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:22:57.884 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:57 smithi187 ceph-mon[20072]: osdmap e636: 12 total, 12 up, 12 in 2024-09-02T10:22:57.884 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:57 smithi187 ceph-mon[20072]: from='client.17697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:57.884 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:57 smithi187 ceph-mon[20072]: osdmap e637: 12 total, 12 up, 12 in 2024-09-02T10:22:57.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:22:57.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:57.933 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:57.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:57.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:57 smithi151 ceph-mon[16639]: osdmap e636: 12 total, 12 up, 12 in 2024-09-02T10:22:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:57 smithi151 ceph-mon[16639]: from='client.17697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:57 smithi151 ceph-mon[16639]: osdmap e637: 12 total, 12 up, 12 in 2024-09-02T10:22:58.035 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:58.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:58.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:58.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:57 smithi162 ceph-mon[20279]: osdmap e636: 12 total, 12 up, 12 in 2024-09-02T10:22:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:57 smithi162 ceph-mon[20279]: from='client.17697 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:22:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:57 smithi162 ceph-mon[20279]: osdmap e637: 12 total, 12 up, 12 in 2024-09-02T10:22:58.152 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:22:58.158 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:58.181 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:58.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:58.283 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:58.306 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:58.307 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:58.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:58.432 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:58.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:58.534 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:58.549 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:58.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:58.556 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:58.583 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1474,"stamp":"2024-09-02T10:22:56.592840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6964,"num_objects_misplaced":294,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25505,"num_bytes_recovered":1287075424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163791,"ondisk_log_size":163791,"up":141,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":6385064,"kb_used_data":797728,"kb_used_omap":18,"kb_used_meta":5587053,"kb_avail":1118556760,"statfs":{"total":1151940427776,"available":1145402122240,"internally_reserved":0,"allocated":816873472,"data_stored":812703589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5721142587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1801,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.017759"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":626,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.781350+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:22:51.781350+0000","last_peered":"2024-09-02T10:22:51.781350+0000","last_clean":"2024-09-02T10:22:51.781350+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:22:51.781350+0000","last_undegraded":"2024-09-02T10:22:51.781350+0000","last_fullsized":"2024-09-02T10:22:51.781350+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":437,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:53.585143+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:22:53.585143+0000","last_peered":"2024-09-02T10:22:53.585143+0000","last_clean":"2024-09-02T10:22:53.585143+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:22:53.585143+0000","last_undegraded":"2024-09-02T10:22:53.585143+0000","last_fullsized":"2024-09-02T10:22:53.585143+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":895,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:53.584426+0000","last_change":"2024-09-02T10:22:32.860224+0000","last_active":"2024-09-02T10:22:53.584426+0000","last_peered":"2024-09-02T10:22:53.584426+0000","last_clean":"2024-09-02T10:22:53.584426+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:22:53.584426+0000","last_undegraded":"2024-09-02T10:22:53.584426+0000","last_fullsized":"2024-09-02T10:22:53.584426+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:32.860180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:32.860180+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.373890+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":887,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:53.584317+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:22:53.584317+0000","last_peered":"2024-09-02T10:22:53.584317+0000","last_clean":"2024-09-02T10:22:53.584317+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:22:53.584317+0000","last_undegraded":"2024-09-02T10:22:53.584317+0000","last_fullsized":"2024-09-02T10:22:53.584317+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.838669+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":191,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:54.113174+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:22:54.113174+0000","last_peered":"2024-09-02T10:22:54.113174+0000","last_clean":"2024-09-02T10:22:54.113174+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:22:54.113174+0000","last_undegraded":"2024-09-02T10:22:54.113174+0000","last_fullsized":"2024-09-02T10:22:54.113174+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":724,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.781102+0000","last_change":"2024-09-02T10:19:45.629126+0000","last_active":"2024-09-02T10:22:51.781102+0000","last_peered":"2024-09-02T10:22:51.781102+0000","last_clean":"2024-09-02T10:22:51.781102+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:22:51.781102+0000","last_undegraded":"2024-09-02T10:22:51.781102+0000","last_fullsized":"2024-09-02T10:22:51.781102+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:45.629080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:45.629080+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1259,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.595849+0000","last_change":"2024-09-02T10:20:52.113290+0000","last_active":"2024-09-02T10:22:51.595849+0000","last_peered":"2024-09-02T10:22:51.595849+0000","last_clean":"2024-09-02T10:22:51.595849+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:22:51.595849+0000","last_undegraded":"2024-09-02T10:22:51.595849+0000","last_fullsized":"2024-09-02T10:22:51.595849+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:29.189593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:29.189593+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":902,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:53.584363+0000","last_change":"2024-09-02T10:21:51.020121+0000","last_active":"2024-09-02T10:22:53.584363+0000","last_peered":"2024-09-02T10:22:53.584363+0000","last_clean":"2024-09-02T10:22:53.584363+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:22:53.584363+0000","last_undegraded":"2024-09-02T10:22:53.584363+0000","last_fullsized":"2024-09-02T10:22:53.584363+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:51.020075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:51.020075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.425292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":894,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:53.584522+0000","last_change":"2024-09-02T10:19:49.116192+0000","last_active":"2024-09-02T10:22:53.584522+0000","last_peered":"2024-09-02T10:22:53.584522+0000","last_clean":"2024-09-02T10:22:53.584522+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:22:53.584522+0000","last_undegraded":"2024-09-02T10:22:53.584522+0000","last_fullsized":"2024-09-02T10:22:53.584522+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:49.116020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:49.116020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":880,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-09-02T10:22:55.630424+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:22:55.630424+0000","last_peered":"2024-09-02T10:22:55.630424+0000","last_clean":"2024-09-02T10:22:55.630424+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:22:55.630424+0000","last_undegraded":"2024-09-02T10:22:55.630424+0000","last_fullsized":"2024-09-02T10:22:55.630424+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.685318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":118,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.599401+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:22:51.599401+0000","last_peered":"2024-09-02T10:22:51.599401+0000","last_clean":"2024-09-02T10:22:51.599401+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:22:51.599401+0000","last_undegraded":"2024-09-02T10:22:51.599401+0000","last_fullsized":"2024-09-02T10:22:51.599401+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1227,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:53.584474+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:22:53.584474+0000","last_peered":"2024-09-02T10:22:53.584474+0000","last_clean":"2024-09-02T10:22:53.584474+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:22:53.584474+0000","last_undegraded":"2024-09-02T10:22:53.584474+0000","last_fullsized":"2024-09-02T10:22:53.584474+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1158,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.595885+0000","last_change":"2024-09-02T10:19:54.188305+0000","last_active":"2024-09-02T10:22:51.595885+0000","last_peered":"2024-09-02T10:22:51.595885+0000","last_clean":"2024-09-02T10:22:51.595885+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:22:51.595885+0000","last_undegraded":"2024-09-02T10:22:51.595885+0000","last_fullsized":"2024-09-02T10:22:51.595885+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:54.188251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.188251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1137,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.909610+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:22:51.909610+0000","last_peered":"2024-09-02T10:22:51.909610+0000","last_clean":"2024-09-02T10:22:51.909610+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:22:51.909610+0000","last_undegraded":"2024-09-02T10:22:51.909610+0000","last_fullsized":"2024-09-02T10:22:51.909610+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1128,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.595902+0000","last_change":"2024-09-02T10:19:53.234818+0000","last_active":"2024-09-02T10:22:51.595902+0000","last_peered":"2024-09-02T10:22:51.595902+0000","last_clean":"2024-09-02T10:22:51.595902+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:22:51.595902+0000","last_undegraded":"2024-09-02T10:22:51.595902+0000","last_fullsized":"2024-09-02T10:22:51.595902+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.234728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.234728+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"626'12394","reported_seq":23943,"reported_epoch":632,"state":"active+recovering+remapped","last_fresh":"2024-09-02T10:22:52.529901+0000","last_change":"2024-09-02T10:20:53.492211+0000","last_active":"2024-09-02T10:22:52.529901+0000","last_peered":"2024-09-02T10:22:52.529901+0000","last_clean":"2024-09-02T10:20:51.027001+0000","last_became_active":"2024-09-02T10:20:53.416507+0000","last_became_peered":"2024-09-02T10:20:53.416507+0000","last_unstale":"2024-09-02T10:22:52.529901+0000","last_undegraded":"2024-09-02T10:22:52.529901+0000","last_fullsized":"2024-09-02T10:22:52.529901+0000","mapping_epoch":586,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":294,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12394,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5639,"num_bytes_recovered":237961216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[5,10,9],"avail_no_missing":["5","9","10"],"object_location_counts":[{"shards":"5,9,10","objects":-294}],"blocked_by":[],"up_primary":2,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1222,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.596018+0000","last_change":"2024-09-02T10:20:52.111563+0000","last_active":"2024-09-02T10:22:51.596018+0000","last_peered":"2024-09-02T10:22:51.596018+0000","last_clean":"2024-09-02T10:22:51.596018+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:22:51.596018+0000","last_undegraded":"2024-09-02T10:22:51.596018+0000","last_fullsized":"2024-09-02T10:22:51.596018+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:52.109153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.109153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"626'12580","reported_seq":27864,"reported_epoch":634,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:54.543913+0000","last_change":"2024-09-02T10:20:53.422779+0000","last_active":"2024-09-02T10:22:54.543913+0000","last_peered":"2024-09-02T10:22:54.543913+0000","last_clean":"2024-09-02T10:20:51.098224+0000","last_became_active":"2024-09-02T10:20:53.370526+0000","last_became_peered":"2024-09-02T10:20:53.370526+0000","last_unstale":"2024-09-02T10:22:54.543913+0000","last_undegraded":"2024-09-02T10:20:53.137669+0000","last_fullsized":"2024-09-02T10:20:53.103100+0000","mapping_epoch":586,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":559,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7498,"num_bytes_recovered":374538240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4","objects":-475}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1359,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-09-02T10:22:55.630249+0000","last_change":"2024-09-02T10:22:37.332903+0000","last_active":"2024-09-02T10:22:55.630249+0000","last_peered":"2024-09-02T10:22:55.630249+0000","last_clean":"2024-09-02T10:22:55.630249+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:22:55.630249+0000","last_undegraded":"2024-09-02T10:22:55.630249+0000","last_fullsized":"2024-09-02T10:22:55.630249+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:37.332859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:37.332859+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.675027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"626'12351","reported_seq":12622,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.595986+0000","last_change":"2024-09-02T10:20:27.715367+0000","last_active":"2024-09-02T10:22:51.595986+0000","last_peered":"2024-09-02T10:22:51.595986+0000","last_clean":"2024-09-02T10:22:51.595986+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:22:51.595986+0000","last_undegraded":"2024-09-02T10:22:51.595986+0000","last_fullsized":"2024-09-02T10:22:51.595986+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"577'7181","last_scrub_stamp":"2024-09-02T10:20:27.715126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:27.715126+0000","objects_scrubbed":5211,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12351,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":988,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.781555+0000","last_change":"2024-09-02T10:20:18.812280+0000","last_active":"2024-09-02T10:22:51.781555+0000","last_peered":"2024-09-02T10:22:51.781555+0000","last_clean":"2024-09-02T10:22:51.781555+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:22:51.781555+0000","last_undegraded":"2024-09-02T10:22:51.781555+0000","last_fullsized":"2024-09-02T10:22:51.781555+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:18.812212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:18.812212+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"626'12382","reported_seq":12669,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-02T10:22:52.668815+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:22:52.668815+0000","last_peered":"2024-09-02T10:22:52.668815+0000","last_clean":"2024-09-02T10:22:52.668815+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:22:52.668815+0000","last_undegraded":"2024-09-02T10:22:52.668815+0000","last_fullsized":"2024-09-02T10:22:52.668815+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":393,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.781146+0000","last_change":"2024-09-02T10:22:15.563077+0000","last_active":"2024-09-02T10:22:51.781146+0000","last_peered":"2024-09-02T10:22:51.781146+0000","last_clean":"2024-09-02T10:22:51.781146+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:22:51.781146+0000","last_undegraded":"2024-09-02T10:22:51.781146+0000","last_fullsized":"2024-09-02T10:22:51.781146+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:15.563022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:15.563022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:20.809417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"626'12606","reported_seq":12914,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-09-02T10:22:55.630365+0000","last_change":"2024-09-02T10:21:41.827351+0000","last_active":"2024-09-02T10:22:55.630365+0000","last_peered":"2024-09-02T10:22:55.630365+0000","last_clean":"2024-09-02T10:22:55.630365+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:22:55.630365+0000","last_undegraded":"2024-09-02T10:22:55.630365+0000","last_fullsized":"2024-09-02T10:22:55.630365+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10385","last_scrub_stamp":"2024-09-02T10:21:41.827172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:41.827172+0000","objects_scrubbed":2236,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":660,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1302,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.781145+0000","last_change":"2024-09-02T10:22:34.647196+0000","last_active":"2024-09-02T10:22:51.781145+0000","last_peered":"2024-09-02T10:22:51.781145+0000","last_clean":"2024-09-02T10:22:51.781145+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:22:51.781145+0000","last_undegraded":"2024-09-02T10:22:51.781145+0000","last_fullsized":"2024-09-02T10:22:51.781145+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:34.647144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:34.647144+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:02.024298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"626'12544","reported_seq":28887,"reported_epoch":634,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:54.540413+0000","last_change":"2024-09-02T10:20:53.407005+0000","last_active":"2024-09-02T10:22:54.540413+0000","last_peered":"2024-09-02T10:22:54.540413+0000","last_clean":"2024-09-02T10:20:51.097394+0000","last_became_active":"2024-09-02T10:20:53.379154+0000","last_became_peered":"2024-09-02T10:20:53.379154+0000","last_unstale":"2024-09-02T10:22:54.540413+0000","last_undegraded":"2024-09-02T10:20:53.138449+0000","last_fullsized":"2024-09-02T10:20:53.102891+0000","mapping_epoch":586,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5178","last_scrub_stamp":"2024-09-02T10:19:54.548007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:54.548007+0000","objects_scrubbed":5123,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8027,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3],"avail_no_missing":["4","3","11"],"object_location_counts":[{"shards":"3,4","objects":-195}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":760,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.781210+0000","last_change":"2024-09-02T10:20:52.104138+0000","last_active":"2024-09-02T10:22:51.781210+0000","last_peered":"2024-09-02T10:22:51.781210+0000","last_clean":"2024-09-02T10:22:51.781210+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:22:51.781210+0000","last_undegraded":"2024-09-02T10:22:51.781210+0000","last_fullsized":"2024-09-02T10:22:51.781210+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:00.749804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.749804+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"626'12380","reported_seq":12688,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:53.584339+0000","last_change":"2024-09-02T10:21:39.544819+0000","last_active":"2024-09-02T10:22:53.584339+0000","last_peered":"2024-09-02T10:22:53.584339+0000","last_clean":"2024-09-02T10:22:53.584339+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:22:53.584339+0000","last_undegraded":"2024-09-02T10:22:53.584339+0000","last_fullsized":"2024-09-02T10:22:53.584339+0000","mapping_epoch":533,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"603'10078","last_scrub_stamp":"2024-09-02T10:21:39.544630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.544630+0000","objects_scrubbed":2309,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:04.692572+0000","scrub_duration":711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12380,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":724,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.781121+0000","last_change":"2024-09-02T10:19:50.631876+0000","last_active":"2024-09-02T10:22:51.781121+0000","last_peered":"2024-09-02T10:22:51.781121+0000","last_clean":"2024-09-02T10:22:51.781121+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:22:51.781121+0000","last_undegraded":"2024-09-02T10:22:51.781121+0000","last_fullsized":"2024-09-02T10:22:51.781121+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:50.631823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:50.631823+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"626'12678","reported_seq":13039,"reported_epoch":632,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:22:51.909495+0000","last_change":"2024-09-02T10:20:53.439960+0000","last_active":"2024-09-02T10:22:51.909495+0000","last_peered":"2024-09-02T10:22:51.909495+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:22:51.909495+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-4089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":492,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.573744+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:22:51.573744+0000","last_peered":"2024-09-02T10:22:51.573744+0000","last_clean":"2024-09-02T10:22:51.573744+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:22:51.573744+0000","last_undegraded":"2024-09-02T10:22:51.573744+0000","last_fullsized":"2024-09-02T10:22:51.573744+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:16.363558+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"626'12542","reported_seq":17173,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-02T10:22:52.668787+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:22:52.668787+0000","last_peered":"2024-09-02T10:22:52.668787+0000","last_clean":"2024-09-02T10:22:52.668787+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:22:52.668787+0000","last_undegraded":"2024-09-02T10:22:52.668787+0000","last_fullsized":"2024-09-02T10:22:52.668787+0000","mapping_epoch":559,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12542,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":725,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.781209+0000","last_change":"2024-09-02T10:19:51.719482+0000","last_active":"2024-09-02T10:22:51.781209+0000","last_peered":"2024-09-02T10:22:51.781209+0000","last_clean":"2024-09-02T10:22:51.781209+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:22:51.781209+0000","last_undegraded":"2024-09-02T10:22:51.781209+0000","last_fullsized":"2024-09-02T10:22:51.781209+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:51.719422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:51.719422+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"626'12632","reported_seq":12945,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-09-02T10:22:55.630372+0000","last_change":"2024-09-02T10:21:39.988693+0000","last_active":"2024-09-02T10:22:55.630372+0000","last_peered":"2024-09-02T10:22:55.630372+0000","last_clean":"2024-09-02T10:22:55.630372+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:22:55.630372+0000","last_undegraded":"2024-09-02T10:22:55.630372+0000","last_fullsized":"2024-09-02T10:22:55.630372+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"604'10274","last_scrub_stamp":"2024-09-02T10:21:39.988364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:39.988364+0000","objects_scrubbed":2383,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:57.287011+0000","scrub_duration":866,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":993,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-09-02T10:22:55.630322+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:22:55.630322+0000","last_peered":"2024-09-02T10:22:55.630322+0000","last_clean":"2024-09-02T10:22:55.630322+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:22:55.630322+0000","last_undegraded":"2024-09-02T10:22:55.630322+0000","last_fullsized":"2024-09-02T10:22:55.630322+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"626'12586","reported_seq":12860,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:53.584641+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:22:53.584641+0000","last_peered":"2024-09-02T10:22:53.584641+0000","last_clean":"2024-09-02T10:22:53.584641+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:22:53.584641+0000","last_undegraded":"2024-09-02T10:22:53.584641+0000","last_fullsized":"2024-09-02T10:22:53.584641+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12586,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1251,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:53.584579+0000","last_change":"2024-09-02T10:19:53.376105+0000","last_active":"2024-09-02T10:22:53.584579+0000","last_peered":"2024-09-02T10:22:53.584579+0000","last_clean":"2024-09-02T10:22:53.584579+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:22:53.584579+0000","last_undegraded":"2024-09-02T10:22:53.584579+0000","last_fullsized":"2024-09-02T10:22:53.584579+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:53.376058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.376058+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"626'12632","reported_seq":17150,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.596060+0000","last_change":"2024-09-02T10:19:56.369956+0000","last_active":"2024-09-02T10:22:51.596060+0000","last_peered":"2024-09-02T10:22:51.596060+0000","last_clean":"2024-09-02T10:22:51.596060+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:22:51.596060+0000","last_undegraded":"2024-09-02T10:22:51.596060+0000","last_fullsized":"2024-09-02T10:22:51.596060+0000","mapping_epoch":565,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5332","last_scrub_stamp":"2024-09-02T10:19:56.369748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:56.369748+0000","objects_scrubbed":5302,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1005,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-09-02T10:22:55.630276+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:22:55.630276+0000","last_peered":"2024-09-02T10:22:55.630276+0000","last_clean":"2024-09-02T10:22:55.630276+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:22:55.630276+0000","last_undegraded":"2024-09-02T10:22:55.630276+0000","last_fullsized":"2024-09-02T10:22:55.630276+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"626'12650","reported_seq":12967,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-09-02T10:22:55.630277+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:22:55.630277+0000","last_peered":"2024-09-02T10:22:55.630277+0000","last_clean":"2024-09-02T10:22:55.630277+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:22:55.630277+0000","last_undegraded":"2024-09-02T10:22:55.630277+0000","last_fullsized":"2024-09-02T10:22:55.630277+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12650,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1253,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.573771+0000","last_change":"2024-09-02T10:21:35.696117+0000","last_active":"2024-09-02T10:22:51.573771+0000","last_peered":"2024-09-02T10:22:51.573771+0000","last_clean":"2024-09-02T10:22:51.573771+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:22:51.573771+0000","last_undegraded":"2024-09-02T10:22:51.573771+0000","last_fullsized":"2024-09-02T10:22:51.573771+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:35.696044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.696044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:22:55.467370+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"626'12496","reported_seq":12770,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.909493+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:22:51.909493+0000","last_peered":"2024-09-02T10:22:51.909493+0000","last_clean":"2024-09-02T10:22:51.909493+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:22:51.909493+0000","last_undegraded":"2024-09-02T10:22:51.909493+0000","last_fullsized":"2024-09-02T10:22:51.909493+0000","mapping_epoch":533,"log_start":"553'2400","ondisk_log_start":"553'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":206,"reported_epoch":634,"state":"active+clean","last_fresh":"2024-09-02T10:22:54.113434+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:22:54.113434+0000","last_peered":"2024-09-02T10:22:54.113434+0000","last_clean":"2024-09-02T10:22:54.113434+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:22:54.113434+0000","last_undegraded":"2024-09-02T10:22:54.113434+0000","last_fullsized":"2024-09-02T10:22:54.113434+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"626'12520","reported_seq":12797,"reported_epoch":632,"state":"active+clean","last_fresh":"2024-09-02T10:22:51.596021+0000","last_change":"2024-09-02T10:20:21.937271+0000","last_active":"2024-09-02T10:22:51.596021+0000","last_peered":"2024-09-02T10:22:51.596021+0000","last_clean":"2024-09-02T10:22:51.596021+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:22:51.596021+0000","last_undegraded":"2024-09-02T10:22:51.596021+0000","last_fullsized":"2024-09-02T10:22:51.596021+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6863","last_scrub_stamp":"2024-09-02T10:20:21.937140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.937140+0000","objects_scrubbed":5699,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1783,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12520,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1040,"reported_epoch":635,"state":"active+clean","last_fresh":"2024-09-02T10:22:55.630488+0000","last_change":"2024-09-02T10:22:30.358040+0000","last_active":"2024-09-02T10:22:55.630488+0000","last_peered":"2024-09-02T10:22:55.630488+0000","last_clean":"2024-09-02T10:22:55.630488+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:22:55.630488+0000","last_undegraded":"2024-09-02T10:22:55.630488+0000","last_fullsized":"2024-09-02T10:22:55.630488+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:30.357970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:30.357970+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.221263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"626'12566","reported_seq":12841,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-02T10:22:52.668632+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:22:52.668632+0000","last_peered":"2024-09-02T10:22:52.668632+0000","last_clean":"2024-09-02T10:22:52.668632+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:22:52.668632+0000","last_undegraded":"2024-09-02T10:22:52.668632+0000","last_fullsized":"2024-09-02T10:22:52.668632+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":418,"reported_epoch":633,"state":"active+clean","last_fresh":"2024-09-02T10:22:52.668743+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:22:52.668743+0000","last_peered":"2024-09-02T10:22:52.668743+0000","last_clean":"2024-09-02T10:22:52.668743+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:22:52.668743+0000","last_undegraded":"2024-09-02T10:22:52.668743+0000","last_fullsized":"2024-09-02T10:22:52.668743+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6964,"num_objects_misplaced":294,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25501,"num_bytes_recovered":1286144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":804007936,"data_stored":803995720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":48,"acting":45,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383953,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":742424,"kb_used_data":260036,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":93002728,"statfs":{"total":95995035648,"available":95234793472,"internally_reserved":0,"allocated":266276864,"data_stored":265937356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":493943236},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612886,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":356960,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93388192,"statfs":{"total":95995035648,"available":95629508608,"internally_reserved":0,"allocated":925696,"data_stored":582116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841819,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545772,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199380,"statfs":{"total":95995035648,"available":95436165120,"internally_reserved":0,"allocated":933888,"data_stored":590097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070753,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338940,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93406212,"statfs":{"total":95995035648,"available":95647961088,"internally_reserved":0,"allocated":958464,"data_stored":606057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299687,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582100,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93163052,"statfs":{"total":95995035648,"available":95398965248,"internally_reserved":0,"allocated":1175552,"data_stored":815306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528617,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365596,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379556,"statfs":{"total":95995035648,"available":95620665344,"internally_reserved":0,"allocated":1249280,"data_stored":909772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724846,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1140016,"kb_used_data":527132,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92605136,"statfs":{"total":95995035648,"available":94827659264,"internally_reserved":0,"allocated":539783168,"data_stored":539431753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953780,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334768,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410384,"statfs":{"total":95995035648,"available":95652233216,"internally_reserved":0,"allocated":946176,"data_stored":598102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150009,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569716,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175436,"statfs":{"total":95995035648,"available":95411646464,"internally_reserved":0,"allocated":1273856,"data_stored":925782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411644,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307176,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93437976,"statfs":{"total":95995035648,"available":95680487424,"internally_reserved":0,"allocated":1138688,"data_stored":791345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575170,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509684,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235468,"statfs":{"total":95995035648,"available":95473119232,"internally_reserved":0,"allocated":950272,"data_stored":598126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738696,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":591912,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93153240,"statfs":{"total":95995035648,"available":95388917760,"internally_reserved":0,"allocated":1261568,"data_stored":917777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":538644480,"data_stored":538640408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":265355264,"data_stored":265355264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:22:58.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:22:58.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:22:58.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:58.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:58.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:22:58.704 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:22:58.704 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:22:58.704 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:22:58.704 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:22:58.781 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:58.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:58.803 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:58.804 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:58.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:22:58 smithi151 ceph-mon[16639]: pgmap v1475: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 0 B/s, 314 objects/s recovering 2024-09-02T10:22:58.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:58.927 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:58.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:58.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:22:58 smithi187 ceph-mon[20072]: pgmap v1475: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 0 B/s, 314 objects/s recovering 2024-09-02T10:22:59.029 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:59.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:59.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:59.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:22:58 smithi162 ceph-mon[20279]: pgmap v1475: 47 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6964/9 objects degraded (77377.778%); 0 B/s, 314 objects/s recovering 2024-09-02T10:22:59.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:22:59.168 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:22:59.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:22:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:59.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:22:59.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:22:59.308 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:59.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:59.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:22:59.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:22:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:59.434 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:59.434 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:22:59.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:22:59.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:22:59.556 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:59.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:22:59.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:22:59.659 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:22:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:59.683 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:22:59.684 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:22:59.721 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:22:59.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:59.807 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:59.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:59.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:22:59.909 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:22:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:22:59.931 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:22:59.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:00.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:00.055 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:00.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:00.134 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:23:00.135 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:23:00.135 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:23:00.135 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:23:00.135 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:23:00.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:00.172 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:23:00.180 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:00.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:00.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:00.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:00.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:00.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:00.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:00.368 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:00 smithi151 ceph-mon[16639]: osdmap e638: 12 total, 12 up, 12 in 2024-09-02T10:23:00.368 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:00 smithi151 ceph-mon[16639]: 6.5 scrub starts 2024-09-02T10:23:00.368 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:00 smithi151 ceph-mon[16639]: 6.5 scrub ok 2024-09-02T10:23:00.368 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/744170451' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:23:00.369 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:00 smithi151 ceph-mon[16639]: pgmap v1478: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4759/9 objects degraded (52877.778%); 0 B/s, 191 objects/s recovering 2024-09-02T10:23:00.411 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:00.434 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:00.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:00.464 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:00 smithi187 ceph-mon[20072]: osdmap e638: 12 total, 12 up, 12 in 2024-09-02T10:23:00.465 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:00 smithi187 ceph-mon[20072]: 6.5 scrub starts 2024-09-02T10:23:00.465 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:00 smithi187 ceph-mon[20072]: 6.5 scrub ok 2024-09-02T10:23:00.465 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/744170451' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:23:00.465 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:00 smithi187 ceph-mon[20072]: pgmap v1478: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4759/9 objects degraded (52877.778%); 0 B/s, 191 objects/s recovering 2024-09-02T10:23:00.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:00.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:00.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:00 smithi162 ceph-mon[20279]: osdmap e638: 12 total, 12 up, 12 in 2024-09-02T10:23:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:00 smithi162 ceph-mon[20279]: 6.5 scrub starts 2024-09-02T10:23:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:00 smithi162 ceph-mon[20279]: 6.5 scrub ok 2024-09-02T10:23:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/744170451' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:23:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:00 smithi162 ceph-mon[20279]: pgmap v1478: 47 pgs: 3 active+recovering+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4759/9 objects degraded (52877.778%); 0 B/s, 191 objects/s recovering 2024-09-02T10:23:00.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:00.681 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:00.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:00.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:00.778 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:23:00.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:00.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:00.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:00.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:00.886 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:23:00.886 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4759/9 objects degraded (52877.778%), 3 pgs degraded, 3 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1652,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":639,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":12,"osd_in_since":1725272451,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":44},{"state_name":"active+recovering+undersized+degraded+remapped","count":3}],"num_pgs":47,"num_pools":2,"num_objects":3,"data_bytes":590392,"bytes_used":6538756096,"bytes_avail":1145401671680,"bytes_total":1151940427776,"degraded_objects":4759,"degraded_total":9,"degraded_ratio":528.77777777777783,"recovering_objects_per_sec":191,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":767,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":37,"modified":"2024-09-02T10:21:41.376228+0000","services":{}},"progress_events":{}} 2024-09-02T10:23:00.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:00.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:00.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:01.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:01.057 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:01.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:01.060 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:23:01.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:01.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:01.183 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:01.183 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:01.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:01.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:01.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:01.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:01.409 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:01.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:01.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:23:01.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:23:01.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 4759/9 objects degraded (52877.778%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:23:01.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:23:01.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/744170451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:23:01.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: osdmap e639: 12 total, 12 up, 12 in 2024-09-02T10:23:01.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/864723420' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:23:01.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:23:01.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:23:01.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:23:01.440 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:01 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:23:01.444 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:23:01.445 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:23:01.445 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 4759/9 objects degraded (52877.778%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:23:01.445 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:23:01.445 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/744170451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:23:01.445 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: osdmap e639: 12 total, 12 up, 12 in 2024-09-02T10:23:01.445 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/864723420' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:23:01.446 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:23:01.446 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:23:01.446 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:23:01.446 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:01 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:23:01.534 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:01.543 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:23:01.543 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:23:01.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:23:01.543 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:23:01.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62902 5.0 5:0a3860dd:::benchmark_data_smithi187_89405_object62901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 446.906072704, 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28493924999999998, 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62902 2024-09-02T10:23:01.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635067+0000", 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635068+0000", 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.502e-06 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635089+0000", 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0916000000000001e-05 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635100+0000", 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1243000000000001e-05 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635107+0000", 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5359999999999998e-06 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635155+0000", 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7939000000000003e-05 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635250+0000", 2024-09-02T10:23:01.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5571999999999999e-05 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.635339+0000", 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9158000000000003e-05 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.636945+0000", 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001605555 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.637559+0000", 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00061385400000000005 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.919922+0000", 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28236284 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.919962+0000", 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0227e-05 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.920006+0000", 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3908e-05 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:126223 5.4 5:2d883f96:::benchmark_data_smithi187_89405_object22618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 379.61521162700001, 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32322968800000001, 2024-09-02T10:23:01.550 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126223 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925928+0000", 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925929+0000", 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2020000000000001e-06 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925933+0000", 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3479999999999997e-06 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925945+0000", 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1559999999999999e-05 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925952+0000", 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4120000000000004e-06 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.925997+0000", 2024-09-02T10:23:01.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4759999999999998e-05 2024-09-02T10:23:01.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.926077+0000", 2024-09-02T10:23:01.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0203000000000004e-05 2024-09-02T10:23:01.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.926145+0000", 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8311000000000004e-05 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929053+0000", 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029072289999999999 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.989494+0000", 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.060441307999999999 2024-09-02T10:23:01.555 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249062+0000", 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25956764199999999 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249115+0000", 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2896000000000003e-05 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249157+0000", 2024-09-02T10:23:01.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2817e-05 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:126223 5.4 e490/431 ondisk, result = 0)", 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 379.61186751299999, 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31989100399999998, 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:01.557 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929272+0000", 2024-09-02T10:23:01.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2900000000000003e-07 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.929276+0000", 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4989999999999999e-06 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.248945+0000", 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31966908500000002 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.248950+0000", 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1749999999999996e-06 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249048+0000", 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7676000000000005e-05 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249056+0000", 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2439999999999993e-06 2024-09-02T10:23:01.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.249163+0000", 2024-09-02T10:23:01.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106696 2024-09-02T10:23:01.561 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.561 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.561 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162650 5.0 5:0595c7e6:::benchmark_data_smithi187_89405_object59045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 337.23844962499999, 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29653194100000002, 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162650 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302691+0000", 2024-09-02T10:23:01.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6139999999999999e-06 2024-09-02T10:23:01.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302696+0000", 2024-09-02T10:23:01.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1599999999999997e-06 2024-09-02T10:23:01.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302706+0000", 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3959999999999995e-06 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302711+0000", 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2920000000000003e-06 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302770+0000", 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8545999999999998e-05 2024-09-02T10:23:01.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302912+0000", 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014217599999999999 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.303032+0000", 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120482 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:23:01.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.307337+0000", 2024-09-02T10:23:01.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004304993 2024-09-02T10:23:01.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.307512+0000", 2024-09-02T10:23:01.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017463799999999999 2024-09-02T10:23:01.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598774+0000", 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29126201499999999 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599166+0000", 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039243999999999998 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599222+0000", 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5189000000000003e-05 2024-09-02T10:23:01.568 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:162688 5.8 e507/490 5:1af44aef:::benchmark_data_smithi187_89405_object59083:head v 507'10194, mlcod=507'10191)", 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 337.19323563099999, 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.262002719, 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8700000000000002e-07 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347908+0000", 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2030000000000002e-06 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347966+0000", 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7603999999999999e-05 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347970+0000", 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1160000000000001e-06 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.348622+0000", 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00065184499999999996 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.348639+0000", 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7731e-05 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609811+0000", 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26117189299999999 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609906+0000", 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5039999999999998e-05 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:01.575 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:162690 5.f e507/431 5:f963755e:::benchmark_data_smithi187_89405_object59085:head v 507'10319, mlcod=507'10317)", 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 337.18953170200001, 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26002222000000003, 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:23:01.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:23:01.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3699999999999999e-07 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351615+0000", 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1940000000000003e-06 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353238+0000", 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001623471 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353245+0000", 2024-09-02T10:23:01.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4540000000000004e-06 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353298+0000", 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3186999999999997e-05 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353317+0000", 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9083999999999999e-05 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611594+0000", 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.258277337 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611630+0000", 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5655999999999999e-05 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162691 5.4 5:2922879e:::benchmark_data_smithi187_89405_object59086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:23:01.582 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 337.18907184099999, 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25893540799999998, 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162691 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352069+0000", 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6869999999999999e-06 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352075+0000", 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5940000000000001e-06 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352087+0000", 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.216e-05 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352093+0000", 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5969999999999997e-06 2024-09-02T10:23:01.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599399+0000", 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24730590099999999 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599528+0000", 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012868299999999999 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599654+0000", 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000125959 2024-09-02T10:23:01.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610381+0000", 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.01072701 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610393+0000", 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1921e-05 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610740+0000", 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00034708900000000002 2024-09-02T10:23:01.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610837+0000", 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7175000000000004e-05 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611003+0000", 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000165632 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.588 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162695 5.0 5:02845cd7:::benchmark_data_smithi187_89405_object59090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 337.187306197, 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.266211531, 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.589 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162695 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353834+0000", 2024-09-02T10:23:01.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3900000000000004e-07 2024-09-02T10:23:01.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353838+0000", 2024-09-02T10:23:01.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3460000000000004e-06 2024-09-02T10:23:01.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598349+0000", 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24451044699999999 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598367+0000", 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8236999999999998e-05 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609481+0000", 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011113576 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609606+0000", 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012559100000000001 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610188+0000", 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058149899999999997 2024-09-02T10:23:01.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619903+0000", 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0097155599999999998 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619914+0000", 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1126e-05 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619947+0000", 2024-09-02T10:23:01.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2432999999999999e-05 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.620001+0000", 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4295000000000001e-05 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.620045+0000", 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3581999999999997e-05 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.595 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:173247 5.8 e509/490 5:173cf939:::benchmark_data_smithi187_89405_object69642:head v 509'10875, mlcod=509'10872)", 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 324.39872319199998, 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25939572700000002, 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142417+0000", 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6099999999999999e-07 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142424+0000", 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9489999999999999e-06 2024-09-02T10:23:01.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396116+0000", 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25369161699999998 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396123+0000", 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5059999999999998e-06 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396178+0000", 2024-09-02T10:23:01.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5271e-05 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396199+0000", 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1009000000000002e-05 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.401729+0000", 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0055293260000000002 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.401812+0000", 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3088000000000001e-05 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207153 5.4 5:2d15d89f:::benchmark_data_smithi187_89405_object103548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 281.49662251900003, 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.39443205100000001, 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.600 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207153 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:23:01.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044518+0000", 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1179999999999999e-06 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044523+0000", 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.113e-06 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.602 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044535+0000", 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1868999999999999e-05 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044540+0000", 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9370000000000003e-06 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044599+0000", 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9589000000000002e-05 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044722+0000", 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122166 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044855+0000", 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133774 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.048607+0000", 2024-09-02T10:23:01.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037520180000000002 2024-09-02T10:23:01.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.075260+0000", 2024-09-02T10:23:01.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.026652973 2024-09-02T10:23:01.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438874+0000", 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.363613939 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438913+0000", 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8795999999999999e-05 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438949+0000", 2024-09-02T10:23:01.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5759000000000001e-05 2024-09-02T10:23:01.607 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.607 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.607 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.607 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207153 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:23:01.607 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 281.45340816300001, 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35122426299999998, 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:23:01.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087732+0000", 2024-09-02T10:23:01.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5300000000000004e-07 2024-09-02T10:23:01.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087736+0000", 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0659999999999997e-06 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438760+0000", 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.351023794 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438766+0000", 2024-09-02T10:23:01.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6499999999999999e-06 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438864+0000", 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8116999999999996e-05 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438871+0000", 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1430000000000003e-06 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438955+0000", 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4839999999999994e-05 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:207198 5.a e525/431 5:57bb5924:::benchmark_data_smithi187_89405_object103593:head v 525'12960, mlcod=525'12956)", 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 281.42455266799999, 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44214100000000001, 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.612 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1399999999999996e-07 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116593+0000", 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0079999999999999e-06 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554609+0000", 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43801511199999998 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.614 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554615+0000", 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8349999999999998e-06 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554662+0000", 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7018000000000003e-05 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554678+0000", 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5116000000000001e-05 2024-09-02T10:23:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558692+0000", 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040144350000000002 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558728+0000", 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5661999999999998e-05 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207204 5.4 5:2473bd20:::benchmark_data_smithi187_89405_object103599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 281.414020381, 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73061778899999996, 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207204 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.617 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127120+0000", 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2810000000000001e-06 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127126+0000", 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1210000000000003e-06 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127137+0000", 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0474e-05 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127142+0000", 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5419999999999996e-06 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127191+0000", 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8485000000000001e-05 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127278+0000", 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7492999999999997e-05 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127352+0000", 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3591999999999996e-05 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131106+0000", 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003754496 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850393+0000", 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.719286219 2024-09-02T10:23:01.621 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857585+0000", 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071919810000000001 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857692+0000", 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107038 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857737+0000", 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5067000000000001e-05 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207204 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 281.409493666, 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72609860400000004, 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.623 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131647+0000", 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2529999999999999e-06 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131653+0000", 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4119999999999998e-06 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857394+0000", 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72574053699999996 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857411+0000", 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7283e-05 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857560+0000", 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014918700000000001 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857574+0000", 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.329e-05 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857744+0000", 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017064200000000001 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207204 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 280.98655245600003, 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.295830757, 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.627 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.628 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554588+0000", 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8899999999999998e-07 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554593+0000", 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1039999999999998e-06 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850206+0000", 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29561359100000001 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850223+0000", 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6730000000000001e-05 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850366+0000", 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014310500000000001 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850383+0000", 2024-09-02T10:23:01.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6419999999999999e-05 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850418+0000", 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4918e-05 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(1.1,GRANT,e:638,from:9,reservation_nonce:0)", 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:23:00.145120+0000", 2024-09-02T10:23:01.631 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 1.3960194210000001, 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025720000000000002, 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.145120+0000", 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.145120+0000", 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.145120+0000", 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2300000000000001e-07 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.145123+0000", 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.976e-06 2024-09-02T10:23:01.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.145149+0000", 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5060999999999999e-05 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.145153+0000", 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4100000000000001e-06 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.145200+0000", 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6882000000000003e-05 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.145220+0000", 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9936999999999999e-05 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.145377+0000", 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015731100000000001 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.635 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(1.10,QREQUEST,e:639,from:5,reservation_nonce:15)", 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:23:00.620524+0000", 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.92061572400000002, 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00027404799999999999, 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.620524+0000", 2024-09-02T10:23:01.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.620524+0000", 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.620525+0000", 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6330000000000001e-06 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.620534+0000", 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0869999999999996e-06 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.620599+0000", 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5055999999999995e-05 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.620613+0000", 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3900000000000001e-05 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.620751+0000", 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013810100000000001 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.620768+0000", 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.683e-05 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:00.620798+0000", 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9441000000000002e-05 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.639 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(1.19,GRANT,e:639,from:0,reservation_nonce:0)", 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:23:01.183547+0000", 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.35759218599999998, 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000153144, 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.183547+0000", 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.183547+0000", 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.183549+0000", 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4050000000000001e-06 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.183552+0000", 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.517e-06 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.183578+0000", 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.588e-05 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.183586+0000", 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4069999999999993e-06 2024-09-02T10:23:01.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.183637+0000", 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1054999999999998e-05 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.183658+0000", 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0712000000000001e-05 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.183700+0000", 2024-09-02T10:23:01.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2168e-05 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(1.19,GRANT,e:639,from:6,reservation_nonce:0)", 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:23:01.184041+0000", 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.35709878499999997, 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016835600000000001, 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:01.644 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184041+0000", 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184041+0000", 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184041+0000", 2024-09-02T10:23:01.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.53e-07 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184043+0000", 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3879999999999998e-06 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184058+0000", 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4442e-05 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184061+0000", 2024-09-02T10:23:01.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6559999999999998e-06 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184091+0000", 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0018e-05 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184103+0000", 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1694e-05 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184209+0000", 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010580499999999999 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "rep_scrubmap(1.19 e301 from shard 6)", 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:23:01.184809+0000", 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.35633028700000002, 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015691899999999999, 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:01.648 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184809+0000", 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184809+0000", 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184809+0000", 2024-09-02T10:23:01.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4200000000000001e-07 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184812+0000", 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9409999999999999e-06 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184855+0000", 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2116000000000001e-05 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184859+0000", 2024-09-02T10:23:01.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5839999999999996e-06 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184940+0000", 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0657999999999999e-05 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184946+0000", 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4330000000000001e-06 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:01.651 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:01.184966+0000", 2024-09-02T10:23:01.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9745000000000002e-05 2024-09-02T10:23:01.652 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.652 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.652 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.652 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:01.652 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:01.652 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:23:01.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 4759/9 objects degraded (52877.778%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/744170451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: osdmap e639: 12 total, 12 up, 12 in 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/864723420' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:23:01.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:01 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:23:01.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:01.779 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:01.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:01.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:01.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:01.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:02.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:02.027 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:02.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:02.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:02.155 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:02.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:02.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:02.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:02.280 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:02.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:02.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:02.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:23:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:02.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:02 smithi151 ceph-mon[16639]: osdmap e640: 12 total, 12 up, 12 in 2024-09-02T10:23:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:02 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:23:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:02 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:23:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:02 smithi151 ceph-mon[16639]: pgmap v1481: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4251/9 objects degraded (47233.333%); 0 B/s, 332 objects/s recovering 2024-09-02T10:23:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:02 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:23:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:02 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:23:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:02 smithi187 ceph-mon[20072]: osdmap e640: 12 total, 12 up, 12 in 2024-09-02T10:23:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:02 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:23:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:02 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:23:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:02 smithi187 ceph-mon[20072]: pgmap v1481: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4251/9 objects degraded (47233.333%); 0 B/s, 332 objects/s recovering 2024-09-02T10:23:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:02 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:23:02.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:02 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:23:02.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:02.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:02.528 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:02 smithi162 ceph-mon[20279]: osdmap e640: 12 total, 12 up, 12 in 2024-09-02T10:23:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:02 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:23:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:02 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:23:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:02 smithi162 ceph-mon[20279]: pgmap v1481: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4251/9 objects degraded (47233.333%); 0 B/s, 332 objects/s recovering 2024-09-02T10:23:02.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:02 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:23:02.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:02 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:23:02.627 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:23:02.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:02.656 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:02.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:02.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:02.780 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:02.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:02.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:02.903 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:02.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:03.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:03.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:03.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:03.104 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:23:03.104 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:23:03.104 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:23:03.104 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:23:03.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:03.157 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:03.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:03.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:03 smithi187 ceph-mon[20072]: osdmap e641: 12 total, 12 up, 12 in 2024-09-02T10:23:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:03 smithi187 ceph-mon[20072]: 6.7 scrub starts 2024-09-02T10:23:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:03 smithi187 ceph-mon[20072]: 6.7 scrub ok 2024-09-02T10:23:03.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:03.285 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:03.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:03.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:03.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:03.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:03.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:03 smithi151 ceph-mon[16639]: osdmap e641: 12 total, 12 up, 12 in 2024-09-02T10:23:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:03 smithi151 ceph-mon[16639]: 6.7 scrub starts 2024-09-02T10:23:03.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:03 smithi151 ceph-mon[16639]: 6.7 scrub ok 2024-09-02T10:23:03.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:03.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:03.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:03.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:03 smithi162 ceph-mon[20279]: osdmap e641: 12 total, 12 up, 12 in 2024-09-02T10:23:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:03 smithi162 ceph-mon[20279]: 6.7 scrub starts 2024-09-02T10:23:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:03 smithi162 ceph-mon[20279]: 6.7 scrub ok 2024-09-02T10:23:03.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:03.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:03.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:03.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:03.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:03.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:23:03.785 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:03.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:03.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:03.913 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:03.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:04.001 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:23:04.015 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:04.036 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:04.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:04.037 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:04.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:04.162 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:04.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:04.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:04.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:04.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:04.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:04.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:04.414 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:04.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:04.417 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:04 smithi187 ceph-mon[20072]: pgmap v1483: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4251/9 objects degraded (47233.333%); 0 B/s, 365 objects/s recovering 2024-09-02T10:23:04.417 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:04 smithi187 ceph-mon[20072]: osdmap e642: 12 total, 12 up, 12 in 2024-09-02T10:23:04.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:04 smithi151 ceph-mon[16639]: pgmap v1483: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4251/9 objects degraded (47233.333%); 0 B/s, 365 objects/s recovering 2024-09-02T10:23:04.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:04 smithi151 ceph-mon[16639]: osdmap e642: 12 total, 12 up, 12 in 2024-09-02T10:23:04.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:04.542 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:04.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:04.579 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:23:04.579 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:23:04.579 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:23:04.579 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:23:04.579 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:23:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:04 smithi162 ceph-mon[20279]: pgmap v1483: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4251/9 objects degraded (47233.333%); 0 B/s, 365 objects/s recovering 2024-09-02T10:23:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:04 smithi162 ceph-mon[20279]: osdmap e642: 12 total, 12 up, 12 in 2024-09-02T10:23:04.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:04.670 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:04.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:04.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:04.795 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:04.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:04.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:04.920 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:04.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:04.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:05.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:05.047 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:05.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:05.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:05.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:05.172 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:05.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:05.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:05.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:23:05.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:05.295 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:05.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:05.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:05.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:05.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:05.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:05.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:05 smithi151 ceph-mon[16639]: 6.d scrub starts 2024-09-02T10:23:05.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:05 smithi151 ceph-mon[16639]: 6.d scrub ok 2024-09-02T10:23:05.439 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:23:05.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:05 smithi187 ceph-mon[20072]: 6.d scrub starts 2024-09-02T10:23:05.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:05 smithi187 ceph-mon[20072]: 6.d scrub ok 2024-09-02T10:23:05.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:05.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:05.544 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:05 smithi162 ceph-mon[20279]: 6.d scrub starts 2024-09-02T10:23:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:05 smithi162 ceph-mon[20279]: 6.d scrub ok 2024-09-02T10:23:05.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:05.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:05.669 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:05.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:05.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:05.798 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:05.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:05.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:05.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:05.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:05.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:05.930 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:23:05.930 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25507.0:126679 4.1 4:86205e4c:::benchmark_data_smithi187_76542_object34343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:13:15.467394+0000", 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 590.46100220699998, 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44570083700000002, 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25507", 2024-09-02T10:23:05.931 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3862031660", 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126679 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467394+0000", 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467394+0000", 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467395+0000", 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1459999999999999e-06 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467400+0000", 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.451e-06 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467413+0000", 2024-09-02T10:23:05.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2799000000000001e-05 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467421+0000", 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4830000000000002e-06 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467468+0000", 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7494999999999998e-05 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467552+0000", 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3823e-05 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.467631+0000", 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8720000000000005e-05 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.469349+0000", 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017183739999999999 2024-09-02T10:23:05.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:05.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.470063+0000", 2024-09-02T10:23:05.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000713731 2024-09-02T10:23:05.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.912992+0000", 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44292904900000002 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.913049+0000", 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7197999999999997e-05 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:13:15.913095+0000", 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5568000000000001e-05 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "rep_scrubmap(1.4 e124 from shard 8)", 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:14:03.322332+0000", 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 542.60606342000005, 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96974535299999998, 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.938 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322332+0000", 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322332+0000", 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322333+0000", 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9000000000000002e-07 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:03.322336+0000", 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.785e-06 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.291988+0000", 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.96965220600000002 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.291994+0000", 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3520000000000003e-06 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.292044+0000", 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9968e-05 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.292054+0000", 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0244000000000001e-05 2024-09-02T10:23:05.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:14:04.292078+0000", 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3408e-05 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62748 5.8 5.b553c768 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.205545+0000", 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.72285062600002, 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99588105199999999, 2024-09-02T10:23:05.942 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62748 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205545+0000", 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205545+0000", 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205546+0000", 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3200000000000003e-07 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205565+0000", 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8519e-05 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205574+0000", 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.448e-06 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205579+0000", 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0980000000000001e-06 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205627+0000", 2024-09-02T10:23:05.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7806000000000003e-05 2024-09-02T10:23:05.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.947 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.947 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.205634+0000", 2024-09-02T10:23:05.947 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5860000000000001e-06 2024-09-02T10:23:05.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.947 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.947 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.947 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201410+0000", 2024-09-02T10:23:05.947 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.99577610400000005 2024-09-02T10:23:05.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201426+0000", 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6558999999999998e-05 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62759 5.8 5.f9616988 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.288073+0000", 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.64032331999999, 2024-09-02T10:23:05.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91353117399999995, 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62759 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288073+0000", 2024-09-02T10:23:05.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288073+0000", 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288074+0000", 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5060000000000001e-06 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288099+0000", 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4862999999999999e-05 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288113+0000", 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4521e-05 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288121+0000", 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8150000000000006e-06 2024-09-02T10:23:05.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288184+0000", 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2637000000000004e-05 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288192+0000", 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5189999999999992e-06 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.952 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201570+0000", 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91337743400000004 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201604+0000", 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3878999999999999e-05 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62760 5.8 5.be230818 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.953 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.288254+0000", 2024-09-02T10:23:05.954 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.64014186899999, 2024-09-02T10:23:05.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91340091000000001, 2024-09-02T10:23:05.954 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.954 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.954 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.954 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.954 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.954 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.954 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62760 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288254+0000", 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288254+0000", 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288255+0000", 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1200000000000002e-07 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288277+0000", 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2768e-05 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288292+0000", 2024-09-02T10:23:05.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4171999999999999e-05 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288298+0000", 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5950000000000004e-06 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288350+0000", 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1588999999999997e-05 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.288356+0000", 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4450000000000001e-06 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201646+0000", 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91328941100000005 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201655+0000", 2024-09-02T10:23:05.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2180000000000005e-06 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62763 5.8 5.1d470d68 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.289624+0000", 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.63877199799998, 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91237609600000003, 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.959 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.960 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.960 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62763 2024-09-02T10:23:05.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.960 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289624+0000", 2024-09-02T10:23:05.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289624+0000", 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289626+0000", 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6500000000000001e-06 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289649+0000", 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3442e-05 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289664+0000", 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.526e-05 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289672+0000", 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7170000000000008e-06 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289731+0000", 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8931999999999998e-05 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.289738+0000", 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1069999999999999e-06 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.201987+0000", 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.91224905899999997 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202000+0000", 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2928999999999999e-05 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62767 5.8 5.9781c208 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.295609+0000", 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.63278701399997, 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90642393600000004, 2024-09-02T10:23:05.964 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62767 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295609+0000", 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295609+0000", 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295610+0000", 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.621e-06 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295647+0000", 2024-09-02T10:23:05.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6955000000000001e-05 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295665+0000", 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7795000000000002e-05 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295674+0000", 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2210000000000003e-06 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295743+0000", 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8320999999999999e-05 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.295755+0000", 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2145e-05 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202026+0000", 2024-09-02T10:23:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.90627074100000005 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202033+0000", 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1369999999999996e-06 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62787 5.8 5.93d7ce28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.316402+0000", 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.61199424699998, 2024-09-02T10:23:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88566921200000004, 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62787 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316402+0000", 2024-09-02T10:23:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316402+0000", 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316404+0000", 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1950000000000002e-06 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316427+0000", 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3611000000000001e-05 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316443+0000", 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5903999999999999e-05 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316451+0000", 2024-09-02T10:23:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4569999999999999e-06 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316511+0000", 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0031999999999997e-05 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.316520+0000", 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3470000000000005e-06 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202063+0000", 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88554330000000003 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202071+0000", 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3660000000000003e-06 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62828 5.8 5.ced072f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.387938+0000", 2024-09-02T10:23:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.54045806900001, 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81416300900000005, 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62828 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387938+0000", 2024-09-02T10:23:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387938+0000", 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387939+0000", 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.22e-06 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387960+0000", 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0571e-05 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387971+0000", 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1659e-05 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.387977+0000", 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2099999999999998e-06 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.388039+0000", 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1771000000000002e-05 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.388048+0000", 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9600000000000006e-06 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202094+0000", 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81404575899999998 2024-09-02T10:23:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202101+0000", 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8589999999999999e-06 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62833 5.8 5.47f28d28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.391791+0000", 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.53660470199998, 2024-09-02T10:23:05.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81033902899999999, 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62833 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391791+0000", 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391791+0000", 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391793+0000", 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7129999999999999e-06 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.981 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391815+0000", 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2514000000000002e-05 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391830+0000", 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4115e-05 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391837+0000", 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3869999999999997e-06 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391895+0000", 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7762e-05 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.391901+0000", 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4910000000000002e-06 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202123+0000", 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.81022218700000004 2024-09-02T10:23:05.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202130+0000", 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8600000000000004e-06 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62834 5.8 5.af89fd28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.393180+0000", 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.53521606999999, 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.80897919100000004, 2024-09-02T10:23:05.984 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62834 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393180+0000", 2024-09-02T10:23:05.985 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393180+0000", 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393181+0000", 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3900000000000004e-07 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393200+0000", 2024-09-02T10:23:05.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8949000000000001e-05 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393210+0000", 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0550000000000001e-05 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393215+0000", 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2789999999999998e-06 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393255+0000", 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9678000000000002e-05 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.393260+0000", 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.989e-06 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202152+0000", 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.80889200100000003 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202159+0000", 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9059999999999996e-06 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62856 5.8 5.39aa96f8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.474887+0000", 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.45350921300002, 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72730094999999995, 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.989 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62856 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474887+0000", 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474887+0000", 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474888+0000", 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2640000000000001e-06 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474909+0000", 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1347000000000001e-05 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474922+0000", 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2448e-05 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474928+0000", 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1519999999999997e-06 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474974+0000", 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6578e-05 2024-09-02T10:23:05.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.474980+0000", 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8880000000000002e-06 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202181+0000", 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72720049399999998 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202188+0000", 2024-09-02T10:23:05.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7789999999999998e-06 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62861 5.8 5.909d6bb8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.497010+0000", 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.43138568799998, 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.70520625100000001, 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.994 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62861 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497010+0000", 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497010+0000", 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:05.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497012+0000", 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.012e-06 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497039+0000", 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6999e-05 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:05.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497060+0000", 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0590000000000001e-05 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497067+0000", 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9199999999999998e-06 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497118+0000", 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1029999999999998e-05 2024-09-02T10:23:05.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.497125+0000", 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8210000000000004e-06 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202209+0000", 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.70508442100000002 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202216+0000", 2024-09-02T10:23:05.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4580000000000004e-06 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62865 5.8 5.cf598d28 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.545057+0000", 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.38333906100002, 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65719031900000002, 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:05.999 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62865 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545057+0000", 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:06.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545057+0000", 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545058+0000", 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4950000000000001e-06 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545081+0000", 2024-09-02T10:23:06.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2793999999999999e-05 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545095+0000", 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3619e-05 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545101+0000", 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9710000000000001e-06 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545150+0000", 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8814999999999999e-05 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.545157+0000", 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9870000000000002e-06 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202240+0000", 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.65708368900000003 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202247+0000", 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9489999999999999e-06 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62879 5.8 5.80da3848 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.568069+0000", 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.36032738300003, 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63420845199999998, 2024-09-02T10:23:06.004 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62879 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568069+0000", 2024-09-02T10:23:06.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568069+0000", 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568070+0000", 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.719e-06 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:06.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568097+0000", 2024-09-02T10:23:06.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6455e-05 2024-09-02T10:23:06.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568112+0000", 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4916000000000001e-05 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568121+0000", 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4690000000000003e-06 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568184+0000", 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2724000000000005e-05 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.568193+0000", 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6840000000000008e-06 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202270+0000", 2024-09-02T10:23:06.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63407665199999996 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202277+0000", 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8329999999999996e-06 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62911 5.8 5.47f6e298 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:34.672293+0000", 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 451.25610247200001, 2024-09-02T10:23:06.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.53001293400000005, 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62911 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672293+0000", 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672293+0000", 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672295+0000", 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.626e-06 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672316+0000", 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1151000000000001e-05 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672328+0000", 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2269000000000001e-05 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672335+0000", 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6649999999999997e-06 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672382+0000", 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6575e-05 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:34.672389+0000", 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8809999999999998e-06 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:06.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202300+0000", 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.52991091899999998 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.202306+0000", 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8480000000000003e-06 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:62929 5.d 5:b28584c9:::benchmark_data_smithi187_89405_object62928:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:35.215954+0000", 2024-09-02T10:23:06.015 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 450.71244203399999, 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.45650502599999998, 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 62929 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215954+0000", 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215954+0000", 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215955+0000", 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3540000000000001e-06 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215975+0000", 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9556999999999999e-05 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215988+0000", 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3335e-05 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.215994+0000", 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2729999999999999e-06 2024-09-02T10:23:06.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.216041+0000", 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7110999999999997e-05 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.216134+0000", 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3008000000000001e-05 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.216218+0000", 2024-09-02T10:23:06.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3017999999999997e-05 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.217972+0000", 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001754979 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.218530+0000", 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00055731300000000004 2024-09-02T10:23:06.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.672383+0000", 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.45385332499999997 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.672426+0000", 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2540999999999998e-05 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:06.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:35.672459+0000", 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3212e-05 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:126219 5.d 5:b74c10c7:::benchmark_data_smithi187_89405_object22614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.906310+0000", 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.02208631799999, 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.330515067, 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:06.022 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126219 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906310+0000", 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906310+0000", 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906311+0000", 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7199999999999997e-07 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906315+0000", 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0439999999999997e-06 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906325+0000", 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0905e-05 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906331+0000", 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6710000000000004e-06 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906372+0000", 2024-09-02T10:23:06.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1084999999999997e-05 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906446+0000", 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3543999999999999e-05 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.906507+0000", 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1243999999999998e-05 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.909064+0000", 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025565589999999999 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.988945+0000", 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.079880947999999993 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236544+0000", 2024-09-02T10:23:06.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24759898499999999 2024-09-02T10:23:06.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:06.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236681+0000", 2024-09-02T10:23:06.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013784499999999999 2024-09-02T10:23:06.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:06.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236825+0000", 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000143265 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:126219 5.d e490/431 ondisk, result = 0)", 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.910662+0000", 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 384.01773419800003, 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32617011499999998, 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:23:06.029 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.910662+0000", 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.910662+0000", 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.910662+0000", 2024-09-02T10:23:06.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7700000000000004e-07 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.910666+0000", 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.332e-06 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236345+0000", 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32567926400000002 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:06.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236357+0000", 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2213e-05 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236518+0000", 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016120800000000001 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236534+0000", 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5622999999999999e-05 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:42.236832+0000", 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000297898 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:157848 6.3 e599/533 6:c44b60a2:::benchmark_data_smithi187_102448_object57576:head v 599'9964, mlcod=599'9961)", 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:21:29.391857+0000", 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 96.536539196000007, 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.106929974, 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:23:06.033 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:29.391857+0000", 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:29.391857+0000", 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:29.391858+0000", 2024-09-02T10:23:06.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5120000000000001e-06 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:29.391868+0000", 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7289999999999995e-06 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:29.494963+0000", 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.103095388 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:29.494970+0000", 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8179999999999998e-06 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:29.495030+0000", 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0303999999999997e-05 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:29.495049+0000", 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.819e-05 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:29.498731+0000", 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003682758 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:29.498787+0000", 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5275000000000002e-05 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:23:06.037 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:23:06.038 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:06.063 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:06.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:06.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:06.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:06.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:06.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:06.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:06.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:06.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:06.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:06.440 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:06.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:06 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:23:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:06 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:23:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:06 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:23:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:06 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:23:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:06 smithi187 ceph-mon[20072]: pgmap v1485: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 113 objects/s recovering 2024-09-02T10:23:06.541 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:06.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:06.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:06 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:23:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:06 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:23:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:06 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:23:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:06 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:23:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:06 smithi162 ceph-mon[20279]: pgmap v1485: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 113 objects/s recovering 2024-09-02T10:23:06.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:06 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:23:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:06 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:23:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:06 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:23:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:06 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:23:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:06 smithi151 ceph-mon[16639]: pgmap v1485: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 113 objects/s recovering 2024-09-02T10:23:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:06.693 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:06.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:06.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:23:06.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:06.814 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:06.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:06.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:06.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:06.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:07.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:07.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:07.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:07.167 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:07.194 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:07.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:07.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:07.319 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:07.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:07.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:07.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:07.447 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:07.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:07.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:07.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:07 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 4089/9 objects degraded (45433.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:07.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:07 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:23:07.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:07 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:23:07.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:07 smithi187 ceph-mon[20072]: 6.a scrub starts 2024-09-02T10:23:07.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:07 smithi187 ceph-mon[20072]: 6.a scrub ok 2024-09-02T10:23:07.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:07.575 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:07.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:07 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 4089/9 objects degraded (45433.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:07 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:23:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:07 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:23:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:07 smithi162 ceph-mon[20279]: 6.a scrub starts 2024-09-02T10:23:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:07 smithi162 ceph-mon[20279]: 6.a scrub ok 2024-09-02T10:23:07.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:07.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:07 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 4089/9 objects degraded (45433.333%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:07 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:23:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:07 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:23:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:07 smithi151 ceph-mon[16639]: 6.a scrub starts 2024-09-02T10:23:07.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:07 smithi151 ceph-mon[16639]: 6.a scrub ok 2024-09-02T10:23:07.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:07.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:07.790 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:07.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:07.827 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:07.828 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:07.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:07.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:07.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:07.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:08.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:08.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:08.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:08.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:08.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:08.198 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:08.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:08.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:08.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:08.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:08.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:08.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:08.452 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:08 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:23:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:08 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:23:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:08 smithi187 ceph-mon[20072]: 6.1 scrub ok 2024-09-02T10:23:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:08 smithi187 ceph-mon[20072]: osdmap e643: 12 total, 12 up, 12 in 2024-09-02T10:23:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:08 smithi187 ceph-mon[20072]: pgmap v1487: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 98 objects/s recovering 2024-09-02T10:23:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:08 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:23:08.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:08 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:23:08.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:08.580 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:08.580 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:08 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:23:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:08 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:23:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:08 smithi162 ceph-mon[20279]: 6.1 scrub ok 2024-09-02T10:23:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:08 smithi162 ceph-mon[20279]: osdmap e643: 12 total, 12 up, 12 in 2024-09-02T10:23:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:08 smithi162 ceph-mon[20279]: pgmap v1487: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 98 objects/s recovering 2024-09-02T10:23:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:08 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:23:08.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:08 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:23:08.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:08 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:23:08.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:08 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:23:08.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:08 smithi151 ceph-mon[16639]: 6.1 scrub ok 2024-09-02T10:23:08.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:08 smithi151 ceph-mon[16639]: osdmap e643: 12 total, 12 up, 12 in 2024-09-02T10:23:08.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:08 smithi151 ceph-mon[16639]: pgmap v1487: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 98 objects/s recovering 2024-09-02T10:23:08.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:08 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:23:08.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:08 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:23:08.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:08.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:08.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:08.707 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:08.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:08.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:08.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:08.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:08.958 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:08.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:09.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:09.086 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:09.087 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:09.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:09.213 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:09.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:09.315 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:09.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:09.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:09.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:09.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:09.467 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:09.468 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:09 smithi187 ceph-mon[20072]: 6.2 scrub starts 2024-09-02T10:23:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:09 smithi187 ceph-mon[20072]: 6.2 scrub ok 2024-09-02T10:23:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:09 smithi187 ceph-mon[20072]: osdmap e644: 12 total, 12 up, 12 in 2024-09-02T10:23:09.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:09 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:23:09.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:09 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:23:09.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:09.595 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:09 smithi162 ceph-mon[20279]: 6.2 scrub starts 2024-09-02T10:23:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:09 smithi162 ceph-mon[20279]: 6.2 scrub ok 2024-09-02T10:23:09.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:09 smithi162 ceph-mon[20279]: osdmap e644: 12 total, 12 up, 12 in 2024-09-02T10:23:09.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:09 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:23:09.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:09 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:23:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:09.597 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:09.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:09 smithi151 ceph-mon[16639]: 6.2 scrub starts 2024-09-02T10:23:09.672 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:09 smithi151 ceph-mon[16639]: 6.2 scrub ok 2024-09-02T10:23:09.673 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:09 smithi151 ceph-mon[16639]: osdmap e644: 12 total, 12 up, 12 in 2024-09-02T10:23:09.673 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:09 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:23:09.673 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:09 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:23:09.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:09.723 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:09.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:09.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:23:09.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:09.850 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:09.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:09.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:09.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:09.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:09.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:10.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:10.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:10.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:10.148 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:10.178 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:23:10.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:23:10.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:10.230 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:10.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:10.332 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:10.358 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:10.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:10.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:10 smithi151 ceph-mon[16639]: pgmap v1489: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 110 objects/s recovering 2024-09-02T10:23:10.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:10 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:23:10.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:10 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:23:10.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:10.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:10.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:10.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:10.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:10 smithi187 ceph-mon[20072]: pgmap v1489: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 110 objects/s recovering 2024-09-02T10:23:10.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:10 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:23:10.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:10 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:23:10.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:10 smithi162 ceph-mon[20279]: pgmap v1489: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 110 objects/s recovering 2024-09-02T10:23:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:10 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:23:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:10 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:23:10.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:10.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:10.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:10.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:10.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:10.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:10.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:10.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:10.867 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:10.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:10.964 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:23:10.964 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:23:10.964 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:23:10.964 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:23:10.964 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:23:10.964 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:23:10.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:23:10.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:23:10.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:23:10.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:23:10.969 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:10.982 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:23:10.982 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:23:10.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:10.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:10.998 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:11.004 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:23:11.004 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:23:11.026 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:23:11.026 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:23:11.047 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:23:11.048 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:23:11.068 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:23:11.069 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:23:11.090 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:23:11.090 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:23:11.090 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:23:11.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:23:11.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:11.111 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:23:11.112 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:23:11.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:11.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:11.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:11.248 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:11.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:11.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:11 smithi151 ceph-mon[16639]: 6.e scrub starts 2024-09-02T10:23:11.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:11 smithi151 ceph-mon[16639]: 6.e scrub ok 2024-09-02T10:23:11.301 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/38427197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:11.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:11.376 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:11.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:11.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:11.503 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:11.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:11 smithi187 ceph-mon[20072]: 6.e scrub starts 2024-09-02T10:23:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:11 smithi187 ceph-mon[20072]: 6.e scrub ok 2024-09-02T10:23:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/38427197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:11.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:11 smithi162 ceph-mon[20279]: 6.e scrub starts 2024-09-02T10:23:11.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:11 smithi162 ceph-mon[20279]: 6.e scrub ok 2024-09-02T10:23:11.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/38427197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:11.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:11.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:11.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:11.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:11.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:11.756 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:11.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:11.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:11.883 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:11.884 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:11.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:11.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:23:12.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:12.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:12.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:12.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:12.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:12.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:12.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:12.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:12.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:12.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:12.386 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:12.387 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:12.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:12.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:12 smithi151 ceph-mon[16639]: pgmap v1490: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 91 objects/s recovering 2024-09-02T10:23:12.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:12 smithi151 ceph-mon[16639]: 6.3 scrub starts 2024-09-02T10:23:12.500 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:12 smithi151 ceph-mon[16639]: 6.3 scrub ok 2024-09-02T10:23:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:12 smithi187 ceph-mon[20072]: pgmap v1490: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 91 objects/s recovering 2024-09-02T10:23:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:12 smithi187 ceph-mon[20072]: 6.3 scrub starts 2024-09-02T10:23:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:12 smithi187 ceph-mon[20072]: 6.3 scrub ok 2024-09-02T10:23:12.513 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:12.514 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:12 smithi162 ceph-mon[20279]: pgmap v1490: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 91 objects/s recovering 2024-09-02T10:23:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:12 smithi162 ceph-mon[20279]: 6.3 scrub starts 2024-09-02T10:23:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:12 smithi162 ceph-mon[20279]: 6.3 scrub ok 2024-09-02T10:23:12.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:12.618 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:12.619 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:12.620 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:12.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:12.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:12.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:12.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:12.771 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:12.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:12.900 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:12.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:13.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:13.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:13.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:13.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:13.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:13.169 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:13.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:13.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:13.296 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:13.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:13.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:13.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:13.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:13.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:13.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:13.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:13.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:13.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:13.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:13.676 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:13.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:13.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:13.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:13.801 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:13.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:13.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:13.928 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:13.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:13.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:14.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:14.034 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:23:14.034 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:23:14.052 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:14.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:14.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:14.181 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:14.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:14.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:14.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:14.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:14.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:14.407 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:14.433 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:14.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:14.435 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:23:14.435 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:23:14.467 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:23:14.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:23:14.535 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:14.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:14 smithi151 ceph-mon[16639]: pgmap v1491: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 111 objects/s recovering 2024-09-02T10:23:14.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2898898737' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:23:14.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:14 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:23:14.555 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:14 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:23:14.561 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:14.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:14.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:14.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:14.688 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:14.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:14 smithi187 ceph-mon[20072]: pgmap v1491: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 111 objects/s recovering 2024-09-02T10:23:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2898898737' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:23:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:14 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:23:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:14 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:23:14.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:14.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:14.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:14 smithi162 ceph-mon[20279]: pgmap v1491: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 111 objects/s recovering 2024-09-02T10:23:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2898898737' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:23:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:14 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:23:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:14 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:23:14.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:14.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:14.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:14.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:15.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:15.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:15.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:15.165 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:15.191 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:15.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:15.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:15.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:15.319 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:15.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:15.421 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:15.447 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:15.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:15.549 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:15.575 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:15.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:15.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:15.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:15.704 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:15.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:15 smithi187 ceph-mon[20072]: from='client.17724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:15 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:23:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2898898737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:23:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:15 smithi187 ceph-mon[20072]: osdmap e645: 12 total, 12 up, 12 in 2024-09-02T10:23:15.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:15.827 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:15.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:15 smithi162 ceph-mon[20279]: from='client.17724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:15 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:23:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2898898737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:23:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:15 smithi162 ceph-mon[20279]: osdmap e645: 12 total, 12 up, 12 in 2024-09-02T10:23:15.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:15 smithi151 ceph-mon[16639]: from='client.17724 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:15 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:23:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2898898737' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:23:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:15 smithi151 ceph-mon[16639]: osdmap e645: 12 total, 12 up, 12 in 2024-09-02T10:23:15.955 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:15.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:16.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:16.085 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:16.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:16.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:16.210 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:16.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:16.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:16.336 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:16.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:16.438 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:16.464 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:16.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:16.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:16.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:16.591 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:16.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:16.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:16.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:16.722 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:16 smithi187 ceph-mon[20072]: pgmap v1493: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 111 objects/s recovering 2024-09-02T10:23:16.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:16.907 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:16 smithi162 ceph-mon[20279]: pgmap v1493: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 111 objects/s recovering 2024-09-02T10:23:16.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:16.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:16 smithi151 ceph-mon[16639]: pgmap v1493: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 111 objects/s recovering 2024-09-02T10:23:17.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:17.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:17.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:17.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:17.182 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:17.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:17.284 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:17.309 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:17.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:17.310 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:17.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:17.436 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:17.437 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:17.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:17.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:17.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:17.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:17.662 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:17.687 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:17.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:17.689 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:17.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:17.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:17.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:17.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:17.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:17.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:17.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:18.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:18.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:18.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:18.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:18.188 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:18.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:18.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:18.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:18.316 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:18.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:18.317 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:18.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:18.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:18.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:18.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:18.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:18.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:18.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:18.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:18.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:18.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:18.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:18 smithi187 ceph-mon[20072]: pgmap v1494: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 42 objects/s recovering 2024-09-02T10:23:18.791 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:18.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:18.817 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:18.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:18 smithi162 ceph-mon[20279]: pgmap v1494: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 42 objects/s recovering 2024-09-02T10:23:18.919 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:18.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:18 smithi151 ceph-mon[16639]: pgmap v1494: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 42 objects/s recovering 2024-09-02T10:23:18.945 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:18.946 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:19.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:19.074 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:19.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:19.176 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:19.201 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:19.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:19.202 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:19.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:19.328 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:19.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:19.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:19.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:19.455 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:19.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:19.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:19.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:19.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:19.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:19.708 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:19.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:19.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:19.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:19.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:19.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:19.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:19.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:23:19.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:19.962 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:19.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:19.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:19.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:20.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:20.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:20.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:20.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:20.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:20.237 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:20.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:20.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:20.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:20.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:20.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:20.398 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:20 smithi151 ceph-mon[16639]: osdmap e646: 12 total, 12 up, 12 in 2024-09-02T10:23:20.398 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:20 smithi151 ceph-mon[16639]: pgmap v1496: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 93 objects/s recovering 2024-09-02T10:23:20.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:20.493 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:20.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:20.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:20.495 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1491,"stamp":"2024-09-02T10:23:13.407711+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28878,"num_bytes_recovered":1287075424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163791,"ondisk_log_size":163791,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5866128,"kb_used_data":278808,"kb_used_omap":18,"kb_used_meta":5587053,"kb_avail":1119075696,"statfs":{"total":1151940427776,"available":1145933512704,"internally_reserved":0,"allocated":285499392,"data_stored":280726350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5721142587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":894,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006593"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":650,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213053+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:23:08.213053+0000","last_peered":"2024-09-02T10:23:08.213053+0000","last_clean":"2024-09-02T10:23:08.213053+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:23:08.213053+0000","last_undegraded":"2024-09-02T10:23:08.213053+0000","last_fullsized":"2024-09-02T10:23:08.213053+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":457,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212733+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:23:08.212733+0000","last_peered":"2024-09-02T10:23:08.212733+0000","last_clean":"2024-09-02T10:23:08.212733+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:23:08.212733+0000","last_undegraded":"2024-09-02T10:23:08.212733+0000","last_fullsized":"2024-09-02T10:23:08.212733+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":915,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213035+0000","last_change":"2024-09-02T10:22:32.860224+0000","last_active":"2024-09-02T10:23:08.213035+0000","last_peered":"2024-09-02T10:23:08.213035+0000","last_clean":"2024-09-02T10:23:08.213035+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:23:08.213035+0000","last_undegraded":"2024-09-02T10:23:08.213035+0000","last_fullsized":"2024-09-02T10:23:08.213035+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:32.860180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:32.860180+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.373890+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":907,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212675+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:23:08.212675+0000","last_peered":"2024-09-02T10:23:08.212675+0000","last_clean":"2024-09-02T10:23:08.212675+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:23:08.212675+0000","last_undegraded":"2024-09-02T10:23:08.212675+0000","last_fullsized":"2024-09-02T10:23:08.212675+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.838669+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":211,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212997+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:23:08.212997+0000","last_peered":"2024-09-02T10:23:08.212997+0000","last_clean":"2024-09-02T10:23:08.212997+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:23:08.212997+0000","last_undegraded":"2024-09-02T10:23:08.212997+0000","last_fullsized":"2024-09-02T10:23:08.212997+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":756,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.199889+0000","last_change":"2024-09-02T10:23:01.185167+0000","last_active":"2024-09-02T10:23:08.199889+0000","last_peered":"2024-09-02T10:23:08.199889+0000","last_clean":"2024-09-02T10:23:08.199889+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:23:08.199889+0000","last_undegraded":"2024-09-02T10:23:08.199889+0000","last_fullsized":"2024-09-02T10:23:08.199889+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.185112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.185112+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.992983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1291,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:09.691782+0000","last_change":"2024-09-02T10:23:09.691782+0000","last_active":"2024-09-02T10:23:09.691782+0000","last_peered":"2024-09-02T10:23:09.691782+0000","last_clean":"2024-09-02T10:23:09.691782+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:23:09.691782+0000","last_undegraded":"2024-09-02T10:23:09.691782+0000","last_fullsized":"2024-09-02T10:23:09.691782+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:09.691691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:09.691691+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.662853+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":930,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213121+0000","last_change":"2024-09-02T10:23:05.131265+0000","last_active":"2024-09-02T10:23:08.213121+0000","last_peered":"2024-09-02T10:23:08.213121+0000","last_clean":"2024-09-02T10:23:08.213121+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:23:08.213121+0000","last_undegraded":"2024-09-02T10:23:08.213121+0000","last_fullsized":"2024-09-02T10:23:08.213121+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.131213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.131213+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.083779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":922,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212597+0000","last_change":"2024-09-02T10:23:06.476818+0000","last_active":"2024-09-02T10:23:08.212597+0000","last_peered":"2024-09-02T10:23:08.212597+0000","last_clean":"2024-09-02T10:23:08.212597+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:23:08.212597+0000","last_undegraded":"2024-09-02T10:23:08.212597+0000","last_fullsized":"2024-09-02T10:23:08.212597+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:06.476768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.476768+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:35.403622+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":898,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213436+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:23:08.213436+0000","last_peered":"2024-09-02T10:23:08.213436+0000","last_clean":"2024-09-02T10:23:08.213436+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:23:08.213436+0000","last_undegraded":"2024-09-02T10:23:08.213436+0000","last_fullsized":"2024-09-02T10:23:08.213436+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:15.685318+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":143,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212607+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:23:08.212607+0000","last_peered":"2024-09-02T10:23:08.212607+0000","last_clean":"2024-09-02T10:23:08.212607+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:23:08.212607+0000","last_undegraded":"2024-09-02T10:23:08.212607+0000","last_fullsized":"2024-09-02T10:23:08.212607+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1247,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212491+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:23:08.212491+0000","last_peered":"2024-09-02T10:23:08.212491+0000","last_clean":"2024-09-02T10:23:08.212491+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:23:08.212491+0000","last_undegraded":"2024-09-02T10:23:08.212491+0000","last_fullsized":"2024-09-02T10:23:08.212491+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1190,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.199686+0000","last_change":"2024-09-02T10:22:55.516683+0000","last_active":"2024-09-02T10:23:08.199686+0000","last_peered":"2024-09-02T10:23:08.199686+0000","last_clean":"2024-09-02T10:23:08.199686+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:23:08.199686+0000","last_undegraded":"2024-09-02T10:23:08.199686+0000","last_fullsized":"2024-09-02T10:23:08.199686+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.516633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.516633+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:58.850702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1161,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212645+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:23:08.212645+0000","last_peered":"2024-09-02T10:23:08.212645+0000","last_clean":"2024-09-02T10:23:08.212645+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:23:08.212645+0000","last_undegraded":"2024-09-02T10:23:08.212645+0000","last_fullsized":"2024-09-02T10:23:08.212645+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1160,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.199683+0000","last_change":"2024-09-02T10:23:00.621966+0000","last_active":"2024-09-02T10:23:08.199683+0000","last_peered":"2024-09-02T10:23:08.199683+0000","last_clean":"2024-09-02T10:23:08.199683+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:23:08.199683+0000","last_undegraded":"2024-09-02T10:23:08.199683+0000","last_fullsized":"2024-09-02T10:23:08.199683+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.621916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.621916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:21.445915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"626'12394","reported_seq":20734,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212725+0000","last_change":"2024-09-02T10:22:56.637995+0000","last_active":"2024-09-02T10:23:08.212725+0000","last_peered":"2024-09-02T10:23:08.212725+0000","last_clean":"2024-09-02T10:23:08.212725+0000","last_became_active":"2024-09-02T10:22:56.595915+0000","last_became_peered":"2024-09-02T10:22:56.595915+0000","last_unstale":"2024-09-02T10:23:08.212725+0000","last_undegraded":"2024-09-02T10:23:08.212725+0000","last_fullsized":"2024-09-02T10:23:08.212725+0000","mapping_epoch":635,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12394,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":237961216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1254,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.199830+0000","last_change":"2024-09-02T10:23:01.892373+0000","last_active":"2024-09-02T10:23:08.199830+0000","last_peered":"2024-09-02T10:23:08.199830+0000","last_clean":"2024-09-02T10:23:08.199830+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:23:08.199830+0000","last_undegraded":"2024-09-02T10:23:08.199830+0000","last_fullsized":"2024-09-02T10:23:08.199830+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.892288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.892288+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:30.607091+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"626'12580","reported_seq":29698,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213099+0000","last_change":"2024-09-02T10:23:02.215418+0000","last_active":"2024-09-02T10:23:08.213099+0000","last_peered":"2024-09-02T10:23:08.213099+0000","last_clean":"2024-09-02T10:23:08.213099+0000","last_became_active":"2024-09-02T10:23:02.182530+0000","last_became_peered":"2024-09-02T10:23:02.182530+0000","last_unstale":"2024-09-02T10:23:08.213099+0000","last_undegraded":"2024-09-02T10:23:08.213099+0000","last_fullsized":"2024-09-02T10:23:08.213099+0000","mapping_epoch":640,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5293","last_scrub_stamp":"2024-09-02T10:19:55.677122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:55.677122+0000","objects_scrubbed":5264,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1480,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8401,"num_bytes_recovered":374538240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1377,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213159+0000","last_change":"2024-09-02T10:22:37.332903+0000","last_active":"2024-09-02T10:23:08.213159+0000","last_peered":"2024-09-02T10:23:08.213159+0000","last_clean":"2024-09-02T10:23:08.213159+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:23:08.213159+0000","last_undegraded":"2024-09-02T10:23:08.213159+0000","last_fullsized":"2024-09-02T10:23:08.213159+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:37.332859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:37.332859+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.675027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"626'12351","reported_seq":12654,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.199909+0000","last_change":"2024-09-02T10:23:04.675691+0000","last_active":"2024-09-02T10:23:08.199909+0000","last_peered":"2024-09-02T10:23:08.199909+0000","last_clean":"2024-09-02T10:23:08.199909+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:23:08.199909+0000","last_undegraded":"2024-09-02T10:23:08.199909+0000","last_fullsized":"2024-09-02T10:23:08.199909+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12351","last_scrub_stamp":"2024-09-02T10:23:04.675613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:04.675613+0000","objects_scrubbed":1,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.698140+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12351,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1020,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212917+0000","last_change":"2024-09-02T10:22:55.068228+0000","last_active":"2024-09-02T10:23:08.212917+0000","last_peered":"2024-09-02T10:23:08.212917+0000","last_clean":"2024-09-02T10:23:08.212917+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:23:08.212917+0000","last_undegraded":"2024-09-02T10:23:08.212917+0000","last_fullsized":"2024-09-02T10:23:08.212917+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.068166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.068166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:04.696428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"626'12382","reported_seq":12690,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-02T10:23:07.326976+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:23:07.326976+0000","last_peered":"2024-09-02T10:23:07.326976+0000","last_clean":"2024-09-02T10:23:07.326976+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:23:07.326976+0000","last_undegraded":"2024-09-02T10:23:07.326976+0000","last_fullsized":"2024-09-02T10:23:07.326976+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":417,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213194+0000","last_change":"2024-09-02T10:22:15.563077+0000","last_active":"2024-09-02T10:23:08.213194+0000","last_peered":"2024-09-02T10:23:08.213194+0000","last_clean":"2024-09-02T10:23:08.213194+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:23:08.213194+0000","last_undegraded":"2024-09-02T10:23:08.213194+0000","last_fullsized":"2024-09-02T10:23:08.213194+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:15.563022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:15.563022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:20.809417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"626'12606","reported_seq":12940,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:11.425512+0000","last_change":"2024-09-02T10:23:11.425512+0000","last_active":"2024-09-02T10:23:11.425512+0000","last_peered":"2024-09-02T10:23:11.425512+0000","last_clean":"2024-09-02T10:23:11.425512+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:23:11.425512+0000","last_undegraded":"2024-09-02T10:23:11.425512+0000","last_fullsized":"2024-09-02T10:23:11.425512+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12606","last_scrub_stamp":"2024-09-02T10:23:11.425432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:11.425432+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:12.626473+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1326,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212935+0000","last_change":"2024-09-02T10:22:34.647196+0000","last_active":"2024-09-02T10:23:08.212935+0000","last_peered":"2024-09-02T10:23:08.212935+0000","last_clean":"2024-09-02T10:23:08.212935+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:23:08.212935+0000","last_undegraded":"2024-09-02T10:23:08.212935+0000","last_fullsized":"2024-09-02T10:23:08.212935+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:34.647144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:34.647144+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:02.024298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"626'12544","reported_seq":29683,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213036+0000","last_change":"2024-09-02T10:23:07.704034+0000","last_active":"2024-09-02T10:23:08.213036+0000","last_peered":"2024-09-02T10:23:08.213036+0000","last_clean":"2024-09-02T10:23:08.213036+0000","last_became_active":"2024-09-02T10:22:57.600280+0000","last_became_peered":"2024-09-02T10:22:57.600280+0000","last_unstale":"2024-09-02T10:23:08.213036+0000","last_undegraded":"2024-09-02T10:23:08.213036+0000","last_fullsized":"2024-09-02T10:23:08.213036+0000","mapping_epoch":636,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12544","last_scrub_stamp":"2024-09-02T10:23:07.703956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:07.703956+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:08.077242+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8407,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":792,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.200015+0000","last_change":"2024-09-02T10:23:05.243285+0000","last_active":"2024-09-02T10:23:08.200015+0000","last_peered":"2024-09-02T10:23:08.200015+0000","last_clean":"2024-09-02T10:23:08.200015+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:23:08.200015+0000","last_undegraded":"2024-09-02T10:23:08.200015+0000","last_fullsized":"2024-09-02T10:23:08.200015+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.243236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.243236+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:16.091958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"626'12380","reported_seq":12716,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212855+0000","last_change":"2024-09-02T10:23:07.151869+0000","last_active":"2024-09-02T10:23:08.212855+0000","last_peered":"2024-09-02T10:23:08.212855+0000","last_clean":"2024-09-02T10:23:08.212855+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:23:08.212855+0000","last_undegraded":"2024-09-02T10:23:08.212855+0000","last_fullsized":"2024-09-02T10:23:08.212855+0000","mapping_epoch":533,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12380","last_scrub_stamp":"2024-09-02T10:23:07.151772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:07.151772+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:33.568723+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12380,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":757,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.200000+0000","last_change":"2024-09-02T10:22:55.086083+0000","last_active":"2024-09-02T10:23:08.200000+0000","last_peered":"2024-09-02T10:23:08.200000+0000","last_clean":"2024-09-02T10:23:08.200000+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:23:08.200000+0000","last_undegraded":"2024-09-02T10:23:08.200000+0000","last_fullsized":"2024-09-02T10:23:08.200000+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.086036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.086036+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.714243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"626'12678","reported_seq":16128,"reported_epoch":644,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:23:13.002045+0000","last_change":"2024-09-02T10:22:55.007293+0000","last_active":"2024-09-02T10:23:13.002045+0000","last_peered":"2024-09-02T10:23:13.002045+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:23:13.002045+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1577,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-4089}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":517,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.200376+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:23:08.200376+0000","last_peered":"2024-09-02T10:23:08.200376+0000","last_clean":"2024-09-02T10:23:08.200376+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:23:08.200376+0000","last_undegraded":"2024-09-02T10:23:08.200376+0000","last_fullsized":"2024-09-02T10:23:08.200376+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:16.363558+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"626'12542","reported_seq":17193,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-02T10:23:07.326926+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:23:07.326926+0000","last_peered":"2024-09-02T10:23:07.326926+0000","last_clean":"2024-09-02T10:23:07.326926+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:23:07.326926+0000","last_undegraded":"2024-09-02T10:23:07.326926+0000","last_fullsized":"2024-09-02T10:23:07.326926+0000","mapping_epoch":559,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12542,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":758,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.199941+0000","last_change":"2024-09-02T10:23:00.146326+0000","last_active":"2024-09-02T10:23:08.199941+0000","last_peered":"2024-09-02T10:23:08.199941+0000","last_clean":"2024-09-02T10:23:08.199941+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:23:08.199941+0000","last_undegraded":"2024-09-02T10:23:08.199941+0000","last_fullsized":"2024-09-02T10:23:08.199941+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.146278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.146278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:09.373850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"626'12632","reported_seq":12971,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213347+0000","last_change":"2024-09-02T10:22:59.338566+0000","last_active":"2024-09-02T10:23:08.213347+0000","last_peered":"2024-09-02T10:23:08.213347+0000","last_clean":"2024-09-02T10:23:08.213347+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:23:08.213347+0000","last_undegraded":"2024-09-02T10:23:08.213347+0000","last_fullsized":"2024-09-02T10:23:08.213347+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12632","last_scrub_stamp":"2024-09-02T10:22:59.338517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:22:59.338517+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.748681+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1011,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213265+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:23:08.213265+0000","last_peered":"2024-09-02T10:23:08.213265+0000","last_clean":"2024-09-02T10:23:08.213265+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:23:08.213265+0000","last_undegraded":"2024-09-02T10:23:08.213265+0000","last_fullsized":"2024-09-02T10:23:08.213265+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"626'12586","reported_seq":12880,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212489+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:23:08.212489+0000","last_peered":"2024-09-02T10:23:08.212489+0000","last_clean":"2024-09-02T10:23:08.212489+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:23:08.212489+0000","last_undegraded":"2024-09-02T10:23:08.212489+0000","last_fullsized":"2024-09-02T10:23:08.212489+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12586,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1279,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.500316+0000","last_change":"2024-09-02T10:23:08.500316+0000","last_active":"2024-09-02T10:23:08.500316+0000","last_peered":"2024-09-02T10:23:08.500316+0000","last_clean":"2024-09-02T10:23:08.500316+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:23:08.500316+0000","last_undegraded":"2024-09-02T10:23:08.500316+0000","last_fullsized":"2024-09-02T10:23:08.500316+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:08.500268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:08.500268+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:14.723023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"626'12632","reported_seq":17183,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.199967+0000","last_change":"2024-09-02T10:23:02.673893+0000","last_active":"2024-09-02T10:23:08.199967+0000","last_peered":"2024-09-02T10:23:08.199967+0000","last_clean":"2024-09-02T10:23:08.199967+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:23:08.199967+0000","last_undegraded":"2024-09-02T10:23:08.199967+0000","last_fullsized":"2024-09-02T10:23:08.199967+0000","mapping_epoch":565,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12632","last_scrub_stamp":"2024-09-02T10:23:02.673828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:02.673828+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:31.609539+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1023,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213247+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:23:08.213247+0000","last_peered":"2024-09-02T10:23:08.213247+0000","last_clean":"2024-09-02T10:23:08.213247+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:23:08.213247+0000","last_undegraded":"2024-09-02T10:23:08.213247+0000","last_fullsized":"2024-09-02T10:23:08.213247+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"626'12650","reported_seq":12985,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213108+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:23:08.213108+0000","last_peered":"2024-09-02T10:23:08.213108+0000","last_clean":"2024-09-02T10:23:08.213108+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:23:08.213108+0000","last_undegraded":"2024-09-02T10:23:08.213108+0000","last_fullsized":"2024-09-02T10:23:08.213108+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12650,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1286,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.200376+0000","last_change":"2024-09-02T10:23:00.386934+0000","last_active":"2024-09-02T10:23:08.200376+0000","last_peered":"2024-09-02T10:23:08.200376+0000","last_clean":"2024-09-02T10:23:08.200376+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:23:08.200376+0000","last_undegraded":"2024-09-02T10:23:08.200376+0000","last_fullsized":"2024-09-02T10:23:08.200376+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.386875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.386875+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.366845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"626'12496","reported_seq":12795,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.212411+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:23:08.212411+0000","last_peered":"2024-09-02T10:23:08.212411+0000","last_clean":"2024-09-02T10:23:08.212411+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:23:08.212411+0000","last_undegraded":"2024-09-02T10:23:08.212411+0000","last_fullsized":"2024-09-02T10:23:08.212411+0000","mapping_epoch":533,"log_start":"553'2400","ondisk_log_start":"553'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":226,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213073+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:23:08.213073+0000","last_peered":"2024-09-02T10:23:08.213073+0000","last_clean":"2024-09-02T10:23:08.213073+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:23:08.213073+0000","last_undegraded":"2024-09-02T10:23:08.213073+0000","last_fullsized":"2024-09-02T10:23:08.213073+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"626'12520","reported_seq":12830,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.199713+0000","last_change":"2024-09-02T10:23:06.653266+0000","last_active":"2024-09-02T10:23:08.199713+0000","last_peered":"2024-09-02T10:23:08.199713+0000","last_clean":"2024-09-02T10:23:08.199713+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:23:08.199713+0000","last_undegraded":"2024-09-02T10:23:08.199713+0000","last_fullsized":"2024-09-02T10:23:08.199713+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12520","last_scrub_stamp":"2024-09-02T10:23:06.653191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.653191+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:17.096908+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12520,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1058,"reported_epoch":644,"state":"active+clean","last_fresh":"2024-09-02T10:23:08.213457+0000","last_change":"2024-09-02T10:22:30.358040+0000","last_active":"2024-09-02T10:23:08.213457+0000","last_peered":"2024-09-02T10:23:08.213457+0000","last_clean":"2024-09-02T10:23:08.213457+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:23:08.213457+0000","last_undegraded":"2024-09-02T10:23:08.213457+0000","last_fullsized":"2024-09-02T10:23:08.213457+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:30.357970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:30.357970+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.221263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"626'12566","reported_seq":12862,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-02T10:23:07.326883+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:23:07.326883+0000","last_peered":"2024-09-02T10:23:07.326883+0000","last_clean":"2024-09-02T10:23:07.326883+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:23:07.326883+0000","last_undegraded":"2024-09-02T10:23:07.326883+0000","last_fullsized":"2024-09-02T10:23:07.326883+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":438,"reported_epoch":643,"state":"active+clean","last_fresh":"2024-09-02T10:23:07.327032+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:23:07.327032+0000","last_peered":"2024-09-02T10:23:07.327032+0000","last_clean":"2024-09-02T10:23:07.327032+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:23:07.327032+0000","last_undegraded":"2024-09-02T10:23:07.327032+0000","last_fullsized":"2024-09-02T10:23:07.327032+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4089,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28874,"num_bytes_recovered":1286144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270938112,"data_stored":270925896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383957,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487608,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93257544,"statfs":{"total":95995035648,"available":95495725056,"internally_reserved":0,"allocated":1081344,"data_stored":685132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":498203076},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612890,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357116,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93388036,"statfs":{"total":95995035648,"available":95629348864,"internally_reserved":0,"allocated":1085440,"data_stored":685156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841823,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545908,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199244,"statfs":{"total":95995035648,"available":95436025856,"internally_reserved":0,"allocated":1081344,"data_stored":685132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070756,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339060,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93406092,"statfs":{"total":95995035648,"available":95647838208,"internally_reserved":0,"allocated":1081344,"data_stored":685132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299690,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582204,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162948,"statfs":{"total":95995035648,"available":95398858752,"internally_reserved":0,"allocated":1286144,"data_stored":886380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528621,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365752,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379400,"statfs":{"total":95995035648,"available":95620505600,"internally_reserved":0,"allocated":1409024,"data_stored":1012812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724850,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":878716,"kb_used_data":265836,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92866436,"statfs":{"total":95995035648,"available":95095230464,"internally_reserved":0,"allocated":272216064,"data_stored":271812228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953783,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334904,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410248,"statfs":{"total":95995035648,"available":95652093952,"internally_reserved":0,"allocated":1081344,"data_stored":685132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150012,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569832,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175320,"statfs":{"total":95995035648,"available":95411527680,"internally_reserved":0,"allocated":1396736,"data_stored":1004898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411648,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303160,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93441992,"statfs":{"total":95995035648,"available":95684599808,"internally_reserved":0,"allocated":1286144,"data_stored":886380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575173,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509816,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235336,"statfs":{"total":95995035648,"available":95472984064,"internally_reserved":0,"allocated":1085440,"data_stored":685156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738700,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592052,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93153100,"statfs":{"total":95995035648,"available":95388774400,"internally_reserved":0,"allocated":1409024,"data_stored":1012812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":270929920,"data_stored":270925848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:23:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:20 smithi187 ceph-mon[20072]: osdmap e646: 12 total, 12 up, 12 in 2024-09-02T10:23:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:20 smithi187 ceph-mon[20072]: pgmap v1496: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 93 objects/s recovering 2024-09-02T10:23:20.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:20 smithi162 ceph-mon[20279]: osdmap e646: 12 total, 12 up, 12 in 2024-09-02T10:23:20.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:20 smithi162 ceph-mon[20279]: pgmap v1496: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 93 objects/s recovering 2024-09-02T10:23:20.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:20.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:20.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:20.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:20.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:20.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:20.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:20.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:20.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:20.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:20.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:20.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:20.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:21.065 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:23:21.066 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:23:21.078 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:21.093 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:23:21.093 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:23:21.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:21.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:23:21.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:23:21.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:21.125 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:21.138 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:23:21.138 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:23:21.161 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:23:21.161 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:23:21.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:23:21.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:23:21.204 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/544241357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:21.206 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:23:21.206 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:23:21.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:21.229 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:23:21.229 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:23:21.251 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:23:21.251 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:23:21.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:21.253 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:21.273 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:23:21.273 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:23:21.295 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:23:21.295 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:23:21.316 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:23:21.317 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:23:21.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:21.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:21.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:21.480 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:21.504 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:21.505 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/544241357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:21.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/544241357' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:21.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:21.633 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:21.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:21.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:21.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:23:21.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:21.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:21.760 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:23:21.760 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(5.8 479/463 [PushReplyOp(5:1936829f:::benchmark_data_smithi187_89405_object32750:head)])", 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 425.57960861700002, 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36677579799999999, 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178505+0000", 2024-09-02T10:23:21.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9100000000000004e-07 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.178510+0000", 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.51e-06 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.543260+0000", 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36475006799999998 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.543276+0000", 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5837000000000001e-05 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.543392+0000", 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115862 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:16.545280+0000", 2024-09-02T10:23:21.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00188893 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.25933.0:205325 5.e e524/431 5:726c3a45:::benchmark_data_smithi187_89405_object101720:head v 524'12690, mlcod=524'12688)", 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.618253+0000", 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 304.13986025499997, 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38099444199999999, 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618253+0000", 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618253+0000", 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618254+0000", 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1600000000000001e-07 2024-09-02T10:23:21.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.618258+0000", 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3320000000000002e-06 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995375+0000", 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37711745899999999 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995380+0000", 2024-09-02T10:23:21.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7330000000000003e-06 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995450+0000", 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9474000000000006e-05 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.995475+0000", 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5579999999999999e-05 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.999213+0000", 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003737476 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.999247+0000", 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4771999999999998e-05 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:207192 5.3 5:c539d868:::benchmark_data_smithi187_89405_object103587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:20.104200+0000", 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 301.65391355999998, 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52651428300000003, 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 207192 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.770 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104200+0000", 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104200+0000", 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104201+0000", 2024-09-02T10:23:21.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4399999999999999e-07 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104205+0000", 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.98e-06 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104213+0000", 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7560000000000005e-06 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104218+0000", 2024-09-02T10:23:21.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104263+0000", 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4708e-05 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104392+0000", 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128987 2024-09-02T10:23:21.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.104522+0000", 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012982799999999999 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.109243+0000", 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047210799999999999 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.109411+0000", 2024-09-02T10:23:21.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016770899999999999 2024-09-02T10:23:21.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630621+0000", 2024-09-02T10:23:21.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52120988899999998 2024-09-02T10:23:21.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630666+0000", 2024-09-02T10:23:21.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4883999999999999e-05 2024-09-02T10:23:21.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630714+0000", 2024-09-02T10:23:21.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8497e-05 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.25933.0:207192 5.3 e525/431 ondisk, result = 0)", 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:20.108691+0000", 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 301.64942185699999, 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52202865700000001, 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:23:21.777 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108691+0000", 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108691+0000", 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108692+0000", 2024-09-02T10:23:21.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9100000000000004e-07 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.108695+0000", 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.529e-06 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630413+0000", 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.52171714700000005 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630428+0000", 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5843999999999999e-05 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630579+0000", 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000150252 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630598+0000", 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8865999999999999e-05 2024-09-02T10:23:21.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:20.630720+0000", 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122528 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:19723 6.9 6:94e8ee31:::benchmark_data_smithi187_102448_object19722:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:53.705822+0000", 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 268.05229097699998, 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21665701200000001, 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.781 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 19723 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705822+0000", 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705822+0000", 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705823+0000", 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.082e-06 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705916+0000", 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2105e-05 2024-09-02T10:23:21.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705929+0000", 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.397e-05 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705937+0000", 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7330000000000003e-06 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.705981+0000", 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3949999999999998e-05 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.706119+0000", 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013788900000000001 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.706208+0000", 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9197000000000007e-05 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.707009+0000", 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00080078799999999998 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.714056+0000", 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0070474170000000003 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922382+0000", 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20832547500000001 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922434+0000", 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2473999999999999e-05 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.922479+0000", 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4932000000000003e-05 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.788 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:19732 6.9 6:9f276e57:::benchmark_data_smithi187_102448_object19731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:23:21.788 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:53.712327+0000", 2024-09-02T10:23:21.788 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 268.04578662400002, 2024-09-02T10:23:21.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.216236916, 2024-09-02T10:23:21.788 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.788 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.788 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.788 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.788 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 19732 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712327+0000", 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712327+0000", 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712328+0000", 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0449999999999999e-06 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712379+0000", 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0779999999999998e-05 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712391+0000", 2024-09-02T10:23:21.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2085e-05 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712398+0000", 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7540000000000006e-06 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712448+0000", 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9880999999999999e-05 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712542+0000", 2024-09-02T10:23:21.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3450999999999997e-05 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.712634+0000", 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2317000000000004e-05 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.713395+0000", 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00076103200000000003 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.787692+0000", 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.074297236000000003 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.928492+0000", 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.14080002899999999 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.928534+0000", 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1220000000000002e-05 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:53.928564+0000", 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0086e-05 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:39567 6.0 6:0ea24a11:::benchmark_data_smithi187_102448_object39566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e553)", 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:11.107391+0000", 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 250.650721965, 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26867963700000003, 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.794 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 39567 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107391+0000", 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107391+0000", 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107393+0000", 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.341e-06 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107430+0000", 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7738999999999999e-05 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107447+0000", 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6472000000000001e-05 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107456+0000", 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6279999999999994e-06 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107520+0000", 2024-09-02T10:23:21.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4442000000000004e-05 2024-09-02T10:23:21.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107677+0000", 2024-09-02T10:23:21.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015673999999999999 2024-09-02T10:23:21.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.107865+0000", 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018834400000000001 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.109307+0000", 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014422720000000001 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.109639+0000", 2024-09-02T10:23:21.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033185300000000001 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.375963+0000", 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.266324318 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.376022+0000", 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8971999999999997e-05 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:11.376071+0000", 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8516000000000001e-05 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:54769 6.9 6:9e64a741:::benchmark_data_smithi187_102448_object54768:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:24.601031+0000", 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 237.15708225500001, 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21469951200000001, 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.801 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 54769 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601031+0000", 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601031+0000", 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601032+0000", 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3960000000000001e-06 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601100+0000", 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7464000000000003e-05 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601111+0000", 2024-09-02T10:23:21.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1240000000000001e-05 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601119+0000", 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4739999999999999e-06 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601187+0000", 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8659999999999997e-05 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601349+0000", 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000161991 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601506+0000", 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000157009 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.605407+0000", 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039008810000000001 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.605419+0000", 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1984000000000001e-05 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815640+0000", 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21022085700000001 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815682+0000", 2024-09-02T10:23:21.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2066e-05 2024-09-02T10:23:21.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815731+0000", 2024-09-02T10:23:21.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8489999999999998e-05 2024-09-02T10:23:21.808 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.808 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.808 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:54772 6.9 6:97f3a42a:::benchmark_data_smithi187_102448_object54771:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e556)", 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:19:24.601836+0000", 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 237.15627717800001, 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21406335900000001, 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 54772 2024-09-02T10:23:21.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601836+0000", 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601836+0000", 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601837+0000", 2024-09-02T10:23:21.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6300000000000004e-07 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601910+0000", 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2541999999999998e-05 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601924+0000", 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4063000000000001e-05 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.601931+0000", 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0509999999999999e-06 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.603340+0000", 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001409039 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.603478+0000", 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013832499999999999 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.603655+0000", 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017739000000000001 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.605744+0000", 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020888360000000002 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.606900+0000", 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001155339 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815803+0000", 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20890336700000001 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815861+0000", 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8230999999999999e-05 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:19:24.815899+0000", 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8312999999999997e-05 2024-09-02T10:23:21.814 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:135986 6.0 6:053ae704:::benchmark_data_smithi187_102448_object35714:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135293+0000", 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.622820233, 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32009341200000002, 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135986 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135293+0000", 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135293+0000", 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135295+0000", 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.894e-06 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135302+0000", 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8249999999999999e-06 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135316+0000", 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4620000000000001e-05 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135326+0000", 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5759999999999995e-06 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136831+0000", 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015045989999999999 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136843+0000", 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2804e-05 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.435574+0000", 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29873042100000002 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.435695+0000", 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012073 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.435805+0000", 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011039099999999999 2024-09-02T10:23:21.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.442308+0000", 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0065034020000000001 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.447422+0000", 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051138219999999996 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.452276+0000", 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048543129999999999 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.452357+0000", 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0165000000000002e-05 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455386+0000", 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030298500000000002 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:135994 6.0 6:0fdb7607:::benchmark_data_smithi187_102448_object35722:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135349+0000", 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.62276395800001, 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34087867300000002, 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135994 2024-09-02T10:23:21.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135349+0000", 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135349+0000", 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135350+0000", 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0400000000000002e-07 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135352+0000", 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3489999999999999e-06 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135357+0000", 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3770000000000004e-06 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135361+0000", 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3759999999999999e-06 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136882+0000", 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015210169999999999 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136890+0000", 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.225e-06 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.442387+0000", 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.30549738100000001 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.442561+0000", 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017320399999999999 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.442754+0000", 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019324399999999999 2024-09-02T10:23:21.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455410+0000", 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.012656499 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455455+0000", 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4912e-05 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.472639+0000", 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017184003999999999 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.472712+0000", 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3100000000000001e-05 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.476228+0000", 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035155809999999998 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.828 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:136020 6.0 6:06e73c6f:::benchmark_data_smithi187_102448_object35748:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135365+0000", 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.62274787000001, 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34402223500000001, 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136020 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.829 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135365+0000", 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135365+0000", 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135366+0000", 2024-09-02T10:23:21.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3999999999999998e-07 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135367+0000", 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.367e-06 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135372+0000", 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5120000000000002e-06 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.831 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135375+0000", 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9809999999999999e-06 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136925+0000", 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015500749999999999 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136931+0000", 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2260000000000002e-06 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.447464+0000", 2024-09-02T10:23:21.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31053328699999999 2024-09-02T10:23:21.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.447637+0000", 2024-09-02T10:23:21.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017254299999999999 2024-09-02T10:23:21.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:23:21.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.447810+0000", 2024-09-02T10:23:21.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000173623 2024-09-02T10:23:21.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455501+0000", 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0076903800000000001 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455514+0000", 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3022e-05 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.476298+0000", 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.020784764000000001 2024-09-02T10:23:21.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.476368+0000", 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9745000000000004e-05 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.479388+0000", 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030194699999999998 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:136029 6.0 6:0a2545ae:::benchmark_data_smithi187_102448_object35757:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.62273516299999, 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34708892600000002, 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136029 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1500000000000001e-07 2024-09-02T10:23:21.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135380+0000", 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2359999999999999e-06 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135383+0000", 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.641e-06 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135386+0000", 2024-09-02T10:23:21.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7920000000000001e-06 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136954+0000", 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001567492 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136959+0000", 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.49e-06 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455542+0000", 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31858257099999998 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455661+0000", 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119422 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455779+0000", 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011832699999999999 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.464952+0000", 2024-09-02T10:23:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091722350000000008 2024-09-02T10:23:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.465554+0000", 2024-09-02T10:23:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060224400000000004 2024-09-02T10:23:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.842 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.479463+0000", 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013909425 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.479532+0000", 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8818000000000004e-05 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.482467+0000", 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029350180000000002 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.843 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:136038 6.0 6:0e293183:::benchmark_data_smithi187_102448_object35766:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135389+0000", 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 148.622723925, 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34942438399999998, 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.844 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136038 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135389+0000", 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135389+0000", 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135390+0000", 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1299999999999999e-07 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135391+0000", 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.249e-06 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135395+0000", 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2150000000000003e-06 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135399+0000", 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4649999999999999e-06 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.137534+0000", 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021359560000000001 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.137545+0000", 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0312999999999999e-05 2024-09-02T10:23:21.847 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.460231+0000", 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32268618199999999 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.460387+0000", 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015636900000000001 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.848 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.460536+0000", 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014892599999999999 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.465596+0000", 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050595570000000001 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.465617+0000", 2024-09-02T10:23:21.849 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1362e-05 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.482532+0000", 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016914565999999999 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.482602+0000", 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9809999999999997e-05 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.850 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.484814+0000", 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002212201 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:144155 6.a e590/533 6:53bb1869:::benchmark_data_smithi187_102448_object43883:head v 590'8984, mlcod=590'8979)", 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:06.775838+0000", 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 134.982275112, 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.214603619, 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.851 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.775838+0000", 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.775838+0000", 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.775839+0000", 2024-09-02T10:23:21.852 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2500000000000004e-07 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.775844+0000", 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2599999999999996e-06 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.981434+0000", 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20558944700000001 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.853 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.981442+0000", 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3999999999999992e-06 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.981528+0000", 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5721000000000001e-05 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.981551+0000", 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3544999999999999e-05 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.854 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.990342+0000", 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087902310000000008 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.990442+0000", 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010019 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:144161 6.a e590/533 6:52a794b3:::benchmark_data_smithi187_102448_object43889:head v 590'8985, mlcod=590'8980)", 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:06.783206+0000", 2024-09-02T10:23:21.855 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 134.974907331, 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.212957436, 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.783206+0000", 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.783206+0000", 2024-09-02T10:23:21.856 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.783207+0000", 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7199999999999998e-07 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.783212+0000", 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4829999999999999e-06 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.857 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.991155+0000", 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20794227500000001 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.991163+0000", 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3289999999999994e-06 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.991223+0000", 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0111000000000001e-05 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.858 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.991244+0000", 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1484999999999998e-05 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.996076+0000", 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048314170000000002 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:06.996163+0000", 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7564000000000004e-05 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.859 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157046 6.0 6:0d4250f7:::benchmark_data_smithi187_102448_object56774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e597)", 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.309407+0000", 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 114.448705824, 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20892961199999999, 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157046 2024-09-02T10:23:21.860 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.309407+0000", 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.309407+0000", 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.309410+0000", 2024-09-02T10:23:21.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.119e-06 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.309418+0000", 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1459999999999996e-06 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.473327+0000", 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.163909369 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.473350+0000", 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3249000000000001e-05 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.476461+0000", 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003110416 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.476641+0000", 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000179968 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.476812+0000", 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017107900000000001 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.498766+0000", 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.021954195999999999 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.498871+0000", 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010496 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.515406+0000", 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016535065000000002 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.515480+0000", 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3497000000000005e-05 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.518337+0000", 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0028575480000000001 2024-09-02T10:23:21.865 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.26524.0:157834 6.a e599/533 6:58679a78:::benchmark_data_smithi187_102448_object57562:head v 599'9829, mlcod=599'9827)", 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:29.325820+0000", 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 112.43229321699999, 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.222988454, 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.325820+0000", 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.325820+0000", 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.325821+0000", 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2580000000000001e-06 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.325830+0000", 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5129999999999996e-06 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.543764+0000", 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21793457899999999 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.543773+0000", 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6899999999999998e-06 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.543837+0000", 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3645000000000005e-05 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.543857+0000", 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0611e-05 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.548702+0000", 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048444430000000004 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:29.548809+0000", 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000106715 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e640,586 6:01803c4b:::benchmark_data_smithi187_102448_object10256:head,576'6972)", 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:02.039244+0000", 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 19.718868875999998, 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.89444717100000004, 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.870 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.039244+0000", 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.039244+0000", 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.039246+0000", 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.077e-06 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.039250+0000", 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4429999999999996e-06 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.931762+0000", 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.89251206999999999 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.931772+0000", 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0360000000000001e-05 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.931837+0000", 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4980999999999993e-05 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.933692+0000", 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018542400000000001 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.873 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e641,586 6:039db2d1:::benchmark_data_smithi187_102448_object12140:head,577'7087)", 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:04.102791+0000", 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 17.655322362, 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.252130301, 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:04.102791+0000", 2024-09-02T10:23:21.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:04.102791+0000", 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:04.102792+0000", 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2799999999999995e-07 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:04.102797+0000", 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3619999999999997e-06 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:04.352880+0000", 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25008253000000003 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:04.352887+0000", 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.452e-06 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:04.352966+0000", 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8535999999999995e-05 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:04.354921+0000", 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019555929999999998 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:23:21.877 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:23:21.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:21.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:21.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:22.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:22.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:22.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:22.128 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:22.153 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:22.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:22.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:22.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:22.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:22.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:22.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:22.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:22.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:22.431 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:22 smithi187 ceph-mon[20072]: pgmap v1497: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 44 objects/s recovering 2024-09-02T10:23:22.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:22 smithi151 ceph-mon[16639]: pgmap v1497: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 44 objects/s recovering 2024-09-02T10:23:22.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:22 smithi162 ceph-mon[20279]: pgmap v1497: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4089/9 objects degraded (45433.333%); 0 B/s, 44 objects/s recovering 2024-09-02T10:23:22.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:22.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:22.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:22.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:22.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:22.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:22.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:22.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:22.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:22.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:22.776 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:22.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:22.900 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:22.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:23.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:23.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:23.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:23.059 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:23:23.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:23.147 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:23.148 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:23.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:23.269 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:23.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:23.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:23.392 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:23.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:23.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:23.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:23.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:23:23.515 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:23.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:23.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:23.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:23.638 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:23.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:23.739 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:23.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:23.762 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:23 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 3953/9 objects degraded (43922.222%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:23.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:23.883 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:23.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:23.884 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:23 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 3953/9 objects degraded (43922.222%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:23.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:23 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 3953/9 objects degraded (43922.222%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:24.007 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:24.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:24.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:24.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:24.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:24.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:24.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:24.255 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:24.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:24.357 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:24.379 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:24.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:24.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:24.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:24.503 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:24.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:24.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:24.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:24.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:24.629 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:24.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:24.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:24.754 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:24 smithi162 ceph-mon[20279]: pgmap v1498: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3953/9 objects degraded (43922.222%); 0 B/s, 90 objects/s recovering 2024-09-02T10:23:24.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:24.881 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:24.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:24 smithi151 ceph-mon[16639]: pgmap v1498: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3953/9 objects degraded (43922.222%); 0 B/s, 90 objects/s recovering 2024-09-02T10:23:24.983 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:24 smithi187 ceph-mon[20072]: pgmap v1498: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3953/9 objects degraded (43922.222%); 0 B/s, 90 objects/s recovering 2024-09-02T10:23:25.008 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:25.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:25.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:25.134 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:25.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:25.236 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:25.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:25.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:25.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:25.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:25.389 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:25.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:25.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:25.516 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:25.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:25.618 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:25.643 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:25.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:25.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:25.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:25.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:25.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:25.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:25.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:25.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:25.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:26.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:26.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:26.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:26.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:26.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:26.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:26.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:26.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:26.275 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:26.313 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:26.314 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:26.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:26.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:26.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:26.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:26.503 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:26.529 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:26.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:26.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:26.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:26.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:26.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:26.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:26 smithi151 ceph-mon[16639]: pgmap v1499: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3953/9 objects degraded (43922.222%); 0 B/s, 80 objects/s recovering 2024-09-02T10:23:26.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:26.781 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:26.782 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:26 smithi162 ceph-mon[20279]: pgmap v1499: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3953/9 objects degraded (43922.222%); 0 B/s, 80 objects/s recovering 2024-09-02T10:23:26.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:26.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:26.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:27.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:27.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:26 smithi187 ceph-mon[20072]: pgmap v1499: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3953/9 objects degraded (43922.222%); 0 B/s, 80 objects/s recovering 2024-09-02T10:23:27.026 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:23:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:23:27.027 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:23:27.027 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:23:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:27.029 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:27.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:27.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:27.156 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:27.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:27.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:27.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:27.284 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:27.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:27.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:27.411 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:27.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:27.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:27.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:27.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:27.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:27.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:27.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:27.760 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:27.786 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:27.787 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:27.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:27.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:27.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:28.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:28.036 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:28.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:28.038 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:28.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:28.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:28.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:28.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:23:28.262 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:28.288 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:28.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:28.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:23:28.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:28.415 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:28.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:28.469 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:23:28.470 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:23:28.517 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:28.543 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:28.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:28.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:28.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:28.673 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:28.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:28.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:28.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:28 smithi151 ceph-mon[16639]: pgmap v1500: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3953/9 objects degraded (43922.222%); 0 B/s, 80 objects/s recovering 2024-09-02T10:23:28.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:28 smithi151 ceph-mon[16639]: from='client.17733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:28.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:28 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:23:28.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:28 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:23:28.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:28.802 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:28 smithi162 ceph-mon[20279]: pgmap v1500: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3953/9 objects degraded (43922.222%); 0 B/s, 80 objects/s recovering 2024-09-02T10:23:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:28 smithi162 ceph-mon[20279]: from='client.17733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:28 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:23:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:28 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:23:28.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:28.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:28.929 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:28.929 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:28 smithi187 ceph-mon[20072]: pgmap v1500: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3953/9 objects degraded (43922.222%); 0 B/s, 80 objects/s recovering 2024-09-02T10:23:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:28 smithi187 ceph-mon[20072]: from='client.17733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:28 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:23:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:28 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:23:29.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:29.057 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:29.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:29.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:29.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:29.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:29.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:29.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:29.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:29.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:29.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:29.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:29.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:29.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:29.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:29.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:29.664 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:29.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:29.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:29.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:29.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:29.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:29.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:29 smithi162 ceph-mon[20279]: osdmap e647: 12 total, 12 up, 12 in 2024-09-02T10:23:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:29 smithi162 ceph-mon[20279]: osdmap e648: 12 total, 12 up, 12 in 2024-09-02T10:23:29.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:29 smithi151 ceph-mon[16639]: osdmap e647: 12 total, 12 up, 12 in 2024-09-02T10:23:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:29 smithi151 ceph-mon[16639]: osdmap e648: 12 total, 12 up, 12 in 2024-09-02T10:23:29.941 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:29.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:29.942 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:29 smithi187 ceph-mon[20072]: osdmap e647: 12 total, 12 up, 12 in 2024-09-02T10:23:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:29 smithi187 ceph-mon[20072]: osdmap e648: 12 total, 12 up, 12 in 2024-09-02T10:23:30.043 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:30.069 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:30.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:30.070 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:30.171 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:30.196 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:30.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:30.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:30.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:30.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:30.326 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:30.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:30.452 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:30.454 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:30.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:30.580 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:30.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:30.617 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1500,"stamp":"2024-09-02T10:23:27.411025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29687,"num_bytes_recovered":1287075424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163791,"ondisk_log_size":163791,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5874736,"kb_used_data":279108,"kb_used_omap":18,"kb_used_meta":5595373,"kb_avail":1119067088,"statfs":{"total":1151940427776,"available":1145924698112,"internally_reserved":0,"allocated":285806592,"data_stored":280923600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5729662267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":809,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002496"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":654,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556751+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:23:19.556751+0000","last_peered":"2024-09-02T10:23:19.556751+0000","last_clean":"2024-09-02T10:23:19.556751+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:23:19.556751+0000","last_undegraded":"2024-09-02T10:23:19.556751+0000","last_fullsized":"2024-09-02T10:23:19.556751+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":461,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.164091+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:23:19.164091+0000","last_peered":"2024-09-02T10:23:19.164091+0000","last_clean":"2024-09-02T10:23:19.164091+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:23:19.164091+0000","last_undegraded":"2024-09-02T10:23:19.164091+0000","last_fullsized":"2024-09-02T10:23:19.164091+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":919,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.712141+0000","last_change":"2024-09-02T10:22:32.860224+0000","last_active":"2024-09-02T10:23:19.712141+0000","last_peered":"2024-09-02T10:23:19.712141+0000","last_clean":"2024-09-02T10:23:19.712141+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:23:19.712141+0000","last_undegraded":"2024-09-02T10:23:19.712141+0000","last_fullsized":"2024-09-02T10:23:19.712141+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:32.860180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:32.860180+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.373890+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":911,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.712008+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:23:19.712008+0000","last_peered":"2024-09-02T10:23:19.712008+0000","last_clean":"2024-09-02T10:23:19.712008+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:23:19.712008+0000","last_undegraded":"2024-09-02T10:23:19.712008+0000","last_fullsized":"2024-09-02T10:23:19.712008+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:24.029956+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":215,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.557016+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:23:19.557016+0000","last_peered":"2024-09-02T10:23:19.557016+0000","last_clean":"2024-09-02T10:23:19.557016+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:23:19.557016+0000","last_undegraded":"2024-09-02T10:23:19.557016+0000","last_fullsized":"2024-09-02T10:23:19.557016+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":760,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.166736+0000","last_change":"2024-09-02T10:23:01.185167+0000","last_active":"2024-09-02T10:23:19.166736+0000","last_peered":"2024-09-02T10:23:19.166736+0000","last_clean":"2024-09-02T10:23:19.166736+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:23:19.166736+0000","last_undegraded":"2024-09-02T10:23:19.166736+0000","last_fullsized":"2024-09-02T10:23:19.166736+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.185112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.185112+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.992983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1295,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556640+0000","last_change":"2024-09-02T10:23:09.691782+0000","last_active":"2024-09-02T10:23:19.556640+0000","last_peered":"2024-09-02T10:23:19.556640+0000","last_clean":"2024-09-02T10:23:19.556640+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:23:19.556640+0000","last_undegraded":"2024-09-02T10:23:19.556640+0000","last_fullsized":"2024-09-02T10:23:19.556640+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:09.691691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:09.691691+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.662853+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":934,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.712051+0000","last_change":"2024-09-02T10:23:05.131265+0000","last_active":"2024-09-02T10:23:19.712051+0000","last_peered":"2024-09-02T10:23:19.712051+0000","last_clean":"2024-09-02T10:23:19.712051+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:23:19.712051+0000","last_undegraded":"2024-09-02T10:23:19.712051+0000","last_fullsized":"2024-09-02T10:23:19.712051+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.131213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.131213+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.083779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":926,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.164119+0000","last_change":"2024-09-02T10:23:06.476818+0000","last_active":"2024-09-02T10:23:19.164119+0000","last_peered":"2024-09-02T10:23:19.164119+0000","last_clean":"2024-09-02T10:23:19.164119+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:23:19.164119+0000","last_undegraded":"2024-09-02T10:23:19.164119+0000","last_fullsized":"2024-09-02T10:23:19.164119+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:06.476768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.476768+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:35.403622+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":902,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.390837+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:23:19.390837+0000","last_peered":"2024-09-02T10:23:19.390837+0000","last_clean":"2024-09-02T10:23:19.390837+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:23:19.390837+0000","last_undegraded":"2024-09-02T10:23:19.390837+0000","last_fullsized":"2024-09-02T10:23:19.390837+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":147,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556477+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:23:19.556477+0000","last_peered":"2024-09-02T10:23:19.556477+0000","last_clean":"2024-09-02T10:23:19.556477+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:23:19.556477+0000","last_undegraded":"2024-09-02T10:23:19.556477+0000","last_fullsized":"2024-09-02T10:23:19.556477+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1251,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.163972+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:23:19.163972+0000","last_peered":"2024-09-02T10:23:19.163972+0000","last_clean":"2024-09-02T10:23:19.163972+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:23:19.163972+0000","last_undegraded":"2024-09-02T10:23:19.163972+0000","last_fullsized":"2024-09-02T10:23:19.163972+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1194,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556836+0000","last_change":"2024-09-02T10:22:55.516683+0000","last_active":"2024-09-02T10:23:19.556836+0000","last_peered":"2024-09-02T10:23:19.556836+0000","last_clean":"2024-09-02T10:23:19.556836+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:23:19.556836+0000","last_undegraded":"2024-09-02T10:23:19.556836+0000","last_fullsized":"2024-09-02T10:23:19.556836+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.516633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.516633+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:58.850702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1165,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556947+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:23:19.556947+0000","last_peered":"2024-09-02T10:23:19.556947+0000","last_clean":"2024-09-02T10:23:19.556947+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:23:19.556947+0000","last_undegraded":"2024-09-02T10:23:19.556947+0000","last_fullsized":"2024-09-02T10:23:19.556947+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1164,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556703+0000","last_change":"2024-09-02T10:23:00.621966+0000","last_active":"2024-09-02T10:23:19.556703+0000","last_peered":"2024-09-02T10:23:19.556703+0000","last_clean":"2024-09-02T10:23:19.556703+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:23:19.556703+0000","last_undegraded":"2024-09-02T10:23:19.556703+0000","last_fullsized":"2024-09-02T10:23:19.556703+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.621916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.621916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:21.445915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"626'12394","reported_seq":20738,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556545+0000","last_change":"2024-09-02T10:22:56.637995+0000","last_active":"2024-09-02T10:23:19.556545+0000","last_peered":"2024-09-02T10:23:19.556545+0000","last_clean":"2024-09-02T10:23:19.556545+0000","last_became_active":"2024-09-02T10:22:56.595915+0000","last_became_peered":"2024-09-02T10:22:56.595915+0000","last_unstale":"2024-09-02T10:23:19.556545+0000","last_undegraded":"2024-09-02T10:23:19.556545+0000","last_fullsized":"2024-09-02T10:23:19.556545+0000","mapping_epoch":635,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12394,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":237961216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1258,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556771+0000","last_change":"2024-09-02T10:23:01.892373+0000","last_active":"2024-09-02T10:23:19.556771+0000","last_peered":"2024-09-02T10:23:19.556771+0000","last_clean":"2024-09-02T10:23:19.556771+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:23:19.556771+0000","last_undegraded":"2024-09-02T10:23:19.556771+0000","last_fullsized":"2024-09-02T10:23:19.556771+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.892288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.892288+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:30.607091+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"626'12580","reported_seq":29710,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.557374+0000","last_change":"2024-09-02T10:23:09.693242+0000","last_active":"2024-09-02T10:23:19.557374+0000","last_peered":"2024-09-02T10:23:19.557374+0000","last_clean":"2024-09-02T10:23:19.557374+0000","last_became_active":"2024-09-02T10:23:02.182530+0000","last_became_peered":"2024-09-02T10:23:02.182530+0000","last_unstale":"2024-09-02T10:23:19.557374+0000","last_undegraded":"2024-09-02T10:23:19.557374+0000","last_fullsized":"2024-09-02T10:23:19.557374+0000","mapping_epoch":640,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12580","last_scrub_stamp":"2024-09-02T10:23:09.693181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:09.693181+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:24.508961+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8401,"num_bytes_recovered":374538240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1381,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.390672+0000","last_change":"2024-09-02T10:22:37.332903+0000","last_active":"2024-09-02T10:23:19.390672+0000","last_peered":"2024-09-02T10:23:19.390672+0000","last_clean":"2024-09-02T10:23:19.390672+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:23:19.390672+0000","last_undegraded":"2024-09-02T10:23:19.390672+0000","last_fullsized":"2024-09-02T10:23:19.390672+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:37.332859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:37.332859+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.675027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"626'12351","reported_seq":12658,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.557035+0000","last_change":"2024-09-02T10:23:04.675691+0000","last_active":"2024-09-02T10:23:19.557035+0000","last_peered":"2024-09-02T10:23:19.557035+0000","last_clean":"2024-09-02T10:23:19.557035+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:23:19.557035+0000","last_undegraded":"2024-09-02T10:23:19.557035+0000","last_fullsized":"2024-09-02T10:23:19.557035+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12351","last_scrub_stamp":"2024-09-02T10:23:04.675613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:04.675613+0000","objects_scrubbed":1,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.698140+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12351,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1024,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.557363+0000","last_change":"2024-09-02T10:22:55.068228+0000","last_active":"2024-09-02T10:23:19.557363+0000","last_peered":"2024-09-02T10:23:19.557363+0000","last_clean":"2024-09-02T10:23:19.557363+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:23:19.557363+0000","last_undegraded":"2024-09-02T10:23:19.557363+0000","last_fullsized":"2024-09-02T10:23:19.557363+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.068166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.068166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:04.696428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"626'12382","reported_seq":12696,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556995+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:23:19.556995+0000","last_peered":"2024-09-02T10:23:19.556995+0000","last_clean":"2024-09-02T10:23:19.556995+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:23:19.556995+0000","last_undegraded":"2024-09-02T10:23:19.556995+0000","last_fullsized":"2024-09-02T10:23:19.556995+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":421,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556837+0000","last_change":"2024-09-02T10:22:15.563077+0000","last_active":"2024-09-02T10:23:19.556837+0000","last_peered":"2024-09-02T10:23:19.556837+0000","last_clean":"2024-09-02T10:23:19.556837+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:23:19.556837+0000","last_undegraded":"2024-09-02T10:23:19.556837+0000","last_fullsized":"2024-09-02T10:23:19.556837+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:15.563022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:15.563022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:20.809417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"626'12606","reported_seq":12944,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.390737+0000","last_change":"2024-09-02T10:23:11.425512+0000","last_active":"2024-09-02T10:23:19.390737+0000","last_peered":"2024-09-02T10:23:19.390737+0000","last_clean":"2024-09-02T10:23:19.390737+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:23:19.390737+0000","last_undegraded":"2024-09-02T10:23:19.390737+0000","last_fullsized":"2024-09-02T10:23:19.390737+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12606","last_scrub_stamp":"2024-09-02T10:23:11.425432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:11.425432+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:12.626473+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1330,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556899+0000","last_change":"2024-09-02T10:22:34.647196+0000","last_active":"2024-09-02T10:23:19.556899+0000","last_peered":"2024-09-02T10:23:19.556899+0000","last_clean":"2024-09-02T10:23:19.556899+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:23:19.556899+0000","last_undegraded":"2024-09-02T10:23:19.556899+0000","last_fullsized":"2024-09-02T10:23:19.556899+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:34.647144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:34.647144+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:02.024298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"626'12544","reported_seq":29687,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.557217+0000","last_change":"2024-09-02T10:23:07.704034+0000","last_active":"2024-09-02T10:23:19.557217+0000","last_peered":"2024-09-02T10:23:19.557217+0000","last_clean":"2024-09-02T10:23:19.557217+0000","last_became_active":"2024-09-02T10:22:57.600280+0000","last_became_peered":"2024-09-02T10:22:57.600280+0000","last_unstale":"2024-09-02T10:23:19.557217+0000","last_undegraded":"2024-09-02T10:23:19.557217+0000","last_fullsized":"2024-09-02T10:23:19.557217+0000","mapping_epoch":636,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12544","last_scrub_stamp":"2024-09-02T10:23:07.703956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:07.703956+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:08.077242+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8407,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":796,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.166872+0000","last_change":"2024-09-02T10:23:05.243285+0000","last_active":"2024-09-02T10:23:19.166872+0000","last_peered":"2024-09-02T10:23:19.166872+0000","last_clean":"2024-09-02T10:23:19.166872+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:23:19.166872+0000","last_undegraded":"2024-09-02T10:23:19.166872+0000","last_fullsized":"2024-09-02T10:23:19.166872+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.243236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.243236+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:16.091958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"626'12380","reported_seq":12720,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.712106+0000","last_change":"2024-09-02T10:23:07.151869+0000","last_active":"2024-09-02T10:23:19.712106+0000","last_peered":"2024-09-02T10:23:19.712106+0000","last_clean":"2024-09-02T10:23:19.712106+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:23:19.712106+0000","last_undegraded":"2024-09-02T10:23:19.712106+0000","last_fullsized":"2024-09-02T10:23:19.712106+0000","mapping_epoch":533,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12380","last_scrub_stamp":"2024-09-02T10:23:07.151772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:07.151772+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:33.568723+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12380,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":761,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.166897+0000","last_change":"2024-09-02T10:22:55.086083+0000","last_active":"2024-09-02T10:23:19.166897+0000","last_peered":"2024-09-02T10:23:19.166897+0000","last_clean":"2024-09-02T10:23:19.166897+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:23:19.166897+0000","last_undegraded":"2024-09-02T10:23:19.166897+0000","last_fullsized":"2024-09-02T10:23:19.166897+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.086036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.086036+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.714243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"626'12678","reported_seq":17750,"reported_epoch":646,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:23:23.009041+0000","last_change":"2024-09-02T10:22:55.007293+0000","last_active":"2024-09-02T10:23:23.009041+0000","last_peered":"2024-09-02T10:23:23.009041+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:23:23.009041+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-3953}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":521,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.164658+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:23:19.164658+0000","last_peered":"2024-09-02T10:23:19.164658+0000","last_clean":"2024-09-02T10:23:19.164658+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:23:19.164658+0000","last_undegraded":"2024-09-02T10:23:19.164658+0000","last_fullsized":"2024-09-02T10:23:19.164658+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:22.506281+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"626'12542","reported_seq":17199,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556942+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:23:19.556942+0000","last_peered":"2024-09-02T10:23:19.556942+0000","last_clean":"2024-09-02T10:23:19.556942+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:23:19.556942+0000","last_undegraded":"2024-09-02T10:23:19.556942+0000","last_fullsized":"2024-09-02T10:23:19.556942+0000","mapping_epoch":559,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12542,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":762,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.166798+0000","last_change":"2024-09-02T10:23:00.146326+0000","last_active":"2024-09-02T10:23:19.166798+0000","last_peered":"2024-09-02T10:23:19.166798+0000","last_clean":"2024-09-02T10:23:19.166798+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:23:19.166798+0000","last_undegraded":"2024-09-02T10:23:19.166798+0000","last_fullsized":"2024-09-02T10:23:19.166798+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.146278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.146278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:09.373850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"626'12632","reported_seq":12975,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.390814+0000","last_change":"2024-09-02T10:22:59.338566+0000","last_active":"2024-09-02T10:23:19.390814+0000","last_peered":"2024-09-02T10:23:19.390814+0000","last_clean":"2024-09-02T10:23:19.390814+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:23:19.390814+0000","last_undegraded":"2024-09-02T10:23:19.390814+0000","last_fullsized":"2024-09-02T10:23:19.390814+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12632","last_scrub_stamp":"2024-09-02T10:22:59.338517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:22:59.338517+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.748681+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1015,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.390702+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:23:19.390702+0000","last_peered":"2024-09-02T10:23:19.390702+0000","last_clean":"2024-09-02T10:23:19.390702+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:23:19.390702+0000","last_undegraded":"2024-09-02T10:23:19.390702+0000","last_fullsized":"2024-09-02T10:23:19.390702+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"626'12586","reported_seq":12884,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.164001+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:23:19.164001+0000","last_peered":"2024-09-02T10:23:19.164001+0000","last_clean":"2024-09-02T10:23:19.164001+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:23:19.164001+0000","last_undegraded":"2024-09-02T10:23:19.164001+0000","last_fullsized":"2024-09-02T10:23:19.164001+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12586,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1283,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.164030+0000","last_change":"2024-09-02T10:23:08.500316+0000","last_active":"2024-09-02T10:23:19.164030+0000","last_peered":"2024-09-02T10:23:19.164030+0000","last_clean":"2024-09-02T10:23:19.164030+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:23:19.164030+0000","last_undegraded":"2024-09-02T10:23:19.164030+0000","last_fullsized":"2024-09-02T10:23:19.164030+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:08.500268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:08.500268+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:14.723023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"626'12632","reported_seq":17187,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.557165+0000","last_change":"2024-09-02T10:23:02.673893+0000","last_active":"2024-09-02T10:23:19.557165+0000","last_peered":"2024-09-02T10:23:19.557165+0000","last_clean":"2024-09-02T10:23:19.557165+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:23:19.557165+0000","last_undegraded":"2024-09-02T10:23:19.557165+0000","last_fullsized":"2024-09-02T10:23:19.557165+0000","mapping_epoch":565,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12632","last_scrub_stamp":"2024-09-02T10:23:02.673828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:02.673828+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:31.609539+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1027,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.390637+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:23:19.390637+0000","last_peered":"2024-09-02T10:23:19.390637+0000","last_clean":"2024-09-02T10:23:19.390637+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:23:19.390637+0000","last_undegraded":"2024-09-02T10:23:19.390637+0000","last_fullsized":"2024-09-02T10:23:19.390637+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:22.322242+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"626'12650","reported_seq":12989,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.390863+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:23:19.390863+0000","last_peered":"2024-09-02T10:23:19.390863+0000","last_clean":"2024-09-02T10:23:19.390863+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:23:19.390863+0000","last_undegraded":"2024-09-02T10:23:19.390863+0000","last_fullsized":"2024-09-02T10:23:19.390863+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:24.389647+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12650,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1290,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.164683+0000","last_change":"2024-09-02T10:23:00.386934+0000","last_active":"2024-09-02T10:23:19.164683+0000","last_peered":"2024-09-02T10:23:19.164683+0000","last_clean":"2024-09-02T10:23:19.164683+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:23:19.164683+0000","last_undegraded":"2024-09-02T10:23:19.164683+0000","last_fullsized":"2024-09-02T10:23:19.164683+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.386875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.386875+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.366845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"626'12496","reported_seq":12799,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556748+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:23:19.556748+0000","last_peered":"2024-09-02T10:23:19.556748+0000","last_clean":"2024-09-02T10:23:19.556748+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:23:19.556748+0000","last_undegraded":"2024-09-02T10:23:19.556748+0000","last_fullsized":"2024-09-02T10:23:19.556748+0000","mapping_epoch":533,"log_start":"553'2400","ondisk_log_start":"553'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":230,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.557312+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:23:19.557312+0000","last_peered":"2024-09-02T10:23:19.557312+0000","last_clean":"2024-09-02T10:23:19.557312+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:23:19.557312+0000","last_undegraded":"2024-09-02T10:23:19.557312+0000","last_fullsized":"2024-09-02T10:23:19.557312+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:22.657515+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"626'12520","reported_seq":12834,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556705+0000","last_change":"2024-09-02T10:23:06.653266+0000","last_active":"2024-09-02T10:23:19.556705+0000","last_peered":"2024-09-02T10:23:19.556705+0000","last_clean":"2024-09-02T10:23:19.556705+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:23:19.556705+0000","last_undegraded":"2024-09-02T10:23:19.556705+0000","last_fullsized":"2024-09-02T10:23:19.556705+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12520","last_scrub_stamp":"2024-09-02T10:23:06.653191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.653191+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:17.096908+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12520,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1062,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.390935+0000","last_change":"2024-09-02T10:22:30.358040+0000","last_active":"2024-09-02T10:23:19.390935+0000","last_peered":"2024-09-02T10:23:19.390935+0000","last_clean":"2024-09-02T10:23:19.390935+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:23:19.390935+0000","last_undegraded":"2024-09-02T10:23:19.390935+0000","last_fullsized":"2024-09-02T10:23:19.390935+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:30.357970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:30.357970+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.221263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"626'12566","reported_seq":12868,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.556654+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:23:19.556654+0000","last_peered":"2024-09-02T10:23:19.556654+0000","last_clean":"2024-09-02T10:23:19.556654+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:23:19.556654+0000","last_undegraded":"2024-09-02T10:23:19.556654+0000","last_fullsized":"2024-09-02T10:23:19.556654+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":444,"reported_epoch":646,"state":"active+clean","last_fresh":"2024-09-02T10:23:19.557092+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:23:19.557092+0000","last_peered":"2024-09-02T10:23:19.557092+0000","last_clean":"2024-09-02T10:23:19.557092+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:23:19.557092+0000","last_undegraded":"2024-09-02T10:23:19.557092+0000","last_fullsized":"2024-09-02T10:23:19.557092+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3953,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29683,"num_bytes_recovered":1286144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270938112,"data_stored":270925896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383960,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487628,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93257524,"statfs":{"total":95995035648,"available":95495704576,"internally_reserved":0,"allocated":1105920,"data_stored":700910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":498203076},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612893,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357136,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93388016,"statfs":{"total":95995035648,"available":95629328384,"internally_reserved":0,"allocated":1110016,"data_stored":700934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841825,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545932,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199220,"statfs":{"total":95995035648,"available":95436001280,"internally_reserved":0,"allocated":1105920,"data_stored":700910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070759,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339084,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93406068,"statfs":{"total":95995035648,"available":95647813632,"internally_reserved":0,"allocated":1105920,"data_stored":700910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299693,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582228,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162924,"statfs":{"total":95995035648,"available":95398834176,"internally_reserved":0,"allocated":1310720,"data_stored":902158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528624,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365772,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379380,"statfs":{"total":95995035648,"available":95620485120,"internally_reserved":0,"allocated":1433600,"data_stored":1028590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724852,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":878744,"kb_used_data":265860,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92866408,"statfs":{"total":95995035648,"available":95095201792,"internally_reserved":0,"allocated":272240640,"data_stored":271828006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953786,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334924,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410228,"statfs":{"total":95995035648,"available":95652073472,"internally_reserved":0,"allocated":1105920,"data_stored":700910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150015,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569868,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175284,"statfs":{"total":95995035648,"available":95411490816,"internally_reserved":0,"allocated":1433600,"data_stored":1028590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411651,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311504,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93433648,"statfs":{"total":95995035648,"available":95676055552,"internally_reserved":0,"allocated":1310720,"data_stored":902158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575176,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509840,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235312,"statfs":{"total":95995035648,"available":95472959488,"internally_reserved":0,"allocated":1110016,"data_stored":700934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738702,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592076,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93153076,"statfs":{"total":95995035648,"available":95388749824,"internally_reserved":0,"allocated":1433600,"data_stored":1028590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":270929920,"data_stored":270925848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:23:30.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:23:30.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:30.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:30.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:30.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:30 smithi151 ceph-mon[16639]: pgmap v1503: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3616/9 objects degraded (40177.778%); 0 B/s, 98 objects/s recovering 2024-09-02T10:23:30.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:30 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 3616/9 objects degraded (40177.778%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:30.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:30.836 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:30.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:30 smithi162 ceph-mon[20279]: pgmap v1503: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3616/9 objects degraded (40177.778%); 0 B/s, 98 objects/s recovering 2024-09-02T10:23:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:30 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 3616/9 objects degraded (40177.778%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:30.919 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:30.919 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:30.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:30.948 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:30.971 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:30.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:30.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:30 smithi187 ceph-mon[20072]: pgmap v1503: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3616/9 objects degraded (40177.778%); 0 B/s, 98 objects/s recovering 2024-09-02T10:23:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:30 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 3616/9 objects degraded (40177.778%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:31.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:31.099 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:31.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:31.100 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:31.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:31.230 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:31.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:31.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:31.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:31.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:31.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:31.359 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:31.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:31.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:31.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:31.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:31.540 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:23:31.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:31.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:31.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:31.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/162901840' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:23:31.706 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:31.732 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:31.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:31.795 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:23:31.795 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 3616/9 objects degraded (40177.778%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1683,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":649,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":12,"osd_in_since":1725272451,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":3,"data_bytes":590392,"bytes_used":6015754240,"bytes_avail":1145924673536,"bytes_total":1151940427776,"degraded_objects":3616,"degraded_total":9,"degraded_ratio":401.77777777777771,"recovering_objects_per_sec":42,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":337,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":37,"modified":"2024-09-02T10:21:41.376228+0000","services":{}},"progress_events":{}} 2024-09-02T10:23:31.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:31.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/162901840' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:23:31.859 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:31.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:31.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:31.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:31.987 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:31.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:31.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/162901840' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:23:32.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:32.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:32.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:32.117 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:32.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:32.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:32.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:32.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:32.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:32.368 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:32.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:32.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:32.497 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:32.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:32.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:32.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:32.624 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:23:32.625 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:23:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:32.625 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:32.646 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:23:32.646 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:23:32.667 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:23:32.668 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:23:32.688 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:23:32.688 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:23:32.709 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:23:32.710 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:23:32.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:32.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:23:32.731 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:23:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:32.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:32.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:32.754 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:23:32.754 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:23:32.776 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:23:32.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:23:32.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:32 smithi151 ceph-mon[16639]: pgmap v1504: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3616/9 objects degraded (40177.778%); 0 B/s, 42 objects/s recovering 2024-09-02T10:23:32.777 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:32 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:23:32.777 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/162901840' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:23:32.777 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:32 smithi151 ceph-mon[16639]: osdmap e649: 12 total, 12 up, 12 in 2024-09-02T10:23:32.777 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/491146413' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:23:32.797 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:23:32.797 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:23:32.819 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:23:32.819 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:23:32.840 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:23:32.841 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:23:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:32 smithi162 ceph-mon[20279]: pgmap v1504: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3616/9 objects degraded (40177.778%); 0 B/s, 42 objects/s recovering 2024-09-02T10:23:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:32 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:23:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/162901840' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:23:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:32 smithi162 ceph-mon[20279]: osdmap e649: 12 total, 12 up, 12 in 2024-09-02T10:23:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/491146413' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:23:32.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:32.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:23:32.862 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:23:32.877 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:32.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:32.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:32.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:33.005 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:33.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:32 smithi187 ceph-mon[20072]: pgmap v1504: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3616/9 objects degraded (40177.778%); 0 B/s, 42 objects/s recovering 2024-09-02T10:23:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:32 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:23:33.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/162901840' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:23:33.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:32 smithi187 ceph-mon[20072]: osdmap e649: 12 total, 12 up, 12 in 2024-09-02T10:23:33.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/491146413' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:23:33.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:33.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:33.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:33.231 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:33.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:33.256 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:33.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:33.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:33.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:33.430 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:23:33.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:23:33.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:33.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:33.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:33.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:33.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:33.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:33.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1184907102' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:33.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:33.755 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:33.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:33.756 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1184907102' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:33.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:33.881 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:33.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:33.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:33.983 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1184907102' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:34.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:34.008 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:34.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:34.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:34.134 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:34.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:34.237 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:34.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:34.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:34.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:34.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:34.392 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:34.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:34.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:34.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:34.520 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:34.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:34.521 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:34.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:34.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:34.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:34.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:34.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:34.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:34.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:34 smithi162 ceph-mon[20279]: pgmap v1506: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3264/9 objects degraded (36266.667%); 0 B/s, 114 objects/s recovering 2024-09-02T10:23:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:34 smithi162 ceph-mon[20279]: osdmap e650: 12 total, 12 up, 12 in 2024-09-02T10:23:34.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:34.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:34.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:34.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:34 smithi151 ceph-mon[16639]: pgmap v1506: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3264/9 objects degraded (36266.667%); 0 B/s, 114 objects/s recovering 2024-09-02T10:23:34.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:34 smithi151 ceph-mon[16639]: osdmap e650: 12 total, 12 up, 12 in 2024-09-02T10:23:34.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:34 smithi187 ceph-mon[20072]: pgmap v1506: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3264/9 objects degraded (36266.667%); 0 B/s, 114 objects/s recovering 2024-09-02T10:23:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:34 smithi187 ceph-mon[20072]: osdmap e650: 12 total, 12 up, 12 in 2024-09-02T10:23:35.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:35.023 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:35.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:35.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:35.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:35.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:35.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:35.279 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:35.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:35.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:35.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:35.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:35.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:35.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:35.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:35.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:35.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:35.655 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:35.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:35.757 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:35.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:35.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:35.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:35.911 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:35.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:35.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:36.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:36.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:36.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:36.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:36.163 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:36.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:36.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:36.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:36.291 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:36.292 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:36.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:36.419 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:36.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:36.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:36.546 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:36.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:36.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:36.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:36.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:36.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:36.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:36.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:36.799 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:36.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:36 smithi162 ceph-mon[20279]: pgmap v1508: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3264/9 objects degraded (36266.667%); 0 B/s, 56 objects/s recovering 2024-09-02T10:23:36.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:36.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:36.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:36.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:36 smithi151 ceph-mon[16639]: pgmap v1508: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3264/9 objects degraded (36266.667%); 0 B/s, 56 objects/s recovering 2024-09-02T10:23:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:36 smithi187 ceph-mon[20072]: pgmap v1508: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3264/9 objects degraded (36266.667%); 0 B/s, 56 objects/s recovering 2024-09-02T10:23:37.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:37.052 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:37.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:37.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:37.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:37.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:37.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:37.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:37.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:37.303 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:37.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:37.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:37.429 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:37.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:37.531 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:37.557 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:37.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:37.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:37.659 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:37.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:37.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:37.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:37.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:37.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:37.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:37.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:37.942 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:37.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:38.043 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:38.066 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:38.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:38.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:38.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:38.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:38.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:38.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:38.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:38.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:38.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:38.316 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:38.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:38.440 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:38.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:38.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:38.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:23:38.509 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:38.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:38.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:38.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:38.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:38.688 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:38.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:38.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:38.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:38 smithi151 ceph-mon[16639]: pgmap v1509: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3264/9 objects degraded (36266.667%); 0 B/s, 43 objects/s recovering 2024-09-02T10:23:38.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:23:38.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:23:38.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:38 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:23:38.778 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:38 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:23:38.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:38.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:38.812 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:38.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:38 smithi162 ceph-mon[20279]: pgmap v1509: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3264/9 objects degraded (36266.667%); 0 B/s, 43 objects/s recovering 2024-09-02T10:23:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:23:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:23:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:38 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:23:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:38 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:23:38.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:38.936 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:38.937 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:38 smithi187 ceph-mon[20072]: pgmap v1509: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3264/9 objects degraded (36266.667%); 0 B/s, 43 objects/s recovering 2024-09-02T10:23:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:23:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:23:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:38 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:23:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:38 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:23:39.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:39.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:39.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:39.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:39.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:39.185 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:39.186 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:39.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:39.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:39.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:39.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:39.362 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:23:39.363 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:23:39.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:39.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:39.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:39.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:39.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:39.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:39.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:39.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:39.689 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:39.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:39.791 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:39.817 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:39.817 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:23:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:23:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 3264/9 objects degraded (36266.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:39.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:23:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:23:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 3264/9 objects degraded (36266.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:39.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:39.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:39.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:23:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:23:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 3264/9 objects degraded (36266.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:40.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:40.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:40.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:40.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:40.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:40.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:40.194 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:40.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:23:40.295 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:40.321 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:40.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:40.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:40.449 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:40.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:40.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:23:40.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:40.576 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:40.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:40.632 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:40.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:40.703 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:40.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:40.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:40.741 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:23:40.742 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:23:40.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:40.830 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:40.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:40 smithi162 ceph-mon[20279]: pgmap v1510: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2561/9 objects degraded (28455.556%); 0 B/s, 131 objects/s recovering 2024-09-02T10:23:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:40 smithi162 ceph-mon[20279]: osdmap e651: 12 total, 12 up, 12 in 2024-09-02T10:23:40.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:40 smithi151 ceph-mon[16639]: pgmap v1510: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2561/9 objects degraded (28455.556%); 0 B/s, 131 objects/s recovering 2024-09-02T10:23:40.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:40 smithi151 ceph-mon[16639]: osdmap e651: 12 total, 12 up, 12 in 2024-09-02T10:23:40.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:40.959 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:40.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:40.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:40 smithi187 ceph-mon[20072]: pgmap v1510: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2561/9 objects degraded (28455.556%); 0 B/s, 131 objects/s recovering 2024-09-02T10:23:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:40 smithi187 ceph-mon[20072]: osdmap e651: 12 total, 12 up, 12 in 2024-09-02T10:23:41.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:41.086 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:41.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:41.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:41.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:41.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:41.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:41.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:41.341 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:41.342 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:41.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:41.470 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:41.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:41.570 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:41.594 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:41.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:41.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:41.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:41.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:41.728 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1511,"stamp":"2024-09-02T10:23:39.557179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31079,"num_bytes_recovered":1287075424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163791,"ondisk_log_size":163791,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5871176,"kb_used_data":279684,"kb_used_omap":18,"kb_used_meta":5591213,"kb_avail":1119070648,"statfs":{"total":1151940427776,"available":1145928343552,"internally_reserved":0,"allocated":286396416,"data_stored":281301672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5725402427},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1055,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1055,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.016635"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":662,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.167902+0000","last_change":"2024-09-02T10:22:10.460243+0000","last_active":"2024-09-02T10:23:34.167902+0000","last_peered":"2024-09-02T10:23:34.167902+0000","last_clean":"2024-09-02T10:23:34.167902+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:23:34.167902+0000","last_undegraded":"2024-09-02T10:23:34.167902+0000","last_fullsized":"2024-09-02T10:23:34.167902+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:10.460196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:10.460196+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.460142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":469,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186225+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:23:34.186225+0000","last_peered":"2024-09-02T10:23:34.186225+0000","last_clean":"2024-09-02T10:23:34.186225+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:23:34.186225+0000","last_undegraded":"2024-09-02T10:23:34.186225+0000","last_fullsized":"2024-09-02T10:23:34.186225+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:35.456477+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":927,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186193+0000","last_change":"2024-09-02T10:22:32.860224+0000","last_active":"2024-09-02T10:23:34.186193+0000","last_peered":"2024-09-02T10:23:34.186193+0000","last_clean":"2024-09-02T10:23:34.186193+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:23:34.186193+0000","last_undegraded":"2024-09-02T10:23:34.186193+0000","last_fullsized":"2024-09-02T10:23:34.186193+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:32.860180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:32.860180+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.373890+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":919,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186061+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:23:34.186061+0000","last_peered":"2024-09-02T10:23:34.186061+0000","last_clean":"2024-09-02T10:23:34.186061+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:23:34.186061+0000","last_undegraded":"2024-09-02T10:23:34.186061+0000","last_fullsized":"2024-09-02T10:23:34.186061+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:36.125606+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":223,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186113+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:23:34.186113+0000","last_peered":"2024-09-02T10:23:34.186113+0000","last_clean":"2024-09-02T10:23:34.186113+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:23:34.186113+0000","last_undegraded":"2024-09-02T10:23:34.186113+0000","last_fullsized":"2024-09-02T10:23:34.186113+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:37.615648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":768,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.185877+0000","last_change":"2024-09-02T10:23:01.185167+0000","last_active":"2024-09-02T10:23:34.185877+0000","last_peered":"2024-09-02T10:23:34.185877+0000","last_clean":"2024-09-02T10:23:34.185877+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:23:34.185877+0000","last_undegraded":"2024-09-02T10:23:34.185877+0000","last_fullsized":"2024-09-02T10:23:34.185877+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.185112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.185112+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.992983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1304,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186350+0000","last_change":"2024-09-02T10:23:09.691782+0000","last_active":"2024-09-02T10:23:34.186350+0000","last_peered":"2024-09-02T10:23:34.186350+0000","last_clean":"2024-09-02T10:23:34.186350+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:23:34.186350+0000","last_undegraded":"2024-09-02T10:23:34.186350+0000","last_fullsized":"2024-09-02T10:23:34.186350+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:09.691691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:09.691691+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.662853+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":943,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186187+0000","last_change":"2024-09-02T10:23:05.131265+0000","last_active":"2024-09-02T10:23:34.186187+0000","last_peered":"2024-09-02T10:23:34.186187+0000","last_clean":"2024-09-02T10:23:34.186187+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:23:34.186187+0000","last_undegraded":"2024-09-02T10:23:34.186187+0000","last_fullsized":"2024-09-02T10:23:34.186187+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.131213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.131213+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.083779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":934,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186130+0000","last_change":"2024-09-02T10:23:06.476818+0000","last_active":"2024-09-02T10:23:34.186130+0000","last_peered":"2024-09-02T10:23:34.186130+0000","last_clean":"2024-09-02T10:23:34.186130+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:23:34.186130+0000","last_undegraded":"2024-09-02T10:23:34.186130+0000","last_fullsized":"2024-09-02T10:23:34.186130+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:06.476768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.476768+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:35.403622+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":911,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186694+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:23:34.186694+0000","last_peered":"2024-09-02T10:23:34.186694+0000","last_clean":"2024-09-02T10:23:34.186694+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:23:34.186694+0000","last_undegraded":"2024-09-02T10:23:34.186694+0000","last_fullsized":"2024-09-02T10:23:34.186694+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":155,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.189665+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:23:34.189665+0000","last_peered":"2024-09-02T10:23:34.189665+0000","last_clean":"2024-09-02T10:23:34.189665+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:23:34.189665+0000","last_undegraded":"2024-09-02T10:23:34.189665+0000","last_fullsized":"2024-09-02T10:23:34.189665+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1259,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186100+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:23:34.186100+0000","last_peered":"2024-09-02T10:23:34.186100+0000","last_clean":"2024-09-02T10:23:34.186100+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:23:34.186100+0000","last_undegraded":"2024-09-02T10:23:34.186100+0000","last_fullsized":"2024-09-02T10:23:34.186100+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1202,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186416+0000","last_change":"2024-09-02T10:22:55.516683+0000","last_active":"2024-09-02T10:23:34.186416+0000","last_peered":"2024-09-02T10:23:34.186416+0000","last_clean":"2024-09-02T10:23:34.186416+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:23:34.186416+0000","last_undegraded":"2024-09-02T10:23:34.186416+0000","last_fullsized":"2024-09-02T10:23:34.186416+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.516633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.516633+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:58.850702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1173,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186443+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:23:34.186443+0000","last_peered":"2024-09-02T10:23:34.186443+0000","last_clean":"2024-09-02T10:23:34.186443+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:23:34.186443+0000","last_undegraded":"2024-09-02T10:23:34.186443+0000","last_fullsized":"2024-09-02T10:23:34.186443+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1172,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186362+0000","last_change":"2024-09-02T10:23:00.621966+0000","last_active":"2024-09-02T10:23:34.186362+0000","last_peered":"2024-09-02T10:23:34.186362+0000","last_clean":"2024-09-02T10:23:34.186362+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:23:34.186362+0000","last_undegraded":"2024-09-02T10:23:34.186362+0000","last_fullsized":"2024-09-02T10:23:34.186362+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.621916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.621916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:21.445915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"626'12394","reported_seq":20746,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.189854+0000","last_change":"2024-09-02T10:22:56.637995+0000","last_active":"2024-09-02T10:23:34.189854+0000","last_peered":"2024-09-02T10:23:34.189854+0000","last_clean":"2024-09-02T10:23:34.189854+0000","last_became_active":"2024-09-02T10:22:56.595915+0000","last_became_peered":"2024-09-02T10:22:56.595915+0000","last_unstale":"2024-09-02T10:23:34.189854+0000","last_undegraded":"2024-09-02T10:23:34.189854+0000","last_fullsized":"2024-09-02T10:23:34.189854+0000","mapping_epoch":635,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12394,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":237961216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1267,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186471+0000","last_change":"2024-09-02T10:23:01.892373+0000","last_active":"2024-09-02T10:23:34.186471+0000","last_peered":"2024-09-02T10:23:34.186471+0000","last_clean":"2024-09-02T10:23:34.186471+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:23:34.186471+0000","last_undegraded":"2024-09-02T10:23:34.186471+0000","last_fullsized":"2024-09-02T10:23:34.186471+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.892288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.892288+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:30.607091+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"626'12580","reported_seq":29718,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186404+0000","last_change":"2024-09-02T10:23:09.693242+0000","last_active":"2024-09-02T10:23:34.186404+0000","last_peered":"2024-09-02T10:23:34.186404+0000","last_clean":"2024-09-02T10:23:34.186404+0000","last_became_active":"2024-09-02T10:23:02.182530+0000","last_became_peered":"2024-09-02T10:23:02.182530+0000","last_unstale":"2024-09-02T10:23:34.186404+0000","last_undegraded":"2024-09-02T10:23:34.186404+0000","last_fullsized":"2024-09-02T10:23:34.186404+0000","mapping_epoch":640,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12580","last_scrub_stamp":"2024-09-02T10:23:09.693181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:09.693181+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:24.508961+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8401,"num_bytes_recovered":374538240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1390,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.185991+0000","last_change":"2024-09-02T10:22:37.332903+0000","last_active":"2024-09-02T10:23:34.185991+0000","last_peered":"2024-09-02T10:23:34.185991+0000","last_clean":"2024-09-02T10:23:34.185991+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:23:34.185991+0000","last_undegraded":"2024-09-02T10:23:34.185991+0000","last_fullsized":"2024-09-02T10:23:34.185991+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:37.332859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:37.332859+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.675027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"626'12351","reported_seq":12666,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186569+0000","last_change":"2024-09-02T10:23:04.675691+0000","last_active":"2024-09-02T10:23:34.186569+0000","last_peered":"2024-09-02T10:23:34.186569+0000","last_clean":"2024-09-02T10:23:34.186569+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:23:34.186569+0000","last_undegraded":"2024-09-02T10:23:34.186569+0000","last_fullsized":"2024-09-02T10:23:34.186569+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12351","last_scrub_stamp":"2024-09-02T10:23:04.675613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:04.675613+0000","objects_scrubbed":1,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.698140+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12351,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1032,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.459995+0000","last_change":"2024-09-02T10:22:55.068228+0000","last_active":"2024-09-02T10:23:34.459995+0000","last_peered":"2024-09-02T10:23:34.459995+0000","last_clean":"2024-09-02T10:23:34.459995+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:23:34.459995+0000","last_undegraded":"2024-09-02T10:23:34.459995+0000","last_fullsized":"2024-09-02T10:23:34.459995+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.068166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.068166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:04.696428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"626'12382","reported_seq":12704,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.168074+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:23:34.168074+0000","last_peered":"2024-09-02T10:23:34.168074+0000","last_clean":"2024-09-02T10:23:34.168074+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:23:34.168074+0000","last_undegraded":"2024-09-02T10:23:34.168074+0000","last_fullsized":"2024-09-02T10:23:34.168074+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":429,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.167793+0000","last_change":"2024-09-02T10:22:15.563077+0000","last_active":"2024-09-02T10:23:34.167793+0000","last_peered":"2024-09-02T10:23:34.167793+0000","last_clean":"2024-09-02T10:23:34.167793+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:23:34.167793+0000","last_undegraded":"2024-09-02T10:23:34.167793+0000","last_fullsized":"2024-09-02T10:23:34.167793+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:15.563022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:15.563022+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:36.203183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"626'12606","reported_seq":12952,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186477+0000","last_change":"2024-09-02T10:23:11.425512+0000","last_active":"2024-09-02T10:23:34.186477+0000","last_peered":"2024-09-02T10:23:34.186477+0000","last_clean":"2024-09-02T10:23:34.186477+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:23:34.186477+0000","last_undegraded":"2024-09-02T10:23:34.186477+0000","last_fullsized":"2024-09-02T10:23:34.186477+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12606","last_scrub_stamp":"2024-09-02T10:23:11.425432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:11.425432+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:12.626473+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1338,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.167863+0000","last_change":"2024-09-02T10:22:34.647196+0000","last_active":"2024-09-02T10:23:34.167863+0000","last_peered":"2024-09-02T10:23:34.167863+0000","last_clean":"2024-09-02T10:23:34.167863+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:23:34.167863+0000","last_undegraded":"2024-09-02T10:23:34.167863+0000","last_fullsized":"2024-09-02T10:23:34.167863+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:34.647144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:34.647144+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:02.024298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"626'12544","reported_seq":29695,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186229+0000","last_change":"2024-09-02T10:23:07.704034+0000","last_active":"2024-09-02T10:23:34.186229+0000","last_peered":"2024-09-02T10:23:34.186229+0000","last_clean":"2024-09-02T10:23:34.186229+0000","last_became_active":"2024-09-02T10:22:57.600280+0000","last_became_peered":"2024-09-02T10:22:57.600280+0000","last_unstale":"2024-09-02T10:23:34.186229+0000","last_undegraded":"2024-09-02T10:23:34.186229+0000","last_fullsized":"2024-09-02T10:23:34.186229+0000","mapping_epoch":636,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12544","last_scrub_stamp":"2024-09-02T10:23:07.703956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:07.703956+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:08.077242+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8407,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":804,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186006+0000","last_change":"2024-09-02T10:23:05.243285+0000","last_active":"2024-09-02T10:23:34.186006+0000","last_peered":"2024-09-02T10:23:34.186006+0000","last_clean":"2024-09-02T10:23:34.186006+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:23:34.186006+0000","last_undegraded":"2024-09-02T10:23:34.186006+0000","last_fullsized":"2024-09-02T10:23:34.186006+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.243236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.243236+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:16.091958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"626'12380","reported_seq":12728,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186167+0000","last_change":"2024-09-02T10:23:07.151869+0000","last_active":"2024-09-02T10:23:34.186167+0000","last_peered":"2024-09-02T10:23:34.186167+0000","last_clean":"2024-09-02T10:23:34.186167+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:23:34.186167+0000","last_undegraded":"2024-09-02T10:23:34.186167+0000","last_fullsized":"2024-09-02T10:23:34.186167+0000","mapping_epoch":533,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12380","last_scrub_stamp":"2024-09-02T10:23:07.151772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:07.151772+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:33.568723+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12380,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":769,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186080+0000","last_change":"2024-09-02T10:22:55.086083+0000","last_active":"2024-09-02T10:23:34.186080+0000","last_peered":"2024-09-02T10:23:34.186080+0000","last_clean":"2024-09-02T10:23:34.186080+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:23:34.186080+0000","last_undegraded":"2024-09-02T10:23:34.186080+0000","last_fullsized":"2024-09-02T10:23:34.186080+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.086036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.086036+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.714243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"626'12678","reported_seq":20542,"reported_epoch":650,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:23:37.780729+0000","last_change":"2024-09-02T10:22:55.007293+0000","last_active":"2024-09-02T10:23:37.780729+0000","last_peered":"2024-09-02T10:23:37.780729+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:23:37.780729+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3778,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-2561}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":529,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.167917+0000","last_change":"2024-09-02T10:22:07.546066+0000","last_active":"2024-09-02T10:23:34.167917+0000","last_peered":"2024-09-02T10:23:34.167917+0000","last_clean":"2024-09-02T10:23:34.167917+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:23:34.167917+0000","last_undegraded":"2024-09-02T10:23:34.167917+0000","last_fullsized":"2024-09-02T10:23:34.167917+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.545990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.545990+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:38.679683+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"626'12542","reported_seq":17207,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.168228+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:23:34.168228+0000","last_peered":"2024-09-02T10:23:34.168228+0000","last_clean":"2024-09-02T10:23:34.168228+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:23:34.168228+0000","last_undegraded":"2024-09-02T10:23:34.168228+0000","last_fullsized":"2024-09-02T10:23:34.168228+0000","mapping_epoch":559,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12542,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":770,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.185907+0000","last_change":"2024-09-02T10:23:00.146326+0000","last_active":"2024-09-02T10:23:34.185907+0000","last_peered":"2024-09-02T10:23:34.185907+0000","last_clean":"2024-09-02T10:23:34.185907+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:23:34.185907+0000","last_undegraded":"2024-09-02T10:23:34.185907+0000","last_fullsized":"2024-09-02T10:23:34.185907+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.146278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.146278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:09.373850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"626'12632","reported_seq":12983,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186590+0000","last_change":"2024-09-02T10:22:59.338566+0000","last_active":"2024-09-02T10:23:34.186590+0000","last_peered":"2024-09-02T10:23:34.186590+0000","last_clean":"2024-09-02T10:23:34.186590+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:23:34.186590+0000","last_undegraded":"2024-09-02T10:23:34.186590+0000","last_fullsized":"2024-09-02T10:23:34.186590+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12632","last_scrub_stamp":"2024-09-02T10:22:59.338517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:22:59.338517+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.748681+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1024,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186200+0000","last_change":"2024-09-02T10:21:40.149911+0000","last_active":"2024-09-02T10:23:34.186200+0000","last_peered":"2024-09-02T10:23:34.186200+0000","last_clean":"2024-09-02T10:23:34.186200+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:23:34.186200+0000","last_undegraded":"2024-09-02T10:23:34.186200+0000","last_fullsized":"2024-09-02T10:23:34.186200+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:40.149862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:40.149862+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"626'12586","reported_seq":12892,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.185869+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:23:34.185869+0000","last_peered":"2024-09-02T10:23:34.185869+0000","last_clean":"2024-09-02T10:23:34.185869+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:23:34.185869+0000","last_undegraded":"2024-09-02T10:23:34.185869+0000","last_fullsized":"2024-09-02T10:23:34.185869+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:37.448288+0000","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12586,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1291,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186155+0000","last_change":"2024-09-02T10:23:08.500316+0000","last_active":"2024-09-02T10:23:34.186155+0000","last_peered":"2024-09-02T10:23:34.186155+0000","last_clean":"2024-09-02T10:23:34.186155+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:23:34.186155+0000","last_undegraded":"2024-09-02T10:23:34.186155+0000","last_fullsized":"2024-09-02T10:23:34.186155+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:08.500268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:08.500268+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:14.723023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"626'12632","reported_seq":17195,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186594+0000","last_change":"2024-09-02T10:23:02.673893+0000","last_active":"2024-09-02T10:23:34.186594+0000","last_peered":"2024-09-02T10:23:34.186594+0000","last_clean":"2024-09-02T10:23:34.186594+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:23:34.186594+0000","last_undegraded":"2024-09-02T10:23:34.186594+0000","last_fullsized":"2024-09-02T10:23:34.186594+0000","mapping_epoch":565,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12632","last_scrub_stamp":"2024-09-02T10:23:02.673828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:02.673828+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:31.609539+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1035,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186007+0000","last_change":"2024-09-02T10:21:38.108542+0000","last_active":"2024-09-02T10:23:34.186007+0000","last_peered":"2024-09-02T10:23:34.186007+0000","last_clean":"2024-09-02T10:23:34.186007+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:23:34.186007+0000","last_undegraded":"2024-09-02T10:23:34.186007+0000","last_fullsized":"2024-09-02T10:23:34.186007+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:38.107558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:38.107558+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:38.486891+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"626'12650","reported_seq":12997,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.185950+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:23:34.185950+0000","last_peered":"2024-09-02T10:23:34.185950+0000","last_clean":"2024-09-02T10:23:34.185950+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:23:34.185950+0000","last_undegraded":"2024-09-02T10:23:34.185950+0000","last_fullsized":"2024-09-02T10:23:34.185950+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12650,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1298,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.167922+0000","last_change":"2024-09-02T10:23:00.386934+0000","last_active":"2024-09-02T10:23:34.167922+0000","last_peered":"2024-09-02T10:23:34.167922+0000","last_clean":"2024-09-02T10:23:34.167922+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:23:34.167922+0000","last_undegraded":"2024-09-02T10:23:34.167922+0000","last_fullsized":"2024-09-02T10:23:34.167922+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.386875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.386875+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.366845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"626'12496","reported_seq":12807,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186334+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:23:34.186334+0000","last_peered":"2024-09-02T10:23:34.186334+0000","last_clean":"2024-09-02T10:23:34.186334+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:23:34.186334+0000","last_undegraded":"2024-09-02T10:23:34.186334+0000","last_fullsized":"2024-09-02T10:23:34.186334+0000","mapping_epoch":533,"log_start":"553'2400","ondisk_log_start":"553'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":239,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186322+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:23:34.186322+0000","last_peered":"2024-09-02T10:23:34.186322+0000","last_clean":"2024-09-02T10:23:34.186322+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:23:34.186322+0000","last_undegraded":"2024-09-02T10:23:34.186322+0000","last_fullsized":"2024-09-02T10:23:34.186322+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:36.571496+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"626'12520","reported_seq":12842,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186377+0000","last_change":"2024-09-02T10:23:06.653266+0000","last_active":"2024-09-02T10:23:34.186377+0000","last_peered":"2024-09-02T10:23:34.186377+0000","last_clean":"2024-09-02T10:23:34.186377+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:23:34.186377+0000","last_undegraded":"2024-09-02T10:23:34.186377+0000","last_fullsized":"2024-09-02T10:23:34.186377+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12520","last_scrub_stamp":"2024-09-02T10:23:06.653191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.653191+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:17.096908+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12520,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1070,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.186764+0000","last_change":"2024-09-02T10:22:30.358040+0000","last_active":"2024-09-02T10:23:34.186764+0000","last_peered":"2024-09-02T10:23:34.186764+0000","last_clean":"2024-09-02T10:23:34.186764+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:23:34.186764+0000","last_undegraded":"2024-09-02T10:23:34.186764+0000","last_fullsized":"2024-09-02T10:23:34.186764+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:30.357970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:30.357970+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:41.221263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"626'12566","reported_seq":12876,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.167957+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:23:34.167957+0000","last_peered":"2024-09-02T10:23:34.167957+0000","last_clean":"2024-09-02T10:23:34.167957+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:23:34.167957+0000","last_undegraded":"2024-09-02T10:23:34.167957+0000","last_fullsized":"2024-09-02T10:23:34.167957+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":452,"reported_epoch":650,"state":"active+clean","last_fresh":"2024-09-02T10:23:34.168164+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:23:34.168164+0000","last_peered":"2024-09-02T10:23:34.168164+0000","last_clean":"2024-09-02T10:23:34.168164+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:23:34.168164+0000","last_undegraded":"2024-09-02T10:23:34.168164+0000","last_fullsized":"2024-09-02T10:23:34.168164+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2561,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31075,"num_bytes_recovered":1286144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270938112,"data_stored":270925896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383962,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487676,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93257476,"statfs":{"total":95995035648,"available":95495655424,"internally_reserved":0,"allocated":1155072,"data_stored":732416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":498203076},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612895,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357184,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93387968,"statfs":{"total":95995035648,"available":95629279232,"internally_reserved":0,"allocated":1159168,"data_stored":732440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841828,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545980,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199172,"statfs":{"total":95995035648,"available":95435952128,"internally_reserved":0,"allocated":1155072,"data_stored":732416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070762,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339132,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93406020,"statfs":{"total":95995035648,"available":95647764480,"internally_reserved":0,"allocated":1155072,"data_stored":732416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299695,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582280,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162872,"statfs":{"total":95995035648,"available":95398780928,"internally_reserved":0,"allocated":1359872,"data_stored":933664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528626,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365820,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379332,"statfs":{"total":95995035648,"available":95620435968,"internally_reserved":0,"allocated":1482752,"data_stored":1060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724855,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":878792,"kb_used_data":265908,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92866360,"statfs":{"total":95995035648,"available":95095152640,"internally_reserved":0,"allocated":272289792,"data_stored":271859512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953789,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334976,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410176,"statfs":{"total":95995035648,"available":95652020224,"internally_reserved":0,"allocated":1155072,"data_stored":732416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150018,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569920,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175232,"statfs":{"total":95995035648,"available":95411437568,"internally_reserved":0,"allocated":1482752,"data_stored":1060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411653,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307396,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93437756,"statfs":{"total":95995035648,"available":95680262144,"internally_reserved":0,"allocated":1359872,"data_stored":933664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575179,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509892,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235260,"statfs":{"total":95995035648,"available":95472906240,"internally_reserved":0,"allocated":1159168,"data_stored":732440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738705,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592128,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93153024,"statfs":{"total":95995035648,"available":95388696576,"internally_reserved":0,"allocated":1482752,"data_stored":1060096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":270929920,"data_stored":270925848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:23:41.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:41 smithi162 ceph-mon[20279]: from='client.17757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:41.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:41.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:41.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:41.885 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:41 smithi151 ceph-mon[16639]: from='client.17757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:41.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:41.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:41.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:41.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:41.992 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:41 smithi187 ceph-mon[20072]: from='client.17757 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:42.029 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:42.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:42.097 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:42.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:42.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:42.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:42.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:42.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:42.332 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:42.358 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:42.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:42.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:42.486 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:42.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:42.505 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:23:42.505 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:23:42.570 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:23:42.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:42.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:42.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:42.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:42.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:42.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:42.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:42 smithi151 ceph-mon[16639]: pgmap v1512: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2561/9 objects degraded (28455.556%); 0 B/s, 87 objects/s recovering 2024-09-02T10:23:42.763 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3612094380' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:23:42.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:42.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:42 smithi162 ceph-mon[20279]: pgmap v1512: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2561/9 objects degraded (28455.556%); 0 B/s, 87 objects/s recovering 2024-09-02T10:23:42.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3612094380' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:23:42.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:42.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:42.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:42.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:42.994 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:42 smithi187 ceph-mon[20072]: pgmap v1512: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2561/9 objects degraded (28455.556%); 0 B/s, 87 objects/s recovering 2024-09-02T10:23:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3612094380' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:23:43.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:43.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:43.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:43.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:43.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:43.245 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:43.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:43.347 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:43.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:43.374 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:43.475 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:43.501 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:43.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:43.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:43.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:43.629 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:43.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:43.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:43.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:43.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:43 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:23:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3612094380' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:23:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:43 smithi162 ceph-mon[20279]: osdmap e652: 12 total, 12 up, 12 in 2024-09-02T10:23:43.854 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:43.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:43.881 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:43 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:23:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3612094380' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:23:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:43 smithi151 ceph-mon[16639]: osdmap e652: 12 total, 12 up, 12 in 2024-09-02T10:23:43.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:44.004 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:44.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:43 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:23:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3612094380' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:23:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:43 smithi187 ceph-mon[20072]: osdmap e652: 12 total, 12 up, 12 in 2024-09-02T10:23:44.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:44.131 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:44.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:44.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:44.259 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:44.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:44.361 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:44.387 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:44.388 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:44.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:44.518 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:44.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:44.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:44.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:44.644 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:44.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:44.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:23:44.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:44.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:44 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:23:44.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:44 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:23:44.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:44 smithi151 ceph-mon[16639]: pgmap v1514: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2052/9 objects degraded (22800.000%); 0 B/s, 151 objects/s recovering 2024-09-02T10:23:44.754 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:44 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2052/9 objects degraded (22800.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:44.772 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:44.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:44.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:44 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:23:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:44 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:23:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:44 smithi162 ceph-mon[20279]: pgmap v1514: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2052/9 objects degraded (22800.000%); 0 B/s, 151 objects/s recovering 2024-09-02T10:23:44.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:44 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2052/9 objects degraded (22800.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:44.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:44.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:44.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:45.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:44 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:23:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:44 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:23:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:44 smithi187 ceph-mon[20072]: pgmap v1514: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2052/9 objects degraded (22800.000%); 0 B/s, 151 objects/s recovering 2024-09-02T10:23:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:44 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2052/9 objects degraded (22800.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:45.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:45.029 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:45.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:45.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:45.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:45.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:45.283 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:45.285 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:45.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:45.411 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:45.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:45.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:45.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:45.537 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:45.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:45.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:23:45.639 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:45.663 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:45.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:45.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:45.793 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:45.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:45 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:23:45.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:45 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:23:45.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:45.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:45.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:45 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:23:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:45 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:23:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:45 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:23:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:45 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:23:46.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:46.048 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:46.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:46.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:46.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:23:46.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:46.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:46.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:46.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:46.301 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:46.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:46.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:46.424 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:46.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:46.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:46.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:46.552 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:46.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:46.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:46 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:23:46.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:46 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:23:46.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:46 smithi151 ceph-mon[16639]: pgmap v1515: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2052/9 objects degraded (22800.000%); 0 B/s, 151 objects/s recovering 2024-09-02T10:23:46.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:46 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:23:46.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:46 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:23:46.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:46 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:23:46.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:46 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:23:46.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:46 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:23:46.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:46 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:23:46.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:46.662 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:23:46.663 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:23:46.670 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:46.670 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:23:46.670 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:23:46.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:46.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:46.703 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:23:46.703 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:23:46.711 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:46.727 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:23:46.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:23:46.749 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:23:46.750 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:23:46.771 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:23:46.771 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:23:46.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:46.793 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:23:46.793 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:23:46.808 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:46.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:46.813 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:23:46.813 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:23:46.834 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:23:46.835 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:23:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:46 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:23:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:46 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:23:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:46 smithi162 ceph-mon[20279]: pgmap v1515: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2052/9 objects degraded (22800.000%); 0 B/s, 151 objects/s recovering 2024-09-02T10:23:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:46 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:23:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:46 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:23:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:46 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:23:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:46 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:23:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:46 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:23:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:46 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:23:46.861 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:23:46.862 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:23:46.883 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:23:46.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:23:46.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:23:46.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:23:46.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:46.936 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:46.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:46.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:46 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:23:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:46 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:23:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:46 smithi187 ceph-mon[20072]: pgmap v1515: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2052/9 objects degraded (22800.000%); 0 B/s, 151 objects/s recovering 2024-09-02T10:23:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:46 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:23:47.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:46 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:23:47.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:46 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:23:47.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:46 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:23:47.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:46 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:23:47.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:46 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:23:47.037 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:47.063 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:47.065 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:47.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:47.192 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:47.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:47.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:47.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:47.319 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:47.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:47.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:47.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:47.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:47.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:47.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:47.576 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:47.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:47.577 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:47.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:47 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:23:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:47 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:23:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:47 smithi151 ceph-mon[16639]: 6.8 scrub starts 2024-09-02T10:23:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:47 smithi151 ceph-mon[16639]: 6.8 scrub ok 2024-09-02T10:23:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/948749594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:47.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:47.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:47.763 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:23:47.764 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:23:47.805 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:47.840 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:47.841 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:47.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:47.968 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:47.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:47.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:47 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:23:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:47 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:23:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:47 smithi187 ceph-mon[20072]: 6.8 scrub starts 2024-09-02T10:23:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:47 smithi187 ceph-mon[20072]: 6.8 scrub ok 2024-09-02T10:23:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/948749594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:48.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:47 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:23:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:47 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:23:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:47 smithi162 ceph-mon[20279]: 6.8 scrub starts 2024-09-02T10:23:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:47 smithi162 ceph-mon[20279]: 6.8 scrub ok 2024-09-02T10:23:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/948749594' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:48.098 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:48.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:48.099 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:48.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:48.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:48.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:48.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:48.351 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:48.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:48.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:48.474 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:48.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:48.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:48.604 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:48.605 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:48.606 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1517,"stamp":"2024-09-02T10:23:47.599428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31588,"num_bytes_recovered":1287075424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163791,"ondisk_log_size":163791,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5879712,"kb_used_data":279936,"kb_used_omap":18,"kb_used_meta":5599533,"kb_avail":1119062112,"statfs":{"total":1151940427776,"available":1145919602688,"internally_reserved":0,"allocated":286654464,"data_stored":281467416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5733922107},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.042248"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":674,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:46.278826+0000","last_change":"2024-09-02T10:23:46.278826+0000","last_active":"2024-09-02T10:23:46.278826+0000","last_peered":"2024-09-02T10:23:46.278826+0000","last_clean":"2024-09-02T10:23:46.278826+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:23:46.278826+0000","last_undegraded":"2024-09-02T10:23:46.278826+0000","last_fullsized":"2024-09-02T10:23:46.278826+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:46.278766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:46.278766+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:11.240675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":473,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.439242+0000","last_change":"2024-09-02T10:20:52.103883+0000","last_active":"2024-09-02T10:23:43.439242+0000","last_peered":"2024-09-02T10:23:43.439242+0000","last_clean":"2024-09-02T10:23:43.439242+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:23:43.439242+0000","last_undegraded":"2024-09-02T10:23:43.439242+0000","last_fullsized":"2024-09-02T10:23:43.439242+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.121618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.121618+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":931,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.573858+0000","last_change":"2024-09-02T10:22:32.860224+0000","last_active":"2024-09-02T10:23:42.573858+0000","last_peered":"2024-09-02T10:23:42.573858+0000","last_clean":"2024-09-02T10:23:42.573858+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:23:42.573858+0000","last_undegraded":"2024-09-02T10:23:42.573858+0000","last_fullsized":"2024-09-02T10:23:42.573858+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:32.860180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:32.860180+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":923,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.573820+0000","last_change":"2024-09-02T10:22:04.875381+0000","last_active":"2024-09-02T10:23:42.573820+0000","last_peered":"2024-09-02T10:23:42.573820+0000","last_clean":"2024-09-02T10:23:42.573820+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:23:42.573820+0000","last_undegraded":"2024-09-02T10:23:42.573820+0000","last_fullsized":"2024-09-02T10:23:42.573820+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:04.875301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:04.875301+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:47.184519+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":227,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.504984+0000","last_change":"2024-09-02T10:20:23.232765+0000","last_active":"2024-09-02T10:23:43.504984+0000","last_peered":"2024-09-02T10:23:43.504984+0000","last_clean":"2024-09-02T10:23:43.504984+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:23:43.504984+0000","last_undegraded":"2024-09-02T10:23:43.504984+0000","last_fullsized":"2024-09-02T10:23:43.504984+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:23.232706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:23.232706+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:47.772472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":772,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.383033+0000","last_change":"2024-09-02T10:23:01.185167+0000","last_active":"2024-09-02T10:23:43.383033+0000","last_peered":"2024-09-02T10:23:43.383033+0000","last_clean":"2024-09-02T10:23:43.383033+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:23:43.383033+0000","last_undegraded":"2024-09-02T10:23:43.383033+0000","last_fullsized":"2024-09-02T10:23:43.383033+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.185112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.185112+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.992983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1308,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.004221+0000","last_change":"2024-09-02T10:23:09.691782+0000","last_active":"2024-09-02T10:23:43.004221+0000","last_peered":"2024-09-02T10:23:43.004221+0000","last_clean":"2024-09-02T10:23:43.004221+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:23:43.004221+0000","last_undegraded":"2024-09-02T10:23:43.004221+0000","last_fullsized":"2024-09-02T10:23:43.004221+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:09.691691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:09.691691+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.662853+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":947,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.573983+0000","last_change":"2024-09-02T10:23:05.131265+0000","last_active":"2024-09-02T10:23:42.573983+0000","last_peered":"2024-09-02T10:23:42.573983+0000","last_clean":"2024-09-02T10:23:42.573983+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:23:42.573983+0000","last_undegraded":"2024-09-02T10:23:42.573983+0000","last_fullsized":"2024-09-02T10:23:42.573983+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.131213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.131213+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.083779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":938,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.439056+0000","last_change":"2024-09-02T10:23:06.476818+0000","last_active":"2024-09-02T10:23:43.439056+0000","last_peered":"2024-09-02T10:23:43.439056+0000","last_clean":"2024-09-02T10:23:43.439056+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:23:43.439056+0000","last_undegraded":"2024-09-02T10:23:43.439056+0000","last_fullsized":"2024-09-02T10:23:43.439056+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:06.476768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.476768+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:35.403622+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":915,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.664448+0000","last_change":"2024-09-02T10:22:07.403894+0000","last_active":"2024-09-02T10:23:42.664448+0000","last_peered":"2024-09-02T10:23:42.664448+0000","last_clean":"2024-09-02T10:23:42.664448+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:23:42.664448+0000","last_undegraded":"2024-09-02T10:23:42.664448+0000","last_fullsized":"2024-09-02T10:23:42.664448+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:07.403797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:07.403797+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":159,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.410480+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:23:43.410480+0000","last_peered":"2024-09-02T10:23:43.410480+0000","last_clean":"2024-09-02T10:23:43.410480+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:23:43.410480+0000","last_undegraded":"2024-09-02T10:23:43.410480+0000","last_fullsized":"2024-09-02T10:23:43.410480+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1264,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.438980+0000","last_change":"2024-09-02T10:20:45.804848+0000","last_active":"2024-09-02T10:23:43.438980+0000","last_peered":"2024-09-02T10:23:43.438980+0000","last_clean":"2024-09-02T10:23:43.438980+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:23:43.438980+0000","last_undegraded":"2024-09-02T10:23:43.438980+0000","last_fullsized":"2024-09-02T10:23:43.438980+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:45.804782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:45.804782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1207,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.004335+0000","last_change":"2024-09-02T10:22:55.516683+0000","last_active":"2024-09-02T10:23:43.004335+0000","last_peered":"2024-09-02T10:23:43.004335+0000","last_clean":"2024-09-02T10:23:43.004335+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:23:43.004335+0000","last_undegraded":"2024-09-02T10:23:43.004335+0000","last_fullsized":"2024-09-02T10:23:43.004335+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.516633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.516633+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:58.850702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1175,"reported_epoch":651,"state":"active+clean","last_fresh":"2024-09-02T10:23:39.832441+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:23:39.832441+0000","last_peered":"2024-09-02T10:23:39.832441+0000","last_clean":"2024-09-02T10:23:39.832441+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:23:39.832441+0000","last_undegraded":"2024-09-02T10:23:39.832441+0000","last_fullsized":"2024-09-02T10:23:39.832441+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1177,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.004065+0000","last_change":"2024-09-02T10:23:00.621966+0000","last_active":"2024-09-02T10:23:43.004065+0000","last_peered":"2024-09-02T10:23:43.004065+0000","last_clean":"2024-09-02T10:23:43.004065+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:23:43.004065+0000","last_undegraded":"2024-09-02T10:23:43.004065+0000","last_fullsized":"2024-09-02T10:23:43.004065+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.621916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.621916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:21.445915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"626'12394","reported_seq":20750,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.410583+0000","last_change":"2024-09-02T10:22:56.637995+0000","last_active":"2024-09-02T10:23:43.410583+0000","last_peered":"2024-09-02T10:23:43.410583+0000","last_clean":"2024-09-02T10:23:43.410583+0000","last_became_active":"2024-09-02T10:22:56.595915+0000","last_became_peered":"2024-09-02T10:22:56.595915+0000","last_unstale":"2024-09-02T10:23:43.410583+0000","last_undegraded":"2024-09-02T10:23:43.410583+0000","last_fullsized":"2024-09-02T10:23:43.410583+0000","mapping_epoch":635,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12394,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":237961216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1271,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.004233+0000","last_change":"2024-09-02T10:23:01.892373+0000","last_active":"2024-09-02T10:23:43.004233+0000","last_peered":"2024-09-02T10:23:43.004233+0000","last_clean":"2024-09-02T10:23:43.004233+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:23:43.004233+0000","last_undegraded":"2024-09-02T10:23:43.004233+0000","last_fullsized":"2024-09-02T10:23:43.004233+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.892288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.892288+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:30.607091+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"626'12580","reported_seq":29722,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.505095+0000","last_change":"2024-09-02T10:23:09.693242+0000","last_active":"2024-09-02T10:23:43.505095+0000","last_peered":"2024-09-02T10:23:43.505095+0000","last_clean":"2024-09-02T10:23:43.505095+0000","last_became_active":"2024-09-02T10:23:02.182530+0000","last_became_peered":"2024-09-02T10:23:02.182530+0000","last_unstale":"2024-09-02T10:23:43.505095+0000","last_undegraded":"2024-09-02T10:23:43.505095+0000","last_fullsized":"2024-09-02T10:23:43.505095+0000","mapping_epoch":640,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12580","last_scrub_stamp":"2024-09-02T10:23:09.693181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:09.693181+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:24.508961+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8401,"num_bytes_recovered":374538240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1394,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.664314+0000","last_change":"2024-09-02T10:22:37.332903+0000","last_active":"2024-09-02T10:23:42.664314+0000","last_peered":"2024-09-02T10:23:42.664314+0000","last_clean":"2024-09-02T10:23:42.664314+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:23:42.664314+0000","last_undegraded":"2024-09-02T10:23:42.664314+0000","last_fullsized":"2024-09-02T10:23:42.664314+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:37.332859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:37.332859+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.675027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"626'12351","reported_seq":12670,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.004177+0000","last_change":"2024-09-02T10:23:04.675691+0000","last_active":"2024-09-02T10:23:43.004177+0000","last_peered":"2024-09-02T10:23:43.004177+0000","last_clean":"2024-09-02T10:23:43.004177+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:23:43.004177+0000","last_undegraded":"2024-09-02T10:23:43.004177+0000","last_fullsized":"2024-09-02T10:23:43.004177+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12351","last_scrub_stamp":"2024-09-02T10:23:04.675613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:04.675613+0000","objects_scrubbed":1,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.698140+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12351,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1034,"reported_epoch":651,"state":"active+clean","last_fresh":"2024-09-02T10:23:39.832779+0000","last_change":"2024-09-02T10:22:55.068228+0000","last_active":"2024-09-02T10:23:39.832779+0000","last_peered":"2024-09-02T10:23:39.832779+0000","last_clean":"2024-09-02T10:23:39.832779+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:23:39.832779+0000","last_undegraded":"2024-09-02T10:23:39.832779+0000","last_fullsized":"2024-09-02T10:23:39.832779+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.068166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.068166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:04.696428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"626'12382","reported_seq":12708,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.571443+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:23:42.571443+0000","last_peered":"2024-09-02T10:23:42.571443+0000","last_clean":"2024-09-02T10:23:42.571443+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:23:42.571443+0000","last_undegraded":"2024-09-02T10:23:42.571443+0000","last_fullsized":"2024-09-02T10:23:42.571443+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":441,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:45.263493+0000","last_change":"2024-09-02T10:23:45.263493+0000","last_active":"2024-09-02T10:23:45.263493+0000","last_peered":"2024-09-02T10:23:45.263493+0000","last_clean":"2024-09-02T10:23:45.263493+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:23:45.263493+0000","last_undegraded":"2024-09-02T10:23:45.263493+0000","last_fullsized":"2024-09-02T10:23:45.263493+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:45.263415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:45.263415+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:04.210025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"626'12606","reported_seq":12957,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.664540+0000","last_change":"2024-09-02T10:23:11.425512+0000","last_active":"2024-09-02T10:23:42.664540+0000","last_peered":"2024-09-02T10:23:42.664540+0000","last_clean":"2024-09-02T10:23:42.664540+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:23:42.664540+0000","last_undegraded":"2024-09-02T10:23:42.664540+0000","last_fullsized":"2024-09-02T10:23:42.664540+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12606","last_scrub_stamp":"2024-09-02T10:23:11.425432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:11.425432+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:12.626473+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1342,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.633350+0000","last_change":"2024-09-02T10:22:34.647196+0000","last_active":"2024-09-02T10:23:42.633350+0000","last_peered":"2024-09-02T10:23:42.633350+0000","last_clean":"2024-09-02T10:23:42.633350+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:23:42.633350+0000","last_undegraded":"2024-09-02T10:23:42.633350+0000","last_fullsized":"2024-09-02T10:23:42.633350+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:34.647144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:34.647144+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:02.024298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"626'12544","reported_seq":29699,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.505000+0000","last_change":"2024-09-02T10:23:07.704034+0000","last_active":"2024-09-02T10:23:43.505000+0000","last_peered":"2024-09-02T10:23:43.505000+0000","last_clean":"2024-09-02T10:23:43.505000+0000","last_became_active":"2024-09-02T10:22:57.600280+0000","last_became_peered":"2024-09-02T10:22:57.600280+0000","last_unstale":"2024-09-02T10:23:43.505000+0000","last_undegraded":"2024-09-02T10:23:43.505000+0000","last_fullsized":"2024-09-02T10:23:43.505000+0000","mapping_epoch":636,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12544","last_scrub_stamp":"2024-09-02T10:23:07.703956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:07.703956+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:08.077242+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8407,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":808,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.383252+0000","last_change":"2024-09-02T10:23:05.243285+0000","last_active":"2024-09-02T10:23:43.383252+0000","last_peered":"2024-09-02T10:23:43.383252+0000","last_clean":"2024-09-02T10:23:43.383252+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:23:43.383252+0000","last_undegraded":"2024-09-02T10:23:43.383252+0000","last_fullsized":"2024-09-02T10:23:43.383252+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.243236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.243236+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:16.091958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"626'12380","reported_seq":12733,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.573941+0000","last_change":"2024-09-02T10:23:07.151869+0000","last_active":"2024-09-02T10:23:42.573941+0000","last_peered":"2024-09-02T10:23:42.573941+0000","last_clean":"2024-09-02T10:23:42.573941+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:23:42.573941+0000","last_undegraded":"2024-09-02T10:23:42.573941+0000","last_fullsized":"2024-09-02T10:23:42.573941+0000","mapping_epoch":533,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12380","last_scrub_stamp":"2024-09-02T10:23:07.151772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:07.151772+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:33.568723+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12380,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":773,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.383030+0000","last_change":"2024-09-02T10:22:55.086083+0000","last_active":"2024-09-02T10:23:43.383030+0000","last_peered":"2024-09-02T10:23:43.383030+0000","last_clean":"2024-09-02T10:23:43.383030+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:23:43.383030+0000","last_undegraded":"2024-09-02T10:23:43.383030+0000","last_fullsized":"2024-09-02T10:23:43.383030+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.086036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.086036+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.714243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"626'12678","reported_seq":21562,"reported_epoch":651,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:23:43.013921+0000","last_change":"2024-09-02T10:22:55.007293+0000","last_active":"2024-09-02T10:23:43.013921+0000","last_peered":"2024-09-02T10:23:43.013921+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:23:43.013921+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4287,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-2052}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":541,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.635086+0000","last_change":"2024-09-02T10:23:42.635086+0000","last_active":"2024-09-02T10:23:42.635086+0000","last_peered":"2024-09-02T10:23:42.635086+0000","last_clean":"2024-09-02T10:23:42.635086+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:23:42.635086+0000","last_undegraded":"2024-09-02T10:23:42.635086+0000","last_fullsized":"2024-09-02T10:23:42.635086+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:42.635029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:42.635029+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:11.935109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"626'12542","reported_seq":17211,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.571462+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:23:42.571462+0000","last_peered":"2024-09-02T10:23:42.571462+0000","last_clean":"2024-09-02T10:23:42.571462+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:23:42.571462+0000","last_undegraded":"2024-09-02T10:23:42.571462+0000","last_fullsized":"2024-09-02T10:23:42.571462+0000","mapping_epoch":559,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12542,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":774,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.383312+0000","last_change":"2024-09-02T10:23:00.146326+0000","last_active":"2024-09-02T10:23:43.383312+0000","last_peered":"2024-09-02T10:23:43.383312+0000","last_clean":"2024-09-02T10:23:43.383312+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:23:43.383312+0000","last_undegraded":"2024-09-02T10:23:43.383312+0000","last_fullsized":"2024-09-02T10:23:43.383312+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.146278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.146278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:09.373850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"626'12632","reported_seq":12987,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.664362+0000","last_change":"2024-09-02T10:22:59.338566+0000","last_active":"2024-09-02T10:23:42.664362+0000","last_peered":"2024-09-02T10:23:42.664362+0000","last_clean":"2024-09-02T10:23:42.664362+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:23:42.664362+0000","last_undegraded":"2024-09-02T10:23:42.664362+0000","last_fullsized":"2024-09-02T10:23:42.664362+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12632","last_scrub_stamp":"2024-09-02T10:22:59.338517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:22:59.338517+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.748681+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1036,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:44.545777+0000","last_change":"2024-09-02T10:23:44.545777+0000","last_active":"2024-09-02T10:23:44.545777+0000","last_peered":"2024-09-02T10:23:44.545777+0000","last_clean":"2024-09-02T10:23:44.545777+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:23:44.545777+0000","last_undegraded":"2024-09-02T10:23:44.545777+0000","last_fullsized":"2024-09-02T10:23:44.545777+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:44.545693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:44.545693+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:53.740167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"626'12586","reported_seq":12896,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.439109+0000","last_change":"2024-09-02T10:19:53.592192+0000","last_active":"2024-09-02T10:23:43.439109+0000","last_peered":"2024-09-02T10:23:43.439109+0000","last_clean":"2024-09-02T10:23:43.439109+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:23:43.439109+0000","last_undegraded":"2024-09-02T10:23:43.439109+0000","last_fullsized":"2024-09-02T10:23:43.439109+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"566'5208","last_scrub_stamp":"2024-09-02T10:19:53.592053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:53.592053+0000","objects_scrubbed":5205,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1418,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12586,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1295,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.439155+0000","last_change":"2024-09-02T10:23:08.500316+0000","last_active":"2024-09-02T10:23:43.439155+0000","last_peered":"2024-09-02T10:23:43.439155+0000","last_clean":"2024-09-02T10:23:43.439155+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:23:43.439155+0000","last_undegraded":"2024-09-02T10:23:43.439155+0000","last_fullsized":"2024-09-02T10:23:43.439155+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:08.500268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:08.500268+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:14.723023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"626'12632","reported_seq":17199,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.004331+0000","last_change":"2024-09-02T10:23:02.673893+0000","last_active":"2024-09-02T10:23:43.004331+0000","last_peered":"2024-09-02T10:23:43.004331+0000","last_clean":"2024-09-02T10:23:43.004331+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:23:43.004331+0000","last_undegraded":"2024-09-02T10:23:43.004331+0000","last_fullsized":"2024-09-02T10:23:43.004331+0000","mapping_epoch":565,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12632","last_scrub_stamp":"2024-09-02T10:23:02.673828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:02.673828+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:31.609539+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1047,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:45.553424+0000","last_change":"2024-09-02T10:23:45.553424+0000","last_active":"2024-09-02T10:23:45.553424+0000","last_peered":"2024-09-02T10:23:45.553424+0000","last_clean":"2024-09-02T10:23:45.553424+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:23:45.553424+0000","last_undegraded":"2024-09-02T10:23:45.553424+0000","last_fullsized":"2024-09-02T10:23:45.553424+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:45.553364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:45.553364+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:06.163823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"626'12650","reported_seq":13001,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.664264+0000","last_change":"2024-09-02T10:21:35.443947+0000","last_active":"2024-09-02T10:23:42.664264+0000","last_peered":"2024-09-02T10:23:42.664264+0000","last_clean":"2024-09-02T10:23:42.664264+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:23:42.664264+0000","last_undegraded":"2024-09-02T10:23:42.664264+0000","last_fullsized":"2024-09-02T10:23:42.664264+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'10135","last_scrub_stamp":"2024-09-02T10:21:35.443742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:21:35.443742+0000","objects_scrubbed":2539,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:45.541673+0000","scrub_duration":1332,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12650,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1302,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.572352+0000","last_change":"2024-09-02T10:23:00.386934+0000","last_active":"2024-09-02T10:23:42.572352+0000","last_peered":"2024-09-02T10:23:42.572352+0000","last_clean":"2024-09-02T10:23:42.572352+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:23:42.572352+0000","last_undegraded":"2024-09-02T10:23:42.572352+0000","last_fullsized":"2024-09-02T10:23:42.572352+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.386875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.386875+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.366845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"626'12496","reported_seq":12809,"reported_epoch":651,"state":"active+clean","last_fresh":"2024-09-02T10:23:39.832237+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:23:39.832237+0000","last_peered":"2024-09-02T10:23:39.832237+0000","last_clean":"2024-09-02T10:23:39.832237+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:23:39.832237+0000","last_undegraded":"2024-09-02T10:23:39.832237+0000","last_fullsized":"2024-09-02T10:23:39.832237+0000","mapping_epoch":533,"log_start":"553'2400","ondisk_log_start":"553'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":243,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.505026+0000","last_change":"2024-09-02T10:20:52.105222+0000","last_active":"2024-09-02T10:23:43.505026+0000","last_peered":"2024-09-02T10:23:43.505026+0000","last_clean":"2024-09-02T10:23:43.505026+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:23:43.505026+0000","last_undegraded":"2024-09-02T10:23:43.505026+0000","last_fullsized":"2024-09-02T10:23:43.505026+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:26.313597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:26.313597+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:45.745886+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"626'12520","reported_seq":12846,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:43.004125+0000","last_change":"2024-09-02T10:23:06.653266+0000","last_active":"2024-09-02T10:23:43.004125+0000","last_peered":"2024-09-02T10:23:43.004125+0000","last_clean":"2024-09-02T10:23:43.004125+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:23:43.004125+0000","last_undegraded":"2024-09-02T10:23:43.004125+0000","last_fullsized":"2024-09-02T10:23:43.004125+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12520","last_scrub_stamp":"2024-09-02T10:23:06.653191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.653191+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:17.096908+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12520,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1074,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.664480+0000","last_change":"2024-09-02T10:22:30.358040+0000","last_active":"2024-09-02T10:23:42.664480+0000","last_peered":"2024-09-02T10:23:42.664480+0000","last_clean":"2024-09-02T10:23:42.664480+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:23:42.664480+0000","last_undegraded":"2024-09-02T10:23:42.664480+0000","last_fullsized":"2024-09-02T10:23:42.664480+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:30.357970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:30.357970+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"626'12566","reported_seq":12880,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.571338+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:23:42.571338+0000","last_peered":"2024-09-02T10:23:42.571338+0000","last_clean":"2024-09-02T10:23:42.571338+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:23:42.571338+0000","last_undegraded":"2024-09-02T10:23:42.571338+0000","last_fullsized":"2024-09-02T10:23:42.571338+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":456,"reported_epoch":652,"state":"active+clean","last_fresh":"2024-09-02T10:23:42.571476+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:23:42.571476+0000","last_peered":"2024-09-02T10:23:42.571476+0000","last_clean":"2024-09-02T10:23:42.571476+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:23:42.571476+0000","last_undegraded":"2024-09-02T10:23:42.571476+0000","last_fullsized":"2024-09-02T10:23:42.571476+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2052,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31584,"num_bytes_recovered":1286144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270938112,"data_stored":270925896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383964,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487696,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93257456,"statfs":{"total":95995035648,"available":95495634944,"internally_reserved":0,"allocated":1179648,"data_stored":748194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":498203076},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612897,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357208,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93387944,"statfs":{"total":95995035648,"available":95629254656,"internally_reserved":0,"allocated":1183744,"data_stored":748218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841829,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":545992,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199160,"statfs":{"total":95995035648,"available":95435939840,"internally_reserved":0,"allocated":1167360,"data_stored":740330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070763,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339156,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93405996,"statfs":{"total":95995035648,"available":95647739904,"internally_reserved":0,"allocated":1179648,"data_stored":748194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299697,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582296,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162856,"statfs":{"total":95995035648,"available":95398764544,"internally_reserved":0,"allocated":1384448,"data_stored":949442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528628,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365840,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379312,"statfs":{"total":95995035648,"available":95620415488,"internally_reserved":0,"allocated":1507328,"data_stored":1075874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724857,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":878816,"kb_used_data":265932,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92866336,"statfs":{"total":95995035648,"available":95095128064,"internally_reserved":0,"allocated":272314368,"data_stored":271875290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953790,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334996,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410156,"statfs":{"total":95995035648,"available":95651999744,"internally_reserved":0,"allocated":1179648,"data_stored":748194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150019,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569940,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175212,"statfs":{"total":95995035648,"available":95411417088,"internally_reserved":0,"allocated":1507328,"data_stored":1075874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411655,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315736,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":314366,"kb_avail":93429416,"statfs":{"total":95995035648,"available":95671721984,"internally_reserved":0,"allocated":1384448,"data_stored":949442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":321911237},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575180,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509900,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235252,"statfs":{"total":95995035648,"available":95472898048,"internally_reserved":0,"allocated":1171456,"data_stored":740354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738706,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592136,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93153016,"statfs":{"total":95995035648,"available":95388688384,"internally_reserved":0,"allocated":1495040,"data_stored":1068010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":270929920,"data_stored":270925848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:23:48.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph status --format=json 2024-09-02T10:23:48.663 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:48.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:48.730 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:48.731 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:48.832 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:48.848 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:48 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:23:48.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:48 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:23:48.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:48 smithi151 ceph-mon[16639]: pgmap v1516: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2052/9 objects degraded (22800.000%); 0 B/s, 63 objects/s recovering 2024-09-02T10:23:48.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:48 smithi151 ceph-mon[16639]: osdmap e653: 12 total, 12 up, 12 in 2024-09-02T10:23:48.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:48 smithi151 ceph-mon[16639]: from='client.17772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:48.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:48.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:48.959 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:48.984 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:48.985 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:48 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:23:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:48 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:23:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:48 smithi187 ceph-mon[20072]: pgmap v1516: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2052/9 objects degraded (22800.000%); 0 B/s, 63 objects/s recovering 2024-09-02T10:23:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:48 smithi187 ceph-mon[20072]: osdmap e653: 12 total, 12 up, 12 in 2024-09-02T10:23:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:48 smithi187 ceph-mon[20072]: from='client.17772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:49.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:48 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:23:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:48 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:23:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:48 smithi162 ceph-mon[20279]: pgmap v1516: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2052/9 objects degraded (22800.000%); 0 B/s, 63 objects/s recovering 2024-09-02T10:23:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:48 smithi162 ceph-mon[20279]: osdmap e653: 12 total, 12 up, 12 in 2024-09-02T10:23:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:48 smithi162 ceph-mon[20279]: from='client.17772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:49.112 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:49.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:49.213 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:49.237 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:49.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:49.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:49.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:49.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:49.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:49.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:49.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:49.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:49.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:49.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:49 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:23:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:49 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:23:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:49 smithi151 ceph-mon[16639]: osdmap e654: 12 total, 12 up, 12 in 2024-09-02T10:23:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:49 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:23:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:49 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:23:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:49 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:23:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:49 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:23:49.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:49.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:49.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:49.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:49.871 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:49.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:49.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:49.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:49.996 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:49.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:49 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:23:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:49 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:23:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:49 smithi187 ceph-mon[20072]: osdmap e654: 12 total, 12 up, 12 in 2024-09-02T10:23:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:49 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:23:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:49 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:23:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:49 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:23:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:49 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:23:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:49 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:23:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:49 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:23:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:49 smithi162 ceph-mon[20279]: osdmap e654: 12 total, 12 up, 12 in 2024-09-02T10:23:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:49 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:23:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:49 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:23:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:49 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:23:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:49 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:23:50.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:50.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:50.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:50.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:50.250 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:50.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:50.308 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:23:50.309 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:23:50.309 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:23:50.309 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:23:50.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:50.376 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:50.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:50.377 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:50.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:50.504 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:50.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:50.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:50.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:50.632 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:50.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:50.760 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:50.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:50.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:50.887 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:50.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:50 smithi151 ceph-mon[16639]: pgmap v1519: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1305/9 objects degraded (14500.000%); 0 B/s, 183 objects/s recovering 2024-09-02T10:23:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:50 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:23:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:50 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:23:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:50 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1305/9 objects degraded (14500.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3241086810' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:23:50.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:50 smithi187 ceph-mon[20072]: pgmap v1519: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1305/9 objects degraded (14500.000%); 0 B/s, 183 objects/s recovering 2024-09-02T10:23:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:50 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:23:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:50 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:23:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:50 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1305/9 objects degraded (14500.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3241086810' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:23:51.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:51.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:51.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:23:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:50 smithi162 ceph-mon[20279]: pgmap v1519: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1305/9 objects degraded (14500.000%); 0 B/s, 183 objects/s recovering 2024-09-02T10:23:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:50 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:23:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:50 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:23:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:50 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1305/9 objects degraded (14500.000%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3241086810' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:23:51.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:51.142 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:51.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:51.169 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:51.174 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:23:51.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:23:51.243 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:51.269 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:51.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:51.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:51.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:51.396 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:51.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:51.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:51.525 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:51.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:51.627 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:51.654 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:51.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:51.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:51.783 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:51.784 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:51.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:51 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:23:51.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3241086810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:23:51.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:51 smithi151 ceph-mon[16639]: osdmap e655: 12 total, 12 up, 12 in 2024-09-02T10:23:51.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:51.912 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:51.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:51 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:23:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3241086810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:23:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:51 smithi187 ceph-mon[20072]: osdmap e655: 12 total, 12 up, 12 in 2024-09-02T10:23:52.014 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:52.037 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:52.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:51 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:23:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3241086810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:23:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:51 smithi162 ceph-mon[20279]: osdmap e655: 12 total, 12 up, 12 in 2024-09-02T10:23:52.138 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:52.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:52.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:52.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:52.290 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:52.291 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:52.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:52.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:52.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:52.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:52.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:52.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:52.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:52.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:52.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:52.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:52.673 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:52.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:52.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:52.799 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:52.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:52.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:52.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:52 smithi187 ceph-mon[20072]: pgmap v1521: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1305/9 objects degraded (14500.000%); 0 B/s, 124 objects/s recovering 2024-09-02T10:23:52.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:52 smithi187 ceph-mon[20072]: 6.4 scrub starts 2024-09-02T10:23:52.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:52 smithi187 ceph-mon[20072]: 6.4 scrub ok 2024-09-02T10:23:52.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:52.928 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:52.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:52 smithi151 ceph-mon[16639]: pgmap v1521: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1305/9 objects degraded (14500.000%); 0 B/s, 124 objects/s recovering 2024-09-02T10:23:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:52 smithi151 ceph-mon[16639]: 6.4 scrub starts 2024-09-02T10:23:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:52 smithi151 ceph-mon[16639]: 6.4 scrub ok 2024-09-02T10:23:52.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:52 smithi162 ceph-mon[20279]: pgmap v1521: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1305/9 objects degraded (14500.000%); 0 B/s, 124 objects/s recovering 2024-09-02T10:23:52.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:52 smithi162 ceph-mon[20279]: 6.4 scrub starts 2024-09-02T10:23:52.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:52 smithi162 ceph-mon[20279]: 6.4 scrub ok 2024-09-02T10:23:53.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:53.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:53.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:53.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:53.158 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:53.179 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:53.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:53.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:53.308 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:53.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:53.409 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:53.434 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:53.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:53.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:53.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:53.559 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:53.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:53.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:53.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:53.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:53 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:23:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:53 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:23:53.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:53.809 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:53.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:53.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:53.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:53.936 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:53.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:53 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:23:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:53 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:23:54.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:54.064 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:54.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:54.073 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:23:54.074 INFO:teuthology.orchestra.run.smithi151.stdout:{"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 403/9 objects degraded (4477.778%), 1 pg degraded, 1 pg undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":1705,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":655,"num_osds":12,"num_up_osds":12,"osd_up_since":1725271225,"num_in_osds":12,"osd_in_since":1725272451,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":46},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":47,"num_pools":2,"num_objects":3,"data_bytes":590392,"bytes_used":6012690432,"bytes_avail":1145927737344,"bytes_total":1151940427776,"degraded_objects":403,"degraded_total":9,"degraded_ratio":44.777777777777786,"recovering_objects_per_sec":274,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1649,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2024-09-02T09:53:53:306254+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.151:8443/"}},"servicemap":{"epoch":37,"modified":"2024-09-02T10:21:41.376228+0000","services":{}},"progress_events":{}} 2024-09-02T10:23:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:53 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:23:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:53 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:23:54.101 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Removed 100269 objects 2024-09-02T10:23:54.101 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Clean up completed and total clean up time :222.141 2024-09-02T10:23:54.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:54.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:54.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:54.265 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:54.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:54.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:54.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:54.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:54.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:54.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:54.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:54.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:54.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:54.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:54.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:54.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:23:54.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:54 smithi151 ceph-mon[16639]: pgmap v1522: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403/9 objects degraded (4477.778%); 0 B/s, 274 objects/s recovering 2024-09-02T10:23:54.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3746497187' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:23:54.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:54 smithi151 ceph-mon[16639]: osdmap e656: 12 total, 12 up, 12 in 2024-09-02T10:23:54.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:54 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:23:54.624 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:54 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:23:54.663 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:54.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:54.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:54.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:54.809 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:54.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:54.816 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-09-02T10:23:54.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:23:54.911 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:23:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:54.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:54.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:23:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:54 smithi187 ceph-mon[20072]: pgmap v1522: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403/9 objects degraded (4477.778%); 0 B/s, 274 objects/s recovering 2024-09-02T10:23:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3746497187' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:23:55.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:54 smithi187 ceph-mon[20072]: osdmap e656: 12 total, 12 up, 12 in 2024-09-02T10:23:55.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:54 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:23:55.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:54 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:23:55.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:55.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:55.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:55.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:54 smithi162 ceph-mon[20279]: pgmap v1522: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403/9 objects degraded (4477.778%); 0 B/s, 274 objects/s recovering 2024-09-02T10:23:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3746497187' entity='client.admin' cmd={"prefix": "status", "format": "json"} : dispatch 2024-09-02T10:23:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:54 smithi162 ceph-mon[20279]: osdmap e656: 12 total, 12 up, 12 in 2024-09-02T10:23:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:54 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:23:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:54 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:23:55.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:55.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:55.187 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:55.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:55.288 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:55.314 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:55.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:55.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:55.382 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:55.393 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:23:55.393 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:23:55.416 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:55.419 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:23:55.419 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:23:55.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:55.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:55.440 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:55.447 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:23:55.447 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:23:55.468 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:23:55.468 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:23:55.490 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:23:55.490 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:23:55.511 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:23:55.512 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:23:55.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:23:55.533 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:23:55.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:55.567 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:55.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:55.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:55.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:55 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:23:55.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:55 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:23:55.682 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:55 smithi151 ceph-mon[16639]: 6.f scrub starts 2024-09-02T10:23:55.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:55 smithi151 ceph-mon[16639]: 6.f scrub ok 2024-09-02T10:23:55.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:55 smithi151 ceph-mon[16639]: osdmap e657: 12 total, 12 up, 12 in 2024-09-02T10:23:55.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:55 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3717176888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:55.695 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:55.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:55.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:55.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:55.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:55.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:55.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:55.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:55.947 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:55.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:55.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:55 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:23:56.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:55 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:23:56.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:55 smithi187 ceph-mon[20072]: 6.f scrub starts 2024-09-02T10:23:56.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:55 smithi187 ceph-mon[20072]: 6.f scrub ok 2024-09-02T10:23:56.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:55 smithi187 ceph-mon[20072]: osdmap e657: 12 total, 12 up, 12 in 2024-09-02T10:23:56.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:55 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3717176888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:56.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:56.074 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:56.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:55 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:23:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:55 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:23:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:55 smithi162 ceph-mon[20279]: 6.f scrub starts 2024-09-02T10:23:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:55 smithi162 ceph-mon[20279]: 6.f scrub ok 2024-09-02T10:23:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:55 smithi162 ceph-mon[20279]: osdmap e657: 12 total, 12 up, 12 in 2024-09-02T10:23:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:55 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3717176888' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:23:56.176 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:56.201 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:56.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:56.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:56.328 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:56.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:56.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:23:56.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:23:56.379 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:23:56.379 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:23:56.401 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:23:56.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:23:56.405 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:23:56.405 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:23:56.423 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:23:56.424 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:23:56.429 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:23:56.449 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:56.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:56.456 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:23:56.456 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:23:56.456 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:23:56.464 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:23:56.464 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:23:56.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:56.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:56.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:56.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:56.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:23:56.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:56.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:56.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:23:56.808 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:56.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:56.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:56.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:56 smithi151 ceph-mon[16639]: pgmap v1525: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403/9 objects degraded (4477.778%); 0 B/s, 150 objects/s recovering 2024-09-02T10:23:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:56 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:23:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:56 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:23:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:56 smithi151 ceph-mon[16639]: osdmap e658: 12 total, 12 up, 12 in 2024-09-02T10:23:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:56.956 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:56.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:56 smithi187 ceph-mon[20072]: pgmap v1525: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403/9 objects degraded (4477.778%); 0 B/s, 150 objects/s recovering 2024-09-02T10:23:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:56 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:23:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:56 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:23:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:56 smithi187 ceph-mon[20072]: osdmap e658: 12 total, 12 up, 12 in 2024-09-02T10:23:57.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:57.083 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:57.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:57.084 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:56 smithi162 ceph-mon[20279]: pgmap v1525: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403/9 objects degraded (4477.778%); 0 B/s, 150 objects/s recovering 2024-09-02T10:23:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:56 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:23:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:56 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:23:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:56 smithi162 ceph-mon[20279]: osdmap e658: 12 total, 12 up, 12 in 2024-09-02T10:23:57.125 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1526,"stamp":"2024-09-02T10:23:56.185304+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33237,"num_bytes_recovered":1287075424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163791,"ondisk_log_size":163791,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5871856,"kb_used_data":280380,"kb_used_omap":18,"kb_used_meta":5591213,"kb_avail":1119069968,"statfs":{"total":1151940427776,"available":1145927647232,"internally_reserved":0,"allocated":287109120,"data_stored":281759584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5725402427},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":902,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.011373"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":681,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.289563+0000","last_change":"2024-09-02T10:23:46.278826+0000","last_active":"2024-09-02T10:23:51.289563+0000","last_peered":"2024-09-02T10:23:51.289563+0000","last_clean":"2024-09-02T10:23:51.289563+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:23:51.289563+0000","last_undegraded":"2024-09-02T10:23:51.289563+0000","last_fullsized":"2024-09-02T10:23:51.289563+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:46.278766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:46.278766+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:11.240675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":487,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.361775+0000","last_change":"2024-09-02T10:23:45.449362+0000","last_active":"2024-09-02T10:23:51.361775+0000","last_peered":"2024-09-02T10:23:51.361775+0000","last_clean":"2024-09-02T10:23:51.361775+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:23:51.361775+0000","last_undegraded":"2024-09-02T10:23:51.361775+0000","last_fullsized":"2024-09-02T10:23:51.361775+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:45.449313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:45.449313+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:02.805553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":947,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-09-02T10:23:54.172514+0000","last_change":"2024-09-02T10:23:46.236367+0000","last_active":"2024-09-02T10:23:54.172514+0000","last_peered":"2024-09-02T10:23:54.172514+0000","last_clean":"2024-09-02T10:23:54.172514+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:23:54.172514+0000","last_undegraded":"2024-09-02T10:23:54.172514+0000","last_fullsized":"2024-09-02T10:23:54.172514+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:46.236302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:46.236302+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:47.877815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":939,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-09-02T10:23:54.226114+0000","last_change":"2024-09-02T10:23:54.226114+0000","last_active":"2024-09-02T10:23:54.226114+0000","last_peered":"2024-09-02T10:23:54.226114+0000","last_clean":"2024-09-02T10:23:54.226114+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:23:54.226114+0000","last_undegraded":"2024-09-02T10:23:54.226114+0000","last_fullsized":"2024-09-02T10:23:54.226114+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:54.226069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:54.226069+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:19.565391+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":241,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.691031+0000","last_change":"2024-09-02T10:23:49.799236+0000","last_active":"2024-09-02T10:23:51.691031+0000","last_peered":"2024-09-02T10:23:51.691031+0000","last_clean":"2024-09-02T10:23:51.691031+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:23:51.691031+0000","last_undegraded":"2024-09-02T10:23:51.691031+0000","last_fullsized":"2024-09-02T10:23:51.691031+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:49.799176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:49.799176+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:58.525188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":778,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.421788+0000","last_change":"2024-09-02T10:23:01.185167+0000","last_active":"2024-09-02T10:23:51.421788+0000","last_peered":"2024-09-02T10:23:51.421788+0000","last_clean":"2024-09-02T10:23:51.421788+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:23:51.421788+0000","last_undegraded":"2024-09-02T10:23:51.421788+0000","last_fullsized":"2024-09-02T10:23:51.421788+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.185112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.185112+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.992983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1314,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.513468+0000","last_change":"2024-09-02T10:23:09.691782+0000","last_active":"2024-09-02T10:23:51.513468+0000","last_peered":"2024-09-02T10:23:51.513468+0000","last_clean":"2024-09-02T10:23:51.513468+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:23:51.513468+0000","last_undegraded":"2024-09-02T10:23:51.513468+0000","last_fullsized":"2024-09-02T10:23:51.513468+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:09.691691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:09.691691+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.662853+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":955,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-09-02T10:23:54.172597+0000","last_change":"2024-09-02T10:23:05.131265+0000","last_active":"2024-09-02T10:23:54.172597+0000","last_peered":"2024-09-02T10:23:54.172597+0000","last_clean":"2024-09-02T10:23:54.172597+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:23:54.172597+0000","last_undegraded":"2024-09-02T10:23:54.172597+0000","last_fullsized":"2024-09-02T10:23:54.172597+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.131213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.131213+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.083779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":944,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.361793+0000","last_change":"2024-09-02T10:23:06.476818+0000","last_active":"2024-09-02T10:23:51.361793+0000","last_peered":"2024-09-02T10:23:51.361793+0000","last_clean":"2024-09-02T10:23:51.361793+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:23:51.361793+0000","last_undegraded":"2024-09-02T10:23:51.361793+0000","last_fullsized":"2024-09-02T10:23:51.361793+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:06.476768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.476768+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:35.403622+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":929,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.175215+0000","last_change":"2024-09-02T10:23:48.667403+0000","last_active":"2024-09-02T10:23:51.175215+0000","last_peered":"2024-09-02T10:23:51.175215+0000","last_clean":"2024-09-02T10:23:51.175215+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:23:51.175215+0000","last_undegraded":"2024-09-02T10:23:51.175215+0000","last_fullsized":"2024-09-02T10:23:51.175215+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:48.667341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:48.667341+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:09.201391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":165,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.181530+0000","last_change":"2024-09-02T10:21:03.639078+0000","last_active":"2024-09-02T10:23:51.181530+0000","last_peered":"2024-09-02T10:23:51.181530+0000","last_clean":"2024-09-02T10:23:51.181530+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:23:51.181530+0000","last_undegraded":"2024-09-02T10:23:51.181530+0000","last_fullsized":"2024-09-02T10:23:51.181530+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:21:03.639023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:21:03.639023+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1278,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:52.504541+0000","last_change":"2024-09-02T10:23:52.504541+0000","last_active":"2024-09-02T10:23:52.504541+0000","last_peered":"2024-09-02T10:23:52.504541+0000","last_clean":"2024-09-02T10:23:52.504541+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:23:52.504541+0000","last_undegraded":"2024-09-02T10:23:52.504541+0000","last_fullsized":"2024-09-02T10:23:52.504541+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:52.504495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:52.504495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:19.309680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1213,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.513504+0000","last_change":"2024-09-02T10:22:55.516683+0000","last_active":"2024-09-02T10:23:51.513504+0000","last_peered":"2024-09-02T10:23:51.513504+0000","last_clean":"2024-09-02T10:23:51.513504+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:23:51.513504+0000","last_undegraded":"2024-09-02T10:23:51.513504+0000","last_fullsized":"2024-09-02T10:23:51.513504+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.516633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.516633+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:23:58.850702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1184,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.178443+0000","last_change":"2024-09-02T10:19:47.496747+0000","last_active":"2024-09-02T10:23:51.178443+0000","last_peered":"2024-09-02T10:23:51.178443+0000","last_clean":"2024-09-02T10:23:51.178443+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:23:51.178443+0000","last_undegraded":"2024-09-02T10:23:51.178443+0000","last_fullsized":"2024-09-02T10:23:51.178443+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:19:47.496648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:19:47.496648+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1183,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.513512+0000","last_change":"2024-09-02T10:23:00.621966+0000","last_active":"2024-09-02T10:23:51.513512+0000","last_peered":"2024-09-02T10:23:51.513512+0000","last_clean":"2024-09-02T10:23:51.513512+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:23:51.513512+0000","last_undegraded":"2024-09-02T10:23:51.513512+0000","last_fullsized":"2024-09-02T10:23:51.513512+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.621916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.621916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:21.445915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"626'12394","reported_seq":20756,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.181783+0000","last_change":"2024-09-02T10:22:56.637995+0000","last_active":"2024-09-02T10:23:51.181783+0000","last_peered":"2024-09-02T10:23:51.181783+0000","last_clean":"2024-09-02T10:23:51.181783+0000","last_became_active":"2024-09-02T10:22:56.595915+0000","last_became_peered":"2024-09-02T10:22:56.595915+0000","last_unstale":"2024-09-02T10:23:51.181783+0000","last_undegraded":"2024-09-02T10:23:51.181783+0000","last_fullsized":"2024-09-02T10:23:51.181783+0000","mapping_epoch":635,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":636,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5500","last_scrub_stamp":"2024-09-02T10:19:59.986579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:59.986579+0000","objects_scrubbed":5468,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12394,"num_write_kb":396608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6197,"num_bytes_recovered":237961216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1277,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.513594+0000","last_change":"2024-09-02T10:23:01.892373+0000","last_active":"2024-09-02T10:23:51.513594+0000","last_peered":"2024-09-02T10:23:51.513594+0000","last_clean":"2024-09-02T10:23:51.513594+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:23:51.513594+0000","last_undegraded":"2024-09-02T10:23:51.513594+0000","last_fullsized":"2024-09-02T10:23:51.513594+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.892288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.892288+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:30.607091+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"626'12580","reported_seq":29728,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.691110+0000","last_change":"2024-09-02T10:23:09.693242+0000","last_active":"2024-09-02T10:23:51.691110+0000","last_peered":"2024-09-02T10:23:51.691110+0000","last_clean":"2024-09-02T10:23:51.691110+0000","last_became_active":"2024-09-02T10:23:02.182530+0000","last_became_peered":"2024-09-02T10:23:02.182530+0000","last_unstale":"2024-09-02T10:23:51.691110+0000","last_undegraded":"2024-09-02T10:23:51.691110+0000","last_fullsized":"2024-09-02T10:23:51.691110+0000","mapping_epoch":640,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":641,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12580","last_scrub_stamp":"2024-09-02T10:23:09.693181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:09.693181+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:24.508961+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12580,"num_write_kb":402560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8401,"num_bytes_recovered":374538240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1400,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.174966+0000","last_change":"2024-09-02T10:22:37.332903+0000","last_active":"2024-09-02T10:23:51.174966+0000","last_peered":"2024-09-02T10:23:51.174966+0000","last_clean":"2024-09-02T10:23:51.174966+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:23:51.174966+0000","last_undegraded":"2024-09-02T10:23:51.174966+0000","last_fullsized":"2024-09-02T10:23:51.174966+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:37.332859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:37.332859+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.675027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"626'12351","reported_seq":12676,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.513590+0000","last_change":"2024-09-02T10:23:04.675691+0000","last_active":"2024-09-02T10:23:51.513590+0000","last_peered":"2024-09-02T10:23:51.513590+0000","last_clean":"2024-09-02T10:23:51.513590+0000","last_became_active":"2024-09-02T10:18:37.062133+0000","last_became_peered":"2024-09-02T10:18:37.062133+0000","last_unstale":"2024-09-02T10:23:51.513590+0000","last_undegraded":"2024-09-02T10:23:51.513590+0000","last_fullsized":"2024-09-02T10:23:51.513590+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12351","last_scrub_stamp":"2024-09-02T10:23:04.675613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:04.675613+0000","objects_scrubbed":1,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.698140+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12351,"num_write_kb":395201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,10],"acting":[5,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1042,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.940621+0000","last_change":"2024-09-02T10:22:55.068228+0000","last_active":"2024-09-02T10:23:51.940621+0000","last_peered":"2024-09-02T10:23:51.940621+0000","last_clean":"2024-09-02T10:23:51.940621+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:23:51.940621+0000","last_undegraded":"2024-09-02T10:23:51.940621+0000","last_fullsized":"2024-09-02T10:23:51.940621+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.068166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.068166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:04.696428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"6.c","version":"626'12382","reported_seq":12714,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.511684+0000","last_change":"2024-09-02T10:20:20.717885+0000","last_active":"2024-09-02T10:23:51.511684+0000","last_peered":"2024-09-02T10:23:51.511684+0000","last_clean":"2024-09-02T10:23:51.511684+0000","last_became_active":"2024-09-02T10:18:37.848152+0000","last_became_peered":"2024-09-02T10:18:37.848152+0000","last_unstale":"2024-09-02T10:23:51.511684+0000","last_undegraded":"2024-09-02T10:23:51.511684+0000","last_fullsized":"2024-09-02T10:23:51.511684+0000","mapping_epoch":533,"log_start":"551'2300","ondisk_log_start":"551'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6728","last_scrub_stamp":"2024-09-02T10:20:20.717733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:20.717733+0000","objects_scrubbed":5690,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1810,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12382,"num_write_kb":396224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":447,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.289339+0000","last_change":"2024-09-02T10:23:45.263493+0000","last_active":"2024-09-02T10:23:51.289339+0000","last_peered":"2024-09-02T10:23:51.289339+0000","last_clean":"2024-09-02T10:23:51.289339+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:23:51.289339+0000","last_undegraded":"2024-09-02T10:23:51.289339+0000","last_fullsized":"2024-09-02T10:23:51.289339+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:45.263415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:45.263415+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:04.210025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.3","version":"626'12606","reported_seq":12963,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.175052+0000","last_change":"2024-09-02T10:23:11.425512+0000","last_active":"2024-09-02T10:23:51.175052+0000","last_peered":"2024-09-02T10:23:51.175052+0000","last_clean":"2024-09-02T10:23:51.175052+0000","last_became_active":"2024-09-02T10:18:37.850585+0000","last_became_peered":"2024-09-02T10:18:37.850585+0000","last_unstale":"2024-09-02T10:23:51.175052+0000","last_undegraded":"2024-09-02T10:23:51.175052+0000","last_fullsized":"2024-09-02T10:23:51.175052+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12606","last_scrub_stamp":"2024-09-02T10:23:11.425432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:11.425432+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:12.626473+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,11],"acting":[7,6,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1348,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.289477+0000","last_change":"2024-09-02T10:22:34.647196+0000","last_active":"2024-09-02T10:23:51.289477+0000","last_peered":"2024-09-02T10:23:51.289477+0000","last_clean":"2024-09-02T10:23:51.289477+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:23:51.289477+0000","last_undegraded":"2024-09-02T10:23:51.289477+0000","last_fullsized":"2024-09-02T10:23:51.289477+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:34.647144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:34.647144+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:02.024298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"6.2","version":"626'12544","reported_seq":29705,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.691037+0000","last_change":"2024-09-02T10:23:07.704034+0000","last_active":"2024-09-02T10:23:51.691037+0000","last_peered":"2024-09-02T10:23:51.691037+0000","last_clean":"2024-09-02T10:23:51.691037+0000","last_became_active":"2024-09-02T10:22:57.600280+0000","last_became_peered":"2024-09-02T10:22:57.600280+0000","last_unstale":"2024-09-02T10:23:51.691037+0000","last_undegraded":"2024-09-02T10:23:51.691037+0000","last_fullsized":"2024-09-02T10:23:51.691037+0000","mapping_epoch":636,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":637,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12544","last_scrub_stamp":"2024-09-02T10:23:07.703956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:07.703956+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:08.077242+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12544,"num_write_kb":401408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8407,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":814,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.421799+0000","last_change":"2024-09-02T10:23:05.243285+0000","last_active":"2024-09-02T10:23:51.421799+0000","last_peered":"2024-09-02T10:23:51.421799+0000","last_clean":"2024-09-02T10:23:51.421799+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:23:51.421799+0000","last_undegraded":"2024-09-02T10:23:51.421799+0000","last_fullsized":"2024-09-02T10:23:51.421799+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.243236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.243236+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:16.091958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.1","version":"626'12380","reported_seq":12741,"reported_epoch":656,"state":"active+clean","last_fresh":"2024-09-02T10:23:54.172452+0000","last_change":"2024-09-02T10:23:07.151869+0000","last_active":"2024-09-02T10:23:54.172452+0000","last_peered":"2024-09-02T10:23:54.172452+0000","last_clean":"2024-09-02T10:23:54.172452+0000","last_became_active":"2024-09-02T10:18:37.600290+0000","last_became_peered":"2024-09-02T10:18:37.600290+0000","last_unstale":"2024-09-02T10:23:54.172452+0000","last_undegraded":"2024-09-02T10:23:54.172452+0000","last_fullsized":"2024-09-02T10:23:54.172452+0000","mapping_epoch":533,"log_start":"552'2300","ondisk_log_start":"552'2300","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12380","last_scrub_stamp":"2024-09-02T10:23:07.151772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:07.151772+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:33.568723+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12380,"num_write_kb":396160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,10],"acting":[8,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":779,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.421811+0000","last_change":"2024-09-02T10:22:55.086083+0000","last_active":"2024-09-02T10:23:51.421811+0000","last_peered":"2024-09-02T10:23:51.421811+0000","last_clean":"2024-09-02T10:23:51.421811+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:23:51.421811+0000","last_undegraded":"2024-09-02T10:23:51.421811+0000","last_fullsized":"2024-09-02T10:23:51.421811+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.086036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.086036+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.714243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.0","version":"626'12678","reported_seq":24868,"reported_epoch":655,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:23:52.555344+0000","last_change":"2024-09-02T10:22:55.007293+0000","last_active":"2024-09-02T10:23:52.555344+0000","last_peered":"2024-09-02T10:23:52.555344+0000","last_clean":"2024-09-02T10:20:51.081664+0000","last_became_active":"2024-09-02T10:20:53.407909+0000","last_became_peered":"2024-09-02T10:20:53.407909+0000","last_unstale":"2024-09-02T10:23:52.555344+0000","last_undegraded":"2024-09-02T10:20:53.134972+0000","last_fullsized":"2024-09-02T10:20:53.101903+0000","mapping_epoch":586,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'5088","last_scrub_stamp":"2024-09-02T10:19:52.017075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:52.017075+0000","objects_scrubbed":5020,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1424,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12678,"num_write_kb":405696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5936,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":-403}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":547,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.504281+0000","last_change":"2024-09-02T10:23:42.635086+0000","last_active":"2024-09-02T10:23:51.504281+0000","last_peered":"2024-09-02T10:23:51.504281+0000","last_clean":"2024-09-02T10:23:51.504281+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:23:51.504281+0000","last_undegraded":"2024-09-02T10:23:51.504281+0000","last_fullsized":"2024-09-02T10:23:51.504281+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:42.635029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:42.635029+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:11.935109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"626'12542","reported_seq":17217,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.511513+0000","last_change":"2024-09-02T10:19:58.464592+0000","last_active":"2024-09-02T10:23:51.511513+0000","last_peered":"2024-09-02T10:23:51.511513+0000","last_clean":"2024-09-02T10:23:51.511513+0000","last_became_active":"2024-09-02T10:19:33.590380+0000","last_became_peered":"2024-09-02T10:19:33.590380+0000","last_unstale":"2024-09-02T10:23:51.511513+0000","last_undegraded":"2024-09-02T10:23:51.511513+0000","last_fullsized":"2024-09-02T10:23:51.511513+0000","mapping_epoch":559,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"567'5456","last_scrub_stamp":"2024-09-02T10:19:58.464466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:19:58.464466+0000","objects_scrubbed":5400,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12542,"num_write_kb":401344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2174,"num_bytes_recovered":142475264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":780,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.421794+0000","last_change":"2024-09-02T10:23:00.146326+0000","last_active":"2024-09-02T10:23:51.421794+0000","last_peered":"2024-09-02T10:23:51.421794+0000","last_clean":"2024-09-02T10:23:51.421794+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:23:51.421794+0000","last_undegraded":"2024-09-02T10:23:51.421794+0000","last_fullsized":"2024-09-02T10:23:51.421794+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.146278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.146278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:09.373850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"6.5","version":"626'12632","reported_seq":12993,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.175196+0000","last_change":"2024-09-02T10:22:59.338566+0000","last_active":"2024-09-02T10:23:51.175196+0000","last_peered":"2024-09-02T10:23:51.175196+0000","last_clean":"2024-09-02T10:23:51.175196+0000","last_became_active":"2024-09-02T10:18:37.847983+0000","last_became_peered":"2024-09-02T10:18:37.847983+0000","last_unstale":"2024-09-02T10:23:51.175196+0000","last_undegraded":"2024-09-02T10:23:51.175196+0000","last_fullsized":"2024-09-02T10:23:51.175196+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12632","last_scrub_stamp":"2024-09-02T10:22:59.338517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:22:59.338517+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.748681+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1042,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.175070+0000","last_change":"2024-09-02T10:23:44.545777+0000","last_active":"2024-09-02T10:23:51.175070+0000","last_peered":"2024-09-02T10:23:51.175070+0000","last_clean":"2024-09-02T10:23:51.175070+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:23:51.175070+0000","last_undegraded":"2024-09-02T10:23:51.175070+0000","last_fullsized":"2024-09-02T10:23:51.175070+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:44.545693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:44.545693+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:53.740167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"626'12586","reported_seq":12910,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.522412+0000","last_change":"2024-09-02T10:23:51.522412+0000","last_active":"2024-09-02T10:23:51.522412+0000","last_peered":"2024-09-02T10:23:51.522412+0000","last_clean":"2024-09-02T10:23:51.522412+0000","last_became_active":"2024-09-02T10:18:37.064656+0000","last_became_peered":"2024-09-02T10:18:37.064656+0000","last_unstale":"2024-09-02T10:23:51.522412+0000","last_undegraded":"2024-09-02T10:23:51.522412+0000","last_fullsized":"2024-09-02T10:23:51.522412+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12586","last_scrub_stamp":"2024-09-02T10:23:51.522321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:51.522321+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:09.538071+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12586,"num_write_kb":402752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1301,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.361583+0000","last_change":"2024-09-02T10:23:08.500316+0000","last_active":"2024-09-02T10:23:51.361583+0000","last_peered":"2024-09-02T10:23:51.361583+0000","last_clean":"2024-09-02T10:23:51.361583+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:23:51.361583+0000","last_undegraded":"2024-09-02T10:23:51.361583+0000","last_fullsized":"2024-09-02T10:23:51.361583+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:08.500268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:08.500268+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:14.723023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"626'12632","reported_seq":17205,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.513679+0000","last_change":"2024-09-02T10:23:02.673893+0000","last_active":"2024-09-02T10:23:51.513679+0000","last_peered":"2024-09-02T10:23:51.513679+0000","last_clean":"2024-09-02T10:23:51.513679+0000","last_became_active":"2024-09-02T10:19:53.490510+0000","last_became_peered":"2024-09-02T10:19:53.490510+0000","last_unstale":"2024-09-02T10:23:51.513679+0000","last_undegraded":"2024-09-02T10:23:51.513679+0000","last_fullsized":"2024-09-02T10:23:51.513679+0000","mapping_epoch":565,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":566,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12632","last_scrub_stamp":"2024-09-02T10:23:02.673828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:02.673828+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:31.609539+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12632,"num_write_kb":404224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2118,"num_bytes_recovered":138805248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1053,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.175001+0000","last_change":"2024-09-02T10:23:45.553424+0000","last_active":"2024-09-02T10:23:51.175001+0000","last_peered":"2024-09-02T10:23:51.175001+0000","last_clean":"2024-09-02T10:23:51.175001+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:23:51.175001+0000","last_undegraded":"2024-09-02T10:23:51.175001+0000","last_fullsized":"2024-09-02T10:23:51.175001+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:45.553364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:45.553364+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:06.163823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"626'12650","reported_seq":13015,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.174939+0000","last_change":"2024-09-02T10:23:46.589005+0000","last_active":"2024-09-02T10:23:51.174939+0000","last_peered":"2024-09-02T10:23:51.174939+0000","last_clean":"2024-09-02T10:23:51.174939+0000","last_became_active":"2024-09-02T10:18:37.850614+0000","last_became_peered":"2024-09-02T10:18:37.850614+0000","last_unstale":"2024-09-02T10:23:51.174939+0000","last_undegraded":"2024-09-02T10:23:51.174939+0000","last_fullsized":"2024-09-02T10:23:51.174939+0000","mapping_epoch":533,"log_start":"553'2600","ondisk_log_start":"553'2600","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12650","last_scrub_stamp":"2024-09-02T10:23:46.588940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:46.588940+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:51.714504+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12650,"num_write_kb":404800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,3],"acting":[7,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1308,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.504286+0000","last_change":"2024-09-02T10:23:00.386934+0000","last_active":"2024-09-02T10:23:51.504286+0000","last_peered":"2024-09-02T10:23:51.504286+0000","last_clean":"2024-09-02T10:23:51.504286+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:23:51.504286+0000","last_undegraded":"2024-09-02T10:23:51.504286+0000","last_fullsized":"2024-09-02T10:23:51.504286+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.386875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.386875+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.366845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"626'12496","reported_seq":12817,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.178266+0000","last_change":"2024-09-02T10:20:00.442267+0000","last_active":"2024-09-02T10:23:51.178266+0000","last_peered":"2024-09-02T10:23:51.178266+0000","last_clean":"2024-09-02T10:23:51.178266+0000","last_became_active":"2024-09-02T10:18:37.600448+0000","last_became_peered":"2024-09-02T10:18:37.600448+0000","last_unstale":"2024-09-02T10:23:51.178266+0000","last_undegraded":"2024-09-02T10:23:51.178266+0000","last_fullsized":"2024-09-02T10:23:51.178266+0000","mapping_epoch":533,"log_start":"553'2400","ondisk_log_start":"553'2400","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"568'5543","last_scrub_stamp":"2024-09-02T10:20:00.442143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:00.442143+0000","objects_scrubbed":5491,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12496,"num_write_kb":399872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,8],"acting":[0,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":257,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.691031+0000","last_change":"2024-09-02T10:23:48.789737+0000","last_active":"2024-09-02T10:23:51.691031+0000","last_peered":"2024-09-02T10:23:51.691031+0000","last_clean":"2024-09-02T10:23:51.691031+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:23:51.691031+0000","last_undegraded":"2024-09-02T10:23:51.691031+0000","last_fullsized":"2024-09-02T10:23:51.691031+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:48.789677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:48.789677+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:14.919249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"626'12520","reported_seq":12852,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.513463+0000","last_change":"2024-09-02T10:23:06.653266+0000","last_active":"2024-09-02T10:23:51.513463+0000","last_peered":"2024-09-02T10:23:51.513463+0000","last_clean":"2024-09-02T10:23:51.513463+0000","last_became_active":"2024-09-02T10:18:37.848151+0000","last_became_peered":"2024-09-02T10:18:37.848151+0000","last_unstale":"2024-09-02T10:23:51.513463+0000","last_undegraded":"2024-09-02T10:23:51.513463+0000","last_fullsized":"2024-09-02T10:23:51.513463+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'12520","last_scrub_stamp":"2024-09-02T10:23:06.653191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.653191+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:17.096908+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12520,"num_write_kb":400640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1080,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.175264+0000","last_change":"2024-09-02T10:22:30.358040+0000","last_active":"2024-09-02T10:23:51.175264+0000","last_peered":"2024-09-02T10:23:51.175264+0000","last_clean":"2024-09-02T10:23:51.175264+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:23:51.175264+0000","last_undegraded":"2024-09-02T10:23:51.175264+0000","last_fullsized":"2024-09-02T10:23:51.175264+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:30.357970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:30.357970+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"626'12566","reported_seq":12886,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.511459+0000","last_change":"2024-09-02T10:20:21.113390+0000","last_active":"2024-09-02T10:23:51.511459+0000","last_peered":"2024-09-02T10:23:51.511459+0000","last_clean":"2024-09-02T10:23:51.511459+0000","last_became_active":"2024-09-02T10:18:37.065723+0000","last_became_peered":"2024-09-02T10:18:37.065723+0000","last_unstale":"2024-09-02T10:23:51.511459+0000","last_undegraded":"2024-09-02T10:23:51.511459+0000","last_fullsized":"2024-09-02T10:23:51.511459+0000","mapping_epoch":533,"log_start":"553'2500","ondisk_log_start":"553'2500","created":533,"last_epoch_clean":534,"parent":"0.0","parent_split_bits":0,"last_scrub":"576'6832","last_scrub_stamp":"2024-09-02T10:20:21.113264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:18:36.041320+0000","last_clean_scrub_stamp":"2024-09-02T10:20:21.113264+0000","objects_scrubbed":5760,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12566,"num_write_kb":402112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":462,"reported_epoch":655,"state":"active+clean","last_fresh":"2024-09-02T10:23:51.511718+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:23:51.511718+0000","last_peered":"2024-09-02T10:23:51.511718+0000","last_clean":"2024-09-02T10:23:51.511718+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:23:51.511718+0000","last_undegraded":"2024-09-02T10:23:51.511718+0000","last_fullsized":"2024-09-02T10:23:51.511718+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":16,"stat_sum":{"num_bytes":24,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":200539,"num_write_kb":6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33233,"num_bytes_recovered":1286144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270938112,"data_stored":270925896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160939,"ondisk_log_size":160939,"up":48,"acting":47,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383965,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487728,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93257424,"statfs":{"total":95995035648,"available":95495602176,"internally_reserved":0,"allocated":1204224,"data_stored":764022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":498203076},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612898,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357244,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93387908,"statfs":{"total":95995035648,"available":95629217792,"internally_reserved":0,"allocated":1220608,"data_stored":771910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841831,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546040,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199112,"statfs":{"total":95995035648,"available":95435890688,"internally_reserved":0,"allocated":1216512,"data_stored":771886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070765,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339204,"kb_used_data":1200,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93405948,"statfs":{"total":95995035648,"available":95647690752,"internally_reserved":0,"allocated":1228800,"data_stored":779750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299698,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582328,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162824,"statfs":{"total":95995035648,"available":95398731776,"internally_reserved":0,"allocated":1409024,"data_stored":965270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528629,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365868,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379284,"statfs":{"total":95995035648,"available":95620386816,"internally_reserved":0,"allocated":1531904,"data_stored":1091702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724858,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":878852,"kb_used_data":265968,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92866300,"statfs":{"total":95995035648,"available":95095091200,"internally_reserved":0,"allocated":272351232,"data_stored":271898982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953792,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":335032,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410120,"statfs":{"total":95995035648,"available":95651962880,"internally_reserved":0,"allocated":1216512,"data_stored":771886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150021,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":569976,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175176,"statfs":{"total":95995035648,"available":95411380224,"internally_reserved":0,"allocated":1544192,"data_stored":1099566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411656,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307456,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":93437696,"statfs":{"total":95995035648,"available":95680200704,"internally_reserved":0,"allocated":1421312,"data_stored":973134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575182,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509944,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235208,"statfs":{"total":95995035648,"available":95472852992,"internally_reserved":0,"allocated":1220608,"data_stored":771910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738708,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592184,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93152968,"statfs":{"total":95995035648,"available":95388639232,"internally_reserved":0,"allocated":1544192,"data_stored":1099566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":270929920,"data_stored":270925848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:23:57.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:57.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:57.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:57.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:23:57.335 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:57.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:23:57.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:23:57.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:57.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:23:57.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:23:57.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:57.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:57.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:23:57.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:57.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:57.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:57.717 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:57.751 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:57 smithi151 ceph-mon[16639]: from='client.17796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:57.751 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:57 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:23:57.751 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:57 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:23:57.751 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:57 smithi151 ceph-mon[16639]: 6.0 scrub starts 2024-09-02T10:23:57.751 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:57 smithi151 ceph-mon[16639]: 6.0 scrub ok 2024-09-02T10:23:57.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:23:57.845 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:57.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:57.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:23:57.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:57.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:57.974 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:57.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:57 smithi187 ceph-mon[20072]: from='client.17796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:57 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:23:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:57 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:23:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:57 smithi187 ceph-mon[20072]: 6.0 scrub starts 2024-09-02T10:23:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:57 smithi187 ceph-mon[20072]: 6.0 scrub ok 2024-09-02T10:23:58.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:58.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:57 smithi162 ceph-mon[20279]: from='client.17796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:23:58.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:57 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:23:58.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:57 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:23:58.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:57 smithi162 ceph-mon[20279]: 6.0 scrub starts 2024-09-02T10:23:58.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:57 smithi162 ceph-mon[20279]: 6.0 scrub ok 2024-09-02T10:23:58.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:58.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:58.102 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:58.184 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-02T10:23:58.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-02T10:23:58.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:58.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:58.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:58.229 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:58.330 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:58.356 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:58.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:58.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:58.484 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:58.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:58.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:23:58.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:58.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:58.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:58.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:58.747 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:58.773 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:58.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:58.774 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:58.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:58 smithi151 ceph-mon[16639]: pgmap v1527: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403/9 objects degraded (4477.778%); 0 B/s, 150 objects/s recovering 2024-09-02T10:23:58.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:58 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:23:58.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:58 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:23:58.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:23:58.896 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:23:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:58.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:23:58.998 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:58 smithi187 ceph-mon[20072]: pgmap v1527: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403/9 objects degraded (4477.778%); 0 B/s, 150 objects/s recovering 2024-09-02T10:23:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:58 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:23:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:58 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:23:59.023 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:59.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:59.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:59.064 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:58 smithi162 ceph-mon[20279]: pgmap v1527: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 403/9 objects degraded (4477.778%); 0 B/s, 150 objects/s recovering 2024-09-02T10:23:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:58 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:23:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:58 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:23:59.097 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:23:59.126 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:23:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:59.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:59.152 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:23:59.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:59.279 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:59.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:59.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:59.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:59.407 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:59.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:23:59.534 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:59.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:23:59.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:23:59.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:23:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:59.662 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:23:59.724 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:59 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:23:59.724 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:59 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:23:59.725 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:59 smithi151 ceph-mon[16639]: 6.6 scrub starts 2024-09-02T10:23:59.725 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:59 smithi151 ceph-mon[16639]: 6.6 scrub ok 2024-09-02T10:23:59.725 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 403/9 objects degraded (4477.778%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:23:59.725 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:23:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1196769481' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:23:59.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:59.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:23:59.788 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:59.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:59.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:23:59.916 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:23:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:23:59.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:23:59.935 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:23:59.935 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:59 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:24:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:59 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:24:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:59 smithi187 ceph-mon[20072]: 6.6 scrub starts 2024-09-02T10:24:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:59 smithi187 ceph-mon[20072]: 6.6 scrub ok 2024-09-02T10:24:00.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 403/9 objects degraded (4477.778%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:24:00.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:23:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1196769481' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:24:00.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:00.043 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:00.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:59 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:24:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:59 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:24:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:59 smithi162 ceph-mon[20279]: 6.6 scrub starts 2024-09-02T10:24:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:59 smithi162 ceph-mon[20279]: 6.6 scrub ok 2024-09-02T10:24:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 403/9 objects degraded (4477.778%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:24:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:23:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1196769481' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:24:00.128 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:24:00.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:00.171 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:00.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:00.182 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:24:00.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:00.295 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:00.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:00.397 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:00.420 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:00.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:00.521 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:00.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:00.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:00.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:00.672 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:00.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:00.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:00.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:24:00.774 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:00.795 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:00.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:00.856 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_4' removed 2024-09-02T10:24:00.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:00.918 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:00.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:00 smithi151 ceph-mon[16639]: pgmap v1528: 47 pgs: 47 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 0 B/s, 67 objects/s recovering 2024-09-02T10:24:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3455138357' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:24:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:00 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 403/9 objects degraded (4477.778%), 1 pg degraded, 1 pg undersized) 2024-09-02T10:24:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:00 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:24:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:00 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1196769481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:24:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3455138357' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:24:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:00 smithi151 ceph-mon[16639]: osdmap e659: 12 total, 12 up, 12 in 2024-09-02T10:24:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:00 smithi187 ceph-mon[20072]: pgmap v1528: 47 pgs: 47 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 0 B/s, 67 objects/s recovering 2024-09-02T10:24:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3455138357' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:24:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:00 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 403/9 objects degraded (4477.778%), 1 pg degraded, 1 pg undersized) 2024-09-02T10:24:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:00 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:24:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:00 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1196769481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:24:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3455138357' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:24:01.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:00 smithi187 ceph-mon[20072]: osdmap e659: 12 total, 12 up, 12 in 2024-09-02T10:24:01.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:01.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:01.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:01.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:00 smithi162 ceph-mon[20279]: pgmap v1528: 47 pgs: 47 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 0 B/s, 67 objects/s recovering 2024-09-02T10:24:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3455138357' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:24:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:00 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 403/9 objects degraded (4477.778%), 1 pg degraded, 1 pg undersized) 2024-09-02T10:24:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:00 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:24:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:00 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1196769481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:24:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3455138357' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:24:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:00 smithi162 ceph-mon[20279]: osdmap e659: 12 total, 12 up, 12 in 2024-09-02T10:24:01.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:01.165 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:01.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:01.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:01.293 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:01.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:01.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:01.417 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:01.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:01.470 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-02T10:24:01.471 INFO:tasks.radosbench:Beginning radosbench... 2024-09-02T10:24:01.471 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_5 2024-09-02T10:24:01.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_5 16 2024-09-02T10:24:01.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:01.540 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:01.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:01.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:01.663 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:01.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:01.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:01.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:01.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:01.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:01.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:01.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:01.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:02.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:02.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:02.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:02.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:02.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:02.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:02.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:02.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:02.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:02.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:02.305 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:02.322 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:02.326 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:02.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:02.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:02.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:02.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:02.545 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:02.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:02.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:02.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:02.670 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:02.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:02 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:24:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:02 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:24:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:02 smithi151 ceph-mon[16639]: pgmap v1530: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:02 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:24:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:02 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:24:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:02 smithi151 ceph-mon[16639]: osdmap e660: 12 total, 12 up, 12 in 2024-09-02T10:24:02.771 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:02.796 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:02.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:02.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:02.902 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:24:02.902 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:24:02.918 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:24:02.918 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:24:02.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:02.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:02.938 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:24:02.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:24:02.959 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:24:02.959 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:24:02.982 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:24:02.982 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:24:03.004 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:24:03.004 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:24:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:02 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:24:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:02 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:24:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:02 smithi187 ceph-mon[20072]: pgmap v1530: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:02 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:24:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:02 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:24:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:02 smithi187 ceph-mon[20072]: osdmap e660: 12 total, 12 up, 12 in 2024-09-02T10:24:03.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:03.026 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:24:03.026 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:24:03.044 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:03.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:03.048 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:24:03.048 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:24:03.070 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:24:03.070 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:24:03.091 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:24:03.092 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:24:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:02 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:24:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:02 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:24:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:02 smithi162 ceph-mon[20279]: pgmap v1530: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:02 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:24:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:02 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:24:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:02 smithi162 ceph-mon[20279]: osdmap e660: 12 total, 12 up, 12 in 2024-09-02T10:24:03.113 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:24:03.113 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:24:03.139 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:24:03.146 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:03.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:03.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:03.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:03.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:03.334 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:03.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:03.436 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:03.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:03.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:03.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:03.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:03.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:03.594 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:03.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:03.677 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:24:03.677 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:24:03.678 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:24:03.678 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:24:03.695 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:03.722 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:03.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:03.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:03.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:03.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:03.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1280477894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:03.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:03.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:03.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1280477894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:04.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1280477894' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:04.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:04.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:04.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:04.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:04.221 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:04.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:04.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:24:04.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:04.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:04.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:04.441 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:04.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:04.474 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:04.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:04.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:04.587 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:24:04.587 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:24:04.601 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:04.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:04.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:04.729 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:04.730 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:04.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:04.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:04.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:04.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:04 smithi151 ceph-mon[16639]: pgmap v1532: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:04.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:04.981 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:04.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:04 smithi187 ceph-mon[20072]: pgmap v1532: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:05.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:04 smithi162 ceph-mon[20279]: pgmap v1532: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:05.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:05.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:05.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:05.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:05.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:05.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:05.361 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:05.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:05.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:05.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:05.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:05.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:05.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:05.614 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:05.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:05.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:05.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:05.740 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:05.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:05.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:05.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:05.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:05.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:05 smithi151 ceph-mon[16639]: from='client.17820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:24:05.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:05.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:05.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:05 smithi187 ceph-mon[20072]: from='client.17820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:24:06.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:24:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:05 smithi162 ceph-mon[20279]: from='client.17820 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:24:06.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:06.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:06.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:06.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:06.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:06.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:06.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:06.389 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:06.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:06.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:06.512 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:06.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:06.614 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:06.640 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:06.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:06.669 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_5' created 2024-09-02T10:24:06.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:06.767 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:06.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:06.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:06.825 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1532,"stamp":"2024-09-02T10:24:03.418782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":123,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":5612040,"kb_used_data":16396,"kb_used_omap":18,"kb_used_meta":5595373,"kb_avail":1119329784,"statfs":{"total":1151940427776,"available":1146193698816,"internally_reserved":0,"allocated":16789504,"data_stored":11211079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":5729662267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-403,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-1,"num_read_kb":-1,"num_write":-200539,"num_write_kb":-6417217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-33233,"num_bytes_recovered":-1286144000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.233477"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":689,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:00.679973+0000","last_change":"2024-09-02T10:23:46.278826+0000","last_active":"2024-09-02T10:24:00.679973+0000","last_peered":"2024-09-02T10:24:00.679973+0000","last_clean":"2024-09-02T10:24:00.679973+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:24:00.679973+0000","last_undegraded":"2024-09-02T10:24:00.679973+0000","last_fullsized":"2024-09-02T10:24:00.679973+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:46.278766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:46.278766+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:11.240675+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":493,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-02T10:23:56.580099+0000","last_change":"2024-09-02T10:23:45.449362+0000","last_active":"2024-09-02T10:23:56.580099+0000","last_peered":"2024-09-02T10:23:56.580099+0000","last_clean":"2024-09-02T10:23:56.580099+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:23:56.580099+0000","last_undegraded":"2024-09-02T10:23:56.580099+0000","last_fullsized":"2024-09-02T10:23:56.580099+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:45.449313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:45.449313+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:02.805553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":951,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-02T10:23:56.186169+0000","last_change":"2024-09-02T10:23:46.236367+0000","last_active":"2024-09-02T10:23:56.186169+0000","last_peered":"2024-09-02T10:23:56.186169+0000","last_clean":"2024-09-02T10:23:56.186169+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:23:56.186169+0000","last_undegraded":"2024-09-02T10:23:56.186169+0000","last_fullsized":"2024-09-02T10:23:56.186169+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:46.236302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:46.236302+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:47.877815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":943,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-02T10:23:56.186156+0000","last_change":"2024-09-02T10:23:54.226114+0000","last_active":"2024-09-02T10:23:56.186156+0000","last_peered":"2024-09-02T10:23:56.186156+0000","last_clean":"2024-09-02T10:23:56.186156+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:23:56.186156+0000","last_undegraded":"2024-09-02T10:23:56.186156+0000","last_fullsized":"2024-09-02T10:23:56.186156+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:54.226069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:54.226069+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:19.565391+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":246,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-02T10:23:56.581716+0000","last_change":"2024-09-02T10:23:49.799236+0000","last_active":"2024-09-02T10:23:56.581716+0000","last_peered":"2024-09-02T10:23:56.581716+0000","last_clean":"2024-09-02T10:23:56.581716+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:23:56.581716+0000","last_undegraded":"2024-09-02T10:23:56.581716+0000","last_fullsized":"2024-09-02T10:23:56.581716+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:49.799176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:49.799176+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:58.525188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":786,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:01.341970+0000","last_change":"2024-09-02T10:23:01.185167+0000","last_active":"2024-09-02T10:24:01.341970+0000","last_peered":"2024-09-02T10:24:01.341970+0000","last_clean":"2024-09-02T10:24:01.341970+0000","last_became_active":"2024-09-02T10:09:58.922443+0000","last_became_peered":"2024-09-02T10:09:58.922443+0000","last_unstale":"2024-09-02T10:24:01.341970+0000","last_undegraded":"2024-09-02T10:24:01.341970+0000","last_fullsized":"2024-09-02T10:24:01.341970+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.185112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.185112+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.992983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1323,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-02T10:24:02.207290+0000","last_change":"2024-09-02T10:23:09.691782+0000","last_active":"2024-09-02T10:24:02.207290+0000","last_peered":"2024-09-02T10:24:02.207290+0000","last_clean":"2024-09-02T10:24:02.207290+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:24:02.207290+0000","last_undegraded":"2024-09-02T10:24:02.207290+0000","last_fullsized":"2024-09-02T10:24:02.207290+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:09.691691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:09.691691+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:18.662853+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":959,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-02T10:23:56.186227+0000","last_change":"2024-09-02T10:23:05.131265+0000","last_active":"2024-09-02T10:23:56.186227+0000","last_peered":"2024-09-02T10:23:56.186227+0000","last_clean":"2024-09-02T10:23:56.186227+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:23:56.186227+0000","last_undegraded":"2024-09-02T10:23:56.186227+0000","last_fullsized":"2024-09-02T10:23:56.186227+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.131213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.131213+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:19.083779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":950,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-02T10:23:56.580175+0000","last_change":"2024-09-02T10:23:06.476818+0000","last_active":"2024-09-02T10:23:56.580175+0000","last_peered":"2024-09-02T10:23:56.580175+0000","last_clean":"2024-09-02T10:23:56.580175+0000","last_became_active":"2024-09-02T10:15:12.086620+0000","last_became_peered":"2024-09-02T10:15:12.086620+0000","last_unstale":"2024-09-02T10:23:56.580175+0000","last_undegraded":"2024-09-02T10:23:56.580175+0000","last_fullsized":"2024-09-02T10:23:56.580175+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:06.476768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:06.476768+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:35.403622+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":936,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:00.683400+0000","last_change":"2024-09-02T10:23:48.667403+0000","last_active":"2024-09-02T10:24:00.683400+0000","last_peered":"2024-09-02T10:24:00.683400+0000","last_clean":"2024-09-02T10:24:00.683400+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:24:00.683400+0000","last_undegraded":"2024-09-02T10:24:00.683400+0000","last_fullsized":"2024-09-02T10:24:00.683400+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:48.667341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:48.667341+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:09.201391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":183,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-02T10:24:01.638135+0000","last_change":"2024-09-02T10:23:58.113236+0000","last_active":"2024-09-02T10:24:01.638135+0000","last_peered":"2024-09-02T10:24:01.638135+0000","last_clean":"2024-09-02T10:24:01.638135+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:24:01.638135+0000","last_undegraded":"2024-09-02T10:24:01.638135+0000","last_fullsized":"2024-09-02T10:24:01.638135+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:58.113166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:58.113166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:18.551316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1284,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-02T10:23:56.580022+0000","last_change":"2024-09-02T10:23:52.504541+0000","last_active":"2024-09-02T10:23:56.580022+0000","last_peered":"2024-09-02T10:23:56.580022+0000","last_clean":"2024-09-02T10:23:56.580022+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:23:56.580022+0000","last_undegraded":"2024-09-02T10:23:56.580022+0000","last_fullsized":"2024-09-02T10:23:56.580022+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:52.504495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:52.504495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:19.309680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1222,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-02T10:24:02.207320+0000","last_change":"2024-09-02T10:22:55.516683+0000","last_active":"2024-09-02T10:24:02.207320+0000","last_peered":"2024-09-02T10:24:02.207320+0000","last_clean":"2024-09-02T10:24:02.207320+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:24:02.207320+0000","last_undegraded":"2024-09-02T10:24:02.207320+0000","last_fullsized":"2024-09-02T10:24:02.207320+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.516633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.516633+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:05.755589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":1202,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-02T10:24:02.364730+0000","last_change":"2024-09-02T10:23:58.615781+0000","last_active":"2024-09-02T10:24:02.364730+0000","last_peered":"2024-09-02T10:24:02.364730+0000","last_clean":"2024-09-02T10:24:02.364730+0000","last_became_active":"2024-09-02T10:15:29.170344+0000","last_became_peered":"2024-09-02T10:15:29.170344+0000","last_unstale":"2024-09-02T10:24:02.364730+0000","last_undegraded":"2024-09-02T10:24:02.364730+0000","last_fullsized":"2024-09-02T10:24:02.364730+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:58.615723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:58.615723+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:16.827516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1192,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-02T10:24:02.207304+0000","last_change":"2024-09-02T10:23:00.621966+0000","last_active":"2024-09-02T10:24:02.207304+0000","last_peered":"2024-09-02T10:24:02.207304+0000","last_clean":"2024-09-02T10:24:02.207304+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:24:02.207304+0000","last_undegraded":"2024-09-02T10:24:02.207304+0000","last_fullsized":"2024-09-02T10:24:02.207304+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.621916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.621916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:21.445915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1286,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-02T10:24:02.207432+0000","last_change":"2024-09-02T10:23:01.892373+0000","last_active":"2024-09-02T10:24:02.207432+0000","last_peered":"2024-09-02T10:24:02.207432+0000","last_clean":"2024-09-02T10:24:02.207432+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:24:02.207432+0000","last_undegraded":"2024-09-02T10:24:02.207432+0000","last_fullsized":"2024-09-02T10:24:02.207432+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:01.892288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:01.892288+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:30.607091+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1407,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:00.680213+0000","last_change":"2024-09-02T10:22:37.332903+0000","last_active":"2024-09-02T10:24:00.680213+0000","last_peered":"2024-09-02T10:24:00.680213+0000","last_clean":"2024-09-02T10:24:00.680213+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:24:00.680213+0000","last_undegraded":"2024-09-02T10:24:00.680213+0000","last_fullsized":"2024-09-02T10:24:00.680213+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:37.332859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:37.332859+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.675027+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1052,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-02T10:24:02.364959+0000","last_change":"2024-09-02T10:22:55.068228+0000","last_active":"2024-09-02T10:24:02.364959+0000","last_peered":"2024-09-02T10:24:02.364959+0000","last_clean":"2024-09-02T10:24:02.364959+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:24:02.364959+0000","last_undegraded":"2024-09-02T10:24:02.364959+0000","last_fullsized":"2024-09-02T10:24:02.364959+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.068166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.068166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:04.696428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":455,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:00.679866+0000","last_change":"2024-09-02T10:23:45.263493+0000","last_active":"2024-09-02T10:24:00.679866+0000","last_peered":"2024-09-02T10:24:00.679866+0000","last_clean":"2024-09-02T10:24:00.679866+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:24:00.679866+0000","last_undegraded":"2024-09-02T10:24:00.679866+0000","last_fullsized":"2024-09-02T10:24:00.679866+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:45.263415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:45.263415+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:04.210025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1356,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:00.679867+0000","last_change":"2024-09-02T10:22:34.647196+0000","last_active":"2024-09-02T10:24:00.679867+0000","last_peered":"2024-09-02T10:24:00.679867+0000","last_clean":"2024-09-02T10:24:00.679867+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:24:00.679867+0000","last_undegraded":"2024-09-02T10:24:00.679867+0000","last_fullsized":"2024-09-02T10:24:00.679867+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:34.647144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:34.647144+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:02.024298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":822,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:01.342048+0000","last_change":"2024-09-02T10:23:05.243285+0000","last_active":"2024-09-02T10:24:01.342048+0000","last_peered":"2024-09-02T10:24:01.342048+0000","last_clean":"2024-09-02T10:24:01.342048+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:24:01.342048+0000","last_undegraded":"2024-09-02T10:24:01.342048+0000","last_fullsized":"2024-09-02T10:24:01.342048+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:05.243236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:05.243236+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:16.091958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":787,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:01.342089+0000","last_change":"2024-09-02T10:22:55.086083+0000","last_active":"2024-09-02T10:24:01.342089+0000","last_peered":"2024-09-02T10:24:01.342089+0000","last_clean":"2024-09-02T10:24:01.342089+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:24:01.342089+0000","last_undegraded":"2024-09-02T10:24:01.342089+0000","last_fullsized":"2024-09-02T10:24:01.342089+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:22:55.086036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:22:55.086036+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:06.714243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":557,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-02T10:24:01.636497+0000","last_change":"2024-09-02T10:23:42.635086+0000","last_active":"2024-09-02T10:24:01.636497+0000","last_peered":"2024-09-02T10:24:01.636497+0000","last_clean":"2024-09-02T10:24:01.636497+0000","last_became_active":"2024-09-02T10:15:12.083070+0000","last_became_peered":"2024-09-02T10:15:12.083070+0000","last_unstale":"2024-09-02T10:24:01.636497+0000","last_undegraded":"2024-09-02T10:24:01.636497+0000","last_fullsized":"2024-09-02T10:24:01.636497+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:42.635029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:42.635029+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:11.935109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":788,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:01.342151+0000","last_change":"2024-09-02T10:23:00.146326+0000","last_active":"2024-09-02T10:24:01.342151+0000","last_peered":"2024-09-02T10:24:01.342151+0000","last_clean":"2024-09-02T10:24:01.342151+0000","last_became_active":"2024-09-02T10:09:58.922368+0000","last_became_peered":"2024-09-02T10:09:58.922368+0000","last_unstale":"2024-09-02T10:24:01.342151+0000","last_undegraded":"2024-09-02T10:24:01.342151+0000","last_fullsized":"2024-09-02T10:24:01.342151+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.146278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.146278+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:09.373850+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1049,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:00.681173+0000","last_change":"2024-09-02T10:23:44.545777+0000","last_active":"2024-09-02T10:24:00.681173+0000","last_peered":"2024-09-02T10:24:00.681173+0000","last_clean":"2024-09-02T10:24:00.681173+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:24:00.681173+0000","last_undegraded":"2024-09-02T10:24:00.681173+0000","last_fullsized":"2024-09-02T10:24:00.681173+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:44.545693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:44.545693+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:53.740167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1307,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-02T10:23:56.580067+0000","last_change":"2024-09-02T10:23:08.500316+0000","last_active":"2024-09-02T10:23:56.580067+0000","last_peered":"2024-09-02T10:23:56.580067+0000","last_clean":"2024-09-02T10:23:56.580067+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:23:56.580067+0000","last_undegraded":"2024-09-02T10:23:56.580067+0000","last_fullsized":"2024-09-02T10:23:56.580067+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:08.500268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:08.500268+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:14.723023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1060,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:00.680025+0000","last_change":"2024-09-02T10:23:45.553424+0000","last_active":"2024-09-02T10:24:00.680025+0000","last_peered":"2024-09-02T10:24:00.680025+0000","last_clean":"2024-09-02T10:24:00.680025+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:24:00.680025+0000","last_undegraded":"2024-09-02T10:24:00.680025+0000","last_fullsized":"2024-09-02T10:24:00.680025+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:45.553364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:45.553364+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:06.163823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1318,"reported_epoch":660,"state":"active+clean","last_fresh":"2024-09-02T10:24:01.636519+0000","last_change":"2024-09-02T10:23:00.386934+0000","last_active":"2024-09-02T10:24:01.636519+0000","last_peered":"2024-09-02T10:24:01.636519+0000","last_clean":"2024-09-02T10:24:01.636519+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:24:01.636519+0000","last_undegraded":"2024-09-02T10:24:01.636519+0000","last_fullsized":"2024-09-02T10:24:01.636519+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:00.386875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:00.386875+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:24:03.366845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":262,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-02T10:23:56.581695+0000","last_change":"2024-09-02T10:23:48.789737+0000","last_active":"2024-09-02T10:23:56.581695+0000","last_peered":"2024-09-02T10:23:56.581695+0000","last_clean":"2024-09-02T10:23:56.581695+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:23:56.581695+0000","last_undegraded":"2024-09-02T10:23:56.581695+0000","last_fullsized":"2024-09-02T10:23:56.581695+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:48.789677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:48.789677+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:14.919249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1095,"reported_epoch":659,"state":"active+clean","last_fresh":"2024-09-02T10:24:00.683503+0000","last_change":"2024-09-02T10:23:56.583117+0000","last_active":"2024-09-02T10:24:00.683503+0000","last_peered":"2024-09-02T10:24:00.683503+0000","last_clean":"2024-09-02T10:24:00.683503+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:24:00.683503+0000","last_undegraded":"2024-09-02T10:24:00.683503+0000","last_fullsized":"2024-09-02T10:24:00.683503+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:23:56.583065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:23:56.583065+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:25:15.384474+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":468,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-02T10:23:56.188422+0000","last_change":"2024-09-02T10:20:28.001502+0000","last_active":"2024-09-02T10:23:56.188422+0000","last_peered":"2024-09-02T10:23:56.188422+0000","last_clean":"2024-09-02T10:23:56.188422+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:23:56.188422+0000","last_undegraded":"2024-09-02T10:23:56.188422+0000","last_fullsized":"2024-09-02T10:23:56.188422+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:20:28.001422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:20:28.001422+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383967,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487788,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":93257364,"statfs":{"total":95995035648,"available":95495540736,"internally_reserved":0,"allocated":1269760,"data_stored":803400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":498203076},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612900,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357292,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93387860,"statfs":{"total":95995035648,"available":95629168640,"internally_reserved":0,"allocated":1269760,"data_stored":803400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":364575168},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":546108,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":93199044,"statfs":{"total":95995035648,"available":95435821056,"internally_reserved":0,"allocated":1282048,"data_stored":810970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":557906368},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070766,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339228,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93405924,"statfs":{"total":95995035648,"available":95647666176,"internally_reserved":0,"allocated":1253376,"data_stored":795557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299700,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":582372,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":93162780,"statfs":{"total":95995035648,"available":95398686720,"internally_reserved":0,"allocated":1458176,"data_stored":996805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528631,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":365932,"kb_used_data":1560,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93379220,"statfs":{"total":95995035648,"available":95620321280,"internally_reserved":0,"allocated":1597440,"data_stored":1131080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,5,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614324,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93130828,"statfs":{"total":95995035648,"available":95365967872,"internally_reserved":0,"allocated":1474560,"data_stored":1004648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953793,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":335068,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93410084,"statfs":{"total":95995035648,"available":95651926016,"internally_reserved":0,"allocated":1253376,"data_stored":795557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150022,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":570020,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":93175132,"statfs":{"total":95995035648,"available":95411335168,"internally_reserved":0,"allocated":1581056,"data_stored":1123237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":582089160},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411658,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311680,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":93433472,"statfs":{"total":95995035648,"available":95675875328,"internally_reserved":0,"allocated":1486848,"data_stored":1012218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575183,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":509980,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93235172,"statfs":{"total":95995035648,"available":95472816128,"internally_reserved":0,"allocated":1253376,"data_stored":795557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":592248,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93152904,"statfs":{"total":95995035648,"available":95388573696,"internally_reserved":0,"allocated":1609728,"data_stored":1138650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:24:06.828 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-02T10:24:06.829 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:24:06.829 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-02T10:24:06.829 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-02T10:24:06.829 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-02T10:24:06.830 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-02T10:24:06.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:06.896 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:06.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:06 smithi151 ceph-mon[16639]: pgmap v1533: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2140256020' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-02T10:24:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:06 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-02T10:24:06.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:06 smithi187 ceph-mon[20072]: pgmap v1533: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2140256020' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-02T10:24:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:06 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-02T10:24:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:07.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:07.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:07.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:24:07.086 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:06 smithi162 ceph-mon[20279]: pgmap v1533: 31 pgs: 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2140256020' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-02T10:24:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:06 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16} : dispatch 2024-09-02T10:24:07.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:07.143 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:07.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:07.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:07.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:07.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:07.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:07.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:07.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:07.396 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:07.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:07.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:07.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:07.572 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:24:07.573 INFO:teuthology.orchestra.run.smithi162.stdout: "success": "" 2024-09-02T10:24:07.573 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:24:07.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:07.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:07.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:07.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:07.769 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:07.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:07.794 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:07.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_5 rados --yes-i-really-mean-it || true 2024-09-02T10:24:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:07 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-02T10:24:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:07 smithi162 ceph-mon[20279]: osdmap e661: 12 total, 12 up, 12 in 2024-09-02T10:24:07.871 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:07.897 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:07.897 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:07.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:07 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-02T10:24:07.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:07 smithi151 ceph-mon[16639]: osdmap e661: 12 total, 12 up, 12 in 2024-09-02T10:24:07.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:07 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_5", "pg_num": 16}]': finished 2024-09-02T10:24:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:07 smithi187 ceph-mon[20072]: osdmap e661: 12 total, 12 up, 12 in 2024-09-02T10:24:08.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:08.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:08.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:08.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:08.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:08.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:08.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:08.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:08.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:08.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:08.366 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:08.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:08.378 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:08.382 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:08.392 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:08.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:08.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:08.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:08.521 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:08.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:08.523 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-02T10:24:08.522+0000 7fd0252cb640 0 --2- 172.21.15.151:0/2640406280 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fd020101a60 0x7fd020197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-02T10:24:08.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:08.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:08.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:08.749 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:08.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:08 smithi151 ceph-mon[16639]: pgmap v1535: 47 pgs: 6 creating+peering, 10 unknown, 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:08.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:08 smithi151 ceph-mon[16639]: osdmap e662: 12 total, 12 up, 12 in 2024-09-02T10:24:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:08.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:08.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:08.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:08.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:08.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:09.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:08 smithi187 ceph-mon[20072]: pgmap v1535: 47 pgs: 6 creating+peering, 10 unknown, 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:08 smithi187 ceph-mon[20072]: osdmap e662: 12 total, 12 up, 12 in 2024-09-02T10:24:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:09.025 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:09.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:08 smithi162 ceph-mon[20279]: pgmap v1535: 47 pgs: 6 creating+peering, 10 unknown, 31 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:08 smithi162 ceph-mon[20279]: osdmap e662: 12 total, 12 up, 12 in 2024-09-02T10:24:09.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:09.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:09.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:09.175 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:24:09.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:09.277 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:09.277 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:09.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:09.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:09.405 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:09.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:09.533 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:09.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:09.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:09.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:09.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:09.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:09 smithi187 ceph-mon[20072]: osdmap e663: 12 total, 12 up, 12 in 2024-09-02T10:24:09.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2640406280' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:09.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:09 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:09.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2640406280' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:09.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:09 smithi187 ceph-mon[20072]: osdmap e664: 12 total, 12 up, 12 in 2024-09-02T10:24:09.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3750113351' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:24:09.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:09 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:24:09.681 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:09 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:24:09.750 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:24:09.750 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:24:09.757 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:24:09.757 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:24:09.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:09.777 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:24:09.777 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:24:09.787 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:09.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:09.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:09.797 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:24:09.797 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:24:09.819 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:24:09.819 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:24:09.840 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:24:09.840 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:24:09.861 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:24:09.861 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:24:09.861 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:24:09.861 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:24:09.861 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:24:09.862 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:24:09.862 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:24:09.883 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:24:09.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:24:09.889 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:09.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:24:09.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:24:09.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:09.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:09.916 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:09.925 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:24:09.925 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:24:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:09 smithi151 ceph-mon[16639]: osdmap e663: 12 total, 12 up, 12 in 2024-09-02T10:24:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2640406280' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:09 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2640406280' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:09 smithi151 ceph-mon[16639]: osdmap e664: 12 total, 12 up, 12 in 2024-09-02T10:24:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3750113351' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:24:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:09 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:24:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:09 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:24:09.946 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:24:09.946 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:24:09.969 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:24:09.970 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:24:10.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:10.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:10.042 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:10.042 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:09 smithi162 ceph-mon[20279]: osdmap e663: 12 total, 12 up, 12 in 2024-09-02T10:24:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2640406280' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:09 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2640406280' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:09 smithi162 ceph-mon[20279]: osdmap e664: 12 total, 12 up, 12 in 2024-09-02T10:24:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3750113351' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:24:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:09 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:24:10.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:09 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:24:10.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:10.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:10.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:10.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:10.182 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_5' 2024-09-02T10:24:10.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:10.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:10.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:10.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:10.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:10.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:10.513 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:10.539 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:10.540 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:10.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:10.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:10.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:10.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:10.765 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:10.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:10.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:10.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:10.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:10.921 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:10.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:10 smithi151 ceph-mon[16639]: pgmap v1539: 47 pgs: 7 creating+peering, 1 unknown, 39 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2588948223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3750113351' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:24:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:10 smithi151 ceph-mon[16639]: osdmap e665: 12 total, 12 up, 12 in 2024-09-02T10:24:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:10 smithi187 ceph-mon[20072]: pgmap v1539: 47 pgs: 7 creating+peering, 1 unknown, 39 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2588948223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3750113351' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:24:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:10 smithi187 ceph-mon[20072]: osdmap e665: 12 total, 12 up, 12 in 2024-09-02T10:24:11.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:11.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:11.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:10 smithi162 ceph-mon[20279]: pgmap v1539: 47 pgs: 7 creating+peering, 1 unknown, 39 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2588948223' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3750113351' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_5", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:24:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:10 smithi162 ceph-mon[20279]: osdmap e665: 12 total, 12 up, 12 in 2024-09-02T10:24:11.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:11.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:11.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:11.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:11.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:11.306 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:11.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:11.408 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:11.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:11.435 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:11.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:11.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:11.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:11.563 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:11.664 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:11.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:11.691 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:11.792 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:11.819 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:11.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:11.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:11.946 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:11.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:11.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:12.048 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:12.073 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:12.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:12.175 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:12.201 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:12.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:12.203 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:12.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:12.330 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:12.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:12.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:12.459 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:12.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:12.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:12.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:12.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:12.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:24:12.687 DEBUG:teuthology.orchestra.run.smithi187:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_5 bench 90 write' 2024-09-02T10:24:12.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:12.690 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-02T10:24:12.690 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-02T10:24:12.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:24:12.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:12.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:12.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:12.754 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:hints = 1 2024-09-02T10:24:12.755 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-02T10:24:12.755 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object prefix: benchmark_data_smithi187_120712 2024-09-02T10:24:12.755 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:24:12.755 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 0 0 0 0 0 0 - 0 2024-09-02T10:24:12.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:12.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:12.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:12.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:12 smithi151 ceph-mon[16639]: pgmap v1541: 47 pgs: 7 creating+peering, 40 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:12.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:12 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:24:12.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:12 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:24:12.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:12 smithi151 ceph-mon[16639]: osdmap e666: 12 total, 12 up, 12 in 2024-09-02T10:24:12.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:12.974 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:12.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:12.997 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:12 smithi187 ceph-mon[20072]: pgmap v1541: 47 pgs: 7 creating+peering, 40 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:12 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:24:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:12 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:24:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:12 smithi187 ceph-mon[20072]: osdmap e666: 12 total, 12 up, 12 in 2024-09-02T10:24:13.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:12 smithi162 ceph-mon[20279]: pgmap v1541: 47 pgs: 7 creating+peering, 40 active+clean; 577 KiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:24:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:12 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:24:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:12 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:24:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:12 smithi162 ceph-mon[20279]: osdmap e666: 12 total, 12 up, 12 in 2024-09-02T10:24:13.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:13.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:13.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:13.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:13.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:13.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:13.355 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:13.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:13.378 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:24:13.378 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:24:13.378 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-02T10:24:13.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 1000 3.0 2024-09-02T10:24:13.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:13.490 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:13.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:13.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:13.598 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:13.598 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:13.615 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:13.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:13.717 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:13.743 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:13.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:13.744 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:13.755 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 1 16 2173 2157 134.805 134.812 0.0143629 0.00720542 2024-09-02T10:24:13.837 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:13.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:13.866 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:13.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:13.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:13.991 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:13.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:14.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:14.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:24:14.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:14.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:14.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:14.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:14.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:14.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:14.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:14.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:14.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:14.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:14.394 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:14.478 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:24:14.478 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:24:14.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:14.506 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:24:14.506 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:24:14.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:14.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:14.518 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:14.532 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:24:14.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:24:14.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:24:14.582 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:24:14.583 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:24:14.612 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:24:14.613 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:24:14.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:14.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:24:14.644 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:24:14.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:14.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:14.646 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:14.670 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:24:14.671 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:24:14.694 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:24:14.694 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:24:14.717 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:24:14.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:24:14.739 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:24:14.740 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:24:14.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:14.755 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 2 15 3529 3514 109.806 84.8125 0.0038176 0.00906909 2024-09-02T10:24:14.768 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:24:14.769 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:24:14.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:14.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:14.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:14.900 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:14.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:14 smithi151 ceph-mon[16639]: pgmap v1543: 47 pgs: 47 active+clean; 4.8 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s wr, 14 op/s 2024-09-02T10:24:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1945879415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:15.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:14 smithi187 ceph-mon[20072]: pgmap v1543: 47 pgs: 47 active+clean; 4.8 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s wr, 14 op/s 2024-09-02T10:24:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1945879415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:15.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:15.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:15.024 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:14 smithi162 ceph-mon[20279]: pgmap v1543: 47 pgs: 47 active+clean; 4.8 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s wr, 14 op/s 2024-09-02T10:24:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1945879415' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:15.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:15.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:15.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:15.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:24:15.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:15.283 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:15.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:15.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:15.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:15.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:15.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:15.493 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:15.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:15.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:15.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:15.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:15.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:15.659 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:15.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:15.673 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:15.695 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:15.755 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 3 16 5164 5148 107.243 102.125 0.0179682 0.00928633 2024-09-02T10:24:15.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:15.783 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:15.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:15.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:15.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:15.906 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:15.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:16.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:16.031 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:16.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:16.096 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:24:16.096 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27220.0:5351 7.3 e666/661 7:c2bfeeba:::benchmark_data_smithi187_120712_object5350:head v 666'364, mlcod=666'362)", 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:24:16.031629+0000", 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.000222189, 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024527400000000002, 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:16.031629+0000", 2024-09-02T10:24:16.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:16.031629+0000", 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:16.031631+0000", 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.065e-06 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:16.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:16.031679+0000", 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8000999999999997e-05 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:16.031712+0000", 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3185000000000002e-05 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:16.031721+0000", 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.064e-06 2024-09-02T10:24:16.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:16.031799+0000", 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8196999999999997e-05 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:16.031820+0000", 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0843e-05 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:16.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:16.101 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:24:16.101 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-02T10:24:16.101 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:24:16.133 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:16.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:16.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:16.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:16.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:16.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:16.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:16.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:16.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:16.412 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:16.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:16.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:16.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:16.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:16.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:16.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:16.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:16.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:16.755 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 4 16 6432 6416 100.243 79.25 0.0123908 0.00995705 2024-09-02T10:24:16.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:16.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:16.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:16.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:16.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:24:16.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:16.923 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:16.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:16 smithi151 ceph-mon[16639]: pgmap v1544: 47 pgs: 47 active+clean; 50 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 127 op/s 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 141 (0%) 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout:avg 11.75 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 1.29904 -> 1.29904 (expected baseline 3.28189) 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.5 with 10 -> 10 pgs (0.851064 -> 0.851064 * mean) 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.0 with 14 -> 14 pgs (1.19149 -> 1.19149 * mean) 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0051 2024-09-02T10:24:16.957 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0511 2024-09-02T10:24:16.958 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:24:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:16 smithi187 ceph-mon[20072]: pgmap v1544: 47 pgs: 47 active+clean; 50 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 127 op/s 2024-09-02T10:24:17.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:17.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:17.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:17.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:16 smithi162 ceph-mon[20279]: pgmap v1544: 47 pgs: 47 active+clean; 50 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s wr, 127 op/s 2024-09-02T10:24:17.140 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:17.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:17.182 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:17.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:17.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:17.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:17.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:17.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:17.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:17.437 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:17.437 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:17.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:17.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:17.565 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:17.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:24:17.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:17.687 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:17.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:17.700 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:24:17.722 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:24:17.722 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:24:17.722 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:24:17.722 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:24:17.722 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:24:17.755 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 5 16 7840 7824 97.7929 88 0.0516544 0.0101431 2024-09-02T10:24:17.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:17.810 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:17.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:17.811 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:17.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:17.935 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:17.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/748809308' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:24:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:17 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:24:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:17 smithi151 ceph-mon[16639]: from='client.27272 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/748809308' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:24:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:17 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:24:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:17 smithi187 ceph-mon[20072]: from='client.27272 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:18.038 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:18.062 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:18.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:18.075 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-02T10:24:18.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-09-02T10:24:18.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/748809308' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:24:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:17 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:24:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:17 smithi162 ceph-mon[20279]: from='client.27272 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:18.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:18.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:18.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:18.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:18.314 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:18.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:18.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:18.437 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:18.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:18.539 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:18.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:18.564 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:18.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:24:18.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:18.694 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:18.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:18.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:18.755 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 6 16 8793 8777 91.4205 59.5625 0.00180159 0.0108786 2024-09-02T10:24:18.796 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:18.802 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:18.819 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:18.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:18.921 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:18.925 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:18.946 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:18.947 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:18.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:18 smithi151 ceph-mon[16639]: pgmap v1545: 47 pgs: 47 active+clean; 219 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 435 op/s 2024-09-02T10:24:18.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:18 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:18.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:18 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:24:18.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:18 smithi151 ceph-mon[16639]: osdmap e667: 12 total, 12 up, 12 in 2024-09-02T10:24:18.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:18 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:24:18.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:18 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:24:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:18 smithi187 ceph-mon[20072]: pgmap v1545: 47 pgs: 47 active+clean; 219 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 435 op/s 2024-09-02T10:24:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:18 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:18 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:24:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:18 smithi187 ceph-mon[20072]: osdmap e667: 12 total, 12 up, 12 in 2024-09-02T10:24:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:18 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:24:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:18 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:24:19.048 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:19.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:18 smithi162 ceph-mon[20279]: pgmap v1545: 47 pgs: 47 active+clean; 219 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 435 op/s 2024-09-02T10:24:19.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:18 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:19.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:18 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:24:19.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:18 smithi162 ceph-mon[20279]: osdmap e667: 12 total, 12 up, 12 in 2024-09-02T10:24:19.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:18 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:24:19.052 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:18 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:24:19.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:19.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:19.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:19.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:19.197 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:19.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:19.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:19.326 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:19.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:19.390 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:19.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:19.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:19.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:19.455 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:19.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:19.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:19.583 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:19.671 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 141 (0%) 2024-09-02T10:24:19.671 INFO:teuthology.orchestra.run.smithi151.stdout:avg 11.75 2024-09-02T10:24:19.671 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 1.29904 -> 1.29904 (expected baseline 3.28189) 2024-09-02T10:24:19.671 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.5 with 10 -> 10 pgs (0.851064 -> 0.851064 * mean) 2024-09-02T10:24:19.671 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.0 with 14 -> 14 pgs (1.19149 -> 1.19149 * mean) 2024-09-02T10:24:19.671 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:24:19.671 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-09-02T10:24:19.671 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 3 2024-09-02T10:24:19.672 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:24:19.672 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0062 2024-09-02T10:24:19.672 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0620 2024-09-02T10:24:19.672 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:24:19.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:19.692 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:24:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:24:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:24:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:24:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:63394 6.2 6:45958ca0:::benchmark_data_smithi187_102448_object63393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-02T10:24:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:35.329719+0000", 2024-09-02T10:24:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 284.35983618500001, 2024-09-02T10:24:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.245115466, 2024-09-02T10:24:19.692 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 63394 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329719+0000", 2024-09-02T10:24:19.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329719+0000", 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329720+0000", 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0600000000000001e-07 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329791+0000", 2024-09-02T10:24:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0882000000000003e-05 2024-09-02T10:24:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329799+0000", 2024-09-02T10:24:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9799999999999998e-06 2024-09-02T10:24:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.329803+0000", 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.459e-06 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.331811+0000", 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020077670000000001 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.331940+0000", 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000129333 2024-09-02T10:24:19.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.332083+0000", 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014232000000000001 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.333709+0000", 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016269209999999999 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.334159+0000", 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000449591 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.574771+0000", 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24061200899999999 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.574805+0000", 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4241e-05 2024-09-02T10:24:19.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.574835+0000", 2024-09-02T10:24:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9556999999999998e-05 2024-09-02T10:24:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:63461 6.2 6:457e4280:::benchmark_data_smithi187_102448_object63460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:35.571810+0000", 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 284.11774513900002, 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24504642700000001, 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 63461 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.700 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571810+0000", 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571810+0000", 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571817+0000", 2024-09-02T10:24:19.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.64e-06 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571864+0000", 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7219000000000003e-05 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571897+0000", 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2789000000000001e-05 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.571913+0000", 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5503999999999999e-05 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.572024+0000", 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111838 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.572167+0000", 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000142762 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.572280+0000", 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113335 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.573933+0000", 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001652803 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.574669+0000", 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073586099999999998 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.816773+0000", 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24210336299999999 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.816825+0000", 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.236e-05 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.816857+0000", 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1952999999999997e-05 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:110262 6.2 6:40592b50:::benchmark_data_smithi187_102448_object9990:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e576)", 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:22.062831+0000", 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 237.62672479899999, 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21039696099999999, 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.706 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.707 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 110262 2024-09-02T10:24:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.708 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062831+0000", 2024-09-02T10:24:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062831+0000", 2024-09-02T10:24:19.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062832+0000", 2024-09-02T10:24:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6190000000000001e-06 2024-09-02T10:24:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062838+0000", 2024-09-02T10:24:19.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8159999999999999e-06 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062863+0000", 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4516999999999999e-05 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062870+0000", 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7870000000000002e-06 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.062928+0000", 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7073000000000001e-05 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.063033+0000", 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010552600000000001 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.063137+0000", 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104378 2024-09-02T10:24:19.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.065961+0000", 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002823365 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.067130+0000", 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001169074 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.273125+0000", 2024-09-02T10:24:19.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20599530199999999 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.273182+0000", 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7024000000000002e-05 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.273228+0000", 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5479999999999998e-05 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.716 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:110281 6.2 6:466107a3:::benchmark_data_smithi187_102448_object10009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e576)", 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:22.085562+0000", 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 237.603993417, 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20897753599999999, 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 110281 2024-09-02T10:24:19.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085562+0000", 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085562+0000", 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085564+0000", 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0999999999999998e-06 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085572+0000", 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2500000000000006e-06 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085596+0000", 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3176e-05 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085605+0000", 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5400000000000001e-06 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085661+0000", 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5930999999999997e-05 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085806+0000", 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000144988 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.085916+0000", 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110084 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.089195+0000", 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032792860000000002 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.089560+0000", 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036478200000000002 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.294468+0000", 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204907331 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.294502+0000", 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4137999999999997e-05 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:22.294540+0000", 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7929999999999998e-05 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:121269 6.7 e579/565 6:e666b9e5:::benchmark_data_smithi187_102448_object20997:head v 579'7625, mlcod=579'7622)", 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:34.395659+0000", 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 225.293896596, 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20071814499999999, 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.723 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.395659+0000", 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.395659+0000", 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.395660+0000", 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0299999999999999e-07 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.395664+0000", 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8409999999999999e-06 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.584140+0000", 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18847551700000001 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.584147+0000", 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6470000000000001e-06 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.584221+0000", 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4035000000000005e-05 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.584246+0000", 2024-09-02T10:24:19.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5167000000000001e-05 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.596164+0000", 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011918529000000001 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.596377+0000", 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00021280600000000001 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:121271 6.e 6:75ac4dfb:::benchmark_data_smithi187_102448_object20999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e579)", 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:34.398988+0000", 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 225.290567603, 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.223294623, 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 121271 2024-09-02T10:24:19.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.398988+0000", 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.398988+0000", 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.398989+0000", 2024-09-02T10:24:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.085e-06 2024-09-02T10:24:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.398994+0000", 2024-09-02T10:24:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:24:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.399006+0000", 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1814000000000001e-05 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.399013+0000", 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2799999999999998e-06 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.399066+0000", 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.287e-05 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.399186+0000", 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011956299999999999 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.399298+0000", 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111878 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.402719+0000", 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034216699999999999 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.402824+0000", 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010517400000000001 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.622131+0000", 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21930701699999999 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.622232+0000", 2024-09-02T10:24:19.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100552 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.622283+0000", 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0599000000000001e-05 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:135976 6.e 6:7520f53c:::benchmark_data_smithi187_102448_object35704:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.137894+0000", 2024-09-02T10:24:19.734 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 206.551661862, 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.248802408, 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135976 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137894+0000", 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137894+0000", 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137896+0000", 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0899999999999999e-06 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137902+0000", 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1789999999999996e-06 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137918+0000", 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5664000000000001e-05 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137929+0000", 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1711e-05 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139544+0000", 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001614905 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139556+0000", 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1411e-05 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.386584+0000", 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.247028571 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.386696+0000", 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111877 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:135977 6.e 6:7fb0371e:::benchmark_data_smithi187_102448_object35705:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.137955+0000", 2024-09-02T10:24:19.739 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 206.55160093399999, 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.250164689, 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135977 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137955+0000", 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137955+0000", 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137955+0000", 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6899999999999998e-07 2024-09-02T10:24:19.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137958+0000", 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0620000000000001e-06 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137969+0000", 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0808000000000001e-05 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137974+0000", 2024-09-02T10:24:19.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1869999999999996e-06 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139586+0000", 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001612112 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139593+0000", 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2400000000000001e-06 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.388023+0000", 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.248429916 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.388119+0000", 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5895000000000003e-05 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:135999 6.e 6:73eba362:::benchmark_data_smithi187_102448_object35727:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:24:19.744 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.137984+0000", 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 206.55157107400001, 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28469357299999998, 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135999 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137984+0000", 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137984+0000", 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137985+0000", 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9900000000000001e-07 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137987+0000", 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.227e-06 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137993+0000", 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8490000000000002e-06 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.137997+0000", 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2250000000000002e-06 2024-09-02T10:24:19.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139619+0000", 2024-09-02T10:24:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016216850000000001 2024-09-02T10:24:19.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:24:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139625+0000", 2024-09-02T10:24:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4880000000000004e-06 2024-09-02T10:24:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.388162+0000", 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24853692999999999 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.388319+0000", 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015642299999999999 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.388478+0000", 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015945999999999999 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395778+0000", 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072996850000000002 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395788+0000", 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0217999999999999e-05 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.420479+0000", 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024690614999999999 2024-09-02T10:24:19.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.420548+0000", 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9344000000000006e-05 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.422678+0000", 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021300249999999998 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:136003 6.e 6:75c4b550:::benchmark_data_smithi187_102448_object35731:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.138003+0000", 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 206.55155272900001, 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.289552264, 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136003 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.753 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138003+0000", 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138003+0000", 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138003+0000", 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8500000000000002e-07 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138005+0000", 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8449999999999999e-06 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138010+0000", 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3510000000000001e-06 2024-09-02T10:24:19.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138014+0000", 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1520000000000002e-06 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139659+0000", 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016441649999999999 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139666+0000", 2024-09-02T10:24:19.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1810000000000004e-06 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.392451+0000", 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25278470600000003 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.392588+0000", 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013727600000000001 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.392676+0000", 2024-09-02T10:24:19.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8603000000000005e-05 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.398948+0000", 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062717700000000003 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.400446+0000", 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001498044 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.425036+0000", 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024589508 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.425131+0000", 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5035000000000001e-05 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.427555+0000", 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024243429999999998 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.759 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:136017 6.e 6:7f8ba5a7:::benchmark_data_smithi187_102448_object35745:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.138020+0000", 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 206.55153587000001, 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29218707199999999, 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.760 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136017 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138020+0000", 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138020+0000", 2024-09-02T10:24:19.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138020+0000", 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0199999999999998e-07 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138022+0000", 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1280000000000002e-06 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138027+0000", 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3159999999999996e-06 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138032+0000", 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2849999999999996e-06 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139738+0000", 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017061380000000001 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.139745+0000", 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1409999999999999e-06 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395833+0000", 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25608827899999997 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395931+0000", 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7783999999999998e-05 2024-09-02T10:24:19.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:24:19.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.396015+0000", 2024-09-02T10:24:19.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4227999999999994e-05 2024-09-02T10:24:19.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:19.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.399079+0000", 2024-09-02T10:24:19.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030640709999999998 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.400485+0000", 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001405917 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.427645+0000", 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027160153999999999 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.427728+0000", 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2545000000000003e-05 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.430207+0000", 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024787839999999999 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:136022 6.2 6:49ff13c6:::benchmark_data_smithi187_102448_object35750:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.138646+0000", 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 206.55090987, 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28079392800000003, 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136022 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138646+0000", 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138646+0000", 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:19.770 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:19.770 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 7 16 10256 10240 91.4218 91.4375 0.00229575 0.0109105 2024-09-02T10:24:19.771 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138646+0000", 2024-09-02T10:24:19.771 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9100000000000003e-07 2024-09-02T10:24:19.771 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138650+0000", 2024-09-02T10:24:19.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8430000000000003e-06 2024-09-02T10:24:19.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.772 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.772 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.772 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138662+0000", 2024-09-02T10:24:19.772 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1304999999999999e-05 2024-09-02T10:24:19.772 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.138668+0000", 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.438e-06 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.140143+0000", 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001474593 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:24:19.773 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.140150+0000", 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.391e-06 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395737+0000", 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25558662700000001 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395868+0000", 2024-09-02T10:24:19.774 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013133400000000001 2024-09-02T10:24:19.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:24:19.775 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.395955+0000", 2024-09-02T10:24:19.775 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6730999999999993e-05 2024-09-02T10:24:19.775 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.775 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.775 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.400421+0000", 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044657969999999996 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.400835+0000", 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041487400000000003 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.416898+0000", 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016062238 2024-09-02T10:24:19.776 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.416971+0000", 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3213000000000005e-05 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.419440+0000", 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024687530000000002 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.777 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:193645 6.6 e622/559 6:6289853a:::benchmark_data_smithi187_102448_object93373:head v 622'12130, mlcod=622'12126)", 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:28.333935+0000", 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 111.35562059199999, 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28267591600000003, 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.333935+0000", 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.778 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.333935+0000", 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.333936+0000", 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.71e-07 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.779 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.333941+0000", 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2220000000000001e-06 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.611497+0000", 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.277555634 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.611504+0000", 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3080000000000002e-06 2024-09-02T10:24:19.780 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.611561+0000", 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7142999999999998e-05 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.611581+0000", 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9658999999999999e-05 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.616574+0000", 2024-09-02T10:24:19.781 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049932040000000002 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.616611+0000", 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6875000000000002e-05 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(6.2 e628,586 6:45d7b3b9:::benchmark_data_smithi187_102448_object17060:head,577'7375)", 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:45.983143+0000", 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 93.706412209000007, 2024-09-02T10:24:19.782 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20322337300000001, 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:45.983143+0000", 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:45.983143+0000", 2024-09-02T10:24:19.783 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:45.983144+0000", 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.54e-07 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:45.983148+0000", 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1339999999999997e-06 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.784 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:46.185861+0000", 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20271291699999999 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:46.185886+0000", 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5120999999999999e-05 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:46.185947+0000", 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0911e-05 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.785 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:46.186367+0000", 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00041943599999999997 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(6.e e629,586 6:7d7ad0b0:::benchmark_data_smithi187_102448_object17910:head,578'7365)", 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:47.531620+0000", 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.157935748, 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33011676699999998, 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.786 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.531620+0000", 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.531620+0000", 2024-09-02T10:24:19.787 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.531620+0000", 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5499999999999998e-07 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.531624+0000", 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.788e-06 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.788 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.860823+0000", 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32919923299999998 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.860842+0000", 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9023000000000001e-05 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.860972+0000", 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012946100000000001 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.789 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.861737+0000", 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00076460700000000005 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(6.2 e629,586 6:468a5e7b:::benchmark_data_smithi187_102448_object20227:head,578'7570)", 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:47.537486+0000", 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 92.152069295999993, 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33096709800000002, 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.790 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.537486+0000", 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.537486+0000", 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.791 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.537488+0000", 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.567e-06 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.537498+0000", 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7189999999999996e-06 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.867624+0000", 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33012665600000002 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.792 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.867639+0000", 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5099999999999999e-05 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.867805+0000", 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016597699999999999 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:47.868453+0000", 2024-09-02T10:24:19.793 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00064807899999999997 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(6.2 e630,586 6:42f617a9:::benchmark_data_smithi187_102448_object24238:head,580'7819)", 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:49.554959+0000", 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 90.134596735000002, 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24105380200000001, 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.794 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.554959+0000", 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.554959+0000", 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.795 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.554960+0000", 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2200000000000003e-07 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.554963+0000", 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.743e-06 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.796 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.795503+0000", 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24053967600000001 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.795509+0000", 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4740000000000002e-06 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.795571+0000", 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1538999999999995e-05 2024-09-02T10:24:19.797 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.796013+0000", 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000441648 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(6.e e637,586 6:7c8529ae:::benchmark_data_smithi187_102448_object41022:head,589'8868)", 2024-09-02T10:24:19.798 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:57.679851+0000", 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 82.009704829, 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.76652717000000004, 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:57.679851+0000", 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:57.679851+0000", 2024-09-02T10:24:19.799 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:57.679851+0000", 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4899999999999995e-07 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:57.679856+0000", 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2699999999999998e-06 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.800 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:58.445866+0000", 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.76601032099999999 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:58.445886+0000", 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9686000000000001e-05 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:58.445969+0000", 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2971000000000002e-05 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.801 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:58.446378+0000", 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040927299999999998 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(6.e e637,586 6:7191cc18:::benchmark_data_smithi187_102448_object55190:head,596'9785)", 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:59.306903+0000", 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 80.382652735999997, 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.76836126599999999, 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.802 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:59.306903+0000", 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:59.306903+0000", 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.803 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:59.306904+0000", 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4300000000000002e-07 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:59.306908+0000", 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0960000000000003e-06 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:23:00.074945+0000", 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.76803782399999998 2024-09-02T10:24:19.804 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:23:00.074954+0000", 2024-09-02T10:24:19.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7250000000000003e-06 2024-09-02T10:24:19.805 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.805 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.805 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.805 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:23:00.075043+0000", 2024-09-02T10:24:19.805 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8663000000000001e-05 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:23:00.075264+0000", 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000221215 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:7778 7.1 7:8c9ea63b:::benchmark_data_smithi187_120712_object7777:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e666)", 2024-09-02T10:24:19.806 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:24:17.652896+0000", 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.0366593220000002, 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20130980000000001, 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7778 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.652896+0000", 2024-09-02T10:24:19.807 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.652896+0000", 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.652898+0000", 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9709999999999998e-06 2024-09-02T10:24:19.808 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.652926+0000", 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7832999999999999e-05 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.652939+0000", 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3315e-05 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.652963+0000", 2024-09-02T10:24:19.809 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3431e-05 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.653024+0000", 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1387e-05 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.653156+0000", 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000131388 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,10", 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.653300+0000", 2024-09-02T10:24:19.810 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014397600000000001 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.653864+0000", 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056453099999999995 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.654185+0000", 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00032143800000000001 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:19.811 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.854137+0000", 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19995128700000001 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.854168+0000", 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1349000000000001e-05 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:17.854206+0000", 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7894e-05 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.812 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.813 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:19.813 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:19.813 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:24:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:19 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:24:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:19 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:24:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:19 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:24:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:19 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:24:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:19 smithi162 ceph-mon[20279]: osdmap e668: 12 total, 12 up, 12 in 2024-09-02T10:24:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:19 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:24:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:19 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:24:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:19 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:24:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:19 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:24:19.871 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:19.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:19.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:19.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:19 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:24:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:19 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:24:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:19 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:24:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:19 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:24:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:19 smithi151 ceph-mon[16639]: osdmap e668: 12 total, 12 up, 12 in 2024-09-02T10:24:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:19 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:24:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:19 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:24:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:19 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:24:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:19 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:24:20.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:19 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:24:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:19 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:24:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:19 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:24:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:19 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:24:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:19 smithi187 ceph-mon[20072]: osdmap e668: 12 total, 12 up, 12 in 2024-09-02T10:24:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:19 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:24:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:19 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:24:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:19 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:24:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:19 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:24:20.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:20.027 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:20.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:20.154 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:20.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:20.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:20.282 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:20.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:20.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:20.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:20.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:20.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:20.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:24:20.576 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:20.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:20.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:20.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:20.706 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:20.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:20.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:20.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 8 16 12037 12021 93.9071 111.312 0.00250381 0.0106316 2024-09-02T10:24:20.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:24:20.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:20.833 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:20.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:20 smithi162 ceph-mon[20279]: pgmap v1548: 47 pgs: 47 active+clean; 470 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 970 op/s 2024-09-02T10:24:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:20 smithi162 ceph-mon[20279]: from='client.27244 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:20 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:24:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:20 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:24:20.854 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:20.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:20.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:20.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:20.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:20 smithi151 ceph-mon[16639]: pgmap v1548: 47 pgs: 47 active+clean; 470 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 970 op/s 2024-09-02T10:24:20.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:20 smithi151 ceph-mon[16639]: from='client.27244 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:20.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:20 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:24:20.997 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:20 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:24:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:20 smithi187 ceph-mon[20072]: pgmap v1548: 47 pgs: 47 active+clean; 470 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 MiB/s wr, 970 op/s 2024-09-02T10:24:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:20 smithi187 ceph-mon[20072]: from='client.27244 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:20 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:24:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:20 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:24:21.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:21.086 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:21.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:21.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:21.215 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:21.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:21.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:21.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:21.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:21.380 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:24:21.380 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:24:21.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.380 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:12957 7.0 7:0afd638a:::benchmark_data_smithi187_120712_object12956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e668)", 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:24:21.369931+0000", 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0090880949999999992, 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0091108920000000006, 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 12957 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:24:21.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:21.369931+0000", 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:21.369931+0000", 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:21.369932+0000", 2024-09-02T10:24:21.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3699999999999999e-07 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:21.369985+0000", 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2664000000000003e-05 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:21.369995+0000", 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0328e-05 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:24:21.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:21.370000+0000", 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1649999999999997e-06 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:21.370040+0000", 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9334e-05 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:21.370166+0000", 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126704 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:21.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:21.370300+0000", 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013334600000000001 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:21.370910+0000", 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00061024899999999999 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:24:21.385 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-02T10:24:21.386 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:24:21.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:21.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:21.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:21.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:21.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:21.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:21.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:21.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:21.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:21.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 9 16 13561 13545 94.0555 95.25 0.00430508 0.0106189 2024-09-02T10:24:21.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:21.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:21.860 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:21.961 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:21.979 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:21 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:24:21.979 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:21 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:24:21.979 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:21 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:24:21.979 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:21 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:24:21.979 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1730250676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:21.979 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:21 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:24:21.979 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:21 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:24:21.980 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:24:21.980 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:24:21.987 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:21.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:22.001 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:24:22.002 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:24:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:21 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:24:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:21 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:24:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:21 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:24:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:21 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:24:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1730250676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:21 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:24:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:21 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:24:22.027 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:24:22.027 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:24:22.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:21 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:24:22.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:21 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:24:22.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:21 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:24:22.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:21 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:24:22.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1730250676' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:22.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:21 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:24:22.036 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:21 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:24:22.054 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:24:22.054 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:24:22.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:24:22.086 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:24:22.086 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:24:22.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:22.091 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-02T10:24:22.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 110 1.0 2024-09-02T10:24:22.092 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:22.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:24:22.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:24:22.121 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:22.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:22.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:22.149 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:24:22.149 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:24:22.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:24:22.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:24:22.214 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:24:22.215 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:24:22.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:22.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:24:22.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:24:22.248 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:22.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:22.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:22.272 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:24:22.272 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:24:22.304 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:24:22.304 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:24:22.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:22.363 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:22.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:22.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:22.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:22.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:22.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:22.501 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:22.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:22.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:22.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:22.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:22.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 10 16 14964 14948 93.418 87.6875 0.0135026 0.0106923 2024-09-02T10:24:22.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:22.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:22 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:24:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:22 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:24:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:22 smithi162 ceph-mon[20279]: pgmap v1549: 47 pgs: 47 active+clean; 497 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 984 op/s 2024-09-02T10:24:22.863 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:22.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:22.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:22.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:22.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:22 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:24:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:22 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:24:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:22 smithi187 ceph-mon[20072]: pgmap v1549: 47 pgs: 47 active+clean; 497 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 984 op/s 2024-09-02T10:24:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:23.024 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:23.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:23.126 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:23.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:23.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:23.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:22 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:24:23.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:22 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:24:23.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:22 smithi151 ceph-mon[16639]: pgmap v1549: 47 pgs: 47 active+clean; 497 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 984 op/s 2024-09-02T10:24:23.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:23.256 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:23.256 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:23.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:23.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:23.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:23.384 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:23.410 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:23.411 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:23.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:23.538 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:23.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:23.595 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:23.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:23.663 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:23.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:23.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:23.722 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:24:23.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 11 16 16484 16468 93.5612 95 0.0411604 0.0106518 2024-09-02T10:24:23.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:23.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:23.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4022643276' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:23.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:23.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:23.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:23.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4022643276' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:24.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:24.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:24.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:24.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:24.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:24:24.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4022643276' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:24.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:24.169 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:24.169 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:24.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:24.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:24.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:24.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:24.404 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:24.428 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:24.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:24.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:24.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:24.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:24.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:24.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:24.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:24.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:24.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 12 16 17960 17944 93.4513 92.25 0.0016496 0.0106767 2024-09-02T10:24:24.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:24.818 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:24.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:24.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:24.916 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:24.917 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:24.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:24 smithi162 ceph-mon[20279]: pgmap v1550: 47 pgs: 47 active+clean; 837 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-02T10:24:24.927 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:24 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:24:24.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:24 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:24:24.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4022643276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:24.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:24 smithi162 ceph-mon[20279]: osdmap e669: 12 total, 12 up, 12 in 2024-09-02T10:24:24.928 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:24 smithi162 ceph-mon[20279]: osdmap e670: 12 total, 12 up, 12 in 2024-09-02T10:24:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:24 smithi151 ceph-mon[16639]: pgmap v1550: 47 pgs: 47 active+clean; 837 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-02T10:24:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:24 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:24:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:24 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:24:24.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4022643276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:24.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:24 smithi151 ceph-mon[16639]: osdmap e669: 12 total, 12 up, 12 in 2024-09-02T10:24:24.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:24 smithi151 ceph-mon[16639]: osdmap e670: 12 total, 12 up, 12 in 2024-09-02T10:24:24.952 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:24.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:24 smithi187 ceph-mon[20072]: pgmap v1550: 47 pgs: 47 active+clean; 837 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-02T10:24:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:24 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:24:25.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:24 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:24:25.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4022643276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:25.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:24 smithi187 ceph-mon[20072]: osdmap e669: 12 total, 12 up, 12 in 2024-09-02T10:24:25.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:24 smithi187 ceph-mon[20072]: osdmap e670: 12 total, 12 up, 12 in 2024-09-02T10:24:25.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:25.078 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:25.079 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:25.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:25.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:25.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:25.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:25.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:25.338 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:25.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:25.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:25.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:25.462 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:25.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:25.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:25.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:24:25.565 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:25.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:25.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:25.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:24:25.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:25.715 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:25.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:25.735 INFO:teuthology.orchestra.run.smithi151.stdout:moved 12 / 141 (8.51064%) 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout:avg 11.75 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 1.29904 -> 2.31391 (expected baseline 3.28189) 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.5 with 10 -> 11 pgs (0.851064 -> 0.93617 * mean) 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.0 with 14 -> 8 pgs (1.19149 -> 0.680851 * mean) 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0075 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0082 2024-09-02T10:24:25.736 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 1.0000 -> 0.6769 2024-09-02T10:24:25.737 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 1.0000 -> 0.7279 2024-09-02T10:24:25.737 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9 weight 1.0000 -> 0.7972 2024-09-02T10:24:25.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 13 16 19607 19591 94.1802 102.938 0.0247913 0.0105854 2024-09-02T10:24:25.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:25.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:25.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:25.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:25 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'47705','3':'44361','9':'52245'}"} : dispatch 2024-09-02T10:24:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:25 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'47705','3':'44361','9':'52245'}"} : dispatch 2024-09-02T10:24:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:25 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:24:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:25 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:24:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:25 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:24:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:25 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:24:25.869 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:25 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'47705','3':'44361','9':'52245'}"} : dispatch 2024-09-02T10:24:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:25 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'47705','3':'44361','9':'52245'}"} : dispatch 2024-09-02T10:24:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:25 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:24:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:25 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:24:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:25 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:24:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:25 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:24:25.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:25.961 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:25.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:25 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'47705','3':'44361','9':'52245'}"} : dispatch 2024-09-02T10:24:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:25 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd reweightn", "weights": "{'0':'47705','3':'44361','9':'52245'}"} : dispatch 2024-09-02T10:24:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:25 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:24:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:25 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:24:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:25 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:24:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:25 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:24:26.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:26.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:26.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:26.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:26.208 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:26.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:26.309 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:26.320 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:26.321 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:26.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:26.333 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:26.333 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:26.434 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:26.456 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:26.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:26.457 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:26.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:26.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:26.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:26.610 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:24:26.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:26.710 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:26.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:26.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 14 16 19617 19601 87.4975 0.625 0.0021622 0.0108051 2024-09-02T10:24:26.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:26.835 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:26.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:26.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:26.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:26 smithi151 ceph-mon[16639]: from='client.27296 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:26 smithi151 ceph-mon[16639]: pgmap v1553: 47 pgs: 47 active+clean; 958 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-02T10:24:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:26 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:24:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:26 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:24:26.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:26 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'47705','3':'44361','9':'52245'}"}]': finished 2024-09-02T10:24:26.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:26 smithi151 ceph-mon[16639]: osdmap e671: 12 total, 12 up, 12 in 2024-09-02T10:24:26.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:26.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:26 smithi187 ceph-mon[20072]: from='client.27296 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:27.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:26 smithi187 ceph-mon[20072]: pgmap v1553: 47 pgs: 47 active+clean; 958 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-02T10:24:27.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:26 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:24:27.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:26 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:24:27.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:26 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'47705','3':'44361','9':'52245'}"}]': finished 2024-09-02T10:24:27.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:26 smithi187 ceph-mon[20072]: osdmap e671: 12 total, 12 up, 12 in 2024-09-02T10:24:27.061 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:27.083 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:27.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:26 smithi162 ceph-mon[20279]: from='client.27296 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:26 smithi162 ceph-mon[20279]: pgmap v1553: 47 pgs: 47 active+clean; 958 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-02T10:24:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:26 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:24:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:26 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:24:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:26 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'47705','3':'44361','9':'52245'}"}]': finished 2024-09-02T10:24:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:26 smithi162 ceph-mon[20279]: osdmap e671: 12 total, 12 up, 12 in 2024-09-02T10:24:27.185 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:27.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:27.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:27.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:27.334 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:27.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:27.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:27.457 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:27.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:27.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:27.584 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:27.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:27.585 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:27.586 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:24:27.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:27.711 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:27.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:27.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:27.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 15 16 19617 19601 81.6643 0 - 0.0108051 2024-09-02T10:24:27.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:27.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:27.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:27.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:27 smithi162 ceph-mon[20279]: osdmap e672: 12 total, 12 up, 12 in 2024-09-02T10:24:27.865 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:27.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:27.969 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:27.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:27 smithi187 ceph-mon[20072]: osdmap e672: 12 total, 12 up, 12 in 2024-09-02T10:24:28.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:28.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:28.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:27 smithi151 ceph-mon[16639]: osdmap e672: 12 total, 12 up, 12 in 2024-09-02T10:24:28.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:28.218 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:28.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:28.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:28.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:28.343 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:28.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:28.395 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:28.396 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:28.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:28.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:28.469 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:28.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:28.571 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:28.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:28.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:28.601 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:28.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:28.738 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:28.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:28.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 16 16 20049 20033 78.2476 13.5 0.00244638 0.0127716 2024-09-02T10:24:28.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:28.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:28.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:28.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:28.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:28.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:28.995 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:28 smithi187 ceph-mon[20072]: pgmap v1556: 47 pgs: 3 remapped+peering, 44 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-02T10:24:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:28 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:24:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:28 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:24:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:28 smithi187 ceph-mon[20072]: osdmap e673: 12 total, 12 up, 12 in 2024-09-02T10:24:29.018 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:28 smithi162 ceph-mon[20279]: pgmap v1556: 47 pgs: 3 remapped+peering, 44 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-02T10:24:29.019 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:28 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:24:29.019 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:28 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:24:29.019 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:28 smithi162 ceph-mon[20279]: osdmap e673: 12 total, 12 up, 12 in 2024-09-02T10:24:29.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:24:29.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:28 smithi151 ceph-mon[16639]: pgmap v1556: 47 pgs: 3 remapped+peering, 44 active+clean; 1.1 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.05k op/s 2024-09-02T10:24:29.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:28 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:24:29.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:28 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:24:29.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:28 smithi151 ceph-mon[16639]: osdmap e673: 12 total, 12 up, 12 in 2024-09-02T10:24:29.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:29.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:29.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:29.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:29.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:29.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:29.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:29.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:29.350 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:29.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:29.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:29.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:29.514 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:29.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:29.562 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-02T10:24:29.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight-by-utilization 1000 1.0 2024-09-02T10:24:29.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:29.628 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:29.646 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:29.655 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:29.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:29.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 17 16 21035 21019 77.2695 61.625 0.032706 0.0129303 2024-09-02T10:24:29.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:29.788 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:29.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:29.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:29.905 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:29.905 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:29.917 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:29.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:29.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:30.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:30.046 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:30.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:30.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:30.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:30.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:30.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:30.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:30.299 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:30.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:30.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:30.401 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:30.434 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:30.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:30.435 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:30.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:30 smithi151 ceph-mon[16639]: osdmap e674: 12 total, 12 up, 12 in 2024-09-02T10:24:30.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:30 smithi151 ceph-mon[16639]: pgmap v1559: 47 pgs: 1 active+recovering+undersized+remapped, 3 remapped+peering, 43 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 937 op/s; 1207/57195 objects misplaced (2.110%); 480 KiB/s, 7 objects/s recovering 2024-09-02T10:24:30.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:30 smithi187 ceph-mon[20072]: osdmap e674: 12 total, 12 up, 12 in 2024-09-02T10:24:30.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:30 smithi187 ceph-mon[20072]: pgmap v1559: 47 pgs: 1 active+recovering+undersized+remapped, 3 remapped+peering, 43 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 937 op/s; 1207/57195 objects misplaced (2.110%); 480 KiB/s, 7 objects/s recovering 2024-09-02T10:24:30.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:30.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:30.574 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:30.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:30 smithi162 ceph-mon[20279]: osdmap e674: 12 total, 12 up, 12 in 2024-09-02T10:24:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:30 smithi162 ceph-mon[20279]: pgmap v1559: 47 pgs: 1 active+recovering+undersized+remapped, 3 remapped+peering, 43 active+clean; 1.2 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 937 op/s; 1207/57195 objects misplaced (2.110%); 480 KiB/s, 7 objects/s recovering 2024-09-02T10:24:30.675 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:30.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:30.708 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:30.756 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 18 16 22405 22389 77.7333 85.625 0.0106639 0.0128527 2024-09-02T10:24:30.809 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:24:30.809 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:24:30.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:30.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:30.840 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:24:30.840 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:24:30.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:30.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:30.866 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:24:30.867 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:24:30.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:24:30.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:24:30.924 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:24:30.924 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:24:30.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:30.950 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:24:30.950 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:24:30.970 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:30.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:30.982 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:24:30.982 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:24:31.013 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:24:31.013 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:24:31.041 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:24:31.042 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:24:31.063 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:24:31.064 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:24:31.072 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:31.094 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:24:31.094 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:24:31.097 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:31.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:31.126 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:24:31.127 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:24:31.199 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:31.226 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:31.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:31.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:31.354 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:31.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:31.379 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:31.458 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:31.480 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:31.481 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:31.504 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2554076284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2554076284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:31.581 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:31.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2554076284' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:31.606 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:31.607 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:31.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:31.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:31.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:31.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 19 16 23333 23317 76.6945 58 0.0263878 0.0130151 2024-09-02T10:24:31.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:31.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:31.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:31.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:32.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:32.006 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:32.107 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:32.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:32.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:32.196 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:24:32.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:32.262 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:32.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:32.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:32.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:32.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/886425385' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:24:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:32 smithi151 ceph-mon[16639]: pgmap v1560: 47 pgs: 4 active+recovering+undersized+remapped, 3 remapped+peering, 40 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 811 op/s; 4283/59817 objects misplaced (7.160%); 7.8 MiB/s, 124 objects/s recovering 2024-09-02T10:24:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:32 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:24:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:32 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:24:32.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/886425385' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:24:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:32 smithi187 ceph-mon[20072]: pgmap v1560: 47 pgs: 4 active+recovering+undersized+remapped, 3 remapped+peering, 40 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 811 op/s; 4283/59817 objects misplaced (7.160%); 7.8 MiB/s, 124 objects/s recovering 2024-09-02T10:24:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:32 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:24:32.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:32 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:24:32.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:32.526 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/886425385' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:24:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:32 smithi162 ceph-mon[20279]: pgmap v1560: 47 pgs: 4 active+recovering+undersized+remapped, 3 remapped+peering, 40 active+clean; 1.2 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 811 op/s; 4283/59817 objects misplaced (7.160%); 7.8 MiB/s, 124 objects/s recovering 2024-09-02T10:24:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:32 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:24:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:32 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:24:32.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:32.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:32.656 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:32.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:32.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:24:32.756553+0000 min lat: 0.00108091 max lat: 2.89813 avg lat: 0.0131063 2024-09-02T10:24:32.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:24:32.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 20 16 24391 24375 76.1657 66.125 0.00181513 0.0131063 2024-09-02T10:24:32.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:32.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:32.779 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:32.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:32.880 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:32.911 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:32.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:32.930 INFO:teuthology.orchestra.run.smithi151.stdout:moved 0 / 141 (0%) 2024-09-02T10:24:32.930 INFO:teuthology.orchestra.run.smithi151.stdout:avg 11.75 2024-09-02T10:24:32.930 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.31391 -> 2.31391 (expected baseline 3.28189) 2024-09-02T10:24:32.930 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.3 with 7 -> 7 pgs (0.595745 -> 0.595745 * mean) 2024-09-02T10:24:32.930 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.2 with 14 -> 14 pgs (1.19149 -> 1.19149 * mean) 2024-09-02T10:24:32.930 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:24:32.930 INFO:teuthology.orchestra.run.smithi151.stdout:oload 1000 2024-09-02T10:24:32.930 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-09-02T10:24:32.931 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:24:32.931 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0087 2024-09-02T10:24:32.931 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0866 2024-09-02T10:24:32.931 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:24:33.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:33.043 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:33.044 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:33.145 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:33.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:33.176 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:33.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:24:33.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:33.304 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:33.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:33.305 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:33.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:33.436 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:33.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:33.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:33 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/886425385' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:24:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:33 smithi187 ceph-mon[20072]: osdmap e675: 12 total, 12 up, 12 in 2024-09-02T10:24:33.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:33 smithi187 ceph-mon[20072]: from='client.17913 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:33.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:33.561 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:33.572 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:33.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:33.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:33 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/886425385' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:24:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:33 smithi162 ceph-mon[20279]: osdmap e675: 12 total, 12 up, 12 in 2024-09-02T10:24:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:33 smithi162 ceph-mon[20279]: from='client.17913 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:33.617 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:33 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:33.617 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/886425385' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:24:33.617 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:33 smithi151 ceph-mon[16639]: osdmap e675: 12 total, 12 up, 12 in 2024-09-02T10:24:33.617 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:33 smithi151 ceph-mon[16639]: from='client.17913 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:33.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:33.699 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:33.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:33.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:33.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 21 16 25361 25345 75.4254 60.625 0.0022021 0.0132298 2024-09-02T10:24:33.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:33.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:33.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:33.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:33.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:24:33.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:33.959 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:33.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:34.054 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-09-02T10:24:34.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd test-reweight-by-utilization 110 1.0 2024-09-02T10:24:34.061 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:34.085 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:34.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:34.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:34.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:34.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:34.292 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:34.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:34.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:34.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:34.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:34.470 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:34 smithi151 ceph-mon[16639]: pgmap v1562: 47 pgs: 7 active+recovering+undersized+remapped, 40 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 862 op/s; 5755/68244 objects misplaced (8.433%); 30 MiB/s, 485 objects/s recovering 2024-09-02T10:24:34.471 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:34 smithi151 ceph-mon[16639]: osdmap e676: 12 total, 12 up, 12 in 2024-09-02T10:24:34.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:34.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:34 smithi162 ceph-mon[20279]: pgmap v1562: 47 pgs: 7 active+recovering+undersized+remapped, 40 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 862 op/s; 5755/68244 objects misplaced (8.433%); 30 MiB/s, 485 objects/s recovering 2024-09-02T10:24:34.494 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:34 smithi162 ceph-mon[20279]: osdmap e676: 12 total, 12 up, 12 in 2024-09-02T10:24:34.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:34.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:34 smithi187 ceph-mon[20072]: pgmap v1562: 47 pgs: 7 active+recovering+undersized+remapped, 40 active+clean; 1.4 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 862 op/s; 5755/68244 objects misplaced (8.433%); 30 MiB/s, 485 objects/s recovering 2024-09-02T10:24:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:34 smithi187 ceph-mon[20072]: osdmap e676: 12 total, 12 up, 12 in 2024-09-02T10:24:34.603 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:34.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:34.632 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:34.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:34.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 22 16 26575 26559 75.4456 75.875 0.00351582 0.0132454 2024-09-02T10:24:34.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:34.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:34.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:34.893 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:34.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:34.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:24:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:24:34.923 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:34.926 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:24:34.927 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:24:34.927 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:24:34.927 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:24:34.953 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:24:34.953 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:24:34.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:35.027 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:35.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:35.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:35.161 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:35.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:35.162 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:35.182 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:24:35.183 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:24:35.212 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:24:35.212 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:24:35.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:24:35.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:24:35.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:24:35.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:35.275 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:24:35.275 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:24:35.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:35.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:35.303 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:24:35.303 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:24:35.335 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:24:35.335 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:24:35.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1695123508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:35.366 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:24:35.367 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:24:35.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:35.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:24:35.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:24:35.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:35.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:35.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:35.436 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:24:35.437 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:24:35.468 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:24:35.468 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:24:35.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1695123508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:35.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:35.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:35.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:35.559 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1695123508' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:35.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:24:35.647 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:35.713 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:35.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:35.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:35.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 23 15 27975 27960 75.9721 87.5625 0.00219347 0.0131546 2024-09-02T10:24:35.814 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:35.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:35.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:35.946 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:35.978 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:35.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:35.979 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:36.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:36.104 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:36.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:36.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:36.235 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:36.236 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:36.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:36.360 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:36.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:36.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:36.484 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:36.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:36 smithi187 ceph-mon[20072]: pgmap v1564: 47 pgs: 7 active+recovering+undersized+remapped, 40 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 914 op/s; 4134/74331 objects misplaced (5.562%); 45 MiB/s, 721 objects/s recovering 2024-09-02T10:24:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:36 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:24:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:36 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:24:36.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:36 smithi151 ceph-mon[16639]: pgmap v1564: 47 pgs: 7 active+recovering+undersized+remapped, 40 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 914 op/s; 4134/74331 objects misplaced (5.562%); 45 MiB/s, 721 objects/s recovering 2024-09-02T10:24:36.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:36 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:24:36.546 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:36 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:24:36.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:36 smithi162 ceph-mon[20279]: pgmap v1564: 47 pgs: 7 active+recovering+undersized+remapped, 40 active+clean; 1.5 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 914 op/s; 4134/74331 objects misplaced (5.562%); 45 MiB/s, 721 objects/s recovering 2024-09-02T10:24:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:36 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:24:36.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:36 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:24:36.607 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:36.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:36.710 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:36.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:36.734 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:36.738 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:36.739 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:36.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 24 16 28739 28723 74.7935 47.6875 0.00215255 0.0132126 2024-09-02T10:24:36.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:36.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:36.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:36.938 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 10:24:36 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y[16863]: 2024-09-02T10:24:36.544+0000 7fd7798ae640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-09-02T10:24:36.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:36.979 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:36.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:37.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:37.102 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:37.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:37.105 INFO:teuthology.orchestra.run.smithi151.stdout:moved 10 / 141 (7.0922%) 2024-09-02T10:24:37.105 INFO:teuthology.orchestra.run.smithi151.stdout:avg 11.75 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:stddev 2.31391 -> 3.56195 (expected baseline 3.28189) 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:min osd.3 with 7 -> 2 pgs (0.595745 -> 0.170213 * mean) 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:max osd.2 with 14 -> 17 pgs (1.19149 -> 1.44681 * mean) 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:oload 110 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:max_change 1 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:max_change_osds 4 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:average_utilization 0.0096 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:overload_utilization 0.0105 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3 weight 0.6769 -> 0.4946 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9 weight 0.7972 -> 0.6259 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0 weight 0.7279 -> 0.5884 2024-09-02T10:24:37.106 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11 weight 1.0000 -> 0.8710 2024-09-02T10:24:37.107 INFO:teuthology.orchestra.run.smithi151.stderr:no change 2024-09-02T10:24:37.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:37.229 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:37.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:37.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:37.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:37.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:37.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:37 smithi151 ceph-mon[16639]: osdmap e677: 12 total, 12 up, 12 in 2024-09-02T10:24:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:37 smithi151 ceph-mon[16639]: from='client.17925 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:37.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:37.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:37 smithi162 ceph-mon[20279]: osdmap e677: 12 total, 12 up, 12 in 2024-09-02T10:24:37.469 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:37 smithi162 ceph-mon[20279]: from='client.17925 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:37.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:37.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:37.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:24:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:37 smithi187 ceph-mon[20072]: osdmap e677: 12 total, 12 up, 12 in 2024-09-02T10:24:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:37 smithi187 ceph-mon[20072]: from='client.17925 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:24:37.580 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:37.603 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:37.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:37.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:37.725 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:37.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:37.726 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:37.744 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:24:37.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 25 16 28898 28882 72.1992 9.9375 0.00239351 0.0136212 2024-09-02T10:24:37.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:37.848 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:37.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:37.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:37.950 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:37.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:37.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:37.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:38.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:38.097 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:38.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:38.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:38.219 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:38.219 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:38.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:38.227 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:38.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:38.349 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:38.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:38.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:38 smithi162 ceph-mon[20279]: osdmap e678: 12 total, 12 up, 12 in 2024-09-02T10:24:38.386 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:38 smithi162 ceph-mon[20279]: pgmap v1567: 47 pgs: 6 active+recovering+undersized+remapped, 41 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 2428/80838 objects misplaced (3.004%); 66 MiB/s, 1.06k objects/s recovering 2024-09-02T10:24:38.451 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:38.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:38.478 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:38.478 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:38 smithi187 ceph-mon[20072]: osdmap e678: 12 total, 12 up, 12 in 2024-09-02T10:24:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:38 smithi187 ceph-mon[20072]: pgmap v1567: 47 pgs: 6 active+recovering+undersized+remapped, 41 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 2428/80838 objects misplaced (3.004%); 66 MiB/s, 1.06k objects/s recovering 2024-09-02T10:24:38.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:38.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:38.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:38.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:38 smithi151 ceph-mon[16639]: osdmap e678: 12 total, 12 up, 12 in 2024-09-02T10:24:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:38 smithi151 ceph-mon[16639]: pgmap v1567: 47 pgs: 6 active+recovering+undersized+remapped, 41 active+clean; 1.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 84 MiB/s wr, 1.34k op/s; 2428/80838 objects misplaced (3.004%); 66 MiB/s, 1.06k objects/s recovering 2024-09-02T10:24:38.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:38.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:38.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:38.724 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:38.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 26 16 28983 28967 69.6266 5.3125 0.415776 0.0141142 2024-09-02T10:24:38.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:38.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:24:38.890 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:38.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:38.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:38.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:39.016 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:39.018 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:39.083 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:39.119 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:39.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:39.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:39.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:39.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:39.268 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:39.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:39.370 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:39.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:39.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:39.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:39 smithi151 ceph-mon[16639]: osdmap e679: 12 total, 12 up, 12 in 2024-09-02T10:24:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:24:39.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:39 smithi151 ceph-mon[16639]: osdmap e680: 12 total, 12 up, 12 in 2024-09-02T10:24:39.497 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:39 smithi187 ceph-mon[20072]: osdmap e679: 12 total, 12 up, 12 in 2024-09-02T10:24:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:24:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:39 smithi187 ceph-mon[20072]: osdmap e680: 12 total, 12 up, 12 in 2024-09-02T10:24:39.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:39.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:39 smithi162 ceph-mon[20279]: osdmap e679: 12 total, 12 up, 12 in 2024-09-02T10:24:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:24:39.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:39 smithi162 ceph-mon[20279]: osdmap e680: 12 total, 12 up, 12 in 2024-09-02T10:24:39.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:39.639 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:39.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:39.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:39.645 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:39.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:39.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 27 16 29142 29126 67.4159 9.9375 0.0101881 0.0145911 2024-09-02T10:24:39.762 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:39.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:39.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:39.886 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:39.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:39.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:40.009 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:40.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:40.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:40.111 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:40.134 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:40.134 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:40.235 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:40.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:40.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:40.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:40.409 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:40.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:40.410 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:40.500 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:40 smithi187 ceph-mon[20072]: pgmap v1570: 47 pgs: 1 active+undersized+remapped, 5 active+recovering+undersized+remapped, 41 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 905 op/s; 1858/85200 objects misplaced (2.181%); 36 MiB/s, 568 objects/s recovering 2024-09-02T10:24:40.500 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:24:40.500 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:24:40.500 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:40 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:24:40.500 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:40 smithi187 ceph-mon[20072]: osdmap e681: 12 total, 12 up, 12 in 2024-09-02T10:24:40.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:40.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:40.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:40 smithi162 ceph-mon[20279]: pgmap v1570: 47 pgs: 1 active+undersized+remapped, 5 active+recovering+undersized+remapped, 41 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 905 op/s; 1858/85200 objects misplaced (2.181%); 36 MiB/s, 568 objects/s recovering 2024-09-02T10:24:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:24:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:24:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:40 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:24:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:40 smithi162 ceph-mon[20279]: osdmap e681: 12 total, 12 up, 12 in 2024-09-02T10:24:40.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:40.671 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:40.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:40.679 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:40.679 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:40 smithi151 ceph-mon[16639]: pgmap v1570: 47 pgs: 1 active+undersized+remapped, 5 active+recovering+undersized+remapped, 41 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 905 op/s; 1858/85200 objects misplaced (2.181%); 36 MiB/s, 568 objects/s recovering 2024-09-02T10:24:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:24:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:24:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:40 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:24:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:40 smithi151 ceph-mon[16639]: osdmap e681: 12 total, 12 up, 12 in 2024-09-02T10:24:40.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 28 16 30099 30083 67.1441 59.8125 0.00988592 0.0148863 2024-09-02T10:24:40.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:40.806 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:40.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:40.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:40.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:40.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:40.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:40.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:41.032 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:41.054 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:41.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:41.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:41.180 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:41.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:41.241 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:24:41.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:41.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:41.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:41.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:41.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:41.428 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:41.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:41.494 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3203105870' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:41.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:24:41.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:41.550 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:41.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3203105870' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:41.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:24:41.632 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3203105870' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:41.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:41.675 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:41.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:41.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:41.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 29 16 30798 30782 66.3352 43.6875 0.00180371 0.0149551 2024-09-02T10:24:41.760 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:41.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:41.798 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:41.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:41.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:41.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:41.923 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:42.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:42.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:42.046 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:42.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:42.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:42.169 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:42.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:42.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:42.200 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:42.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:42.294 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:42.294 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:42.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:42.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:42.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:42 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:42.321 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3203105870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:42.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:42 smithi151 ceph-mon[16639]: osdmap e682: 12 total, 12 up, 12 in 2024-09-02T10:24:42.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:42 smithi151 ceph-mon[16639]: pgmap v1573: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 497 op/s; 403/86805 objects misplaced (0.464%); 32 MiB/s, 506 objects/s recovering 2024-09-02T10:24:42.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:42.417 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:42.417 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:42 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3203105870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:42 smithi187 ceph-mon[20072]: osdmap e682: 12 total, 12 up, 12 in 2024-09-02T10:24:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:42 smithi187 ceph-mon[20072]: pgmap v1573: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 497 op/s; 403/86805 objects misplaced (0.464%); 32 MiB/s, 506 objects/s recovering 2024-09-02T10:24:42.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:42.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:42.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:42 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3203105870' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:42 smithi162 ceph-mon[20279]: osdmap e682: 12 total, 12 up, 12 in 2024-09-02T10:24:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:42 smithi162 ceph-mon[20279]: pgmap v1573: 47 pgs: 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 44 active+clean; 1.8 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s wr, 497 op/s; 403/86805 objects misplaced (0.464%); 32 MiB/s, 506 objects/s recovering 2024-09-02T10:24:42.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:42.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:42.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:42.731 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:24:42.731 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:24:42.757 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:24:42.757 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:24:42.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 30 16 31284 31268 65.1364 30.375 0.00272769 0.0153449 2024-09-02T10:24:42.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:42.786 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:24:42.787 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:24:42.798 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:42.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:42.812 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:24:42.813 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:24:42.842 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:24:42.842 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:24:42.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:24:42.870 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:24:42.894 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:24:42.894 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:24:42.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:42.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:24:42.920 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:24:42.925 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:42.926 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:42.944 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:24:42.944 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:24:42.972 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:24:42.973 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:24:42.997 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:24:42.997 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:24:43.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:24:43.024 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:24:43.024 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:24:43.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:43.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:43.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:43.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:43.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:43.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:43.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:43.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:43.309 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:43.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:43.310 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:43.326 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:43.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:43.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:43.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:43.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:43 smithi187 ceph-mon[20072]: osdmap e683: 12 total, 12 up, 12 in 2024-09-02T10:24:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4271459659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:43.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:43.562 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:43.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:43 smithi162 ceph-mon[20279]: osdmap e683: 12 total, 12 up, 12 in 2024-09-02T10:24:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4271459659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:43.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:43.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:43.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:43.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:43 smithi151 ceph-mon[16639]: osdmap e683: 12 total, 12 up, 12 in 2024-09-02T10:24:43.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4271459659' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:43.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 31 16 32548 32532 65.5834 79 0.00563966 0.0152369 2024-09-02T10:24:43.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:43.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:43.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:43.886 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:24:43.886 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:24:43.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:24:43.886 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63248 5.a 5:5adbc538:::benchmark_data_smithi187_89405_object63247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 547.36889623299999, 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21653889600000001, 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63248 2024-09-02T10:24:43.887 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514160+0000", 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.888 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514161+0000", 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.099e-06 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514181+0000", 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.999e-05 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514212+0000", 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0648999999999997e-05 2024-09-02T10:24:43.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514371+0000", 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015914799999999999 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514421+0000", 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9804e-05 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514561+0000", 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000140285 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.514666+0000", 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104963 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.516942+0000", 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022756780000000002 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521266+0000", 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043245380000000002 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730584+0000", 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20931819099999999 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730647+0000", 2024-09-02T10:24:43.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3040999999999995e-05 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730699+0000", 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.151e-05 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63255 5.a 5:57c5feec:::benchmark_data_smithi187_89405_object63254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:24:43.893 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 547.36508633300002, 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21287715600000001, 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63255 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:24:43.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517970+0000", 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517972+0000", 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.832e-06 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.517998+0000", 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6004e-05 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518024+0000", 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6290000000000001e-05 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518039+0000", 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.484e-05 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518103+0000", 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4350000000000006e-05 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518230+0000", 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012653999999999999 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.518365+0000", 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013489900000000001 2024-09-02T10:24:43.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.520755+0000", 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023907780000000001 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.523888+0000", 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0031328179999999999 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730745+0000", 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.206856442 2024-09-02T10:24:43.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730800+0000", 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5055000000000001e-05 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.730847+0000", 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7308000000000002e-05 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.899 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:63260 5.a 5:581436a1:::benchmark_data_smithi187_89405_object63259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 547.36178106099999, 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21179099900000001, 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.900 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.906 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:24:43.906 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:24:43.906 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 63260 2024-09-02T10:24:43.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.906 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:24:43.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521275+0000", 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521277+0000", 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.782e-06 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521301+0000", 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3893e-05 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521320+0000", 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8879999999999999e-05 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521327+0000", 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6890000000000004e-06 2024-09-02T10:24:43.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521399+0000", 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1439000000000005e-05 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521552+0000", 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000153268 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.521685+0000", 2024-09-02T10:24:43.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013246699999999999 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.524092+0000", 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024077679999999998 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.525479+0000", 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001386916 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.910 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.732968+0000", 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20748902399999999 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.733024+0000", 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5269999999999998e-05 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:36.733066+0000", 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2602999999999999e-05 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.911 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:86595 5.a 5:52430871:::benchmark_data_smithi187_89405_object86594:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e473)", 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 525.37697176699999, 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.225552366, 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:24:43.912 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 86595 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506084+0000", 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506087+0000", 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2469999999999998e-06 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506114+0000", 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7141999999999999e-05 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506133+0000", 2024-09-02T10:24:43.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9137e-05 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506142+0000", 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8859999999999993e-06 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506210+0000", 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7992999999999998e-05 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506361+0000", 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015085899999999999 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.506506+0000", 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014534300000000001 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.509602+0000", 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030956009999999999 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.597950+0000", 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.088348135999999994 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731542+0000", 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13359195700000001 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731597+0000", 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5368999999999998e-05 2024-09-02T10:24:43.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:58.731637+0000", 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9696000000000001e-05 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:122098 5.e e488/431 5:783becf1:::benchmark_data_smithi187_89405_object18493:head v 488'7571, mlcod=488'7567)", 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 487.311376645, 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21171738700000001, 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.918 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571680+0000", 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1999999999999999e-07 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.571686+0000", 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6010000000000002e-06 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780330+0000", 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20864460000000001 2024-09-02T10:24:43.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780339+0000", 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1869999999999997e-06 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780399+0000", 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9941000000000001e-05 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.780424+0000", 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5751000000000001e-05 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.783348+0000", 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029234819999999998 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:36.783397+0000", 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9104999999999998e-05 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.922 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:126123 5.e e490/431 5:7800327a:::benchmark_data_smithi187_89405_object22518:head v 490'7825, mlcod=490'7821)", 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 482.343031365, 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25454940500000001, 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540025+0000", 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540026+0000", 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1000000000000001e-06 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.540033+0000", 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8009999999999998e-06 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791598+0000", 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25156501399999998 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791605+0000", 2024-09-02T10:24:43.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9569999999999996e-06 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791659+0000", 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4088000000000001e-05 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.791680+0000", 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1450999999999998e-05 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.794543+0000", 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028628120000000002 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.794574+0000", 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1182000000000001e-05 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:194962 5.a 5:5b6b76d9:::benchmark_data_smithi187_89405_object91357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-02T10:24:43.927 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 400.924351495, 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22579173, 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194962 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958706+0000", 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.269e-06 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958711+0000", 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7010000000000004e-06 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958723+0000", 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2641e-05 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958731+0000", 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9980000000000003e-06 2024-09-02T10:24:43.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958775+0000", 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3399999999999998e-05 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958862+0000", 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7163999999999994e-05 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:24:43.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958935+0000", 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2804000000000003e-05 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.962637+0000", 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037025320000000001 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.963124+0000", 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048696900000000002 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184411+0000", 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.221286813 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184450+0000", 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9230999999999998e-05 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184496+0000", 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6208000000000002e-05 2024-09-02T10:24:43.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:194969 5.a 5:503ca7c3:::benchmark_data_smithi187_89405_object91364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 400.91345435599999, 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21939375699999999, 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:24:43.934 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194969 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.935 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969603+0000", 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.068e-06 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969607+0000", 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3850000000000004e-06 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969617+0000", 2024-09-02T10:24:43.936 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7750000000000004e-06 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969622+0000", 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2449999999999998e-06 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969678+0000", 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5784e-05 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.937 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969792+0000", 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113993 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969919+0000", 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012692700000000001 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.975053+0000", 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051336569999999998 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.938 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.975381+0000", 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00032868799999999999 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188892+0000", 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21351113099999999 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188951+0000", 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8777000000000003e-05 2024-09-02T10:24:43.939 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188996+0000", 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4326999999999998e-05 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207198 5.a 5:57bb5924:::benchmark_data_smithi187_89405_object103593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:24:43.940 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 383.76678235999998, 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44270453799999998, 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207198 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.941 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116275+0000", 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.393e-06 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.942 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116281+0000", 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.694e-06 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116296+0000", 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5234e-05 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116305+0000", 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5650000000000002e-06 2024-09-02T10:24:43.943 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116360+0000", 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5164999999999998e-05 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116445+0000", 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5158e-05 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116517+0000", 2024-09-02T10:24:43.944 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2376e-05 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.121133+0000", 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046159479999999999 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.121221+0000", 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7831999999999995e-05 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.945 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558874+0000", 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43765253900000001 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558932+0000", 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7908000000000002e-05 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558978+0000", 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6726e-05 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.946 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:207204 5.4 e525/431 5:2473bd20:::benchmark_data_smithi187_89405_object103599:head v 525'13184, mlcod=525'13181)", 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 383.75562740300001, 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42712563199999998, 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.947 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127430+0000", 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.251e-06 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.948 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127438+0000", 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9039999999999992e-06 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550537+0000", 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42309940200000001 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550550+0000", 2024-09-02T10:24:43.949 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2262999999999999e-05 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550695+0000", 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000145845 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550728+0000", 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2923000000000003e-05 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.950 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554509+0000", 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037808550000000001 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554554+0000", 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5188999999999997e-05 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:19698 6.1 6:8f6a7646:::benchmark_data_smithi187_102448_object19697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:24:43.951 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:53.693434+0000", 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 350.189622402, 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22523997900000001, 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 19698 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.952 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693434+0000", 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693434+0000", 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693436+0000", 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7400000000000001e-06 2024-09-02T10:24:43.953 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693457+0000", 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1824999999999999e-05 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693471+0000", 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3200999999999999e-05 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693478+0000", 2024-09-02T10:24:43.954 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5820000000000004e-06 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693538+0000", 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9824999999999997e-05 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693698+0000", 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015993299999999999 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.955 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693861+0000", 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016271300000000001 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.694864+0000", 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010034600000000001 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.695808+0000", 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00094396200000000001 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.956 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.918543+0000", 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22273512600000001 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.918616+0000", 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2878000000000003e-05 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.918674+0000", 2024-09-02T10:24:43.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7734e-05 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:19714 6.1 6:8a5fdd45:::benchmark_data_smithi187_102448_object19713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:53.701207+0000", 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 350.18184872, 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21946518000000001, 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.958 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 19714 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701207+0000", 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701207+0000", 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701209+0000", 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2839999999999999e-06 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701234+0000", 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5704999999999999e-05 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701250+0000", 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5739999999999998e-05 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701259+0000", 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3310000000000004e-06 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701320+0000", 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1755999999999996e-05 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701450+0000", 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000129794 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701592+0000", 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014243400000000001 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.703694+0000", 2024-09-02T10:24:43.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002101416 2024-09-02T10:24:43.963 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:43.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.703718+0000", 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3785999999999999e-05 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.920572+0000", 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21685423000000001 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.920630+0000", 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8134000000000003e-05 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.920673+0000", 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2570000000000001e-05 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:19717 6.1 6:8971b800:::benchmark_data_smithi187_102448_object19716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:24:43.966 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:53.702495+0000", 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 350.180560684, 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.219151597, 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 19717 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702495+0000", 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702495+0000", 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702497+0000", 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.567e-06 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702518+0000", 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0837999999999999e-05 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702531+0000", 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3467e-05 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702539+0000", 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7419999999999997e-06 2024-09-02T10:24:43.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702609+0000", 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9890999999999999e-05 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702775+0000", 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000165575 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:24:43.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702939+0000", 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016468799999999999 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.703801+0000", 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086205600000000004 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.704004+0000", 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000203177 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.971 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.921562+0000", 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21755740500000001 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.921611+0000", 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9421e-05 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.921647+0000", 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5769999999999998e-05 2024-09-02T10:24:43.972 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:100226 6.1 6:80d0f12d:::benchmark_data_smithi187_102448_object100225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:20:10.905652+0000", 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 272.97740372200002, 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63928361700000003, 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:24:43.973 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 100226 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905652+0000", 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905652+0000", 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905654+0000", 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.593e-06 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905727+0000", 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3153999999999998e-05 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905744+0000", 2024-09-02T10:24:43.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6371e-05 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905753+0000", 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2299999999999997e-06 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905816+0000", 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3464000000000001e-05 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.976 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905925+0000", 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010829599999999999 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906047+0000", 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012230699999999999 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.908790+0000", 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027432860000000002 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.977 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.909351+0000", 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00056067900000000002 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544856+0000", 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63550532299999996 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544894+0000", 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8143999999999999e-05 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.978 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544936+0000", 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1770000000000002e-05 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:137822 6.1 6:8813a8dd:::benchmark_data_smithi187_102448_object37550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:20:56.005408+0000", 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 227.877648241, 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25302631599999997, 2024-09-02T10:24:43.979 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 137822 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005408+0000", 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.980 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005408+0000", 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005409+0000", 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4890000000000001e-06 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005416+0000", 2024-09-02T10:24:43.981 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2380000000000002e-06 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005430+0000", 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4027e-05 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005436+0000", 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2750000000000001e-06 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.982 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005482+0000", 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6344999999999998e-05 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005574+0000", 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1984000000000006e-05 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005649+0000", 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4694e-05 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.983 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.009865+0000", 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004215964 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.010096+0000", 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00023086800000000001 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.258316+0000", 2024-09-02T10:24:43.984 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.248219841 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.258372+0000", 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6681000000000002e-05 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.258434+0000", 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1909999999999995e-05 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.985 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:191080 6.1 6:83094aee:::benchmark_data_smithi187_102448_object90808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:24.184440+0000", 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 139.69861578999999, 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.210722517, 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 191080 2024-09-02T10:24:43.986 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184440+0000", 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184440+0000", 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.987 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184442+0000", 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3289999999999999e-06 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184447+0000", 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.356e-06 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184460+0000", 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3213e-05 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.988 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184466+0000", 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9320000000000001e-06 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184523+0000", 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6706999999999998e-05 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184654+0000", 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000131211 2024-09-02T10:24:43.989 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184794+0000", 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013967999999999999 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.188749+0000", 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039552550000000004 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.189060+0000", 2024-09-02T10:24:43.990 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031115299999999999 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.395061+0000", 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20600111700000001 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.395116+0000", 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4283999999999997e-05 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.991 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.395163+0000", 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7280000000000001e-05 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:23891 7.f 7:fd726100:::benchmark_data_smithi187_120712_object23890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:32.086879+0000", 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 11.796177173, 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.219302261, 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.992 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 23891 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086879+0000", 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.993 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086879+0000", 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086881+0000", 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.069e-06 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086915+0000", 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3998000000000002e-05 2024-09-02T10:24:43.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086944+0000", 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9300000000000001e-05 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086955+0000", 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0796e-05 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:43.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.087024+0000", 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8351999999999999e-05 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.087186+0000", 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016218 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.087364+0000", 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017785500000000001 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.089462+0000", 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020980199999999999 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.097470+0000", 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0080084630000000004 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.305360+0000", 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20789026099999999 2024-09-02T10:24:43.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.305438+0000", 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8193000000000002e-05 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.306181+0000", 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00074277399999999997 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:23904 7.f 7:f7b9f367:::benchmark_data_smithi187_120712_object23903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:32.095666+0000", 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 11.787390644, 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21120496499999999, 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 23904 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:43.999 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095666+0000", 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095666+0000", 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:44.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095668+0000", 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1239999999999999e-06 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095692+0000", 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4825e-05 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095713+0000", 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0588e-05 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095721+0000", 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2689999999999999e-06 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095786+0000", 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4481999999999997e-05 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.096004+0000", 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00021850400000000001 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.096181+0000", 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000176305 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.097203+0000", 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010226409999999999 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.101678+0000", 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044750529999999997 2024-09-02T10:24:44.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.306240+0000", 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.204561403 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.306305+0000", 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5547000000000001e-05 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.306870+0000", 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00056522400000000002 2024-09-02T10:24:44.004 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:23909 7.f 7:f5ee81a7:::benchmark_data_smithi187_120712_object23908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:32.099943+0000", 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 11.783113072999999, 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20944562999999999, 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:44.005 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 23909 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.099943+0000", 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.099943+0000", 2024-09-02T10:24:44.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.099945+0000", 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2170000000000001e-06 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.099973+0000", 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7277e-05 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:44.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.099993+0000", 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0720000000000002e-05 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.100001+0000", 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0209999999999999e-06 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.100075+0000", 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3533000000000002e-05 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.100241+0000", 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016628199999999999 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.100369+0000", 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012790100000000001 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.101352+0000", 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00098319800000000001 2024-09-02T10:24:44.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.114398+0000", 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013045929 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.308648+0000", 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19424997299999999 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.308707+0000", 2024-09-02T10:24:44.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8406000000000003e-05 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.309389+0000", 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000682173 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:23920 7.f 7:f5f3e507:::benchmark_data_smithi187_120712_object23919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:32.112637+0000", 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 11.770419029999999, 2024-09-02T10:24:44.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19743918899999999, 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 23920 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.112637+0000", 2024-09-02T10:24:44.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.112637+0000", 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.112639+0000", 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8610000000000001e-06 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:44.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.112660+0000", 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0596000000000001e-05 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.112677+0000", 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7671000000000001e-05 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.112686+0000", 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5199999999999997e-06 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.112753+0000", 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7251999999999999e-05 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.112981+0000", 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00022833099999999999 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.113094+0000", 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112934 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.114428+0000", 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013332470000000001 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.285742+0000", 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.17131441 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.309459+0000", 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.023717477000000001 2024-09-02T10:24:44.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.309538+0000", 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8937999999999995e-05 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.310076+0000", 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053795199999999996 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:44.017 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:44.018 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:24:44.045 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:44.046 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:44.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:24:44.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:44.173 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:44.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:44.174 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:44.195 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:24:44.195 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:24:44.195 INFO:tasks.thrashosds.thrasher:inject_pause on osd.9 2024-09-02T10:24:44.195 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-02T10:24:44.195 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-02T10:24:44.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 config set filestore_inject_stall 3 2024-09-02T10:24:44.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:44.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:44.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:44.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:44.398 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:44.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:44.428 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:44.429 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:44 smithi151 ceph-mon[16639]: pgmap v1575: 47 pgs: 1 activating, 46 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 437 op/s; 27 MiB/s, 437 objects/s recovering 2024-09-02T10:24:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:44 smithi151 ceph-mon[16639]: osdmap e684: 12 total, 12 up, 12 in 2024-09-02T10:24:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:44 smithi187 ceph-mon[20072]: pgmap v1575: 47 pgs: 1 activating, 46 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 437 op/s; 27 MiB/s, 437 objects/s recovering 2024-09-02T10:24:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:44 smithi187 ceph-mon[20072]: osdmap e684: 12 total, 12 up, 12 in 2024-09-02T10:24:44.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:44.553 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:44.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:44.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:44 smithi162 ceph-mon[20279]: pgmap v1575: 47 pgs: 1 activating, 46 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 27 MiB/s wr, 437 op/s; 27 MiB/s, 437 objects/s recovering 2024-09-02T10:24:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:44 smithi162 ceph-mon[20279]: osdmap e684: 12 total, 12 up, 12 in 2024-09-02T10:24:44.655 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:44.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:44.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:44.757 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 32 16 33803 33787 65.9849 78.4375 0.00174193 0.0151318 2024-09-02T10:24:44.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:44.808 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:44.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:44.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:44.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:44.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:45.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:45.064 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:45.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:45.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:24:45.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:45.169 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:45.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:45.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:45.257 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:24:45.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:45.297 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3256320979' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:24:45.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:45.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:45.426 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:45.450 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:45.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:45.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:45.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:45.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:45.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3256320979' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:24:45.637 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3256320979' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:24:45.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:45.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:45.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:45.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:45.758 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 33 15 35156 35141 66.5495 84.625 0.00686934 0.0150207 2024-09-02T10:24:45.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:45.836 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:45.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:45.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:24:45.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:45.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:45.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:46.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:46.106 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:46.106 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:46.207 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:46.216 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:46.232 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:46.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:46.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:46.354 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:24:46.354 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "" 2024-09-02T10:24:46.354 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:24:46.356 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:46.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:46.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:46.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:46 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:24:46.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:46 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:24:46.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3256320979' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:24:46.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:46 smithi151 ceph-mon[16639]: osdmap e685: 12 total, 12 up, 12 in 2024-09-02T10:24:46.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:46 smithi151 ceph-mon[16639]: pgmap v1578: 47 pgs: 1 activating, 46 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 663 op/s; 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:24:46.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:46.483 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:46.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:46 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:24:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:46 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:24:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3256320979' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:24:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:46 smithi187 ceph-mon[20072]: osdmap e685: 12 total, 12 up, 12 in 2024-09-02T10:24:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:46 smithi187 ceph-mon[20072]: pgmap v1578: 47 pgs: 1 activating, 46 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 663 op/s; 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:24:46.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:46 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:24:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:46 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:24:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3256320979' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:24:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:46 smithi162 ceph-mon[20279]: osdmap e685: 12 total, 12 up, 12 in 2024-09-02T10:24:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:46 smithi162 ceph-mon[20279]: pgmap v1578: 47 pgs: 1 activating, 46 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 663 op/s; 6.0 MiB/s, 96 objects/s recovering 2024-09-02T10:24:46.605 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:46.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:46.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:46.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:46.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:46.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:46.758 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 34 16 36600 36584 67.2445 90.1875 0.0852576 0.0148255 2024-09-02T10:24:46.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:24:46.796 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:24:46.817 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:24:46.818 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:24:46.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:46.844 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:24:46.844 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:24:46.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:46.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:46.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:46.876 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:24:46.876 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:24:46.902 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:24:46.902 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:24:46.931 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:24:46.932 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:24:46.957 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:24:46.958 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:24:46.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:46.986 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:24:46.986 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:24:47.004 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:47.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:47.006 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:47.011 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:24:47.011 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:24:47.038 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:24:47.039 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:24:47.068 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:24:47.068 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:24:47.097 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:24:47.097 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:24:47.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:47.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:47.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:47.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:47.265 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:47.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:47.318 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:47 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:24:47.318 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:47 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:24:47.319 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2016991216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:47.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:47.389 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:47.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:47.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:47.491 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:47.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:47.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:47 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:24:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:47 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:24:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2016991216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:47.596 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:47.597 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:47.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:47.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:47.652 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:47 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:24:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:47 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:24:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2016991216' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:47.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:47.758 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 35 16 38160 38144 68.1087 97.5 0.0149566 0.0146726 2024-09-02T10:24:47.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:47.778 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:47.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:47.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:47.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:47.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:48.003 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:48.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:48.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:48.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:48.149 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:48.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:48.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:48.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:24:48.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:48.278 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:48.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:48.315 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:24:48.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:48.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:48.409 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:48 smithi151 ceph-mon[16639]: pgmap v1579: 47 pgs: 47 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s; 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:24:48.465 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:48.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:48.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:48.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:48 smithi162 ceph-mon[20279]: pgmap v1579: 47 pgs: 47 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s; 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:24:48.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:48.644 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:48.658 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:48.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:48.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:48.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:48 smithi187 ceph-mon[20072]: pgmap v1579: 47 pgs: 47 active+clean; 2.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 958 op/s; 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:24:48.758 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 36 16 39325 39309 68.2392 72.8125 0.0137023 0.0146458 2024-09-02T10:24:48.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:48.784 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:48.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:48.885 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:48.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:48.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:48.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:49.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:49.042 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:49.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:49.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:49.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:49.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:49.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:49.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:49.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:49.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:49.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:49.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:49.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:49.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:49.555 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:49.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:49 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:24:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:49 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:24:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:49 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:49 smithi162 ceph-mon[20279]: osdmap e686: 12 total, 12 up, 12 in 2024-09-02T10:24:49.605 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:49 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:24:49.605 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:49 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:24:49.605 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:49 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:49.605 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:49 smithi187 ceph-mon[20072]: osdmap e686: 12 total, 12 up, 12 in 2024-09-02T10:24:49.658 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:49.680 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:49.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:49.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:49 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:24:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:49 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:24:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:49 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:49.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:49 smithi151 ceph-mon[16639]: osdmap e686: 12 total, 12 up, 12 in 2024-09-02T10:24:49.749 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:49.749 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:49.758 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 37 16 40778 40762 68.8491 90.8125 0.0025979 0.0145166 2024-09-02T10:24:49.783 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:49.811 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:49.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:49.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:49.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:24:49.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:49.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:49.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:50.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:50.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:50.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:50.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:50.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:50.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:50.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:50.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:50.234 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:50.280 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:24:50.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:50.475 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:50.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:50.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:50.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:24:50.526 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/435723823' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:50.526 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:50 smithi151 ceph-mon[16639]: pgmap v1581: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-02T10:24:50.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/435723823' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:50 smithi162 ceph-mon[20279]: pgmap v1581: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-02T10:24:50.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:50.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:50.710 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:50.732 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:50.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:50.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/435723823' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:24:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:50 smithi187 ceph-mon[20072]: pgmap v1581: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 72 MiB/s wr, 1.16k op/s 2024-09-02T10:24:50.758 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 38 16 42018 42002 69.0766 77.5 0.00487129 0.0144705 2024-09-02T10:24:50.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:50.835 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:50.858 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:50.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:50.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:50.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:50.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:50.998 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:51.020 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:24:51.020 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:24:51.049 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:24:51.049 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:24:51.080 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:24:51.080 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:24:51.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:51.107 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:24:51.107 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:24:51.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:51.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:51.135 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:24:51.135 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:24:51.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:24:51.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:24:51.179 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:24:51.179 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:24:51.210 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:24:51.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:24:51.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:51.241 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:24:51.241 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:24:51.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:51.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:51.273 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:24:51.273 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:24:51.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:24:51.303 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:24:51.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:24:51.330 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:24:51.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:24:51.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:51.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:24:51.405 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:24:51.405 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:24:51.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:24:51.405 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:24:51.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.405 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:50217 5.1 e455/431 5:84aaa397:::benchmark_data_smithi187_89405_object50216:head v 455'3114, mlcod=455'3111)", 2024-09-02T10:24:51.405 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:15:23.792228+0000", 2024-09-02T10:24:51.405 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 567.60921018600004, 2024-09-02T10:24:51.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.382032398, 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:23.792228+0000", 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:23.792228+0000", 2024-09-02T10:24:51.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:23.792230+0000", 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.393e-06 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:23.792299+0000", 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8938999999999998e-05 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.172069+0000", 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.37976990700000002 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.172075+0000", 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1090000000000003e-06 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.172418+0000", 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00034368799999999998 2024-09-02T10:24:51.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.172439+0000", 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0704e-05 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.174214+0000", 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00177505 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:15:24.174261+0000", 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6607999999999998e-05 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:126036 5.e 5:76dd882f:::benchmark_data_smithi187_89405_object22431:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e490)", 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:16:41.216999+0000", 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 490.18443964599999, 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32619644800000003, 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.410 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 126036 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.216999+0000", 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.216999+0000", 2024-09-02T10:24:51.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.217000+0000", 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1299999999999998e-07 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.217004+0000", 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3189999999999996e-06 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.217017+0000", 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3086e-05 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.217024+0000", 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8889999999999996e-06 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.218835+0000", 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0018111360000000001 2024-09-02T10:24:51.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.218937+0000", 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101393 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,8", 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.219003+0000", 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6853999999999994e-05 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.221568+0000", 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002564481 2024-09-02T10:24:51.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.221904+0000", 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00033607100000000002 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.543122+0000", 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32121784399999997 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.543155+0000", 2024-09-02T10:24:51.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3068999999999998e-05 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:16:41.543195+0000", 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0392999999999998e-05 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.416 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:205325 5.e 5:726c3a45:::benchmark_data_smithi187_89405_object101720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:17.617930+0000", 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 393.78350815599998, 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.38179982800000001, 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 205325 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.417 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617930+0000", 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617930+0000", 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617931+0000", 2024-09-02T10:24:51.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1999999999999998e-07 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617936+0000", 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3499999999999999e-06 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617946+0000", 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0223e-05 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617951+0000", 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3159999999999996e-06 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617992+0000", 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0435000000000002e-05 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.618066+0000", 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4764000000000004e-05 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,8", 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.618133+0000", 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6779999999999994e-05 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.622877+0000", 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047439980000000001 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.623239+0000", 2024-09-02T10:24:51.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036236100000000001 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.999558+0000", 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.37631872100000002 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.999642+0000", 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4165000000000005e-05 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.999730+0000", 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7794999999999995e-05 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:100226 6.1 e572/533 6:80d0f12d:::benchmark_data_smithi187_102448_object100225:head v 572'6190, mlcod=572'6187)", 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 280.49531685900001, 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63855390000000001, 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.423 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906123+0000", 2024-09-02T10:24:51.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.269e-06 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906147+0000", 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3620000000000001e-05 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.541864+0000", 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.635717054 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.541879+0000", 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5546e-05 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.542040+0000", 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000161022 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.542059+0000", 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8697000000000001e-05 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544638+0000", 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025792850000000002 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544676+0000", 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7407000000000003e-05 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.427 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:157058 6.5 e597/533 6:a63ec91b:::benchmark_data_smithi187_102448_object56786:head v 597'9839, mlcod=597'9836)", 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:21:27.330318+0000", 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 204.07112075399999, 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19032256, 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.330318+0000", 2024-09-02T10:24:51.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.330318+0000", 2024-09-02T10:24:51.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.330318+0000", 2024-09-02T10:24:51.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1999999999999999e-07 2024-09-02T10:24:51.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.330324+0000", 2024-09-02T10:24:51.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1030000000000001e-06 2024-09-02T10:24:51.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.517156+0000", 2024-09-02T10:24:51.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18683244900000001 2024-09-02T10:24:51.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.517162+0000", 2024-09-02T10:24:51.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.297e-06 2024-09-02T10:24:51.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.517232+0000", 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9531999999999998e-05 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.517246+0000", 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3764000000000001e-05 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.520611+0000", 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033651459999999998 2024-09-02T10:24:51.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.520640+0000", 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9549000000000001e-05 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:157286 6.5 e598/533 6:aeee93d8:::benchmark_data_smithi187_102448_object57014:head v 598'9853, mlcod=598'9850)", 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 203.45142395299999, 2024-09-02T10:24:51.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29959694599999998, 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:24:51.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:51 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:24:51.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:51 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:24:51.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:51 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:24:51.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:51 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:24:51.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/435723823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:51.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:51 smithi151 ceph-mon[16639]: osdmap e687: 12 total, 12 up, 12 in 2024-09-02T10:24:51.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2714840412' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:51.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2900000000000002e-07 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950022+0000", 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8739999999999997e-06 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245512+0000", 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29548945300000001 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245524+0000", 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2347000000000001e-05 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245666+0000", 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014145800000000001 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245702+0000", 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6853000000000001e-05 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.249576+0000", 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003873385 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.249611+0000", 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5646999999999999e-05 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.439 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:174779 6.1 e609/533 6:81b1e1cd:::benchmark_data_smithi187_102448_object74507:head v 609'10785, mlcod=609'10782)", 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:21:56.468798+0000", 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 174.932640623, 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14462064999999999, 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.468798+0000", 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.468798+0000", 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.468799+0000", 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4900000000000005e-07 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.468804+0000", 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.818e-06 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.608450+0000", 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.139645936 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.608455+0000", 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6639999999999997e-06 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.608527+0000", 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1967999999999999e-05 2024-09-02T10:24:51.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.608546+0000", 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8525e-05 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.613381+0000", 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0048350789999999999 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:56.613419+0000", 2024-09-02T10:24:51.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7910999999999997e-05 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:183270 6.1 e619/533 6:867e70f2:::benchmark_data_smithi187_102448_object82998:head v 619'11309, mlcod=619'11306)", 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:11.229360+0000", 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.17207853900001, 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.142139139, 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.229360+0000", 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.229360+0000", 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.229361+0000", 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7199999999999997e-07 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.229367+0000", 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.241e-06 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.367448+0000", 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.138081221 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.367455+0000", 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7089999999999996e-06 2024-09-02T10:24:51.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.367513+0000", 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8069000000000003e-05 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.367539+0000", 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.546e-05 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.371452+0000", 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039135130000000004 2024-09-02T10:24:51.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:11.371499+0000", 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7054e-05 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:183760 6.5 e619/533 6:a105bd12:::benchmark_data_smithi187_102448_object83488:head v 619'11547, mlcod=619'11543)", 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:12.127198+0000", 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 159.27424020000001, 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13583553700000001, 2024-09-02T10:24:51.448 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.127198+0000", 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.127198+0000", 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.127200+0000", 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5e-06 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.127210+0000", 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0349999999999999e-05 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.259243+0000", 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13203271499999999 2024-09-02T10:24:51.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.259248+0000", 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4750000000000001e-06 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.259297+0000", 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9109e-05 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.259320+0000", 2024-09-02T10:24:51.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2501999999999999e-05 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.262999+0000", 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036790249999999998 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:12.263034+0000", 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4860999999999997e-05 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.452 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:199541 6.9 e625/533 6:99e7e980:::benchmark_data_smithi187_102448_object99269:head v 625'12423, mlcod=625'12420)", 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:39.132919+0000", 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 132.26851909300001, 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.133414848, 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.132919+0000", 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.132919+0000", 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.132920+0000", 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5700000000000001e-07 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.132926+0000", 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3310000000000003e-06 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.261661+0000", 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.12873506200000001 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.261668+0000", 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5580000000000003e-06 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.261767+0000", 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9121000000000002e-05 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.261791+0000", 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3553999999999998e-05 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.266292+0000", 2024-09-02T10:24:51.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045013040000000002 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:39.266334+0000", 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2061000000000003e-05 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:7924 7.a 7:57270faf:::benchmark_data_smithi187_120712_object7923:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:17.879815+0000", 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 33.521623746000003, 2024-09-02T10:24:51.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13918634499999999, 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 7924 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.879815+0000", 2024-09-02T10:24:51.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.879815+0000", 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.879816+0000", 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.254e-06 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.879835+0000", 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9199999999999999e-05 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.879856+0000", 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0534e-05 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.879861+0000", 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0889999999999999e-06 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.879911+0000", 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9721e-05 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.880010+0000", 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9857000000000003e-05 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.880102+0000", 2024-09-02T10:24:51.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1509999999999996e-05 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.880559+0000", 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00045701800000000001 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.880942+0000", 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00038332399999999999 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.018938+0000", 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13799526300000001 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.018971+0000", 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3855e-05 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.019001+0000", 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.972e-05 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.463 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:8307 7.a 7:58091577:::benchmark_data_smithi187_120712_object8306:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:18.374943+0000", 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 33.026495619000002, 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.17937603999999999, 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 8307 2024-09-02T10:24:51.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.374943+0000", 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.374943+0000", 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.374944+0000", 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1090000000000001e-06 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.374965+0000", 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0703000000000002e-05 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.374976+0000", 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1116e-05 2024-09-02T10:24:51.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.374982+0000", 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9909999999999999e-06 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.375024+0000", 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2203999999999998e-05 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.375117+0000", 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3228999999999998e-05 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.375201+0000", 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3707000000000003e-05 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.375739+0000", 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00053855999999999999 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.381806+0000", 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0060665989999999998 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.554210+0000", 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.17240378200000001 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.554276+0000", 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6185000000000004e-05 2024-09-02T10:24:51.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:18.554319+0000", 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2855000000000002e-05 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:12194 7.0 e668/661 7:03a8fc00:::benchmark_data_smithi187_120712_object12193:head v 668'780, mlcod=668'777)", 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:20.858789+0000", 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 30.542649721, 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.1442676, 2024-09-02T10:24:51.470 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:20.858789+0000", 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:20.858789+0000", 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:20.858789+0000", 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4499999999999997e-07 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:20.858850+0000", 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0767999999999998e-05 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:21.002338+0000", 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.143488109 2024-09-02T10:24:51.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:21.002351+0000", 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2327000000000001e-05 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:21.002441+0000", 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0365000000000006e-05 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:21.002456+0000", 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4932999999999999e-05 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:21.003027+0000", 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00057081199999999997 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:21.003056+0000", 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9740999999999999e-05 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.474 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGPush(7.4 674/672 [PushOp(7:2da41cfd:::benchmark_data_smithi187_120712_object8329:head, version: 667'509, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:2da41cfd:::benchmark_data_smithi187_120712_object8329:head@667'509, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:31.414247+0000", 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 19.987191637999999, 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13194886, 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.414247+0000", 2024-09-02T10:24:51.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.414247+0000", 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.414248+0000", 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.44e-06 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.414306+0000", 2024-09-02T10:24:51.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7442000000000003e-05 2024-09-02T10:24:51.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.545358+0000", 2024-09-02T10:24:51.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13105259599999999 2024-09-02T10:24:51.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.545373+0000", 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4472e-05 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.545452+0000", 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9109000000000004e-05 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.545461+0000", 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3919999999999992e-06 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.546196+0000", 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00073440899999999999 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:23159 7.a 7:531267a1:::benchmark_data_smithi187_120712_object23158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:31.415771+0000", 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 19.985667108000001, 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13268749499999999, 2024-09-02T10:24:51.479 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 23159 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.415771+0000", 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.415771+0000", 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.415773+0000", 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3349999999999999e-06 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.415792+0000", 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9264999999999999e-05 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.415803+0000", 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0715e-05 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.415809+0000", 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0549999999999996e-06 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.415859+0000", 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0257999999999999e-05 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.415958+0000", 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8528000000000003e-05 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.416042+0000", 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4892e-05 2024-09-02T10:24:51.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.416833+0000", 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00079098899999999997 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.438625+0000", 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.021791119000000001 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.548356+0000", 2024-09-02T10:24:51.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.10973195099999999 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.548416+0000", 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9398000000000003e-05 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.548459+0000", 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.299e-05 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:23159 7.a e674/661 ondisk, result = 0)", 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:31.417025+0000", 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 19.984413352000001, 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13144061200000001, 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.417025+0000", 2024-09-02T10:24:51.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.417025+0000", 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.417026+0000", 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8199999999999999e-07 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.417029+0000", 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4400000000000001e-06 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.548291+0000", 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13126154000000001 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.548298+0000", 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6340000000000001e-06 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.548343+0000", 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4409000000000001e-05 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.548350+0000", 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0400000000000004e-06 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.548466+0000", 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011586699999999999 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.489 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:23941 7.a 7:54134a82:::benchmark_data_smithi187_120712_object23940:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:32.137153+0000", 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 19.264285903000001, 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.17711252999999999, 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:24:51.490 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 23941 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.137153+0000", 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.137153+0000", 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.137154+0000", 2024-09-02T10:24:51.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7489999999999999e-06 2024-09-02T10:24:51.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:51.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.137179+0000", 2024-09-02T10:24:51.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4749999999999999e-05 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.137195+0000", 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6042999999999999e-05 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.137203+0000", 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8669999999999994e-06 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.137258+0000", 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4700999999999997e-05 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.137361+0000", 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010356099999999999 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.137455+0000", 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3368999999999993e-05 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.138414+0000", 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00095918100000000003 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.305657+0000", 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.167242896 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.314181+0000", 2024-09-02T10:24:51.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0085240320000000008 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.314222+0000", 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1022000000000001e-05 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.314265+0000", 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3359000000000003e-05 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:23941 7.a e674/661 ondisk, result = 0)", 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:32.138630+0000", 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 19.262808199999998, 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16703343000000001, 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.138630+0000", 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.138630+0000", 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.138631+0000", 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1349999999999999e-06 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.138636+0000", 2024-09-02T10:24:51.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8489999999999997e-06 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.305467+0000", 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.166830756 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.305549+0000", 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1629000000000001e-05 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.305642+0000", 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3022000000000005e-05 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.305651+0000", 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2089999999999994e-06 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.305664+0000", 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2830000000000001e-05 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.501 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:23941 7.a e674/661 ondisk, result = 0)", 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:32.138647+0000", 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 19.262791657000001, 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.175625788, 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.138647+0000", 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.138647+0000", 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.138648+0000", 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1600000000000001e-07 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.138653+0000", 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0289999999999996e-06 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.314075+0000", 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.17542286000000001 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.314097+0000", 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1353e-05 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.314167+0000", 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0304999999999999e-05 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.314175+0000", 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5059999999999998e-06 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.314273+0000", 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8018999999999998e-05 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:25331 7.f e675/672 7:f2e00bea:::benchmark_data_smithi187_120712_object25330:head v 675'1655, mlcod=668'708)", 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:33.605324+0000", 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 17.796114760000002, 2024-09-02T10:24:51.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.13208483500000001, 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:33.605324+0000", 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:24:51.507 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:33.605324+0000", 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:33.605325+0000", 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0210000000000001e-06 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:33.605351+0000", 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5834999999999999e-05 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:33.736098+0000", 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.130747001 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:33.736106+0000", 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2679999999999994e-06 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:33.736167+0000", 2024-09-02T10:24:51.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0859000000000001e-05 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:33.736193+0000", 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6590000000000001e-05 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:33.737365+0000", 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001171896 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:24:51.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:33.737409+0000", 2024-09-02T10:24:51.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3365000000000002e-05 2024-09-02T10:24:51.511 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.511 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.511 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.511 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:24:51.511 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:24:51.511 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:24:51.517 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:51.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:51 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:24:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:51 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:24:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:51 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:24:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:51 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:24:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/435723823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:51 smithi162 ceph-mon[20279]: osdmap e687: 12 total, 12 up, 12 in 2024-09-02T10:24:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2714840412' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:51.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:51.640 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:51.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:51.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:51.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:51 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:24:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:51 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:24:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:51 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:24:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:51 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:24:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/435723823' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:24:51.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:51 smithi187 ceph-mon[20072]: osdmap e687: 12 total, 12 up, 12 in 2024-09-02T10:24:51.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2714840412' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:51.758 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 39 16 43317 43301 69.3868 81.1875 0.0769233 0.0143512 2024-09-02T10:24:51.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:51.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:51.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:51.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:51.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:51.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:51.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:52.016 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:52.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:52.102 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:24:52.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-09-02T10:24:52.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:52.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:52.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:52.151 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:52.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:52.277 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:52.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:52.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:52.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:24:52.379 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:52.384 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:52 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:24:52.384 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:52 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:24:52.384 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:52 smithi187 ceph-mon[20072]: osdmap e688: 12 total, 12 up, 12 in 2024-09-02T10:24:52.384 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:52 smithi187 ceph-mon[20072]: pgmap v1584: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-02T10:24:52.417 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:52.418 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:52.447 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:52.519 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:52.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:52.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:52.575 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:52 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:24:52.575 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:52 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:24:52.575 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:52 smithi151 ceph-mon[16639]: osdmap e688: 12 total, 12 up, 12 in 2024-09-02T10:24:52.575 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:52 smithi151 ceph-mon[16639]: pgmap v1584: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-02T10:24:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:52 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:24:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:52 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:24:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:52 smithi162 ceph-mon[20279]: osdmap e688: 12 total, 12 up, 12 in 2024-09-02T10:24:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:52 smithi162 ceph-mon[20279]: pgmap v1584: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-02T10:24:52.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:52.680 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:52.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:52.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:52.720 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:52.758 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:24:52.758311+0000 min lat: 0.00108091 max lat: 2.89813 avg lat: 0.0142673 2024-09-02T10:24:52.758 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:24:52.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 40 16 44858 44842 70.0597 96.3125 0.010624 0.0142673 2024-09-02T10:24:52.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:52.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:52.814 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:52.815 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:52.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:52.939 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:52.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:53.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:53.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:53.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:53.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:53.173 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:53.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:53.211 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:53.290 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-09-02T10:24:53.311 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:53.324 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:24:53.324 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:24:53.324 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:24:53.324 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:24:53.334 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:53.335 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:53.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4245301856' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:24:53.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:53.456 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:53.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4245301856' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:24:53.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:53.579 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:53.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:53.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:53.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:24:53.664 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4245301856' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:24:53.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:53.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:53.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:53.705 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:53.758 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 41 16 45655 45639 69.5658 49.8125 0.0781916 0.0142498 2024-09-02T10:24:53.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:53.826 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:53.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:53.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:53.950 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:53.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:53.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-09-02T10:24:54.012 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:54.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:54.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:54.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:54.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:24:54.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:54.198 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:54.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:54.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:54.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:54.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:54.324 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:54.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:24:54.402 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:54.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:54.450 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:54.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4245301856' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:24:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:54 smithi187 ceph-mon[20072]: osdmap e689: 12 total, 12 up, 12 in 2024-09-02T10:24:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:54 smithi187 ceph-mon[20072]: pgmap v1586: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:24:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:54 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:24:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:54 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:24:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:54 smithi187 ceph-mon[20072]: osdmap e690: 12 total, 12 up, 12 in 2024-09-02T10:24:54.529 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:54.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:54.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:54.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4245301856' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:24:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:54 smithi162 ceph-mon[20279]: osdmap e689: 12 total, 12 up, 12 in 2024-09-02T10:24:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:54 smithi162 ceph-mon[20279]: pgmap v1586: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:24:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:54 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:24:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:54 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:24:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:54 smithi162 ceph-mon[20279]: osdmap e690: 12 total, 12 up, 12 in 2024-09-02T10:24:54.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4245301856' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:24:54.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:54 smithi151 ceph-mon[16639]: osdmap e689: 12 total, 12 up, 12 in 2024-09-02T10:24:54.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:54 smithi151 ceph-mon[16639]: pgmap v1586: 47 pgs: 47 active+clean; 2.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:24:54.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:54 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:24:54.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:54 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:24:54.650 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:54 smithi151 ceph-mon[16639]: osdmap e690: 12 total, 12 up, 12 in 2024-09-02T10:24:54.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:54.699 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:54.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:54.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:54.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 42 16 45666 45650 67.9257 0.6875 0.00262892 0.0142729 2024-09-02T10:24:54.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:54.824 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:54.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:54.882 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:24:54.882 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:24:54.882 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:24:54.882 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:24:54.882 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:24:54.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:54.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:54.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:55.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:55.072 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:55.076 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:55.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:55.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:55.190 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:24:55.205 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:55.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:55.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:55.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:55.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:55.437 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:55.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:55.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:55.476 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:55.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:55.578 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:55 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2352167533' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:24:55.578 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:55 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:55.578 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:55 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2352167533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:24:55.578 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:55 smithi187 ceph-mon[20072]: osdmap e691: 12 total, 12 up, 12 in 2024-09-02T10:24:55.578 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:55 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3325013212' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:24:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:55 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2352167533' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:24:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:55 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:55 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2352167533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:24:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:55 smithi162 ceph-mon[20279]: osdmap e691: 12 total, 12 up, 12 in 2024-09-02T10:24:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:55 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3325013212' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:24:55.599 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:55.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:55.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:24:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:55 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2352167533' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:24:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:55 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:24:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:55 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2352167533' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:24:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:55 smithi151 ceph-mon[16639]: osdmap e691: 12 total, 12 up, 12 in 2024-09-02T10:24:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:55 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3325013212' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:24:55.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:55.724 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:55.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:55.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:55.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 43 16 46139 46123 67.0335 29.5625 0.021297 0.0149128 2024-09-02T10:24:55.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:55.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:55.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:55.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:55.955 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:24:55.956 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:24:55.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:55.972 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:55.991 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:24:55.991 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:24:55.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:55.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:55.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:56.018 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:24:56.019 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:24:56.043 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:24:56.044 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:24:56.070 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:24:56.070 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:24:56.093 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:24:56.093 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:24:56.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:56.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:56.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:24:56.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:24:56.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:56.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:56.138 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:24:56.138 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:24:56.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:24:56.159 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:24:56.179 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:24:56.179 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:24:56.192 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-02T10:24:56.203 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:24:56.203 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:24:56.219 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:56.227 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:24:56.227 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:24:56.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:56.241 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:56.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:56.372 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:56.373 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:56.392 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:56 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:24:56.392 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:56 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:24:56.392 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:56 smithi187 ceph-mon[20072]: 7.9 starting backfill to osd.0 from (0'0,670'1282] MIN to 688'2942 2024-09-02T10:24:56.392 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:56 smithi187 ceph-mon[20072]: 7.f starting backfill to osd.0 from (0'0,670'1266] MIN to 688'2906 2024-09-02T10:24:56.393 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:56 smithi187 ceph-mon[20072]: 7.4 starting backfill to osd.0 from (0'0,670'1243] MIN to 688'2953 2024-09-02T10:24:56.393 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:56 smithi187 ceph-mon[20072]: 7.5 starting backfill to osd.0 from (0'0,670'1187] MIN to 688'2870 2024-09-02T10:24:56.393 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:56 smithi187 ceph-mon[20072]: pgmap v1589: 47 pgs: 2 remapped+peering, 45 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-02T10:24:56.393 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3221862588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:56.393 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3325013212' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:24:56.393 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:56 smithi187 ceph-mon[20072]: osdmap e692: 12 total, 12 up, 12 in 2024-09-02T10:24:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:56 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:24:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:56 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:24:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:56 smithi151 ceph-mon[16639]: 7.9 starting backfill to osd.0 from (0'0,670'1282] MIN to 688'2942 2024-09-02T10:24:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:56 smithi151 ceph-mon[16639]: 7.f starting backfill to osd.0 from (0'0,670'1266] MIN to 688'2906 2024-09-02T10:24:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:56 smithi151 ceph-mon[16639]: 7.4 starting backfill to osd.0 from (0'0,670'1243] MIN to 688'2953 2024-09-02T10:24:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:56 smithi151 ceph-mon[16639]: 7.5 starting backfill to osd.0 from (0'0,670'1187] MIN to 688'2870 2024-09-02T10:24:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:56 smithi151 ceph-mon[16639]: pgmap v1589: 47 pgs: 2 remapped+peering, 45 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-02T10:24:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3221862588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3325013212' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:24:56.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:56 smithi151 ceph-mon[16639]: osdmap e692: 12 total, 12 up, 12 in 2024-09-02T10:24:56.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:56.508 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:56.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:56.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:56.553 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:24:56.553 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:24:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:56 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:24:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:56 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:24:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:56 smithi162 ceph-mon[20279]: 7.9 starting backfill to osd.0 from (0'0,670'1282] MIN to 688'2942 2024-09-02T10:24:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:56 smithi162 ceph-mon[20279]: 7.f starting backfill to osd.0 from (0'0,670'1266] MIN to 688'2906 2024-09-02T10:24:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:56 smithi162 ceph-mon[20279]: 7.4 starting backfill to osd.0 from (0'0,670'1243] MIN to 688'2953 2024-09-02T10:24:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:56 smithi162 ceph-mon[20279]: 7.5 starting backfill to osd.0 from (0'0,670'1187] MIN to 688'2870 2024-09-02T10:24:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:56 smithi162 ceph-mon[20279]: pgmap v1589: 47 pgs: 2 remapped+peering, 45 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 93 MiB/s wr, 1.49k op/s 2024-09-02T10:24:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3221862588' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:24:56.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3325013212' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:24:56.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:56 smithi162 ceph-mon[20279]: osdmap e692: 12 total, 12 up, 12 in 2024-09-02T10:24:56.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:56.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:56.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:56.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:56.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 44 16 47060 47044 66.8181 57.5625 0.00823817 0.0149584 2024-09-02T10:24:56.766 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:56.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:56.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:56.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:56.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:56.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:56.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:57.025 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:57.026 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:57.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:57.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:57.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:57.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:24:57.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:57.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:57.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:57.316 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:57.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:57.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:57.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:57.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:57.568 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:57.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:57.620 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:24:57.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:24:57.693 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:57.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:24:57.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 45 16 48114 48098 66.797 65.875 0.0228629 0.0149638 2024-09-02T10:24:57.795 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:57.820 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:57.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:57.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:57.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:57.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:57.944 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:57.944 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:58.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:58.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:58.081 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:58.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:58.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:58.208 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:58.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:58.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:58.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:24:58.333 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:58.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:24:58.426 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-09-02T10:24:58.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:24:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:58.504 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:58.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:24:58.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:24:58.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:58.631 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:24:58 smithi151 ceph-mon[16639]: pgmap v1591: 47 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 44 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 765 op/s; 2853/137748 objects misplaced (2.071%); 2.0 MiB/s, 32 objects/s recovering 2024-09-02T10:24:58.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:58.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:58.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:58.737 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:58.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:24:58 smithi187 ceph-mon[20072]: pgmap v1591: 47 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 44 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 765 op/s; 2853/137748 objects misplaced (2.071%); 2.0 MiB/s, 32 objects/s recovering 2024-09-02T10:24:58.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 46 16 49509 49493 67.2402 87.1875 0.00439963 0.0148644 2024-09-02T10:24:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:58.766 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:58.766 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:58.792 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:58.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:24:58 smithi162 ceph-mon[20279]: pgmap v1591: 47 pgs: 1 active+remapped+backfilling, 2 remapped+peering, 44 active+clean; 2.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 765 op/s; 2853/137748 objects misplaced (2.071%); 2.0 MiB/s, 32 objects/s recovering 2024-09-02T10:24:58.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:24:58.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:58.888 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:24:58.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:24:59.016 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:59.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:59.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:24:59.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:24:59.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:59.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:59.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:24:59.246 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:59.274 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:59.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:59.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:24:59.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:24:59.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:24:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:59.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:24:59.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:24:59.530 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:59.531 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:24:59.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:24:59.632 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:24:59.663 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:59.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:59.665 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:24:59.730 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:24:59.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 47 16 50443 50427 67.0515 58.375 0.0337433 0.0149064 2024-09-02T10:24:59.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:24:59.789 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:24:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:59.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:24:59.891 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:24:59.925 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:24:59.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:24:59.927 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:00.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:00.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:00.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:00.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:00.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:00.187 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:00.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:00.209 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-02T10:25:00.289 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:00.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:00.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:00.321 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:25:00.322 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:25:00.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:00.437 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:00 smithi151 ceph-mon[16639]: pgmap v1592: 47 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 remapped+peering, 43 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 833 op/s; 6087/143247 objects misplaced (4.249%); 1.4 MiB/s, 22 objects/s recovering 2024-09-02T10:25:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3412521569' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:25:00.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:00.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:00.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:00 smithi187 ceph-mon[20072]: pgmap v1592: 47 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 remapped+peering, 43 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 833 op/s; 6087/143247 objects misplaced (4.249%); 1.4 MiB/s, 22 objects/s recovering 2024-09-02T10:25:00.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3412521569' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:25:00.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:00.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:00.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:00.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:00 smithi162 ceph-mon[20279]: pgmap v1592: 47 pgs: 1 active+remapped+backfill_wait, 1 active+remapped+backfilling, 2 remapped+peering, 43 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 833 op/s; 6087/143247 objects misplaced (4.249%); 1.4 MiB/s, 22 objects/s recovering 2024-09-02T10:25:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3412521569' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:25:00.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:00.689 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:00.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:00.776 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 48 16 51097 51081 66.5061 40.875 0.379496 0.0148956 2024-09-02T10:25:00.790 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:00.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:00.814 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:00.915 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:00.943 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:00.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:01.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:01.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:01.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:01.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:01.201 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:01.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:01.206 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:25:01.300 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-09-02T10:25:01.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:01.328 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:01.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:01.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:01.452 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:01.453 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:01.496 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3412521569' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:25:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:01 smithi187 ceph-mon[20072]: osdmap e693: 12 total, 12 up, 12 in 2024-09-02T10:25:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2833082699' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:01.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:01.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:01.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:01.577 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:01.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3412521569' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:25:01.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:01 smithi162 ceph-mon[20279]: osdmap e693: 12 total, 12 up, 12 in 2024-09-02T10:25:01.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2833082699' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:01.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3412521569' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:25:01.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:01 smithi151 ceph-mon[16639]: osdmap e693: 12 total, 12 up, 12 in 2024-09-02T10:25:01.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2833082699' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:01.677 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:01.704 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:01.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:01.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:01.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 49 16 51099 51083 65.1513 0.125 0.227193 0.0149099 2024-09-02T10:25:01.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:01.832 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:01.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:01.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:01.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:01.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:02.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:02.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:02.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:02.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:02.210 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:02.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:02.253 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:02.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:02.334 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:02.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:02.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:02.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:02 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2833082699' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:25:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:02 smithi151 ceph-mon[16639]: osdmap e694: 12 total, 12 up, 12 in 2024-09-02T10:25:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:02 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:25:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:02 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:25:02.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:02 smithi151 ceph-mon[16639]: pgmap v1595: 47 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 43 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 668 op/s; 11609/145998 objects misplaced (7.951%); 11 MiB/s, 168 objects/s recovering 2024-09-02T10:25:02.467 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:02.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:02 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2833082699' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:25:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:02 smithi187 ceph-mon[20072]: osdmap e694: 12 total, 12 up, 12 in 2024-09-02T10:25:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:02 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:25:02.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:02 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:25:02.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:02 smithi187 ceph-mon[20072]: pgmap v1595: 47 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 43 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 668 op/s; 11609/145998 objects misplaced (7.951%); 11 MiB/s, 168 objects/s recovering 2024-09-02T10:25:02.569 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:02 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2833082699' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:25:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:02 smithi162 ceph-mon[20279]: osdmap e694: 12 total, 12 up, 12 in 2024-09-02T10:25:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:02 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:25:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:02 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:25:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:02 smithi162 ceph-mon[20279]: pgmap v1595: 47 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 43 active+clean; 3.0 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 668 op/s; 11609/145998 objects misplaced (7.951%); 11 MiB/s, 168 objects/s recovering 2024-09-02T10:25:02.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:02.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:02.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:02.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:02.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:02.735 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:02.735 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:02.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 50 16 51568 51552 64.4345 29.3125 0.0210147 0.015512 2024-09-02T10:25:02.836 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:02.860 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:02.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:02.861 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:02.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:02.990 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:02.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:03.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:03.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:03.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:03.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:03.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:03.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:03.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:03.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:03.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:03.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:03.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:03 smithi187 ceph-mon[20072]: osdmap e695: 12 total, 12 up, 12 in 2024-09-02T10:25:03.511 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:03.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:03 smithi162 ceph-mon[20279]: osdmap e695: 12 total, 12 up, 12 in 2024-09-02T10:25:03.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:03.639 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:03.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:03 smithi151 ceph-mon[16639]: osdmap e695: 12 total, 12 up, 12 in 2024-09-02T10:25:03.741 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:03.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 51 16 52373 52357 64.1575 50.3125 0.00333752 0.01558 2024-09-02T10:25:03.767 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:03.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:03.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:03.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:03.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:03.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:03.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:04.019 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:04.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:04.121 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:04.145 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:04.152 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:04.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:25:04.253 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:04.277 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:04.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:04.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:04.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:04.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:04.460 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:04 smithi151 ceph-mon[16639]: pgmap v1597: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 41 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 775 op/s; 17582/151716 objects misplaced (11.589%); 15 MiB/s, 247 objects/s recovering 2024-09-02T10:25:04.460 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:04 smithi151 ceph-mon[16639]: osdmap e696: 12 total, 12 up, 12 in 2024-09-02T10:25:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:04 smithi187 ceph-mon[20072]: pgmap v1597: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 41 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 775 op/s; 17582/151716 objects misplaced (11.589%); 15 MiB/s, 247 objects/s recovering 2024-09-02T10:25:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:04 smithi187 ceph-mon[20072]: osdmap e696: 12 total, 12 up, 12 in 2024-09-02T10:25:04.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:04.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:04.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:04.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:04 smithi162 ceph-mon[20279]: pgmap v1597: 47 pgs: 2 active+recovering+undersized+remapped, 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 41 active+clean; 3.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 775 op/s; 17582/151716 objects misplaced (11.589%); 15 MiB/s, 247 objects/s recovering 2024-09-02T10:25:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:04 smithi162 ceph-mon[20279]: osdmap e696: 12 total, 12 up, 12 in 2024-09-02T10:25:04.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:04.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:04.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:04.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 52 16 53161 53145 63.8707 49.25 0.00702566 0.0156478 2024-09-02T10:25:04.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:04.790 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:04.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:04.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:04.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:04.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:04.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:04.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:05.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:05.061 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:05.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:05.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:05.192 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:05.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:05.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:05.322 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:05.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:05.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:05.453 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:05.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:05.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:05.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:05.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:05.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:05.715 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:05.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:05.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 53 16 54033 54017 63.6938 54.5 0.00962162 0.0156944 2024-09-02T10:25:05.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:05.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:05.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:05.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:05.967 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:05.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:06.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:06.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:06.097 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:06.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:06.221 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:06.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:06.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:25:06.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:06.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:06.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:06.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:06.473 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:06.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:06.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:06.605 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:06.606 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:06.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:06.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:06.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:06 smithi187 ceph-mon[20072]: pgmap v1599: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 609 op/s; 20064/155559 objects misplaced (12.898%); 20 MiB/s, 323 objects/s recovering 2024-09-02T10:25:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/353252302' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:06.759 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 54 16 54752 54736 63.3464 44.9375 0.0405374 0.0157704 2024-09-02T10:25:06.788 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:25:06.789 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:25:06.789 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:25:06.789 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:25:06.789 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:25:06.789 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:25:06.789 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:25:06.789 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:25:06.789 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:25:06.789 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:25:06.790 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:25:06.790 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:25:06.790 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:25:06.790 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:25:06.790 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:25:06.790 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:25:06.790 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:25:06.790 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:25:06.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:25:06.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:25:06.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:25:06.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:25:06.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:25:06.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:25:06.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:06 smithi151 ceph-mon[16639]: pgmap v1599: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 609 op/s; 20064/155559 objects misplaced (12.898%); 20 MiB/s, 323 objects/s recovering 2024-09-02T10:25:06.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/353252302' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:06.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:06.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:06 smithi162 ceph-mon[20279]: pgmap v1599: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 609 op/s; 20064/155559 objects misplaced (12.898%); 20 MiB/s, 323 objects/s recovering 2024-09-02T10:25:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/353252302' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:06.861 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:06.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:06.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:06.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:06.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:06.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:07.046 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:25:07.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:07.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:07.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:07.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:07.254 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:07.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:07.356 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:07.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:07.383 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:07.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:07.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:07.510 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:07.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:07.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:07.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:07.636 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:07.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:07.737 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:07.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 55 16 55510 55494 63.056 47.375 0.0121101 0.0158326 2024-09-02T10:25:07.763 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:07.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:07.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:07.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:07.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:07.888 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:07.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:08.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:08.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:08.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:08.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:08.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:08.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:08.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:08.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:08.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:08.374 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:08.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:08.428 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:08.428 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:08.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:08.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:08.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:08.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:08.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:08.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:08 smithi187 ceph-mon[20072]: pgmap v1600: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 763 op/s; 19370/160260 objects misplaced (12.087%); 26 MiB/s, 411 objects/s recovering 2024-09-02T10:25:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:08 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:25:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:08 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:25:08.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 56 16 56206 56190 62.7067 43.5 0.00495218 0.0159405 2024-09-02T10:25:08.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:08 smithi162 ceph-mon[20279]: pgmap v1600: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 763 op/s; 19370/160260 objects misplaced (12.087%); 26 MiB/s, 411 objects/s recovering 2024-09-02T10:25:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:08 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:25:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:08 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:25:08.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:08.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:08.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:08.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:08 smithi151 ceph-mon[16639]: pgmap v1600: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.3 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 48 MiB/s wr, 763 op/s; 19370/160260 objects misplaced (12.087%); 26 MiB/s, 411 objects/s recovering 2024-09-02T10:25:08.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:08 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:25:08.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:08 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:25:08.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:08.976 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:08.977 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:09.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:09.102 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:09.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:09.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:09.198 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-02T10:25:09.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:09.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:09.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:09.235 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:09.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:09.356 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:09.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:09.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:09.491 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:09.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:09.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:09.618 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:09.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:09.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:09.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:09.752 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:09.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:09.753 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:09 smithi187 ceph-mon[20072]: osdmap e697: 12 total, 12 up, 12 in 2024-09-02T10:25:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1883682671' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:25:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:09 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:25:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:09 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:25:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:09 smithi187 ceph-mon[20072]: osdmap e698: 12 total, 12 up, 12 in 2024-09-02T10:25:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3230625833' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:25:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:09 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:25:09.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 57 16 57057 57041 62.5396 53.1875 0.00648872 0.0159836 2024-09-02T10:25:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:09 smithi162 ceph-mon[20279]: osdmap e697: 12 total, 12 up, 12 in 2024-09-02T10:25:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1883682671' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:25:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:09 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:25:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:09 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:25:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:09 smithi162 ceph-mon[20279]: osdmap e698: 12 total, 12 up, 12 in 2024-09-02T10:25:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3230625833' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:25:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:09 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:25:09.854 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:09.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:09.885 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:09 smithi151 ceph-mon[16639]: osdmap e697: 12 total, 12 up, 12 in 2024-09-02T10:25:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1883682671' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:25:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:09 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:25:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:09 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:25:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:09 smithi151 ceph-mon[16639]: osdmap e698: 12 total, 12 up, 12 in 2024-09-02T10:25:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3230625833' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:25:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:09 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:25:09.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:10.007 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:10.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:10.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:10.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:10.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:10.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:10.223 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:25:10.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:10.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:10.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:10.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:10.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:10.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:10.411 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:10.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:10.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:10.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:10.545 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:10.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:10.675 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:10.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:10.676 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:10.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-09-02T10:25:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:10 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:25:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:10 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:25:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:10 smithi187 ceph-mon[20072]: pgmap v1603: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 759 op/s; 19404/165396 objects misplaced (11.732%); 23 MiB/s, 360 objects/s recovering 2024-09-02T10:25:10.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:10 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:10.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:10 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:25:10.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:10 smithi187 ceph-mon[20072]: osdmap e699: 12 total, 12 up, 12 in 2024-09-02T10:25:10.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 58 16 57788 57772 62.249 45.6875 0.00807157 0.0160573 2024-09-02T10:25:10.777 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:10.804 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:10.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:10 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:25:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:10 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:25:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:10 smithi162 ceph-mon[20279]: pgmap v1603: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 759 op/s; 19404/165396 objects misplaced (11.732%); 23 MiB/s, 360 objects/s recovering 2024-09-02T10:25:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:10 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:10 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:25:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:10 smithi162 ceph-mon[20279]: osdmap e699: 12 total, 12 up, 12 in 2024-09-02T10:25:10.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:10.935 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:10.937 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:10 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:25:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:10 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:25:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:10 smithi151 ceph-mon[16639]: pgmap v1603: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 759 op/s; 19404/165396 objects misplaced (11.732%); 23 MiB/s, 360 objects/s recovering 2024-09-02T10:25:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:10 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:10 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:25:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:10 smithi151 ceph-mon[16639]: osdmap e699: 12 total, 12 up, 12 in 2024-09-02T10:25:11.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:11.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:11.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:11.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:11.172 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:11.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:11.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:11.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:11.344 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:11.345 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:11.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:11.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:11.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:11.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:11.601 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:11.609 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:11.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:11.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:11.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:25:11.711 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:11.729 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:11 smithi151 ceph-mon[16639]: 7.0 scrub starts 2024-09-02T10:25:11.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:11.740 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:11 smithi187 ceph-mon[20072]: 7.0 scrub starts 2024-09-02T10:25:11.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 59 16 58667 58651 62.125 54.9375 0.0134275 0.0160811 2024-09-02T10:25:11.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:11 smithi162 ceph-mon[20279]: 7.0 scrub starts 2024-09-02T10:25:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:11.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:11.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:11.889 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:25:11.889 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:25:11.889 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:25:11.889 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:25:11.889 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:25:11.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:11.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:11.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:11.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:12.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:12.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:12.123 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:12.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:12.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:12.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:12.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:12.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:12.339 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:12.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:12.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:12.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:12.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:12.506 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-02T10:25:12.509 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:12.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:12.539 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:12.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:12.663 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:12.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:12.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:25:12.760071+0000 min lat: 0.00108091 max lat: 2.89813 avg lat: 0.0161592 2024-09-02T10:25:12.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:25:12.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 60 16 59399 59383 61.852 45.75 0.00337247 0.0161592 2024-09-02T10:25:12.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:12.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:12.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:12 smithi162 ceph-mon[20279]: 7.0 scrub ok 2024-09-02T10:25:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:12 smithi162 ceph-mon[20279]: pgmap v1605: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 668 op/s; 17829/167586 objects misplaced (10.639%); 26 MiB/s, 413 objects/s recovering 2024-09-02T10:25:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2517430806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:25:12.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:12 smithi151 ceph-mon[16639]: 7.0 scrub ok 2024-09-02T10:25:12.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:12 smithi151 ceph-mon[16639]: pgmap v1605: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 668 op/s; 17829/167586 objects misplaced (10.639%); 26 MiB/s, 413 objects/s recovering 2024-09-02T10:25:12.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2517430806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:25:12.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:12.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:12.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:12 smithi187 ceph-mon[20072]: 7.0 scrub ok 2024-09-02T10:25:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:12 smithi187 ceph-mon[20072]: pgmap v1605: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 42 MiB/s wr, 668 op/s; 17829/167586 objects misplaced (10.639%); 26 MiB/s, 413 objects/s recovering 2024-09-02T10:25:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2517430806' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:25:13.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:13.055 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:13.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:13.157 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:13.185 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:13.186 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:13.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:13.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:13.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:13.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-09-02T10:25:13.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:13.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:13.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:13.444 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:13.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:25:13.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:13.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2517430806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:25:13.554 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:13 smithi151 ceph-mon[16639]: osdmap e700: 12 total, 12 up, 12 in 2024-09-02T10:25:13.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:13.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:13.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:13.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:13.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:13.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:13.724 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:13.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 61 16 60255 60239 61.715 53.5 0.00393422 0.0161966 2024-09-02T10:25:13.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2517430806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:25:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:13 smithi162 ceph-mon[20279]: osdmap e700: 12 total, 12 up, 12 in 2024-09-02T10:25:13.853 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:13.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:13.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:13.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:25:13.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:25:13.909 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:25:13.910 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:25:13.946 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:25:13.946 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:25:13.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:13.981 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:25:13.981 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:25:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2517430806' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:25:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:13 smithi187 ceph-mon[20072]: osdmap e700: 12 total, 12 up, 12 in 2024-09-02T10:25:14.012 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:25:14.012 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:25:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:14.013 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:14.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:14.047 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:25:14.047 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:25:14.081 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:25:14.081 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:25:14.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:14.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:25:14.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:25:14.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:14.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:14.157 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:25:14.157 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:25:14.191 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:25:14.191 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:25:14.229 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:25:14.230 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:25:14.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:14.274 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:25:14.275 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:25:14.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:14.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:14.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:14.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:14.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:14.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:14.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:14.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:14.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:14.676 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:14.677 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:14.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 62 16 61310 61294 61.783 65.9375 0.0153506 0.0161767 2024-09-02T10:25:14.779 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:14.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:14.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:14 smithi162 ceph-mon[20279]: pgmap v1607: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 787 op/s; 16897/174024 objects misplaced (9.710%); 30 MiB/s, 478 objects/s recovering 2024-09-02T10:25:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2909076924' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:14.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:14 smithi151 ceph-mon[16639]: pgmap v1607: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 787 op/s; 16897/174024 objects misplaced (9.710%); 30 MiB/s, 478 objects/s recovering 2024-09-02T10:25:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2909076924' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:14.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:14.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:14.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:14 smithi187 ceph-mon[20072]: pgmap v1607: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 49 MiB/s wr, 787 op/s; 16897/174024 objects misplaced (9.710%); 30 MiB/s, 478 objects/s recovering 2024-09-02T10:25:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2909076924' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:15.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:15.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:15.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:15.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:15.214 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:15.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:15.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:25:15.315 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:15.337 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:15.346 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:15.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:15.348 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:15.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:15.483 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:15.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:15.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:15.546 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-02T10:25:15.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:15.610 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:15.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:15.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:15.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:15 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:25:15.647 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:15 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:25:15.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3798116092' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:25:15.648 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:15 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:25:15.713 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:15.737 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:15.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:15.760 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 63 16 62299 62283 61.7834 61.8125 0.00520302 0.0161788 2024-09-02T10:25:15.762 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:25:15.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:15.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:15 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:25:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:15 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:25:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3798116092' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:25:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:15 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:25:15.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:15.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:15.895 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:15 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:25:15.895 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:15 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:25:15.895 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3798116092' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:25:15.895 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:15 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:25:15.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:15.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:15.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:16.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:16.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:16.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:16.124 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:16.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:16.249 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:16.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:16.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:16.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:16.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:16.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:16.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:16.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-09-02T10:25:16.512 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:16.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:16.613 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:16.638 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:16.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:16.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:16.650 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:16 smithi187 ceph-mon[20072]: pgmap v1608: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 751 op/s; 15152/179457 objects misplaced (8.443%); 39 MiB/s, 630 objects/s recovering 2024-09-02T10:25:16.651 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:16 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:25:16.651 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:16 smithi187 ceph-mon[20072]: osdmap e701: 12 total, 12 up, 12 in 2024-09-02T10:25:16.651 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:16 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:25:16.651 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:16 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:25:16.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:16 smithi151 ceph-mon[16639]: pgmap v1608: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 751 op/s; 15152/179457 objects misplaced (8.443%); 39 MiB/s, 630 objects/s recovering 2024-09-02T10:25:16.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:16 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:25:16.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:16 smithi151 ceph-mon[16639]: osdmap e701: 12 total, 12 up, 12 in 2024-09-02T10:25:16.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:16 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:25:16.684 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:16 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:25:16.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:16.762 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:16.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:16.766 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 64 16 63291 63275 61.7831 62 0.024484 0.0161787 2024-09-02T10:25:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:16 smithi162 ceph-mon[20279]: pgmap v1608: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.7 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 47 MiB/s wr, 751 op/s; 15152/179457 objects misplaced (8.443%); 39 MiB/s, 630 objects/s recovering 2024-09-02T10:25:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:16 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:25:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:16 smithi162 ceph-mon[20279]: osdmap e701: 12 total, 12 up, 12 in 2024-09-02T10:25:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:16 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:25:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:16 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:25:16.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:16.889 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:16.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:16.930 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:16.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:17.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:17.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:17.028 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:17.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:17.154 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:17.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:17.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:17.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:17.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:17.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:17.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:17.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:25:17.402 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:25:17.402 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:25:17.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:17.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:17.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:17.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:17.567 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:25:17.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:17.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:17.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:17.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:17.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:17 smithi151 ceph-mon[16639]: osdmap e702: 12 total, 12 up, 12 in 2024-09-02T10:25:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/394031522' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:17.705 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:17.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:17 smithi187 ceph-mon[20072]: osdmap e702: 12 total, 12 up, 12 in 2024-09-02T10:25:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/394031522' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:17.764 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 65 16 64206 64190 61.7124 57.1875 0.0821939 0.0161969 2024-09-02T10:25:17.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:17.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:17.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:17 smithi162 ceph-mon[20279]: osdmap e702: 12 total, 12 up, 12 in 2024-09-02T10:25:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/394031522' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:17.938 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:17.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:17.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:17.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:18.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:18.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:18.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:18.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:25:18.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:18.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:18.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:18.337 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:18.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:18.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:18.459 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:18.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:18.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:18.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:18.568 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-09-02T10:25:18.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:18.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:18.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:18.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:18 smithi151 ceph-mon[16639]: pgmap v1611: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 960 op/s; 14364/184878 objects misplaced (7.769%); 34 MiB/s, 541 objects/s recovering 2024-09-02T10:25:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:18 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:25:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:18 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:25:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/394031522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:25:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:18 smithi151 ceph-mon[16639]: osdmap e703: 12 total, 12 up, 12 in 2024-09-02T10:25:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1165483213' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:25:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:18 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:25:18.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:18.762 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:18.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:18.764 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 66 16 65005 64989 61.5339 49.9375 0.015536 0.0162439 2024-09-02T10:25:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:18 smithi162 ceph-mon[20279]: pgmap v1611: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 960 op/s; 14364/184878 objects misplaced (7.769%); 34 MiB/s, 541 objects/s recovering 2024-09-02T10:25:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:18 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:25:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:18 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:25:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/394031522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:25:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:18 smithi162 ceph-mon[20279]: osdmap e703: 12 total, 12 up, 12 in 2024-09-02T10:25:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1165483213' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:25:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:18 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:25:18.864 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:18.889 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:18.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:18.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:18.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:18 smithi187 ceph-mon[20072]: pgmap v1611: 47 pgs: 1 active+remapped+backfill_wait, 3 active+recovering+undersized+remapped, 3 active+remapped+backfilling, 40 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 960 op/s; 14364/184878 objects misplaced (7.769%); 34 MiB/s, 541 objects/s recovering 2024-09-02T10:25:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:18 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:25:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:18 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:25:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/394031522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:25:19.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:18 smithi187 ceph-mon[20072]: osdmap e703: 12 total, 12 up, 12 in 2024-09-02T10:25:19.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1165483213' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:25:19.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:18 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:25:19.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:19.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:19.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:19.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:19.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:19.153 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:19.255 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:19.278 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:19.279 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:19.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:19.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:19.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:19.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:19.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:19.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:19.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:19.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:19.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:19.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:19.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:19.764 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 67 16 65458 65442 61.038 28.3125 0.386989 0.0162906 2024-09-02T10:25:19.776 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:19.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:19 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:25:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:19 smithi162 ceph-mon[20279]: osdmap e704: 12 total, 12 up, 12 in 2024-09-02T10:25:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:19 smithi162 ceph-mon[20279]: 7.e scrub starts 2024-09-02T10:25:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:19 smithi162 ceph-mon[20279]: osdmap e705: 12 total, 12 up, 12 in 2024-09-02T10:25:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1899502920' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:19.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:19 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:25:19.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:19 smithi151 ceph-mon[16639]: osdmap e704: 12 total, 12 up, 12 in 2024-09-02T10:25:19.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:19 smithi151 ceph-mon[16639]: 7.e scrub starts 2024-09-02T10:25:19.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:19 smithi151 ceph-mon[16639]: osdmap e705: 12 total, 12 up, 12 in 2024-09-02T10:25:19.854 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1899502920' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:19.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:19.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-09-02T10:25:19.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:19.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:19.928 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:20.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:19 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:25:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:19 smithi187 ceph-mon[20072]: osdmap e704: 12 total, 12 up, 12 in 2024-09-02T10:25:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:19 smithi187 ceph-mon[20072]: 7.e scrub starts 2024-09-02T10:25:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:19 smithi187 ceph-mon[20072]: osdmap e705: 12 total, 12 up, 12 in 2024-09-02T10:25:20.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1899502920' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:20.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:20.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:20.111 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:25:20.112 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:25:20.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:20.143 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:25:20.143 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:25:20.149 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:20.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:20.165 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:25:20.166 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:25:20.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:25:20.187 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:25:20.213 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:25:20.213 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:25:20.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:20.251 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:25:20.252 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:25:20.294 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:25:20.294 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:25:20.297 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:20.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:20.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:25:20.331 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:25:20.366 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:25:20.367 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:25:20.396 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:25:20.397 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:25:20.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:20.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:20.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:20.429 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:25:20.429 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:25:20.456 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:25:20.456 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:25:20.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:25:20.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:20.534 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:25:20.534 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:25:20.534 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:25:20.534 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:25:20.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:20.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:20.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:20.668 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:20.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:20 smithi151 ceph-mon[16639]: pgmap v1615: 47 pgs: 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 40 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 989 op/s; 14425/191013 objects misplaced (7.552%); 15 MiB/s, 238 objects/s recovering 2024-09-02T10:25:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:20 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:25:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:20 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:25:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:20 smithi151 ceph-mon[16639]: osdmap e706: 12 total, 12 up, 12 in 2024-09-02T10:25:20.764 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 68 16 65695 65679 60.3582 14.8125 0.00502551 0.0164731 2024-09-02T10:25:20.770 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:20.791 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:20.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:20.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:20 smithi162 ceph-mon[20279]: pgmap v1615: 47 pgs: 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 40 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 989 op/s; 14425/191013 objects misplaced (7.552%); 15 MiB/s, 238 objects/s recovering 2024-09-02T10:25:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:20 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:25:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:20 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:25:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:20 smithi162 ceph-mon[20279]: osdmap e706: 12 total, 12 up, 12 in 2024-09-02T10:25:20.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:20.916 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:20.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:20.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:20 smithi187 ceph-mon[20072]: pgmap v1615: 47 pgs: 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 40 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 989 op/s; 14425/191013 objects misplaced (7.552%); 15 MiB/s, 238 objects/s recovering 2024-09-02T10:25:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:20 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:25:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:20 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:25:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:20 smithi187 ceph-mon[20072]: osdmap e706: 12 total, 12 up, 12 in 2024-09-02T10:25:21.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:21.042 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:21.043 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:21.144 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:21.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:21.167 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:21.268 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:21.294 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:21.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:21.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:21.420 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:21.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:21.521 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:21.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:21.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:21.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:21.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:21.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:21.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 69 16 66279 66263 60.0124 36.5 0.0233417 0.0166576 2024-09-02T10:25:21.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:21.811 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:21.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:21.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:21 smithi162 ceph-mon[20279]: 7.e scrub ok 2024-09-02T10:25:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:21 smithi162 ceph-mon[20279]: osdmap e707: 12 total, 12 up, 12 in 2024-09-02T10:25:21.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:21 smithi151 ceph-mon[16639]: 7.e scrub ok 2024-09-02T10:25:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:21 smithi151 ceph-mon[16639]: osdmap e707: 12 total, 12 up, 12 in 2024-09-02T10:25:21.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:21.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:21.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:21 smithi187 ceph-mon[20072]: 7.e scrub ok 2024-09-02T10:25:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:21 smithi187 ceph-mon[20072]: osdmap e707: 12 total, 12 up, 12 in 2024-09-02T10:25:22.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:22.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:22.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:22.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:22.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:25:22.166 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:22.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:22.205 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:22.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:22.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:22.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:22.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:22.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:22.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:22.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:22.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:22.546 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:22.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:22.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:22 smithi151 ceph-mon[16639]: pgmap v1618: 47 pgs: 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 40 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 595 op/s; 13104/192624 objects misplaced (6.803%); 20 MiB/s, 322 objects/s recovering 2024-09-02T10:25:22.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:22 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:25:22.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:22 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:25:22.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:22.674 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:22.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:22.675 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:22.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 70 16 67189 67173 59.9675 56.875 0.007966 0.0166677 2024-09-02T10:25:22.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:22.800 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:22.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:22.801 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:22 smithi162 ceph-mon[20279]: pgmap v1618: 47 pgs: 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 40 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 595 op/s; 13104/192624 objects misplaced (6.803%); 20 MiB/s, 322 objects/s recovering 2024-09-02T10:25:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:22 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:25:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:22 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:25:22.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:22.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:22 smithi187 ceph-mon[20072]: pgmap v1618: 47 pgs: 3 active+recovering+undersized+remapped, 4 active+remapped+backfilling, 40 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 37 MiB/s wr, 595 op/s; 13104/192624 objects misplaced (6.803%); 20 MiB/s, 322 objects/s recovering 2024-09-02T10:25:22.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:22 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:25:22.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:22 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:25:22.926 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:22.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:23.028 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:23.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:23.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:23.165 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:23.187 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:23.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:23.188 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:23.289 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:23.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:23.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:23.316 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:23.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:25:23.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:23.437 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:23.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:23.438 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:23.539 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:23.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:23.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:23.591 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-09-02T10:25:23.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:23.708 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:23.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:23.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:23.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 71 16 67962 67946 59.8033 48.3125 0.0038435 0.0167144 2024-09-02T10:25:23.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:23.834 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:23.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:23.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1781417228' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:25:23.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:23.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1781417228' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:25:23.960 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:23.961 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1781417228' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:25:24.062 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:24.087 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:24.088 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:24.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:24.213 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:24.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:24.294 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:25:24.294 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:25:24.295 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:25:24.295 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:25:24.295 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:25:24.314 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:24.341 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:24.342 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:24.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:24.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:24.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:24.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:24.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:24 smithi151 ceph-mon[16639]: pgmap v1619: 47 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 42 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 809 op/s; 11049/199392 objects misplaced (5.541%); 35 MiB/s, 554 objects/s recovering 2024-09-02T10:25:24.588 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1781417228' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:25:24.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:24 smithi151 ceph-mon[16639]: osdmap e708: 12 total, 12 up, 12 in 2024-09-02T10:25:24.589 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2928147067' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:25:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:25:24.614 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:24.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:24.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:24.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:24.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:24.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:24.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 72 16 68818 68802 59.7157 53.5 0.00619968 0.016737 2024-09-02T10:25:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:24 smithi162 ceph-mon[20279]: pgmap v1619: 47 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 42 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 809 op/s; 11049/199392 objects misplaced (5.541%); 35 MiB/s, 554 objects/s recovering 2024-09-02T10:25:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1781417228' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:25:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:24 smithi162 ceph-mon[20279]: osdmap e708: 12 total, 12 up, 12 in 2024-09-02T10:25:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2928147067' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:25:24.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:24.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:24.877 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:24.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:25.000 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:25.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:24 smithi187 ceph-mon[20072]: pgmap v1619: 47 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 42 active+clean; 4.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 809 op/s; 11049/199392 objects misplaced (5.541%); 35 MiB/s, 554 objects/s recovering 2024-09-02T10:25:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1781417228' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:25:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:24 smithi187 ceph-mon[20072]: osdmap e708: 12 total, 12 up, 12 in 2024-09-02T10:25:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2928147067' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:25:25.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:25.129 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:25.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:25.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:25.252 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:25.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:25.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:25.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:25.379 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:25.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:25.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:25.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:25.510 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:25.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:25.638 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:25.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:25.640 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:25.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:25.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 73 16 69679 69663 59.6348 53.8125 0.00482993 0.0167625 2024-09-02T10:25:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:25.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:25.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:25 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2928147067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:25:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:25 smithi162 ceph-mon[20279]: osdmap e709: 12 total, 12 up, 12 in 2024-09-02T10:25:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:25 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:25:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:25 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:25:25.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:25.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:25.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:25.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:25 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2928147067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:25:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:25 smithi151 ceph-mon[16639]: osdmap e709: 12 total, 12 up, 12 in 2024-09-02T10:25:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:25 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:25:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:25 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:25:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:25 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2928147067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:25:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:25 smithi187 ceph-mon[20072]: osdmap e709: 12 total, 12 up, 12 in 2024-09-02T10:25:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:25 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:25:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:25 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:25:26.020 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:26.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:26.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:26.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:26.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:26.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:26.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:26.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:26.276 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:26.300 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:26.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:26.301 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:26.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:26.425 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:26.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:26.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:26.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:26.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:26.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:26.550 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:26.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:26.675 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:26.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:26.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-09-02T10:25:26.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 74 16 70554 70538 59.5679 54.6875 0.0275214 0.0167808 2024-09-02T10:25:26.777 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:26.811 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:26.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:26.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:26.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:26.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:26 smithi151 ceph-mon[16639]: pgmap v1622: 47 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 42 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 615 op/s; 10321/202269 objects misplaced (5.103%); 30 MiB/s, 487 objects/s recovering 2024-09-02T10:25:26.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:26 smithi151 ceph-mon[16639]: osdmap e710: 12 total, 12 up, 12 in 2024-09-02T10:25:26.949 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:26.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:26.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:26 smithi187 ceph-mon[20072]: pgmap v1622: 47 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 42 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 615 op/s; 10321/202269 objects misplaced (5.103%); 30 MiB/s, 487 objects/s recovering 2024-09-02T10:25:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:26 smithi187 ceph-mon[20072]: osdmap e710: 12 total, 12 up, 12 in 2024-09-02T10:25:27.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:27.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:27.089 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:27.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:26 smithi162 ceph-mon[20279]: pgmap v1622: 47 pgs: 3 active+recovering+undersized+remapped, 2 active+remapped+backfilling, 42 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 38 MiB/s wr, 615 op/s; 10321/202269 objects misplaced (5.103%); 30 MiB/s, 487 objects/s recovering 2024-09-02T10:25:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:26 smithi162 ceph-mon[20279]: osdmap e710: 12 total, 12 up, 12 in 2024-09-02T10:25:27.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:27.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:27.215 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:27.316 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:27.340 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:27.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:27.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:27.442 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:27.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:27.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:27.471 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:27.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:27.595 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:27.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:27.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:25:27.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:27.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:27.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:27.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:27.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 75 16 71591 71575 59.6378 64.8125 0.0513774 0.0167544 2024-09-02T10:25:27.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:27.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:27.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:27.908 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:25:27.909 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:25:27.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:27.994 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:27.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:27.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:25:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:25:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:25:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:25:28.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:28.117 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:28.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:28.118 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:28.219 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:28.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:28.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:28.241 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:28.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:28.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:28.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:28.365 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:28.378 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:28.410 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:28.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:28.486 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:28.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:28.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:28.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:28.609 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:28.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:28.711 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:28.738 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:28.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:28.739 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:28.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 76 16 71876 71860 59.0874 17.8125 0.00790098 0.0169186 2024-09-02T10:25:28.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:28.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:28.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:28.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: pgmap v1624: 47 pgs: 1 active+clean+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 42 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 793 op/s; 9189/206913 objects misplaced (4.441%); 36 MiB/s, 579 objects/s recovering 2024-09-02T10:25:28.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: osdmap e711: 12 total, 12 up, 12 in 2024-09-02T10:25:28.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1724031736,config=2353167246,health=551204407,kv=1882860337}) 2024-09-02T10:25:28.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=3924037727,logm=1647095347}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296273027}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166248726}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=727458006}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2529803842}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308479023}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091331278}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2315938611}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393095931}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4178599514}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3618048317}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2691813089}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671697270}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1552197543}) 2024-09-02T10:25:28.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1409444253}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2669344278}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3813674171}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3006334389}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568249257}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2789559848}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594758198}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=305580906}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208973784}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004751970}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=91862954}) 2024-09-02T10:25:28.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=89714523}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323017441}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692709277}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569885932}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2794309755}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1981420100}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2893956828}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773126159}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1489725274}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1929070581}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1368284741}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3325487098}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2777176067}) 2024-09-02T10:25:28.933 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=206125427}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187635356}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427201889}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=110005666}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2193391750}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2604399484}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3627376188}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=15139780}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2718991919}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242489683}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098924072}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3078017242}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=418591900}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1478603077}) 2024-09-02T10:25:28.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2900987319}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3775517813}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486250112}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1774543063}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3790232823}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634024861}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105915972}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179002247}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3738624884}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104319614}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=754108524}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1974835919}) 2024-09-02T10:25:28.935 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432785945}) 2024-09-02T10:25:28.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=53,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=10} crc {logm=2723207841,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=3970626954,mon_config_key=240619806}) 2024-09-02T10:25:28.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2606227617}) 2024-09-02T10:25:28.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=895370485}) 2024-09-02T10:25:28.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=1} crc {mon_config_key=670442692,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=318870949,osdmap=3822152966}) 2024-09-02T10:25:28.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3578095896}) 2024-09-02T10:25:28.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=110482080}) 2024-09-02T10:25:28.936 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=59} crc {osdmap=3574454615}) 2024-09-02T10:25:28.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: pgmap v1624: 47 pgs: 1 active+clean+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 42 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 793 op/s; 9189/206913 objects misplaced (4.441%); 36 MiB/s, 579 objects/s recovering 2024-09-02T10:25:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: osdmap e711: 12 total, 12 up, 12 in 2024-09-02T10:25:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1724031736,config=2353167246,health=551204407,kv=1882860337}) 2024-09-02T10:25:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=3924037727,logm=1647095347}) 2024-09-02T10:25:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296273027}) 2024-09-02T10:25:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166248726}) 2024-09-02T10:25:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=727458006}) 2024-09-02T10:25:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2529803842}) 2024-09-02T10:25:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308479023}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091331278}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2315938611}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393095931}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4178599514}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3618048317}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2691813089}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671697270}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1552197543}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1409444253}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2669344278}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3813674171}) 2024-09-02T10:25:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3006334389}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568249257}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2789559848}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594758198}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=305580906}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208973784}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004751970}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=91862954}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=89714523}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323017441}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692709277}) 2024-09-02T10:25:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569885932}) 2024-09-02T10:25:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2794309755}) 2024-09-02T10:25:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1981420100}) 2024-09-02T10:25:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2893956828}) 2024-09-02T10:25:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773126159}) 2024-09-02T10:25:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1489725274}) 2024-09-02T10:25:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1929070581}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1368284741}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3325487098}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2777176067}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=206125427}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187635356}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427201889}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=110005666}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2193391750}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2604399484}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3627376188}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=15139780}) 2024-09-02T10:25:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2718991919}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242489683}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098924072}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3078017242}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=418591900}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1478603077}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2900987319}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3775517813}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486250112}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1774543063}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3790232823}) 2024-09-02T10:25:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634024861}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105915972}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179002247}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3738624884}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104319614}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=754108524}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1974835919}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432785945}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=53,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=10} crc {logm=2723207841,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=3970626954,mon_config_key=240619806}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2606227617}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=895370485}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=1} crc {mon_config_key=670442692,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=318870949,osdmap=3822152966}) 2024-09-02T10:25:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3578095896}) 2024-09-02T10:25:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=110482080}) 2024-09-02T10:25:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=59} crc {osdmap=3574454615}) 2024-09-02T10:25:29.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:29.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: pgmap v1624: 47 pgs: 1 active+clean+remapped, 3 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 42 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 793 op/s; 9189/206913 objects misplaced (4.441%); 36 MiB/s, 579 objects/s recovering 2024-09-02T10:25:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: osdmap e711: 12 total, 12 up, 12 in 2024-09-02T10:25:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=1724031736,config=2353167246,health=551204407,kv=1882860337}) 2024-09-02T10:25:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {kv=7,logm=93} crc {kv=3924037727,logm=1647095347}) 2024-09-02T10:25:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2296273027}) 2024-09-02T10:25:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1166248726}) 2024-09-02T10:25:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=727458006}) 2024-09-02T10:25:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2529803842}) 2024-09-02T10:25:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308479023}) 2024-09-02T10:25:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2091331278}) 2024-09-02T10:25:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2315938611}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3393095931}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4178599514}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3618048317}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2691813089}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671697270}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1552197543}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1409444253}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2669344278}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3813674171}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3006334389}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568249257}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2789559848}) 2024-09-02T10:25:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1594758198}) 2024-09-02T10:25:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=305580906}) 2024-09-02T10:25:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2208973784}) 2024-09-02T10:25:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1004751970}) 2024-09-02T10:25:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=91862954}) 2024-09-02T10:25:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=89714523}) 2024-09-02T10:25:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3323017441}) 2024-09-02T10:25:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2692709277}) 2024-09-02T10:25:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569885932}) 2024-09-02T10:25:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2794309755}) 2024-09-02T10:25:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1981420100}) 2024-09-02T10:25:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2893956828}) 2024-09-02T10:25:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1773126159}) 2024-09-02T10:25:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1489725274}) 2024-09-02T10:25:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1929070581}) 2024-09-02T10:25:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1368284741}) 2024-09-02T10:25:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3325487098}) 2024-09-02T10:25:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2777176067}) 2024-09-02T10:25:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=206125427}) 2024-09-02T10:25:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1187635356}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3427201889}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=110005666}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2193391750}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2604399484}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3627376188}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=15139780}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2718991919}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2242489683}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3098924072}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3078017242}) 2024-09-02T10:25:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=418591900}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1478603077}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2900987319}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3775517813}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=486250112}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1774543063}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3790232823}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=634024861}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=105915972}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4179002247}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3738624884}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104319614}) 2024-09-02T10:25:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=754108524}) 2024-09-02T10:25:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1974835919}) 2024-09-02T10:25:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3432785945}) 2024-09-02T10:25:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=53,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=10} crc {logm=2723207841,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=3970626954,mon_config_key=240619806}) 2024-09-02T10:25:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2606227617}) 2024-09-02T10:25:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=895370485}) 2024-09-02T10:25:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=81,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=1} crc {mon_config_key=670442692,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=318870949,osdmap=3822152966}) 2024-09-02T10:25:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3578095896}) 2024-09-02T10:25:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=110482080}) 2024-09-02T10:25:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=59} crc {osdmap=3574454615}) 2024-09-02T10:25:29.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:29.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:29.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:29.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:29.290 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:29.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:29.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:29.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:29.415 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:29.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:29.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:29.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:29.542 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:29.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:29.667 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:29.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:29.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 77 16 72857 72841 59.1163 61.3125 0.03192 0.0169077 2024-09-02T10:25:29.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:29.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:29.793 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:29.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:29.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:29.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:29 smithi151 ceph-mon[16639]: osdmap e712: 12 total, 12 up, 12 in 2024-09-02T10:25:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:29 smithi187 ceph-mon[20072]: osdmap e712: 12 total, 12 up, 12 in 2024-09-02T10:25:30.023 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:30.046 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:30.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:30.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:29 smithi162 ceph-mon[20279]: osdmap e712: 12 total, 12 up, 12 in 2024-09-02T10:25:30.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:30.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:30.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:30.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:30.305 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:30.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:30.306 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:30.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:25:30.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:30.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:30.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:30.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:30.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:30.579 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:30.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:30.704 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:30.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:30.766 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 78 15 73771 73756 59.0915 57.1875 0.0257098 0.016918 2024-09-02T10:25:30.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:30.828 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:30.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:30.930 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:30 smithi151 ceph-mon[16639]: pgmap v1627: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+recovering+undersized+remapped, 42 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 805 op/s; 8673/213975 objects misplaced (4.053%); 21 MiB/s, 343 objects/s recovering 2024-09-02T10:25:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1163482105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:30.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:30.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:30 smithi187 ceph-mon[20072]: pgmap v1627: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+recovering+undersized+remapped, 42 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 805 op/s; 8673/213975 objects misplaced (4.053%); 21 MiB/s, 343 objects/s recovering 2024-09-02T10:25:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1163482105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:31.066 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:31.094 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:31.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:31.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:30 smithi162 ceph-mon[20279]: pgmap v1627: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+recovering+undersized+remapped, 42 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 50 MiB/s wr, 805 op/s; 8673/213975 objects misplaced (4.053%); 21 MiB/s, 343 objects/s recovering 2024-09-02T10:25:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1163482105' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:31.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:31.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:31.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:31.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:31.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:31.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:31.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:31.447 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:25:31.447 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:25:31.447 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:25:31.447 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:25:31.448 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:25:31.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:25:31.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:25:31.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:25:31.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:25:31.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:25:31.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:25:31.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:25:31.449 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:25:31.450 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:25:31.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:31.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:31.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:31.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:31.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:31.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:31.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:31.747 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:31.748 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:31.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 79 16 74736 74720 59.1061 60.25 0.0037452 0.0169123 2024-09-02T10:25:31.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:31.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:31.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:31.872 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:31 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:25:31.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:31 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:25:31.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:31.996 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:31.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:31 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:25:32.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:31 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:25:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:31 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:25:32.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:31 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:25:32.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:32.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:32.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:32.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:32.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:32.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:32.356 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:32.379 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:32.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:32.380 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:32.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:32.504 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:32.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:32.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:32.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:32.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:32.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:32.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:32.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:32.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:32.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:25:32.765461+0000 min lat: 0.00108091 max lat: 2.89813 avg lat: 0.0169777 2024-09-02T10:25:32.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:25:32.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 80 16 75396 75380 58.8829 41.25 0.0241255 0.0169777 2024-09-02T10:25:32.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:32.884 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:32.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:32 smithi151 ceph-mon[16639]: pgmap v1628: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+recovering+undersized+remapped, 42 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 690 op/s; 8308/214701 objects misplaced (3.870%); 21 MiB/s, 338 objects/s recovering 2024-09-02T10:25:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:32 smithi151 ceph-mon[16639]: osdmap e713: 12 total, 12 up, 12 in 2024-09-02T10:25:32.951 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:32 smithi187 ceph-mon[20072]: pgmap v1628: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+recovering+undersized+remapped, 42 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 690 op/s; 8308/214701 objects misplaced (3.870%); 21 MiB/s, 338 objects/s recovering 2024-09-02T10:25:32.951 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:32 smithi187 ceph-mon[20072]: osdmap e713: 12 total, 12 up, 12 in 2024-09-02T10:25:32.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:33.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:33.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:32 smithi162 ceph-mon[20279]: pgmap v1628: 47 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 3 active+recovering+undersized+remapped, 42 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 43 MiB/s wr, 690 op/s; 8308/214701 objects misplaced (3.870%); 21 MiB/s, 338 objects/s recovering 2024-09-02T10:25:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:32 smithi162 ceph-mon[20279]: osdmap e713: 12 total, 12 up, 12 in 2024-09-02T10:25:33.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:33.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:33.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:33.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:33.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:33.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:33.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:33.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:33.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:33.423 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:33.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:33.545 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:33.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:33.647 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:33.675 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:33.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:33.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 81 16 76269 76253 58.8295 54.5625 0.0167611 0.0169932 2024-09-02T10:25:33.777 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:33.800 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:33.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:33.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:33.902 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:33.923 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:33.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:33.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:34.025 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:34.047 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:34.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:34.047 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:34.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:34.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:34.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:34.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:34.308 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:34.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:34.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:34.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:34.414 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:34.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:34.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:34.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:34.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:34.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:34.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:34.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:34.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:34.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:34.718 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:34.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 82 16 77023 77007 58.6867 47.125 0.0230041 0.0170333 2024-09-02T10:25:34.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:34.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:34.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:34.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:34 smithi151 ceph-mon[16639]: pgmap v1630: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 885 op/s; 7517/222864 objects misplaced (3.373%); 20 MiB/s, 314 objects/s recovering 2024-09-02T10:25:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:34 smithi151 ceph-mon[16639]: osdmap e714: 12 total, 12 up, 12 in 2024-09-02T10:25:34.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:34.971 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:34.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:34.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:34 smithi187 ceph-mon[20072]: pgmap v1630: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 885 op/s; 7517/222864 objects misplaced (3.373%); 20 MiB/s, 314 objects/s recovering 2024-09-02T10:25:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:34 smithi187 ceph-mon[20072]: osdmap e714: 12 total, 12 up, 12 in 2024-09-02T10:25:35.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:34 smithi162 ceph-mon[20279]: pgmap v1630: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 55 MiB/s wr, 885 op/s; 7517/222864 objects misplaced (3.373%); 20 MiB/s, 314 objects/s recovering 2024-09-02T10:25:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:34 smithi162 ceph-mon[20279]: osdmap e714: 12 total, 12 up, 12 in 2024-09-02T10:25:35.099 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:35.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:35.201 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:35.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:35.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:35.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:35.361 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:35.362 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:35.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:35.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:35.496 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:35.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:35.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:35.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:35.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:35.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:35.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:35.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:35.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:35.765 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 83 16 77895 77879 58.6362 54.5 0.0632848 0.0170467 2024-09-02T10:25:35.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:35.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:35.879 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:35.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:36.004 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:36.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:36.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:36.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:36.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:36.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:36.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:36.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:36.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:36.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:36.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:36.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:36.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:36.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:36.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:36.655 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:36.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:36.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:36.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:36.766 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 84 16 78705 78689 58.5408 50.625 0.00335845 0.0170653 2024-09-02T10:25:36.778 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:36.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:36.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:36.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:36.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:36.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:36 smithi151 ceph-mon[16639]: pgmap v1632: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 704 op/s; 7024/226887 objects misplaced (3.096%); 21 MiB/s, 337 objects/s recovering 2024-09-02T10:25:37.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:36 smithi187 ceph-mon[20072]: pgmap v1632: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 704 op/s; 7024/226887 objects misplaced (3.096%); 21 MiB/s, 337 objects/s recovering 2024-09-02T10:25:37.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:37.035 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:37.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:37.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:36 smithi162 ceph-mon[20279]: pgmap v1632: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 44 MiB/s wr, 704 op/s; 7024/226887 objects misplaced (3.096%); 21 MiB/s, 337 objects/s recovering 2024-09-02T10:25:37.137 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:37.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:37.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:37.245 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:25:37.246 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:25:37.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:37.300 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:37.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:37.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:37.427 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:37.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:37.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:37.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:37.550 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:37.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:37.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:37.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:37.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:37.766 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 85 16 79741 79725 58.6137 64.75 0.028732 0.0170545 2024-09-02T10:25:37.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:37.811 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:37.812 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:37.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:37.936 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:37.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:38.038 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:38.061 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:38.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:38.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:38.200 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:38.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:38.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:38.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:38.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:38.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:38.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:38.453 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:38.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:38.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:38.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:38.580 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:38.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:38.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:38.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:38.703 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:38.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:38.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:38.766 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 86 16 80604 80588 58.5593 53.9375 0.123789 0.0170482 2024-09-02T10:25:38.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:38.831 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:38.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:38.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:38 smithi151 ceph-mon[16639]: pgmap v1633: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 734 op/s; 6565/231615 objects misplaced (2.834%); 18 MiB/s, 288 objects/s recovering 2024-09-02T10:25:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:38 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:25:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:38 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:25:38.962 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:38.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:38 smithi187 ceph-mon[20072]: pgmap v1633: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 734 op/s; 6565/231615 objects misplaced (2.834%); 18 MiB/s, 288 objects/s recovering 2024-09-02T10:25:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:38 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:25:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:38 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:25:39.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:39.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:39.093 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:39.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:38 smithi162 ceph-mon[20279]: pgmap v1633: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 46 MiB/s wr, 734 op/s; 6565/231615 objects misplaced (2.834%); 18 MiB/s, 288 objects/s recovering 2024-09-02T10:25:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:38 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:25:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:38 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:25:39.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:39.217 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:39.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:39.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:39.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:39.344 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:39.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:39.346 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:39.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:39.471 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:39.473 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:39.573 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:39.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:39.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:39.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:39.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:39.721 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:39.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:39.766 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 87 16 81543 81527 58.5607 58.6875 0.0134913 0.0170698 2024-09-02T10:25:39.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:39.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:39.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:39.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:39.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:39.970 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:40.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:39 smithi162 ceph-mon[20279]: osdmap e715: 12 total, 12 up, 12 in 2024-09-02T10:25:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:39 smithi162 ceph-mon[20279]: osdmap e716: 12 total, 12 up, 12 in 2024-09-02T10:25:40.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:40.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:40.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:39 smithi151 ceph-mon[16639]: osdmap e715: 12 total, 12 up, 12 in 2024-09-02T10:25:40.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:39 smithi151 ceph-mon[16639]: osdmap e716: 12 total, 12 up, 12 in 2024-09-02T10:25:40.201 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:40.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:40.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:40.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:39 smithi187 ceph-mon[20072]: osdmap e715: 12 total, 12 up, 12 in 2024-09-02T10:25:40.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:39 smithi187 ceph-mon[20072]: osdmap e716: 12 total, 12 up, 12 in 2024-09-02T10:25:40.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:40.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:40.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:40.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:40.481 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:40.482 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:40.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:40.611 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:40.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:40.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:40.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:40.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:40.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:40.766 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 88 16 82251 82235 58.398 44.25 0.00669963 0.0171088 2024-09-02T10:25:40.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:40.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:40.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:40.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:40.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:40.996 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:40.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:40 smithi162 ceph-mon[20279]: pgmap v1636: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s; 6250/239271 objects misplaced (2.612%); 17 MiB/s, 273 objects/s recovering 2024-09-02T10:25:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:25:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:40 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:40 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:41.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:41.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:41.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:40 smithi151 ceph-mon[16639]: pgmap v1636: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s; 6250/239271 objects misplaced (2.612%); 17 MiB/s, 273 objects/s recovering 2024-09-02T10:25:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:25:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:40 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:40 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:41.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:41.251 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:41.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:41.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:40 smithi187 ceph-mon[20072]: pgmap v1636: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 911 op/s; 6250/239271 objects misplaced (2.612%); 17 MiB/s, 273 objects/s recovering 2024-09-02T10:25:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:25:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:40 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:40 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:41.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:41.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:41.377 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:41.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:41.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:41.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:41.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:41.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:41.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:41.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:41.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:41.751 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:41.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:41.766 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 89 16 83040 83024 58.2959 49.3125 0.0176713 0.0171467 2024-09-02T10:25:41.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:41.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:41.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:41.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:42.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:42.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:42.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:42.121 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:42.146 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:42.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:42.147 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:42.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:42.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:42.273 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:42 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:42.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:42 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:42.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:42.397 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:42.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:42 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:42.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:42 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:42.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:42 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:42.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:42 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:42.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:42.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:42.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:42.658 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:42.659 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:42.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:42.766 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 90 9 83939 83930 58.2773 56.625 0.026961 0.0171524 2024-09-02T10:25:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:42.783 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:42.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:42.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:42.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:42.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:43.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:43.030 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:43.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:43.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:43.153 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:43.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:43.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:43.276 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:43.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:43 smithi162 ceph-mon[20279]: pgmap v1637: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 640 op/s; 5900/240072 objects misplaced (2.458%); 14 MiB/s, 225 objects/s recovering 2024-09-02T10:25:43.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:43.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:43.402 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:43 smithi151 ceph-mon[16639]: pgmap v1637: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 640 op/s; 5900/240072 objects misplaced (2.458%); 14 MiB/s, 225 objects/s recovering 2024-09-02T10:25:43.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:43 smithi187 ceph-mon[20072]: pgmap v1637: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 640 op/s; 5900/240072 objects misplaced (2.458%); 14 MiB/s, 225 objects/s recovering 2024-09-02T10:25:43.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:43.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:43.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:43.627 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:43.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:43.650 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:43.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:43.767 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total time run: 90.669 2024-09-02T10:25:43.767 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total writes made: 83939 2024-09-02T10:25:43.767 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Write size: 65536 2024-09-02T10:25:43.767 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object size: 65536 2024-09-02T10:25:43.767 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Bandwidth (MB/sec): 57.8609 2024-09-02T10:25:43.767 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Bandwidth: 26.8663 2024-09-02T10:25:43.767 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max bandwidth (MB/sec): 134.812 2024-09-02T10:25:43.767 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min bandwidth (MB/sec): 0 2024-09-02T10:25:43.768 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average IOPS: 925 2024-09-02T10:25:43.768 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev IOPS: 429.861 2024-09-02T10:25:43.768 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max IOPS: 2157 2024-09-02T10:25:43.768 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min IOPS: 0 2024-09-02T10:25:43.768 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average Latency(s): 0.0171937 2024-09-02T10:25:43.768 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Latency(s): 0.0617802 2024-09-02T10:25:43.768 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max latency(s): 2.89813 2024-09-02T10:25:43.768 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min latency(s): 0.00108091 2024-09-02T10:25:43.773 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Cleaning up (deleting benchmark objects) 2024-09-02T10:25:43.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:43.776 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:43.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:43.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:43.904 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:43.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:43.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:44.006 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:44.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:44.038 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:44.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:44.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:44.162 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:44.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:44.264 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:44.291 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:44.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:44.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:44.393 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:44.419 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:44.420 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:44.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:44.542 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:44.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:44.644 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:44.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:44.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:44.669 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:44.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:44.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:44.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:44.893 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:44.915 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:44.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:45.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:45.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:45.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:45.151 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:45.174 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:45.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:45.276 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:45.299 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:45.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:45 smithi162 ceph-mon[20279]: pgmap v1638: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s; 5172/248463 objects misplaced (2.082%); 16 MiB/s, 253 objects/s recovering 2024-09-02T10:25:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:45 smithi162 ceph-mon[20279]: osdmap e717: 12 total, 12 up, 12 in 2024-09-02T10:25:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:45 smithi162 ceph-mon[20279]: osdmap e718: 12 total, 12 up, 12 in 2024-09-02T10:25:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:45 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:45 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:45.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:45.423 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:45.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:45.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:45 smithi151 ceph-mon[16639]: pgmap v1638: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s; 5172/248463 objects misplaced (2.082%); 16 MiB/s, 253 objects/s recovering 2024-09-02T10:25:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:45 smithi151 ceph-mon[16639]: osdmap e717: 12 total, 12 up, 12 in 2024-09-02T10:25:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:45 smithi151 ceph-mon[16639]: osdmap e718: 12 total, 12 up, 12 in 2024-09-02T10:25:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:45 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:45 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:45 smithi187 ceph-mon[20072]: pgmap v1638: 47 pgs: 1 active+remapped+backfilling, 3 active+recovering+undersized+remapped, 43 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 898 op/s; 5172/248463 objects misplaced (2.082%); 16 MiB/s, 253 objects/s recovering 2024-09-02T10:25:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:45 smithi187 ceph-mon[20072]: osdmap e717: 12 total, 12 up, 12 in 2024-09-02T10:25:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:45 smithi187 ceph-mon[20072]: osdmap e718: 12 total, 12 up, 12 in 2024-09-02T10:25:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:45 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:45 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:45.525 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:45.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:45.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:45.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:45.672 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:45.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:45.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:45.774 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:45.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:45.799 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:45.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:45.921 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:45.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:45.922 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:46.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:46.046 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:46.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:46.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:46.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:46.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:46.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:46.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:46.296 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:46.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:46.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:46.420 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:46.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:46.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:46.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:46.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:46.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:46.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:46.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:46.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:46.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:46.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:46.815 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:46.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:46.917 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:46.941 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:46.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:47.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:47.067 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:47.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:47.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:47.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:47.190 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:47.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:47.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:47.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:47.327 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:47.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:47.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:47 smithi162 ceph-mon[20279]: pgmap v1641: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 44 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 660 op/s; 4434/251253 objects misplaced (1.765%); 19 MiB/s, 303 objects/s recovering 2024-09-02T10:25:47.429 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:47.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:47 smithi151 ceph-mon[16639]: pgmap v1641: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 44 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 660 op/s; 4434/251253 objects misplaced (1.765%); 19 MiB/s, 303 objects/s recovering 2024-09-02T10:25:47.452 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:47.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:47.453 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:47 smithi187 ceph-mon[20072]: pgmap v1641: 47 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 44 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 660 op/s; 4434/251253 objects misplaced (1.765%); 19 MiB/s, 303 objects/s recovering 2024-09-02T10:25:47.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:47.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:47.581 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:47.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:47.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:47.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:47.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:47.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:47.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:47.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:47.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:47.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:47.982 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:47.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:48.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:48.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:48.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:48.221 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:48.243 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:48.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:48.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:48.346 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:48.370 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:48.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:48.371 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:48.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:48.495 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:48.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:48.496 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:48.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:48.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:48.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:48.622 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:48.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:48.754 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:48.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:48.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:48.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:48.878 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:48.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:48.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:49.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:49.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:49.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:49.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:49.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:49.128 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:49.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:49.261 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:49.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:49 smithi162 ceph-mon[20279]: pgmap v1642: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 33 MiB/s wr, 683 op/s; 4313/247920 objects misplaced (1.740%); 16 MiB/s, 251 objects/s recovering 2024-09-02T10:25:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:49 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:49 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:25:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:25:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:49 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:49.363 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:49.389 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:49.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:49.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:49 smithi151 ceph-mon[16639]: pgmap v1642: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 33 MiB/s wr, 683 op/s; 4313/247920 objects misplaced (1.740%); 16 MiB/s, 251 objects/s recovering 2024-09-02T10:25:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:49 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:49 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:25:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:25:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:49 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:49.491 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:49 smithi187 ceph-mon[20072]: pgmap v1642: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 33 MiB/s wr, 683 op/s; 4313/247920 objects misplaced (1.740%); 16 MiB/s, 251 objects/s recovering 2024-09-02T10:25:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:49 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:49 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:25:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:25:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:49 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:25:49.516 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:49.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:49.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:49.618 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:49.643 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:49.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:49.745 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:49.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:49.773 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:49.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:49.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:49.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:25:49.914 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:49.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:50.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:50.038 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:50.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:50.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:50.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:50.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:50.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:50.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:25:50.198 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:50.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:50.321 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:50.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:50.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:50.444 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:50.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:50.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:50.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:50.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:50.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:50.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:50.695 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:50.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:50.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:50.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:50.820 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:50.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:50.894 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:50.902 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:50.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:50.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:50.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:50.949 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:51.042 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-09-02T10:25:51.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:51.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:51 smithi151 ceph-mon[16639]: pgmap v1643: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 45 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 31 MiB/s wr, 970 op/s; 3573/240279 objects misplaced (1.487%); 17 MiB/s, 269 objects/s recovering 2024-09-02T10:25:51.085 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2868265466' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:25:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:51.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:51.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:51.192 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:51.218 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:51.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:51.320 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:51 smithi162 ceph-mon[20279]: pgmap v1643: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 45 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 31 MiB/s wr, 970 op/s; 3573/240279 objects misplaced (1.487%); 17 MiB/s, 269 objects/s recovering 2024-09-02T10:25:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2868265466' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:25:51.347 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:51.348 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:51.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:51.472 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:51.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:51 smithi187 ceph-mon[20072]: pgmap v1643: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 45 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 31 MiB/s wr, 970 op/s; 3573/240279 objects misplaced (1.487%); 17 MiB/s, 269 objects/s recovering 2024-09-02T10:25:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2868265466' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:25:51.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:51.596 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:51.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:51.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:51.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:51.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:51.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:51.724 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:51.738 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:25:51.738 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:25:51.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:51.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:51.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:51.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:51.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:51.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:51.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:52.061 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:25:52.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:52.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:52.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:52.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:52.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:52.243 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:52.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:52.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2868265466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:25:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:52 smithi162 ceph-mon[20279]: osdmap e719: 12 total, 12 up, 12 in 2024-09-02T10:25:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/675195914' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:52 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:52 smithi162 ceph-mon[20279]: pgmap v1645: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 45 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 138 B/s rd, 2.5 MiB/s wr, 603 op/s; 3573/239268 objects misplaced (1.493%); 9.8 MiB/s, 156 objects/s recovering 2024-09-02T10:25:52.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:52.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:52.407 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2868265466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:25:52.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:52 smithi187 ceph-mon[20072]: osdmap e719: 12 total, 12 up, 12 in 2024-09-02T10:25:52.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/675195914' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:52.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:52 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:52.408 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:52 smithi187 ceph-mon[20072]: pgmap v1645: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 45 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 138 B/s rd, 2.5 MiB/s wr, 603 op/s; 3573/239268 objects misplaced (1.493%); 9.8 MiB/s, 156 objects/s recovering 2024-09-02T10:25:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2868265466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:25:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:52 smithi151 ceph-mon[16639]: osdmap e719: 12 total, 12 up, 12 in 2024-09-02T10:25:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/675195914' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:52 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:25:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:52 smithi151 ceph-mon[16639]: pgmap v1645: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 45 active+clean; 4.9 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 138 B/s rd, 2.5 MiB/s wr, 603 op/s; 3573/239268 objects misplaced (1.493%); 9.8 MiB/s, 156 objects/s recovering 2024-09-02T10:25:52.466 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:52.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:52.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:52.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:52.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:52.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:52.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:52.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:52.974 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:53.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:25:53.105 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:53.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:53.106 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:25:53.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:53.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:53.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:53 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:53 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:25:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:53 smithi187 ceph-mon[20072]: osdmap e720: 12 total, 12 up, 12 in 2024-09-02T10:25:53.281 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:53 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:53.281 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:53 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:25:53.281 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:53 smithi151 ceph-mon[16639]: osdmap e720: 12 total, 12 up, 12 in 2024-09-02T10:25:53.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:53.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:53 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:25:53.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:53 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:25:53.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:53 smithi162 ceph-mon[20279]: osdmap e720: 12 total, 12 up, 12 in 2024-09-02T10:25:53.370 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:53.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:53.371 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:53.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 10 1 2024-09-02T10:25:53.459 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:25:53.459 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:25:53.459 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:25:53.459 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:25:53.460 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:25:53.460 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:25:53.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:25:53.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:25:53.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:25:53.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:25:53.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:25:53.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:25:53.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:53.490 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:25:53.490 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:25:53.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:53.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:53.523 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:25:53.523 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:25:53.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:25:53.559 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:25:53.589 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:25:53.590 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:25:53.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:53.622 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:25:53.623 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:25:53.634 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:53.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:53.636 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:53.655 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:25:53.655 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:25:53.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:53.766 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:53.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:53.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:53.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:53.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:53.895 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:53.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:54.019 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:54.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:54.121 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:54.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:54.149 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:54.250 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:54.271 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:54.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1443848802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:54 smithi162 ceph-mon[20279]: pgmap v1647: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 45 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 1007 KiB/s wr, 974 op/s; 3216/228606 objects misplaced (1.407%); 7.6 MiB/s, 121 objects/s recovering 2024-09-02T10:25:54.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:54.396 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:54.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1443848802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:54 smithi151 ceph-mon[16639]: pgmap v1647: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 45 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 1007 KiB/s wr, 974 op/s; 3216/228606 objects misplaced (1.407%); 7.6 MiB/s, 121 objects/s recovering 2024-09-02T10:25:54.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1443848802' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:25:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:54 smithi187 ceph-mon[20072]: pgmap v1647: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+remapped+backfilling, 45 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 1007 KiB/s wr, 974 op/s; 3216/228606 objects misplaced (1.407%); 7.6 MiB/s, 121 objects/s recovering 2024-09-02T10:25:54.520 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:54.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:54.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:54.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:54.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:54.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:54.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:54.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:54.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:54.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:54.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:54.917 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:54.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:55.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:55.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:55.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:55.143 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:55.164 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:55.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:55.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:55.290 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:55.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:55.392 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:55.415 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:55.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:55.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:55.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:55.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:55.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:55.659 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:55.681 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:55.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:55.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:55.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:55.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:55.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:55.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:55.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:55.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:56.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:56.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:25:56.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:56.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:56.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:56.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:56.210 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:56.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:56.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:56.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:56.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:56.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:56.469 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:56.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:56.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:56.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:56.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:56.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:56.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:56.701 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:56.721 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:56.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:56.725 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:56 smithi187 ceph-mon[20072]: pgmap v1648: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s wr, 996 op/s; 2620/224046 objects misplaced (1.169%); 9.7 MiB/s, 155 objects/s recovering 2024-09-02T10:25:56.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:56 smithi151 ceph-mon[16639]: pgmap v1648: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s wr, 996 op/s; 2620/224046 objects misplaced (1.169%); 9.7 MiB/s, 155 objects/s recovering 2024-09-02T10:25:56.768 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:25:56.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:56.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:56 smithi162 ceph-mon[20279]: pgmap v1648: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.6 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s wr, 996 op/s; 2620/224046 objects misplaced (1.169%); 9.7 MiB/s, 155 objects/s recovering 2024-09-02T10:25:56.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:56.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:56.953 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:56.977 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:56.978 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:57.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:57.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:57.101 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:57.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:25:57.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:57.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:57.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:57.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:57.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:57.344 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:57.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:57.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:57.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:57.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:57.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:57.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:57.621 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:57.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:57.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:57.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:25:57.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:25:57.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:57.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:25:57.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:57.872 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:57.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:57.924 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:25:57.958 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:25:57.958 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:25:57.958 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:25:57.959 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:25:57.959 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:25:57.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:57.995 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:57.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:58.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:58.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:58.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:58.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:58.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:25:58.264 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:58.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:25:58.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:58.390 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:58.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:58.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:58.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:25:58.518 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:58.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:25:58.523 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-09-02T10:25:58.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:58.648 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:58.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:58.649 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:58.750 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:58.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:58.776 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:58 smithi162 ceph-mon[20279]: pgmap v1649: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 883 op/s; 2620/219063 objects misplaced (1.196%); 5.9 MiB/s, 93 objects/s recovering 2024-09-02T10:25:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1522953882' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:25:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:58 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:25:58.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:25:58.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:58.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:25:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:58 smithi151 ceph-mon[16639]: pgmap v1649: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 883 op/s; 2620/219063 objects misplaced (1.196%); 5.9 MiB/s, 93 objects/s recovering 2024-09-02T10:25:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1522953882' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:25:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:58 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:25:59.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:25:59.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:58 smithi187 ceph-mon[20072]: pgmap v1649: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.5 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 883 op/s; 2620/219063 objects misplaced (1.196%); 5.9 MiB/s, 93 objects/s recovering 2024-09-02T10:25:59.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1522953882' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:25:59.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:58 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:25:59.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:59.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:25:59.131 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:25:59.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:59.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:25:59.259 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:59.284 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:59.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:59.285 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:59.386 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:25:59.408 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:59.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:25:59.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:25:59.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:25:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:59.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:25:59.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:25:59.662 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:59.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:59.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:25:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:59 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:25:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:59 smithi151 ceph-mon[16639]: osdmap e721: 12 total, 12 up, 12 in 2024-09-02T10:25:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:25:59 smithi151 ceph-mon[16639]: osdmap e722: 12 total, 12 up, 12 in 2024-09-02T10:25:59.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:25:59.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:25:59.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:59.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:25:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:59 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:25:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:59 smithi162 ceph-mon[20279]: osdmap e721: 12 total, 12 up, 12 in 2024-09-02T10:25:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:25:59 smithi162 ceph-mon[20279]: osdmap e722: 12 total, 12 up, 12 in 2024-09-02T10:25:59.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:25:59.911 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:25:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:25:59.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:59 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:26:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:59 smithi187 ceph-mon[20072]: osdmap e721: 12 total, 12 up, 12 in 2024-09-02T10:26:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:25:59 smithi187 ceph-mon[20072]: osdmap e722: 12 total, 12 up, 12 in 2024-09-02T10:26:00.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:00.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:00.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:00.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:00.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:00.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:00.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:00.302 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:00.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:00.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:00.426 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:00.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:00.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-09-02T10:26:00.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:00.550 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:00.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:00.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:00.675 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:00.676 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:00 smithi151 ceph-mon[16639]: pgmap v1652: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 1725/211320 objects misplaced (0.816%); 12 MiB/s, 198 objects/s recovering 2024-09-02T10:26:00.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:00.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:00.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:00.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:00 smithi162 ceph-mon[20279]: pgmap v1652: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 1725/211320 objects misplaced (0.816%); 12 MiB/s, 198 objects/s recovering 2024-09-02T10:26:00.920 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:00.945 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:00.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:00 smithi187 ceph-mon[20072]: pgmap v1652: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.26k op/s; 1725/211320 objects misplaced (0.816%); 12 MiB/s, 198 objects/s recovering 2024-09-02T10:26:01.047 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:01.071 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:01.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:01.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:01.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:01.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:01.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:26:01.289 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:01.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:01.343 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:01.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:01.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:01.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:01.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:01.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:01.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:01.529 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:26:01.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:01.618 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:01.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:01.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:01.741 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:01.742 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:01.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:01.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3784580455' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:26:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:01.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:01.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:01.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3784580455' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:26:01.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:01.992 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:01.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3784580455' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:26:02.096 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:02.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:02.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:02.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:02.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:02.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:02.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:02.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:02.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:02.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:02.488 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:02.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:02.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:02.591 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:02.619 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:02.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:02.620 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:02.639 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:02 smithi151 ceph-mon[16639]: pgmap v1653: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 751 op/s; 1725/210570 objects misplaced (0.819%); 8.7 MiB/s, 138 objects/s recovering 2024-09-02T10:26:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:02 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3784580455' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:26:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:02 smithi151 ceph-mon[16639]: osdmap e723: 12 total, 12 up, 12 in 2024-09-02T10:26:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:02 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:26:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:02 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:26:02.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:02.745 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:02.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:02 smithi162 ceph-mon[20279]: pgmap v1653: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 751 op/s; 1725/210570 objects misplaced (0.819%); 8.7 MiB/s, 138 objects/s recovering 2024-09-02T10:26:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:02 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3784580455' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:26:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:02 smithi162 ceph-mon[20279]: osdmap e723: 12 total, 12 up, 12 in 2024-09-02T10:26:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:02 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:26:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:02 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:26:02.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:02.869 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:02.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:02.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:02.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:02.999 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:02 smithi187 ceph-mon[20072]: pgmap v1653: 47 pgs: 1 active+remapped+backfilling, 1 active+recovering+undersized+remapped, 45 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 751 op/s; 1725/210570 objects misplaced (0.819%); 8.7 MiB/s, 138 objects/s recovering 2024-09-02T10:26:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:02 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3784580455' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:26:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:02 smithi187 ceph-mon[20072]: osdmap e723: 12 total, 12 up, 12 in 2024-09-02T10:26:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:02 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:26:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:02 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:26:03.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:03.137 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:03.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:03.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:03.239 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:03.263 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:03.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:03.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:03.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:03.389 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:03.390 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:03.441 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:26:03.441 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:26:03.476 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:26:03.477 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:26:03.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:03.508 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:26:03.509 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:26:03.516 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:03.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:03.517 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:03.544 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:26:03.545 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:26:03.579 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:26:03.580 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:26:03.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:03 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:26:03.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:03 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:26:03.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:03 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:26:03.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:03 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:26:03.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:03 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:26:03.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:03 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:26:03.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:03 smithi151 ceph-mon[16639]: osdmap e724: 12 total, 12 up, 12 in 2024-09-02T10:26:03.580 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/272277537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:03.581 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:03 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:26:03.581 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:03 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:26:03.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:03.619 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:26:03.619 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:26:03.641 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:03.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:03.656 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:26:03.657 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:26:03.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:03.767 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:03.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:03.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:03 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:26:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:03 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:26:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:03 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:26:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:03 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:26:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:03 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:26:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:03 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:26:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:03 smithi162 ceph-mon[20279]: osdmap e724: 12 total, 12 up, 12 in 2024-09-02T10:26:03.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/272277537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:03.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:03 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:26:03.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:03 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:26:03.868 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:03.892 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:03.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:03.897 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:03 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:26:03.898 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:03 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:26:03.898 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:03 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:26:03.898 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:03 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:26:03.898 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:03 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:26:03.898 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:03 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:26:03.898 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:03 smithi187 ceph-mon[20072]: osdmap e724: 12 total, 12 up, 12 in 2024-09-02T10:26:03.898 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/272277537' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:03.898 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:03 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:26:03.898 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:03 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:26:03.989 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:26:03.990 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:26:03.994 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:04.023 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:26:04.023 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:26:04.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:04.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:04.056 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:26:04.056 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:26:04.089 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:26:04.090 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:26:04.123 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:26:04.124 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:26:04.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:04.167 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:04.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:04.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:04.290 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:04.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:04.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:26:04.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:04.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:04.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:04.414 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:04.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:04.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:04.545 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:04.546 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-09-02T10:26:04.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:04.672 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:04.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:04.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:04.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:04 smithi151 ceph-mon[16639]: pgmap v1656: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s; 1494/202377 objects misplaced (0.738%); 8.1 MiB/s, 129 objects/s recovering 2024-09-02T10:26:04.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:04 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4122538485' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:26:04.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:04 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:26:04.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:04.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:04.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:04 smithi162 ceph-mon[20279]: pgmap v1656: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s; 1494/202377 objects misplaced (0.738%); 8.1 MiB/s, 129 objects/s recovering 2024-09-02T10:26:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:04 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4122538485' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:26:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:04 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:26:04.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:04.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:04.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:04.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:04 smithi187 ceph-mon[20072]: pgmap v1656: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 841 op/s; 1494/202377 objects misplaced (0.738%); 8.1 MiB/s, 129 objects/s recovering 2024-09-02T10:26:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:04 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4122538485' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:26:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:04 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:26:05.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:05.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:05.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:05.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:05.192 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:05.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:05.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:05.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:05.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:05.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:05.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:05.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:05.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:05.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:05.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:05.588 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:05.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:05.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:05.711 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:05.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:05.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:05 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:26:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:05 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:26:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:05 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:26:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:05 smithi162 ceph-mon[20279]: osdmap e725: 12 total, 12 up, 12 in 2024-09-02T10:26:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:05 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:05 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:05.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:05 smithi162 ceph-mon[20279]: 7.8 scrub starts 2024-09-02T10:26:05.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:05.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:05.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:05.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:05 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:26:05.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:05 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:26:05.924 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:05 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:26:05.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:05 smithi151 ceph-mon[16639]: osdmap e725: 12 total, 12 up, 12 in 2024-09-02T10:26:05.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:05 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:05.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:05 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:05.925 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:05 smithi151 ceph-mon[16639]: 7.8 scrub starts 2024-09-02T10:26:05.925 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:26:05.925 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:26:05.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:05.972 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:05.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:05 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:26:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:05 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:26:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:05 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:26:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:05 smithi187 ceph-mon[20072]: osdmap e725: 12 total, 12 up, 12 in 2024-09-02T10:26:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:05 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:06.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:05 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:06.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:05 smithi187 ceph-mon[20072]: 7.8 scrub starts 2024-09-02T10:26:06.073 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:06.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:06.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:06.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:06.228 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:06.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:06.330 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:06.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:06.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:06.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:06.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:06.474 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:06.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:06.475 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:06.576 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:06.605 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:06.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:06.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:06.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:06.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:06.737 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:06.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:06 smithi162 ceph-mon[20279]: 7.5 scrub starts 2024-09-02T10:26:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:06 smithi162 ceph-mon[20279]: pgmap v1658: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 968/198588 objects misplaced (0.487%); 7.1 MiB/s, 113 objects/s recovering 2024-09-02T10:26:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:06 smithi162 ceph-mon[20279]: osdmap e726: 12 total, 12 up, 12 in 2024-09-02T10:26:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:06 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:26:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:06 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:26:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:06 smithi162 ceph-mon[20279]: 7.8 scrub ok 2024-09-02T10:26:06.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:06.869 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:06 smithi151 ceph-mon[16639]: 7.5 scrub starts 2024-09-02T10:26:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:06 smithi151 ceph-mon[16639]: pgmap v1658: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 968/198588 objects misplaced (0.487%); 7.1 MiB/s, 113 objects/s recovering 2024-09-02T10:26:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:06 smithi151 ceph-mon[16639]: osdmap e726: 12 total, 12 up, 12 in 2024-09-02T10:26:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:06 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:26:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:06 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:26:06.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:06 smithi151 ceph-mon[16639]: 7.8 scrub ok 2024-09-02T10:26:06.971 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:06.996 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:06.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:06 smithi187 ceph-mon[20072]: 7.5 scrub starts 2024-09-02T10:26:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:06 smithi187 ceph-mon[20072]: pgmap v1658: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 968/198588 objects misplaced (0.487%); 7.1 MiB/s, 113 objects/s recovering 2024-09-02T10:26:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:06 smithi187 ceph-mon[20072]: osdmap e726: 12 total, 12 up, 12 in 2024-09-02T10:26:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:06 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:26:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:06 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:26:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:06 smithi187 ceph-mon[20072]: 7.8 scrub ok 2024-09-02T10:26:07.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:07.122 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:07.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:07.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:07.258 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:07.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:07.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:07.386 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:07.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:07.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:07.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:07.512 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:07.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:07.513 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:07.614 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:07.637 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:07.638 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:07.739 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:07.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:07.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:07 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:26:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:07 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:26:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:07 smithi162 ceph-mon[20279]: 7.5 scrub ok 2024-09-02T10:26:07.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:07 smithi162 ceph-mon[20279]: 7.3 scrub starts 2024-09-02T10:26:07.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:07.886 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:07.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:07 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:26:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:07 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:26:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:07 smithi151 ceph-mon[16639]: 7.5 scrub ok 2024-09-02T10:26:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:07 smithi151 ceph-mon[16639]: 7.3 scrub starts 2024-09-02T10:26:07.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:07 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:26:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:07 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:26:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:07 smithi187 ceph-mon[20072]: 7.5 scrub ok 2024-09-02T10:26:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:07 smithi187 ceph-mon[20072]: 7.3 scrub starts 2024-09-02T10:26:08.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:08.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:08.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:08.123 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:08.147 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:08.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:08.148 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:08.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:08.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:08.276 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:08.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:08.400 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:08.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:08.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:08.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:08.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:08.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:08.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:08.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:08.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:08.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:08.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:08.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:08.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:08 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:26:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:08 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:26:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:08 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:26:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:08 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:26:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:08 smithi162 ceph-mon[20279]: pgmap v1660: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s; 968/193731 objects misplaced (0.500%); 7.2 MiB/s, 114 objects/s recovering 2024-09-02T10:26:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:08 smithi162 ceph-mon[20279]: 7.3 scrub ok 2024-09-02T10:26:08.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:08 smithi162 ceph-mon[20279]: 7.a scrub starts 2024-09-02T10:26:08.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:08.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:08.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:08 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:26:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:08 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:26:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:08 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:26:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:08 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:26:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:08 smithi151 ceph-mon[16639]: pgmap v1660: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s; 968/193731 objects misplaced (0.500%); 7.2 MiB/s, 114 objects/s recovering 2024-09-02T10:26:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:08 smithi151 ceph-mon[16639]: 7.3 scrub ok 2024-09-02T10:26:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:08 smithi151 ceph-mon[16639]: 7.a scrub starts 2024-09-02T10:26:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:08 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:26:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:08 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:26:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:08 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:26:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:08 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:26:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:08 smithi187 ceph-mon[20072]: pgmap v1660: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 947 op/s; 968/193731 objects misplaced (0.500%); 7.2 MiB/s, 114 objects/s recovering 2024-09-02T10:26:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:08 smithi187 ceph-mon[20072]: 7.3 scrub ok 2024-09-02T10:26:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:08 smithi187 ceph-mon[20072]: 7.a scrub starts 2024-09-02T10:26:09.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:09.046 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:09.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:09.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:09.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:09.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:09.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:09.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:09.311 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:09.312 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:09.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:09.436 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:09.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:09.538 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:09.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:09.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:09.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:09.690 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:09.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:09.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:09.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:09.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:09 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:26:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:09 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:26:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:09 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:09 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:09 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:26:09.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:09 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:26:09.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:09 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:26:09.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:09 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:26:09.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:09 smithi162 ceph-mon[20279]: 7.2 scrub starts 2024-09-02T10:26:09.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:09 smithi162 ceph-mon[20279]: 7.a scrub ok 2024-09-02T10:26:09.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:09 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:26:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:09 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:26:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:09 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:09 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:09 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:26:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:09 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:26:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:09 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:26:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:09 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:26:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:09 smithi151 ceph-mon[16639]: 7.2 scrub starts 2024-09-02T10:26:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:09 smithi151 ceph-mon[16639]: 7.a scrub ok 2024-09-02T10:26:09.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:09.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:09.950 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:09 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:26:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:09 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:26:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:09 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:09 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:09 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:26:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:09 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:26:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:09 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:26:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:09 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:26:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:09 smithi187 ceph-mon[20072]: 7.2 scrub starts 2024-09-02T10:26:10.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:09 smithi187 ceph-mon[20072]: 7.a scrub ok 2024-09-02T10:26:10.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:10.078 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:10.079 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:10.181 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:10.204 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:10.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:10.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:10.336 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:10.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:10.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:10.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:10.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:26:10.466 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:10.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:10.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:10.598 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:10.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: 7.1 scrub starts 2024-09-02T10:26:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:26:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:26:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: pgmap v1661: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 959 op/s; 359/186108 objects misplaced (0.193%); 11 MiB/s, 180 objects/s recovering 2024-09-02T10:26:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:26:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:26:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: osdmap e727: 12 total, 12 up, 12 in 2024-09-02T10:26:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:26:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:26:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:26:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:26:10.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:10 smithi151 ceph-mon[16639]: 7.2 scrub ok 2024-09-02T10:26:10.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:10.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:10.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:10.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:10.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: 7.1 scrub starts 2024-09-02T10:26:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:26:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:26:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: pgmap v1661: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 959 op/s; 359/186108 objects misplaced (0.193%); 11 MiB/s, 180 objects/s recovering 2024-09-02T10:26:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:26:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:26:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: osdmap e727: 12 total, 12 up, 12 in 2024-09-02T10:26:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:26:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:26:10.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:26:10.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:26:10.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:10 smithi162 ceph-mon[20279]: 7.2 scrub ok 2024-09-02T10:26:10.858 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:10.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:10.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:10.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:10.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: 7.1 scrub starts 2024-09-02T10:26:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:26:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:26:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: pgmap v1661: 47 pgs: 1 active+remapped+backfilling, 46 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 959 op/s; 359/186108 objects misplaced (0.193%); 11 MiB/s, 180 objects/s recovering 2024-09-02T10:26:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:26:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:26:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: osdmap e727: 12 total, 12 up, 12 in 2024-09-02T10:26:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:26:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:26:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:26:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:26:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:10 smithi187 ceph-mon[20072]: 7.2 scrub ok 2024-09-02T10:26:11.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 2 1 2024-09-02T10:26:11.070 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:11.081 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:11.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:11.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:11.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:11.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:11.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:11.248 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:11.249 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:11.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:11.382 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:11.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:11.383 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:11.484 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:11.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:11.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:11.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:11.644 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:11.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:11.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:11.746 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:11.770 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:11.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:11 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:26:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:11 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:26:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:11 smithi162 ceph-mon[20279]: 7.7 scrub starts 2024-09-02T10:26:11.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:11.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:11.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:11.931 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:11 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:26:11.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:11 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:26:11.932 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:11 smithi151 ceph-mon[16639]: 7.7 scrub starts 2024-09-02T10:26:12.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:12.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:11 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:26:12.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:11 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:26:12.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:11 smithi187 ceph-mon[20072]: 7.7 scrub starts 2024-09-02T10:26:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:12.029 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:12.030 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:12.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:12.168 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:12.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:12.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:12.213 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:26:12.213 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:26:12.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:12.300 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:12.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:12.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:12.402 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:12.430 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:12.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:12.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:12.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:12.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:12.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:12.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:12.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:12.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:12.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:12.799 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:12.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:12.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: 7.1 scrub ok 2024-09-02T10:26:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: 7.7 scrub ok 2024-09-02T10:26:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:26:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:26:12.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: osdmap e728: 12 total, 12 up, 12 in 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: 7.d scrub starts 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2623512205' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:26:12.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:26:12.847 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:12 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:12.926 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:12.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: 7.1 scrub ok 2024-09-02T10:26:12.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: 7.7 scrub ok 2024-09-02T10:26:12.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:12.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:12.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: osdmap e728: 12 total, 12 up, 12 in 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: 7.d scrub starts 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2623512205' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:12.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:26:12.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:26:12.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:12 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:12.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:12.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:12.953 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: 7.1 scrub ok 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: 7.7 scrub ok 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: osdmap e728: 12 total, 12 up, 12 in 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: 7.d scrub starts 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:13.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:13.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:13.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2623512205' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:13.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:13.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:13.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:26:13.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:26:13.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:12 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:26:13.043 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:26:13.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:13.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:26:13.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:13.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:13.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:13.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:13.213 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:13.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:13.316 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:13.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:13.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:13.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:13.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:13.466 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:13.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:13.467 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:13.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:13.590 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:13.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:13 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:26:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:13 smithi151 ceph-mon[16639]: 7.c scrub starts 2024-09-02T10:26:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:13 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:26:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:13 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:26:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:13 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:26:13.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:13 smithi151 ceph-mon[16639]: osdmap e729: 12 total, 12 up, 12 in 2024-09-02T10:26:13.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:13.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:13.713 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:13.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:13.815 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:13.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:13.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:13.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:13 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:26:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:13 smithi162 ceph-mon[20279]: 7.c scrub starts 2024-09-02T10:26:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:13 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:26:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:13 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:26:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:13 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:26:13.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:13 smithi162 ceph-mon[20279]: osdmap e729: 12 total, 12 up, 12 in 2024-09-02T10:26:13.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:13.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:13.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:13 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:26:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:13 smithi187 ceph-mon[20072]: 7.c scrub starts 2024-09-02T10:26:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:13 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:26:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:13 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:26:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:13 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:26:14.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:13 smithi187 ceph-mon[20072]: osdmap e729: 12 total, 12 up, 12 in 2024-09-02T10:26:14.049 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-09-02T10:26:14.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:14.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:14.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:14.204 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:14.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:14.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:14.347 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:14.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:14.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:14.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:14.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:14.505 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:14.606 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:14.632 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:14.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:14 smithi151 ceph-mon[16639]: 7.c scrub ok 2024-09-02T10:26:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:14 smithi151 ceph-mon[16639]: pgmap v1666: 47 pgs: 1 peering, 2 active+clean+scrubbing, 44 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 9.5 MiB/s, 152 objects/s recovering 2024-09-02T10:26:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:14 smithi151 ceph-mon[16639]: 7.d scrub ok 2024-09-02T10:26:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:14 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:26:14.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:14 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:26:14.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2767579582' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:26:14.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:14 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:26:14.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:14 smithi151 ceph-mon[16639]: 7.6 scrub starts 2024-09-02T10:26:14.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:14 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:26:14.690 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:14 smithi151 ceph-mon[16639]: osdmap e730: 12 total, 12 up, 12 in 2024-09-02T10:26:14.734 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:14.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:14.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:14 smithi162 ceph-mon[20279]: 7.c scrub ok 2024-09-02T10:26:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:14 smithi162 ceph-mon[20279]: pgmap v1666: 47 pgs: 1 peering, 2 active+clean+scrubbing, 44 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 9.5 MiB/s, 152 objects/s recovering 2024-09-02T10:26:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:14 smithi162 ceph-mon[20279]: 7.d scrub ok 2024-09-02T10:26:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:14 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:26:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:14 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:26:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2767579582' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:26:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:14 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:26:14.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:14 smithi162 ceph-mon[20279]: 7.6 scrub starts 2024-09-02T10:26:14.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:14 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:26:14.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:14 smithi162 ceph-mon[20279]: osdmap e730: 12 total, 12 up, 12 in 2024-09-02T10:26:14.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:14.884 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:14.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:14.986 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:14 smithi187 ceph-mon[20072]: 7.c scrub ok 2024-09-02T10:26:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:14 smithi187 ceph-mon[20072]: pgmap v1666: 47 pgs: 1 peering, 2 active+clean+scrubbing, 44 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.09k op/s; 9.5 MiB/s, 152 objects/s recovering 2024-09-02T10:26:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:14 smithi187 ceph-mon[20072]: 7.d scrub ok 2024-09-02T10:26:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:14 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:26:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:14 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:26:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2767579582' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:26:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:14 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:26:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:14 smithi187 ceph-mon[20072]: 7.6 scrub starts 2024-09-02T10:26:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:14 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:26:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:14 smithi187 ceph-mon[20072]: osdmap e730: 12 total, 12 up, 12 in 2024-09-02T10:26:15.008 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:15.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:15.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:26:15.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:15.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:15.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:15.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:15.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:15.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:15.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:15.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:15.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:15.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:15.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:15.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:15.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:15.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:15.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:15.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:15.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:15.698 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:15.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:15.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:15.736 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-02T10:26:15.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:26:15.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:15.825 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:15.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:15.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: 7.6 scrub ok 2024-09-02T10:26:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:26:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:26:15.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:26:15.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:26:15.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:26:15.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:26:15.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: osdmap e731: 12 total, 12 up, 12 in 2024-09-02T10:26:15.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:26:15.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:15 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:26:15.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: 7.6 scrub ok 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: osdmap e731: 12 total, 12 up, 12 in 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:26:15.865 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:15 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:26:15.928 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:15.953 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:15.954 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:16.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: 7.6 scrub ok 2024-09-02T10:26:16.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:16.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:16.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:26:16.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:26:16.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:26:16.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:26:16.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:26:16.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:26:16.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: osdmap e731: 12 total, 12 up, 12 in 2024-09-02T10:26:16.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:26:16.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:15 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:26:16.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:16.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:16.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:16.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:16.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:16.218 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:16.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:16.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:16.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:16.342 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:16.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:16.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:16.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:26:16.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:26:16.383 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:16.393 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:16.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:26:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:26:16.402 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:26:16.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:16.445 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:26:16.445 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:26:16.482 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:26:16.483 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:26:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:16.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:16.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:16.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:26:16.517 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:26:16.551 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:26:16.551 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:26:16.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:26:16.585 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:26:16.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:16.622 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:26:16.622 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:26:16.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:16.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:16.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:16.653 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:26:16.653 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:26:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:16 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:26:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:16 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:26:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:16 smithi151 ceph-mon[16639]: pgmap v1669: 47 pgs: 1 peering, 1 active+clean+scrubbing, 45 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:26:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:16 smithi151 ceph-mon[16639]: 7.0 scrub starts 2024-09-02T10:26:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2054601163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:16.691 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:26:16.691 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:26:16.728 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:26:16.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:26:16.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:16.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:16.752 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:16.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:26:16.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:26:16.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:16.880 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:16.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:16.882 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:16.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:16 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:26:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:16 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:26:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:16 smithi187 ceph-mon[20072]: pgmap v1669: 47 pgs: 1 peering, 1 active+clean+scrubbing, 45 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:26:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:16 smithi187 ceph-mon[20072]: 7.0 scrub starts 2024-09-02T10:26:17.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2054601163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:17.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:17.011 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:16 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:26:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:16 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:26:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:16 smithi162 ceph-mon[20279]: pgmap v1669: 47 pgs: 1 peering, 1 active+clean+scrubbing, 45 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.19k op/s; 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:26:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:16 smithi162 ceph-mon[20279]: 7.0 scrub starts 2024-09-02T10:26:17.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2054601163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:17.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:17.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:17.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:17.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:17.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:17.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:17.365 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:17.388 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:17.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:17.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:17.513 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:17.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:17.615 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:17.642 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:17.643 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:17.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:17.768 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:17.769 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:17.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:17.898 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:17.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:17.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:17 smithi151 ceph-mon[16639]: 7.0 scrub ok 2024-09-02T10:26:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:17 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:26:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:17 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:26:18.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:17 smithi187 ceph-mon[20072]: 7.0 scrub ok 2024-09-02T10:26:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:17 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:26:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:17 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:26:18.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:18.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:18.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:17 smithi162 ceph-mon[20279]: 7.0 scrub ok 2024-09-02T10:26:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:17 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:26:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:17 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:26:18.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:18.147 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:18.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:18.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:18.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:18.276 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:18.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:18.378 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:18.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:18.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:18.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:18.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:18.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:18.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:18.636 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:18.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:18.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:18.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:18.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:18.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:18.907 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:18.933 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:18.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:18 smithi151 ceph-mon[16639]: pgmap v1670: 47 pgs: 1 peering, 46 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 3.7 MiB/s, 59 objects/s recovering 2024-09-02T10:26:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:18 smithi187 ceph-mon[20072]: pgmap v1670: 47 pgs: 1 peering, 46 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 3.7 MiB/s, 59 objects/s recovering 2024-09-02T10:26:19.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:19.057 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:19.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:18 smithi162 ceph-mon[20279]: pgmap v1670: 47 pgs: 1 peering, 46 active+clean; 3.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.07k op/s; 3.7 MiB/s, 59 objects/s recovering 2024-09-02T10:26:19.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:19.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:19.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:19.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:19.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:19.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:19.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:19.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:19.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:19.443 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:19.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:19.567 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:19.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:19.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:19.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:19.696 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:19.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:19.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:19.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:19.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:19.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:19.852 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:26:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:26:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:26:19.852 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:26:19.852 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:26:19.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:19.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:19 smithi151 ceph-mon[16639]: 7.9 scrub starts 2024-09-02T10:26:19.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:19.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:19.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:19 smithi187 ceph-mon[20072]: 7.9 scrub starts 2024-09-02T10:26:20.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:20.085 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:20.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:20.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:19 smithi162 ceph-mon[20279]: 7.9 scrub starts 2024-09-02T10:26:20.188 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:20.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:20.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:20.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:20.333 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:20.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:20.436 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:20.457 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:20.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:20.560 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:20.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:20.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:20.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:20.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:20.716 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:20.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:20.818 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:20.845 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:20.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:20.846 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:20 smithi151 ceph-mon[16639]: pgmap v1671: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 3.4 MiB/s, 54 objects/s recovering 2024-09-02T10:26:20.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:20.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:20.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:20.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:20 smithi187 ceph-mon[20072]: pgmap v1671: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 3.4 MiB/s, 54 objects/s recovering 2024-09-02T10:26:21.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:21.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:20 smithi162 ceph-mon[20279]: pgmap v1671: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.02k op/s; 3.4 MiB/s, 54 objects/s recovering 2024-09-02T10:26:21.104 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:21.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:21.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:21.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:21.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:21.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:21.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:21.371 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:21.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:21.472 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:21.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:21.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:21.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:21.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:21.624 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:21.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:21.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:21.748 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:21.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:21.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:21.873 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:21.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:21 smithi151 ceph-mon[16639]: 7.9 scrub ok 2024-09-02T10:26:21.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:21.998 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:21.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:22.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:21 smithi187 ceph-mon[20072]: 7.9 scrub ok 2024-09-02T10:26:22.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:26:22.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:21 smithi162 ceph-mon[20279]: 7.9 scrub ok 2024-09-02T10:26:22.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:22.124 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:22.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:22.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:22.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:22.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:22.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:22.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:22.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:22.509 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:22.510 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:22.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:22.538 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:22.539 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:22.613 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:22 smithi151 ceph-mon[16639]: pgmap v1672: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 657 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:22.613 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:22 smithi151 ceph-mon[16639]: 7.b scrub starts 2024-09-02T10:26:22.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:22.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:22.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:22.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:22.712 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:26:22.712 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:26:22.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:22.791 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:22.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:22.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:22.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:22 smithi187 ceph-mon[20072]: pgmap v1672: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 657 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:22.907 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:22 smithi187 ceph-mon[20072]: 7.b scrub starts 2024-09-02T10:26:22.919 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:22.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:22.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:22.995 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:22 smithi162 ceph-mon[20279]: pgmap v1672: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 657 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:22.995 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:22 smithi162 ceph-mon[20279]: 7.b scrub starts 2024-09-02T10:26:23.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:23.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:23.049 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:23.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:23.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:23.174 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:23.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:23.269 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:26:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:26:23.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.f e628,586 6:f0620152:::benchmark_data_smithi187_102448_object15839:head,577'7198)", 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.172751543, 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69156394899999996, 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094472+0000", 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1230000000000002e-06 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785890+0000", 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69141827 2024-09-02T10:26:23.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785894+0000", 2024-09-02T10:26:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1520000000000002e-06 2024-09-02T10:26:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785977+0000", 2024-09-02T10:26:23.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3077000000000004e-05 2024-09-02T10:26:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785987+0000", 2024-09-02T10:26:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8379999999999997e-06 2024-09-02T10:26:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.786031+0000", 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3763000000000001e-05 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.f e629,586 6:fb6fcd13:::benchmark_data_smithi187_102448_object18483:head,578'7347)", 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:47.434406+0000", 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 215.83281202699999, 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67700380800000004, 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.275 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434406+0000", 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434406+0000", 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434407+0000", 2024-09-02T10:26:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-02T10:26:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434411+0000", 2024-09-02T10:26:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6449999999999998e-06 2024-09-02T10:26:23.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111277+0000", 2024-09-02T10:26:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67686604399999994 2024-09-02T10:26:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111286+0000", 2024-09-02T10:26:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0529999999999996e-06 2024-09-02T10:26:23.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111349+0000", 2024-09-02T10:26:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000004e-05 2024-09-02T10:26:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111359+0000", 2024-09-02T10:26:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9250000000000007e-06 2024-09-02T10:26:23.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111410+0000", 2024-09-02T10:26:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0819999999999998e-05 2024-09-02T10:26:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.e e630,586 6:7c96fe1c:::benchmark_data_smithi187_102448_object21225:head,579'7600)", 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:49.550784+0000", 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 213.716433811, 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24060901500000001, 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:49.550784+0000", 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:49.550784+0000", 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:49.550785+0000", 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7899999999999998e-07 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:49.550788+0000", 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1240000000000001e-06 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:49.791298+0000", 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.240509748 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:49.791302+0000", 2024-09-02T10:26:23.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3989999999999997e-06 2024-09-02T10:26:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:49.791347+0000", 2024-09-02T10:26:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4604999999999997e-05 2024-09-02T10:26:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:49.791355+0000", 2024-09-02T10:26:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.712e-06 2024-09-02T10:26:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:49.791393+0000", 2024-09-02T10:26:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8847999999999998e-05 2024-09-02T10:26:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.2 e630,586 6:440f6173:::benchmark_data_smithi187_102448_object24822:head,580'7852)", 2024-09-02T10:26:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:50.119586+0000", 2024-09-02T10:26:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 213.14763223200001, 2024-09-02T10:26:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.259760771, 2024-09-02T10:26:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.286 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.119586+0000", 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.119586+0000", 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.119586+0000", 2024-09-02T10:26:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8500000000000002e-07 2024-09-02T10:26:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.288 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:23.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.119589+0000", 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7099999999999999e-06 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.379059+0000", 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25946972499999998 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.379077+0000", 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8406999999999998e-05 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.379253+0000", 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175995 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.379268+0000", 2024-09-02T10:26:23.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.435e-05 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.379347+0000", 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9098999999999996e-05 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.e e630,586 6:7bfdd3b2:::benchmark_data_smithi187_102448_object21710:head,579'7633)", 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:50.124080+0000", 2024-09-02T10:26:23.292 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 213.14313811400001, 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26112460300000001, 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.124080+0000", 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.124080+0000", 2024-09-02T10:26:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.124081+0000", 2024-09-02T10:26:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8400000000000002e-07 2024-09-02T10:26:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.124085+0000", 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1250000000000003e-06 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.385085+0000", 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26099998899999999 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.385092+0000", 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9059999999999996e-06 2024-09-02T10:26:23.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.385152+0000", 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9524000000000001e-05 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.385161+0000", 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3640000000000005e-06 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:50.385205+0000", 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3711000000000002e-05 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e635,586 6:033be843:::benchmark_data_smithi187_102448_object1499:head,573'6439)", 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:56.345932+0000", 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 206.92128600699999, 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39558264700000001, 2024-09-02T10:26:23.297 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345932+0000", 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345932+0000", 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345933+0000", 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345938+0000", 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0640000000000001e-06 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741308+0000", 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39536988899999997 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741316+0000", 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9009999999999994e-06 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741421+0000", 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105486 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741436+0000", 2024-09-02T10:26:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4617000000000001e-05 2024-09-02T10:26:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741515+0000", 2024-09-02T10:26:23.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8910000000000002e-05 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e643,586 6:0a7160e0:::benchmark_data_smithi187_102448_object18383:head,578'7477)", 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:07.703494+0000", 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 195.56372393199999, 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44123607500000001, 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.302 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703494+0000", 2024-09-02T10:26:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703494+0000", 2024-09-02T10:26:23.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703496+0000", 2024-09-02T10:26:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4529999999999999e-06 2024-09-02T10:26:23.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703502+0000", 2024-09-02T10:26:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2709999999999998e-06 2024-09-02T10:26:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144429+0000", 2024-09-02T10:26:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44092676800000002 2024-09-02T10:26:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.306 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144444+0000", 2024-09-02T10:26:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5146e-05 2024-09-02T10:26:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144589+0000", 2024-09-02T10:26:23.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014536200000000001 2024-09-02T10:26:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144604+0000", 2024-09-02T10:26:23.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4817000000000001e-05 2024-09-02T10:26:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144730+0000", 2024-09-02T10:26:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126258 2024-09-02T10:26:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.309 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e648,586 6:065bcc16:::benchmark_data_smithi187_102448_object45489:head,591'9175)", 2024-09-02T10:26:23.310 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:29.598657+0000", 2024-09-02T10:26:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 173.66856108299999, 2024-09-02T10:26:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67037543899999996, 2024-09-02T10:26:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.311 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598657+0000", 2024-09-02T10:26:23.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598657+0000", 2024-09-02T10:26:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598658+0000", 2024-09-02T10:26:23.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7800000000000001e-07 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598663+0000", 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268825+0000", 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.670161754 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268834+0000", 2024-09-02T10:26:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3449999999999995e-06 2024-09-02T10:26:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268939+0000", 2024-09-02T10:26:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010511899999999999 2024-09-02T10:26:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.315 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268949+0000", 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7599999999999997e-06 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.269033+0000", 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3561999999999997e-05 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e648,586 6:0cd50fba:::benchmark_data_smithi187_102448_object46855:head,592'9264)", 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:31.097253+0000", 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 172.16996509800001, 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.83171925999999996, 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097253+0000", 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097253+0000", 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097254+0000", 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.61e-07 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097259+0000", 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9520000000000002e-06 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928794+0000", 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.83153501299999999 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928806+0000", 2024-09-02T10:26:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1847e-05 2024-09-02T10:26:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928905+0000", 2024-09-02T10:26:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9277999999999994e-05 2024-09-02T10:26:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928913+0000", 2024-09-02T10:26:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7039999999999994e-06 2024-09-02T10:26:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928972+0000", 2024-09-02T10:26:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9805000000000001e-05 2024-09-02T10:26:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e653,586 6:01d685fb:::benchmark_data_smithi187_102448_object80695:head,616'11374)", 2024-09-02T10:26:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:48.578010+0000", 2024-09-02T10:26:23.322 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 154.689208079, 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54628079200000002, 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578010+0000", 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578010+0000", 2024-09-02T10:26:23.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578011+0000", 2024-09-02T10:26:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9400000000000005e-07 2024-09-02T10:26:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578016+0000", 2024-09-02T10:26:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9069999999999998e-06 2024-09-02T10:26:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124136+0000", 2024-09-02T10:26:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54612002800000004 2024-09-02T10:26:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124160+0000", 2024-09-02T10:26:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4731000000000001e-05 2024-09-02T10:26:23.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124241+0000", 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0457999999999994e-05 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124249+0000", 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3620000000000006e-06 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124291+0000", 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1712000000000003e-05 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:45621 7.8 7:1a107a8d:::benchmark_data_smithi187_120712_object45620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.307311+0000", 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 89.959907416999997, 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36183037800000001, 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.328 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45621 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307311+0000", 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307311+0000", 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307315+0000", 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.566e-06 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307366+0000", 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0365999999999999e-05 2024-09-02T10:26:23.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307379+0000", 2024-09-02T10:26:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3134e-05 2024-09-02T10:26:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307386+0000", 2024-09-02T10:26:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.362e-06 2024-09-02T10:26:23.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307441+0000", 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4568999999999999e-05 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307551+0000", 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109806 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:26:23.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307723+0000", 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017262399999999999 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.309378+0000", 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016545729999999999 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.309672+0000", 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029370500000000001 2024-09-02T10:26:23.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669068+0000", 2024-09-02T10:26:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35939658099999999 2024-09-02T10:26:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669108+0000", 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9860999999999997e-05 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669141+0000", 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3231000000000001e-05 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:45621 7.8 e689/661 ondisk, result = 0)", 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.383905+0000", 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 89.883312877999998, 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28524120600000002, 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383905+0000", 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383905+0000", 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383907+0000", 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0470000000000001e-06 2024-09-02T10:26:23.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383920+0000", 2024-09-02T10:26:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2989e-05 2024-09-02T10:26:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669023+0000", 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28510220400000003 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669028+0000", 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4759999999999997e-06 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.339 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669062+0000", 2024-09-02T10:26:23.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3779999999999998e-05 2024-09-02T10:26:23.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.340 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.340 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669065+0000", 2024-09-02T10:26:23.340 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4089999999999999e-06 2024-09-02T10:26:23.340 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.340 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.341 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.341 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669147+0000", 2024-09-02T10:26:23.341 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1301e-05 2024-09-02T10:26:23.341 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.341 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.341 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.341 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.341 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.341 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:45650 7.8 7:1c05cd80:::benchmark_data_smithi187_120712_object45649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.666730+0000", 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 89.600487982999994, 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.325110548, 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45650 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.342 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666730+0000", 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666730+0000", 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666732+0000", 2024-09-02T10:26:23.343 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.835e-06 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666773+0000", 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1121999999999997e-05 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666787+0000", 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3908e-05 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.344 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666793+0000", 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4589999999999997e-06 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666848+0000", 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5037999999999997e-05 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666935+0000", 2024-09-02T10:26:23.345 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7781000000000005e-05 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.667024+0000", 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8331000000000005e-05 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.668407+0000", 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013827869999999999 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.346 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.347 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.347 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.668945+0000", 2024-09-02T10:26:23.348 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053888899999999997 2024-09-02T10:26:23.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.349 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.349 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991726+0000", 2024-09-02T10:26:23.349 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32278023700000003 2024-09-02T10:26:23.349 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.349 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991790+0000", 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4288000000000005e-05 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991841+0000", 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0872999999999999e-05 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.350 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:45650 7.8 e689/661 ondisk, result = 0)", 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.669003+0000", 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 89.598215202000006, 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32284557699999999, 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669003+0000", 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.351 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669003+0000", 2024-09-02T10:26:23.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.352 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669004+0000", 2024-09-02T10:26:23.352 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-02T10:26:23.352 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.352 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.352 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669007+0000", 2024-09-02T10:26:23.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4340000000000001e-06 2024-09-02T10:26:23.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.353 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991617+0000", 2024-09-02T10:26:23.353 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.322609692 2024-09-02T10:26:23.353 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.353 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.353 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991628+0000", 2024-09-02T10:26:23.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0998e-05 2024-09-02T10:26:23.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.354 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.354 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991700+0000", 2024-09-02T10:26:23.354 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2343999999999997e-05 2024-09-02T10:26:23.354 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.354 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991718+0000", 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8142e-05 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991849+0000", 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130207 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.355 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:55500 7.8 7:1ef2c8c5:::benchmark_data_smithi187_120712_object55499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:07.741235+0000", 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 75.525983053999994, 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24081252, 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55500 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.356 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.741235+0000", 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.741235+0000", 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.357 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.741237+0000", 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8789999999999999e-06 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.741295+0000", 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7645000000000001e-05 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.358 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.741307+0000", 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1847e-05 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.741314+0000", 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8140000000000001e-06 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.741367+0000", 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2868000000000003e-05 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.359 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.741471+0000", 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104165 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.741570+0000", 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8887000000000004e-05 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.743239+0000", 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001668484 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.360 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.743504+0000", 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000264959 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.981973+0000", 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23846893599999999 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.982014+0000", 2024-09-02T10:26:23.361 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1211999999999998e-05 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.982048+0000", 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3824e-05 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:55533 7.e 7:752a3ea5:::benchmark_data_smithi187_120712_object55532:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:07.797219+0000", 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 75.469999420999997, 2024-09-02T10:26:23.362 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24933429400000001, 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55533 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.797219+0000", 2024-09-02T10:26:23.363 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.797219+0000", 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.797221+0000", 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8169999999999999e-06 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.364 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.797281+0000", 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9985000000000003e-05 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.797300+0000", 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8995e-05 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.797310+0000", 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0445000000000001e-05 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.365 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.797374+0000", 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3584000000000007e-05 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.797505+0000", 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013112099999999999 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.797661+0000", 2024-09-02T10:26:23.366 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000156558 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.799676+0000", 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020144 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.799824+0000", 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014840099999999999 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.367 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.046457+0000", 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24663254300000001 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.046509+0000", 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2580000000000001e-05 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.046553+0000", 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3865000000000001e-05 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.368 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:71627 7.8 7:158e78ca:::benchmark_data_smithi187_120712_object71626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e711)", 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:27.818033+0000", 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 55.449185661999998, 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48315623299999999, 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:23.369 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 71627 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818033+0000", 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818033+0000", 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.370 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818035+0000", 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2529999999999999e-06 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818066+0000", 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1619000000000003e-05 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818083+0000", 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6721000000000001e-05 2024-09-02T10:26:23.371 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818093+0000", 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3680000000000008e-06 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818169+0000", 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6487000000000001e-05 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818309+0000", 2024-09-02T10:26:23.372 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014026699999999999 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818454+0000", 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144964 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.821157+0000", 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027031830000000001 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:23.373 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.821177+0000", 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0055e-05 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.301111+0000", 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47993339899999998 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.301149+0000", 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8605000000000001e-05 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.374 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.301189+0000", 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9311999999999999e-05 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:99739 7.8 7:121bd1d6:::benchmark_data_smithi187_120712_object15797:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:00.273261+0000", 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.993957591000001, 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22864082099999999, 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.375 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 99739 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.273261+0000", 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.376 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.273261+0000", 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.273262+0000", 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.341e-06 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.273267+0000", 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8690000000000003e-06 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.377 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.273281+0000", 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4012999999999999e-05 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.273288+0000", 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9550000000000003e-06 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.273377+0000", 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9062999999999998e-05 2024-09-02T10:26:23.378 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.273497+0000", 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012023000000000001 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.273607+0000", 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109787 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.379 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.276195+0000", 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0025880909999999998 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.276208+0000", 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2561999999999999e-05 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.501764+0000", 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22555598600000001 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.380 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.501855+0000", 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1847000000000003e-05 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.501901+0000", 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6076999999999999e-05 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:99751 7.e 7:70408914:::benchmark_data_smithi187_120712_object15809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-09-02T10:26:23.381 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:00.289876+0000", 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.977342721999999, 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27411807599999999, 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 99751 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.382 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289876+0000", 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289876+0000", 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289876+0000", 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7999999999999993e-07 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.383 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289881+0000", 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7299999999999996e-06 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289893+0000", 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1924e-05 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289901+0000", 2024-09-02T10:26:23.384 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0360000000000006e-06 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.293250+0000", 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033488630000000001 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.293376+0000", 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012641599999999999 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:26:23.385 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.293494+0000", 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011747999999999999 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.296970+0000", 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0034765529999999998 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.556677+0000", 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25970602100000001 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.386 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.563870+0000", 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0071933509999999997 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.563943+0000", 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.326e-05 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.563994+0000", 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0461999999999999e-05 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.387 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:99751 7.e e722/661 ondisk, result = 0)", 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:00.296758+0000", 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.970460732999999, 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.267245758, 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:23.388 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.296758+0000", 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.296758+0000", 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.296758+0000", 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2900000000000003e-07 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.389 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.296763+0000", 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9599999999999999e-06 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.562082+0000", 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26531871499999998 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.562090+0000", 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0940000000000008e-06 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.390 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.563856+0000", 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001766175 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.563864+0000", 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3710000000000002e-06 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.564003+0000", 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013971400000000001 2024-09-02T10:26:23.391 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.392 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.392 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.392 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:23.392 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:23.392 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:26:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:23.392 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:23.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:23.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:23.521 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:23.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:23.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:26:23.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:23.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:23.650 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:23.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:23.775 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:23.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:23.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:23.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:23.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:23.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:23 smithi151 ceph-mon[16639]: 7.b scrub ok 2024-09-02T10:26:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:23 smithi151 ceph-mon[16639]: from='client.18123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:26:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:23 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:26:23.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:23 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:26:23.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:23 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:23.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:23 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:24.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:23 smithi187 ceph-mon[20072]: 7.b scrub ok 2024-09-02T10:26:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:23 smithi187 ceph-mon[20072]: from='client.18123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:26:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:23 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:26:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:23 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:26:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:23 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:24.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:23 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:24.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:24.024 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:24.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:23 smithi162 ceph-mon[20279]: 7.b scrub ok 2024-09-02T10:26:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:23 smithi162 ceph-mon[20279]: from='client.18123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:26:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:23 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:26:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:23 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:26:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:23 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:23 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:24.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:24.147 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:24.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:24.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:24.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:24.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:24.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:24.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:24.395 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:24.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:24.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:24.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:24.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:24.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:24.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:24.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:24.655 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:24.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:24.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:24.779 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:24.781 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:24.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:24.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:24.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:24.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:24 smithi151 ceph-mon[16639]: pgmap v1673: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:24 smithi151 ceph-mon[16639]: osdmap e732: 12 total, 12 up, 12 in 2024-09-02T10:26:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:24 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:26:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:24 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:26:25.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:25.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:24 smithi187 ceph-mon[20072]: pgmap v1673: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:25.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:24 smithi187 ceph-mon[20072]: osdmap e732: 12 total, 12 up, 12 in 2024-09-02T10:26:25.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:24 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:26:25.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:24 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:26:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:25.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:25.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:24 smithi162 ceph-mon[20279]: pgmap v1673: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 844 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:24 smithi162 ceph-mon[20279]: osdmap e732: 12 total, 12 up, 12 in 2024-09-02T10:26:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:24 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:26:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:24 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:26:25.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:25.166 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:25.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:25.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:25.296 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:25.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:25.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:25.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:25.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:25.423 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:25.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:25.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:25.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:25.549 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:25.629 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:26:25.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:25.674 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:25.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:25.675 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:25.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:25.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:25.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:25.800 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:25.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:25.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:25.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:25.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2027674304' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:26:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2027674304' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:26:26.036 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:26.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:26.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:26.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2027674304' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:26:26.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:26.200 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:26.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:26.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:26.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:26.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:26.408 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1672,"stamp":"2024-09-02T10:26:21.461590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3457483320,"num_objects":52751,"num_object_clones":0,"num_object_copies":158253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52751,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115131,"num_write_kb":5372097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30937,"num_bytes_recovered":2028090976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117983,"ondisk_log_size":117983,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16834612,"kb_used_data":10362260,"kb_used_omap":18,"kb_used_meta":6471405,"kb_avail":1108107212,"statfs":{"total":1151940427776,"available":1134701785088,"internally_reserved":0,"allocated":10610954240,"data_stored":10605516776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":6626719035},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-344850432,"num_objects":-5262,"num_object_clones":0,"num_object_copies":-15786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5262,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001958"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":841,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.234183+0000","last_change":"2024-09-02T10:26:02.405331+0000","last_active":"2024-09-02T10:26:15.234183+0000","last_peered":"2024-09-02T10:26:15.234183+0000","last_clean":"2024-09-02T10:26:15.234183+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:26:15.234183+0000","last_undegraded":"2024-09-02T10:26:15.234183+0000","last_fullsized":"2024-09-02T10:26:15.234183+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.405283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.405283+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:17.160849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":648,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.243880+0000","last_change":"2024-09-02T10:26:14.793294+0000","last_active":"2024-09-02T10:26:15.243880+0000","last_peered":"2024-09-02T10:26:15.243880+0000","last_clean":"2024-09-02T10:26:15.243880+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:26:15.243880+0000","last_undegraded":"2024-09-02T10:26:15.243880+0000","last_fullsized":"2024-09-02T10:26:15.243880+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:14.793044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.793044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:31.973724+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1114,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.227776+0000","last_change":"2024-09-02T10:26:09.987463+0000","last_active":"2024-09-02T10:26:15.227776+0000","last_peered":"2024-09-02T10:26:15.227776+0000","last_clean":"2024-09-02T10:26:15.227776+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:26:15.227776+0000","last_undegraded":"2024-09-02T10:26:15.227776+0000","last_fullsized":"2024-09-02T10:26:15.227776+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.987389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.987389+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:39.899046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1098,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.227667+0000","last_change":"2024-09-02T10:25:22.185960+0000","last_active":"2024-09-02T10:26:15.227667+0000","last_peered":"2024-09-02T10:26:15.227667+0000","last_clean":"2024-09-02T10:26:15.227667+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:26:15.227667+0000","last_undegraded":"2024-09-02T10:26:15.227667+0000","last_fullsized":"2024-09-02T10:26:15.227667+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:25:22.185894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:25:22.185894+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:26:25.443076+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":401,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.232299+0000","last_change":"2024-09-02T10:26:06.766310+0000","last_active":"2024-09-02T10:26:15.232299+0000","last_peered":"2024-09-02T10:26:15.232299+0000","last_clean":"2024-09-02T10:26:15.232299+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:26:15.232299+0000","last_undegraded":"2024-09-02T10:26:15.232299+0000","last_fullsized":"2024-09-02T10:26:15.232299+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:06.766238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:06.766238+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:09.577073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":960,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.236495+0000","last_change":"2024-09-02T10:26:12.294448+0000","last_active":"2024-09-02T10:26:15.236495+0000","last_peered":"2024-09-02T10:26:15.236495+0000","last_clean":"2024-09-02T10:26:15.236495+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:26:15.236495+0000","last_undegraded":"2024-09-02T10:26:15.236495+0000","last_fullsized":"2024-09-02T10:26:15.236495+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:12.294402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:12.294402+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:13.584271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1480,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.246230+0000","last_change":"2024-09-02T10:26:04.538470+0000","last_active":"2024-09-02T10:26:15.246230+0000","last_peered":"2024-09-02T10:26:15.246230+0000","last_clean":"2024-09-02T10:26:15.246230+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:26:15.246230+0000","last_undegraded":"2024-09-02T10:26:15.246230+0000","last_fullsized":"2024-09-02T10:26:15.246230+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:04.538409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:04.538409+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:23.977037+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1122,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.227969+0000","last_change":"2024-09-02T10:26:08.951582+0000","last_active":"2024-09-02T10:26:15.227969+0000","last_peered":"2024-09-02T10:26:15.227969+0000","last_clean":"2024-09-02T10:26:15.227969+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:26:15.227969+0000","last_undegraded":"2024-09-02T10:26:15.227969+0000","last_fullsized":"2024-09-02T10:26:15.227969+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:08.951514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.951514+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:10.021565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":1119,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.239867+0000","last_change":"2024-09-02T10:25:01.211529+0000","last_active":"2024-09-02T10:26:15.239867+0000","last_peered":"2024-09-02T10:26:15.239867+0000","last_clean":"2024-09-02T10:26:15.239867+0000","last_became_active":"2024-09-02T10:25:01.211125+0000","last_became_peered":"2024-09-02T10:25:01.211125+0000","last_unstale":"2024-09-02T10:26:15.239867+0000","last_undegraded":"2024-09-02T10:26:15.239867+0000","last_fullsized":"2024-09-02T10:26:15.239867+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:24:46.621773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:24:46.621773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1089,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.232490+0000","last_change":"2024-09-02T10:26:05.853283+0000","last_active":"2024-09-02T10:26:15.232490+0000","last_peered":"2024-09-02T10:26:15.232490+0000","last_clean":"2024-09-02T10:26:15.232490+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:26:15.232490+0000","last_undegraded":"2024-09-02T10:26:15.232490+0000","last_fullsized":"2024-09-02T10:26:15.232490+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:05.853228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:05.853228+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:33.199800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":333,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.228416+0000","last_change":"2024-09-02T10:25:20.120097+0000","last_active":"2024-09-02T10:26:15.228416+0000","last_peered":"2024-09-02T10:26:15.228416+0000","last_clean":"2024-09-02T10:26:15.228416+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:26:15.228416+0000","last_undegraded":"2024-09-02T10:26:15.228416+0000","last_fullsized":"2024-09-02T10:26:15.228416+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:25:20.120024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:25:20.120024+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:26:47.878922+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1439,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:16.774967+0000","last_change":"2024-09-02T10:26:16.774967+0000","last_active":"2024-09-02T10:26:16.774967+0000","last_peered":"2024-09-02T10:26:16.774967+0000","last_clean":"2024-09-02T10:26:16.774967+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:26:16.774967+0000","last_undegraded":"2024-09-02T10:26:16.774967+0000","last_fullsized":"2024-09-02T10:26:16.774967+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:16.774772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:16.774772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:37.484948+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1379,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.246261+0000","last_change":"2024-09-02T10:26:09.562148+0000","last_active":"2024-09-02T10:26:15.246261+0000","last_peered":"2024-09-02T10:26:15.246261+0000","last_clean":"2024-09-02T10:26:15.246261+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:26:15.246261+0000","last_undegraded":"2024-09-02T10:26:15.246261+0000","last_fullsized":"2024-09-02T10:26:15.246261+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.562104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.562104+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:16.950256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":102,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.232206+0000","last_change":"2024-09-02T10:26:14.840384+0000","last_active":"2024-09-02T10:26:15.232206+0000","last_peered":"2024-09-02T10:26:15.232206+0000","last_clean":"2024-09-02T10:26:15.232206+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:26:15.232206+0000","last_undegraded":"2024-09-02T10:26:15.232206+0000","last_fullsized":"2024-09-02T10:26:15.232206+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:14.840310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.840310+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:19.191366+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1349,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.246345+0000","last_change":"2024-09-02T10:26:06.555868+0000","last_active":"2024-09-02T10:26:15.246345+0000","last_peered":"2024-09-02T10:26:15.246345+0000","last_clean":"2024-09-02T10:26:15.246345+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:26:15.246345+0000","last_undegraded":"2024-09-02T10:26:15.246345+0000","last_fullsized":"2024-09-02T10:26:15.246345+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:06.555813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:06.555813+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:17.827776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"731'6908","reported_seq":7106,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:17.178453+0000","last_change":"2024-09-02T10:25:20.361425+0000","last_active":"2024-09-02T10:26:17.178453+0000","last_peered":"2024-09-02T10:26:17.178453+0000","last_clean":"2024-09-02T10:26:17.178453+0000","last_became_active":"2024-09-02T10:24:07.681518+0000","last_became_peered":"2024-09-02T10:24:07.681518+0000","last_unstale":"2024-09-02T10:26:17.178453+0000","last_undegraded":"2024-09-02T10:26:17.178453+0000","last_fullsized":"2024-09-02T10:26:17.178453+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"706'3999","last_scrub_stamp":"2024-09-02T10:25:20.360400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:25:20.360400+0000","objects_scrubbed":3994,"log_size":6908,"log_dups_size":0,"ondisk_log_size":6908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:26:38.772833+0000","scrub_duration":1241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220725248,"num_objects":3368,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6908,"num_write_kb":328832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1443,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.246467+0000","last_change":"2024-09-02T10:26:02.497906+0000","last_active":"2024-09-02T10:26:15.246467+0000","last_peered":"2024-09-02T10:26:15.246467+0000","last_clean":"2024-09-02T10:26:15.246467+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:26:15.246467+0000","last_undegraded":"2024-09-02T10:26:15.246467+0000","last_fullsized":"2024-09-02T10:26:15.246467+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.497853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.497853+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:21.295805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"731'7373","reported_seq":15651,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:17.984514+0000","last_change":"2024-09-02T10:25:21.220340+0000","last_active":"2024-09-02T10:26:17.984514+0000","last_peered":"2024-09-02T10:26:17.984514+0000","last_clean":"2024-09-02T10:26:17.984514+0000","last_became_active":"2024-09-02T10:25:21.217238+0000","last_became_peered":"2024-09-02T10:25:21.217238+0000","last_unstale":"2024-09-02T10:26:17.984514+0000","last_undegraded":"2024-09-02T10:26:17.984514+0000","last_fullsized":"2024-09-02T10:26:17.984514+0000","mapping_epoch":706,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":707,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:24:06.651660+0000","objects_scrubbed":0,"log_size":7373,"log_dups_size":0,"ondisk_log_size":7373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216334336,"num_objects":3301,"num_object_clones":0,"num_object_copies":9903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7373,"num_write_kb":341568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4053,"num_bytes_recovered":265617408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1568,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.232359+0000","last_change":"2024-09-02T10:26:09.933550+0000","last_active":"2024-09-02T10:26:15.232359+0000","last_peered":"2024-09-02T10:26:15.232359+0000","last_clean":"2024-09-02T10:26:15.232359+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:26:15.232359+0000","last_undegraded":"2024-09-02T10:26:15.232359+0000","last_fullsized":"2024-09-02T10:26:15.232359+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.933487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.933487+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:19.759531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"731'7106","reported_seq":16039,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:17.381426+0000","last_change":"2024-09-02T10:26:13.178680+0000","last_active":"2024-09-02T10:26:17.381426+0000","last_peered":"2024-09-02T10:26:17.381426+0000","last_clean":"2024-09-02T10:26:17.381426+0000","last_became_active":"2024-09-02T10:25:44.212783+0000","last_became_peered":"2024-09-02T10:25:44.212783+0000","last_unstale":"2024-09-02T10:26:17.381426+0000","last_undegraded":"2024-09-02T10:26:17.381426+0000","last_fullsized":"2024-09-02T10:26:17.381426+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"729'6901","last_scrub_stamp":"2024-09-02T10:26:13.178572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.178572+0000","objects_scrubbed":3559,"log_size":7106,"log_dups_size":0,"ondisk_log_size":7106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:39.044105+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218890240,"num_objects":3340,"num_object_clones":0,"num_object_copies":10020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7106,"num_write_kb":334272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4359,"num_bytes_recovered":285671424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1211,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.240372+0000","last_change":"2024-09-02T10:26:02.369141+0000","last_active":"2024-09-02T10:26:15.240372+0000","last_peered":"2024-09-02T10:26:15.240372+0000","last_clean":"2024-09-02T10:26:15.240372+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:26:15.240372+0000","last_undegraded":"2024-09-02T10:26:15.240372+0000","last_fullsized":"2024-09-02T10:26:15.240372+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.369090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.369090+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:20.449364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.d","version":"731'7180","reported_seq":7379,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:17.394475+0000","last_change":"2024-09-02T10:26:13.549346+0000","last_active":"2024-09-02T10:26:17.394475+0000","last_peered":"2024-09-02T10:26:17.394475+0000","last_clean":"2024-09-02T10:26:17.394475+0000","last_became_active":"2024-09-02T10:24:07.671900+0000","last_became_peered":"2024-09-02T10:24:07.671900+0000","last_unstale":"2024-09-02T10:26:17.394475+0000","last_undegraded":"2024-09-02T10:26:17.394475+0000","last_fullsized":"2024-09-02T10:26:17.394475+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"729'6967","last_scrub_stamp":"2024-09-02T10:26:13.549295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.549295+0000","objects_scrubbed":3613,"log_size":7180,"log_dups_size":0,"ondisk_log_size":7180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:30.818673+0000","scrub_duration":1966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219480088,"num_objects":3350,"num_object_clones":0,"num_object_copies":10050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7180,"num_write_kb":336897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":607,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.236188+0000","last_change":"2024-09-02T10:26:07.441225+0000","last_active":"2024-09-02T10:26:15.236188+0000","last_peered":"2024-09-02T10:26:15.236188+0000","last_clean":"2024-09-02T10:26:15.236188+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:26:15.236188+0000","last_undegraded":"2024-09-02T10:26:15.236188+0000","last_fullsized":"2024-09-02T10:26:15.236188+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:07.441159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:07.441159+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:21.111255+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"731'7142","reported_seq":7340,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:18.536657+0000","last_change":"2024-09-02T10:26:10.244647+0000","last_active":"2024-09-02T10:26:18.536657+0000","last_peered":"2024-09-02T10:26:18.536657+0000","last_clean":"2024-09-02T10:26:18.536657+0000","last_became_active":"2024-09-02T10:24:07.681580+0000","last_became_peered":"2024-09-02T10:24:07.681580+0000","last_unstale":"2024-09-02T10:26:18.536657+0000","last_undegraded":"2024-09-02T10:26:18.536657+0000","last_fullsized":"2024-09-02T10:26:18.536657+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'6721","last_scrub_stamp":"2024-09-02T10:26:10.244419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:10.244419+0000","objects_scrubbed":3718,"log_size":7142,"log_dups_size":0,"ondisk_log_size":7142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:18.436180+0000","scrub_duration":1044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214564864,"num_objects":3274,"num_object_clones":0,"num_object_copies":9822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7142,"num_write_kb":333312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1516,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.237284+0000","last_change":"2024-09-02T10:26:13.596384+0000","last_active":"2024-09-02T10:26:15.237284+0000","last_peered":"2024-09-02T10:26:15.237284+0000","last_clean":"2024-09-02T10:26:15.237284+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:26:15.237284+0000","last_undegraded":"2024-09-02T10:26:15.237284+0000","last_fullsized":"2024-09-02T10:26:15.237284+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:13.596339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.596339+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:42.860522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"731'7126","reported_seq":7324,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:18.555463+0000","last_change":"2024-09-02T10:26:08.197217+0000","last_active":"2024-09-02T10:26:18.555463+0000","last_peered":"2024-09-02T10:26:18.555463+0000","last_clean":"2024-09-02T10:26:18.555463+0000","last_became_active":"2024-09-02T10:24:07.674582+0000","last_became_peered":"2024-09-02T10:24:07.674582+0000","last_unstale":"2024-09-02T10:26:18.555463+0000","last_undegraded":"2024-09-02T10:26:18.555463+0000","last_fullsized":"2024-09-02T10:26:18.555463+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6563","last_scrub_stamp":"2024-09-02T10:26:08.197134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.197134+0000","objects_scrubbed":3761,"log_size":7126,"log_dups_size":0,"ondisk_log_size":7126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:38.138983+0000","scrub_duration":1013,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207880192,"num_objects":3172,"num_object_clones":0,"num_object_copies":9516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7126,"num_write_kb":329536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":982,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.352217+0000","last_change":"2024-09-02T10:26:15.352217+0000","last_active":"2024-09-02T10:26:15.352217+0000","last_peered":"2024-09-02T10:26:15.352217+0000","last_clean":"2024-09-02T10:26:15.352217+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:26:15.352217+0000","last_undegraded":"2024-09-02T10:26:15.352217+0000","last_fullsized":"2024-09-02T10:26:15.352217+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:15.352153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:15.352153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:33.222325+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"731'7205","reported_seq":7441,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:17.506544+0000","last_change":"2024-09-02T10:26:16.241362+0000","last_active":"2024-09-02T10:26:17.506544+0000","last_peered":"2024-09-02T10:26:17.506544+0000","last_clean":"2024-09-02T10:26:17.506544+0000","last_became_active":"2024-09-02T10:24:07.673747+0000","last_became_peered":"2024-09-02T10:24:07.673747+0000","last_unstale":"2024-09-02T10:26:17.506544+0000","last_undegraded":"2024-09-02T10:26:17.506544+0000","last_fullsized":"2024-09-02T10:26:17.506544+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'7131","last_scrub_stamp":"2024-09-02T10:26:16.241303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:16.241303+0000","objects_scrubbed":3422,"log_size":7205,"log_dups_size":0,"ondisk_log_size":7205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:29.657946+0000","scrub_duration":564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218431488,"num_objects":3333,"num_object_clones":0,"num_object_copies":9999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7205,"num_write_kb":337216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":947,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.233790+0000","last_change":"2024-09-02T10:26:09.304259+0000","last_active":"2024-09-02T10:26:15.233790+0000","last_peered":"2024-09-02T10:26:15.233790+0000","last_clean":"2024-09-02T10:26:15.233790+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:26:15.233790+0000","last_undegraded":"2024-09-02T10:26:15.233790+0000","last_fullsized":"2024-09-02T10:26:15.233790+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.303950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.303950+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:10.581116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"731'7249","reported_seq":16282,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:19.584982+0000","last_change":"2024-09-02T10:26:10.642407+0000","last_active":"2024-09-02T10:26:19.584982+0000","last_peered":"2024-09-02T10:26:19.584982+0000","last_clean":"2024-09-02T10:26:19.584982+0000","last_became_active":"2024-09-02T10:25:44.211739+0000","last_became_peered":"2024-09-02T10:25:44.211739+0000","last_unstale":"2024-09-02T10:26:19.584982+0000","last_undegraded":"2024-09-02T10:26:19.584982+0000","last_fullsized":"2024-09-02T10:26:19.584982+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'6786","last_scrub_stamp":"2024-09-02T10:26:10.642208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:10.642208+0000","objects_scrubbed":3711,"log_size":7249,"log_dups_size":0,"ondisk_log_size":7249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:27.257300+0000","scrub_duration":1800,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209780736,"num_objects":3201,"num_object_clones":0,"num_object_copies":9603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3201,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7249,"num_write_kb":334400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4407,"num_bytes_recovered":288817152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":729,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.234079+0000","last_change":"2024-09-02T10:26:14.593855+0000","last_active":"2024-09-02T10:26:15.234079+0000","last_peered":"2024-09-02T10:26:15.234079+0000","last_clean":"2024-09-02T10:26:15.234079+0000","last_became_active":"2024-09-02T10:25:02.065175+0000","last_became_peered":"2024-09-02T10:25:02.065175+0000","last_unstale":"2024-09-02T10:26:15.234079+0000","last_undegraded":"2024-09-02T10:26:15.234079+0000","last_fullsized":"2024-09-02T10:26:15.234079+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:14.593805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.593805+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:42.832993+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"731'7103","reported_seq":7299,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:17.418902+0000","last_change":"2024-09-02T10:26:11.397647+0000","last_active":"2024-09-02T10:26:17.418902+0000","last_peered":"2024-09-02T10:26:17.418902+0000","last_clean":"2024-09-02T10:26:17.418902+0000","last_became_active":"2024-09-02T10:24:07.670909+0000","last_became_peered":"2024-09-02T10:24:07.670909+0000","last_unstale":"2024-09-02T10:26:17.418902+0000","last_undegraded":"2024-09-02T10:26:17.418902+0000","last_fullsized":"2024-09-02T10:26:17.418902+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'6764","last_scrub_stamp":"2024-09-02T10:26:11.397518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:11.397518+0000","objects_scrubbed":3666,"log_size":7103,"log_dups_size":0,"ondisk_log_size":7103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:13.343075+0000","scrub_duration":854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216727552,"num_objects":3307,"num_object_clones":0,"num_object_copies":9921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7103,"num_write_kb":333120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":962,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.236598+0000","last_change":"2024-09-02T10:26:08.256406+0000","last_active":"2024-09-02T10:26:15.236598+0000","last_peered":"2024-09-02T10:26:15.236598+0000","last_clean":"2024-09-02T10:26:15.236598+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:26:15.236598+0000","last_undegraded":"2024-09-02T10:26:15.236598+0000","last_fullsized":"2024-09-02T10:26:15.236598+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:08.256332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.256332+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:32.379501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.4","version":"731'7367","reported_seq":18530,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:18.010864+0000","last_change":"2024-09-02T10:26:14.088498+0000","last_active":"2024-09-02T10:26:18.010864+0000","last_peered":"2024-09-02T10:26:18.010864+0000","last_clean":"2024-09-02T10:26:18.010864+0000","last_became_active":"2024-09-02T10:26:14.049049+0000","last_became_peered":"2024-09-02T10:26:14.049049+0000","last_unstale":"2024-09-02T10:26:18.010864+0000","last_undegraded":"2024-09-02T10:26:18.010864+0000","last_fullsized":"2024-09-02T10:26:18.010864+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:24:06.651660+0000","objects_scrubbed":0,"log_size":7367,"log_dups_size":0,"ondisk_log_size":7367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219348992,"num_objects":3347,"num_object_clones":0,"num_object_copies":10041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7367,"num_write_kb":342848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5491,"num_bytes_recovered":359792640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1210,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.232483+0000","last_change":"2024-09-02T10:26:11.944736+0000","last_active":"2024-09-02T10:26:15.232483+0000","last_peered":"2024-09-02T10:26:15.232483+0000","last_clean":"2024-09-02T10:26:15.232483+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:26:15.232483+0000","last_undegraded":"2024-09-02T10:26:15.232483+0000","last_fullsized":"2024-09-02T10:26:15.232483+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:11.944689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:11.944689+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:23.172770+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"731'7109","reported_seq":15737,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:17.267856+0000","last_change":"2024-09-02T10:26:06.663497+0000","last_active":"2024-09-02T10:26:17.267856+0000","last_peered":"2024-09-02T10:26:17.267856+0000","last_clean":"2024-09-02T10:26:17.267856+0000","last_became_active":"2024-09-02T10:25:28.620158+0000","last_became_peered":"2024-09-02T10:25:28.620158+0000","last_unstale":"2024-09-02T10:26:17.267856+0000","last_undegraded":"2024-09-02T10:26:17.267856+0000","last_fullsized":"2024-09-02T10:26:17.267856+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6440","last_scrub_stamp":"2024-09-02T10:26:06.663370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:06.663370+0000","objects_scrubbed":3970,"log_size":7109,"log_dups_size":0,"ondisk_log_size":7109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:12.071995+0000","scrub_duration":1255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213975040,"num_objects":3265,"num_object_clones":0,"num_object_copies":9795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7109,"num_write_kb":331968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4199,"num_bytes_recovered":275185664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1462,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.239873+0000","last_change":"2024-09-02T10:24:20.598008+0000","last_active":"2024-09-02T10:26:15.239873+0000","last_peered":"2024-09-02T10:26:15.239873+0000","last_clean":"2024-09-02T10:26:15.239873+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:26:15.239873+0000","last_undegraded":"2024-09-02T10:26:15.239873+0000","last_fullsized":"2024-09-02T10:26:15.239873+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:24:20.597925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:24:20.597925+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"731'7362","reported_seq":16515,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:18.082016+0000","last_change":"2024-09-02T10:26:14.504599+0000","last_active":"2024-09-02T10:26:18.082016+0000","last_peered":"2024-09-02T10:26:18.082016+0000","last_clean":"2024-09-02T10:26:18.082016+0000","last_became_active":"2024-09-02T10:26:02.534985+0000","last_became_peered":"2024-09-02T10:26:02.534985+0000","last_unstale":"2024-09-02T10:26:18.082016+0000","last_undegraded":"2024-09-02T10:26:18.082016+0000","last_fullsized":"2024-09-02T10:26:18.082016+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'7133","last_scrub_stamp":"2024-09-02T10:26:14.504505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.504505+0000","objects_scrubbed":3617,"log_size":7362,"log_dups_size":0,"ondisk_log_size":7362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:24.932727+0000","scrub_duration":694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219545600,"num_objects":3350,"num_object_clones":0,"num_object_copies":10050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7362,"num_write_kb":342784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4468,"num_bytes_recovered":292814848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1213,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.233911+0000","last_change":"2024-09-02T10:26:01.795185+0000","last_active":"2024-09-02T10:26:15.233911+0000","last_peered":"2024-09-02T10:26:15.233911+0000","last_clean":"2024-09-02T10:26:15.233911+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:26:15.233911+0000","last_undegraded":"2024-09-02T10:26:15.233911+0000","last_fullsized":"2024-09-02T10:26:15.233911+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:01.795140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:01.795140+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:19.964214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"731'7449","reported_seq":15536,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:18.078105+0000","last_change":"2024-09-02T10:25:20.222974+0000","last_active":"2024-09-02T10:26:18.078105+0000","last_peered":"2024-09-02T10:26:18.078105+0000","last_clean":"2024-09-02T10:26:18.078105+0000","last_became_active":"2024-09-02T10:25:20.217872+0000","last_became_peered":"2024-09-02T10:25:20.217872+0000","last_unstale":"2024-09-02T10:26:18.078105+0000","last_undegraded":"2024-09-02T10:26:18.078105+0000","last_fullsized":"2024-09-02T10:26:18.078105+0000","mapping_epoch":705,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:24:06.651660+0000","objects_scrubbed":0,"log_size":7449,"log_dups_size":0,"ondisk_log_size":7449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221446144,"num_objects":3379,"num_object_clones":0,"num_object_copies":10137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3379,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7449,"num_write_kb":346496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":259260416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1476,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.234179+0000","last_change":"2024-09-02T10:26:02.582294+0000","last_active":"2024-09-02T10:26:15.234179+0000","last_peered":"2024-09-02T10:26:15.234179+0000","last_clean":"2024-09-02T10:26:15.234179+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:26:15.234179+0000","last_undegraded":"2024-09-02T10:26:15.234179+0000","last_fullsized":"2024-09-02T10:26:15.234179+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.582251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.582251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:03.315097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"731'7110","reported_seq":7308,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:17.293439+0000","last_change":"2024-09-02T10:26:05.976133+0000","last_active":"2024-09-02T10:26:17.293439+0000","last_peered":"2024-09-02T10:26:17.293439+0000","last_clean":"2024-09-02T10:26:17.293439+0000","last_became_active":"2024-09-02T10:24:07.681130+0000","last_became_peered":"2024-09-02T10:24:07.681130+0000","last_unstale":"2024-09-02T10:26:17.293439+0000","last_undegraded":"2024-09-02T10:26:17.293439+0000","last_fullsized":"2024-09-02T10:26:17.293439+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6487","last_scrub_stamp":"2024-09-02T10:26:05.975865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:05.975865+0000","objects_scrubbed":3945,"log_size":7110,"log_dups_size":0,"ondisk_log_size":7110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:18.517262+0000","scrub_duration":1219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216399872,"num_objects":3302,"num_object_clones":0,"num_object_copies":9906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7110,"num_write_kb":333184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":417,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.232357+0000","last_change":"2024-09-02T10:26:09.806628+0000","last_active":"2024-09-02T10:26:15.232357+0000","last_peered":"2024-09-02T10:26:15.232357+0000","last_clean":"2024-09-02T10:26:15.232357+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:26:15.232357+0000","last_undegraded":"2024-09-02T10:26:15.232357+0000","last_fullsized":"2024-09-02T10:26:15.232357+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.806582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.806582+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:38.336600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"731'7202","reported_seq":7353,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:18.844136+0000","last_change":"2024-09-02T10:24:07.673626+0000","last_active":"2024-09-02T10:26:18.844136+0000","last_peered":"2024-09-02T10:26:18.844136+0000","last_clean":"2024-09-02T10:26:18.844136+0000","last_became_active":"2024-09-02T10:24:07.673332+0000","last_became_peered":"2024-09-02T10:24:07.673332+0000","last_unstale":"2024-09-02T10:26:18.844136+0000","last_undegraded":"2024-09-02T10:26:18.844136+0000","last_fullsized":"2024-09-02T10:26:18.844136+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:24:06.651660+0000","objects_scrubbed":0,"log_size":7202,"log_dups_size":0,"ondisk_log_size":7202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208797696,"num_objects":3186,"num_object_clones":0,"num_object_copies":9558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7202,"num_write_kb":332416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1248,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.232859+0000","last_change":"2024-09-02T10:26:08.915296+0000","last_active":"2024-09-02T10:26:15.232859+0000","last_peered":"2024-09-02T10:26:15.232859+0000","last_clean":"2024-09-02T10:26:15.232859+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:26:15.232859+0000","last_undegraded":"2024-09-02T10:26:15.232859+0000","last_fullsized":"2024-09-02T10:26:15.232859+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:08.915248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.915248+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:30.109523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"731'7140","reported_seq":7337,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:18.118238+0000","last_change":"2024-09-02T10:26:09.242371+0000","last_active":"2024-09-02T10:26:18.118238+0000","last_peered":"2024-09-02T10:26:18.118238+0000","last_clean":"2024-09-02T10:26:18.118238+0000","last_became_active":"2024-09-02T10:24:07.673813+0000","last_became_peered":"2024-09-02T10:24:07.673813+0000","last_unstale":"2024-09-02T10:26:18.118238+0000","last_undegraded":"2024-09-02T10:26:18.118238+0000","last_fullsized":"2024-09-02T10:26:18.118238+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6670","last_scrub_stamp":"2024-09-02T10:26:09.242311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.242311+0000","objects_scrubbed":3772,"log_size":7140,"log_dups_size":0,"ondisk_log_size":7140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:18.078348+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214564864,"num_objects":3274,"num_object_clones":0,"num_object_copies":9822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7140,"num_write_kb":333248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":631,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.301143+0000","last_change":"2024-09-02T10:26:15.301143+0000","last_active":"2024-09-02T10:26:15.301143+0000","last_peered":"2024-09-02T10:26:15.301143+0000","last_clean":"2024-09-02T10:26:15.301143+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:26:15.301143+0000","last_undegraded":"2024-09-02T10:26:15.301143+0000","last_fullsized":"2024-09-02T10:26:15.301143+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:15.300992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:15.300992+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:45.194647+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3456892952,"num_objects":52749,"num_object_clones":0,"num_object_copies":158247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52749,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":115131,"num_write_kb":5372097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30933,"num_bytes_recovered":2027159552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10511986688,"data_stored":10511974472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115131,"ondisk_log_size":115131,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383994,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1648764,"kb_used_data":1080116,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92096388,"statfs":{"total":95995035648,"available":94306701312,"internally_reserved":0,"allocated":1106038784,"data_stored":1105587568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":582089156},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612927,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1065012,"kb_used_data":654452,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92680140,"statfs":{"total":95995035648,"available":94904463360,"internally_reserved":0,"allocated":670158848,"data_stored":669707632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":420018624},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841860,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1696516,"kb_used_data":1051700,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":92048636,"statfs":{"total":95995035648,"available":94257803264,"internally_reserved":0,"allocated":1076940800,"data_stored":1076489584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":660273600},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070795,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1004932,"kb_used_data":626676,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92740220,"statfs":{"total":95995035648,"available":94965985280,"internally_reserved":0,"allocated":641716224,"data_stored":641265008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299728,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1000588,"kb_used_data":405756,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92744564,"statfs":{"total":95995035648,"available":94970433536,"internally_reserved":0,"allocated":415494144,"data_stored":415039376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":609089994},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528658,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1068680,"kb_used_data":650168,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92676472,"statfs":{"total":95995035648,"available":94900707328,"internally_reserved":0,"allocated":665772032,"data_stored":665316744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":428538311},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724888,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1052172,"kb_used_data":429564,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92692980,"statfs":{"total":95995035648,"available":94917611520,"internally_reserved":0,"allocated":439873536,"data_stored":439418768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":637532613},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953821,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1238916,"kb_used_data":828724,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92506236,"statfs":{"total":95995035648,"available":94726385664,"internally_reserved":0,"allocated":848613376,"data_stored":848162160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150050,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2179396,"kb_used_data":1472116,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":91565756,"statfs":{"total":95995035648,"available":93763334144,"internally_reserved":0,"allocated":1507446784,"data_stored":1506995568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":724236744},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411685,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1490316,"kb_used_data":1067644,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":92254836,"statfs":{"total":95995035648,"available":94468952064,"internally_reserved":0,"allocated":1093267456,"data_stored":1092812688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":432798149},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575211,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1406728,"kb_used_data":816056,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92338424,"statfs":{"total":95995035648,"available":94554546176,"internally_reserved":0,"allocated":835641344,"data_stored":835186056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738737,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1982592,"kb_used_data":1279288,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91762560,"statfs":{"total":95995035648,"available":93964861440,"internally_reserved":0,"allocated":1309990912,"data_stored":1309535624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1303515136,"data_stored":1303511064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":830803968,"data_stored":830799896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1087504384,"data_stored":1087504384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1494024192,"data_stored":1494024192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":837156864,"data_stored":837156864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":431882240,"data_stored":431882240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":655101952,"data_stored":655097880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":407502848,"data_stored":407502848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":635568128,"data_stored":635568128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1075642368,"data_stored":1075642368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":663617536,"data_stored":663617536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1089667072,"data_stored":1089667072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:26:26.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:26:26.435 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:26.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:26.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:26.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:26.599 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:26.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:26.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:26.724 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:26.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:26.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:26.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:26.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:26.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:26 smithi151 ceph-mon[16639]: pgmap v1675: 47 pgs: 47 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:26 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2027674304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:26:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:26 smithi151 ceph-mon[16639]: osdmap e733: 12 total, 12 up, 12 in 2024-09-02T10:26:26.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:26.984 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:26.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:26 smithi187 ceph-mon[20072]: pgmap v1675: 47 pgs: 47 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:26 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2027674304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:26:27.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:26 smithi187 ceph-mon[20072]: osdmap e733: 12 total, 12 up, 12 in 2024-09-02T10:26:27.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:26 smithi162 ceph-mon[20279]: pgmap v1675: 47 pgs: 47 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 876 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:26 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2027674304' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:26:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:26 smithi162 ceph-mon[20279]: osdmap e733: 12 total, 12 up, 12 in 2024-09-02T10:26:27.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:27.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:27.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:27.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:27.241 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:27.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:27.343 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:27.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:27.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:27.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:27.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:27.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:27.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:27.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:26:27.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:27.618 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:27.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:27.670 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:27.673 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:27.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:27.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:27.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:27.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:27.863 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:27.887 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:27.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:27.989 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:28.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:28.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:28.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:28.148 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:28.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:28.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:28.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:28.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:28.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:28.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:28.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:28.397 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:28.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:28.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:28.460 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:26:28.461 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:26:28.492 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:26:28.492 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:26:28.499 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:28.523 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:28.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:28.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:28.526 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:26:28.526 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:26:28.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:26:28.558 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:26:28.588 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:26:28.588 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:26:28.623 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:26:28.623 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:26:28.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:28.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:28.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:28.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:28.657 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:26:28.657 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:26:28.658 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:28 smithi151 ceph-mon[16639]: pgmap v1677: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s 2024-09-02T10:26:28.658 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:28 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:28.658 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:28 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:28.658 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:28 smithi151 ceph-mon[16639]: osdmap e734: 12 total, 12 up, 12 in 2024-09-02T10:26:28.658 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3976695949' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:28.691 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:26:28.691 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:26:28.727 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:26:28.728 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:26:28.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:28.759 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:26:28.759 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:26:28.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:28.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:28.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:28.797 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:26:28.797 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:26:28.811 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:28.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:28.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:28.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:28.911 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:28 smithi187 ceph-mon[20072]: pgmap v1677: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s 2024-09-02T10:26:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:28 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:28 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:28 smithi187 ceph-mon[20072]: osdmap e734: 12 total, 12 up, 12 in 2024-09-02T10:26:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3976695949' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:29.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:29.036 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:29.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:29.039 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:26:29.040 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:26:29.052 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:26:29.052 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:26:29.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:26:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:28 smithi162 ceph-mon[20279]: pgmap v1677: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s 2024-09-02T10:26:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:28 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:28 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:28 smithi162 ceph-mon[20279]: osdmap e734: 12 total, 12 up, 12 in 2024-09-02T10:26:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3976695949' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:29.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:29.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:29.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:29.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:29.305 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:29.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:29.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:29.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:29.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:29.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:29.560 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:29.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:29.662 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:29.684 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:29.685 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:29.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:29.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:29.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:29.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:29 smithi151 ceph-mon[16639]: from='client.18141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:26:29.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:29.946 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:29.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:29 smithi187 ceph-mon[20072]: from='client.18141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:26:30.046 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:30.073 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:30.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:30.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:30.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:29 smithi162 ceph-mon[20279]: from='client.18141 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:26:30.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:30.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:30.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:30.306 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:30.338 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:30.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:30.339 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:30.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:30.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:30.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:30.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:30.572 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:30.594 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:30.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:30.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:30.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:30.725 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:30.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:30.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:30.827 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:30.859 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:30.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:30 smithi151 ceph-mon[16639]: pgmap v1679: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s 2024-09-02T10:26:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:30 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:30 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:30.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:30 smithi151 ceph-mon[16639]: osdmap e735: 12 total, 12 up, 12 in 2024-09-02T10:26:30.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:30.985 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:30.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:30.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:30 smithi187 ceph-mon[20072]: pgmap v1679: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s 2024-09-02T10:26:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:30 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:30 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:30 smithi187 ceph-mon[20072]: osdmap e735: 12 total, 12 up, 12 in 2024-09-02T10:26:31.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:31.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:30 smithi162 ceph-mon[20279]: pgmap v1679: 47 pgs: 47 active+clean; 2.7 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 868 op/s 2024-09-02T10:26:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:30 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:30 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:30 smithi162 ceph-mon[20279]: osdmap e735: 12 total, 12 up, 12 in 2024-09-02T10:26:31.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:31.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:31.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:31.250 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:31.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:31.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:31.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:31.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:31.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:31.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:31.504 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:31.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:31.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:31.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:31.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:31.629 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:31.630 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:31.656 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1678,"stamp":"2024-09-02T10:26:27.651649+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3031499320,"num_objects":46251,"num_object_clones":0,"num_object_copies":138753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46251,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121631,"num_write_kb":5372097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30937,"num_bytes_recovered":2028090976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":124483,"ondisk_log_size":124483,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":15673844,"kb_used_data":9153404,"kb_used_omap":18,"kb_used_meta":6518573,"kb_avail":1109267980,"statfs":{"total":1151940427776,"available":1135890411520,"internally_reserved":0,"allocated":9373085696,"data_stored":9368570417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":6675019067},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,7,8,8],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-425984000,"num_objects":-6500,"num_object_clones":0,"num_object_copies":-19500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6500,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6500,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.190058"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":845,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.636489+0000","last_change":"2024-09-02T10:26:02.405331+0000","last_active":"2024-09-02T10:26:25.636489+0000","last_peered":"2024-09-02T10:26:25.636489+0000","last_clean":"2024-09-02T10:26:25.636489+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:26:25.636489+0000","last_undegraded":"2024-09-02T10:26:25.636489+0000","last_fullsized":"2024-09-02T10:26:25.636489+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.405283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.405283+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:17.160849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":650,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.641444+0000","last_change":"2024-09-02T10:26:14.793294+0000","last_active":"2024-09-02T10:26:23.641444+0000","last_peered":"2024-09-02T10:26:23.641444+0000","last_clean":"2024-09-02T10:26:23.641444+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:26:23.641444+0000","last_undegraded":"2024-09-02T10:26:23.641444+0000","last_fullsized":"2024-09-02T10:26:23.641444+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:14.793044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.793044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:31.973724+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1116,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.622108+0000","last_change":"2024-09-02T10:26:09.987463+0000","last_active":"2024-09-02T10:26:23.622108+0000","last_peered":"2024-09-02T10:26:23.622108+0000","last_clean":"2024-09-02T10:26:23.622108+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:26:23.622108+0000","last_undegraded":"2024-09-02T10:26:23.622108+0000","last_fullsized":"2024-09-02T10:26:23.622108+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.987389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.987389+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:39.899046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1100,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.621969+0000","last_change":"2024-09-02T10:25:22.185960+0000","last_active":"2024-09-02T10:26:23.621969+0000","last_peered":"2024-09-02T10:26:23.621969+0000","last_clean":"2024-09-02T10:26:23.621969+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:26:23.621969+0000","last_undegraded":"2024-09-02T10:26:23.621969+0000","last_fullsized":"2024-09-02T10:26:23.621969+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:25:22.185894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:25:22.185894+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:26:25.443076+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":403,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.630393+0000","last_change":"2024-09-02T10:26:06.766310+0000","last_active":"2024-09-02T10:26:23.630393+0000","last_peered":"2024-09-02T10:26:23.630393+0000","last_clean":"2024-09-02T10:26:23.630393+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:26:23.630393+0000","last_undegraded":"2024-09-02T10:26:23.630393+0000","last_fullsized":"2024-09-02T10:26:23.630393+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:06.766238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:06.766238+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:09.577073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":964,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.649550+0000","last_change":"2024-09-02T10:26:12.294448+0000","last_active":"2024-09-02T10:26:25.649550+0000","last_peered":"2024-09-02T10:26:25.649550+0000","last_clean":"2024-09-02T10:26:25.649550+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:26:25.649550+0000","last_undegraded":"2024-09-02T10:26:25.649550+0000","last_fullsized":"2024-09-02T10:26:25.649550+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:12.294402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:12.294402+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:13.584271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1484,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.637363+0000","last_change":"2024-09-02T10:26:04.538470+0000","last_active":"2024-09-02T10:26:25.637363+0000","last_peered":"2024-09-02T10:26:25.637363+0000","last_clean":"2024-09-02T10:26:25.637363+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:26:25.637363+0000","last_undegraded":"2024-09-02T10:26:25.637363+0000","last_fullsized":"2024-09-02T10:26:25.637363+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:04.538409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:04.538409+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:23.977037+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1124,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.623299+0000","last_change":"2024-09-02T10:26:08.951582+0000","last_active":"2024-09-02T10:26:23.623299+0000","last_peered":"2024-09-02T10:26:23.623299+0000","last_clean":"2024-09-02T10:26:23.623299+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:26:23.623299+0000","last_undegraded":"2024-09-02T10:26:23.623299+0000","last_fullsized":"2024-09-02T10:26:23.623299+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:08.951514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.951514+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:10.021565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":1129,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.753612+0000","last_change":"2024-09-02T10:26:23.753612+0000","last_active":"2024-09-02T10:26:23.753612+0000","last_peered":"2024-09-02T10:26:23.753612+0000","last_clean":"2024-09-02T10:26:23.753612+0000","last_became_active":"2024-09-02T10:25:01.211125+0000","last_became_peered":"2024-09-02T10:25:01.211125+0000","last_unstale":"2024-09-02T10:26:23.753612+0000","last_undegraded":"2024-09-02T10:26:23.753612+0000","last_fullsized":"2024-09-02T10:26:23.753612+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:23.753547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:23.753547+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:43.401582+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1093,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.636482+0000","last_change":"2024-09-02T10:26:05.853283+0000","last_active":"2024-09-02T10:26:25.636482+0000","last_peered":"2024-09-02T10:26:25.636482+0000","last_clean":"2024-09-02T10:26:25.636482+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:26:25.636482+0000","last_undegraded":"2024-09-02T10:26:25.636482+0000","last_fullsized":"2024-09-02T10:26:25.636482+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:05.853228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:05.853228+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:33.199800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":337,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.647970+0000","last_change":"2024-09-02T10:25:20.120097+0000","last_active":"2024-09-02T10:26:25.647970+0000","last_peered":"2024-09-02T10:26:25.647970+0000","last_clean":"2024-09-02T10:26:25.647970+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:26:25.647970+0000","last_undegraded":"2024-09-02T10:26:25.647970+0000","last_fullsized":"2024-09-02T10:26:25.647970+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:25:20.120024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:25:20.120024+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:26:47.878922+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1441,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.641216+0000","last_change":"2024-09-02T10:26:16.774967+0000","last_active":"2024-09-02T10:26:23.641216+0000","last_peered":"2024-09-02T10:26:23.641216+0000","last_clean":"2024-09-02T10:26:23.641216+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:26:23.641216+0000","last_undegraded":"2024-09-02T10:26:23.641216+0000","last_fullsized":"2024-09-02T10:26:23.641216+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:16.774772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:16.774772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:37.484948+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1383,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.637446+0000","last_change":"2024-09-02T10:26:09.562148+0000","last_active":"2024-09-02T10:26:25.637446+0000","last_peered":"2024-09-02T10:26:25.637446+0000","last_clean":"2024-09-02T10:26:25.637446+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:26:25.637446+0000","last_undegraded":"2024-09-02T10:26:25.637446+0000","last_fullsized":"2024-09-02T10:26:25.637446+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.562104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.562104+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:16.950256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":102,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.232206+0000","last_change":"2024-09-02T10:26:14.840384+0000","last_active":"2024-09-02T10:26:15.232206+0000","last_peered":"2024-09-02T10:26:15.232206+0000","last_clean":"2024-09-02T10:26:15.232206+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:26:15.232206+0000","last_undegraded":"2024-09-02T10:26:15.232206+0000","last_fullsized":"2024-09-02T10:26:15.232206+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:14.840310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.840310+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:19.191366+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1353,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.637437+0000","last_change":"2024-09-02T10:26:06.555868+0000","last_active":"2024-09-02T10:26:25.637437+0000","last_peered":"2024-09-02T10:26:25.637437+0000","last_clean":"2024-09-02T10:26:25.637437+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:26:25.637437+0000","last_undegraded":"2024-09-02T10:26:25.637437+0000","last_fullsized":"2024-09-02T10:26:25.637437+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:06.555813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:06.555813+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:17.827776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"733'7517","reported_seq":7720,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.221526+0000","last_change":"2024-09-02T10:25:20.361425+0000","last_active":"2024-09-02T10:26:27.221526+0000","last_peered":"2024-09-02T10:26:27.221526+0000","last_clean":"2024-09-02T10:26:27.221526+0000","last_became_active":"2024-09-02T10:24:07.681518+0000","last_became_peered":"2024-09-02T10:24:07.681518+0000","last_unstale":"2024-09-02T10:26:27.221526+0000","last_undegraded":"2024-09-02T10:26:27.221526+0000","last_fullsized":"2024-09-02T10:26:27.221526+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"706'3999","last_scrub_stamp":"2024-09-02T10:25:20.360400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:25:20.360400+0000","objects_scrubbed":3994,"log_size":7517,"log_dups_size":0,"ondisk_log_size":7517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:26:38.772833+0000","scrub_duration":1241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180813824,"num_objects":2759,"num_object_clones":0,"num_object_copies":8277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7517,"num_write_kb":328832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1447,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.637475+0000","last_change":"2024-09-02T10:26:02.497906+0000","last_active":"2024-09-02T10:26:25.637475+0000","last_peered":"2024-09-02T10:26:25.637475+0000","last_clean":"2024-09-02T10:26:25.637475+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:26:25.637475+0000","last_undegraded":"2024-09-02T10:26:25.637475+0000","last_fullsized":"2024-09-02T10:26:25.637475+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.497853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.497853+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:21.295805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"731'7657","reported_seq":15936,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.080952+0000","last_change":"2024-09-02T10:25:21.220340+0000","last_active":"2024-09-02T10:26:23.080952+0000","last_peered":"2024-09-02T10:26:23.080952+0000","last_clean":"2024-09-02T10:26:23.080952+0000","last_became_active":"2024-09-02T10:25:21.217238+0000","last_became_peered":"2024-09-02T10:25:21.217238+0000","last_unstale":"2024-09-02T10:26:23.080952+0000","last_undegraded":"2024-09-02T10:26:23.080952+0000","last_fullsized":"2024-09-02T10:26:23.080952+0000","mapping_epoch":706,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":707,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:24:06.651660+0000","objects_scrubbed":0,"log_size":7657,"log_dups_size":0,"ondisk_log_size":7657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197722112,"num_objects":3017,"num_object_clones":0,"num_object_copies":9051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7657,"num_write_kb":341568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4053,"num_bytes_recovered":265617408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1572,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.636377+0000","last_change":"2024-09-02T10:26:09.933550+0000","last_active":"2024-09-02T10:26:25.636377+0000","last_peered":"2024-09-02T10:26:25.636377+0000","last_clean":"2024-09-02T10:26:25.636377+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:26:25.636377+0000","last_undegraded":"2024-09-02T10:26:25.636377+0000","last_fullsized":"2024-09-02T10:26:25.636377+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.933487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.933487+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:19.759531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"733'7673","reported_seq":16612,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.374367+0000","last_change":"2024-09-02T10:26:13.178680+0000","last_active":"2024-09-02T10:26:27.374367+0000","last_peered":"2024-09-02T10:26:27.374367+0000","last_clean":"2024-09-02T10:26:27.374367+0000","last_became_active":"2024-09-02T10:25:44.212783+0000","last_became_peered":"2024-09-02T10:25:44.212783+0000","last_unstale":"2024-09-02T10:26:27.374367+0000","last_undegraded":"2024-09-02T10:26:27.374367+0000","last_fullsized":"2024-09-02T10:26:27.374367+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"729'6901","last_scrub_stamp":"2024-09-02T10:26:13.178572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.178572+0000","objects_scrubbed":3559,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:39.044105+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181731328,"num_objects":2773,"num_object_clones":0,"num_object_copies":8319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7673,"num_write_kb":334272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4359,"num_bytes_recovered":285671424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1211,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.240372+0000","last_change":"2024-09-02T10:26:02.369141+0000","last_active":"2024-09-02T10:26:15.240372+0000","last_peered":"2024-09-02T10:26:15.240372+0000","last_clean":"2024-09-02T10:26:15.240372+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:26:15.240372+0000","last_undegraded":"2024-09-02T10:26:15.240372+0000","last_fullsized":"2024-09-02T10:26:15.240372+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.369090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.369090+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:20.449364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.d","version":"733'7755","reported_seq":7958,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.406876+0000","last_change":"2024-09-02T10:26:13.549346+0000","last_active":"2024-09-02T10:26:27.406876+0000","last_peered":"2024-09-02T10:26:27.406876+0000","last_clean":"2024-09-02T10:26:27.406876+0000","last_became_active":"2024-09-02T10:24:07.671900+0000","last_became_peered":"2024-09-02T10:24:07.671900+0000","last_unstale":"2024-09-02T10:26:27.406876+0000","last_undegraded":"2024-09-02T10:26:27.406876+0000","last_fullsized":"2024-09-02T10:26:27.406876+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"729'6967","last_scrub_stamp":"2024-09-02T10:26:13.549295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.549295+0000","objects_scrubbed":3613,"log_size":7755,"log_dups_size":0,"ondisk_log_size":7755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:30.818673+0000","scrub_duration":1966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181796888,"num_objects":2775,"num_object_clones":0,"num_object_copies":8325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2775,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7755,"num_write_kb":336897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":611,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.636638+0000","last_change":"2024-09-02T10:26:07.441225+0000","last_active":"2024-09-02T10:26:25.636638+0000","last_peered":"2024-09-02T10:26:25.636638+0000","last_clean":"2024-09-02T10:26:25.636638+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:26:25.636638+0000","last_undegraded":"2024-09-02T10:26:25.636638+0000","last_fullsized":"2024-09-02T10:26:25.636638+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:07.441159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:07.441159+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:21.111255+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"731'7407","reported_seq":7604,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.553030+0000","last_change":"2024-09-02T10:26:10.244647+0000","last_active":"2024-09-02T10:26:23.553030+0000","last_peered":"2024-09-02T10:26:23.553030+0000","last_clean":"2024-09-02T10:26:23.553030+0000","last_became_active":"2024-09-02T10:24:07.681580+0000","last_became_peered":"2024-09-02T10:24:07.681580+0000","last_unstale":"2024-09-02T10:26:23.553030+0000","last_undegraded":"2024-09-02T10:26:23.553030+0000","last_fullsized":"2024-09-02T10:26:23.553030+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'6721","last_scrub_stamp":"2024-09-02T10:26:10.244419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:10.244419+0000","objects_scrubbed":3718,"log_size":7407,"log_dups_size":0,"ondisk_log_size":7407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:18.436180+0000","scrub_duration":1044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197197824,"num_objects":3009,"num_object_clones":0,"num_object_copies":9027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7407,"num_write_kb":333312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1520,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.636521+0000","last_change":"2024-09-02T10:26:13.596384+0000","last_active":"2024-09-02T10:26:25.636521+0000","last_peered":"2024-09-02T10:26:25.636521+0000","last_clean":"2024-09-02T10:26:25.636521+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:26:25.636521+0000","last_undegraded":"2024-09-02T10:26:25.636521+0000","last_fullsized":"2024-09-02T10:26:25.636521+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:13.596339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.596339+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:42.860522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"731'7369","reported_seq":7568,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.577741+0000","last_change":"2024-09-02T10:26:08.197217+0000","last_active":"2024-09-02T10:26:23.577741+0000","last_peered":"2024-09-02T10:26:23.577741+0000","last_clean":"2024-09-02T10:26:23.577741+0000","last_became_active":"2024-09-02T10:24:07.674582+0000","last_became_peered":"2024-09-02T10:24:07.674582+0000","last_unstale":"2024-09-02T10:26:23.577741+0000","last_undegraded":"2024-09-02T10:26:23.577741+0000","last_fullsized":"2024-09-02T10:26:23.577741+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6563","last_scrub_stamp":"2024-09-02T10:26:08.197134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.197134+0000","objects_scrubbed":3761,"log_size":7369,"log_dups_size":0,"ondisk_log_size":7369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:38.138983+0000","scrub_duration":1013,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191954944,"num_objects":2929,"num_object_clones":0,"num_object_copies":8787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7369,"num_write_kb":329536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":986,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.649573+0000","last_change":"2024-09-02T10:26:15.352217+0000","last_active":"2024-09-02T10:26:25.649573+0000","last_peered":"2024-09-02T10:26:25.649573+0000","last_clean":"2024-09-02T10:26:25.649573+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:26:25.649573+0000","last_undegraded":"2024-09-02T10:26:25.649573+0000","last_fullsized":"2024-09-02T10:26:25.649573+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:15.352153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:15.352153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:33.222325+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"733'7778","reported_seq":8018,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.538679+0000","last_change":"2024-09-02T10:26:16.241362+0000","last_active":"2024-09-02T10:26:27.538679+0000","last_peered":"2024-09-02T10:26:27.538679+0000","last_clean":"2024-09-02T10:26:27.538679+0000","last_became_active":"2024-09-02T10:24:07.673747+0000","last_became_peered":"2024-09-02T10:24:07.673747+0000","last_unstale":"2024-09-02T10:26:27.538679+0000","last_undegraded":"2024-09-02T10:26:27.538679+0000","last_fullsized":"2024-09-02T10:26:27.538679+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'7131","last_scrub_stamp":"2024-09-02T10:26:16.241303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:16.241303+0000","objects_scrubbed":3422,"log_size":7778,"log_dups_size":0,"ondisk_log_size":7778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:29.657946+0000","scrub_duration":564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180879360,"num_objects":2760,"num_object_clones":0,"num_object_copies":8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7778,"num_write_kb":337216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":951,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.652312+0000","last_change":"2024-09-02T10:26:09.304259+0000","last_active":"2024-09-02T10:26:25.652312+0000","last_peered":"2024-09-02T10:26:25.652312+0000","last_clean":"2024-09-02T10:26:25.652312+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:26:25.652312+0000","last_undegraded":"2024-09-02T10:26:25.652312+0000","last_fullsized":"2024-09-02T10:26:25.652312+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.303950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.303950+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:10.581116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"732'7519","reported_seq":16553,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:24.581492+0000","last_change":"2024-09-02T10:26:10.642407+0000","last_active":"2024-09-02T10:26:24.581492+0000","last_peered":"2024-09-02T10:26:24.581492+0000","last_clean":"2024-09-02T10:26:24.581492+0000","last_became_active":"2024-09-02T10:25:44.211739+0000","last_became_peered":"2024-09-02T10:25:44.211739+0000","last_unstale":"2024-09-02T10:26:24.581492+0000","last_undegraded":"2024-09-02T10:26:24.581492+0000","last_fullsized":"2024-09-02T10:26:24.581492+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'6786","last_scrub_stamp":"2024-09-02T10:26:10.642208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:10.642208+0000","objects_scrubbed":3711,"log_size":7519,"log_dups_size":0,"ondisk_log_size":7519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:27.257300+0000","scrub_duration":1800,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192086016,"num_objects":2931,"num_object_clones":0,"num_object_copies":8793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7519,"num_write_kb":334400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4407,"num_bytes_recovered":288817152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":733,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.632977+0000","last_change":"2024-09-02T10:26:14.593855+0000","last_active":"2024-09-02T10:26:25.632977+0000","last_peered":"2024-09-02T10:26:25.632977+0000","last_clean":"2024-09-02T10:26:25.632977+0000","last_became_active":"2024-09-02T10:25:02.065175+0000","last_became_peered":"2024-09-02T10:25:02.065175+0000","last_unstale":"2024-09-02T10:26:25.632977+0000","last_undegraded":"2024-09-02T10:26:25.632977+0000","last_fullsized":"2024-09-02T10:26:25.632977+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:14.593805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.593805+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:42.832993+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"733'7647","reported_seq":7847,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.397578+0000","last_change":"2024-09-02T10:26:11.397647+0000","last_active":"2024-09-02T10:26:27.397578+0000","last_peered":"2024-09-02T10:26:27.397578+0000","last_clean":"2024-09-02T10:26:27.397578+0000","last_became_active":"2024-09-02T10:24:07.670909+0000","last_became_peered":"2024-09-02T10:24:07.670909+0000","last_unstale":"2024-09-02T10:26:27.397578+0000","last_undegraded":"2024-09-02T10:26:27.397578+0000","last_fullsized":"2024-09-02T10:26:27.397578+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'6764","last_scrub_stamp":"2024-09-02T10:26:11.397518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:11.397518+0000","objects_scrubbed":3666,"log_size":7647,"log_dups_size":0,"ondisk_log_size":7647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:13.343075+0000","scrub_duration":854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181075968,"num_objects":2763,"num_object_clones":0,"num_object_copies":8289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7647,"num_write_kb":333120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":966,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.649656+0000","last_change":"2024-09-02T10:26:08.256406+0000","last_active":"2024-09-02T10:26:25.649656+0000","last_peered":"2024-09-02T10:26:25.649656+0000","last_clean":"2024-09-02T10:26:25.649656+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:26:25.649656+0000","last_undegraded":"2024-09-02T10:26:25.649656+0000","last_fullsized":"2024-09-02T10:26:25.649656+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:08.256332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.256332+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:32.379501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.4","version":"731'7663","reported_seq":18826,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.061750+0000","last_change":"2024-09-02T10:26:14.088498+0000","last_active":"2024-09-02T10:26:23.061750+0000","last_peered":"2024-09-02T10:26:23.061750+0000","last_clean":"2024-09-02T10:26:23.061750+0000","last_became_active":"2024-09-02T10:26:14.049049+0000","last_became_peered":"2024-09-02T10:26:14.049049+0000","last_unstale":"2024-09-02T10:26:23.061750+0000","last_undegraded":"2024-09-02T10:26:23.061750+0000","last_fullsized":"2024-09-02T10:26:23.061750+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:24:06.651660+0000","objects_scrubbed":0,"log_size":7663,"log_dups_size":0,"ondisk_log_size":7663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199950336,"num_objects":3051,"num_object_clones":0,"num_object_copies":9153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7663,"num_write_kb":342848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5491,"num_bytes_recovered":359792640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1214,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.636430+0000","last_change":"2024-09-02T10:26:11.944736+0000","last_active":"2024-09-02T10:26:25.636430+0000","last_peered":"2024-09-02T10:26:25.636430+0000","last_clean":"2024-09-02T10:26:25.636430+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:26:25.636430+0000","last_undegraded":"2024-09-02T10:26:25.636430+0000","last_fullsized":"2024-09-02T10:26:25.636430+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:11.944689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:11.944689+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:23.172770+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"733'7682","reported_seq":16314,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.275300+0000","last_change":"2024-09-02T10:26:06.663497+0000","last_active":"2024-09-02T10:26:27.275300+0000","last_peered":"2024-09-02T10:26:27.275300+0000","last_clean":"2024-09-02T10:26:27.275300+0000","last_became_active":"2024-09-02T10:25:28.620158+0000","last_became_peered":"2024-09-02T10:25:28.620158+0000","last_unstale":"2024-09-02T10:26:27.275300+0000","last_undegraded":"2024-09-02T10:26:27.275300+0000","last_fullsized":"2024-09-02T10:26:27.275300+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6440","last_scrub_stamp":"2024-09-02T10:26:06.663370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:06.663370+0000","objects_scrubbed":3970,"log_size":7682,"log_dups_size":0,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:12.071995+0000","scrub_duration":1255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176422912,"num_objects":2692,"num_object_clones":0,"num_object_copies":8076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7682,"num_write_kb":331968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4199,"num_bytes_recovered":275185664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1472,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.641240+0000","last_change":"2024-09-02T10:26:22.765464+0000","last_active":"2024-09-02T10:26:23.641240+0000","last_peered":"2024-09-02T10:26:23.641240+0000","last_clean":"2024-09-02T10:26:23.641240+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:26:23.641240+0000","last_undegraded":"2024-09-02T10:26:23.641240+0000","last_fullsized":"2024-09-02T10:26:23.641240+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:22.765406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:22.765406+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:38.157020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"731'7645","reported_seq":16799,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.051524+0000","last_change":"2024-09-02T10:26:14.504599+0000","last_active":"2024-09-02T10:26:23.051524+0000","last_peered":"2024-09-02T10:26:23.051524+0000","last_clean":"2024-09-02T10:26:23.051524+0000","last_became_active":"2024-09-02T10:26:02.534985+0000","last_became_peered":"2024-09-02T10:26:02.534985+0000","last_unstale":"2024-09-02T10:26:23.051524+0000","last_undegraded":"2024-09-02T10:26:23.051524+0000","last_fullsized":"2024-09-02T10:26:23.051524+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'7133","last_scrub_stamp":"2024-09-02T10:26:14.504505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.504505+0000","objects_scrubbed":3617,"log_size":7645,"log_dups_size":0,"ondisk_log_size":7645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:24.932727+0000","scrub_duration":694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200998912,"num_objects":3067,"num_object_clones":0,"num_object_copies":9201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7645,"num_write_kb":342784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4468,"num_bytes_recovered":292814848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1217,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.636582+0000","last_change":"2024-09-02T10:26:01.795185+0000","last_active":"2024-09-02T10:26:25.636582+0000","last_peered":"2024-09-02T10:26:25.636582+0000","last_clean":"2024-09-02T10:26:25.636582+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:26:25.636582+0000","last_undegraded":"2024-09-02T10:26:25.636582+0000","last_fullsized":"2024-09-02T10:26:25.636582+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:01.795140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:01.795140+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:19.964214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"731'7731","reported_seq":15862,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.072045+0000","last_change":"2024-09-02T10:26:20.104150+0000","last_active":"2024-09-02T10:26:23.072045+0000","last_peered":"2024-09-02T10:26:23.072045+0000","last_clean":"2024-09-02T10:26:23.072045+0000","last_became_active":"2024-09-02T10:25:20.217872+0000","last_became_peered":"2024-09-02T10:25:20.217872+0000","last_unstale":"2024-09-02T10:26:23.072045+0000","last_undegraded":"2024-09-02T10:26:23.072045+0000","last_fullsized":"2024-09-02T10:26:23.072045+0000","mapping_epoch":705,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'7570","last_scrub_stamp":"2024-09-02T10:26:20.103959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:20.103959+0000","objects_scrubbed":3294,"log_size":7731,"log_dups_size":0,"ondisk_log_size":7731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:27.513393+0000","scrub_duration":1260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202964992,"num_objects":3097,"num_object_clones":0,"num_object_copies":9291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7731,"num_write_kb":346496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":259260416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1480,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.633055+0000","last_change":"2024-09-02T10:26:02.582294+0000","last_active":"2024-09-02T10:26:25.633055+0000","last_peered":"2024-09-02T10:26:25.633055+0000","last_clean":"2024-09-02T10:26:25.633055+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:26:25.633055+0000","last_undegraded":"2024-09-02T10:26:25.633055+0000","last_fullsized":"2024-09-02T10:26:25.633055+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.582251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.582251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:03.315097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"733'7670","reported_seq":7872,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.287892+0000","last_change":"2024-09-02T10:26:05.976133+0000","last_active":"2024-09-02T10:26:27.287892+0000","last_peered":"2024-09-02T10:26:27.287892+0000","last_clean":"2024-09-02T10:26:27.287892+0000","last_became_active":"2024-09-02T10:24:07.681130+0000","last_became_peered":"2024-09-02T10:24:07.681130+0000","last_unstale":"2024-09-02T10:26:27.287892+0000","last_undegraded":"2024-09-02T10:26:27.287892+0000","last_fullsized":"2024-09-02T10:26:27.287892+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6487","last_scrub_stamp":"2024-09-02T10:26:05.975865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:05.975865+0000","objects_scrubbed":3945,"log_size":7670,"log_dups_size":0,"ondisk_log_size":7670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:18.517262+0000","scrub_duration":1219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179699712,"num_objects":2742,"num_object_clones":0,"num_object_copies":8226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2742,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7670,"num_write_kb":333184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":419,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.630447+0000","last_change":"2024-09-02T10:26:09.806628+0000","last_active":"2024-09-02T10:26:23.630447+0000","last_peered":"2024-09-02T10:26:23.630447+0000","last_clean":"2024-09-02T10:26:23.630447+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:26:23.630447+0000","last_undegraded":"2024-09-02T10:26:23.630447+0000","last_fullsized":"2024-09-02T10:26:23.630447+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.806582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.806582+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:38.336600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"732'7492","reported_seq":7683,"reported_epoch":732,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.804717+0000","last_change":"2024-09-02T10:26:22.680654+0000","last_active":"2024-09-02T10:26:23.804717+0000","last_peered":"2024-09-02T10:26:23.804717+0000","last_clean":"2024-09-02T10:26:23.804717+0000","last_became_active":"2024-09-02T10:24:07.673332+0000","last_became_peered":"2024-09-02T10:24:07.673332+0000","last_unstale":"2024-09-02T10:26:23.804717+0000","last_undegraded":"2024-09-02T10:26:23.804717+0000","last_fullsized":"2024-09-02T10:26:23.804717+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'7425","last_scrub_stamp":"2024-09-02T10:26:22.680454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:22.680454+0000","objects_scrubbed":2997,"log_size":7492,"log_dups_size":0,"ondisk_log_size":7492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:50.130765+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189792256,"num_objects":2896,"num_object_clones":0,"num_object_copies":8688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2896,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7492,"num_write_kb":332416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1252,"reported_epoch":733,"state":"active+clean","last_fresh":"2024-09-02T10:26:25.636502+0000","last_change":"2024-09-02T10:26:08.915296+0000","last_active":"2024-09-02T10:26:25.636502+0000","last_peered":"2024-09-02T10:26:25.636502+0000","last_clean":"2024-09-02T10:26:25.636502+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:26:25.636502+0000","last_undegraded":"2024-09-02T10:26:25.636502+0000","last_fullsized":"2024-09-02T10:26:25.636502+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:08.915248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.915248+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:30.109523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"731'7426","reported_seq":7624,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:23.123814+0000","last_change":"2024-09-02T10:26:09.242371+0000","last_active":"2024-09-02T10:26:23.123814+0000","last_peered":"2024-09-02T10:26:23.123814+0000","last_clean":"2024-09-02T10:26:23.123814+0000","last_became_active":"2024-09-02T10:24:07.673813+0000","last_became_peered":"2024-09-02T10:24:07.673813+0000","last_unstale":"2024-09-02T10:26:23.123814+0000","last_undegraded":"2024-09-02T10:26:23.123814+0000","last_fullsized":"2024-09-02T10:26:23.123814+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6670","last_scrub_stamp":"2024-09-02T10:26:09.242311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.242311+0000","objects_scrubbed":3772,"log_size":7426,"log_dups_size":0,"ondisk_log_size":7426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:18.078348+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195821568,"num_objects":2988,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7426,"num_write_kb":333248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":631,"reported_epoch":731,"state":"active+clean","last_fresh":"2024-09-02T10:26:15.301143+0000","last_change":"2024-09-02T10:26:15.301143+0000","last_active":"2024-09-02T10:26:15.301143+0000","last_peered":"2024-09-02T10:26:15.301143+0000","last_clean":"2024-09-02T10:26:15.301143+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:26:15.301143+0000","last_undegraded":"2024-09-02T10:26:15.301143+0000","last_fullsized":"2024-09-02T10:26:15.301143+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:15.300992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:15.300992+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:45.194647+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3030908952,"num_objects":46249,"num_object_clones":0,"num_object_copies":138747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46249,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":121631,"num_write_kb":5372097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30933,"num_bytes_recovered":2027159552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9282990080,"data_stored":9282977864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":121631,"ondisk_log_size":121631,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383996,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1499692,"kb_used_data":900516,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92245460,"statfs":{"total":95995035648,"available":94459351040,"internally_reserved":0,"allocated":922128384,"data_stored":921800200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":613349828},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612929,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":964268,"kb_used_data":545572,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92780884,"statfs":{"total":95995035648,"available":95007625216,"internally_reserved":0,"allocated":558665728,"data_stored":558337544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":428538304},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841861,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1613316,"kb_used_data":960180,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":92131836,"statfs":{"total":95995035648,"available":94343000064,"internally_reserved":0,"allocated":983224320,"data_stored":982773104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":668793280},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070796,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":955796,"kb_used_data":573196,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92789356,"statfs":{"total":95995035648,"available":95016300544,"internally_reserved":0,"allocated":586952704,"data_stored":586562819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299729,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":968428,"kb_used_data":369068,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92776724,"statfs":{"total":95995035648,"available":95003365376,"internally_reserved":0,"allocated":377925632,"data_stored":377593896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":613349834},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528660,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":950960,"kb_used_data":540584,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92794192,"statfs":{"total":95995035648,"available":95021252608,"internally_reserved":0,"allocated":553558016,"data_stored":553225760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724890,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":987508,"kb_used_data":356396,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":92757644,"statfs":{"total":95995035648,"available":94983827456,"internally_reserved":0,"allocated":364949504,"data_stored":364617768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953822,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1171796,"kb_used_data":757260,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92573356,"statfs":{"total":95995035648,"available":94795116544,"internally_reserved":0,"allocated":775434240,"data_stored":775044355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150051,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2042244,"kb_used_data":1343284,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":91702908,"statfs":{"total":95995035648,"available":93903777792,"internally_reserved":0,"allocated":1375522816,"data_stored":1375071600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":715717064},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411687,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302588,"kb_used_data":888236,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92442564,"statfs":{"total":95995035648,"available":94661185536,"internally_reserved":0,"allocated":909553664,"data_stored":909221928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575212,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1340376,"kb_used_data":749520,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92404776,"statfs":{"total":95995035648,"available":94622490624,"internally_reserved":0,"allocated":767508480,"data_stored":767114523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738738,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1876872,"kb_used_data":1169592,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":91868280,"statfs":{"total":95995035648,"available":94073118720,"internally_reserved":0,"allocated":1197662208,"data_stored":1197206920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1190596608,"data_stored":1190592536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":763367424,"data_stored":763363352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":904069120,"data_stored":904069120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1365508096,"data_stored":1365508096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":765132800,"data_stored":765132800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":359006208,"data_stored":359006208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":544542720,"data_stored":544538648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":371064832,"data_stored":371064832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":582418432,"data_stored":582418432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":980221952,"data_stored":980221952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":552075264,"data_stored":552075264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":904986624,"data_stored":904986624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:26:31.659 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-02T10:26:31.659 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:26:31.660 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_5 2024-09-02T10:26:31.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:26:31.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:31.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:31.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:31.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:31.900 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:26:31.900 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:26:31.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:31.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:31.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:32.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:32.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:32.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:32.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:32.170 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:32.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:32.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:32.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:32.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:32.302 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:32.404 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:32.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:32.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:32.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:32.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:32.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:32.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:32.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:26:32.624 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:32.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:32.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:32.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:32.807 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:32 smithi151 ceph-mon[16639]: pgmap v1681: 47 pgs: 47 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s 2024-09-02T10:26:32.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:32.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:32.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:32.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:33.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:32 smithi187 ceph-mon[20072]: pgmap v1681: 47 pgs: 47 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s 2024-09-02T10:26:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:33.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:32 smithi162 ceph-mon[20279]: pgmap v1681: 47 pgs: 47 active+clean; 2.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 734 op/s 2024-09-02T10:26:33.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:33.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:33.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:33.221 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:33.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:33.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:33.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:33.383 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:33.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:33.410 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:33.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:33.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:33.523 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:33.533 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:33.534 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:33.550 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:26:33.552 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:26:33.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:33.658 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:33.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:33.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:33.761 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:33.801 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:26:33.804 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:33.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:33.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:33.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:33 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:33.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:33 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:33.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:33 smithi151 ceph-mon[16639]: osdmap e736: 12 total, 12 up, 12 in 2024-09-02T10:26:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:33.958 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:33.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:33 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:33 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:33 smithi187 ceph-mon[20072]: osdmap e736: 12 total, 12 up, 12 in 2024-09-02T10:26:34.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:34.082 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:34.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:33 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:33 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:34.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:33 smithi162 ceph-mon[20279]: osdmap e736: 12 total, 12 up, 12 in 2024-09-02T10:26:34.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:34.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:34.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:34.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:34.321 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:34.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:34.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:34.343 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:34.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:34.474 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:34.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:34.476 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:34.577 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:34.602 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:34.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:34.603 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:34.646 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1683,"stamp":"2024-09-02T10:26:33.465338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2642870840,"num_objects":40321,"num_object_clones":0,"num_object_copies":120963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40321,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127561,"num_write_kb":5372097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30937,"num_bytes_recovered":2028090976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130413,"ondisk_log_size":130413,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":142,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":14597076,"kb_used_data":8076572,"kb_used_omap":18,"kb_used_meta":6519981,"kb_avail":1110344748,"statfs":{"total":1151940427776,"available":1136993021952,"internally_reserved":0,"allocated":8270409728,"data_stored":8266970646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":6676460859},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,4,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-408420352,"num_objects":-6232,"num_object_clones":0,"num_object_copies":-18696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6232,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002420"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":849,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.671977+0000","last_change":"2024-09-02T10:26:02.405331+0000","last_active":"2024-09-02T10:26:29.671977+0000","last_peered":"2024-09-02T10:26:29.671977+0000","last_clean":"2024-09-02T10:26:29.671977+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:26:29.671977+0000","last_undegraded":"2024-09-02T10:26:29.671977+0000","last_fullsized":"2024-09-02T10:26:29.671977+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.405283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.405283+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:17.160849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":654,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.655718+0000","last_change":"2024-09-02T10:26:14.793294+0000","last_active":"2024-09-02T10:26:27.655718+0000","last_peered":"2024-09-02T10:26:27.655718+0000","last_clean":"2024-09-02T10:26:27.655718+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:26:27.655718+0000","last_undegraded":"2024-09-02T10:26:27.655718+0000","last_fullsized":"2024-09-02T10:26:27.655718+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:14.793044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.793044+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:31.973724+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1120,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.651891+0000","last_change":"2024-09-02T10:26:09.987463+0000","last_active":"2024-09-02T10:26:27.651891+0000","last_peered":"2024-09-02T10:26:27.651891+0000","last_clean":"2024-09-02T10:26:27.651891+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:26:27.651891+0000","last_undegraded":"2024-09-02T10:26:27.651891+0000","last_fullsized":"2024-09-02T10:26:27.651891+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.987389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.987389+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:39.899046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1104,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.651829+0000","last_change":"2024-09-02T10:25:22.185960+0000","last_active":"2024-09-02T10:26:27.651829+0000","last_peered":"2024-09-02T10:26:27.651829+0000","last_clean":"2024-09-02T10:26:27.651829+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:26:27.651829+0000","last_undegraded":"2024-09-02T10:26:27.651829+0000","last_fullsized":"2024-09-02T10:26:27.651829+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:25:22.185894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:25:22.185894+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":407,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.658050+0000","last_change":"2024-09-02T10:26:06.766310+0000","last_active":"2024-09-02T10:26:27.658050+0000","last_peered":"2024-09-02T10:26:27.658050+0000","last_clean":"2024-09-02T10:26:27.658050+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:26:27.658050+0000","last_undegraded":"2024-09-02T10:26:27.658050+0000","last_fullsized":"2024-09-02T10:26:27.658050+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:06.766238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:06.766238+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:09.577073+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":968,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.681041+0000","last_change":"2024-09-02T10:26:12.294448+0000","last_active":"2024-09-02T10:26:29.681041+0000","last_peered":"2024-09-02T10:26:29.681041+0000","last_clean":"2024-09-02T10:26:29.681041+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:26:29.681041+0000","last_undegraded":"2024-09-02T10:26:29.681041+0000","last_fullsized":"2024-09-02T10:26:29.681041+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:12.294402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:12.294402+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:13.584271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1488,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.679838+0000","last_change":"2024-09-02T10:26:04.538470+0000","last_active":"2024-09-02T10:26:29.679838+0000","last_peered":"2024-09-02T10:26:29.679838+0000","last_clean":"2024-09-02T10:26:29.679838+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:26:29.679838+0000","last_undegraded":"2024-09-02T10:26:29.679838+0000","last_fullsized":"2024-09-02T10:26:29.679838+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:04.538409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:04.538409+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:23.977037+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1128,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.652067+0000","last_change":"2024-09-02T10:26:08.951582+0000","last_active":"2024-09-02T10:26:27.652067+0000","last_peered":"2024-09-02T10:26:27.652067+0000","last_clean":"2024-09-02T10:26:27.652067+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:26:27.652067+0000","last_undegraded":"2024-09-02T10:26:27.652067+0000","last_fullsized":"2024-09-02T10:26:27.652067+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:08.951514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.951514+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:10.021565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":1133,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.655524+0000","last_change":"2024-09-02T10:26:23.753612+0000","last_active":"2024-09-02T10:26:27.655524+0000","last_peered":"2024-09-02T10:26:27.655524+0000","last_clean":"2024-09-02T10:26:27.655524+0000","last_became_active":"2024-09-02T10:25:01.211125+0000","last_became_peered":"2024-09-02T10:25:01.211125+0000","last_unstale":"2024-09-02T10:26:27.655524+0000","last_undegraded":"2024-09-02T10:26:27.655524+0000","last_fullsized":"2024-09-02T10:26:27.655524+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:23.753547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:23.753547+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:43.401582+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1097,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.687262+0000","last_change":"2024-09-02T10:26:05.853283+0000","last_active":"2024-09-02T10:26:29.687262+0000","last_peered":"2024-09-02T10:26:29.687262+0000","last_clean":"2024-09-02T10:26:29.687262+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:26:29.687262+0000","last_undegraded":"2024-09-02T10:26:29.687262+0000","last_fullsized":"2024-09-02T10:26:29.687262+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:05.853228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:05.853228+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:33.199800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":341,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.661789+0000","last_change":"2024-09-02T10:25:20.120097+0000","last_active":"2024-09-02T10:26:29.661789+0000","last_peered":"2024-09-02T10:26:29.661789+0000","last_clean":"2024-09-02T10:26:29.661789+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:26:29.661789+0000","last_undegraded":"2024-09-02T10:26:29.661789+0000","last_fullsized":"2024-09-02T10:26:29.661789+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:25:20.120024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:25:20.120024+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:26:47.878922+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1445,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.655443+0000","last_change":"2024-09-02T10:26:16.774967+0000","last_active":"2024-09-02T10:26:27.655443+0000","last_peered":"2024-09-02T10:26:27.655443+0000","last_clean":"2024-09-02T10:26:27.655443+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:26:27.655443+0000","last_undegraded":"2024-09-02T10:26:27.655443+0000","last_fullsized":"2024-09-02T10:26:27.655443+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:16.774772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:16.774772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:37.484948+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1387,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.679880+0000","last_change":"2024-09-02T10:26:09.562148+0000","last_active":"2024-09-02T10:26:29.679880+0000","last_peered":"2024-09-02T10:26:29.679880+0000","last_clean":"2024-09-02T10:26:29.679880+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:26:29.679880+0000","last_undegraded":"2024-09-02T10:26:29.679880+0000","last_fullsized":"2024-09-02T10:26:29.679880+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.562104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.562104+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:16.950256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":112,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-09-02T10:26:32.688628+0000","last_change":"2024-09-02T10:26:14.840384+0000","last_active":"2024-09-02T10:26:32.688628+0000","last_peered":"2024-09-02T10:26:32.688628+0000","last_clean":"2024-09-02T10:26:32.688628+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:26:32.688628+0000","last_undegraded":"2024-09-02T10:26:32.688628+0000","last_fullsized":"2024-09-02T10:26:32.688628+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:14.840310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.840310+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:19.191366+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1357,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.679966+0000","last_change":"2024-09-02T10:26:06.555868+0000","last_active":"2024-09-02T10:26:29.679966+0000","last_peered":"2024-09-02T10:26:29.679966+0000","last_clean":"2024-09-02T10:26:29.679966+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:26:29.679966+0000","last_undegraded":"2024-09-02T10:26:29.679966+0000","last_fullsized":"2024-09-02T10:26:29.679966+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:06.555813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:06.555813+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:17.827776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"735'7810","reported_seq":8016,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:32.311881+0000","last_change":"2024-09-02T10:25:20.361425+0000","last_active":"2024-09-02T10:26:32.311881+0000","last_peered":"2024-09-02T10:26:32.311881+0000","last_clean":"2024-09-02T10:26:32.311881+0000","last_became_active":"2024-09-02T10:24:07.681518+0000","last_became_peered":"2024-09-02T10:24:07.681518+0000","last_unstale":"2024-09-02T10:26:32.311881+0000","last_undegraded":"2024-09-02T10:26:32.311881+0000","last_fullsized":"2024-09-02T10:26:32.311881+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"706'3999","last_scrub_stamp":"2024-09-02T10:25:20.360400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:25:20.360400+0000","objects_scrubbed":3994,"log_size":7810,"log_dups_size":0,"ondisk_log_size":7810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:26:38.772833+0000","scrub_duration":1241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161611776,"num_objects":2466,"num_object_clones":0,"num_object_copies":7398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7810,"num_write_kb":328832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,9],"acting":[2,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1451,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.679987+0000","last_change":"2024-09-02T10:26:02.497906+0000","last_active":"2024-09-02T10:26:29.679987+0000","last_peered":"2024-09-02T10:26:29.679987+0000","last_clean":"2024-09-02T10:26:29.679987+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:26:29.679987+0000","last_undegraded":"2024-09-02T10:26:29.679987+0000","last_fullsized":"2024-09-02T10:26:29.679987+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.497853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.497853+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:21.295805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.f","version":"736'8180","reported_seq":16470,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-09-02T10:26:33.022106+0000","last_change":"2024-09-02T10:25:21.220340+0000","last_active":"2024-09-02T10:26:33.022106+0000","last_peered":"2024-09-02T10:26:33.022106+0000","last_clean":"2024-09-02T10:26:33.022106+0000","last_became_active":"2024-09-02T10:25:21.217238+0000","last_became_peered":"2024-09-02T10:25:21.217238+0000","last_unstale":"2024-09-02T10:26:33.022106+0000","last_undegraded":"2024-09-02T10:26:33.022106+0000","last_fullsized":"2024-09-02T10:26:33.022106+0000","mapping_epoch":706,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":707,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:24:06.651660+0000","objects_scrubbed":0,"log_size":8180,"log_dups_size":0,"ondisk_log_size":8180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:26:34.893329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163446784,"num_objects":2494,"num_object_clones":0,"num_object_copies":7482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8180,"num_write_kb":341568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4053,"num_bytes_recovered":265617408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1576,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.687220+0000","last_change":"2024-09-02T10:26:09.933550+0000","last_active":"2024-09-02T10:26:29.687220+0000","last_peered":"2024-09-02T10:26:29.687220+0000","last_clean":"2024-09-02T10:26:29.687220+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:26:29.687220+0000","last_undegraded":"2024-09-02T10:26:29.687220+0000","last_fullsized":"2024-09-02T10:26:29.687220+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.933487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.933487+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:19.759531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"735'7962","reported_seq":16905,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:32.417137+0000","last_change":"2024-09-02T10:26:13.178680+0000","last_active":"2024-09-02T10:26:32.417137+0000","last_peered":"2024-09-02T10:26:32.417137+0000","last_clean":"2024-09-02T10:26:32.417137+0000","last_became_active":"2024-09-02T10:25:44.212783+0000","last_became_peered":"2024-09-02T10:25:44.212783+0000","last_unstale":"2024-09-02T10:26:32.417137+0000","last_undegraded":"2024-09-02T10:26:32.417137+0000","last_fullsized":"2024-09-02T10:26:32.417137+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"729'6901","last_scrub_stamp":"2024-09-02T10:26:13.178572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.178572+0000","objects_scrubbed":3559,"log_size":7962,"log_dups_size":0,"ondisk_log_size":7962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:39.044105+0000","scrub_duration":615,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162791424,"num_objects":2484,"num_object_clones":0,"num_object_copies":7452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7962,"num_write_kb":334272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4359,"num_bytes_recovered":285671424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,11],"acting":[6,3,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1221,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-09-02T10:26:32.685003+0000","last_change":"2024-09-02T10:26:02.369141+0000","last_active":"2024-09-02T10:26:32.685003+0000","last_peered":"2024-09-02T10:26:32.685003+0000","last_clean":"2024-09-02T10:26:32.685003+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:26:32.685003+0000","last_undegraded":"2024-09-02T10:26:32.685003+0000","last_fullsized":"2024-09-02T10:26:32.685003+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.369090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.369090+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:20.449364+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"7.d","version":"735'8059","reported_seq":8266,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:32.408509+0000","last_change":"2024-09-02T10:26:13.549346+0000","last_active":"2024-09-02T10:26:32.408509+0000","last_peered":"2024-09-02T10:26:32.408509+0000","last_clean":"2024-09-02T10:26:32.408509+0000","last_became_active":"2024-09-02T10:24:07.671900+0000","last_became_peered":"2024-09-02T10:24:07.671900+0000","last_unstale":"2024-09-02T10:26:32.408509+0000","last_undegraded":"2024-09-02T10:26:32.408509+0000","last_fullsized":"2024-09-02T10:26:32.408509+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"729'6967","last_scrub_stamp":"2024-09-02T10:26:13.549295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.549295+0000","objects_scrubbed":3613,"log_size":8059,"log_dups_size":0,"ondisk_log_size":8059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:30.818673+0000","scrub_duration":1966,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161873944,"num_objects":2471,"num_object_clones":0,"num_object_copies":7413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2471,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8059,"num_write_kb":336897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":615,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.672061+0000","last_change":"2024-09-02T10:26:07.441225+0000","last_active":"2024-09-02T10:26:29.672061+0000","last_peered":"2024-09-02T10:26:29.672061+0000","last_clean":"2024-09-02T10:26:29.672061+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:26:29.672061+0000","last_undegraded":"2024-09-02T10:26:29.672061+0000","last_fullsized":"2024-09-02T10:26:29.672061+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:07.441159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:07.441159+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:21.111255+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.2","version":"734'7673","reported_seq":7877,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:28.584442+0000","last_change":"2024-09-02T10:26:10.244647+0000","last_active":"2024-09-02T10:26:28.584442+0000","last_peered":"2024-09-02T10:26:28.584442+0000","last_clean":"2024-09-02T10:26:28.584442+0000","last_became_active":"2024-09-02T10:24:07.681580+0000","last_became_peered":"2024-09-02T10:24:07.681580+0000","last_unstale":"2024-09-02T10:26:28.584442+0000","last_undegraded":"2024-09-02T10:26:28.584442+0000","last_fullsized":"2024-09-02T10:26:28.584442+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'6721","last_scrub_stamp":"2024-09-02T10:26:10.244419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:10.244419+0000","objects_scrubbed":3718,"log_size":7673,"log_dups_size":0,"ondisk_log_size":7673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:18.436180+0000","scrub_duration":1044,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179765248,"num_objects":2743,"num_object_clones":0,"num_object_copies":8229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7673,"num_write_kb":333312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,2],"acting":[3,10,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1524,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.671956+0000","last_change":"2024-09-02T10:26:13.596384+0000","last_active":"2024-09-02T10:26:29.671956+0000","last_peered":"2024-09-02T10:26:29.671956+0000","last_clean":"2024-09-02T10:26:29.671956+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:26:29.671956+0000","last_undegraded":"2024-09-02T10:26:29.671956+0000","last_fullsized":"2024-09-02T10:26:29.671956+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:13.596339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.596339+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:42.860522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"7.3","version":"734'7641","reported_seq":7846,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:28.555225+0000","last_change":"2024-09-02T10:26:08.197217+0000","last_active":"2024-09-02T10:26:28.555225+0000","last_peered":"2024-09-02T10:26:28.555225+0000","last_clean":"2024-09-02T10:26:28.555225+0000","last_became_active":"2024-09-02T10:24:07.674582+0000","last_became_peered":"2024-09-02T10:24:07.674582+0000","last_unstale":"2024-09-02T10:26:28.555225+0000","last_undegraded":"2024-09-02T10:26:28.555225+0000","last_fullsized":"2024-09-02T10:26:28.555225+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6563","last_scrub_stamp":"2024-09-02T10:26:08.197134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.197134+0000","objects_scrubbed":3761,"log_size":7641,"log_dups_size":0,"ondisk_log_size":7641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:38.138983+0000","scrub_duration":1013,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174129152,"num_objects":2657,"num_object_clones":0,"num_object_copies":7971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7641,"num_write_kb":329536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":990,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.680823+0000","last_change":"2024-09-02T10:26:15.352217+0000","last_active":"2024-09-02T10:26:29.680823+0000","last_peered":"2024-09-02T10:26:29.680823+0000","last_clean":"2024-09-02T10:26:29.680823+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:26:29.680823+0000","last_undegraded":"2024-09-02T10:26:29.680823+0000","last_fullsized":"2024-09-02T10:26:29.680823+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:15.352153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:15.352153+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:33.222325+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.0","version":"735'8063","reported_seq":8306,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:32.437167+0000","last_change":"2024-09-02T10:26:16.241362+0000","last_active":"2024-09-02T10:26:32.437167+0000","last_peered":"2024-09-02T10:26:32.437167+0000","last_clean":"2024-09-02T10:26:32.437167+0000","last_became_active":"2024-09-02T10:24:07.673747+0000","last_became_peered":"2024-09-02T10:24:07.673747+0000","last_unstale":"2024-09-02T10:26:32.437167+0000","last_undegraded":"2024-09-02T10:26:32.437167+0000","last_fullsized":"2024-09-02T10:26:32.437167+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'7131","last_scrub_stamp":"2024-09-02T10:26:16.241303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:16.241303+0000","objects_scrubbed":3422,"log_size":8063,"log_dups_size":0,"ondisk_log_size":8063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:29.657946+0000","scrub_duration":564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162201600,"num_objects":2475,"num_object_clones":0,"num_object_copies":7425,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8063,"num_write_kb":337216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":955,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.683150+0000","last_change":"2024-09-02T10:26:09.304259+0000","last_active":"2024-09-02T10:26:29.683150+0000","last_peered":"2024-09-02T10:26:29.683150+0000","last_clean":"2024-09-02T10:26:29.683150+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:26:29.683150+0000","last_undegraded":"2024-09-02T10:26:29.683150+0000","last_fullsized":"2024-09-02T10:26:29.683150+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.303950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.303950+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:10.581116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.1","version":"734'7779","reported_seq":16818,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.569426+0000","last_change":"2024-09-02T10:26:10.642407+0000","last_active":"2024-09-02T10:26:29.569426+0000","last_peered":"2024-09-02T10:26:29.569426+0000","last_clean":"2024-09-02T10:26:29.569426+0000","last_became_active":"2024-09-02T10:25:44.211739+0000","last_became_peered":"2024-09-02T10:25:44.211739+0000","last_unstale":"2024-09-02T10:26:29.569426+0000","last_undegraded":"2024-09-02T10:26:29.569426+0000","last_fullsized":"2024-09-02T10:26:29.569426+0000","mapping_epoch":717,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":718,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'6786","last_scrub_stamp":"2024-09-02T10:26:10.642208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:10.642208+0000","objects_scrubbed":3711,"log_size":7779,"log_dups_size":0,"ondisk_log_size":7779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:27.257300+0000","scrub_duration":1800,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175046656,"num_objects":2671,"num_object_clones":0,"num_object_copies":8013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7779,"num_write_kb":334400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4407,"num_bytes_recovered":288817152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,3],"acting":[4,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":737,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.659831+0000","last_change":"2024-09-02T10:26:14.593855+0000","last_active":"2024-09-02T10:26:29.659831+0000","last_peered":"2024-09-02T10:26:29.659831+0000","last_clean":"2024-09-02T10:26:29.659831+0000","last_became_active":"2024-09-02T10:25:02.065175+0000","last_became_peered":"2024-09-02T10:25:02.065175+0000","last_unstale":"2024-09-02T10:26:29.659831+0000","last_undegraded":"2024-09-02T10:26:29.659831+0000","last_fullsized":"2024-09-02T10:26:29.659831+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:14.593805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.593805+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:42.832993+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"735'7940","reported_seq":8144,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:32.398413+0000","last_change":"2024-09-02T10:26:11.397647+0000","last_active":"2024-09-02T10:26:32.398413+0000","last_peered":"2024-09-02T10:26:32.398413+0000","last_clean":"2024-09-02T10:26:32.398413+0000","last_became_active":"2024-09-02T10:24:07.670909+0000","last_became_peered":"2024-09-02T10:24:07.670909+0000","last_unstale":"2024-09-02T10:26:32.398413+0000","last_undegraded":"2024-09-02T10:26:32.398413+0000","last_fullsized":"2024-09-02T10:26:32.398413+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"727'6764","last_scrub_stamp":"2024-09-02T10:26:11.397518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:11.397518+0000","objects_scrubbed":3666,"log_size":7940,"log_dups_size":0,"ondisk_log_size":7940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:13.343075+0000","scrub_duration":854,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161873920,"num_objects":2470,"num_object_clones":0,"num_object_copies":7410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7940,"num_write_kb":333120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,3],"acting":[6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":970,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.680785+0000","last_change":"2024-09-02T10:26:08.256406+0000","last_active":"2024-09-02T10:26:29.680785+0000","last_peered":"2024-09-02T10:26:29.680785+0000","last_clean":"2024-09-02T10:26:29.680785+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:26:29.680785+0000","last_undegraded":"2024-09-02T10:26:29.680785+0000","last_fullsized":"2024-09-02T10:26:29.680785+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:08.256332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.256332+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:32.379501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"7.4","version":"736'8251","reported_seq":19424,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-09-02T10:26:33.070732+0000","last_change":"2024-09-02T10:26:14.088498+0000","last_active":"2024-09-02T10:26:33.070732+0000","last_peered":"2024-09-02T10:26:33.070732+0000","last_clean":"2024-09-02T10:26:33.070732+0000","last_became_active":"2024-09-02T10:26:14.049049+0000","last_became_peered":"2024-09-02T10:26:14.049049+0000","last_unstale":"2024-09-02T10:26:33.070732+0000","last_undegraded":"2024-09-02T10:26:33.070732+0000","last_fullsized":"2024-09-02T10:26:33.070732+0000","mapping_epoch":729,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":730,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:24:06.651660+0000","objects_scrubbed":0,"log_size":8251,"log_dups_size":0,"ondisk_log_size":8251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161415168,"num_objects":2463,"num_object_clones":0,"num_object_copies":7389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8251,"num_write_kb":342848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5491,"num_bytes_recovered":359792640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1218,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.687273+0000","last_change":"2024-09-02T10:26:11.944736+0000","last_active":"2024-09-02T10:26:29.687273+0000","last_peered":"2024-09-02T10:26:29.687273+0000","last_clean":"2024-09-02T10:26:29.687273+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:26:29.687273+0000","last_undegraded":"2024-09-02T10:26:29.687273+0000","last_fullsized":"2024-09-02T10:26:29.687273+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:11.944689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:11.944689+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:23.172770+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"735'7978","reported_seq":16614,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:32.265514+0000","last_change":"2024-09-02T10:26:06.663497+0000","last_active":"2024-09-02T10:26:32.265514+0000","last_peered":"2024-09-02T10:26:32.265514+0000","last_clean":"2024-09-02T10:26:32.265514+0000","last_became_active":"2024-09-02T10:25:28.620158+0000","last_became_peered":"2024-09-02T10:25:28.620158+0000","last_unstale":"2024-09-02T10:26:32.265514+0000","last_undegraded":"2024-09-02T10:26:32.265514+0000","last_fullsized":"2024-09-02T10:26:32.265514+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6440","last_scrub_stamp":"2024-09-02T10:26:06.663370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:06.663370+0000","objects_scrubbed":3970,"log_size":7978,"log_dups_size":0,"ondisk_log_size":7978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:12.071995+0000","scrub_duration":1255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157024256,"num_objects":2396,"num_object_clones":0,"num_object_copies":7188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7978,"num_write_kb":331968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4199,"num_bytes_recovered":275185664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,0],"acting":[11,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1476,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.655463+0000","last_change":"2024-09-02T10:26:22.765464+0000","last_active":"2024-09-02T10:26:27.655463+0000","last_peered":"2024-09-02T10:26:27.655463+0000","last_clean":"2024-09-02T10:26:27.655463+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:26:27.655463+0000","last_undegraded":"2024-09-02T10:26:27.655463+0000","last_fullsized":"2024-09-02T10:26:27.655463+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:22.765406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:22.765406+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:38.157020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"736'8204","reported_seq":17368,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-09-02T10:26:33.079480+0000","last_change":"2024-09-02T10:26:14.504599+0000","last_active":"2024-09-02T10:26:33.079480+0000","last_peered":"2024-09-02T10:26:33.079480+0000","last_clean":"2024-09-02T10:26:33.079480+0000","last_became_active":"2024-09-02T10:26:02.534985+0000","last_became_peered":"2024-09-02T10:26:02.534985+0000","last_unstale":"2024-09-02T10:26:33.079480+0000","last_undegraded":"2024-09-02T10:26:33.079480+0000","last_fullsized":"2024-09-02T10:26:33.079480+0000","mapping_epoch":723,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":724,"parent":"0.0","parent_split_bits":0,"last_scrub":"730'7133","last_scrub_stamp":"2024-09-02T10:26:14.504505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:14.504505+0000","objects_scrubbed":3617,"log_size":8204,"log_dups_size":0,"ondisk_log_size":8204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:24.932727+0000","scrub_duration":694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164364288,"num_objects":2508,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8204,"num_write_kb":342784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4468,"num_bytes_recovered":292814848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1221,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.687232+0000","last_change":"2024-09-02T10:26:01.795185+0000","last_active":"2024-09-02T10:26:29.687232+0000","last_peered":"2024-09-02T10:26:29.687232+0000","last_clean":"2024-09-02T10:26:29.687232+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:26:29.687232+0000","last_undegraded":"2024-09-02T10:26:29.687232+0000","last_fullsized":"2024-09-02T10:26:29.687232+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:01.795140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:01.795140+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:19.964214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"736'8308","reported_seq":16449,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-09-02T10:26:33.094157+0000","last_change":"2024-09-02T10:26:20.104150+0000","last_active":"2024-09-02T10:26:33.094157+0000","last_peered":"2024-09-02T10:26:33.094157+0000","last_clean":"2024-09-02T10:26:33.094157+0000","last_became_active":"2024-09-02T10:25:20.217872+0000","last_became_peered":"2024-09-02T10:25:20.217872+0000","last_unstale":"2024-09-02T10:26:33.094157+0000","last_undegraded":"2024-09-02T10:26:33.094157+0000","last_fullsized":"2024-09-02T10:26:33.094157+0000","mapping_epoch":705,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":706,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'7570","last_scrub_stamp":"2024-09-02T10:26:20.103959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:20.103959+0000","objects_scrubbed":3294,"log_size":8308,"log_dups_size":0,"ondisk_log_size":8308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:27.513393+0000","scrub_duration":1260,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165150720,"num_objects":2520,"num_object_clones":0,"num_object_copies":7560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2520,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8308,"num_write_kb":346496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3956,"num_bytes_recovered":259260416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,8],"acting":[0,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1484,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.659875+0000","last_change":"2024-09-02T10:26:02.582294+0000","last_active":"2024-09-02T10:26:29.659875+0000","last_peered":"2024-09-02T10:26:29.659875+0000","last_clean":"2024-09-02T10:26:29.659875+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:26:29.659875+0000","last_undegraded":"2024-09-02T10:26:29.659875+0000","last_fullsized":"2024-09-02T10:26:29.659875+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:02.582251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:02.582251+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:03.315097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"735'7984","reported_seq":8189,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:32.305796+0000","last_change":"2024-09-02T10:26:05.976133+0000","last_active":"2024-09-02T10:26:32.305796+0000","last_peered":"2024-09-02T10:26:32.305796+0000","last_clean":"2024-09-02T10:26:32.305796+0000","last_became_active":"2024-09-02T10:24:07.681130+0000","last_became_peered":"2024-09-02T10:24:07.681130+0000","last_unstale":"2024-09-02T10:26:32.305796+0000","last_undegraded":"2024-09-02T10:26:32.305796+0000","last_fullsized":"2024-09-02T10:26:32.305796+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6487","last_scrub_stamp":"2024-09-02T10:26:05.975865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:05.975865+0000","objects_scrubbed":3945,"log_size":7984,"log_dups_size":0,"ondisk_log_size":7984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:18.517262+0000","scrub_duration":1219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159121408,"num_objects":2428,"num_object_clones":0,"num_object_copies":7284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7984,"num_write_kb":333184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":423,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.658111+0000","last_change":"2024-09-02T10:26:09.806628+0000","last_active":"2024-09-02T10:26:27.658111+0000","last_peered":"2024-09-02T10:26:27.658111+0000","last_clean":"2024-09-02T10:26:27.658111+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:26:27.658111+0000","last_undegraded":"2024-09-02T10:26:27.658111+0000","last_fullsized":"2024-09-02T10:26:27.658111+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:09.806582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.806582+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:38.336600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.b","version":"734'7730","reported_seq":7926,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:28.823450+0000","last_change":"2024-09-02T10:26:22.680654+0000","last_active":"2024-09-02T10:26:28.823450+0000","last_peered":"2024-09-02T10:26:28.823450+0000","last_clean":"2024-09-02T10:26:28.823450+0000","last_became_active":"2024-09-02T10:24:07.673332+0000","last_became_peered":"2024-09-02T10:24:07.673332+0000","last_unstale":"2024-09-02T10:26:28.823450+0000","last_undegraded":"2024-09-02T10:26:28.823450+0000","last_fullsized":"2024-09-02T10:26:28.823450+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"731'7425","last_scrub_stamp":"2024-09-02T10:26:22.680454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:22.680454+0000","objects_scrubbed":2997,"log_size":7730,"log_dups_size":0,"ondisk_log_size":7730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:50.130765+0000","scrub_duration":889,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174194688,"num_objects":2658,"num_object_clones":0,"num_object_copies":7974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7730,"num_write_kb":332416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1256,"reported_epoch":735,"state":"active+clean","last_fresh":"2024-09-02T10:26:29.687374+0000","last_change":"2024-09-02T10:26:08.915296+0000","last_active":"2024-09-02T10:26:29.687374+0000","last_peered":"2024-09-02T10:26:29.687374+0000","last_clean":"2024-09-02T10:26:29.687374+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:26:29.687374+0000","last_undegraded":"2024-09-02T10:26:29.687374+0000","last_fullsized":"2024-09-02T10:26:29.687374+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:08.915248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:08.915248+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:30.109523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"736'7999","reported_seq":8207,"reported_epoch":736,"state":"active+clean","last_fresh":"2024-09-02T10:26:33.132639+0000","last_change":"2024-09-02T10:26:09.242371+0000","last_active":"2024-09-02T10:26:33.132639+0000","last_peered":"2024-09-02T10:26:33.132639+0000","last_clean":"2024-09-02T10:26:33.132639+0000","last_became_active":"2024-09-02T10:24:07.673813+0000","last_became_peered":"2024-09-02T10:24:07.673813+0000","last_unstale":"2024-09-02T10:26:33.132639+0000","last_undegraded":"2024-09-02T10:26:33.132639+0000","last_fullsized":"2024-09-02T10:26:33.132639+0000","mapping_epoch":661,"log_start":"0'0","ondisk_log_start":"0'0","created":661,"last_epoch_clean":662,"parent":"0.0","parent_split_bits":0,"last_scrub":"726'6670","last_scrub_stamp":"2024-09-02T10:26:09.242311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:24:06.651660+0000","last_clean_scrub_stamp":"2024-09-02T10:26:09.242311+0000","objects_scrubbed":3772,"log_size":7999,"log_dups_size":0,"ondisk_log_size":7999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:18.078348+0000","scrub_duration":1017,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158269440,"num_objects":2415,"num_object_clones":0,"num_object_copies":7245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7999,"num_write_kb":333248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,11],"acting":[9,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":638,"reported_epoch":734,"state":"active+clean","last_fresh":"2024-09-02T10:26:27.656519+0000","last_change":"2024-09-02T10:26:15.301143+0000","last_active":"2024-09-02T10:26:27.656519+0000","last_peered":"2024-09-02T10:26:27.656519+0000","last_clean":"2024-09-02T10:26:27.656519+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:26:27.656519+0000","last_undegraded":"2024-09-02T10:26:27.656519+0000","last_fullsized":"2024-09-02T10:26:27.656519+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:15.300992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:15.300992+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:27:45.194647+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2642280472,"num_objects":40319,"num_object_clones":0,"num_object_copies":120957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":127561,"num_write_kb":5372097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30933,"num_bytes_recovered":2027159552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8183689216,"data_stored":8183677000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127561,"ondisk_log_size":127561,"up":48,"acting":48,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383997,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1395772,"kb_used_data":805100,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92349380,"statfs":{"total":95995035648,"available":94565765120,"internally_reserved":0,"allocated":824422400,"data_stored":824136982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":604830148},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,3,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612930,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":912832,"kb_used_data":490156,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":92832320,"statfs":{"total":95995035648,"available":95060295680,"internally_reserved":0,"allocated":501919744,"data_stored":501634326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":432798144},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841863,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1427596,"kb_used_data":778616,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92317556,"statfs":{"total":95995035648,"available":94533177344,"internally_reserved":0,"allocated":797302784,"data_stored":797012963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":664533440},"hb_peers":[0,1,2,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070797,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893124,"kb_used_data":523040,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92852028,"statfs":{"total":95995035648,"available":95080476672,"internally_reserved":0,"allocated":535592960,"data_stored":535311945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299730,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":928200,"kb_used_data":329204,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92816952,"statfs":{"total":95995035648,"available":95044558848,"internally_reserved":0,"allocated":337104896,"data_stored":336815926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":613349834},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528661,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":900032,"kb_used_data":485680,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92845120,"statfs":{"total":95995035648,"available":95073402880,"internally_reserved":0,"allocated":497336320,"data_stored":497046830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":424278471},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724891,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":969220,"kb_used_data":320244,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92775932,"statfs":{"total":95995035648,"available":95002554368,"internally_reserved":0,"allocated":327929856,"data_stored":327640886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953823,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1105008,"kb_used_data":686496,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":92640144,"statfs":{"total":95995035648,"available":94863507456,"internally_reserved":0,"allocated":702971904,"data_stored":702690889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150052,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1923184,"kb_used_data":1220064,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91821968,"statfs":{"total":95995035648,"available":94025695232,"internally_reserved":0,"allocated":1249345536,"data_stored":1249064521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":719976904},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411688,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1203524,"kb_used_data":797492,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":92541628,"statfs":{"total":95995035648,"available":94762627072,"internally_reserved":0,"allocated":816631808,"data_stored":816342838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575213,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1282440,"kb_used_data":687460,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":92462712,"statfs":{"total":95995035648,"available":94681817088,"internally_reserved":0,"allocated":703959040,"data_stored":703673953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738740,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1656144,"kb_used_data":953020,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92089008,"statfs":{"total":95995035648,"available":94299144192,"internally_reserved":0,"allocated":975892480,"data_stored":975598587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":969281536,"data_stored":969277464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":700321792,"data_stored":700317720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":809631744,"data_stored":809631744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1243545600,"data_stored":1243545600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":690225152,"data_stored":690225152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":321650688,"data_stored":321650688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":486543360,"data_stored":486539288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":331087872,"data_stored":331087872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":528482304,"data_stored":528482304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":795738112,"data_stored":795738112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":498204672,"data_stored":498204672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":808976384,"data_stored":808976384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:26:34.649 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-02T10:26:34.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set unique_pool_5 pg_num 26 2024-09-02T10:26:34.697 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:26:34.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:34.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:34.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:34.854 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:34.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:34.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:34.879 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:34.904 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:34 smithi151 ceph-mon[16639]: pgmap v1683: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-09-02T10:26:34.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:34 smithi151 ceph-mon[16639]: from='client.18147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:26:34.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1910705935' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:34.905 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:34 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:34.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:35.004 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:35.005 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:34 smithi187 ceph-mon[20072]: pgmap v1683: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-09-02T10:26:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:34 smithi187 ceph-mon[20072]: from='client.18147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:26:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1910705935' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:34 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:34 smithi162 ceph-mon[20279]: pgmap v1683: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.04k op/s 2024-09-02T10:26:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:34 smithi162 ceph-mon[20279]: from='client.18147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:26:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1910705935' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:34 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:35.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:35.128 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:35.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:35.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:35.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:35.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:35.256 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:35.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:35.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:35.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:35.382 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:35.445 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:26:35.445 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:26:35.483 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:35.507 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:35.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:35.609 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:35.632 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:35.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:35.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:35.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:35.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:35.870 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:35.892 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:35.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:35.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:35 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:35 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:26:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:35 smithi151 ceph-mon[16639]: osdmap e737: 12 total, 12 up, 12 in 2024-09-02T10:26:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:35 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:35.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:35 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:35.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:35 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:35 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:26:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:35 smithi187 ceph-mon[20072]: osdmap e737: 12 total, 12 up, 12 in 2024-09-02T10:26:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:35 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:36.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:35 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:36.015 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:36.016 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:36.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:35 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:36.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:35 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:26:36.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:35 smithi162 ceph-mon[20279]: osdmap e737: 12 total, 12 up, 12 in 2024-09-02T10:26:36.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:35 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:36.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:35 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:36.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:36.145 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:36.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:36.146 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:36.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:36.276 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:36.276 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:36.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:36.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:36.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:36.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:36.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:36.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:36.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:36.660 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:36.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:36.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:36.784 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:36.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:36.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:36.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:36.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:36 smithi151 ceph-mon[16639]: pgmap v1685: 47 pgs: 47 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-09-02T10:26:36.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:36 smithi151 ceph-mon[16639]: osdmap e738: 12 total, 12 up, 12 in 2024-09-02T10:26:36.908 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:36.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:36 smithi187 ceph-mon[20072]: pgmap v1685: 47 pgs: 47 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-09-02T10:26:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:36 smithi187 ceph-mon[20072]: osdmap e738: 12 total, 12 up, 12 in 2024-09-02T10:26:37.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:37.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:37.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:37.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:36 smithi162 ceph-mon[20279]: pgmap v1685: 47 pgs: 47 active+clean; 2.4 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 805 op/s 2024-09-02T10:26:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:36 smithi162 ceph-mon[20279]: osdmap e738: 12 total, 12 up, 12 in 2024-09-02T10:26:37.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:37.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:37.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:37.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:37.286 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:37.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:37.367 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:37.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:26:37.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:37.399 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:37.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:37.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:37.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:37.572 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:37.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:37.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:37.700 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:37.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:37.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:37.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:37.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:37.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:37.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:37.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:37.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:37.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:38.022 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:26:38.022 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:26:38.058 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:38.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:38.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:38.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:38.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:38.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:38.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:38.243 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:38.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:26:38.265 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:26:38.298 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:26:38.298 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:26:38.328 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:26:38.329 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:26:38.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:38.363 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:26:38.363 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:26:38.368 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:38.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:38.369 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:38.397 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:26:38.416 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:26:38.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:38.493 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:38.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:38.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:38.558 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:26:38.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:38.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:38.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:38.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:38.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:38.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:38 smithi151 ceph-mon[16639]: pgmap v1687: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-09-02T10:26:38.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:38 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:38.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:38 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:38.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2945705306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:38.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:38.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:38.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:38.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:39.001 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:38 smithi187 ceph-mon[20072]: pgmap v1687: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-09-02T10:26:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:38 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:38 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2945705306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:39.023 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:39.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:39.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:38 smithi162 ceph-mon[20279]: pgmap v1687: 47 pgs: 47 active+clean; 2.3 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 1.00k op/s 2024-09-02T10:26:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:38 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:38 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2945705306' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:39.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:39.148 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:39.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:39.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.16,RELEASE,e:497,from:1,reservation_nonce:0)", 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 580.56013120600005, 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58412980699999995, 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.201 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636692+0000", 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636693+0000", 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6e-07 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:58.636695+0000", 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.841e-06 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.203 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220721+0000", 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.58402521600000001 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220727+0000", 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4219999999999997e-06 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220798+0000", 2024-09-02T10:26:39.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1131999999999996e-05 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220810+0000", 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1863e-05 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:16:59.220822+0000", 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2072999999999999e-05 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.205 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:205302 5.1 5:85fde85c:::benchmark_data_smithi187_89405_object101697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 501.95530297800002, 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31845405399999999, 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:26:39.206 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 205302 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241521+0000", 2024-09-02T10:26:39.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241523+0000", 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3429999999999998e-06 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241533+0000", 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0139999999999999e-05 2024-09-02T10:26:39.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241562+0000", 2024-09-02T10:26:39.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8600000000000001e-05 2024-09-02T10:26:39.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241581+0000", 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9656e-05 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241772+0000", 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019121200000000001 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241906+0000", 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013322299999999999 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.241975+0000", 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9129999999999997e-05 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.245690+0000", 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037148390000000002 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.246234+0000", 2024-09-02T10:26:39.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00054468699999999997 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559894+0000", 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31365966099999998 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559928+0000", 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3813000000000003e-05 2024-09-02T10:26:39.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.559975+0000", 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6749999999999998e-05 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.25933.0:205324 5.1 5:85edc945:::benchmark_data_smithi187_89405_object101719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:26:39.213 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 501.59456520999998, 2024-09-02T10:26:39.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38627865099999997, 2024-09-02T10:26:39.214 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.214 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.214 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.214 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.214 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.25933", 2024-09-02T10:26:39.214 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:26:39.214 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 205324 2024-09-02T10:26:39.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.214 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602258+0000", 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602260+0000", 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4300000000000001e-06 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602266+0000", 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6599999999999998e-06 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602283+0000", 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6646e-05 2024-09-02T10:26:39.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602307+0000", 2024-09-02T10:26:39.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3853999999999999e-05 2024-09-02T10:26:39.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602361+0000", 2024-09-02T10:26:39.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4200000000000003e-05 2024-09-02T10:26:39.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602447+0000", 2024-09-02T10:26:39.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6116e-05 2024-09-02T10:26:39.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 7,9", 2024-09-02T10:26:39.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.602524+0000", 2024-09-02T10:26:39.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6596999999999999e-05 2024-09-02T10:26:39.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.606604+0000", 2024-09-02T10:26:39.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040804589999999998 2024-09-02T10:26:39.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.606691+0000", 2024-09-02T10:26:39.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6608999999999997e-05 2024-09-02T10:26:39.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988466+0000", 2024-09-02T10:26:39.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38177503699999998 2024-09-02T10:26:39.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988502+0000", 2024-09-02T10:26:39.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6217000000000002e-05 2024-09-02T10:26:39.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:18:17.988537+0000", 2024-09-02T10:26:39.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4826000000000002e-05 2024-09-02T10:26:39.222 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.222 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.222 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:100267 6.b 6:dbd5fb01:::benchmark_data_smithi187_102448_object100266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:10.945411+0000", 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 388.25141219900001, 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54857257299999995, 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.223 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.224 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 100267 2024-09-02T10:26:39.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.224 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945411+0000", 2024-09-02T10:26:39.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945411+0000", 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945413+0000", 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3850000000000001e-06 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945444+0000", 2024-09-02T10:26:39.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0843999999999998e-05 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945455+0000", 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1107000000000001e-05 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945462+0000", 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9029999999999998e-06 2024-09-02T10:26:39.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945541+0000", 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9658000000000002e-05 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945679+0000", 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013753800000000001 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:26:39.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945835+0000", 2024-09-02T10:26:39.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015657799999999999 2024-09-02T10:26:39.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949089+0000", 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032539589999999998 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949384+0000", 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029429199999999998 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493901+0000", 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54451749699999996 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493945+0000", 2024-09-02T10:26:39.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3469e-05 2024-09-02T10:26:39.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493984+0000", 2024-09-02T10:26:39.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9342999999999999e-05 2024-09-02T10:26:39.230 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26524.0:100267 6.b e572/533 ondisk, result = 0)", 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:10.949464+0000", 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 388.24735960599997, 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54452646699999996, 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949464+0000", 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949464+0000", 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949465+0000", 2024-09-02T10:26:39.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5099999999999999e-07 2024-09-02T10:26:39.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949468+0000", 2024-09-02T10:26:39.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3440000000000001e-06 2024-09-02T10:26:39.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493766+0000", 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54429846599999998 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493783+0000", 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7013000000000001e-05 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493889+0000", 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105537 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493897+0000", 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1009999999999992e-06 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493990+0000", 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3355000000000003e-05 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157268 6.b 6:db1c5c67:::benchmark_data_smithi187_102448_object56996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 311.37197732700002, 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62417882300000005, 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157268 2024-09-02T10:26:39.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824847+0000", 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824850+0000", 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.76e-06 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824860+0000", 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2690000000000005e-06 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824876+0000", 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5882000000000001e-05 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824934+0000", 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8780000000000003e-05 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.825044+0000", 2024-09-02T10:26:39.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010938900000000001 2024-09-02T10:26:39.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:26:39.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.825142+0000", 2024-09-02T10:26:39.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8009000000000003e-05 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830102+0000", 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049605530000000004 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.838185+0000", 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080826430000000005 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.448954+0000", 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61076896000000003 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.448992+0000", 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7732999999999999e-05 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449025+0000", 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3315999999999997e-05 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.242 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157271 6.6 6:69d4aafc:::benchmark_data_smithi187_102448_object56999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 311.366422688, 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62368362300000002, 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157271 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:26:39.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830403+0000", 2024-09-02T10:26:39.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7179999999999999e-06 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830409+0000", 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.703e-06 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830424+0000", 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4961e-05 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830432+0000", 2024-09-02T10:26:39.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4880000000000001e-06 2024-09-02T10:26:39.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830495+0000", 2024-09-02T10:26:39.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3758999999999997e-05 2024-09-02T10:26:39.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830625+0000", 2024-09-02T10:26:39.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012918000000000001 2024-09-02T10:26:39.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:26:39.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830756+0000", 2024-09-02T10:26:39.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013138200000000001 2024-09-02T10:26:39.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.248 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.835882+0000", 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051260960000000001 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.836280+0000", 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039814100000000002 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454013+0000", 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61773251699999998 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454048+0000", 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5559999999999998e-05 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454084+0000", 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6118000000000002e-05 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.251 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157274 6.6 6:6387cbb8:::benchmark_data_smithi187_102448_object57002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:26:39.251 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:26:39.251 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 311.35939453399999, 2024-09-02T10:26:39.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61674791900000003, 2024-09-02T10:26:39.251 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.251 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.251 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.251 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157274 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.252 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:26:39.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837431+0000", 2024-09-02T10:26:39.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7010000000000001e-06 2024-09-02T10:26:39.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837437+0000", 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0379999999999996e-06 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837448+0000", 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1557e-05 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837456+0000", 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.396e-06 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837510+0000", 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4666000000000001e-05 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837636+0000", 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125313 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837757+0000", 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121212 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.845343+0000", 2024-09-02T10:26:39.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075863880000000003 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.847219+0000", 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00187546 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454127+0000", 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60690782600000004 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454156+0000", 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9281999999999998e-05 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454177+0000", 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1080000000000001e-05 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.258 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157277 6.6 6:67446ce0:::benchmark_data_smithi187_102448_object57005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 311.35886292399999, 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61628705500000003, 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157277 2024-09-02T10:26:39.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.260 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:26:39.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:26:39.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.261 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.261 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:26:39.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0699999999999996e-07 2024-09-02T10:26:39.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837964+0000", 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8569999999999999e-06 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837973+0000", 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0159999999999997e-06 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837978+0000", 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7770000000000002e-06 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842046+0000", 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040683289999999999 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842130+0000", 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3751000000000005e-05 2024-09-02T10:26:39.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842199+0000", 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8411999999999995e-05 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.845512+0000", 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003313795 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.847365+0000", 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018529009999999999 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454203+0000", 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60683791600000003 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454229+0000", 2024-09-02T10:26:39.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5609000000000002e-05 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454248+0000", 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8885e-05 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157282 6.b 6:d751a93a:::benchmark_data_smithi187_102448_object57010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 311.32289246699997, 2024-09-02T10:26:39.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57523664399999996, 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157282 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873933+0000", 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.08e-06 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873941+0000", 2024-09-02T10:26:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0919999999999998e-06 2024-09-02T10:26:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873961+0000", 2024-09-02T10:26:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0231e-05 2024-09-02T10:26:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.269 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:39.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873971+0000", 2024-09-02T10:26:39.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2969999999999992e-06 2024-09-02T10:26:39.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.272 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874034+0000", 2024-09-02T10:26:39.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3257999999999996e-05 2024-09-02T10:26:39.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874177+0000", 2024-09-02T10:26:39.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142712 2024-09-02T10:26:39.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.273 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:26:39.273 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874303+0000", 2024-09-02T10:26:39.273 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126248 2024-09-02T10:26:39.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.878010+0000", 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037068610000000001 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.946302+0000", 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.068292383999999998 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.274 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449070+0000", 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50276810199999999 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449124+0000", 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3712999999999999e-05 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449168+0000", 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3665999999999998e-05 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.275 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.276 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.276 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157307 6.6 6:675a1a62:::benchmark_data_smithi187_102448_object57035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:26:39.276 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:26:39.276 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 311.178803786, 2024-09-02T10:26:39.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43718100199999999, 2024-09-02T10:26:39.276 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.276 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.276 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.277 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.277 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.277 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.277 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157307 2024-09-02T10:26:39.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.277 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:26:39.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:26:39.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018021+0000", 2024-09-02T10:26:39.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6559999999999999e-06 2024-09-02T10:26:39.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018029+0000", 2024-09-02T10:26:39.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6310000000000003e-06 2024-09-02T10:26:39.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018046+0000", 2024-09-02T10:26:39.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6779999999999999e-05 2024-09-02T10:26:39.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018054+0000", 2024-09-02T10:26:39.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3180000000000007e-06 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018619+0000", 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056491400000000002 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018723+0000", 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103619 2024-09-02T10:26:39.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018800+0000", 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7859000000000001e-05 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.022686+0000", 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038855399999999998 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031392+0000", 2024-09-02T10:26:39.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087064819999999998 2024-09-02T10:26:39.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455118+0000", 2024-09-02T10:26:39.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42372546999999999 2024-09-02T10:26:39.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455167+0000", 2024-09-02T10:26:39.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8773000000000002e-05 2024-09-02T10:26:39.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455201+0000", 2024-09-02T10:26:39.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.396e-05 2024-09-02T10:26:39.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.285 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.285 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157310 6.b 6:d937f28e:::benchmark_data_smithi187_102448_object57038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 311.16569936000002, 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42301055599999998, 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.286 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157310 2024-09-02T10:26:39.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.287 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:26:39.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:26:39.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031125+0000", 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8599999999999997e-07 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031130+0000", 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7949999999999998e-06 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031141+0000", 2024-09-02T10:26:39.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0735e-05 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031148+0000", 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4189999999999996e-06 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031196+0000", 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7571000000000002e-05 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031278+0000", 2024-09-02T10:26:39.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2262000000000006e-05 2024-09-02T10:26:39.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:26:39.291 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:39.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031363+0000", 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5279999999999997e-05 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.035027+0000", 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036643140000000001 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.290856+0000", 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25582839899999998 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454040+0000", 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.163183774 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454092+0000", 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2224e-05 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.293 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454135+0000", 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2896999999999999e-05 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:193645 6.6 6:6289853a:::benchmark_data_smithi187_102448_object93373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:28.326730+0000", 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 250.870093766, 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29017263799999998, 2024-09-02T10:26:39.294 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 193645 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.326730+0000", 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.295 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.326730+0000", 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.326731+0000", 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2389999999999999e-06 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.326736+0000", 2024-09-02T10:26:39.296 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6110000000000004e-06 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.326749+0000", 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.33e-05 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.326756+0000", 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2439999999999996e-06 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.297 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.331131+0000", 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0043745440000000002 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.331239+0000", 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108498 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.331347+0000", 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108336 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.298 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.337161+0000", 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0058130109999999999 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.342647+0000", 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0054860020000000002 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.616802+0000", 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27415514299999999 2024-09-02T10:26:39.299 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.616854+0000", 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2772000000000002e-05 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.616902+0000", 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7938000000000001e-05 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.300 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:193653 6.c 6:3563a971:::benchmark_data_smithi187_102448_object93381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:28.339122+0000", 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 250.85770144, 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25693641299999997, 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 193653 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.301 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.339122+0000", 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.339122+0000", 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.339123+0000", 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-02T10:26:39.302 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.339127+0000", 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0400000000000003e-06 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.339135+0000", 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1710000000000002e-06 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.339142+0000", 2024-09-02T10:26:39.303 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6170000000000003e-06 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.345508+0000", 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0063660929999999998 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.345662+0000", 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000154182 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:26:39.304 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.345801+0000", 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013868800000000001 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.350333+0000", 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0045328579999999999 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.350567+0000", 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00023365300000000001 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.305 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.306 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.595953+0000", 2024-09-02T10:26:39.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:26:39.306 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24538633300000001 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.596011+0000", 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7975e-05 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:28.596058+0000", 2024-09-02T10:26:39.307 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7076999999999997e-05 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:200538 6.b 6:dbd5fb01:::benchmark_data_smithi187_102448_object100266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:40.755486+0000", 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 238.441337272, 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70349968900000004, 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.308 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200538 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755486+0000", 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755486+0000", 2024-09-02T10:26:39.309 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755488+0000", 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3680000000000001e-06 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755494+0000", 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6059999999999999e-06 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.310 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755507+0000", 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2850999999999999e-05 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755514+0000", 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0480000000000001e-06 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755579+0000", 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4465e-05 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.311 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755693+0000", 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114207 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755818+0000", 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124998 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.760931+0000", 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051127849999999999 2024-09-02T10:26:39.312 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.762153+0000", 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00122255 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458876+0000", 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69672282799999996 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458936+0000", 2024-09-02T10:26:39.313 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9864000000000001e-05 2024-09-02T10:26:39.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.314 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.314 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458986+0000", 2024-09-02T10:26:39.314 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0118999999999999e-05 2024-09-02T10:26:39.314 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.314 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.314 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.314 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.314 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.315 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26524.0:200538 6.b e626/533 ondisk, result = 0)", 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 238.435169962, 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69733888300000002, 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.316 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:26:39.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.317 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.317 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:26:39.317 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1900000000000001e-07 2024-09-02T10:26:39.317 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.317 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761658+0000", 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0430000000000001e-06 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458732+0000", 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69707383899999997 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458742+0000", 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4120000000000007e-06 2024-09-02T10:26:39.318 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458857+0000", 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115513 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458869+0000", 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1325000000000001e-05 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458992+0000", 2024-09-02T10:26:39.319 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012383199999999999 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "replica_scrub(pg: 1.8,from:0'0,to:0'0,epoch:640/585,start:1:10000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:01.625713+0000", 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.57111004800001, 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.265666917, 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.320 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.625713+0000", 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.625713+0000", 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.625714+0000", 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4000000000000002e-07 2024-09-02T10:26:39.321 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.625720+0000", 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5189999999999998e-06 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.891275+0000", 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26555459100000001 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.891282+0000", 2024-09-02T10:26:39.322 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7150000000000001e-06 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.891330+0000", 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7905000000000003e-05 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.891343+0000", 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3658e-05 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.323 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.891380+0000", 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6989000000000001e-05 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.8,RELEASE,e:640,from:5,reservation_nonce:0)", 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 217.30422672399999, 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54587120899999997, 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.324 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.325 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5900000000000001e-07 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892598+0000", 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5209999999999999e-06 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438366+0000", 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54576760899999999 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.326 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438376+0000", 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.029e-05 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438438+0000", 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1400000000000002e-05 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438454+0000", 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6132000000000001e-05 2024-09-02T10:26:39.327 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438468+0000", 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4098e-05 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(7.c 706/694 [PushOp(7:3cc4d02d:::benchmark_data_smithi187_120712_object20411:head, version: 673'1268, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:3cc4d02d:::benchmark_data_smithi187_120712_object20411:head@673'1268, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:20.795258+0000", 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 78.401565085000001, 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26708424600000003, 2024-09-02T10:26:39.328 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:20.795258+0000", 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:20.795258+0000", 2024-09-02T10:26:39.329 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:20.795259+0000", 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7400000000000002e-07 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:20.795322+0000", 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2786000000000005e-05 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.330 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:21.060765+0000", 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26544319599999999 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:21.060773+0000", 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5660000000000001e-06 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:21.060829+0000", 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6348000000000003e-05 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.331 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:21.060838+0000", 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8300000000000002e-06 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:21.062343+0000", 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015047459999999999 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.332 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:99745 7.2 7:41c66d73:::benchmark_data_smithi187_120712_object15803:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:00.281756+0000", 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 38.915067710999999, 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.201925363, 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 99745 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.333 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.281756+0000", 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.281756+0000", 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:26:39.334 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.281757+0000", 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.64e-06 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.281764+0000", 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1659999999999999e-06 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.281780+0000", 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6714e-05 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.335 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.281788+0000", 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7160000000000003e-06 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.281879+0000", 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0989000000000006e-05 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.282006+0000", 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012723500000000001 2024-09-02T10:26:39.336 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.282121+0000", 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114485 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286075+0000", 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039546709999999999 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.288473+0000", 2024-09-02T10:26:39.337 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023972109999999998 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.483608+0000", 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19513497199999999 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.483650+0000", 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2641000000000001e-05 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.483681+0000", 2024-09-02T10:26:39.338 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0923000000000002e-05 2024-09-02T10:26:39.339 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.339 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.339 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.339 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:26:39.339 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:26:39.339 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:26:39.339 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:26:39.340 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:26:39.340 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:26:39.340 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:26:39.340 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:26:39.341 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:26:39.341 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:26:39.341 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:26:39.362 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:39.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:39.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:39.368 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:26:39.368 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:26:39.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:39.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:39.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:39.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:39.622 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:39.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:39.718 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 7 pg_num to 26 2024-09-02T10:26:39.724 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:39.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:39.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:39.750 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:39.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:39.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:39.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:39.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:39.994 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:39 smithi151 ceph-mon[16639]: osdmap e739: 12 total, 12 up, 12 in 2024-09-02T10:26:39.995 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3368716142' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:26:39.995 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:39 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:26:40.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:40.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:39 smithi187 ceph-mon[20072]: osdmap e739: 12 total, 12 up, 12 in 2024-09-02T10:26:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3368716142' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:26:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:39 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:26:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:39 smithi162 ceph-mon[20279]: osdmap e739: 12 total, 12 up, 12 in 2024-09-02T10:26:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3368716142' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:26:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:39 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:26:40.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:40.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:40.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:40.131 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:40.232 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:40.257 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:40.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:40.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:40.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:40.383 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:40.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:40.485 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:40.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:40.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:40.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:40.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:40.645 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:40.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:40.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:40.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:40.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:40.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:40.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:40.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:40 smithi187 ceph-mon[20072]: pgmap v1689: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s 2024-09-02T10:26:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:40 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-09-02T10:26:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:40 smithi187 ceph-mon[20072]: osdmap e740: 12 total, 12 up, 12 in 2024-09-02T10:26:41.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:41.029 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:41.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:40 smithi162 ceph-mon[20279]: pgmap v1689: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s 2024-09-02T10:26:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:40 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-09-02T10:26:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:40 smithi162 ceph-mon[20279]: osdmap e740: 12 total, 12 up, 12 in 2024-09-02T10:26:41.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:41.159 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:41.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:40 smithi151 ceph-mon[16639]: pgmap v1689: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 808 op/s 2024-09-02T10:26:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:40 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num", "val": "26"}]': finished 2024-09-02T10:26:41.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:40 smithi151 ceph-mon[16639]: osdmap e740: 12 total, 12 up, 12 in 2024-09-02T10:26:41.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:41.285 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:41.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:41.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:41.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:41.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:41.409 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:41.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:41.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:41.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:41.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:41.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:41.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:41.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:41.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:41.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:41.790 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:41.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:41.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:41.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:41.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:41.933 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:41.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:41 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:41 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:42.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:42.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:42.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:42.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:42.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:41 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:42.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:41 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:42.187 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:41 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:42.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:41 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:42.208 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:42.209 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:42.310 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:42.331 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:42.332 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:42.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:42.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:42.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:42.455 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:42.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:42.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:42.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:42.681 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:42.705 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:42.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:42.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:42.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:42.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:42.835 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:42.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:42.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:42 smithi151 ceph-mon[16639]: pgmap v1691: 47 pgs: 47 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s 2024-09-02T10:26:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:42 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-02T10:26:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:42 smithi151 ceph-mon[16639]: osdmap e741: 12 total, 12 up, 12 in 2024-09-02T10:26:42.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:42.963 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:42.964 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:42 smithi187 ceph-mon[20072]: pgmap v1691: 47 pgs: 47 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s 2024-09-02T10:26:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:42 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-02T10:26:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:42 smithi187 ceph-mon[20072]: osdmap e741: 12 total, 12 up, 12 in 2024-09-02T10:26:43.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:43.094 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:43.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:43.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:42 smithi162 ceph-mon[20279]: pgmap v1691: 47 pgs: 47 active+clean; 2.1 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 674 op/s 2024-09-02T10:26:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:42 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-02T10:26:43.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:42 smithi162 ceph-mon[20279]: osdmap e741: 12 total, 12 up, 12 in 2024-09-02T10:26:43.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:43.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:43.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:43.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:43.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:43.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:43.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:43.475 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:43.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:43.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:43.488 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:26:43.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:43.599 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:43.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:43.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:43.724 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:43.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:43.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:43.791 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:26:43.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:43 smithi162 ceph-mon[20279]: osdmap e742: 12 total, 12 up, 12 in 2024-09-02T10:26:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:43 smithi162 ceph-mon[20279]: 7.11 scrub starts 2024-09-02T10:26:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:43 smithi162 ceph-mon[20279]: 7.0 scrub starts 2024-09-02T10:26:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:43 smithi162 ceph-mon[20279]: 7.8 scrub starts 2024-09-02T10:26:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:43 smithi162 ceph-mon[20279]: 7.3 scrub starts 2024-09-02T10:26:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:43 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:43 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:43.848 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:43.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:43.950 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:43.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:43.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:43.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:43 smithi187 ceph-mon[20072]: osdmap e742: 12 total, 12 up, 12 in 2024-09-02T10:26:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:43 smithi187 ceph-mon[20072]: 7.11 scrub starts 2024-09-02T10:26:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:43 smithi187 ceph-mon[20072]: 7.0 scrub starts 2024-09-02T10:26:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:43 smithi187 ceph-mon[20072]: 7.8 scrub starts 2024-09-02T10:26:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:43 smithi187 ceph-mon[20072]: 7.3 scrub starts 2024-09-02T10:26:44.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:43 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:44.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:43 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:44.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:43 smithi151 ceph-mon[16639]: osdmap e742: 12 total, 12 up, 12 in 2024-09-02T10:26:44.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:43 smithi151 ceph-mon[16639]: 7.11 scrub starts 2024-09-02T10:26:44.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:43 smithi151 ceph-mon[16639]: 7.0 scrub starts 2024-09-02T10:26:44.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:43 smithi151 ceph-mon[16639]: 7.8 scrub starts 2024-09-02T10:26:44.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:43 smithi151 ceph-mon[16639]: 7.3 scrub starts 2024-09-02T10:26:44.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:43 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:44.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:43 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:26:44.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:44.114 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:44.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:44.216 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:44.242 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:44.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:44.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:44.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:44.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:44.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:44.401 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:26:44.401 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:26:44.402 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:26:44.402 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:26:44.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:44.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:44.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:44.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:44.592 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:44.615 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:44.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:44.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:44.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:44.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:44.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:44.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:44.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:44.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:44.987 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:44.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: 7.5 scrub starts 2024-09-02T10:26:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: pgmap v1694: 57 pgs: 4 peering, 7 unknown, 46 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: 7.7 scrub starts 2024-09-02T10:26:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-02T10:26:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: osdmap e743: 12 total, 12 up, 12 in 2024-09-02T10:26:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: 7.9 scrub starts 2024-09-02T10:26:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: 7.0 scrub starts 2024-09-02T10:26:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: 7.1 scrub starts 2024-09-02T10:26:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: 7.8 scrub starts 2024-09-02T10:26:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: osdmap e744: 12 total, 12 up, 12 in 2024-09-02T10:26:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: 7.3 scrub starts 2024-09-02T10:26:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2521696452' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:26:45.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:45.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: 7.5 scrub starts 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: pgmap v1694: 57 pgs: 4 peering, 7 unknown, 46 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: 7.7 scrub starts 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: osdmap e743: 12 total, 12 up, 12 in 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: 7.9 scrub starts 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: 7.0 scrub starts 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: 7.1 scrub starts 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: 7.8 scrub starts 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: osdmap e744: 12 total, 12 up, 12 in 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: 7.3 scrub starts 2024-09-02T10:26:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2521696452' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:26:45.109 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:45.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: 7.5 scrub starts 2024-09-02T10:26:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: pgmap v1694: 57 pgs: 4 peering, 7 unknown, 46 active+clean; 2.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 673 op/s; 0 B/s, 0 objects/s recovering 2024-09-02T10:26:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: 7.7 scrub starts 2024-09-02T10:26:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_5", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-02T10:26:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: osdmap e743: 12 total, 12 up, 12 in 2024-09-02T10:26:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: 7.9 scrub starts 2024-09-02T10:26:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: 7.0 scrub starts 2024-09-02T10:26:45.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: 7.1 scrub starts 2024-09-02T10:26:45.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: 7.8 scrub starts 2024-09-02T10:26:45.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: osdmap e744: 12 total, 12 up, 12 in 2024-09-02T10:26:45.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: 7.3 scrub starts 2024-09-02T10:26:45.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2521696452' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:26:45.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:45.227 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:26:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:45.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:45.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:45.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:26:45.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:45.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:45.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:45.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:45.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:45.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:45.508 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:45.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:45.632 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:26:45.633 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:45.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:45.736 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:45.781 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:45.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:45.882 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:45.912 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:45.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:45.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:45 smithi151 ceph-mon[16639]: 7.5 scrub starts 2024-09-02T10:26:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:45 smithi151 ceph-mon[16639]: 7.7 scrub starts 2024-09-02T10:26:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:45 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:26:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:45 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:26:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:45 smithi151 ceph-mon[16639]: 7.4 scrub starts 2024-09-02T10:26:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:45 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2521696452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:26:45.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:45 smithi151 ceph-mon[16639]: osdmap e745: 12 total, 12 up, 12 in 2024-09-02T10:26:45.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:45 smithi151 ceph-mon[16639]: 7.2 scrub starts 2024-09-02T10:26:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:45 smithi187 ceph-mon[20072]: 7.5 scrub starts 2024-09-02T10:26:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:45 smithi187 ceph-mon[20072]: 7.7 scrub starts 2024-09-02T10:26:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:45 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:26:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:45 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:26:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:45 smithi187 ceph-mon[20072]: 7.4 scrub starts 2024-09-02T10:26:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:45 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2521696452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:26:46.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:45 smithi187 ceph-mon[20072]: osdmap e745: 12 total, 12 up, 12 in 2024-09-02T10:26:46.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:45 smithi187 ceph-mon[20072]: 7.2 scrub starts 2024-09-02T10:26:46.014 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:46.039 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:46.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:46.077 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:45 smithi162 ceph-mon[20279]: 7.5 scrub starts 2024-09-02T10:26:46.077 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:45 smithi162 ceph-mon[20279]: 7.7 scrub starts 2024-09-02T10:26:46.077 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:45 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:26:46.078 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:45 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:26:46.078 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:45 smithi162 ceph-mon[20279]: 7.4 scrub starts 2024-09-02T10:26:46.078 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:45 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:46.078 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2521696452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:26:46.078 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:45 smithi162 ceph-mon[20279]: osdmap e745: 12 total, 12 up, 12 in 2024-09-02T10:26:46.078 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:45 smithi162 ceph-mon[20279]: 7.2 scrub starts 2024-09-02T10:26:46.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:26:46.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:46.165 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:46.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:46.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:46.204 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:26:46.204 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:26:46.204 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:26:46.204 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:26:46.204 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:26:46.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:46.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:46.291 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:46.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:46.391 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:26:46.391 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:26:46.391 INFO:tasks.thrashosds.thrasher:inject_pause on osd.8 2024-09-02T10:26:46.391 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-02T10:26:46.392 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-02T10:26:46.392 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 config set filestore_inject_stall 3 2024-09-02T10:26:46.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:46.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:46.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:46.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:46.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:46.574 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:46.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:46.576 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:46.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:46.694 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:26:46.699 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:46.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:46.801 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:46.826 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:46.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:46.827 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:46 smithi162 ceph-mon[20279]: 7.15 starting backfill to osd.1 from (0'0,710'4472] MIN to 742'8466 2024-09-02T10:26:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:46 smithi162 ceph-mon[20279]: pgmap v1698: 57 pgs: 4 peering, 5 unknown, 48 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:26:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:46 smithi162 ceph-mon[20279]: 7.6 scrub starts 2024-09-02T10:26:46.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:46 smithi162 ceph-mon[20279]: 7.2 scrub ok 2024-09-02T10:26:46.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:26:46.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:46.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:46.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:46 smithi187 ceph-mon[20072]: 7.15 starting backfill to osd.1 from (0'0,710'4472] MIN to 742'8466 2024-09-02T10:26:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:46 smithi187 ceph-mon[20072]: pgmap v1698: 57 pgs: 4 peering, 5 unknown, 48 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:26:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:46 smithi187 ceph-mon[20072]: 7.6 scrub starts 2024-09-02T10:26:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:46 smithi187 ceph-mon[20072]: 7.2 scrub ok 2024-09-02T10:26:47.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:46 smithi151 ceph-mon[16639]: 7.15 starting backfill to osd.1 from (0'0,710'4472] MIN to 742'8466 2024-09-02T10:26:47.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:46 smithi151 ceph-mon[16639]: pgmap v1698: 57 pgs: 4 peering, 5 unknown, 48 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:26:47.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:46 smithi151 ceph-mon[16639]: 7.6 scrub starts 2024-09-02T10:26:47.010 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:46 smithi151 ceph-mon[16639]: 7.2 scrub ok 2024-09-02T10:26:47.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:47.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:47.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:47.109 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:47.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:47.227 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:26:47.227 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:26:47.230 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:26:47.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:47.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:47.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:47.262 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:26:47.262 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:26:47.290 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:26:47.291 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:26:47.316 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:26:47.317 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:26:47.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:47.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:47.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:47.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:47.508 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:47.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:47.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:47.553 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:26:47.553 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:26:47.587 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:26:47.587 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:26:47.611 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:47.625 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:26:47.626 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:26:47.635 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:47.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:47.660 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:26:47.661 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:26:47.697 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:26:47.697 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:26:47.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:47.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:26:47.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:26:47.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:47.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:47.776 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:26:47.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:26:47.813 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:26:47.814 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:26:47.814 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:47 smithi151 ceph-mon[16639]: 7.6 scrub ok 2024-09-02T10:26:47.814 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/104584163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:47.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:47.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:47.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:47.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:48.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:47 smithi187 ceph-mon[20072]: 7.6 scrub ok 2024-09-02T10:26:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/104584163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:48.024 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:48.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:48.048 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:26:48.048 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "" 2024-09-02T10:26:48.048 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:26:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:47 smithi162 ceph-mon[20279]: 7.6 scrub ok 2024-09-02T10:26:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/104584163' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:48.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:26:48.126 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:48.149 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:48.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:48.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:48.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:48.275 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:48.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:48.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:48.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:48.400 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:48.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:48.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:48.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:48.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:48.588 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:26:48.588 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:26:48.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:48.686 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:48.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:48.788 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:48.814 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:48.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:48.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:48.943 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:48.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:49.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:49.069 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:49.070 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:48 smithi162 ceph-mon[20279]: pgmap v1699: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 4 unknown, 49 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5179/89988 objects degraded (5.755%); 6131/89988 objects misplaced (6.813%) 2024-09-02T10:26:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:48 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 5179/89988 objects degraded (5.755%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:26:49.170 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:49.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:48 smithi151 ceph-mon[16639]: pgmap v1699: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 4 unknown, 49 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5179/89988 objects degraded (5.755%); 6131/89988 objects misplaced (6.813%) 2024-09-02T10:26:49.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:48 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 5179/89988 objects degraded (5.755%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:26:49.196 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:49.197 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:49.224 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:48 smithi187 ceph-mon[20072]: pgmap v1699: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 4 unknown, 49 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5179/89988 objects degraded (5.755%); 6131/89988 objects misplaced (6.813%) 2024-09-02T10:26:49.224 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:48 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 5179/89988 objects degraded (5.755%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:26:49.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:26:49.298 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:49.322 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:49.323 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:49.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:49.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:49.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:49.554 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:26:49.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:49.591 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:49.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:49.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:49.721 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:49.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:49.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:49.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:49.852 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:49.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:49.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:49.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:49.980 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:49 smithi162 ceph-mon[20279]: 7.9 scrub starts 2024-09-02T10:26:49.981 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:49 smithi162 ceph-mon[20279]: 7.0 scrub starts 2024-09-02T10:26:49.981 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:49 smithi162 ceph-mon[20279]: 7.0 scrub ok 2024-09-02T10:26:49.981 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:49 smithi162 ceph-mon[20279]: osdmap e746: 12 total, 12 up, 12 in 2024-09-02T10:26:49.981 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:49 smithi162 ceph-mon[20279]: 7.3 scrub starts 2024-09-02T10:26:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:49.986 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:49.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:50.036 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:50.087 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:50.103 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:26:50.104 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:26:50.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:50.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:50.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:49 smithi151 ceph-mon[16639]: 7.9 scrub starts 2024-09-02T10:26:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:49 smithi151 ceph-mon[16639]: 7.0 scrub starts 2024-09-02T10:26:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:49 smithi151 ceph-mon[16639]: 7.0 scrub ok 2024-09-02T10:26:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:49 smithi151 ceph-mon[16639]: osdmap e746: 12 total, 12 up, 12 in 2024-09-02T10:26:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:49 smithi151 ceph-mon[16639]: 7.3 scrub starts 2024-09-02T10:26:50.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:49 smithi187 ceph-mon[20072]: 7.9 scrub starts 2024-09-02T10:26:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:49 smithi187 ceph-mon[20072]: 7.0 scrub starts 2024-09-02T10:26:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:49 smithi187 ceph-mon[20072]: 7.0 scrub ok 2024-09-02T10:26:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:49 smithi187 ceph-mon[20072]: osdmap e746: 12 total, 12 up, 12 in 2024-09-02T10:26:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:49 smithi187 ceph-mon[20072]: 7.3 scrub starts 2024-09-02T10:26:50.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:50.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:50.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:50.421 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:50.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:50.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:50.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:50.548 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:50.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:50.673 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:50.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:50.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:50.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:50.799 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:50.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:50.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:26:50.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:50.924 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:50.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: 7.9 scrub ok 2024-09-02T10:26:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: pgmap v1701: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14022/89625 objects degraded (15.645%); 22168/89625 objects misplaced (24.734%) 2024-09-02T10:26:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: 7.3 scrub ok 2024-09-02T10:26:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: 7.5 scrub starts 2024-09-02T10:26:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: 7.7 scrub starts 2024-09-02T10:26:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: 7.4 scrub starts 2024-09-02T10:26:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: 7.1 scrub starts 2024-09-02T10:26:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: 7.7 scrub ok 2024-09-02T10:26:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: 7.8 scrub starts 2024-09-02T10:26:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: 7.1 scrub ok 2024-09-02T10:26:50.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:50 smithi151 ceph-mon[16639]: 7.8 scrub ok 2024-09-02T10:26:51.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:51.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:51.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:51.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: 7.9 scrub ok 2024-09-02T10:26:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: pgmap v1701: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14022/89625 objects degraded (15.645%); 22168/89625 objects misplaced (24.734%) 2024-09-02T10:26:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: 7.3 scrub ok 2024-09-02T10:26:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: 7.5 scrub starts 2024-09-02T10:26:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: 7.7 scrub starts 2024-09-02T10:26:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: 7.4 scrub starts 2024-09-02T10:26:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: 7.1 scrub starts 2024-09-02T10:26:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: 7.7 scrub ok 2024-09-02T10:26:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: 7.8 scrub starts 2024-09-02T10:26:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: 7.1 scrub ok 2024-09-02T10:26:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:50 smithi162 ceph-mon[20279]: 7.8 scrub ok 2024-09-02T10:26:51.138 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:26:51.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:51.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:51.177 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: 7.9 scrub ok 2024-09-02T10:26:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: pgmap v1701: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14022/89625 objects degraded (15.645%); 22168/89625 objects misplaced (24.734%) 2024-09-02T10:26:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: 7.3 scrub ok 2024-09-02T10:26:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: 7.5 scrub starts 2024-09-02T10:26:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: 7.7 scrub starts 2024-09-02T10:26:51.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: 7.4 scrub starts 2024-09-02T10:26:51.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: 7.1 scrub starts 2024-09-02T10:26:51.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: 7.7 scrub ok 2024-09-02T10:26:51.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: 7.8 scrub starts 2024-09-02T10:26:51.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: 7.1 scrub ok 2024-09-02T10:26:51.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:50 smithi187 ceph-mon[20072]: 7.8 scrub ok 2024-09-02T10:26:51.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:51.304 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:51.304 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:51.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:51.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:51.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:51.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:51.574 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:51.575 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:51.658 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:26:51.658 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:26:51.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:51.701 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:51.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:51.782 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:26:51.802 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:51 smithi162 ceph-mon[20279]: 7.5 scrub ok 2024-09-02T10:26:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:51 smithi162 ceph-mon[20279]: 7.4 scrub ok 2024-09-02T10:26:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1629211510' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:51.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:51.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:51.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:26:51.958 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:51.980 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:51.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:52.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:51 smithi151 ceph-mon[16639]: 7.5 scrub ok 2024-09-02T10:26:52.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:51 smithi151 ceph-mon[16639]: 7.4 scrub ok 2024-09-02T10:26:52.043 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1629211510' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:52.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:52.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:52.118 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:52.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:52.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:51 smithi187 ceph-mon[20072]: 7.5 scrub ok 2024-09-02T10:26:52.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:51 smithi187 ceph-mon[20072]: 7.4 scrub ok 2024-09-02T10:26:52.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1629211510' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:26:52.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:52.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:52.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:52.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:52.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:52.387 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:52.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:52.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:52.489 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:52.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:26:52.514 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:52.515 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:52.615 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:52.639 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:52.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:52.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:52.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:52.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:52.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:52.788 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:26:52.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:52.907 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:52.908 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:53.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:53.033 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:53.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:53.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:52 smithi162 ceph-mon[20279]: pgmap v1702: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14022/89466 objects degraded (15.673%); 22168/89466 objects misplaced (24.778%) 2024-09-02T10:26:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:52 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:26:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1629211510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:26:53.059 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:52 smithi162 ceph-mon[20279]: osdmap e747: 12 total, 12 up, 12 in 2024-09-02T10:26:53.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:53.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:53.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:52 smithi151 ceph-mon[16639]: pgmap v1702: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14022/89466 objects degraded (15.673%); 22168/89466 objects misplaced (24.778%) 2024-09-02T10:26:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:52 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:26:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1629211510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:26:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:52 smithi151 ceph-mon[16639]: osdmap e747: 12 total, 12 up, 12 in 2024-09-02T10:26:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:52 smithi187 ceph-mon[20072]: pgmap v1702: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.8 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 14022/89466 objects degraded (15.673%); 22168/89466 objects misplaced (24.778%) 2024-09-02T10:26:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:52 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:26:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1629211510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:26:53.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:52 smithi187 ceph-mon[20072]: osdmap e747: 12 total, 12 up, 12 in 2024-09-02T10:26:53.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:53.298 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:53.299 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:53.307 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:26:53.307 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:26:53.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:53.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:53.425 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:53.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:53.433 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:53.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:53.550 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:53.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:53.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:53.686 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:53.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:53.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:53.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:53.811 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:53.812 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:53.907 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:26:53.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-09-02T10:26:53.913 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:53.936 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:53.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:54.038 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:54.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:54.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:54.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:53 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 13483/83667 objects degraded (16.115%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:26:54.145 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:26:54.145 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:26:54.166 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:53 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 13483/83667 objects degraded (16.115%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:26:54.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:54.186 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:26:54.186 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:26:54.203 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:26:54.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:54.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:54.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:54.225 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:26:54.226 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:26:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:53 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 13483/83667 objects degraded (16.115%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:26:54.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:26:54.268 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:26:54.306 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:26:54.306 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:26:54.310 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:54.344 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:26:54.344 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:26:54.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:54.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:54.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:54.383 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:26:54.383 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:26:54.383 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:54.419 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:26:54.419 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:26:54.460 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:26:54.460 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:26:54.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:54.489 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:26:54.497 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:26:54.497 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:26:54.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:54.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:54.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:26:54.534 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:26:54.570 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:26:54.571 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:26:54.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:54.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:54.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:54.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:54.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:54.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:54.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:54.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:54.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:54.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:54.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:54.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:54 smithi151 ceph-mon[16639]: pgmap v1704: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13483/83667 objects degraded (16.115%); 20587/83667 objects misplaced (24.606%) 2024-09-02T10:26:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1994828032' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:55.016 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:26:55.016 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:26:55.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:141753 7.d 7:b58e8e82:::benchmark_data_smithi187_120712_object57811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:55.005732+0000", 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0088178649999999994, 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0088381139999999993, 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141753 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.017 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.005732+0000", 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.005732+0000", 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:55.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.005733+0000", 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7699999999999992e-07 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.005737+0000", 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4730000000000002e-06 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.005748+0000", 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0781000000000001e-05 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.005753+0000", 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4110000000000004e-06 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.005805+0000", 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1860000000000002e-05 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.005932+0000", 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126635 2024-09-02T10:26:55.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.006049+0000", 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116826 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.009675+0000", 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036266250000000001 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:55.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.011573+0000", 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001897656 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:141736 7.c 7:33834245:::benchmark_data_smithi187_120712_object57794:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:54.974699+0000", 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.039850586, 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.039918072999999998, 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:55.022 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141736 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.974699+0000", 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:55.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.974699+0000", 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.974700+0000", 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.393e-06 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.974706+0000", 2024-09-02T10:26:55.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6799999999999998e-06 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.974719+0000", 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2534000000000001e-05 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.974726+0000", 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6480000000000003e-06 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:55.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.974788+0000", 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2088000000000006e-05 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.974923+0000", 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013520900000000001 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.975033+0000", 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010922 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.978321+0000", 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003288188 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.986141+0000", 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0078200920000000007 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.027 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:141748 7.d 7:b9621361:::benchmark_data_smithi187_120712_object57806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e747)", 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:54.993488+0000", 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.021062014, 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021169332999999999, 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 141748 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.993488+0000", 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.993488+0000", 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.993489+0000", 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.934e-06 2024-09-02T10:26:55.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.993496+0000", 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7789999999999998e-06 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.993515+0000", 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8402000000000001e-05 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.993525+0000", 2024-09-02T10:26:55.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0271e-05 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.994414+0000", 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00088876099999999996 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.994541+0000", 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000127341 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:26:55.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.994661+0000", 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011974800000000001 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:54.998249+0000", 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035887409999999999 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:55.002844+0000", 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0045940750000000004 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:55.032 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:55.033 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:55.033 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:55.033 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:26:55.033 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 3 2024-09-02T10:26:55.033 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:26:55.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:55.061 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:55.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:55.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:55.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:26:55.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:55.189 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:26:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:54 smithi187 ceph-mon[20072]: pgmap v1704: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13483/83667 objects degraded (16.115%); 20587/83667 objects misplaced (24.606%) 2024-09-02T10:26:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1994828032' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:55.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:55.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:26:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:55.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:55.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:55.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:54 smithi162 ceph-mon[20279]: pgmap v1704: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13483/83667 objects degraded (16.115%); 20587/83667 objects misplaced (24.606%) 2024-09-02T10:26:55.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1994828032' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:26:55.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:55.471 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:55.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:55.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:55.573 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:55.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:55.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:55.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:55.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:55.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:55.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:26:55.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:55.848 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:55.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:55.860 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:26:55.950 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:55.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:55.973 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:56.060 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:26:56.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:56.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:56.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:56.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:56.201 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:26:56.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:56.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:26:56.331 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:56.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:56.360 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:56.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:56.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:56.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:56.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:56.567 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:26:56.568 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:26:56.568 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:26:56.568 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:26:56.568 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:26:56.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:56.633 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:56.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:56.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:56.760 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:56.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:56.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:56.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:56.882 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-09-02T10:26:56.886 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:56.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:56.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:57.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:57.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:57.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:57.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:57.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:56 smithi151 ceph-mon[16639]: pgmap v1705: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13483/81426 objects degraded (16.559%); 19561/81426 objects misplaced (24.023%) 2024-09-02T10:26:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/634236199' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:26:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:56 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:26:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:56 smithi187 ceph-mon[20072]: pgmap v1705: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13483/81426 objects degraded (16.559%); 19561/81426 objects misplaced (24.023%) 2024-09-02T10:26:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/634236199' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:26:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:56 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:26:57.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:26:57.308 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:57.309 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:26:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:56 smithi162 ceph-mon[20279]: pgmap v1705: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13483/81426 objects degraded (16.559%); 19561/81426 objects misplaced (24.023%) 2024-09-02T10:26:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/634236199' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:26:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:56 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:26:57.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:57.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:57.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:57.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:26:57.552 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:57.575 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:57.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:57.640 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:26:57.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:26:57.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:57.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:57.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:26:57.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:57.850 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:57.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:57.905 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:26:57.932 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:26:57.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:57.977 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:57.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:57.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:58.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:58.104 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:58.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:57 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:26:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:57 smithi162 ceph-mon[20279]: osdmap e748: 12 total, 12 up, 12 in 2024-09-02T10:26:58.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3896061159' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:26:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:57 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:26:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:57 smithi151 ceph-mon[16639]: osdmap e748: 12 total, 12 up, 12 in 2024-09-02T10:26:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3896061159' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:26:58.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:58.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:58.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:58.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:57 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:26:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:57 smithi187 ceph-mon[20072]: osdmap e748: 12 total, 12 up, 12 in 2024-09-02T10:26:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3896061159' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:26:58.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:26:58.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:58.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:58.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:26:58.416 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:26:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:26:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:26:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:26:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162692 5.2 5:4dd653e2:::benchmark_data_smithi187_89405_object59087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:26:58.417 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 574.06109607799999, 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25261044100000002, 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162692 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353165+0000", 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353166+0000", 2024-09-02T10:26:58.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8200000000000008e-07 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353170+0000", 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3880000000000002e-06 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353180+0000", 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0336999999999999e-05 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353188+0000", 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7060000000000004e-06 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353233+0000", 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4453000000000003e-05 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353306+0000", 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3559999999999994e-05 2024-09-02T10:26:58.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353375+0000", 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9202999999999995e-05 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360656+0000", 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0072804360000000004 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360676+0000", 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0075e-05 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605656+0000", 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24498062500000001 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605693+0000", 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6377000000000001e-05 2024-09-02T10:26:58.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605775+0000", 2024-09-02T10:26:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2299000000000006e-05 2024-09-02T10:26:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.424 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:162694 5.2 5:4549dcce:::benchmark_data_smithi187_89405_object59089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 574.06049989099995, 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25215807000000001, 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:26:58.425 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 162694 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353761+0000", 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353762+0000", 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3600000000000003e-07 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353765+0000", 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6559999999999998e-06 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353776+0000", 2024-09-02T10:26:58.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0243e-05 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.353781+0000", 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0540000000000002e-06 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.356988+0000", 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0032069899999999998 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.357116+0000", 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012815 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,10", 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.357235+0000", 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011919 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.360716+0000", 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003481254 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.409925+0000", 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049208533999999998 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605820+0000", 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19589521600000001 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605878+0000", 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8149000000000002e-05 2024-09-02T10:26:58.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:17:24.605919+0000", 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0898e-05 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:121249 6.c e579/533 6:31dbf052:::benchmark_data_smithi187_102448_object20977:head v 579'7531, mlcod=579'7526)", 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:34.371591+0000", 2024-09-02T10:26:58.431 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 384.04267007700003, 2024-09-02T10:26:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.231073748, 2024-09-02T10:26:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.371591+0000", 2024-09-02T10:26:58.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.371591+0000", 2024-09-02T10:26:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.371592+0000", 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8000000000000004e-07 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.371598+0000", 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4790000000000001e-06 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.583533+0000", 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21193474500000001 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.583542+0000", 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8589999999999994e-06 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.583622+0000", 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9844000000000003e-05 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.583646+0000", 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4394999999999999e-05 2024-09-02T10:26:58.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.602623+0000", 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018977099000000001 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:34.602664+0000", 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1446999999999998e-05 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:193652 6.c e622/533 6:35889ade:::benchmark_data_smithi187_102448_object93380:head v 622'11980, mlcod=622'11978)", 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:28.340137+0000", 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 270.07412396900003, 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24338019899999999, 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.340137+0000", 2024-09-02T10:26:58.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.340137+0000", 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.340138+0000", 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.023e-06 2024-09-02T10:26:58.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.340145+0000", 2024-09-02T10:26:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1130000000000003e-06 2024-09-02T10:26:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.340176+0000", 2024-09-02T10:26:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1424999999999998e-05 2024-09-02T10:26:58.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.340184+0000", 2024-09-02T10:26:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8110000000000003e-06 2024-09-02T10:26:58.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.340237+0000", 2024-09-02T10:26:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2796000000000001e-05 2024-09-02T10:26:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.340258+0000", 2024-09-02T10:26:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1121999999999999e-05 2024-09-02T10:26:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.344312+0000", 2024-09-02T10:26:58.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040534220000000001 2024-09-02T10:26:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.583517+0000", 2024-09-02T10:26:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23920548699999999 2024-09-02T10:26:58.443 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:193653 6.c e622/533 6:3563a971:::benchmark_data_smithi187_102448_object93381:head v 622'11981, mlcod=622'11978)", 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:28.345924+0000", 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 270.06833644900001, 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24993547199999999, 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.444 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345924+0000", 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345924+0000", 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345925+0000", 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.068e-06 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345932+0000", 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0790000000000003e-06 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.583538+0000", 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23760562399999999 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.583550+0000", 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.217e-05 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.583616+0000", 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5530000000000004e-05 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.583641+0000", 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4788000000000001e-05 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.595556+0000", 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011915627 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.595860+0000", 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000303586 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.26524.0:197503 6.5 e625/533 6:abdd639a:::benchmark_data_smithi187_102448_object97231:head v 625'12442, mlcod=625'12439)", 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:35.724436+0000", 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 262.68982439000001, 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22813137, 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.724436+0000", 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.724436+0000", 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.724437+0000", 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.035e-06 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.724445+0000", 2024-09-02T10:26:58.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6799999999999993e-06 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.738864+0000", 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0144189 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.738870+0000", 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0000000000000002e-06 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.738919+0000", 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8683000000000001e-05 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.928240+0000", 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.189320871 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.952524+0000", 2024-09-02T10:26:58.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.024284549999999998 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:35.952568+0000", 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3650999999999999e-05 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:55454 7.d 7:b8d34c18:::benchmark_data_smithi187_120712_object55453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:26:58.453 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:07.546586+0000", 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 110.86767466000001, 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23446257500000001, 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55454 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.546586+0000", 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.546586+0000", 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.546588+0000", 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4619999999999999e-06 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.546638+0000", 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0608000000000001e-05 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.546650+0000", 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1586000000000001e-05 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.546655+0000", 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0250000000000002e-06 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.546715+0000", 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0421999999999998e-05 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.546815+0000", 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100035 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.546902+0000", 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6463000000000002e-05 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.548551+0000", 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001649569 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.738146+0000", 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18959511700000001 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.780966+0000", 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.042819889999999999 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.781006+0000", 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9453999999999999e-05 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.781049+0000", 2024-09-02T10:26:58.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2944000000000001e-05 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:55454 7.d e696/661 ondisk, result = 0)", 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:07.549268+0000", 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 110.864992528, 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.231787888, 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.460 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.549268+0000", 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.549268+0000", 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.549269+0000", 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0299999999999998e-07 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.549272+0000", 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.439e-06 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.780160+0000", 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23088721200000001 2024-09-02T10:26:58.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.780166+0000", 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.37e-06 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.780958+0000", 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00079182799999999995 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.780963+0000", 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9320000000000004e-06 2024-09-02T10:26:58.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.781056+0000", 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3404000000000002e-05 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:65671 7.d 7:b84e577b:::benchmark_data_smithi187_120712_object65670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:20.550861+0000", 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.863399340000001, 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22606066, 2024-09-02T10:26:58.464 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 65671 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.550861+0000", 2024-09-02T10:26:58.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.550861+0000", 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.550862+0000", 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6499999999999998e-07 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.550923+0000", 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0748000000000002e-05 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.550933+0000", 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8369999999999992e-06 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.550938+0000", 2024-09-02T10:26:58.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.485e-06 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.550973+0000", 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4193000000000003e-05 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.551059+0000", 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6594000000000005e-05 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.551142+0000", 2024-09-02T10:26:58.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2675000000000004e-05 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.552482+0000", 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001340085 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.554340+0000", 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018582049999999999 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776853+0000", 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222512819 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776891+0000", 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8115999999999999e-05 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776922+0000", 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1038000000000003e-05 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.470 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:65697 7.d 7:b3a5eaf4:::benchmark_data_smithi187_120712_object65696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:20.774076+0000", 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.640185216999996, 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48971618, 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:58.471 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 65697 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774076+0000", 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774076+0000", 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774077+0000", 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8950000000000001e-06 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774127+0000", 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9299999999999999e-05 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774144+0000", 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6858999999999999e-05 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774151+0000", 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9500000000000004e-06 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774205+0000", 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4392999999999999e-05 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774314+0000", 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010882699999999999 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774413+0000", 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9253999999999995e-05 2024-09-02T10:26:58.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776257+0000", 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001843896 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776758+0000", 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000501294 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263673+0000", 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48691518299999997 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263745+0000", 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1769999999999999e-05 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263792+0000", 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6558999999999999e-05 2024-09-02T10:26:58.477 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:65697 7.d e706/661 ondisk, result = 0)", 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:20.776789+0000", 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.637471896999998, 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48700850400000001, 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776789+0000", 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776789+0000", 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776789+0000", 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3999999999999997e-07 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776792+0000", 2024-09-02T10:26:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.576e-06 2024-09-02T10:26:58.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263583+0000", 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.486791272 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263589+0000", 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5779999999999998e-06 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263660+0000", 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1297000000000005e-05 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263667+0000", 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0879999999999997e-06 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263797+0000", 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013035299999999999 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.483 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:65699 7.c 7:3d3e89d8:::benchmark_data_smithi187_120712_object65698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:20.788356+0000", 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.625904413000001, 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27550548699999999, 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.484 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 65699 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788356+0000", 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788356+0000", 2024-09-02T10:26:58.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788359+0000", 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.131e-06 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788410+0000", 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1156999999999998e-05 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788427+0000", 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7198999999999999e-05 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788437+0000", 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7680000000000003e-06 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788502+0000", 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5049999999999996e-05 2024-09-02T10:26:58.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788627+0000", 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012572000000000001 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788741+0000", 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113211 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.790598+0000", 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018573159999999999 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.792472+0000", 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001874393 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062706+0000", 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27023351600000001 2024-09-02T10:26:58.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062753+0000", 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6978000000000003e-05 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.063862+0000", 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011090480000000001 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:65699 7.c e706/694 ondisk, result = 0)", 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:20.797961+0000", 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 97.616299421999997, 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26590793699999998, 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.797961+0000", 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.797961+0000", 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.797962+0000", 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8700000000000005e-07 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.797966+0000", 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.614e-06 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062630+0000", 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26466413300000002 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062638+0000", 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4689999999999997e-06 2024-09-02T10:26:58.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062692+0000", 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4214e-05 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062701+0000", 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.063869+0000", 2024-09-02T10:26:58.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00116807 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:100943 7.c 7:3c179c28:::benchmark_data_smithi187_120712_object17001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:01.695117+0000", 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.719143307000003, 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26333421299999998, 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.495 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 100943 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695117+0000", 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695117+0000", 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695120+0000", 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2139999999999999e-06 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695130+0000", 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8959999999999998e-06 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695159+0000", 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9539e-05 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695174+0000", 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695336+0000", 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016231700000000001 2024-09-02T10:26:58.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695535+0000", 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019830499999999999 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695643+0000", 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108521 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.698192+0000", 2024-09-02T10:26:58.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025493569999999999 2024-09-02T10:26:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.698560+0000", 2024-09-02T10:26:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036743600000000001 2024-09-02T10:26:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958382+0000", 2024-09-02T10:26:58.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25982233300000002 2024-09-02T10:26:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958419+0000", 2024-09-02T10:26:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6749999999999999e-05 2024-09-02T10:26:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958452+0000", 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2678000000000001e-05 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:100999 7.d 7:bd5ad45b:::benchmark_data_smithi187_120712_object17057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:01.958635+0000", 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.455625343999998, 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42676124199999999, 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.502 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 100999 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958635+0000", 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958635+0000", 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958636+0000", 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1699999999999997e-07 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958640+0000", 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.286e-06 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958649+0000", 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9889999999999998e-06 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958653+0000", 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6439999999999999e-06 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958693+0000", 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0079e-05 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958790+0000", 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6644000000000005e-05 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958854+0000", 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3767000000000001e-05 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.961222+0000", 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023684969999999998 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.961990+0000", 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000768178 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385337+0000", 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.423346471 2024-09-02T10:26:58.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385370+0000", 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3349999999999997e-05 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385397+0000", 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.652e-05 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.508 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:100999 7.d e723/661 ondisk, result = 0)", 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:01.962006+0000", 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 56.452254508999999, 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42339489600000002, 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.962006+0000", 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.962006+0000", 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.962007+0000", 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0200000000000002e-07 2024-09-02T10:26:58.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.962010+0000", 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1810000000000001e-06 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385252+0000", 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42324162900000001 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385257+0000", 2024-09-02T10:26:58.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2959999999999998e-06 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385328+0000", 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0612999999999996e-05 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385333+0000", 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.933e-06 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385401+0000", 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8641999999999998e-05 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:121238 7.d 7:bd43be96:::benchmark_data_smithi187_120712_object37296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e732)", 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:24.780132+0000", 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 33.634128867000001, 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.225465363, 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.513 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 121238 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.780132+0000", 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.780132+0000", 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.780133+0000", 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5199999999999996e-07 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.780136+0000", 2024-09-02T10:26:58.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.145e-06 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.780144+0000", 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9999999999999996e-06 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.780148+0000", 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5739999999999997e-06 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.516 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.780180+0000", 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1813999999999997e-05 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.780250+0000", 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9651000000000001e-05 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.780310+0000", 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9790000000000002e-05 2024-09-02T10:26:58.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.781892+0000", 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0015828229999999999 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:24.784288+0000", 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023960420000000001 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:25.005495+0000", 2024-09-02T10:26:58.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22120677999999999 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:25.005552+0000", 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6579000000000002e-05 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:25.005597+0000", 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5513000000000003e-05 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:135857 7.17 7:e979241f:::benchmark_data_smithi187_120712_object51915:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:41.733787+0000", 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.680474264000001, 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0657094760000001, 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135857 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.520 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733787+0000", 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733787+0000", 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733787+0000", 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2200000000000003e-07 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733791+0000", 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.213e-06 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733801+0000", 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733807+0000", 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7089999999999996e-06 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.882008+0000", 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14820015 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.882017+0000", 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9900000000000003e-06 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.799411+0000", 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91739405500000004 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.799496+0000", 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5364000000000005e-05 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:135859 7.17 7:eb549113:::benchmark_data_smithi187_120712_object51917:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:26:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:41.733821+0000", 2024-09-02T10:26:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 16.680439405000001, 2024-09-02T10:26:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.066307179, 2024-09-02T10:26:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.525 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135859 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733821+0000", 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733821+0000", 2024-09-02T10:26:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733822+0000", 2024-09-02T10:26:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4499999999999997e-07 2024-09-02T10:26:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.527 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733824+0000", 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6139999999999999e-06 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733831+0000", 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.827e-06 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733837+0000", 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1940000000000003e-06 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.528 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.882038+0000", 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14820045000000001 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.882043+0000", 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4079999999999997e-06 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.800039+0000", 2024-09-02T10:26:58.529 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91799532699999997 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.800129+0000", 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9913999999999993e-05 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:136915 7.19 7.95b520b9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:43.759447+0000", 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 14.654814008000001, 2024-09-02T10:26:58.530 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.47116866499999999, 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136915 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759447+0000", 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.531 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759447+0000", 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759448+0000", 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1069999999999999e-06 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:26:58.532 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759453+0000", 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.096e-06 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759465+0000", 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2265e-05 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759471+0000", 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.519e-06 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.533 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759524+0000", 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2970999999999998e-05 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759531+0000", 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3080000000000002e-06 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.230604+0000", 2024-09-02T10:26:58.534 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.47107291099999998 2024-09-02T10:26:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:26:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:26:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:26:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.230615+0000", 2024-09-02T10:26:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1488000000000001e-05 2024-09-02T10:26:58.535 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:26:58.536 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:26:58.536 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:26:58.564 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:58.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:58.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:58.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:58.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:58.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:26:58.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:58.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:26:58.941 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:58.965 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:58.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:58.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:59.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:26:59.095 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:59.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:26:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:58 smithi151 ceph-mon[16639]: pgmap v1707: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13298/78444 objects degraded (16.952%); 18769/78444 objects misplaced (23.927%) 2024-09-02T10:26:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:58 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3896061159' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:26:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:58 smithi151 ceph-mon[16639]: osdmap e749: 12 total, 12 up, 12 in 2024-09-02T10:26:59.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:26:59.218 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:59.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:26:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:58 smithi187 ceph-mon[20072]: pgmap v1707: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13298/78444 objects degraded (16.952%); 18769/78444 objects misplaced (23.927%) 2024-09-02T10:26:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:58 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3896061159' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:26:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:58 smithi187 ceph-mon[20072]: osdmap e749: 12 total, 12 up, 12 in 2024-09-02T10:26:59.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:26:59.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:58 smithi162 ceph-mon[20279]: pgmap v1707: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.6 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 13298/78444 objects degraded (16.952%); 18769/78444 objects misplaced (23.927%) 2024-09-02T10:26:59.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:58 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:26:59.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3896061159' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:26:59.321 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:58 smithi162 ceph-mon[20279]: osdmap e749: 12 total, 12 up, 12 in 2024-09-02T10:26:59.352 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:59.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:59.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:26:59.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:26:59.454 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:59.478 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:59.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:26:59.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:59.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:59.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:59.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:59.712 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:26:59.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:26:59.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:26:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:59.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:26:59.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:26:59.870 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:26:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:59.871 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:26:59.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:26:59.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:26:59.997 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:26:59.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:26:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 13298/78444 objects degraded (16.952%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:00.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:00.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:00.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:00.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:00.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:26:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 13298/78444 objects degraded (16.952%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:00.224 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:27:00.224 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:27:00.224 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:27:00.224 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:27:00.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:26:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 13298/78444 objects degraded (16.952%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:00.267 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:00.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:00.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:00.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:00.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:00.393 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:00.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:00.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:00.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:00.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:00.661 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:00.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:00.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:00.787 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:00.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:00.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:00.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:00.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:27:00.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:00.916 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:01.016 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:01.038 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:01.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:01.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:01.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:01.167 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:27:01.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:01.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:01.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:00 smithi151 ceph-mon[16639]: pgmap v1709: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12950/73791 objects degraded (17.550%); 16764/73791 objects misplaced (22.718%) 2024-09-02T10:27:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:00 smithi187 ceph-mon[20072]: pgmap v1709: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12950/73791 objects degraded (17.550%); 16764/73791 objects misplaced (22.718%) 2024-09-02T10:27:01.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:01.292 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:00 smithi162 ceph-mon[20279]: pgmap v1709: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.5 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 12950/73791 objects degraded (17.550%); 16764/73791 objects misplaced (22.718%) 2024-09-02T10:27:01.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:01.309 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:01.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:01.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:01.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:01.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:01.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:01.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:01.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:01.667 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:27:01.668 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:27:01.668 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:27:01.668 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:27:01.668 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:27:01.688 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:01.713 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:01.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:01.816 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:01.840 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:01.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:01.841 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:01.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:01.967 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:01.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:02.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:02.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:02.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:02.201 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:02.226 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:02.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:02.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:02.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:02.355 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:02.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:02.356 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:02.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:02.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:27:02.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:02.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:02.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:02.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:02.626 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:02.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:02.728 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:02.758 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:02.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:02.773 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:27:02.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:02.883 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:02.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:02.884 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:02.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:03.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:03.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:03.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:03.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:03.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:03.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:03.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:03.172 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:02 smithi162 ceph-mon[20279]: pgmap v1710: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12950/73545 objects degraded (17.608%); 16764/73545 objects misplaced (22.794%) 2024-09-02T10:27:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:02 smithi151 ceph-mon[16639]: pgmap v1710: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12950/73545 objects degraded (17.608%); 16764/73545 objects misplaced (22.794%) 2024-09-02T10:27:03.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:02 smithi187 ceph-mon[20072]: pgmap v1710: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 12950/73545 objects degraded (17.608%); 16764/73545 objects misplaced (22.794%) 2024-09-02T10:27:03.273 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:03.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:03.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:03.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:03.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:03.401 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:03.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:03.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:03.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:03.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:03.580 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:27:03.580 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:205302 5.1 e524/431 5:85fde85c:::benchmark_data_smithi187_89405_object101697:head v 524'12779, mlcod=524'12776)", 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:18:17.242083+0000", 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 526.33563442699995, 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31752161400000001, 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.242083+0000", 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.242083+0000", 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.242084+0000", 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5000000000000001e-07 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.242090+0000", 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5300000000000002e-06 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.555639+0000", 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31354897399999998 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.555645+0000", 2024-09-02T10:27:03.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7529999999999998e-06 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.555691+0000", 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5738999999999997e-05 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.555708+0000", 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6651e-05 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.559576+0000", 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003868696 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.559605+0000", 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8421000000000001e-05 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.585 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.25933.0:205307 5.9 5:95d5dc53:::benchmark_data_smithi187_89405_object101702:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:18:17.254106+0000", 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 526.32361120200005, 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28016561099999998, 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.25933", 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 205307 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.254106+0000", 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.254106+0000", 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.254107+0000", 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1599999999999999e-06 2024-09-02T10:27:03.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.254113+0000", 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4199999999999998e-06 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.254124+0000", 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1109e-05 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.254130+0000", 2024-09-02T10:27:03.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6570000000000002e-06 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.254190+0000", 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0615999999999997e-05 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.254329+0000", 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013915499999999999 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,11", 2024-09-02T10:27:03.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.254461+0000", 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013197999999999999 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.258456+0000", 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039946499999999998 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.258533+0000", 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7479999999999997e-05 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.534164+0000", 2024-09-02T10:27:03.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.275630606 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.534225+0000", 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0689000000000001e-05 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.534272+0000", 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7089000000000003e-05 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.25933.0:205324 5.1 e524/431 5:85edc945:::benchmark_data_smithi187_89405_object101719:head v 524'12781, mlcod=524'12778)", 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:18:17.602685+0000", 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 525.97503199400001, 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.385615027, 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.602685+0000", 2024-09-02T10:27:03.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.602685+0000", 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.602687+0000", 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.717e-06 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.602698+0000", 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1233000000000001e-05 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.984288+0000", 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38158929699999999 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.984292+0000", 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6e-06 2024-09-02T10:27:03.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.984347+0000", 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4528999999999999e-05 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.984363+0000", 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5784999999999999e-05 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.988265+0000", 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039021659999999999 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:18:17.988300+0000", 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.57e-05 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:63466 6.8 6:16b63edb:::benchmark_data_smithi187_102448_object63465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-02T10:27:03.596 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:19:35.652969+0000", 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 447.92474801899999, 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.311925112, 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 63466 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.652969+0000", 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.652969+0000", 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.652970+0000", 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4099999999999998e-07 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.653025+0000", 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5368999999999998e-05 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.653034+0000", 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.259e-06 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.653055+0000", 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1186e-05 2024-09-02T10:27:03.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.653110+0000", 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5084999999999999e-05 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.653189+0000", 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9536000000000006e-05 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.653272+0000", 2024-09-02T10:27:03.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2844999999999997e-05 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.654313+0000", 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010408660000000001 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.654785+0000", 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00047135200000000002 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.601 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.964795+0000", 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31001089300000001 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.964852+0000", 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6909e-05 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:19:35.964895+0000", 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2171e-05 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.602 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.9 e585 from shard 2)", 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:10.193840+0000", 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 353.38387772900001, 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.84180176500000004, 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:10.193840+0000", 2024-09-02T10:27:03.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:10.193840+0000", 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:10.193840+0000", 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4899999999999995e-07 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:10.193844+0000", 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0799999999999999e-06 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:11.035350+0000", 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.84150551299999998 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:11.035359+0000", 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3789999999999995e-06 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:11.035601+0000", 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024139499999999999 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:11.035613+0000", 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1926000000000001e-05 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:11.035641+0000", 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8822999999999998e-05 2024-09-02T10:27:03.606 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157286 6.5 6:aeee93d8:::benchmark_data_smithi187_102448_object57014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:27.949571+0000", 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 335.62814680600002, 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30036963500000002, 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.607 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157286 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949571+0000", 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949571+0000", 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949573+0000", 2024-09-02T10:27:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0779999999999998e-06 2024-09-02T10:27:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949581+0000", 2024-09-02T10:27:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7839999999999992e-06 2024-09-02T10:27:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949604+0000", 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2959000000000002e-05 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949613+0000", 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9989999999999997e-06 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949683+0000", 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9142999999999999e-05 2024-09-02T10:27:03.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949797+0000", 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114734 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949886+0000", 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8478000000000002e-05 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.955070+0000", 2024-09-02T10:27:03.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051841719999999999 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.971047+0000", 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015976596999999999 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.249846+0000", 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.278799621 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.249889+0000", 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2719000000000002e-05 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.249940+0000", 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1350999999999997e-05 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157292 6.8 6:12f72cdc:::benchmark_data_smithi187_102448_object57020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:27:03.614 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:27.961774+0000", 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 335.61594314299998, 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48684550700000001, 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157292 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961774+0000", 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961774+0000", 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961776+0000", 2024-09-02T10:27:03.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0509999999999999e-06 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961784+0000", 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0870000000000007e-06 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961801+0000", 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6909e-05 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961809+0000", 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4139999999999997e-06 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961866+0000", 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7178e-05 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961956+0000", 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9728000000000006e-05 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.962033+0000", 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7099000000000002e-05 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.967020+0000", 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049875379999999997 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.967034+0000", 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4061e-05 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448511+0000", 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48147646599999999 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448573+0000", 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1878000000000006e-05 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.620 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448620+0000", 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7098000000000002e-05 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157297 6.8 6:13945e88:::benchmark_data_smithi187_102448_object57025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:27.976886+0000", 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 335.60083178399998, 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.47188472599999998, 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157297 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976886+0000", 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976886+0000", 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976888+0000", 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0080000000000001e-06 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976899+0000", 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0943e-05 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976916+0000", 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7176999999999998e-05 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976929+0000", 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860999999999999e-05 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.977006+0000", 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7248000000000003e-05 2024-09-02T10:27:03.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.977142+0000", 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136267 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.977282+0000", 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014029099999999999 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.983648+0000", 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0063655580000000003 2024-09-02T10:27:03.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.984252+0000", 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00060420199999999997 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448667+0000", 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46441474300000002 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448724+0000", 2024-09-02T10:27:03.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7235999999999999e-05 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448770+0000", 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6192000000000001e-05 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157311 6.8 6:152b8c45:::benchmark_data_smithi187_102448_object57039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:28.031794+0000", 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 335.54592377799997, 2024-09-02T10:27:03.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42239910000000003, 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157311 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031794+0000", 2024-09-02T10:27:03.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031794+0000", 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031796+0000", 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3539999999999998e-06 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031805+0000", 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8720000000000008e-06 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031827+0000", 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1678000000000001e-05 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031837+0000", 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0345e-05 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031909+0000", 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2026000000000005e-05 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.032041+0000", 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013162100000000001 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.032161+0000", 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012061999999999999 2024-09-02T10:27:03.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.036863+0000", 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047015859999999998 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.036882+0000", 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9409e-05 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454076+0000", 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.417193376 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454142+0000", 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6501000000000006e-05 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454193+0000", 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0711999999999998e-05 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.633 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157316 6.8 6:10c735aa:::benchmark_data_smithi187_102448_object57044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:28.069591+0000", 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 335.50812671300002, 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38474562000000001, 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.634 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157316 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069591+0000", 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069591+0000", 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069592+0000", 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069599+0000", 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7390000000000002e-06 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069615+0000", 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5988e-05 2024-09-02T10:27:03.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069624+0000", 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3990000000000004e-06 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069683+0000", 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9341e-05 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069819+0000", 2024-09-02T10:27:03.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135742 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069934+0000", 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115708 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.074383+0000", 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044484529999999998 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.074508+0000", 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012532299999999999 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454242+0000", 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37973358800000001 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454295+0000", 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3288000000000002e-05 2024-09-02T10:27:03.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454336+0000", 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1307000000000003e-05 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157317 6.8 6:19b233e5:::benchmark_data_smithi187_102448_object57045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:28.093866+0000", 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 335.48385167200001, 2024-09-02T10:27:03.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36060196900000002, 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157317 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093866+0000", 2024-09-02T10:27:03.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093866+0000", 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093868+0000", 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7540000000000001e-06 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093876+0000", 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3680000000000002e-06 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093895+0000", 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9585999999999998e-05 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093904+0000", 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0940000000000008e-06 2024-09-02T10:27:03.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093974+0000", 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0454e-05 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.094070+0000", 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5610000000000001e-05 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.094211+0000", 2024-09-02T10:27:03.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014128400000000001 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.098351+0000", 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041397889999999996 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.098481+0000", 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013057699999999999 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454389+0000", 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35590732400000002 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454433+0000", 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4869e-05 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454468+0000", 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4260000000000001e-05 2024-09-02T10:27:03.646 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157323 6.8 6:14bbaefe:::benchmark_data_smithi187_102448_object57051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:28.157047+0000", 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 335.42066994100003, 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29754121, 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157323 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157047+0000", 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157047+0000", 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157050+0000", 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1600000000000001e-06 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157058+0000", 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9449999999999993e-06 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157074+0000", 2024-09-02T10:27:03.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6833e-05 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157081+0000", 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.657e-06 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157128+0000", 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6421000000000002e-05 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157218+0000", 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0778000000000004e-05 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157299+0000", 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0711999999999996e-05 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.161246+0000", 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039468059999999998 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.161419+0000", 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017325399999999999 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454510+0000", 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29309048900000001 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454551+0000", 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1878000000000001e-05 2024-09-02T10:27:03.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454589+0000", 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7277000000000002e-05 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:191032 6.8 6:141909d5:::benchmark_data_smithi187_102448_object90760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:24.090741+0000", 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.48697618800003, 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.261754549, 2024-09-02T10:27:03.653 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191032 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090741+0000", 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090741+0000", 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090743+0000", 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.333e-06 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090748+0000", 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.857e-06 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090769+0000", 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0494000000000001e-05 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090776+0000", 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6329999999999999e-06 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090855+0000", 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9370999999999996e-05 2024-09-02T10:27:03.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090971+0000", 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115963 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.091080+0000", 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108837 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.096121+0000", 2024-09-02T10:27:03.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050410940000000003 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.099730+0000", 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036096359999999998 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.352348+0000", 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25261799699999998 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.352412+0000", 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3746999999999998e-05 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.352496+0000", 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3586999999999998e-05 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.659 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:193656 6.5 6:a5525aaf:::benchmark_data_smithi187_102448_object93384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:28.345700+0000", 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 275.23201787599999, 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26458540800000002, 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 193656 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345700+0000", 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345700+0000", 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345701+0000", 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2529999999999999e-06 2024-09-02T10:27:03.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345706+0000", 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.057e-06 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345723+0000", 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7232e-05 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345729+0000", 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0059999999999998e-06 2024-09-02T10:27:03.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345792+0000", 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2606000000000003e-05 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345882+0000", 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0539999999999997e-05 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345992+0000", 2024-09-02T10:27:03.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010925300000000001 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.350336+0000", 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043448200000000001 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.350360+0000", 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3213e-05 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.610142+0000", 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25978261800000002 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.610237+0000", 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5206999999999998e-05 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.610285+0000", 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7602999999999998e-05 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.665 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.666 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.666 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:200540 6.5 6:ac93dc06:::benchmark_data_smithi187_102448_object100268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-09-02T10:27:03.666 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:40.759488+0000", 2024-09-02T10:27:03.666 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 262.81822977600001, 2024-09-02T10:27:03.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.261497865, 2024-09-02T10:27:03.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:03.667 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 200540 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759488+0000", 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759488+0000", 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759489+0000", 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7570000000000001e-06 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759495+0000", 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1120000000000004e-06 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759509+0000", 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4606999999999999e-05 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759515+0000", 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3199999999999996e-06 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759558+0000", 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2401999999999998e-05 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759642+0000", 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4380000000000002e-05 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759716+0000", 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3795999999999996e-05 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764547+0000", 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048309219999999996 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764639+0000", 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1902000000000002e-05 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020875+0000", 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25623643800000001 2024-09-02T10:27:03.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020938+0000", 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2754000000000003e-05 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020986+0000", 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7475000000000002e-05 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.26524.0:200540 6.5 e626/533 ondisk, result = 0)", 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:40.764375+0000", 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 262.81334287599998, 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25661853200000001, 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764375+0000", 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764375+0000", 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764375+0000", 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7999999999999995e-07 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764380+0000", 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3089999999999997e-06 2024-09-02T10:27:03.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020777+0000", 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25639733999999997 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020786+0000", 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020857+0000", 2024-09-02T10:27:03.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1361999999999999e-05 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020868+0000", 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0528e-05 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020993+0000", 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012516 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27220.0:55500 7.8 e696/661 7:1ef2c8c5:::benchmark_data_smithi187_120712_object55499:head v 696'3479, mlcod=696'3476)", 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:07.741713+0000", 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.836004454, 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24015054999999999, 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.741713+0000", 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.741713+0000", 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.741714+0000", 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7400000000000002e-07 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.741789+0000", 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5412000000000002e-05 2024-09-02T10:27:03.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.980006+0000", 2024-09-02T10:27:03.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.238217016 2024-09-02T10:27:03.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.980014+0000", 2024-09-02T10:27:03.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1289999999999996e-06 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.980074+0000", 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9769999999999999e-05 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.980095+0000", 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0330999999999999e-05 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.981817+0000", 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001722339 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.981864+0000", 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6678999999999998e-05 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27220.0:55527 7.6 e696/694 7:6a5de1a3:::benchmark_data_smithi187_120712_object55526:head v 696'3508, mlcod=666'246)", 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:07.792451+0000", 2024-09-02T10:27:03.682 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 115.78526618399999, 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17637931500000001, 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.792451+0000", 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.792451+0000", 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.792452+0000", 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.13e-06 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.792507+0000", 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4234999999999998e-05 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.966578+0000", 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17407172800000001 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.966585+0000", 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1239999999999999e-06 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.966643+0000", 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7034000000000003e-05 2024-09-02T10:27:03.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.966669+0000", 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6239000000000001e-05 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.968793+0000", 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002123794 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.968831+0000", 2024-09-02T10:27:03.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8031000000000002e-05 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27220.0:99739 7.8 e722/661 7:121bd1d6:::benchmark_data_smithi187_120712_object15797:head v 722'6207, mlcod=722'6204)", 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:00.273723+0000", 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 63.303994158000002, 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22788001299999999, 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:03.687 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.273723+0000", 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.273723+0000", 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.273724+0000", 2024-09-02T10:27:03.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5899999999999995e-07 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.273728+0000", 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3329999999999998e-06 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.498479+0000", 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224750224 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.498486+0000", 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4549999999999998e-06 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.498551+0000", 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5413000000000005e-05 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.498573+0000", 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1880000000000001e-05 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.501563+0000", 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00298986 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.501603+0000", 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0089000000000002e-05 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.691 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(7.19 748/744 [PushOp(7:9a16227e:::benchmark_data_smithi187_120712_object59153:head, version: 699'3819, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:9a16227e:::benchmark_data_smithi187_120712_object59153:head@699'3819, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:57.449290+0000", 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 6.1284273230000004, 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18166254400000001, 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:57.449290+0000", 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:57.449290+0000", 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:57.449291+0000", 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0419999999999999e-06 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:57.449395+0000", 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103662 2024-09-02T10:27:03.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:57.629051+0000", 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17965656399999999 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:57.629073+0000", 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2010000000000001e-05 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:57.629191+0000", 2024-09-02T10:27:03.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117383 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:57.629205+0000", 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4425e-05 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:57.630953+0000", 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017474579999999999 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.695 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:27:03.696 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:27:03.696 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:27:03.720 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:03.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:03.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:03.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:03.848 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:03.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:03.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:03.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:03.973 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:03.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:04.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:04.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:04.104 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:04.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:04.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:04.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:04.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:04.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:04.378 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:04.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:04.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:04.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:04.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:27:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:04.508 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:04.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:04.610 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:04.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:04.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:04.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:04.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:04.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:04.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:04.794 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:04.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:04.903 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:04.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:05.005 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:05.046 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:05.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:05.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:05.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:05.171 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:04 smithi151 ceph-mon[16639]: pgmap v1711: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 12395/67530 objects degraded (18.355%); 15168/67530 objects misplaced (22.461%); 18 MiB/s, 293 objects/s recovering 2024-09-02T10:27:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 12395/67530 objects degraded (18.355%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:05.201 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:04 smithi187 ceph-mon[20072]: pgmap v1711: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 12395/67530 objects degraded (18.355%); 15168/67530 objects misplaced (22.461%); 18 MiB/s, 293 objects/s recovering 2024-09-02T10:27:05.201 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 12395/67530 objects degraded (18.355%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:05.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:05.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:05.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:04 smithi162 ceph-mon[20279]: pgmap v1711: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.4 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 12395/67530 objects degraded (18.355%); 15168/67530 objects misplaced (22.461%); 18 MiB/s, 293 objects/s recovering 2024-09-02T10:27:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 12395/67530 objects degraded (18.355%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:05.400 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:05.423 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:05.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:05.463 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:147320 7.f e749/706 7:f93c5a1e:::benchmark_data_smithi187_120712_object63378:head v 749'9344, mlcod=749'9341)", 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:05.456508+0000", 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0056813200000000001, 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0057084199999999996, 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.456508+0000", 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.456508+0000", 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.456509+0000", 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.302e-06 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.456518+0000", 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3399999999999998e-06 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.456560+0000", 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2799000000000001e-05 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.456569+0000", 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0059999999999998e-06 2024-09-02T10:27:05.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.456624+0000", 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4735999999999999e-05 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.456643+0000", 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8549000000000001e-05 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.467 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:147321 7.f e749/706 7:f426ec93:::benchmark_data_smithi187_120712_object63379:head v 749'9345, mlcod=749'9341)", 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:05.458015+0000", 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0041743279999999997, 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042342309999999998, 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.458015+0000", 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.458015+0000", 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.458016+0000", 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8599999999999997e-07 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.458033+0000", 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7384000000000002e-05 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.458067+0000", 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4075999999999997e-05 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.458074+0000", 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8730000000000001e-06 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.461450+0000", 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033759609999999998 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:05.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:05.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:05.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:05.461470+0000", 2024-09-02T10:27:05.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0017000000000002e-05 2024-09-02T10:27:05.471 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:05.471 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:05.471 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:05.471 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:05.471 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:27:05.471 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 2 2024-09-02T10:27:05.471 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:27:05.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:05.548 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:05.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:05.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:05.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:05.688 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:05.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:05.790 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:05.812 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:05.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:05.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:05.941 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:05.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:06.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:06.064 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:06.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:06.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:06.190 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:06.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:06.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:27:06.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:06.314 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:06.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:06.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:06.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:06.440 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:06.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:06.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:06.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:06.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:06.597 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:06.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:06.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:06.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:06.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:06.834 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:06.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:06.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:06.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:06.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:06 smithi187 ceph-mon[20072]: pgmap v1712: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 12395/65523 objects degraded (18.917%); 14062/65523 objects misplaced (21.461%); 21 MiB/s, 349 objects/s recovering 2024-09-02T10:27:07.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:07.104 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:27:07.104 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:27:07.104 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:27:07.104 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:27:07.104 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:27:07.105 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:07.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:07.106 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:06 smithi151 ceph-mon[16639]: pgmap v1712: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 12395/65523 objects degraded (18.917%); 14062/65523 objects misplaced (21.461%); 21 MiB/s, 349 objects/s recovering 2024-09-02T10:27:07.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:07.232 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:07.233 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:07.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:06 smithi162 ceph-mon[20279]: pgmap v1712: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 12395/65523 objects degraded (18.917%); 14062/65523 objects misplaced (21.461%); 21 MiB/s, 349 objects/s recovering 2024-09-02T10:27:07.357 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:07.358 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:07.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:07.483 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:07.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:07.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:07.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:07.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:07.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:07.738 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:07.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:07.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:07.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:27:07.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:07.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:07.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:07.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:08.000 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:08.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:08.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:08.127 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:08.139 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:08.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:08.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:08.241 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:08.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:08.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:08.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:08.400 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:08.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:08.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-09-02T10:27:08.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:08.523 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:08.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:08.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:08.649 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:08.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:08.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:08.663 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:27:08.663 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:27:08.663 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:27:08.663 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:27:08.663 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.663 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:194962 5.a 5:5b6b76d9:::benchmark_data_smithi187_89405_object91357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 545.70227710999995, 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22579173, 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194962 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.664 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.665 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:27:08.665 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.665 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.665 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.665 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958705+0000", 2024-09-02T10:27:08.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.666 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.666 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.666 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958706+0000", 2024-09-02T10:27:08.666 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.269e-06 2024-09-02T10:27:08.666 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958711+0000", 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7010000000000004e-06 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958723+0000", 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2641e-05 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.667 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958731+0000", 2024-09-02T10:27:08.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9980000000000003e-06 2024-09-02T10:27:08.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.668 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.668 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.668 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958775+0000", 2024-09-02T10:27:08.668 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3399999999999998e-05 2024-09-02T10:27:08.668 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958862+0000", 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7163999999999994e-05 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.958935+0000", 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2804000000000003e-05 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.669 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.962637+0000", 2024-09-02T10:27:08.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037025320000000001 2024-09-02T10:27:08.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.670 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.670 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.963124+0000", 2024-09-02T10:27:08.670 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00048696900000000002 2024-09-02T10:27:08.670 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.670 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184411+0000", 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.221286813 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184450+0000", 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9230999999999998e-05 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.671 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.184496+0000", 2024-09-02T10:27:08.672 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6208000000000002e-05 2024-09-02T10:27:08.672 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.672 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.672 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.672 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.672 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.672 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:194969 5.a 5:503ca7c3:::benchmark_data_smithi187_89405_object91364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e516)", 2024-09-02T10:27:08.672 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:27:08.673 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 545.69137997099995, 2024-09-02T10:27:08.673 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21939375699999999, 2024-09-02T10:27:08.673 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.673 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.673 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.673 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.673 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:27:08.673 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:27:08.673 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 194969 2024-09-02T10:27:08.673 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969602+0000", 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.674 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969603+0000", 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.068e-06 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969607+0000", 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3850000000000004e-06 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969617+0000", 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7750000000000004e-06 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.675 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969622+0000", 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2449999999999998e-06 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969678+0000", 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5784e-05 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.676 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969792+0000", 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113993 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.969919+0000", 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012692700000000001 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.975053+0000", 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051336569999999998 2024-09-02T10:27:08.677 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:02.975381+0000", 2024-09-02T10:27:08.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00032868799999999999 2024-09-02T10:27:08.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.678 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.678 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.678 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188892+0000", 2024-09-02T10:27:08.678 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21351113099999999 2024-09-02T10:27:08.678 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188951+0000", 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8777000000000003e-05 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:03.188996+0000", 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4326999999999998e-05 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.679 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207198 5.a 5:57bb5924:::benchmark_data_smithi187_89405_object103593:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 528.54470797500005, 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44270453799999998, 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:27:08.680 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207198 2024-09-02T10:27:08.681 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.681 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.681 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116274+0000", 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116275+0000", 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.393e-06 2024-09-02T10:27:08.682 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116281+0000", 2024-09-02T10:27:08.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.694e-06 2024-09-02T10:27:08.683 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.683 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.683 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.683 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116296+0000", 2024-09-02T10:27:08.683 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5234e-05 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116305+0000", 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5650000000000002e-06 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116360+0000", 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5164999999999998e-05 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.684 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116445+0000", 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5158e-05 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 10,11", 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116517+0000", 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2376e-05 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.121133+0000", 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0046159479999999999 2024-09-02T10:27:08.685 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.121221+0000", 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7831999999999995e-05 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558874+0000", 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43765253900000001 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558932+0000", 2024-09-02T10:27:08.686 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7908000000000002e-05 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558978+0000", 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6726e-05 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:207204 5.4 e525/431 5:2473bd20:::benchmark_data_smithi187_89405_object103599:head v 525'13184, mlcod=525'13181)", 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 528.53355301800002, 2024-09-02T10:27:08.687 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42712563199999998, 2024-09-02T10:27:08.688 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.688 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.688 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.688 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.688 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.688 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.688 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:27:08.688 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.688 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127429+0000", 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127430+0000", 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.251e-06 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.689 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127438+0000", 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9039999999999992e-06 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550537+0000", 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.42309940200000001 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550550+0000", 2024-09-02T10:27:08.690 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2262999999999999e-05 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550695+0000", 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000145845 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.550728+0000", 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2923000000000003e-05 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.691 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554509+0000", 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037808550000000001 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554554+0000", 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5188999999999997e-05 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.692 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:19698 6.1 6:8f6a7646:::benchmark_data_smithi187_102448_object19697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:53.693434+0000", 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 494.96754801700001, 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22523997900000001, 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 19698 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.693 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693434+0000", 2024-09-02T10:27:08.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.694 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.694 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.694 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.694 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693434+0000", 2024-09-02T10:27:08.694 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.695 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.695 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.695 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.695 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693436+0000", 2024-09-02T10:27:08.695 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7400000000000001e-06 2024-09-02T10:27:08.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.696 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.696 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.696 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693457+0000", 2024-09-02T10:27:08.696 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1824999999999999e-05 2024-09-02T10:27:08.696 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693471+0000", 2024-09-02T10:27:08.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3200999999999999e-05 2024-09-02T10:27:08.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.697 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.697 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.697 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693478+0000", 2024-09-02T10:27:08.697 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5820000000000004e-06 2024-09-02T10:27:08.697 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693538+0000", 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9824999999999997e-05 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693698+0000", 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015993299999999999 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.698 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.693861+0000", 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016271300000000001 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.694864+0000", 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010034600000000001 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.699 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.695808+0000", 2024-09-02T10:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00094396200000000001 2024-09-02T10:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.918543+0000", 2024-09-02T10:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22273512600000001 2024-09-02T10:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.700 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.918616+0000", 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2878000000000003e-05 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.918674+0000", 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7734e-05 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:19714 6.1 6:8a5fdd45:::benchmark_data_smithi187_102448_object19713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:53.701207+0000", 2024-09-02T10:27:08.701 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 494.95977433500002, 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21946518000000001, 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 19714 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.702 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701207+0000", 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701207+0000", 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.703 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701209+0000", 2024-09-02T10:27:08.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2839999999999999e-06 2024-09-02T10:27:08.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701234+0000", 2024-09-02T10:27:08.704 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5704999999999999e-05 2024-09-02T10:27:08.704 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.704 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.704 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.704 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701250+0000", 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5739999999999998e-05 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701259+0000", 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3310000000000004e-06 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701320+0000", 2024-09-02T10:27:08.705 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1755999999999996e-05 2024-09-02T10:27:08.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701450+0000", 2024-09-02T10:27:08.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000129794 2024-09-02T10:27:08.706 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.706 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.706 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:27:08.706 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.701592+0000", 2024-09-02T10:27:08.706 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014243400000000001 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.703694+0000", 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002101416 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.703718+0000", 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3785999999999999e-05 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.707 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.920572+0000", 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21685423000000001 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.920630+0000", 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8134000000000003e-05 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.920673+0000", 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2570000000000001e-05 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.708 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.709 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.709 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:19717 6.1 6:8971b800:::benchmark_data_smithi187_102448_object19716:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e543)", 2024-09-02T10:27:08.709 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:53.702495+0000", 2024-09-02T10:27:08.709 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 494.95848629900001, 2024-09-02T10:27:08.709 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.219151597, 2024-09-02T10:27:08.709 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.709 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.709 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.709 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 19717 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702495+0000", 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702495+0000", 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.710 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702497+0000", 2024-09-02T10:27:08.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.567e-06 2024-09-02T10:27:08.711 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.711 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.711 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.711 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702518+0000", 2024-09-02T10:27:08.711 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0837999999999999e-05 2024-09-02T10:27:08.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702531+0000", 2024-09-02T10:27:08.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3467e-05 2024-09-02T10:27:08.712 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.712 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.712 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.712 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702539+0000", 2024-09-02T10:27:08.712 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7419999999999997e-06 2024-09-02T10:27:08.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702609+0000", 2024-09-02T10:27:08.713 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9890999999999999e-05 2024-09-02T10:27:08.713 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.713 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.713 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.713 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702775+0000", 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000165575 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.702939+0000", 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016468799999999999 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.703801+0000", 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086205600000000004 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.714 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.704004+0000", 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000203177 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.921562+0000", 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21755740500000001 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.921611+0000", 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9421e-05 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.715 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:53.921647+0000", 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5769999999999998e-05 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:100226 6.1 6:80d0f12d:::benchmark_data_smithi187_102448_object100225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:20:10.905652+0000", 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 417.75532933699998, 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63928361700000003, 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.716 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 100226 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905652+0000", 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.717 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905652+0000", 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905654+0000", 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.593e-06 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905727+0000", 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3153999999999998e-05 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.718 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905744+0000", 2024-09-02T10:27:08.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6371e-05 2024-09-02T10:27:08.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.719 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.719 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.719 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905753+0000", 2024-09-02T10:27:08.719 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2299999999999997e-06 2024-09-02T10:27:08.719 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905816+0000", 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3464000000000001e-05 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.905925+0000", 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010829599999999999 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.720 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906047+0000", 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012230699999999999 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.908790+0000", 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027432860000000002 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.909351+0000", 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00056067900000000002 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.721 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544856+0000", 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63550532299999996 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544894+0000", 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8143999999999999e-05 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.722 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544936+0000", 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1770000000000002e-05 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:137822 6.1 6:8813a8dd:::benchmark_data_smithi187_102448_object37550:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:20:56.005408+0000", 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 372.65557385599999, 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25302631599999997, 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.723 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 137822 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005408+0000", 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.724 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005408+0000", 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005409+0000", 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4890000000000001e-06 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005416+0000", 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2380000000000002e-06 2024-09-02T10:27:08.725 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005430+0000", 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4027e-05 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005436+0000", 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2750000000000001e-06 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.726 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005482+0000", 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6344999999999998e-05 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005574+0000", 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1984000000000006e-05 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.005649+0000", 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4694e-05 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.727 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.009865+0000", 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004215964 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.010096+0000", 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00023086800000000001 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.728 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.258316+0000", 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.248219841 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.258372+0000", 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6681000000000002e-05 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:56.258434+0000", 2024-09-02T10:27:08.729 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1909999999999995e-05 2024-09-02T10:27:08.730 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.731 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.731 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.731 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.731 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.731 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.26524.0:191080 6.1 6:83094aee:::benchmark_data_smithi187_102448_object90808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-09-02T10:27:08.731 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:22:24.184440+0000", 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 284.47654140499998, 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.210722517, 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.26524", 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 191080 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.732 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184440+0000", 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184440+0000", 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184442+0000", 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3289999999999999e-06 2024-09-02T10:27:08.733 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184447+0000", 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.356e-06 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184460+0000", 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3213e-05 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.734 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184466+0000", 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9320000000000001e-06 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184523+0000", 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6706999999999998e-05 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184654+0000", 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000131211 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.735 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.184794+0000", 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013967999999999999 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.188749+0000", 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0039552550000000004 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.189060+0000", 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00031115299999999999 2024-09-02T10:27:08.736 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.395061+0000", 2024-09-02T10:27:08.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20600111700000001 2024-09-02T10:27:08.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.737 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.737 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.737 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.395116+0000", 2024-09-02T10:27:08.737 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4283999999999997e-05 2024-09-02T10:27:08.737 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:22:24.395163+0000", 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7280000000000001e-05 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:23891 7.f 7:fd726100:::benchmark_data_smithi187_120712_object23890:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:32.086879+0000", 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 156.574102788, 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.219302261, 2024-09-02T10:27:08.738 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.739 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.739 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.739 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.739 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:08.739 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:08.739 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 23891 2024-09-02T10:27:08.739 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.739 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.739 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.739 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086879+0000", 2024-09-02T10:27:08.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.740 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.740 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086879+0000", 2024-09-02T10:27:08.740 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.740 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.740 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086881+0000", 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.069e-06 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086915+0000", 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3998000000000002e-05 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086944+0000", 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9300000000000001e-05 2024-09-02T10:27:08.741 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.086955+0000", 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0796e-05 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.087024+0000", 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8351999999999999e-05 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.742 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.087186+0000", 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016218 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.087364+0000", 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017785500000000001 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.743 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.089462+0000", 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020980199999999999 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.097470+0000", 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0080084630000000004 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.305360+0000", 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20789026099999999 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.744 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.305438+0000", 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8193000000000002e-05 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.306181+0000", 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00074277399999999997 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:23904 7.f 7:f7b9f367:::benchmark_data_smithi187_120712_object23903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:32.095666+0000", 2024-09-02T10:27:08.745 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 156.56531625900001, 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21120496499999999, 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 23904 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095666+0000", 2024-09-02T10:27:08.746 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095666+0000", 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095668+0000", 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1239999999999999e-06 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.747 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095692+0000", 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4825e-05 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095713+0000", 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0588e-05 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095721+0000", 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2689999999999999e-06 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.748 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.095786+0000", 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4481999999999997e-05 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.096004+0000", 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00021850400000000001 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.096181+0000", 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000176305 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.749 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.097203+0000", 2024-09-02T10:27:08.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010226409999999999 2024-09-02T10:27:08.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.750 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.750 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.750 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.101678+0000", 2024-09-02T10:27:08.750 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0044750529999999997 2024-09-02T10:27:08.750 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.306240+0000", 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.204561403 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.306305+0000", 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5547000000000001e-05 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.306870+0000", 2024-09-02T10:27:08.751 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00056522400000000002 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:23909 7.f 7:f5ee81a7:::benchmark_data_smithi187_120712_object23908:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:32.099943+0000", 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 156.561038688, 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20944562999999999, 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.752 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 23909 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.099943+0000", 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.099943+0000", 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.753 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.099945+0000", 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2170000000000001e-06 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.099973+0000", 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7277e-05 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.099993+0000", 2024-09-02T10:27:08.754 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0720000000000002e-05 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.100001+0000", 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0209999999999999e-06 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.100075+0000", 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3533000000000002e-05 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.100241+0000", 2024-09-02T10:27:08.755 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016628199999999999 2024-09-02T10:27:08.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 9,11", 2024-09-02T10:27:08.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.100369+0000", 2024-09-02T10:27:08.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012790100000000001 2024-09-02T10:27:08.756 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.756 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.756 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.756 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.101352+0000", 2024-09-02T10:27:08.756 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00098319800000000001 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.114398+0000", 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013045929 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.308648+0000", 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19424997299999999 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.757 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.308707+0000", 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8406000000000003e-05 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:32.309389+0000", 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000682173 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55464 7.f 7:f848bf24:::benchmark_data_smithi187_120712_object55463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.558211+0000", 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 121.102770631, 2024-09-02T10:27:08.758 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.219308693, 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55464 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.759 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558211+0000", 2024-09-02T10:27:08.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558211+0000", 2024-09-02T10:27:08.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558213+0000", 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5579999999999999e-06 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558264+0000", 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0971999999999999e-05 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558276+0000", 2024-09-02T10:27:08.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2336e-05 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558283+0000", 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8410000000000002e-06 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.564067+0000", 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0057837560000000001 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.564243+0000", 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017600800000000001 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,9,11", 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.564442+0000", 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000199469 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.566944+0000", 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002502116 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.567038+0000", 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3987999999999995e-05 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.593916+0000", 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.026877429000000001 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.777338+0000", 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18342222399999999 2024-09-02T10:27:08.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.777398+0000", 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0353000000000003e-05 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.777520+0000", 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121643 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.765 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.766 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.766 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:99761 7.f e722/706 7:f08c5da3:::benchmark_data_smithi187_120712_object15819:head v 722'6376, mlcod=722'6374)", 2024-09-02T10:27:08.766 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:00.298147+0000", 2024-09-02T10:27:08.767 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 68.362834926999994, 2024-09-02T10:27:08.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.171438336, 2024-09-02T10:27:08.767 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.767 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.767 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.767 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.298147+0000", 2024-09-02T10:27:08.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.298147+0000", 2024-09-02T10:27:08.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.298148+0000", 2024-09-02T10:27:08.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2899999999999999e-06 2024-09-02T10:27:08.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.298156+0000", 2024-09-02T10:27:08.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3699999999999997e-06 2024-09-02T10:27:08.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.466615+0000", 2024-09-02T10:27:08.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16845906399999999 2024-09-02T10:27:08.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.466621+0000", 2024-09-02T10:27:08.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.426e-06 2024-09-02T10:27:08.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.466670+0000", 2024-09-02T10:27:08.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9032000000000001e-05 2024-09-02T10:27:08.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.774 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.466691+0000", 2024-09-02T10:27:08.774 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0965e-05 2024-09-02T10:27:08.774 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.774 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.774 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.469549+0000", 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0028581209999999999 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.469585+0000", 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6068000000000001e-05 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.775 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:120770 7.7 e732/661 7:ef170c21:::benchmark_data_smithi187_120712_object36828:head v 732'7502, mlcod=732'7499)", 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:24.150568+0000", 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 44.510414130999997, 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15003277500000001, 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:24.150568+0000", 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:24.150568+0000", 2024-09-02T10:27:08.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.777 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:24.150569+0000", 2024-09-02T10:27:08.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9299999999999996e-07 2024-09-02T10:27:08.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:24.150576+0000", 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0260000000000002e-06 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:24.297376+0000", 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14680041499999999 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:24.297384+0000", 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0539999999999995e-06 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:24.297434+0000", 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9975000000000002e-05 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:24.297449+0000", 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4683e-05 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:24.300512+0000", 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0030634740000000001 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:24.300600+0000", 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8255000000000001e-05 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.780 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:136911 7.16 7.a2712c76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:43.756684+0000", 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 24.904297563, 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.473282537, 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 136911 2024-09-02T10:27:08.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756684+0000", 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756684+0000", 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756686+0000", 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8729999999999999e-06 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756692+0000", 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1430000000000003e-06 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756705+0000", 2024-09-02T10:27:08.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2863e-05 2024-09-02T10:27:08.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756712+0000", 2024-09-02T10:27:08.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7680000000000003e-06 2024-09-02T10:27:08.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756763+0000", 2024-09-02T10:27:08.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1360000000000003e-05 2024-09-02T10:27:08.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:27:08.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756772+0000", 2024-09-02T10:27:08.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6119999999999999e-06 2024-09-02T10:27:08.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.229956+0000", 2024-09-02T10:27:08.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.473184084 2024-09-02T10:27:08.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.229967+0000", 2024-09-02T10:27:08.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0834e-05 2024-09-02T10:27:08.788 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.788 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.789 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.789 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:143885 7.7 e749/741 7:e3d45447:::benchmark_data_smithi187_120712_object59943:head v 749'8672, mlcod=749'8670)", 2024-09-02T10:27:08.789 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:59.060553+0000", 2024-09-02T10:27:08.789 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 9.6004290510000008, 2024-09-02T10:27:08.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.155215835, 2024-09-02T10:27:08.789 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.789 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.060553+0000", 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.060553+0000", 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.060554+0000", 2024-09-02T10:27:08.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.212e-06 2024-09-02T10:27:08.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.060564+0000", 2024-09-02T10:27:08.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0178e-05 2024-09-02T10:27:08.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.213603+0000", 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.15303875 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.213609+0000", 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4289999999999998e-06 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.213658+0000", 2024-09-02T10:27:08.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8714999999999997e-05 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.213672+0000", 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3481e-05 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.215728+0000", 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0020568460000000002 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.215769+0000", 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0224e-05 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:143968 7.7 e749/741 7:e02b8863:::benchmark_data_smithi187_120712_object60026:head v 749'8673, mlcod=749'8670)", 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:59.213581+0000", 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 9.4474004570000005, 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20551592099999999, 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.794 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.213581+0000", 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.213581+0000", 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.213582+0000", 2024-09-02T10:27:08.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8700000000000004e-07 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.213588+0000", 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3079999999999998e-06 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.416848+0000", 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.203259987 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.416863+0000", 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5637e-05 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.416917+0000", 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3838999999999997e-05 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.416938+0000", 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0707999999999999e-05 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.419050+0000", 2024-09-02T10:27:08.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021127759999999998 2024-09-02T10:27:08.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.419097+0000", 2024-09-02T10:27:08.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6779000000000001e-05 2024-09-02T10:27:08.799 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.799 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.799 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.800 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:144199 7.7 e749/741 7:e22c0a38:::benchmark_data_smithi187_120712_object60257:head v 749'8678, mlcod=749'8675)", 2024-09-02T10:27:08.800 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:59.587094+0000", 2024-09-02T10:27:08.800 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 9.0738875990000007, 2024-09-02T10:27:08.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.249879298, 2024-09-02T10:27:08.801 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:08.801 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:08.801 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:08.801 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:08.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:08.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.587094+0000", 2024-09-02T10:27:08.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:08.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.587094+0000", 2024-09-02T10:27:08.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:08.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:08.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.587095+0000", 2024-09-02T10:27:08.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2999999999999999e-07 2024-09-02T10:27:08.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:08.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.587102+0000", 2024-09-02T10:27:08.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5119999999999997e-06 2024-09-02T10:27:08.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:08.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.834486+0000", 2024-09-02T10:27:08.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.247384403 2024-09-02T10:27:08.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:08.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.834490+0000", 2024-09-02T10:27:08.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5210000000000004e-06 2024-09-02T10:27:08.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:08.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.834533+0000", 2024-09-02T10:27:08.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2959e-05 2024-09-02T10:27:08.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:08.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.834547+0000", 2024-09-02T10:27:08.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3328e-05 2024-09-02T10:27:08.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.836929+0000", 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002382103 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.836973+0000", 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4641999999999997e-05 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.809 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:08.810 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:08.810 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:27:08.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:08.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:08.834 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:08.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:08.935 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:08.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:08.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:09.062 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:09.087 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:09.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:09.088 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:08 smithi151 ceph-mon[16639]: pgmap v1713: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 12219/62352 objects degraded (19.597%); 13372/62352 objects misplaced (21.446%); 18 MiB/s, 294 objects/s recovering 2024-09-02T10:27:09.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:09.218 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:09.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:09.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:08 smithi187 ceph-mon[20072]: pgmap v1713: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 12219/62352 objects degraded (19.597%); 13372/62352 objects misplaced (21.446%); 18 MiB/s, 294 objects/s recovering 2024-09-02T10:27:09.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:08 smithi162 ceph-mon[20279]: pgmap v1713: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.3 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 12219/62352 objects degraded (19.597%); 13372/62352 objects misplaced (21.446%); 18 MiB/s, 294 objects/s recovering 2024-09-02T10:27:09.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:09.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:09.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:09.490 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:09.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:09.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:09.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:09.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:09.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:27:09.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:09.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:09.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:09.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:09.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:09.886 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:09.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:09.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:10.021 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:10.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:10.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:10.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:10.181 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:10.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 12219/62352 objects degraded (19.597%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 12219/62352 objects degraded (19.597%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:10.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:10.306 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:10.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:10.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 12219/62352 objects degraded (19.597%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:10.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:10.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:10.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:10.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:10.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:10.565 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:27:10.565 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:27:10.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:150371 7.0 e749/741 7:01280226:::benchmark_data_smithi187_120712_object66429:head v 749'8950, mlcod=749'8948)", 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:10.562050+0000", 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.002332462, 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023584019999999999, 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562050+0000", 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.566 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562050+0000", 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562051+0000", 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3570000000000001e-06 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562062+0000", 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1046999999999999e-05 2024-09-02T10:27:10.567 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562087+0000", 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4256e-05 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562098+0000", 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1038e-05 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562169+0000", 2024-09-02T10:27:10.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1223999999999994e-05 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562189+0000", 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0531e-05 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:150372 7.11 e749/744 7:8f6bf651:::benchmark_data_smithi187_120712_object66430:head v 749'8912, mlcod=712'4582)", 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:10.562068+0000", 2024-09-02T10:27:10.569 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.0023140869999999998, 2024-09-02T10:27:10.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023959250000000001, 2024-09-02T10:27:10.570 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:10.570 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:10.570 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:10.570 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:10.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:10.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562068+0000", 2024-09-02T10:27:10.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:10.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562068+0000", 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562069+0000", 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3300000000000001e-07 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562072+0000", 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.196e-06 2024-09-02T10:27:10.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562127+0000", 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5025999999999998e-05 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562132+0000", 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7160000000000002e-06 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562696+0000", 2024-09-02T10:27:10.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00056426399999999998 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:10.562715+0000", 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8950999999999999e-05 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 2 2024-09-02T10:27:10.573 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:27:10.575 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:10.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:10.677 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:10.703 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:10.704 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:10.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:10.829 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:10.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:10.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:10.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:10.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:10.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:11.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:11.094 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:11.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:11.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:11.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:11.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:10 smithi187 ceph-mon[20072]: pgmap v1714: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 11866/57432 objects degraded (20.661%); 11867/57432 objects misplaced (20.663%); 19 MiB/s, 348 objects/s recovering 2024-09-02T10:27:11.334 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:11.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:10 smithi162 ceph-mon[20279]: pgmap v1714: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 11866/57432 objects degraded (20.661%); 11867/57432 objects misplaced (20.663%); 19 MiB/s, 348 objects/s recovering 2024-09-02T10:27:11.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:11.373 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:11.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:11.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:27:11.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:10 smithi151 ceph-mon[16639]: pgmap v1714: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 11866/57432 objects degraded (20.661%); 11867/57432 objects misplaced (20.663%); 19 MiB/s, 348 objects/s recovering 2024-09-02T10:27:11.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:11.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:11.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:11.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:11.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:11.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:11.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:11.712 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:11.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:11.781 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:11.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:11.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:11.909 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:11.910 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:12.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:12.045 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:12.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:12.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:12.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:12.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:12.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:12.232 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:27:12.232 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:27:12.232 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:27:12.233 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:27:12.233 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:27:12.276 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:12.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:12.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:12.418 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:12.440 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:12.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:12.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:12.542 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:12.564 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:12.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:12.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:12.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:12.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:12.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:12.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:12.822 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:12.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:12.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:27:12.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:12.948 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:12.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:13.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:13.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:13.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:13.173 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:13.197 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:13.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:13.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:13.219 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:12 smithi187 ceph-mon[20072]: pgmap v1715: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 462 op/s; 11866/57147 objects degraded (20.764%); 11867/57147 objects misplaced (20.766%); 13 MiB/s, 250 objects/s recovering 2024-09-02T10:27:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:27:13.299 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:13.322 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:13.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:13.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:12 smithi162 ceph-mon[20279]: pgmap v1715: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 462 op/s; 11866/57147 objects degraded (20.764%); 11867/57147 objects misplaced (20.766%); 13 MiB/s, 250 objects/s recovering 2024-09-02T10:27:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:27:13.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:12 smithi151 ceph-mon[16639]: pgmap v1715: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 462 op/s; 11866/57147 objects degraded (20.764%); 11867/57147 objects misplaced (20.766%); 13 MiB/s, 250 objects/s recovering 2024-09-02T10:27:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:13 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:27:13.451 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:13.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:13.553 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:13.580 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:13.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:13.581 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:13.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:13.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:13.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:13.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:13.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:13.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:13.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:13.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:13.963 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:13.988 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:13.989 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:14.045 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:27:14.045 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:27:14.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:27:14.045 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:27:14.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:205325 5.e 5:726c3a45:::benchmark_data_smithi187_89405_object101720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e524)", 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:17.617930+0000", 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 536.42442702300002, 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.38179982800000001, 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 205325 2024-09-02T10:27:14.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617930+0000", 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617930+0000", 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617931+0000", 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1999999999999998e-07 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617936+0000", 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3499999999999999e-06 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617946+0000", 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0223e-05 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617951+0000", 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3159999999999996e-06 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.617992+0000", 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0435000000000002e-05 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.618066+0000", 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4764000000000004e-05 2024-09-02T10:27:14.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,8", 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.618133+0000", 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6779999999999994e-05 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.622877+0000", 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047439980000000001 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.623239+0000", 2024-09-02T10:27:14.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00036236100000000001 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.999558+0000", 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.37631872100000002 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.999642+0000", 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4165000000000005e-05 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:17.999730+0000", 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7794999999999995e-05 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:100226 6.1 e572/533 6:80d0f12d:::benchmark_data_smithi187_102448_object100225:head v 572'6190, mlcod=572'6187)", 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 423.136235726, 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63855390000000001, 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.052 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906123+0000", 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.269e-06 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906147+0000", 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3620000000000001e-05 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.541864+0000", 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.635717054 2024-09-02T10:27:14.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.541879+0000", 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5546e-05 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.542040+0000", 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000161022 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.542059+0000", 2024-09-02T10:27:14.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8697000000000001e-05 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544638+0000", 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025792850000000002 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544676+0000", 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7407000000000003e-05 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.056 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:157286 6.5 e598/533 6:aeee93d8:::benchmark_data_smithi187_102448_object57014:head v 598'9853, mlcod=598'9850)", 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 346.09234282, 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29959694599999998, 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:27:14.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2900000000000002e-07 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950022+0000", 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8739999999999997e-06 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245512+0000", 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29548945300000001 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245524+0000", 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2347000000000001e-05 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245666+0000", 2024-09-02T10:27:14.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014145800000000001 2024-09-02T10:27:14.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245702+0000", 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6853000000000001e-05 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.249576+0000", 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003873385 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.249611+0000", 2024-09-02T10:27:14.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5646999999999999e-05 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:43285 7.a 7:505fa4fc:::benchmark_data_smithi187_120712_object43284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:51.571815+0000", 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 142.47054269200001, 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21327285000000001, 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:14.062 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 43285 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571815+0000", 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571815+0000", 2024-09-02T10:27:14.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571816+0000", 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8109999999999999e-06 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571848+0000", 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1999999999999999e-05 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571863+0000", 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4976e-05 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571871+0000", 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6930000000000007e-06 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571922+0000", 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0884999999999998e-05 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.572019+0000", 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6482000000000002e-05 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.572104+0000", 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5686000000000006e-05 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573298+0000", 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001193646 2024-09-02T10:27:14.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.574986+0000", 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001687893 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.784959+0000", 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20997289899999999 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.785035+0000", 2024-09-02T10:27:14.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6780999999999996e-05 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.785088+0000", 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2098000000000002e-05 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:43288 7.a 7:52f650c0:::benchmark_data_smithi187_120712_object43287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:51.573470+0000", 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 142.46888784699999, 2024-09-02T10:27:14.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.215809268, 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 43288 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573470+0000", 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573470+0000", 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573470+0000", 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6000000000000003e-07 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573490+0000", 2024-09-02T10:27:14.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9558000000000001e-05 2024-09-02T10:27:14.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.071 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.071 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573500+0000", 2024-09-02T10:27:14.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.696e-06 2024-09-02T10:27:14.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.071 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.071 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573504+0000", 2024-09-02T10:27:14.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.87e-06 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573544+0000", 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9749999999999997e-05 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573653+0000", 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108455 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:27:14.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573733+0000", 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0520999999999997e-05 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.575073+0000", 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013397000000000001 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.575346+0000", 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000272716 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789047+0000", 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21370119100000001 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789231+0000", 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018440700000000001 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789279+0000", 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7644e-05 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.074 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:43296 7.a 7:53509a3f:::benchmark_data_smithi187_120712_object43295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:51.582447+0000", 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 142.459910009, 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20711470400000001, 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:14.075 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 43296 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582447+0000", 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582447+0000", 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582449+0000", 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2309999999999999e-06 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582470+0000", 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1141e-05 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582481+0000", 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1581e-05 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582487+0000", 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4790000000000004e-06 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582531+0000", 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4206999999999999e-05 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582632+0000", 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100882 2024-09-02T10:27:14.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582720+0000", 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7806999999999995e-05 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.584239+0000", 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001519734 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.584541+0000", 2024-09-02T10:27:14.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00030194999999999999 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789368+0000", 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.204827017 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789525+0000", 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015684 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.080 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789562+0000", 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6835000000000002e-05 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:45631 7.a 7:579b01e3:::benchmark_data_smithi187_120712_object45630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:53.384584+0000", 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 140.65777369099999, 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25830757599999998, 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.081 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 45631 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384584+0000", 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384584+0000", 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384586+0000", 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4090000000000001e-06 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384615+0000", 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9200000000000002e-05 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384639+0000", 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3901e-05 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384651+0000", 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1793999999999999e-05 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384726+0000", 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4813000000000004e-05 2024-09-02T10:27:14.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384857+0000", 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013135800000000001 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384960+0000", 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000102682 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.386554+0000", 2024-09-02T10:27:14.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001593742 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.386810+0000", 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025653200000000002 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.642810+0000", 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25600023 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.642853+0000", 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2657999999999997e-05 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.642891+0000", 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8257000000000003e-05 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55524 7.a 7:5888f7f6:::benchmark_data_smithi187_120712_object55523:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.784823+0000", 2024-09-02T10:27:14.087 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 126.257534158, 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25595624, 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55524 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784823+0000", 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784823+0000", 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784825+0000", 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.289e-06 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784866+0000", 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0327000000000002e-05 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784884+0000", 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7697999999999999e-05 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784893+0000", 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5510000000000005e-06 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784968+0000", 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4975999999999995e-05 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.785119+0000", 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000151404 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.785270+0000", 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015045799999999999 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.787505+0000", 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022353439999999998 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.787795+0000", 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028950500000000001 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040702+0000", 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25290737699999999 2024-09-02T10:27:14.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040745+0000", 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.337e-05 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040779+0000", 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3940999999999999e-05 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:55525 7.5 e696/690 7:a5f161ff:::benchmark_data_smithi187_120712_object55524:head v 696'3504, mlcod=696'3501)", 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.788514+0000", 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 126.253843632, 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.2522201, 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788514+0000", 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788514+0000", 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788515+0000", 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9299999999999996e-07 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788550+0000", 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5129000000000002e-05 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788586+0000", 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6294000000000002e-05 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788593+0000", 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9299999999999997e-06 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788652+0000", 2024-09-02T10:27:14.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9002999999999997e-05 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788676+0000", 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4290999999999998e-05 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040675+0000", 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25199830000000001 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040734+0000", 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9259999999999998e-05 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:55524 7.a e696/661 ondisk, result = 0)", 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.788585+0000", 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 126.253772645, 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25220463100000001, 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.098 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788585+0000", 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788585+0000", 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788586+0000", 2024-09-02T10:27:14.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1399999999999996e-07 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788593+0000", 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5510000000000002e-06 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040578+0000", 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25198481900000003 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040594+0000", 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6563e-05 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040692+0000", 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7800999999999994e-05 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040698+0000", 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8390000000000003e-06 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040789+0000", 2024-09-02T10:27:14.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:14.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1243999999999996e-05 2024-09-02T10:27:14.103 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.103 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.103 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55529 7.a 7:5c0a98d2:::benchmark_data_smithi187_120712_object55528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.789084+0000", 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 126.25327346900001, 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25619412899999999, 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55529 2024-09-02T10:27:14.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789084+0000", 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789084+0000", 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789086+0000", 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3910000000000001e-06 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789109+0000", 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2734e-05 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789121+0000", 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2435e-05 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789129+0000", 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.52e-06 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789235+0000", 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010628000000000001 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789361+0000", 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000125351 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789484+0000", 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012378000000000001 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040657+0000", 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25117236700000001 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040831+0000", 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017463 2024-09-02T10:27:14.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045180+0000", 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043483080000000004 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045232+0000", 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2074000000000003e-05 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045278+0000", 2024-09-02T10:27:14.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6258999999999999e-05 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:55530 7.4 e696/690 7:2b921397:::benchmark_data_smithi187_120712_object55529:head v 696'3599, mlcod=696'3597)", 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.789951+0000", 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 126.252406824, 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.2507779, 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789951+0000", 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789951+0000", 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789954+0000", 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1530000000000001e-06 2024-09-02T10:27:14.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790032+0000", 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8168000000000001e-05 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790048+0000", 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5862000000000002e-05 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790054+0000", 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.584e-06 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790120+0000", 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5668999999999998e-05 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790141+0000", 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0818999999999998e-05 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040685+0000", 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25054374800000001 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040728+0000", 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3896999999999997e-05 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.114 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:55529 7.a e696/661 ondisk, result = 0)", 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.791795+0000", 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 126.250562131, 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25349172599999997, 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791795+0000", 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791795+0000", 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791796+0000", 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9599999999999999e-07 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791803+0000", 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9299999999999997e-06 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040780+0000", 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.248977435 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040785+0000", 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9910000000000002e-06 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045163+0000", 2024-09-02T10:27:14.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043775949999999998 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045171+0000", 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2109999999999998e-06 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045287+0000", 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115868 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:55529 7.a e696/661 ondisk, result = 0)", 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.793833+0000", 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 126.248524479, 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24701554000000001, 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793833+0000", 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793833+0000", 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793834+0000", 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2900000000000003e-07 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793838+0000", 2024-09-02T10:27:14.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8720000000000001e-06 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040586+0000", 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.246747148 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040598+0000", 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1943999999999999e-05 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040817+0000", 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00021921700000000001 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040825+0000", 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5939999999999994e-06 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040848+0000", 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3036000000000001e-05 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.122 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55539 7.a 7:5096ce06:::benchmark_data_smithi187_120712_object55538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.802709+0000", 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 126.239648622, 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24498555799999999, 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:14.123 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55539 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802709+0000", 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802709+0000", 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802711+0000", 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5840000000000002e-06 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802737+0000", 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5948e-05 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040797+0000", 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23805955500000001 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040805+0000", 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9629999999999998e-06 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040878+0000", 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.305e-05 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.041029+0000", 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000151337 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.041152+0000", 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012326200000000001 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045348+0000", 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041960260000000003 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045394+0000", 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.587e-05 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.047524+0000", 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021297120000000002 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.047622+0000", 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8387999999999994e-05 2024-09-02T10:27:14.128 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.047694+0000", 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1863e-05 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:71694 7.a 7:589b7e94:::benchmark_data_smithi187_120712_object71693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e711)", 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:28.310151+0000", 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 105.732206022, 2024-09-02T10:27:14.129 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31770000700000001, 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 71694 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310151+0000", 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.130 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310151+0000", 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310153+0000", 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5910000000000001e-06 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.131 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310173+0000", 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.052e-05 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310187+0000", 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.397e-05 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310194+0000", 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1580000000000002e-06 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.132 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310240+0000", 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6564999999999999e-05 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310346+0000", 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010611999999999999 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310438+0000", 2024-09-02T10:27:14.133 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2083999999999995e-05 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312967+0000", 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025282730000000002 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.313222+0000", 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025570999999999998 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.134 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627778+0000", 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31455519399999998 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627814+0000", 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6374000000000001e-05 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627851+0000", 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7447999999999998e-05 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.135 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:71694 7.a e711/661 ondisk, result = 0)", 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:28.312737+0000", 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 105.729620001, 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.315119227, 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.136 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312737+0000", 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312737+0000", 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312738+0000", 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1500000000000003e-07 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.137 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312742+0000", 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0389999999999998e-06 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627726+0000", 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31498364600000001 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627732+0000", 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0789999999999997e-06 2024-09-02T10:27:14.138 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627771+0000", 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8642e-05 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627774+0000", 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9060000000000004e-06 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.139 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627857+0000", 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2100000000000003e-05 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:99751 7.e e722/661 7:70408914:::benchmark_data_smithi187_120712_object15809:head v 722'6054, mlcod=722'6050)", 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:00.293617+0000", 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 73.748740506000004, 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26051671900000001, 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.140 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293617+0000", 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293617+0000", 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293618+0000", 2024-09-02T10:27:14.141 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5099999999999999e-07 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293623+0000", 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8810000000000001e-06 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551233+0000", 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.257609745 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.142 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551238+0000", 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9609999999999996e-06 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551289+0000", 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0492999999999999e-05 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551309+0000", 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0426999999999999e-05 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.143 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.554093+0000", 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027834660000000001 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.554134+0000", 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0995000000000003e-05 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.144 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:151430 7.a 7:557e0f20:::benchmark_data_smithi187_120712_object67488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:12.399799+0000", 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 1.6425580200000001, 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22159694199999999, 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 151430 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.145 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399799+0000", 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399799+0000", 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399801+0000", 2024-09-02T10:27:14.146 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6750000000000001e-06 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399807+0000", 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3840000000000002e-06 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399824+0000", 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6257999999999999e-05 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.147 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399831+0000", 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2459999999999997e-06 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399892+0000", 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1234000000000004e-05 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399987+0000", 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4999000000000003e-05 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.148 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.400061+0000", 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3603000000000007e-05 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.403732+0000", 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036712530000000002 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.403903+0000", 2024-09-02T10:27:14.149 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000170645 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621303+0000", 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21740003899999999 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621359+0000", 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5983000000000003e-05 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.150 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621396+0000", 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7623000000000002e-05 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:151430 7.a e749/661 ondisk, result = 0)", 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:12.404953+0000", 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 1.6374047790000001, 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.216450315, 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:14.151 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.404953+0000", 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.404953+0000", 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.404953+0000", 2024-09-02T10:27:14.152 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6700000000000003e-07 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.404958+0000", 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6249999999999998e-06 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621131+0000", 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.216173486 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:14.153 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621144+0000", 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.254e-05 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621259+0000", 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011462499999999999 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621296+0000", 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7696e-05 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:14.154 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:14.155 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621403+0000", 2024-09-02T10:27:14.155 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010667600000000001 2024-09-02T10:27:14.155 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.155 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.155 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.155 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:14.155 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:14.155 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:27:14.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:14.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:14.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:14.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:14.287 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:14.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:14.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:14.412 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:14.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:14.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:14.539 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:14.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:14.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:14.674 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:14.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:14.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:14.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:14.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:14.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:14.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:14.928 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:15.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:15.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:15.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:15.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:15.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:15.184 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:15.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:14 smithi187 ceph-mon[20072]: pgmap v1716: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 644 op/s; 11305/50340 objects degraded (22.457%); 10140/50340 objects misplaced (20.143%); 18 MiB/s, 327 objects/s recovering 2024-09-02T10:27:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:27:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:14 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:27:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:14 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:27:15.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 11305/50340 objects degraded (22.457%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:15.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:15.323 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:15.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:15.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:14 smithi162 ceph-mon[20279]: pgmap v1716: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 644 op/s; 11305/50340 objects degraded (22.457%); 10140/50340 objects misplaced (20.143%); 18 MiB/s, 327 objects/s recovering 2024-09-02T10:27:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:14 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:27:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:14 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:27:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:14 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:27:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 11305/50340 objects degraded (22.457%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:15.362 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:27:15.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:14 smithi151 ceph-mon[16639]: pgmap v1716: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1.0 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 644 op/s; 11305/50340 objects degraded (22.457%); 10140/50340 objects misplaced (20.143%); 18 MiB/s, 327 objects/s recovering 2024-09-02T10:27:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:14 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:27:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:14 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:27:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:14 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:27:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 11305/50340 objects degraded (22.457%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:15.449 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:15.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:15.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:15.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:15.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:15.678 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:15.687 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:15.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:15.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:15.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:15.723 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:27:15.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:15.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:15.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:15.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:15.975 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:15.976 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:16.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:16.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:16.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:16.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:16.227 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:153594 7.10 e749/744 7:0a250376:::benchmark_data_smithi187_120712_object69652:head v 749'9064, mlcod=732'7570)", 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:16.225036+0000", 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00098815300000000008, 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010138689999999999, 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:16.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:16.225036+0000", 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:16.225036+0000", 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:16.225036+0000", 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8700000000000005e-07 2024-09-02T10:27:16.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:16.225042+0000", 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1159999999999998e-06 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:16.225068+0000", 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6536000000000001e-05 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:16.225074+0000", 2024-09-02T10:27:16.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8930000000000002e-06 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:16.225126+0000", 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1678999999999998e-05 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:16.225151+0000", 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4833000000000001e-05 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:16.231 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:16.232 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:27:16.232 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:27:16.232 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:27:16.236 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:16.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:16.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:16.360 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:16.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:16.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:16.486 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:16.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:16.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:16.610 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:16.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:16.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:16.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:16.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:16.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:16.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:16.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:16.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:16.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:16.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:17.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:17.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:17.042 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:27:17.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:17.171 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:17.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:16 smithi187 ceph-mon[20072]: pgmap v1717: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1005 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 11305/48207 objects degraded (23.451%); 9307/48207 objects misplaced (19.306%); 15 MiB/s, 308 objects/s recovering 2024-09-02T10:27:17.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:17.300 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:17.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:17.301 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:16 smithi162 ceph-mon[20279]: pgmap v1717: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1005 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 11305/48207 objects degraded (23.451%); 9307/48207 objects misplaced (19.306%); 15 MiB/s, 308 objects/s recovering 2024-09-02T10:27:17.386 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:17.402 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:16 smithi151 ceph-mon[16639]: pgmap v1717: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 47 active+clean; 1005 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 536 op/s; 11305/48207 objects degraded (23.451%); 9307/48207 objects misplaced (19.306%); 15 MiB/s, 308 objects/s recovering 2024-09-02T10:27:17.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:17.439 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:17.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:17.564 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:17.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:17.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:17.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:17.691 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:17.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:17.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:27:17.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:17.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:17.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:17.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:17.939 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:27:17.939 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:27:17.939 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:27:17.939 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:27:17.939 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:27:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:17.963 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:17.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:18.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:18.088 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:18.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:18.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:18.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:18.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:18.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:18.356 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:18.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:18.458 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:18.483 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:18.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:18.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:18.585 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:18.610 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:18.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:18.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:18.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:18.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:27:18.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:18.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:18.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:18.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:19.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:19.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:19.068 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:19.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:19.132 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:19.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:19.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:19.256 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:18 smithi151 ceph-mon[16639]: pgmap v1718: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 938 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 11093/45006 objects degraded (24.648%); 8791/45006 objects misplaced (19.533%); 13 MiB/s, 301 objects/s recovering 2024-09-02T10:27:19.256 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:18 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:27:19.257 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:18 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:27:19.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:18 smithi187 ceph-mon[20072]: pgmap v1718: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 938 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 11093/45006 objects degraded (24.648%); 8791/45006 objects misplaced (19.533%); 13 MiB/s, 301 objects/s recovering 2024-09-02T10:27:19.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:18 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:27:19.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:18 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:27:19.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:19.258 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:19.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:18 smithi162 ceph-mon[20279]: pgmap v1718: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 938 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 569 op/s; 11093/45006 objects degraded (24.648%); 8791/45006 objects misplaced (19.533%); 13 MiB/s, 301 objects/s recovering 2024-09-02T10:27:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:18 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:27:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:18 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:27:19.359 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:19.384 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:19.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:19.489 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:19.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:19.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:19.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:19.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:19.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:19.758 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162650 5.0 5:0595c7e6:::benchmark_data_smithi187_89405_object59045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 595.45182800199996, 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29653194100000002, 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:19.759 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162650 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302690+0000", 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.760 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302691+0000", 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6139999999999999e-06 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302696+0000", 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1599999999999997e-06 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.761 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302706+0000", 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3959999999999995e-06 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302711+0000", 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2920000000000003e-06 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302770+0000", 2024-09-02T10:27:19.762 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8545999999999998e-05 2024-09-02T10:27:19.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.302912+0000", 2024-09-02T10:27:19.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014217599999999999 2024-09-02T10:27:19.763 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.763 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.763 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:27:19.763 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.303032+0000", 2024-09-02T10:27:19.763 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120482 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.307337+0000", 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004304993 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.307512+0000", 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017463799999999999 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:19.764 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598774+0000", 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29126201499999999 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599166+0000", 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039243999999999998 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599222+0000", 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5189000000000003e-05 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.765 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:162688 5.8 e507/490 5:1af44aef:::benchmark_data_smithi187_89405_object59083:head v 507'10194, mlcod=507'10191)", 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 595.40661400800002, 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.262002719, 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:27:19.766 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347904+0000", 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8700000000000002e-07 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.767 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347908+0000", 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2030000000000002e-06 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347966+0000", 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7603999999999999e-05 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.347970+0000", 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1160000000000001e-06 2024-09-02T10:27:19.768 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.348622+0000", 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00065184499999999996 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.348639+0000", 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7731e-05 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609811+0000", 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26117189299999999 2024-09-02T10:27:19.769 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609906+0000", 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5039999999999998e-05 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:162690 5.f e507/431 5:f963755e:::benchmark_data_smithi187_89405_object59085:head v 507'10319, mlcod=507'10317)", 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:27:19.770 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 595.40291007899998, 2024-09-02T10:27:19.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26002222000000003, 2024-09-02T10:27:19.771 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.771 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.771 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.771 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.771 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.771 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.771 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:27:19.771 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.771 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351608+0000", 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3699999999999999e-07 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.351615+0000", 2024-09-02T10:27:19.772 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1940000000000003e-06 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353238+0000", 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001623471 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353245+0000", 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4540000000000004e-06 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.773 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:19.775 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.775 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353298+0000", 2024-09-02T10:27:19.775 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3186999999999997e-05 2024-09-02T10:27:19.775 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353317+0000", 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9083999999999999e-05 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611594+0000", 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.258277337 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.776 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611630+0000", 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5655999999999999e-05 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162691 5.4 5:2922879e:::benchmark_data_smithi187_89405_object59086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 595.40245021800001, 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25893540799999998, 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.777 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162691 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352067+0000", 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.778 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352069+0000", 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6869999999999999e-06 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352075+0000", 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5940000000000001e-06 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352087+0000", 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.216e-05 2024-09-02T10:27:19.779 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.352093+0000", 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5969999999999997e-06 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599399+0000", 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24730590099999999 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.780 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599528+0000", 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012868299999999999 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.599654+0000", 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000125959 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610381+0000", 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.01072701 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.781 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610393+0000", 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1921e-05 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610740+0000", 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00034708900000000002 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610837+0000", 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7175000000000004e-05 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.782 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.611003+0000", 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000165632 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:162695 5.0 5:02845cd7:::benchmark_data_smithi187_89405_object59090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e507)", 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 595.40068457400002, 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.266211531, 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.783 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 162695 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.784 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353833+0000", 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353834+0000", 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3900000000000004e-07 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.353838+0000", 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3460000000000004e-06 2024-09-02T10:27:19.785 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598349+0000", 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24451044699999999 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.598367+0000", 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8236999999999998e-05 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609481+0000", 2024-09-02T10:27:19.786 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011113576 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.609606+0000", 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012559100000000001 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.610188+0000", 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058149899999999997 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:19.787 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619903+0000", 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0097155599999999998 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619914+0000", 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1126e-05 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.619947+0000", 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2432999999999999e-05 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.788 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.620001+0000", 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4295000000000001e-05 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:24.620045+0000", 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3581999999999997e-05 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:173247 5.8 e509/490 5:173cf939:::benchmark_data_smithi187_89405_object69642:head v 509'10875, mlcod=509'10872)", 2024-09-02T10:27:19.789 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 582.61210156899995, 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25939572700000002, 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142416+0000", 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142417+0000", 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6099999999999999e-07 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.142424+0000", 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9489999999999999e-06 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396116+0000", 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25369161699999998 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396123+0000", 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5059999999999998e-06 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396178+0000", 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5271e-05 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.396199+0000", 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1009000000000002e-05 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.401729+0000", 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0055293260000000002 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:17:37.401812+0000", 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3088000000000001e-05 2024-09-02T10:27:19.793 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207153 5.4 5:2d15d89f:::benchmark_data_smithi187_89405_object103548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 539.710000896, 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.39443205100000001, 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:19.794 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207153 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044517+0000", 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044518+0000", 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1179999999999999e-06 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044523+0000", 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.113e-06 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044535+0000", 2024-09-02T10:27:19.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1868999999999999e-05 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044540+0000", 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9370000000000003e-06 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044599+0000", 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9589000000000002e-05 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044722+0000", 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000122166 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.044855+0000", 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000133774 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.048607+0000", 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037520180000000002 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.075260+0000", 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.026652973 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438874+0000", 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.363613939 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438913+0000", 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8795999999999999e-05 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438949+0000", 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5759000000000001e-05 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207153 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 539.66678653999998, 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35122426299999998, 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.800 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087731+0000", 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087732+0000", 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5300000000000004e-07 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.087736+0000", 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0659999999999997e-06 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438760+0000", 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.351023794 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438766+0000", 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6499999999999999e-06 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438864+0000", 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8116999999999996e-05 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438871+0000", 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1430000000000003e-06 2024-09-02T10:27:19.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.438955+0000", 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4839999999999994e-05 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.25933.0:207198 5.a e525/431 5:57bb5924:::benchmark_data_smithi187_89405_object103593:head v 525'12960, mlcod=525'12956)", 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 539.63793104499996, 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44214100000000001, 2024-09-02T10:27:19.804 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116587+0000", 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1399999999999996e-07 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.116593+0000", 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0079999999999999e-06 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554609+0000", 2024-09-02T10:27:19.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43801511199999998 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554615+0000", 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8349999999999998e-06 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554662+0000", 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7018000000000003e-05 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554678+0000", 2024-09-02T10:27:19.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5116000000000001e-05 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558692+0000", 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0040144350000000002 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.558728+0000", 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5661999999999998e-05 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.808 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.25933.0:207204 5.4 5:2473bd20:::benchmark_data_smithi187_89405_object103599:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e525)", 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 539.62739875800003, 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.73061778899999996, 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.25933", 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/1305848809", 2024-09-02T10:27:19.809 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207204 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127119+0000", 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127120+0000", 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2810000000000001e-06 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127126+0000", 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1210000000000003e-06 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127137+0000", 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0474e-05 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127142+0000", 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5419999999999996e-06 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127191+0000", 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8485000000000001e-05 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127278+0000", 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7492999999999997e-05 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,8", 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.127352+0000", 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3591999999999996e-05 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131106+0000", 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003754496 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850393+0000", 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.719286219 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857585+0000", 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0071919810000000001 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857692+0000", 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000107038 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857737+0000", 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5067000000000001e-05 2024-09-02T10:27:19.814 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207204 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 539.62287204300003, 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72609860400000004, 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131646+0000", 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131647+0000", 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2529999999999999e-06 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.816 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.131653+0000", 2024-09-02T10:27:19.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4119999999999998e-06 2024-09-02T10:27:19.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857394+0000", 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.72574053699999996 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857411+0000", 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7283e-05 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857560+0000", 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014918700000000001 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857574+0000", 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.329e-05 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.857744+0000", 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017064200000000001 2024-09-02T10:27:19.820 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.25933.0:207204 5.4 e525/431 ondisk, result = 0)", 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 539.19993083300005, 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.295830757, 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554587+0000", 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554588+0000", 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8899999999999998e-07 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.822 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.554593+0000", 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1039999999999998e-06 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850206+0000", 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29561359100000001 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850223+0000", 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6730000000000001e-05 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850366+0000", 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014310500000000001 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850383+0000", 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6419999999999999e-05 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:18:20.850418+0000", 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4918e-05 2024-09-02T10:27:19.824 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:7778 7.1 e666/661 7:8c9ea63b:::benchmark_data_smithi187_120712_object7777:head v 666'505, mlcod=666'502)", 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:17.653454+0000", 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 182.10106381700001, 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20058453800000001, 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.825 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.653454+0000", 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.653454+0000", 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.653454+0000", 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4300000000000003e-07 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.653525+0000", 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0949000000000001e-05 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.827 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.853476+0000", 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19995095600000001 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.853484+0000", 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.126e-06 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.853532+0000", 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7973000000000003e-05 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.853546+0000", 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4968e-05 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.854011+0000", 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00046434500000000001 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:17.854038+0000", 2024-09-02T10:27:19.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7577999999999999e-05 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:23110 7.1 e674/672 7:8822f6ec:::benchmark_data_smithi187_120712_object23109:head v 674'1481, mlcod=666'311)", 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:31.346366+0000", 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 168.40815167299999, 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21175090599999999, 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.830 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.346366+0000", 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.346366+0000", 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.346367+0000", 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3200000000000005e-07 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.346425+0000", 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8740999999999999e-05 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.557147+0000", 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21072139000000001 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.557153+0000", 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3740000000000003e-06 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.557199+0000", 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6190000000000003e-05 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.557217+0000", 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7941999999999999e-05 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.558084+0000", 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00086684299999999995 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.558117+0000", 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2793999999999998e-05 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.834 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:23135 7.1 e674/672 7:824a1355:::benchmark_data_smithi187_120712_object23134:head v 674'1482, mlcod=666'312)", 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:31.363635+0000", 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 168.39088226000001, 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20122978, 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.363635+0000", 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.363635+0000", 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.363636+0000", 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5099999999999999e-07 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.363690+0000", 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4290999999999999e-05 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.563852+0000", 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20016155799999999 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.563856+0000", 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5619999999999997e-06 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.563907+0000", 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0430000000000003e-05 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.563933+0000", 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5601000000000001e-05 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.564816+0000", 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00088335499999999995 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:31.564865+0000", 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9332000000000001e-05 2024-09-02T10:27:19.838 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:43258 7.2 e688/661 7:49444d4c:::benchmark_data_smithi187_120712_object43257:head v 688'2623, mlcod=688'2619)", 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:51.553410+0000", 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 148.20110735700001, 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.231937966, 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.553410+0000", 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.553410+0000", 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.553411+0000", 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0399999999999996e-07 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.553471+0000", 2024-09-02T10:27:19.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0463e-05 2024-09-02T10:27:19.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.841 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.841 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.841 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.783710+0000", 2024-09-02T10:27:19.841 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.230238832 2024-09-02T10:27:19.841 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.783724+0000", 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3334e-05 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.783836+0000", 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112781 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.783898+0000", 2024-09-02T10:27:19.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2123000000000001e-05 2024-09-02T10:27:19.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.785316+0000", 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014171559999999999 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.785348+0000", 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2672999999999997e-05 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:43269 7.1 e688/679 7:8e1e088a:::benchmark_data_smithi187_120712_object43268:head v 688'2659, mlcod=688'2656)", 2024-09-02T10:27:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:51.561315+0000", 2024-09-02T10:27:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 148.19320284400001, 2024-09-02T10:27:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23091329199999999, 2024-09-02T10:27:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.845 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.561315+0000", 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.561315+0000", 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.561316+0000", 2024-09-02T10:27:19.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.136e-06 2024-09-02T10:27:19.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.561358+0000", 2024-09-02T10:27:19.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2144000000000002e-05 2024-09-02T10:27:19.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.788687+0000", 2024-09-02T10:27:19.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.227328581 2024-09-02T10:27:19.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.788694+0000", 2024-09-02T10:27:19.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1729999999999998e-06 2024-09-02T10:27:19.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.849 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.849 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.788759+0000", 2024-09-02T10:27:19.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5592000000000005e-05 2024-09-02T10:27:19.850 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.850 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.850 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.850 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.788778+0000", 2024-09-02T10:27:19.850 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8573999999999999e-05 2024-09-02T10:27:19.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.851 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.851 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.851 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.792152+0000", 2024-09-02T10:27:19.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033742389999999998 2024-09-02T10:27:19.851 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.852 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.852 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.852 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.792228+0000", 2024-09-02T10:27:19.852 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5852999999999994e-05 2024-09-02T10:27:19.852 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.852 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.853 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.853 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:149412 7.e e749/661 7:78676864:::benchmark_data_smithi187_120712_object65470:head v 749'9132, mlcod=749'9129)", 2024-09-02T10:27:19.853 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:09.042681+0000", 2024-09-02T10:27:19.853 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 10.711836898, 2024-09-02T10:27:19.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.174122951, 2024-09-02T10:27:19.854 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.854 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.854 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.854 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:09.042681+0000", 2024-09-02T10:27:19.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:09.042681+0000", 2024-09-02T10:27:19.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:09.042682+0000", 2024-09-02T10:27:19.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.505e-06 2024-09-02T10:27:19.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:09.042690+0000", 2024-09-02T10:27:19.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8210000000000002e-06 2024-09-02T10:27:19.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:09.211930+0000", 2024-09-02T10:27:19.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16924017099999999 2024-09-02T10:27:19.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:09.211938+0000", 2024-09-02T10:27:19.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9330000000000001e-06 2024-09-02T10:27:19.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:09.212009+0000", 2024-09-02T10:27:19.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0384999999999998e-05 2024-09-02T10:27:19.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.861 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.861 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.861 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:09.212029+0000", 2024-09-02T10:27:19.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0775e-05 2024-09-02T10:27:19.861 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:09.216741+0000", 2024-09-02T10:27:19.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004711181 2024-09-02T10:27:19.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:09.216804+0000", 2024-09-02T10:27:19.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3180000000000002e-05 2024-09-02T10:27:19.863 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.863 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.864 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.864 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGRecoveryDelete(7.10 e749,744 7:0b45e5be:::benchmark_data_smithi187_120712_object36086:head,731'7527)", 2024-09-02T10:27:19.864 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:15.672912+0000", 2024-09-02T10:27:19.864 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 4.081605895, 2024-09-02T10:27:19.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18250243699999999, 2024-09-02T10:27:19.865 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.865 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.865 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:27:19.865 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:15.672912+0000", 2024-09-02T10:27:19.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:15.672912+0000", 2024-09-02T10:27:19.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:15.672913+0000", 2024-09-02T10:27:19.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3500000000000005e-07 2024-09-02T10:27:19.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:15.672918+0000", 2024-09-02T10:27:19.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.806e-06 2024-09-02T10:27:19.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:15.855298+0000", 2024-09-02T10:27:19.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.182379394 2024-09-02T10:27:19.870 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.870 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.870 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.870 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:15.855304+0000", 2024-09-02T10:27:19.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4840000000000001e-06 2024-09-02T10:27:19.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:15.855353+0000", 2024-09-02T10:27:19.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8435e-05 2024-09-02T10:27:19.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:15.855359+0000", 2024-09-02T10:27:19.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9970000000000004e-06 2024-09-02T10:27:19.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:15.855414+0000", 2024-09-02T10:27:19.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5485999999999997e-05 2024-09-02T10:27:19.873 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.873 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.873 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.874 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:155272 7.e e749/661 7:7956d247:::benchmark_data_smithi187_120712_object71330:head v 749'9472, mlcod=749'9469)", 2024-09-02T10:27:19.874 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:18.844679+0000", 2024-09-02T10:27:19.874 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.90983888300000004, 2024-09-02T10:27:19.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.17317275900000001, 2024-09-02T10:27:19.875 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:27:19.875 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:27:19.875 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:27:19.875 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:27:19.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:27:19.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:18.844679+0000", 2024-09-02T10:27:19.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:27:19.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:18.844679+0000", 2024-09-02T10:27:19.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:27:19.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:27:19.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:18.844680+0000", 2024-09-02T10:27:19.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.46e-06 2024-09-02T10:27:19.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:27:19.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:18.844689+0000", 2024-09-02T10:27:19.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9719999999999998e-06 2024-09-02T10:27:19.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.879 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:27:19.879 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:19.013360+0000", 2024-09-02T10:27:19.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.16867111300000001 2024-09-02T10:27:19.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:27:19.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:19.013368+0000", 2024-09-02T10:27:19.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.391e-06 2024-09-02T10:27:19.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:27:19.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:19.013484+0000", 2024-09-02T10:27:19.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115958 2024-09-02T10:27:19.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:27:19.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:19.013508+0000", 2024-09-02T10:27:19.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4766999999999999e-05 2024-09-02T10:27:19.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:27:19.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:19.017792+0000", 2024-09-02T10:27:19.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0042838779999999996 2024-09-02T10:27:19.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:27:19.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:27:19.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:27:19.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:19.017852+0000", 2024-09-02T10:27:19.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9219999999999999e-05 2024-09-02T10:27:19.884 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.884 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.884 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.884 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:27:19.885 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:27:19.885 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:27:19.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:19.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:19.987 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:20.018 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:20.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:20.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:20.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:20.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:19 smithi187 ceph-mon[20072]: osdmap e750: 12 total, 12 up, 12 in 2024-09-02T10:27:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 11093/45006 objects degraded (24.648%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:19 smithi187 ceph-mon[20072]: osdmap e751: 12 total, 12 up, 12 in 2024-09-02T10:27:20.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:20.280 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:20.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:20.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:19 smithi162 ceph-mon[20279]: osdmap e750: 12 total, 12 up, 12 in 2024-09-02T10:27:20.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 11093/45006 objects degraded (24.648%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:20.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:19 smithi162 ceph-mon[20279]: osdmap e751: 12 total, 12 up, 12 in 2024-09-02T10:27:20.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:20.407 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:20.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:19 smithi151 ceph-mon[16639]: osdmap e750: 12 total, 12 up, 12 in 2024-09-02T10:27:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 11093/45006 objects degraded (24.648%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:19 smithi151 ceph-mon[16639]: osdmap e751: 12 total, 12 up, 12 in 2024-09-02T10:27:20.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:20.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:20.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:20.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:20.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:20.683 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:20.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:20.806 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:20.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:20.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:20.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:20.933 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:20.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:20.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:21.035 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:21.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:21.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:21.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:21.095 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:20 smithi151 ceph-mon[16639]: pgmap v1721: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 821 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 10670/39405 objects degraded (27.078%); 7018/39405 objects misplaced (17.810%); 18 MiB/s, 430 objects/s recovering 2024-09-02T10:27:21.095 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/241890054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:27:21.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:27:21.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:21.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:21.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:20 smithi187 ceph-mon[20072]: pgmap v1721: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 821 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 10670/39405 objects degraded (27.078%); 7018/39405 objects misplaced (17.810%); 18 MiB/s, 430 objects/s recovering 2024-09-02T10:27:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/241890054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:27:21.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:21.338 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:21.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:21.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:20 smithi162 ceph-mon[20279]: pgmap v1721: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 821 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 739 op/s; 10670/39405 objects degraded (27.078%); 7018/39405 objects misplaced (17.810%); 18 MiB/s, 430 objects/s recovering 2024-09-02T10:27:21.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/241890054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:27:21.440 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:21.454 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:21.462 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:21.463 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:21.564 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:21.591 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:21.592 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:21.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:21.715 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:21.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:21.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:21.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:21.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:21.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:21.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:27:21.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:27:21.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:27:21.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:27:21.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:27:21.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:27:21.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:27:21.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:27:21.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:27:21.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:27:21.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:27:21.914 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:27:21.915 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:27:21.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:21.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:21.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:21.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:21.999 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:27:21.999 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:27:22.081 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:22.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:22.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:22.109 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:22.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:22.233 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:22.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:22.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:22.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:22.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:22.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:22.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:22.499 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:22.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:22.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:22.627 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:22.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:22.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:27:22.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:22.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:22.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:22.753 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:22.857 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:22.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:22.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:22.940 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:22.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:23.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:23.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:23.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:23.146 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:23.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:23.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:23.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:22 smithi187 ceph-mon[20072]: pgmap v1722: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 816 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 465 op/s; 10670/39153 objects degraded (27.252%); 7018/39153 objects misplaced (17.925%); 11 MiB/s, 313 objects/s recovering 2024-09-02T10:27:23.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:22 smithi187 ceph-mon[20072]: osdmap e752: 12 total, 12 up, 12 in 2024-09-02T10:27:23.278 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:23.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:23.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:23.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:22 smithi162 ceph-mon[20279]: pgmap v1722: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 816 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 465 op/s; 10670/39153 objects degraded (27.252%); 7018/39153 objects misplaced (17.925%); 11 MiB/s, 313 objects/s recovering 2024-09-02T10:27:23.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:22 smithi162 ceph-mon[20279]: osdmap e752: 12 total, 12 up, 12 in 2024-09-02T10:27:23.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:23.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:23.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:23.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:22 smithi151 ceph-mon[16639]: pgmap v1722: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 816 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 465 op/s; 10670/39153 objects degraded (27.252%); 7018/39153 objects misplaced (17.925%); 11 MiB/s, 313 objects/s recovering 2024-09-02T10:27:23.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:22 smithi151 ceph-mon[16639]: osdmap e752: 12 total, 12 up, 12 in 2024-09-02T10:27:23.505 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:27:23.505 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:27:23.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:23.551 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:23.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:23.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:23.681 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:23.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:23.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:23.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:23.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:23.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:23.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:23.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:23.953 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:23.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:24.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:24.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:24.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:24.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:24.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:24.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:24.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:24 smithi187 ceph-mon[20072]: osdmap e753: 12 total, 12 up, 12 in 2024-09-02T10:27:24.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:24 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:27:24.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:24 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:27:24.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:24 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:27:24.344 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:24.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:24 smithi162 ceph-mon[20279]: osdmap e753: 12 total, 12 up, 12 in 2024-09-02T10:27:24.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:24 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:27:24.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:24 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:27:24.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:24 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:27:24.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:24.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:24.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:24.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:27:24.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:24 smithi151 ceph-mon[16639]: osdmap e753: 12 total, 12 up, 12 in 2024-09-02T10:27:24.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:24 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:27:24.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:24 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:27:24.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:24 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:27:24.480 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:24.504 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:24.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:24.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:24.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:24.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:24.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:24.660 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:27:24.727 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:27:24.736 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:27:24.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:24.775 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:24.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:24.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:24.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:24.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:24.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:25.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:25.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:25.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:25.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:25.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:25.154 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:25 smithi187 ceph-mon[20072]: pgmap v1725: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 684 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 485 op/s; 10212/32814 objects degraded (31.121%); 6377/32814 objects misplaced (19.434%); 5.1 MiB/s, 246 objects/s recovering 2024-09-02T10:27:25.154 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:25 smithi187 ceph-mon[20072]: osdmap e754: 12 total, 12 up, 12 in 2024-09-02T10:27:25.154 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:25 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 10212/32814 objects degraded (31.121%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:25.154 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:25 smithi187 ceph-mon[20072]: osdmap e755: 12 total, 12 up, 12 in 2024-09-02T10:27:25.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:25 smithi151 ceph-mon[16639]: pgmap v1725: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 684 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 485 op/s; 10212/32814 objects degraded (31.121%); 6377/32814 objects misplaced (19.434%); 5.1 MiB/s, 246 objects/s recovering 2024-09-02T10:27:25.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:25 smithi151 ceph-mon[16639]: osdmap e754: 12 total, 12 up, 12 in 2024-09-02T10:27:25.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:25 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 10212/32814 objects degraded (31.121%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:25.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:25 smithi151 ceph-mon[16639]: osdmap e755: 12 total, 12 up, 12 in 2024-09-02T10:27:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:25.191 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:25.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:25.286 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:27:25.286 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:27:25.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:25.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:25.326 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:25 smithi162 ceph-mon[20279]: pgmap v1725: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 47 active+clean; 684 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 485 op/s; 10212/32814 objects degraded (31.121%); 6377/32814 objects misplaced (19.434%); 5.1 MiB/s, 246 objects/s recovering 2024-09-02T10:27:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:25 smithi162 ceph-mon[20279]: osdmap e754: 12 total, 12 up, 12 in 2024-09-02T10:27:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:25 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 10212/32814 objects degraded (31.121%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:25 smithi162 ceph-mon[20279]: osdmap e755: 12 total, 12 up, 12 in 2024-09-02T10:27:25.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:25.449 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:25.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:25.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:25.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:25.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:25.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:25.714 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:25.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:25.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:25.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:25.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:25.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:25.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:25.982 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:25.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:26.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:26.110 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:26.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:26.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:26.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:27:26.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:26.236 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:26.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:26.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:26.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:26.361 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:26.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:26.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:26.655 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:26.656 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:26.757 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:26.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:26.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:26.779 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:26.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:26.905 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:26.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:27.007 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:27.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:27.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:27.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:27.167 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:27.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:27.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:27.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:27.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:27.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:27 smithi162 ceph-mon[20279]: pgmap v1728: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 644 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 10209/30891 objects degraded (33.048%); 5966/30891 objects misplaced (19.313%); 7.9 MiB/s, 414 objects/s recovering 2024-09-02T10:27:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:27:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:27:27.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:27.422 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:27.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:27.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:27.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:27 smithi151 ceph-mon[16639]: pgmap v1728: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 644 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 10209/30891 objects degraded (33.048%); 5966/30891 objects misplaced (19.313%); 7.9 MiB/s, 414 objects/s recovering 2024-09-02T10:27:27.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:27:27.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:27:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:27 smithi187 ceph-mon[20072]: pgmap v1728: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 644 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 688 op/s; 10209/30891 objects degraded (33.048%); 5966/30891 objects misplaced (19.313%); 7.9 MiB/s, 414 objects/s recovering 2024-09-02T10:27:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:27:27.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:27:27.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:27.549 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:27.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:27.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:27.674 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:27.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:27.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:27.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:27.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:27.800 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:27.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:27.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:27.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:27.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:28.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:28.058 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:28.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:28.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:28.186 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:28.187 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:28.288 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:28.312 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:28.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:28.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:28 smithi162 ceph-mon[20279]: osdmap e756: 12 total, 12 up, 12 in 2024-09-02T10:27:28.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:28.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:28 smithi151 ceph-mon[16639]: osdmap e756: 12 total, 12 up, 12 in 2024-09-02T10:27:28.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:28.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:28.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:28 smithi187 ceph-mon[20072]: osdmap e756: 12 total, 12 up, 12 in 2024-09-02T10:27:28.553 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:28.583 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:28.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:28.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:28.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:28.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:28.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:28.835 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:28.836 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:28.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:28.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:28.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:28.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:29.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:29.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:29.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:29.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:29.224 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:29.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:29.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:29 smithi162 ceph-mon[20279]: pgmap v1730: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 579 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 10034/27798 objects degraded (36.096%); 5880/27798 objects misplaced (21.153%); 7.0 MiB/s, 410 objects/s recovering 2024-09-02T10:27:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:29 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:27:29.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:29.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:29 smithi151 ceph-mon[16639]: pgmap v1730: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 579 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 10034/27798 objects degraded (36.096%); 5880/27798 objects misplaced (21.153%); 7.0 MiB/s, 410 objects/s recovering 2024-09-02T10:27:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:29 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:27:29.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:29.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:29.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:29.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:29 smithi187 ceph-mon[20072]: pgmap v1730: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 48 active+clean; 579 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 579 op/s; 10034/27798 objects degraded (36.096%); 5880/27798 objects misplaced (21.153%); 7.0 MiB/s, 410 objects/s recovering 2024-09-02T10:27:29.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:29 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:27:29.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:29.619 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:29.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:29.620 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:29.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:29.744 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:29.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:29.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:29.875 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:29.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:29.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:29.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:30.001 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:30.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:30.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:30.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:30.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:30.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:30.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:30.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:30.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:30 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 10034/27798 objects degraded (36.096%), 7 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:30 smithi162 ceph-mon[20279]: osdmap e757: 12 total, 12 up, 12 in 2024-09-02T10:27:30.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:30.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:30.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:30.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:30.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:30 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 10034/27798 objects degraded (36.096%), 7 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:30.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:30 smithi151 ceph-mon[16639]: osdmap e757: 12 total, 12 up, 12 in 2024-09-02T10:27:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:30 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 10034/27798 objects degraded (36.096%), 7 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:30.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:30 smithi187 ceph-mon[20072]: osdmap e757: 12 total, 12 up, 12 in 2024-09-02T10:27:30.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:30.538 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:30.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:30.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:30.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:30.661 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:30.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:30.763 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:30.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:30.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:30.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:30.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:30.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:30.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:31.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:31.052 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:31.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:31.154 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:31.180 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:31.181 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:31.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:31.308 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:31.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:31 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:27:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:31 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:27:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:31 smithi162 ceph-mon[20279]: pgmap v1732: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 465 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 9658/22326 objects degraded (43.259%); 5692/22326 objects misplaced (25.495%); 3.1 MiB/s, 356 objects/s recovering 2024-09-02T10:27:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:31 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:27:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:31 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:27:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:31 smithi162 ceph-mon[20279]: osdmap e758: 12 total, 12 up, 12 in 2024-09-02T10:27:31.410 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:31.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:31.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:31.436 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:31 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:27:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:31 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:27:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:31 smithi151 ceph-mon[16639]: pgmap v1732: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 465 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 9658/22326 objects degraded (43.259%); 5692/22326 objects misplaced (25.495%); 3.1 MiB/s, 356 objects/s recovering 2024-09-02T10:27:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:31 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:27:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:31 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:27:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:31 smithi151 ceph-mon[16639]: osdmap e758: 12 total, 12 up, 12 in 2024-09-02T10:27:31.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:31 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:27:31.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:31 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:27:31.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:31 smithi187 ceph-mon[20072]: pgmap v1732: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 465 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 537 op/s; 9658/22326 objects degraded (43.259%); 5692/22326 objects misplaced (25.495%); 3.1 MiB/s, 356 objects/s recovering 2024-09-02T10:27:31.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:31 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:27:31.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:31 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:27:31.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:31 smithi187 ceph-mon[20072]: osdmap e758: 12 total, 12 up, 12 in 2024-09-02T10:27:31.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:31.561 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:31.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:31.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:31.687 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:31.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:31.789 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:31.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:31.811 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:31.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:31.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:31.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:31.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:31.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:32.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:32.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:32.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:32.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:32.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:32.212 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:32.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:32.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:32.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:32.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:32 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:27:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:32 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:27:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:32 smithi151 ceph-mon[16639]: osdmap e759: 12 total, 12 up, 12 in 2024-09-02T10:27:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:32 smithi151 ceph-mon[16639]: pgmap v1735: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 459 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 9658/22038 objects degraded (43.824%); 5692/22038 objects misplaced (25.828%); 3.8 MiB/s, 437 objects/s recovering 2024-09-02T10:27:32.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:32.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:32.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:32.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:32.573 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:32 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:27:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:32 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:27:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:32 smithi162 ceph-mon[20279]: osdmap e759: 12 total, 12 up, 12 in 2024-09-02T10:27:32.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:32 smithi162 ceph-mon[20279]: pgmap v1735: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 459 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 9658/22038 objects degraded (43.824%); 5692/22038 objects misplaced (25.828%); 3.8 MiB/s, 437 objects/s recovering 2024-09-02T10:27:32.596 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:32.597 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:32.698 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:32.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:32.724 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:32 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:27:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:32 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:27:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:32 smithi187 ceph-mon[20072]: osdmap e759: 12 total, 12 up, 12 in 2024-09-02T10:27:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:32 smithi187 ceph-mon[20072]: pgmap v1735: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 459 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 662 op/s; 9658/22038 objects degraded (43.824%); 5692/22038 objects misplaced (25.828%); 3.8 MiB/s, 437 objects/s recovering 2024-09-02T10:27:32.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:32.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:32.856 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:32.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:32.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:32.988 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:32.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:32.989 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:33.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:33.114 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:33.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:33.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:33.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:33.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:33.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:33.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:33.366 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:33.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:33.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:33.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:33.489 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:33.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:33.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:33.614 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:33.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:33.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:33.740 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:33.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:33.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:33.863 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:33.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:33.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:33.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:33.989 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:33.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:34.091 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:34.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:34.115 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:34.115 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:34.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:34.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:34.239 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:34.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:34.362 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:34.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:34.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:34.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:34.490 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:34.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:34.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:34.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:34.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:34.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:34.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:34.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:34 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:27:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:34 smithi187 ceph-mon[20072]: pgmap v1736: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 48 active+clean; 314 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 9047/15084 objects degraded (59.977%); 5385/15084 objects misplaced (35.700%); 5.2 MiB/s, 452 objects/s recovering 2024-09-02T10:27:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:34 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 9047/15084 objects degraded (59.977%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:34 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:27:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:34 smithi162 ceph-mon[20279]: pgmap v1736: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 48 active+clean; 314 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 9047/15084 objects degraded (59.977%); 5385/15084 objects misplaced (35.700%); 5.2 MiB/s, 452 objects/s recovering 2024-09-02T10:27:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:34 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 9047/15084 objects degraded (59.977%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:34.851 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:34.879 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:34.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:34 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:27:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:34 smithi151 ceph-mon[16639]: pgmap v1736: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 48 active+clean; 314 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 706 op/s; 9047/15084 objects degraded (59.977%); 5385/15084 objects misplaced (35.700%); 5.2 MiB/s, 452 objects/s recovering 2024-09-02T10:27:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:34 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 9047/15084 objects degraded (59.977%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:34.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:35.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:35.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:35.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:35.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:35.136 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:35.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:35.266 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:35.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:35.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:35.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:35.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:35.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:35.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:35.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:35.526 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:35.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:35.650 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:35.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:35.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:35.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:35.779 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:35.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:35.780 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:35.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:35.902 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:35.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:36.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:36.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:36.033 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:36.137 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:36.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:36.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:36.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:36.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:36.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:36.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:36.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:36.415 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:36.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:36.517 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:36.541 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:36.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:36.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:36.675 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:36.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:36.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:36.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:36.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:36.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:36.799 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:36 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:27:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:36 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:27:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:36 smithi162 ceph-mon[20279]: pgmap v1737: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 271 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 9045/12993 objects degraded (69.614%); 5385/12993 objects misplaced (41.445%); 3.1 MiB/s, 297 objects/s recovering 2024-09-02T10:27:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:36 smithi162 ceph-mon[20279]: osdmap e760: 12 total, 12 up, 12 in 2024-09-02T10:27:36.900 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:36.925 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:36.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:36 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:27:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:36 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:27:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:36 smithi151 ceph-mon[16639]: pgmap v1737: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 271 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 9045/12993 objects degraded (69.614%); 5385/12993 objects misplaced (41.445%); 3.1 MiB/s, 297 objects/s recovering 2024-09-02T10:27:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:36 smithi151 ceph-mon[16639]: osdmap e760: 12 total, 12 up, 12 in 2024-09-02T10:27:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:36 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:27:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:36 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:27:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:36 smithi187 ceph-mon[20072]: pgmap v1737: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 271 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 498 op/s; 9045/12993 objects degraded (69.614%); 5385/12993 objects misplaced (41.445%); 3.1 MiB/s, 297 objects/s recovering 2024-09-02T10:27:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:36 smithi187 ceph-mon[20072]: osdmap e760: 12 total, 12 up, 12 in 2024-09-02T10:27:37.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:37.048 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:37.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:37.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:37.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:37.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:37.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:37.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:37.319 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:37.320 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:37.421 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:37.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:37.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:37.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:37.576 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:37.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:37.588 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:27:37.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:37.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:37.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:37.808 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:37.831 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:37.832 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:37.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:37.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:37.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:38.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:38.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:38.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:38.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:38.216 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:38.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:38.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:38.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:38.345 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:38.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:38.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:38.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:38.470 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:38.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:38.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:38.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:38.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:38.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:38.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:38.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:38.740 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:38.841 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:38 smithi162 ceph-mon[20279]: pgmap v1739: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 210 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 551 op/s; 8909/10074 objects degraded (88.436%); 5316/10074 objects misplaced (52.770%); 3.2 MiB/s, 352 objects/s recovering 2024-09-02T10:27:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1100347250' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:27:38.848 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-02T10:27:38.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:38 smithi151 ceph-mon[16639]: pgmap v1739: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 210 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 551 op/s; 8909/10074 objects degraded (88.436%); 5316/10074 objects misplaced (52.770%); 3.2 MiB/s, 352 objects/s recovering 2024-09-02T10:27:38.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1100347250' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:27:38.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:38.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:38.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:39.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:39.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:38 smithi187 ceph-mon[20072]: pgmap v1739: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 48 active+clean; 210 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 551 op/s; 8909/10074 objects degraded (88.436%); 5316/10074 objects misplaced (52.770%); 3.2 MiB/s, 352 objects/s recovering 2024-09-02T10:27:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1100347250' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:27:39.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:39.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:39.130 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:39.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:39.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:39.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:39.364 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:39.396 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:39.397 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:39.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:39.524 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:39.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:39.626 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:39.648 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:39.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:39.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:39.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:39.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:39.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:39 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:27:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1100347250' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:27:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:39 smithi162 ceph-mon[20279]: osdmap e761: 12 total, 12 up, 12 in 2024-09-02T10:27:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 8909/10074 objects degraded (88.436%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:39.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:39.910 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:39.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:39.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:39 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:27:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1100347250' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:27:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:39 smithi151 ceph-mon[16639]: osdmap e761: 12 total, 12 up, 12 in 2024-09-02T10:27:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 8909/10074 objects degraded (88.436%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:39 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:27:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1100347250' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:27:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:39 smithi187 ceph-mon[20072]: osdmap e761: 12 total, 12 up, 12 in 2024-09-02T10:27:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 8909/10074 objects degraded (88.436%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:40.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:40.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:40.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:40.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:40.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:40.164 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:40.165 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:40.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:40.292 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:40.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:40.395 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:40.419 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:40.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:40.521 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:40.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:40.544 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:40.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:40.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:40.672 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:40.672 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:40.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:40.795 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:40.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:40 smithi162 ceph-mon[20279]: pgmap v1741: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 48 active+clean; 112 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 8566/5382 objects degraded (159.160%); 5144/5382 objects misplaced (95.578%); 4.3 MiB/s, 433 objects/s recovering 2024-09-02T10:27:40.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:40.922 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:40.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:40.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:40 smithi151 ceph-mon[16639]: pgmap v1741: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 48 active+clean; 112 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 8566/5382 objects degraded (159.160%); 5144/5382 objects misplaced (95.578%); 4.3 MiB/s, 433 objects/s recovering 2024-09-02T10:27:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:40 smithi187 ceph-mon[20072]: pgmap v1741: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 48 active+clean; 112 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 693 op/s; 8566/5382 objects degraded (159.160%); 5144/5382 objects misplaced (95.578%); 4.3 MiB/s, 433 objects/s recovering 2024-09-02T10:27:41.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:41.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:41.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:41.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:41.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:41.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:41.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:41.297 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:41.298 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:41.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:41.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:41.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:41.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:41.544 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:41.545 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:41.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:41.681 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:41.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:41.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:41.783 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:41.805 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:41.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:41 smithi162 ceph-mon[20279]: osdmap e762: 12 total, 12 up, 12 in 2024-09-02T10:27:41.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:41.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:41.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:41.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:41.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:41 smithi151 ceph-mon[16639]: osdmap e762: 12 total, 12 up, 12 in 2024-09-02T10:27:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:41 smithi187 ceph-mon[20072]: osdmap e762: 12 total, 12 up, 12 in 2024-09-02T10:27:42.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:42.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:42.058 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:42.059 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:42.118 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:27:42.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:42.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:42.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:42.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:42.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:42.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:42.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:42.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:42.439 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:42.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:42.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:42.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:42.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:42.684 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Removed 83939 objects 2024-09-02T10:27:42.684 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Clean up completed and total clean up time :118.911 2024-09-02T10:27:42.691 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:42.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:42.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:42.814 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:42.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:42.815 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:42.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:42.937 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:42 smithi151 ceph-mon[16639]: pgmap v1743: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 48 active+clean; 107 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 436 op/s; 8566/5127 objects degraded (167.076%); 5144/5127 objects misplaced (100.332%); 2.5 MiB/s, 268 objects/s recovering 2024-09-02T10:27:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:42 smithi151 ceph-mon[16639]: osdmap e763: 12 total, 12 up, 12 in 2024-09-02T10:27:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:42 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:27:42.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:42.939 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:42 smithi187 ceph-mon[20072]: pgmap v1743: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 48 active+clean; 107 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 436 op/s; 8566/5127 objects degraded (167.076%); 5144/5127 objects misplaced (100.332%); 2.5 MiB/s, 268 objects/s recovering 2024-09-02T10:27:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:42 smithi187 ceph-mon[20072]: osdmap e763: 12 total, 12 up, 12 in 2024-09-02T10:27:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:42 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:27:43.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:43.061 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:43.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:42 smithi162 ceph-mon[20279]: pgmap v1743: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 48 active+clean; 107 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 436 op/s; 8566/5127 objects degraded (167.076%); 5144/5127 objects misplaced (100.332%); 2.5 MiB/s, 268 objects/s recovering 2024-09-02T10:27:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:42 smithi162 ceph-mon[20279]: osdmap e763: 12 total, 12 up, 12 in 2024-09-02T10:27:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:42 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:27:43.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:43.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:43.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:43.289 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:43.311 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:43.311 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:43.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:43.434 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:43.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:43.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:43.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:43.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:43.559 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:43.660 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:43.683 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:43.684 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:43.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:43.806 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:43.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:43.807 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:43.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:43.929 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:43.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:43.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:43 smithi151 ceph-mon[16639]: osdmap e764: 12 total, 12 up, 12 in 2024-09-02T10:27:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:43 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:27:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:43 smithi187 ceph-mon[20072]: osdmap e764: 12 total, 12 up, 12 in 2024-09-02T10:27:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:43 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:27:44.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:44.051 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:44.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:43 smithi162 ceph-mon[20279]: osdmap e764: 12 total, 12 up, 12 in 2024-09-02T10:27:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:43 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:27:44.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:44.175 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:44.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:44.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:44.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:44.299 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:44.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:44.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:44.427 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:44.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:44.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:44.529 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:44.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:44.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:44.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:44.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:44.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:44.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:44.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:44.784 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:44.806 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:44.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:44.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:44.907 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:44.930 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:44.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:44 smithi151 ceph-mon[16639]: pgmap v1746: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 49 active+clean; 27 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 5053/1260 objects degraded (401.032%); 3372/1260 objects misplaced (267.619%); 2.0 MiB/s, 386 objects/s recovering 2024-09-02T10:27:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:44 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:27:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:44 smithi151 ceph-mon[16639]: osdmap e765: 12 total, 12 up, 12 in 2024-09-02T10:27:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:44 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 5053/1260 objects degraded (401.032%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:44 smithi187 ceph-mon[20072]: pgmap v1746: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 49 active+clean; 27 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 5053/1260 objects degraded (401.032%); 3372/1260 objects misplaced (267.619%); 2.0 MiB/s, 386 objects/s recovering 2024-09-02T10:27:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:44 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:27:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:44 smithi187 ceph-mon[20072]: osdmap e765: 12 total, 12 up, 12 in 2024-09-02T10:27:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:44 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 5053/1260 objects degraded (401.032%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:45.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:45.054 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:45.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:44 smithi162 ceph-mon[20279]: pgmap v1746: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 49 active+clean; 27 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 5053/1260 objects degraded (401.032%); 3372/1260 objects misplaced (267.619%); 2.0 MiB/s, 386 objects/s recovering 2024-09-02T10:27:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:44 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:27:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:44 smithi162 ceph-mon[20279]: osdmap e765: 12 total, 12 up, 12 in 2024-09-02T10:27:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:44 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 5053/1260 objects degraded (401.032%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:27:45.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:45.178 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:45.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:45.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:45.302 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:45.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:45.303 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:45.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:45.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:45.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:45.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:45.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:45.551 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:45.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:45.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:45.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:45.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:45.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:45.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:45.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:45.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:45.924 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:45.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:45 smithi151 ceph-mon[16639]: osdmap e766: 12 total, 12 up, 12 in 2024-09-02T10:27:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:45 smithi187 ceph-mon[20072]: osdmap e766: 12 total, 12 up, 12 in 2024-09-02T10:27:46.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:46.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:46.049 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:46.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:45 smithi162 ceph-mon[20279]: osdmap e766: 12 total, 12 up, 12 in 2024-09-02T10:27:46.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:46.172 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:46.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:46.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:46.295 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:46.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:46.304 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_5 2024-09-02T10:27:46.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_5 unique_pool_5 --yes-i-really-really-mean-it 2024-09-02T10:27:46.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:46.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:46.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:46.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:46.519 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:46.541 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:46.542 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:46.642 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:46.669 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:46.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:46.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:46.793 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:46.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:46.796 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:46.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:46.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:46.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:46.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:46 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:27:46.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:46 smithi151 ceph-mon[16639]: pgmap v1749: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 50 active+clean; 4.2 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 443 op/s; 5051/186 objects degraded (2715.591%); 3372/186 objects misplaced (1812.903%); 2.4 MiB/s, 548 objects/s recovering 2024-09-02T10:27:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:46 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:27:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:46 smithi187 ceph-mon[20072]: pgmap v1749: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 50 active+clean; 4.2 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 443 op/s; 5051/186 objects degraded (2715.591%); 3372/186 objects misplaced (1812.903%); 2.4 MiB/s, 548 objects/s recovering 2024-09-02T10:27:47.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:47.043 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:47.044 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:47.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:46 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:27:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:46 smithi162 ceph-mon[20279]: pgmap v1749: 57 pgs: 4 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 50 active+clean; 4.2 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 443 op/s; 5051/186 objects degraded (2715.591%); 3372/186 objects misplaced (1812.903%); 2.4 MiB/s, 548 objects/s recovering 2024-09-02T10:27:47.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:47.167 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:47.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:47.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:47.289 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:47.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:47.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:47.413 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:47.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:47.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:47.535 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:47.536 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:47.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:47.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:47.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:47.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:47.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:47.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:47.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:47.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:47.908 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:47.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:47.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:47 smithi151 ceph-mon[16639]: osdmap e767: 12 total, 12 up, 12 in 2024-09-02T10:27:47.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:47 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:27:47.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:47 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:27:47.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:47 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:27:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:47 smithi187 ceph-mon[20072]: osdmap e767: 12 total, 12 up, 12 in 2024-09-02T10:27:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:47 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:27:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:47 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:27:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:47 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:27:48.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:48.031 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:48.032 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:47 smithi162 ceph-mon[20279]: osdmap e767: 12 total, 12 up, 12 in 2024-09-02T10:27:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:47 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:27:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:47 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:27:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:47 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:27:48.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:48.156 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:48.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:48.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:48.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:48.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:48.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:48.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:48.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:48.406 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:48.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:48.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:48.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:48.530 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:48.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:48.652 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:48.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:48.653 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:48.754 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:48.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:48.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:48.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:48.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:48.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:48.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:49.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:48 smithi187 ceph-mon[20072]: pgmap v1751: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 51 active+clean; 4.2 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 197 op/s; 5049/183 objects degraded (2759.016%); 3372/183 objects misplaced (1842.623%); 1.2 MiB/s, 442 objects/s recovering 2024-09-02T10:27:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:48 smithi187 ceph-mon[20072]: osdmap e768: 12 total, 12 up, 12 in 2024-09-02T10:27:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:48 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:27:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:48 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:27:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/278671584' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:27:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:48 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:27:49.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:49.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:49.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:48 smithi162 ceph-mon[20279]: pgmap v1751: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 51 active+clean; 4.2 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 197 op/s; 5049/183 objects degraded (2759.016%); 3372/183 objects misplaced (1842.623%); 1.2 MiB/s, 442 objects/s recovering 2024-09-02T10:27:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:48 smithi162 ceph-mon[20279]: osdmap e768: 12 total, 12 up, 12 in 2024-09-02T10:27:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:48 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:27:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:48 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:27:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/278671584' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:27:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:48 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:27:49.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:49.137 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:27:49.137 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:48 smithi151 ceph-mon[16639]: pgmap v1751: 57 pgs: 2 active+recovering+undersized+degraded+remapped, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 51 active+clean; 4.2 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 197 op/s; 5049/183 objects degraded (2759.016%); 3372/183 objects misplaced (1842.623%); 1.2 MiB/s, 442 objects/s recovering 2024-09-02T10:27:49.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:48 smithi151 ceph-mon[16639]: osdmap e768: 12 total, 12 up, 12 in 2024-09-02T10:27:49.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:48 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:27:49.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:48 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:27:49.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/278671584' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:27:49.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:48 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:27:49.150 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:49.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:49.151 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:49.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:49.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:49.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:49.275 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:49.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:49.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:49.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:49.502 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:49.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:49.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:49.524 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:49.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:49.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:49.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:49.749 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:49.770 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:49.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:49.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:49.895 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:49.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:49.997 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:49 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:27:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:49 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:27:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:49 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:27:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/278671584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:27:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:49 smithi187 ceph-mon[20072]: osdmap e769: 12 total, 12 up, 12 in 2024-09-02T10:27:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:49 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 5049/183 objects degraded (2759.016%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:27:50.019 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:50.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:49 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:27:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:49 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:27:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:49 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:27:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/278671584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:27:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:49 smithi162 ceph-mon[20279]: osdmap e769: 12 total, 12 up, 12 in 2024-09-02T10:27:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:49 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 5049/183 objects degraded (2759.016%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:27:50.120 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:50.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:50.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:49 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:27:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:49 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:27:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:49 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:27:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/278671584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:27:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:49 smithi151 ceph-mon[16639]: osdmap e769: 12 total, 12 up, 12 in 2024-09-02T10:27:50.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:49 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 5049/183 objects degraded (2759.016%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:27:50.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:50.269 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:50.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:50.371 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:50.393 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:50.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:50.494 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:50.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:50.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:50.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:50.641 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:50.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:50.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:50.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:50.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:50.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:50.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:50.867 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:50.888 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:50.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:50.890 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:50.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:50 smithi187 ceph-mon[20072]: pgmap v1754: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 5054/183 objects degraded (2761.749%); 3372/183 objects misplaced (1842.623%); 0 B/s, 219 objects/s recovering 2024-09-02T10:27:51.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:51.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:50 smithi162 ceph-mon[20279]: pgmap v1754: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 5054/183 objects degraded (2761.749%); 3372/183 objects misplaced (1842.623%); 0 B/s, 219 objects/s recovering 2024-09-02T10:27:51.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:51.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:51.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:51.139 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:51.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:50 smithi151 ceph-mon[16639]: pgmap v1754: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 5054/183 objects degraded (2761.749%); 3372/183 objects misplaced (1842.623%); 0 B/s, 219 objects/s recovering 2024-09-02T10:27:51.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:51.261 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:51.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:51.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:51.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:51.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:51.487 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:51.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:51.511 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:51.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:51.633 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:51.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:51.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:51.757 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:51.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:51.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:51.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:51.882 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:51.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:51.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:52.005 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:52.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:52.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:52.107 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:52.129 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:52.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:52.231 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:52.253 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:52.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:52.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:27:52.376 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:52.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:27:52.478 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:52.500 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:52.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:52.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:52.623 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:52.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:52.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:52.746 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:52.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:52.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:52.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:52.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:52.872 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:52.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:52.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:52.995 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:52 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:27:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:52 smithi187 ceph-mon[20072]: pgmap v1755: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 5054/183 objects degraded (2761.749%); 3372/183 objects misplaced (1842.623%); 0 B/s, 176 objects/s recovering 2024-09-02T10:27:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:52 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:27:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:52 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:27:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:52 smithi162 ceph-mon[20279]: pgmap v1755: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 5054/183 objects degraded (2761.749%); 3372/183 objects misplaced (1842.623%); 0 B/s, 176 objects/s recovering 2024-09-02T10:27:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:52 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:27:53.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:53.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:53.117 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:53.118 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:52 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:27:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:52 smithi151 ceph-mon[16639]: pgmap v1755: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 1 creating+peering, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 5054/183 objects degraded (2761.749%); 3372/183 objects misplaced (1842.623%); 0 B/s, 176 objects/s recovering 2024-09-02T10:27:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:52 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:27:53.219 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:53.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:53.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:53.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:53.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:53.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:53.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:53.487 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:53.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:53.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:53.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:53.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:53.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:53.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:53.735 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:53.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:53.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:53.858 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:53.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:53.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:53.982 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:53.982 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:54.083 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:54.104 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:54.105 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:54.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:54.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:54.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:54.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:54.328 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:54.352 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:54.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:54.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:54.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:54.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:54.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:54.579 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:54.600 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:54.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:54.701 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:54.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:54.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:54.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:54.846 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:54.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:54.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:54.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:54.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:54.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:54.970 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:54 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:27:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:54 smithi187 ceph-mon[20072]: pgmap v1756: 57 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 49 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 5055/183 objects degraded (2762.295%); 3372/183 objects misplaced (1842.623%); 0 B/s, 259 objects/s recovering 2024-09-02T10:27:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:54 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:27:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:54 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 5055/183 objects degraded (2762.295%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:27:55.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:55.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:55.093 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:54 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:27:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:54 smithi162 ceph-mon[20279]: pgmap v1756: 57 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 49 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 5055/183 objects degraded (2762.295%); 3372/183 objects misplaced (1842.623%); 0 B/s, 259 objects/s recovering 2024-09-02T10:27:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:54 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:27:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:54 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 5055/183 objects degraded (2762.295%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:27:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:54 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:27:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:54 smithi151 ceph-mon[16639]: pgmap v1756: 57 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 49 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 5055/183 objects degraded (2762.295%); 3372/183 objects misplaced (1842.623%); 0 B/s, 259 objects/s recovering 2024-09-02T10:27:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:54 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:27:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:54 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 5055/183 objects degraded (2762.295%), 3 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:27:55.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:55.214 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:55.215 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:55.316 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:55.337 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:55.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:55.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:55.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:55.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:55.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:55.563 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:55.584 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:55.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:55.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:55.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:55.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:55.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:55.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:55.833 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:55.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:55.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:55.957 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:55.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:55 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:27:56.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:56.081 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:56.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:56.082 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:55 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:27:56.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:56.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:55 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:27:56.205 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:56.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:56.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:56.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:56.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:56.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:56.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:56.451 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:56.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:56.553 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:56.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:56.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:56.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:56.701 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:56.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:56.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:56.803 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:56.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:56.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:56.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:56.933 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:27:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:56 smithi151 ceph-mon[16639]: pgmap v1757: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 49 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5050/183 objects degraded (2759.563%); 3372/183 objects misplaced (1842.623%); 0 B/s, 193 objects/s recovering 2024-09-02T10:27:56.955 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:56.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:27:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:56 smithi187 ceph-mon[20072]: pgmap v1757: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 49 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5050/183 objects degraded (2759.563%); 3372/183 objects misplaced (1842.623%); 0 B/s, 193 objects/s recovering 2024-09-02T10:27:57.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:27:57.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:57.079 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:57.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:27:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:56 smithi162 ceph-mon[20279]: pgmap v1757: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 49 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5050/183 objects degraded (2759.563%); 3372/183 objects misplaced (1842.623%); 0 B/s, 193 objects/s recovering 2024-09-02T10:27:57.180 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:57.203 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:57.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:57.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:57.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:57.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:57.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:57.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:57.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:57.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:57.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:27:57.576 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:57.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:27:57.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:57.745 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:57.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:57.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:57.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:57.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:57.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:57.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:57.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:57.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:57 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:27:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:57 smithi187 ceph-mon[20072]: osdmap e770: 12 total, 12 up, 12 in 2024-09-02T10:27:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:57 smithi187 ceph-mon[20072]: osdmap e771: 12 total, 12 up, 12 in 2024-09-02T10:27:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:57 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:27:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:57 smithi162 ceph-mon[20279]: osdmap e770: 12 total, 12 up, 12 in 2024-09-02T10:27:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:57 smithi162 ceph-mon[20279]: osdmap e771: 12 total, 12 up, 12 in 2024-09-02T10:27:58.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:27:58.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:58.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:27:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:57 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:27:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:57 smithi151 ceph-mon[16639]: osdmap e770: 12 total, 12 up, 12 in 2024-09-02T10:27:58.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:57 smithi151 ceph-mon[16639]: osdmap e771: 12 total, 12 up, 12 in 2024-09-02T10:27:58.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:58.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:58.247 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:58.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:58.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:58.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:58.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:58.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:58.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:58.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:58.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:58.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:27:58.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:58.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:58.615 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:27:58.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:58.738 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:58.739 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:58.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:58.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:58.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:58.967 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:58.989 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:58.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:58 smithi187 ceph-mon[20072]: pgmap v1759: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5050/183 objects degraded (2759.563%); 3372/183 objects misplaced (1842.623%); 0 B/s, 175 objects/s recovering 2024-09-02T10:27:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:58 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:27:59.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:27:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:58 smithi162 ceph-mon[20279]: pgmap v1759: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5050/183 objects degraded (2759.563%); 3372/183 objects misplaced (1842.623%); 0 B/s, 175 objects/s recovering 2024-09-02T10:27:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:58 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:27:59.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:59.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:27:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:58 smithi151 ceph-mon[16639]: pgmap v1759: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 2 active+recovery_wait+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5050/183 objects degraded (2759.563%); 3372/183 objects misplaced (1842.623%); 0 B/s, 175 objects/s recovering 2024-09-02T10:27:59.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:58 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:27:59.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:27:59.236 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:59.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:59.237 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:27:59.338 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:27:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:59.361 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:59.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:27:59.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:59.484 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:59.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:59.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:59.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:27:59.608 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:27:59.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:59.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:27:59.710 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:59.731 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:59.732 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:27:59.833 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:27:59.855 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:27:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:59.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:27:59.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:27:59.978 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:27:59.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:27:59.979 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:59 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:28:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:59 smithi187 ceph-mon[20072]: osdmap e772: 12 total, 12 up, 12 in 2024-09-02T10:28:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:59 smithi187 ceph-mon[20072]: 7.13 scrub starts 2024-09-02T10:28:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 5050/183 objects degraded (2759.563%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:27:59 smithi187 ceph-mon[20072]: osdmap e773: 12 total, 12 up, 12 in 2024-09-02T10:28:00.080 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:59 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:28:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:59 smithi162 ceph-mon[20279]: osdmap e772: 12 total, 12 up, 12 in 2024-09-02T10:28:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:59 smithi162 ceph-mon[20279]: 7.13 scrub starts 2024-09-02T10:28:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 5050/183 objects degraded (2759.563%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:27:59 smithi162 ceph-mon[20279]: osdmap e773: 12 total, 12 up, 12 in 2024-09-02T10:28:00.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:00.102 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:00.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:59 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:28:00.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:59 smithi151 ceph-mon[16639]: osdmap e772: 12 total, 12 up, 12 in 2024-09-02T10:28:00.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:59 smithi151 ceph-mon[16639]: 7.13 scrub starts 2024-09-02T10:28:00.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 5050/183 objects degraded (2759.563%), 4 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:00.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:27:59 smithi151 ceph-mon[16639]: osdmap e773: 12 total, 12 up, 12 in 2024-09-02T10:28:00.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:00.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:00.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:00.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:00.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:00.348 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:00.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:00.450 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:00.471 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:00.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:00.472 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:00.573 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:00.595 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:00.595 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:00.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:00.718 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:00.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:00.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:00.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:00.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:00.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:00.965 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:00.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:00 smithi187 ceph-mon[20072]: pgmap v1763: 57 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 51 active+clean; 4.2 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%) 2024-09-02T10:28:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:00 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:28:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:00 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:28:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:00 smithi187 ceph-mon[20072]: 7.17 scrub starts 2024-09-02T10:28:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:00 smithi187 ceph-mon[20072]: osdmap e774: 12 total, 12 up, 12 in 2024-09-02T10:28:01.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:01.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:01.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:01.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:00 smithi162 ceph-mon[20279]: pgmap v1763: 57 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 51 active+clean; 4.2 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%) 2024-09-02T10:28:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:00 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:28:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:00 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:28:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:00 smithi162 ceph-mon[20279]: 7.17 scrub starts 2024-09-02T10:28:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:00 smithi162 ceph-mon[20279]: osdmap e774: 12 total, 12 up, 12 in 2024-09-02T10:28:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:00 smithi151 ceph-mon[16639]: pgmap v1763: 57 pgs: 2 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+clean+scrubbing, 51 active+clean; 4.2 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%) 2024-09-02T10:28:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:00 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:28:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:00 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:28:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:00 smithi151 ceph-mon[16639]: 7.17 scrub starts 2024-09-02T10:28:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:00 smithi151 ceph-mon[16639]: osdmap e774: 12 total, 12 up, 12 in 2024-09-02T10:28:01.190 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:01.211 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:01.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:01.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:01.335 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:01.336 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:01.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:01.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:01.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:01.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:01.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:01.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:01.582 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:01.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:01.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:01.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:01.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:01.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:01.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:01.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:01.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:01.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:02.054 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:02.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:02.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:02.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:02.200 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:02.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:02.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:02.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:02.323 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:02.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:02.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:02.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:02.446 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:02.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:02.548 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:02.569 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:02.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:02 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:02 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:02 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:28:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:02 smithi162 ceph-mon[20279]: osdmap e775: 12 total, 12 up, 12 in 2024-09-02T10:28:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:02 smithi162 ceph-mon[20279]: pgmap v1766: 57 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 53 active+clean; 4.2 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%) 2024-09-02T10:28:02.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:02.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:02 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:02 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:02 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:28:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:02 smithi151 ceph-mon[16639]: osdmap e775: 12 total, 12 up, 12 in 2024-09-02T10:28:02.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:02 smithi151 ceph-mon[16639]: pgmap v1766: 57 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 53 active+clean; 4.2 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%) 2024-09-02T10:28:02.692 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:02.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:02 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:02.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:02 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:02.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:02 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:28:02.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:02 smithi187 ceph-mon[20072]: osdmap e775: 12 total, 12 up, 12 in 2024-09-02T10:28:02.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:02 smithi187 ceph-mon[20072]: pgmap v1766: 57 pgs: 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+clean+scrubbing, 53 active+clean; 4.2 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%) 2024-09-02T10:28:02.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:02.815 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:02.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:02.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:02.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:02.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:02.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:03.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:03.064 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:03.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:03.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:03.166 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:03.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:03.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:03.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:03.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:03.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:03.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:03.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:03.435 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:03.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:03.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:03.560 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:03.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:03 smithi162 ceph-mon[20279]: 7.11 scrub starts 2024-09-02T10:28:03.662 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:03.683 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:03.684 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:03.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:03 smithi151 ceph-mon[16639]: 7.11 scrub starts 2024-09-02T10:28:03.694 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:28:03.694 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:28:03.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:03 smithi187 ceph-mon[20072]: 7.11 scrub starts 2024-09-02T10:28:03.785 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:03.806 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:03.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:03.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:03.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:03.929 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:03.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:03.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:04.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:04.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:04.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:04.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:04.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:04.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:04.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:04.300 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:04.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:04.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:04.402 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:04.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:04.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:04.425 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:04 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:04 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:04 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:28:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:04 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:28:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:04 smithi187 ceph-mon[20072]: osdmap e776: 12 total, 12 up, 12 in 2024-09-02T10:28:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:04 smithi187 ceph-mon[20072]: pgmap v1768: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 55 active+clean; 4.2 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%); 0 B/s, 126 objects/s recovering 2024-09-02T10:28:04.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 3/183 objects degraded (1.639%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:04.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:04 smithi187 ceph-mon[20072]: osdmap e777: 12 total, 12 up, 12 in 2024-09-02T10:28:04.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:04.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:04.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:04.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:04 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:04.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:04 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:04.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:04 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:28:04.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:04 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:28:04.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:04 smithi162 ceph-mon[20279]: osdmap e776: 12 total, 12 up, 12 in 2024-09-02T10:28:04.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:04 smithi162 ceph-mon[20279]: pgmap v1768: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 55 active+clean; 4.2 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%); 0 B/s, 126 objects/s recovering 2024-09-02T10:28:04.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 3/183 objects degraded (1.639%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:04.597 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:04 smithi162 ceph-mon[20279]: osdmap e777: 12 total, 12 up, 12 in 2024-09-02T10:28:04.649 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:04.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:04.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:04.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:04 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:04 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:04 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:28:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:04 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:28:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:04 smithi151 ceph-mon[16639]: osdmap e776: 12 total, 12 up, 12 in 2024-09-02T10:28:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:04 smithi151 ceph-mon[16639]: pgmap v1768: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 55 active+clean; 4.2 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%); 0 B/s, 126 objects/s recovering 2024-09-02T10:28:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 3/183 objects degraded (1.639%), 1 pg degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:04.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:04 smithi151 ceph-mon[16639]: osdmap e777: 12 total, 12 up, 12 in 2024-09-02T10:28:04.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:04.794 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:04.795 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:04.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:04.917 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:04.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:05.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:05.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:05.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:05.143 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:05.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:05.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:05.267 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:05.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:05.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:05.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:05.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:05.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:05.514 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:05.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:05.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:05 smithi162 ceph-mon[20279]: 7.13 scrub starts 2024-09-02T10:28:05.637 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:05.660 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:05.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:05.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:05.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:05 smithi151 ceph-mon[16639]: 7.13 scrub starts 2024-09-02T10:28:05.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:05 smithi187 ceph-mon[20072]: 7.13 scrub starts 2024-09-02T10:28:05.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:05.782 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:05.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:05.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:05.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:05.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:05.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:06.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:06.031 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:06.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:06.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:06.153 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:06.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:06.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:06.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:06.277 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:06.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:06.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:06.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:06.401 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:06.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:06.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:06.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:06.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:06.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:06 smithi162 ceph-mon[20279]: pgmap v1770: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 55 active+clean; 4.2 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%); 0 B/s, 102 objects/s recovering 2024-09-02T10:28:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:06 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:28:06.627 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:06.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:06.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:06.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:06 smithi151 ceph-mon[16639]: pgmap v1770: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 55 active+clean; 4.2 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%); 0 B/s, 102 objects/s recovering 2024-09-02T10:28:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:06 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:28:06.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:06 smithi187 ceph-mon[20072]: pgmap v1770: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 55 active+clean; 4.2 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%); 0 B/s, 102 objects/s recovering 2024-09-02T10:28:06.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:06 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:28:06.772 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:06.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:06.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:06.874 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:06.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:06.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:06.899 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:07.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:07.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:07.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:07.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:07.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:07.145 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:07.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:07.248 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:07.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:07.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:07.372 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:07.397 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:07.398 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:07.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:07.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:07.525 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:07.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:07 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:28:07.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:07 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:28:07.557 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:07 smithi151 ceph-mon[16639]: 7.17 scrub starts 2024-09-02T10:28:07.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:07 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:28:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:07 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:28:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:07 smithi162 ceph-mon[20279]: 7.17 scrub starts 2024-09-02T10:28:07.598 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-09-02T10:28:07.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:07.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:07.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:07.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:07 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:28:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:07 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:28:07.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:07 smithi187 ceph-mon[20072]: 7.17 scrub starts 2024-09-02T10:28:07.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:07.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:07.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:07.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:07.910 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:07.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:08.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:08.033 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:08.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:08.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:08.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:08.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:08.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:08.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:08.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:08.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:08.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:08.410 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:08.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:08.511 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:08.534 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:08.535 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:08.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:08 smithi162 ceph-mon[20279]: pgmap v1771: 57 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 53 active+clean; 4.2 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%); 0 B/s, 86 objects/s recovering 2024-09-02T10:28:08.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:08.657 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:08.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:08.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:08.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:08 smithi151 ceph-mon[16639]: pgmap v1771: 57 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 53 active+clean; 4.2 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%); 0 B/s, 86 objects/s recovering 2024-09-02T10:28:08.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:08 smithi187 ceph-mon[20072]: pgmap v1771: 57 pgs: 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 53 active+clean; 4.2 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3/183 objects degraded (1.639%); 0 B/s, 86 objects/s recovering 2024-09-02T10:28:08.758 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:08.780 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:08.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:08.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:08.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:08.905 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:08.906 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:09.008 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:09.029 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:09.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:09.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:09.130 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:09.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:09.153 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:09.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:09.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:09.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:09.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:09.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:09.397 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:09.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:09.398 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:09.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:09.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:09.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:09.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:09 smithi162 ceph-mon[20279]: 7.11 scrub starts 2024-09-02T10:28:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:09 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:28:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:09 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:09 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 3/183 objects degraded (1.639%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:09.622 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:09.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:09.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:09.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:09 smithi151 ceph-mon[16639]: 7.11 scrub starts 2024-09-02T10:28:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:09 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:28:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:09 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:09 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:09.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 3/183 objects degraded (1.639%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:09.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:09 smithi187 ceph-mon[20072]: 7.11 scrub starts 2024-09-02T10:28:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:09 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:28:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:09 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:09 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:09.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 3/183 objects degraded (1.639%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:09.766 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:09.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:09.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:09.891 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:09.892 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:09.993 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:10.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:10.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:10.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:10.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:10.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:10.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:10.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:10.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:10.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:10.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:10.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:10.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:10.487 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:10.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:10.509 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:10.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:10 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:28:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:10 smithi162 ceph-mon[20279]: osdmap e778: 12 total, 12 up, 12 in 2024-09-02T10:28:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:10 smithi162 ceph-mon[20279]: pgmap v1773: 57 pgs: 5 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 250 objects/s recovering 2024-09-02T10:28:10.610 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:10.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:10.633 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:10 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:28:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:10 smithi151 ceph-mon[16639]: osdmap e778: 12 total, 12 up, 12 in 2024-09-02T10:28:10.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:10 smithi151 ceph-mon[16639]: pgmap v1773: 57 pgs: 5 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 250 objects/s recovering 2024-09-02T10:28:10.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:10.755 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:10.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:10.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:10 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:28:10.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:10 smithi187 ceph-mon[20072]: osdmap e778: 12 total, 12 up, 12 in 2024-09-02T10:28:10.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:10 smithi187 ceph-mon[20072]: pgmap v1773: 57 pgs: 5 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 50 active+clean; 4.2 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 250 objects/s recovering 2024-09-02T10:28:10.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:10.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:10.878 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:10.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:11.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:11.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:11.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:11.124 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:11.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:11.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:11.226 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:11.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:11.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:11.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:11.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:11.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:11.372 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:11.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:11.498 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:11.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:11.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:11.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:11 smithi162 ceph-mon[20279]: 7.13 scrub starts 2024-09-02T10:28:11.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:11.620 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:11.621 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:11.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:11 smithi151 ceph-mon[16639]: 7.13 scrub starts 2024-09-02T10:28:11.722 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:11.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:11.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:11.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:11 smithi187 ceph-mon[20072]: 7.13 scrub starts 2024-09-02T10:28:11.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:11.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:11.870 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:11.971 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:11.994 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:11.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:12.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:12.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:12.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:12.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:12.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:12.240 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:12.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:12.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:12.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:12.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:12.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:12.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:12.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:12.488 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:12.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:12.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:12 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:28:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:12 smithi162 ceph-mon[20279]: pgmap v1774: 57 pgs: 4 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 70 objects/s recovering 2024-09-02T10:28:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:12 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:28:12.611 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:12.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:12.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:12 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:28:12.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:12 smithi151 ceph-mon[16639]: pgmap v1774: 57 pgs: 4 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 70 objects/s recovering 2024-09-02T10:28:12.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:12 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:28:12.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:12.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:12.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:12 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:28:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:12 smithi187 ceph-mon[20072]: pgmap v1774: 57 pgs: 4 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 70 objects/s recovering 2024-09-02T10:28:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:12 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:28:12.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:12.860 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:12.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:12.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:12.963 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:12.984 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:12.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:12.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:13.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:13.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:13.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:13.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:13.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:13.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:13.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:13.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:13.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:13.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:13.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:13.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:13.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:13.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:13 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:28:13.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:13 smithi162 ceph-mon[20279]: 7.17 scrub starts 2024-09-02T10:28:13.608 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:13.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:13.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:13 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:28:13.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:13 smithi151 ceph-mon[16639]: 7.17 scrub starts 2024-09-02T10:28:13.710 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:13.732 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:13.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:13.733 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:13 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:28:13.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:13 smithi187 ceph-mon[20072]: 7.17 scrub starts 2024-09-02T10:28:13.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:13.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:13.857 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:13.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:13.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:13.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:13.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:14.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:14.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:14.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:14.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:14.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:14.232 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:14.233 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:14.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:14.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:14.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:14.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:14.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:14.481 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:14.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:14.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:14.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:14 smithi162 ceph-mon[20279]: pgmap v1775: 57 pgs: 3 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 4.2 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 61 objects/s recovering 2024-09-02T10:28:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:14 smithi162 ceph-mon[20279]: osdmap e779: 12 total, 12 up, 12 in 2024-09-02T10:28:14.604 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:14.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:14.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:14 smithi151 ceph-mon[16639]: pgmap v1775: 57 pgs: 3 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 4.2 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 61 objects/s recovering 2024-09-02T10:28:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:14.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:14 smithi151 ceph-mon[16639]: osdmap e779: 12 total, 12 up, 12 in 2024-09-02T10:28:14.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:14.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:14.728 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:14 smithi187 ceph-mon[20072]: pgmap v1775: 57 pgs: 3 active+clean+scrubbing, 2 active+recovering+undersized+degraded+remapped, 52 active+clean; 4.2 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 61 objects/s recovering 2024-09-02T10:28:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:14 smithi187 ceph-mon[20072]: osdmap e779: 12 total, 12 up, 12 in 2024-09-02T10:28:14.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:14.851 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:14.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:14.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:14.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:14.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:14.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:15.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:15.099 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:15.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:15.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:15.201 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:15.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:15.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:15.225 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:15.250 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:15.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:28:15.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:15.347 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:15.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:15.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:15.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:15.470 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:15.471 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:15.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:15.594 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:15.595 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:15.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:15 smithi162 ceph-mon[20279]: 7.11 scrub starts 2024-09-02T10:28:15.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:15 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:28:15.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:15 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:28:15.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:15 smithi151 ceph-mon[16639]: 7.11 scrub starts 2024-09-02T10:28:15.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:15 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:28:15.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:15 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:28:15.696 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:15.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:15.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:15 smithi187 ceph-mon[20072]: 7.11 scrub starts 2024-09-02T10:28:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:15 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:28:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:15 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:28:15.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:15.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:15.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:15.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:15.967 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:15.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:15.968 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:16.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:16.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:16.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:16.215 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:16.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:16.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:16.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:16.338 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:16.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:16.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:16.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:16.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:16.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:16.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:16.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:16.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:16 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:28:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:16 smithi162 ceph-mon[20279]: pgmap v1777: 57 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%); 0 B/s, 153 objects/s recovering 2024-09-02T10:28:16.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:16.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:16 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:28:16.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:16 smithi151 ceph-mon[16639]: pgmap v1777: 57 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%); 0 B/s, 153 objects/s recovering 2024-09-02T10:28:16.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:16.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:16 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:28:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:16 smithi187 ceph-mon[20072]: pgmap v1777: 57 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%); 0 B/s, 153 objects/s recovering 2024-09-02T10:28:16.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:16.833 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:16.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:16.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:16.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:16.958 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:17.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:17.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:17.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:17.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:17.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:17.206 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:17.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:17.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:17.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:17.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:17.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:17.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:17.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:17.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:17.577 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:17.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:17.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:17.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:17 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:28:17.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:17 smithi162 ceph-mon[20279]: 7.13 scrub starts 2024-09-02T10:28:17.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:17.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:17 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:28:17.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:17 smithi151 ceph-mon[16639]: 7.13 scrub starts 2024-09-02T10:28:17.699 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:17.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:17.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:17 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:28:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:17 smithi187 ceph-mon[20072]: 7.13 scrub starts 2024-09-02T10:28:17.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:17.823 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:17.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:17.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:17.946 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:17.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:17.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:18.048 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:18.070 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:18.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:18.071 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:18.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:18.193 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:18.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:18.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:18.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:18.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:18.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:18.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:18.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:18.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:18.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:18.543 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:18.565 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:18.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:18.566 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:18 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:28:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:18 smithi162 ceph-mon[20279]: pgmap v1778: 57 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%); 0 B/s, 80 objects/s recovering 2024-09-02T10:28:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:18 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:28:18.667 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:18 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:28:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:18 smithi151 ceph-mon[16639]: pgmap v1778: 57 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%); 0 B/s, 80 objects/s recovering 2024-09-02T10:28:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:18 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:28:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:18.689 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:18.689 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:18 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:28:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:18 smithi187 ceph-mon[20072]: pgmap v1778: 57 pgs: 1 active+recovering+undersized+remapped, 4 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 51 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%); 0 B/s, 80 objects/s recovering 2024-09-02T10:28:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:18 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:28:18.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:18.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:18.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:18.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:18.938 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:18.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:19.039 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:19.061 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:19.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:19.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:19.185 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:19.185 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:19.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:19.309 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:19.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:19.411 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:19.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:19.434 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:19.535 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:19.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:19.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:19.557 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:19 smithi162 ceph-mon[20279]: 7.17 scrub starts 2024-09-02T10:28:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:19 smithi162 ceph-mon[20279]: osdmap e780: 12 total, 12 up, 12 in 2024-09-02T10:28:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2/183 objects degraded (1.093%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:19 smithi162 ceph-mon[20279]: osdmap e781: 12 total, 12 up, 12 in 2024-09-02T10:28:19.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:19.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:19.680 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:19.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:19 smithi151 ceph-mon[16639]: 7.17 scrub starts 2024-09-02T10:28:19.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:19 smithi151 ceph-mon[16639]: osdmap e780: 12 total, 12 up, 12 in 2024-09-02T10:28:19.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2/183 objects degraded (1.093%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:19.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:19 smithi151 ceph-mon[16639]: osdmap e781: 12 total, 12 up, 12 in 2024-09-02T10:28:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:19 smithi187 ceph-mon[20072]: 7.17 scrub starts 2024-09-02T10:28:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:19 smithi187 ceph-mon[20072]: osdmap e780: 12 total, 12 up, 12 in 2024-09-02T10:28:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2/183 objects degraded (1.093%), 1 pg degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:28:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:19 smithi187 ceph-mon[20072]: osdmap e781: 12 total, 12 up, 12 in 2024-09-02T10:28:19.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:19.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:19.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:19.904 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:19.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:19.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:20.026 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:20.049 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:20.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:20.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:20.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:20.172 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:20.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:20.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:20.295 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:20.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:20.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:20.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:20.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:20.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:20.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:20.543 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:20.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:20.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:20 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:28:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:20 smithi162 ceph-mon[20279]: 7.15 continuing backfill to osd.1 from (0'0,763'9436] 7:aba22644:::benchmark_data_smithi187_120712_object25156:head to 763'9436 2024-09-02T10:28:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:20 smithi162 ceph-mon[20279]: pgmap v1781: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 53 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%); 0 B/s, 147 objects/s recovering 2024-09-02T10:28:20.645 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:20.669 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:20.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:20.670 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:20 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:28:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:20 smithi151 ceph-mon[16639]: 7.15 continuing backfill to osd.1 from (0'0,763'9436] 7:aba22644:::benchmark_data_smithi187_120712_object25156:head to 763'9436 2024-09-02T10:28:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:20 smithi151 ceph-mon[16639]: pgmap v1781: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 53 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%); 0 B/s, 147 objects/s recovering 2024-09-02T10:28:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:20 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:28:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:20 smithi187 ceph-mon[20072]: 7.15 continuing backfill to osd.1 from (0'0,763'9436] 7:aba22644:::benchmark_data_smithi187_120712_object25156:head to 763'9436 2024-09-02T10:28:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:20 smithi187 ceph-mon[20072]: pgmap v1781: 57 pgs: 1 active+recovering+undersized+remapped, 2 active+clean+scrubbing, 1 active+recovering+undersized+degraded+remapped, 53 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%); 0 B/s, 147 objects/s recovering 2024-09-02T10:28:20.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:20.793 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:20.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:20.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:20.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:20.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:21.019 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:21.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:21.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:21.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:21.168 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:21.169 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:21.270 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:21.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:21.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:21.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:21.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:21.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:21.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:21.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:21.545 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:21.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:21.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:21 smithi151 ceph-mon[16639]: 7.11 scrub starts 2024-09-02T10:28:21.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:21 smithi151 ceph-mon[16639]: 7.15 scrub starts 2024-09-02T10:28:21.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:21 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:28:21.625 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:21 smithi151 ceph-mon[16639]: osdmap e782: 12 total, 12 up, 12 in 2024-09-02T10:28:21.647 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:21.673 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:21.674 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:21 smithi187 ceph-mon[20072]: 7.11 scrub starts 2024-09-02T10:28:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:21 smithi187 ceph-mon[20072]: 7.15 scrub starts 2024-09-02T10:28:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:21 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:28:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:21 smithi187 ceph-mon[20072]: osdmap e782: 12 total, 12 up, 12 in 2024-09-02T10:28:21.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:21.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:21.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:21 smithi162 ceph-mon[20279]: 7.11 scrub starts 2024-09-02T10:28:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:21 smithi162 ceph-mon[20279]: 7.15 scrub starts 2024-09-02T10:28:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:21 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:28:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:21 smithi162 ceph-mon[20279]: osdmap e782: 12 total, 12 up, 12 in 2024-09-02T10:28:21.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:21.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:21.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:22.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:22.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:22.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:22.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:22.146 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:22.172 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:22.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:22.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:22.299 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:22.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:22.300 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:22.401 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:22.424 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:22.425 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:22.526 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:22.551 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:22.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:22.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:22 smithi187 ceph-mon[20072]: 7.13 scrub starts 2024-09-02T10:28:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:22 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:28:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:22 smithi187 ceph-mon[20072]: osdmap e783: 12 total, 12 up, 12 in 2024-09-02T10:28:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:22 smithi187 ceph-mon[20072]: pgmap v1784: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 55 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%) 2024-09-02T10:28:22.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:22.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:22.680 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:22 smithi151 ceph-mon[16639]: 7.13 scrub starts 2024-09-02T10:28:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:22 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:28:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:22 smithi151 ceph-mon[16639]: osdmap e783: 12 total, 12 up, 12 in 2024-09-02T10:28:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:22 smithi151 ceph-mon[16639]: pgmap v1784: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 55 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%) 2024-09-02T10:28:22.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:22.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:22.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:22 smithi162 ceph-mon[20279]: 7.13 scrub starts 2024-09-02T10:28:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:22 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:28:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:22 smithi162 ceph-mon[20279]: osdmap e783: 12 total, 12 up, 12 in 2024-09-02T10:28:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:22 smithi162 ceph-mon[20279]: pgmap v1784: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 55 active+clean; 4.2 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2/183 objects degraded (1.093%) 2024-09-02T10:28:22.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:22.934 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:22.935 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:23.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:23.062 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:23.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:23.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:23.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:23.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:23.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:23.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:23.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:23.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:23.443 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:23.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:23.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:23.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:23.571 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:23.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:23.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:23 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:23 smithi151 ceph-mon[16639]: 7.15 scrub starts 2024-09-02T10:28:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:23 smithi151 ceph-mon[16639]: osdmap e784: 12 total, 12 up, 12 in 2024-09-02T10:28:23.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:23.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:23 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:28:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:23 smithi187 ceph-mon[20072]: 7.15 scrub starts 2024-09-02T10:28:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:23 smithi187 ceph-mon[20072]: osdmap e784: 12 total, 12 up, 12 in 2024-09-02T10:28:23.799 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:23.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:23.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:23.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:23 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:28:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:23 smithi162 ceph-mon[20279]: 7.15 scrub starts 2024-09-02T10:28:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:23 smithi162 ceph-mon[20279]: osdmap e784: 12 total, 12 up, 12 in 2024-09-02T10:28:23.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:23.944 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:23.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:23.945 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:24.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:24.069 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:24.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:24.171 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:24.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:24.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:24.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:24.294 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:24.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:24.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:24.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:24.444 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:24.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:24.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:24.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:24.570 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:24.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:24.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:24 smithi151 ceph-mon[16639]: 7.17 scrub starts 2024-09-02T10:28:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:24 smithi151 ceph-mon[16639]: 7.19 scrub starts 2024-09-02T10:28:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:24 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:28:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:24 smithi151 ceph-mon[16639]: pgmap v1786: 57 pgs: 57 active+clean; 4.2 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2027142701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:28:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2/183 objects degraded (1.093%), 1 pg undersized (PG_DEGRADED) 2024-09-02T10:28:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:24 smithi151 ceph-mon[16639]: osdmap e785: 12 total, 12 up, 12 in 2024-09-02T10:28:24.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:24.697 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:24.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:24 smithi162 ceph-mon[20279]: 7.17 scrub starts 2024-09-02T10:28:24.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:24 smithi162 ceph-mon[20279]: 7.19 scrub starts 2024-09-02T10:28:24.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:24 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:28:24.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:24 smithi162 ceph-mon[20279]: pgmap v1786: 57 pgs: 57 active+clean; 4.2 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:24.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2027142701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:28:24.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2/183 objects degraded (1.093%), 1 pg undersized (PG_DEGRADED) 2024-09-02T10:28:24.747 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:24 smithi162 ceph-mon[20279]: osdmap e785: 12 total, 12 up, 12 in 2024-09-02T10:28:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:24 smithi187 ceph-mon[20072]: 7.17 scrub starts 2024-09-02T10:28:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:24 smithi187 ceph-mon[20072]: 7.19 scrub starts 2024-09-02T10:28:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:24 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:28:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:24 smithi187 ceph-mon[20072]: pgmap v1786: 57 pgs: 57 active+clean; 4.2 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2027142701' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:28:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2/183 objects degraded (1.093%), 1 pg undersized (PG_DEGRADED) 2024-09-02T10:28:24.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:24 smithi187 ceph-mon[20072]: osdmap e785: 12 total, 12 up, 12 in 2024-09-02T10:28:24.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:24.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:28:24.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:24.823 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:24.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:24.948 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:24.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:25.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:25.074 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:25.075 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:25.075 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:28:25.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:28:25.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:28:25.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:28:25.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:28:25.076 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:28:25.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:28:25.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:28:25.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:28:25.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:28:25.076 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:28:25.077 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:28:25.077 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:28:25.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:28:25.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:28:25.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:28:25.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:28:25.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:28:25.077 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:28:25.077 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:28:25.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:28:25.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:28:25.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:28:25.078 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:28:25.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:25.201 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:25.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:25.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:25.327 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:25.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:25.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:25.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:25.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:25.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:25.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:25.580 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:25.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:25.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:25 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/183 objects degraded (1.093%), 1 pg undersized) 2024-09-02T10:28:25.706 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:25.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:25 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/183 objects degraded (1.093%), 1 pg undersized) 2024-09-02T10:28:25.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:25.832 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:25.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:25 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/183 objects degraded (1.093%), 1 pg undersized) 2024-09-02T10:28:25.935 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:25.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:25.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:26.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:26.091 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:26.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:26.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:26.219 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:26.220 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:26.321 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:26.347 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:26.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:26.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:26.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:26.474 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:26.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:26.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:26.602 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:26.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:26 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:28:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:26 smithi151 ceph-mon[16639]: pgmap v1788: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.2 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:26.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:26.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:26.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:26 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:28:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:26 smithi187 ceph-mon[20072]: pgmap v1788: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.2 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:26.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:26 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:28:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:26 smithi162 ceph-mon[20279]: pgmap v1788: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 4.2 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:26.857 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:26.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:26.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:26.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:26.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:27.089 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:27.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:27.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:27.116 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:27.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:27.230 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:27.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:27.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:27.271 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:27.271 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:27.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:27.370 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:27.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:27.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:27.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:27.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:27.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:27.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:27.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:27.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:27 smithi151 ceph-mon[16639]: 7.11 scrub starts 2024-09-02T10:28:27.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:27 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:28:27.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:27 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:28:27.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:27.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:27.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:27 smithi187 ceph-mon[20072]: 7.11 scrub starts 2024-09-02T10:28:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:27 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:28:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:27 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:28:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:27 smithi162 ceph-mon[20279]: 7.11 scrub starts 2024-09-02T10:28:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:27 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:28:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:27 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:28:27.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:27.873 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:27.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:27.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:27.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:28.000 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:28.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:28.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:28.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:28.128 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:28.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:28.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:28.256 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:28.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:28.257 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:28.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:28.391 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:28.392 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:28.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:28.518 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:28.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:28.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:28.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:28.647 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:28.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:28 smithi151 ceph-mon[16639]: 7.15 scrub starts 2024-09-02T10:28:28.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:28 smithi151 ceph-mon[16639]: 7.13 scrub starts 2024-09-02T10:28:28.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:28 smithi151 ceph-mon[16639]: pgmap v1789: 57 pgs: 3 active+clean+scrubbing, 54 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:28.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:28 smithi187 ceph-mon[20072]: 7.15 scrub starts 2024-09-02T10:28:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:28 smithi187 ceph-mon[20072]: 7.13 scrub starts 2024-09-02T10:28:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:28 smithi187 ceph-mon[20072]: pgmap v1789: 57 pgs: 3 active+clean+scrubbing, 54 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:28.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:28.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:28 smithi162 ceph-mon[20279]: 7.15 scrub starts 2024-09-02T10:28:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:28 smithi162 ceph-mon[20279]: 7.13 scrub starts 2024-09-02T10:28:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:28 smithi162 ceph-mon[20279]: pgmap v1789: 57 pgs: 3 active+clean+scrubbing, 54 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:28.876 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:28.902 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:28.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:28.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:29.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:29.029 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:29.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:29.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:29.157 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:29.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:29.258 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:29.283 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:29.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:29.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:29.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:29.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:29.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:29.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:29.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:29.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:29.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:29.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:29.666 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:29 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:28:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:29 smithi151 ceph-mon[16639]: 7.17 scrub starts 2024-09-02T10:28:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:29 smithi151 ceph-mon[16639]: 7.19 scrub starts 2024-09-02T10:28:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:29 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:28:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:29 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:28:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:29 smithi187 ceph-mon[20072]: 7.17 scrub starts 2024-09-02T10:28:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:29 smithi187 ceph-mon[20072]: 7.19 scrub starts 2024-09-02T10:28:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:29 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:28:29.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:29.793 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:29.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:29 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:28:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:29 smithi162 ceph-mon[20279]: 7.17 scrub starts 2024-09-02T10:28:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:29 smithi162 ceph-mon[20279]: 7.19 scrub starts 2024-09-02T10:28:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:29 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:28:29.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:29.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:29.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:29.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:30.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:30.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:30.050 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:30.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:30.176 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:30.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:30.177 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:30.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:30.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:30.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:30.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:30.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:30.431 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:30.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:30.533 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:30.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:30.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:30.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:30.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:30 smithi151 ceph-mon[16639]: pgmap v1790: 57 pgs: 6 active+clean+scrubbing, 51 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 34 objects/s recovering 2024-09-02T10:28:30.688 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:30.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:30 smithi187 ceph-mon[20072]: pgmap v1790: 57 pgs: 6 active+clean+scrubbing, 51 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 34 objects/s recovering 2024-09-02T10:28:30.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:30.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:30.816 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:30 smithi162 ceph-mon[20279]: pgmap v1790: 57 pgs: 6 active+clean+scrubbing, 51 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 34 objects/s recovering 2024-09-02T10:28:30.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:30.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:30.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:30.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:31.044 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:31.070 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:31.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:31.172 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:31.198 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:31.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:31.199 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:31.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:31.325 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:31.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:31.326 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:31.427 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:31.453 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:31.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:31.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:31.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:31.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:31.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:31.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:31.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:31.709 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:31.809 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:31.835 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:31.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:31.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:31.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:31.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:32.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:32.090 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:32.091 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:32.192 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:32.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:32.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:32.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:32.345 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:32.346 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:32.447 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:32.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:32.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:32.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:32.600 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:32.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:32.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:32.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:32.727 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:32.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:32.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:32.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:32 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:28:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:32 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:28:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:32 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:28:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:32 smithi162 ceph-mon[20279]: pgmap v1791: 57 pgs: 9 active+clean+scrubbing, 48 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 30 objects/s recovering 2024-09-02T10:28:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:32 smithi162 ceph-mon[20279]: 7.11 scrub starts 2024-09-02T10:28:32.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:32.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:32 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:28:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:32 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:28:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:32 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:28:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:32 smithi151 ceph-mon[16639]: pgmap v1791: 57 pgs: 9 active+clean+scrubbing, 48 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 30 objects/s recovering 2024-09-02T10:28:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:32 smithi151 ceph-mon[16639]: 7.11 scrub starts 2024-09-02T10:28:32.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:32.982 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:32.983 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:32 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:28:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:32 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:28:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:32 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:28:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:32 smithi187 ceph-mon[20072]: pgmap v1791: 57 pgs: 9 active+clean+scrubbing, 48 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s, 30 objects/s recovering 2024-09-02T10:28:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:32 smithi187 ceph-mon[20072]: 7.11 scrub starts 2024-09-02T10:28:33.084 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:33.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:33.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:33.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:33.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:33.238 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:33.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:33.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:33.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:33.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:33.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:33.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:33.493 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:33.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:33.617 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:33.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:33.618 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:33.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:33.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:33.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:33.743 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:33 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:28:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:33 smithi187 ceph-mon[20072]: 7.15 scrub starts 2024-09-02T10:28:33.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:33 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:28:33.811 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:33 smithi151 ceph-mon[16639]: 7.15 scrub starts 2024-09-02T10:28:33.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:33 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:28:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:33 smithi162 ceph-mon[20279]: 7.15 scrub starts 2024-09-02T10:28:33.870 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:33.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:33.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:33.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:33.995 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:33.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:33.996 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:34.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:34.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:34.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:34.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:34.220 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:34.246 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:34.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:34.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:34.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:34.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:34.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:34.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:34.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:34.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:34.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:34.630 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:34.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:34.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:34.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:34.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:34.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:34 smithi151 ceph-mon[16639]: 7.13 scrub starts 2024-09-02T10:28:34.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:34 smithi151 ceph-mon[16639]: 7.19 scrub starts 2024-09-02T10:28:34.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:34 smithi151 ceph-mon[16639]: pgmap v1792: 57 pgs: 10 active+clean+scrubbing, 47 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:34.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:34 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:28:34.829 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:34 smithi151 ceph-mon[16639]: 7.17 scrub starts 2024-09-02T10:28:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:34 smithi162 ceph-mon[20279]: 7.13 scrub starts 2024-09-02T10:28:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:34 smithi162 ceph-mon[20279]: 7.19 scrub starts 2024-09-02T10:28:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:34 smithi162 ceph-mon[20279]: pgmap v1792: 57 pgs: 10 active+clean+scrubbing, 47 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:34 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:28:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:34 smithi162 ceph-mon[20279]: 7.17 scrub starts 2024-09-02T10:28:34.860 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:34.887 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:34.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:34.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:34 smithi187 ceph-mon[20072]: 7.13 scrub starts 2024-09-02T10:28:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:34 smithi187 ceph-mon[20072]: 7.19 scrub starts 2024-09-02T10:28:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:34 smithi187 ceph-mon[20072]: pgmap v1792: 57 pgs: 10 active+clean+scrubbing, 47 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:34 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:28:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:34 smithi187 ceph-mon[20072]: 7.17 scrub starts 2024-09-02T10:28:35.014 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:35.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:35.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:35.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:35.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:35.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:35.270 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:35.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:35.371 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:35.397 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:35.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:35.499 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:35.524 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:35.525 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:35.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:35.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:35.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:35.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:35.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:35.780 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:35.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:35.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:35.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:35.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:35.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:35.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:36.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:36.033 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:36.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:36.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:36.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:36.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:36.161 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:36.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:36.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:36.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:36.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:36.389 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:36.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:36.416 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:36.517 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:36.542 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:36.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:36.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:36.644 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:36.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:36.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:36.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:36.797 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:36.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:36.798 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:36.831 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:28:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:36 smithi162 ceph-mon[20279]: pgmap v1793: 57 pgs: 10 active+clean+scrubbing, 47 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:36 smithi162 ceph-mon[20279]: 7.18 scrub starts 2024-09-02T10:28:36.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:36 smithi151 ceph-mon[16639]: pgmap v1793: 57 pgs: 10 active+clean+scrubbing, 47 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:36.857 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:36 smithi151 ceph-mon[16639]: 7.18 scrub starts 2024-09-02T10:28:36.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:36.925 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:36.926 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:36 smithi187 ceph-mon[20072]: pgmap v1793: 57 pgs: 10 active+clean+scrubbing, 47 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:36 smithi187 ceph-mon[20072]: 7.18 scrub starts 2024-09-02T10:28:37.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:37.052 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:37.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:37.053 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:37.154 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:37.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:37.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:37.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:37.301 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:37.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:37.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:37.430 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:37.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:37.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:37.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:37.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:37.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:37.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:37.684 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:37.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:37.685 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:37.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:37.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:37.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:37.912 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:37.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:37 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:28:37.938 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:37.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:38.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:37 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:28:38.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:38.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:38.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:37 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:28:38.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:38.194 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:38.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:38.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:38.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:38.322 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:38.323 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:38.424 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:38.450 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:38.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:38.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:38.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:38.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:38.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:38.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:38.707 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:38.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:38.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:38.835 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:38.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:38.835 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:38.937 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:38 smithi151 ceph-mon[16639]: 7.12 scrub starts 2024-09-02T10:28:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:38 smithi151 ceph-mon[16639]: 7.11 scrub starts 2024-09-02T10:28:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:38 smithi151 ceph-mon[16639]: pgmap v1794: 57 pgs: 10 active+clean+scrubbing, 47 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:38 smithi151 ceph-mon[16639]: 7.15 scrub starts 2024-09-02T10:28:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:38 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:38 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:38.962 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:38.963 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:38 smithi187 ceph-mon[20072]: 7.12 scrub starts 2024-09-02T10:28:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:38 smithi187 ceph-mon[20072]: 7.11 scrub starts 2024-09-02T10:28:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:38 smithi187 ceph-mon[20072]: pgmap v1794: 57 pgs: 10 active+clean+scrubbing, 47 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:38 smithi187 ceph-mon[20072]: 7.15 scrub starts 2024-09-02T10:28:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:38 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:38 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:39.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:39.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:39.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:39.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:38 smithi162 ceph-mon[20279]: 7.12 scrub starts 2024-09-02T10:28:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:38 smithi162 ceph-mon[20279]: 7.11 scrub starts 2024-09-02T10:28:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:38 smithi162 ceph-mon[20279]: pgmap v1794: 57 pgs: 10 active+clean+scrubbing, 47 active+clean; 4.2 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:38 smithi162 ceph-mon[20279]: 7.15 scrub starts 2024-09-02T10:28:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:38 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:38 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:39.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:39.218 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:39.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:39.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:39.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:39.344 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:39.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:39.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:39.446 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:39.468 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:39.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:39.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:39.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:39.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:39.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:39.697 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:39.721 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:39.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:39.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:39.845 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:39.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:39 smithi151 ceph-mon[16639]: osdmap e786: 12 total, 12 up, 12 in 2024-09-02T10:28:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:39 smithi151 ceph-mon[16639]: 7.16 scrub starts 2024-09-02T10:28:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:39 smithi151 ceph-mon[16639]: 7.10 scrub starts 2024-09-02T10:28:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:39 smithi151 ceph-mon[16639]: 7.17 scrub starts 2024-09-02T10:28:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:39 smithi151 ceph-mon[16639]: 7.13 scrub starts 2024-09-02T10:28:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:39 smithi151 ceph-mon[16639]: osdmap e787: 12 total, 12 up, 12 in 2024-09-02T10:28:39.947 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:39.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:39.969 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:39.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:39 smithi187 ceph-mon[20072]: osdmap e786: 12 total, 12 up, 12 in 2024-09-02T10:28:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:39 smithi187 ceph-mon[20072]: 7.16 scrub starts 2024-09-02T10:28:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:39 smithi187 ceph-mon[20072]: 7.10 scrub starts 2024-09-02T10:28:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:39 smithi187 ceph-mon[20072]: 7.17 scrub starts 2024-09-02T10:28:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:39 smithi187 ceph-mon[20072]: 7.13 scrub starts 2024-09-02T10:28:40.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:39 smithi187 ceph-mon[20072]: osdmap e787: 12 total, 12 up, 12 in 2024-09-02T10:28:40.070 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:39 smithi162 ceph-mon[20279]: osdmap e786: 12 total, 12 up, 12 in 2024-09-02T10:28:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:39 smithi162 ceph-mon[20279]: 7.16 scrub starts 2024-09-02T10:28:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:39 smithi162 ceph-mon[20279]: 7.10 scrub starts 2024-09-02T10:28:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:39 smithi162 ceph-mon[20279]: 7.17 scrub starts 2024-09-02T10:28:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:39 smithi162 ceph-mon[20279]: 7.13 scrub starts 2024-09-02T10:28:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:39 smithi162 ceph-mon[20279]: osdmap e787: 12 total, 12 up, 12 in 2024-09-02T10:28:40.097 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:40.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:40.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:40.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:40.226 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:40.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:40.349 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:40.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:40.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:40.473 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:40.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:40.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:40.575 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:40.598 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:40.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:40.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:40 smithi151 ceph-mon[16639]: 7.19 scrub starts 2024-09-02T10:28:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:28:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:28:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:28:40.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/165052413' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:28:40.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:40.721 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:40.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:40.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:40.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:40.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:40.847 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:28:40.847 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:28:40.847 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:28:40.847 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:28:40.847 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:28:40.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:40.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:40.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:40 smithi187 ceph-mon[20072]: 7.19 scrub starts 2024-09-02T10:28:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:28:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:28:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:28:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/165052413' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:28:41.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:41.094 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:40 smithi162 ceph-mon[20279]: 7.19 scrub starts 2024-09-02T10:28:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:28:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:28:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:28:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/165052413' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:28:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:41.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:41.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:41.221 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:41.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:41.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:41.290 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-02T10:28:41.323 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:41.346 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:41.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:41.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:41.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:41.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:41.474 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:41.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:28:41.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:41.593 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_5' removed 2024-09-02T10:28:41.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:41.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:41.600 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:41.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:41.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:41.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:41.825 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:41.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:41.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:41.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:41.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4070094021' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:28:41.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/165052413' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:28:41.917 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4070094021' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:28:41.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:41 smithi151 ceph-mon[16639]: osdmap e788: 12 total, 12 up, 12 in 2024-09-02T10:28:41.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:41.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:41.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4070094021' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:28:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/165052413' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:28:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4070094021' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:28:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:41 smithi187 ceph-mon[20072]: osdmap e788: 12 total, 12 up, 12 in 2024-09-02T10:28:42.074 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:42.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4070094021' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:28:42.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/165052413' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:28:42.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4070094021' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_5", "pool2": "unique_pool_5", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:28:42.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:41 smithi162 ceph-mon[20279]: osdmap e788: 12 total, 12 up, 12 in 2024-09-02T10:28:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:42.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:42.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:42.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:42.220 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:42.221 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:42.319 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-02T10:28:42.320 INFO:tasks.radosbench:Beginning radosbench... 2024-09-02T10:28:42.321 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_6 2024-09-02T10:28:42.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_6 16 2024-09-02T10:28:42.321 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:42.343 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:42.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:42.344 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:42.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:42.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:42.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:42.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:42.590 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:42.591 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:42.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:42.718 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:42.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:42.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:42.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:42.844 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:42.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:42 smithi151 ceph-mon[16639]: 7.14 scrub starts 2024-09-02T10:28:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:42 smithi151 ceph-mon[16639]: pgmap v1799: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:42.946 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:42.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:42.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:42.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:42 smithi187 ceph-mon[20072]: 7.14 scrub starts 2024-09-02T10:28:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:42 smithi187 ceph-mon[20072]: pgmap v1799: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:43.033 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:43.071 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:43.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:42 smithi162 ceph-mon[20279]: 7.14 scrub starts 2024-09-02T10:28:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:42 smithi162 ceph-mon[20279]: pgmap v1799: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:43.097 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:43.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:43.198 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:43.222 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:43.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:43.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:43.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:43.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:43.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:43.448 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:43.472 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:43.473 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:43.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:43.599 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:43.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:43.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:43.701 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:43.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:43.725 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:43.826 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:43.852 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:43.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:43.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:43.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-09-02T10:28:43.954 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:43.979 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:43.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:43.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:44.007 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:44.007 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:44.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:44.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:44.108 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:44.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:44.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:44.231 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:44.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:44.333 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:44.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:44.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:44.360 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:44.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:44.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:44.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:44.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:44.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:44.631 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:44.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:44.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:44 smithi151 ceph-mon[16639]: pgmap v1800: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:44.734 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:44 smithi151 ceph-mon[16639]: osdmap e789: 12 total, 12 up, 12 in 2024-09-02T10:28:44.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:44.758 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:44.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:44.880 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:44.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:44.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:44 smithi187 ceph-mon[20072]: pgmap v1800: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:44 smithi187 ceph-mon[20072]: osdmap e789: 12 total, 12 up, 12 in 2024-09-02T10:28:45.008 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:45.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:44 smithi162 ceph-mon[20279]: pgmap v1800: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:44 smithi162 ceph-mon[20279]: osdmap e789: 12 total, 12 up, 12 in 2024-09-02T10:28:45.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:45.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:45.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:45.137 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:45.205 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:136029 6.0 6:0a2545ae:::benchmark_data_smithi187_102448_object35757:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 472.06756705200002, 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34708892600000002, 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:28:45.206 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136029 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135378+0000", 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1500000000000001e-07 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135380+0000", 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2359999999999999e-06 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135383+0000", 2024-09-02T10:28:45.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.641e-06 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135386+0000", 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7920000000000001e-06 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136954+0000", 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001567492 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:28:45.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.136959+0000", 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.49e-06 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455542+0000", 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31858257099999998 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455661+0000", 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119422 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.455779+0000", 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011832699999999999 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.464952+0000", 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0091722350000000008 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.465554+0000", 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060224400000000004 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.479463+0000", 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.013909425 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.479532+0000", 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8818000000000004e-05 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.482467+0000", 2024-09-02T10:28:45.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0029350180000000002 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:136038 6.0 6:0e293183:::benchmark_data_smithi187_102448_object35766:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e586)", 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:53.135389+0000", 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 472.067555814, 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34942438399999998, 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:28:45.213 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136038 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135389+0000", 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135389+0000", 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135390+0000", 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1299999999999999e-07 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135391+0000", 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.249e-06 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135395+0000", 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2150000000000003e-06 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.135399+0000", 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4649999999999999e-06 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.137534+0000", 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021359560000000001 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:28:45.216 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.137545+0000", 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0312999999999999e-05 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.460231+0000", 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32268618199999999 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.460387+0000", 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015636900000000001 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,3", 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.460536+0000", 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014892599999999999 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.465596+0000", 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050595570000000001 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.465617+0000", 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1362e-05 2024-09-02T10:28:45.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.482532+0000", 2024-09-02T10:28:45.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016914565999999999 2024-09-02T10:28:45.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:28:45.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.482602+0000", 2024-09-02T10:28:45.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9809999999999997e-05 2024-09-02T10:28:45.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:53.484814+0000", 2024-09-02T10:28:45.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002212201 2024-09-02T10:28:45.220 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.220 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.220 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e640,586 6:01803c4b:::benchmark_data_smithi187_102448_object10256:head,576'6972)", 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:02.039244+0000", 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 343.16370076499999, 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.89444717100000004, 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.039244+0000", 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.039244+0000", 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.039246+0000", 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.077e-06 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.039250+0000", 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4429999999999996e-06 2024-09-02T10:28:45.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.931762+0000", 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.89251206999999999 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.931772+0000", 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0360000000000001e-05 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.931837+0000", 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4980999999999993e-05 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.933692+0000", 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018542400000000001 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e646,586 6:0a36e23f:::benchmark_data_smithi187_102448_object40966:head,589'8871)", 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:24.826375+0000", 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 320.37657043600001, 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.94590308400000001, 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.224 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:24.826375+0000", 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:24.826375+0000", 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:24.826375+0000", 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4899999999999995e-07 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:24.826379+0000", 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9910000000000004e-06 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:25.770077+0000", 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.94369714100000002 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:25.770090+0000", 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3488e-05 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:25.770173+0000", 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2980999999999997e-05 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:25.772278+0000", 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021048339999999999 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.227 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e646,586 6:0b3a9c49:::benchmark_data_smithi187_102448_object41773:head,590'8922)", 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:26.221623+0000", 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 318.98132183799999, 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.95374069100000003, 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:26.221623+0000", 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:26.221623+0000", 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:26.221625+0000", 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3379999999999999e-06 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:26.221633+0000", 2024-09-02T10:28:45.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9519999999999994e-06 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:27.173768+0000", 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.95213511100000003 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:27.173796+0000", 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7767999999999999e-05 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:27.173856+0000", 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0217000000000003e-05 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:27.175364+0000", 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015083049999999999 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.231 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e653,586 6:0ef61dca:::benchmark_data_smithi187_102448_object80200:head,615'11351)", 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:47.999763+0000", 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 297.203182529, 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48810655800000002, 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:47.999763+0000", 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:47.999763+0000", 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:47.999763+0000", 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:47.999767+0000", 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1019999999999999e-06 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.233 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.486785+0000", 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48701780300000003 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.486805+0000", 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9879999999999999e-05 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.486938+0000", 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013259799999999999 2024-09-02T10:28:45.234 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.487869+0000", 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093163600000000001 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e655,586 6:03970f42:::benchmark_data_smithi187_102448_object92390:head,622'12173)", 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:51.641573+0000", 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 293.56137241099998, 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62893131199999996, 2024-09-02T10:28:45.235 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:51.641573+0000", 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:51.641573+0000", 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:51.641573+0000", 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3499999999999996e-07 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:51.641577+0000", 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0990000000000001e-06 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:52.269760+0000", 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62818215200000005 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:52.269777+0000", 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7750000000000001e-05 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:52.269903+0000", 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125785 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:52.270504+0000", 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00060099100000000005 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.238 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(6.0 e655,586 6:001e5900:::benchmark_data_smithi187_102448_object93843:head,623'12275)", 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:52.560921+0000", 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 292.642024139, 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46799331999999999, 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.239 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:52.560921+0000", 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:52.560921+0000", 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:52.560923+0000", 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.454e-06 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:52.560932+0000", 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3819999999999993e-06 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:53.028211+0000", 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.46727914100000001 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:53.028220+0000", 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1619999999999997e-06 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:53.028330+0000", 2024-09-02T10:28:45.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010935299999999999 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:53.028914+0000", 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00058482800000000002 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:71625 7.6 7:6b509032:::benchmark_data_smithi187_120712_object71624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e711)", 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:27.814721+0000", 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 197.38822419499999, 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.40169136599999999, 2024-09-02T10:28:45.242 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 71625 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.814721+0000", 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.814721+0000", 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.814723+0000", 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7740000000000001e-06 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.814766+0000", 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3624000000000001e-05 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.814869+0000", 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000102983 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.814881+0000", 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1328000000000001e-05 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.815728+0000", 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00084696100000000002 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:28:45.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.815898+0000", 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017059 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.816082+0000", 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00018364299999999999 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.819214+0000", 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031320710000000002 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.825861+0000", 2024-09-02T10:28:45.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.006647043 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.215213+0000", 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38935154300000002 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.215265+0000", 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2617999999999997e-05 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.216412+0000", 2024-09-02T10:28:45.247 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011471879999999999 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:71625 7.6 e711/694 ondisk, result = 0)", 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:27.819320+0000", 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 197.383625284, 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39710403599999999, 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.248 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.819320+0000", 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.819320+0000", 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.819320+0000", 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8899999999999999e-07 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.249 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.819324+0000", 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3550000000000001e-06 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.212725+0000", 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39340159600000002 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.212733+0000", 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0790000000000003e-06 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.250 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.215197+0000", 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002464047 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.215207+0000", 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0049e-05 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.216424+0000", 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0012173209999999999 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.251 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135855 7.16 7:68fd4dab:::benchmark_data_smithi187_120712_object51913:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.785399+0000", 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 123.417546122, 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0146758810000001, 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135855 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.252 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785399+0000", 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785399+0000", 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785399+0000", 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.01e-07 2024-09-02T10:28:45.253 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785402+0000", 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1490000000000001e-06 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785408+0000", 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5370000000000003e-06 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785413+0000", 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9710000000000003e-06 2024-09-02T10:28:45.254 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.047551+0000", 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26213762000000002 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.047559+0000", 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6810000000000004e-06 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.799983+0000", 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75242376399999999 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.255 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.800075+0000", 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1757999999999998e-05 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135862 7.14 7:2bd061c1:::benchmark_data_smithi187_120712_object51920:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.785429+0000", 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 123.417516613, 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.035955309, 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.256 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135862 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785429+0000", 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.257 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785429+0000", 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785429+0000", 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8099999999999998e-07 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785431+0000", 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1780000000000002e-06 2024-09-02T10:28:45.258 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785438+0000", 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.562e-06 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785443+0000", 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9389999999999996e-06 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.004860+0000", 2024-09-02T10:28:45.259 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21941693500000001 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.004869+0000", 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8810000000000008e-06 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821304+0000", 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.81643467300000006 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821384+0000", 2024-09-02T10:28:45.260 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9759999999999995e-05 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135864 7.14 7:2e1c8040:::benchmark_data_smithi187_120712_object51922:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.785449+0000", 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 123.417496632, 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0362721509999999, 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:28:45.261 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135864 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785449+0000", 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785449+0000", 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.262 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785449+0000", 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4700000000000002e-07 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785451+0000", 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2249999999999999e-06 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785457+0000", 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0190000000000003e-06 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.263 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.785462+0000", 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2989999999999996e-06 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.004890+0000", 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21942751499999999 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.004896+0000", 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8699999999999997e-06 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.264 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821565+0000", 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.81666898300000001 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821721+0000", 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015589299999999999 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.265 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135870 7.14 7:2968342d:::benchmark_data_smithi187_120712_object51928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.835745+0000", 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 123.3672, 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.99478446600000003, 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135870 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.266 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.835745+0000", 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.835745+0000", 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.835748+0000", 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3369999999999998e-06 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.835762+0000", 2024-09-02T10:28:45.267 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4072e-05 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.835800+0000", 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7948000000000003e-05 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.835815+0000", 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5290000000000001e-05 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.004916+0000", 2024-09-02T10:28:45.268 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16910067500000001 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.004920+0000", 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8489999999999997e-06 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821744+0000", 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.81682342900000005 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821835+0000", 2024-09-02T10:28:45.269 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1132999999999996e-05 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821934+0000", 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9388000000000005e-05 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.829740+0000", 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0078055060000000002 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.829828+0000", 2024-09-02T10:28:45.270 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7975999999999999e-05 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.830389+0000", 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056124099999999998 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.830440+0000", 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1028e-05 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.271 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.830530+0000", 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9593999999999996e-05 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135873 7.16 7:6bf1f557:::benchmark_data_smithi187_120712_object51931:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:42.006784+0000", 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 123.19616126699999, 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.80843002399999997, 2024-09-02T10:28:45.272 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.273 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.273 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:28:45.273 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:28:45.273 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:28:45.273 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:28:45.273 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135873 2024-09-02T10:28:45.273 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.273 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.273 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.273 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:45.274 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.006784+0000", 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.006784+0000", 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.006786+0000", 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.697e-06 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.275 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.006791+0000", 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.899e-06 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.006818+0000", 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7424e-05 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.006831+0000", 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3488e-05 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.276 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.047578+0000", 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.040746676000000003 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.047583+0000", 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7859999999999996e-06 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.800530+0000", 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75294724000000002 2024-09-02T10:28:45.277 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.800655+0000", 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124602 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.800775+0000", 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012048399999999999 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.802885+0000", 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021100170000000001 2024-09-02T10:28:45.278 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.806470+0000", 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035847140000000001 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.815082+0000", 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.008612471 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.815171+0000", 2024-09-02T10:28:45.279 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8211e-05 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.815214+0000", 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3315000000000001e-05 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:136912 7.f 7:fcec4aa2:::benchmark_data_smithi187_120712_object52970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.757981+0000", 2024-09-02T10:28:45.280 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 121.44496391, 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34458326700000003, 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136912 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.281 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.757981+0000", 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.757981+0000", 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.757983+0000", 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.68e-06 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.282 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.757990+0000", 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5699999999999998e-06 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758002+0000", 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2933e-05 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758009+0000", 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9970000000000001e-06 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.283 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758073+0000", 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3169000000000005e-05 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758217+0000", 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014449800000000001 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9", 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758330+0000", 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011268 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.284 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.762167+0000", 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038371009999999999 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.762182+0000", 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4882000000000001e-05 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.102474+0000", 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34029198900000002 2024-09-02T10:28:45.285 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.102519+0000", 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5707000000000001e-05 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.102565+0000", 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5061000000000001e-05 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.286 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:154719 7.6 7:65814d4b:::benchmark_data_smithi187_120712_object70777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:17.825670+0000", 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 87.377274955000004, 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35717151499999999, 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 154719 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.287 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.825670+0000", 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.825670+0000", 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.825672+0000", 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6220000000000001e-06 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.288 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.825677+0000", 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2100000000000001e-06 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.825687+0000", 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7759999999999992e-06 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.825694+0000", 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3699999999999997e-06 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.289 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.830749+0000", 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050547589999999998 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.897726+0000", 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.066976890999999997 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 3,7", 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.897840+0000", 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011455699999999999 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:28:45.290 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.902723+0000", 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048828220000000002 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.903631+0000", 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00090824099999999995 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.172116+0000", 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.268484373 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:28:45.291 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.182797+0000", 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010681342999999999 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.182842+0000", 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4551000000000001e-05 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:28:45.292 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:28:45.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:45.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:45.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:45.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:45.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:45.538 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:45.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:45.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:45.621 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:28:45.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:45.682 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:45.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:45.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:45.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:45.805 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:45.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:45.899 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:45 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:45.900 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:45 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:45.900 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2650826935' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:28:45.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:45.932 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:45.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:45.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:45 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:45 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2650826935' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:28:46.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:46.059 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:46.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:46.060 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:46.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:45 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:46.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:45 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:46.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2650826935' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:28:46.161 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:46.186 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:46.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:46.288 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:46.314 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:46.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:46.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:46.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:46.443 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:46.544 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:46.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:46.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:46.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:46.695 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:46.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:46.736 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:46.796 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:46.821 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:46.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:46.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:46 smithi151 ceph-mon[16639]: pgmap v1802: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:46.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:46 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:28:46.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2650826935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:28:46.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:46 smithi151 ceph-mon[16639]: osdmap e790: 12 total, 12 up, 12 in 2024-09-02T10:28:46.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:46.949 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:46.950 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:46 smithi187 ceph-mon[20072]: pgmap v1802: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:46 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:28:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2650826935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:28:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:46 smithi187 ceph-mon[20072]: osdmap e790: 12 total, 12 up, 12 in 2024-09-02T10:28:47.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:47.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:47.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:47.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:46 smithi162 ceph-mon[20279]: pgmap v1802: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:46 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:28:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2650826935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:28:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:46 smithi162 ceph-mon[20279]: osdmap e790: 12 total, 12 up, 12 in 2024-09-02T10:28:47.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:47.201 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:47.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:47.303 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:47.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:47.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:47.427 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:47.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:47.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:47.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:47.551 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:28:47.552 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:28:47.556 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:28:47.556 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:28:47.576 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:28:47.577 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:28:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:47.578 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:47.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:47.598 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:28:47.598 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:28:47.619 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:28:47.619 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:28:47.641 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:28:47.641 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:28:47.662 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:28:47.663 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:28:47.679 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:47 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:28:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:47 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:28:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:47 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:28:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:47 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:28:47.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2192845256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:28:47.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:47.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:47.705 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:47.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:47.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:47.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:47.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:47.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:47.956 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:47.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:47.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:47 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:28:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:47 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:28:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:47 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:28:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:47 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:28:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2192845256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:28:48.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:48.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:48.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:47 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:28:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:47 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:28:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:47 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:28:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:47 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:28:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2192845256' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:28:48.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:48.210 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:48.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:48.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:48.336 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:48.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:48.438 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:48.463 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:48.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:48.464 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:48.487 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:28:48.488 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:28:48.510 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:28:48.510 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:28:48.531 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:28:48.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:28:48.554 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:28:48.554 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:28:48.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:48.575 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:28:48.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:28:48.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:48.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:28:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:28:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: pgmap v1804: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:28:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:28:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:48.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:28:48.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:48 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:28:48.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:48.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:28:48.714 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:48.715 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:48.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:48.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:48.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:48.943 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:48.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:48.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: pgmap v1804: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:28:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:49.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:49.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:28:49.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:28:49.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:49.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:28:49.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:28:49.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:48 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:28:49.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:49.092 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:49.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:49.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:28:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:28:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: pgmap v1804: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:28:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:28:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:28:49.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:28:49.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:48 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:28:49.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:28:49.195 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:49.220 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:49.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:49.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:49.280 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:49.322 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:49.346 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:49.347 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:49.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:49.474 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:49.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:49.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:49.601 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:49.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:49.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:49.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:49.728 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:49.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:49.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:49.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:49 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:28:49.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:49 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:28:49.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:49 smithi151 ceph-mon[16639]: osdmap e791: 12 total, 12 up, 12 in 2024-09-02T10:28:49.827 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:49 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:28:49.828 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:49 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:28:49.830 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:49.854 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:49.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:49.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:49.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:49.982 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:49.983 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:49 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:28:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:49 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:28:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:49 smithi187 ceph-mon[20072]: osdmap e791: 12 total, 12 up, 12 in 2024-09-02T10:28:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:49 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:28:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:49 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:28:50.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:49 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:28:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:49 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:28:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:49 smithi162 ceph-mon[20279]: osdmap e791: 12 total, 12 up, 12 in 2024-09-02T10:28:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:49 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:28:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:49 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:28:50.110 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:50.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:50.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:50.236 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:50.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:50.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:50.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:50.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:50.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:50.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:50.415 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:50.430 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:50.465 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:50.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:50.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:50.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:50.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:50.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:50.642 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-02T10:28:50.642 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_6' created 2024-09-02T10:28:50.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:50.738 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:50.739 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:50.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:50.864 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:50.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:50 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:28:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:50 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:28:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:50 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:28:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:50 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:28:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:50 smithi151 ceph-mon[16639]: pgmap v1806: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1856933860' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-02T10:28:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/565271624' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:28:50.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:50.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:50.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:50.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:50 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:28:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:50 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:28:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:50 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:28:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:50 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:28:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:50 smithi187 ceph-mon[20072]: pgmap v1806: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1856933860' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-02T10:28:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/565271624' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:28:51.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:50 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:28:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:50 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:28:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:50 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:28:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:50 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:28:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:50 smithi162 ceph-mon[20279]: pgmap v1806: 31 pgs: 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1856933860' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16} : dispatch 2024-09-02T10:28:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/565271624' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:28:51.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:51.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:51.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:51.240 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:51.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:51.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:28:51.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:51.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:51.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:51.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:51.494 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:51.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:51.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:51.596 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:51.622 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:51.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:51.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:51.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:51.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:51.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:51.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:51.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:51.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:28:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:28:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1856933860' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-02T10:28:51.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/565271624' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:28:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: osdmap e792: 12 total, 12 up, 12 in 2024-09-02T10:28:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:28:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:28:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:51.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:51 smithi151 ceph-mon[16639]: osdmap e793: 12 total, 12 up, 12 in 2024-09-02T10:28:51.970 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:51.996 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:51.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:28:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:28:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1856933860' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-02T10:28:52.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/565271624' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:28:52.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: osdmap e792: 12 total, 12 up, 12 in 2024-09-02T10:28:52.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:28:52.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:28:52.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:52.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:52.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:51 smithi187 ceph-mon[20072]: osdmap e793: 12 total, 12 up, 12 in 2024-09-02T10:28:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:28:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:28:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1856933860' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_6", "pg_num": 16}]': finished 2024-09-02T10:28:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/565271624' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:28:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: osdmap e792: 12 total, 12 up, 12 in 2024-09-02T10:28:52.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:28:52.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:28:52.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:52.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:52.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:51 smithi162 ceph-mon[20279]: osdmap e793: 12 total, 12 up, 12 in 2024-09-02T10:28:52.098 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:52.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:52.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:52.222 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:52.244 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:52.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:52.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:52.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:52.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:52.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:52.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:52.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:52.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:52.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:52.619 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:52.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:52.621 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:52.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:52.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:52.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:52.747 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:52.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_6 rados --yes-i-really-mean-it || true 2024-09-02T10:28:52.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:52.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:52.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:52 smithi151 ceph-mon[16639]: pgmap v1808: 47 pgs: 16 unknown, 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:52 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:28:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:52 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:28:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:52 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:28:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:52 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:28:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:52 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:28:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:52 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:28:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:52 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:28:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:52 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:28:52.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:52 smithi151 ceph-mon[16639]: osdmap e794: 12 total, 12 up, 12 in 2024-09-02T10:28:52.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:52.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:52.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:52.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:52 smithi187 ceph-mon[20072]: pgmap v1808: 47 pgs: 16 unknown, 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:52 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:28:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:52 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:28:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:52 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:28:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:52 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:28:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:52 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:28:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:52 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:28:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:52 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:28:53.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:52 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:28:53.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:52 smithi187 ceph-mon[20072]: osdmap e794: 12 total, 12 up, 12 in 2024-09-02T10:28:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:52 smithi162 ceph-mon[20279]: pgmap v1808: 47 pgs: 16 unknown, 31 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:52 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:28:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:52 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:28:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:52 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:28:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:52 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:28:53.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:52 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:28:53.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:52 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:28:53.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:52 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:28:53.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:52 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:28:53.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:52 smithi162 ceph-mon[20279]: osdmap e794: 12 total, 12 up, 12 in 2024-09-02T10:28:53.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:53.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:53.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:53.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:53.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:53.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:53.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:53.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:53.380 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:53.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:53.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:53.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-09-02T10:28:53.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:53.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:53.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:53.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:53.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:53.648 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:53.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:53.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:53.667 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:28:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:53 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:28:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:53 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:28:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3179166319' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:28:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:53 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:28:53.683 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:53 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:28:53.705 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:28:53.705 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:28:53.750 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:53.771 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:53.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:53.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:53.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:53.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:53.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:54.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:53 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:28:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:53 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:28:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3179166319' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:28:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:53 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:28:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:53 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:28:54.026 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:54.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:54.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:53 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:28:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:53 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:28:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3179166319' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:28:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:53 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:28:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:53 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:28:54.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:54.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:54.154 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:54.154 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:54.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:54.155 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:54.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:54.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:54.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:54.382 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:54.407 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:54.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:54.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:54.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:54.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:54.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:54.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:54.660 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:54.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:54.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:54.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:28:54.761 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:54.786 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:54.787 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:54.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:54.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:54.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:54.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: pgmap v1811: 47 pgs: 5 unknown, 42 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3179166319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: osdmap e795: 12 total, 12 up, 12 in 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:28:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:54 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:28:54.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:28:54.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:28:54.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: pgmap v1811: 47 pgs: 5 unknown, 42 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:54.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:28:54.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:28:54.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3179166319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:28:54.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: osdmap e795: 12 total, 12 up, 12 in 2024-09-02T10:28:54.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:54.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:54.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:28:54.997 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:54 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:28:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:28:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:28:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: pgmap v1811: 47 pgs: 5 unknown, 42 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:28:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:28:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3179166319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:28:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: osdmap e795: 12 total, 12 up, 12 in 2024-09-02T10:28:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:55.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:28:55.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:54 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:28:55.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:55.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:55.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:55.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:28:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:55.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:55.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:28:55.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:55.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:55.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:55.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:55.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:55.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:55.511 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:55.537 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:55.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:55.639 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:55.640 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:55.664 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:55.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:55.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:55.790 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:55.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:55.891 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:55 smithi151 ceph-mon[16639]: osdmap e796: 12 total, 12 up, 12 in 2024-09-02T10:28:55.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:55.916 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:55.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:56.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:55 smithi187 ceph-mon[20072]: osdmap e796: 12 total, 12 up, 12 in 2024-09-02T10:28:56.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:56.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:56.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:55 smithi162 ceph-mon[20279]: osdmap e796: 12 total, 12 up, 12 in 2024-09-02T10:28:56.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:28:56.121 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:28:56.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:56.145 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:28:56.148 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:28:56.165 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:56.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:56.167 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:28:56.167 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:28:56.189 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:28:56.189 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:28:56.210 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:28:56.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:28:56.232 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:28:56.232 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:28:56.256 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:28:56.256 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:28:56.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:56.277 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:28:56.277 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:28:56.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:56.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:56.299 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:28:56.299 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:28:56.320 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:28:56.321 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:28:56.341 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:28:56.341 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:28:56.362 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:28:56.363 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:28:56.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:28:56.412 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:56.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:28:56.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:56.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:56.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:56.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:56.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:56.665 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:56.686 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_6' 2024-09-02T10:28:56.689 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:28:56.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:56.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:56.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:56.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:28:56.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:28:56.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: pgmap v1814: 47 pgs: 47 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:56.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:28:56.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:28:56.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4217615060' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:28:56.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1393158680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:28:56.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:56.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:56.800 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:28:56.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:28:56.801 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:56 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:56.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:56.915 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:56.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: pgmap v1814: 47 pgs: 47 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4217615060' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1393158680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:28:57.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:56 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:57.017 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:57.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:57.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: pgmap v1814: 47 pgs: 47 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4217615060' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1393158680' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:28:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:28:57.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:56 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:28:57.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:57.165 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:57.166 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:57.267 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:57.292 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:57.293 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:57.394 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:28:57.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:57.421 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:28:57.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:57.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:57.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:57.646 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:57.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:28:57.672 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:57.672 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:57.686 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-02T10:28:57.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:28:57.799 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:57.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:28:57.901 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:57.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:57.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:57.928 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4217615060' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:28:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:57 smithi151 ceph-mon[16639]: osdmap e797: 12 total, 12 up, 12 in 2024-09-02T10:28:57.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2365336261' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:28:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4217615060' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:28:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:57 smithi187 ceph-mon[20072]: osdmap e797: 12 total, 12 up, 12 in 2024-09-02T10:28:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2365336261' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:28:58.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:58.054 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:58.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4217615060' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_6", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:28:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:57 smithi162 ceph-mon[20279]: osdmap e797: 12 total, 12 up, 12 in 2024-09-02T10:28:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2365336261' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:28:58.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:28:58.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:58.182 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:28:58.209 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:28:58.209 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:28:58.209 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:28:58.209 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:28:58.209 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:28:58.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:58.308 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:58.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:58.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:58.410 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:28:58.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:58.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:58.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:28:58.534 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:58.560 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:58.561 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:58.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:58.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:58.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:28:58.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:58.813 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:58.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:58.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:58.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:58 smithi151 ceph-mon[16639]: pgmap v1816: 47 pgs: 47 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:58.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2365336261' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:28:58.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:58 smithi151 ceph-mon[16639]: osdmap e798: 12 total, 12 up, 12 in 2024-09-02T10:28:58.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:58 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:28:58.847 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:58 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:28:58.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:28:58.936 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:58.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:58.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:28:58.997 DEBUG:teuthology.orchestra.run.smithi187:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_6 bench 90 write' 2024-09-02T10:28:58.999 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-02T10:28:59.000 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-02T10:28:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:58 smithi187 ceph-mon[20072]: pgmap v1816: 47 pgs: 47 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2365336261' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:28:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:58 smithi187 ceph-mon[20072]: osdmap e798: 12 total, 12 up, 12 in 2024-09-02T10:28:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:58 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:28:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:58 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:28:59.038 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:59.062 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:59.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:59.063 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:58 smithi162 ceph-mon[20279]: pgmap v1816: 47 pgs: 47 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:28:59.133 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2365336261' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:28:59.133 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:58 smithi162 ceph-mon[20279]: osdmap e798: 12 total, 12 up, 12 in 2024-09-02T10:28:59.133 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:58 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:28:59.133 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:58 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:28:59.134 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:hints = 1 2024-09-02T10:28:59.134 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-02T10:28:59.134 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object prefix: benchmark_data_smithi187_136059 2024-09-02T10:28:59.134 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:28:59.134 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 0 0 0 0 0 0 - 0 2024-09-02T10:28:59.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:59.191 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:59.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:59.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:59.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:28:59.320 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:59.321 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:28:59.422 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:28:59.444 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:59.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:28:59.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:28:59.546 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:59.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:59.574 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:59.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:28:59.697 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:28:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:59.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:28:59.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:28:59.824 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:28:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:59.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:28:59.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:28:59.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:28:59.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:28:59.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:28:59 smithi187 ceph-mon[20072]: osdmap e799: 12 total, 12 up, 12 in 2024-09-02T10:29:00.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:00.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:00.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:00.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:00.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:28:59 smithi162 ceph-mon[20279]: osdmap e799: 12 total, 12 up, 12 in 2024-09-02T10:29:00.113 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 1 16 1776 1760 109.994 110 0.053491 0.00874324 2024-09-02T10:29:00.180 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:00.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:28:59 smithi151 ceph-mon[16639]: osdmap e799: 12 total, 12 up, 12 in 2024-09-02T10:29:00.201 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:00.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:00.303 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:00.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:00.328 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:00.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-09-02T10:29:00.412 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:00.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:00.446 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:00.450 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:00.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:00.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:00.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:00.578 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:00.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:00.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:00.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:00.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:00.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:00.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:00.811 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:00.835 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:00.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:00.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:00.875 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:29:00.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:00.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:00 smithi151 ceph-mon[16639]: pgmap v1819: 47 pgs: 47 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:29:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:00 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:29:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:00 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:29:00.958 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:00.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:00.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:00 smithi187 ceph-mon[20072]: pgmap v1819: 47 pgs: 47 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:29:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:00 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:29:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:00 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:29:01.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:01.083 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:01.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:01.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:00 smithi162 ceph-mon[20279]: pgmap v1819: 47 pgs: 47 active+clean; 577 KiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:29:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:00 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:29:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:00 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:29:01.113 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 2 16 3342 3326 103.931 97.875 0.00573042 0.00957816 2024-09-02T10:29:01.185 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:01.207 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:01.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:01.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:01.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:01.331 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:01.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:01.417 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:29:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:29:01.417 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:24824 7.b 7:dcd0dd6d:::benchmark_data_smithi187_120712_object24823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:33.128771+0000", 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 268.28303237699998, 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21211126899999999, 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.418 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 24824 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128771+0000", 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128771+0000", 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.419 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128773+0000", 2024-09-02T10:29:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.95e-06 2024-09-02T10:29:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128832+0000", 2024-09-02T10:29:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8831e-05 2024-09-02T10:29:01.420 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128844+0000", 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2425999999999999e-05 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128851+0000", 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.289e-06 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.421 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128915+0000", 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4090000000000005e-05 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.129050+0000", 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013484 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.129182+0000", 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000132194 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.422 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.130289+0000", 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011068479999999999 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.130725+0000", 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00043614899999999999 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.340817+0000", 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.210091533 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.423 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.340853+0000", 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6480999999999999e-05 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.340882+0000", 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8637999999999999e-05 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.424 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:24845 7.b 7:dee0a0e0:::benchmark_data_smithi187_120712_object24844:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:33.144319+0000", 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 268.26748374800002, 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.197350796, 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 24845 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.425 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.144319+0000", 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.144319+0000", 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.144321+0000", 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.389e-06 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.426 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.144343+0000", 2024-09-02T10:29:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2099e-05 2024-09-02T10:29:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.144355+0000", 2024-09-02T10:29:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1849e-05 2024-09-02T10:29:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.427 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.144360+0000", 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.524e-06 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.144413+0000", 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2673999999999997e-05 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.144571+0000", 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015765899999999999 2024-09-02T10:29:01.428 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.144727+0000", 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015673299999999999 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.145656+0000", 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000928179 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.146310+0000", 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00065435199999999997 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.429 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.341613+0000", 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19530354999999999 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.341646+0000", 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2199000000000001e-05 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.341670+0000", 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4589000000000001e-05 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.430 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:25297 7.b 7:d6727adf:::benchmark_data_smithi187_120712_object25296:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:33.574550+0000", 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 267.83725364899999, 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.196644341, 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 25297 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.431 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.574550+0000", 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.574550+0000", 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.574552+0000", 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0020000000000001e-06 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.574671+0000", 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119335 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.574688+0000", 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7428999999999999e-05 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.574696+0000", 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0229999999999992e-06 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.574767+0000", 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0740000000000004e-05 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.574886+0000", 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119417 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.575028+0000", 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014201900000000001 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.576182+0000", 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001153196 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.577494+0000", 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001312782 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.771097+0000", 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19360256000000001 2024-09-02T10:29:01.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.771149+0000", 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2141000000000001e-05 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.771194+0000", 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4697000000000003e-05 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.436 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:25297 7.b e675/661 ondisk, result = 0)", 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:33.576449+0000", 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 267.83535422599999, 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19475184200000001, 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.576449+0000", 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.437 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.576449+0000", 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.576450+0000", 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6400000000000001e-07 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.576454+0000", 2024-09-02T10:29:01.438 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4610000000000001e-06 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.771045+0000", 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.194590338 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.771051+0000", 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4779999999999996e-06 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.439 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.771085+0000", 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3549e-05 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.771092+0000", 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4399999999999999e-06 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.771201+0000", 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000108812 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.440 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:45635 7.b 7:d320ff39:::benchmark_data_smithi187_120712_object45634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.398280+0000", 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 248.01352341500001, 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24220898799999999, 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.441 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45635 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398280+0000", 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398280+0000", 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.442 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398282+0000", 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8309999999999999e-06 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398330+0000", 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8127000000000002e-05 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398345+0000", 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5690999999999999e-05 2024-09-02T10:29:01.443 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398352+0000", 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8079999999999999e-06 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398404+0000", 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1823000000000002e-05 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398508+0000", 2024-09-02T10:29:01.444 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103457 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398593+0000", 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5700999999999998e-05 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.399971+0000", 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013773360000000001 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.445 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.400538+0000", 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056756000000000005 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.640408+0000", 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23987022799999999 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.640452+0000", 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3565e-05 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.446 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.640489+0000", 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6860999999999998e-05 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:55478 7.b 7:de76706b:::benchmark_data_smithi187_120712_object55477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:07.589026+0000", 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 233.822777108, 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.190039492, 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.447 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55478 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.589026+0000", 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.589026+0000", 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.448 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.589027+0000", 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.218e-06 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.589069+0000", 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1279000000000002e-05 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.589079+0000", 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0351e-05 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.449 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.589084+0000", 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0819999999999998e-06 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.589134+0000", 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0164000000000003e-05 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.589229+0000", 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5260000000000006e-05 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.450 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.589317+0000", 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7482e-05 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.590665+0000", 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013481179999999999 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.451 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.591132+0000", 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00046705200000000003 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.778999+0000", 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.187867326 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.779034+0000", 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.489e-05 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.452 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.779066+0000", 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1269999999999997e-05 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:55530 7.4 7:2b921397:::benchmark_data_smithi187_120712_object55529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:07.789283+0000", 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 233.62251984400001, 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25236621799999998, 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.453 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55530 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789283+0000", 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.454 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789283+0000", 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789285+0000", 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.438e-06 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789326+0000", 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1672999999999999e-05 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.455 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789337+0000", 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0895e-05 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789343+0000", 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2569999999999998e-06 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789405+0000", 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2550000000000003e-05 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.456 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789568+0000", 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016324500000000001 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4,9", 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789765+0000", 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000197097 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791887+0000", 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021217290000000001 2024-09-02T10:29:01.457 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791969+0000", 2024-09-02T10:29:01.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1676999999999997e-05 2024-09-02T10:29:01.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.458 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.458 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.458 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.792487+0000", 2024-09-02T10:29:01.458 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051796399999999999 2024-09-02T10:29:01.458 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:01.459 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.041461+0000", 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24897453 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.041524+0000", 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2496999999999995e-05 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.460 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.041650+0000", 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125666 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:55531 7.b 7:ddf5f000:::benchmark_data_smithi187_120712_object55530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:07.791489+0000", 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 233.62031421500001, 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22225244299999999, 2024-09-02T10:29:01.461 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55531 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.462 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791489+0000", 2024-09-02T10:29:01.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.463 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791489+0000", 2024-09-02T10:29:01.463 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.463 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.463 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.463 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791490+0000", 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5290000000000001e-06 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791541+0000", 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.041e-05 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791555+0000", 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4256999999999999e-05 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.464 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791563+0000", 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.191e-06 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791624+0000", 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0288000000000003e-05 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.465 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791822+0000", 2024-09-02T10:29:01.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000198381 2024-09-02T10:29:01.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:29:01.466 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791949+0000", 2024-09-02T10:29:01.466 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012674600000000001 2024-09-02T10:29:01.466 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.466 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.466 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.793670+0000", 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017215469999999999 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.794018+0000", 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034720600000000002 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013559+0000", 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21954131399999999 2024-09-02T10:29:01.467 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013656+0000", 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6859000000000002e-05 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013741+0000", 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5715000000000002e-05 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.468 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:55531 7.b e696/661 ondisk, result = 0)", 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:07.794053+0000", 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 233.617750341, 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21969546700000001, 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.794053+0000", 2024-09-02T10:29:01.469 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.794053+0000", 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.794053+0000", 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8800000000000003e-07 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.470 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.794056+0000", 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1489999999999998e-06 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013396+0000", 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21933929699999999 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.471 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013416+0000", 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9927000000000001e-05 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013530+0000", 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011441499999999999 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013548+0000", 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8085999999999999e-05 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.472 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013748+0000", 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020010500000000001 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:71617 7.b 7:d1c8b968:::benchmark_data_smithi187_120712_object71616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e711)", 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:27.802425+0000", 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 213.60937828600001, 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33678569200000003, 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.473 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 71617 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802425+0000", 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.474 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802425+0000", 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802427+0000", 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.731e-06 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802470+0000", 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2948000000000003e-05 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.475 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802486+0000", 2024-09-02T10:29:01.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6790999999999999e-05 2024-09-02T10:29:01.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.476 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.476 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.476 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802496+0000", 2024-09-02T10:29:01.476 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.696e-06 2024-09-02T10:29:01.476 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802583+0000", 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6898999999999996e-05 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802729+0000", 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145656 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:29:01.477 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802878+0000", 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014961999999999999 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.805442+0000", 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002563723 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.805818+0000", 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037650999999999998 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.478 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.139128+0000", 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33330957999999999 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.139168+0000", 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0169000000000001e-05 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.139211+0000", 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2369e-05 2024-09-02T10:29:01.479 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.4 722/690 [PushReplyOp(7:2b12bdff:::benchmark_data_smithi187_120712_object64093:head)])", 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:00.285664+0000", 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 181.12613883500001, 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22833324099999999, 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.285664+0000", 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.285664+0000", 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.285665+0000", 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6299999999999999e-07 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.285668+0000", 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1389999999999999e-06 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.513679+0000", 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22801164900000001 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.513687+0000", 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4159999999999998e-06 2024-09-02T10:29:01.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.513779+0000", 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2192000000000001e-05 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.513998+0000", 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021868199999999999 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:99748 7.4 7:2ea4c3b6:::benchmark_data_smithi187_120712_object15806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:00.286187+0000", 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 181.12561594799999, 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23885245899999999, 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 99748 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.484 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286187+0000", 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286187+0000", 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286189+0000", 2024-09-02T10:29:01.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.671e-06 2024-09-02T10:29:01.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286195+0000", 2024-09-02T10:29:01.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0340000000000002e-06 2024-09-02T10:29:01.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286210+0000", 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5291999999999999e-05 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286218+0000", 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6979999999999998e-06 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286277+0000", 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9361000000000003e-05 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286406+0000", 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012913900000000001 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4,9", 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286536+0000", 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012913099999999999 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289876+0000", 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033401699999999999 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.290140+0000", 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000264567 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509411+0000", 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21927023300000001 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.524839+0000", 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015428496 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.524915+0000", 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6193999999999996e-05 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.525040+0000", 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124473 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:99748 7.4 e722/690 ondisk, result = 0)", 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:00.289806+0000", 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 181.12199767600001, 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21961228499999999, 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289806+0000", 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289806+0000", 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289806+0000", 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.16e-07 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289814+0000", 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9100000000000005e-06 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509136+0000", 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.219321604 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509140+0000", 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7579999999999998e-06 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509389+0000", 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024907500000000002 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509395+0000", 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1079999999999998e-06 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509418+0000", 2024-09-02T10:29:01.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2914000000000001e-05 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.4 723/690 [PushReplyOp(7:2b550f09:::benchmark_data_smithi187_120712_object82889:head)])", 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:01.652741+0000", 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 179.759061831, 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20101590699999999, 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:29:01.496 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.652741+0000", 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.652741+0000", 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.652742+0000", 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2399999999999999e-07 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.652745+0000", 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3969999999999998e-06 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.853508+0000", 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20076253099999999 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.853515+0000", 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1760000000000004e-06 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.853556+0000", 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1325999999999997e-05 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.853757+0000", 2024-09-02T10:29:01.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000201153 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.4 723/690 [PushReplyOp(7:2b8179cd:::benchmark_data_smithi187_120712_object66421:head)])", 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:02.076155+0000", 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 179.33564777300001, 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25945981000000001, 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:29:01.500 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.076155+0000", 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.076155+0000", 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.076156+0000", 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4999999999999999e-07 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.076158+0000", 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.335405+0000", 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25924723 2024-09-02T10:29:01.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.335411+0000", 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.767e-06 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.335458+0000", 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6761999999999997e-05 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.335615+0000", 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000157677 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.503 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:101021 7.4 7:2274b89d:::benchmark_data_smithi187_120712_object17079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:02.078078+0000", 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 179.33372559, 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26144672400000002, 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.504 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 101021 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078078+0000", 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078078+0000", 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078079+0000", 2024-09-02T10:29:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2750000000000001e-06 2024-09-02T10:29:01.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078085+0000", 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9410000000000004e-06 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078097+0000", 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1695999999999999e-05 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078102+0000", 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.587e-06 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078151+0000", 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078281+0000", 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013052199999999999 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4,9", 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078403+0000", 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121857 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.081030+0000", 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026272330000000001 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.081176+0000", 2024-09-02T10:29:01.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014553299999999999 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.081342+0000", 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016558600000000001 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.339408+0000", 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25806603 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.510 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.339458+0000", 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0525999999999997e-05 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.339524+0000", 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6153000000000002e-05 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.511 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:143463 7.d e749/661 7:bdddf662:::benchmark_data_smithi187_120712_object59521:head v 749'9005, mlcod=749'9001)", 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:58.379297+0000", 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 123.03250608899999, 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20072000000000001, 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.379297+0000", 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.379297+0000", 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.379299+0000", 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5960000000000001e-06 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.379308+0000", 2024-09-02T10:29:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2949999999999999e-06 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.574020+0000", 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19471186900000001 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.574026+0000", 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1369999999999999e-06 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.574224+0000", 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019798099999999999 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.574293+0000", 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8864999999999999e-05 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.579973+0000", 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005679829 2024-09-02T10:29:01.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.580017+0000", 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4428000000000002e-05 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:144568 7.4 e749/741 7:21ef4de9:::benchmark_data_smithi187_120712_object60626:head v 749'8996, mlcod=749'8994)", 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:00.270570+0000", 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 121.14123315000001, 2024-09-02T10:29:01.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22573947899999999, 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.270570+0000", 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.270570+0000", 2024-09-02T10:29:01.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.270570+0000", 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.05e-07 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.270573+0000", 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7020000000000001e-06 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.518 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.493109+0000", 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.222536013 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.493118+0000", 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5909999999999996e-06 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.493188+0000", 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0698000000000006e-05 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.493215+0000", 2024-09-02T10:29:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6727999999999999e-05 2024-09-02T10:29:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.496252+0000", 2024-09-02T10:29:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030365890000000001 2024-09-02T10:29:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.496310+0000", 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7952999999999999e-05 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:146199 7.b 7:d34aba4d:::benchmark_data_smithi187_120712_object62257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:03.405908+0000", 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 118.00589535100001, 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.196440846, 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.521 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 146199 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.405908+0000", 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.405908+0000", 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.405910+0000", 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.576e-06 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.405919+0000", 2024-09-02T10:29:01.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9039999999999998e-06 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.405938+0000", 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8505e-05 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.405947+0000", 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.884e-06 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.406016+0000", 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9789999999999994e-05 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.406153+0000", 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013689299999999999 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.406280+0000", 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126391 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.411343+0000", 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0050628089999999997 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.411492+0000", 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014906899999999999 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.602205+0000", 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19071316999999999 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.602302+0000", 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7050000000000001e-05 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.602349+0000", 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6805000000000003e-05 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27559.0:1392 8.b e799/792 8:d8a1f253:::benchmark_data_smithi187_136059_object1391:head v 799'77, mlcod=799'74)", 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:28:59.818180+0000", 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.593623352, 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.087871540999999997, 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:28:59.818180+0000", 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:28:59.818180+0000", 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:28:59.818180+0000", 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3900000000000005e-07 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:28:59.818258+0000", 2024-09-02T10:29:01.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7280999999999994e-05 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:28:59.905566+0000", 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.087308679 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:28:59.905570+0000", 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7510000000000002e-06 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:28:59.905618+0000", 2024-09-02T10:29:01.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7840000000000003e-05 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:28:59.905637+0000", 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8987e-05 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:28:59.906012+0000", 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037528099999999999 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:01.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:28:59.906051+0000", 2024-09-02T10:29:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9183000000000001e-05 2024-09-02T10:29:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:01.532 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:01.532 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:29:01.534 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:01.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:01.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:01.658 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:01.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:01.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:01.744 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:29:01.760 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:01.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:01.792 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:01.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:01.924 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:01.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:01.925 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:01 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:29:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:01 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:29:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1279141548' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:02.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:02.049 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:02.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:02.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:01 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:29:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:01 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:29:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1279141548' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:02.113 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 3 15 5067 5052 105.243 107.875 0.0488356 0.0094703 2024-09-02T10:29:02.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:02.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:02.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:01 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:29:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:01 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:29:02.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1279141548' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:02.278 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:02.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:02.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:02.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:02.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:02.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:02.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:02.537 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:02.560 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:02.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:02.662 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:02.689 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:02.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:02.690 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:02.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:02.813 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:02.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:02.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:02.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:02.937 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:02.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:02.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:02 smithi151 ceph-mon[16639]: pgmap v1820: 47 pgs: 47 active+clean; 6.3 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s wr, 15 op/s 2024-09-02T10:29:02.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:02 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:02.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1279141548' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:29:02.975 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:02 smithi151 ceph-mon[16639]: osdmap e800: 12 total, 12 up, 12 in 2024-09-02T10:29:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:02 smithi187 ceph-mon[20072]: pgmap v1820: 47 pgs: 47 active+clean; 6.3 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s wr, 15 op/s 2024-09-02T10:29:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:02 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1279141548' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:29:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:02 smithi187 ceph-mon[20072]: osdmap e800: 12 total, 12 up, 12 in 2024-09-02T10:29:03.038 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:03.065 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:03.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:03.066 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:02 smithi162 ceph-mon[20279]: pgmap v1820: 47 pgs: 47 active+clean; 6.3 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s wr, 15 op/s 2024-09-02T10:29:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:02 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1279141548' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:29:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:02 smithi162 ceph-mon[20279]: osdmap e800: 12 total, 12 up, 12 in 2024-09-02T10:29:03.113 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 4 16 6846 6830 106.711 111.125 0.00157801 0.00935084 2024-09-02T10:29:03.167 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:03.192 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:03.193 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:03.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:03.315 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:03.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:03.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:03.417 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:03.441 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:03.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:03.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:03.567 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:03.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:03.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:03.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:03.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:03.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:03.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:03.819 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:03.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:03.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:03.921 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:03.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:03.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:03.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:04.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:04.069 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:04.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:04.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 5 16 8194 8178 102.218 84.25 0.0145889 0.00976393 2024-09-02T10:29:04.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:04.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:29:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:04.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:04.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:04.212 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:04.213 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:04.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:04.335 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:04.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:04.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:04.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:04.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:04.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:04.591 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:04.592 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:04.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:04.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:04.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:04.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:04.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:04.847 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:04.875 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:29:04.876 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:29:04.901 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:29:04.901 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:29:04.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:29:04.928 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:29:04.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:04.953 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:29:04.953 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:29:04.953 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:04 smithi151 ceph-mon[16639]: pgmap v1822: 47 pgs: 47 active+clean; 250 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 663 op/s 2024-09-02T10:29:04.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:29:04.979 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:04.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:04.984 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:29:04.984 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:29:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:04 smithi187 ceph-mon[20072]: pgmap v1822: 47 pgs: 47 active+clean; 250 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 663 op/s 2024-09-02T10:29:05.014 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:29:05.014 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:29:05.050 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:29:05.050 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:29:05.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:05.082 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:29:05.082 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:29:05.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:04 smithi162 ceph-mon[20279]: pgmap v1822: 47 pgs: 47 active+clean; 250 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 41 MiB/s wr, 663 op/s 2024-09-02T10:29:05.105 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:05.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:05.110 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:29:05.110 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:29:05.113 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 6 15 9723 9708 101.118 95.625 0.00167693 0.00987096 2024-09-02T10:29:05.136 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:29:05.137 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:29:05.158 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:29:05.158 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:29:05.179 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:29:05.179 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:29:05.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:05.228 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:05.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:05.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:05.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:05.361 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:05.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:05.362 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:05.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:05.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:05.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:05.493 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:05.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:05.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:05.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:05.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:05.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:05.747 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:05.748 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:05.849 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:05.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:05.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:05.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:05.977 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:06.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:06.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:06.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:05 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:29:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:05 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:29:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:05 smithi187 ceph-mon[20072]: osdmap e801: 12 total, 12 up, 12 in 2024-09-02T10:29:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1669438645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:05 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:29:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:05 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:29:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:05 smithi162 ceph-mon[20279]: osdmap e801: 12 total, 12 up, 12 in 2024-09-02T10:29:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1669438645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:06.102 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:06.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 7 16 11025 11009 98.2874 81.3125 0.00209084 0.0101623 2024-09-02T10:29:06.127 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:06.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:05 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:29:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:05 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:29:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:05 smithi151 ceph-mon[16639]: osdmap e801: 12 total, 12 up, 12 in 2024-09-02T10:29:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1669438645' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:06.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:06.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:06.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:06.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:06.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:06.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:06.482 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:06.515 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:06.516 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:06.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:06.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:06.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:06.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:06.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:06.764 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:06.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:06.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:06.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:06.889 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:06.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:06.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:06.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:06 smithi187 ceph-mon[20072]: pgmap v1824: 47 pgs: 47 active+clean; 405 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 950 op/s 2024-09-02T10:29:07.014 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:07.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:07.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:07.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:06 smithi162 ceph-mon[20279]: pgmap v1824: 47 pgs: 47 active+clean; 405 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 950 op/s 2024-09-02T10:29:07.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 8 16 12644 12628 98.6489 101.188 0.0214618 0.0101238 2024-09-02T10:29:07.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:07.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:07.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:07.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:06 smithi151 ceph-mon[16639]: pgmap v1824: 47 pgs: 47 active+clean; 405 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 950 op/s 2024-09-02T10:29:07.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:07.265 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:07.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:07.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:07.389 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:07.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:07.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:07.514 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:07.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:07.555 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:07.603 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:07.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:07.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:07.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:07.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:07.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:07.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:07.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:07.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:07.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:07.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:07 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:29:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:07 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:29:07.995 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:07 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:29:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:07 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:29:08.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:08.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:07 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:29:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:07 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:29:08.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 9 16 14040 14024 97.3816 87.25 0.0217974 0.0102455 2024-09-02T10:29:08.121 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:08.143 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:08.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:08.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:08.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:08.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:08.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:08.413 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:08.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:08.414 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:08.515 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:08.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:08.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:08.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:08.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:08.663 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:08.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:08.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:08.728 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-02T10:29:08.765 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:08.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:08 smithi151 ceph-mon[16639]: pgmap v1825: 47 pgs: 47 active+clean; 615 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-02T10:29:08.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:08 smithi151 ceph-mon[16639]: osdmap e802: 12 total, 12 up, 12 in 2024-09-02T10:29:08.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/405635461' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:29:08.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:08 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:29:08.788 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:08.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:08.811 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:08.811 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:08.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:08.921 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:08.922 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:08 smithi187 ceph-mon[20072]: pgmap v1825: 47 pgs: 47 active+clean; 615 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-02T10:29:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:08 smithi187 ceph-mon[20072]: osdmap e802: 12 total, 12 up, 12 in 2024-09-02T10:29:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/405635461' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:29:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:08 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:29:09.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:09.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:09.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:09.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:09.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:08 smithi162 ceph-mon[20279]: pgmap v1825: 47 pgs: 47 active+clean; 615 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 77 MiB/s wr, 1.23k op/s 2024-09-02T10:29:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:08 smithi162 ceph-mon[20279]: osdmap e802: 12 total, 12 up, 12 in 2024-09-02T10:29:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/405635461' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:29:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:08 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:29:09.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 10 16 15684 15668 97.9177 102.75 0.00814207 0.0102018 2024-09-02T10:29:09.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:09.168 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:09.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:09.270 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:09.295 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:09.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:09.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:29:09.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:09.425 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:09.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:09.427 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:09.528 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:09.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:09.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:09.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:09.658 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:09.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:09.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:09.735 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:29:09.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:09.791 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-09-02T10:29:09.808 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:09.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:09.910 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:09.943 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:09.944 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:09 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:29:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:09 smithi187 ceph-mon[20072]: osdmap e803: 12 total, 12 up, 12 in 2024-09-02T10:29:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3907879197' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:10.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:10.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:10.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:10.087 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:09 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:29:10.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:09 smithi151 ceph-mon[16639]: osdmap e803: 12 total, 12 up, 12 in 2024-09-02T10:29:10.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3907879197' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:10.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:09 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:29:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:09 smithi162 ceph-mon[20279]: osdmap e803: 12 total, 12 up, 12 in 2024-09-02T10:29:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3907879197' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:10.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 11 16 17063 17047 96.8507 86.1875 0.0133892 0.0103149 2024-09-02T10:29:10.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:10.195 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:10.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:10.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:10.297 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:10.321 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:10.322 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:10.423 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:10.445 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:10.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:10.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:10.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:10.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:10.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:10.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:29:10.678 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:10.711 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:10.712 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:10.813 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:10.835 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:10.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:10.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:10 smithi151 ceph-mon[16639]: pgmap v1828: 47 pgs: 47 active+clean; 816 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-02T10:29:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:10 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3907879197' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:10 smithi151 ceph-mon[16639]: osdmap e804: 12 total, 12 up, 12 in 2024-09-02T10:29:10.961 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:10.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:10 smithi187 ceph-mon[20072]: pgmap v1828: 47 pgs: 47 active+clean; 816 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-02T10:29:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:10 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3907879197' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:10 smithi187 ceph-mon[20072]: osdmap e804: 12 total, 12 up, 12 in 2024-09-02T10:29:11.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:11.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:11.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:11.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:11.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:10 smithi162 ceph-mon[20279]: pgmap v1828: 47 pgs: 47 active+clean; 816 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.51k op/s 2024-09-02T10:29:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:10 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3907879197' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:10 smithi162 ceph-mon[20279]: osdmap e804: 12 total, 12 up, 12 in 2024-09-02T10:29:11.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 12 15 18436 18421 95.9355 85.875 0.00280791 0.0104128 2024-09-02T10:29:11.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:11.217 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:11.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:11.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:11.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:11.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:11.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:11.434 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:11.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:11.449 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:11.454 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:11.463 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:11.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:11.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:11.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:11.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:11.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:11.718 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:11.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:11.820 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:11.843 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:11.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:11.945 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:11.968 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:11.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:11.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:12.070 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:12.094 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:12.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:12.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:12.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 13 16 20000 19984 96.0696 97.6875 0.00178894 0.010397 2024-09-02T10:29:12.196 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:12.217 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:12.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:12.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:12.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:12.341 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:12.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:12.342 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:12.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:12.465 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:12.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:12.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:12.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:12.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:12.599 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:12.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:12.727 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:12.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:12.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:12.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:12 smithi151 ceph-mon[16639]: pgmap v1830: 47 pgs: 47 active+clean; 873 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-02T10:29:12.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:12.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:12.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:12.902 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:12.902 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:12.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:12.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:12.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:12 smithi187 ceph-mon[20072]: pgmap v1830: 47 pgs: 47 active+clean; 873 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-02T10:29:13.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:13.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:12 smithi162 ceph-mon[20279]: pgmap v1830: 47 pgs: 47 active+clean; 873 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-02T10:29:13.107 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:13.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:13.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:13.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 14 16 21067 21051 93.9705 66.6875 0.00177243 0.0106021 2024-09-02T10:29:13.153 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:29:13.153 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:29:13.189 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:29:13.189 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:29:13.209 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:13.215 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:29:13.215 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:29:13.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:13.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:13.240 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:29:13.240 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:29:13.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:29:13.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:29:13.290 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:29:13.290 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:29:13.315 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:29:13.315 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:29:13.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:13.339 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:29:13.340 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:29:13.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:13.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:13.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:13.362 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:29:13.362 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:29:13.389 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:29:13.389 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:29:13.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:29:13.422 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:29:13.422 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:29:13.451 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:29:13.451 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:29:13.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:13.485 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:13.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:13.486 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:13.587 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:13.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:13.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:13.714 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:13.734 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:13.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:13.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:13.738 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-09-02T10:29:13.836 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:13.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:13.870 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:13.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:14.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:14.003 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/302752990' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:29:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3807090957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:13 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:29:14.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:13 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:29:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/302752990' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:29:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3807090957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:13 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:29:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:13 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:29:14.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:14.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 15 16 22489 22473 93.6303 88.875 0.00188102 0.0106671 2024-09-02T10:29:14.131 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:14.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:14.132 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/302752990' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:29:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3807090957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:13 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:29:14.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:13 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:29:14.233 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:14.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:14.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:14.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:14.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:14.387 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:14.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:14.412 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:29:14.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:14.510 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:14.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:14.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:14.612 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:14.634 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:14.635 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:14.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:14.758 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:14.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:14.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:14.860 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:14.886 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:14.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:14.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:14 smithi187 ceph-mon[20072]: pgmap v1831: 47 pgs: 47 active+clean; 1.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-02T10:29:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/302752990' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:29:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:14 smithi187 ceph-mon[20072]: osdmap e805: 12 total, 12 up, 12 in 2024-09-02T10:29:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:15.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:15.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:14 smithi162 ceph-mon[20279]: pgmap v1831: 47 pgs: 47 active+clean; 1.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-02T10:29:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/302752990' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:29:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:14 smithi162 ceph-mon[20279]: osdmap e805: 12 total, 12 up, 12 in 2024-09-02T10:29:15.113 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:15.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 16 16 23455 23439 91.5516 60.375 0.0142276 0.0108747 2024-09-02T10:29:15.134 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:15.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:15.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:14 smithi151 ceph-mon[16639]: pgmap v1831: 47 pgs: 47 active+clean; 1.1 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 94 MiB/s wr, 1.50k op/s 2024-09-02T10:29:15.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/302752990' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:29:15.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:14 smithi151 ceph-mon[16639]: osdmap e805: 12 total, 12 up, 12 in 2024-09-02T10:29:15.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:15.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:15.267 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:15.368 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:15.407 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:15.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:15.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:15.536 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:15.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:15.538 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:15.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:15.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:15.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:15.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:15.788 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:15.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:15.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:15.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:15.918 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:15.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:15.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:15.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-09-02T10:29:15.970 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:15.970 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:16.020 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:16.052 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:16.053 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:16.114 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 17 16 25176 25160 92.4929 107.562 0.0115576 0.0108047 2024-09-02T10:29:16.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:16.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:16.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:16.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:16.309 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:16.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:16.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:16.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:16.433 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:16.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:16.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:16.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:16.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:16.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:16.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:16.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:16.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:16.689 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:16.731 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.f e628,586 6:f0620152:::benchmark_data_smithi187_102448_object15839:head,577'7198)", 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 390.53988085999998, 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69156394899999996, 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-02T10:29:16.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094472+0000", 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1230000000000002e-06 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785890+0000", 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69141827 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785894+0000", 2024-09-02T10:29:16.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1520000000000002e-06 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785977+0000", 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3077000000000004e-05 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785987+0000", 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8379999999999997e-06 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.786031+0000", 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3763000000000001e-05 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.f e629,586 6:fb6fcd13:::benchmark_data_smithi187_102448_object18483:head,578'7347)", 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:47.434406+0000", 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 389.19994134400002, 2024-09-02T10:29:16.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67700380800000004, 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434406+0000", 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434406+0000", 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434407+0000", 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434411+0000", 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6449999999999998e-06 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111277+0000", 2024-09-02T10:29:16.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67686604399999994 2024-09-02T10:29:16.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111286+0000", 2024-09-02T10:29:16.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0529999999999996e-06 2024-09-02T10:29:16.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111349+0000", 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000004e-05 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111359+0000", 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9250000000000007e-06 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111410+0000", 2024-09-02T10:29:16.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0819999999999998e-05 2024-09-02T10:29:16.741 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.741 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.741 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e635,586 6:033be843:::benchmark_data_smithi187_102448_object1499:head,573'6439)", 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:56.345932+0000", 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 380.28841532400003, 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39558264700000001, 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345932+0000", 2024-09-02T10:29:16.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345932+0000", 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345933+0000", 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345938+0000", 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0640000000000001e-06 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741308+0000", 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39536988899999997 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741316+0000", 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9009999999999994e-06 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741421+0000", 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105486 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741436+0000", 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4617000000000001e-05 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741515+0000", 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8910000000000002e-05 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.745 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e643,586 6:0a7160e0:::benchmark_data_smithi187_102448_object18383:head,578'7477)", 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:07.703494+0000", 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 368.93085324899999, 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44123607500000001, 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703494+0000", 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703494+0000", 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703496+0000", 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4529999999999999e-06 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703502+0000", 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2709999999999998e-06 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144429+0000", 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44092676800000002 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144444+0000", 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5146e-05 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144589+0000", 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014536200000000001 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144604+0000", 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4817000000000001e-05 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144730+0000", 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126258 2024-09-02T10:29:16.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e648,586 6:065bcc16:::benchmark_data_smithi187_102448_object45489:head,591'9175)", 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:29.598657+0000", 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 347.03569040000002, 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67037543899999996, 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598657+0000", 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598657+0000", 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598658+0000", 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7800000000000001e-07 2024-09-02T10:29:16.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598663+0000", 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268825+0000", 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.670161754 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268834+0000", 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3449999999999995e-06 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268939+0000", 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010511899999999999 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268949+0000", 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7599999999999997e-06 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.269033+0000", 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3561999999999997e-05 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e648,586 6:0cd50fba:::benchmark_data_smithi187_102448_object46855:head,592'9264)", 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:31.097253+0000", 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 345.53709441500001, 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.83171925999999996, 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.754 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097253+0000", 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097253+0000", 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097254+0000", 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.61e-07 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097259+0000", 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9520000000000002e-06 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928794+0000", 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.83153501299999999 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928806+0000", 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1847e-05 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928905+0000", 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9277999999999994e-05 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928913+0000", 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7039999999999994e-06 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928972+0000", 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9805000000000001e-05 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e653,586 6:01d685fb:::benchmark_data_smithi187_102448_object80695:head,616'11374)", 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:48.578010+0000", 2024-09-02T10:29:16.758 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 328.056337396, 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54628079200000002, 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578010+0000", 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578010+0000", 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578011+0000", 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9400000000000005e-07 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578016+0000", 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9069999999999998e-06 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124136+0000", 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54612002800000004 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124160+0000", 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4731000000000001e-05 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124241+0000", 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0457999999999994e-05 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124249+0000", 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3620000000000006e-06 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124291+0000", 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1712000000000003e-05 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:45621 7.8 7:1a107a8d:::benchmark_data_smithi187_120712_object45620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.307311+0000", 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 263.32703673399999, 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36183037800000001, 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45621 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.763 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307311+0000", 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307311+0000", 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307315+0000", 2024-09-02T10:29:16.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.566e-06 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307366+0000", 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0365999999999999e-05 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307379+0000", 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3134e-05 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307386+0000", 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.362e-06 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307441+0000", 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4568999999999999e-05 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307551+0000", 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109806 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:29:16.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307723+0000", 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017262399999999999 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.309378+0000", 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016545729999999999 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.309672+0000", 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029370500000000001 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669068+0000", 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35939658099999999 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669108+0000", 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9860999999999997e-05 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669141+0000", 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3231000000000001e-05 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.768 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:45621 7.8 e689/661 ondisk, result = 0)", 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.383905+0000", 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 263.25044219500001, 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28524120600000002, 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383905+0000", 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383905+0000", 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383907+0000", 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0470000000000001e-06 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383920+0000", 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2989e-05 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669023+0000", 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28510220400000003 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669028+0000", 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4759999999999997e-06 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669062+0000", 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3779999999999998e-05 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669065+0000", 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4089999999999999e-06 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669147+0000", 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1301e-05 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.772 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:45650 7.8 7:1c05cd80:::benchmark_data_smithi187_120712_object45649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.666730+0000", 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 262.96761729999997, 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.325110548, 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45650 2024-09-02T10:29:16.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666730+0000", 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666730+0000", 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666732+0000", 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.835e-06 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666773+0000", 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1121999999999997e-05 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666787+0000", 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3908e-05 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666793+0000", 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4589999999999997e-06 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666848+0000", 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5037999999999997e-05 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666935+0000", 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7781000000000005e-05 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.667024+0000", 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8331000000000005e-05 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.668407+0000", 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013827869999999999 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.668945+0000", 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053888899999999997 2024-09-02T10:29:16.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991726+0000", 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32278023700000003 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991790+0000", 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4288000000000005e-05 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991841+0000", 2024-09-02T10:29:16.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0872999999999999e-05 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:45650 7.8 e689/661 ondisk, result = 0)", 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.669003+0000", 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 262.96534451899998, 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32284557699999999, 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669003+0000", 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669003+0000", 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669004+0000", 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-02T10:29:16.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669007+0000", 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4340000000000001e-06 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991617+0000", 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.322609692 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991628+0000", 2024-09-02T10:29:16.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0998e-05 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991700+0000", 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2343999999999997e-05 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991718+0000", 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8142e-05 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991849+0000", 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130207 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:71627 7.8 7:158e78ca:::benchmark_data_smithi187_120712_object71626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e711)", 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:27.818033+0000", 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 228.816314979, 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48315623299999999, 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:16.783 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 71627 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818033+0000", 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818033+0000", 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818035+0000", 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2529999999999999e-06 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818066+0000", 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1619000000000003e-05 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818083+0000", 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6721000000000001e-05 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818093+0000", 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3680000000000008e-06 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818169+0000", 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6487000000000001e-05 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818309+0000", 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014026699999999999 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818454+0000", 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144964 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.821157+0000", 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027031830000000001 2024-09-02T10:29:16.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.821177+0000", 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0055e-05 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.301111+0000", 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47993339899999998 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.301149+0000", 2024-09-02T10:29:16.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8605000000000001e-05 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.301189+0000", 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9311999999999999e-05 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:136914 7.3 e743/741 7:c30a193a:::benchmark_data_smithi187_120712_object52972:head v 743'8327, mlcod=742'8324)", 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.758995+0000", 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 152.87535222899999, 2024-09-02T10:29:16.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33365049099999999, 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758995+0000", 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758995+0000", 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758996+0000", 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1100000000000002e-07 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.759001+0000", 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9309999999999999e-06 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090734+0000", 2024-09-02T10:29:16.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33173325799999998 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090757+0000", 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3033000000000001e-05 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090882+0000", 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012556700000000001 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090902+0000", 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9519999999999999e-05 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092615+0000", 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001713387 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092646+0000", 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0484000000000002e-05 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.793 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:145939 7.12 7:4b7e5994:::benchmark_data_smithi187_120712_object61997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.921348+0000", 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 133.71299985900001, 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38239470399999997, 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 145939 2024-09-02T10:29:16.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.795 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921348+0000", 2024-09-02T10:29:16.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921348+0000", 2024-09-02T10:29:16.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921349+0000", 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5349999999999999e-06 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921355+0000", 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3940000000000004e-06 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921368+0000", 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3499000000000001e-05 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921375+0000", 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4729999999999997e-06 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.928654+0000", 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072797640000000002 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.928971+0000", 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000317127 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.929306+0000", 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033418199999999998 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.939106+0000", 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0098002920000000004 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.951104+0000", 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011997733 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302905+0000", 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.351801429 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.303056+0000", 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015083300000000001 2024-09-02T10:29:16.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.303742+0000", 2024-09-02T10:29:16.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068644300000000001 2024-09-02T10:29:16.800 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.800 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.800 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.800 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:145939 7.12 e749/744 ondisk, result = 0)", 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.932006+0000", 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 133.702341623, 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37174302799999998, 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932006+0000", 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932006+0000", 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932007+0000", 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1039999999999999e-06 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932012+0000", 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8289999999999998e-06 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302586+0000", 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37057435599999999 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302593+0000", 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4300000000000003e-06 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302888+0000", 2024-09-02T10:29:16.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000295378 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302898+0000", 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6469999999999993e-06 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.303749+0000", 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000851284 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.804 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.12 749/744 [PushReplyOp(7:481a1f6c:::benchmark_data_smithi187_120712_object69540:head)])", 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.953782+0000", 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 133.680565548, 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.375520674, 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953782+0000", 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953782+0000", 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953783+0000", 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6099999999999999e-07 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953789+0000", 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6279999999999999e-06 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.328024+0000", 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37423416799999998 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.328033+0000", 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1700000000000003e-06 2024-09-02T10:29:16.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.328279+0000", 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000246519 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.329303+0000", 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001023328 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:145959 7.12 7:4e913d9f:::benchmark_data_smithi187_120712_object62017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.957341+0000", 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 133.677006823, 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39245286600000001, 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 145959 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957341+0000", 2024-09-02T10:29:16.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957341+0000", 2024-09-02T10:29:16.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957342+0000", 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5549999999999999e-06 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957350+0000", 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5889999999999997e-06 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957361+0000", 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1075e-05 2024-09-02T10:29:16.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957369+0000", 2024-09-02T10:29:16.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8010000000000004e-06 2024-09-02T10:29:16.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957569+0000", 2024-09-02T10:29:16.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020020500000000001 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957715+0000", 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014592899999999999 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957956+0000", 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024078399999999999 2024-09-02T10:29:16.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.972919+0000", 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014963476999999999 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.974010+0000", 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010906589999999999 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349010+0000", 2024-09-02T10:29:16.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37499981799999998 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349077+0000", 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599000000000001e-05 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349794+0000", 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000716375 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:145959 7.12 e749/744 ondisk, result = 0)", 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.961683+0000", 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 133.67266473000001, 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38811612099999998, 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961683+0000", 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961683+0000", 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961684+0000", 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5600000000000005e-07 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961688+0000", 2024-09-02T10:29:16.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.831e-06 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.348905+0000", 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38721641600000001 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.348912+0000", 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2559999999999996e-06 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.348993+0000", 2024-09-02T10:29:16.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1050999999999994e-05 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349002+0000", 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3309999999999993e-06 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349799+0000", 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00079648000000000002 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.819 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.18 749/744 [PushReplyOp(7:18b04759:::benchmark_data_smithi187_120712_object77911:head)])", 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:06.705518+0000", 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 129.928829597, 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32887954400000002, 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:06.705518+0000", 2024-09-02T10:29:16.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:06.705518+0000", 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:06.705519+0000", 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1900000000000002e-07 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:06.705528+0000", 2024-09-02T10:29:16.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2119999999999992e-06 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:07.031820+0000", 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.326292305 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:07.031827+0000", 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0929999999999997e-06 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.822 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:07.033462+0000", 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016343569999999999 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:07.034397+0000", 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093585800000000002 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:152368 7.8 7:17e54ee4:::benchmark_data_smithi187_120712_object68426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:14.166120+0000", 2024-09-02T10:29:16.823 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 122.46822747, 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28226089599999998, 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 152368 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:16.824 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166120+0000", 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166120+0000", 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166122+0000", 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.761e-06 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:16.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166128+0000", 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7379999999999999e-06 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166140+0000", 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2480000000000001e-05 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166147+0000", 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7920000000000004e-06 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166231+0000", 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4176000000000002e-05 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166376+0000", 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014515799999999999 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166519+0000", 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014257300000000001 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.170193+0000", 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003673782 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.177004+0000", 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068118199999999997 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.448215+0000", 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27121096900000002 2024-09-02T10:29:16.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.448323+0000", 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107253 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.448381+0000", 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8393999999999997e-05 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:16.829 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:29:16.830 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:16.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:16.855 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:16 smithi151 ceph-mon[16639]: pgmap v1833: 47 pgs: 47 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-02T10:29:16.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:16.981 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:16.982 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:16 smithi187 ceph-mon[20072]: pgmap v1833: 47 pgs: 47 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-02T10:29:17.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:17.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:16 smithi162 ceph-mon[20279]: pgmap v1833: 47 pgs: 47 active+clean; 1.3 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-02T10:29:17.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:17.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:17.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:17.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 18 16 26697 26681 92.6352 95.0625 0.00405764 0.0107878 2024-09-02T10:29:17.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:17.243 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:17.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:17.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:17.345 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:17.372 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:17.373 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:17.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:29:17.474 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:17.497 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:17.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:17.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:17.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:17.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:17.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:17.746 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:29:17.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:17.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:17.755 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:17.856 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:17.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:17.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:17.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1669339594' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:29:18.008 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:18.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:18.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:18.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1669339594' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:29:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1669339594' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:29:18.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:29:18.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:18.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 19 16 28308 28292 93.0586 100.688 0.00283082 0.0107378 2024-09-02T10:29:18.131 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:18.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:18.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:18.254 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:18.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:18.356 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:18.399 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:18.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:18.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:18.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:18.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:18.537 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:18.538 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:18.639 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:18.662 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:18.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:18.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:18.788 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:18.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:18.890 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:18.918 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:18.919 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:18 smithi187 ceph-mon[20072]: pgmap v1834: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-02T10:29:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:18 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1669339594' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:29:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:18 smithi187 ceph-mon[20072]: osdmap e806: 12 total, 12 up, 12 in 2024-09-02T10:29:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:18 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:29:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:18 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:29:19.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:19.036 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:18 smithi151 ceph-mon[16639]: pgmap v1834: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-02T10:29:19.036 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:18 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:19.036 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1669339594' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:29:19.036 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:18 smithi151 ceph-mon[16639]: osdmap e806: 12 total, 12 up, 12 in 2024-09-02T10:29:19.036 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:18 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:29:19.036 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:18 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:29:19.041 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:19.053 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:19.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:18 smithi162 ceph-mon[20279]: pgmap v1834: 47 pgs: 47 active+clean; 1.5 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.36k op/s 2024-09-02T10:29:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:18 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1669339594' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:29:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:18 smithi162 ceph-mon[20279]: osdmap e806: 12 total, 12 up, 12 in 2024-09-02T10:29:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:18 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:29:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:18 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:29:19.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:29:19.114803+0000 min lat: 0.00107552 max lat: 0.254684 avg lat: 0.0107708 2024-09-02T10:29:19.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:29:19.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 20 16 29707 29691 92.7772 87.4375 0.0102972 0.0107708 2024-09-02T10:29:19.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:19.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:19.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:19.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:19.201 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:19.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:19.302 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:19.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:19.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:19.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:19.426 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:19.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:19.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:19.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:19.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:19.558 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:19.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:19.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:19.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:19.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:19.814 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:19.815 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:19.916 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:19.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:19.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:20.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:20.067 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:20.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:20.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 21 15 31139 31124 92.6237 89.5625 0.0214708 0.0107892 2024-09-02T10:29:20.168 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:20.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:20.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:20.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:20.328 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:20.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:20.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:20.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:20.452 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:20.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:20.554 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:20.575 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:20.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:20.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:20.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:20 smithi162 ceph-mon[20279]: osdmap e807: 12 total, 12 up, 12 in 2024-09-02T10:29:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:20 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:29:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:20 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:29:20.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:20 smithi162 ceph-mon[20279]: pgmap v1837: 47 pgs: 47 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-02T10:29:20.677 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:20.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:20 smithi151 ceph-mon[16639]: osdmap e807: 12 total, 12 up, 12 in 2024-09-02T10:29:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:20 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:29:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:20 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:29:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:20 smithi151 ceph-mon[16639]: pgmap v1837: 47 pgs: 47 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-02T10:29:20.701 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:20.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:20.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:20 smithi187 ceph-mon[20072]: osdmap e807: 12 total, 12 up, 12 in 2024-09-02T10:29:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:20 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:29:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:20 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:29:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:20 smithi187 ceph-mon[20072]: pgmap v1837: 47 pgs: 47 active+clean; 1.7 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 87 MiB/s wr, 1.39k op/s 2024-09-02T10:29:20.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:20.825 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:20.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:20.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:20.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:20.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:20.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:21.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:21.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:21.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:21.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:29:21.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 22 16 32654 32638 92.7143 94.625 0.0040104 0.0107798 2024-09-02T10:29:21.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:21.227 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:21.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:21.329 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:21.353 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:21.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:21.406 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:29:21.406 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:29:21.425 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:29:21.425 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:29:21.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:21.464 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:29:21.464 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:29:21.477 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:21.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:21.497 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:29:21.497 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:29:21.533 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:29:21.533 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:29:21.562 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:29:21.563 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:29:21.564 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2329341132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:21.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:21.589 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:29:21.590 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:29:21.603 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:21.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:21.617 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:29:21.617 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:29:21.644 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:29:21.644 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:29:21.673 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:29:21.674 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:29:21.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:21.707 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:29:21.708 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:29:21.735 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:29:21.735 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:29:21.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:21.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:21.744 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2329341132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2329341132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:21.845 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:21.875 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:21.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:21.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:21.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:22.000 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:22.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:22.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:22.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 23 16 34506 34490 93.7154 115.75 0.0218118 0.0106629 2024-09-02T10:29:22.129 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:22.130 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:22.231 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:22.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:22.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:22.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:22.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:22.379 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:22.409 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-09-02T10:29:22.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:22 smithi151 ceph-mon[16639]: pgmap v1838: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-02T10:29:22.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3368943820' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:29:22.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:22.501 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:22.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:22.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:22.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:22.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:22.646 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:22.647 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:22 smithi187 ceph-mon[20072]: pgmap v1838: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-02T10:29:22.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3368943820' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:29:22.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:22.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:22.749 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:22 smithi162 ceph-mon[20279]: pgmap v1838: 47 pgs: 47 active+clean; 1.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 74 MiB/s wr, 1.19k op/s 2024-09-02T10:29:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3368943820' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:29:22.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:22.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:22.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:22.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:23.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:23.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:23.115 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:23.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 24 16 35314 35298 91.9146 50.5 0.00234088 0.0108695 2024-09-02T10:29:23.142 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:23.143 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:23.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:23.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:23.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:23.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:23.395 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:23.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:23.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:23.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:23.520 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:23.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:23.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:23.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:23.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3368943820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:29:23.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:23 smithi151 ceph-mon[16639]: osdmap e808: 12 total, 12 up, 12 in 2024-09-02T10:29:23.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:23 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:29:23.627 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:23 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:29:23.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:23.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:23.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:23.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3368943820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:29:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:23 smithi187 ceph-mon[20072]: osdmap e808: 12 total, 12 up, 12 in 2024-09-02T10:29:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:23 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:29:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:23 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:29:23.776 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:23.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3368943820' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:29:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:23 smithi162 ceph-mon[20279]: osdmap e808: 12 total, 12 up, 12 in 2024-09-02T10:29:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:23 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:29:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:23 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:29:23.868 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:23.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:23.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:23.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:23.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:24.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:24.025 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:24.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:24.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:24.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 25 16 36841 36825 92.0552 95.4375 0.00279489 0.0108571 2024-09-02T10:29:24.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:24.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:24.159 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:24.260 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:24.283 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:24.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:24.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-09-02T10:29:24.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:24.406 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:24.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:24.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:24.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:24.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:24.530 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:24.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:24 smithi151 ceph-mon[16639]: pgmap v1840: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-02T10:29:24.569 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:24 smithi151 ceph-mon[16639]: osdmap e809: 12 total, 12 up, 12 in 2024-09-02T10:29:24.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:24.658 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:24.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:24.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:24 smithi162 ceph-mon[20279]: pgmap v1840: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-02T10:29:24.745 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:24 smithi162 ceph-mon[20279]: osdmap e809: 12 total, 12 up, 12 in 2024-09-02T10:29:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:24 smithi187 ceph-mon[20072]: pgmap v1840: 47 pgs: 47 active+clean; 2.0 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-02T10:29:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:24 smithi187 ceph-mon[20072]: osdmap e809: 12 total, 12 up, 12 in 2024-09-02T10:29:24.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:24.791 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:24.792 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:24.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:24.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:24.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:25.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:25.047 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:25.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:25.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 26 16 37802 37786 90.8246 60.0625 0.154488 0.0109705 2024-09-02T10:29:25.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:25.179 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:25.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:25.180 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:25.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:25.312 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:25.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:25.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:25.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:25.424 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:29:25.442 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:25.443 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:25.449 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:25.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:29:25.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:25.566 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:25.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:25.567 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:25.668 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:25.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:29:25.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2504650963' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:25.691 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:25.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2504650963' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:25.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:25.825 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:25.826 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:25.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2504650963' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:25.927 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:25.954 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:25.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:25.955 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:26.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:26.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:26.083 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:26.083 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:26.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 27 15 39298 39283 90.9257 93.5625 0.00515155 0.0109916 2024-09-02T10:29:26.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:26.197 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:26.201 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:26.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:26.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:26.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:26.342 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:26.343 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:26.444 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:26.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:26.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:26.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:26.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:26.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:26 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:29:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:26 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:29:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2504650963' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:26 smithi151 ceph-mon[16639]: osdmap e810: 12 total, 12 up, 12 in 2024-09-02T10:29:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:26 smithi151 ceph-mon[16639]: pgmap v1843: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-02T10:29:26.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:26.730 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:26.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:26.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:26 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:29:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:26 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:29:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2504650963' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:26 smithi187 ceph-mon[20072]: osdmap e810: 12 total, 12 up, 12 in 2024-09-02T10:29:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:26 smithi187 ceph-mon[20072]: pgmap v1843: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-02T10:29:26.832 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:26 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:29:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:26 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:29:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2504650963' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:26 smithi162 ceph-mon[20279]: osdmap e810: 12 total, 12 up, 12 in 2024-09-02T10:29:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:26 smithi162 ceph-mon[20279]: pgmap v1843: 47 pgs: 47 active+clean; 2.2 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.35k op/s 2024-09-02T10:29:26.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:26.864 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:26.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:26.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:26.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:27.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:27.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 28 16 40650 40634 90.6936 84.4375 0.0167958 0.01102 2024-09-02T10:29:27.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:27.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:27.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:27.228 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:27.241 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:27.241 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:27.251 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:27.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:27.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:27.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:27.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:27.376 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:27.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:27.478 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:27.500 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:27.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:27.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:27.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:27.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:27.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:27.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:27.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:27.754 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:27.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:27.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:27.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:27.877 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:27.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:27.978 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:28.001 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:28.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:28.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:28.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 29 16 41717 41701 89.8656 66.6875 0.0717818 0.0111016 2024-09-02T10:29:28.124 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:28.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:28.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:29:28.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:28.248 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:28.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:28.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:28.350 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:28.373 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:28.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:28.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:28.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:28.505 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:28.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:28.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:28.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:28.733 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:28.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:28.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:28 smithi162 ceph-mon[20279]: pgmap v1844: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-02T10:29:28.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:28.885 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:28.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:28 smithi151 ceph-mon[16639]: pgmap v1844: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-02T10:29:28.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:28 smithi187 ceph-mon[20072]: pgmap v1844: 47 pgs: 47 active+clean; 2.3 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 104 MiB/s wr, 1.66k op/s 2024-09-02T10:29:29.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:29.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:29.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:29.115 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 30 16 43049 43033 89.6449 83.25 0.0187212 0.0111495 2024-09-02T10:29:29.140 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:29.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:29.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:29.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:29.266 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:29.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:29.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:29.391 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:29.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:29.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:29.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:29:29.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:29.515 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:29.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:29.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:29.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:29.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:29.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:29.749 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:29.774 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:29.775 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:29.876 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:29.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:29.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:30.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:30.023 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:30.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:30.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 31 16 44310 44294 89.2951 78.8125 0.0360029 0.0111675 2024-09-02T10:29:30.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:30.149 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:30.150 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:30.245 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:29:30.245 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:29:30.251 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:30.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:29:30.268 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:29:30.273 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:30.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:30.274 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:30.279 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-09-02T10:29:30.291 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:29:30.291 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:29:30.322 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:29:30.322 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:29:30.347 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:29:30.347 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:29:30.371 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:29:30.371 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:29:30.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:30.398 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:30.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:30.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:29:30.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:29:30.401 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:30 smithi151 ceph-mon[16639]: osdmap e811: 12 total, 12 up, 12 in 2024-09-02T10:29:30.401 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:30 smithi151 ceph-mon[16639]: pgmap v1846: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-02T10:29:30.401 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/447202792' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:29:30.401 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:30 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:29:30.401 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1270602133' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:30.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:29:30.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:29:30.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:29:30.458 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:29:30.485 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:29:30.486 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:29:30.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:30.513 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:29:30.513 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:29:30.523 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:30.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:30.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:30.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:29:30.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:29:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:30 smithi162 ceph-mon[20279]: osdmap e811: 12 total, 12 up, 12 in 2024-09-02T10:29:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:30 smithi162 ceph-mon[20279]: pgmap v1846: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-02T10:29:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/447202792' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:29:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:30 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:29:30.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1270602133' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:30.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:30.647 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:30.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:30.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:30 smithi187 ceph-mon[20072]: osdmap e811: 12 total, 12 up, 12 in 2024-09-02T10:29:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:30 smithi187 ceph-mon[20072]: pgmap v1846: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 80 MiB/s wr, 1.28k op/s 2024-09-02T10:29:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/447202792' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:29:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:30 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:29:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1270602133' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:30.772 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:30.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:30.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:30.896 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:30.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:30.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:31.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:31.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:31.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 32 16 45464 45448 88.7584 72.125 0.0264407 0.0112597 2024-09-02T10:29:31.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:31.147 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:31.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:31.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:31.274 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:31.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:31.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:31.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:31.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:31.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:31.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:31 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:29:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:31 smithi151 ceph-mon[16639]: osdmap e812: 12 total, 12 up, 12 in 2024-09-02T10:29:31.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:31.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:31.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:31 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:29:31.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:31 smithi162 ceph-mon[20279]: osdmap e812: 12 total, 12 up, 12 in 2024-09-02T10:29:31.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:31.663 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:31.664 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:31.750 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:31.750 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:31 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:29:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:31 smithi187 ceph-mon[20072]: osdmap e812: 12 total, 12 up, 12 in 2024-09-02T10:29:31.764 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:31.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 10 1 2024-09-02T10:29:31.786 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:31.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:31.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:31.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:31.915 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:31.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:32.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:32.046 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:32.047 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:32.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 33 16 46821 46805 88.6386 84.8125 0.0166816 0.0112743 2024-09-02T10:29:32.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:32.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:32.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:32.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:32.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:32.296 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:32 smithi151 ceph-mon[16639]: pgmap v1848: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:29:32.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:32.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:32.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:32.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:32.437 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:32.438 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:32.539 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:32.569 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:32.569 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:32.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:32 smithi162 ceph-mon[20279]: pgmap v1848: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:29:32.669 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:32.691 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:32.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:32.707 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:29:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:29:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:29:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:29:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:100267 6.b 6:dbd5fb01:::benchmark_data_smithi187_102448_object100266:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:10.945411+0000", 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 561.69437361799999, 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54857257299999995, 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 100267 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945411+0000", 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945411+0000", 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945413+0000", 2024-09-02T10:29:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3850000000000001e-06 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945444+0000", 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0843999999999998e-05 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945455+0000", 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1107000000000001e-05 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945462+0000", 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9029999999999998e-06 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945541+0000", 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9658000000000002e-05 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945679+0000", 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013753800000000001 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.945835+0000", 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015657799999999999 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949089+0000", 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0032539589999999998 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949384+0000", 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029429199999999998 2024-09-02T10:29:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493901+0000", 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54451749699999996 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493945+0000", 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3469e-05 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493984+0000", 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9342999999999999e-05 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26524.0:100267 6.b e572/533 ondisk, result = 0)", 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:20:10.949464+0000", 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 561.690321025, 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54452646699999996, 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949464+0000", 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949464+0000", 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949465+0000", 2024-09-02T10:29:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5099999999999999e-07 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:10.949468+0000", 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3440000000000001e-06 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493766+0000", 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54429846599999998 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493783+0000", 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7013000000000001e-05 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493889+0000", 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105537 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493897+0000", 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1009999999999992e-06 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:20:11.493990+0000", 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3355000000000003e-05 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157268 6.b 6:db1c5c67:::benchmark_data_smithi187_102448_object56996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 484.814938746, 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62417882300000005, 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157268 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:29:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824847+0000", 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824850+0000", 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.76e-06 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824860+0000", 2024-09-02T10:29:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2690000000000005e-06 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824876+0000", 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5882000000000001e-05 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824934+0000", 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8780000000000003e-05 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.825044+0000", 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010938900000000001 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.825142+0000", 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8009000000000003e-05 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830102+0000", 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049605530000000004 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.838185+0000", 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080826430000000005 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.448954+0000", 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61076896000000003 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.448992+0000", 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7732999999999999e-05 2024-09-02T10:29:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449025+0000", 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3315999999999997e-05 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157271 6.6 6:69d4aafc:::benchmark_data_smithi187_102448_object56999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 484.80938410700003, 2024-09-02T10:29:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62368362300000002, 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157271 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830403+0000", 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7179999999999999e-06 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830409+0000", 2024-09-02T10:29:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.703e-06 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830424+0000", 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4961e-05 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830432+0000", 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4880000000000001e-06 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830495+0000", 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3758999999999997e-05 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830625+0000", 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012918000000000001 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830756+0000", 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013138200000000001 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.835882+0000", 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051260960000000001 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.836280+0000", 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039814100000000002 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454013+0000", 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61773251699999998 2024-09-02T10:29:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454048+0000", 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5559999999999998e-05 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454084+0000", 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6118000000000002e-05 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157274 6.6 6:6387cbb8:::benchmark_data_smithi187_102448_object57002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 484.80235595300002, 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61674791900000003, 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157274 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837431+0000", 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7010000000000001e-06 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837437+0000", 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0379999999999996e-06 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837448+0000", 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1557e-05 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837456+0000", 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.396e-06 2024-09-02T10:29:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837510+0000", 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4666000000000001e-05 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837636+0000", 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125313 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837757+0000", 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121212 2024-09-02T10:29:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.845343+0000", 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075863880000000003 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.847219+0000", 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00187546 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454127+0000", 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60690782600000004 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454156+0000", 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9281999999999998e-05 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454177+0000", 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1080000000000001e-05 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157277 6.6 6:67446ce0:::benchmark_data_smithi187_102448_object57005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 484.80182434300002, 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61628705500000003, 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157277 2024-09-02T10:29:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0699999999999996e-07 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837964+0000", 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8569999999999999e-06 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837973+0000", 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0159999999999997e-06 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837978+0000", 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7770000000000002e-06 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842046+0000", 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040683289999999999 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842130+0000", 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3751000000000005e-05 2024-09-02T10:29:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842199+0000", 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8411999999999995e-05 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.845512+0000", 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003313795 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.847365+0000", 2024-09-02T10:29:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018529009999999999 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454203+0000", 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60683791600000003 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454229+0000", 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5609000000000002e-05 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454248+0000", 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8885e-05 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157282 6.b 6:d751a93a:::benchmark_data_smithi187_102448_object57010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 484.765853886, 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57523664399999996, 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157282 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873933+0000", 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.08e-06 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873941+0000", 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0919999999999998e-06 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873961+0000", 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0231e-05 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873971+0000", 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2969999999999992e-06 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874034+0000", 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3257999999999996e-05 2024-09-02T10:29:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874177+0000", 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142712 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874303+0000", 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126248 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.878010+0000", 2024-09-02T10:29:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037068610000000001 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.946302+0000", 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.068292383999999998 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449070+0000", 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50276810199999999 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449124+0000", 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3712999999999999e-05 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449168+0000", 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3665999999999998e-05 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157307 6.6 6:675a1a62:::benchmark_data_smithi187_102448_object57035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 484.62176520499997, 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43718100199999999, 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157307 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018021+0000", 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6559999999999999e-06 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018029+0000", 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6310000000000003e-06 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018046+0000", 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6779999999999999e-05 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018054+0000", 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3180000000000007e-06 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018619+0000", 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056491400000000002 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018723+0000", 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103619 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018800+0000", 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7859000000000001e-05 2024-09-02T10:29:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.022686+0000", 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038855399999999998 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031392+0000", 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087064819999999998 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455118+0000", 2024-09-02T10:29:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42372546999999999 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455167+0000", 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8773000000000002e-05 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455201+0000", 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.396e-05 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157310 6.b 6:d937f28e:::benchmark_data_smithi187_102448_object57038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 484.60866077899999, 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42301055599999998, 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157310 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031125+0000", 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8599999999999997e-07 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031130+0000", 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7949999999999998e-06 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031141+0000", 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0735e-05 2024-09-02T10:29:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031148+0000", 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4189999999999996e-06 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031196+0000", 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7571000000000002e-05 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031278+0000", 2024-09-02T10:29:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2262000000000006e-05 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031363+0000", 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5279999999999997e-05 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.035027+0000", 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036643140000000001 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.290856+0000", 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25582839899999998 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454040+0000", 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.163183774 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454092+0000", 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2224e-05 2024-09-02T10:29:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454135+0000", 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2896999999999999e-05 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:200538 6.b 6:dbd5fb01:::benchmark_data_smithi187_102448_object100266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:40.755486+0000", 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 411.88429869100003, 2024-09-02T10:29:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70349968900000004, 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200538 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755486+0000", 2024-09-02T10:29:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755486+0000", 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755488+0000", 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3680000000000001e-06 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755494+0000", 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6059999999999999e-06 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755507+0000", 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2850999999999999e-05 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755514+0000", 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0480000000000001e-06 2024-09-02T10:29:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755579+0000", 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4465e-05 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755693+0000", 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114207 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755818+0000", 2024-09-02T10:29:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124998 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.760931+0000", 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051127849999999999 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.762153+0000", 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00122255 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458876+0000", 2024-09-02T10:29:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69672282799999996 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458936+0000", 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9864000000000001e-05 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458986+0000", 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0118999999999999e-05 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26524.0:200538 6.b e626/533 ondisk, result = 0)", 2024-09-02T10:29:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:29:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 411.878131381, 2024-09-02T10:29:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69733888300000002, 2024-09-02T10:29:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1900000000000001e-07 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761658+0000", 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0430000000000001e-06 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458732+0000", 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69707383899999997 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458742+0000", 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4120000000000007e-06 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458857+0000", 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115513 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458869+0000", 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1325000000000001e-05 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458992+0000", 2024-09-02T10:29:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012383199999999999 2024-09-02T10:29:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.8,RELEASE,e:640,from:5,reservation_nonce:0)", 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 390.74718814300002, 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54587120899999997, 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:29:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5900000000000001e-07 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892598+0000", 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5209999999999999e-06 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438366+0000", 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54576760899999999 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438376+0000", 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.029e-05 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438438+0000", 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1400000000000002e-05 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438454+0000", 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6132000000000001e-05 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438468+0000", 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4098e-05 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:135860 7.1 e740/717)", 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.726925+0000", 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 170.912859946, 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.362811098, 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726925+0000", 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726925+0000", 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726926+0000", 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8400000000000004e-07 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726931+0000", 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6339999999999998e-06 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872187+0000", 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.145255784 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872192+0000", 2024-09-02T10:29:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7990000000000001e-06 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.089719+0000", 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.217526953 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.089736+0000", 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7244e-05 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135866 7.2 7.308b4c22 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.733884+0000", 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 170.90590049599999, 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38588288599999998, 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135866 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.784 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733884+0000", 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733884+0000", 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733886+0000", 2024-09-02T10:29:32.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7269999999999999e-06 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733891+0000", 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8690000000000003e-06 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733902+0000", 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1139e-05 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733910+0000", 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5820000000000004e-06 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.119723+0000", 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38581336799999999 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.119767+0000", 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4200999999999999e-05 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135869 7.12 7:4e2fbeef:::benchmark_data_smithi187_120712_object51927:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.872032+0000", 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 170.76775277499999, 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.95676479699999994, 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135869 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.788 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872032+0000", 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872032+0000", 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872034+0000", 2024-09-02T10:29:32.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9099999999999999e-06 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872043+0000", 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6319999999999997e-06 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872054+0000", 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1352000000000001e-05 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872063+0000", 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1470000000000007e-06 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.120273+0000", 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24820998499999999 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.120282+0000", 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7499999999999992e-06 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.820947+0000", 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70066504299999999 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821053+0000", 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010563100000000001 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821156+0000", 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010351199999999999 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.823544+0000", 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002387672 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.823920+0000", 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037607899999999997 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.828702+0000", 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047823479999999996 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.828757+0000", 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5010999999999999e-05 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.828797+0000", 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9725000000000003e-05 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.794 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:136913 7.10 7.335c64d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.758150+0000", 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 168.881634846, 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47337745399999998, 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136913 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758150+0000", 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758150+0000", 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758152+0000", 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0839999999999999e-06 2024-09-02T10:29:32.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758158+0000", 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8889999999999999e-06 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758173+0000", 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4831000000000001e-05 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758181+0000", 2024-09-02T10:29:32.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4120000000000001e-06 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758256+0000", 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4867e-05 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758265+0000", 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1710000000000008e-06 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231504+0000", 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47323864300000001 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231528+0000", 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3557000000000002e-05 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:136914 7.3 7:c30a193a:::benchmark_data_smithi187_120712_object52972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.758512+0000", 2024-09-02T10:29:32.799 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 168.88127247599999, 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33439487099999998, 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136914 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758512+0000", 2024-09-02T10:29:32.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758512+0000", 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758513+0000", 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0299999999999998e-07 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758517+0000", 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6799999999999999e-06 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758527+0000", 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0404999999999999e-05 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758534+0000", 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3729999999999998e-06 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758589+0000", 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5315999999999997e-05 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758687+0000", 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7879000000000003e-05 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,4", 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758807+0000", 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119839 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.760731+0000", 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001924413 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.771256+0000", 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010524939000000001 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092834+0000", 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.321577844 2024-09-02T10:29:32.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092874+0000", 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0209000000000001e-05 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092907+0000", 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3271e-05 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.805 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:136923 7.10 7.4a3ead10 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.891481+0000", 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 168.748303577, 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34014060299999999, 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136923 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.806 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891481+0000", 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891481+0000", 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891483+0000", 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2899999999999999e-06 2024-09-02T10:29:32.807 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891489+0000", 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8830000000000003e-06 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891502+0000", 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3086e-05 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.808 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891508+0000", 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9889999999999997e-06 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891569+0000", 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1042000000000003e-05 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891577+0000", 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1289999999999996e-06 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.809 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231610+0000", 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34003330300000001 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231622+0000", 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1881e-05 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.810 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:154719 7.6 e749/741 7:65814d4b:::benchmark_data_smithi187_120712_object70777:head v 749'9238, mlcod=749'9235)", 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 134.73516225099999, 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26617824499999998, 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.811 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3999999999999997e-07 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904629+0000", 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5419999999999996e-06 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.812 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.164896+0000", 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26026724899999998 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.164907+0000", 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0862999999999999e-05 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.165442+0000", 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053543600000000003 2024-09-02T10:29:32.813 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.165465+0000", 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2442e-05 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.170745+0000", 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0052807360000000003 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.170801+0000", 2024-09-02T10:29:32.814 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5637000000000003e-05 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:7047 8.0 8:007ff306:::benchmark_data_smithi187_136059_object7046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:29:03.222949+0000", 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 29.416835479, 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.16784033400000001, 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:29:32.815 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7047 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.222949+0000", 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.222949+0000", 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:29:32.816 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.222958+0000", 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5369999999999997e-06 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.222989+0000", 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.057e-05 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.223000+0000", 2024-09-02T10:29:32.817 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1956999999999999e-05 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.223006+0000", 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0749999999999997e-06 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.223053+0000", 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7808e-05 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:29:32.818 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.223189+0000", 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013560699999999999 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,9", 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.223343+0000", 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015440000000000001 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.223791+0000", 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00044728200000000002 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.819 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.224092+0000", 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00030158300000000002 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.390719+0000", 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.166627104 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.390757+0000", 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7997999999999998e-05 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:29:32.820 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:29:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:29:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.390790+0000", 2024-09-02T10:29:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2413000000000003e-05 2024-09-02T10:29:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:29:32.821 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:29:32.821 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:29:32.821 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:32.822 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:32 smithi187 ceph-mon[20072]: pgmap v1848: 47 pgs: 47 active+clean; 2.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:29:32.845 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:32.846 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:32.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:32.975 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:32.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:32.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:33.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:33.111 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:33.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:33.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 34 16 48090 48074 88.364 79.3125 0.0204369 0.0113112 2024-09-02T10:29:33.213 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:33.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:29:33.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:33.244 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:33.293 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:29:33.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:33.368 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:33.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:33.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:33.494 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:33.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:33 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:29:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:33 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:29:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/324651723' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:33.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:33 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:33.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:33.618 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:33.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:33.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:33 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:29:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:33 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:29:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/324651723' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:33.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:33 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:33.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:33.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:33.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:33.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:33 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:29:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:33 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:29:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/324651723' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:33 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:33.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:33.866 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:33.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:33.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:29:33.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:33.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:33.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:34.095 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:34.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 35 16 49310 49294 88.0177 76.25 0.0143523 0.0113532 2024-09-02T10:29:34.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:34.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:34.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:34.175 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:34.191 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:34.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:34.228 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:34.248 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:34.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:34.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:34.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:34.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:34.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:34.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:34.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:34.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:34.566 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:34 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:34.566 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:34 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:29:34.566 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:34 smithi151 ceph-mon[16639]: osdmap e813: 12 total, 12 up, 12 in 2024-09-02T10:29:34.566 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:34 smithi151 ceph-mon[16639]: pgmap v1850: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:29:34.566 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:34 smithi151 ceph-mon[16639]: osdmap e814: 12 total, 12 up, 12 in 2024-09-02T10:29:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:34 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:34 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:29:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:34 smithi162 ceph-mon[20279]: osdmap e813: 12 total, 12 up, 12 in 2024-09-02T10:29:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:34 smithi162 ceph-mon[20279]: pgmap v1850: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:29:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:34 smithi162 ceph-mon[20279]: osdmap e814: 12 total, 12 up, 12 in 2024-09-02T10:29:34.603 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:34.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:34.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:34.631 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:34.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:34 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:34 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:29:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:34 smithi187 ceph-mon[20072]: osdmap e813: 12 total, 12 up, 12 in 2024-09-02T10:29:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:34 smithi187 ceph-mon[20072]: pgmap v1850: 47 pgs: 47 active+clean; 2.8 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 82 MiB/s wr, 1.31k op/s 2024-09-02T10:29:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:34 smithi187 ceph-mon[20072]: osdmap e814: 12 total, 12 up, 12 in 2024-09-02T10:29:34.764 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:34.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:34.765 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:34.773 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:29:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:29:34.773 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:29:34.773 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:29:34.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:34.896 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:34.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:34.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:35.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:35.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:35.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 36 16 50327 50311 87.3383 63.5625 0.00280553 0.0114422 2024-09-02T10:29:35.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:35.155 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:35.156 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:35.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:35.278 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:35.278 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:35.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:35.401 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:35.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:35.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:35.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:35.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:35.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:35.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:35.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:35.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:35.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:35.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:35.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:35.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:35.920 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:35.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:36.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:36.047 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:36.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:36.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 37 16 51319 51303 86.6532 62 0.00859336 0.0115338 2024-09-02T10:29:36.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:36.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:36.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:36.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:36.306 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:36.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:36.408 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:36.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:36.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:36.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:36.579 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:36.581 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:36.597 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-09-02T10:29:36.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:36.722 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:36.723 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:36.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:29:36.824 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:36 smithi162 ceph-mon[20279]: pgmap v1852: 47 pgs: 47 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:29:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3944653065' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:29:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:36 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:29:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:29:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:29:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:29:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:36 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:29:36.848 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:36.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:36 smithi151 ceph-mon[16639]: pgmap v1852: 47 pgs: 47 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:29:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3944653065' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:29:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:36 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:29:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:29:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:29:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:29:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:36 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:29:36.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:36.981 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:36.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:36 smithi187 ceph-mon[20072]: pgmap v1852: 47 pgs: 47 active+clean; 2.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.17k op/s 2024-09-02T10:29:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3944653065' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:29:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:36 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:29:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:29:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:29:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:29:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:36 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:29:37.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:37.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:37.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:37.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:37.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 38 16 52604 52588 86.4862 80.3125 0.0143329 0.0115555 2024-09-02T10:29:37.212 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:37.241 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:37.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:37.243 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:37.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:37.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:37.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:37.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:37.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:37.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:37.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:37.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-09-02T10:29:37.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:37.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:37.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:37.757 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:37.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:37.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:37.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:37 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:29:37.864 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:37 smithi151 ceph-mon[16639]: osdmap e815: 12 total, 12 up, 12 in 2024-09-02T10:29:37.884 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:37.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:37.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:37.919 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:29:37.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:29:37.948 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:29:37.949 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:29:37.953 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:37.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:37.987 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:29:38.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:37 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:29:38.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:37 smithi187 ceph-mon[20072]: osdmap e815: 12 total, 12 up, 12 in 2024-09-02T10:29:38.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:38.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:38.013 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:29:38.013 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:29:38.039 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:29:38.039 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:29:38.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:29:38.067 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:29:38.090 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:29:38.090 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:29:38.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:37 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:29:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:37 smithi162 ceph-mon[20279]: osdmap e815: 12 total, 12 up, 12 in 2024-09-02T10:29:38.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:38.114 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:29:38.115 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:29:38.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 39 16 53867 53851 86.2925 78.9375 0.0437492 0.0115793 2024-09-02T10:29:38.117 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:38.139 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:38.140 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:29:38.140 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:29:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:38.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:38.162 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:29:38.163 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:29:38.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:29:38.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:29:38.205 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:29:38.205 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:29:38.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:38.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:38.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:38.266 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:38.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:38.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:38.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:38.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:38.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:29:38.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:38.520 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:38.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:38.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:38.650 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:38.651 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:38.653 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:38.653 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:38.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:38.774 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:38.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:38 smithi162 ceph-mon[20279]: pgmap v1854: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-02T10:29:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2625491394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:38.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:38 smithi151 ceph-mon[16639]: pgmap v1854: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-02T10:29:38.870 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2625491394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:38.876 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:38.900 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:38.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:38.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:39.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:38 smithi187 ceph-mon[20072]: pgmap v1854: 47 pgs: 47 active+clean; 3.1 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 91 MiB/s wr, 1.45k op/s 2024-09-02T10:29:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2625491394' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:39.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:39.038 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:39.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:29:39.116590+0000 min lat: 0.00107552 max lat: 0.623325 avg lat: 0.0116463 2024-09-02T10:29:39.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:29:39.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 40 16 54944 54928 85.8178 67.3125 0.0040739 0.0116463 2024-09-02T10:29:39.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:39.163 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:39.164 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:39.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:39.281 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-09-02T10:29:39.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:39.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:39.299 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:39.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:39.414 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:39.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:39.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:29:39.517 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:39.540 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:39.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:39.541 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:39.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:39.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:39.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:39.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:39.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:39.797 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:39.799 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:39.800 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3949101033' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:29:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:39 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:29:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:39 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:29:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:39 smithi162 ceph-mon[20279]: osdmap e816: 12 total, 12 up, 12 in 2024-09-02T10:29:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:39.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:39.862 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3949101033' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:29:39.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:39 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:29:39.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:39 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:29:39.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:39 smithi151 ceph-mon[16639]: osdmap e816: 12 total, 12 up, 12 in 2024-09-02T10:29:39.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:39.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:39.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:39.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:39.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3949101033' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:29:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:39 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:29:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:39 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:29:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:39 smithi187 ceph-mon[20072]: osdmap e816: 12 total, 12 up, 12 in 2024-09-02T10:29:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:40.026 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:40.056 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:40.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:40.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:40.116 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 41 16 55891 55875 85.1682 59.1875 0.0340111 0.0117334 2024-09-02T10:29:40.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:40.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:40.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:40.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:40.290 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:40.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:40.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:40.316 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:40.347 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:40.347 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:40.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:40.439 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:40.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:40.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:40.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:40.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:40.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:40.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 2 1 2024-09-02T10:29:40.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:40.702 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:40.702 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:40.804 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:40.829 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:40.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:40.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:40.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:40 smithi151 ceph-mon[16639]: pgmap v1856: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-02T10:29:40.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4036585864' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:40.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:40 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:40.954 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:40.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:40.972 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:40 smithi187 ceph-mon[20072]: pgmap v1856: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-02T10:29:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4036585864' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:40 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:41.020 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:40 smithi162 ceph-mon[20279]: pgmap v1856: 47 pgs: 47 active+clean; 3.2 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 MiB/s wr, 1.16k op/s 2024-09-02T10:29:41.020 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4036585864' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:41.020 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:40 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:41.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:29:41.057 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:41.081 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:41.082 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:41.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 42 16 56897 56881 84.6373 62.875 0.117084 0.0118046 2024-09-02T10:29:41.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:41.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:41.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:41.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:41.244 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:29:41.288 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:29:41.310 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:41.334 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:41.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:41.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:41.465 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:41.466 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:41.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:41.595 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:41.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:41.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:41.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:41.722 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:41.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:41.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:41.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:41 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:41 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:41 smithi162 ceph-mon[20279]: osdmap e817: 12 total, 12 up, 12 in 2024-09-02T10:29:41.846 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:41.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:41.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:41 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:41.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:41 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:41.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:41 smithi151 ceph-mon[16639]: osdmap e817: 12 total, 12 up, 12 in 2024-09-02T10:29:41.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:41.965 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:41.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:41.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:41.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:41 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:41 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:41 smithi187 ceph-mon[20072]: osdmap e817: 12 total, 12 up, 12 in 2024-09-02T10:29:42.074 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:42.096 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:42.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:42.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 43 16 57976 57960 84.2372 67.4375 0.00439912 0.0118651 2024-09-02T10:29:42.198 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:42.228 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:42.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:42.229 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:42.331 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:42.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:42.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:42.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:42.463 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:42.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:42.493 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:42.498 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:42.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:42.618 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:42.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:42.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:42.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:42.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:42.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:42.870 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:42.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:42.898 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:42 smithi162 ceph-mon[20279]: pgmap v1858: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 961 op/s 2024-09-02T10:29:42.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:42 smithi151 ceph-mon[16639]: pgmap v1858: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 961 op/s 2024-09-02T10:29:42.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:42.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:42.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:42 smithi187 ceph-mon[20072]: pgmap v1858: 47 pgs: 47 active+clean; 3.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 961 op/s 2024-09-02T10:29:43.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:43.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 44 16 59144 59128 83.9816 73 0.0312062 0.0118996 2024-09-02T10:29:43.123 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:29:43.123 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:29:43.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:43.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:43.147 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:29:43.148 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:29:43.172 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:29:43.172 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:29:43.197 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:29:43.197 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:29:43.217 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:29:43.217 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:29:43.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:43.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:29:43.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:29:43.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:43.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:43.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:29:43.269 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:29:43.295 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:29:43.295 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:29:43.319 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:29:43.319 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:29:43.345 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:29:43.345 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:29:43.354 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:43.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:29:43.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:29:43.376 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:43.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:43.392 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:29:43.392 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:29:43.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:43.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:43.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:43.504 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:43.522 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:43.522 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:43.605 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:43.615 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-09-02T10:29:43.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:43.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:43.633 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:43.734 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:43.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:43.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:43.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3468407961' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:29:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:43 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:29:43.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/930098697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:43.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:43.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:43.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:43.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3468407961' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:29:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:43 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:29:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/930098697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:43.987 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3468407961' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:29:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:43 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:29:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/930098697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:44.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:44.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:44.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:44.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 45 16 60150 60134 83.5125 62.875 0.0131552 0.0119679 2024-09-02T10:29:44.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:44.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:44.140 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:44.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:44.249 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:29:44.264 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:44.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:44.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:44.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:44.395 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:44.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:44.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:44.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:44.524 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:44.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:44.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:44.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:44.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:29:44.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:44.779 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:44.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:44.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:44.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:44.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:44 smithi151 ceph-mon[16639]: pgmap v1859: 47 pgs: 47 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-09-02T10:29:44.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:44 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:29:44.907 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:44 smithi151 ceph-mon[16639]: osdmap e818: 12 total, 12 up, 12 in 2024-09-02T10:29:44.909 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:44.910 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:44.944 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-02T10:29:44.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:29:44.961 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:44 smithi187 ceph-mon[20072]: pgmap v1859: 47 pgs: 47 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-09-02T10:29:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:44 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:29:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:44 smithi187 ceph-mon[20072]: osdmap e818: 12 total, 12 up, 12 in 2024-09-02T10:29:45.010 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:45.031 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:45.038 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:45.039 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:45.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:44 smithi162 ceph-mon[20279]: pgmap v1859: 47 pgs: 47 active+clean; 3.5 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 85 MiB/s wr, 1.37k op/s 2024-09-02T10:29:45.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:44 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:29:45.088 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:44 smithi162 ceph-mon[20279]: osdmap e818: 12 total, 12 up, 12 in 2024-09-02T10:29:45.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 46 16 61389 61373 83.3803 77.4375 0.0160508 0.0119882 2024-09-02T10:29:45.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:45.166 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:45.167 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:45.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:45.294 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:45.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:45.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:45.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:45.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:45.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:45.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:45.493 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:29:45.493 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:29:45.493 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:29:45.493 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:29:45.493 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:29:45.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:45.571 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:45.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:45.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:45.712 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:45.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:45.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:45.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:45.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:45.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:45.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:45.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:45.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:45.973 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:45.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:45.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:46.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:46.102 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:46.103 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:46.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 47 16 62779 62763 83.4545 86.875 0.00395785 0.0119769 2024-09-02T10:29:46.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:29:46.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:46.234 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:46.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:46.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:46.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:29:46.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:46.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:46.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:46.364 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:46.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:46.495 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:46.496 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:46.549 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:29:46.550 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:29:46.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:46.624 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:29:46.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:46.627 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:46.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:46.752 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:46.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:46.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:46 smithi162 ceph-mon[20279]: pgmap v1861: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 964 op/s 2024-09-02T10:29:46.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1897358269' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:29:46.755 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:46 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:29:46.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:46 smithi151 ceph-mon[16639]: pgmap v1861: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 964 op/s 2024-09-02T10:29:46.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1897358269' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:29:46.818 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:46 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:29:46.839 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:46.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:46.879 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:46.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:46.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:46.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:47.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:47.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:46 smithi187 ceph-mon[20072]: pgmap v1861: 47 pgs: 47 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 964 op/s 2024-09-02T10:29:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1897358269' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:29:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:46 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:29:47.053 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:47.106 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:47.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 48 16 63971 63955 83.2678 74.5 0.0169454 0.012004 2024-09-02T10:29:47.130 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:47.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:47.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:47.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:47.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:47.254 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:47.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:47.377 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:29:47.377 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:29:47.377 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:29:47.377 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:29:47.377 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.377 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:135981 6.f 6:fb1fb44a:::benchmark_data_smithi187_102448_object35709:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.144443+0000", 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 534.22984333299996, 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30923446599999999, 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135981 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.378 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144443+0000", 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144443+0000", 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144445+0000", 2024-09-02T10:29:47.379 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8819999999999999e-06 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144451+0000", 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0379999999999996e-06 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144462+0000", 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0954999999999999e-05 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.380 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144470+0000", 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1349999999999992e-06 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.147086+0000", 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026162360000000001 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.147101+0000", 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5065000000000001e-05 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.381 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.438894+0000", 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29179251899999997 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.442304+0000", 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034102730000000001 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,9,10", 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.442472+0000", 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016778999999999999 2024-09-02T10:29:47.382 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.451929+0000", 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0094568650000000001 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.451945+0000", 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5993000000000001e-05 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.451993+0000", 2024-09-02T10:29:47.383 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.88e-05 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.452029+0000", 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5639000000000002e-05 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.452067+0000", 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7465000000000001e-05 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.384 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.453677+0000", 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001610811 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:135985 6.f 6:f0ede17e:::benchmark_data_smithi187_102448_object35713:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.144486+0000", 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 534.22980029999997, 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.34346973200000003, 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.385 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135985 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144486+0000", 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144486+0000", 2024-09-02T10:29:47.386 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144486+0000", 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7e-07 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144488+0000", 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.902e-06 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.387 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144493+0000", 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5959999999999997e-06 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144496+0000", 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4070000000000002e-06 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.147137+0000", 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026404140000000002 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.388 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.147142+0000", 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6810000000000003e-06 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.448184+0000", 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.301041901 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.448283+0000", 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8607e-05 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.389 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,9,10", 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.448371+0000", 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8688000000000002e-05 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.453841+0000", 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0054700069999999998 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.453858+0000", 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6373999999999999e-05 2024-09-02T10:29:47.390 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.453905+0000", 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7147000000000001e-05 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.485839+0000", 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.031934322000000001 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.391 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.485894+0000", 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4892999999999998e-05 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.487956+0000", 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020614230000000002 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:135987 6.f 6:f2d06ddf:::benchmark_data_smithi187_102448_object35715:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.144502+0000", 2024-09-02T10:29:47.392 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 534.22978424400003, 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.34525122400000002, 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135987 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.393 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144502+0000", 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144502+0000", 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144502+0000", 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.53e-07 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.394 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144504+0000", 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.361e-06 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144508+0000", 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2400000000000001e-06 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144512+0000", 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8399999999999997e-06 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.395 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.147188+0000", 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026762370000000001 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.147198+0000", 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9720000000000004e-06 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.453942+0000", 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30674434299999997 2024-09-02T10:29:47.396 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.454095+0000", 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015266600000000001 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,9,10", 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.454244+0000", 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014888 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.397 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.462538+0000", 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0082938249999999995 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.462578+0000", 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0540000000000001e-05 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.466986+0000", 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044079560000000002 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.398 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.488014+0000", 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.021027441000000001 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.488064+0000", 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0591999999999999e-05 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.489753+0000", 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016890780000000001 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.399 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:136012 6.f 6:faa86e2e:::benchmark_data_smithi187_102448_object35740:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.144516+0000", 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 534.22977049199994, 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.34722961899999999, 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:47.400 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136012 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144516+0000", 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144516+0000", 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.401 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144516+0000", 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4900000000000002e-07 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144517+0000", 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2929999999999999e-06 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144520+0000", 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2279999999999998e-06 2024-09-02T10:29:47.402 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144523+0000", 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9890000000000001e-06 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.147264+0000", 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027407270000000001 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.147274+0000", 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5510000000000005e-06 2024-09-02T10:29:47.403 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.458415+0000", 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.311141006 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.458561+0000", 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014663100000000001 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,9,10", 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.458708+0000", 2024-09-02T10:29:47.404 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014709099999999999 2024-09-02T10:29:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:47.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.406 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.406 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.471254+0000", 2024-09-02T10:29:47.406 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012545163 2024-09-02T10:29:47.406 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.406 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.471330+0000", 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6774999999999996e-05 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.471351+0000", 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0273000000000001e-05 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.489818+0000", 2024-09-02T10:29:47.407 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018467772 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.489868+0000", 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9410999999999998e-05 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.491745+0000", 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018774600000000001 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.408 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:136043 6.f 6:f9189865:::benchmark_data_smithi187_102448_object35771:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e587)", 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:20:53.144527+0000", 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 534.22975931099995, 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36173080600000002, 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136043 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.409 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144527+0000", 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144527+0000", 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144527+0000", 2024-09-02T10:29:47.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2000000000000001e-07 2024-09-02T10:29:47.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144528+0000", 2024-09-02T10:29:47.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.176e-06 2024-09-02T10:29:47.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144532+0000", 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4800000000000001e-06 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.144536+0000", 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1930000000000003e-06 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.147295+0000", 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027591500000000001 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.147299+0000", 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2799999999999997e-06 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.462618+0000", 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31531838800000001 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.462781+0000", 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016368399999999999 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,9,10", 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.462890+0000", 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108941 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.471394+0000", 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0085038100000000005 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.471439+0000", 2024-09-02T10:29:47.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4311999999999999e-05 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.471454+0000", 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5301000000000001e-05 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.504507+0000", 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.033053072000000003 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.504558+0000", 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1208000000000002e-05 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:20:53.506258+0000", 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016995910000000001 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(6.f e629,586 6:fcd958cb:::benchmark_data_smithi187_102448_object21195:head,579'7484)", 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:48.751501+0000", 2024-09-02T10:29:47.416 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 418.62278541299997, 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.54048170500000003, 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:48.751501+0000", 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:48.751501+0000", 2024-09-02T10:29:47.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:48.751502+0000", 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8100000000000001e-07 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:48.751510+0000", 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9209999999999992e-06 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.291397+0000", 2024-09-02T10:29:47.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.53988699900000003 2024-09-02T10:29:47.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.291410+0000", 2024-09-02T10:29:47.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3529e-05 2024-09-02T10:29:47.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.419 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.419 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.419 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.291582+0000", 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000171774 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.291983+0000", 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040050099999999999 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:3735 7.0 7:0bf36798:::benchmark_data_smithi187_120712_object3734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e666)", 2024-09-02T10:29:47.420 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:24:14.834852+0000", 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 332.53943421899999, 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21021257700000001, 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3735 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834852+0000", 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834852+0000", 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834856+0000", 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3290000000000004e-06 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834901+0000", 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4601000000000002e-05 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834912+0000", 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0749e-05 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834919+0000", 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1559999999999998e-06 2024-09-02T10:29:47.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834960+0000", 2024-09-02T10:29:47.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.138e-05 2024-09-02T10:29:47.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.835059+0000", 2024-09-02T10:29:47.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8540000000000002e-05 2024-09-02T10:29:47.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.835139+0000", 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920999999999996e-05 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.835504+0000", 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036490200000000001 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.835891+0000", 2024-09-02T10:29:47.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038723099999999998 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:15.044999+0000", 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209108608 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:15.045036+0000", 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6634000000000002e-05 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:15.045065+0000", 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8526000000000001e-05 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:8296 7.0 7:06654a29:::benchmark_data_smithi187_120712_object8295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:24:18.337323+0000", 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 329.03696280399998, 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21776643000000001, 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.427 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 8296 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337323+0000", 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337323+0000", 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337324+0000", 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5499999999999996e-07 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337407+0000", 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2157000000000006e-05 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337420+0000", 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.382e-05 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337426+0000", 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3360000000000002e-06 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337473+0000", 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.757e-05 2024-09-02T10:29:47.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337586+0000", 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113199 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337673+0000", 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6339000000000001e-05 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.338207+0000", 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00053426400000000001 2024-09-02T10:29:47.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.338706+0000", 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049890899999999996 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.555033+0000", 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21632686100000001 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.555064+0000", 2024-09-02T10:29:47.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1501000000000002e-05 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.555090+0000", 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5519000000000001e-05 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:55534 7.0 7:0cb06b0e:::benchmark_data_smithi187_120712_object55533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:07.797396+0000", 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.57689020399999, 2024-09-02T10:29:47.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.247318706, 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55534 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797396+0000", 2024-09-02T10:29:47.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797396+0000", 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797398+0000", 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6619999999999999e-06 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797481+0000", 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3710999999999999e-05 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797497+0000", 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5536000000000002e-05 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797505+0000", 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3610000000000001e-06 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797566+0000", 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1115e-05 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797709+0000", 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014281 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797855+0000", 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014629700000000001 2024-09-02T10:29:47.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.799694+0000", 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018388269999999999 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800428+0000", 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073367199999999999 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044600+0000", 2024-09-02T10:29:47.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.244171631 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044664+0000", 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4329e-05 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044715+0000", 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0754999999999998e-05 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:55537 7.0 7:0ece8a38:::benchmark_data_smithi187_120712_object55536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:07.800289+0000", 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.57399678799999, 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.244576865, 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55537 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.440 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800289+0000", 2024-09-02T10:29:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800289+0000", 2024-09-02T10:29:47.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800291+0000", 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6649999999999999e-06 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800356+0000", 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4936000000000003e-05 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800368+0000", 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1798e-05 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800375+0000", 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3329999999999999e-06 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800454+0000", 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8854000000000001e-05 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800548+0000", 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4458999999999998e-05 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800631+0000", 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2812000000000006e-05 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.802282+0000", 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001650861 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.802849+0000", 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056718700000000003 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044778+0000", 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.241928643 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044830+0000", 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1833000000000003e-05 2024-09-02T10:29:47.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044866+0000", 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6483999999999998e-05 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:116084 7.0 7:07eb9730:::benchmark_data_smithi187_120712_object32142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e731)", 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:18.919750+0000", 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 208.45453595000001, 2024-09-02T10:29:47.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222370436, 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 116084 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919750+0000", 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919750+0000", 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919752+0000", 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.59e-06 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919764+0000", 2024-09-02T10:29:47.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2203e-05 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919779+0000", 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4457999999999999e-05 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919785+0000", 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8979999999999999e-06 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.922669+0000", 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028836299999999999 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.922803+0000", 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133972 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.922916+0000", 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113479 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.926242+0000", 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033256919999999999 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.926412+0000", 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016997299999999999 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.142054+0000", 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21564232799999999 2024-09-02T10:29:47.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.142092+0000", 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7494999999999999e-05 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.142121+0000", 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8717999999999998e-05 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:116098 7.0 7:07a476f9:::benchmark_data_smithi187_120712_object32156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e731)", 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:18.931907+0000", 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 208.44237903499999, 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22223031300000001, 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 116098 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.453 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931907+0000", 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931907+0000", 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931909+0000", 2024-09-02T10:29:47.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3039999999999999e-06 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931918+0000", 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2239999999999995e-06 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931939+0000", 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1155e-05 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931949+0000", 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6979999999999993e-06 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.932018+0000", 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9766999999999998e-05 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.932138+0000", 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119337 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:29:47.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.932252+0000", 2024-09-02T10:29:47.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114528 2024-09-02T10:29:47.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.457 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.457 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.936161+0000", 2024-09-02T10:29:47.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003908758 2024-09-02T10:29:47.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.936253+0000", 2024-09-02T10:29:47.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2379000000000004e-05 2024-09-02T10:29:47.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.154031+0000", 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21777723500000001 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.154088+0000", 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7559000000000002e-05 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.154137+0000", 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9369000000000001e-05 2024-09-02T10:29:47.459 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:135858 7.10 7:0b5a73a5:::benchmark_data_smithi187_120712_object51916:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:41.733794+0000", 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 185.64049239600001, 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.072650061, 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.460 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135858 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733794+0000", 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733794+0000", 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733795+0000", 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6080000000000001e-06 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733802+0000", 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1120000000000001e-06 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733819+0000", 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7064000000000001e-05 2024-09-02T10:29:47.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733828+0000", 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2900000000000008e-06 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.946794+0000", 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21296567199999999 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:47.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.946804+0000", 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0910999999999999e-05 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.806352+0000", 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.85954753399999995 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.806444+0000", 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.187e-05 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:136917 7.0 7:009dde76:::benchmark_data_smithi187_120712_object52975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:43.760124+0000", 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 183.614162443, 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33985228200000001, 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.466 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136917 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760124+0000", 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760124+0000", 2024-09-02T10:29:47.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760126+0000", 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8610000000000001e-06 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760131+0000", 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.575e-06 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760144+0000", 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2942000000000001e-05 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760150+0000", 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1179999999999997e-06 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760198+0000", 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7546000000000002e-05 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760283+0000", 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5561000000000003e-05 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760352+0000", 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8633000000000005e-05 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762399+0000", 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020470330000000002 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762673+0000", 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027366000000000001 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099886+0000", 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33721340999999999 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099938+0000", 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1635000000000003e-05 2024-09-02T10:29:47.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099976+0000", 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8308e-05 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:136917 7.0 e743/741 ondisk, result = 0)", 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:43.762433+0000", 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 183.61185307400001, 2024-09-02T10:29:47.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33754895299999998, 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762433+0000", 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762433+0000", 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762434+0000", 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.02e-06 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762440+0000", 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0449999999999997e-06 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099808+0000", 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33736824799999998 2024-09-02T10:29:47.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099816+0000", 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1799999999999999e-06 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099871+0000", 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5730999999999999e-05 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099881+0000", 2024-09-02T10:29:47.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5759999999999995e-06 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099982+0000", 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101153 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:7006 8.8 8:1d422b62:::benchmark_data_smithi187_136059_object7005:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:29:03.201313+0000", 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.172973304000003, 2024-09-02T10:29:47.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18690911700000001, 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7006 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.201313+0000", 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.201313+0000", 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.201314+0000", 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9400000000000005e-07 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.201372+0000", 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8177999999999998e-05 2024-09-02T10:29:47.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.201382+0000", 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0339e-05 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.201387+0000", 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5390000000000001e-06 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.201421+0000", 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4338999999999998e-05 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.201499+0000", 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8359e-05 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.201579+0000", 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9418000000000004e-05 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.205643+0000", 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040646290000000002 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.223594+0000", 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.017950544999999998 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.388109+0000", 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16451544900000001 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.388173+0000", 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3730000000000001e-05 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.388222+0000", 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8897999999999998e-05 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.482 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:7036 8.4 8:2df0ab2f:::benchmark_data_smithi187_136059_object7035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:29:03.216472+0000", 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.157814385999998, 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.1735746, 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 7036 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.216472+0000", 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.216472+0000", 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.216473+0000", 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7599999999999996e-07 2024-09-02T10:29:47.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.216520+0000", 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.702e-05 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.216531+0000", 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1256e-05 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.216536+0000", 2024-09-02T10:29:47.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4310000000000002e-06 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.216580+0000", 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3538000000000002e-05 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.216671+0000", 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0659000000000001e-05 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:29:47.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.216761+0000", 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0676999999999999e-05 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.217206+0000", 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00044452199999999998 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.217716+0000", 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00050994899999999999 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.389975+0000", 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17225889699999999 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.390010+0000", 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5191000000000002e-05 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.390046+0000", 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6584000000000001e-05 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.488 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27559.0:7006 8.8 e800/792 ondisk, result = 0)", 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:29:03.217870+0000", 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 44.156416649999997, 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17035873700000001, 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.217870+0000", 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.217870+0000", 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.217870+0000", 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9399999999999995e-07 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.217873+0000", 2024-09-02T10:29:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.399e-06 2024-09-02T10:29:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.388013+0000", 2024-09-02T10:29:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17013929899999999 2024-09-02T10:29:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.388017+0000", 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5719999999999996e-06 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.388093+0000", 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6125999999999996e-05 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.388102+0000", 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.524e-06 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:03.388228+0000", 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012632299999999999 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:48920 8.8 8:1673ed4b:::benchmark_data_smithi187_136059_object48919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:29:33.671380+0000", 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 13.702906235, 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209033422, 2024-09-02T10:29:47.494 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48920 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671380+0000", 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671380+0000", 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671381+0000", 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.09e-07 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671439+0000", 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8641999999999999e-05 2024-09-02T10:29:47.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671450+0000", 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0216e-05 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671454+0000", 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3100000000000002e-06 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671518+0000", 2024-09-02T10:29:47.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3646999999999996e-05 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671637+0000", 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119815 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671809+0000", 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000171776 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:47.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.673208+0000", 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013985340000000001 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.677308+0000", 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041007040000000002 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880310+0000", 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203001297 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.499 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880365+0000", 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5544000000000002e-05 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880413+0000", 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8127999999999997e-05 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27559.0:48920 8.8 e813/792 ondisk, result = 0)", 2024-09-02T10:29:47.500 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:29:33.674779+0000", 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 13.699507197999999, 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20564031099999999, 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.674779+0000", 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.674779+0000", 2024-09-02T10:29:47.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.674780+0000", 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8700000000000005e-07 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.674784+0000", 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2810000000000002e-06 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880258+0000", 2024-09-02T10:29:47.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20547411600000001 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880261+0000", 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2349999999999999e-06 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880295+0000", 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3649999999999998e-05 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:47.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880304+0000", 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0100000000000001e-06 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880419+0000", 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115332 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:47.504 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:29:47.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:47.586 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:47.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:47.688 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:47.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:47.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:47.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:47.835 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:47.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:47.836 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:47 smithi162 ceph-mon[20279]: from='client.27673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:29:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:47 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:47 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:29:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:47 smithi162 ceph-mon[20279]: osdmap e819: 12 total, 12 up, 12 in 2024-09-02T10:29:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:29:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:29:47.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:47.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:47 smithi151 ceph-mon[16639]: from='client.27673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:29:47.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:47 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:47.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:47 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:29:47.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:47 smithi151 ceph-mon[16639]: osdmap e819: 12 total, 12 up, 12 in 2024-09-02T10:29:47.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:47.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:47.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:29:47.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:29:47.888 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:47.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:47.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:47.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:47.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:47 smithi187 ceph-mon[20072]: from='client.27673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:29:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:47 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:29:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:47 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:29:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:47 smithi187 ceph-mon[20072]: osdmap e819: 12 total, 12 up, 12 in 2024-09-02T10:29:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:29:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:29:48.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:29:48.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:48.085 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:48.085 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:48.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 49 15 64759 64744 82.5748 49.3125 0.00877651 0.0121047 2024-09-02T10:29:48.181 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:29:48.181 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:29:48.186 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:48.215 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:29:48.215 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:29:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:48.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:48.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:48.247 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:29:48.247 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:29:48.277 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:29:48.277 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:29:48.302 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:29:48.302 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:29:48.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:48.330 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:29:48.330 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:29:48.348 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:48.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:48.349 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:48.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:29:48.355 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:29:48.382 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:29:48.382 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:29:48.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:29:48.408 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:29:48.408 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:29:48.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:29:48.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:29:48.450 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:48.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:29:48.458 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:29:48.473 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:48.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:48.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:48.487 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:29:48.488 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:29:48.574 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:48.602 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:48.603 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:48.703 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:48.704 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:48.732 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1861,"stamp":"2024-09-02T10:29:45.515880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3860922912,"num_objects":58906,"num_object_clones":0,"num_object_copies":176718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58904,"num_write_kb":3769856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61756,"ondisk_log_size":61756,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":18382100,"kb_used_data":11230132,"kb_used_omap":18,"kb_used_meta":7151725,"kb_avail":1106559724,"statfs":{"total":1151940427776,"available":1133117157376,"internally_reserved":0,"allocated":11499655168,"data_stored":11495282315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":7323366715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":394199040,"num_objects":6015,"num_object_clones":0,"num_object_copies":18045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6015,"num_write_kb":384960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.238329"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":1021,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.289929+0000","last_change":"2024-09-02T10:28:52.430883+0000","last_active":"2024-09-02T10:29:41.289929+0000","last_peered":"2024-09-02T10:29:41.289929+0000","last_clean":"2024-09-02T10:29:41.289929+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:29:41.289929+0000","last_undegraded":"2024-09-02T10:29:41.289929+0000","last_fullsized":"2024-09-02T10:29:41.289929+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:52.430813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:52.430813+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:07.918104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":831,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.620562+0000","last_change":"2024-09-02T10:28:48.078204+0000","last_active":"2024-09-02T10:29:43.620562+0000","last_peered":"2024-09-02T10:29:43.620562+0000","last_clean":"2024-09-02T10:29:43.620562+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:29:43.620562+0000","last_undegraded":"2024-09-02T10:29:43.620562+0000","last_fullsized":"2024-09-02T10:29:43.620562+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:48.078152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:48.078152+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:54.410021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1298,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.616402+0000","last_change":"2024-09-02T10:28:52.456682+0000","last_active":"2024-09-02T10:29:43.616402+0000","last_peered":"2024-09-02T10:29:43.616402+0000","last_clean":"2024-09-02T10:29:43.616402+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:29:43.616402+0000","last_undegraded":"2024-09-02T10:29:43.616402+0000","last_fullsized":"2024-09-02T10:29:43.616402+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:52.456625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:52.456625+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:06.978613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1282,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.616317+0000","last_change":"2024-09-02T10:28:50.517611+0000","last_active":"2024-09-02T10:29:43.616317+0000","last_peered":"2024-09-02T10:29:43.616317+0000","last_clean":"2024-09-02T10:29:43.616317+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:29:43.616317+0000","last_undegraded":"2024-09-02T10:29:43.616317+0000","last_fullsized":"2024-09-02T10:29:43.616317+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:50.517550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.517550+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:16.401759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":585,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.620285+0000","last_change":"2024-09-02T10:29:01.533562+0000","last_active":"2024-09-02T10:29:43.620285+0000","last_peered":"2024-09-02T10:29:43.620285+0000","last_clean":"2024-09-02T10:29:43.620285+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:29:43.620285+0000","last_undegraded":"2024-09-02T10:29:43.620285+0000","last_fullsized":"2024-09-02T10:29:43.620285+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:01.533495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:01.533495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:14.562689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":1142,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.292421+0000","last_change":"2024-09-02T10:28:49.258215+0000","last_active":"2024-09-02T10:29:41.292421+0000","last_peered":"2024-09-02T10:29:41.292421+0000","last_clean":"2024-09-02T10:29:41.292421+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:29:41.292421+0000","last_undegraded":"2024-09-02T10:29:41.292421+0000","last_fullsized":"2024-09-02T10:29:41.292421+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:49.258156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:49.258156+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:56.969009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1661,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.291374+0000","last_change":"2024-09-02T10:28:47.961752+0000","last_active":"2024-09-02T10:29:41.291374+0000","last_peered":"2024-09-02T10:29:41.291374+0000","last_clean":"2024-09-02T10:29:41.291374+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:29:41.291374+0000","last_undegraded":"2024-09-02T10:29:41.291374+0000","last_fullsized":"2024-09-02T10:29:41.291374+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:47.961678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:47.961678+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:55.766413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1305,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.616503+0000","last_change":"2024-09-02T10:28:47.556551+0000","last_active":"2024-09-02T10:29:43.616503+0000","last_peered":"2024-09-02T10:29:43.616503+0000","last_clean":"2024-09-02T10:29:43.616503+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:29:43.616503+0000","last_undegraded":"2024-09-02T10:29:43.616503+0000","last_fullsized":"2024-09-02T10:29:43.616503+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:47.556478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:47.556478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:05.628713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":1311,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.619438+0000","last_change":"2024-09-02T10:28:54.049984+0000","last_active":"2024-09-02T10:29:43.619438+0000","last_peered":"2024-09-02T10:29:43.619438+0000","last_clean":"2024-09-02T10:29:43.619438+0000","last_became_active":"2024-09-02T10:25:01.211125+0000","last_became_peered":"2024-09-02T10:25:01.211125+0000","last_unstale":"2024-09-02T10:29:43.619438+0000","last_undegraded":"2024-09-02T10:29:43.619438+0000","last_fullsized":"2024-09-02T10:29:43.619438+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:54.049936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:54.049936+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:00.725690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1270,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.292443+0000","last_change":"2024-09-02T10:29:22.476684+0000","last_active":"2024-09-02T10:29:41.292443+0000","last_peered":"2024-09-02T10:29:41.292443+0000","last_clean":"2024-09-02T10:29:41.292443+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:29:41.292443+0000","last_undegraded":"2024-09-02T10:29:41.292443+0000","last_fullsized":"2024-09-02T10:29:41.292443+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:22.476642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:22.476642+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:43.763817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":514,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.285019+0000","last_change":"2024-09-02T10:28:48.100781+0000","last_active":"2024-09-02T10:29:41.285019+0000","last_peered":"2024-09-02T10:29:41.285019+0000","last_clean":"2024-09-02T10:29:41.285019+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:29:41.285019+0000","last_undegraded":"2024-09-02T10:29:41.285019+0000","last_fullsized":"2024-09-02T10:29:41.285019+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:48.100733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:48.100733+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:05.383446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1622,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.619659+0000","last_change":"2024-09-02T10:28:49.036559+0000","last_active":"2024-09-02T10:29:43.619659+0000","last_peered":"2024-09-02T10:29:43.619659+0000","last_clean":"2024-09-02T10:29:43.619659+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:29:43.619659+0000","last_undegraded":"2024-09-02T10:29:43.619659+0000","last_fullsized":"2024-09-02T10:29:43.619659+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:49.036511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:49.036511+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:16.505702+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1560,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.291298+0000","last_change":"2024-09-02T10:28:46.987718+0000","last_active":"2024-09-02T10:29:41.291298+0000","last_peered":"2024-09-02T10:29:41.291298+0000","last_clean":"2024-09-02T10:29:41.291298+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:29:41.291298+0000","last_undegraded":"2024-09-02T10:29:41.291298+0000","last_fullsized":"2024-09-02T10:29:41.291298+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:46.987655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:46.987655+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:11.197274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":284,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.288921+0000","last_change":"2024-09-02T10:28:48.043362+0000","last_active":"2024-09-02T10:29:41.288921+0000","last_peered":"2024-09-02T10:29:41.288921+0000","last_clean":"2024-09-02T10:29:41.288921+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:29:41.288921+0000","last_undegraded":"2024-09-02T10:29:41.288921+0000","last_fullsized":"2024-09-02T10:29:41.288921+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:48.043289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:48.043289+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:50.931931+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1530,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.291416+0000","last_change":"2024-09-02T10:28:51.944117+0000","last_active":"2024-09-02T10:29:41.291416+0000","last_peered":"2024-09-02T10:29:41.291416+0000","last_clean":"2024-09-02T10:29:41.291416+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:29:41.291416+0000","last_undegraded":"2024-09-02T10:29:41.291416+0000","last_fullsized":"2024-09-02T10:29:41.291416+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:51.944062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:51.944062+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:58.245654+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"818'3684","reported_seq":3747,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.858857+0000","last_change":"2024-09-02T10:28:51.932251+0000","last_active":"2024-09-02T10:29:43.858857+0000","last_peered":"2024-09-02T10:29:43.858857+0000","last_clean":"2024-09-02T10:29:43.858857+0000","last_became_active":"2024-09-02T10:28:51.931930+0000","last_became_peered":"2024-09-02T10:28:51.931930+0000","last_unstale":"2024-09-02T10:29:43.858857+0000","last_undegraded":"2024-09-02T10:29:43.858857+0000","last_fullsized":"2024-09-02T10:29:43.858857+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3684,"log_dups_size":0,"ondisk_log_size":3684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:57.164235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241434624,"num_objects":3684,"num_object_clones":0,"num_object_copies":11052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3684,"num_write_kb":235776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1624,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.291455+0000","last_change":"2024-09-02T10:28:55.953519+0000","last_active":"2024-09-02T10:29:41.291455+0000","last_peered":"2024-09-02T10:29:41.291455+0000","last_clean":"2024-09-02T10:29:41.291455+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:29:41.291455+0000","last_undegraded":"2024-09-02T10:29:41.291455+0000","last_fullsized":"2024-09-02T10:29:41.291455+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:55.953460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:55.953460+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:07.072567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"818'3792","reported_seq":3854,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.906884+0000","last_change":"2024-09-02T10:28:51.652862+0000","last_active":"2024-09-02T10:29:43.906884+0000","last_peered":"2024-09-02T10:29:43.906884+0000","last_clean":"2024-09-02T10:29:43.906884+0000","last_became_active":"2024-09-02T10:28:51.652599+0000","last_became_peered":"2024-09-02T10:28:51.652599+0000","last_unstale":"2024-09-02T10:29:43.906884+0000","last_undegraded":"2024-09-02T10:29:43.906884+0000","last_fullsized":"2024-09-02T10:29:43.906884+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3792,"log_dups_size":0,"ondisk_log_size":3792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:09.609839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248512512,"num_objects":3792,"num_object_clones":0,"num_object_copies":11376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3792,"num_write_kb":242688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1749,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.292330+0000","last_change":"2024-09-02T10:28:47.136735+0000","last_active":"2024-09-02T10:29:41.292330+0000","last_peered":"2024-09-02T10:29:41.292330+0000","last_clean":"2024-09-02T10:29:41.292330+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:29:41.292330+0000","last_undegraded":"2024-09-02T10:29:41.292330+0000","last_fullsized":"2024-09-02T10:29:41.292330+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:47.136663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:47.136663+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:01.269934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"817'3690","reported_seq":3750,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:42.349597+0000","last_change":"2024-09-02T10:28:51.651512+0000","last_active":"2024-09-02T10:29:42.349597+0000","last_peered":"2024-09-02T10:29:42.349597+0000","last_clean":"2024-09-02T10:29:42.349597+0000","last_became_active":"2024-09-02T10:28:51.651237+0000","last_became_peered":"2024-09-02T10:28:51.651237+0000","last_unstale":"2024-09-02T10:29:42.349597+0000","last_undegraded":"2024-09-02T10:29:42.349597+0000","last_fullsized":"2024-09-02T10:29:42.349597+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:59.354913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241827840,"num_objects":3690,"num_object_clones":0,"num_object_copies":11070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3690,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3690,"num_write_kb":236160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1393,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.288682+0000","last_change":"2024-09-02T10:28:48.285450+0000","last_active":"2024-09-02T10:29:41.288682+0000","last_peered":"2024-09-02T10:29:41.288682+0000","last_clean":"2024-09-02T10:29:41.288682+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:29:41.288682+0000","last_undegraded":"2024-09-02T10:29:41.288682+0000","last_fullsized":"2024-09-02T10:29:41.288682+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:48.285368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:48.285368+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:55.329615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"817'3633","reported_seq":3694,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.919507+0000","last_change":"2024-09-02T10:28:51.668966+0000","last_active":"2024-09-02T10:29:41.919507+0000","last_peered":"2024-09-02T10:29:41.919507+0000","last_clean":"2024-09-02T10:29:41.919507+0000","last_became_active":"2024-09-02T10:28:51.668558+0000","last_became_peered":"2024-09-02T10:28:51.668558+0000","last_unstale":"2024-09-02T10:29:41.919507+0000","last_undegraded":"2024-09-02T10:29:41.919507+0000","last_fullsized":"2024-09-02T10:29:41.919507+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:59.884382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238092288,"num_objects":3633,"num_object_clones":0,"num_object_copies":10899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3633,"num_write_kb":232512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":780,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.289795+0000","last_change":"2024-09-02T10:26:07.441225+0000","last_active":"2024-09-02T10:29:41.289795+0000","last_peered":"2024-09-02T10:29:41.289795+0000","last_clean":"2024-09-02T10:29:41.289795+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:29:41.289795+0000","last_undegraded":"2024-09-02T10:29:41.289795+0000","last_fullsized":"2024-09-02T10:29:41.289795+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:07.441159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:07.441159+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.d","version":"817'3633","reported_seq":3694,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:42.311517+0000","last_change":"2024-09-02T10:28:51.933292+0000","last_active":"2024-09-02T10:29:42.311517+0000","last_peered":"2024-09-02T10:29:42.311517+0000","last_clean":"2024-09-02T10:29:42.311517+0000","last_became_active":"2024-09-02T10:28:51.932338+0000","last_became_peered":"2024-09-02T10:28:51.932338+0000","last_unstale":"2024-09-02T10:29:42.311517+0000","last_undegraded":"2024-09-02T10:29:42.311517+0000","last_fullsized":"2024-09-02T10:29:42.311517+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3633,"log_dups_size":0,"ondisk_log_size":3633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:51.834588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238092288,"num_objects":3633,"num_object_clones":0,"num_object_copies":10899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3633,"num_write_kb":232512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1689,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.289820+0000","last_change":"2024-09-02T10:26:13.596384+0000","last_active":"2024-09-02T10:29:41.289820+0000","last_peered":"2024-09-02T10:29:41.289820+0000","last_clean":"2024-09-02T10:29:41.289820+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:29:41.289820+0000","last_undegraded":"2024-09-02T10:29:41.289820+0000","last_fullsized":"2024-09-02T10:29:41.289820+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:13.596339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.596339+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"817'3702","reported_seq":3763,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:42.925899+0000","last_change":"2024-09-02T10:28:51.653019+0000","last_active":"2024-09-02T10:29:42.925899+0000","last_peered":"2024-09-02T10:29:42.925899+0000","last_clean":"2024-09-02T10:29:42.925899+0000","last_became_active":"2024-09-02T10:28:51.652606+0000","last_became_peered":"2024-09-02T10:28:51.652606+0000","last_unstale":"2024-09-02T10:29:42.925899+0000","last_undegraded":"2024-09-02T10:29:42.925899+0000","last_fullsized":"2024-09-02T10:29:42.925899+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3702,"log_dups_size":0,"ondisk_log_size":3702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:11.138883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242614272,"num_objects":3702,"num_object_clones":0,"num_object_copies":11106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3702,"num_write_kb":236928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":1164,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.292790+0000","last_change":"2024-09-02T10:29:00.307437+0000","last_active":"2024-09-02T10:29:41.292790+0000","last_peered":"2024-09-02T10:29:41.292790+0000","last_clean":"2024-09-02T10:29:41.292790+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:29:41.292790+0000","last_undegraded":"2024-09-02T10:29:41.292790+0000","last_fullsized":"2024-09-02T10:29:41.292790+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:00.307367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:00.307367+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:25.326214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"817'3538","reported_seq":3599,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:42.323575+0000","last_change":"2024-09-02T10:28:51.932940+0000","last_active":"2024-09-02T10:29:42.323575+0000","last_peered":"2024-09-02T10:29:42.323575+0000","last_clean":"2024-09-02T10:29:42.323575+0000","last_became_active":"2024-09-02T10:28:51.932528+0000","last_became_peered":"2024-09-02T10:28:51.932528+0000","last_unstale":"2024-09-02T10:29:42.323575+0000","last_undegraded":"2024-09-02T10:29:42.323575+0000","last_fullsized":"2024-09-02T10:29:42.323575+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3538,"log_dups_size":0,"ondisk_log_size":3538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:05.406510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231866368,"num_objects":3538,"num_object_clones":0,"num_object_copies":10614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3538,"num_write_kb":226432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":1129,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.293804+0000","last_change":"2024-09-02T10:28:47.273603+0000","last_active":"2024-09-02T10:29:41.293804+0000","last_peered":"2024-09-02T10:29:41.293804+0000","last_clean":"2024-09-02T10:29:41.293804+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:29:41.293804+0000","last_undegraded":"2024-09-02T10:29:41.293804+0000","last_fullsized":"2024-09-02T10:29:41.293804+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:47.273542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:47.273542+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:50.537830+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"817'3667","reported_seq":3728,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:42.640629+0000","last_change":"2024-09-02T10:28:51.932270+0000","last_active":"2024-09-02T10:29:42.640629+0000","last_peered":"2024-09-02T10:29:42.640629+0000","last_clean":"2024-09-02T10:29:42.640629+0000","last_became_active":"2024-09-02T10:28:51.931936+0000","last_became_peered":"2024-09-02T10:28:51.931936+0000","last_unstale":"2024-09-02T10:29:42.640629+0000","last_undegraded":"2024-09-02T10:29:42.640629+0000","last_fullsized":"2024-09-02T10:29:42.640629+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3667,"log_dups_size":0,"ondisk_log_size":3667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:18.707827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240320512,"num_objects":3667,"num_object_clones":0,"num_object_copies":11001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3667,"num_write_kb":234688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":911,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.285720+0000","last_change":"2024-09-02T10:28:49.154105+0000","last_active":"2024-09-02T10:29:41.285720+0000","last_peered":"2024-09-02T10:29:41.285720+0000","last_clean":"2024-09-02T10:29:41.285720+0000","last_became_active":"2024-09-02T10:25:02.065175+0000","last_became_peered":"2024-09-02T10:25:02.065175+0000","last_unstale":"2024-09-02T10:29:41.285720+0000","last_undegraded":"2024-09-02T10:29:41.285720+0000","last_fullsized":"2024-09-02T10:29:41.285720+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:49.154043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:49.154043+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:06.042892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"817'3734","reported_seq":3795,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:42.571571+0000","last_change":"2024-09-02T10:28:51.652867+0000","last_active":"2024-09-02T10:29:42.571571+0000","last_peered":"2024-09-02T10:29:42.571571+0000","last_clean":"2024-09-02T10:29:42.571571+0000","last_became_active":"2024-09-02T10:28:51.652541+0000","last_became_peered":"2024-09-02T10:28:51.652541+0000","last_unstale":"2024-09-02T10:29:42.571571+0000","last_undegraded":"2024-09-02T10:29:42.571571+0000","last_fullsized":"2024-09-02T10:29:42.571571+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3734,"log_dups_size":0,"ondisk_log_size":3734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:17.961249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":244711424,"num_objects":3734,"num_object_clones":0,"num_object_copies":11202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3734,"num_write_kb":238976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":1144,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.292526+0000","last_change":"2024-09-02T10:28:55.242301+0000","last_active":"2024-09-02T10:29:41.292526+0000","last_peered":"2024-09-02T10:29:41.292526+0000","last_clean":"2024-09-02T10:29:41.292526+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:29:41.292526+0000","last_undegraded":"2024-09-02T10:29:41.292526+0000","last_fullsized":"2024-09-02T10:29:41.292526+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:55.242242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:55.242242+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:12.221925+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"818'3781","reported_seq":3843,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:44.524890+0000","last_change":"2024-09-02T10:28:51.668957+0000","last_active":"2024-09-02T10:29:44.524890+0000","last_peered":"2024-09-02T10:29:44.524890+0000","last_clean":"2024-09-02T10:29:44.524890+0000","last_became_active":"2024-09-02T10:28:51.668549+0000","last_became_peered":"2024-09-02T10:28:51.668549+0000","last_unstale":"2024-09-02T10:29:44.524890+0000","last_undegraded":"2024-09-02T10:29:44.524890+0000","last_fullsized":"2024-09-02T10:29:44.524890+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:54.920722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247791616,"num_objects":3781,"num_object_clones":0,"num_object_copies":11343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3781,"num_write_kb":241984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1391,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.292377+0000","last_change":"2024-09-02T10:28:53.261629+0000","last_active":"2024-09-02T10:29:41.292377+0000","last_peered":"2024-09-02T10:29:41.292377+0000","last_clean":"2024-09-02T10:29:41.292377+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:29:41.292377+0000","last_undegraded":"2024-09-02T10:29:41.292377+0000","last_fullsized":"2024-09-02T10:29:41.292377+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:53.261572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:53.261572+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:00.182258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"818'3722","reported_seq":3784,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:44.533206+0000","last_change":"2024-09-02T10:28:51.932687+0000","last_active":"2024-09-02T10:29:44.533206+0000","last_peered":"2024-09-02T10:29:44.533206+0000","last_clean":"2024-09-02T10:29:44.533206+0000","last_became_active":"2024-09-02T10:28:51.932243+0000","last_became_peered":"2024-09-02T10:28:51.932243+0000","last_unstale":"2024-09-02T10:29:44.533206+0000","last_undegraded":"2024-09-02T10:29:44.533206+0000","last_fullsized":"2024-09-02T10:29:44.533206+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:53.389270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243924992,"num_objects":3722,"num_object_clones":0,"num_object_copies":11166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3722,"num_write_kb":238208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1654,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.619853+0000","last_change":"2024-09-02T10:28:51.032822+0000","last_active":"2024-09-02T10:29:43.619853+0000","last_peered":"2024-09-02T10:29:43.619853+0000","last_clean":"2024-09-02T10:29:43.619853+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:29:43.619853+0000","last_undegraded":"2024-09-02T10:29:43.619853+0000","last_fullsized":"2024-09-02T10:29:43.619853+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:51.032772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:51.032772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:06.504659+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"817'3658","reported_seq":3719,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.958133+0000","last_change":"2024-09-02T10:28:51.651229+0000","last_active":"2024-09-02T10:29:41.958133+0000","last_peered":"2024-09-02T10:29:41.958133+0000","last_clean":"2024-09-02T10:29:41.958133+0000","last_became_active":"2024-09-02T10:28:51.650885+0000","last_became_peered":"2024-09-02T10:28:51.650885+0000","last_unstale":"2024-09-02T10:29:41.958133+0000","last_undegraded":"2024-09-02T10:29:41.958133+0000","last_fullsized":"2024-09-02T10:29:41.958133+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3658,"log_dups_size":0,"ondisk_log_size":3658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:57.436857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239730688,"num_objects":3658,"num_object_clones":0,"num_object_copies":10974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3658,"num_write_kb":234112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1395,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.292323+0000","last_change":"2024-09-02T10:28:52.220285+0000","last_active":"2024-09-02T10:29:41.292323+0000","last_peered":"2024-09-02T10:29:41.292323+0000","last_clean":"2024-09-02T10:29:41.292323+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:29:41.292323+0000","last_undegraded":"2024-09-02T10:29:41.292323+0000","last_fullsized":"2024-09-02T10:29:41.292323+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:52.220218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:52.220218+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:57.411003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"817'3611","reported_seq":3671,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:42.349590+0000","last_change":"2024-09-02T10:28:51.933496+0000","last_active":"2024-09-02T10:29:42.349590+0000","last_peered":"2024-09-02T10:29:42.349590+0000","last_clean":"2024-09-02T10:29:42.349590+0000","last_became_active":"2024-09-02T10:28:51.933116+0000","last_became_peered":"2024-09-02T10:28:51.933116+0000","last_unstale":"2024-09-02T10:29:42.349590+0000","last_undegraded":"2024-09-02T10:29:42.349590+0000","last_fullsized":"2024-09-02T10:29:42.349590+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:00.906748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":236650496,"num_objects":3611,"num_object_clones":0,"num_object_copies":10833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3611,"num_write_kb":231104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1658,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.285800+0000","last_change":"2024-09-02T10:28:53.146541+0000","last_active":"2024-09-02T10:29:41.285800+0000","last_peered":"2024-09-02T10:29:41.285800+0000","last_clean":"2024-09-02T10:29:41.285800+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:29:41.285800+0000","last_undegraded":"2024-09-02T10:29:41.285800+0000","last_fullsized":"2024-09-02T10:29:41.285800+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:53.146476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:53.146476+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:09.095447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"818'3769","reported_seq":3832,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.903128+0000","last_change":"2024-09-02T10:28:51.651017+0000","last_active":"2024-09-02T10:29:43.903128+0000","last_peered":"2024-09-02T10:29:43.903128+0000","last_clean":"2024-09-02T10:29:43.903128+0000","last_became_active":"2024-09-02T10:28:51.650798+0000","last_became_peered":"2024-09-02T10:28:51.650798+0000","last_unstale":"2024-09-02T10:29:43.903128+0000","last_undegraded":"2024-09-02T10:29:43.903128+0000","last_fullsized":"2024-09-02T10:29:43.903128+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3769,"log_dups_size":0,"ondisk_log_size":3769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:18.357633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247005184,"num_objects":3769,"num_object_clones":0,"num_object_copies":11307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3769,"num_write_kb":241216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":600,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.619184+0000","last_change":"2024-09-02T10:29:18.561961+0000","last_active":"2024-09-02T10:29:43.619184+0000","last_peered":"2024-09-02T10:29:43.619184+0000","last_clean":"2024-09-02T10:29:43.619184+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:29:43.619184+0000","last_undegraded":"2024-09-02T10:29:43.619184+0000","last_fullsized":"2024-09-02T10:29:43.619184+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:18.561913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:18.561913+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:27.219602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"817'3643","reported_seq":3703,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:42.578993+0000","last_change":"2024-09-02T10:28:51.651937+0000","last_active":"2024-09-02T10:29:42.578993+0000","last_peered":"2024-09-02T10:29:42.578993+0000","last_clean":"2024-09-02T10:29:42.578993+0000","last_became_active":"2024-09-02T10:28:51.651609+0000","last_became_peered":"2024-09-02T10:28:51.651609+0000","last_unstale":"2024-09-02T10:29:42.578993+0000","last_undegraded":"2024-09-02T10:29:42.578993+0000","last_fullsized":"2024-09-02T10:29:42.578993+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3643,"log_dups_size":0,"ondisk_log_size":3643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:16.543808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238747648,"num_objects":3643,"num_object_clones":0,"num_object_copies":10929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3643,"num_write_kb":233152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1430,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.292499+0000","last_change":"2024-09-02T10:29:19.460821+0000","last_active":"2024-09-02T10:29:41.292499+0000","last_peered":"2024-09-02T10:29:41.292499+0000","last_clean":"2024-09-02T10:29:41.292499+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:29:41.292499+0000","last_undegraded":"2024-09-02T10:29:41.292499+0000","last_fullsized":"2024-09-02T10:29:41.292499+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:19.460773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:19.460773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:29.978713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"817'3647","reported_seq":3706,"reported_epoch":817,"state":"active+clean","last_fresh":"2024-09-02T10:29:41.935565+0000","last_change":"2024-09-02T10:28:51.668957+0000","last_active":"2024-09-02T10:29:41.935565+0000","last_peered":"2024-09-02T10:29:41.935565+0000","last_clean":"2024-09-02T10:29:41.935565+0000","last_became_active":"2024-09-02T10:28:51.668549+0000","last_became_peered":"2024-09-02T10:28:51.668549+0000","last_unstale":"2024-09-02T10:29:41.935565+0000","last_undegraded":"2024-09-02T10:29:41.935565+0000","last_fullsized":"2024-09-02T10:29:41.935565+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3647,"log_dups_size":0,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:59.955444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":239009792,"num_objects":3647,"num_object_clones":0,"num_object_copies":10941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3647,"num_write_kb":233408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":815,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.612555+0000","last_change":"2024-09-02T10:28:51.687935+0000","last_active":"2024-09-02T10:29:43.612555+0000","last_peered":"2024-09-02T10:29:43.612555+0000","last_clean":"2024-09-02T10:29:43.612555+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:29:43.612555+0000","last_undegraded":"2024-09-02T10:29:43.612555+0000","last_fullsized":"2024-09-02T10:29:43.612555+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:51.687888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:51.687888+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:10.849080+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3860332544,"num_objects":58904,"num_object_clones":0,"num_object_copies":176712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":58904,"num_write_kb":3769856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11591811072,"data_stored":11591811072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":58904,"ondisk_log_size":58904,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384035,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1802064,"kb_used_data":1161404,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":91943088,"statfs":{"total":95995035648,"available":94149722112,"internally_reserved":0,"allocated":1189277696,"data_stored":1188914548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":656013764},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612968,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1850496,"kb_used_data":1372272,"kb_used_omap":1,"kb_used_meta":478206,"kb_avail":91894656,"statfs":{"total":95995035648,"available":94100127744,"internally_reserved":0,"allocated":1405206528,"data_stored":1404847833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":489683392},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841901,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2109964,"kb_used_data":1402684,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":91635188,"statfs":{"total":95995035648,"available":93834432512,"internally_reserved":0,"allocated":1436348416,"data_stored":1435985268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":724236736},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070836,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1126300,"kb_used_data":716104,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92618852,"statfs":{"total":95995035648,"available":94841704448,"internally_reserved":0,"allocated":733290496,"data_stored":732922903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299768,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1113548,"kb_used_data":454840,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":92631604,"statfs":{"total":95995035648,"available":94854762496,"internally_reserved":0,"allocated":465756160,"data_stored":465393913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":674494922},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528700,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425360,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93319792,"statfs":{"total":95995035648,"available":95559467008,"internally_reserved":0,"allocated":1306624,"data_stored":943476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":434239943},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724929,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1853144,"kb_used_data":1168068,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":91892008,"statfs":{"total":95995035648,"available":94097416192,"internally_reserved":0,"allocated":1196101632,"data_stored":1195734932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":701495749},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418456,"kb_used_data":972168,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92326696,"statfs":{"total":95995035648,"available":94542536704,"internally_reserved":0,"allocated":995500032,"data_stored":995132439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150092,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1460252,"kb_used_data":725192,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":92284900,"statfs":{"total":95995035648,"available":94499737600,"internally_reserved":0,"allocated":742596608,"data_stored":742229015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":752679368},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411726,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1630984,"kb_used_data":1126392,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92114168,"statfs":{"total":95995035648,"available":94324908032,"internally_reserved":0,"allocated":1153425408,"data_stored":1153063161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575252,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2071132,"kb_used_data":1430472,"kb_used_omap":1,"kb_used_meta":640638,"kb_avail":91674020,"statfs":{"total":95995035648,"available":93874196480,"internally_reserved":0,"allocated":1464803328,"data_stored":1464435735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":656013765},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738778,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1520400,"kb_used_data":699260,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92224752,"statfs":{"total":95995035648,"available":94438146048,"internally_reserved":0,"allocated":716042240,"data_stored":715679092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":726073344,"data_stored":726073344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1465384960,"data_stored":1465384960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1170472960,"data_stored":1170472960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":745078784,"data_stored":745078784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":997064704,"data_stored":997064704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1213988864,"data_stored":1213988864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":467402752,"data_stored":467402752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":742588416,"data_stored":742588416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1448738816,"data_stored":1448738816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1425932288,"data_stored":1425932288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1189085184,"data_stored":1189085184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:29:48.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:29:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:48.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:48.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:48.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:48.863 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:48.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:48.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:48.994 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:48.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:49.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:48 smithi151 ceph-mon[16639]: pgmap v1863: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-02T10:29:49.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3810304543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:49.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:48 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:29:49.055 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:48 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:29:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:48 smithi162 ceph-mon[20279]: pgmap v1863: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-02T10:29:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3810304543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:48 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:29:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:48 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:29:49.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:49.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 50 16 65974 65958 82.4407 75.875 0.00262326 0.0121224 2024-09-02T10:29:49.123 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:49.124 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:49.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:49.254 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:49.255 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:48 smithi187 ceph-mon[20072]: pgmap v1863: 47 pgs: 47 active+clean; 3.8 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.26k op/s 2024-09-02T10:29:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3810304543' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:48 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:29:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:48 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:29:49.357 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:49.382 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:49.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:49.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:49.421 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:49.421 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:49.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:49.507 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:49.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:49.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:49.528 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:49.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:29:49.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:49.657 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:49.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:49.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:49.758 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:49.785 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:49.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:49.786 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:49.887 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:49.916 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:49.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:49.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:50.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:50.040 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:50.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:50.111 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:29:50.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 51 16 67226 67210 82.3584 78.25 0.00355327 0.0121356 2024-09-02T10:29:50.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:50.148 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:50.175 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:50.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:50.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:50.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:50.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:50.305 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:50.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:50 smithi162 ceph-mon[20279]: osdmap e820: 12 total, 12 up, 12 in 2024-09-02T10:29:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:50 smithi162 ceph-mon[20279]: pgmap v1865: 47 pgs: 47 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-02T10:29:50.380 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:29:50.381 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:29:50.387 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:50.406 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:50.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:50.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:50.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:50.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:50.565 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:50.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:50.667 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:50 smithi151 ceph-mon[16639]: osdmap e820: 12 total, 12 up, 12 in 2024-09-02T10:29:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:50 smithi151 ceph-mon[16639]: pgmap v1865: 47 pgs: 47 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-02T10:29:50.694 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:50.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:50 smithi187 ceph-mon[20072]: osdmap e820: 12 total, 12 up, 12 in 2024-09-02T10:29:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:50 smithi187 ceph-mon[20072]: pgmap v1865: 47 pgs: 47 active+clean; 3.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.12k op/s 2024-09-02T10:29:50.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:50.831 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:50.832 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:50.933 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:50.957 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:50.957 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:50.965 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:50.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:51.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:51.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:51.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:51.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:51.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 52 16 68454 68438 82.2504 76.75 0.00550578 0.0121518 2024-09-02T10:29:51.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:51.223 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:51.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:51.325 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:51.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:51.352 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:51.402 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:29:51.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:51.476 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:51.477 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:51.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:51 smithi162 ceph-mon[20279]: from='client.27685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:29:51.601 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:51.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:51.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:51.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:29:51.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:51 smithi151 ceph-mon[16639]: from='client.27685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:29:51.703 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:51.726 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:51.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:51.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:51 smithi187 ceph-mon[20072]: from='client.27685 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:29:51.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:51.851 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:51.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:51.871 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1865,"stamp":"2024-09-02T10:29:49.516738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4194173472,"num_objects":63991,"num_object_clones":0,"num_object_copies":191973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63989,"num_write_kb":4095296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66841,"ondisk_log_size":66841,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19350244,"kb_used_data":12167756,"kb_used_omap":18,"kb_used_meta":7182253,"kb_avail":1105591580,"statfs":{"total":1151940427776,"available":1132125777920,"internally_reserved":0,"allocated":12459782144,"data_stored":12455329201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":7354627387},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,10,4,5,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":440664064,"num_objects":6724,"num_object_clones":0,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":430336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001269"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":1025,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.629344+0000","last_change":"2024-09-02T10:28:52.430883+0000","last_active":"2024-09-02T10:29:46.629344+0000","last_peered":"2024-09-02T10:29:46.629344+0000","last_clean":"2024-09-02T10:29:46.629344+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:29:46.629344+0000","last_undegraded":"2024-09-02T10:29:46.629344+0000","last_fullsized":"2024-09-02T10:29:46.629344+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:52.430813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:52.430813+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:07.918104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":833,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.626906+0000","last_change":"2024-09-02T10:28:48.078204+0000","last_active":"2024-09-02T10:29:46.626906+0000","last_peered":"2024-09-02T10:29:46.626906+0000","last_clean":"2024-09-02T10:29:46.626906+0000","last_became_active":"2024-09-02T10:20:52.103473+0000","last_became_peered":"2024-09-02T10:20:52.103473+0000","last_unstale":"2024-09-02T10:29:46.626906+0000","last_undegraded":"2024-09-02T10:29:46.626906+0000","last_fullsized":"2024-09-02T10:29:46.626906+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:48.078152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:48.078152+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:54.410021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1298,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.616402+0000","last_change":"2024-09-02T10:28:52.456682+0000","last_active":"2024-09-02T10:29:43.616402+0000","last_peered":"2024-09-02T10:29:43.616402+0000","last_clean":"2024-09-02T10:29:43.616402+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:29:43.616402+0000","last_undegraded":"2024-09-02T10:29:43.616402+0000","last_fullsized":"2024-09-02T10:29:43.616402+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:52.456625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:52.456625+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:06.978613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1282,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.616317+0000","last_change":"2024-09-02T10:28:50.517611+0000","last_active":"2024-09-02T10:29:43.616317+0000","last_peered":"2024-09-02T10:29:43.616317+0000","last_clean":"2024-09-02T10:29:43.616317+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:29:43.616317+0000","last_undegraded":"2024-09-02T10:29:43.616317+0000","last_fullsized":"2024-09-02T10:29:43.616317+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:50.517550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.517550+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:16.401759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":585,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.620285+0000","last_change":"2024-09-02T10:29:01.533562+0000","last_active":"2024-09-02T10:29:43.620285+0000","last_peered":"2024-09-02T10:29:43.620285+0000","last_clean":"2024-09-02T10:29:43.620285+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:29:43.620285+0000","last_undegraded":"2024-09-02T10:29:43.620285+0000","last_fullsized":"2024-09-02T10:29:43.620285+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:01.533495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:01.533495+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:14.562689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":1146,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.628396+0000","last_change":"2024-09-02T10:28:49.258215+0000","last_active":"2024-09-02T10:29:46.628396+0000","last_peered":"2024-09-02T10:29:46.628396+0000","last_clean":"2024-09-02T10:29:46.628396+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:29:46.628396+0000","last_undegraded":"2024-09-02T10:29:46.628396+0000","last_fullsized":"2024-09-02T10:29:46.628396+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:49.258156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:49.258156+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:56.969009+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1665,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.626951+0000","last_change":"2024-09-02T10:28:47.961752+0000","last_active":"2024-09-02T10:29:46.626951+0000","last_peered":"2024-09-02T10:29:46.626951+0000","last_clean":"2024-09-02T10:29:46.626951+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:29:46.626951+0000","last_undegraded":"2024-09-02T10:29:46.626951+0000","last_fullsized":"2024-09-02T10:29:46.626951+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:47.961678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:47.961678+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:55.766413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1305,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.616503+0000","last_change":"2024-09-02T10:28:47.556551+0000","last_active":"2024-09-02T10:29:43.616503+0000","last_peered":"2024-09-02T10:29:43.616503+0000","last_clean":"2024-09-02T10:29:43.616503+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:29:43.616503+0000","last_undegraded":"2024-09-02T10:29:43.616503+0000","last_fullsized":"2024-09-02T10:29:43.616503+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:47.556478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:47.556478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:05.628713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":1313,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.626839+0000","last_change":"2024-09-02T10:28:54.049984+0000","last_active":"2024-09-02T10:29:46.626839+0000","last_peered":"2024-09-02T10:29:46.626839+0000","last_clean":"2024-09-02T10:29:46.626839+0000","last_became_active":"2024-09-02T10:25:01.211125+0000","last_became_peered":"2024-09-02T10:25:01.211125+0000","last_unstale":"2024-09-02T10:29:46.626839+0000","last_undegraded":"2024-09-02T10:29:46.626839+0000","last_fullsized":"2024-09-02T10:29:46.626839+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:54.049936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:54.049936+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:00.725690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1272,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.627067+0000","last_change":"2024-09-02T10:29:22.476684+0000","last_active":"2024-09-02T10:29:43.627067+0000","last_peered":"2024-09-02T10:29:43.627067+0000","last_clean":"2024-09-02T10:29:43.627067+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:29:43.627067+0000","last_undegraded":"2024-09-02T10:29:43.627067+0000","last_fullsized":"2024-09-02T10:29:43.627067+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:22.476642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:22.476642+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:43.763817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":518,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.627025+0000","last_change":"2024-09-02T10:28:48.100781+0000","last_active":"2024-09-02T10:29:46.627025+0000","last_peered":"2024-09-02T10:29:46.627025+0000","last_clean":"2024-09-02T10:29:46.627025+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:29:46.627025+0000","last_undegraded":"2024-09-02T10:29:46.627025+0000","last_fullsized":"2024-09-02T10:29:46.627025+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:48.100733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:48.100733+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:05.383446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1624,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.626854+0000","last_change":"2024-09-02T10:28:49.036559+0000","last_active":"2024-09-02T10:29:46.626854+0000","last_peered":"2024-09-02T10:29:46.626854+0000","last_clean":"2024-09-02T10:29:46.626854+0000","last_became_active":"2024-09-02T10:09:58.906274+0000","last_became_peered":"2024-09-02T10:09:58.906274+0000","last_unstale":"2024-09-02T10:29:46.626854+0000","last_undegraded":"2024-09-02T10:29:46.626854+0000","last_fullsized":"2024-09-02T10:29:46.626854+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:49.036511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:49.036511+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:16.505702+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1564,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.626943+0000","last_change":"2024-09-02T10:28:46.987718+0000","last_active":"2024-09-02T10:29:46.626943+0000","last_peered":"2024-09-02T10:29:46.626943+0000","last_clean":"2024-09-02T10:29:46.626943+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:29:46.626943+0000","last_undegraded":"2024-09-02T10:29:46.626943+0000","last_fullsized":"2024-09-02T10:29:46.626943+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:46.987655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:46.987655+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:11.197274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":288,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.645433+0000","last_change":"2024-09-02T10:28:48.043362+0000","last_active":"2024-09-02T10:29:46.645433+0000","last_peered":"2024-09-02T10:29:46.645433+0000","last_clean":"2024-09-02T10:29:46.645433+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:29:46.645433+0000","last_undegraded":"2024-09-02T10:29:46.645433+0000","last_fullsized":"2024-09-02T10:29:46.645433+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:48.043289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:48.043289+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:50.931931+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1534,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.627013+0000","last_change":"2024-09-02T10:28:51.944117+0000","last_active":"2024-09-02T10:29:46.627013+0000","last_peered":"2024-09-02T10:29:46.627013+0000","last_clean":"2024-09-02T10:29:46.627013+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:29:46.627013+0000","last_undegraded":"2024-09-02T10:29:46.627013+0000","last_fullsized":"2024-09-02T10:29:46.627013+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:51.944062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:51.944062+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:58.245654+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"819'4055","reported_seq":4119,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:48.876698+0000","last_change":"2024-09-02T10:28:51.932251+0000","last_active":"2024-09-02T10:29:48.876698+0000","last_peered":"2024-09-02T10:29:48.876698+0000","last_clean":"2024-09-02T10:29:48.876698+0000","last_became_active":"2024-09-02T10:28:51.931930+0000","last_became_peered":"2024-09-02T10:28:51.931930+0000","last_unstale":"2024-09-02T10:29:48.876698+0000","last_undegraded":"2024-09-02T10:29:48.876698+0000","last_fullsized":"2024-09-02T10:29:48.876698+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:57.164235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265748480,"num_objects":4055,"num_object_clones":0,"num_object_copies":12165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4055,"num_write_kb":259520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1628,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.627087+0000","last_change":"2024-09-02T10:28:55.953519+0000","last_active":"2024-09-02T10:29:46.627087+0000","last_peered":"2024-09-02T10:29:46.627087+0000","last_clean":"2024-09-02T10:29:46.627087+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:29:46.627087+0000","last_undegraded":"2024-09-02T10:29:46.627087+0000","last_fullsized":"2024-09-02T10:29:46.627087+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:55.953460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:55.953460+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:07.072567+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"819'4184","reported_seq":4247,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:48.894967+0000","last_change":"2024-09-02T10:28:51.652862+0000","last_active":"2024-09-02T10:29:48.894967+0000","last_peered":"2024-09-02T10:29:48.894967+0000","last_clean":"2024-09-02T10:29:48.894967+0000","last_became_active":"2024-09-02T10:28:51.652599+0000","last_became_peered":"2024-09-02T10:28:51.652599+0000","last_unstale":"2024-09-02T10:29:48.894967+0000","last_undegraded":"2024-09-02T10:29:48.894967+0000","last_fullsized":"2024-09-02T10:29:48.894967+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:09.609839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274202624,"num_objects":4184,"num_object_clones":0,"num_object_copies":12552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4184,"num_write_kb":267776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1751,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.626957+0000","last_change":"2024-09-02T10:28:47.136735+0000","last_active":"2024-09-02T10:29:43.626957+0000","last_peered":"2024-09-02T10:29:43.626957+0000","last_clean":"2024-09-02T10:29:43.626957+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:29:43.626957+0000","last_undegraded":"2024-09-02T10:29:43.626957+0000","last_fullsized":"2024-09-02T10:29:43.626957+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:47.136663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:47.136663+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:01.269934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"819'4032","reported_seq":4097,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:47.129249+0000","last_change":"2024-09-02T10:28:51.651512+0000","last_active":"2024-09-02T10:29:47.129249+0000","last_peered":"2024-09-02T10:29:47.129249+0000","last_clean":"2024-09-02T10:29:47.129249+0000","last_became_active":"2024-09-02T10:28:51.651237+0000","last_became_peered":"2024-09-02T10:28:51.651237+0000","last_unstale":"2024-09-02T10:29:47.129249+0000","last_undegraded":"2024-09-02T10:29:47.129249+0000","last_fullsized":"2024-09-02T10:29:47.129249+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:59.354913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264241152,"num_objects":4032,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4032,"num_write_kb":258048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1397,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.626575+0000","last_change":"2024-09-02T10:28:48.285450+0000","last_active":"2024-09-02T10:29:46.626575+0000","last_peered":"2024-09-02T10:29:46.626575+0000","last_clean":"2024-09-02T10:29:46.626575+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:29:46.626575+0000","last_undegraded":"2024-09-02T10:29:46.626575+0000","last_fullsized":"2024-09-02T10:29:46.626575+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:48.285368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:48.285368+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:55.329615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"819'4005","reported_seq":4070,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.948584+0000","last_change":"2024-09-02T10:28:51.668966+0000","last_active":"2024-09-02T10:29:46.948584+0000","last_peered":"2024-09-02T10:29:46.948584+0000","last_clean":"2024-09-02T10:29:46.948584+0000","last_became_active":"2024-09-02T10:28:51.668558+0000","last_became_peered":"2024-09-02T10:28:51.668558+0000","last_unstale":"2024-09-02T10:29:46.948584+0000","last_undegraded":"2024-09-02T10:29:46.948584+0000","last_fullsized":"2024-09-02T10:29:46.948584+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4005,"log_dups_size":0,"ondisk_log_size":4005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:59.884382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262471680,"num_objects":4005,"num_object_clones":0,"num_object_copies":12015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4005,"num_write_kb":256320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":784,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.629198+0000","last_change":"2024-09-02T10:26:07.441225+0000","last_active":"2024-09-02T10:29:46.629198+0000","last_peered":"2024-09-02T10:29:46.629198+0000","last_clean":"2024-09-02T10:29:46.629198+0000","last_became_active":"2024-09-02T10:15:35.207810+0000","last_became_peered":"2024-09-02T10:15:35.207810+0000","last_unstale":"2024-09-02T10:29:46.629198+0000","last_undegraded":"2024-09-02T10:29:46.629198+0000","last_fullsized":"2024-09-02T10:29:46.629198+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:07.441159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:07.441159+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:47.320430+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.d","version":"819'4017","reported_seq":4082,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:47.183914+0000","last_change":"2024-09-02T10:28:51.933292+0000","last_active":"2024-09-02T10:29:47.183914+0000","last_peered":"2024-09-02T10:29:47.183914+0000","last_clean":"2024-09-02T10:29:47.183914+0000","last_became_active":"2024-09-02T10:28:51.932338+0000","last_became_peered":"2024-09-02T10:28:51.932338+0000","last_unstale":"2024-09-02T10:29:47.183914+0000","last_undegraded":"2024-09-02T10:29:47.183914+0000","last_fullsized":"2024-09-02T10:29:47.183914+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:51.834588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263258112,"num_objects":4017,"num_object_clones":0,"num_object_copies":12051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4017,"num_write_kb":257088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,7,2],"acting":[11,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1693,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.629224+0000","last_change":"2024-09-02T10:26:13.596384+0000","last_active":"2024-09-02T10:29:46.629224+0000","last_peered":"2024-09-02T10:29:46.629224+0000","last_clean":"2024-09-02T10:29:46.629224+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:29:46.629224+0000","last_undegraded":"2024-09-02T10:29:46.629224+0000","last_fullsized":"2024-09-02T10:29:46.629224+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:26:13.596339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:26:13.596339+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:50.308900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"819'4041","reported_seq":4106,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:47.930459+0000","last_change":"2024-09-02T10:28:51.653019+0000","last_active":"2024-09-02T10:29:47.930459+0000","last_peered":"2024-09-02T10:29:47.930459+0000","last_clean":"2024-09-02T10:29:47.930459+0000","last_became_active":"2024-09-02T10:28:51.652606+0000","last_became_peered":"2024-09-02T10:28:51.652606+0000","last_unstale":"2024-09-02T10:29:47.930459+0000","last_undegraded":"2024-09-02T10:29:47.930459+0000","last_fullsized":"2024-09-02T10:29:47.930459+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4041,"log_dups_size":0,"ondisk_log_size":4041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:11.138883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264830976,"num_objects":4041,"num_object_clones":0,"num_object_copies":12123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4041,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4041,"num_write_kb":258624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":1168,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.628711+0000","last_change":"2024-09-02T10:29:00.307437+0000","last_active":"2024-09-02T10:29:46.628711+0000","last_peered":"2024-09-02T10:29:46.628711+0000","last_clean":"2024-09-02T10:29:46.628711+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:29:46.628711+0000","last_undegraded":"2024-09-02T10:29:46.628711+0000","last_fullsized":"2024-09-02T10:29:46.628711+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:00.307367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:00.307367+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:25.326214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"819'3888","reported_seq":3953,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:47.308379+0000","last_change":"2024-09-02T10:28:51.932940+0000","last_active":"2024-09-02T10:29:47.308379+0000","last_peered":"2024-09-02T10:29:47.308379+0000","last_clean":"2024-09-02T10:29:47.308379+0000","last_became_active":"2024-09-02T10:28:51.932528+0000","last_became_peered":"2024-09-02T10:28:51.932528+0000","last_unstale":"2024-09-02T10:29:47.308379+0000","last_undegraded":"2024-09-02T10:29:47.308379+0000","last_fullsized":"2024-09-02T10:29:47.308379+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3888,"log_dups_size":0,"ondisk_log_size":3888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:05.406510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254803968,"num_objects":3888,"num_object_clones":0,"num_object_copies":11664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3888,"num_write_kb":248832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":1133,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.628780+0000","last_change":"2024-09-02T10:28:47.273603+0000","last_active":"2024-09-02T10:29:46.628780+0000","last_peered":"2024-09-02T10:29:46.628780+0000","last_clean":"2024-09-02T10:29:46.628780+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:29:46.628780+0000","last_undegraded":"2024-09-02T10:29:46.628780+0000","last_fullsized":"2024-09-02T10:29:46.628780+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:47.273542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:47.273542+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:50.537830+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"819'4024","reported_seq":4089,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:47.658231+0000","last_change":"2024-09-02T10:28:51.932270+0000","last_active":"2024-09-02T10:29:47.658231+0000","last_peered":"2024-09-02T10:29:47.658231+0000","last_clean":"2024-09-02T10:29:47.658231+0000","last_became_active":"2024-09-02T10:28:51.931936+0000","last_became_peered":"2024-09-02T10:28:51.931936+0000","last_unstale":"2024-09-02T10:29:47.658231+0000","last_undegraded":"2024-09-02T10:29:47.658231+0000","last_fullsized":"2024-09-02T10:29:47.658231+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4024,"log_dups_size":0,"ondisk_log_size":4024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:18.707827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263716864,"num_objects":4024,"num_object_clones":0,"num_object_copies":12072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4024,"num_write_kb":257536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":915,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.621647+0000","last_change":"2024-09-02T10:28:49.154105+0000","last_active":"2024-09-02T10:29:46.621647+0000","last_peered":"2024-09-02T10:29:46.621647+0000","last_clean":"2024-09-02T10:29:46.621647+0000","last_became_active":"2024-09-02T10:25:02.065175+0000","last_became_peered":"2024-09-02T10:25:02.065175+0000","last_unstale":"2024-09-02T10:29:46.621647+0000","last_undegraded":"2024-09-02T10:29:46.621647+0000","last_fullsized":"2024-09-02T10:29:46.621647+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:49.154043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:49.154043+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:06.042892+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"819'4095","reported_seq":4158,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:47.579417+0000","last_change":"2024-09-02T10:28:51.652867+0000","last_active":"2024-09-02T10:29:47.579417+0000","last_peered":"2024-09-02T10:29:47.579417+0000","last_clean":"2024-09-02T10:29:47.579417+0000","last_became_active":"2024-09-02T10:28:51.652541+0000","last_became_peered":"2024-09-02T10:28:51.652541+0000","last_unstale":"2024-09-02T10:29:47.579417+0000","last_undegraded":"2024-09-02T10:29:47.579417+0000","last_fullsized":"2024-09-02T10:29:47.579417+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4095,"log_dups_size":0,"ondisk_log_size":4095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:17.961249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268369920,"num_objects":4095,"num_object_clones":0,"num_object_copies":12285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4095,"num_write_kb":262080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":1148,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.628504+0000","last_change":"2024-09-02T10:28:55.242301+0000","last_active":"2024-09-02T10:29:46.628504+0000","last_peered":"2024-09-02T10:29:46.628504+0000","last_clean":"2024-09-02T10:29:46.628504+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:29:46.628504+0000","last_undegraded":"2024-09-02T10:29:46.628504+0000","last_fullsized":"2024-09-02T10:29:46.628504+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:55.242242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:55.242242+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:12.221925+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"818'3781","reported_seq":3843,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:44.524890+0000","last_change":"2024-09-02T10:28:51.668957+0000","last_active":"2024-09-02T10:29:44.524890+0000","last_peered":"2024-09-02T10:29:44.524890+0000","last_clean":"2024-09-02T10:29:44.524890+0000","last_became_active":"2024-09-02T10:28:51.668549+0000","last_became_peered":"2024-09-02T10:28:51.668549+0000","last_unstale":"2024-09-02T10:29:44.524890+0000","last_undegraded":"2024-09-02T10:29:44.524890+0000","last_fullsized":"2024-09-02T10:29:44.524890+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3781,"log_dups_size":0,"ondisk_log_size":3781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:54.920722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247791616,"num_objects":3781,"num_object_clones":0,"num_object_copies":11343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3781,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3781,"num_write_kb":241984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1393,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.626981+0000","last_change":"2024-09-02T10:28:53.261629+0000","last_active":"2024-09-02T10:29:43.626981+0000","last_peered":"2024-09-02T10:29:43.626981+0000","last_clean":"2024-09-02T10:29:43.626981+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:29:43.626981+0000","last_undegraded":"2024-09-02T10:29:43.626981+0000","last_fullsized":"2024-09-02T10:29:43.626981+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:53.261572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:53.261572+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:00.182258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"818'3722","reported_seq":3784,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:44.533206+0000","last_change":"2024-09-02T10:28:51.932687+0000","last_active":"2024-09-02T10:29:44.533206+0000","last_peered":"2024-09-02T10:29:44.533206+0000","last_clean":"2024-09-02T10:29:44.533206+0000","last_became_active":"2024-09-02T10:28:51.932243+0000","last_became_peered":"2024-09-02T10:28:51.932243+0000","last_unstale":"2024-09-02T10:29:44.533206+0000","last_undegraded":"2024-09-02T10:29:44.533206+0000","last_fullsized":"2024-09-02T10:29:44.533206+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3722,"log_dups_size":0,"ondisk_log_size":3722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:53.389270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":243924992,"num_objects":3722,"num_object_clones":0,"num_object_copies":11166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3722,"num_write_kb":238208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1656,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.626952+0000","last_change":"2024-09-02T10:28:51.032822+0000","last_active":"2024-09-02T10:29:46.626952+0000","last_peered":"2024-09-02T10:29:46.626952+0000","last_clean":"2024-09-02T10:29:46.626952+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:29:46.626952+0000","last_undegraded":"2024-09-02T10:29:46.626952+0000","last_fullsized":"2024-09-02T10:29:46.626952+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:51.032772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:51.032772+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:06.504659+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"819'4036","reported_seq":4101,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.944561+0000","last_change":"2024-09-02T10:28:51.651229+0000","last_active":"2024-09-02T10:29:46.944561+0000","last_peered":"2024-09-02T10:29:46.944561+0000","last_clean":"2024-09-02T10:29:46.944561+0000","last_became_active":"2024-09-02T10:28:51.650885+0000","last_became_peered":"2024-09-02T10:28:51.650885+0000","last_unstale":"2024-09-02T10:29:46.944561+0000","last_undegraded":"2024-09-02T10:29:46.944561+0000","last_fullsized":"2024-09-02T10:29:46.944561+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4036,"log_dups_size":0,"ondisk_log_size":4036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:57.436857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264503296,"num_objects":4036,"num_object_clones":0,"num_object_copies":12108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4036,"num_write_kb":258304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1397,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.626947+0000","last_change":"2024-09-02T10:28:52.220285+0000","last_active":"2024-09-02T10:29:43.626947+0000","last_peered":"2024-09-02T10:29:43.626947+0000","last_clean":"2024-09-02T10:29:43.626947+0000","last_became_active":"2024-09-02T10:15:35.202445+0000","last_became_peered":"2024-09-02T10:15:35.202445+0000","last_unstale":"2024-09-02T10:29:43.626947+0000","last_undegraded":"2024-09-02T10:29:43.626947+0000","last_fullsized":"2024-09-02T10:29:43.626947+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:52.220218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:52.220218+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:57.411003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"819'3971","reported_seq":4036,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:47.332092+0000","last_change":"2024-09-02T10:28:51.933496+0000","last_active":"2024-09-02T10:29:47.332092+0000","last_peered":"2024-09-02T10:29:47.332092+0000","last_clean":"2024-09-02T10:29:47.332092+0000","last_became_active":"2024-09-02T10:28:51.933116+0000","last_became_peered":"2024-09-02T10:28:51.933116+0000","last_unstale":"2024-09-02T10:29:47.332092+0000","last_undegraded":"2024-09-02T10:29:47.332092+0000","last_fullsized":"2024-09-02T10:29:47.332092+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3971,"log_dups_size":0,"ondisk_log_size":3971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:00.906748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260243456,"num_objects":3971,"num_object_clones":0,"num_object_copies":11913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3971,"num_write_kb":254144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,7],"acting":[11,10,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1662,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.621708+0000","last_change":"2024-09-02T10:28:53.146541+0000","last_active":"2024-09-02T10:29:46.621708+0000","last_peered":"2024-09-02T10:29:46.621708+0000","last_clean":"2024-09-02T10:29:46.621708+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:29:46.621708+0000","last_undegraded":"2024-09-02T10:29:46.621708+0000","last_fullsized":"2024-09-02T10:29:46.621708+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:53.146476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:53.146476+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:09.095447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"819'4143","reported_seq":4208,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:48.902325+0000","last_change":"2024-09-02T10:28:51.651017+0000","last_active":"2024-09-02T10:29:48.902325+0000","last_peered":"2024-09-02T10:29:48.902325+0000","last_clean":"2024-09-02T10:29:48.902325+0000","last_became_active":"2024-09-02T10:28:51.650798+0000","last_became_peered":"2024-09-02T10:28:51.650798+0000","last_unstale":"2024-09-02T10:29:48.902325+0000","last_undegraded":"2024-09-02T10:29:48.902325+0000","last_fullsized":"2024-09-02T10:29:48.902325+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4143,"log_dups_size":0,"ondisk_log_size":4143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:18.357633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271515648,"num_objects":4143,"num_object_clones":0,"num_object_copies":12429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4143,"num_write_kb":265152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,10],"acting":[1,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":600,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.619184+0000","last_change":"2024-09-02T10:29:18.561961+0000","last_active":"2024-09-02T10:29:43.619184+0000","last_peered":"2024-09-02T10:29:43.619184+0000","last_clean":"2024-09-02T10:29:43.619184+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:29:43.619184+0000","last_undegraded":"2024-09-02T10:29:43.619184+0000","last_fullsized":"2024-09-02T10:29:43.619184+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:18.561913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:18.561913+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:27.219602+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"819'3986","reported_seq":4051,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:47.516795+0000","last_change":"2024-09-02T10:28:51.651937+0000","last_active":"2024-09-02T10:29:47.516795+0000","last_peered":"2024-09-02T10:29:47.516795+0000","last_clean":"2024-09-02T10:29:47.516795+0000","last_became_active":"2024-09-02T10:28:51.651609+0000","last_became_peered":"2024-09-02T10:28:51.651609+0000","last_unstale":"2024-09-02T10:29:47.516795+0000","last_undegraded":"2024-09-02T10:29:47.516795+0000","last_fullsized":"2024-09-02T10:29:47.516795+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:16.543808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261226496,"num_objects":3986,"num_object_clones":0,"num_object_copies":11958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3986,"num_write_kb":255104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1432,"reported_epoch":818,"state":"active+clean","last_fresh":"2024-09-02T10:29:43.627110+0000","last_change":"2024-09-02T10:29:19.460821+0000","last_active":"2024-09-02T10:29:43.627110+0000","last_peered":"2024-09-02T10:29:43.627110+0000","last_clean":"2024-09-02T10:29:43.627110+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:29:43.627110+0000","last_undegraded":"2024-09-02T10:29:43.627110+0000","last_fullsized":"2024-09-02T10:29:43.627110+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:19.460773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:19.460773+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:29.978713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"819'4009","reported_seq":4073,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.951512+0000","last_change":"2024-09-02T10:28:51.668957+0000","last_active":"2024-09-02T10:29:46.951512+0000","last_peered":"2024-09-02T10:29:46.951512+0000","last_clean":"2024-09-02T10:29:46.951512+0000","last_became_active":"2024-09-02T10:28:51.668549+0000","last_became_peered":"2024-09-02T10:28:51.668549+0000","last_unstale":"2024-09-02T10:29:46.951512+0000","last_undegraded":"2024-09-02T10:29:46.951512+0000","last_fullsized":"2024-09-02T10:29:46.951512+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:28:50.639462+0000","objects_scrubbed":0,"log_size":4009,"log_dups_size":0,"ondisk_log_size":4009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:29:59.955444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262733824,"num_objects":4009,"num_object_clones":0,"num_object_copies":12027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4009,"num_write_kb":256576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":817,"reported_epoch":819,"state":"active+clean","last_fresh":"2024-09-02T10:29:46.629230+0000","last_change":"2024-09-02T10:28:51.687935+0000","last_active":"2024-09-02T10:29:46.629230+0000","last_peered":"2024-09-02T10:29:46.629230+0000","last_clean":"2024-09-02T10:29:46.629230+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:29:46.629230+0000","last_undegraded":"2024-09-02T10:29:46.629230+0000","last_fullsized":"2024-09-02T10:29:46.629230+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:28:51.687888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:28:51.687888+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:30:10.849080+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4193583104,"num_objects":63989,"num_object_clones":0,"num_object_copies":191967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":63989,"num_write_kb":4095296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12569477120,"data_stored":12569477120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63989,"ondisk_log_size":63989,"up":48,"acting":48,"num_store_stats":11},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384036,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1944360,"kb_used_data":1277332,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":91800792,"statfs":{"total":95995035648,"available":94004011008,"internally_reserved":0,"allocated":1307987968,"data_stored":1307615922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":683014596},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612969,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1998808,"kb_used_data":1512264,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":91746344,"statfs":{"total":95995035648,"available":93948256256,"internally_reserved":0,"allocated":1548558336,"data_stored":1548190743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":498203072},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841902,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2235112,"kb_used_data":1536148,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":91510040,"statfs":{"total":95995035648,"available":93706280960,"internally_reserved":0,"allocated":1573015552,"data_stored":1572643506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":715717056},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070836,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1126300,"kb_used_data":716104,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92618852,"statfs":{"total":95995035648,"available":94841704448,"internally_reserved":0,"allocated":733290496,"data_stored":732922903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299769,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1166880,"kb_used_data":504016,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":92578272,"statfs":{"total":95995035648,"available":94800150528,"internally_reserved":0,"allocated":516112384,"data_stored":515741239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":678754762},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528701,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425384,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":93319768,"statfs":{"total":95995035648,"available":95559442432,"internally_reserved":0,"allocated":1331200,"data_stored":959154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":434239943},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724930,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1965488,"kb_used_data":1288732,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":91779664,"statfs":{"total":95995035648,"available":93982375936,"internally_reserved":0,"allocated":1319661568,"data_stored":1319285970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":692976069},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418456,"kb_used_data":972168,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92326696,"statfs":{"total":95995035648,"available":94542536704,"internally_reserved":0,"allocated":995500032,"data_stored":995132439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150093,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1536420,"kb_used_data":797204,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92208732,"statfs":{"total":95995035648,"available":94421741568,"internally_reserved":0,"allocated":816336896,"data_stored":815964850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":756939208},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411727,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1737056,"kb_used_data":1240784,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92008096,"statfs":{"total":95995035648,"available":94216290304,"internally_reserved":0,"allocated":1270562816,"data_stored":1270191671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575253,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2197928,"kb_used_data":1548948,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91547224,"statfs":{"total":95995035648,"available":93744357376,"internally_reserved":0,"allocated":1586122752,"data_stored":1585750706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738779,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1598052,"kb_used_data":772756,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92147100,"statfs":{"total":95995035648,"available":94358630400,"internally_reserved":0,"allocated":791302144,"data_stored":790930098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":795017216,"data_stored":795017216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1613955072,"data_stored":1613955072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1291845632,"data_stored":1291845632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":817889280,"data_stored":817889280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":997064704,"data_stored":997064704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1328545792,"data_stored":1328545792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":517931008,"data_stored":517931008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":742588416,"data_stored":742588416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1584922624,"data_stored":1584922624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1572077568,"data_stored":1572077568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1307639808,"data_stored":1307639808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:29:51.874 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-02T10:29:51.874 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:29:51.874 INFO:tasks.thrashosds.thrasher:inject_pause on osd.11 2024-09-02T10:29:51.874 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-02T10:29:51.874 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-02T10:29:51.874 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 config set filestore_inject_stall 3 2024-09-02T10:29:51.938 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:51.953 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:51.984 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:51.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:51.988 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:52.090 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:52.113 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:52.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:52.117 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 53 16 69250 69234 81.6371 49.75 0.056353 0.0122345 2024-09-02T10:29:52.183 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:29:52.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:52.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:52.244 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:52.302 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:52.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:52 smithi162 ceph-mon[20279]: pgmap v1866: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 945 op/s 2024-09-02T10:29:52.366 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:52.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:52.429 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:52 smithi151 ceph-mon[16639]: pgmap v1866: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 945 op/s 2024-09-02T10:29:52.468 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:52.486 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:52.486 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:52.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:52.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:52.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:52.595 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:52.622 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:52.624 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:52.725 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:52.749 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:52.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:52.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:52 smithi187 ceph-mon[20072]: pgmap v1866: 47 pgs: 47 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 MiB/s wr, 945 op/s 2024-09-02T10:29:52.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:52.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:52.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:52.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:53.002 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:53.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:53.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:53.112 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:29:53.113 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:29:53.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 54 16 70194 70178 81.2178 59 0.0128142 0.0123065 2024-09-02T10:29:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:53.131 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:53.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:53.142 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:29:53.143 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:29:53.144 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:29:53.167 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:29:53.168 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:29:53.193 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:29:53.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:29:53.218 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:29:53.218 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:29:53.233 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:53.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:29:53.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:29:53.257 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:53.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:53.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:53.271 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:29:53.271 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:29:53.295 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:29:53.295 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:29:53.306 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:29:53.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:29:53.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:29:53.354 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:29:53.354 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:29:53.358 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:53.382 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:29:53.382 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:29:53.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:53 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:29:53.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:53 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:29:53.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:53 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:29:53.382 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:53 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:29:53.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1035996401' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:53.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:53 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:29:53.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:53 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:29:53.383 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/648849190' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:53.384 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:53.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:53.411 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:29:53.411 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:29:53.440 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:53.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:29:53.509 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:53.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:29:53.565 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:53 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:29:53.565 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:53 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:29:53.565 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:53 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:29:53.565 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:53 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:29:53.565 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1035996401' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:53.565 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:53 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:29:53.565 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:53 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:29:53.565 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/648849190' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:53.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:53 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:29:53.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:53 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:29:53.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:53 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:29:53.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:53 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:29:53.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1035996401' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:29:53.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:53 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:29:53.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:53 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:29:53.566 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/648849190' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:53.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:53.640 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:53.641 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:53.742 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:53.779 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:53.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:53.780 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:53.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:53.881 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:29:53.882 INFO:teuthology.orchestra.run.smithi187.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-02T10:29:53.882 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:29:53.904 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:53.905 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:54.006 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:54.036 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:54.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:54.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 55 16 71265 71249 80.9581 66.9375 0.00552702 0.0123464 2024-09-02T10:29:54.137 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:54.161 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:54.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:54.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:54.195 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:29:54.195 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:29:54.196 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:29:54.196 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:29:54.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:54.286 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:54.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:54 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:29:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:54 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:29:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1035996401' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:54 smithi162 ceph-mon[20279]: osdmap e821: 12 total, 12 up, 12 in 2024-09-02T10:29:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:54 smithi162 ceph-mon[20279]: 8.d scrub starts 2024-09-02T10:29:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:54 smithi162 ceph-mon[20279]: pgmap v1868: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-02T10:29:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:54 smithi162 ceph-mon[20279]: osdmap e822: 12 total, 12 up, 12 in 2024-09-02T10:29:54.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:54.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:54.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:54.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:54.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:54.539 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:54.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:54.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:54.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:54 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:29:54.658 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:54 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:29:54.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1035996401' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:54.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:54 smithi187 ceph-mon[20072]: osdmap e821: 12 total, 12 up, 12 in 2024-09-02T10:29:54.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:54 smithi187 ceph-mon[20072]: 8.d scrub starts 2024-09-02T10:29:54.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:54 smithi187 ceph-mon[20072]: pgmap v1868: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-02T10:29:54.659 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:54 smithi187 ceph-mon[20072]: osdmap e822: 12 total, 12 up, 12 in 2024-09-02T10:29:54.666 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:54.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:54.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:54 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:29:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:54 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:29:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1035996401' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:29:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:54 smithi151 ceph-mon[16639]: osdmap e821: 12 total, 12 up, 12 in 2024-09-02T10:29:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:54 smithi151 ceph-mon[16639]: 8.d scrub starts 2024-09-02T10:29:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:54 smithi151 ceph-mon[16639]: pgmap v1868: 47 pgs: 47 active+clean; 4.2 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 MiB/s wr, 1.37k op/s 2024-09-02T10:29:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:54 smithi151 ceph-mon[16639]: osdmap e822: 12 total, 12 up, 12 in 2024-09-02T10:29:54.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:54.791 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:54.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:54.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:54.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:54.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:54.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:55.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:55.063 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:55.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:55.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:29:55.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 56 16 72505 72489 80.8962 77.5 0.00333862 0.0123551 2024-09-02T10:29:55.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:55.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:55.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:55.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:55.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:55.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:55.324 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:55.415 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:55.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:55.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:55.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:55.552 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:29:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:55.586 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:55.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:29:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:55 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:29:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:55 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:29:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:55 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:29:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:55 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:29:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:55 smithi162 ceph-mon[20279]: 8.d scrub ok 2024-09-02T10:29:55.687 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:55 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:29:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:55 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:29:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:55 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:29:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:55 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:29:55.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:55 smithi151 ceph-mon[16639]: 8.d scrub ok 2024-09-02T10:29:55.719 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:55.720 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:55 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:29:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:55 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:29:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:55 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:29:55.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:55 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:29:55.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:55 smithi187 ceph-mon[20072]: 8.d scrub ok 2024-09-02T10:29:55.821 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:55.854 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:55.855 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:55.929 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:29:55.929 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:29:55.929 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:29:55.929 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:29:55.929 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:29:55.956 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:55.980 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:55.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:56.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:56.108 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:56.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:56.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:56.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 57 16 73723 73707 80.8124 76.125 0.0101944 0.0123683 2024-09-02T10:29:56.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:56.240 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:56.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:56.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:56.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:29:56.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:29:56.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:56.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:56.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:29:56.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:56 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:29:56.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:56 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:29:56.506 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:56 smithi162 ceph-mon[20279]: pgmap v1870: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 966 op/s 2024-09-02T10:29:56.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:56.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:56.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:56.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:56.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:29:56.640 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:56.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:56.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:56 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:29:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:56 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:29:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:56 smithi151 ceph-mon[16639]: pgmap v1870: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 966 op/s 2024-09-02T10:29:56.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:56.698 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:56.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:56 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:29:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:56 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:29:56.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:56 smithi187 ceph-mon[20072]: pgmap v1870: 47 pgs: 47 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 60 MiB/s wr, 966 op/s 2024-09-02T10:29:56.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:56.829 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:56.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:56.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:56.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:29:56.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:56.941 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:29:56.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:56.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:56.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:57.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:57.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:57.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:57.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 58 16 74779 74763 80.5569 66 0.0230887 0.012409 2024-09-02T10:29:57.198 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:57.224 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:57.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:57.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:57.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:57.358 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:57.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:29:57.490 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:57.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:57.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:29:57.556 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "rep_scrubmap(1.9 e585 from shard 2)", 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:10.193840+0000", 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 527.35950830900003, 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.84180176500000004, 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.557 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:10.193840+0000", 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:10.193840+0000", 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:10.193840+0000", 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4899999999999995e-07 2024-09-02T10:29:57.558 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:10.193844+0000", 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0799999999999999e-06 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:11.035350+0000", 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.84150551299999998 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:11.035359+0000", 2024-09-02T10:29:57.559 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3789999999999995e-06 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:11.035601+0000", 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00024139499999999999 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:11.035613+0000", 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1926000000000001e-05 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.560 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:11.035641+0000", 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8822999999999998e-05 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157286 6.5 6:aeee93d8:::benchmark_data_smithi187_102448_object57014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:27.949571+0000", 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 509.60377738599999, 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30036963500000002, 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.561 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157286 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949571+0000", 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.562 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949571+0000", 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949573+0000", 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0779999999999998e-06 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949581+0000", 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7839999999999992e-06 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.563 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949604+0000", 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2959000000000002e-05 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949613+0000", 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9989999999999997e-06 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949683+0000", 2024-09-02T10:29:57.564 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9142999999999999e-05 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949797+0000", 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114734 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.949886+0000", 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8478000000000002e-05 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.565 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.955070+0000", 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0051841719999999999 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.971047+0000", 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.015976596999999999 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.249846+0000", 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.278799621 2024-09-02T10:29:57.566 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.249889+0000", 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2719000000000002e-05 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.249940+0000", 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1350999999999997e-05 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.567 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157292 6.8 6:12f72cdc:::benchmark_data_smithi187_102448_object57020:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:27.961774+0000", 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 509.59157372300001, 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48684550700000001, 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157292 2024-09-02T10:29:57.568 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961774+0000", 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961774+0000", 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.569 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961776+0000", 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0509999999999999e-06 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961784+0000", 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0870000000000007e-06 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961801+0000", 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6909e-05 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.570 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961809+0000", 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4139999999999997e-06 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961866+0000", 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7178e-05 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.961956+0000", 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9728000000000006e-05 2024-09-02T10:29:57.571 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.962033+0000", 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7099000000000002e-05 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.967020+0000", 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049875379999999997 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.967034+0000", 2024-09-02T10:29:57.572 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4061e-05 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448511+0000", 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48147646599999999 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448573+0000", 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1878000000000006e-05 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.573 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448620+0000", 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7098000000000002e-05 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157297 6.8 6:13945e88:::benchmark_data_smithi187_102448_object57025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:27.976886+0000", 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 509.57646236400001, 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.47188472599999998, 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.574 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157297 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976886+0000", 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.575 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976886+0000", 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976888+0000", 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0080000000000001e-06 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976899+0000", 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0943e-05 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.576 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976916+0000", 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7176999999999998e-05 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.976929+0000", 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2860999999999999e-05 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.977006+0000", 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7248000000000003e-05 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.577 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.977142+0000", 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000136267 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.977282+0000", 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014029099999999999 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.983648+0000", 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0063655580000000003 2024-09-02T10:29:57.578 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:27.984252+0000", 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00060420199999999997 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448667+0000", 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.46441474300000002 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448724+0000", 2024-09-02T10:29:57.579 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7235999999999999e-05 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.448770+0000", 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6192000000000001e-05 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157311 6.8 6:152b8c45:::benchmark_data_smithi187_102448_object57039:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:28.031794+0000", 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 509.521554358, 2024-09-02T10:29:57.580 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42239910000000003, 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157311 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.581 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031794+0000", 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031794+0000", 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031796+0000", 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3539999999999998e-06 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.582 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031805+0000", 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8720000000000008e-06 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031827+0000", 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1678000000000001e-05 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031837+0000", 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0345e-05 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.583 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.031909+0000", 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2026000000000005e-05 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.032041+0000", 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013162100000000001 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.032161+0000", 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012061999999999999 2024-09-02T10:29:57.584 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.585 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.585 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.585 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.036863+0000", 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0047015859999999998 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.036882+0000", 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9409e-05 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454076+0000", 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.417193376 2024-09-02T10:29:57.586 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454142+0000", 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6501000000000006e-05 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454193+0000", 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0711999999999998e-05 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.587 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157316 6.8 6:10c735aa:::benchmark_data_smithi187_102448_object57044:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:28.069591+0000", 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 509.483757293, 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.38474562000000001, 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157316 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.588 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069591+0000", 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069591+0000", 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069592+0000", 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7439999999999999e-06 2024-09-02T10:29:57.589 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069599+0000", 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7390000000000002e-06 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069615+0000", 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5988e-05 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.590 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069624+0000", 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3990000000000004e-06 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069683+0000", 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9341e-05 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069819+0000", 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000135742 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:29:57.591 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.069934+0000", 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115708 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.074383+0000", 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0044484529999999998 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.074508+0000", 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012532299999999999 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.592 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454242+0000", 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37973358800000001 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454295+0000", 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3288000000000002e-05 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454336+0000", 2024-09-02T10:29:57.593 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1307000000000003e-05 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157317 6.8 6:19b233e5:::benchmark_data_smithi187_102448_object57045:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:28.093866+0000", 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 509.45948225199999, 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36060196900000002, 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:57.594 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157317 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093866+0000", 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093866+0000", 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.595 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093868+0000", 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7540000000000001e-06 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093876+0000", 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3680000000000002e-06 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093895+0000", 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9585999999999998e-05 2024-09-02T10:29:57.596 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093904+0000", 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0940000000000008e-06 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.093974+0000", 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0454e-05 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.597 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.094070+0000", 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5610000000000001e-05 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.094211+0000", 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014128400000000001 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.098351+0000", 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041397889999999996 2024-09-02T10:29:57.598 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.098481+0000", 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013057699999999999 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454389+0000", 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35590732400000002 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.599 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454433+0000", 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4869e-05 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454468+0000", 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4260000000000001e-05 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.600 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:157323 6.8 6:14bbaefe:::benchmark_data_smithi187_102448_object57051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:21:28.157047+0000", 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 509.396300521, 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29754121, 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 157323 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.601 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157047+0000", 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157047+0000", 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157050+0000", 2024-09-02T10:29:57.602 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1600000000000001e-06 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157058+0000", 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9449999999999993e-06 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157074+0000", 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6833e-05 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.603 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157081+0000", 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.657e-06 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157128+0000", 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6421000000000002e-05 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157218+0000", 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0778000000000004e-05 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.604 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.157299+0000", 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0711999999999996e-05 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.161246+0000", 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039468059999999998 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.161419+0000", 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017325399999999999 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.605 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454510+0000", 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29309048900000001 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454551+0000", 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1878000000000001e-05 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:21:28.454589+0000", 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7277000000000002e-05 2024-09-02T10:29:57.606 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:191032 6.8 6:141909d5:::benchmark_data_smithi187_102448_object90760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:24.090741+0000", 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 453.462606768, 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.261754549, 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.607 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 191032 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090741+0000", 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.608 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090741+0000", 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090743+0000", 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.333e-06 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090748+0000", 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.857e-06 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.609 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090769+0000", 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0494000000000001e-05 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090776+0000", 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6329999999999999e-06 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090855+0000", 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9370999999999996e-05 2024-09-02T10:29:57.610 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.090971+0000", 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115963 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.091080+0000", 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108837 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.096121+0000", 2024-09-02T10:29:57.611 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0050410940000000003 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.099730+0000", 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036096359999999998 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.352348+0000", 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25261799699999998 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.612 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.352412+0000", 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3746999999999998e-05 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:24.352496+0000", 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3586999999999998e-05 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:193656 6.5 6:a5525aaf:::benchmark_data_smithi187_102448_object93384:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e622)", 2024-09-02T10:29:57.613 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:28.345700+0000", 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 449.20764845600002, 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26458540800000002, 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 193656 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.614 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345700+0000", 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345700+0000", 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345701+0000", 2024-09-02T10:29:57.615 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2529999999999999e-06 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345706+0000", 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.057e-06 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345723+0000", 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7232e-05 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.616 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345729+0000", 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0059999999999998e-06 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345792+0000", 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2606000000000003e-05 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345882+0000", 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0539999999999997e-05 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.617 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.345992+0000", 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010925300000000001 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.350336+0000", 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043448200000000001 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.350360+0000", 2024-09-02T10:29:57.618 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3213e-05 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.610142+0000", 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25978261800000002 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.610237+0000", 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5206999999999998e-05 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.619 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:28.610285+0000", 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7602999999999998e-05 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.26524.0:200540 6.5 6:ac93dc06:::benchmark_data_smithi187_102448_object100268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:40.759488+0000", 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 436.79386035599998, 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.261497865, 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.620 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.26524", 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 200540 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759488+0000", 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.621 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759488+0000", 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759489+0000", 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7570000000000001e-06 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759495+0000", 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1120000000000004e-06 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759509+0000", 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4606999999999999e-05 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759515+0000", 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3199999999999996e-06 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759558+0000", 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2401999999999998e-05 2024-09-02T10:29:57.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759642+0000", 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4380000000000002e-05 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.759716+0000", 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3795999999999996e-05 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764547+0000", 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0048309219999999996 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764639+0000", 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1902000000000002e-05 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020875+0000", 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25623643800000001 2024-09-02T10:29:57.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020938+0000", 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2754000000000003e-05 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020986+0000", 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7475000000000002e-05 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.26524.0:200540 6.5 e626/533 ondisk, result = 0)", 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:40.764375+0000", 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 436.78897345600001, 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25661853200000001, 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764375+0000", 2024-09-02T10:29:57.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764375+0000", 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764375+0000", 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7999999999999995e-07 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.628 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:40.764380+0000", 2024-09-02T10:29:57.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:57.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3089999999999997e-06 2024-09-02T10:29:57.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020777+0000", 2024-09-02T10:29:57.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25639733999999997 2024-09-02T10:29:57.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020786+0000", 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1530000000000003e-06 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020857+0000", 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1361999999999999e-05 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020868+0000", 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0528e-05 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:41.020993+0000", 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012516 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27220.0:55500 7.8 e696/661 7:1ef2c8c5:::benchmark_data_smithi187_120712_object55499:head v 696'3479, mlcod=696'3476)", 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:07.741713+0000", 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 289.81163503400001, 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24015054999999999, 2024-09-02T10:29:57.632 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.741713+0000", 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.741713+0000", 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.741714+0000", 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7400000000000002e-07 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.741789+0000", 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5412000000000002e-05 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.980006+0000", 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.238217016 2024-09-02T10:29:57.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.980014+0000", 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1289999999999996e-06 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.980074+0000", 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9769999999999999e-05 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.980095+0000", 2024-09-02T10:29:57.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0330999999999999e-05 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.981817+0000", 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001722339 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.981864+0000", 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6678999999999998e-05 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.636 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27220.0:99739 7.8 e722/661 7:121bd1d6:::benchmark_data_smithi187_120712_object15797:head v 722'6207, mlcod=722'6204)", 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:00.273723+0000", 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 237.279624738, 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22788001299999999, 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.273723+0000", 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.273723+0000", 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.273724+0000", 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5899999999999995e-07 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.273728+0000", 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3329999999999998e-06 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.498479+0000", 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.224750224 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.498486+0000", 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4549999999999998e-06 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.498551+0000", 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5413000000000005e-05 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.498573+0000", 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1880000000000001e-05 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.501563+0000", 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00298986 2024-09-02T10:29:57.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:00.501603+0000", 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0089000000000002e-05 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27220.0:151392 7.15 e749/744 7:abb8bf64:::benchmark_data_smithi187_120712_object67450:head v 749'8947, mlcod=707'4218)", 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:27:12.330043+0000", 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 165.223305031, 2024-09-02T10:29:57.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.243870749, 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.330043+0000", 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.330043+0000", 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.330044+0000", 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.64e-07 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.330051+0000", 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.114e-06 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.569889+0000", 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23983832399999999 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.569900+0000", 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.041e-05 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.569971+0000", 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1188999999999999e-05 2024-09-02T10:29:57.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.570005+0000", 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4448e-05 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.573872+0000", 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003866759 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.573914+0000", 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1541000000000001e-05 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27220.0:151431 7.13 e749/744 7:cd9d2b1f:::benchmark_data_smithi187_120712_object67489:head v 749'8832, mlcod=707'4144)", 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:27:12.414200+0000", 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 165.13914761000001, 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18708517199999999, 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.646 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.414200+0000", 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.414200+0000", 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.414201+0000", 2024-09-02T10:29:57.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2399999999999998e-07 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.414205+0000", 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.754e-06 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.598196+0000", 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.183991404 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.598206+0000", 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8910000000000007e-06 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.598263+0000", 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.698e-05 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.598283+0000", 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0264000000000001e-05 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.601235+0000", 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002951731 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.601286+0000", 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0624000000000002e-05 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.650 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:49403 8.6 e813/792 8:6b4d2932:::benchmark_data_smithi187_136059_object49402:head v 813'3069, mlcod=813'3065)", 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:29:34.205813+0000", 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 23.347535447999999, 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.11281205499999999, 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.205813+0000", 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.205813+0000", 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.205814+0000", 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0410000000000001e-06 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.205869+0000", 2024-09-02T10:29:57.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5279999999999999e-05 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.316736+0000", 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.110866824 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.316742+0000", 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4370000000000004e-06 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.316826+0000", 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4207000000000002e-05 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.316852+0000", 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5874e-05 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.318579+0000", 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001726289 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.318625+0000", 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6103000000000002e-05 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:49416 8.d e813/792 8:bac7226e:::benchmark_data_smithi187_136059_object49415:head v 813'3086, mlcod=813'3083)", 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:29:34.217716+0000", 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 23.335631828, 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.101006265, 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.655 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.217716+0000", 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.217716+0000", 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.217717+0000", 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2720000000000001e-06 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.217749+0000", 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1656000000000003e-05 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.316786+0000", 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.099036434000000007 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.316789+0000", 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9799999999999998e-06 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.316829+0000", 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0179999999999998e-05 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.316855+0000", 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5826e-05 2024-09-02T10:29:57.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.318696+0000", 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018418169999999999 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:34.318722+0000", 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6100000000000001e-05 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:64017 8.6 e819/792 8:6d9d70af:::benchmark_data_smithi187_136059_object64016:head v 819'3968, mlcod=819'3965)", 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:29:47.173344+0000", 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.380003772, 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.129595498, 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.173344+0000", 2024-09-02T10:29:57.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.173344+0000", 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.173348+0000", 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6519999999999999e-06 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.173392+0000", 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.409e-05 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.300945+0000", 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12755324400000001 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.300952+0000", 2024-09-02T10:29:57.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6699999999999997e-06 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.301010+0000", 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8368000000000001e-05 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.301033+0000", 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2282000000000001e-05 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.302905+0000", 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018727749999999999 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.302940+0000", 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4416999999999999e-05 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.664 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:64021 8.d e819/792 8:b5025052:::benchmark_data_smithi187_136059_object64020:head v 819'4017, mlcod=819'4014)", 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:29:47.181980+0000", 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.371367728999999, 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12872088500000001, 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.181980+0000", 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.181980+0000", 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.181981+0000", 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7899999999999998e-07 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.182048+0000", 2024-09-02T10:29:57.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6598000000000002e-05 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.308913+0000", 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.12686567000000001 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.308918+0000", 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8500000000000002e-06 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:29:57.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.308972+0000", 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4233000000000001e-05 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.308995+0000", 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2874000000000002e-05 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.310665+0000", 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001670244 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:29:57.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:29:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:47.310701+0000", 2024-09-02T10:29:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5737e-05 2024-09-02T10:29:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:29:57.669 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:29:57.669 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:29:57.701 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:57.702 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:57.803 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:57.829 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:57.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:57.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:57.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:57.961 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:57.962 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:58.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:58.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:58.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:58.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:58.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 59 16 76118 76102 80.6098 83.6875 0.0105417 0.0124003 2024-09-02T10:29:58.206 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:29:58.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:58.241 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:29:58.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:58.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:58.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:58.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:58.503 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:58.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:58.521 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:29:58.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:29:58.641 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:58.642 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:29:58.668 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:29:58.743 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:58.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:58.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:58.796 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:58 smithi151 ceph-mon[16639]: pgmap v1871: 47 pgs: 47 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-02T10:29:58.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:58 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:29:58.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:58 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:29:58.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:58 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:29:58.797 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:58 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:29:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:58 smithi162 ceph-mon[20279]: pgmap v1871: 47 pgs: 47 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-02T10:29:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:58 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:29:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:58 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:29:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:58 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:29:58.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:58 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:29:58.872 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:58.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:58.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:59.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:29:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:58 smithi187 ceph-mon[20072]: pgmap v1871: 47 pgs: 47 active+clean; 4.4 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 66 MiB/s wr, 1.06k op/s 2024-09-02T10:29:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:58 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:29:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:58 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:29:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:58 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:29:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:58 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:29:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:59.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:59.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:29:59.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:29:59.118252+0000 min lat: 0.00107552 max lat: 0.623325 avg lat: 0.0124285 2024-09-02T10:29:59.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:29:59.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 60 16 77230 77214 80.4246 69.5 0.00558199 0.0124285 2024-09-02T10:29:59.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:59.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:59.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:59.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:59.220 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:29:59.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:29:59.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:59.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:29:59.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:59.429 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:59.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:29:59.443 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:29:59.443 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:29:59.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:29:59.474 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:29:59.502 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:29:59.502 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:29:59.530 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:29:59.531 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:29:59.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:29:59.554 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:59.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:29:59.555 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:29:59.555 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:29:59.579 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:29:59.579 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:29:59.601 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:29:59.602 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:29:59.627 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:29:59.627 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:29:59.654 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:29:59.654 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:29:59.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:29:59.673 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:59 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:29:59.673 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:59 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:29:59.673 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2635673007' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:59.673 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:59 smithi187 ceph-mon[20072]: 8.a scrub starts 2024-09-02T10:29:59.673 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:29:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3905815615' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:59.684 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:29:59.684 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:29:59.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:29:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:59.693 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:29:59.709 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:29:59.709 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:29:59.709 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-02T10:29:59.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-02T10:29:59.712 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:29:59.713 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:29:59.713 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:59 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:29:59.713 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:59 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:29:59.713 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2635673007' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:59.713 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:59 smithi151 ceph-mon[16639]: 8.a scrub starts 2024-09-02T10:29:59.713 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:29:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3905815615' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:59.743 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:29:59.744 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:29:59.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:29:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:59.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:29:59.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:29:59.825 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:29:59.825 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:29:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:59 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:29:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:59 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:29:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2635673007' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:29:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:59 smithi162 ceph-mon[20279]: 8.a scrub starts 2024-09-02T10:29:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:29:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3905815615' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:29:59.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:29:59.945 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:29:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:29:59.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:00.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:00.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:00.086 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:00.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 61 16 78155 78139 80.0538 57.8125 0.00986842 0.0124861 2024-09-02T10:30:00.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:00.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:00.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:00.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:00.294 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:30:00.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:00.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:00.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:00.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:00.444 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:00.466 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:00.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:00.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:00.607 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:00.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:00.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:00.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:00 smithi187 ceph-mon[20072]: 8.9 scrub starts 2024-09-02T10:30:00.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:00 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:30:00.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:00 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:30:00.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:00 smithi187 ceph-mon[20072]: pgmap v1872: 47 pgs: 47 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-02T10:30:00.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:00 smithi187 ceph-mon[20072]: 8.1 scrub starts 2024-09-02T10:30:00.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:00 smithi187 ceph-mon[20072]: overall HEALTH_OK 2024-09-02T10:30:00.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:00 smithi187 ceph-mon[20072]: 8.a scrub ok 2024-09-02T10:30:00.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:00 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:00.660 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3905815615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:30:00.661 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:00 smithi187 ceph-mon[20072]: osdmap e823: 12 total, 12 up, 12 in 2024-09-02T10:30:00.662 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:30:00.708 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:00.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:00.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:00.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:00.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:00 smithi162 ceph-mon[20279]: 8.9 scrub starts 2024-09-02T10:30:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:00 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:30:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:00 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:30:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:00 smithi162 ceph-mon[20279]: pgmap v1872: 47 pgs: 47 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-02T10:30:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:00 smithi162 ceph-mon[20279]: 8.1 scrub starts 2024-09-02T10:30:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:00 smithi162 ceph-mon[20279]: overall HEALTH_OK 2024-09-02T10:30:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:00 smithi162 ceph-mon[20279]: 8.a scrub ok 2024-09-02T10:30:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:00 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3905815615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:30:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:00 smithi162 ceph-mon[20279]: osdmap e823: 12 total, 12 up, 12 in 2024-09-02T10:30:00.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:00.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:00.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:00 smithi151 ceph-mon[16639]: 8.9 scrub starts 2024-09-02T10:30:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:00 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:30:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:00 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:30:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:00 smithi151 ceph-mon[16639]: pgmap v1872: 47 pgs: 47 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 81 MiB/s wr, 1.29k op/s 2024-09-02T10:30:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:00 smithi151 ceph-mon[16639]: 8.1 scrub starts 2024-09-02T10:30:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:00 smithi151 ceph-mon[16639]: overall HEALTH_OK 2024-09-02T10:30:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:00 smithi151 ceph-mon[16639]: 8.a scrub ok 2024-09-02T10:30:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:00 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3905815615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:30:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:00 smithi151 ceph-mon[16639]: osdmap e823: 12 total, 12 up, 12 in 2024-09-02T10:30:00.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:00.981 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:00.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:00.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:01.003 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:01.083 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:01.105 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:01.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:01.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 62 16 79075 79059 79.6899 57.5 0.0407818 0.0125419 2024-09-02T10:30:01.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:01.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:01.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:01.234 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:01.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:01.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:01.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:01.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:01.425 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:01.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:01.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:01.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:01.513 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:01.513 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:01.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:01.611 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:01.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:01.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:01.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:01.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:01.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:01.869 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:01.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:01.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:01.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:02.003 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:02.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:02.004 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:02.068 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:02.069 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:02.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:02.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 63 16 80132 80116 79.4735 66.0625 0.00556815 0.0125773 2024-09-02T10:30:02.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:02.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:02.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:30:02.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:02.283 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:02.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:02.284 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:02.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:02.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:02.493 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:02.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:02.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:02.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:02.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:30:02.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:02.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:02.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:02.771 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:02.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:02.793 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:02.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:02.811 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:02.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-02T10:30:02.833 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:02 smithi151 ceph-mon[16639]: pgmap v1874: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-02T10:30:02.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:02 smithi162 ceph-mon[20279]: pgmap v1874: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-02T10:30:02.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:02.902 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:02.916 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:02.917 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:02.918 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:02 smithi187 ceph-mon[20072]: pgmap v1874: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.03k op/s 2024-09-02T10:30:03.018 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:03.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:03.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:03.069 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:03.069 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:03.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 64 16 80999 80983 79.0784 54.1875 0.0186902 0.0126398 2024-09-02T10:30:03.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:03.183 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:03.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:03.184 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:03.285 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:03.314 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:03.315 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:03.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:30:03.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:03.441 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:03.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:03.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:03.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:03.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:03.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:03.591 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:03.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:03.715 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:03.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:03.817 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:03.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3241222904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:03.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:03.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:03.853 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3241222904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:03.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:30:03.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3241222904' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:03.949 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:03.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:03.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:04.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:04.111 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:04.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:04.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:04.118 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 65 16 82214 82198 79.0299 75.9375 0.00456217 0.0126472 2024-09-02T10:30:04.213 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:04.250 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:04.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:04.251 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:04.334 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:04.335 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:30:04.335 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:30:04.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:04.369 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:30:04.369 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:30:04.375 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:04.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:04.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:04.400 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:30:04.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:30:04.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:30:04.434 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:30:04.472 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:30:04.472 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:30:04.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:04.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:04.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:04.508 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:04.511 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:30:04.511 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:30:04.546 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:30:04.546 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:30:04.581 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:30:04.581 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:30:04.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:04.621 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:30:04.621 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:30:04.621 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:04 smithi151 ceph-mon[16639]: pgmap v1875: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:30:04.621 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:04 smithi151 ceph-mon[16639]: osdmap e824: 12 total, 12 up, 12 in 2024-09-02T10:30:04.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:04.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:04.656 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:30:04.657 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:30:04.687 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:30:04.687 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:30:04.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:04.748 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:30:04.748 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:30:04.756 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:04 smithi187 ceph-mon[20072]: pgmap v1875: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:30:04.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:04 smithi187 ceph-mon[20072]: osdmap e824: 12 total, 12 up, 12 in 2024-09-02T10:30:04.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:04.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:04 smithi162 ceph-mon[20279]: pgmap v1875: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 4.8 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 68 MiB/s wr, 1.08k op/s 2024-09-02T10:30:04.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:04 smithi162 ceph-mon[20279]: osdmap e824: 12 total, 12 up, 12 in 2024-09-02T10:30:04.859 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:04.882 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:04.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:04.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:04.985 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:05.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:05.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:05.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:05.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 66 16 83319 83303 78.8788 69.0625 0.0166384 0.0126724 2024-09-02T10:30:05.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:05.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:05.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:05.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:05.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:05.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:05.372 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:05.394 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:05.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:05.495 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:05.517 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:05.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:05.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:05.639 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:05.640 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:05.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:05.772 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:05.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:05.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:05.874 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:05.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:05.908 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:06.009 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:06.030 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:06.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:06.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:06.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 67 16 84084 84068 78.4151 47.8125 0.00579063 0.0127462 2024-09-02T10:30:06.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:06.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:06.162 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:06.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:06.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:06.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:06.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:06.360 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:06.360 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:06.360 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:06.360 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:06.360 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:06.360 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:06.361 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:06.361 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:06.361 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:06.361 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:06.361 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:06.361 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:06.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:06.422 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:06.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:06.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:06.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:06.541 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:30:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:30:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.541 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:84026 8.c 8:3b563e53:::benchmark_data_smithi187_136059_object84025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:30:05.972110+0000", 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.106856515, 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.106887549, 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 84026 2024-09-02T10:30:06.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.972110+0000", 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.972110+0000", 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:06.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.972113+0000", 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8590000000000001e-06 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.972140+0000", 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7342e-05 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.972163+0000", 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.285e-05 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.972173+0000", 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0175e-05 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.972272+0000", 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8187e-05 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:06.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.972434+0000", 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016210899999999999 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.972578+0000", 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000144103 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.975369+0000", 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027916460000000001 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:06.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.991758+0000", 2024-09-02T10:30:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.016389048 2024-09-02T10:30:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:30:06.547 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:30:06.547 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:30:06.547 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:06.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:06.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:06.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:06.677 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:06.678 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:06.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:06 smithi151 ceph-mon[16639]: pgmap v1877: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-02T10:30:06.778 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:06.801 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:06.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:06 smithi162 ceph-mon[20279]: pgmap v1877: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-02T10:30:06.860 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:06 smithi187 ceph-mon[20072]: pgmap v1877: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.05k op/s 2024-09-02T10:30:06.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:06.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:06.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:07.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:07.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:07.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:07.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 68 16 85182 85166 78.271 68.625 0.00802294 0.0127703 2024-09-02T10:30:07.154 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:07.161 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:07.200 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:07.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:07.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:07.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:30:07.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:07.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:07.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:07.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:07.466 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:07.467 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:07.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:07.592 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:07.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:07.644 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:07.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-02T10:30:07.650 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:07.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:07.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:07.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:07.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:07.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:07.848 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:07.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:07.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:07.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:07.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:08.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:08.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:08.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:08.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 69 16 86193 86177 78.0524 63.1875 0.00798889 0.0128064 2024-09-02T10:30:08.209 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:30:08.210 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:30:08.210 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:30:08.210 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:30:08.210 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:30:08.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:08.219 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:08.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:08.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:08.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:08.383 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:08.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:08.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:08.506 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:08.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:08.604 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:30:08.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:08.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:08.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:08.733 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:08.763 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:08.764 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:08.865 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:08.872 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:08 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:08.872 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:08 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:08.872 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:08 smithi187 ceph-mon[20072]: pgmap v1878: 47 pgs: 47 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s 2024-09-02T10:30:08.872 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:08 smithi187 ceph-mon[20072]: osdmap e825: 12 total, 12 up, 12 in 2024-09-02T10:30:08.872 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2286359046' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:08.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:08.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:08.898 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:08.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:30:08.921 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:30:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:08 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:08 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:08 smithi151 ceph-mon[16639]: pgmap v1878: 47 pgs: 47 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s 2024-09-02T10:30:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:08 smithi151 ceph-mon[16639]: osdmap e825: 12 total, 12 up, 12 in 2024-09-02T10:30:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2286359046' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:08.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:09.033 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:09.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:08 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:08 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:08 smithi162 ceph-mon[20279]: pgmap v1878: 47 pgs: 47 active+clean; 5.1 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 62 MiB/s wr, 988 op/s 2024-09-02T10:30:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:08 smithi162 ceph-mon[20279]: osdmap e825: 12 total, 12 up, 12 in 2024-09-02T10:30:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2286359046' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:09.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 70 16 87251 87235 77.8819 66.125 0.0110792 0.0128338 2024-09-02T10:30:09.134 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:09.157 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:09.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:09.168 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:09.169 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:09.259 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:09.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:09.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:09.324 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:09.386 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:09.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:09.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:09.515 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:09.545 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:09.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:09.546 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:09.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:09.674 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:09.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:09.718 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:09 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:09.718 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2286359046' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:09.718 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:09 smithi187 ceph-mon[20072]: osdmap e826: 12 total, 12 up, 12 in 2024-09-02T10:30:09.776 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:09.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:09.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:09.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:09.926 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:09.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:09.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:09 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2286359046' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:09 smithi151 ceph-mon[16639]: osdmap e826: 12 total, 12 up, 12 in 2024-09-02T10:30:10.019 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:30:10.019 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:30:10.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:30:10.019 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:30:10.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.019 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:100226 6.1 e572/533 6:80d0f12d:::benchmark_data_smithi187_102448_object100225:head v 572'6190, mlcod=572'6187)", 2024-09-02T10:30:10.019 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 599.11058290999995, 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.63855390000000001, 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906122+0000", 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906123+0000", 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.269e-06 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:10.906147+0000", 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3620000000000001e-05 2024-09-02T10:30:10.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.541864+0000", 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.635717054 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.541879+0000", 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5546e-05 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.542040+0000", 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000161022 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.542059+0000", 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8697000000000001e-05 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544638+0000", 2024-09-02T10:30:10.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025792850000000002 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:20:11.544676+0000", 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7407000000000003e-05 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.26524.0:157286 6.5 e598/533 6:aeee93d8:::benchmark_data_smithi187_102448_object57014:head v 598'9853, mlcod=598'9850)", 2024-09-02T10:30:10.024 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 522.06669000399995, 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29959694599999998, 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950015+0000", 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2900000000000002e-07 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:27.950022+0000", 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8739999999999997e-06 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245512+0000", 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29548945300000001 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245524+0000", 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2347000000000001e-05 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245666+0000", 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014145800000000001 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.245702+0000", 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6853000000000001e-05 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.249576+0000", 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003873385 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:21:28.249611+0000", 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5646999999999999e-05 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:43285 7.a 7:505fa4fc:::benchmark_data_smithi187_120712_object43284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:51.571815+0000", 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 318.44488987599999, 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21327285000000001, 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.029 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 43285 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571815+0000", 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571815+0000", 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571816+0000", 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8109999999999999e-06 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571848+0000", 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1999999999999999e-05 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571863+0000", 2024-09-02T10:30:10.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4976e-05 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571871+0000", 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6930000000000007e-06 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.571922+0000", 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0884999999999998e-05 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.572019+0000", 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6482000000000002e-05 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.572104+0000", 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5686000000000006e-05 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573298+0000", 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001193646 2024-09-02T10:30:10.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:10.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.574986+0000", 2024-09-02T10:30:10.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001687893 2024-09-02T10:30:10.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.784959+0000", 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20997289899999999 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.785035+0000", 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6780999999999996e-05 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.785088+0000", 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2098000000000002e-05 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:43288 7.a 7:52f650c0:::benchmark_data_smithi187_120712_object43287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:51.573470+0000", 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 318.44323503099997, 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.215809268, 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:10.037 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 43288 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573470+0000", 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573470+0000", 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573470+0000", 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6000000000000003e-07 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573490+0000", 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9558000000000001e-05 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573500+0000", 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.696e-06 2024-09-02T10:30:10.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573504+0000", 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.87e-06 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573544+0000", 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9749999999999997e-05 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573653+0000", 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108455 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.573733+0000", 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0520999999999997e-05 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.575073+0000", 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013397000000000001 2024-09-02T10:30:10.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.575346+0000", 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000272716 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789047+0000", 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21370119100000001 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789231+0000", 2024-09-02T10:30:10.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018440700000000001 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789279+0000", 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7644e-05 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:43296 7.a 7:53509a3f:::benchmark_data_smithi187_120712_object43295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:51.582447+0000", 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 318.43425719300001, 2024-09-02T10:30:10.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20711470400000001, 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 43296 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582447+0000", 2024-09-02T10:30:10.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582447+0000", 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582449+0000", 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2309999999999999e-06 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582470+0000", 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1141e-05 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582481+0000", 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1581e-05 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582487+0000", 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4790000000000004e-06 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582531+0000", 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4206999999999999e-05 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582632+0000", 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000100882 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.582720+0000", 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7806999999999995e-05 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.584239+0000", 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001519734 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.584541+0000", 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00030194999999999999 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789368+0000", 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.204827017 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789525+0000", 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015684 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.789562+0000", 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6835000000000002e-05 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.049 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:45631 7.a 7:579b01e3:::benchmark_data_smithi187_120712_object45630:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:53.384584+0000", 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 316.632120875, 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25830757599999998, 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 45631 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384584+0000", 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384584+0000", 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384586+0000", 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4090000000000001e-06 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384615+0000", 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9200000000000002e-05 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384639+0000", 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3901e-05 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384651+0000", 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1793999999999999e-05 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384726+0000", 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4813000000000004e-05 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384857+0000", 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013135800000000001 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:30:10.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.384960+0000", 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000102682 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.386554+0000", 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001593742 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.386810+0000", 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025653200000000002 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.642810+0000", 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25600023 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.642853+0000", 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2657999999999997e-05 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.642891+0000", 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8257000000000003e-05 2024-09-02T10:30:10.055 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55524 7.a 7:5888f7f6:::benchmark_data_smithi187_120712_object55523:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.784823+0000", 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.23188134200001, 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25595624, 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:30:10.056 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55524 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784823+0000", 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784823+0000", 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784825+0000", 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.289e-06 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784866+0000", 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0327000000000002e-05 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784884+0000", 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7697999999999999e-05 2024-09-02T10:30:10.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784893+0000", 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5510000000000005e-06 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784968+0000", 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4975999999999995e-05 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.785119+0000", 2024-09-02T10:30:10.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000151404 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.785270+0000", 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015045799999999999 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.787505+0000", 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022353439999999998 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.787795+0000", 2024-09-02T10:30:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028950500000000001 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040702+0000", 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25290737699999999 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040745+0000", 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.337e-05 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040779+0000", 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3940999999999999e-05 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:55525 7.5 e696/690 7:a5f161ff:::benchmark_data_smithi187_120712_object55524:head v 696'3504, mlcod=696'3501)", 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.788514+0000", 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.22819081599999, 2024-09-02T10:30:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.2522201, 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788514+0000", 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788514+0000", 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788515+0000", 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9299999999999996e-07 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788550+0000", 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5129000000000002e-05 2024-09-02T10:30:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788586+0000", 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6294000000000002e-05 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788593+0000", 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9299999999999997e-06 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788652+0000", 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9002999999999997e-05 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788676+0000", 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4290999999999998e-05 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040675+0000", 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25199830000000001 2024-09-02T10:30:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040734+0000", 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9259999999999998e-05 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:55524 7.a e696/661 ondisk, result = 0)", 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.788585+0000", 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.22811982899998, 2024-09-02T10:30:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25220463100000001, 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788585+0000", 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788585+0000", 2024-09-02T10:30:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788586+0000", 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1399999999999996e-07 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788593+0000", 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5510000000000002e-06 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040578+0000", 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25198481900000003 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040594+0000", 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6563e-05 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040692+0000", 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7800999999999994e-05 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040698+0000", 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8390000000000003e-06 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040789+0000", 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1243999999999996e-05 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55529 7.a 7:5c0a98d2:::benchmark_data_smithi187_120712_object55528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.789084+0000", 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.22762065299997, 2024-09-02T10:30:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25619412899999999, 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55529 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789084+0000", 2024-09-02T10:30:10.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789084+0000", 2024-09-02T10:30:10.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789086+0000", 2024-09-02T10:30:10.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3910000000000001e-06 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789109+0000", 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2734e-05 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789121+0000", 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2435e-05 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.074 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789129+0000", 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.52e-06 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789235+0000", 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010628000000000001 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789361+0000", 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000125351 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789484+0000", 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012378000000000001 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040657+0000", 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25117236700000001 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040831+0000", 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017463 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045180+0000", 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043483080000000004 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045232+0000", 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2074000000000003e-05 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:10.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045278+0000", 2024-09-02T10:30:10.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6258999999999999e-05 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:55530 7.4 e696/690 7:2b921397:::benchmark_data_smithi187_120712_object55529:head v 696'3599, mlcod=696'3597)", 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.789951+0000", 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.226754008, 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.2507779, 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.080 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789951+0000", 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789951+0000", 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789954+0000", 2024-09-02T10:30:10.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1530000000000001e-06 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790032+0000", 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8168000000000001e-05 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790048+0000", 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5862000000000002e-05 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790054+0000", 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.584e-06 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790120+0000", 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5668999999999998e-05 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790141+0000", 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0818999999999998e-05 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040685+0000", 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25054374800000001 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040728+0000", 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3896999999999997e-05 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:55529 7.a e696/661 ondisk, result = 0)", 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.791795+0000", 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.22490931499999, 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25349172599999997, 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791795+0000", 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791795+0000", 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791796+0000", 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9599999999999999e-07 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791803+0000", 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9299999999999997e-06 2024-09-02T10:30:10.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040780+0000", 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.248977435 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040785+0000", 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9910000000000002e-06 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045163+0000", 2024-09-02T10:30:10.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043775949999999998 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045171+0000", 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2109999999999998e-06 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045287+0000", 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115868 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:55529 7.a e696/661 ondisk, result = 0)", 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.793833+0000", 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.22287166299998, 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24701554000000001, 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793833+0000", 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793833+0000", 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793834+0000", 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2900000000000003e-07 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793838+0000", 2024-09-02T10:30:10.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8720000000000001e-06 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040586+0000", 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.246747148 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040598+0000", 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1943999999999999e-05 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040817+0000", 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00021921700000000001 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040825+0000", 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5939999999999994e-06 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040848+0000", 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3036000000000001e-05 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.092 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55539 7.a 7:5096ce06:::benchmark_data_smithi187_120712_object55538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.802709+0000", 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 302.21399580600001, 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24498555799999999, 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:10.093 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55539 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802709+0000", 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802709+0000", 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802711+0000", 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5840000000000002e-06 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802737+0000", 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5948e-05 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040797+0000", 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23805955500000001 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040805+0000", 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9629999999999998e-06 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040878+0000", 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.305e-05 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.041029+0000", 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000151337 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.041152+0000", 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012326200000000001 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045348+0000", 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041960260000000003 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045394+0000", 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.587e-05 2024-09-02T10:30:10.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.047524+0000", 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021297120000000002 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.047622+0000", 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8387999999999994e-05 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.047694+0000", 2024-09-02T10:30:10.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1863e-05 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:71694 7.a 7:589b7e94:::benchmark_data_smithi187_120712_object71693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e711)", 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:28.310151+0000", 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 281.70655320600002, 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31770000700000001, 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:10.099 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 71694 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310151+0000", 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310151+0000", 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310153+0000", 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5910000000000001e-06 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310173+0000", 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.052e-05 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310187+0000", 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.397e-05 2024-09-02T10:30:10.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310194+0000", 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1580000000000002e-06 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310240+0000", 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6564999999999999e-05 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310346+0000", 2024-09-02T10:30:10.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010611999999999999 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310438+0000", 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2083999999999995e-05 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312967+0000", 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025282730000000002 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.313222+0000", 2024-09-02T10:30:10.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025570999999999998 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627778+0000", 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31455519399999998 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627814+0000", 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6374000000000001e-05 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627851+0000", 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7447999999999998e-05 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:71694 7.a e711/661 ondisk, result = 0)", 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:28.312737+0000", 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 281.70396718500001, 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.315119227, 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.105 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312737+0000", 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312737+0000", 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312738+0000", 2024-09-02T10:30:10.106 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1500000000000003e-07 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312742+0000", 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0389999999999998e-06 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627726+0000", 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31498364600000001 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.107 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627732+0000", 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0789999999999997e-06 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627771+0000", 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8642e-05 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627774+0000", 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9060000000000004e-06 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.108 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627857+0000", 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2100000000000003e-05 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:99751 7.e e722/661 7:70408914:::benchmark_data_smithi187_120712_object15809:head v 722'6054, mlcod=722'6050)", 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:00.293617+0000", 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 249.72308769, 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26051671900000001, 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.109 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293617+0000", 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293617+0000", 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293618+0000", 2024-09-02T10:30:10.110 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5099999999999999e-07 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293623+0000", 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8810000000000001e-06 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551233+0000", 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.257609745 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.111 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551238+0000", 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9609999999999996e-06 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551289+0000", 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0492999999999999e-05 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551309+0000", 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0426999999999999e-05 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.112 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.554093+0000", 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027834660000000001 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.554134+0000", 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0995000000000003e-05 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:151430 7.a 7:557e0f20:::benchmark_data_smithi187_120712_object67488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:30:10.113 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:12.399799+0000", 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 177.61690520400001, 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22159694199999999, 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 151430 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.114 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399799+0000", 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399799+0000", 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399801+0000", 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6750000000000001e-06 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.115 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399807+0000", 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3840000000000002e-06 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399824+0000", 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6257999999999999e-05 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399831+0000", 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2459999999999997e-06 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.116 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399892+0000", 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1234000000000004e-05 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.399987+0000", 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4999000000000003e-05 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.400061+0000", 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3603000000000007e-05 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.117 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.403732+0000", 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036712530000000002 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.403903+0000", 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000170645 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621303+0000", 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21740003899999999 2024-09-02T10:30:10.118 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621359+0000", 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5983000000000003e-05 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621396+0000", 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7623000000000002e-05 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.119 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:151430 7.a e749/661 ondisk, result = 0)", 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:27:12.404953+0000", 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 177.61175196299999, 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.216450315, 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.404953+0000", 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.120 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.404953+0000", 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.404953+0000", 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6700000000000003e-07 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.404958+0000", 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6249999999999998e-06 2024-09-02T10:30:10.121 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621131+0000", 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.216173486 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621144+0000", 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.254e-05 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621259+0000", 2024-09-02T10:30:10.122 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011462499999999999 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621296+0000", 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7696e-05 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:27:12.621403+0000", 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010667600000000001 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.123 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "replica_scrub(pg: 1.6,from:0'0,to:0'0,epoch:821/301,start:1:60000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:53.986902+0000", 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 16.029802298, 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35521325199999998, 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:53.986902+0000", 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.124 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:10.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:53.986902+0000", 2024-09-02T10:30:10.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:10.125 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.125 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.125 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:10.125 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:53.986903+0000", 2024-09-02T10:30:10.125 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5499999999999997e-07 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:53.986909+0000", 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0399999999999998e-06 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:54.341139+0000", 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35423013199999998 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.126 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:54.341149+0000", 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6150000000000003e-06 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:54.341954+0000", 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00080544199999999996 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:54.342075+0000", 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012106 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:10.127 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:10.128 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:30:10.128 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:54.342115+0000", 2024-09-02T10:30:10.128 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0108000000000003e-05 2024-09-02T10:30:10.128 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.128 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.128 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.128 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:10.128 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:10.128 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:30:10.129 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:10.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:10.129 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:09 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:10.129 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2286359046' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:10.129 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:09 smithi162 ceph-mon[20279]: osdmap e826: 12 total, 12 up, 12 in 2024-09-02T10:30:10.130 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 71 16 88348 88332 77.7506 68.5625 0.015423 0.0128565 2024-09-02T10:30:10.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:10.261 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:10.262 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:10.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:10.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:10.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:10.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:10.479 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:10.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:10.501 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:10.501 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:10.501 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-02T10:30:10.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:10.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:10.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:10.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:10.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:10.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:10.727 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:10 smithi151 ceph-mon[16639]: pgmap v1881: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-02T10:30:10.727 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:10 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:10.727 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:10 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:10.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:10.778 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:10.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:10.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:10.880 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:10.904 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:10.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:11.006 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:10 smithi187 ceph-mon[20072]: pgmap v1881: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-02T10:30:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:10 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:10 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:11.028 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:11.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:10 smithi162 ceph-mon[20279]: pgmap v1881: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 63 MiB/s wr, 1.01k op/s 2024-09-02T10:30:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:10 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:10 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:11.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 72 15 89473 89458 77.648 70.375 0.0181773 0.012872 2024-09-02T10:30:11.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:11.131 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:30:11.131 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:30:11.148 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:11.152 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:11.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:11.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:11.163 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:30:11.163 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:30:11.187 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:30:11.187 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:30:11.217 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:30:11.217 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:30:11.245 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:30:11.245 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:30:11.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:11.268 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:30:11.268 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:30:11.276 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:11.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:11.295 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:30:11.296 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:30:11.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:30:11.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:30:11.352 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:30:11.352 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:30:11.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:11.386 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:30:11.386 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:30:11.406 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:11.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:11.407 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:11.413 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:30:11.414 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:30:11.442 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:30:11.442 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:30:11.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:11.531 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:11.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:11.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:11.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:11.655 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:11.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:11.679 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:30:11.757 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:11.779 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:11.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:11.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:11.877 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:11 smithi187 ceph-mon[20072]: osdmap e827: 12 total, 12 up, 12 in 2024-09-02T10:30:11.877 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3616934027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:11.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:11.911 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:11.912 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:11.928 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:30:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:11 smithi151 ceph-mon[16639]: osdmap e827: 12 total, 12 up, 12 in 2024-09-02T10:30:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3616934027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:12.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:12.038 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:12.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:12.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:11 smithi162 ceph-mon[20279]: osdmap e827: 12 total, 12 up, 12 in 2024-09-02T10:30:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3616934027' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:12.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 73 16 90476 90460 77.4422 62.625 0.0340801 0.0129059 2024-09-02T10:30:12.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:12.168 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:12.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:12.233 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:12.233 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:12.233 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:12.233 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:12.233 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:12.233 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:12.234 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:12.234 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:12.234 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:12.234 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:12.234 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:12.234 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:12.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:12.292 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:12.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:12.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:12.294 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:12.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:12.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:12.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:12.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:12.552 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:12.553 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:12.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:12.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:12.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:12.740 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:12 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:12.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:12 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:12.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:12 smithi187 ceph-mon[20072]: pgmap v1883: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-02T10:30:12.741 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:12 smithi187 ceph-mon[20072]: osdmap e828: 12 total, 12 up, 12 in 2024-09-02T10:30:12.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:12.810 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:12.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:12.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:12.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:12 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:12.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:12 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:12.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:12 smithi151 ceph-mon[16639]: pgmap v1883: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-02T10:30:12.838 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:12 smithi151 ceph-mon[16639]: osdmap e828: 12 total, 12 up, 12 in 2024-09-02T10:30:12.882 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:12.882 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:12.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:12.937 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:12.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:13.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:13.060 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:13.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:12 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:12 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:12 smithi162 ceph-mon[20279]: pgmap v1883: 47 pgs: 47 active+clean; 5.3 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 64 MiB/s wr, 1.02k op/s 2024-09-02T10:30:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:12 smithi162 ceph-mon[20279]: osdmap e828: 12 total, 12 up, 12 in 2024-09-02T10:30:13.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 74 16 91341 91325 77.1262 54.0625 0.00610009 0.0129511 2024-09-02T10:30:13.162 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:13.183 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:13.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:13.245 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:13.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:13.307 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:13.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:13.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:13.430 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:13.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:13.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:13.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:13.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:13.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:13.657 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:13.685 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:13.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:13.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:30:13.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:13.816 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:13.817 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:13.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:13.944 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:13.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:14.030 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:14.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:14.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:14.077 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:14.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 75 16 92060 92044 76.6969 44.9375 0.00794787 0.0130322 2024-09-02T10:30:14.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:14.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:14.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:14.292 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:30:14.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:14.328 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:14.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:14.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:14.404 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:14.405 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:14.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-02T10:30:14.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:14.454 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:14.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:14.556 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:14.578 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:14.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:14.595 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:14.595 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:14.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:14 smithi151 ceph-mon[16639]: pgmap v1885: 47 pgs: 47 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-02T10:30:14.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:14 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:14.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:14 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:14.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:14 smithi151 ceph-mon[16639]: osdmap e829: 12 total, 12 up, 12 in 2024-09-02T10:30:14.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2037140787' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:14.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:14 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:14.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:14 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:14.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:14 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:30:14.636 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:14 smithi151 ceph-mon[16639]: osdmap e830: 12 total, 12 up, 12 in 2024-09-02T10:30:14.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:14.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:14.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:14 smithi187 ceph-mon[20072]: pgmap v1885: 47 pgs: 47 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-02T10:30:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:14 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:14 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:14 smithi187 ceph-mon[20072]: osdmap e829: 12 total, 12 up, 12 in 2024-09-02T10:30:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2037140787' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:14 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:14 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:14 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:30:14.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:14 smithi187 ceph-mon[20072]: osdmap e830: 12 total, 12 up, 12 in 2024-09-02T10:30:14.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:14.832 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:14.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:14.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:14.934 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:14.958 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:14.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:15.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:15.083 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:15.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:15.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:14 smithi162 ceph-mon[20279]: pgmap v1885: 47 pgs: 47 active+clean; 5.5 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 70 MiB/s wr, 1.11k op/s 2024-09-02T10:30:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:14 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:14 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:14 smithi162 ceph-mon[20279]: osdmap e829: 12 total, 12 up, 12 in 2024-09-02T10:30:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2037140787' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:14 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:14 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:14 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:30:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:14 smithi162 ceph-mon[20279]: osdmap e830: 12 total, 12 up, 12 in 2024-09-02T10:30:15.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 76 15 93033 93018 76.4887 60.875 0.0117306 0.0130686 2024-09-02T10:30:15.152 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:15.184 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:15.207 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:15.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:15.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:15.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:30:15.308 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:15.330 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:15.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:15.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:15.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:30:15.432 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:15.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:15.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:15.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:15.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:15.602 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:15.602 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:15.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:15.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:15.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:15.767 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:15.810 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:15.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:15.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:15.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:15.900 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:15 smithi187 ceph-mon[20072]: 8.3 scrub starts 2024-09-02T10:30:15.900 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:15 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:15.900 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:15 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:15.901 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:15 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:30:15.901 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:15 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:30:15.901 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:15 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:30:15.901 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:15 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:30:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:15 smithi151 ceph-mon[16639]: 8.3 scrub starts 2024-09-02T10:30:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:15 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:15 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:15 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:30:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:15 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:30:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:15 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:30:15.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:15 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:30:15.977 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:16.008 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:16.009 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:15 smithi162 ceph-mon[20279]: 8.3 scrub starts 2024-09-02T10:30:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:15 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:15 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:15 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:30:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:15 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:30:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:15 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:30:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:15 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:30:16.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:16.120 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 77 16 93855 93839 76.1617 51.3125 0.0144013 0.0131244 2024-09-02T10:30:16.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:16.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:16.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:16.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:16.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:16.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:16.384 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:16.384 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:16.390 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:16.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:16.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:16.527 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:16.528 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:16.630 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:16.652 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:16.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:16.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:16 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:30:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:16 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:30:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:16 smithi151 ceph-mon[16639]: 8.2 scrub starts 2024-09-02T10:30:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:16 smithi151 ceph-mon[16639]: 8.6 scrub starts 2024-09-02T10:30:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:16 smithi151 ceph-mon[16639]: pgmap v1888: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:30:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:16 smithi151 ceph-mon[16639]: osdmap e831: 12 total, 12 up, 12 in 2024-09-02T10:30:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:16 smithi151 ceph-mon[16639]: 8.0 scrub starts 2024-09-02T10:30:16.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:16.775 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:16.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:16.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:16.905 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:16.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:16.906 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:16 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:30:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:16 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:30:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:16 smithi187 ceph-mon[20072]: 8.2 scrub starts 2024-09-02T10:30:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:16 smithi187 ceph-mon[20072]: 8.6 scrub starts 2024-09-02T10:30:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:16 smithi187 ceph-mon[20072]: pgmap v1888: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:30:17.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:16 smithi187 ceph-mon[20072]: osdmap e831: 12 total, 12 up, 12 in 2024-09-02T10:30:17.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:16 smithi187 ceph-mon[20072]: 8.0 scrub starts 2024-09-02T10:30:17.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:17.030 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:17.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:17.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:17.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:30:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:16 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:30:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:16 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:30:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:16 smithi162 ceph-mon[20279]: 8.2 scrub starts 2024-09-02T10:30:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:16 smithi162 ceph-mon[20279]: 8.6 scrub starts 2024-09-02T10:30:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:16 smithi162 ceph-mon[20279]: pgmap v1888: 47 pgs: 47 active+clean; 5.6 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 65 MiB/s wr, 1.04k op/s 2024-09-02T10:30:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:16 smithi162 ceph-mon[20279]: osdmap e831: 12 total, 12 up, 12 in 2024-09-02T10:30:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:16 smithi162 ceph-mon[20279]: 8.0 scrub starts 2024-09-02T10:30:17.119 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 78 16 94735 94719 75.8903 55 0.0182806 0.0131724 2024-09-02T10:30:17.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:17.154 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:17.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:17.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:17.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:17.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:17.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:17.286 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:17.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:17.421 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:17.422 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:17.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:17.543 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:17.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:17.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:17.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:17.669 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:17.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:17.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:17.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:17.792 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:17.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:17.793 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:17.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:17.916 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:17.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:17 smithi151 ceph-mon[16639]: 8.3 scrub ok 2024-09-02T10:30:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:17 smithi151 ceph-mon[16639]: 8.2 scrub ok 2024-09-02T10:30:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:17 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:17 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:17 smithi151 ceph-mon[16639]: osdmap e832: 12 total, 12 up, 12 in 2024-09-02T10:30:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:17 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:30:17.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:17 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:30:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:17 smithi187 ceph-mon[20072]: 8.3 scrub ok 2024-09-02T10:30:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:17 smithi187 ceph-mon[20072]: 8.2 scrub ok 2024-09-02T10:30:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:17 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:17 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:18.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:17 smithi187 ceph-mon[20072]: osdmap e832: 12 total, 12 up, 12 in 2024-09-02T10:30:18.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:17 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:30:18.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:17 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:30:18.018 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:18.048 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:18.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:17 smithi162 ceph-mon[20279]: 8.3 scrub ok 2024-09-02T10:30:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:17 smithi162 ceph-mon[20279]: 8.2 scrub ok 2024-09-02T10:30:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:17 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:17 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:17 smithi162 ceph-mon[20279]: osdmap e832: 12 total, 12 up, 12 in 2024-09-02T10:30:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:17 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:30:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:17 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:30:18.120 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 79 16 95494 95478 75.5301 47.4375 0.0062539 0.0132349 2024-09-02T10:30:18.149 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:18.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:18.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:18.171 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:18.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:18.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:18.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:18.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:18.400 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:18.431 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:18.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:18.449 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:18.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-02T10:30:18.521 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:18.533 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:18.550 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:18.555 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:18.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:18.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:18.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:18.686 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:18.687 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:18.787 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:18.809 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:18.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:18.911 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:18.933 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:18.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:18.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:18 smithi151 ceph-mon[16639]: 8.6 scrub ok 2024-09-02T10:30:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:18 smithi151 ceph-mon[16639]: 8.0 scrub ok 2024-09-02T10:30:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:18 smithi151 ceph-mon[16639]: pgmap v1891: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 846 op/s 2024-09-02T10:30:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:18 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:30:18.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:18 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:30:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:18 smithi187 ceph-mon[20072]: 8.6 scrub ok 2024-09-02T10:30:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:18 smithi187 ceph-mon[20072]: 8.0 scrub ok 2024-09-02T10:30:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:18 smithi187 ceph-mon[20072]: pgmap v1891: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 846 op/s 2024-09-02T10:30:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:18 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:30:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:18 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:30:19.035 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:19.057 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:19.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:19.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:18 smithi162 ceph-mon[20279]: 8.6 scrub ok 2024-09-02T10:30:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:18 smithi162 ceph-mon[20279]: 8.0 scrub ok 2024-09-02T10:30:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:18 smithi162 ceph-mon[20279]: pgmap v1891: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 53 MiB/s wr, 846 op/s 2024-09-02T10:30:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:18 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:30:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:18 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:30:19.120 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:30:19.119910+0000 min lat: 0.00107552 max lat: 0.623325 avg lat: 0.0132845 2024-09-02T10:30:19.120 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:30:19.120 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 80 15 96329 96314 75.2391 52.25 0.020636 0.0132845 2024-09-02T10:30:19.133 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:30:19.133 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:30:19.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:19.165 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:30:19.165 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:30:19.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:19.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:19.198 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:30:19.198 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:30:19.228 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:30:19.229 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:30:19.259 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:30:19.260 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:30:19.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:19.294 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:30:19.294 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:30:19.315 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:30:19.315 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:30:19.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:19.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:19.318 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:19.338 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:30:19.339 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:30:19.365 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:30:19.366 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:30:19.389 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:30:19.389 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:30:19.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:19.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:19.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:19.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:19.463 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:30:19.463 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:30:19.492 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:30:19.492 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:30:19.544 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:19.565 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:19.566 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:19.666 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:19.690 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:19.691 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:19.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:19.826 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:19.826 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:19.927 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:19 smithi151 ceph-mon[16639]: 8.f scrub starts 2024-09-02T10:30:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:19 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:30:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:19 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:30:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2824105656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:19.949 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:19.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:19 smithi187 ceph-mon[20072]: 8.f scrub starts 2024-09-02T10:30:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:19 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:30:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:19 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:30:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2824105656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:20.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:20.073 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:20.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:20.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:20.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:19 smithi162 ceph-mon[20279]: 8.f scrub starts 2024-09-02T10:30:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:19 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:30:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:19 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:30:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2824105656' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:20.120 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 81 16 97023 97007 74.8448 43.3125 0.00985524 0.0133556 2024-09-02T10:30:20.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:20.198 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:20.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:20.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:20.332 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:20.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:20.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:20.382 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:20.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:20.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:20.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:20.562 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:20.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:20.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:20.662 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:30:20.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:20.731 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:20.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:20 smithi187 ceph-mon[20072]: 8.e scrub starts 2024-09-02T10:30:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:20 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:30:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:20 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:30:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:20 smithi187 ceph-mon[20072]: pgmap v1892: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s 2024-09-02T10:30:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2554347563' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:20.832 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:20.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:20.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:20 smithi151 ceph-mon[16639]: 8.e scrub starts 2024-09-02T10:30:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:20 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:30:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:20 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:30:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:20 smithi151 ceph-mon[16639]: pgmap v1892: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s 2024-09-02T10:30:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2554347563' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:20.960 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:20.993 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:20.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:20.994 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:20 smithi162 ceph-mon[20279]: 8.e scrub starts 2024-09-02T10:30:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:20 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:30:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:20 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:30:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:20 smithi162 ceph-mon[20279]: pgmap v1892: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 888 op/s 2024-09-02T10:30:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2554347563' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:21.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:21.119 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:21.120 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 82 16 97941 97925 74.6317 57.375 0.0144314 0.013394 2024-09-02T10:30:21.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:21.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:21.208 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:21.208 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:21.208 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:21.209 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:21.209 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:21.209 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:21.209 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:21.209 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:21.210 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:21.210 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:21.210 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:21.210 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:21.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:21.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:21.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:21.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:21.380 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:21.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:21.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:21.505 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:21.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:21.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:21.607 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:21.631 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:21.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:21.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:21.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:21.755 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:21.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:21.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:21.881 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:21.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:21.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:21 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:30:21.930 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:21 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:30:21.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2554347563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:21.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:21 smithi187 ceph-mon[20072]: osdmap e833: 12 total, 12 up, 12 in 2024-09-02T10:30:21.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:21 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:30:21.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:21 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:30:21.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:21 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:30:21.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:21 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:30:21.931 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:21 smithi187 ceph-mon[20072]: 8.b scrub starts 2024-09-02T10:30:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:21 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:30:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:21 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:30:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2554347563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:21 smithi151 ceph-mon[16639]: osdmap e833: 12 total, 12 up, 12 in 2024-09-02T10:30:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:21 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:30:21.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:21 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:30:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:21 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:30:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:21 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:30:21.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:21 smithi151 ceph-mon[16639]: 8.b scrub starts 2024-09-02T10:30:21.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:30:21.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:22.004 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:22.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:22.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:21 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:30:22.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:21 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:30:22.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2554347563' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:21 smithi162 ceph-mon[20279]: osdmap e833: 12 total, 12 up, 12 in 2024-09-02T10:30:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:21 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:30:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:21 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:30:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:21 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:30:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:21 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:30:22.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:21 smithi162 ceph-mon[20279]: 8.b scrub starts 2024-09-02T10:30:22.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:22.120 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 83 16 98874 98858 74.4351 58.3125 0.0114866 0.0134293 2024-09-02T10:30:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:22.146 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:22.146 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:22.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:30:22.247 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:22.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:22.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:22.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:22.355 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:22.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:22.395 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:22.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:22.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:22.520 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:22.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:22.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:22.640 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:22.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.8 'trim stale osdmaps' 2024-09-02T10:30:22.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:22.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:22.726 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:22.748 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:22.777 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:22.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:22.795 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:22 smithi187 ceph-mon[20072]: 8.f scrub ok 2024-09-02T10:30:22.795 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:22 smithi187 ceph-mon[20072]: 8.e scrub ok 2024-09-02T10:30:22.795 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:22 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:30:22.795 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:22 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:30:22.795 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:22 smithi187 ceph-mon[20072]: pgmap v1894: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s 2024-09-02T10:30:22.796 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:22 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:30:22.796 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:22 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:30:22.796 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:22 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:30:22.796 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:22 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:30:22.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:22 smithi151 ceph-mon[16639]: 8.f scrub ok 2024-09-02T10:30:22.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:22 smithi151 ceph-mon[16639]: 8.e scrub ok 2024-09-02T10:30:22.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:22 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:30:22.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:22 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:30:22.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:22 smithi151 ceph-mon[16639]: pgmap v1894: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s 2024-09-02T10:30:22.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:22 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:30:22.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:22 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:30:22.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:22 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:30:22.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:22 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:30:22.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:22.918 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:22.918 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:22.957 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:30:22.957 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:30:22.957 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.957 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:99567 8.d 8:b8da58cb:::benchmark_data_smithi187_136059_object99566:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e833)", 2024-09-02T10:30:22.957 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:30:22.937165+0000", 2024-09-02T10:30:22.957 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.018625745999999999, 2024-09-02T10:30:22.957 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.018649170999999999, 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 99567 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.937165+0000", 2024-09-02T10:30:22.958 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.937165+0000", 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.937166+0000", 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4300000000000002e-07 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:22.959 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.937198+0000", 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1574e-05 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.937209+0000", 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1272e-05 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.937220+0000", 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0978000000000001e-05 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.960 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:22.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.942638+0000", 2024-09-02T10:30:22.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0054178500000000001 2024-09-02T10:30:22.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.961 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:22.961 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.942779+0000", 2024-09-02T10:30:22.961 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000141583 2024-09-02T10:30:22.961 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.961 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.942945+0000", 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016538999999999999 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.947479+0000", 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.004534639 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.952244+0000", 2024-09-02T10:30:22.962 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047647990000000001 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:99570 8.d 8:b7caf146:::benchmark_data_smithi187_136059_object99569:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e833)", 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:30:22.941357+0000", 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.014434587, 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014514281, 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:22.963 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 99570 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.941357+0000", 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.941357+0000", 2024-09-02T10:30:22.964 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.941359+0000", 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9999999999999999e-06 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.941389+0000", 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0051000000000001e-05 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:22.965 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.941409+0000", 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0420000000000001e-05 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.941419+0000", 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7710000000000001e-06 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.952745+0000", 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011325864 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.966 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.952903+0000", 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000158677 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.953436+0000", 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000532728 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:99580 8.d 8.9f63410d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e833)", 2024-09-02T10:30:22.967 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:30:22.954065+0000", 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.001725641, 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001838955, 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "queued for pg", 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 99580 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:22.968 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.954065+0000", 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.954065+0000", 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.954067+0000", 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.53e-06 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.969 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.954089+0000", 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1594999999999999e-05 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.954101+0000", 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2480000000000001e-05 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.954108+0000", 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9550000000000003e-06 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:22.970 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:22.971 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:22.971 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.971 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.971 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:99582 8.3 8:cbda8e3b:::benchmark_data_smithi187_136059_object99581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e833)", 2024-09-02T10:30:22.971 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:30:22.955012+0000", 2024-09-02T10:30:22.971 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00077919299999999996, 2024-09-02T10:30:22.971 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00091932600000000004, 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 99582 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:30:22.972 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.955012+0000", 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.955012+0000", 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.955013+0000", 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.37e-07 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.973 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.955039+0000", 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6069999999999999e-05 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.955051+0000", 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2447e-05 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.955057+0000", 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2979999999999997e-06 2024-09-02T10:30:22.974 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.955129+0000", 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1874999999999999e-05 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.955263+0000", 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013361300000000001 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,5", 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:22.955388+0000", 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012467000000000001 2024-09-02T10:30:22.975 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:22.976 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:30:22.976 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:22.976 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:30:22.976 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:30:22.976 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 4 2024-09-02T10:30:22.976 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:30:23.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:23.042 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:23.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:23.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:22 smithi162 ceph-mon[20279]: 8.f scrub ok 2024-09-02T10:30:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:22 smithi162 ceph-mon[20279]: 8.e scrub ok 2024-09-02T10:30:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:22 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:30:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:22 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:30:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:22 smithi162 ceph-mon[20279]: pgmap v1894: 47 pgs: 47 active+clean; 5.9 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 52 MiB/s wr, 835 op/s 2024-09-02T10:30:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:22 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:30:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:22 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:30:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:22 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:30:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:22 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:30:23.120 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 84 16 99715 99699 74.1746 52.5625 0.00905289 0.0134758 2024-09-02T10:30:23.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:23.166 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:23.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:23.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:23.268 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:23.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:23.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:23.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:23.327 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:30:23.327 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:30:23.339 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:23.359 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:30:23.360 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:30:23.392 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:30:23.393 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:30:23.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:23.420 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:30:23.420 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:30:23.429 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:23.430 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:23.445 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:30:23.445 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:30:23.469 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:30:23.470 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:30:23.496 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:30:23.497 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:30:23.518 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:30:23.518 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:30:23.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:23.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:30:23.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:30:23.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:23.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:23.575 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:30:23.575 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:30:23.609 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:30:23.609 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:30:23.643 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:30:23.643 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:30:23.661 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:23.684 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:23.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:23.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:23.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:30:23.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:23 smithi187 ceph-mon[20072]: 8.c scrub starts 2024-09-02T10:30:23.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:23 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:30:23.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:23 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:30:23.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:23 smithi187 ceph-mon[20072]: 8.b scrub ok 2024-09-02T10:30:23.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/521291431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:23.785 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:23 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:30:23.786 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:23 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:30:23.786 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:23.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:30:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:23.828 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:23.829 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:23.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:23 smithi151 ceph-mon[16639]: 8.c scrub starts 2024-09-02T10:30:23.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:23 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:30:23.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:23 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:30:23.841 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:23 smithi151 ceph-mon[16639]: 8.b scrub ok 2024-09-02T10:30:23.842 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/521291431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:23.842 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:23 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:30:23.842 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:23 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:30:23.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:23.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:23.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:24.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:23 smithi162 ceph-mon[20279]: 8.c scrub starts 2024-09-02T10:30:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:23 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:30:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:23 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:30:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:23 smithi162 ceph-mon[20279]: 8.b scrub ok 2024-09-02T10:30:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/521291431' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:23 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:30:24.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:23 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:30:24.105 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:24.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:24.121 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 85 16 100576 100560 73.9344 53.8125 0.010495 0.0135201 2024-09-02T10:30:24.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:24.215 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:24.236 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:24.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:24.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:24.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:24.372 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:24.374 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:24.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:24.497 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:24.498 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:24.599 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:24.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:24.621 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:24.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:24.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:24.752 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:24.753 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:24.753 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:24.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:24.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:24.780 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:24.780 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:24.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:24.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:24.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:24 smithi151 ceph-mon[16639]: pgmap v1895: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 895 op/s 2024-09-02T10:30:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:24 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:24 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:24 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:30:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:24 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:30:24.992 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:24.995 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:24 smithi162 ceph-mon[20279]: pgmap v1895: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 895 op/s 2024-09-02T10:30:24.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:24 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:24.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:24 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:24.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:24 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:30:24.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:24 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:30:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:24 smithi187 ceph-mon[20072]: pgmap v1895: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 MiB/s wr, 895 op/s 2024-09-02T10:30:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:24 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:24 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:24 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:30:25.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:24 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:30:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:25.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:25.121 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:25.121 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 86 16 101357 101341 73.6422 48.8125 0.00489804 0.0135735 2024-09-02T10:30:25.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:25.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:25.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:25.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:25.272 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:25.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:25.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:25.398 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:25.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:25.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:25.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:25.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:25.628 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:25.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:30:25.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:25.666 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:25.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:25.789 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:25.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:25.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:25 smithi162 ceph-mon[20279]: 8.c scrub ok 2024-09-02T10:30:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:25 smithi162 ceph-mon[20279]: osdmap e834: 12 total, 12 up, 12 in 2024-09-02T10:30:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:25 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:30:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:25 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:30:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:25 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:25 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:25.892 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:25.915 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:25.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:25 smithi151 ceph-mon[16639]: 8.c scrub ok 2024-09-02T10:30:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:25 smithi151 ceph-mon[16639]: osdmap e834: 12 total, 12 up, 12 in 2024-09-02T10:30:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:25 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:30:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:25 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:30:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:25 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:25.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:25 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:25 smithi187 ceph-mon[20072]: 8.c scrub ok 2024-09-02T10:30:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:25 smithi187 ceph-mon[20072]: osdmap e834: 12 total, 12 up, 12 in 2024-09-02T10:30:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:25 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:30:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:25 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:30:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:25 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:26.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:25 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:26.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:26.021 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:26.037 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:30:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:26.053 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:26.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.9 'trim stale osdmaps' 2024-09-02T10:30:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:26.055 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:26.055 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:26.121 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 87 15 102337 102322 73.5002 61.3125 0.00485693 0.0135998 2024-09-02T10:30:26.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:26.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:26.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:26.188 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:26.289 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:26.315 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:26.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:26.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:26.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:26.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:26.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:26.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:26.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:26.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:26.582 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:26.633 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:26.637 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:26.637 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:26.683 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:26.715 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:26.716 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:26 smithi187 ceph-mon[20072]: 8.5 scrub starts 2024-09-02T10:30:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:26 smithi187 ceph-mon[20072]: pgmap v1897: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 809 op/s 2024-09-02T10:30:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:26 smithi187 ceph-mon[20072]: osdmap e835: 12 total, 12 up, 12 in 2024-09-02T10:30:26.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:26.839 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:26.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:26.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:26 smithi151 ceph-mon[16639]: 8.5 scrub starts 2024-09-02T10:30:26.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:26 smithi151 ceph-mon[16639]: pgmap v1897: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 809 op/s 2024-09-02T10:30:26.916 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:26 smithi151 ceph-mon[16639]: osdmap e835: 12 total, 12 up, 12 in 2024-09-02T10:30:26.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:26.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:26.970 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:27.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:26 smithi162 ceph-mon[20279]: 8.5 scrub starts 2024-09-02T10:30:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:26 smithi162 ceph-mon[20279]: pgmap v1897: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 6.1 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 51 MiB/s wr, 809 op/s 2024-09-02T10:30:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:26 smithi162 ceph-mon[20279]: osdmap e835: 12 total, 12 up, 12 in 2024-09-02T10:30:27.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:27.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:27.121 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 88 16 103280 103264 73.334 58.875 0.0459988 0.01363 2024-09-02T10:30:27.203 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:27.203 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:27.203 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:27.203 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:27.203 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:27.203 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:27.203 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:27.203 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:27.203 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:27.204 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:27.204 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:27.204 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:27.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:27.240 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:27.240 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:27.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:30:27.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:27.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:30:27.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:27.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:27.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:27.539 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:27.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:27.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:27.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:27.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:27.712 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:30:27.715 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:27.716 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:27.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:27.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:27.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:27.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3342852889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:30:27.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:27 smithi151 ceph-mon[16639]: 8.4 scrub starts 2024-09-02T10:30:27.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:27.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:27.972 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3342852889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:30:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:27 smithi187 ceph-mon[20072]: 8.4 scrub starts 2024-09-02T10:30:28.028 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:28.029 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:28.029 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.11 'trim stale osdmaps' 2024-09-02T10:30:28.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3342852889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:30:28.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:27 smithi162 ceph-mon[20279]: 8.4 scrub starts 2024-09-02T10:30:28.104 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:28.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:28.121 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 89 16 104181 104165 73.1427 56.3125 0.0285539 0.0136668 2024-09-02T10:30:28.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:28.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:28.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:28.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:28.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:28.461 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:28.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:28.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:28.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:28.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:28.611 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:28.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:28.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:28.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:28.750 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:28.815 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:28.832 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:28.851 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:28.875 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:28.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:28.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: 8.5 scrub ok 2024-09-02T10:30:28.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: pgmap v1899: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 863 op/s 2024-09-02T10:30:28.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:28.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3342852889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:30:28.965 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: osdmap e836: 12 total, 12 up, 12 in 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1085270876,config=2353167246,health=551204407,kv=4028617667}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3755806917,logm=2932910307}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1687224646}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359695692}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3049432331}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616085451}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=447675300}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4286176094}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906336597}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2277579298}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3951522783}) 2024-09-02T10:30:28.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2387660508}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1796675424}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3699565650}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4104585631}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1133187332}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646197985}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4127499315}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1414810619}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2596196412}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1238874972}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1926252686}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1948771250}) 2024-09-02T10:30:28.967 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3572157891}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071262198}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3223603283}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2737864798}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=284641475}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1875891670}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639846181}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108891258}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3881218301}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999149336}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=64521341}) 2024-09-02T10:30:28.968 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221717504}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569158190}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=548499738}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3919362059}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3354399171}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1602422217}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241899215}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=294897037}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460168738}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1117520745}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3066292357}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=391565825}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=211319397}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3071144008}) 2024-09-02T10:30:28.969 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641165272}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=839626543}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711645376}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3583540961}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=532867966}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4265178065}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=49030164}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2655662072}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1621291590}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4008687318}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4066241515}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203367659}) 2024-09-02T10:30:28.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2048834184}) 2024-09-02T10:30:28.971 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1679327986}) 2024-09-02T10:30:28.971 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413397617}) 2024-09-02T10:30:28.971 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474034438}) 2024-09-02T10:30:28.971 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=697199978}) 2024-09-02T10:30:28.971 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=985293915}) 2024-09-02T10:30:28.971 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=676591543}) 2024-09-02T10:30:28.971 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1358361032}) 2024-09-02T10:30:28.971 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287959280}) 2024-09-02T10:30:28.971 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3188037203}) 2024-09-02T10:30:28.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2896866101}) 2024-09-02T10:30:28.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3389192744}) 2024-09-02T10:30:28.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1137273860}) 2024-09-02T10:30:28.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2152923190}) 2024-09-02T10:30:28.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=740955825}) 2024-09-02T10:30:28.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673212987}) 2024-09-02T10:30:28.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=4150923161,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=1816728726}) 2024-09-02T10:30:28.972 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=7,mon_config_key=93} crc {mgrstat=1800733112,mon_config_key=1216732797}) 2024-09-02T10:30:28.973 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=88818814}) 2024-09-02T10:30:28.973 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=98,monmap=2} crc {mon_config_key=1671813372,monmap=4155822442}) 2024-09-02T10:30:28.973 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {monmap=3,osd_metadata=12,osd_pg_creating=1,osdmap=84} crc {monmap=2056657799,osd_metadata=1182194493,osd_pg_creating=101258768,osdmap=2609012902}) 2024-09-02T10:30:28.973 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=29} crc {osdmap=170678690}) 2024-09-02T10:30:28.977 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:29.002 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:29.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: 8.5 scrub ok 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: pgmap v1899: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 863 op/s 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3342852889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: osdmap e836: 12 total, 12 up, 12 in 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1085270876,config=2353167246,health=551204407,kv=4028617667}) 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3755806917,logm=2932910307}) 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1687224646}) 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359695692}) 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3049432331}) 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616085451}) 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=447675300}) 2024-09-02T10:30:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4286176094}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906336597}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2277579298}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3951522783}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2387660508}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1796675424}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3699565650}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4104585631}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1133187332}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646197985}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4127499315}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1414810619}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2596196412}) 2024-09-02T10:30:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1238874972}) 2024-09-02T10:30:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1926252686}) 2024-09-02T10:30:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1948771250}) 2024-09-02T10:30:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3572157891}) 2024-09-02T10:30:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071262198}) 2024-09-02T10:30:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3223603283}) 2024-09-02T10:30:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2737864798}) 2024-09-02T10:30:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=284641475}) 2024-09-02T10:30:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1875891670}) 2024-09-02T10:30:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639846181}) 2024-09-02T10:30:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108891258}) 2024-09-02T10:30:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3881218301}) 2024-09-02T10:30:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999149336}) 2024-09-02T10:30:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=64521341}) 2024-09-02T10:30:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221717504}) 2024-09-02T10:30:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569158190}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=548499738}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3919362059}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3354399171}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1602422217}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241899215}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=294897037}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460168738}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1117520745}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3066292357}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=391565825}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=211319397}) 2024-09-02T10:30:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3071144008}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641165272}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=839626543}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711645376}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3583540961}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=532867966}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4265178065}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=49030164}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2655662072}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1621291590}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4008687318}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4066241515}) 2024-09-02T10:30:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203367659}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2048834184}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1679327986}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413397617}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474034438}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=697199978}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=985293915}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=676591543}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1358361032}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287959280}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3188037203}) 2024-09-02T10:30:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2896866101}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3389192744}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1137273860}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2152923190}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=740955825}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673212987}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=4150923161,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=1816728726}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=7,mon_config_key=93} crc {mgrstat=1800733112,mon_config_key=1216732797}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=88818814}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=98,monmap=2} crc {mon_config_key=1671813372,monmap=4155822442}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {monmap=3,osd_metadata=12,osd_pg_creating=1,osdmap=84} crc {monmap=2056657799,osd_metadata=1182194493,osd_pg_creating=101258768,osdmap=2609012902}) 2024-09-02T10:30:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=29} crc {osdmap=170678690}) 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: 8.5 scrub ok 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: pgmap v1899: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.2 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 863 op/s 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3342852889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: osdmap e836: 12 total, 12 up, 12 in 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1085270876,config=2353167246,health=551204407,kv=4028617667}) 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {kv=3,logm=97} crc {kv=3755806917,logm=2932910307}) 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1687224646}) 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=359695692}) 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3049432331}) 2024-09-02T10:30:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2616085451}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=447675300}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4286176094}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2906336597}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2277579298}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3951522783}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2387660508}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1796675424}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3699565650}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4104585631}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1133187332}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2646197985}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4127499315}) 2024-09-02T10:30:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1414810619}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2596196412}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1238874972}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1926252686}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1948771250}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3572157891}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071262198}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3223603283}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2737864798}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=284641475}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1875891670}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1639846181}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1108891258}) 2024-09-02T10:30:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3881218301}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999149336}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=64521341}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4221717504}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3569158190}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=548499738}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3919362059}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3354399171}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1602422217}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4241899215}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=294897037}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1460168738}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1117520745}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3066292357}) 2024-09-02T10:30:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=391565825}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=211319397}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3071144008}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2641165272}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=839626543}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3711645376}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3583540961}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=532867966}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4265178065}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=49030164}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2655662072}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1621291590}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4008687318}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4066241515}) 2024-09-02T10:30:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=203367659}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2048834184}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1679327986}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1413397617}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2474034438}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=697199978}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=985293915}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=676591543}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1358361032}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287959280}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3188037203}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2896866101}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3389192744}) 2024-09-02T10:30:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1137273860}) 2024-09-02T10:30:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2152923190}) 2024-09-02T10:30:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=740955825}) 2024-09-02T10:30:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673212987}) 2024-09-02T10:30:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=70,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=6} crc {logm=4150923161,mdsmap=1083786272,mgr=3566298445,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=1816728726}) 2024-09-02T10:30:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=7,mon_config_key=93} crc {mgrstat=1800733112,mon_config_key=1216732797}) 2024-09-02T10:30:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=88818814}) 2024-09-02T10:30:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=98,monmap=2} crc {mon_config_key=1671813372,monmap=4155822442}) 2024-09-02T10:30:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {monmap=3,osd_metadata=12,osd_pg_creating=1,osdmap=84} crc {monmap=2056657799,osd_metadata=1182194493,osd_pg_creating=101258768,osdmap=2609012902}) 2024-09-02T10:30:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=29} crc {osdmap=170678690}) 2024-09-02T10:30:29.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:29.122 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 90 11 105278 105267 73.0952 68.875 0.0113658 0.0136748 2024-09-02T10:30:29.126 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:29.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:29.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:29.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:29.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:29.352 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:29.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:29.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:29.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:29.497 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:29.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:29.599 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:29.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:29.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:29.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:29.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:29.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:29.749 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:29.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:29.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:29.872 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:29.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:29.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:29.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:30.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:30.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:30.122 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total time run: 90.7015 2024-09-02T10:30:30.122 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total writes made: 105278 2024-09-02T10:30:30.122 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Write size: 65536 2024-09-02T10:30:30.122 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object size: 65536 2024-09-02T10:30:30.122 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Bandwidth (MB/sec): 72.5443 2024-09-02T10:30:30.122 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Bandwidth: 17.3785 2024-09-02T10:30:30.122 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max bandwidth (MB/sec): 115.75 2024-09-02T10:30:30.123 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min bandwidth (MB/sec): 43.3125 2024-09-02T10:30:30.123 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average IOPS: 1160 2024-09-02T10:30:30.123 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev IOPS: 278.055 2024-09-02T10:30:30.123 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max IOPS: 1852 2024-09-02T10:30:30.123 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min IOPS: 693 2024-09-02T10:30:30.123 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average Latency(s): 0.0136949 2024-09-02T10:30:30.123 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Latency(s): 0.0186782 2024-09-02T10:30:30.123 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max latency(s): 0.704115 2024-09-02T10:30:30.123 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min latency(s): 0.00107552 2024-09-02T10:30:30.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:30.124 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:30.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:30.129 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Cleaning up (deleting benchmark objects) 2024-09-02T10:30:30.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:30.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:30.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:30.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:30.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:30.371 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:30.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:30.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:30.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:30.508 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:30.609 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:30.632 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:30.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:30.735 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:30.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:30.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:30.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:30.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:30:30.861 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:30.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:30.885 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:30.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:30.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:30 smithi187 ceph-mon[20072]: pgmap v1901: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s 2024-09-02T10:30:31.026 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:30 smithi151 ceph-mon[16639]: pgmap v1901: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s 2024-09-02T10:30:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:31.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:31.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:31.066 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:31.066 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:30 smithi162 ceph-mon[20279]: pgmap v1901: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 54 MiB/s wr, 865 op/s 2024-09-02T10:30:31.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:31.155 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:31.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:31.257 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:31.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:31.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:31.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:31.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:31.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:31.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:31.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:31.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:31.568 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:31.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:31.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:31.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:31.692 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:31.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:31.747 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:30:31.748 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:30:31.774 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:30:31.774 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:30:31.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:31.797 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:30:31.798 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:30:31.816 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:31.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:31.825 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:30:31.825 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:30:31.854 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:30:31.855 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:30:31.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:30:31.882 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:30:31.909 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:30:31.909 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:30:31.918 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:31.936 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:30:31.936 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:30:31.939 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:31.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:31.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:31.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:30:31.964 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:30:31.986 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:30:31.987 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:30:32.009 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:30:32.009 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:30:32.030 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:30:32.030 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:30:32.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:32.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:32.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:32.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:32.166 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:32.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:32.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:32.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:32.296 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:32.299 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:30:32.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:32.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:32.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:32.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:32.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:32.459 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:32.560 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:32.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:32.584 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:32.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:32.686 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:32.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:32.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:32.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:32.817 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:32.839 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:32.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:32 smithi151 ceph-mon[16639]: pgmap v1902: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 643 op/s 2024-09-02T10:30:32.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3204986896' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:32.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:32.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:32.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:32.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:32 smithi187 ceph-mon[20072]: pgmap v1902: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 643 op/s 2024-09-02T10:30:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3204986896' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:33.031 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:33.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:32 smithi162 ceph-mon[20279]: pgmap v1902: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 40 MiB/s wr, 643 op/s 2024-09-02T10:30:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3204986896' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:33.100 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:33.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:33.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:33.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:33.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:33.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:33.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:33.357 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:33.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:33.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:33.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:33.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:33.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:33.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:33.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:33.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:33.615 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:33.643 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:33.644 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:33.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:33.740 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:33.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:33.741 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:33.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:33.872 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:33.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:33.874 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:33.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:33.999 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:33.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:34.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:34.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:34.128 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:34.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:34.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:34.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:34.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:34.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:34.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:34.360 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:34.381 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:34.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:34.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:34.483 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:34.513 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:34.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:34.615 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:34.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:34.645 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:34.729 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:30:34.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:34.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:34.784 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:34.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:34.923 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:34.924 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:34.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:34 smithi151 ceph-mon[16639]: pgmap v1903: 47 pgs: 47 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 45 MiB/s wr, 902 op/s 2024-09-02T10:30:34.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/10630732' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:34 smithi187 ceph-mon[20072]: pgmap v1903: 47 pgs: 47 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 45 MiB/s wr, 902 op/s 2024-09-02T10:30:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/10630732' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:35.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:30:35.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:35.082 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:35.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:35.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:34 smithi162 ceph-mon[20279]: pgmap v1903: 47 pgs: 47 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 45 MiB/s wr, 902 op/s 2024-09-02T10:30:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/10630732' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:35.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:35.210 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:35.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:35.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:35.335 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:35.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:35.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:35.461 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:35.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:35.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:35.495 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:30:35.495 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:30:35.496 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:30:35.496 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:30:35.496 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:30:35.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:35.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:35.590 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:35.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:35.692 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:35.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:35.718 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:35.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:35.841 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:35.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:35.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:35.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:35.965 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:35.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:36.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:35 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:36.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:35 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:36.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:35 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:36.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/10630732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:36.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:35 smithi187 ceph-mon[20072]: osdmap e837: 12 total, 12 up, 12 in 2024-09-02T10:30:36.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:36.090 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:35 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:35 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:35 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/10630732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:36.091 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:35 smithi162 ceph-mon[20279]: osdmap e837: 12 total, 12 up, 12 in 2024-09-02T10:30:36.095 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:36.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:35 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:35 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:35 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/10630732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:35 smithi151 ceph-mon[16639]: osdmap e837: 12 total, 12 up, 12 in 2024-09-02T10:30:36.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:36.220 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:36.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:36.321 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:36.345 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:36.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:36.447 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:36.469 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:36.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:36.572 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:36.576 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:36.576 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:36.576 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:36.576 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:36.577 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:36.577 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:36.577 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:36.577 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:36.577 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:36.577 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:36.577 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:36.577 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:36.600 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:36.601 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:36.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:36.726 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:36.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:36.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:36.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:36.851 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:36.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:36.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:36.899 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:36 smithi151 ceph-mon[16639]: pgmap v1905: 47 pgs: 47 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 31 MiB/s wr, 836 op/s 2024-09-02T10:30:36.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:36.977 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:36.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:36 smithi187 ceph-mon[20072]: pgmap v1905: 47 pgs: 47 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 31 MiB/s wr, 836 op/s 2024-09-02T10:30:37.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:36 smithi162 ceph-mon[20279]: pgmap v1905: 47 pgs: 47 active+clean; 6.3 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 127 B/s rd, 31 MiB/s wr, 836 op/s 2024-09-02T10:30:37.106 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:37.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:37.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:37.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:37.231 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:37.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:37.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:37.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:37.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:37.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:37.402 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:37.403 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:37.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.10 'trim stale osdmaps' 2024-09-02T10:30:37.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:37.485 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:37.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:37.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:37.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:37.616 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:37.617 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:37.718 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:37.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:37.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:37.851 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:37.886 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:37.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:37.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:30:37.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:38.014 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:38.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:38.015 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:38.018 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:38.036 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:38.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:38.151 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:38.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:38.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:30:38.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:38.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:38.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:38.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:38.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:38.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:38.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:38.560 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:38.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:38.662 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:38.691 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:38.693 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:38.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:38 smithi151 ceph-mon[16639]: pgmap v1906: 47 pgs: 47 active+clean; 6.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 104 B/s rd, 20 MiB/s wr, 796 op/s 2024-09-02T10:30:38.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:38.817 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:38.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:38.901 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:30:38.901 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:30:38.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:38.930 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:30:38.931 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:30:38.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:38.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:38.963 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:30:38.963 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:30:38.990 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:30:38.991 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:30:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:38 smithi187 ceph-mon[20072]: pgmap v1906: 47 pgs: 47 active+clean; 6.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 104 B/s rd, 20 MiB/s wr, 796 op/s 2024-09-02T10:30:39.021 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:30:39.021 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:30:39.049 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:30:39.050 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:30:39.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:39.075 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:39.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:39.080 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:30:39.081 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:30:39.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:38 smithi162 ceph-mon[20279]: pgmap v1906: 47 pgs: 47 active+clean; 6.1 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 104 B/s rd, 20 MiB/s wr, 796 op/s 2024-09-02T10:30:39.104 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:30:39.104 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:30:39.136 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:30:39.136 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:30:39.162 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:30:39.163 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:30:39.167 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:39.174 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:39.174 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:39.174 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:39.174 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:39.174 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:39.174 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:39.174 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:39.175 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:39.175 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:39.175 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:39.175 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:39.175 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:39.176 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:39.194 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:30:39.194 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:30:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:39.211 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:39.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:39.220 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:30:39.220 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:30:39.312 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:39.335 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:39.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:39.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:39.437 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:39.460 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:39.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:39.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:39.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:39.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:39.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:39.688 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:39.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:39.722 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:39.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:39.847 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:39.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/107623131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:39.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:39.972 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:39.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/107623131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:40.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/107623131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:40.096 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:40.096 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:40.109 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:40.109 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:40.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:40.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:40.237 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:40.238 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:40.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:40.367 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:40.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:40.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:40.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:40.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:40.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:40.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:40.549 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:40.550 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-02T10:30:40.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:40.619 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:40.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:40.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:40.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:40.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:40.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:40.754 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:40.764 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:40 smithi151 ceph-mon[16639]: pgmap v1907: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 12 MiB/s wr, 861 op/s 2024-09-02T10:30:40.855 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:40.878 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:40.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:40.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:40.992 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:40 smithi187 ceph-mon[20072]: pgmap v1907: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 12 MiB/s wr, 861 op/s 2024-09-02T10:30:41.007 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:41.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:41.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:40 smithi162 ceph-mon[20279]: pgmap v1907: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 12 MiB/s wr, 861 op/s 2024-09-02T10:30:41.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:41.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:41.139 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:41.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:41.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:41.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:41.269 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:41.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:41.394 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:41.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:41.496 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:41.522 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:41.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:41.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:41.649 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:41.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:41.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:41.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:41.774 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:41.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:41.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:41.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:41.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:42.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:42.032 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:42.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:42.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:42.158 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:42.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:42.221 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:30:42.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:42.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:42.289 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:42.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:42.419 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:42.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:42.421 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:42.502 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:42.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:42.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:42.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:42.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:42.668 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:42.700 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:42.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:42.753 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:30:42.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:42.829 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:42.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:42.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:42 smithi151 ceph-mon[16639]: pgmap v1908: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 8.9 MiB/s wr, 865 op/s 2024-09-02T10:30:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2977952302' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:42 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:42.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:42.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:42 smithi187 ceph-mon[20072]: pgmap v1908: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 8.9 MiB/s wr, 865 op/s 2024-09-02T10:30:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2977952302' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:42 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:43.062 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:43.094 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:42 smithi162 ceph-mon[20279]: pgmap v1908: 47 pgs: 47 active+clean; 6.0 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 102 B/s rd, 8.9 MiB/s wr, 865 op/s 2024-09-02T10:30:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2977952302' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:42 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:30:43.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:43.096 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:43.197 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:43.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:43.231 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:43.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:43.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:43.366 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:43.366 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:43.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:43.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:30:43.498 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:43.499 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:43.600 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:43.622 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:43.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:43.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:43.724 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:43.747 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:43.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:43.782 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:43.847 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:43.848 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:43.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:43 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:43.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:43 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:30:43.849 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:43 smithi151 ceph-mon[16639]: osdmap e838: 12 total, 12 up, 12 in 2024-09-02T10:30:43.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:43.875 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:43.875 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:43.977 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:44.009 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:44.010 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:44.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:43 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:44.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:43 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:30:44.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:43 smithi187 ceph-mon[20072]: osdmap e838: 12 total, 12 up, 12 in 2024-09-02T10:30:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:43 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:43 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:30:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:43 smithi162 ceph-mon[20279]: osdmap e838: 12 total, 12 up, 12 in 2024-09-02T10:30:44.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:44.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:44.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:44.140 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:44.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:44.264 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:44.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:44.366 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:44.382 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:44.382 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:44.390 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:44.391 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:44.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:44.520 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:44.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:44.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:44.622 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:44.648 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:44.649 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:44.750 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:44.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:44.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:44.777 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:44.878 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:44.900 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:44.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:44.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:45.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:44 smithi187 ceph-mon[20072]: pgmap v1910: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s 2024-09-02T10:30:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:44 smithi187 ceph-mon[20072]: 8.1 scrub starts 2024-09-02T10:30:45.032 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:45.033 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:44 smithi162 ceph-mon[20279]: pgmap v1910: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s 2024-09-02T10:30:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:44 smithi162 ceph-mon[20279]: 8.1 scrub starts 2024-09-02T10:30:45.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:45.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:45.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:44 smithi151 ceph-mon[16639]: pgmap v1910: 47 pgs: 47 active+clean; 5.8 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 928 op/s 2024-09-02T10:30:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:44 smithi151 ceph-mon[16639]: 8.1 scrub starts 2024-09-02T10:30:45.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:45.288 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:45.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:45.390 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:45.418 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:45.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:45.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:45.520 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:45.544 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:45.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:45.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:45.615 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:30:45.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:45.674 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:45.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:45.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:45.800 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:45.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:45.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:45.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:45.926 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:45.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:45.929 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:45 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:30:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:45 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:30:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:45 smithi151 ceph-mon[16639]: 8.9 scrub starts 2024-09-02T10:30:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:45 smithi151 ceph-mon[16639]: 8.8 scrub starts 2024-09-02T10:30:46.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:46.053 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:46.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:46.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:45 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:30:46.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:45 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:30:46.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:45 smithi162 ceph-mon[20279]: 8.9 scrub starts 2024-09-02T10:30:46.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:45 smithi162 ceph-mon[20279]: 8.8 scrub starts 2024-09-02T10:30:46.154 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:46.176 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:46.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:46.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:45 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:30:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:45 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:30:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:45 smithi187 ceph-mon[20072]: 8.9 scrub starts 2024-09-02T10:30:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:45 smithi187 ceph-mon[20072]: 8.8 scrub starts 2024-09-02T10:30:46.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:46.301 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:46.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:46.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:46.403 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:46.432 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:46.433 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:46.534 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:46.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:46.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:46.661 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:46.693 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:46.693 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:46.794 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:46.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:46.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:46.919 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:46.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:46.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:46.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:47.043 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:47.067 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:47.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:47.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:46 smithi162 ceph-mon[20279]: 8.1 scrub ok 2024-09-02T10:30:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:46 smithi162 ceph-mon[20279]: pgmap v1911: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-09-02T10:30:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:46 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:30:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:46 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:30:47.127 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:47.170 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:46 smithi151 ceph-mon[16639]: 8.1 scrub ok 2024-09-02T10:30:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:46 smithi151 ceph-mon[16639]: pgmap v1911: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-09-02T10:30:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:46 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:30:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:46 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:30:47.200 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:47.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:47.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:46 smithi187 ceph-mon[20072]: 8.1 scrub ok 2024-09-02T10:30:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:46 smithi187 ceph-mon[20072]: pgmap v1911: 47 pgs: 47 active+clean; 5.7 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 979 op/s 2024-09-02T10:30:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:46 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:30:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:46 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:30:47.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:47.328 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:47.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:47.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:47.381 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:47.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-02T10:30:47.430 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:47.452 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:47.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:47.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:47.554 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:47.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:47.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:47.695 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:47.720 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:47.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:47.823 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:47.861 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:47.862 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:47.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:47 smithi151 ceph-mon[16639]: 8.8 scrub ok 2024-09-02T10:30:47.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:30:47.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:47 smithi151 ceph-mon[16639]: 8.9 scrub ok 2024-09-02T10:30:47.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:47 smithi151 ceph-mon[16639]: 8.4 scrub starts 2024-09-02T10:30:47.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:47 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:30:47.915 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:47 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:30:47.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:30:47.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:48.027 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:48.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:48.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:48.062 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:47 smithi187 ceph-mon[20072]: 8.8 scrub ok 2024-09-02T10:30:48.062 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:30:48.062 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:47 smithi187 ceph-mon[20072]: 8.9 scrub ok 2024-09-02T10:30:48.062 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:47 smithi187 ceph-mon[20072]: 8.4 scrub starts 2024-09-02T10:30:48.062 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:47 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:30:48.062 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:47 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:30:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:47 smithi162 ceph-mon[20279]: 8.8 scrub ok 2024-09-02T10:30:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:30:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:47 smithi162 ceph-mon[20279]: 8.9 scrub ok 2024-09-02T10:30:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:47 smithi162 ceph-mon[20279]: 8.4 scrub starts 2024-09-02T10:30:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:47 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:30:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:47 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:30:48.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:48.160 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:48.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:48.161 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:48.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:48.287 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:48.288 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:48.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:48.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:48.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:48.490 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:48.521 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:48.541 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:30:48.542 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:30:48.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:48.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:48.570 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:48.577 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:48.588 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:30:48.588 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:30:48.616 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:30:48.616 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:30:48.645 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:30:48.645 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:30:48.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:48.675 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:30:48.675 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:30:48.682 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:48.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:48.706 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:30:48.706 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:30:48.735 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:30:48.735 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:30:48.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:30:48.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:30:48.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:48.789 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:30:48.789 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:30:48.810 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:48.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:48.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:30:48.823 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:30:48.854 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:30:48.854 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:30:48.877 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:30:48.878 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:30:48.911 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:48.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:48.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:48.943 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:48.990 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:48 smithi151 ceph-mon[16639]: pgmap v1912: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 976 op/s 2024-09-02T10:30:48.990 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:48 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:30:48.990 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:48 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:30:48.990 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:48 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:30:48.991 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:48 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:30:48.991 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1359582867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:49.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:49.066 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:49.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:48 smithi162 ceph-mon[20279]: pgmap v1912: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 976 op/s 2024-09-02T10:30:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:48 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:30:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:48 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:30:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:48 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:30:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:48 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:30:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1359582867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:49.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:49.191 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:49.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:49.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:48 smithi187 ceph-mon[20072]: pgmap v1912: 47 pgs: 1 active+clean+scrubbing, 46 active+clean; 5.5 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 976 op/s 2024-09-02T10:30:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:48 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:30:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:48 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:30:49.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:48 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:30:49.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:48 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:30:49.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1359582867' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:49.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:49.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:49.324 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:49.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:49.456 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:49.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:49.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:49.565 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:30:49.565 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:30:49.565 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:30:49.566 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:30:49.566 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:30:49.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:49.593 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:49.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:49.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:49.727 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:49.828 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:49.845 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:49 smithi187 ceph-mon[20072]: 8.4 scrub ok 2024-09-02T10:30:49.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:49.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:49.866 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:49.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:49.998 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:49.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:49.999 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:50.039 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:49 smithi151 ceph-mon[16639]: 8.4 scrub ok 2024-09-02T10:30:50.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:49 smithi162 ceph-mon[20279]: 8.4 scrub ok 2024-09-02T10:30:50.100 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:50.122 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:50.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:50.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:50.224 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:50.250 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:50.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:50.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:50.380 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:50.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:50.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:50.482 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:50.510 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:50.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:50.613 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:50.648 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:50.649 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:50.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:30:50.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:50.777 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:50.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:50.879 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:50.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:50.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:50.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:50 smithi151 ceph-mon[16639]: pgmap v1913: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 956 op/s 2024-09-02T10:30:51.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:51.036 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:51.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:51.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:51.067 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:50 smithi187 ceph-mon[20072]: pgmap v1913: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 956 op/s 2024-09-02T10:30:51.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:50 smithi162 ceph-mon[20279]: pgmap v1913: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.4 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 956 op/s 2024-09-02T10:30:51.138 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:51.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:51.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:51.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:51.289 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:51.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:51.391 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:51.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:51.427 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:51.427 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:51.482 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:51.508 INFO:teuthology.orchestra.run.smithi151.stderr:no valid command found; 10 closest matches: 2024-09-02T10:30:51.508 INFO:teuthology.orchestra.run.smithi151.stderr:0 2024-09-02T10:30:51.508 INFO:teuthology.orchestra.run.smithi151.stderr:1 2024-09-02T10:30:51.508 INFO:teuthology.orchestra.run.smithi151.stderr:2 2024-09-02T10:30:51.508 INFO:teuthology.orchestra.run.smithi151.stderr:abort 2024-09-02T10:30:51.509 INFO:teuthology.orchestra.run.smithi151.stderr:assert 2024-09-02T10:30:51.509 INFO:teuthology.orchestra.run.smithi151.stderr:bench [] [] [] [] 2024-09-02T10:30:51.509 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs debug_inject_read_zeros 2024-09-02T10:30:51.509 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs files list 2024-09-02T10:30:51.509 INFO:teuthology.orchestra.run.smithi151.stderr:bluefs stats 2024-09-02T10:30:51.509 INFO:teuthology.orchestra.run.smithi151.stderr:bluestore allocator dump block 2024-09-02T10:30:51.509 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: invalid command 2024-09-02T10:30:51.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:51.555 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:51.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:51.556 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:51.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:51.687 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:51.688 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:51.777 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:30:51.789 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:51.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:51.819 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:51.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:51.920 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:51.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:51.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:52.041 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:30:52.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1888369379' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:52.049 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:52.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:52.075 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:52.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:52.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1888369379' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:52.109 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1888369379' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:30:52.176 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:52.200 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:52.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:52.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:52.341 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:52.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:52.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:52.465 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:52.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:52.547 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:24824 7.b 7:dcd0dd6d:::benchmark_data_smithi187_120712_object24823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e675)", 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:33.128771+0000", 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 379.41455391699998, 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21211126899999999, 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.548 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 24824 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128771+0000", 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128771+0000", 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128773+0000", 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.95e-06 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128832+0000", 2024-09-02T10:30:52.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8831e-05 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128844+0000", 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2425999999999999e-05 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128851+0000", 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.289e-06 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.128915+0000", 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4090000000000005e-05 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.129050+0000", 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013484 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.129182+0000", 2024-09-02T10:30:52.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000132194 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.130289+0000", 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011068479999999999 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.130725+0000", 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00043614899999999999 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.340817+0000", 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.210091533 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.340853+0000", 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6480999999999999e-05 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:33.340882+0000", 2024-09-02T10:30:52.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8637999999999999e-05 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:45635 7.b 7:d320ff39:::benchmark_data_smithi187_120712_object45634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.398280+0000", 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 359.145044955, 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24220898799999999, 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.555 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45635 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398280+0000", 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398280+0000", 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398282+0000", 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8309999999999999e-06 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398330+0000", 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8127000000000002e-05 2024-09-02T10:30:52.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398345+0000", 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5690999999999999e-05 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398352+0000", 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8079999999999999e-06 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.558 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398404+0000", 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1823000000000002e-05 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398508+0000", 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103457 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.398593+0000", 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5700999999999998e-05 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.399971+0000", 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013773360000000001 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.400538+0000", 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056756000000000005 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.640408+0000", 2024-09-02T10:30:52.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23987022799999999 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.640452+0000", 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3565e-05 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.640489+0000", 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6860999999999998e-05 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:55530 7.4 7:2b921397:::benchmark_data_smithi187_120712_object55529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:07.789283+0000", 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 344.754041384, 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25236621799999998, 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:30:52.562 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:52.563 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55530 2024-09-02T10:30:52.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.563 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789283+0000", 2024-09-02T10:30:52.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789283+0000", 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789285+0000", 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.438e-06 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789326+0000", 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1672999999999999e-05 2024-09-02T10:30:52.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789337+0000", 2024-09-02T10:30:52.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0895e-05 2024-09-02T10:30:52.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789343+0000", 2024-09-02T10:30:52.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2569999999999998e-06 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789405+0000", 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2550000000000003e-05 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789568+0000", 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016324500000000001 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.566 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4,9", 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.789765+0000", 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000197097 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791887+0000", 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021217290000000001 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791969+0000", 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1676999999999997e-05 2024-09-02T10:30:52.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.792487+0000", 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00051796399999999999 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.041461+0000", 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24897453 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.041524+0000", 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2496999999999995e-05 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.041650+0000", 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125666 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:55531 7.b 7:ddf5f000:::benchmark_data_smithi187_120712_object55530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:07.791489+0000", 2024-09-02T10:30:52.569 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 344.751835755, 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22225244299999999, 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 55531 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791489+0000", 2024-09-02T10:30:52.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791489+0000", 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791490+0000", 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5290000000000001e-06 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791541+0000", 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.041e-05 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791555+0000", 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4256999999999999e-05 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791563+0000", 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.191e-06 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791624+0000", 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0288000000000003e-05 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791822+0000", 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000198381 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.791949+0000", 2024-09-02T10:30:52.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012674600000000001 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.793670+0000", 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0017215469999999999 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.794018+0000", 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00034720600000000002 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013559+0000", 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21954131399999999 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013656+0000", 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6859000000000002e-05 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013741+0000", 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5715000000000002e-05 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.575 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:55531 7.b e696/661 ondisk, result = 0)", 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:07.794053+0000", 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 344.74927188100003, 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21969546700000001, 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.794053+0000", 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.794053+0000", 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.794053+0000", 2024-09-02T10:30:52.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8800000000000003e-07 2024-09-02T10:30:52.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:52.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:07.794056+0000", 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1489999999999998e-06 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013396+0000", 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21933929699999999 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013416+0000", 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9927000000000001e-05 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013530+0000", 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011441499999999999 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013548+0000", 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8085999999999999e-05 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:08.013748+0000", 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020010500000000001 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.581 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:71617 7.b 7:d1c8b968:::benchmark_data_smithi187_120712_object71616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e711)", 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:27.802425+0000", 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 324.74089982599997, 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33678569200000003, 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 71617 2024-09-02T10:30:52.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802425+0000", 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802425+0000", 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802427+0000", 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.731e-06 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802470+0000", 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2948000000000003e-05 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802486+0000", 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6790999999999999e-05 2024-09-02T10:30:52.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802496+0000", 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.696e-06 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802583+0000", 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6898999999999996e-05 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802729+0000", 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000145656 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,11", 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.802878+0000", 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014961999999999999 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.805442+0000", 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002563723 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.805818+0000", 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037650999999999998 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.139128+0000", 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33330957999999999 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.139168+0000", 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0169000000000001e-05 2024-09-02T10:30:52.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.139211+0000", 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2369e-05 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.4 722/690 [PushReplyOp(7:2b12bdff:::benchmark_data_smithi187_120712_object64093:head)])", 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:00.285664+0000", 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 292.257660375, 2024-09-02T10:30:52.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22833324099999999, 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.285664+0000", 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.285664+0000", 2024-09-02T10:30:52.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.285665+0000", 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6299999999999999e-07 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.285668+0000", 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1389999999999999e-06 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.513679+0000", 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22801164900000001 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.513687+0000", 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4159999999999998e-06 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.513779+0000", 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2192000000000001e-05 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.513998+0000", 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021868199999999999 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.594 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:99748 7.4 7:2ea4c3b6:::benchmark_data_smithi187_120712_object15806:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e722)", 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:00.286187+0000", 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 292.25713748800001, 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23885245899999999, 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:52.595 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 99748 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286187+0000", 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286187+0000", 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286189+0000", 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.671e-06 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286195+0000", 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0340000000000002e-06 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286210+0000", 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5291999999999999e-05 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286218+0000", 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6979999999999998e-06 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286277+0000", 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9361000000000003e-05 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286406+0000", 2024-09-02T10:30:52.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012913900000000001 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4,9", 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.286536+0000", 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012913099999999999 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289876+0000", 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033401699999999999 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.290140+0000", 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000264567 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509411+0000", 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21927023300000001 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.524839+0000", 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.015428496 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.524915+0000", 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6193999999999996e-05 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.525040+0000", 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124473 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.601 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:99748 7.4 e722/690 ondisk, result = 0)", 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:00.289806+0000", 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 292.25351921599997, 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21961228499999999, 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289806+0000", 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289806+0000", 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289806+0000", 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.16e-07 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.289814+0000", 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9100000000000005e-06 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509136+0000", 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.219321604 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509140+0000", 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7579999999999998e-06 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509389+0000", 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024907500000000002 2024-09-02T10:30:52.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509395+0000", 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1079999999999998e-06 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:00.509418+0000", 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2914000000000001e-05 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.4 723/690 [PushReplyOp(7:2b550f09:::benchmark_data_smithi187_120712_object82889:head)])", 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:01.652741+0000", 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 290.89058337099999, 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20101590699999999, 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.606 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.652741+0000", 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.652741+0000", 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.652742+0000", 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2399999999999999e-07 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.607 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.652745+0000", 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3969999999999998e-06 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.853508+0000", 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20076253099999999 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.853515+0000", 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1760000000000004e-06 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.853556+0000", 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1325999999999997e-05 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:01.853757+0000", 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000201153 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.4 723/690 [PushReplyOp(7:2b8179cd:::benchmark_data_smithi187_120712_object66421:head)])", 2024-09-02T10:30:52.609 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:02.076155+0000", 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 290.467169313, 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25945981000000001, 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.076155+0000", 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.610 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.076155+0000", 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.076156+0000", 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4999999999999999e-07 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.076158+0000", 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2239999999999998e-06 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.611 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.335405+0000", 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25924723 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.335411+0000", 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.767e-06 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.335458+0000", 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6761999999999997e-05 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.335615+0000", 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000157677 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:101021 7.4 7:2274b89d:::benchmark_data_smithi187_120712_object17079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:02.078078+0000", 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 290.46524713000002, 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26144672400000002, 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.613 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 101021 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078078+0000", 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.614 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078078+0000", 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078079+0000", 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2750000000000001e-06 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078085+0000", 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9410000000000004e-06 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078097+0000", 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1695999999999999e-05 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078102+0000", 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.587e-06 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078151+0000", 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8785000000000001e-05 2024-09-02T10:30:52.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078281+0000", 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013052199999999999 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,4,9", 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.078403+0000", 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121857 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.081030+0000", 2024-09-02T10:30:52.617 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0026272330000000001 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.081176+0000", 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014553299999999999 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.081342+0000", 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016558600000000001 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.618 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.339408+0000", 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25806603 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.339458+0000", 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0525999999999997e-05 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:02.339524+0000", 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6153000000000002e-05 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:143463 7.d e749/661 7:bdddf662:::benchmark_data_smithi187_120712_object59521:head v 749'9005, mlcod=749'9001)", 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:58.379297+0000", 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 234.164027629, 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20072000000000001, 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.379297+0000", 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.379297+0000", 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.379299+0000", 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5960000000000001e-06 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.379308+0000", 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2949999999999999e-06 2024-09-02T10:30:52.621 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.574020+0000", 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19471186900000001 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.574026+0000", 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1369999999999999e-06 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.574224+0000", 2024-09-02T10:30:52.622 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00019798099999999999 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.574293+0000", 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8864999999999999e-05 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.579973+0000", 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005679829 2024-09-02T10:30:52.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.624 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.624 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:58.580017+0000", 2024-09-02T10:30:52.624 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4428000000000002e-05 2024-09-02T10:30:52.624 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.624 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.624 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.624 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.626 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:144568 7.4 e749/741 7:21ef4de9:::benchmark_data_smithi187_120712_object60626:head v 749'8996, mlcod=749'8994)", 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:00.270570+0000", 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 232.27275469, 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.22573947899999999, 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.270570+0000", 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.270570+0000", 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.270570+0000", 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.05e-07 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.628 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.270573+0000", 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7020000000000001e-06 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.493109+0000", 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.222536013 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.493118+0000", 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5909999999999996e-06 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.493188+0000", 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0698000000000006e-05 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.493215+0000", 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6727999999999999e-05 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.496252+0000", 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030365890000000001 2024-09-02T10:30:52.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:00.496310+0000", 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7952999999999999e-05 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:6397 8.1 8:8dc84f7a:::benchmark_data_smithi187_136059_object6396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:29:02.701046+0000", 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 109.842278564, 2024-09-02T10:30:52.631 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.211004938, 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6397 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.701046+0000", 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.701046+0000", 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.701047+0000", 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.23e-07 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.701103+0000", 2024-09-02T10:30:52.633 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5967000000000001e-05 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.701113+0000", 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9550000000000004e-06 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.701118+0000", 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0289999999999996e-06 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.634 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.701152+0000", 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3680000000000003e-05 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.701230+0000", 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8499999999999997e-05 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.701307+0000", 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6588999999999995e-05 2024-09-02T10:30:52.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.701746+0000", 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00043929100000000002 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.739119+0000", 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.037373451000000002 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.636 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.911944+0000", 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.172824795 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.911979+0000", 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5336000000000002e-05 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.912051+0000", 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1622e-05 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.637 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:6397 8.1 8:8dc84f7a:::benchmark_data_smithi187_136059_object6396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e800)", 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:29:02.714326+0000", 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 109.82899919099999, 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.197662427, 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for ondisk", 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 6397 2024-09-02T10:30:52.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.714326+0000", 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.714326+0000", 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.714326+0000", 2024-09-02T10:30:52.639 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2099999999999996e-07 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.714443+0000", 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117236 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.714452+0000", 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8859999999999993e-06 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.640 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.714459+0000", 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2870000000000001e-06 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.714504+0000", 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5355999999999997e-05 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for ondisk", 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.714546+0000", 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1808999999999999e-05 2024-09-02T10:30:52.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:02.911988+0000", 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19744213199999999 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:7043 8.1 8:8bbd3211:::benchmark_data_smithi187_136059_object7042:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e800)", 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:29:03.220324+0000", 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 109.323000256, 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20844415999999999, 2024-09-02T10:30:52.642 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.643 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.643 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.643 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.643 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:30:52.643 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:52.643 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 7043 2024-09-02T10:30:52.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.643 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.220324+0000", 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.220324+0000", 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.220326+0000", 2024-09-02T10:30:52.644 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1209999999999999e-06 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.220370+0000", 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3956999999999999e-05 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.220382+0000", 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2601e-05 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.645 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.220390+0000", 2024-09-02T10:30:52.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9680000000000006e-06 2024-09-02T10:30:52.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.646 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.646 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.220439+0000", 2024-09-02T10:30:52.646 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9138999999999998e-05 2024-09-02T10:30:52.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.220537+0000", 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7756000000000004e-05 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.220618+0000", 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0920000000000005e-05 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.221123+0000", 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00050539900000000002 2024-09-02T10:30:52.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.221527+0000", 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00040319100000000001 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.428680+0000", 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.207153318 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.648 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.428723+0000", 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3164000000000002e-05 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:03.428769+0000", 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5626e-05 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:48853 8.1 8:8398910f:::benchmark_data_smithi187_136059_object48852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:29:33.629914+0000", 2024-09-02T10:30:52.649 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 78.913410526999996, 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.220221581, 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 48853 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.629914+0000", 2024-09-02T10:30:52.650 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.629914+0000", 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.629915+0000", 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1850000000000001e-06 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.651 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.629980+0000", 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4649999999999999e-05 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.629992+0000", 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1936999999999999e-05 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.629997+0000", 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9130000000000002e-06 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.652 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.630046+0000", 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8640000000000002e-05 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.630181+0000", 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013553600000000001 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.630321+0000", 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014028099999999999 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.631842+0000", 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015207650000000001 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.632393+0000", 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00055123399999999995 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.850059+0000", 2024-09-02T10:30:52.654 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.217665629 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.850102+0000", 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3109999999999999e-05 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.850136+0000", 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3701000000000001e-05 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:48872 8.1 8:8d3bba92:::benchmark_data_smithi187_136059_object48871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:29:33.642217+0000", 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 78.901107490000001, 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.215408454, 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 48872 2024-09-02T10:30:52.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.642217+0000", 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.642217+0000", 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.657 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.642219+0000", 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1110000000000002e-06 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.642273+0000", 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3678999999999999e-05 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.642285+0000", 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1531e-05 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.658 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.642290+0000", 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6160000000000001e-06 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.642335+0000", 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5318999999999998e-05 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.642434+0000", 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8110999999999996e-05 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.642548+0000", 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011452599999999999 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.644209+0000", 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016612790000000001 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.644548+0000", 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000338115 2024-09-02T10:30:52.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.857557+0000", 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21300989400000001 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.857594+0000", 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6733999999999998e-05 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.857626+0000", 2024-09-02T10:30:52.661 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1538999999999997e-05 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:48894 8.1 8:86818920:::benchmark_data_smithi187_136059_object48893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:29:33.654751+0000", 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 78.888573781999995, 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.202994127, 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:30:52.662 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 48894 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.654751+0000", 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.654751+0000", 2024-09-02T10:30:52.663 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.654753+0000", 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.68e-06 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.654865+0000", 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000112333 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:30:52.664 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.654883+0000", 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8536e-05 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.654891+0000", 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2840000000000001e-06 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.654944+0000", 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3217000000000002e-05 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.665 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.655045+0000", 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101446 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.655129+0000", 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3549999999999998e-05 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.656748+0000", 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001619065 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.666 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.657203+0000", 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00045460500000000002 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.857669+0000", 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20046609600000001 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.857701+0000", 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2502000000000002e-05 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:30:52.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:30:52.668 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:30:52.668 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:29:33.857745+0000", 2024-09-02T10:30:52.668 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3813000000000002e-05 2024-09-02T10:30:52.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.668 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:30:52.668 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:30:52.668 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:30:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:52.669 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:52.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:52.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:52.801 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:52.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:52.802 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:52.903 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:52.933 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:52.933 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:53.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:53.050 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: pgmap v1914: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 946 op/s 2024-09-02T10:30:53.050 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:30:53.051 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:30:53.051 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1888369379' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:53.051 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: osdmap e839: 12 total, 12 up, 12 in 2024-09-02T10:30:53.051 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: 8.7 scrub starts 2024-09-02T10:30:53.051 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:30:53.051 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:30:53.051 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:30:53.052 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:30:53.052 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:52 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:30:53.066 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:53.067 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: pgmap v1914: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 946 op/s 2024-09-02T10:30:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:30:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:30:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1888369379' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: osdmap e839: 12 total, 12 up, 12 in 2024-09-02T10:30:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: 8.7 scrub starts 2024-09-02T10:30:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:30:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:30:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:30:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:30:53.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:52 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:30:53.168 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: pgmap v1914: 47 pgs: 2 active+clean+scrubbing, 45 active+clean; 5.4 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 946 op/s 2024-09-02T10:30:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:30:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:30:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1888369379' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:30:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: osdmap e839: 12 total, 12 up, 12 in 2024-09-02T10:30:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: 8.7 scrub starts 2024-09-02T10:30:53.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:30:53.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:30:53.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:30:53.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:30:53.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:52 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:30:53.201 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:53.202 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:53.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:53.327 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:53.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:53.429 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:53.453 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:53.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:53.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:53.578 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:53.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:53.680 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:53.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:53.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:53.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:53.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:53.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:53.952 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:53.978 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:53.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:53.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:54.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:54.104 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:54.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:54.205 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:54.227 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:54.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:54.330 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:54.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:30:54.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:54.359 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:54.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:54.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:54.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:54.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:54.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:54.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:54.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:54.746 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:54.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:54.747 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:54.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:54.880 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:54.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:30:54.967 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-02T10:30:54.973 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:54.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:55.010 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:55.011 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:55.014 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:54 smithi162 ceph-mon[20279]: pgmap v1916: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 903 op/s 2024-09-02T10:30:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:54 smithi162 ceph-mon[20279]: 8.7 scrub ok 2024-09-02T10:30:55.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:54 smithi151 ceph-mon[16639]: pgmap v1916: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 903 op/s 2024-09-02T10:30:55.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:54 smithi151 ceph-mon[16639]: 8.7 scrub ok 2024-09-02T10:30:55.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:55.134 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:55.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:55.236 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:54 smithi187 ceph-mon[20072]: pgmap v1916: 47 pgs: 47 active+clean; 5.2 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 903 op/s 2024-09-02T10:30:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:54 smithi187 ceph-mon[20072]: 8.7 scrub ok 2024-09-02T10:30:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:55.262 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:55.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:55.364 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:55.386 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:55.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:55.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:55.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:30:55.488 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:55.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:55.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:55.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:55.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:55.641 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:55.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:55.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:55.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:55.774 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:55.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:55.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:55.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:55.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:30:55.883 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:30:55.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:30:55.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:30:55.899 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:30:55.899 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:30:55.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:55.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:55.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:55.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:30:55.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:30:55.974 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:30:55.974 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:30:55.992 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:55.992 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:56.001 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:30:56.001 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:30:56.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:56.028 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:30:56.029 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:30:56.037 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:56.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:56.039 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:56.061 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:30:56.062 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:30:56.089 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:30:56.089 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:30:56.121 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:30:56.121 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:30:56.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:56.148 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:30:56.149 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:30:56.169 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:56.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:56.170 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:56.180 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:30:56.180 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:30:56.271 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:30:56.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:56.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:56.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:30:56.401 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:56.431 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:56.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:56.432 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:56.533 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:56.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:56.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:56.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:56.666 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:56.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:30:56.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:56.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:56.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:56.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:56.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:56 smithi151 ceph-mon[16639]: pgmap v1917: 47 pgs: 47 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s 2024-09-02T10:30:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3212081245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:56 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:30:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:56 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:30:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:56 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:56 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:56.943 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:56.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:56.972 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:57.073 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:56 smithi162 ceph-mon[20279]: pgmap v1917: 47 pgs: 47 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s 2024-09-02T10:30:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3212081245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:56 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:30:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:56 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:30:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:56 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:57.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:56 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:57.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:57.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:57.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:57.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:57.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:56 smithi187 ceph-mon[20072]: pgmap v1917: 47 pgs: 47 active+clean; 5.1 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 891 op/s 2024-09-02T10:30:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3212081245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:30:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:56 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:30:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:56 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:30:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:56 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:56 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:57.332 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:57.354 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:57.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:57.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:57.456 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:57.469 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:57.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:57.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:57.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:57.609 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:57.610 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:57.711 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:30:57.737 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:57.738 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:30:57.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:57.864 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:57.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:57.965 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:57.989 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:57.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:58.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:57 smithi151 ceph-mon[16639]: osdmap e840: 12 total, 12 up, 12 in 2024-09-02T10:30:58.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:57 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:30:58.051 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:57 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:30:58.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:58.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:57 smithi162 ceph-mon[20279]: osdmap e840: 12 total, 12 up, 12 in 2024-09-02T10:30:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:57 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:30:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:57 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:30:58.110 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:30:58.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:58.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:58.120 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:58.221 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:58.243 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:58.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:57 smithi187 ceph-mon[20072]: osdmap e840: 12 total, 12 up, 12 in 2024-09-02T10:30:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:57 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:30:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:57 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:30:58.345 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:30:58.371 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:58.372 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:30:58.473 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:58.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:58.496 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:58.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:58.597 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:30:58.619 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:58.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:30:58.649 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:30:58.649 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:30:58.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:30:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:58.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:58.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:30:58.802 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:30:58.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:30:58.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:58.881 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:58.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:30:58.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:58 smithi151 ceph-mon[16639]: pgmap v1919: 47 pgs: 47 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s 2024-09-02T10:30:58.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:58 smithi151 ceph-mon[16639]: osdmap e841: 12 total, 12 up, 12 in 2024-09-02T10:30:58.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/14949199' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:30:58.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:58 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:30:58.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:59.004 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:59.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:59.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:58 smithi162 ceph-mon[20279]: pgmap v1919: 47 pgs: 47 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s 2024-09-02T10:30:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:58 smithi162 ceph-mon[20279]: osdmap e841: 12 total, 12 up, 12 in 2024-09-02T10:30:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/14949199' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:30:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:58 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:30:59.106 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:30:59.139 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:59.140 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:30:59.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:58 smithi187 ceph-mon[20072]: pgmap v1919: 47 pgs: 47 active+clean; 5.0 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 905 op/s 2024-09-02T10:30:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:58 smithi187 ceph-mon[20072]: osdmap e841: 12 total, 12 up, 12 in 2024-09-02T10:30:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/14949199' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:30:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:58 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:30:59.263 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:59.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:59.366 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:30:59.388 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:59.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:30:59.455 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:30:59.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:30:59.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:59.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:59.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:30:59.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:30:59.651 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:30:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:59.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:30:59.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:30:59.752 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:30:59.779 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:30:59.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:59.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:30:59.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:30:59.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:30:59.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:30:59.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:30:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:59 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:30:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:59 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:30:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:59 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:30:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:59 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:30:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:30:59 smithi151 ceph-mon[16639]: osdmap e842: 12 total, 12 up, 12 in 2024-09-02T10:30:59.968 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:30:59.968 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:30:59.968 INFO:tasks.thrashosds.thrasher:Reweighting osd 11 to 0.21299633150132005 2024-09-02T10:30:59.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 0.21299633150132005 2024-09-02T10:31:00.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:00.035 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:00.035 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:59 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:31:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:59 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:31:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:59 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:59 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:31:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:30:59 smithi162 ceph-mon[20279]: osdmap e842: 12 total, 12 up, 12 in 2024-09-02T10:31:00.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:00.158 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:00.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:00.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:59 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:31:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:59 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:31:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:59 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:59 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:31:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:30:59 smithi187 ceph-mon[20072]: osdmap e842: 12 total, 12 up, 12 in 2024-09-02T10:31:00.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:00.264 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:00.269 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:00.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:00.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:00.287 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:00.326 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:00.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:00.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:00.413 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:00.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:00.515 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:00.539 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:00.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:00.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:00.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:00.664 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:00.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:00.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:00.767 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:00.792 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:00.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:00.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:00.917 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:00.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:00.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:00 smithi151 ceph-mon[16639]: pgmap v1922: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 934 op/s 2024-09-02T10:31:01.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:01.044 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:01.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:00 smithi162 ceph-mon[20279]: pgmap v1922: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 934 op/s 2024-09-02T10:31:01.146 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:01.152 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:31:01.152 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:31:01.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:01.176 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:01.180 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:31:01.180 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:31:01.214 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:31:01.215 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:31:01.255 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:31:01.256 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:31:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:00 smithi187 ceph-mon[20072]: pgmap v1922: 47 pgs: 47 active+clean; 4.9 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 934 op/s 2024-09-02T10:31:01.277 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:01.291 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:31:01.291 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:31:01.301 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:01.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:01.327 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:31:01.327 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:31:01.361 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:31:01.361 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:31:01.390 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:31:01.390 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:31:01.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:01.424 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:31:01.424 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:31:01.425 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:01.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:01.462 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:31:01.462 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:31:01.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:31:01.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:31:01.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:01.531 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:31:01.532 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:31:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:01.553 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:01.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:01.655 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:01.685 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:01.685 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:01.787 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:01.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:01.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:01.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:01.815 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 0.212996 (3686) 2024-09-02T10:31:01.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:01.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:01.937 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:01.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:02.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:02.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1984078172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:02.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/366717891' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 0.21299633150132005} : dispatch 2024-09-02T10:31:02.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:01 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 0.21299633150132005} : dispatch 2024-09-02T10:31:02.186 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1984078172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:02.186 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/366717891' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 0.21299633150132005} : dispatch 2024-09-02T10:31:02.187 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:01 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 0.21299633150132005} : dispatch 2024-09-02T10:31:02.195 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:31:02.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:02.208 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:02.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1984078172' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/366717891' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 0.21299633150132005} : dispatch 2024-09-02T10:31:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:01 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 0.21299633150132005} : dispatch 2024-09-02T10:31:02.309 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:02.330 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:02.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:02.432 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:02.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:02.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:02.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:02.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:02.579 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:02.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:02.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:02.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:02.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:02.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:02.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:02.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:02.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:02 smithi151 ceph-mon[16639]: pgmap v1923: 47 pgs: 47 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s 2024-09-02T10:31:02.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:02 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 0.21299633150132005}]': finished 2024-09-02T10:31:02.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:02 smithi151 ceph-mon[16639]: osdmap e843: 12 total, 12 up, 12 in 2024-09-02T10:31:02.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:02 smithi151 ceph-mon[16639]: osdmap e844: 12 total, 12 up, 12 in 2024-09-02T10:31:02.932 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:02.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:02.955 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:03.056 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:03.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:03.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:02 smithi162 ceph-mon[20279]: pgmap v1923: 47 pgs: 47 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s 2024-09-02T10:31:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:02 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 0.21299633150132005}]': finished 2024-09-02T10:31:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:02 smithi162 ceph-mon[20279]: osdmap e843: 12 total, 12 up, 12 in 2024-09-02T10:31:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:02 smithi162 ceph-mon[20279]: osdmap e844: 12 total, 12 up, 12 in 2024-09-02T10:31:03.133 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:31:03.133 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:31:03.133 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:31:03.133 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:31:03.133 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.133 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.f e628,586 6:f0620152:::benchmark_data_smithi187_102448_object15839:head,577'7198)", 2024-09-02T10:31:03.133 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:31:03.133 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 496.79719913000002, 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69156394899999996, 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:31:03.134 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094467+0000", 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2600000000000002e-07 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.094472+0000", 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1230000000000002e-06 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.135 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785890+0000", 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69141827 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785894+0000", 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1520000000000002e-06 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785977+0000", 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3077000000000004e-05 2024-09-02T10:31:03.136 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.785987+0000", 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8379999999999997e-06 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:46.786031+0000", 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3763000000000001e-05 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.137 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.f e629,586 6:fb6fcd13:::benchmark_data_smithi187_102448_object18483:head,578'7347)", 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:47.434406+0000", 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 495.45725961400001, 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67700380800000004, 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434406+0000", 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.138 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434406+0000", 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434407+0000", 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0100000000000004e-07 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:47.434411+0000", 2024-09-02T10:31:03.139 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6449999999999998e-06 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111277+0000", 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67686604399999994 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111286+0000", 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0529999999999996e-06 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.140 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111349+0000", 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3620000000000004e-05 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111359+0000", 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9250000000000007e-06 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:48.111410+0000", 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0819999999999998e-05 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.141 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e635,586 6:033be843:::benchmark_data_smithi187_102448_object1499:head,573'6439)", 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:56.345932+0000", 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 486.54573359400001, 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39558264700000001, 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.142 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345932+0000", 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345932+0000", 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345933+0000", 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8000000000000005e-07 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.143 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.345938+0000", 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0640000000000001e-06 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741308+0000", 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39536988899999997 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741316+0000", 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.9009999999999994e-06 2024-09-02T10:31:03.144 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741421+0000", 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000105486 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741436+0000", 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4617000000000001e-05 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:56.741515+0000", 2024-09-02T10:31:03.145 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8910000000000002e-05 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e643,586 6:0a7160e0:::benchmark_data_smithi187_102448_object18383:head,578'7477)", 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:07.703494+0000", 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 475.18817151899998, 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44123607500000001, 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.146 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.148 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703494+0000", 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703494+0000", 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703496+0000", 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4529999999999999e-06 2024-09-02T10:31:03.149 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:07.703502+0000", 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2709999999999998e-06 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144429+0000", 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.44092676800000002 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144444+0000", 2024-09-02T10:31:03.150 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5146e-05 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144589+0000", 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014536200000000001 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144604+0000", 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4817000000000001e-05 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.151 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:08.144730+0000", 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126258 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e648,586 6:065bcc16:::benchmark_data_smithi187_102448_object45489:head,591'9175)", 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:29.598657+0000", 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 453.29300867000001, 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.67037543899999996, 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.152 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598657+0000", 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598657+0000", 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.153 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598658+0000", 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7800000000000001e-07 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:29.598663+0000", 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1209999999999998e-06 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268825+0000", 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.670161754 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.154 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268834+0000", 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3449999999999995e-06 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268939+0000", 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010511899999999999 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.268949+0000", 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7599999999999997e-06 2024-09-02T10:31:03.155 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:30.269033+0000", 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3561999999999997e-05 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e648,586 6:0cd50fba:::benchmark_data_smithi187_102448_object46855:head,592'9264)", 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:31.097253+0000", 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 451.794412685, 2024-09-02T10:31:03.156 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.83171925999999996, 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097253+0000", 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097253+0000", 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.157 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097254+0000", 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.61e-07 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.097259+0000", 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9520000000000002e-06 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928794+0000", 2024-09-02T10:31:03.158 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.83153501299999999 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928806+0000", 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1847e-05 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928905+0000", 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9277999999999994e-05 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.159 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928913+0000", 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7039999999999994e-06 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:31.928972+0000", 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9805000000000001e-05 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(6.0 e653,586 6:01d685fb:::benchmark_data_smithi187_102448_object80695:head,616'11374)", 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:48.578010+0000", 2024-09-02T10:31:03.160 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 434.31365566599999, 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54628079200000002, 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578010+0000", 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578010+0000", 2024-09-02T10:31:03.161 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578011+0000", 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9400000000000005e-07 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:48.578016+0000", 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9069999999999998e-06 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.162 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124136+0000", 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54612002800000004 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124160+0000", 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4731000000000001e-05 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124241+0000", 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0457999999999994e-05 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.163 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124249+0000", 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3620000000000006e-06 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:49.124291+0000", 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1712000000000003e-05 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.164 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:45621 7.8 7:1a107a8d:::benchmark_data_smithi187_120712_object45620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.307311+0000", 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 369.58435500399997, 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36183037800000001, 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45621 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.165 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307311+0000", 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307311+0000", 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307315+0000", 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.566e-06 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.166 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307366+0000", 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0365999999999999e-05 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307379+0000", 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3134e-05 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307386+0000", 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.362e-06 2024-09-02T10:31:03.167 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307441+0000", 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4568999999999999e-05 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307551+0000", 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109806 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:31:03.168 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.307723+0000", 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00017262399999999999 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.309378+0000", 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016545729999999999 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.309672+0000", 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00029370500000000001 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.169 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669068+0000", 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35939658099999999 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669108+0000", 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9860999999999997e-05 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669141+0000", 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3231000000000001e-05 2024-09-02T10:31:03.170 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:45621 7.8 e689/661 ondisk, result = 0)", 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.383905+0000", 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 369.50776046499999, 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28524120600000002, 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.171 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383905+0000", 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383905+0000", 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383907+0000", 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0470000000000001e-06 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.172 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.383920+0000", 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2989e-05 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669023+0000", 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28510220400000003 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669028+0000", 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4759999999999997e-06 2024-09-02T10:31:03.173 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669062+0000", 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3779999999999998e-05 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669065+0000", 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4089999999999999e-06 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669147+0000", 2024-09-02T10:31:03.174 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1301e-05 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:45650 7.8 7:1c05cd80:::benchmark_data_smithi187_120712_object45649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.666730+0000", 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 369.22493557000001, 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.325110548, 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:03.175 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 45650 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666730+0000", 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666730+0000", 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.176 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666732+0000", 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.835e-06 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666773+0000", 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1121999999999997e-05 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666787+0000", 2024-09-02T10:31:03.177 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3908e-05 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666793+0000", 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4589999999999997e-06 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666848+0000", 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5037999999999997e-05 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.178 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.666935+0000", 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7781000000000005e-05 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.667024+0000", 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8331000000000005e-05 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.668407+0000", 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013827869999999999 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.179 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.668945+0000", 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053888899999999997 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991726+0000", 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32278023700000003 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991790+0000", 2024-09-02T10:31:03.180 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4288000000000005e-05 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991841+0000", 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0872999999999999e-05 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:45650 7.8 e689/661 ondisk, result = 0)", 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:24:53.669003+0000", 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 369.22266278900003, 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32284557699999999, 2024-09-02T10:31:03.181 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669003+0000", 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669003+0000", 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.182 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669004+0000", 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6000000000000003e-07 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.669007+0000", 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4340000000000001e-06 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991617+0000", 2024-09-02T10:31:03.183 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.322609692 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991628+0000", 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0998e-05 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991700+0000", 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2343999999999997e-05 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.184 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991718+0000", 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8142e-05 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:24:53.991849+0000", 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000130207 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:71627 7.8 7:158e78ca:::benchmark_data_smithi187_120712_object71626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e711)", 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:25:27.818033+0000", 2024-09-02T10:31:03.185 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 335.07363324900001, 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.48315623299999999, 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 71627 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.186 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818033+0000", 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818033+0000", 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.187 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818035+0000", 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2529999999999999e-06 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818066+0000", 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1619000000000003e-05 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818083+0000", 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6721000000000001e-05 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.188 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818093+0000", 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3680000000000008e-06 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818169+0000", 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6487000000000001e-05 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818309+0000", 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014026699999999999 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.189 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.818454+0000", 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000144964 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.821157+0000", 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027031830000000001 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:27.821177+0000", 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0055e-05 2024-09-02T10:31:03.190 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.301111+0000", 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47993339899999998 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.301149+0000", 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8605000000000001e-05 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.191 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:25:28.301189+0000", 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9311999999999999e-05 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:136914 7.3 e743/741 7:c30a193a:::benchmark_data_smithi187_120712_object52972:head v 743'8327, mlcod=742'8324)", 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.758995+0000", 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 259.13267049900003, 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33365049099999999, 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.192 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758995+0000", 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758995+0000", 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.193 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758996+0000", 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1100000000000002e-07 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.759001+0000", 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9309999999999999e-06 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090734+0000", 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33173325799999998 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.194 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090757+0000", 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3033000000000001e-05 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090882+0000", 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012556700000000001 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090902+0000", 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9519999999999999e-05 2024-09-02T10:31:03.195 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092615+0000", 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001713387 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092646+0000", 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0484000000000002e-05 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.196 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:145939 7.12 7:4b7e5994:::benchmark_data_smithi187_120712_object61997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.921348+0000", 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 239.97031812899999, 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38239470399999997, 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 145939 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.197 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921348+0000", 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921348+0000", 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921349+0000", 2024-09-02T10:31:03.198 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5349999999999999e-06 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921355+0000", 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3940000000000004e-06 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921368+0000", 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3499000000000001e-05 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.199 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921375+0000", 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4729999999999997e-06 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.928654+0000", 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072797640000000002 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.928971+0000", 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000317127 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.200 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.929306+0000", 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033418199999999998 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.939106+0000", 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0098002920000000004 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.951104+0000", 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011997733 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.201 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302905+0000", 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.351801429 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.303056+0000", 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015083300000000001 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.303742+0000", 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068644300000000001 2024-09-02T10:31:03.202 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:145939 7.12 e749/744 ondisk, result = 0)", 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.932006+0000", 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 239.95965989300001, 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37174302799999998, 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.203 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.204 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932006+0000", 2024-09-02T10:31:03.204 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.204 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.204 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.204 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932006+0000", 2024-09-02T10:31:03.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.205 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.205 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932007+0000", 2024-09-02T10:31:03.205 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1039999999999999e-06 2024-09-02T10:31:03.205 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.205 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932012+0000", 2024-09-02T10:31:03.206 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8289999999999998e-06 2024-09-02T10:31:03.206 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.206 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.206 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.206 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302586+0000", 2024-09-02T10:31:03.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37057435599999999 2024-09-02T10:31:03.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.207 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302593+0000", 2024-09-02T10:31:03.207 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4300000000000003e-06 2024-09-02T10:31:03.207 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.207 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.207 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302888+0000", 2024-09-02T10:31:03.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000295378 2024-09-02T10:31:03.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.208 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.208 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.208 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302898+0000", 2024-09-02T10:31:03.208 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6469999999999993e-06 2024-09-02T10:31:03.208 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.209 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.209 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.209 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.303749+0000", 2024-09-02T10:31:03.209 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000851284 2024-09-02T10:31:03.209 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.209 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.209 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.12 749/744 [PushReplyOp(7:481a1f6c:::benchmark_data_smithi187_120712_object69540:head)])", 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.953782+0000", 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 239.93788381799999, 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.375520674, 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953782+0000", 2024-09-02T10:31:03.210 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.211 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953782+0000", 2024-09-02T10:31:03.211 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.211 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.211 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.211 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953783+0000", 2024-09-02T10:31:03.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6099999999999999e-07 2024-09-02T10:31:03.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.212 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.212 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.212 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953789+0000", 2024-09-02T10:31:03.212 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6279999999999999e-06 2024-09-02T10:31:03.212 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.328024+0000", 2024-09-02T10:31:03.213 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37423416799999998 2024-09-02T10:31:03.213 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.213 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.213 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.213 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.328033+0000", 2024-09-02T10:31:03.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1700000000000003e-06 2024-09-02T10:31:03.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.214 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.328279+0000", 2024-09-02T10:31:03.214 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000246519 2024-09-02T10:31:03.214 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.214 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.214 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.215 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.329303+0000", 2024-09-02T10:31:03.215 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001023328 2024-09-02T10:31:03.215 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.215 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.215 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.215 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.215 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.215 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:145959 7.12 7:4e913d9f:::benchmark_data_smithi187_120712_object62017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:31:03.216 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.957341+0000", 2024-09-02T10:31:03.216 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 239.93432509300001, 2024-09-02T10:31:03.216 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39245286600000001, 2024-09-02T10:31:03.216 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.216 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.216 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:03.216 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:03.216 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:03.217 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:03.217 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 145959 2024-09-02T10:31:03.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.217 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.217 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.217 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.217 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957341+0000", 2024-09-02T10:31:03.217 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.217 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957341+0000", 2024-09-02T10:31:03.218 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.218 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.218 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.218 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.218 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957342+0000", 2024-09-02T10:31:03.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5549999999999999e-06 2024-09-02T10:31:03.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.219 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.219 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957350+0000", 2024-09-02T10:31:03.219 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5889999999999997e-06 2024-09-02T10:31:03.219 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.219 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957361+0000", 2024-09-02T10:31:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1075e-05 2024-09-02T10:31:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.220 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957369+0000", 2024-09-02T10:31:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8010000000000004e-06 2024-09-02T10:31:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957569+0000", 2024-09-02T10:31:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020020500000000001 2024-09-02T10:31:03.221 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957715+0000", 2024-09-02T10:31:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014592899999999999 2024-09-02T10:31:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.222 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:31:03.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957956+0000", 2024-09-02T10:31:03.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024078399999999999 2024-09-02T10:31:03.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.223 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.223 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.223 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.972919+0000", 2024-09-02T10:31:03.223 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014963476999999999 2024-09-02T10:31:03.223 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:03.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.974010+0000", 2024-09-02T10:31:03.224 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010906589999999999 2024-09-02T10:31:03.224 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.224 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.224 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.224 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349010+0000", 2024-09-02T10:31:03.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37499981799999998 2024-09-02T10:31:03.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.225 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:03.225 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349077+0000", 2024-09-02T10:31:03.225 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599000000000001e-05 2024-09-02T10:31:03.225 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.225 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.226 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.226 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349794+0000", 2024-09-02T10:31:03.226 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000716375 2024-09-02T10:31:03.226 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.226 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.226 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.226 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.226 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.226 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:145959 7.12 e749/744 ondisk, result = 0)", 2024-09-02T10:31:03.227 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.961683+0000", 2024-09-02T10:31:03.227 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 239.92998299999999, 2024-09-02T10:31:03.227 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38811612099999998, 2024-09-02T10:31:03.227 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.227 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.227 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:03.227 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.227 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961683+0000", 2024-09-02T10:31:03.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.228 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.228 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.228 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.228 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961683+0000", 2024-09-02T10:31:03.228 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961684+0000", 2024-09-02T10:31:03.229 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5600000000000005e-07 2024-09-02T10:31:03.229 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.229 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.229 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.229 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961688+0000", 2024-09-02T10:31:03.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.831e-06 2024-09-02T10:31:03.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.230 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.348905+0000", 2024-09-02T10:31:03.230 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38721641600000001 2024-09-02T10:31:03.230 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.230 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.230 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.348912+0000", 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2559999999999996e-06 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.348993+0000", 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1050999999999994e-05 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349002+0000", 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3309999999999993e-06 2024-09-02T10:31:03.231 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.232 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.232 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.232 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349799+0000", 2024-09-02T10:31:03.232 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00079648000000000002 2024-09-02T10:31:03.232 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.232 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.232 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.232 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.233 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.233 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.18 749/744 [PushReplyOp(7:18b04759:::benchmark_data_smithi187_120712_object77911:head)])", 2024-09-02T10:31:03.233 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:06.705518+0000", 2024-09-02T10:31:03.233 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 236.18614786699999, 2024-09-02T10:31:03.233 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32887954400000002, 2024-09-02T10:31:03.233 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.233 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.234 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:31:03.234 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.234 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.234 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.234 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:06.705518+0000", 2024-09-02T10:31:03.234 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.235 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.235 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:06.705518+0000", 2024-09-02T10:31:03.235 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.235 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.235 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:06.705519+0000", 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1900000000000002e-07 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:06.705528+0000", 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2119999999999992e-06 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:07.031820+0000", 2024-09-02T10:31:03.236 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.326292305 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:07.031827+0000", 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0929999999999997e-06 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:07.033462+0000", 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016343569999999999 2024-09-02T10:31:03.237 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.238 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.238 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.238 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:07.034397+0000", 2024-09-02T10:31:03.238 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00093585800000000002 2024-09-02T10:31:03.238 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.238 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.238 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:152368 7.8 7:17e54ee4:::benchmark_data_smithi187_120712_object68426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:14.166120+0000", 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 228.72554574, 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28226089599999998, 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:03.239 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 152368 2024-09-02T10:31:03.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.240 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:03.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.240 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:03.240 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166120+0000", 2024-09-02T10:31:03.240 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.240 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.240 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:03.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166120+0000", 2024-09-02T10:31:03.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:03.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.241 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:03.241 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166122+0000", 2024-09-02T10:31:03.241 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.761e-06 2024-09-02T10:31:03.241 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.241 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166128+0000", 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7379999999999999e-06 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166140+0000", 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2480000000000001e-05 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166147+0000", 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7920000000000004e-06 2024-09-02T10:31:03.242 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166231+0000", 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4176000000000002e-05 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166376+0000", 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014515799999999999 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.166519+0000", 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014257300000000001 2024-09-02T10:31:03.243 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.170193+0000", 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003673782 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.177004+0000", 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0068118199999999997 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.244 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.448215+0000", 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.27121096900000002 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.448323+0000", 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000107253 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:03.245 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:14.448381+0000", 2024-09-02T10:31:03.246 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8393999999999997e-05 2024-09-02T10:31:03.246 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.246 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.246 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.246 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:03.246 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:03.246 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:31:03.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:02 smithi187 ceph-mon[20072]: pgmap v1923: 47 pgs: 47 active+clean; 4.8 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 781 op/s 2024-09-02T10:31:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:02 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 0.21299633150132005}]': finished 2024-09-02T10:31:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:02 smithi187 ceph-mon[20072]: osdmap e843: 12 total, 12 up, 12 in 2024-09-02T10:31:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:02 smithi187 ceph-mon[20072]: osdmap e844: 12 total, 12 up, 12 in 2024-09-02T10:31:03.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:03.269 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:03.270 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:03.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:03.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:03.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:03.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:03.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:03.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:03.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:03.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:03.642 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:03.642 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:03.717 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:03.743 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:03.770 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:03.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:03.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:03.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:03.895 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:03.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:03.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:04.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:04.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:04.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:04.149 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:04.150 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:04.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:04.279 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:04.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:04.380 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:04.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:04.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:04.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:04.549 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:04.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:04.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:04.628 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:31:04.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:04.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:04.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:04.806 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:04.830 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:04.831 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:04.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:04 smithi151 ceph-mon[16639]: pgmap v1926: 47 pgs: 1 peering, 46 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s 2024-09-02T10:31:04.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:04 smithi151 ceph-mon[16639]: osdmap e845: 12 total, 12 up, 12 in 2024-09-02T10:31:04.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:04 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:31:04.932 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:04.956 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:04.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:04.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:05.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:05.081 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:05.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:05.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:04 smithi162 ceph-mon[20279]: pgmap v1926: 47 pgs: 1 peering, 46 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s 2024-09-02T10:31:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:04 smithi162 ceph-mon[20279]: osdmap e845: 12 total, 12 up, 12 in 2024-09-02T10:31:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:04 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:31:05.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:05.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:05.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:05.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:04 smithi187 ceph-mon[20072]: pgmap v1926: 47 pgs: 1 peering, 46 active+clean; 4.7 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 647 op/s 2024-09-02T10:31:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:04 smithi187 ceph-mon[20072]: osdmap e845: 12 total, 12 up, 12 in 2024-09-02T10:31:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:04 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:31:05.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:05.346 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:05.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:05.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:05.448 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:05.473 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:05.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:05.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:05.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:05.612 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:05.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:05.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:05.739 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:05.740 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:05.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:05.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:05.866 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:05.967 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:06.000 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:06.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:06.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:06.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:06.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:06.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:06.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:06.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:05 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 13158/228246 objects degraded (5.765%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:06.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:31:06.165 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:06.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:05 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 13158/228246 objects degraded (5.765%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:06.268 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:06.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:05 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 13158/228246 objects degraded (5.765%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:06.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:06.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:06.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:06.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:06.516 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:06.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:06.566 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:06.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:06.668 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:06.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:06.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:06.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:06.822 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:06.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:06.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:06.844 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:06.851 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:31:06.851 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:31:06.879 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:31:06.896 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:31:06.896 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:31:06.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:06.934 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:31:06.934 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:31:06.948 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:06.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:06.965 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:31:06.965 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:31:07.000 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:31:07.000 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:31:07.034 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:31:07.034 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:31:07.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:06 smithi151 ceph-mon[16639]: pgmap v1928: 47 pgs: 2 activating+undersized+degraded+remapped, 1 peering, 44 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 13158/228246 objects degraded (5.765%) 2024-09-02T10:31:07.034 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1192789141' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:31:07.035 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2184066715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:07.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:07.066 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:31:07.066 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:31:07.072 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:07.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:07.102 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:31:07.102 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:31:07.134 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:31:07.134 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:31:07.160 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:31:07.160 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:31:07.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:07.195 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:31:07.195 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:31:07.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:07.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:07.233 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:31:07.233 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:31:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:06 smithi187 ceph-mon[20072]: pgmap v1928: 47 pgs: 2 activating+undersized+degraded+remapped, 1 peering, 44 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 13158/228246 objects degraded (5.765%) 2024-09-02T10:31:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1192789141' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:31:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2184066715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:07.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:07.332 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:07.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:07.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:06 smithi162 ceph-mon[20279]: pgmap v1928: 47 pgs: 2 activating+undersized+degraded+remapped, 1 peering, 44 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 13158/228246 objects degraded (5.765%) 2024-09-02T10:31:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1192789141' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:31:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2184066715' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:07.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:07.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:07.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:07.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:07.591 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:07.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:07.592 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:07.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:07.719 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:07.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:07.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:07.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:07.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:07.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:07.964 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:31:07.964 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:31:07.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:07.974 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:07.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:08.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:07 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:08.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1192789141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:31:08.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:07 smithi151 ceph-mon[16639]: osdmap e846: 12 total, 12 up, 12 in 2024-09-02T10:31:08.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:08.097 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:08.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:08.200 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:08.233 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:08.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:07 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1192789141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:31:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:07 smithi187 ceph-mon[20072]: osdmap e846: 12 total, 12 up, 12 in 2024-09-02T10:31:08.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:07 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1192789141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:31:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:07 smithi162 ceph-mon[20279]: osdmap e846: 12 total, 12 up, 12 in 2024-09-02T10:31:08.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:08.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:08.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:08.493 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:08.494 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:08.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:08.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:08.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:08.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:08.721 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:08.746 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:08.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:08.849 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:08.875 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:08.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:08.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:08.977 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:09.007 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:09.008 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:09.109 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:09.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:09.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:08 smithi151 ceph-mon[16639]: pgmap v1930: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 1 peering, 42 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 13213/224571 objects degraded (5.884%); 9286/224571 objects misplaced (4.135%); 1.8 MiB/s, 28 objects/s recovering 2024-09-02T10:31:09.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:31:09.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:08 smithi187 ceph-mon[20072]: pgmap v1930: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 1 peering, 42 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 13213/224571 objects degraded (5.884%); 9286/224571 objects misplaced (4.135%); 1.8 MiB/s, 28 objects/s recovering 2024-09-02T10:31:09.268 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:09.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:09.269 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:08 smithi162 ceph-mon[20279]: pgmap v1930: 47 pgs: 2 active+recovering+undersized+degraded+remapped, 2 activating+undersized+degraded+remapped, 1 peering, 42 active+clean; 4.6 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 614 op/s; 13213/224571 objects degraded (5.884%); 9286/224571 objects misplaced (4.135%); 1.8 MiB/s, 28 objects/s recovering 2024-09-02T10:31:09.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:09.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:09.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:09.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:09.528 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:09.528 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:09.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:09.657 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:09.658 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:09.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:09.786 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:09.787 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:09.888 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:09.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:09.915 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:09.999 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:31:09.999 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:31:10.000 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-02T10:31:10.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:31:10.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:10.038 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:10.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:10.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:10.140 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:10.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:10.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:09 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:31:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:09 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:31:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:09 smithi151 ceph-mon[16639]: osdmap e847: 12 total, 12 up, 12 in 2024-09-02T10:31:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:09 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:31:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:09 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:31:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:09 smithi187 ceph-mon[20072]: osdmap e847: 12 total, 12 up, 12 in 2024-09-02T10:31:10.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:10.303 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:10.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:09 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:31:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:09 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:31:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:09 smithi162 ceph-mon[20279]: osdmap e847: 12 total, 12 up, 12 in 2024-09-02T10:31:10.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:10.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:10.437 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:10.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:10.541 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:10.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:31:10.567 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:10.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:10.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:10.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:10.694 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:10.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:10.797 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:10.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:10.824 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:10.924 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:10.948 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:10.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:10.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:11.050 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:11.078 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:11.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:11.180 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:11.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:10 smithi151 ceph-mon[16639]: pgmap v1932: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 225/220599 objects degraded (0.102%); 18257/220599 objects misplaced (8.276%); 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:31:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:10 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-02T10:31:11.208 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:11.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:10 smithi187 ceph-mon[20072]: pgmap v1932: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 225/220599 objects degraded (0.102%); 18257/220599 objects misplaced (8.276%); 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:31:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:10 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-02T10:31:11.310 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:11.333 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:11.334 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:10 smithi162 ceph-mon[20279]: pgmap v1932: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 609 op/s; 225/220599 objects degraded (0.102%); 18257/220599 objects misplaced (8.276%); 4.2 MiB/s, 67 objects/s recovering 2024-09-02T10:31:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:10 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-02T10:31:11.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:11.458 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:11.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:11.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:11.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:11.589 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:11.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:11.717 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:11.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:11.718 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:11.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:11.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:11.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:11.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:11.974 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:11.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:12.075 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:12.104 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:12.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:12.105 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:12.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:12.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:12.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:11 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 229/218829 objects degraded (0.105%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:12.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:12.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:11 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 229/218829 objects degraded (0.105%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:12.364 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:12.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:12.395 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:11 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 229/218829 objects degraded (0.105%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:12.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:12.482 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:12.488 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:12.491 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:12.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:12.593 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:12.623 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:12.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:12.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:12.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:12.751 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:12.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:12.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:12.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:12.880 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:12.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:12.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:13.008 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:13.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:13.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:13.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:13.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:13.135 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:13.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:12 smithi151 ceph-mon[16639]: pgmap v1933: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 229/218829 objects degraded (0.105%); 18064/218829 objects misplaced (8.255%); 4.8 MiB/s, 77 objects/s recovering 2024-09-02T10:31:13.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:12 smithi187 ceph-mon[20072]: pgmap v1933: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 229/218829 objects degraded (0.105%); 18064/218829 objects misplaced (8.255%); 4.8 MiB/s, 77 objects/s recovering 2024-09-02T10:31:13.261 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:13.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:13.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:12 smithi162 ceph-mon[20279]: pgmap v1933: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.5 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 549 op/s; 229/218829 objects degraded (0.105%); 18064/218829 objects misplaced (8.255%); 4.8 MiB/s, 77 objects/s recovering 2024-09-02T10:31:13.363 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:13.392 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:13.393 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:13.494 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:13.519 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:13.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:13.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:13.644 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:13.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:13.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:13.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:13.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:13.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:13.875 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:13.898 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:13.899 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:14.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:14.021 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:14.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:14.022 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:14.123 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:14.153 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:14.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:14.254 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:14.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:14.281 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:14.382 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:14.419 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:14.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:14.454 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:31:14.520 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:14.556 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:14.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:14.658 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:14.681 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:14.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:14.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:14.784 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:14.806 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:14.808 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:14.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:14.935 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:14.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:15.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:15.061 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:15.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:15.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:15.187 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:15.188 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:14 smithi187 ceph-mon[20072]: pgmap v1934: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 258/213192 objects degraded (0.121%); 17869/213192 objects misplaced (8.382%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:31:15.289 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:15.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:15.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:15.314 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:14 smithi162 ceph-mon[20279]: pgmap v1934: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 258/213192 objects degraded (0.121%); 17869/213192 objects misplaced (8.382%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:31:15.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:14 smithi151 ceph-mon[16639]: pgmap v1934: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 627 op/s; 258/213192 objects degraded (0.121%); 17869/213192 objects misplaced (8.382%); 5.9 MiB/s, 95 objects/s recovering 2024-09-02T10:31:15.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:15.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:15.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:15.542 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:15.568 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:15.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:15.569 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:15.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:15.695 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:15.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:15.797 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:15.821 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:15.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:15.922 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:15.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:15.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:16.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:16.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:16.091 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:16.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:16.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:16.220 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:16.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:16.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:16.347 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:16.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:16.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:16.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:16.477 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:16.478 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:16.479 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:31:16.479 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:31:16.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:16.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:16.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:16.709 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:16.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:16.734 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:16.835 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:16.859 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:16.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:16.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:16.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:16.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:16.987 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:17.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:17.114 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:17.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:17.216 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:17.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:17.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:17.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:16 smithi187 ceph-mon[20072]: pgmap v1935: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s; 417/209406 objects degraded (0.199%); 17455/209406 objects misplaced (8.335%); 6.1 MiB/s, 98 objects/s recovering 2024-09-02T10:31:17.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:16 smithi151 ceph-mon[16639]: pgmap v1935: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s; 417/209406 objects degraded (0.199%); 17455/209406 objects misplaced (8.335%); 6.1 MiB/s, 98 objects/s recovering 2024-09-02T10:31:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:16 smithi162 ceph-mon[20279]: pgmap v1935: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.3 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 584 op/s; 417/209406 objects degraded (0.199%); 17455/209406 objects misplaced (8.335%); 6.1 MiB/s, 98 objects/s recovering 2024-09-02T10:31:17.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:17.383 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:17.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:17.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:17.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:17.517 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:17.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:17.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:17.619 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:17.644 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:17.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:17.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:17.775 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:17.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:17.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:17.877 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:17.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:17.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:17.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:18.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:18.026 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:18.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:18.027 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:18.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:18.152 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:18.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:18.153 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:18.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:18 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 479/205794 objects degraded (0.233%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:18.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:18.286 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:18.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:18 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 479/205794 objects degraded (0.233%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:18.387 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:18.397 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:18.420 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:18.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:18.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:18.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:18 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 479/205794 objects degraded (0.233%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:18.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:18.546 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:18.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:18.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:18.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:18.684 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:18.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:18.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:18.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:18.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:18.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:18.940 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:18.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:19.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:19.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:19.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:19.147 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:19 smithi151 ceph-mon[16639]: pgmap v1936: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 479/205794 objects degraded (0.233%); 17056/205794 objects misplaced (8.288%); 7.4 MiB/s, 118 objects/s recovering 2024-09-02T10:31:19.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:19.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:19.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:19.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:19.323 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:31:19.324 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:31:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:19 smithi162 ceph-mon[20279]: pgmap v1936: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 479/205794 objects degraded (0.233%); 17056/205794 objects misplaced (8.288%); 7.4 MiB/s, 118 objects/s recovering 2024-09-02T10:31:19.346 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:19.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:19.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:19.370 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:31:19.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:31:19.403 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:31:19.403 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:31:19.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:19.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:31:19.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:31:19.492 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:31:19.492 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 847, 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:31:08.881065+0000", 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:20:51.089364+0000", 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5806080, 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "nodeep-scrub", 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "noscrub", 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:31:19.493 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 8, 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-09-02T10:31:19.494 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:31:19.495 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-09-02T10:31:19.496 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "264", 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "241", 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:31:19.497 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:31:19.498 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:31:19.499 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:31:19.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 1.940000057220459, 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 1.940000057220459, 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 1.940000057220459, 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 1.940000057220459, 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 8, 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_6", 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:28:50.275834+0000", 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:31:19.502 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:31:19.503 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "797", 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:31:19.504 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:31:19.505 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:31:19.506 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 3, 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 3, 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 3, 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 3, 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.99999988079071045, 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 1 2024-09-02T10:31:19.507 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.508 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 843, 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:31:19.509 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:31:19.510 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:31:19.511 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:31:19.512 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:31:19.513 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 844, 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.514 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.515 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.516 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.517 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:31:19.518 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.519 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 769, 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:31:19.520 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:31:19.521 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:31:19.522 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:31:19.523 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:31:19.524 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 792, 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.525 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.526 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.527 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:31:19.528 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:31:19.528 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:19 smithi187 ceph-mon[20072]: pgmap v1936: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.2 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 625 op/s; 479/205794 objects degraded (0.233%); 17056/205794 objects misplaced (8.288%); 7.4 MiB/s, 118 objects/s recovering 2024-09-02T10:31:19.529 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:31:19.529 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:31:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:31:19.529 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.530 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 741, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.532 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.533 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.534 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.535 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.536 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:31:19.537 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 792, 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.538 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.539 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.540 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:31:19.542 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.543 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 783, 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.544 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.545 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.546 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.547 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:19.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:19.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:31:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:31:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.549 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:31:19.550 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 844, 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.551 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.554 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:31:19.555 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.556 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 792, 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.558 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:31:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:31:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.560 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:31:19.561 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.562 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 792, 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:31:19.564 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:31:19.565 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:31:19.566 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.567 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:31:19.568 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 844, 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:31:19.569 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:31:19.570 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:31:19.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:31:19.572 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:31:19.573 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0.212982177734375, 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 792, 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.574 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.575 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:31:19.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:31:19.578 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.579 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:16.593439+0000", 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:25.181288+0000", 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:55.923810+0000", 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.581 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:13.602274+0000", 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:14.694863+0000", 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:56.916932+0000", 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.583 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:10.851101+0000", 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.584 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:31:08.544244+0000", 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:23.937961+0000", 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:31:19.585 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.586 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.586 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.586 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.586 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:31:19.586 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:09.726435+0000", 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:34.199781+0000", 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:30:07.158807+0000", 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.588 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.e", 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: 4, 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: 2 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.1e", 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: 11, 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: 9 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.589 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [ 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.7", 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: "mappings": [ 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: "from": 10, 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: "to": 11 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:31:19.590 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [ 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "8.3", 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: 5 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "8.6", 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: 7 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "8.7", 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: 10 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "8.d", 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: 7, 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: 2 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.592 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.593 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:31:19.594 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:31:19.595 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:31:19.595 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:31:19.595 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:31:19.595 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:31:19.595 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:31:19.595 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:31:19.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:19.595 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:31:19.595 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:31:19.595 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:31:19.603 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:31:19.603 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:31:19.635 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:31:19.635 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:31:19.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:19.668 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:31:19.669 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:31:19.673 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:19.674 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:19.693 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:31:19.693 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:31:19.719 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:31:19.719 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:31:19.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:19.799 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:19.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:19.901 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:19.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:19.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:19.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:20.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:20.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:20.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:20.058 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:20.159 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:20.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:20.185 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:20.218 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:31:20.286 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:20.311 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:20.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:20.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2909033048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:20.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/902442112' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:31:20.413 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2909033048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:20.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/902442112' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:31:20.445 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:20.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2909033048' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:20.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/902442112' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:31:20.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:20.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:20.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:20.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:20.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:20.735 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:20.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:20.837 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:20.843 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:20.860 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:20.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:20.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:20.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:20.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:20.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:21.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:21.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:21.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:21.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:21.240 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:21.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:21.261 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:21 smithi151 ceph-mon[16639]: pgmap v1937: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 622/201138 objects degraded (0.309%); 16649/201138 objects misplaced (8.277%); 8.5 MiB/s, 136 objects/s recovering 2024-09-02T10:31:21.342 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:21.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:21 smithi162 ceph-mon[20279]: pgmap v1937: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 622/201138 objects degraded (0.309%); 16649/201138 objects misplaced (8.277%); 8.5 MiB/s, 136 objects/s recovering 2024-09-02T10:31:21.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:21.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:21.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:21.499 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:21.500 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:21.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:21 smithi187 ceph-mon[20072]: pgmap v1937: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 681 op/s; 622/201138 objects degraded (0.309%); 16649/201138 objects misplaced (8.277%); 8.5 MiB/s, 136 objects/s recovering 2024-09-02T10:31:21.601 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:21.625 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:21.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:21.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:21.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:21.755 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:21.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:21.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:21.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:21.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:22.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:22.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:22.001 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:22.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:22.133 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:22.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:22.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:22.257 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:22.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:22.259 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:22.360 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:22.387 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:22.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:22.489 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:22.490 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:31:22.513 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:22.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:22 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:31:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:22 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:31:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:22 smithi162 ceph-mon[20279]: osdmap e848: 12 total, 12 up, 12 in 2024-09-02T10:31:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:22 smithi162 ceph-mon[20279]: pgmap v1939: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 630/199296 objects degraded (0.316%); 16445/199296 objects misplaced (8.252%); 7.6 MiB/s, 121 objects/s recovering 2024-09-02T10:31:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1727446263' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:31:22.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:22 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:31:22.616 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:22.646 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:22.646 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:22 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:31:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:22 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:31:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:22 smithi151 ceph-mon[16639]: osdmap e848: 12 total, 12 up, 12 in 2024-09-02T10:31:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:22 smithi151 ceph-mon[16639]: pgmap v1939: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 630/199296 objects degraded (0.316%); 16445/199296 objects misplaced (8.252%); 7.6 MiB/s, 121 objects/s recovering 2024-09-02T10:31:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1727446263' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:31:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:22 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:31:22.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:22 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:31:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:22 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:31:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:22 smithi187 ceph-mon[20072]: osdmap e848: 12 total, 12 up, 12 in 2024-09-02T10:31:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:22 smithi187 ceph-mon[20072]: pgmap v1939: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.1 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 650 op/s; 630/199296 objects degraded (0.316%); 16445/199296 objects misplaced (8.252%); 7.6 MiB/s, 121 objects/s recovering 2024-09-02T10:31:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1727446263' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:31:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:22 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:31:22.771 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:22.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:22.873 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:22.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:22.901 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:23.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:23.030 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:23.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:23.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:23.155 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:23.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:23.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:23.280 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:23.281 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:23.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:23.404 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:23.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:23.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:23.539 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:23.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:23.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:23 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:23 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:31:23.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:23 smithi162 ceph-mon[20279]: osdmap e849: 12 total, 12 up, 12 in 2024-09-02T10:31:23.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:23.668 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:23.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:23.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:23.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:23 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:23 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:31:23.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:23 smithi151 ceph-mon[16639]: osdmap e849: 12 total, 12 up, 12 in 2024-09-02T10:31:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:23 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:23 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:31:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:23 smithi187 ceph-mon[20072]: osdmap e849: 12 total, 12 up, 12 in 2024-09-02T10:31:23.770 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:23.792 INFO:tasks.thrashosds.thrasher:j is {'epoch': 847, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:31:08.881065+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:20:51.089364+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.940000057220459, 'score_stable': 1.940000057220459, 'optimal_score': 1, 'raw_score_acting': 1.940000057220459, 'raw_score_stable': 1.940000057220459, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_6', 'create_time': '2024-09-02T10:28:50.275834+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '797', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 843, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 844, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 769, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 741, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 783, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 844, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 844, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 1, 'weight': 0.212982177734375, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:16.593439+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:25.181288+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:55.923810+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:13.602274+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:14.694863+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:56.916932+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:10.851101+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:31:08.544244+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:23.937961+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:09.726435+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:34.199781+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:30:07.158807+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.e', 'osds': [4, 9, 2]}, {'pgid': '1.1e', 'osds': [11, 10, 9]}], 'pg_upmap_items': [{'pgid': '1.7', 'mappings': [{'from': 10, 'to': 11}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.3', 'osds': [1, 5]}, {'pgid': '8.6', 'osds': [10, 7]}, {'pgid': '8.7', 'osds': [1, 10]}, {'pgid': '8.d', 'osds': [7, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:31:23.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:31:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:23.794 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:23.794 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:23.824 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:23.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:23.924 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:23.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:24.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:24.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:24.054 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:24.055 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:24.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:24.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:24.195 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:24.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:24.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:24.338 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:24.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:24.440 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:24.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:31:24.464 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:24.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:24.477 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:31:24.477 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:31:24.477 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:31:24.477 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:31:24.478 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.478 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157268 6.b 6:db1c5c67:::benchmark_data_smithi187_102448_object56996:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:31:24.478 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:31:24.478 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 596.64723327299998, 2024-09-02T10:31:24.478 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62417882300000005, 2024-09-02T10:31:24.478 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.478 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.479 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:31:24.479 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:31:24.479 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157268 2024-09-02T10:31:24.479 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.479 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.480 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:31:24.480 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.480 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.480 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.480 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824846+0000", 2024-09-02T10:31:24.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.481 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.481 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.481 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.481 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824847+0000", 2024-09-02T10:31:24.481 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2900000000000004e-07 2024-09-02T10:31:24.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.482 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.482 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824850+0000", 2024-09-02T10:31:24.482 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.76e-06 2024-09-02T10:31:24.482 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.482 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.483 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.483 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824860+0000", 2024-09-02T10:31:24.483 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2690000000000005e-06 2024-09-02T10:31:24.483 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.483 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824876+0000", 2024-09-02T10:31:24.484 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5882000000000001e-05 2024-09-02T10:31:24.484 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.484 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.484 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.484 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.824934+0000", 2024-09-02T10:31:24.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8780000000000003e-05 2024-09-02T10:31:24.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.485 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.485 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.825044+0000", 2024-09-02T10:31:24.485 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010938900000000001 2024-09-02T10:31:24.485 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.485 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.825142+0000", 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8009000000000003e-05 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830102+0000", 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049605530000000004 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.838185+0000", 2024-09-02T10:31:24.486 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0080826430000000005 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.448954+0000", 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61076896000000003 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.448992+0000", 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7732999999999999e-05 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.487 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.488 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449025+0000", 2024-09-02T10:31:24.488 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3315999999999997e-05 2024-09-02T10:31:24.488 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.488 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.488 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.488 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.488 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.488 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157271 6.6 6:69d4aafc:::benchmark_data_smithi187_102448_object56999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:31:24.488 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:31:24.488 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 596.64167863399996, 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62368362300000002, 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157271 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.489 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830401+0000", 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830403+0000", 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7179999999999999e-06 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.490 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830409+0000", 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.703e-06 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830424+0000", 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4961e-05 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830432+0000", 2024-09-02T10:31:24.491 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4880000000000001e-06 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830495+0000", 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3758999999999997e-05 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830625+0000", 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012918000000000001 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.492 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.830756+0000", 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013138200000000001 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.835882+0000", 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051260960000000001 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.836280+0000", 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00039814100000000002 2024-09-02T10:31:24.493 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454013+0000", 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61773251699999998 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454048+0000", 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.5559999999999998e-05 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.494 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454084+0000", 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6118000000000002e-05 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157274 6.6 6:6387cbb8:::benchmark_data_smithi187_102448_object57002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 596.63465048, 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61674791900000003, 2024-09-02T10:31:24.495 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157274 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:31:24.496 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837429+0000", 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837431+0000", 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7010000000000001e-06 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.497 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837437+0000", 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0379999999999996e-06 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837448+0000", 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1557e-05 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837456+0000", 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.396e-06 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.498 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837510+0000", 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4666000000000001e-05 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837636+0000", 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000125313 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837757+0000", 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000121212 2024-09-02T10:31:24.499 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.845343+0000", 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0075863880000000003 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.847219+0000", 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00187546 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.500 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454127+0000", 2024-09-02T10:31:24.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60690782600000004 2024-09-02T10:31:24.501 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.501 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.501 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.501 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454156+0000", 2024-09-02T10:31:24.501 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9281999999999998e-05 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454177+0000", 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1080000000000001e-05 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157277 6.6 6:67446ce0:::benchmark_data_smithi187_102448_object57005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:31:24.502 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 596.63411886999995, 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.61628705500000003, 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157277 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.503 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837961+0000", 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0699999999999996e-07 2024-09-02T10:31:24.504 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837964+0000", 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.8569999999999999e-06 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837973+0000", 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0159999999999997e-06 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.837978+0000", 2024-09-02T10:31:24.505 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7770000000000002e-06 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842046+0000", 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040683289999999999 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842130+0000", 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3751000000000005e-05 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.506 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.842199+0000", 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8411999999999995e-05 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.845512+0000", 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003313795 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.847365+0000", 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018529009999999999 2024-09-02T10:31:24.507 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454203+0000", 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.60683791600000003 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454229+0000", 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5609000000000002e-05 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.508 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454248+0000", 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8885e-05 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157282 6.b 6:d751a93a:::benchmark_data_smithi187_102448_object57010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 596.59814841299999, 2024-09-02T10:31:24.509 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.57523664399999996, 2024-09-02T10:31:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:31:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:31:24.510 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157282 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873931+0000", 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.511 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.512 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873933+0000", 2024-09-02T10:31:24.512 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.08e-06 2024-09-02T10:31:24.512 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.512 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.512 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:24.513 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.513 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873941+0000", 2024-09-02T10:31:24.513 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0919999999999998e-06 2024-09-02T10:31:24.513 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.513 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873961+0000", 2024-09-02T10:31:24.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0231e-05 2024-09-02T10:31:24.514 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.514 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.514 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.514 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.873971+0000", 2024-09-02T10:31:24.514 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.2969999999999992e-06 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874034+0000", 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3257999999999996e-05 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874177+0000", 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000142712 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.515 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.874303+0000", 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000126248 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.878010+0000", 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0037068610000000001 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:27.946302+0000", 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.068292383999999998 2024-09-02T10:31:24.516 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449070+0000", 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.50276810199999999 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449124+0000", 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3712999999999999e-05 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.449168+0000", 2024-09-02T10:31:24.517 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3665999999999998e-05 2024-09-02T10:31:24.518 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.518 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.518 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.518 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.518 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.518 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157307 6.6 6:675a1a62:::benchmark_data_smithi187_102448_object57035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:31:24.518 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:31:24.518 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 596.45405973200002, 2024-09-02T10:31:24.518 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43718100199999999, 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157307 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.519 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018020+0000", 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018021+0000", 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6559999999999999e-06 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.520 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018029+0000", 2024-09-02T10:31:24.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6310000000000003e-06 2024-09-02T10:31:24.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.521 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.521 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018046+0000", 2024-09-02T10:31:24.521 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6779999999999999e-05 2024-09-02T10:31:24.521 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.521 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018054+0000", 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3180000000000007e-06 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018619+0000", 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00056491400000000002 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018723+0000", 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000103619 2024-09-02T10:31:24.522 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,7", 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.018800+0000", 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7859000000000001e-05 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.022686+0000", 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0038855399999999998 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.523 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031392+0000", 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0087064819999999998 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455118+0000", 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42372546999999999 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.524 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455167+0000", 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8773000000000002e-05 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.455201+0000", 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.396e-05 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:157310 6.b 6:d937f28e:::benchmark_data_smithi187_102448_object57038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e598)", 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:31:24.525 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 596.44095530599998, 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.42301055599999998, 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 157310 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.526 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031124+0000", 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031125+0000", 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8599999999999997e-07 2024-09-02T10:31:24.527 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031130+0000", 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7949999999999998e-06 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031141+0000", 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0735e-05 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.528 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031148+0000", 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4189999999999996e-06 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031196+0000", 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7571000000000002e-05 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031278+0000", 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2262000000000006e-05 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.529 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.031363+0000", 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5279999999999997e-05 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.035027+0000", 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0036643140000000001 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.290856+0000", 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25582839899999998 2024-09-02T10:31:24.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454040+0000", 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.163183774 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454092+0000", 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2224e-05 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.531 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:21:28.454135+0000", 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2896999999999999e-05 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.26524.0:200538 6.b 6:dbd5fb01:::benchmark_data_smithi187_102448_object100266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e626)", 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:40.755486+0000", 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 523.71659321799996, 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70349968900000004, 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.532 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.26524", 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/770196963", 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 200538 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755486+0000", 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.533 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755486+0000", 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755488+0000", 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3680000000000001e-06 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.534 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755494+0000", 2024-09-02T10:31:24.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6059999999999999e-06 2024-09-02T10:31:24.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.535 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.535 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.535 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755507+0000", 2024-09-02T10:31:24.535 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2850999999999999e-05 2024-09-02T10:31:24.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755514+0000", 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0480000000000001e-06 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755579+0000", 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4465e-05 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.536 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755693+0000", 2024-09-02T10:31:24.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114207 2024-09-02T10:31:24.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:31:24.537 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.755818+0000", 2024-09-02T10:31:24.537 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000124998 2024-09-02T10:31:24.537 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.537 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.760931+0000", 2024-09-02T10:31:24.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0051127849999999999 2024-09-02T10:31:24.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.538 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.538 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.538 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.762153+0000", 2024-09-02T10:31:24.538 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00122255 2024-09-02T10:31:24.538 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458876+0000", 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69672282799999996 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458936+0000", 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9864000000000001e-05 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.540 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.540 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458986+0000", 2024-09-02T10:31:24.540 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0118999999999999e-05 2024-09-02T10:31:24.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.540 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.540 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.26524.0:200538 6.b e626/533 ondisk, result = 0)", 2024-09-02T10:31:24.540 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 523.71042590800005, 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69733888300000002, 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.541 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:31:24.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761654+0000", 2024-09-02T10:31:24.542 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1900000000000001e-07 2024-09-02T10:31:24.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.542 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.542 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:40.761658+0000", 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0430000000000001e-06 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458732+0000", 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.69707383899999997 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458742+0000", 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4120000000000007e-06 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.543 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458857+0000", 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000115513 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458869+0000", 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1325000000000001e-05 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:22:41.458992+0000", 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012383199999999999 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.544 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDScrubReserve(1.8,RELEASE,e:640,from:5,reservation_nonce:0)", 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 502.57948267, 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54587120899999997, 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892597+0000", 2024-09-02T10:31:24.546 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5900000000000001e-07 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:01.892598+0000", 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5209999999999999e-06 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438366+0000", 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.54576760899999999 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.547 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438376+0000", 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.029e-05 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438438+0000", 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1400000000000002e-05 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438454+0000", 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6132000000000001e-05 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:23:02.438468+0000", 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4098e-05 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:135860 7.1 e740/717)", 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.726925+0000", 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 282.74515447300001, 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.362811098, 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.549 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726925+0000", 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726925+0000", 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.550 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726926+0000", 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8400000000000004e-07 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726931+0000", 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6339999999999998e-06 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872187+0000", 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.145255784 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.551 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872192+0000", 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7990000000000001e-06 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.089719+0000", 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.217526953 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.552 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.089736+0000", 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7244e-05 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135866 7.2 7.308b4c22 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.733884+0000", 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 282.738195023, 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38588288599999998, 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135866 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733884+0000", 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733884+0000", 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733886+0000", 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7269999999999999e-06 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733891+0000", 2024-09-02T10:31:24.555 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8690000000000003e-06 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733902+0000", 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1139e-05 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733910+0000", 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5820000000000004e-06 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.556 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.119723+0000", 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38581336799999999 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.119767+0000", 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4200999999999999e-05 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.557 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135869 7.12 7:4e2fbeef:::benchmark_data_smithi187_120712_object51927:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.872032+0000", 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 282.60004730200001, 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.95676479699999994, 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135869 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.558 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872032+0000", 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872032+0000", 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.559 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872034+0000", 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9099999999999999e-06 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872043+0000", 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6319999999999997e-06 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872054+0000", 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1352000000000001e-05 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.561 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.561 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872063+0000", 2024-09-02T10:31:24.561 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1470000000000007e-06 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.120273+0000", 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24820998499999999 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.120282+0000", 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7499999999999992e-06 2024-09-02T10:31:24.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.820947+0000", 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70066504299999999 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821053+0000", 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010563100000000001 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821156+0000", 2024-09-02T10:31:24.563 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010351199999999999 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.823544+0000", 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002387672 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.823920+0000", 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037607899999999997 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.564 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.828702+0000", 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047823479999999996 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.828757+0000", 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5010999999999999e-05 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.828797+0000", 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9725000000000003e-05 2024-09-02T10:31:24.565 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:136913 7.10 7.335c64d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.758150+0000", 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 280.71392937299998, 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47337745399999998, 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:31:24.566 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136913 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758150+0000", 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758150+0000", 2024-09-02T10:31:24.567 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758152+0000", 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0839999999999999e-06 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758158+0000", 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8889999999999999e-06 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.568 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758173+0000", 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4831000000000001e-05 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758181+0000", 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4120000000000001e-06 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.569 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758256+0000", 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4867e-05 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758265+0000", 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1710000000000008e-06 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231504+0000", 2024-09-02T10:31:24.570 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47323864300000001 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231528+0000", 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3557000000000002e-05 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:136914 7.3 7:c30a193a:::benchmark_data_smithi187_120712_object52972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.758512+0000", 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 280.71356700299998, 2024-09-02T10:31:24.571 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33439487099999998, 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136914 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758512+0000", 2024-09-02T10:31:24.572 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758512+0000", 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758513+0000", 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0299999999999998e-07 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758517+0000", 2024-09-02T10:31:24.573 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6799999999999999e-06 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758527+0000", 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0404999999999999e-05 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758534+0000", 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3729999999999998e-06 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.574 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758589+0000", 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5315999999999997e-05 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758687+0000", 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7879000000000003e-05 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,4", 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758807+0000", 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119839 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.575 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.760731+0000", 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001924413 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.771256+0000", 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010524939000000001 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092834+0000", 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.321577844 2024-09-02T10:31:24.576 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092874+0000", 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0209000000000001e-05 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092907+0000", 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3271e-05 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:136923 7.10 7.4a3ead10 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.891481+0000", 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 280.58059810399999, 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34014060299999999, 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136923 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891481+0000", 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891481+0000", 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891483+0000", 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2899999999999999e-06 2024-09-02T10:31:24.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891489+0000", 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8830000000000003e-06 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891502+0000", 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3086e-05 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891508+0000", 2024-09-02T10:31:24.580 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9889999999999997e-06 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891569+0000", 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1042000000000003e-05 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891577+0000", 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1289999999999996e-06 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.581 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231610+0000", 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34003330300000001 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231622+0000", 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1881e-05 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:154719 7.6 e749/741 7:65814d4b:::benchmark_data_smithi187_120712_object70777:head v 749'9238, mlcod=749'9235)", 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:31:24.582 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 246.56745677800001, 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26617824499999998, 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:31:24.583 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3999999999999997e-07 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904629+0000", 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5419999999999996e-06 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.584 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.164896+0000", 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26026724899999998 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.164907+0000", 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0862999999999999e-05 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.165442+0000", 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053543600000000003 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.585 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.165465+0000", 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2442e-05 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.170745+0000", 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0052807360000000003 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.170801+0000", 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5637000000000003e-05 2024-09-02T10:31:24.586 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:109466 8.0 8:09d4259d:::benchmark_data_smithi187_136059_object4185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e836)", 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:30:34.224731+0000", 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.247348596999998, 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26449957800000001, 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:31:24.587 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109466 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224731+0000", 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224731+0000", 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224732+0000", 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4300000000000001e-06 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224738+0000", 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9689999999999999e-06 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224750+0000", 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1479e-05 2024-09-02T10:31:24.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.590 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224756+0000", 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1999999999999999e-06 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224821+0000", 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5056999999999997e-05 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474358+0000", 2024-09-02T10:31:24.592 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.249537286 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,9", 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474480+0000", 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012162799999999999 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.480187+0000", 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005706994 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.593 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.480196+0000", 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5079999999999994e-06 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.489130+0000", 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089338339999999999 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.489180+0000", 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0167999999999998e-05 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.489230+0000", 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0025000000000003e-05 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27559.0:109464 8.c e836/792 8:3bd17a8d:::benchmark_data_smithi187_136059_object4183:head v 836'6827, mlcod=836'6825)", 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:30:34.224895+0000", 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 50.247184212999997, 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25820176, 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.595 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224895+0000", 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224895+0000", 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.596 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224896+0000", 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3600000000000002e-07 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224903+0000", 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.708e-06 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474294+0000", 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.249391523 2024-09-02T10:31:24.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474313+0000", 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8998999999999999e-05 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474382+0000", 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8394999999999998e-05 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474406+0000", 2024-09-02T10:31:24.598 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4085000000000001e-05 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.483074+0000", 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086678429999999997 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.483097+0000", 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3371e-05 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:134986 8.0 8:02e53999:::benchmark_data_smithi187_136059_object29705:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:01.901202+0000", 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 22.570876935000001, 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21155201400000001, 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 134986 2024-09-02T10:31:24.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.901202+0000", 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.901202+0000", 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:31:24.601 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.901206+0000", 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.067e-06 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.901220+0000", 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4693e-05 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.901257+0000", 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6579000000000003e-05 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.602 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.901266+0000", 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.0259999999999996e-06 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.901343+0000", 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7439000000000002e-05 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.901440+0000", 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6433000000000003e-05 2024-09-02T10:31:24.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,9", 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.901516+0000", 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6287999999999999e-05 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.905022+0000", 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0035061229999999999 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.005353+0000", 2024-09-02T10:31:24.604 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.10033128099999999 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.112644+0000", 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.10729021699999999 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.112698+0000", 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4786e-05 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:31:24.605 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.112755+0000", 2024-09-02T10:31:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6082000000000003e-05 2024-09-02T10:31:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:31:24.606 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:31:24.606 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:31:24.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:24.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:24 smithi162 ceph-mon[20279]: pgmap v1941: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 630/194595 objects degraded (0.324%); 16253/194595 objects misplaced (8.352%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:31:24.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:24 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:31:24.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:24 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:31:24.607 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 630/199296 objects degraded (0.316%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:24.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:24 smithi151 ceph-mon[16639]: pgmap v1941: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 630/194595 objects degraded (0.324%); 16253/194595 objects misplaced (8.352%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:31:24.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:24 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:31:24.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:24 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:31:24.608 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 630/199296 objects degraded (0.316%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:24.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:24.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:24.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:24.756 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:24 smithi187 ceph-mon[20072]: pgmap v1941: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 4.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 630/194595 objects degraded (0.324%); 16253/194595 objects misplaced (8.352%); 7.7 MiB/s, 123 objects/s recovering 2024-09-02T10:31:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:24 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:31:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:24 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:31:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 630/199296 objects degraded (0.316%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:24.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:24.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:24.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:24.886 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:24.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:25.018 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:25.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:25.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:25.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:25.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:25.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:25.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:25.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:25.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:25.289 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:31:25.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:25.417 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:25.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:25.418 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:25.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:25 smithi151 ceph-mon[16639]: osdmap e850: 12 total, 12 up, 12 in 2024-09-02T10:31:25.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:25 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:31:25.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:25 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:31:25.519 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:25.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:25.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:25.544 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:25 smithi162 ceph-mon[20279]: osdmap e850: 12 total, 12 up, 12 in 2024-09-02T10:31:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:25 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:31:25.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:25 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:31:25.645 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:25.669 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:25.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:25 smithi187 ceph-mon[20072]: osdmap e850: 12 total, 12 up, 12 in 2024-09-02T10:31:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:25 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:31:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:25 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:31:25.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:25.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:25.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:25.912 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:25.938 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:25.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:26.039 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:26.061 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:26.062 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:26.163 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:26.190 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:26.191 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:26.292 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:26.319 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:26.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:26.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:26.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:26.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:26.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:26.585 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:26.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:26.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:26 smithi151 ceph-mon[16639]: pgmap v1943: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 727/191352 objects degraded (0.380%); 15885/191352 objects misplaced (8.301%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:31:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:26.717 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:26.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:26 smithi187 ceph-mon[20072]: pgmap v1943: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 727/191352 objects degraded (0.380%); 15885/191352 objects misplaced (8.301%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:31:26.818 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:26.842 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:26.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:26 smithi162 ceph-mon[20279]: pgmap v1943: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.9 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 543 op/s; 727/191352 objects degraded (0.380%); 15885/191352 objects misplaced (8.301%); 6.8 MiB/s, 109 objects/s recovering 2024-09-02T10:31:26.944 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:26.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:26.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:27.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:27.103 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:27.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:27.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:27.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:27.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:27.230 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:27.331 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:27.353 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:27.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:27.355 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:27.456 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:27.477 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:27.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:27.479 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:27.580 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:27.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:27.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:27.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:27.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:31:27.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:31:27.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:27 smithi151 ceph-mon[16639]: osdmap e851: 12 total, 12 up, 12 in 2024-09-02T10:31:27.718 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:27.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:27.744 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:31:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:31:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:27 smithi187 ceph-mon[20072]: osdmap e851: 12 total, 12 up, 12 in 2024-09-02T10:31:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:31:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:31:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:27 smithi162 ceph-mon[20279]: osdmap e851: 12 total, 12 up, 12 in 2024-09-02T10:31:27.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:27.871 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:27.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:27.973 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:27.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:27.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:28.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:28.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:28.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:28.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:28.260 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:28.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:28.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:28.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:28.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:28.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:28.520 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:28.521 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:28.561 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:31:28.561 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:31:28.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:28.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:28.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:28.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:28.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:28 smithi151 ceph-mon[16639]: pgmap v1945: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 744/187869 objects degraded (0.396%); 15500/187869 objects misplaced (8.250%); 8.6 MiB/s, 138 objects/s recovering 2024-09-02T10:31:28.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:28 smithi151 ceph-mon[16639]: from='client.18648 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:31:28.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:28 smithi187 ceph-mon[20072]: pgmap v1945: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 744/187869 objects degraded (0.396%); 15500/187869 objects misplaced (8.250%); 8.6 MiB/s, 138 objects/s recovering 2024-09-02T10:31:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:28 smithi187 ceph-mon[20072]: from='client.18648 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:31:28.774 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:28.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:28 smithi162 ceph-mon[20279]: pgmap v1945: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.8 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 634 op/s; 744/187869 objects degraded (0.396%); 15500/187869 objects misplaced (8.250%); 8.6 MiB/s, 138 objects/s recovering 2024-09-02T10:31:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:28 smithi162 ceph-mon[20279]: from='client.18648 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:31:28.876 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:28.904 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:28.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:29.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:29.038 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:29.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:29.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:29.140 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:29.169 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:29.170 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:29.271 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:29.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:29.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:29.399 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:29.421 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:29.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:29.524 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:29.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:29.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:29.655 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:29.679 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:29.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:29 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:31:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:29 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:31:29.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:29 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 744/187869 objects degraded (0.396%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:29 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:31:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:29 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:31:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:29 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 744/187869 objects degraded (0.396%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:29.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:29.806 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:29.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:29.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:29 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:31:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:29 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:31:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:29 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 744/187869 objects degraded (0.396%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:29.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:29.939 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:29.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:29.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:30.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:30.066 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:30.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:30.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:30.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:30.195 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:30.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:30.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:30.323 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:30.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:30.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:30.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:30.452 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:30.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:30.575 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:30.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:30.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:30 smithi151 ceph-mon[16639]: 8.0 scrub starts 2024-09-02T10:31:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:30 smithi151 ceph-mon[16639]: pgmap v1946: 47 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 837/183240 objects degraded (0.457%); 15123/183240 objects misplaced (8.253%); 7.9 MiB/s, 127 objects/s recovering 2024-09-02T10:31:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:30 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:31:30.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:30 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:31:30.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:30.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:30.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:30 smithi187 ceph-mon[20072]: 8.0 scrub starts 2024-09-02T10:31:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:30 smithi187 ceph-mon[20072]: pgmap v1946: 47 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 837/183240 objects degraded (0.457%); 15123/183240 objects misplaced (8.253%); 7.9 MiB/s, 127 objects/s recovering 2024-09-02T10:31:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:30 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:31:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:30 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:31:30.806 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:30.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:30.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:30.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:30 smithi162 ceph-mon[20279]: 8.0 scrub starts 2024-09-02T10:31:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:30 smithi162 ceph-mon[20279]: pgmap v1946: 47 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 622 op/s; 837/183240 objects degraded (0.457%); 15123/183240 objects misplaced (8.253%); 7.9 MiB/s, 127 objects/s recovering 2024-09-02T10:31:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:30 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:31:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:30 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:31:30.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:30.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:30.965 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:30.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:31.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:31.096 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:31.097 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:31.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:31.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:31.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:31.334 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:31.361 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:31.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:31.463 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:31.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:31.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:31.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:31.623 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:31.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:31 smithi151 ceph-mon[16639]: 8.0 scrub ok 2024-09-02T10:31:31.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:31 smithi151 ceph-mon[16639]: 8.f scrub starts 2024-09-02T10:31:31.725 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:31 smithi187 ceph-mon[20072]: 8.0 scrub ok 2024-09-02T10:31:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:31 smithi187 ceph-mon[20072]: 8.f scrub starts 2024-09-02T10:31:31.761 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:31.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:31 smithi162 ceph-mon[20279]: 8.0 scrub ok 2024-09-02T10:31:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:31 smithi162 ceph-mon[20279]: 8.f scrub starts 2024-09-02T10:31:31.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:31.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:31.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:31.995 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:32.021 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:32.022 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:32.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:32.153 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:32.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:32.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:32.279 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:32.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:32.381 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:32.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:32.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:32.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:32.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:32.533 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:32.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:32.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:32.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:32.662 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:32.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:32.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:32 smithi151 ceph-mon[16639]: pgmap v1947: 47 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 837/181560 objects degraded (0.461%); 14898/181560 objects misplaced (8.206%); 8.9 MiB/s, 143 objects/s recovering 2024-09-02T10:31:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:32 smithi187 ceph-mon[20072]: pgmap v1947: 47 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 837/181560 objects degraded (0.461%); 14898/181560 objects misplaced (8.206%); 8.9 MiB/s, 143 objects/s recovering 2024-09-02T10:31:32.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:32.790 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:32.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:32 smithi162 ceph-mon[20279]: pgmap v1947: 47 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.7 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 542 op/s; 837/181560 objects degraded (0.461%); 14898/181560 objects misplaced (8.206%); 8.9 MiB/s, 143 objects/s recovering 2024-09-02T10:31:32.892 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:32.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:32.920 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:33.021 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:33.044 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:33.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:33.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:33.169 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:33.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:33.270 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:33.296 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:33.297 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:33.398 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:33.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:33.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:33.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:33.527 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:31:33.531 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:33.563 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:33.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:33.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:33.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:33 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:31:33.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:33 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:31:33.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:33 smithi151 ceph-mon[16639]: 8.f scrub ok 2024-09-02T10:31:33.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:33 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:31:33.685 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:33 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:31:33.688 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:33.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:33.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:33 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:31:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:33 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:31:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:33 smithi187 ceph-mon[20072]: 8.f scrub ok 2024-09-02T10:31:33.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:33 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:31:33.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:33 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:31:33.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:33.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:33.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:33.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:33 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:31:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:33 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:31:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:33 smithi162 ceph-mon[20279]: 8.f scrub ok 2024-09-02T10:31:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:33 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:31:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:33 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:31:33.869 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:33.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:33.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:33.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:33.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:34.061 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:34.085 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:34.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:34.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:34.121 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:34.175 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1945,"stamp":"2024-09-02T10:31:27.543253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4104454712,"num_objects":62623,"num_object_clones":0,"num_object_copies":187869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":744,"num_objects_misplaced":15500,"num_objects_unfound":0,"num_objects_dirty":62623,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147937,"num_write_kb":6737793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2649,"num_bytes_recovered":173094496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":150789,"ondisk_log_size":150789,"up":141,"acting":137,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":19916044,"kb_used_data":12323992,"kb_used_omap":18,"kb_used_meta":7591789,"kb_avail":1105025780,"statfs":{"total":1151940427776,"available":1131546398720,"internally_reserved":0,"allocated":12619767808,"data_stored":12616295915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":7773992251},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,2,3,5,4,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-249626624,"num_objects":-3809,"num_object_clones":0,"num_object_copies":-11427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":114,"num_objects_misplaced":-945,"num_objects_unfound":0,"num_objects_dirty":-3809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3809,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":831,"num_bytes_recovered":54067200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001357"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":1098,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.376088+0000","last_change":"2024-09-02T10:30:19.443030+0000","last_active":"2024-09-02T10:31:26.376088+0000","last_peered":"2024-09-02T10:31:26.376088+0000","last_clean":"2024-09-02T10:31:26.376088+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:31:26.376088+0000","last_undegraded":"2024-09-02T10:31:26.376088+0000","last_fullsized":"2024-09-02T10:31:26.376088+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:19.442955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:19.442955+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:26.516100+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":909,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.325369+0000","last_change":"2024-09-02T10:31:02.822274+0000","last_active":"2024-09-02T10:31:22.325369+0000","last_peered":"2024-09-02T10:31:22.325369+0000","last_clean":"2024-09-02T10:31:22.325369+0000","last_became_active":"2024-09-02T10:31:02.821847+0000","last_became_peered":"2024-09-02T10:31:02.821847+0000","last_unstale":"2024-09-02T10:31:22.325369+0000","last_undegraded":"2024-09-02T10:31:22.325369+0000","last_fullsized":"2024-09-02T10:31:22.325369+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:54.883293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.883293+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1370,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-02T10:31:24.340871+0000","last_change":"2024-09-02T10:30:14.830001+0000","last_active":"2024-09-02T10:31:24.340871+0000","last_peered":"2024-09-02T10:31:24.340871+0000","last_clean":"2024-09-02T10:31:24.340871+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:31:24.340871+0000","last_undegraded":"2024-09-02T10:31:24.340871+0000","last_fullsized":"2024-09-02T10:31:24.340871+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:14.829717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:14.829717+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1354,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-02T10:31:24.340826+0000","last_change":"2024-09-02T10:30:17.892008+0000","last_active":"2024-09-02T10:31:24.340826+0000","last_peered":"2024-09-02T10:31:24.340826+0000","last_clean":"2024-09-02T10:31:24.340826+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:31:24.340826+0000","last_undegraded":"2024-09-02T10:31:24.340826+0000","last_fullsized":"2024-09-02T10:31:24.340826+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:17.891937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:17.891937+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:40.788304+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":657,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-02T10:31:24.352240+0000","last_change":"2024-09-02T10:30:14.720335+0000","last_active":"2024-09-02T10:31:24.352240+0000","last_peered":"2024-09-02T10:31:24.352240+0000","last_clean":"2024-09-02T10:31:24.352240+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:31:24.352240+0000","last_undegraded":"2024-09-02T10:31:24.352240+0000","last_fullsized":"2024-09-02T10:31:24.352240+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:14.720291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:14.720291+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:32.669400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":1218,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.360683+0000","last_change":"2024-09-02T10:29:58.037831+0000","last_active":"2024-09-02T10:31:26.360683+0000","last_peered":"2024-09-02T10:31:26.360683+0000","last_clean":"2024-09-02T10:31:26.360683+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:31:26.360683+0000","last_undegraded":"2024-09-02T10:31:26.360683+0000","last_fullsized":"2024-09-02T10:31:26.360683+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:58.037744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:58.037744+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1734,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.327979+0000","last_change":"2024-09-02T10:29:57.973361+0000","last_active":"2024-09-02T10:31:22.327979+0000","last_peered":"2024-09-02T10:31:22.327979+0000","last_clean":"2024-09-02T10:31:22.327979+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:31:22.327979+0000","last_undegraded":"2024-09-02T10:31:22.327979+0000","last_fullsized":"2024-09-02T10:31:22.327979+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:57.973289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:57.973289+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1378,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-02T10:31:24.340793+0000","last_change":"2024-09-02T10:30:24.905924+0000","last_active":"2024-09-02T10:31:24.340793+0000","last_peered":"2024-09-02T10:31:24.340793+0000","last_clean":"2024-09-02T10:31:24.340793+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:31:24.340793+0000","last_undegraded":"2024-09-02T10:31:24.340793+0000","last_fullsized":"2024-09-02T10:31:24.340793+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.905855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.905855+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:50.566333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":1381,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.320775+0000","last_change":"2024-09-02T10:30:18.808693+0000","last_active":"2024-09-02T10:31:22.320775+0000","last_peered":"2024-09-02T10:31:22.320775+0000","last_clean":"2024-09-02T10:31:22.320775+0000","last_became_active":"2024-09-02T10:25:01.211125+0000","last_became_peered":"2024-09-02T10:25:01.211125+0000","last_unstale":"2024-09-02T10:31:22.320775+0000","last_undegraded":"2024-09-02T10:31:22.320775+0000","last_fullsized":"2024-09-02T10:31:22.320775+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:18.808615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:18.808615+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1347,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.366881+0000","last_change":"2024-09-02T10:30:47.378511+0000","last_active":"2024-09-02T10:31:26.366881+0000","last_peered":"2024-09-02T10:31:26.366881+0000","last_clean":"2024-09-02T10:31:26.366881+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:31:26.366881+0000","last_undegraded":"2024-09-02T10:31:26.366881+0000","last_fullsized":"2024-09-02T10:31:26.366881+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:47.378461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:47.378461+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:54.553044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":591,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.368743+0000","last_change":"2024-09-02T10:30:21.301133+0000","last_active":"2024-09-02T10:31:26.368743+0000","last_peered":"2024-09-02T10:31:26.368743+0000","last_clean":"2024-09-02T10:31:26.368743+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:31:26.368743+0000","last_undegraded":"2024-09-02T10:31:26.368743+0000","last_fullsized":"2024-09-02T10:31:26.368743+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:21.301078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:21.301078+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:50.908107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1700,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.322830+0000","last_change":"2024-09-02T10:31:02.820524+0000","last_active":"2024-09-02T10:31:22.322830+0000","last_peered":"2024-09-02T10:31:22.322830+0000","last_clean":"2024-09-02T10:31:22.322830+0000","last_became_active":"2024-09-02T10:31:02.820058+0000","last_became_peered":"2024-09-02T10:31:02.820058+0000","last_unstale":"2024-09-02T10:31:22.322830+0000","last_undegraded":"2024-09-02T10:31:22.322830+0000","last_fullsized":"2024-09-02T10:31:22.322830+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:48.014073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.014073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:10.729452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1633,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.327874+0000","last_change":"2024-09-02T10:30:23.108717+0000","last_active":"2024-09-02T10:31:22.327874+0000","last_peered":"2024-09-02T10:31:22.327874+0000","last_clean":"2024-09-02T10:31:22.327874+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:31:22.327874+0000","last_undegraded":"2024-09-02T10:31:22.327874+0000","last_fullsized":"2024-09-02T10:31:22.327874+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:23.108646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:23.108646+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:40.286555+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":356,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.332381+0000","last_change":"2024-09-02T10:29:52.115045+0000","last_active":"2024-09-02T10:31:22.332381+0000","last_peered":"2024-09-02T10:31:22.332381+0000","last_clean":"2024-09-02T10:31:22.332381+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:31:22.332381+0000","last_undegraded":"2024-09-02T10:31:22.332381+0000","last_fullsized":"2024-09-02T10:31:22.332381+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:52.114995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:52.114995+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:24.311916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1603,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.328048+0000","last_change":"2024-09-02T10:30:16.984287+0000","last_active":"2024-09-02T10:31:22.328048+0000","last_peered":"2024-09-02T10:31:22.328048+0000","last_clean":"2024-09-02T10:31:22.328048+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:31:22.328048+0000","last_undegraded":"2024-09-02T10:31:22.328048+0000","last_fullsized":"2024-09-02T10:31:22.328048+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:16.984241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:16.984241+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"849'9089","reported_seq":9315,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:24.037729+0000","last_change":"2024-09-02T10:30:45.515400+0000","last_active":"2024-09-02T10:31:24.037729+0000","last_peered":"2024-09-02T10:31:24.037729+0000","last_clean":"2024-09-02T10:31:24.037729+0000","last_became_active":"2024-09-02T10:28:51.931930+0000","last_became_peered":"2024-09-02T10:28:51.931930+0000","last_unstale":"2024-09-02T10:31:24.037729+0000","last_undegraded":"2024-09-02T10:31:24.037729+0000","last_fullsized":"2024-09-02T10:31:24.037729+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7411","last_scrub_stamp":"2024-09-02T10:30:45.515308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:45.515308+0000","objects_scrubbed":5608,"log_size":9089,"log_dups_size":0,"ondisk_log_size":9089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:07.081713+0000","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255918080,"num_objects":3905,"num_object_clones":0,"num_object_copies":11715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":415808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1697,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.328096+0000","last_change":"2024-09-02T10:30:22.096061+0000","last_active":"2024-09-02T10:31:22.328096+0000","last_peered":"2024-09-02T10:31:22.328096+0000","last_clean":"2024-09-02T10:31:22.328096+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:31:22.328096+0000","last_undegraded":"2024-09-02T10:31:22.328096+0000","last_fullsized":"2024-09-02T10:31:22.328096+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:22.095998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:22.095998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:35.377458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"849'9359","reported_seq":9552,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:23.901941+0000","last_change":"2024-09-02T10:30:17.338013+0000","last_active":"2024-09-02T10:31:23.901941+0000","last_peered":"2024-09-02T10:31:23.901941+0000","last_clean":"2024-09-02T10:31:23.901941+0000","last_became_active":"2024-09-02T10:28:51.652599+0000","last_became_peered":"2024-09-02T10:28:51.652599+0000","last_unstale":"2024-09-02T10:31:23.901941+0000","last_undegraded":"2024-09-02T10:31:23.901941+0000","last_fullsized":"2024-09-02T10:31:23.901941+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'5982","last_scrub_stamp":"2024-09-02T10:30:17.337832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:17.337832+0000","objects_scrubbed":5938,"log_size":9359,"log_dups_size":0,"ondisk_log_size":9359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261947392,"num_objects":3997,"num_object_clones":0,"num_object_copies":11991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9359,"num_write_kb":427392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1826,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.366742+0000","last_change":"2024-09-02T10:30:24.407289+0000","last_active":"2024-09-02T10:31:26.366742+0000","last_peered":"2024-09-02T10:31:26.366742+0000","last_clean":"2024-09-02T10:31:26.366742+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:31:26.366742+0000","last_undegraded":"2024-09-02T10:31:26.366742+0000","last_fullsized":"2024-09-02T10:31:26.366742+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.407218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.407218+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:46.044283+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"849'9271","reported_seq":10488,"reported_epoch":849,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:31:24.078994+0000","last_change":"2024-09-02T10:31:04.242623+0000","last_active":"2024-09-02T10:31:24.078994+0000","last_peered":"2024-09-02T10:31:24.078994+0000","last_clean":"2024-09-02T10:31:01.749812+0000","last_became_active":"2024-09-02T10:31:04.153013+0000","last_became_peered":"2024-09-02T10:31:04.153013+0000","last_unstale":"2024-09-02T10:31:24.078994+0000","last_undegraded":"2024-09-02T10:31:03.853663+0000","last_fullsized":"2024-09-02T10:31:03.819841+0000","mapping_epoch":844,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'5828","last_scrub_stamp":"2024-09-02T10:30:16.082748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:16.082748+0000","objects_scrubbed":5788,"log_size":9271,"log_dups_size":0,"ondisk_log_size":9271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1773,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252903424,"num_objects":3859,"num_object_clones":0,"num_object_copies":11577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":313,"num_objects_misplaced":3859,"num_objects_unfound":0,"num_objects_dirty":3859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9271,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":511,"num_bytes_recovered":32899072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5],"avail_no_missing":["1","5","11"],"object_location_counts":[{"shards":"1,5","objects":-313},{"shards":"1,5,11","objects":3859}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1466,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.323870+0000","last_change":"2024-09-02T10:29:59.479021+0000","last_active":"2024-09-02T10:31:22.323870+0000","last_peered":"2024-09-02T10:31:22.323870+0000","last_clean":"2024-09-02T10:31:22.323870+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:31:22.323870+0000","last_undegraded":"2024-09-02T10:31:22.323870+0000","last_fullsized":"2024-09-02T10:31:22.323870+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:59.478887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:59.478887+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:24.159387+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"851'9290","reported_seq":9485,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.560347+0000","last_change":"2024-09-02T10:30:16.516178+0000","last_active":"2024-09-02T10:31:26.560347+0000","last_peered":"2024-09-02T10:31:26.560347+0000","last_clean":"2024-09-02T10:31:26.560347+0000","last_became_active":"2024-09-02T10:28:51.668558+0000","last_became_peered":"2024-09-02T10:28:51.668558+0000","last_unstale":"2024-09-02T10:31:26.560347+0000","last_undegraded":"2024-09-02T10:31:26.560347+0000","last_fullsized":"2024-09-02T10:31:26.560347+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'5835","last_scrub_stamp":"2024-09-02T10:30:16.515921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:16.515921+0000","objects_scrubbed":5778,"log_size":9290,"log_dups_size":0,"ondisk_log_size":9290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:35.009866+0000","scrub_duration":1503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247201792,"num_objects":3772,"num_object_clones":0,"num_object_copies":11316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9290,"num_write_kb":417984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":435,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.328072+0000","last_change":"2024-09-02T10:31:02.820173+0000","last_active":"2024-09-02T10:31:22.328072+0000","last_peered":"2024-09-02T10:31:22.328072+0000","last_clean":"2024-09-02T10:31:22.328072+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:31:22.328072+0000","last_undegraded":"2024-09-02T10:31:22.328072+0000","last_fullsized":"2024-09-02T10:31:22.328072+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:48.652851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:48.652851+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"851'9304","reported_seq":11202,"reported_epoch":851,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:31:26.497308+0000","last_change":"2024-09-02T10:31:04.087511+0000","last_active":"2024-09-02T10:31:26.497308+0000","last_peered":"2024-09-02T10:31:26.497308+0000","last_clean":"2024-09-02T10:31:01.804575+0000","last_became_active":"2024-09-02T10:31:04.035888+0000","last_became_peered":"2024-09-02T10:31:04.035888+0000","last_unstale":"2024-09-02T10:31:26.497308+0000","last_undegraded":"2024-09-02T10:31:03.853844+0000","last_fullsized":"2024-09-02T10:31:03.818904+0000","mapping_epoch":844,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"822'4517","last_scrub_stamp":"2024-09-02T10:29:54.734394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.734394+0000","objects_scrubbed":4467,"log_size":9304,"log_dups_size":0,"ondisk_log_size":9304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1390,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250937368,"num_objects":3830,"num_object_clones":0,"num_object_copies":11490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":89,"num_objects_misplaced":3807,"num_objects_unfound":0,"num_objects_dirty":3830,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9304,"num_write_kb":420225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":855,"num_bytes_recovered":55836672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2],"avail_no_missing":["7","2","11"],"object_location_counts":[{"shards":"2,7","objects":-66},{"shards":"2,7,11","objects":3807}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1774,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.375898+0000","last_change":"2024-09-02T10:30:58.631533+0000","last_active":"2024-09-02T10:31:26.375898+0000","last_peered":"2024-09-02T10:31:26.375898+0000","last_clean":"2024-09-02T10:31:26.375898+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:31:26.375898+0000","last_undegraded":"2024-09-02T10:31:26.375898+0000","last_fullsized":"2024-09-02T10:31:26.375898+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:58.631463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:58.631463+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:19.925804+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"849'9153","reported_seq":9349,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:23.001660+0000","last_change":"2024-09-02T10:30:24.460290+0000","last_active":"2024-09-02T10:31:23.001660+0000","last_peered":"2024-09-02T10:31:23.001660+0000","last_clean":"2024-09-02T10:31:23.001660+0000","last_became_active":"2024-09-02T10:28:51.652606+0000","last_became_peered":"2024-09-02T10:28:51.652606+0000","last_unstale":"2024-09-02T10:31:23.001660+0000","last_undegraded":"2024-09-02T10:31:23.001660+0000","last_fullsized":"2024-09-02T10:31:23.001660+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'6309","last_scrub_stamp":"2024-09-02T10:30:24.460190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.460190+0000","objects_scrubbed":6243,"log_size":9153,"log_dups_size":0,"ondisk_log_size":9153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:48.861566+0000","scrub_duration":2190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260374528,"num_objects":3973,"num_object_clones":0,"num_object_copies":11919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3973,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9153,"num_write_kb":420032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":1241,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.360888+0000","last_change":"2024-09-02T10:30:55.890418+0000","last_active":"2024-09-02T10:31:26.360888+0000","last_peered":"2024-09-02T10:31:26.360888+0000","last_clean":"2024-09-02T10:31:26.360888+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:31:26.360888+0000","last_undegraded":"2024-09-02T10:31:26.360888+0000","last_fullsized":"2024-09-02T10:31:26.360888+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:55.890351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:55.890351+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:56.943591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"851'9193","reported_seq":9392,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.789957+0000","last_change":"2024-09-02T10:30:20.825467+0000","last_active":"2024-09-02T10:31:26.789957+0000","last_peered":"2024-09-02T10:31:26.789957+0000","last_clean":"2024-09-02T10:31:26.789957+0000","last_became_active":"2024-09-02T10:28:51.932528+0000","last_became_peered":"2024-09-02T10:28:51.932528+0000","last_unstale":"2024-09-02T10:31:26.789957+0000","last_undegraded":"2024-09-02T10:31:26.789957+0000","last_fullsized":"2024-09-02T10:31:26.789957+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'6041","last_scrub_stamp":"2024-09-02T10:30:20.825310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:20.825310+0000","objects_scrubbed":5988,"log_size":9193,"log_dups_size":0,"ondisk_log_size":9193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:30.224599+0000","scrub_duration":2452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250544128,"num_objects":3823,"num_object_clones":0,"num_object_copies":11469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9193,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":1205,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.369742+0000","last_change":"2024-09-02T10:29:54.344897+0000","last_active":"2024-09-02T10:31:26.369742+0000","last_peered":"2024-09-02T10:31:26.369742+0000","last_clean":"2024-09-02T10:31:26.369742+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:31:26.369742+0000","last_undegraded":"2024-09-02T10:31:26.369742+0000","last_fullsized":"2024-09-02T10:31:26.369742+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:54.344678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.344678+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"849'9122","reported_seq":9316,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.641936+0000","last_change":"2024-09-02T10:30:20.863311+0000","last_active":"2024-09-02T10:31:22.641936+0000","last_peered":"2024-09-02T10:31:22.641936+0000","last_clean":"2024-09-02T10:31:22.641936+0000","last_became_active":"2024-09-02T10:28:51.931936+0000","last_became_peered":"2024-09-02T10:28:51.931936+0000","last_unstale":"2024-09-02T10:31:22.641936+0000","last_undegraded":"2024-09-02T10:31:22.641936+0000","last_fullsized":"2024-09-02T10:31:22.641936+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'6100","last_scrub_stamp":"2024-09-02T10:30:20.863163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:20.863163+0000","objects_scrubbed":6054,"log_size":9122,"log_dups_size":0,"ondisk_log_size":9122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:26.262855+0000","scrub_duration":1643,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":259260416,"num_objects":3956,"num_object_clones":0,"num_object_copies":11868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9122,"num_write_kb":418496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":987,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.357999+0000","last_change":"2024-09-02T10:30:14.802221+0000","last_active":"2024-09-02T10:31:26.357999+0000","last_peered":"2024-09-02T10:31:26.357999+0000","last_clean":"2024-09-02T10:31:26.357999+0000","last_became_active":"2024-09-02T10:25:02.065175+0000","last_became_peered":"2024-09-02T10:25:02.065175+0000","last_unstale":"2024-09-02T10:31:26.357999+0000","last_undegraded":"2024-09-02T10:31:26.357999+0000","last_fullsized":"2024-09-02T10:31:26.357999+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:14.802157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:14.802157+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"849'9298","reported_seq":9493,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.598176+0000","last_change":"2024-09-02T10:30:46.822316+0000","last_active":"2024-09-02T10:31:22.598176+0000","last_peered":"2024-09-02T10:31:22.598176+0000","last_clean":"2024-09-02T10:31:22.598176+0000","last_became_active":"2024-09-02T10:28:51.652541+0000","last_became_peered":"2024-09-02T10:28:51.652541+0000","last_unstale":"2024-09-02T10:31:22.598176+0000","last_undegraded":"2024-09-02T10:31:22.598176+0000","last_fullsized":"2024-09-02T10:31:22.598176+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7674","last_scrub_stamp":"2024-09-02T10:30:46.822238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.822238+0000","objects_scrubbed":5790,"log_size":9298,"log_dups_size":0,"ondisk_log_size":9298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:51.931363+0000","scrub_duration":1685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270139392,"num_objects":4122,"num_object_clones":0,"num_object_copies":12366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9298,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,9],"acting":[5,3,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":1221,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.360863+0000","last_change":"2024-09-02T10:30:48.025489+0000","last_active":"2024-09-02T10:31:26.360863+0000","last_peered":"2024-09-02T10:31:26.360863+0000","last_clean":"2024-09-02T10:31:26.360863+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:31:26.360863+0000","last_undegraded":"2024-09-02T10:31:26.360863+0000","last_fullsized":"2024-09-02T10:31:26.360863+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:48.025327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.025327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:57.412128+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"850'9301","reported_seq":9501,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-02T10:31:24.528993+0000","last_change":"2024-09-02T10:30:23.214839+0000","last_active":"2024-09-02T10:31:24.528993+0000","last_peered":"2024-09-02T10:31:24.528993+0000","last_clean":"2024-09-02T10:31:24.528993+0000","last_became_active":"2024-09-02T10:28:51.668549+0000","last_became_peered":"2024-09-02T10:28:51.668549+0000","last_unstale":"2024-09-02T10:31:24.528993+0000","last_undegraded":"2024-09-02T10:31:24.528993+0000","last_fullsized":"2024-09-02T10:31:24.528993+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'6274","last_scrub_stamp":"2024-09-02T10:30:23.214700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:23.214700+0000","objects_scrubbed":6218,"log_size":9301,"log_dups_size":0,"ondisk_log_size":9301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:26.351683+0000","scrub_duration":2309,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257753088,"num_objects":3933,"num_object_clones":0,"num_object_copies":11799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9301,"num_write_kb":423488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1468,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.366828+0000","last_change":"2024-09-02T10:30:23.359450+0000","last_active":"2024-09-02T10:31:26.366828+0000","last_peered":"2024-09-02T10:31:26.366828+0000","last_clean":"2024-09-02T10:31:26.366828+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:31:26.366828+0000","last_undegraded":"2024-09-02T10:31:26.366828+0000","last_fullsized":"2024-09-02T10:31:26.366828+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:23.359405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:23.359405+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:27.036873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"850'9076","reported_seq":9260,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-02T10:31:24.516017+0000","last_change":"2024-09-02T10:30:00.120454+0000","last_active":"2024-09-02T10:31:24.516017+0000","last_peered":"2024-09-02T10:31:24.516017+0000","last_clean":"2024-09-02T10:31:24.516017+0000","last_became_active":"2024-09-02T10:28:51.932243+0000","last_became_peered":"2024-09-02T10:28:51.932243+0000","last_unstale":"2024-09-02T10:31:24.516017+0000","last_undegraded":"2024-09-02T10:31:24.516017+0000","last_fullsized":"2024-09-02T10:31:24.516017+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"822'4808","last_scrub_stamp":"2024-09-02T10:30:00.120321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:00.120321+0000","objects_scrubbed":4780,"log_size":9076,"log_dups_size":0,"ondisk_log_size":9076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252837888,"num_objects":3858,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9076,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1725,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.322926+0000","last_change":"2024-09-02T10:30:20.873940+0000","last_active":"2024-09-02T10:31:22.322926+0000","last_peered":"2024-09-02T10:31:22.322926+0000","last_clean":"2024-09-02T10:31:22.322926+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:31:22.322926+0000","last_undegraded":"2024-09-02T10:31:22.322926+0000","last_fullsized":"2024-09-02T10:31:22.322926+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:20.873897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:20.873897+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:41.753920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"851'9372","reported_seq":9602,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.593098+0000","last_change":"2024-09-02T10:30:46.985280+0000","last_active":"2024-09-02T10:31:26.593098+0000","last_peered":"2024-09-02T10:31:26.593098+0000","last_clean":"2024-09-02T10:31:26.593098+0000","last_became_active":"2024-09-02T10:28:51.650885+0000","last_became_peered":"2024-09-02T10:28:51.650885+0000","last_unstale":"2024-09-02T10:31:26.593098+0000","last_undegraded":"2024-09-02T10:31:26.593098+0000","last_fullsized":"2024-09-02T10:31:26.593098+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7658","last_scrub_stamp":"2024-09-02T10:30:46.985056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.985056+0000","objects_scrubbed":5619,"log_size":9372,"log_dups_size":0,"ondisk_log_size":9372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:47.684250+0000","scrub_duration":2021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252575744,"num_objects":3854,"num_object_clones":0,"num_object_copies":11562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9372,"num_write_kb":423232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1479,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.366720+0000","last_change":"2024-09-02T10:31:03.381315+0000","last_active":"2024-09-02T10:31:26.366720+0000","last_peered":"2024-09-02T10:31:26.366720+0000","last_clean":"2024-09-02T10:31:26.366720+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:31:26.366720+0000","last_undegraded":"2024-09-02T10:31:26.366720+0000","last_fullsized":"2024-09-02T10:31:26.366720+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:58.376013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:58.376013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"851'9375","reported_seq":11088,"reported_epoch":851,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:31:26.597172+0000","last_change":"2024-09-02T10:31:04.128960+0000","last_active":"2024-09-02T10:31:26.597172+0000","last_peered":"2024-09-02T10:31:26.597172+0000","last_clean":"2024-09-02T10:31:01.786664+0000","last_became_active":"2024-09-02T10:31:04.069096+0000","last_became_peered":"2024-09-02T10:31:04.069096+0000","last_unstale":"2024-09-02T10:31:26.597172+0000","last_undegraded":"2024-09-02T10:31:03.844885+0000","last_fullsized":"2024-09-02T10:31:03.820047+0000","mapping_epoch":844,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'5937","last_scrub_stamp":"2024-09-02T10:30:17.013655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:17.013655+0000","objects_scrubbed":5883,"log_size":9375,"log_dups_size":0,"ondisk_log_size":9375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256311296,"num_objects":3911,"num_object_clones":0,"num_object_copies":11733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":86,"num_objects_misplaced":3911,"num_objects_unfound":0,"num_objects_dirty":3911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9375,"num_write_kb":425152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":49545216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7],"avail_no_missing":["10","7","11"],"object_location_counts":[{"shards":"7,10","objects":-86},{"shards":"7,10,11","objects":3911}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1734,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.358004+0000","last_change":"2024-09-02T10:30:20.714039+0000","last_active":"2024-09-02T10:31:26.358004+0000","last_peered":"2024-09-02T10:31:26.358004+0000","last_clean":"2024-09-02T10:31:26.358004+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:31:26.358004+0000","last_undegraded":"2024-09-02T10:31:26.358004+0000","last_fullsized":"2024-09-02T10:31:26.358004+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:20.713975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:20.713975+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:40.167796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"849'9262","reported_seq":10497,"reported_epoch":849,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:31:24.076274+0000","last_change":"2024-09-02T10:31:04.119388+0000","last_active":"2024-09-02T10:31:24.076274+0000","last_peered":"2024-09-02T10:31:24.076274+0000","last_clean":"2024-09-02T10:31:01.796091+0000","last_became_active":"2024-09-02T10:31:04.054394+0000","last_became_peered":"2024-09-02T10:31:04.054394+0000","last_unstale":"2024-09-02T10:31:24.076274+0000","last_undegraded":"2024-09-02T10:31:03.853930+0000","last_fullsized":"2024-09-02T10:31:03.819998+0000","mapping_epoch":844,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"839'7994","last_scrub_stamp":"2024-09-02T10:30:53.579876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:53.579876+0000","objects_scrubbed":5226,"log_size":9262,"log_dups_size":0,"ondisk_log_size":9262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257163264,"num_objects":3924,"num_object_clones":0,"num_object_copies":11772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":256,"num_objects_misplaced":3923,"num_objects_unfound":0,"num_objects_dirty":3924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9262,"num_write_kb":421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":522,"num_bytes_recovered":33882112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10],"avail_no_missing":["1","10","11"],"object_location_counts":[{"shards":"1,10","objects":-255},{"shards":"1,10,11","objects":3923}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":673,"reported_epoch":850,"state":"active+clean","last_fresh":"2024-09-02T10:31:24.349227+0000","last_change":"2024-09-02T10:30:44.835334+0000","last_active":"2024-09-02T10:31:24.349227+0000","last_peered":"2024-09-02T10:31:24.349227+0000","last_clean":"2024-09-02T10:31:24.349227+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:31:24.349227+0000","last_undegraded":"2024-09-02T10:31:24.349227+0000","last_fullsized":"2024-09-02T10:31:24.349227+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:44.835287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:44.835287+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:47.563776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"849'9205","reported_seq":9428,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.571514+0000","last_change":"2024-09-02T10:30:48.613043+0000","last_active":"2024-09-02T10:31:22.571514+0000","last_peered":"2024-09-02T10:31:22.571514+0000","last_clean":"2024-09-02T10:31:22.571514+0000","last_became_active":"2024-09-02T10:28:51.651609+0000","last_became_peered":"2024-09-02T10:28:51.651609+0000","last_unstale":"2024-09-02T10:31:22.571514+0000","last_undegraded":"2024-09-02T10:31:22.571514+0000","last_fullsized":"2024-09-02T10:31:22.571514+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7724","last_scrub_stamp":"2024-09-02T10:30:48.612984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.612984+0000","objects_scrubbed":5646,"log_size":9205,"log_dups_size":0,"ondisk_log_size":9205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:59.218878+0000","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":270467072,"num_objects":4127,"num_object_clones":0,"num_object_copies":12381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9205,"num_write_kb":426624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1506,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.366960+0000","last_change":"2024-09-02T10:30:46.402988+0000","last_active":"2024-09-02T10:31:26.366960+0000","last_peered":"2024-09-02T10:31:26.366960+0000","last_clean":"2024-09-02T10:31:26.366960+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:31:26.366960+0000","last_undegraded":"2024-09-02T10:31:26.366960+0000","last_fullsized":"2024-09-02T10:31:26.366960+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:46.402901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.402901+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:12.184642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"851'9267","reported_seq":9469,"reported_epoch":851,"state":"active+clean","last_fresh":"2024-09-02T10:31:26.588504+0000","last_change":"2024-09-02T10:30:27.169510+0000","last_active":"2024-09-02T10:31:26.588504+0000","last_peered":"2024-09-02T10:31:26.588504+0000","last_clean":"2024-09-02T10:31:26.588504+0000","last_became_active":"2024-09-02T10:28:51.668549+0000","last_became_peered":"2024-09-02T10:28:51.668549+0000","last_unstale":"2024-09-02T10:31:26.588504+0000","last_undegraded":"2024-09-02T10:31:26.588504+0000","last_fullsized":"2024-09-02T10:31:26.588504+0000","mapping_epoch":792,"log_start":"0'0","ondisk_log_start":"0'0","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"835'6417","last_scrub_stamp":"2024-09-02T10:30:27.169341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:27.169341+0000","objects_scrubbed":6355,"log_size":9267,"log_dups_size":0,"ondisk_log_size":9267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:56.046399+0000","scrub_duration":2342,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":247529472,"num_objects":3777,"num_object_clones":0,"num_object_copies":11331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9267,"num_write_kb":417408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":886,"reported_epoch":849,"state":"active+clean","last_fresh":"2024-09-02T10:31:22.329847+0000","last_change":"2024-09-02T10:30:21.773406+0000","last_active":"2024-09-02T10:31:22.329847+0000","last_peered":"2024-09-02T10:31:22.329847+0000","last_clean":"2024-09-02T10:31:22.329847+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:31:22.329847+0000","last_undegraded":"2024-09-02T10:31:22.329847+0000","last_fullsized":"2024-09-02T10:31:22.329847+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:21.773337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:21.773337+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:31:30.611344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4103864344,"num_objects":62621,"num_object_clones":0,"num_object_copies":187863,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":744,"num_objects_misplaced":15500,"num_objects_unfound":0,"num_objects_dirty":62621,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":147937,"num_write_kb":6737793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2645,"num_bytes_recovered":172163072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12548976640,"data_stored":12548964424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":147937,"ondisk_log_size":147937,"up":48,"acting":44,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384056,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2131144,"kb_used_data":1454388,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":91614008,"statfs":{"total":95995035648,"available":93812744192,"internally_reserved":0,"allocated":1489293312,"data_stored":1488995740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":692976068},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612989,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2045560,"kb_used_data":1490980,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91699592,"statfs":{"total":95995035648,"available":93900382208,"internally_reserved":0,"allocated":1526763520,"data_stored":1526474307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":567867840},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841921,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2291888,"kb_used_data":1552664,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":91453264,"statfs":{"total":95995035648,"available":93648142336,"internally_reserved":0,"allocated":1589927936,"data_stored":1589643010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":756939200},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070856,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1201772,"kb_used_data":755480,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":92543380,"statfs":{"total":95995035648,"available":94764421120,"internally_reserved":0,"allocated":773611520,"data_stored":773326594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456980933},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299789,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1174164,"kb_used_data":497404,"kb_used_omap":1,"kb_used_meta":676734,"kb_avail":92570988,"statfs":{"total":95995035648,"available":94792691712,"internally_reserved":0,"allocated":509341696,"data_stored":509040572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":692976074},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528721,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":469636,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":93275516,"statfs":{"total":95995035648,"available":95514128384,"internally_reserved":0,"allocated":1163264,"data_stored":869764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":479721927},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724949,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2049576,"kb_used_data":1306196,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":91695576,"statfs":{"total":95995035648,"available":93896269824,"internally_reserved":0,"allocated":1337544704,"data_stored":1337260563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953882,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1469864,"kb_used_data":1001368,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92275288,"statfs":{"total":95995035648,"available":94489894912,"internally_reserved":0,"allocated":1025400832,"data_stored":1025115906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150112,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1541420,"kb_used_data":770264,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92203732,"statfs":{"total":95995035648,"available":94416621568,"internally_reserved":0,"allocated":788750336,"data_stored":788465410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":789641672},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411748,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1751376,"kb_used_data":1214844,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91993776,"statfs":{"total":95995035648,"available":94201626624,"internally_reserved":0,"allocated":1244000256,"data_stored":1243699132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575272,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2199148,"kb_used_data":1509912,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91546004,"statfs":{"total":95995035648,"available":93743108096,"internally_reserved":0,"allocated":1546149888,"data_stored":1545864962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1590496,"kb_used_data":769356,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92154656,"statfs":{"total":95995035648,"available":94366367744,"internally_reserved":0,"allocated":787820544,"data_stored":787539955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":782630912,"data_stored":782630912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1534328832,"data_stored":1534328832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1241387008,"data_stored":1241382936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":786628608,"data_stored":786628608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1015349248,"data_stored":1015349248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1330315264,"data_stored":1330315264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":196608,"data_stored":196608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":507514880,"data_stored":507510808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":766705664,"data_stored":766705664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1580269568,"data_stored":1580269568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1515388928,"data_stored":1515388928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1488261120,"data_stored":1488257048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:31:34.177 INFO:tasks.thrashosds.thrasher:Setting 8.8 to [5, 10, 2, 11, 3, 8] 2024-09-02T10:31:34.177 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '8.8', '5', '10', '2', '11', '3', '8'] 2024-09-02T10:31:34.178 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 8.8 5 10 2 11 3 8 2024-09-02T10:31:34.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:34.216 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:34.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:34.317 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:34.324 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:34.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:34.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:34.340 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:34.434 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:31:34.434 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:31:34.434 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:31:34.434 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:31:34.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:34.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:34.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:34 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:31:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:34 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:31:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:34 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:31:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:34 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:31:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:34 smithi162 ceph-mon[20279]: 8.e scrub starts 2024-09-02T10:31:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:34 smithi162 ceph-mon[20279]: pgmap v1948: 47 pgs: 2 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 839/176451 objects degraded (0.475%); 14704/176451 objects misplaced (8.333%); 7.2 MiB/s, 116 objects/s recovering 2024-09-02T10:31:34.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:34 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 837/181560 objects degraded (0.461%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:34.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:34.628 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:34.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:34.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:34 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:31:34.661 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:34 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:31:34.662 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:34 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:31:34.662 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:34 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:31:34.662 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:34 smithi151 ceph-mon[16639]: 8.e scrub starts 2024-09-02T10:31:34.662 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:34 smithi151 ceph-mon[16639]: pgmap v1948: 47 pgs: 2 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 839/176451 objects degraded (0.475%); 14704/176451 objects misplaced (8.333%); 7.2 MiB/s, 116 objects/s recovering 2024-09-02T10:31:34.662 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:34 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 837/181560 objects degraded (0.461%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:34.730 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:34.754 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:34.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:34.755 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:34 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:31:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:34 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:31:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:34 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:31:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:34 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:31:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:34 smithi187 ceph-mon[20072]: 8.e scrub starts 2024-09-02T10:31:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:34 smithi187 ceph-mon[20072]: pgmap v1948: 47 pgs: 2 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 41 active+clean; 3.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 839/176451 objects degraded (0.475%); 14704/176451 objects misplaced (8.333%); 7.2 MiB/s, 116 objects/s recovering 2024-09-02T10:31:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:34 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 837/181560 objects degraded (0.461%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:34.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:34.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:34.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:34.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:35.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:35.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:35.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:35.086 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:31:35.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:35.151 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:35.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:35.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:35.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:35.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:35.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:35.362 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:35.380 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:35.403 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:31:35.405 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:35.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:35.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:35.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:31:35.479 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:31:35.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:35.514 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:31:35.514 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:31:35.514 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:35 smithi151 ceph-mon[16639]: 8.e scrub ok 2024-09-02T10:31:35.515 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4203494846' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:31:35.533 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:35.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:35.549 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:31:35.549 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:31:35.583 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:31:35.583 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:31:35.614 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:31:35.614 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:31:35.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:35.649 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:31:35.649 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:31:35.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:35 smithi162 ceph-mon[20279]: 8.e scrub ok 2024-09-02T10:31:35.655 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4203494846' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:31:35.661 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:35.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:35.680 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:35.685 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:31:35.685 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:31:35.732 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:31:35.732 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:31:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:35 smithi187 ceph-mon[20072]: 8.e scrub ok 2024-09-02T10:31:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4203494846' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:31:35.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:35.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:31:35.765 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:31:35.787 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:35.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:35.801 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:31:35.801 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:31:35.833 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:31:35.833 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:31:35.870 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:31:35.871 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:31:35.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:35.898 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:31:35.898 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:31:35.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:35.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:36.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:36.046 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:36.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:36.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:36.176 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:36.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:36.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:36.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:36.303 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:36.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:36.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:36.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:36.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:36.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:36.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:36.560 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:36.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:36.578 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:36 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:31:36.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4203494846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:31:36.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:36 smithi162 ceph-mon[20279]: osdmap e852: 12 total, 12 up, 12 in 2024-09-02T10:31:36.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3275783857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:36.579 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:36 smithi162 ceph-mon[20279]: pgmap v1950: 47 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 921/173193 objects degraded (0.532%); 14334/173193 objects misplaced (8.276%); 9.2 MiB/s, 147 objects/s recovering 2024-09-02T10:31:36.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:31:36.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:36.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:36 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:31:36.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4203494846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:31:36.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:36 smithi151 ceph-mon[16639]: osdmap e852: 12 total, 12 up, 12 in 2024-09-02T10:31:36.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3275783857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:36.679 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:36 smithi151 ceph-mon[16639]: pgmap v1950: 47 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 921/173193 objects degraded (0.532%); 14334/173193 objects misplaced (8.276%); 9.2 MiB/s, 147 objects/s recovering 2024-09-02T10:31:36.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:36.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:36.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:36.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:36 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:31:36.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4203494846' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:31:36.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:36 smithi187 ceph-mon[20072]: osdmap e852: 12 total, 12 up, 12 in 2024-09-02T10:31:36.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3275783857' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:36.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:36 smithi187 ceph-mon[20072]: pgmap v1950: 47 pgs: 1 active+clean+scrubbing, 4 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 658 op/s; 921/173193 objects degraded (0.532%); 14334/173193 objects misplaced (8.276%); 9.2 MiB/s, 147 objects/s recovering 2024-09-02T10:31:36.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:36.822 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:36.822 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:36.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:36.947 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:36.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:36.961 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:37.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:37.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:37.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:37.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:37.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:37.205 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:37.206 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:37.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:37.331 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:37.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:37.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:37.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:37.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:37.503 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:31:37.503 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:31:37.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:37.583 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:37.584 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:37.686 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:37.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:37.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:37.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:37.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:37.864 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:37.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:37.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:37.988 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:37.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:38.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:38.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:38.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:38.187 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:31:38.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:38.251 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:38.252 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:38.324 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:31:38.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:38.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:38.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:38.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:38.515 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:38.534 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:38.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:38.602 INFO:teuthology.orchestra.run.smithi151.stderr:set 8.8 pg_upmap_items mapping to [5->10,2->11,3->8] 2024-09-02T10:31:38.636 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:38.666 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:38.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:38.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:38.768 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:38.794 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:38.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:38.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:38.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:38.921 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:38.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:38 smithi151 ceph-mon[16639]: pgmap v1951: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 921/169983 objects degraded (0.542%); 13981/169983 objects misplaced (8.225%); 8.4 MiB/s, 134 objects/s recovering 2024-09-02T10:31:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:38 smithi151 ceph-mon[16639]: 8.a scrub starts 2024-09-02T10:31:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/95052328' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [5, 10, 2, 11, 3, 8]} : dispatch 2024-09-02T10:31:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:38 smithi151 ceph-mon[16639]: 8.b scrub starts 2024-09-02T10:31:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:38 smithi187 ceph-mon[20072]: pgmap v1951: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 921/169983 objects degraded (0.542%); 13981/169983 objects misplaced (8.225%); 8.4 MiB/s, 134 objects/s recovering 2024-09-02T10:31:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:38 smithi187 ceph-mon[20072]: 8.a scrub starts 2024-09-02T10:31:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/95052328' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [5, 10, 2, 11, 3, 8]} : dispatch 2024-09-02T10:31:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:38 smithi187 ceph-mon[20072]: 8.b scrub starts 2024-09-02T10:31:39.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:39.044 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:39.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:38 smithi162 ceph-mon[20279]: pgmap v1951: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.5 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 596 op/s; 921/169983 objects degraded (0.542%); 13981/169983 objects misplaced (8.225%); 8.4 MiB/s, 134 objects/s recovering 2024-09-02T10:31:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:38 smithi162 ceph-mon[20279]: 8.a scrub starts 2024-09-02T10:31:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/95052328' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [5, 10, 2, 11, 3, 8]} : dispatch 2024-09-02T10:31:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:38 smithi162 ceph-mon[20279]: 8.b scrub starts 2024-09-02T10:31:39.147 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:39.169 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:39.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:39.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:39.170 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:31:39.270 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:39.293 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:39.294 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:39.394 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:39.416 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:39.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:39.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:39.541 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:39.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:39.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:39.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:39.665 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:39.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:39.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:39.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:39.788 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:39.789 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:39.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:39.913 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:39.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:31:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:31:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/95052328' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [5, 10, 2, 11, 3, 8]}]': finished 2024-09-02T10:31:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:39 smithi151 ceph-mon[16639]: osdmap e853: 12 total, 12 up, 12 in 2024-09-02T10:31:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:39 smithi151 ceph-mon[16639]: 8.a scrub ok 2024-09-02T10:31:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:39 smithi151 ceph-mon[16639]: 8.b scrub ok 2024-09-02T10:31:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 921/169983 objects degraded (0.542%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:39 smithi151 ceph-mon[16639]: osdmap e854: 12 total, 12 up, 12 in 2024-09-02T10:31:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:31:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:31:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/95052328' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [5, 10, 2, 11, 3, 8]}]': finished 2024-09-02T10:31:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:39 smithi187 ceph-mon[20072]: osdmap e853: 12 total, 12 up, 12 in 2024-09-02T10:31:40.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:39 smithi187 ceph-mon[20072]: 8.a scrub ok 2024-09-02T10:31:40.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:39 smithi187 ceph-mon[20072]: 8.b scrub ok 2024-09-02T10:31:40.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 921/169983 objects degraded (0.542%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:40.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:39 smithi187 ceph-mon[20072]: osdmap e854: 12 total, 12 up, 12 in 2024-09-02T10:31:40.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:40.038 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:40.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:31:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:31:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/95052328' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.8", "id": [5, 10, 2, 11, 3, 8]}]': finished 2024-09-02T10:31:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:39 smithi162 ceph-mon[20279]: osdmap e853: 12 total, 12 up, 12 in 2024-09-02T10:31:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:39 smithi162 ceph-mon[20279]: 8.a scrub ok 2024-09-02T10:31:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:39 smithi162 ceph-mon[20279]: 8.b scrub ok 2024-09-02T10:31:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 921/169983 objects degraded (0.542%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:39 smithi162 ceph-mon[20279]: osdmap e854: 12 total, 12 up, 12 in 2024-09-02T10:31:40.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:40.161 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:40.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:40.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:40.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:40.286 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:40.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:40.388 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:40.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:40.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:40.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:40.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:40.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:40.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:40.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:40.662 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:40.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:40.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:40.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:40.790 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:40.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:40.792 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:40 smithi162 ceph-mon[20279]: pgmap v1954: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 1007/165795 objects degraded (0.607%); 13615/165795 objects misplaced (8.212%); 9.6 MiB/s, 153 objects/s recovering 2024-09-02T10:31:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:40 smithi162 ceph-mon[20279]: osdmap e855: 12 total, 12 up, 12 in 2024-09-02T10:31:40.893 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:40.924 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:40.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:40.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:40 smithi151 ceph-mon[16639]: pgmap v1954: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 1007/165795 objects degraded (0.607%); 13615/165795 objects misplaced (8.212%); 9.6 MiB/s, 153 objects/s recovering 2024-09-02T10:31:40.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:40 smithi151 ceph-mon[16639]: osdmap e855: 12 total, 12 up, 12 in 2024-09-02T10:31:41.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:40 smithi187 ceph-mon[20072]: pgmap v1954: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 1007/165795 objects degraded (0.607%); 13615/165795 objects misplaced (8.212%); 9.6 MiB/s, 153 objects/s recovering 2024-09-02T10:31:41.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:40 smithi187 ceph-mon[20072]: osdmap e855: 12 total, 12 up, 12 in 2024-09-02T10:31:41.026 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:41.058 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:41.059 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:41.160 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:41.187 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:41.187 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:41.288 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:41.311 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:41.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:41.312 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:41.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:41.439 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:41.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:41.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:41.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:41.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:41.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:41.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:41.670 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:41.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:41.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:41.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:41 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:31:41.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:41 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:31:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:41.846 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:41.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:41.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:41 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:31:41.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:41 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:31:41.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:41.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:41.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:41.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:41 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:31:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:41 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:31:42.082 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:42.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:42.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:42.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:42.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:42.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:42.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:42.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:42.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:42.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:42.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:42.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:31:42.380 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:31:42.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:42.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:42.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:42.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:42.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:42.626 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:42.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:42.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:42.678 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:42 smithi162 ceph-mon[20279]: pgmap v1956: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 457 op/s; 1007/164955 objects degraded (0.610%); 13615/164955 objects misplaced (8.254%); 6.6 MiB/s, 105 objects/s recovering 2024-09-02T10:31:42.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:42.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:42.758 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:42.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:42.890 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:42.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:42 smithi151 ceph-mon[16639]: pgmap v1956: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 457 op/s; 1007/164955 objects degraded (0.610%); 13615/164955 objects misplaced (8.254%); 6.6 MiB/s, 105 objects/s recovering 2024-09-02T10:31:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:42 smithi151 ceph-mon[16639]: osdmap e856: 12 total, 12 up, 12 in 2024-09-02T10:31:42.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:42 smithi187 ceph-mon[20072]: pgmap v1956: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.4 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 457 op/s; 1007/164955 objects degraded (0.610%); 13615/164955 objects misplaced (8.254%); 6.6 MiB/s, 105 objects/s recovering 2024-09-02T10:31:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:42 smithi187 ceph-mon[20072]: osdmap e856: 12 total, 12 up, 12 in 2024-09-02T10:31:43.014 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:43.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:42 smithi162 ceph-mon[20279]: osdmap e856: 12 total, 12 up, 12 in 2024-09-02T10:31:43.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:43.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:43.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:43.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:43.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:43.271 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:43.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:43.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:43.401 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:43.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:43.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:43.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:43.525 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:43.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:43.527 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:43.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:43.654 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:43.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:43.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:43.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:43.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:43.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:43.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:43.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:43 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:31:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:43 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:31:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:43 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:31:43.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:43 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:31:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:43 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:31:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:43 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:31:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:43 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:31:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:43 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:31:44.024 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:44.058 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:44.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:43 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:31:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:43 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:31:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:43 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:31:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:43 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:31:44.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:44.183 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:44.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:44.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:44.288 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:44.316 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:44.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:44.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:44.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:44.450 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:44.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:44.553 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:44.577 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:44.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:44.679 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:44.705 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:44.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:44.706 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:44.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:44.831 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:44.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:44.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:44.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:44 smithi151 ceph-mon[16639]: pgmap v1958: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 406 op/s; 1022/161859 objects degraded (0.631%); 13042/161859 objects misplaced (8.058%); 11 MiB/s, 169 objects/s recovering 2024-09-02T10:31:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:44 smithi151 ceph-mon[16639]: osdmap e857: 12 total, 12 up, 12 in 2024-09-02T10:31:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:44 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1022/161859 objects degraded (0.631%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:44 smithi151 ceph-mon[16639]: osdmap e858: 12 total, 12 up, 12 in 2024-09-02T10:31:44.964 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:44.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:44 smithi187 ceph-mon[20072]: pgmap v1958: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 406 op/s; 1022/161859 objects degraded (0.631%); 13042/161859 objects misplaced (8.058%); 11 MiB/s, 169 objects/s recovering 2024-09-02T10:31:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:44 smithi187 ceph-mon[20072]: osdmap e857: 12 total, 12 up, 12 in 2024-09-02T10:31:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:44 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1022/161859 objects degraded (0.631%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:44 smithi187 ceph-mon[20072]: osdmap e858: 12 total, 12 up, 12 in 2024-09-02T10:31:45.066 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:45.088 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:45.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:45.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:44 smithi162 ceph-mon[20279]: pgmap v1958: 47 pgs: 4 active+recovering+undersized+degraded+remapped, 43 active+clean; 3.3 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 406 op/s; 1022/161859 objects degraded (0.631%); 13042/161859 objects misplaced (8.058%); 11 MiB/s, 169 objects/s recovering 2024-09-02T10:31:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:44 smithi162 ceph-mon[20279]: osdmap e857: 12 total, 12 up, 12 in 2024-09-02T10:31:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:44 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1022/161859 objects degraded (0.631%), 4 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:44 smithi162 ceph-mon[20279]: osdmap e858: 12 total, 12 up, 12 in 2024-09-02T10:31:45.190 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:45.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:45.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:45.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:45.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:45.348 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:45.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:45.350 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:45.451 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:45.474 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:45.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:45.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:45.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:45.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:45.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:45.602 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:45.703 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:45.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:45.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:45.738 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:45.839 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:45.895 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:45.901 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:45.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:45.925 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:45 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:31:45.925 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:45 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:31:45.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:31:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:45 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:31:45.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:45 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:31:46.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:45 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:31:46.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:45 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:31:46.031 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:46.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:46.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:46.133 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:46.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:46.160 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:46.231 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:46.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:46.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:46.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:46.406 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:46.431 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:46.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:46.490 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:46.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:46.548 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:31:46.548 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:31:46.565 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:46.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:46.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:46.580 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:31:46.580 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:31:46.613 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:31:46.613 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:31:46.640 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:31:46.640 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:31:46.667 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:46.676 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:31:46.676 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:31:46.702 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:46.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:46.709 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:31:46.709 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:31:46.744 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:31:46.744 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:31:46.769 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:31:46.769 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:31:46.769 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:31:46.769 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:31:46.769 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:31:46.782 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:31:46.782 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:31:46.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:46.820 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:31:46.821 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:31:46.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:46.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:46.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:46.853 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:31:46.853 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:31:46.895 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:31:46.895 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:31:46.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:46 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:31:46.895 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:46 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:31:46.896 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:46 smithi151 ceph-mon[16639]: pgmap v1961: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 363 op/s; 1077/159252 objects degraded (0.676%); 19494/159252 objects misplaced (12.241%); 12 MiB/s, 185 objects/s recovering 2024-09-02T10:31:46.896 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:46 smithi151 ceph-mon[16639]: osdmap e859: 12 total, 12 up, 12 in 2024-09-02T10:31:46.896 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/970376986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:46.927 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:31:46.928 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:31:46.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:46.975 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:46.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:46.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:46 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:31:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:46 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:31:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:46 smithi187 ceph-mon[20072]: pgmap v1961: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 363 op/s; 1077/159252 objects degraded (0.676%); 19494/159252 objects misplaced (12.241%); 12 MiB/s, 185 objects/s recovering 2024-09-02T10:31:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:46 smithi187 ceph-mon[20072]: osdmap e859: 12 total, 12 up, 12 in 2024-09-02T10:31:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/970376986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:47.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:46 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:31:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:46 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:31:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:46 smithi162 ceph-mon[20279]: pgmap v1961: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 363 op/s; 1077/159252 objects degraded (0.676%); 19494/159252 objects misplaced (12.241%); 12 MiB/s, 185 objects/s recovering 2024-09-02T10:31:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:46 smithi162 ceph-mon[20279]: osdmap e859: 12 total, 12 up, 12 in 2024-09-02T10:31:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/970376986' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:47.122 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:47.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:47.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:47.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:47.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:47.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:47.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:47.409 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:47.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:47.511 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:47.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:31:47.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:47.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:47.647 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:47.652 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1756314568' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:31:47.653 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:47 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:31:47.663 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:31:47.676 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:47.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:47.779 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:47.805 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:47.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:47.840 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:47.908 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:47.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1756314568' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:31:47.922 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:47 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:31:47.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:47.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1756314568' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:31:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:47 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:31:48.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:48.073 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:48.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:48.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:48.174 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:48.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:48.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:48.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:48.307 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:48.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:48.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:48.408 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:31:48.408 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:31:48.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:31:48.408 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:31:48.408 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.408 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDeleteReply(6.f e629,586 6:fcd958cb:::benchmark_data_smithi187_102448_object21195:head,579'7484)", 2024-09-02T10:31:48.408 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:22:48.751501+0000", 2024-09-02T10:31:48.408 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 539.653426212, 2024-09-02T10:31:48.408 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.54048170500000003, 2024-09-02T10:31:48.408 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:48.751501+0000", 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:48.751501+0000", 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.409 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:48.751502+0000", 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8100000000000001e-07 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:48.751510+0000", 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9209999999999992e-06 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.291397+0000", 2024-09-02T10:31:48.410 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.53988699900000003 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.291410+0000", 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3529e-05 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.291582+0000", 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000171774 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.411 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:22:49.291983+0000", 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040050099999999999 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:3735 7.0 7:0bf36798:::benchmark_data_smithi187_120712_object3734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e666)", 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:24:14.834852+0000", 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 453.57007501800001, 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21021257700000001, 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.412 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 3735 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834852+0000", 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.413 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834852+0000", 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834856+0000", 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3290000000000004e-06 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834901+0000", 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4601000000000002e-05 2024-09-02T10:31:48.414 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834912+0000", 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0749e-05 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834919+0000", 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1559999999999998e-06 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.834960+0000", 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.138e-05 2024-09-02T10:31:48.415 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.835059+0000", 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8540000000000002e-05 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.835139+0000", 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9920999999999996e-05 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.416 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.835504+0000", 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036490200000000001 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:14.835891+0000", 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038723099999999998 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:15.044999+0000", 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209108608 2024-09-02T10:31:48.417 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:15.045036+0000", 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6634000000000002e-05 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:15.045065+0000", 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8526000000000001e-05 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.418 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:8296 7.0 7:06654a29:::benchmark_data_smithi187_120712_object8295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e667)", 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:24:18.337323+0000", 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 450.06760360300001, 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21776643000000001, 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 8296 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.419 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337323+0000", 2024-09-02T10:31:48.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.420 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.420 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.420 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.420 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337323+0000", 2024-09-02T10:31:48.420 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337324+0000", 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5499999999999996e-07 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337407+0000", 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2157000000000006e-05 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.421 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337420+0000", 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.382e-05 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337426+0000", 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3360000000000002e-06 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337473+0000", 2024-09-02T10:31:48.422 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.757e-05 2024-09-02T10:31:48.423 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.423 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337586+0000", 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113199 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.337673+0000", 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6339000000000001e-05 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.338207+0000", 2024-09-02T10:31:48.424 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00053426400000000001 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.338706+0000", 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00049890899999999996 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.555033+0000", 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21632686100000001 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.425 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.555064+0000", 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1501000000000002e-05 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:24:18.555090+0000", 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.5519000000000001e-05 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:55534 7.0 7:0cb06b0e:::benchmark_data_smithi187_120712_object55533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:31:48.426 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:07.797396+0000", 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 400.60753100300002, 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.247318706, 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55534 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.427 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797396+0000", 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797396+0000", 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797398+0000", 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6619999999999999e-06 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.428 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797481+0000", 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3710999999999999e-05 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797497+0000", 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5536000000000002e-05 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797505+0000", 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3610000000000001e-06 2024-09-02T10:31:48.429 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797566+0000", 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1115e-05 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797709+0000", 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014281 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.430 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.797855+0000", 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014629700000000001 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.799694+0000", 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018388269999999999 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800428+0000", 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00073367199999999999 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.431 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044600+0000", 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.244171631 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044664+0000", 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4329e-05 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044715+0000", 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0754999999999998e-05 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.432 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:55537 7.0 7:0ece8a38:::benchmark_data_smithi187_120712_object55536:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:07.800289+0000", 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 400.60463758700001, 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.244576865, 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:48.433 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 55537 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800289+0000", 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800289+0000", 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.434 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800291+0000", 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6649999999999999e-06 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800356+0000", 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4936000000000003e-05 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800368+0000", 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1798e-05 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.435 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800375+0000", 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3329999999999999e-06 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800454+0000", 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8854000000000001e-05 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800548+0000", 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4458999999999998e-05 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.436 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.800631+0000", 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2812000000000006e-05 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.802282+0000", 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001650861 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:07.802849+0000", 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00056718700000000003 2024-09-02T10:31:48.437 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044778+0000", 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.241928643 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044830+0000", 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1833000000000003e-05 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:08.044866+0000", 2024-09-02T10:31:48.438 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6483999999999998e-05 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:116084 7.0 7:07eb9730:::benchmark_data_smithi187_120712_object32142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e731)", 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:18.919750+0000", 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 329.485176749, 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222370436, 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.439 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 116084 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919750+0000", 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.440 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919750+0000", 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919752+0000", 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.59e-06 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919764+0000", 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2203e-05 2024-09-02T10:31:48.441 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919779+0000", 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4457999999999999e-05 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.919785+0000", 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8979999999999999e-06 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.922669+0000", 2024-09-02T10:31:48.442 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0028836299999999999 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.922803+0000", 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000133972 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.922916+0000", 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113479 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.443 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.926242+0000", 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0033256919999999999 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.926412+0000", 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016997299999999999 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.142054+0000", 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21564232799999999 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.444 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.142092+0000", 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7494999999999999e-05 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.142121+0000", 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8717999999999998e-05 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.445 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:116098 7.0 7:07a476f9:::benchmark_data_smithi187_120712_object32156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e731)", 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:18.931907+0000", 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 329.47301983400001, 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22223031300000001, 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 116098 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.446 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931907+0000", 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931907+0000", 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931909+0000", 2024-09-02T10:31:48.447 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3039999999999999e-06 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931918+0000", 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2239999999999995e-06 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931939+0000", 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1155e-05 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.448 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.931949+0000", 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6979999999999993e-06 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.932018+0000", 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9766999999999998e-05 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.932138+0000", 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119337 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.449 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.932252+0000", 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114528 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.936161+0000", 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003908758 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:18.936253+0000", 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2379000000000004e-05 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.450 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.154031+0000", 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21777723500000001 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.154088+0000", 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7559000000000002e-05 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:19.154137+0000", 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9369000000000001e-05 2024-09-02T10:31:48.451 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:135858 7.10 7:0b5a73a5:::benchmark_data_smithi187_120712_object51916:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:41.733794+0000", 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 306.67113319499998, 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.072650061, 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.452 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135858 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733794+0000", 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733794+0000", 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.453 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733795+0000", 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6080000000000001e-06 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733802+0000", 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1120000000000001e-06 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733819+0000", 2024-09-02T10:31:48.454 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7064000000000001e-05 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733828+0000", 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2900000000000008e-06 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.946794+0000", 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21296567199999999 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.946804+0000", 2024-09-02T10:31:48.455 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0910999999999999e-05 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.806352+0000", 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.85954753399999995 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.806444+0000", 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.187e-05 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.456 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:136917 7.0 7:009dde76:::benchmark_data_smithi187_120712_object52975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:43.760124+0000", 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 304.64480324200002, 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33985228200000001, 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136917 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.457 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760124+0000", 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760124+0000", 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760126+0000", 2024-09-02T10:31:48.458 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8610000000000001e-06 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760131+0000", 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.575e-06 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760144+0000", 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2942000000000001e-05 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.459 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760150+0000", 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1179999999999997e-06 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760198+0000", 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7546000000000002e-05 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760283+0000", 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5561000000000003e-05 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.460 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.760352+0000", 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8633000000000005e-05 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762399+0000", 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020470330000000002 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762673+0000", 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00027366000000000001 2024-09-02T10:31:48.461 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099886+0000", 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33721340999999999 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099938+0000", 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1635000000000003e-05 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099976+0000", 2024-09-02T10:31:48.462 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8308e-05 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:136917 7.0 e743/741 ondisk, result = 0)", 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:43.762433+0000", 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 304.64249387299998, 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33754895299999998, 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.463 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762433+0000", 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762433+0000", 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762434+0000", 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.02e-06 2024-09-02T10:31:48.464 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.762440+0000", 2024-09-02T10:31:48.465 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0449999999999997e-06 2024-09-02T10:31:48.465 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.465 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.465 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.465 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099808+0000", 2024-09-02T10:31:48.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:48.467 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33736824799999998 2024-09-02T10:31:48.467 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.467 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.467 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.467 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099816+0000", 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1799999999999999e-06 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099871+0000", 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5730999999999999e-05 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099881+0000", 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5759999999999995e-06 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.468 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.099982+0000", 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000101153 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:48920 8.8 8:1673ed4b:::benchmark_data_smithi187_136059_object48919:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e813)", 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:29:33.671380+0000", 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 134.733547034, 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209033422, 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.469 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 48920 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671380+0000", 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.470 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671380+0000", 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671381+0000", 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.09e-07 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671439+0000", 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8641999999999999e-05 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.471 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671450+0000", 2024-09-02T10:31:48.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0216e-05 2024-09-02T10:31:48.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.472 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.472 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671454+0000", 2024-09-02T10:31:48.472 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3100000000000002e-06 2024-09-02T10:31:48.472 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.472 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671518+0000", 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3646999999999996e-05 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671637+0000", 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119815 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.671809+0000", 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000171776 2024-09-02T10:31:48.473 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.673208+0000", 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013985340000000001 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.677308+0000", 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041007040000000002 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880310+0000", 2024-09-02T10:31:48.474 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.203001297 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880365+0000", 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5544000000000002e-05 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:29:33.880413+0000", 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8127999999999997e-05 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.475 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:96546 8.8 8:1d54d21b:::benchmark_data_smithi187_136059_object96545:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e832)", 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:30:19.487805+0000", 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 88.917122159000002, 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20920671900000001, 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 96546 2024-09-02T10:31:48.476 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.487805+0000", 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.487805+0000", 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.477 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.487806+0000", 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0160000000000001e-06 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.487850+0000", 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4279000000000001e-05 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.487861+0000", 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0895e-05 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.478 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.487867+0000", 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5010000000000004e-06 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.487918+0000", 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1019000000000001e-05 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.488063+0000", 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000145093 2024-09-02T10:31:48.479 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.488203+0000", 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000140166 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.491385+0000", 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031824940000000001 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.491712+0000", 2024-09-02T10:31:48.480 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000326254 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.696906+0000", 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20519488299999999 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.696962+0000", 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5182000000000001e-05 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.481 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:19.697012+0000", 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9937e-05 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:109455 8.8 8:14dc587f:::benchmark_data_smithi187_136059_object4174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e836)", 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:30:34.214622+0000", 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.190304655999995, 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26031262100000002, 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.482 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 109455 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214622+0000", 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.483 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214622+0000", 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214624+0000", 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.345e-06 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214629+0000", 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2159999999999997e-06 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.484 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214641+0000", 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2097e-05 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214648+0000", 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1969999999999999e-06 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214704+0000", 2024-09-02T10:31:48.485 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5408000000000002e-05 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214841+0000", 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013739399999999999 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214978+0000", 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013713199999999999 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.486 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218166+0000", 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003187722 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218948+0000", 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00078261400000000005 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474854+0000", 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25590502999999998 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.487 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474907+0000", 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3647999999999999e-05 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474935+0000", 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7818e-05 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.488 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27559.0:109455 8.8 e836/792 ondisk, result = 0)", 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:30:34.218001+0000", 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.186925841000004, 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25693914899999998, 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218001+0000", 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.489 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218001+0000", 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218002+0000", 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5099999999999998e-07 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218006+0000", 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3680000000000004e-06 2024-09-02T10:31:48.490 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474642+0000", 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25663574099999997 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474658+0000", 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5990000000000001e-05 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474828+0000", 2024-09-02T10:31:48.491 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017020300000000001 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474844+0000", 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5889999999999999e-05 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474940+0000", 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6106000000000004e-05 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.492 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:109505 8.8 8:1da32497:::benchmark_data_smithi187_136059_object4224:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e836)", 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:30:34.271211+0000", 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.133716046000004, 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209867728, 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 109505 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.493 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.271211+0000", 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.271211+0000", 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.271212+0000", 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3710000000000001e-06 2024-09-02T10:31:48.494 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.271219+0000", 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4989999999999999e-06 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.271252+0000", 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3148000000000002e-05 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.271261+0000", 2024-09-02T10:31:48.495 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5490000000000006e-06 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.271347+0000", 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6951000000000001e-05 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.271468+0000", 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120489 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.496 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.271589+0000", 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121173 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.274520+0000", 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029307869999999998 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.275112+0000", 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059222300000000001 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.497 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.480985+0000", 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20587285 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.481038+0000", 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3032000000000003e-05 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.481079+0000", 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0655999999999998e-05 2024-09-02T10:31:48.498 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:134989 8.8 8:15a31c2e:::benchmark_data_smithi187_136059_object29708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:01.997645+0000", 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 46.407281922999999, 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48778658600000002, 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:31:48.499 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 134989 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997645+0000", 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997645+0000", 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.500 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997646+0000", 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1480000000000001e-06 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997650+0000", 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0799999999999999e-06 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997662+0000", 2024-09-02T10:31:48.501 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997668+0000", 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4060000000000004e-06 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997739+0000", 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1211999999999995e-05 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.502 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997836+0000", 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6628999999999999e-05 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997908+0000", 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2268000000000007e-05 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001365+0000", 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034576619999999998 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.503 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001726+0000", 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000360997 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485369+0000", 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48364271800000003 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485401+0000", 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2274000000000003e-05 2024-09-02T10:31:48.504 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485432+0000", 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0244999999999999e-05 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27559.0:134989 8.8 e843/792 ondisk, result = 0)", 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:02.001592+0000", 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 46.403335278, 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48384515900000002, 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.505 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001592+0000", 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001592+0000", 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.506 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001592+0000", 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5700000000000002e-07 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001599+0000", 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.669e-06 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485331+0000", 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48373196800000001 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.507 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485335+0000", 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3059999999999999e-06 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485362+0000", 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6495e-05 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485366+0000", 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3889999999999998e-06 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.508 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485437+0000", 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0674999999999997e-05 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:149583 8.8 8:1df6255f:::benchmark_data_smithi187_136059_object44302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e851)", 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:27.306490+0000", 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.098437221000001, 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23605482999999999, 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.509 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 149583 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306490+0000", 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.510 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306490+0000", 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306490+0000", 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4700000000000002e-07 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306493+0000", 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3329999999999999e-06 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.511 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306500+0000", 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5700000000000004e-06 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306504+0000", 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5710000000000002e-06 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306550+0000", 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5964999999999998e-05 2024-09-02T10:31:48.512 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306648+0000", 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8740000000000007e-05 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306746+0000", 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7916000000000002e-05 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.311648+0000", 2024-09-02T10:31:48.513 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004901954 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.336249+0000", 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02460066 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.542476+0000", 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20622670900000001 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.514 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.542514+0000", 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8563000000000003e-05 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.542545+0000", 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0402000000000001e-05 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:149595 8.8 8:173adb21:::benchmark_data_smithi187_136059_object44314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e851)", 2024-09-02T10:31:48.515 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:27.331797+0000", 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 21.073129933000001, 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21089745200000001, 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 149595 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.516 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.331797+0000", 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.331797+0000", 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.517 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.331799+0000", 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9800000000000001e-06 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.331806+0000", 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1119999999999998e-06 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.331820+0000", 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4127e-05 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.518 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.331826+0000", 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1680000000000001e-06 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.331886+0000", 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9144999999999997e-05 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.332018+0000", 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000132096 2024-09-02T10:31:48.519 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.332135+0000", 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000117386 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.336382+0000", 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042466800000000001 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.520 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.386081+0000", 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.049699701999999998 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.542635+0000", 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15655359799999999 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.542667+0000", 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2387e-05 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.521 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.542695+0000", 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7070999999999999e-05 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(8.d 851/844 [PushOp(8:bc6811a2:::benchmark_data_smithi187_136059_object50100:head, version: 814'3129, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:bc6811a2:::benchmark_data_smithi187_136059_object50100:head@814'3129, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:35.080864+0000", 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 13.324062674, 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21813774, 2024-09-02T10:31:48.522 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.080864+0000", 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.080864+0000", 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.523 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.080865+0000", 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5000000000000002e-07 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.080900+0000", 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5438999999999997e-05 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:31:48.524 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.295229+0000", 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.214328291 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.295246+0000", 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7334999999999999e-05 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.295363+0000", 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116778 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.525 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.295375+0000", 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.26e-05 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.299002+0000", 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036266470000000002 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:31:48.526 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:31:48.527 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:31:48.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:48.551 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:48.652 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:48.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:48.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:48.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:48.804 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:48.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:48.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:48.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:48.929 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:48.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:48 smithi151 ceph-mon[16639]: pgmap v1963: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 1077/156318 objects degraded (0.689%); 19051/156318 objects misplaced (12.187%); 12 MiB/s, 189 objects/s recovering 2024-09-02T10:31:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:48 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:48 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:31:48.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:48 smithi151 ceph-mon[16639]: osdmap e860: 12 total, 12 up, 12 in 2024-09-02T10:31:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:48 smithi187 ceph-mon[20072]: pgmap v1963: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 1077/156318 objects degraded (0.689%); 19051/156318 objects misplaced (12.187%); 12 MiB/s, 189 objects/s recovering 2024-09-02T10:31:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:48 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:48 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:31:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:48 smithi187 ceph-mon[20072]: osdmap e860: 12 total, 12 up, 12 in 2024-09-02T10:31:49.030 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:49.054 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:49.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:48 smithi162 ceph-mon[20279]: pgmap v1963: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.2 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 468 op/s; 1077/156318 objects degraded (0.689%); 19051/156318 objects misplaced (12.187%); 12 MiB/s, 189 objects/s recovering 2024-09-02T10:31:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:48 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:31:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:48 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:31:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:48 smithi162 ceph-mon[20279]: osdmap e860: 12 total, 12 up, 12 in 2024-09-02T10:31:49.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:49.184 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:49.184 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:49.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:49.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:49.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:49.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:49.447 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:49.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:49.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:49.577 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:49.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:49.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:49.704 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:49.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:49.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:49.807 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:49.835 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:49.837 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:49.867 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:31:49.927 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:31:49.937 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:49.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:49 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1077/156318 objects degraded (0.689%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:49.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:49 smithi151 ceph-mon[16639]: osdmap e861: 12 total, 12 up, 12 in 2024-09-02T10:31:49.963 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:49.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:49 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1077/156318 objects degraded (0.689%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:49 smithi187 ceph-mon[20072]: osdmap e861: 12 total, 12 up, 12 in 2024-09-02T10:31:50.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:50.091 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:50.092 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:49 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1077/156318 objects degraded (0.689%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:49 smithi162 ceph-mon[20279]: osdmap e861: 12 total, 12 up, 12 in 2024-09-02T10:31:50.165 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:50.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:50.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:50.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:50.297 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:31:50.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:50.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:50.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:50.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:50.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:50.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:50.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:50.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:31:50.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:50.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:50.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:50.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:50.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:50 smithi151 ceph-mon[16639]: pgmap v1966: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 1153/152241 objects degraded (0.757%); 18479/152241 objects misplaced (12.138%); 10 MiB/s, 161 objects/s recovering 2024-09-02T10:31:50.695 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:31:50.695 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:31:50.695 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 0 2024-09-02T10:31:50.695 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:31:50.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:50.792 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:50.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:50 smithi162 ceph-mon[20279]: pgmap v1966: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 1153/152241 objects degraded (0.757%); 18479/152241 objects misplaced (12.138%); 10 MiB/s, 161 objects/s recovering 2024-09-02T10:31:50.889 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:31:50.889 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:31:50.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:31:50.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:50.916 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:50.917 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:50 smithi187 ceph-mon[20072]: pgmap v1966: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 497 op/s; 1153/152241 objects degraded (0.757%); 18479/152241 objects misplaced (12.138%); 10 MiB/s, 161 objects/s recovering 2024-09-02T10:31:51.020 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:51.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:51.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:51.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:51.181 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:51.219 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:51.221 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:51.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:51.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:51.359 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:51.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:51.461 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:51.462 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:31:51.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:51.488 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:51.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:51.614 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:51.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:51.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:51.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:51.744 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:51.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:51.771 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:51.846 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:51.871 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:51.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:51.872 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:51.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:51.997 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:51.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:52.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:52.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:52.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:52.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:52.259 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:52.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:52.260 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:52.312 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:31:52.313 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:31:52.313 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:31:52.313 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:31:52.313 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:31:52.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:52.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:52.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:52.519 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:52.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:52.553 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:52.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:52.654 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:52.677 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:52.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:52.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:52.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:52.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:52.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:52.906 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:52.932 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:52.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:52.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:52 smithi151 ceph-mon[16639]: pgmap v1967: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 444 op/s; 1153/151251 objects degraded (0.762%); 18479/151251 objects misplaced (12.217%); 8.7 MiB/s, 140 objects/s recovering 2024-09-02T10:31:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:52 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:31:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3874098150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:52 smithi187 ceph-mon[20072]: pgmap v1967: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 444 op/s; 1153/151251 objects degraded (0.762%); 18479/151251 objects misplaced (12.217%); 8.7 MiB/s, 140 objects/s recovering 2024-09-02T10:31:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:52 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:31:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3874098150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:53.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:53.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:53.079 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:53.080 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:52 smithi162 ceph-mon[20279]: pgmap v1967: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.1 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 444 op/s; 1153/151251 objects degraded (0.762%); 18479/151251 objects misplaced (12.217%); 8.7 MiB/s, 140 objects/s recovering 2024-09-02T10:31:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:52 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:31:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3874098150' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:31:53.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:53.203 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:53.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:53.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:53.305 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:53.331 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:53.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:53.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:53.455 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:53.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:53.557 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:53.582 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:53.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:53.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:53.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:53.707 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:53.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:53.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:31:53.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:31:53.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:31:53.765 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:31:53.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:31:53.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:31:53.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:31:53.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:31:53.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:31:53.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:31:53.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:31:53.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:31:53.767 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:31:53.767 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:31:53.767 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:31:53.767 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:31:53.767 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:31:53.767 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:31:53.767 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:31:53.768 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:31:53.768 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:31:53.768 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:31:53.768 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:31:53.768 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:31:53.808 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:53.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:53.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:53.914 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:31:53.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:53.981 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:53.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:54.083 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:54.108 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:54.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:54.210 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:54.230 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:54.234 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:54.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:54.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:54.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:54.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:54.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:54.489 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:54.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:54.591 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:54.625 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:54.626 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:54.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:54.753 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:54.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:54.774 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:31:54.774 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:31:54.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:54.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:54.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:54.890 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:54 smithi151 ceph-mon[16639]: pgmap v1968: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 1153/146706 objects degraded (0.786%); 18036/146706 objects misplaced (12.294%); 10 MiB/s, 162 objects/s recovering 2024-09-02T10:31:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:54 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1153/146706 objects degraded (0.786%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:54 smithi151 ceph-mon[16639]: osdmap e862: 12 total, 12 up, 12 in 2024-09-02T10:31:54.991 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:54 smithi187 ceph-mon[20072]: pgmap v1968: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 1153/146706 objects degraded (0.786%); 18036/146706 objects misplaced (12.294%); 10 MiB/s, 162 objects/s recovering 2024-09-02T10:31:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:54 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1153/146706 objects degraded (0.786%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:54 smithi187 ceph-mon[20072]: osdmap e862: 12 total, 12 up, 12 in 2024-09-02T10:31:55.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:55.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:54 smithi162 ceph-mon[20279]: pgmap v1968: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 3.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 1153/146706 objects degraded (0.786%); 18036/146706 objects misplaced (12.294%); 10 MiB/s, 162 objects/s recovering 2024-09-02T10:31:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:54 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1153/146706 objects degraded (0.786%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:31:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:54 smithi162 ceph-mon[20279]: osdmap e862: 12 total, 12 up, 12 in 2024-09-02T10:31:55.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:55.158 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:55.159 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:55.260 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:55.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:55.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:55.295 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:55.397 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:55.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:55.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:55.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:55.530 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:55.559 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:55.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:55.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:31:55.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:55.689 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:55.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:55.792 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:55.820 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:55.821 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:55.922 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:55.954 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:55.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:55.993 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:56.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:56.080 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:56.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:56.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:55 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:31:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:55 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:31:56.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:56.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:31:56.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:55 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:31:56.206 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:56.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:56.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:56.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:55 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:31:56.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:55 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:31:56.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:56.331 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:56.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:56.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:56.433 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:56.466 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:56.467 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:56.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:56.573 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:31:56.574 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:31:56.601 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:56.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:56.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:56.729 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:56.729 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:56.830 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:56.856 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:56.857 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:56.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:56.990 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:56.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:57.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:56 smithi162 ceph-mon[20279]: pgmap v1970: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 529 op/s; 1250/143298 objects degraded (0.872%); 17453/143298 objects misplaced (12.180%); 9.8 MiB/s, 157 objects/s recovering 2024-09-02T10:31:57.119 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:57.121 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:56 smithi151 ceph-mon[16639]: pgmap v1970: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 529 op/s; 1250/143298 objects degraded (0.872%); 17453/143298 objects misplaced (12.180%); 9.8 MiB/s, 157 objects/s recovering 2024-09-02T10:31:57.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:31:57.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:57.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:31:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:56 smithi187 ceph-mon[20072]: pgmap v1970: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.9 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 529 op/s; 1250/143298 objects degraded (0.872%); 17453/143298 objects misplaced (12.180%); 9.8 MiB/s, 157 objects/s recovering 2024-09-02T10:31:57.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:57.375 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:31:57.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:57.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:57.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:57.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:57.511 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:57.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:57.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:57.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:57.647 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:57.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:57.700 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:31:57.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:31:57.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:57.777 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:57.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:31:57.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:57.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:57.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:57.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:58.010 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:58.039 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:58.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:58.040 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:58.141 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:58.169 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:58.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:58.271 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:31:58.293 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:31:58.293 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:31:58.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:58.297 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:58.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:31:58.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:58.430 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:58.431 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:58.532 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:58.560 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:58.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:58.561 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:58.662 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:31:58.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:58.699 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:31:58.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:58.826 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:58.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:58.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:31:58.954 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:58.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:31:59.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:59.083 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:59.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:31:59.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:58 smithi162 ceph-mon[20279]: pgmap v1971: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 477 op/s; 1250/140448 objects degraded (0.890%); 17027/140448 objects misplaced (12.123%); 9.6 MiB/s, 154 objects/s recovering 2024-09-02T10:31:59.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:31:59.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:58 smithi151 ceph-mon[16639]: pgmap v1971: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 477 op/s; 1250/140448 objects degraded (0.890%); 17027/140448 objects misplaced (12.123%); 9.6 MiB/s, 154 objects/s recovering 2024-09-02T10:31:59.220 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:59.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:31:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:58 smithi187 ceph-mon[20072]: pgmap v1971: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.9 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 477 op/s; 1250/140448 objects degraded (0.890%); 17027/140448 objects misplaced (12.123%); 9.6 MiB/s, 154 objects/s recovering 2024-09-02T10:31:59.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:31:59.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:59.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:59.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:31:59.446 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:31:59.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:59.475 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:31:59.475 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:31:59.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:31:59.601 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:59.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:59.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:31:59.703 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:31:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:59.726 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:59.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:31:59.827 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:31:59.851 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:31:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:59.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:31:59.953 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:31:59.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:31:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:31:59.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1250/140448 objects degraded (0.890%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:32:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:31:59 smithi162 ceph-mon[20279]: osdmap e863: 12 total, 12 up, 12 in 2024-09-02T10:32:00.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:00.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:00.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:00.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1250/140448 objects degraded (0.890%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:32:00.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:31:59 smithi151 ceph-mon[16639]: osdmap e863: 12 total, 12 up, 12 in 2024-09-02T10:32:00.225 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:00.256 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:00.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:00.262 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1250/140448 objects degraded (0.890%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:32:00.262 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:31:59 smithi187 ceph-mon[20072]: osdmap e863: 12 total, 12 up, 12 in 2024-09-02T10:32:00.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:00.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:00.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:00.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:00.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:00.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:00.625 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:00.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:00.654 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:00.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:00.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:00.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:00.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:00.920 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:00.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:01.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:01.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:01.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:01.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:00 smithi162 ceph-mon[20279]: pgmap v1973: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 1340/136092 objects degraded (0.985%); 16472/136092 objects misplaced (12.104%); 13 MiB/s, 205 objects/s recovering 2024-09-02T10:32:01.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:01.171 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:01.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:01.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:00 smithi151 ceph-mon[16639]: pgmap v1973: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 1340/136092 objects degraded (0.985%); 16472/136092 objects misplaced (12.104%); 13 MiB/s, 205 objects/s recovering 2024-09-02T10:32:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:00 smithi187 ceph-mon[20072]: pgmap v1973: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.8 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 631 op/s; 1340/136092 objects degraded (0.985%); 16472/136092 objects misplaced (12.104%); 13 MiB/s, 205 objects/s recovering 2024-09-02T10:32:01.273 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:01.299 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:01.299 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:01.401 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:01.432 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:01.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:01.535 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:01.570 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:01.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:01.672 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:01.697 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:01.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:01.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:01.799 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:01.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:32:01.827 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:01.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:01.929 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:01.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:01.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:01.958 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:02.045 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:32:02.059 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:02.075 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:32:02.083 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:02.084 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:32:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:32:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:01 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:32:02.136 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:32:02.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:32:02.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:32:02.177 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:01 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:32:02.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:02.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:02.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:02.210 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:32:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:32:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:01 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:32:02.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:02.333 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:02.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:02.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:02.466 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:02.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:02.568 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:02.591 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:02.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:02.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:02.716 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:02.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:02.818 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:02.843 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:02.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:02.844 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:02.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:02.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:02.975 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:03.076 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:03.105 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:03.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:03.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:03.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:03.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:02 smithi187 ceph-mon[20072]: pgmap v1974: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 484 op/s; 1340/135078 objects degraded (0.992%); 16472/135078 objects misplaced (12.194%); 9.4 MiB/s, 150 objects/s recovering 2024-09-02T10:32:03.332 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:03.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:02 smithi162 ceph-mon[20279]: pgmap v1974: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 484 op/s; 1340/135078 objects degraded (0.992%); 16472/135078 objects misplaced (12.194%); 9.4 MiB/s, 150 objects/s recovering 2024-09-02T10:32:03.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:03.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:03.435 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:02 smithi151 ceph-mon[16639]: pgmap v1974: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 484 op/s; 1340/135078 objects degraded (0.992%); 16472/135078 objects misplaced (12.194%); 9.4 MiB/s, 150 objects/s recovering 2024-09-02T10:32:03.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:03.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:03.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:03.633 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:03.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:03.662 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:03.748 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:32:03.749 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:32:03.763 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:03.802 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:03.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:03.903 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:03.943 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:03.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:03.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:03.965 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:32:04.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:04.072 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:04.073 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:04.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:04.199 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:04.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:03 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:32:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:03 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:32:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:03 smithi187 ceph-mon[20072]: osdmap e864: 12 total, 12 up, 12 in 2024-09-02T10:32:04.301 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:04.326 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:04.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:03 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:32:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:03 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:32:04.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:03 smithi162 ceph-mon[20279]: osdmap e864: 12 total, 12 up, 12 in 2024-09-02T10:32:04.428 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:04.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:03 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:32:04.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:04 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:32:04.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:04 smithi151 ceph-mon[16639]: osdmap e864: 12 total, 12 up, 12 in 2024-09-02T10:32:04.452 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:04.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:04.529 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:32:04.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:04.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:04.598 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:04.699 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:04.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:04.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:04.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:04.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:04.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:04.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:04.888 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:04.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:04.995 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:04.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:05.097 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:05.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:04 smithi151 ceph-mon[16639]: pgmap v1976: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 1340/130335 objects degraded (1.028%); 15991/130335 objects misplaced (12.269%); 10 MiB/s, 160 objects/s recovering 2024-09-02T10:32:05.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1340/130335 objects degraded (1.028%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:32:05.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:04 smithi151 ceph-mon[16639]: osdmap e865: 12 total, 12 up, 12 in 2024-09-02T10:32:05.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:05.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:05.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:05.223 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:04 smithi187 ceph-mon[20072]: pgmap v1976: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 1340/130335 objects degraded (1.028%); 15991/130335 objects misplaced (12.269%); 10 MiB/s, 160 objects/s recovering 2024-09-02T10:32:05.223 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1340/130335 objects degraded (1.028%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:32:05.223 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:04 smithi187 ceph-mon[20072]: osdmap e865: 12 total, 12 up, 12 in 2024-09-02T10:32:05.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:05.247 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:05.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:04 smithi162 ceph-mon[20279]: pgmap v1976: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.7 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 1340/130335 objects degraded (1.028%); 15991/130335 objects misplaced (12.269%); 10 MiB/s, 160 objects/s recovering 2024-09-02T10:32:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1340/130335 objects degraded (1.028%), 5 pgs degraded (PG_DEGRADED) 2024-09-02T10:32:05.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:04 smithi162 ceph-mon[20279]: osdmap e865: 12 total, 12 up, 12 in 2024-09-02T10:32:05.348 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:05.370 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:05.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:05.473 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:05.502 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:32:05.502 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:32:05.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:05.503 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:05.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:05.628 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:05.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:05.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:05.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:05.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:05.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:05.869 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:05.907 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:05.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:06.009 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:06.034 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:06.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:06.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:06.136 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:06.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:06.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:06.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:06.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:06.301 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:06.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:06.322 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:32:06.403 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:06.434 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:06.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:06.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:06.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:06.568 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:06.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:06.570 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:06.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:06.686 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:06.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:06.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:06.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:06.795 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:32:06.797 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:32:06.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:06.844 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:06.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:06.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:06.973 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:06.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:07.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:07.101 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:07.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:07.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:32:07.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:32:07.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:06 smithi187 ceph-mon[20072]: pgmap v1978: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 499 op/s; 1481/126741 objects degraded (1.169%); 15399/126741 objects misplaced (12.150%); 8.5 MiB/s, 136 objects/s recovering 2024-09-02T10:32:07.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:06 smithi187 ceph-mon[20072]: osdmap e866: 12 total, 12 up, 12 in 2024-09-02T10:32:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:32:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:32:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:06 smithi151 ceph-mon[16639]: pgmap v1978: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 499 op/s; 1481/126741 objects degraded (1.169%); 15399/126741 objects misplaced (12.150%); 8.5 MiB/s, 136 objects/s recovering 2024-09-02T10:32:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:06 smithi151 ceph-mon[16639]: osdmap e866: 12 total, 12 up, 12 in 2024-09-02T10:32:07.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:07.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:07.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:07.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:07.316 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:32:07.316 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:32:07.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:32:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:32:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:06 smithi162 ceph-mon[20279]: pgmap v1978: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.6 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 499 op/s; 1481/126741 objects degraded (1.169%); 15399/126741 objects misplaced (12.150%); 8.5 MiB/s, 136 objects/s recovering 2024-09-02T10:32:07.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:06 smithi162 ceph-mon[20279]: osdmap e866: 12 total, 12 up, 12 in 2024-09-02T10:32:07.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:07.370 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:07.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:07.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:07.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:07.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:07.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:07.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:07.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:07.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:07.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:07.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:07.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:07.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:07.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:07.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:07.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:08.021 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:08.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:08.122 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:08.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:08.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:08.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:07 smithi187 ceph-mon[20072]: from='client.18684 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:32:08.274 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:08.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:07 smithi162 ceph-mon[20279]: from='client.18684 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:32:08.376 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:08.398 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:08.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:08.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:07 smithi151 ceph-mon[16639]: from='client.18684 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:32:08.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:08.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:08.530 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:08.632 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:08.654 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:08.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:08.756 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:08.777 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:08.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:08.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:08.879 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:08.917 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:08.918 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:09.019 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:09.050 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:09.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:09.051 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:09.055 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:32:09.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:09.182 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:09.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:08 smithi187 ceph-mon[20072]: pgmap v1980: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 1481/123852 objects degraded (1.196%); 14943/123852 objects misplaced (12.065%); 14 MiB/s, 217 objects/s recovering 2024-09-02T10:32:09.285 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:09.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:09.318 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:09.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:08 smithi162 ceph-mon[20279]: pgmap v1980: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 1481/123852 objects degraded (1.196%); 14943/123852 objects misplaced (12.065%); 14 MiB/s, 217 objects/s recovering 2024-09-02T10:32:09.415 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:09.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:09.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:08 smithi151 ceph-mon[16639]: pgmap v1980: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 623 op/s; 1481/123852 objects degraded (1.196%); 14943/123852 objects misplaced (12.065%); 14 MiB/s, 217 objects/s recovering 2024-09-02T10:32:09.452 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:09.454 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:09.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:09.583 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:09.584 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:09.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:09.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:09.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:09.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:09.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:09.847 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:09.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:09.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:09.975 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:09.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:09.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:10.032 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:32:10.032 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:32:10.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:10.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:10.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:10.202 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:10.224 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:10.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1481/123852 objects degraded (1.196%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:09 smithi187 ceph-mon[20072]: osdmap e867: 12 total, 12 up, 12 in 2024-09-02T10:32:10.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1481/123852 objects degraded (1.196%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:09 smithi162 ceph-mon[20279]: osdmap e867: 12 total, 12 up, 12 in 2024-09-02T10:32:10.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:10.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:10.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1481/123852 objects degraded (1.196%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:10.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:10 smithi151 ceph-mon[16639]: osdmap e867: 12 total, 12 up, 12 in 2024-09-02T10:32:10.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:10.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:10.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:10.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:10.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:10.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:10.722 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:10.746 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:10.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:10.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:32:10.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:10.871 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:10.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:10.974 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:11.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:11.013 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:11.115 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:11.139 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:11.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:11.205 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:11.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:10 smithi187 ceph-mon[20072]: pgmap v1982: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 1580/119688 objects degraded (1.320%); 14348/119688 objects misplaced (11.988%); 13 MiB/s, 204 objects/s recovering 2024-09-02T10:32:11.265 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:11.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:11.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:10 smithi162 ceph-mon[20279]: pgmap v1982: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 1580/119688 objects degraded (1.320%); 14348/119688 objects misplaced (11.988%); 13 MiB/s, 204 objects/s recovering 2024-09-02T10:32:11.367 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:11.398 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:11.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:11.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:10 smithi151 ceph-mon[16639]: pgmap v1982: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 591 op/s; 1580/119688 objects degraded (1.320%); 14348/119688 objects misplaced (11.988%); 13 MiB/s, 204 objects/s recovering 2024-09-02T10:32:11.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:11.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:11.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:11.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:11.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:11.655 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:11.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:11.748 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:32:11.748 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:32:11.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:11.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:11.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:11.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:11.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:11.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:11.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:12.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:12.042 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:12.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:12.043 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:12.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:12.171 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:12.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:12.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:12.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:12.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:12.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:12.412 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:12.436 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:12.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:12.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:32:12.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:12.587 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:12.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:12.588 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:12.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:12.711 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:12.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:12.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:12.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:12.838 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:12.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:12.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:12.891 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:12.940 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:12.969 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:12.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:13.070 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:13.101 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:13.102 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:13.203 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:13.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:13.234 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:13.247 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:12 smithi187 ceph-mon[20072]: pgmap v1983: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 1580/118650 objects degraded (1.332%); 14348/118650 objects misplaced (12.093%); 7.3 MiB/s, 118 objects/s recovering 2024-09-02T10:32:13.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:13.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:12 smithi162 ceph-mon[20279]: pgmap v1983: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 1580/118650 objects degraded (1.332%); 14348/118650 objects misplaced (12.093%); 7.3 MiB/s, 118 objects/s recovering 2024-09-02T10:32:13.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:13.364 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:13.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:12 smithi151 ceph-mon[16639]: pgmap v1983: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 1580/118650 objects degraded (1.332%); 14348/118650 objects misplaced (12.093%); 7.3 MiB/s, 118 objects/s recovering 2024-09-02T10:32:13.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:13.488 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:13.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:13.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:13.613 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:32:13.613 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:32:13.620 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:13.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:13.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:13.749 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:13.749 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:13.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:13.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:13.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:13.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:14.011 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:14.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:14.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:14.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:14.142 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:14.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:14.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:14.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:14.273 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:14.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:14.397 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:14.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:14.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:14.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:14.538 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:14.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:14.639 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:14.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:14.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:14.671 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:14.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:14.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:14.803 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:14.904 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:14.929 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:14.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:15.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:15.060 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:15.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:15.161 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:15.202 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:15.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:14 smithi187 ceph-mon[20072]: pgmap v1984: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 1580/114063 objects degraded (1.385%); 13899/114063 objects misplaced (12.185%); 10 MiB/s, 163 objects/s recovering 2024-09-02T10:32:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1580/114063 objects degraded (1.385%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:15.305 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:15.335 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:15.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:15.336 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:14 smithi162 ceph-mon[20279]: pgmap v1984: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 1580/114063 objects degraded (1.385%); 13899/114063 objects misplaced (12.185%); 10 MiB/s, 163 objects/s recovering 2024-09-02T10:32:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1580/114063 objects degraded (1.385%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:15.437 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:14 smithi151 ceph-mon[16639]: pgmap v1984: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 1580/114063 objects degraded (1.385%); 13899/114063 objects misplaced (12.185%); 10 MiB/s, 163 objects/s recovering 2024-09-02T10:32:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1580/114063 objects degraded (1.385%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:15.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:15.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:15.469 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:15.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:15.606 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:15.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:15.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:15.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:15.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:15.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:15.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:15.871 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:15.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:15.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:15.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:15.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:15.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:16.099 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:16.123 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:16.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:16.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:16.250 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:16.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:16.336 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":1979,"stamp":"2024-09-02T10:32:05.968109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2769093176,"num_objects":42247,"num_object_clones":0,"num_object_copies":126741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1481,"num_objects_misplaced":15399,"num_objects_unfound":0,"num_objects_dirty":42247,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168313,"num_write_kb":6737793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8584,"num_bytes_recovered":560936544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163465,"ondisk_log_size":163465,"up":141,"acting":136,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":16469340,"kb_used_data":8810644,"kb_used_omap":18,"kb_used_meta":7658413,"kb_avail":1108472484,"statfs":{"total":1151940427776,"available":1135075823616,"internally_reserved":0,"allocated":9022099456,"data_stored":9017926120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":7842215227},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-204275712,"num_objects":-3117,"num_object_clones":0,"num_object_copies":-9351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":141,"num_objects_misplaced":-1073,"num_objects_unfound":0,"num_objects_dirty":-3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3117,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":852,"num_bytes_recovered":55836672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.417036"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":1131,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.389257+0000","last_change":"2024-09-02T10:31:29.714733+0000","last_active":"2024-09-02T10:31:59.389257+0000","last_peered":"2024-09-02T10:31:59.389257+0000","last_clean":"2024-09-02T10:31:59.389257+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:31:59.389257+0000","last_undegraded":"2024-09-02T10:31:59.389257+0000","last_fullsized":"2024-09-02T10:31:59.389257+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:29.714679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:29.714679+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:55.882299+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":939,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-09-02T10:32:02.957065+0000","last_change":"2024-09-02T10:31:02.822274+0000","last_active":"2024-09-02T10:32:02.957065+0000","last_peered":"2024-09-02T10:32:02.957065+0000","last_clean":"2024-09-02T10:32:02.957065+0000","last_became_active":"2024-09-02T10:31:02.821847+0000","last_became_peered":"2024-09-02T10:31:02.821847+0000","last_unstale":"2024-09-02T10:32:02.957065+0000","last_undegraded":"2024-09-02T10:32:02.957065+0000","last_fullsized":"2024-09-02T10:32:02.957065+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:54.883293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.883293+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1409,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-09-02T10:32:04.323230+0000","last_change":"2024-09-02T10:31:32.666776+0000","last_active":"2024-09-02T10:32:04.323230+0000","last_peered":"2024-09-02T10:32:04.323230+0000","last_clean":"2024-09-02T10:32:04.323230+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:32:04.323230+0000","last_undegraded":"2024-09-02T10:32:04.323230+0000","last_fullsized":"2024-09-02T10:32:04.323230+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:32.666702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:32.666702+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:41.340915+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1385,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-09-02T10:32:04.323099+0000","last_change":"2024-09-02T10:30:17.892008+0000","last_active":"2024-09-02T10:32:04.323099+0000","last_peered":"2024-09-02T10:32:04.323099+0000","last_clean":"2024-09-02T10:32:04.323099+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:32:04.323099+0000","last_undegraded":"2024-09-02T10:32:04.323099+0000","last_fullsized":"2024-09-02T10:32:04.323099+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:17.891937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:17.891937+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":696,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-09-02T10:32:04.327010+0000","last_change":"2024-09-02T10:31:33.042779+0000","last_active":"2024-09-02T10:32:04.327010+0000","last_peered":"2024-09-02T10:32:04.327010+0000","last_clean":"2024-09-02T10:32:04.327010+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:32:04.327010+0000","last_undegraded":"2024-09-02T10:32:04.327010+0000","last_fullsized":"2024-09-02T10:32:04.327010+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:33.042701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:33.042701+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:33.118824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":1243,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.370113+0000","last_change":"2024-09-02T10:29:58.037831+0000","last_active":"2024-09-02T10:31:59.370113+0000","last_peered":"2024-09-02T10:31:59.370113+0000","last_clean":"2024-09-02T10:31:59.370113+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:31:59.370113+0000","last_undegraded":"2024-09-02T10:31:59.370113+0000","last_fullsized":"2024-09-02T10:31:59.370113+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:58.037744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:58.037744+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1762,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.378224+0000","last_change":"2024-09-02T10:29:57.973361+0000","last_active":"2024-09-02T10:31:59.378224+0000","last_peered":"2024-09-02T10:31:59.378224+0000","last_clean":"2024-09-02T10:31:59.378224+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:31:59.378224+0000","last_undegraded":"2024-09-02T10:31:59.378224+0000","last_fullsized":"2024-09-02T10:31:59.378224+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:57.973289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:57.973289+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1408,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-09-02T10:32:04.323213+0000","last_change":"2024-09-02T10:30:24.905924+0000","last_active":"2024-09-02T10:32:04.323213+0000","last_peered":"2024-09-02T10:32:04.323213+0000","last_clean":"2024-09-02T10:32:04.323213+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:32:04.323213+0000","last_undegraded":"2024-09-02T10:32:04.323213+0000","last_fullsized":"2024-09-02T10:32:04.323213+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.905855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.905855+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":1412,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-09-02T10:32:02.956720+0000","last_change":"2024-09-02T10:30:18.808693+0000","last_active":"2024-09-02T10:32:02.956720+0000","last_peered":"2024-09-02T10:32:02.956720+0000","last_clean":"2024-09-02T10:32:02.956720+0000","last_became_active":"2024-09-02T10:25:01.211125+0000","last_became_peered":"2024-09-02T10:25:01.211125+0000","last_unstale":"2024-09-02T10:32:02.956720+0000","last_undegraded":"2024-09-02T10:32:02.956720+0000","last_fullsized":"2024-09-02T10:32:02.956720+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:18.808615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:18.808615+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1371,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.375964+0000","last_change":"2024-09-02T10:30:47.378511+0000","last_active":"2024-09-02T10:31:59.375964+0000","last_peered":"2024-09-02T10:31:59.375964+0000","last_clean":"2024-09-02T10:31:59.375964+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:31:59.375964+0000","last_undegraded":"2024-09-02T10:31:59.375964+0000","last_fullsized":"2024-09-02T10:31:59.375964+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:47.378461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:47.378461+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":615,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.374461+0000","last_change":"2024-09-02T10:30:21.301133+0000","last_active":"2024-09-02T10:31:59.374461+0000","last_peered":"2024-09-02T10:31:59.374461+0000","last_clean":"2024-09-02T10:31:59.374461+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:31:59.374461+0000","last_undegraded":"2024-09-02T10:31:59.374461+0000","last_fullsized":"2024-09-02T10:31:59.374461+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:21.301078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:21.301078+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1730,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-09-02T10:32:02.972774+0000","last_change":"2024-09-02T10:31:02.820524+0000","last_active":"2024-09-02T10:32:02.972774+0000","last_peered":"2024-09-02T10:32:02.972774+0000","last_clean":"2024-09-02T10:32:02.972774+0000","last_became_active":"2024-09-02T10:31:02.820058+0000","last_became_peered":"2024-09-02T10:31:02.820058+0000","last_unstale":"2024-09-02T10:32:02.972774+0000","last_undegraded":"2024-09-02T10:32:02.972774+0000","last_fullsized":"2024-09-02T10:32:02.972774+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:48.014073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.014073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:10.729452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1661,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.378207+0000","last_change":"2024-09-02T10:30:23.108717+0000","last_active":"2024-09-02T10:31:59.378207+0000","last_peered":"2024-09-02T10:31:59.378207+0000","last_clean":"2024-09-02T10:31:59.378207+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:31:59.378207+0000","last_undegraded":"2024-09-02T10:31:59.378207+0000","last_fullsized":"2024-09-02T10:31:59.378207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:23.108646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:23.108646+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":393,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.324191+0000","last_change":"2024-09-02T10:31:32.353256+0000","last_active":"2024-09-02T10:31:59.324191+0000","last_peered":"2024-09-02T10:31:59.324191+0000","last_clean":"2024-09-02T10:31:59.324191+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:31:59.324191+0000","last_undegraded":"2024-09-02T10:31:59.324191+0000","last_fullsized":"2024-09-02T10:31:59.324191+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:32.353166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:32.353166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:45.856271+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1631,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.378207+0000","last_change":"2024-09-02T10:30:16.984287+0000","last_active":"2024-09-02T10:31:59.378207+0000","last_peered":"2024-09-02T10:31:59.378207+0000","last_clean":"2024-09-02T10:31:59.378207+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:31:59.378207+0000","last_undegraded":"2024-09-02T10:31:59.378207+0000","last_fullsized":"2024-09-02T10:31:59.378207+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:16.984241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:16.984241+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"864'10411","reported_seq":10667,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-09-02T10:32:04.055300+0000","last_change":"2024-09-02T10:30:45.515400+0000","last_active":"2024-09-02T10:32:04.055300+0000","last_peered":"2024-09-02T10:32:04.055300+0000","last_clean":"2024-09-02T10:32:04.055300+0000","last_became_active":"2024-09-02T10:28:51.931930+0000","last_became_peered":"2024-09-02T10:28:51.931930+0000","last_unstale":"2024-09-02T10:32:04.055300+0000","last_undegraded":"2024-09-02T10:32:04.055300+0000","last_fullsized":"2024-09-02T10:32:04.055300+0000","mapping_epoch":792,"log_start":"800'400","ondisk_log_start":"800'400","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7411","last_scrub_stamp":"2024-09-02T10:30:45.515308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:45.515308+0000","objects_scrubbed":5608,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:07.081713+0000","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169279488,"num_objects":2583,"num_object_clones":0,"num_object_copies":7749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10411,"num_write_kb":415808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1725,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.378311+0000","last_change":"2024-09-02T10:30:22.096061+0000","last_active":"2024-09-02T10:31:59.378311+0000","last_peered":"2024-09-02T10:31:59.378311+0000","last_clean":"2024-09-02T10:31:59.378311+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:31:59.378311+0000","last_undegraded":"2024-09-02T10:31:59.378311+0000","last_fullsized":"2024-09-02T10:31:59.378311+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:22.095998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:22.095998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"864'10710","reported_seq":10980,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-09-02T10:32:03.909883+0000","last_change":"2024-09-02T10:31:30.230104+0000","last_active":"2024-09-02T10:32:03.909883+0000","last_peered":"2024-09-02T10:32:03.909883+0000","last_clean":"2024-09-02T10:32:03.909883+0000","last_became_active":"2024-09-02T10:28:51.652599+0000","last_became_peered":"2024-09-02T10:28:51.652599+0000","last_unstale":"2024-09-02T10:32:03.909883+0000","last_undegraded":"2024-09-02T10:32:03.909883+0000","last_fullsized":"2024-09-02T10:32:03.909883+0000","mapping_epoch":792,"log_start":"801'700","ondisk_log_start":"801'700","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'9583","last_scrub_stamp":"2024-09-02T10:31:30.229961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:31:30.229961+0000","objects_scrubbed":3792,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:31.708698+0000","scrub_duration":1422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173408256,"num_objects":2646,"num_object_clones":0,"num_object_copies":7938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2646,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10710,"num_write_kb":427392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1850,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.373686+0000","last_change":"2024-09-02T10:30:24.407289+0000","last_active":"2024-09-02T10:31:59.373686+0000","last_peered":"2024-09-02T10:31:59.373686+0000","last_clean":"2024-09-02T10:31:59.373686+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:31:59.373686+0000","last_undegraded":"2024-09-02T10:31:59.373686+0000","last_fullsized":"2024-09-02T10:31:59.373686+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.407218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.407218+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"864'10573","reported_seq":14371,"reported_epoch":864,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:32:04.201938+0000","last_change":"2024-09-02T10:31:04.242623+0000","last_active":"2024-09-02T10:32:04.201938+0000","last_peered":"2024-09-02T10:32:04.201938+0000","last_clean":"2024-09-02T10:31:01.749812+0000","last_became_active":"2024-09-02T10:31:04.153013+0000","last_became_peered":"2024-09-02T10:31:04.153013+0000","last_unstale":"2024-09-02T10:32:04.201938+0000","last_undegraded":"2024-09-02T10:31:03.853663+0000","last_fullsized":"2024-09-02T10:31:03.819841+0000","mapping_epoch":844,"log_start":"800'500","ondisk_log_start":"800'500","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'5828","last_scrub_stamp":"2024-09-02T10:30:16.082748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:16.082748+0000","objects_scrubbed":5788,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1773,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167575552,"num_objects":2557,"num_object_clones":0,"num_object_copies":7671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":386,"num_objects_misplaced":2509,"num_objects_unfound":0,"num_objects_dirty":2557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10573,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1788,"num_bytes_recovered":116260864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5],"avail_no_missing":["1","5","11"],"object_location_counts":[{"shards":"1,5","objects":-338},{"shards":"1,5,11","objects":2509}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1504,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-09-02T10:32:02.954186+0000","last_change":"2024-09-02T10:31:24.531418+0000","last_active":"2024-09-02T10:32:02.954186+0000","last_peered":"2024-09-02T10:32:02.954186+0000","last_clean":"2024-09-02T10:32:02.954186+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:32:02.954186+0000","last_undegraded":"2024-09-02T10:32:02.954186+0000","last_fullsized":"2024-09-02T10:32:02.954186+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:24.531349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:24.531349+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:31.310250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"863'10460","reported_seq":10679,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:32:01.641523+0000","last_change":"2024-09-02T10:30:16.516178+0000","last_active":"2024-09-02T10:32:01.641523+0000","last_peered":"2024-09-02T10:32:01.641523+0000","last_clean":"2024-09-02T10:32:01.641523+0000","last_became_active":"2024-09-02T10:28:51.668558+0000","last_became_peered":"2024-09-02T10:28:51.668558+0000","last_unstale":"2024-09-02T10:32:01.641523+0000","last_undegraded":"2024-09-02T10:32:01.641523+0000","last_fullsized":"2024-09-02T10:32:01.641523+0000","mapping_epoch":792,"log_start":"800'400","ondisk_log_start":"800'400","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'5835","last_scrub_stamp":"2024-09-02T10:30:16.515921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:16.515921+0000","objects_scrubbed":5778,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170524672,"num_objects":2602,"num_object_clones":0,"num_object_copies":7806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2602,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10460,"num_write_kb":417984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":471,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.368157+0000","last_change":"2024-09-02T10:31:28.385898+0000","last_active":"2024-09-02T10:31:59.368157+0000","last_peered":"2024-09-02T10:31:59.368157+0000","last_clean":"2024-09-02T10:31:59.368157+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:31:59.368157+0000","last_undegraded":"2024-09-02T10:31:59.368157+0000","last_fullsized":"2024-09-02T10:31:59.368157+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:28.385818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:28.385818+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:45.599604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"863'10455","reported_seq":15819,"reported_epoch":863,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:32:01.561583+0000","last_change":"2024-09-02T10:31:04.087511+0000","last_active":"2024-09-02T10:32:01.561583+0000","last_peered":"2024-09-02T10:32:01.561583+0000","last_clean":"2024-09-02T10:31:01.804575+0000","last_became_active":"2024-09-02T10:31:04.035888+0000","last_became_peered":"2024-09-02T10:31:04.035888+0000","last_unstale":"2024-09-02T10:32:01.561583+0000","last_undegraded":"2024-09-02T10:31:03.853844+0000","last_fullsized":"2024-09-02T10:31:03.818904+0000","mapping_epoch":844,"log_start":"800'400","ondisk_log_start":"800'400","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"822'4517","last_scrub_stamp":"2024-09-02T10:29:54.734394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.734394+0000","objects_scrubbed":4467,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1390,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175505432,"num_objects":2679,"num_object_clones":0,"num_object_copies":8037,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":89,"num_objects_misplaced":2086,"num_objects_unfound":0,"num_objects_dirty":2679,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10455,"num_write_kb":420225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2576,"num_bytes_recovered":168624128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2],"avail_no_missing":["7","2","11"],"object_location_counts":[{"shards":"2,7","objects":504},{"shards":"2,7,11","objects":2086}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1798,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.389165+0000","last_change":"2024-09-02T10:30:58.631533+0000","last_active":"2024-09-02T10:31:59.389165+0000","last_peered":"2024-09-02T10:31:59.389165+0000","last_clean":"2024-09-02T10:31:59.389165+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:31:59.389165+0000","last_undegraded":"2024-09-02T10:31:59.389165+0000","last_fullsized":"2024-09-02T10:31:59.389165+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:58.631463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:58.631463+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:19.925804+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"864'10503","reported_seq":10729,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-09-02T10:32:02.973649+0000","last_change":"2024-09-02T10:30:24.460290+0000","last_active":"2024-09-02T10:32:02.973649+0000","last_peered":"2024-09-02T10:32:02.973649+0000","last_clean":"2024-09-02T10:32:02.973649+0000","last_became_active":"2024-09-02T10:28:51.652606+0000","last_became_peered":"2024-09-02T10:28:51.652606+0000","last_unstale":"2024-09-02T10:32:02.973649+0000","last_undegraded":"2024-09-02T10:32:02.973649+0000","last_fullsized":"2024-09-02T10:32:02.973649+0000","mapping_epoch":792,"log_start":"800'500","ondisk_log_start":"800'500","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'6309","last_scrub_stamp":"2024-09-02T10:30:24.460190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.460190+0000","objects_scrubbed":6243,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171900928,"num_objects":2623,"num_object_clones":0,"num_object_copies":7869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10503,"num_write_kb":420032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":1265,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.370222+0000","last_change":"2024-09-02T10:30:55.890418+0000","last_active":"2024-09-02T10:31:59.370222+0000","last_peered":"2024-09-02T10:31:59.370222+0000","last_clean":"2024-09-02T10:31:59.370222+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:31:59.370222+0000","last_undegraded":"2024-09-02T10:31:59.370222+0000","last_fullsized":"2024-09-02T10:31:59.370222+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:55.890351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:55.890351+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"863'10308","reported_seq":10579,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:32:01.779537+0000","last_change":"2024-09-02T10:31:32.737381+0000","last_active":"2024-09-02T10:32:01.779537+0000","last_peered":"2024-09-02T10:32:01.779537+0000","last_clean":"2024-09-02T10:32:01.779537+0000","last_became_active":"2024-09-02T10:28:51.932528+0000","last_became_peered":"2024-09-02T10:28:51.932528+0000","last_unstale":"2024-09-02T10:32:01.779537+0000","last_undegraded":"2024-09-02T10:32:01.779537+0000","last_fullsized":"2024-09-02T10:32:01.779537+0000","mapping_epoch":792,"log_start":"799'300","ondisk_log_start":"799'300","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'9419","last_scrub_stamp":"2024-09-02T10:31:32.737193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:31:32.737193+0000","objects_scrubbed":3630,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:37.250256+0000","scrub_duration":2052,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177471488,"num_objects":2708,"num_object_clones":0,"num_object_copies":8124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10308,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":1230,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.370398+0000","last_change":"2024-09-02T10:29:54.344897+0000","last_active":"2024-09-02T10:31:59.370398+0000","last_peered":"2024-09-02T10:31:59.370398+0000","last_clean":"2024-09-02T10:31:59.370398+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:31:59.370398+0000","last_undegraded":"2024-09-02T10:31:59.370398+0000","last_fullsized":"2024-09-02T10:31:59.370398+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:54.344678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.344678+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"863'10467","reported_seq":10731,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:32:02.700951+0000","last_change":"2024-09-02T10:31:34.212685+0000","last_active":"2024-09-02T10:32:02.700951+0000","last_peered":"2024-09-02T10:32:02.700951+0000","last_clean":"2024-09-02T10:32:02.700951+0000","last_became_active":"2024-09-02T10:28:51.931936+0000","last_became_peered":"2024-09-02T10:28:51.931936+0000","last_unstale":"2024-09-02T10:32:02.700951+0000","last_undegraded":"2024-09-02T10:32:02.700951+0000","last_fullsized":"2024-09-02T10:32:02.700951+0000","mapping_epoch":792,"log_start":"800'400","ondisk_log_start":"800'400","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'9573","last_scrub_stamp":"2024-09-02T10:31:34.212499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:31:34.212499+0000","objects_scrubbed":3519,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:58.429729+0000","scrub_duration":849,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171114496,"num_objects":2611,"num_object_clones":0,"num_object_copies":7833,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2611,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10467,"num_write_kb":418496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1012,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.321200+0000","last_change":"2024-09-02T10:30:14.802221+0000","last_active":"2024-09-02T10:31:59.321200+0000","last_peered":"2024-09-02T10:31:59.321200+0000","last_clean":"2024-09-02T10:31:59.321200+0000","last_became_active":"2024-09-02T10:25:02.065175+0000","last_became_peered":"2024-09-02T10:25:02.065175+0000","last_unstale":"2024-09-02T10:31:59.321200+0000","last_undegraded":"2024-09-02T10:31:59.321200+0000","last_fullsized":"2024-09-02T10:31:59.321200+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:14.802157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:14.802157+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"864'10758","reported_seq":12225,"reported_epoch":864,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:32:04.051091+0000","last_change":"2024-09-02T10:31:40.864360+0000","last_active":"2024-09-02T10:32:04.051091+0000","last_peered":"2024-09-02T10:32:04.051091+0000","last_clean":"2024-09-02T10:31:38.550380+0000","last_became_active":"2024-09-02T10:31:40.658764+0000","last_became_peered":"2024-09-02T10:31:40.658764+0000","last_unstale":"2024-09-02T10:32:04.051091+0000","last_undegraded":"2024-09-02T10:31:40.522684+0000","last_fullsized":"2024-09-02T10:31:40.323854+0000","mapping_epoch":854,"log_start":"801'700","ondisk_log_start":"801'700","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7674","last_scrub_stamp":"2024-09-02T10:30:46.822238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.822238+0000","objects_scrubbed":5790,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174456832,"num_objects":2662,"num_object_clones":0,"num_object_copies":7986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":418,"num_objects_misplaced":5740,"num_objects_unfound":0,"num_objects_dirty":2662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10758,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":413,"num_bytes_recovered":26411008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,9],"acting":[3,9],"avail_no_missing":["3","5","9"],"object_location_counts":[{"shards":"3,5,9","objects":2661},{"shards":"3,9","objects":-417}],"blocked_by":[],"up_primary":10,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":1245,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.370224+0000","last_change":"2024-09-02T10:30:48.025489+0000","last_active":"2024-09-02T10:31:59.370224+0000","last_peered":"2024-09-02T10:31:59.370224+0000","last_clean":"2024-09-02T10:31:59.370224+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:31:59.370224+0000","last_undegraded":"2024-09-02T10:31:59.370224+0000","last_fullsized":"2024-09-02T10:31:59.370224+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:48.025327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.025327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"865'10625","reported_seq":10899,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-09-02T10:32:04.556987+0000","last_change":"2024-09-02T10:31:39.273035+0000","last_active":"2024-09-02T10:32:04.556987+0000","last_peered":"2024-09-02T10:32:04.556987+0000","last_clean":"2024-09-02T10:32:04.556987+0000","last_became_active":"2024-09-02T10:28:51.668549+0000","last_became_peered":"2024-09-02T10:28:51.668549+0000","last_unstale":"2024-09-02T10:32:04.556987+0000","last_undegraded":"2024-09-02T10:32:04.556987+0000","last_fullsized":"2024-09-02T10:32:04.556987+0000","mapping_epoch":792,"log_start":"801'600","ondisk_log_start":"801'600","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"853'9804","last_scrub_stamp":"2024-09-02T10:31:39.272959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:31:39.272959+0000","objects_scrubbed":3434,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:33:05.517671+0000","scrub_duration":710,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170983424,"num_objects":2609,"num_object_clones":0,"num_object_copies":7827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":423488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1493,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.373806+0000","last_change":"2024-09-02T10:30:23.359450+0000","last_active":"2024-09-02T10:31:59.373806+0000","last_peered":"2024-09-02T10:31:59.373806+0000","last_clean":"2024-09-02T10:31:59.373806+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:31:59.373806+0000","last_undegraded":"2024-09-02T10:31:59.373806+0000","last_fullsized":"2024-09-02T10:31:59.373806+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:23.359405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:23.359405+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"865'10369","reported_seq":10626,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-09-02T10:32:04.550157+0000","last_change":"2024-09-02T10:31:38.628550+0000","last_active":"2024-09-02T10:32:04.550157+0000","last_peered":"2024-09-02T10:32:04.550157+0000","last_clean":"2024-09-02T10:32:04.550157+0000","last_became_active":"2024-09-02T10:28:51.932243+0000","last_became_peered":"2024-09-02T10:28:51.932243+0000","last_unstale":"2024-09-02T10:32:04.550157+0000","last_undegraded":"2024-09-02T10:32:04.550157+0000","last_fullsized":"2024-09-02T10:32:04.550157+0000","mapping_epoch":792,"log_start":"800'300","ondisk_log_start":"800'300","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"853'9585","last_scrub_stamp":"2024-09-02T10:31:38.628451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:31:38.628451+0000","objects_scrubbed":3368,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:39.735930+0000","scrub_duration":1037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168099840,"num_objects":2565,"num_object_clones":0,"num_object_copies":7695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10369,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1755,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-09-02T10:32:02.972886+0000","last_change":"2024-09-02T10:30:20.873940+0000","last_active":"2024-09-02T10:32:02.972886+0000","last_peered":"2024-09-02T10:32:02.972886+0000","last_clean":"2024-09-02T10:32:02.972886+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:32:02.972886+0000","last_undegraded":"2024-09-02T10:32:02.972886+0000","last_fullsized":"2024-09-02T10:32:02.972886+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:20.873897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:20.873897+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"863'10561","reported_seq":10815,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:32:01.552383+0000","last_change":"2024-09-02T10:30:46.985280+0000","last_active":"2024-09-02T10:32:01.552383+0000","last_peered":"2024-09-02T10:32:01.552383+0000","last_clean":"2024-09-02T10:32:01.552383+0000","last_became_active":"2024-09-02T10:28:51.650885+0000","last_became_peered":"2024-09-02T10:28:51.650885+0000","last_unstale":"2024-09-02T10:32:01.552383+0000","last_undegraded":"2024-09-02T10:32:01.552383+0000","last_fullsized":"2024-09-02T10:32:01.552383+0000","mapping_epoch":792,"log_start":"800'500","ondisk_log_start":"800'500","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7658","last_scrub_stamp":"2024-09-02T10:30:46.985056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.985056+0000","objects_scrubbed":5619,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174653440,"num_objects":2665,"num_object_clones":0,"num_object_copies":7995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10561,"num_write_kb":423232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1503,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.373654+0000","last_change":"2024-09-02T10:31:03.381315+0000","last_active":"2024-09-02T10:31:59.373654+0000","last_peered":"2024-09-02T10:31:59.373654+0000","last_clean":"2024-09-02T10:31:59.373654+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:31:59.373654+0000","last_undegraded":"2024-09-02T10:31:59.373654+0000","last_fullsized":"2024-09-02T10:31:59.373654+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:58.376013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:58.376013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"863'10513","reported_seq":15050,"reported_epoch":863,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:32:01.652174+0000","last_change":"2024-09-02T10:31:04.128960+0000","last_active":"2024-09-02T10:32:01.652174+0000","last_peered":"2024-09-02T10:32:01.652174+0000","last_clean":"2024-09-02T10:31:01.786664+0000","last_became_active":"2024-09-02T10:31:04.069096+0000","last_became_peered":"2024-09-02T10:31:04.069096+0000","last_unstale":"2024-09-02T10:32:01.652174+0000","last_undegraded":"2024-09-02T10:31:03.844885+0000","last_fullsized":"2024-09-02T10:31:03.820047+0000","mapping_epoch":844,"log_start":"800'500","ondisk_log_start":"800'500","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'5937","last_scrub_stamp":"2024-09-02T10:30:17.013655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:17.013655+0000","objects_scrubbed":5883,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181731328,"num_objects":2773,"num_object_clones":0,"num_object_copies":8319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":103,"num_objects_misplaced":2494,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10513,"num_write_kb":425152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2157,"num_bytes_recovered":141164544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7],"avail_no_missing":["10","7","11"],"object_location_counts":[{"shards":"7,10","objects":176},{"shards":"7,10,11","objects":2494}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1759,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.321220+0000","last_change":"2024-09-02T10:30:20.714039+0000","last_active":"2024-09-02T10:31:59.321220+0000","last_peered":"2024-09-02T10:31:59.321220+0000","last_clean":"2024-09-02T10:31:59.321220+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:31:59.321220+0000","last_undegraded":"2024-09-02T10:31:59.321220+0000","last_fullsized":"2024-09-02T10:31:59.321220+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:20.713975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:20.713975+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"864'10615","reported_seq":14128,"reported_epoch":864,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:32:04.220124+0000","last_change":"2024-09-02T10:31:04.119388+0000","last_active":"2024-09-02T10:32:04.220124+0000","last_peered":"2024-09-02T10:32:04.220124+0000","last_clean":"2024-09-02T10:31:01.796091+0000","last_became_active":"2024-09-02T10:31:04.054394+0000","last_became_peered":"2024-09-02T10:31:04.054394+0000","last_unstale":"2024-09-02T10:32:04.220124+0000","last_undegraded":"2024-09-02T10:31:03.853930+0000","last_fullsized":"2024-09-02T10:31:03.819998+0000","mapping_epoch":844,"log_start":"801'600","ondisk_log_start":"801'600","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"839'7994","last_scrub_stamp":"2024-09-02T10:30:53.579876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:53.579876+0000","objects_scrubbed":5226,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168493056,"num_objects":2571,"num_object_clones":0,"num_object_copies":7713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":485,"num_objects_misplaced":2570,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10615,"num_write_kb":421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1646,"num_bytes_recovered":107544576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10],"avail_no_missing":["1","10","11"],"object_location_counts":[{"shards":"1,10","objects":-484},{"shards":"1,10,11","objects":2570}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":703,"reported_epoch":865,"state":"active+clean","last_fresh":"2024-09-02T10:32:04.326942+0000","last_change":"2024-09-02T10:30:44.835334+0000","last_active":"2024-09-02T10:32:04.326942+0000","last_peered":"2024-09-02T10:32:04.326942+0000","last_clean":"2024-09-02T10:32:04.326942+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:32:04.326942+0000","last_undegraded":"2024-09-02T10:32:04.326942+0000","last_fullsized":"2024-09-02T10:32:04.326942+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:44.835287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:44.835287+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:08.132382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"863'10559","reported_seq":10809,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:32:02.644302+0000","last_change":"2024-09-02T10:30:48.613043+0000","last_active":"2024-09-02T10:32:02.644302+0000","last_peered":"2024-09-02T10:32:02.644302+0000","last_clean":"2024-09-02T10:32:02.644302+0000","last_became_active":"2024-09-02T10:28:51.651609+0000","last_became_peered":"2024-09-02T10:28:51.651609+0000","last_unstale":"2024-09-02T10:32:02.644302+0000","last_undegraded":"2024-09-02T10:32:02.644302+0000","last_fullsized":"2024-09-02T10:32:02.644302+0000","mapping_epoch":792,"log_start":"800'500","ondisk_log_start":"800'500","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7724","last_scrub_stamp":"2024-09-02T10:30:48.612984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.612984+0000","objects_scrubbed":5646,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181731328,"num_objects":2773,"num_object_clones":0,"num_object_copies":8319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2773,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10559,"num_write_kb":426624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1531,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:31:59.376073+0000","last_change":"2024-09-02T10:30:46.402988+0000","last_active":"2024-09-02T10:31:59.376073+0000","last_peered":"2024-09-02T10:31:59.376073+0000","last_clean":"2024-09-02T10:31:59.376073+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:31:59.376073+0000","last_undegraded":"2024-09-02T10:31:59.376073+0000","last_fullsized":"2024-09-02T10:31:59.376073+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:46.402901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.402901+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:12.184642+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"863'10426","reported_seq":10650,"reported_epoch":863,"state":"active+clean","last_fresh":"2024-09-02T10:32:01.580775+0000","last_change":"2024-09-02T10:30:27.169510+0000","last_active":"2024-09-02T10:32:01.580775+0000","last_peered":"2024-09-02T10:32:01.580775+0000","last_clean":"2024-09-02T10:32:01.580775+0000","last_became_active":"2024-09-02T10:28:51.668549+0000","last_became_peered":"2024-09-02T10:28:51.668549+0000","last_unstale":"2024-09-02T10:32:01.580775+0000","last_undegraded":"2024-09-02T10:32:01.580775+0000","last_fullsized":"2024-09-02T10:32:01.580775+0000","mapping_epoch":792,"log_start":"800'400","ondisk_log_start":"800'400","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"835'6417","last_scrub_stamp":"2024-09-02T10:30:27.169341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:27.169341+0000","objects_scrubbed":6355,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2342,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171573248,"num_objects":2618,"num_object_clones":0,"num_object_copies":7854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10426,"num_write_kb":417408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":924,"reported_epoch":864,"state":"active+clean","last_fresh":"2024-09-02T10:32:02.958252+0000","last_change":"2024-09-02T10:31:32.841160+0000","last_active":"2024-09-02T10:32:02.958252+0000","last_peered":"2024-09-02T10:32:02.958252+0000","last_clean":"2024-09-02T10:32:02.958252+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:32:02.958252+0000","last_undegraded":"2024-09-02T10:32:02.958252+0000","last_fullsized":"2024-09-02T10:32:02.958252+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:32.841078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:32.841078+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:47.291610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2768502808,"num_objects":42245,"num_object_clones":0,"num_object_copies":126735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1481,"num_objects_misplaced":15399,"num_objects_unfound":0,"num_objects_dirty":42245,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168313,"num_write_kb":6737793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8580,"num_bytes_recovered":560005120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8956751872,"data_stored":8956739656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160613,"ondisk_log_size":160613,"up":48,"acting":43,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384063,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2064916,"kb_used_data":1384004,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":91680236,"statfs":{"total":95995035648,"available":93880561664,"internally_reserved":0,"allocated":1417220096,"data_stored":1416871559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":697235908},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612996,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1615952,"kb_used_data":1043328,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92129200,"statfs":{"total":95995035648,"available":94340300800,"internally_reserved":0,"allocated":1068367872,"data_stored":1068023407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":586348992},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841929,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1771728,"kb_used_data":1032512,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":91973424,"statfs":{"total":95995035648,"available":94180786176,"internally_reserved":0,"allocated":1057292288,"data_stored":1056947823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":756939200},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070864,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":949148,"kb_used_data":498700,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92796004,"statfs":{"total":95995035648,"available":95023108096,"internally_reserved":0,"allocated":510668800,"data_stored":510320114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299796,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1053852,"kb_used_data":350732,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":92691300,"statfs":{"total":95995035648,"available":94915891200,"internally_reserved":0,"allocated":359149568,"data_stored":358797479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":719976906},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528728,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":464272,"kb_used_data":4096,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93280880,"statfs":{"total":95995035648,"available":95519621120,"internally_reserved":0,"allocated":4194304,"data_stored":3849839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":471202247},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724957,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1684504,"kb_used_data":953608,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92060648,"statfs":{"total":95995035648,"available":94270103552,"internally_reserved":0,"allocated":976494592,"data_stored":976146575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953890,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1129120,"kb_used_data":656460,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92616032,"statfs":{"total":95995035648,"available":94838816768,"internally_reserved":0,"allocated":672215040,"data_stored":671866354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150120,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1278108,"kb_used_data":506956,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92467044,"statfs":{"total":95995035648,"available":94686253056,"internally_reserved":0,"allocated":519122944,"data_stored":518774258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":789641672},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411755,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1434716,"kb_used_data":871756,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92310436,"statfs":{"total":95995035648,"available":94525886464,"internally_reserved":0,"allocated":892678144,"data_stored":892326055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575280,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1693536,"kb_used_data":1004300,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":92051616,"statfs":{"total":95995035648,"available":94260854784,"internally_reserved":0,"allocated":1028403200,"data_stored":1028054514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738806,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1329488,"kb_used_data":504192,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":92415664,"statfs":{"total":95995035648,"available":94633639936,"internally_reserved":0,"allocated":516292608,"data_stored":515948143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":512098304,"data_stored":512098304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1015021568,"data_stored":1015021568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":889720832,"data_stored":889716760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":516947968,"data_stored":516947968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":666107904,"data_stored":666107904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":969015296,"data_stored":969015296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3604480,"data_stored":3604480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":357502976,"data_stored":357498904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":506920960,"data_stored":506920960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1048641536,"data_stored":1048641536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1055981568,"data_stored":1055981568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1415188480,"data_stored":1415184408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:32:16.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg force-recovery 8.3 8.6 8.7 2024-09-02T10:32:16.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:16.421 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:16.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:16.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:16.548 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:16.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:16.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:16.679 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:16.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:16.781 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:16.809 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:16.811 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:16.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:16.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:16.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:17.042 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:17.079 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:17.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:17.181 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:17.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:17.212 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:16 smithi187 ceph-mon[20072]: pgmap v1985: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 1666/110310 objects degraded (1.510%); 13276/110310 objects misplaced (12.035%); 11 MiB/s, 182 objects/s recovering 2024-09-02T10:32:17.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:17.338 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:17.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:17.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:16 smithi162 ceph-mon[20279]: pgmap v1985: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 1666/110310 objects degraded (1.510%); 13276/110310 objects misplaced (12.035%); 11 MiB/s, 182 objects/s recovering 2024-09-02T10:32:17.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:16 smithi151 ceph-mon[16639]: pgmap v1985: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 1666/110310 objects degraded (1.510%); 13276/110310 objects misplaced (12.035%); 11 MiB/s, 182 objects/s recovering 2024-09-02T10:32:17.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:17.462 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:17.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:17.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:17.586 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:17.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:17.688 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:17.715 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:17.716 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:17.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:17.849 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:17.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:17.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:17.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:32:17.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:17.975 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:17.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:18.077 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:18.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:18.108 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:18.177 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:32:18.208 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:18.232 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:18.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:18.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:18.254 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:18.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3088097292' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:32:18.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:17 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:32:18.333 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:18.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3088097292' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:32:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:17 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:32:18.359 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:18.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:18.360 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3088097292' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:32:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:17 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:32:18.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:18.487 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:18.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:18.589 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:18.612 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:18.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:18.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:18.736 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:18.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:18.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:18.868 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:18.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:18.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:19.003 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:19.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:19.081 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:18 smithi187 ceph-mon[20072]: pgmap v1986: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 1666/106890 objects degraded (1.559%); 12813/106890 objects misplaced (11.987%); 11 MiB/s, 175 objects/s recovering 2024-09-02T10:32:19.082 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:18 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:32:19.082 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:18 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:32:19.082 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:18 smithi187 ceph-mon[20072]: osdmap e868: 12 total, 12 up, 12 in 2024-09-02T10:32:19.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:19.133 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:19.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:19.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:19.260 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:19.262 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:19.340 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:32:19.340 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:32:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:18 smithi162 ceph-mon[20279]: pgmap v1986: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 1666/106890 objects degraded (1.559%); 12813/106890 objects misplaced (11.987%); 11 MiB/s, 175 objects/s recovering 2024-09-02T10:32:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:18 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:32:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:18 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:32:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:18 smithi162 ceph-mon[20279]: osdmap e868: 12 total, 12 up, 12 in 2024-09-02T10:32:19.362 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:19.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:19.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:19.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:18 smithi151 ceph-mon[16639]: pgmap v1986: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.2 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 565 op/s; 1666/106890 objects degraded (1.559%); 12813/106890 objects misplaced (11.987%); 11 MiB/s, 175 objects/s recovering 2024-09-02T10:32:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:18 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:32:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:18 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:32:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:18 smithi151 ceph-mon[16639]: osdmap e868: 12 total, 12 up, 12 in 2024-09-02T10:32:19.493 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:19.520 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:19.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:19.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:19.647 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:19.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:19.749 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:19.771 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:19.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:19.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:19.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:19.914 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:19.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:20.015 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:20.047 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:20.047 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:20.104 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:32:20.128 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1666/106890 objects degraded (1.559%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:20.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:20.174 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:32:20.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:20.184 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:20.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:20.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:20.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:20.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:20.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1666/106890 objects degraded (1.559%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:20.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:20.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1666/106890 objects degraded (1.559%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:20.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:20.465 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:20.545 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:20.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:20.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:20.616 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:20.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:20.717 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:20.741 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:20.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:20.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:20.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:20.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:20.886 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:20.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:21.014 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:21.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:21.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:21.120 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:32:21.120 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:32:21.139 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:21.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:21.143 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:32:21.159 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:32:21.159 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:32:21.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:20 smithi151 ceph-mon[16639]: pgmap v1988: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 1766/102456 objects degraded (1.724%); 12147/102456 objects misplaced (11.856%); 11 MiB/s, 181 objects/s recovering 2024-09-02T10:32:21.194 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:32:21.194 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:32:21.225 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:32:21.226 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:32:21.241 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:20 smithi187 ceph-mon[20072]: pgmap v1988: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 1766/102456 objects degraded (1.724%); 12147/102456 objects misplaced (11.856%); 11 MiB/s, 181 objects/s recovering 2024-09-02T10:32:21.257 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:32:21.257 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:32:21.264 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:21.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:21.288 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:32:21.288 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:32:21.317 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:32:21.319 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:32:21.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:20 smithi162 ceph-mon[20279]: pgmap v1988: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 574 op/s; 1766/102456 objects degraded (1.724%); 12147/102456 objects misplaced (11.856%); 11 MiB/s, 181 objects/s recovering 2024-09-02T10:32:21.352 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:32:21.352 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:32:21.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:21.381 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:32:21.381 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:32:21.397 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:21.398 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:21.407 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:32:21.408 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:32:21.440 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:32:21.441 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:32:21.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:32:21.481 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:32:21.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:21.516 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:32:21.517 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:32:21.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:21.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:21.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:21.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:21.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:21.775 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:21.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:21.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:21.861 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:32:21.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:21.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:21.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:21.951 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:22.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:22.077 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:22.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:22.098 INFO:teuthology.orchestra.run.smithi151.stderr:instructing pg(s) [8.3,8.7] on osd.1 to force-recovery; instructing pg(s) [8.6] on osd.10 to force-recovery; 2024-09-02T10:32:22.098 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-09-02T10:32:22.162 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/174787638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:32:22.179 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:22.186 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:22.205 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:22.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:22.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:22.308 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:22.342 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:22.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:22.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/174787638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:32:22.346 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/174787638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:32:22.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:22.482 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:22.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:22.584 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:22.620 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:32:22.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:22.625 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:22.726 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:22.742 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:32:22.743 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:32:22.743 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:32:22.743 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:32:22.743 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:32:22.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:22.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:22.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:22.900 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:22.901 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:23.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:23.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:23.034 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:23.034 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:23.135 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:23.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:23.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:23.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:23.266 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:23.289 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:23.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:23.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:23.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:23 smithi162 ceph-mon[20279]: pgmap v1989: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 1766/101286 objects degraded (1.744%); 12147/101286 objects misplaced (11.993%); 11 MiB/s, 181 objects/s recovering 2024-09-02T10:32:23.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:23 smithi162 ceph-mon[20279]: from='client.18702 -' entity='client.admin' cmd=[{"prefix": "pg force-recovery", "pgid": ["8.3", "8.6", "8.7"], "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:32:23.363 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:23 smithi187 ceph-mon[20072]: pgmap v1989: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 1766/101286 objects degraded (1.744%); 12147/101286 objects misplaced (11.993%); 11 MiB/s, 181 objects/s recovering 2024-09-02T10:32:23.363 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:23 smithi187 ceph-mon[20072]: from='client.18702 -' entity='client.admin' cmd=[{"prefix": "pg force-recovery", "pgid": ["8.3", "8.6", "8.7"], "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:32:23.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:23.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:23.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:23.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:23.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:23 smithi151 ceph-mon[16639]: pgmap v1989: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 578 op/s; 1766/101286 objects degraded (1.744%); 12147/101286 objects misplaced (11.993%); 11 MiB/s, 181 objects/s recovering 2024-09-02T10:32:23.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:23 smithi151 ceph-mon[16639]: from='client.18702 -' entity='client.admin' cmd=[{"prefix": "pg force-recovery", "pgid": ["8.3", "8.6", "8.7"], "target": ["mon-mgr", ""]}]: dispatch 2024-09-02T10:32:23.519 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:23.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:23.551 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:23.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:23.675 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:23.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:23.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:23.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:23.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:23.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:23.916 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:23.942 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:23.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:23.943 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:24.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:24.067 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:24.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:24.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:24.169 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:24.193 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:24.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:24.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:24.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:24.326 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:24.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:24.327 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:24.428 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:24.459 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:24.460 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:24.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:24.569 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:32:24.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:24.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:24.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:24.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:24.730 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:24.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:24.832 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:24.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:24.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:24.928 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:24.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:24.992 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:24.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:25.094 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:25.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:25.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:25.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:25.256 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:25.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:25.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:25.342 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:25 smithi151 ceph-mon[16639]: pgmap v1990: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 1766/96558 objects degraded (1.829%); 11636/96558 objects misplaced (12.051%); 12 MiB/s, 187 objects/s recovering 2024-09-02T10:32:25.343 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:25 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1766/96558 objects degraded (1.829%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:25.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:25 smithi162 ceph-mon[20279]: pgmap v1990: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 1766/96558 objects degraded (1.829%); 11636/96558 objects misplaced (12.051%); 12 MiB/s, 187 objects/s recovering 2024-09-02T10:32:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:25 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1766/96558 objects degraded (1.829%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:25.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:25.372 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:25 smithi187 ceph-mon[20072]: pgmap v1990: 47 pgs: 5 active+recovering+undersized+degraded+remapped, 42 active+clean; 2.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 583 op/s; 1766/96558 objects degraded (1.829%); 11636/96558 objects misplaced (12.051%); 12 MiB/s, 187 objects/s recovering 2024-09-02T10:32:25.373 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:25 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1766/96558 objects degraded (1.829%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:25.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:25.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:25.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:25.486 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:32:25.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:25.523 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:43258 7.2 e688/661 7:49444d4c:::benchmark_data_smithi187_120712_object43257:head v 688'2623, mlcod=688'2619)", 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:51.553410+0000", 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 453.965675154, 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.231937966, 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.553410+0000", 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.553410+0000", 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.553411+0000", 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0399999999999996e-07 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.553471+0000", 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0463e-05 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.783710+0000", 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.230238832 2024-09-02T10:32:25.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.783724+0000", 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3334e-05 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.783836+0000", 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112781 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.783898+0000", 2024-09-02T10:32:25.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2123000000000001e-05 2024-09-02T10:32:25.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.785316+0000", 2024-09-02T10:32:25.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0014171559999999999 2024-09-02T10:32:25.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.785348+0000", 2024-09-02T10:32:25.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2672999999999997e-05 2024-09-02T10:32:25.529 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.529 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.529 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.529 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:43269 7.1 e688/679 7:8e1e088a:::benchmark_data_smithi187_120712_object43268:head v 688'2659, mlcod=688'2656)", 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:51.561315+0000", 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 453.95777064100002, 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23091329199999999, 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.561315+0000", 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.561315+0000", 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.561316+0000", 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.136e-06 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.561358+0000", 2024-09-02T10:32:25.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2144000000000002e-05 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.788687+0000", 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.227328581 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.788694+0000", 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1729999999999998e-06 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.788759+0000", 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5592000000000005e-05 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.788778+0000", 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8573999999999999e-05 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.792152+0000", 2024-09-02T10:32:25.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033742389999999998 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:51.792228+0000", 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5852999999999994e-05 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:83687 8.9 8:9271f805:::benchmark_data_smithi187_136059_object83686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:30:05.479781+0000", 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 140.039303972, 2024-09-02T10:32:25.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.230440322, 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 83687 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479781+0000", 2024-09-02T10:32:25.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479781+0000", 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479784+0000", 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2579999999999998e-06 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479834+0000", 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9975999999999997e-05 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479854+0000", 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9888e-05 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479859+0000", 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3340000000000001e-06 2024-09-02T10:32:25.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479927+0000", 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8322000000000001e-05 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.480048+0000", 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121041 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-02T10:32:25.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.480173+0000", 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012423100000000001 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.483089+0000", 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0029163980000000002 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.483118+0000", 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8558e-05 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.710140+0000", 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22702211799999999 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.710182+0000", 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2765999999999997e-05 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.710222+0000", 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9431999999999998e-05 2024-09-02T10:32:25.540 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:83688 8.2 8:44e0af3d:::benchmark_data_smithi187_136059_object83687:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e824)", 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:30:05.479914+0000", 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 140.03917150500001, 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22828726899999999, 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.541 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 83688 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479914+0000", 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479914+0000", 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479915+0000", 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8800000000000002e-07 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479960+0000", 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5355999999999997e-05 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479967+0000", 2024-09-02T10:32:25.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2060000000000001e-06 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.479972+0000", 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5950000000000001e-06 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.480022+0000", 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9941000000000002e-05 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.480124+0000", 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101987 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.480204+0000", 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0569999999999996e-05 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.482833+0000", 2024-09-02T10:32:25.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0026287770000000001 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.482861+0000", 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7325999999999999e-05 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.708127+0000", 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22526626699999999 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.708169+0000", 2024-09-02T10:32:25.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2386999999999999e-05 2024-09-02T10:32:25.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:05.708201+0000", 2024-09-02T10:32:25.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2069000000000001e-05 2024-09-02T10:32:25.547 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.547 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.548 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.548 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:134976 8.5 8:a145b8b2:::benchmark_data_smithi187_136059_object29695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:32:25.548 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:01.814761+0000", 2024-09-02T10:32:25.548 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 83.704324282000002, 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25719428799999999, 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 134976 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.549 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814761+0000", 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814761+0000", 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814762+0000", 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7300000000000004e-07 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.550 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814767+0000", 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6430000000000003e-06 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814779+0000", 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2524000000000001e-05 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814786+0000", 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0890000000000002e-06 2024-09-02T10:32:25.551 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814848+0000", 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1876000000000002e-05 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op must wait for map", 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814852+0000", 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8840000000000004e-06 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.552 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.818348+0000", 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003496075 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.818479+0000", 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000131247 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.818602+0000", 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012228900000000001 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.553 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.823332+0000", 2024-09-02T10:32:25.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047306409999999998 2024-09-02T10:32:25.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.554 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.554 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.825575+0000", 2024-09-02T10:32:25.554 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022428859999999999 2024-09-02T10:32:25.554 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.554 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.555 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.555 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.071873+0000", 2024-09-02T10:32:25.555 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.246297399 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.071908+0000", 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5500999999999998e-05 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.071955+0000", 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7261e-05 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.556 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:134976 8.5 e843/792 ondisk, result = 0)", 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:01.822449+0000", 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 83.696636075000001, 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.249511027, 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.557 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.822449+0000", 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.822449+0000", 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.558 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.822450+0000", 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8999999999999995e-07 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.822455+0000", 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.797e-06 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.071706+0000", 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24925146100000001 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.559 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.071718+0000", 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1683e-05 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.071863+0000", 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014447700000000001 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.071868+0000", 2024-09-02T10:32:25.560 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6829999999999996e-06 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.071960+0000", 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2136e-05 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:134971 8.6 8:61a0a7d2:::benchmark_data_smithi187_136059_object29690:head [delete] snapc 0=[] RETRY=3 ondisk+retry+write+known_if_redirected+supports_pool_eio e844)", 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:03.845135+0000", 2024-09-02T10:32:25.561 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 81.673950180000006, 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.239296486, 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 134971 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845135+0000", 2024-09-02T10:32:25.562 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845135+0000", 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845137+0000", 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6700000000000001e-06 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.563 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845145+0000", 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9719999999999992e-06 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845158+0000", 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2807e-05 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845167+0000", 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8300000000000002e-06 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.564 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.565 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.846168+0000", 2024-09-02T10:32:25.565 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010017940000000001 2024-09-02T10:32:25.565 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.565 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.565 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:32:25.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.846180+0000", 2024-09-02T10:32:25.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2123000000000001e-05 2024-09-02T10:32:25.568 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.568 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.568 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.568 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.084315+0000", 2024-09-02T10:32:25.568 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23813434 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.084432+0000", 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011695 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:134979 8.6 8:64ac925f:::benchmark_data_smithi187_136059_object29698:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e844)", 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:03.845190+0000", 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 81.673895388000005, 2024-09-02T10:32:25.569 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25484030299999999, 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 134979 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845190+0000", 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.570 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845190+0000", 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845190+0000", 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4400000000000001e-07 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.571 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845193+0000", 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.683e-06 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845199+0000", 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.519e-06 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845202+0000", 2024-09-02T10:32:25.572 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7309999999999999e-06 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.846234+0000", 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001031178 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.846238+0000", 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7380000000000002e-06 2024-09-02T10:32:25.573 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:25.574 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:25.575 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:25.575 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.085613+0000", 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23937467100000001 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.089069+0000", 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034556410000000002 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:25.576 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.089221+0000", 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015230800000000001 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.098517+0000", 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0092962240000000005 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.098572+0000", 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4466000000000003e-05 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.577 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.098662+0000", 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0302000000000004e-05 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.098728+0000", 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6298999999999997e-05 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.578 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.100030+0000", 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001302199 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:134985 8.6 8:63a6e710:::benchmark_data_smithi187_136059_object29704:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e844)", 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:03.845208+0000", 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 81.673877472000001, 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27400817399999999, 2024-09-02T10:32:25.579 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 134985 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845208+0000", 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.580 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845208+0000", 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845208+0000", 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5400000000000002e-07 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.581 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845210+0000", 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.389e-06 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845214+0000", 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6319999999999999e-06 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845217+0000", 2024-09-02T10:32:25.582 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9229999999999998e-06 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.846275+0000", 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010578880000000001 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.846284+0000", 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7010000000000002e-06 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.583 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.092418+0000", 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.246134305 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.092518+0000", 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010037799999999999 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.092609+0000", 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0915000000000006e-05 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.098612+0000", 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0060029139999999998 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.098622+0000", 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.645e-06 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.117381+0000", 2024-09-02T10:32:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.018759350000000001 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.117443+0000", 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1943e-05 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.119216+0000", 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001772937 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.586 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:135000 8.6 8:62d6a576:::benchmark_data_smithi187_136059_object29719:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e844)", 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:03.845221+0000", 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 81.673864512999998, 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.276103879, 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.587 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 135000 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845221+0000", 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845221+0000", 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.588 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845221+0000", 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.16e-07 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845222+0000", 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.297e-06 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845226+0000", 2024-09-02T10:32:25.589 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3220000000000001e-06 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845228+0000", 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6800000000000002e-06 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.846308+0000", 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001079221 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.590 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.846311+0000", 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6830000000000001e-06 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.100076+0000", 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25376431500000002 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.100185+0000", 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010901299999999999 2024-09-02T10:32:25.591 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.100278+0000", 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3244000000000004e-05 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.106934+0000", 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0066560569999999999 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.108320+0000", 2024-09-02T10:32:25.592 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001386172 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.119293+0000", 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010972654 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.119368+0000", 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4954000000000001e-05 2024-09-02T10:32:25.593 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.594 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.594 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.121325+0000", 2024-09-02T10:32:25.594 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001957051 2024-09-02T10:32:25.594 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.594 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.594 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.594 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.594 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.594 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:135007 8.6 8:67330f5b:::benchmark_data_smithi187_136059_object29726:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e844)", 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:03.845232+0000", 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 81.673853859999994, 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.28303472499999999, 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 135007 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.595 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845232+0000", 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845232+0000", 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845232+0000", 2024-09-02T10:32:25.596 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0900000000000001e-07 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845233+0000", 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2669999999999999e-06 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845236+0000", 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3239999999999999e-06 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.597 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.845239+0000", 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8909999999999999e-06 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.846327+0000", 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010874909999999999 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:03.846330+0000", 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5939999999999998e-06 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.598 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.103344+0000", 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25701337299999999 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.103506+0000", 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016175599999999999 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.103655+0000", 2024-09-02T10:32:25.599 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014943500000000001 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.106939+0000", 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032836739999999999 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.108457+0000", 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0015179130000000001 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.600 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.126303+0000", 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.017846603999999999 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.126346+0000", 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3133999999999997e-05 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:04.128266+0000", 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019200599999999999 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.601 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146099 8.6 8:6509e9b8:::benchmark_data_smithi187_136059_object40818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.705971+0000", 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 63.813114767999998, 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.37556117500000002, 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.602 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146099 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705971+0000", 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705971+0000", 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.603 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705972+0000", 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2359999999999999e-06 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705976+0000", 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1470000000000003e-06 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.604 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705986+0000", 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0161e-05 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705993+0000", 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9330000000000004e-06 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.719530+0000", 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013537077999999999 2024-09-02T10:32:25.605 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.719606+0000", 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5570999999999999e-05 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.719685+0000", 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9579000000000005e-05 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.606 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.723078+0000", 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033929590000000001 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.756155+0000", 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.033076326000000003 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.080171+0000", 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32401667499999998 2024-09-02T10:32:25.607 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.080216+0000", 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5018999999999997e-05 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.081532+0000", 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001315491 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.608 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27559.0:146120 8.7 e848/844 8:ef3a04bb:::benchmark_data_smithi187_136059_object40839:head v 848'9181, mlcod=810'2587)", 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.740255+0000", 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 63.778830483, 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35431497699999998, 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.609 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.740255+0000", 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.740255+0000", 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.740256+0000", 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.345e-06 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.610 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.740265+0000", 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4919999999999993e-06 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.089903+0000", 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34963781300000002 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.089907+0000", 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5709999999999999e-06 2024-09-02T10:32:25.611 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.089956+0000", 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8398e-05 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.089983+0000", 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7163999999999999e-05 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.612 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.094533+0000", 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045505709999999998 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.094570+0000", 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6622999999999998e-05 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.613 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146125 8.6 8:68b9d70e:::benchmark_data_smithi187_136059_object40844:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.744380+0000", 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 63.774705232000002, 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36087077000000001, 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146125 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.614 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.744380+0000", 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.744380+0000", 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.615 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.744382+0000", 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3400000000000001e-06 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.744388+0000", 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0850000000000002e-06 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.744404+0000", 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6643e-05 2024-09-02T10:32:25.616 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.744413+0000", 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3469999999999999e-06 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.747212+0000", 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027998200000000002 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.617 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.747290+0000", 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7560999999999998e-05 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.747385+0000", 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4486000000000003e-05 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.756208+0000", 2024-09-02T10:32:25.618 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0088230600000000006 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.756222+0000", 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4309e-05 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.103781+0000", 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34755899899999998 2024-09-02T10:32:25.619 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.103821+0000", 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0139999999999999e-05 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.105251+0000", 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00142998 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.620 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146131 8.6 8:6708c0fe:::benchmark_data_smithi187_136059_object40850:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.749657+0000", 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 63.769428812999998, 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.38950238999999998, 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.621 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146131 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749657+0000", 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749657+0000", 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.622 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749658+0000", 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1799999999999999e-06 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749662+0000", 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6519999999999997e-06 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.623 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749674+0000", 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1518000000000001e-05 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749681+0000", 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8759999999999999e-06 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.751393+0000", 2024-09-02T10:32:25.624 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017116729999999999 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.751501+0000", 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108157 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.751635+0000", 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013469600000000001 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.625 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.756230+0000", 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045950890000000001 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.097478+0000", 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34124748199999999 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.137802+0000", 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.040324368999999999 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.626 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.137851+0000", 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8349000000000001e-05 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.139159+0000", 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013083490000000001 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.627 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:149587 8.9 8:91d8a8a5:::benchmark_data_smithi187_136059_object44306:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e851)", 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:27.311994+0000", 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 58.207091390000002, 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26515585000000003, 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 149587 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.628 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.311994+0000", 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.311994+0000", 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.629 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.311995+0000", 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1379999999999999e-06 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.312001+0000", 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3510000000000001e-06 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.630 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.312010+0000", 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.285e-06 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.312015+0000", 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1959999999999999e-06 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.314202+0000", 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021866939999999999 2024-09-02T10:32:25.631 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.314303+0000", 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000101679 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,5", 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.314394+0000", 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0452999999999996e-05 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.632 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.318068+0000", 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036743140000000001 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.319851+0000", 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017832290000000001 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.577080+0000", 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.257228862 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.633 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.577112+0000", 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1353999999999998e-05 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.577150+0000", 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8294999999999998e-05 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.634 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:149588 8.2 8:47503dca:::benchmark_data_smithi187_136059_object44307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e851)", 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:27.313803+0000", 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 58.205281999, 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.249002801, 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.635 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 149588 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.313803+0000", 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.313803+0000", 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.636 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.313805+0000", 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0389999999999999e-06 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.313809+0000", 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9620000000000001e-06 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.313819+0000", 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7829999999999993e-06 2024-09-02T10:32:25.637 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.313825+0000", 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3569999999999997e-06 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.313888+0000", 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3396000000000001e-05 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.638 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.314013+0000", 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012450100000000001 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,4", 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.314129+0000", 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116937 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.317916+0000", 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0037868469999999999 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.639 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.318480+0000", 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00056347400000000001 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.562697+0000", 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.244217025 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.562748+0000", 2024-09-02T10:32:25.640 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1375999999999997e-05 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.562806+0000", 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8103999999999998e-05 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:149588 8.2 e851/792 ondisk, result = 0)", 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:27.319052+0000", 2024-09-02T10:32:25.641 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 58.200033173000001, 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24375923499999999, 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.319052+0000", 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.319052+0000", 2024-09-02T10:32:25.642 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.319053+0000", 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3300000000000001e-07 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.319057+0000", 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1060000000000002e-06 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.643 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.562625+0000", 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.243567437 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.562630+0000", 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2399999999999998e-06 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.562681+0000", 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1405e-05 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.644 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.562691+0000", 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4099999999999997e-06 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:27.562812+0000", 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012090400000000001 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.645 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:179906 8.6 8:6550510d:::benchmark_data_smithi187_136059_object74625:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e868)", 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:23.481013+0000", 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 2.0380720600000002, 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24370868600000001, 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 179906 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.646 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.481013+0000", 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.481013+0000", 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.481021+0000", 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7919999999999993e-06 2024-09-02T10:32:25.647 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.481026+0000", 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8389999999999997e-06 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.481040+0000", 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4078e-05 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.648 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.481048+0000", 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1340000000000004e-06 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.484727+0000", 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00367907 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.484923+0000", 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00019536799999999999 2024-09-02T10:32:25.649 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.485059+0000", 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013651700000000001 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.495443+0000", 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.010384048 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.650 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.495485+0000", 2024-09-02T10:32:25.651 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2092000000000003e-05 2024-09-02T10:32:25.651 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.651 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.651 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.723587+0000", 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.228101217 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.723663+0000", 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6563000000000005e-05 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.724722+0000", 2024-09-02T10:32:25.652 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001058968 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:179909 8.6 8:6f5f11bd:::benchmark_data_smithi187_136059_object74628:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e868)", 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:23.484221+0000", 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 2.0348648389999999, 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.241527303, 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:25.653 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 179909 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.484221+0000", 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.484221+0000", 2024-09-02T10:32:25.654 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.484222+0000", 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.178e-06 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.484227+0000", 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3380000000000004e-06 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.655 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:25.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.484241+0000", 2024-09-02T10:32:25.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4283e-05 2024-09-02T10:32:25.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:25.656 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.484251+0000", 2024-09-02T10:32:25.656 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.825e-06 2024-09-02T10:32:25.656 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.656 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.656 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.490183+0000", 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0059317989999999998 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.490515+0000", 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00033236799999999999 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.490741+0000", 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00022525299999999999 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.657 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.495917+0000", 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051758990000000003 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.496752+0000", 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00083567500000000002 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.724784+0000", 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22803147900000001 2024-09-02T10:32:25.658 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.724854+0000", 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9867e-05 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:23.725748+0000", 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00089433899999999996 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:25.659 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:25.660 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:32:25.676 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:25.701 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:25.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:25.803 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:25.831 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:25.832 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:25.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:25.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:25.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:26.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:26.103 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:26.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:26.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:26.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:26.237 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:26.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:26.339 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:26.373 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:26.374 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:26.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:26.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:26.507 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:26.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:26.630 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:32:26.636 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:26.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:26.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:26.763 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:26.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:26.865 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:26.891 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:26.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:26.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:32:26.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:27.014 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:32:27.016 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:27.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:27.037 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:32:27.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:27.154 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:27.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:27.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:27.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:27.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:27 smithi187 ceph-mon[20072]: pgmap v1991: 47 pgs: 2 active+recovering+forced_recovery+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 1872/92853 objects degraded (2.016%); 11019/92853 objects misplaced (11.867%); 11 MiB/s, 184 objects/s recovering 2024-09-02T10:32:27.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/172270246' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:32:27.260 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:27 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:32:27.287 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:27.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:27.335 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:27 smithi162 ceph-mon[20279]: pgmap v1991: 47 pgs: 2 active+recovering+forced_recovery+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 1872/92853 objects degraded (2.016%); 11019/92853 objects misplaced (11.867%); 11 MiB/s, 184 objects/s recovering 2024-09-02T10:32:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/172270246' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:32:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:27 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:32:27.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:27.411 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:27.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:27.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:27.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:27 smithi151 ceph-mon[16639]: pgmap v1991: 47 pgs: 2 active+recovering+forced_recovery+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 581 op/s; 1872/92853 objects degraded (2.016%); 11019/92853 objects misplaced (11.867%); 11 MiB/s, 184 objects/s recovering 2024-09-02T10:32:27.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/172270246' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:32:27.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:27 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:32:27.513 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:27.537 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:27.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:27.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:27.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:27.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:27.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:27.798 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:27.799 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:27.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:27.913 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:32:27.913 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:182411 8.f 8:fcc031fd:::benchmark_data_smithi187_136059_object77130:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:27.885265+0000", 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.026952402, 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.026977210000000001, 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 182411 2024-09-02T10:32:27.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.885265+0000", 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.885265+0000", 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:27.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.885268+0000", 2024-09-02T10:32:27.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3970000000000001e-06 2024-09-02T10:32:27.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:27.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.885275+0000", 2024-09-02T10:32:27.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3250000000000002e-06 2024-09-02T10:32:27.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.885296+0000", 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1288999999999999e-05 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.885306+0000", 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4369999999999996e-06 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.885379+0000", 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2958000000000002e-05 2024-09-02T10:32:27.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.885513+0000", 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013397499999999999 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.885635+0000", 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012240100000000001 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.890903+0000", 2024-09-02T10:32:27.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0052676210000000001 2024-09-02T10:32:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:27.896009+0000", 2024-09-02T10:32:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051057150000000003 2024-09-02T10:32:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:27.919 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:32:27.920 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:32:27.920 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:32:27.928 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:27.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:27.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:28.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:28.055 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:28.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:28.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:28.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:28.189 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:28.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:28.291 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:28.318 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:28.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:28 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:32:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:28 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:32:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:28 smithi162 ceph-mon[20279]: osdmap e869: 12 total, 12 up, 12 in 2024-09-02T10:32:28.420 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:28 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:32:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:28 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:32:28.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:28 smithi151 ceph-mon[16639]: osdmap e869: 12 total, 12 up, 12 in 2024-09-02T10:32:28.451 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:28.452 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:28.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:28 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:32:28.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:28 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:32:28.487 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:28 smithi187 ceph-mon[20072]: osdmap e869: 12 total, 12 up, 12 in 2024-09-02T10:32:28.553 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:28.586 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:28.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:28.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:28.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:32:28.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:28.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:28.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:28.816 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:28.840 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:28.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:28.942 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:28.966 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:28.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:28.968 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:29.068 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:29.096 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:29.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:29.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:29.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:29.219 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:29.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:29.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:29.284 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:29 smithi187 ceph-mon[20072]: pgmap v1993: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s; 1872/89550 objects degraded (2.090%); 10545/89550 objects misplaced (11.776%); 12 MiB/s, 194 objects/s recovering 2024-09-02T10:32:29.321 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:29.343 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:29.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:29.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:29 smithi162 ceph-mon[20279]: pgmap v1993: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s; 1872/89550 objects degraded (2.090%); 10545/89550 objects misplaced (11.776%); 12 MiB/s, 194 objects/s recovering 2024-09-02T10:32:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:29 smithi151 ceph-mon[16639]: pgmap v1993: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 563 op/s; 1872/89550 objects degraded (2.090%); 10545/89550 objects misplaced (11.776%); 12 MiB/s, 194 objects/s recovering 2024-09-02T10:32:29.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:29.466 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:29.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:29.486 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:32:29.486 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:32:29.486 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:32:29.486 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:32:29.486 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:32:29.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:29.598 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:29.600 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:29.700 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:29.724 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:29.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:29.726 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:29.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:29.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:29.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:29.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:29.984 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:29.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:30.086 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:30.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:30.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:30.177 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:30 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1872/89550 objects degraded (2.090%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:30.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:30.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:32:30.237 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:30.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:30.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:30.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:30 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1872/89550 objects degraded (2.090%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:30.362 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:30.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:30.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:30.394 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:32:30.394 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:32:30.395 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-02T10:32:30.395 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:32:30.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:30 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1872/89550 objects degraded (2.090%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:30.464 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:30.488 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:30.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:30.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:30.589 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:32:30.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:30.615 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:30.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:30.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:30.740 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:30.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:30.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:30.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:30.864 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:30.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:30.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:30.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:30.991 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:30.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:31.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:31.128 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:31.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:31.129 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:31.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:31.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:31 smithi187 ceph-mon[20072]: pgmap v1994: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 2012/85185 objects degraded (2.362%); 9835/85185 objects misplaced (11.545%); 12 MiB/s, 188 objects/s recovering 2024-09-02T10:32:31.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:31 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:32:31.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:31 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:32:31.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:31 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:32:31.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:31 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:32:31.265 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:31.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:31 smithi162 ceph-mon[20279]: pgmap v1994: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 2012/85185 objects degraded (2.362%); 9835/85185 objects misplaced (11.545%); 12 MiB/s, 188 objects/s recovering 2024-09-02T10:32:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:31 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:32:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:31 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:32:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:31 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:32:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:31 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:32:31.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:31.390 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:31.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:31 smithi151 ceph-mon[16639]: pgmap v1994: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 575 op/s; 2012/85185 objects degraded (2.362%); 9835/85185 objects misplaced (11.545%); 12 MiB/s, 188 objects/s recovering 2024-09-02T10:32:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:31 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:32:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:31 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:32:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:31 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:32:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:31 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:32:31.492 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:31.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:31.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:31.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:31.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:31.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:31.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:31.782 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:31.783 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:31.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:31.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:31.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:31.993 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:32:31.993 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:32:31.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:32:31.993 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:32:31.993 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.993 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDScrubReserve(6.8,RELEASE,e:652,from:7,reservation_nonce:0)", 2024-09-02T10:32:31.993 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:23:46.589381+0000", 2024-09-02T10:32:31.993 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 524.68153932200005, 2024-09-02T10:32:31.993 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.48470698099999998, 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:46.589381+0000", 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:46.589381+0000", 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:31.994 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:46.589381+0000", 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3399999999999999e-07 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:46.589401+0000", 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9202999999999999e-05 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:47.073954+0000", 2024-09-02T10:32:31.995 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.48455304599999999 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:47.073967+0000", 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3286e-05 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:47.074051+0000", 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4165999999999993e-05 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.996 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:47.074072+0000", 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0877e-05 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:23:47.074088+0000", 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5869e-05 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:45631 7.a e689/661 7:579b01e3:::benchmark_data_smithi187_120712_object45630:head v 689'2850, mlcod=688'2848)", 2024-09-02T10:32:31.997 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:24:53.385098+0000", 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 457.88582215999998, 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25756838500000001, 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.385098+0000", 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:31.998 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.385098+0000", 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.385099+0000", 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3370000000000001e-06 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.385128+0000", 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9105999999999999e-05 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:31.999 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.641211+0000", 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25608216299999997 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.641217+0000", 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8329999999999996e-06 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.641273+0000", 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5835000000000003e-05 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.000 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.641295+0000", 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1977e-05 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.642634+0000", 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013388969999999999 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:24:53.642666+0000", 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2237000000000003e-05 2024-09-02T10:32:32.001 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:71617 7.b e711/661 7:d1c8b968:::benchmark_data_smithi187_120712_object71616:head v 711'4410, mlcod=711'4407)", 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:27.803500+0000", 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 423.46742023500002, 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33549837500000002, 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.002 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:27.803500+0000", 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:27.803500+0000", 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:27.803501+0000", 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0449999999999999e-06 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.003 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:27.803556+0000", 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5214999999999999e-05 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.136887+0000", 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33333045 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.136896+0000", 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1300000000000007e-06 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.004 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.136955+0000", 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8792999999999998e-05 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.136974+0000", 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9411000000000001e-05 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.138967+0000", 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019933170000000001 2024-09-02T10:32:32.005 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.138998+0000", 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1013999999999998e-05 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:100943 7.c e723/717 7:3c179c28:::benchmark_data_smithi187_120712_object17001:head v 723'6262, mlcod=723'6259)", 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:01.695823+0000", 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 389.57509737200002, 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26241625899999999, 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.006 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.695823+0000", 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.695823+0000", 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.007 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.695824+0000", 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6599999999999995e-07 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.695830+0000", 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9340000000000003e-06 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.955758+0000", 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25992827400000001 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.008 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.955762+0000", 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1749999999999998e-06 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.955831+0000", 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8554999999999997e-05 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.955859+0000", 2024-09-02T10:32:32.009 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8867999999999999e-05 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.958208+0000", 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023483219999999999 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.958239+0000", 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1365000000000002e-05 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.010 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:136920 7.5 7:a235c29c:::benchmark_data_smithi187_120712_object52978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:43.766046+0000", 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 347.50487397900002, 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46275040899999997, 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 136920 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.011 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766046+0000", 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766046+0000", 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766048+0000", 2024-09-02T10:32:32.012 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.415e-06 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766053+0000", 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0549999999999999e-06 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766064+0000", 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1457999999999999e-05 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766070+0000", 2024-09-02T10:32:32.013 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0700000000000003e-06 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766119+0000", 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9104999999999998e-05 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766205+0000", 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5873999999999998e-05 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:32:32.014 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766280+0000", 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4931999999999993e-05 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768103+0000", 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0018229419999999999 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768304+0000", 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00020122 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.015 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228691+0000", 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46038674499999999 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228748+0000", 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6935000000000003e-05 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228797+0000", 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8658000000000001e-05 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.016 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:136920 7.5 e743/741 ondisk, result = 0)", 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:43.768313+0000", 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 347.50260675300001, 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46048927299999998, 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768313+0000", 2024-09-02T10:32:32.017 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768313+0000", 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768314+0000", 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8500000000000002e-07 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768317+0000", 2024-09-02T10:32:32.018 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6070000000000002e-06 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228601+0000", 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.460284949 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228610+0000", 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8789999999999992e-06 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.019 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228674+0000", 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3646999999999996e-05 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228684+0000", 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0288e-05 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228803+0000", 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118418 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.020 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:136934 7.5 7:a1526dd1:::benchmark_data_smithi187_120712_object52992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:44.230426+0000", 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 347.04049379600002, 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.212588995, 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:32:32.021 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 136934 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230426+0000", 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230426+0000", 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.022 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230428+0000", 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6190000000000001e-06 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230434+0000", 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1709999999999999e-06 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230449+0000", 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4666e-05 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.023 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230457+0000", 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8590000000000005e-06 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.231739+0000", 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00128243 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.231878+0000", 2024-09-02T10:32:32.024 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013864199999999999 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.231991+0000", 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011313499999999999 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.234150+0000", 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021594489999999999 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.025 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.234408+0000", 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025732799999999997 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.442935+0000", 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.208527407 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.442978+0000", 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2558000000000001e-05 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.026 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.443015+0000", 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7731000000000002e-05 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:136934 7.5 e744/741 ondisk, result = 0)", 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:44.234783+0000", 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 347.03613735900001, 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20823810100000001, 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.027 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.234783+0000", 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.234783+0000", 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.028 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.234783+0000", 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7800000000000002e-07 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.234786+0000", 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.729e-06 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.442824+0000", 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20803751500000001 2024-09-02T10:32:32.029 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.442831+0000", 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5769999999999996e-06 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.442922+0000", 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1249000000000007e-05 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.442931+0000", 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1790000000000008e-06 2024-09-02T10:32:32.030 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.443021+0000", 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0373999999999999e-05 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:136942 7.5 7:a20fdecd:::benchmark_data_smithi187_120712_object53000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e745)", 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:45.462956+0000", 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 345.80796434600001, 2024-09-02T10:32:32.031 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23792718600000001, 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 136942 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.462956+0000", 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.032 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.462956+0000", 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.462958+0000", 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.294e-06 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.462968+0000", 2024-09-02T10:32:32.033 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6569999999999992e-06 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.462990+0000", 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2633000000000002e-05 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.463003+0000", 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2517000000000001e-05 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.034 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.463095+0000", 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1850999999999999e-05 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op must wait for map", 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.463104+0000", 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7509999999999997e-06 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.465089+0000", 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001985419 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.035 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.465219+0000", 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012985199999999999 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.465339+0000", 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120386 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.467783+0000", 2024-09-02T10:32:32.036 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024435809999999998 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.468202+0000", 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00041971700000000001 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.700799+0000", 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.232596409 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.037 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.700836+0000", 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6619999999999998e-05 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.700883+0000", 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7499e-05 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:21803 8.6 8:6cea7233:::benchmark_data_smithi187_136059_object21802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e804)", 2024-09-02T10:32:32.038 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:13.620281+0000", 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 197.65063954600001, 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21402042600000001, 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 21803 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.039 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620281+0000", 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620281+0000", 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620282+0000", 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1570000000000001e-06 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.040 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620313+0000", 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1090000000000002e-05 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620324+0000", 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.115e-05 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620332+0000", 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0120000000000005e-06 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.041 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620376+0000", 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3488000000000001e-05 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620478+0000", 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000102246 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620611+0000", 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013364200000000001 2024-09-02T10:32:32.042 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.043 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.621411+0000", 2024-09-02T10:32:32.043 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00079942399999999997 2024-09-02T10:32:32.043 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.043 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.043 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.623938+0000", 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025271460000000001 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834210+0000", 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21027237400000001 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834263+0000", 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3063999999999999e-05 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.044 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834301+0000", 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7632999999999997e-05 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:21820 8.6 8:6a3536fe:::benchmark_data_smithi187_136059_object21819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e804)", 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:13.629639+0000", 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 197.641281392, 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21678998799999999, 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.045 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 21820 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629639+0000", 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.046 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629639+0000", 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629641+0000", 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0169999999999999e-06 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629670+0000", 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9173999999999999e-05 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.047 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629689+0000", 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9045000000000001e-05 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629697+0000", 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6199999999999999e-06 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629747+0000", 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9968e-05 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.048 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629904+0000", 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000157656 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.630021+0000", 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116347 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.631099+0000", 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010783819999999999 2024-09-02T10:32:32.049 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.631117+0000", 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7915e-05 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.846333+0000", 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21521643500000001 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.846391+0000", 2024-09-02T10:32:32.050 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7169000000000001e-05 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.846429+0000", 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8260000000000003e-05 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:21803 8.6 e804/792 ondisk, result = 0)", 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:13.631234+0000", 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 197.63968659299999, 2024-09-02T10:32:32.051 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20307475899999999, 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.631234+0000", 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.631234+0000", 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.052 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.631234+0000", 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3900000000000005e-07 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.631237+0000", 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2140000000000001e-06 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834139+0000", 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.202901214 2024-09-02T10:32:32.053 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834145+0000", 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6490000000000002e-06 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834192+0000", 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6536000000000003e-05 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834204+0000", 2024-09-02T10:32:32.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2298e-05 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834308+0000", 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010430899999999999 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28292 8.d 8:b2a9e2ae:::benchmark_data_smithi187_136059_object28291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.105898+0000", 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 193.165022085, 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21874021900000001, 2024-09-02T10:32:32.055 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28292 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105898+0000", 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105898+0000", 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105899+0000", 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.068e-06 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105924+0000", 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5179000000000001e-05 2024-09-02T10:32:32.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105938+0000", 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3713000000000001e-05 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105946+0000", 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6559999999999992e-06 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.106008+0000", 2024-09-02T10:32:32.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2001999999999994e-05 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.106149+0000", 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014115800000000001 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.106281+0000", 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000132466 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.059 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.107636+0000", 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001355055 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.113043+0000", 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0054062870000000001 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324567+0000", 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21152405399999999 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.060 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324604+0000", 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7017999999999997e-05 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324638+0000", 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4563000000000001e-05 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28300 8.f 8:f195aa31:::benchmark_data_smithi187_136059_object28299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.108959+0000", 2024-09-02T10:32:32.061 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 193.16196146999999, 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21860515799999999, 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28300 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108959+0000", 2024-09-02T10:32:32.062 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108959+0000", 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108960+0000", 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5209999999999999e-06 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.063 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108983+0000", 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3189000000000001e-05 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108998+0000", 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4542000000000001e-05 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109005+0000", 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4170000000000003e-06 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.064 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109063+0000", 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7337999999999999e-05 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109178+0000", 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115203 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109310+0000", 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000132156 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.065 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.110649+0000", 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013386990000000001 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111465+0000", 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0008164 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327469+0000", 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.216004207 2024-09-02T10:32:32.066 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327527+0000", 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7271000000000001e-05 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327564+0000", 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7215000000000002e-05 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.067 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28305 8.d 8:bd02d929:::benchmark_data_smithi187_136059_object28304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.111275+0000", 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 193.15964497499999, 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21380102300000001, 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28305 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.068 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111275+0000", 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111275+0000", 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111276+0000", 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4399999999999999e-07 2024-09-02T10:32:32.069 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111295+0000", 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8816000000000001e-05 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111305+0000", 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0589e-05 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.070 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111311+0000", 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2650000000000004e-06 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111387+0000", 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6338e-05 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111513+0000", 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012598000000000001 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.071 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111655+0000", 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000141726 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112644+0000", 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00098899699999999992 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.127377+0000", 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014733448 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.072 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324992+0000", 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19761451799999999 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.325035+0000", 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3201999999999997e-05 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.325076+0000", 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1300000000000001e-05 2024-09-02T10:32:32.073 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28306 8.f 8:fcc25ce2:::benchmark_data_smithi187_136059_object28305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.112056+0000", 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 193.15886417999999, 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21564313299999999, 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:32.074 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28306 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112056+0000", 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112056+0000", 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.075 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112057+0000", 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.144e-06 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112081+0000", 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3713000000000001e-05 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112091+0000", 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8090000000000004e-06 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.076 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112098+0000", 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0990000000000001e-06 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112150+0000", 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1768999999999999e-05 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112236+0000", 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6237999999999997e-05 2024-09-02T10:32:32.077 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112316+0000", 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0603000000000001e-05 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.113365+0000", 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010481539999999999 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.113551+0000", 2024-09-02T10:32:32.078 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018640599999999999 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327611+0000", 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21405996499999999 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327664+0000", 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2982999999999997e-05 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.079 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327699+0000", 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5250000000000003e-05 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28313 8.f 8:fb500fbb:::benchmark_data_smithi187_136059_object28312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.117307+0000", 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 193.15361297800001, 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.210606557, 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.080 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28313 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117307+0000", 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.081 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117307+0000", 2024-09-02T10:32:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117308+0000", 2024-09-02T10:32:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1990000000000001e-06 2024-09-02T10:32:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.082 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117328+0000", 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9528999999999999e-05 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117342+0000", 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4219000000000001e-05 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117350+0000", 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0120000000000005e-06 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.083 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117456+0000", 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106291 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117573+0000", 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011608 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117677+0000", 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010426500000000001 2024-09-02T10:32:32.084 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.118658+0000", 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00098097000000000006 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.119074+0000", 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000416384 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327819+0000", 2024-09-02T10:32:32.085 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20874446899999999 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327880+0000", 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1617000000000003e-05 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327914+0000", 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3522000000000002e-05 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.086 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146135 8.f 8:ffa01c02:::benchmark_data_smithi187_136059_object40854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.753711+0000", 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 69.517209148000006, 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33445889499999998, 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146135 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.087 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753711+0000", 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753711+0000", 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753713+0000", 2024-09-02T10:32:32.088 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8589999999999999e-06 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753720+0000", 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2579999999999998e-06 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753739+0000", 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8916999999999999e-05 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.089 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753748+0000", 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1190000000000003e-06 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.755012+0000", 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012635649999999999 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.755145+0000", 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013299500000000001 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.090 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.755260+0000", 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115469 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.758833+0000", 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035724749999999999 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.768082+0000", 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0092492670000000003 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.091 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.088097+0000", 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32001463499999999 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.088136+0000", 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9756999999999999e-05 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.088170+0000", 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3578999999999998e-05 2024-09-02T10:32:32.092 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146139 8.f 8:f849bf1d:::benchmark_data_smithi187_136059_object40858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.773477+0000", 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 69.497443240999999, 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31800191100000003, 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:32.093 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146139 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773477+0000", 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773477+0000", 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.094 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773478+0000", 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1790000000000001e-06 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773483+0000", 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8280000000000002e-06 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773494+0000", 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1419e-05 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.095 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773500+0000", 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.874e-06 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773553+0000", 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2957999999999997e-05 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773667+0000", 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113812 2024-09-02T10:32:32.096 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773777+0000", 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010969899999999999 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.777216+0000", 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034393179999999998 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.777885+0000", 2024-09-02T10:32:32.097 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00066886699999999996 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.091413+0000", 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31352819799999998 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.091446+0000", 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2802999999999997e-05 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.091479+0000", 2024-09-02T10:32:32.098 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2956000000000001e-05 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:183126 8.f 8:fec0a0af:::benchmark_data_smithi187_136059_object77845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:29.326892+0000", 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 1.9440283869999999, 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23921741799999999, 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:32:32.099 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183126 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326892+0000", 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326892+0000", 2024-09-02T10:32:32.100 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326894+0000", 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2029999999999999e-06 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326903+0000", 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2240000000000001e-06 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:32:32.101 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326925+0000", 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2033000000000001e-05 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326937+0000", 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1522000000000001e-05 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.329339+0000", 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024024099999999998 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.102 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.329467+0000", 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000127508 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.329604+0000", 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013698200000000001 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.334705+0000", 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051010439999999999 2024-09-02T10:32:32.103 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.347683+0000", 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012978701 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.566044+0000", 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.218360309 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.566080+0000", 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6727999999999998e-05 2024-09-02T10:32:32.104 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:32:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:32:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:32:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.566109+0000", 2024-09-02T10:32:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8753999999999999e-05 2024-09-02T10:32:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:32:32.105 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:32:32.105 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:32:32.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:32.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:32.136 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:32.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:32 smithi187 ceph-mon[20072]: osdmap e870: 12 total, 12 up, 12 in 2024-09-02T10:32:32.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:32.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:32 smithi162 ceph-mon[20279]: osdmap e870: 12 total, 12 up, 12 in 2024-09-02T10:32:32.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:32.396 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:32.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:32.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:32 smithi151 ceph-mon[16639]: osdmap e870: 12 total, 12 up, 12 in 2024-09-02T10:32:32.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:32.524 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:32.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:32.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:32.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:32.648 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:32.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:32.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:32.775 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:32.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:32.877 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:32.908 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:32.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:33.010 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:33.034 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:33.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:33.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:33.136 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:33.161 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:33.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:33.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:33.288 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:33.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:33.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:33.339 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:33 smithi151 ceph-mon[16639]: pgmap v1996: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 2012/84231 objects degraded (2.389%); 9835/84231 objects misplaced (11.676%); 11 MiB/s, 171 objects/s recovering 2024-09-02T10:32:33.339 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:33 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:32:33.339 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:33 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:32:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:33 smithi162 ceph-mon[20279]: pgmap v1996: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 2012/84231 objects degraded (2.389%); 9835/84231 objects misplaced (11.676%); 11 MiB/s, 171 objects/s recovering 2024-09-02T10:32:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:33 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:32:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:33 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:32:33.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:33.433 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:33.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:33.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:33 smithi187 ceph-mon[20072]: pgmap v1996: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 513 op/s; 2012/84231 objects degraded (2.389%); 9835/84231 objects misplaced (11.676%); 11 MiB/s, 171 objects/s recovering 2024-09-02T10:32:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:33 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:32:33.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:33 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:32:33.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:33.568 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:33.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:33.670 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:33.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:33.698 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:33.799 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:33.827 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:33.828 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:33.929 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:33.958 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:33.959 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:34.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:34.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:34.093 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:34.194 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:34.230 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:34.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:34.232 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:34.333 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:34.358 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:34.359 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:34.460 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:34.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:34.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:34.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:34.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:34.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:34.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:34.729 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:34.751 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:34.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:34.853 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:34.877 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:34.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:34.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:35.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:35.008 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:35.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:35.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:35.136 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:35.237 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:35 smithi187 ceph-mon[20072]: pgmap v1997: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 2012/79908 objects degraded (2.518%); 9147/79908 objects misplaced (11.447%); 13 MiB/s, 206 objects/s recovering 2024-09-02T10:32:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:35 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:32:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:35 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:32:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:35 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2012/79908 objects degraded (2.518%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:35.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:35.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:35.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:35 smithi162 ceph-mon[20279]: pgmap v1997: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 2012/79908 objects degraded (2.518%); 9147/79908 objects misplaced (11.447%); 13 MiB/s, 206 objects/s recovering 2024-09-02T10:32:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:35 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:32:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:35 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:32:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:35 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2012/79908 objects degraded (2.518%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:35.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:35.397 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:35.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:35.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:35.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:35 smithi151 ceph-mon[16639]: pgmap v1997: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.6 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 539 op/s; 2012/79908 objects degraded (2.518%); 9147/79908 objects misplaced (11.447%); 13 MiB/s, 206 objects/s recovering 2024-09-02T10:32:35.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:35 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:32:35.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:35 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:32:35.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:35 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2012/79908 objects degraded (2.518%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:35.500 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:35.529 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:35.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:35.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:35.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:35.655 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:35.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:35.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:35.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:32:35.757 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:35.784 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:35.785 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:35.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:35.908 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:35.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:36.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:36.032 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:36.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:36.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:36.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:36.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:36.264 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:36.303 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:36.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:36.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:36.427 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:36.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:36.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:36.481 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:32:36.528 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:36.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:36.558 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:36.558 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:36.660 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:36.681 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:36.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:36.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:36.784 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:36.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:36.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:36.928 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:36.951 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:36.952 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:37.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:37.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:37.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:37.188 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:37.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:37.224 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:37.225 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:37.326 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:37 smithi162 ceph-mon[20279]: pgmap v1998: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 2113/76164 objects degraded (2.774%); 8502/76164 objects misplaced (11.163%); 12 MiB/s, 189 objects/s recovering 2024-09-02T10:32:37.350 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:37.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:37.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:37 smithi151 ceph-mon[16639]: pgmap v1998: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 2113/76164 objects degraded (2.774%); 8502/76164 objects misplaced (11.163%); 12 MiB/s, 189 objects/s recovering 2024-09-02T10:32:37.452 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:37.481 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:37.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:37.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:37 smithi187 ceph-mon[20072]: pgmap v1998: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 522 op/s; 2113/76164 objects degraded (2.774%); 8502/76164 objects misplaced (11.163%); 12 MiB/s, 189 objects/s recovering 2024-09-02T10:32:37.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:37.611 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:37.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:37.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:37.713 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:37.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:37.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:37.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:37.848 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:37.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:37.878 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:37.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:38.005 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:38.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:38.006 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:38.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:38.132 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:38.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:38.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:38.255 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2925157687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:32:38.258 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:38.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:38.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2925157687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:32:38.359 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:38.384 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:38.384 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:38.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:38.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2925157687' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:32:38.509 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:38.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:38.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:38.611 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:38.638 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:38.639 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:38.740 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:38.764 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:38.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:38.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:38.774 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:32:38.774 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:32:38.774 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:32:38.774 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:32:38.774 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:32:38.774 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:32:38.774 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:32:38.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:32:38.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:32:38.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:32:38.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:32:38.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:32:38.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:32:38.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:32:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:32:38.775 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:32:38.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:32:38.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:32:38.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:32:38.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:32:38.776 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:32:38.776 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:32:38.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:32:38.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:32:38.865 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:38.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:38.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:39.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:39.029 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:39.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:39.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:39.131 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:39.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:39.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:39.163 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:39.264 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:39.290 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:39.290 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:39 smithi162 ceph-mon[20279]: pgmap v1999: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 2113/72888 objects degraded (2.899%); 8010/72888 objects misplaced (10.989%); 13 MiB/s, 217 objects/s recovering 2024-09-02T10:32:39.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:39.426 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:39.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:39.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:39.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:39 smithi151 ceph-mon[16639]: pgmap v1999: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 2113/72888 objects degraded (2.899%); 8010/72888 objects misplaced (10.989%); 13 MiB/s, 217 objects/s recovering 2024-09-02T10:32:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:39 smithi187 ceph-mon[20072]: pgmap v1999: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.5 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 555 op/s; 2113/72888 objects degraded (2.899%); 8010/72888 objects misplaced (10.989%); 13 MiB/s, 217 objects/s recovering 2024-09-02T10:32:39.528 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:39.551 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:39.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:39.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:39.653 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:39.682 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:39.682 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:39.783 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:39.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:39.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:39.916 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:39.939 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:39.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:40.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:40.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:40.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:40.174 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:40.197 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:40.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:40.299 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:40.328 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:40.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:40 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2113/72888 objects degraded (2.899%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:40.420 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:40 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2113/72888 objects degraded (2.899%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:40.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:40.458 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:40.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:40.459 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:40.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:40 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2113/72888 objects degraded (2.899%), 5 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:40.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:40.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:40.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:40.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:40.716 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:40.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:40.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:40.819 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:40.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:40.850 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:40.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:40.978 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:40.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:40.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:41.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:41.105 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:41.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:41.207 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:41.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:41.240 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:41.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:41.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:41 smithi162 ceph-mon[20279]: pgmap v2000: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 2233/68307 objects degraded (3.269%); 7362/68307 objects misplaced (10.778%); 13 MiB/s, 213 objects/s recovering 2024-09-02T10:32:41.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:41.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:41.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:41 smithi151 ceph-mon[16639]: pgmap v2000: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 2233/68307 objects degraded (3.269%); 7362/68307 objects misplaced (10.778%); 13 MiB/s, 213 objects/s recovering 2024-09-02T10:32:41.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:41.491 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:41.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:41.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:41 smithi187 ceph-mon[20072]: pgmap v2000: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 562 op/s; 2233/68307 objects degraded (3.269%); 7362/68307 objects misplaced (10.778%); 13 MiB/s, 213 objects/s recovering 2024-09-02T10:32:41.593 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:41.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:41.622 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:41.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:41.748 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:41.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:41.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:41.850 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:41.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:41.882 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:41.884 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:41.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:42.012 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:42.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:42.114 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:42.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:42.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:42.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:42.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:42.271 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:42.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:42.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:42.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:42.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:42.406 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:42.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:42.529 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:42.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:42.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:42.631 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:42.664 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:42.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:42.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:42.766 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:42.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:42.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:42.897 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:42.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:42.927 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:42.928 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:43.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:43.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:43.058 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:43.160 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:43.185 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:43.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:43.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:43.315 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:43.315 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:43 smithi162 ceph-mon[20279]: pgmap v2001: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 2233/67338 objects degraded (3.316%); 7362/67338 objects misplaced (10.933%); 12 MiB/s, 202 objects/s recovering 2024-09-02T10:32:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:43 smithi162 ceph-mon[20279]: 8.f scrub starts 2024-09-02T10:32:43.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:43 smithi162 ceph-mon[20279]: 8.f scrub ok 2024-09-02T10:32:43.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:43 smithi151 ceph-mon[16639]: pgmap v2001: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 2233/67338 objects degraded (3.316%); 7362/67338 objects misplaced (10.933%); 12 MiB/s, 202 objects/s recovering 2024-09-02T10:32:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:43 smithi151 ceph-mon[16639]: 8.f scrub starts 2024-09-02T10:32:43.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:43 smithi151 ceph-mon[16639]: 8.f scrub ok 2024-09-02T10:32:43.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:43.439 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:43 smithi187 ceph-mon[20072]: pgmap v2001: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.4 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 534 op/s; 2233/67338 objects degraded (3.316%); 7362/67338 objects misplaced (10.933%); 12 MiB/s, 202 objects/s recovering 2024-09-02T10:32:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:43 smithi187 ceph-mon[20072]: 8.f scrub starts 2024-09-02T10:32:43.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:43 smithi187 ceph-mon[20072]: 8.f scrub ok 2024-09-02T10:32:43.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:43.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:43.564 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:43.665 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:43.686 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:43.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:43.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:43.788 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:43.813 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:43.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:43.915 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:43.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:43.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:44.049 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:44.074 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:44.074 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:44.175 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:44.199 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:44.200 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:44.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:44.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:44.328 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:44.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:44.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:44 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:32:44.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:44 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:32:44.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:44.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:44 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:32:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:44 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:32:44.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:44 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:32:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:44 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:32:44.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:44.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:44.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:44.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:44.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:44.723 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:44.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:44.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:44.846 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:44.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:44.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:44.972 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:44.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:44.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:45.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:45.101 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:45.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:45.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:45.203 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:45.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:45.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:45.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:45.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:45 smithi162 ceph-mon[20279]: pgmap v2002: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 2233/62412 objects degraded (3.578%); 7120/62412 objects misplaced (11.408%); 12 MiB/s, 225 objects/s recovering 2024-09-02T10:32:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:45 smithi162 ceph-mon[20279]: osdmap e871: 12 total, 12 up, 12 in 2024-09-02T10:32:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:45 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2233/62412 objects degraded (3.578%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:45.359 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:45.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:45 smithi151 ceph-mon[16639]: pgmap v2002: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 2233/62412 objects degraded (3.578%); 7120/62412 objects misplaced (11.408%); 12 MiB/s, 225 objects/s recovering 2024-09-02T10:32:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:45 smithi151 ceph-mon[16639]: osdmap e871: 12 total, 12 up, 12 in 2024-09-02T10:32:45.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:45 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2233/62412 objects degraded (3.578%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:45.461 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:45.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:45.488 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:45 smithi187 ceph-mon[20072]: pgmap v2002: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 605 op/s; 2233/62412 objects degraded (3.578%); 7120/62412 objects misplaced (11.408%); 12 MiB/s, 225 objects/s recovering 2024-09-02T10:32:45.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:45 smithi187 ceph-mon[20072]: osdmap e871: 12 total, 12 up, 12 in 2024-09-02T10:32:45.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:45 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2233/62412 objects degraded (3.578%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:45.590 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:45.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:45.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:45.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:45.747 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:45.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:45.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:45.878 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:45.879 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:45.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:46.001 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:32:46.001 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:32:46.002 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:46.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:46.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:46.048 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:32:46.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:46.128 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:46.129 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:46.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:46 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:32:46.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:46 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:32:46.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:46 smithi151 ceph-mon[16639]: osdmap e872: 12 total, 12 up, 12 in 2024-09-02T10:32:46.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:46.258 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:46.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:46.259 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:46.361 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:46.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:46.411 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:46.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:46 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:32:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:46 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:32:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:46 smithi187 ceph-mon[20072]: osdmap e872: 12 total, 12 up, 12 in 2024-09-02T10:32:46.512 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:46.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:46.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:46.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:46.594 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:46 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:32:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:46 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:32:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:46 smithi162 ceph-mon[20279]: osdmap e872: 12 total, 12 up, 12 in 2024-09-02T10:32:46.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:46.660 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:46.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:46.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:46.784 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:46.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:46.885 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:46.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:46.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:46.953 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:32:46.953 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:32:47.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:47.036 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:47.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:47.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:47.163 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:47.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:47.265 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:47.290 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:47.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:47.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:47.345 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:32:47.349 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:32:47.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:47 smithi151 ceph-mon[16639]: pgmap v2005: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 2332/58527 objects degraded (3.984%); 6406/58527 objects misplaced (10.945%); 9.3 MiB/s, 190 objects/s recovering 2024-09-02T10:32:47.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:47 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:32:47.367 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:47 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:32:47.392 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:47.416 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:47.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:47 smithi187 ceph-mon[20072]: pgmap v2005: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 2332/58527 objects degraded (3.984%); 6406/58527 objects misplaced (10.945%); 9.3 MiB/s, 190 objects/s recovering 2024-09-02T10:32:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:47 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:32:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:47 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:32:47.518 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:47.558 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:47.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:47 smithi162 ceph-mon[20279]: pgmap v2005: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.2 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 2332/58527 objects degraded (3.984%); 6406/58527 objects misplaced (10.945%); 9.3 MiB/s, 190 objects/s recovering 2024-09-02T10:32:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:47 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:32:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:47 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:32:47.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:47.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:47.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:47.793 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:47.824 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:47.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:47.926 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:47.949 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:47.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:47.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:32:48.051 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:48.081 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:48.083 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:48.183 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:48.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:48.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:48.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:48.335 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:48.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:48.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:48.437 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:48 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:32:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:48 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:32:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:48 smithi151 ceph-mon[16639]: osdmap e873: 12 total, 12 up, 12 in 2024-09-02T10:32:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:48 smithi151 ceph-mon[16639]: from='client.18720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:32:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:48 smithi151 ceph-mon[16639]: pgmap v2007: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 2332/55296 objects degraded (4.217%); 6211/55296 objects misplaced (11.232%); 9.9 MiB/s, 291 objects/s recovering 2024-09-02T10:32:48.472 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:48.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:48 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:32:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:48 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:32:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:48 smithi187 ceph-mon[20072]: osdmap e873: 12 total, 12 up, 12 in 2024-09-02T10:32:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:48 smithi187 ceph-mon[20072]: from='client.18720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:32:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:48 smithi187 ceph-mon[20072]: pgmap v2007: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 2332/55296 objects degraded (4.217%); 6211/55296 objects misplaced (11.232%); 9.9 MiB/s, 291 objects/s recovering 2024-09-02T10:32:48.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:48 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:32:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:48 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:32:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:48 smithi162 ceph-mon[20279]: osdmap e873: 12 total, 12 up, 12 in 2024-09-02T10:32:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:48 smithi162 ceph-mon[20279]: from='client.18720 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:32:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:48 smithi162 ceph-mon[20279]: pgmap v2007: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.1 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 668 op/s; 2332/55296 objects degraded (4.217%); 6211/55296 objects misplaced (11.232%); 9.9 MiB/s, 291 objects/s recovering 2024-09-02T10:32:48.599 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:48.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:48.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:48.736 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:48.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:48.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:48.838 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:48.868 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:48.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:48.869 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:48.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:48.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:48.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:48.998 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:49.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:49.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:49.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:49.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:49.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:49.259 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:49.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:49.361 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:49.386 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:49.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:49.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:49 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:32:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:49 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:32:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:49 smithi151 ceph-mon[16639]: osdmap e874: 12 total, 12 up, 12 in 2024-09-02T10:32:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:49 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:32:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:49 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:32:49.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:49 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:32:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:49 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:32:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:49 smithi187 ceph-mon[20072]: osdmap e874: 12 total, 12 up, 12 in 2024-09-02T10:32:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:49 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:32:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:49 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:32:49.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:49.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:49 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:32:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:49 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:32:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:49 smithi162 ceph-mon[20279]: osdmap e874: 12 total, 12 up, 12 in 2024-09-02T10:32:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:49 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:32:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:49 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:32:49.617 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:49.646 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:49.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:49.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:49.772 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:49.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:49.773 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:49.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:49.905 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:49.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:49.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:50.007 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:50.028 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:50.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:50.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:50.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:50.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:50.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:50.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:50.299 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:50.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:50.401 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:50.424 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:50.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:32:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:32:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:50 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2332/55296 objects degraded (4.217%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:32:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:32:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:32:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:32:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:32:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:32:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:50 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:32:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:32:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:32:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:50 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2332/55296 objects degraded (4.217%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:32:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:32:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:32:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:32:50.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:32:50.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:32:50.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:50 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:32:50.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:50.550 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:50.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:32:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:32:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:50 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2332/55296 objects degraded (4.217%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:32:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:32:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:32:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:32:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:32:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:32:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:50 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:32:50.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:50.687 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:50.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:50.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:50.789 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:50.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:50.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:50.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:50.948 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:50.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:50.949 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:51.050 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:51.076 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:51.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:51.178 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:51.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:51.204 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:51.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:51.331 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:51.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:51.433 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:51.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:51 smithi151 ceph-mon[16639]: osdmap e875: 12 total, 12 up, 12 in 2024-09-02T10:32:51.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:51.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:51.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:51 smithi187 ceph-mon[20072]: osdmap e875: 12 total, 12 up, 12 in 2024-09-02T10:32:51.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:51.593 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:51.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:51 smithi162 ceph-mon[20279]: osdmap e875: 12 total, 12 up, 12 in 2024-09-02T10:32:51.696 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:51.727 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:51.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:51.729 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:51.829 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:51.854 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:51.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:51.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:51.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:51.984 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:51.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:52.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:52.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:52.115 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:52.216 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:52.246 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:52.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:52.248 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:52.349 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:52.382 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:52.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:52 smithi151 ceph-mon[16639]: pgmap v2011: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 485 op/s; 2427/49779 objects degraded (4.876%); 5469/49779 objects misplaced (10.987%); 7.7 MiB/s, 201 objects/s recovering 2024-09-02T10:32:52.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:52.500 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:52 smithi187 ceph-mon[20072]: pgmap v2011: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 485 op/s; 2427/49779 objects degraded (4.876%); 5469/49779 objects misplaced (10.987%); 7.7 MiB/s, 201 objects/s recovering 2024-09-02T10:32:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:52.520 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:52.521 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:52 smithi162 ceph-mon[20279]: pgmap v2011: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 485 op/s; 2427/49779 objects degraded (4.876%); 5469/49779 objects misplaced (10.987%); 7.7 MiB/s, 201 objects/s recovering 2024-09-02T10:32:52.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:52.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:52.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:52.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:52.777 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:52.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:52.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:52.879 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:52.902 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:52.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:53.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:53.028 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:53.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:53.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:53.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:53.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:53.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:53.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:53.297 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:53.297 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:53.378 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:32:53.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:53.421 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:53.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:53.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1379123961' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:32:53.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1379123961' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:32:53.523 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:53.546 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:53.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:53.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1379123961' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:32:53.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:53.684 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:53.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:53.685 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:53.786 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:53.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:53.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:53.923 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:53.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:53.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:54.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:54.075 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:54.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:54.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:54.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:54.201 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:54.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:54.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:54.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:54.328 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:54.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:54.430 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:54 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:32:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1379123961' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:32:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:54 smithi151 ceph-mon[16639]: osdmap e876: 12 total, 12 up, 12 in 2024-09-02T10:32:54.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:54 smithi151 ceph-mon[16639]: pgmap v2013: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 928 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 2427/44523 objects degraded (5.451%); 5220/44523 objects misplaced (11.724%); 8.2 MiB/s, 212 objects/s recovering 2024-09-02T10:32:54.467 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:54.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:54.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:54 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:32:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1379123961' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:32:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:54 smithi187 ceph-mon[20072]: osdmap e876: 12 total, 12 up, 12 in 2024-09-02T10:32:54.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:54 smithi187 ceph-mon[20072]: pgmap v2013: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 928 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 2427/44523 objects degraded (5.451%); 5220/44523 objects misplaced (11.724%); 8.2 MiB/s, 212 objects/s recovering 2024-09-02T10:32:54.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:54 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:32:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1379123961' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:32:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:54 smithi162 ceph-mon[20279]: osdmap e876: 12 total, 12 up, 12 in 2024-09-02T10:32:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:54 smithi162 ceph-mon[20279]: pgmap v2013: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 928 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 598 op/s; 2427/44523 objects degraded (5.451%); 5220/44523 objects misplaced (11.724%); 8.2 MiB/s, 212 objects/s recovering 2024-09-02T10:32:54.597 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:54.597 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:54.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:54.732 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:54.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:54.733 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:54.834 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:54.858 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:54.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:54.960 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:54.986 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:54.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:55.088 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:55.112 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:55.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:55.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:55.239 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:55.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:55.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:55.364 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:55.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:55.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:55 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:32:55.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:55 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:32:55.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:55 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2427/44523 objects degraded (5.451%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:55.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:55.491 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:55.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:55.492 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:55.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:55 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:32:55.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:55 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:32:55.495 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:55 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2427/44523 objects degraded (5.451%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:55.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:55 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:32:55.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:55 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:32:55.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:55 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2427/44523 objects degraded (5.451%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:55.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:55.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:55.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:55.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:55.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:55.760 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:55.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:55.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:55.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:55.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:55.891 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:55.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:56.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:56.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:56.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:56.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:56.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:56.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:56.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:56.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:56.384 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:56.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:56.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:56 smithi187 ceph-mon[20072]: osdmap e877: 12 total, 12 up, 12 in 2024-09-02T10:32:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:56 smithi187 ceph-mon[20072]: pgmap v2015: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 846 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 2519/40584 objects degraded (6.207%); 4476/40584 objects misplaced (11.029%); 8.2 MiB/s, 212 objects/s recovering 2024-09-02T10:32:56.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:56.529 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:56.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:56.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:56 smithi162 ceph-mon[20279]: osdmap e877: 12 total, 12 up, 12 in 2024-09-02T10:32:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:56 smithi162 ceph-mon[20279]: pgmap v2015: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 846 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 2519/40584 objects degraded (6.207%); 4476/40584 objects misplaced (11.029%); 8.2 MiB/s, 212 objects/s recovering 2024-09-02T10:32:56.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:56.654 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:56.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:56.655 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:56.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:56 smithi151 ceph-mon[16639]: osdmap e877: 12 total, 12 up, 12 in 2024-09-02T10:32:56.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:56 smithi151 ceph-mon[16639]: pgmap v2015: 47 pgs: 3 active+recovering+forced_recovery+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 42 active+clean; 846 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 571 op/s; 2519/40584 objects degraded (6.207%); 4476/40584 objects misplaced (11.029%); 8.2 MiB/s, 212 objects/s recovering 2024-09-02T10:32:56.756 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:56.779 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:56.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:56.882 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:56.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:56.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:56.903 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:57.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:32:57.026 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:57.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:32:57.086 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2006,"stamp":"2024-09-02T10:32:47.120354+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1208353336,"num_objects":18432,"num_object_clones":0,"num_object_copies":55296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2332,"num_objects_misplaced":6211,"num_objects_unfound":0,"num_objects_dirty":18432,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192128,"num_write_kb":6737793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17027,"num_bytes_recovered":1055405688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163580,"ondisk_log_size":163580,"up":141,"acting":136,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":147,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":12744444,"kb_used_data":4856616,"kb_used_omap":18,"kb_used_meta":7887469,"kb_avail":1112197380,"statfs":{"total":1151940427776,"available":1138890117120,"internally_reserved":0,"allocated":4973174784,"data_stored":4969941044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":8076768571},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-284229632,"num_objects":-4337,"num_object_clones":0,"num_object_copies":-13011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":99,"num_objects_misplaced":-1151,"num_objects_unfound":0,"num_objects_dirty":-4337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4337,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1752,"num_bytes_recovered":62062592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.555051"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":1149,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.136416+0000","last_change":"2024-09-02T10:31:29.714733+0000","last_active":"2024-09-02T10:32:45.136416+0000","last_peered":"2024-09-02T10:32:45.136416+0000","last_clean":"2024-09-02T10:32:45.136416+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:32:45.136416+0000","last_undegraded":"2024-09-02T10:32:45.136416+0000","last_fullsized":"2024-09-02T10:32:45.136416+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:29.714679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:29.714679+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:55.882299+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":954,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.111515+0000","last_change":"2024-09-02T10:31:02.822274+0000","last_active":"2024-09-02T10:32:44.111515+0000","last_peered":"2024-09-02T10:32:44.111515+0000","last_clean":"2024-09-02T10:32:44.111515+0000","last_became_active":"2024-09-02T10:31:02.821847+0000","last_became_peered":"2024-09-02T10:31:02.821847+0000","last_unstale":"2024-09-02T10:32:44.111515+0000","last_undegraded":"2024-09-02T10:32:44.111515+0000","last_fullsized":"2024-09-02T10:32:44.111515+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:54.883293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.883293+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1421,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.092281+0000","last_change":"2024-09-02T10:31:32.666776+0000","last_active":"2024-09-02T10:32:44.092281+0000","last_peered":"2024-09-02T10:32:44.092281+0000","last_clean":"2024-09-02T10:32:44.092281+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:32:44.092281+0000","last_undegraded":"2024-09-02T10:32:44.092281+0000","last_fullsized":"2024-09-02T10:32:44.092281+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:32.666702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:32.666702+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1397,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.092293+0000","last_change":"2024-09-02T10:30:17.892008+0000","last_active":"2024-09-02T10:32:44.092293+0000","last_peered":"2024-09-02T10:32:44.092293+0000","last_clean":"2024-09-02T10:32:44.092293+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:32:44.092293+0000","last_undegraded":"2024-09-02T10:32:44.092293+0000","last_fullsized":"2024-09-02T10:32:44.092293+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:17.891937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:17.891937+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":716,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.112120+0000","last_change":"2024-09-02T10:32:34.109718+0000","last_active":"2024-09-02T10:32:44.112120+0000","last_peered":"2024-09-02T10:32:44.112120+0000","last_clean":"2024-09-02T10:32:44.112120+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:32:44.112120+0000","last_undegraded":"2024-09-02T10:32:44.112120+0000","last_fullsized":"2024-09-02T10:32:44.112120+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:32:34.109667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:32:34.109667+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:33:34.900958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":1261,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.114154+0000","last_change":"2024-09-02T10:29:58.037831+0000","last_active":"2024-09-02T10:32:45.114154+0000","last_peered":"2024-09-02T10:32:45.114154+0000","last_clean":"2024-09-02T10:32:45.114154+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:32:45.114154+0000","last_undegraded":"2024-09-02T10:32:45.114154+0000","last_fullsized":"2024-09-02T10:32:45.114154+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:58.037744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:58.037744+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1776,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:31.042976+0000","last_change":"2024-09-02T10:29:57.973361+0000","last_active":"2024-09-02T10:32:31.042976+0000","last_peered":"2024-09-02T10:32:31.042976+0000","last_clean":"2024-09-02T10:32:31.042976+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:32:31.042976+0000","last_undegraded":"2024-09-02T10:32:31.042976+0000","last_fullsized":"2024-09-02T10:32:31.042976+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:57.973289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:57.973289+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1420,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.092327+0000","last_change":"2024-09-02T10:30:24.905924+0000","last_active":"2024-09-02T10:32:44.092327+0000","last_peered":"2024-09-02T10:32:44.092327+0000","last_clean":"2024-09-02T10:32:44.092327+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:32:44.092327+0000","last_undegraded":"2024-09-02T10:32:44.092327+0000","last_fullsized":"2024-09-02T10:32:44.092327+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.905855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.905855+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":1426,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.111662+0000","last_change":"2024-09-02T10:30:18.808693+0000","last_active":"2024-09-02T10:32:44.111662+0000","last_peered":"2024-09-02T10:32:44.111662+0000","last_clean":"2024-09-02T10:32:44.111662+0000","last_became_active":"2024-09-02T10:25:01.211125+0000","last_became_peered":"2024-09-02T10:25:01.211125+0000","last_unstale":"2024-09-02T10:32:44.111662+0000","last_undegraded":"2024-09-02T10:32:44.111662+0000","last_fullsized":"2024-09-02T10:32:44.111662+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:18.808615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:18.808615+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1389,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.106493+0000","last_change":"2024-09-02T10:30:47.378511+0000","last_active":"2024-09-02T10:32:45.106493+0000","last_peered":"2024-09-02T10:32:45.106493+0000","last_clean":"2024-09-02T10:32:45.106493+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:32:45.106493+0000","last_undegraded":"2024-09-02T10:32:45.106493+0000","last_fullsized":"2024-09-02T10:32:45.106493+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:47.378461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:47.378461+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":630,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:31.047270+0000","last_change":"2024-09-02T10:30:21.301133+0000","last_active":"2024-09-02T10:32:31.047270+0000","last_peered":"2024-09-02T10:32:31.047270+0000","last_clean":"2024-09-02T10:32:31.047270+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:32:31.047270+0000","last_undegraded":"2024-09-02T10:32:31.047270+0000","last_fullsized":"2024-09-02T10:32:31.047270+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:21.301078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:21.301078+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":1744,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.115730+0000","last_change":"2024-09-02T10:31:02.820524+0000","last_active":"2024-09-02T10:32:44.115730+0000","last_peered":"2024-09-02T10:32:44.115730+0000","last_clean":"2024-09-02T10:32:44.115730+0000","last_became_active":"2024-09-02T10:31:02.820058+0000","last_became_peered":"2024-09-02T10:31:02.820058+0000","last_unstale":"2024-09-02T10:32:44.115730+0000","last_undegraded":"2024-09-02T10:32:44.115730+0000","last_fullsized":"2024-09-02T10:32:44.115730+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:48.014073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.014073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,0],"acting":[1,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1675,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:31.042172+0000","last_change":"2024-09-02T10:30:23.108717+0000","last_active":"2024-09-02T10:32:31.042172+0000","last_peered":"2024-09-02T10:32:31.042172+0000","last_clean":"2024-09-02T10:32:31.042172+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:32:31.042172+0000","last_undegraded":"2024-09-02T10:32:31.042172+0000","last_fullsized":"2024-09-02T10:32:31.042172+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:23.108646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:23.108646+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":407,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:31.057257+0000","last_change":"2024-09-02T10:31:32.353256+0000","last_active":"2024-09-02T10:32:31.057257+0000","last_peered":"2024-09-02T10:32:31.057257+0000","last_clean":"2024-09-02T10:32:31.057257+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:32:31.057257+0000","last_undegraded":"2024-09-02T10:32:31.057257+0000","last_fullsized":"2024-09-02T10:32:31.057257+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:32.353166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:32.353166+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:45.856271+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1645,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:31.042984+0000","last_change":"2024-09-02T10:30:16.984287+0000","last_active":"2024-09-02T10:32:31.042984+0000","last_peered":"2024-09-02T10:32:31.042984+0000","last_clean":"2024-09-02T10:32:31.042984+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:32:31.042984+0000","last_undegraded":"2024-09-02T10:32:31.042984+0000","last_fullsized":"2024-09-02T10:32:31.042984+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:16.984241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:16.984241+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.1","version":"871'11836","reported_seq":12106,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.302894+0000","last_change":"2024-09-02T10:30:45.515400+0000","last_active":"2024-09-02T10:32:44.302894+0000","last_peered":"2024-09-02T10:32:44.302894+0000","last_clean":"2024-09-02T10:32:44.302894+0000","last_became_active":"2024-09-02T10:28:51.931930+0000","last_became_peered":"2024-09-02T10:28:51.931930+0000","last_unstale":"2024-09-02T10:32:44.302894+0000","last_undegraded":"2024-09-02T10:32:44.302894+0000","last_fullsized":"2024-09-02T10:32:44.302894+0000","mapping_epoch":792,"log_start":"806'1800","ondisk_log_start":"806'1800","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7411","last_scrub_stamp":"2024-09-02T10:30:45.515308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:45.515308+0000","objects_scrubbed":5608,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1512,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75890688,"num_objects":1158,"num_object_clones":0,"num_object_copies":3474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11836,"num_write_kb":415808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9,2],"acting":[1,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1740,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:31.043076+0000","last_change":"2024-09-02T10:30:22.096061+0000","last_active":"2024-09-02T10:32:31.043076+0000","last_peered":"2024-09-02T10:32:31.043076+0000","last_clean":"2024-09-02T10:32:31.043076+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:32:31.043076+0000","last_undegraded":"2024-09-02T10:32:31.043076+0000","last_fullsized":"2024-09-02T10:32:31.043076+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:22.095998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:22.095998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.0","version":"871'12165","reported_seq":12449,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.121190+0000","last_change":"2024-09-02T10:31:30.230104+0000","last_active":"2024-09-02T10:32:44.121190+0000","last_peered":"2024-09-02T10:32:44.121190+0000","last_clean":"2024-09-02T10:32:44.121190+0000","last_became_active":"2024-09-02T10:28:51.652599+0000","last_became_peered":"2024-09-02T10:28:51.652599+0000","last_unstale":"2024-09-02T10:32:44.121190+0000","last_undegraded":"2024-09-02T10:32:44.121190+0000","last_fullsized":"2024-09-02T10:32:44.121190+0000","mapping_epoch":792,"log_start":"807'2100","ondisk_log_start":"807'2100","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'9583","last_scrub_stamp":"2024-09-02T10:31:30.229961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:31:30.229961+0000","objects_scrubbed":3792,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78053376,"num_objects":1191,"num_object_clones":0,"num_object_copies":3573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12165,"num_write_kb":427392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,9],"acting":[3,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1869,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.105865+0000","last_change":"2024-09-02T10:30:24.407289+0000","last_active":"2024-09-02T10:32:45.105865+0000","last_peered":"2024-09-02T10:32:45.105865+0000","last_clean":"2024-09-02T10:32:45.105865+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:32:45.105865+0000","last_undegraded":"2024-09-02T10:32:45.105865+0000","last_fullsized":"2024-09-02T10:32:45.105865+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.407218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.407218+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3","version":"871'11956","reported_seq":18942,"reported_epoch":871,"state":"active+recovering+forced_recovery+undersized+degraded+remapped","last_fresh":"2024-09-02T10:32:44.313862+0000","last_change":"2024-09-02T10:32:22.102310+0000","last_active":"2024-09-02T10:32:44.313862+0000","last_peered":"2024-09-02T10:32:44.313862+0000","last_clean":"2024-09-02T10:31:01.749812+0000","last_became_active":"2024-09-02T10:31:04.153013+0000","last_became_peered":"2024-09-02T10:31:04.153013+0000","last_unstale":"2024-09-02T10:32:44.313862+0000","last_undegraded":"2024-09-02T10:31:03.853663+0000","last_fullsized":"2024-09-02T10:31:03.819841+0000","mapping_epoch":844,"log_start":"806'1900","ondisk_log_start":"806'1900","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'5828","last_scrub_stamp":"2024-09-02T10:30:16.082748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:16.082748+0000","objects_scrubbed":5788,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1773,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76939264,"num_objects":1174,"num_object_clones":0,"num_object_copies":3522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":386,"num_objects_misplaced":924,"num_objects_unfound":0,"num_objects_dirty":1174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11956,"num_write_kb":420160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3373,"num_bytes_recovered":220135424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5],"avail_no_missing":["1","5","11"],"object_location_counts":[{"shards":"1,5","objects":-136},{"shards":"1,5,11","objects":924}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1516,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:31.045726+0000","last_change":"2024-09-02T10:31:24.531418+0000","last_active":"2024-09-02T10:32:31.045726+0000","last_peered":"2024-09-02T10:32:31.045726+0000","last_clean":"2024-09-02T10:32:31.045726+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:32:31.045726+0000","last_undegraded":"2024-09-02T10:32:31.045726+0000","last_fullsized":"2024-09-02T10:32:31.045726+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:24.531349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:24.531349+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:31.310250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"8.2","version":"872'11978","reported_seq":12215,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:46.653894+0000","last_change":"2024-09-02T10:30:16.516178+0000","last_active":"2024-09-02T10:32:46.653894+0000","last_peered":"2024-09-02T10:32:46.653894+0000","last_clean":"2024-09-02T10:32:46.653894+0000","last_became_active":"2024-09-02T10:28:51.668558+0000","last_became_peered":"2024-09-02T10:28:51.668558+0000","last_unstale":"2024-09-02T10:32:46.653894+0000","last_undegraded":"2024-09-02T10:32:46.653894+0000","last_fullsized":"2024-09-02T10:32:46.653894+0000","mapping_epoch":792,"log_start":"807'1900","ondisk_log_start":"807'1900","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"831'5835","last_scrub_stamp":"2024-09-02T10:30:16.515921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:16.515921+0000","objects_scrubbed":5778,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11978,"num_write_kb":417984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,4],"acting":[10,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":485,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:31.057659+0000","last_change":"2024-09-02T10:31:28.385898+0000","last_active":"2024-09-02T10:32:31.057659+0000","last_peered":"2024-09-02T10:32:31.057659+0000","last_clean":"2024-09-02T10:32:31.057659+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:32:31.057659+0000","last_undegraded":"2024-09-02T10:32:31.057659+0000","last_fullsized":"2024-09-02T10:32:31.057659+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:28.385818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:28.385818+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:45.599604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"872'12036","reported_seq":23324,"reported_epoch":872,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:32:46.577363+0000","last_change":"2024-09-02T10:31:04.087511+0000","last_active":"2024-09-02T10:32:46.577363+0000","last_peered":"2024-09-02T10:32:46.577363+0000","last_clean":"2024-09-02T10:31:01.804575+0000","last_became_active":"2024-09-02T10:31:04.035888+0000","last_became_peered":"2024-09-02T10:31:04.035888+0000","last_unstale":"2024-09-02T10:32:46.577363+0000","last_undegraded":"2024-09-02T10:31:03.853844+0000","last_fullsized":"2024-09-02T10:31:03.818904+0000","mapping_epoch":844,"log_start":"807'2000","ondisk_log_start":"807'2000","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"822'4517","last_scrub_stamp":"2024-09-02T10:29:54.734394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.734394+0000","objects_scrubbed":4467,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1390,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71893016,"num_objects":1098,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":89,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12036,"num_write_kb":420225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5530,"num_bytes_recovered":305266712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2],"avail_no_missing":["7","2","11"],"object_location_counts":[{"shards":"2,7","objects":1009}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1824,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.136352+0000","last_change":"2024-09-02T10:32:30.121351+0000","last_active":"2024-09-02T10:32:45.136352+0000","last_peered":"2024-09-02T10:32:45.136352+0000","last_clean":"2024-09-02T10:32:45.136352+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:32:45.136352+0000","last_undegraded":"2024-09-02T10:32:45.136352+0000","last_fullsized":"2024-09-02T10:32:45.136352+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:32:30.121275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:32:30.121275+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:33:52.321817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"8.c","version":"870'11915","reported_seq":12152,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:43.043515+0000","last_change":"2024-09-02T10:30:24.460290+0000","last_active":"2024-09-02T10:32:43.043515+0000","last_peered":"2024-09-02T10:32:43.043515+0000","last_clean":"2024-09-02T10:32:43.043515+0000","last_became_active":"2024-09-02T10:28:51.652606+0000","last_became_peered":"2024-09-02T10:28:51.652606+0000","last_unstale":"2024-09-02T10:32:43.043515+0000","last_undegraded":"2024-09-02T10:32:43.043515+0000","last_fullsized":"2024-09-02T10:32:43.043515+0000","mapping_epoch":792,"log_start":"807'1900","ondisk_log_start":"807'1900","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'6309","last_scrub_stamp":"2024-09-02T10:30:24.460190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.460190+0000","objects_scrubbed":6243,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79364096,"num_objects":1211,"num_object_clones":0,"num_object_copies":3633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11915,"num_write_kb":420032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":1283,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.114330+0000","last_change":"2024-09-02T10:30:55.890418+0000","last_active":"2024-09-02T10:32:45.114330+0000","last_peered":"2024-09-02T10:32:45.114330+0000","last_clean":"2024-09-02T10:32:45.114330+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:32:45.114330+0000","last_undegraded":"2024-09-02T10:32:45.114330+0000","last_fullsized":"2024-09-02T10:32:45.114330+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:55.890351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:55.890351+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.f","version":"872'11932","reported_seq":12239,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:46.838035+0000","last_change":"2024-09-02T10:32:42.402494+0000","last_active":"2024-09-02T10:32:46.838035+0000","last_peered":"2024-09-02T10:32:46.838035+0000","last_clean":"2024-09-02T10:32:46.838035+0000","last_became_active":"2024-09-02T10:28:51.932528+0000","last_became_peered":"2024-09-02T10:28:51.932528+0000","last_unstale":"2024-09-02T10:32:46.838035+0000","last_undegraded":"2024-09-02T10:32:46.838035+0000","last_fullsized":"2024-09-02T10:32:46.838035+0000","mapping_epoch":792,"log_start":"807'1900","ondisk_log_start":"807'1900","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"870'11757","last_scrub_stamp":"2024-09-02T10:32:42.402272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:32:42.402272+0000","objects_scrubbed":1259,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:12.162229+0000","scrub_duration":210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11932,"num_write_kb":416512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,1],"acting":[11,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":1248,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.116294+0000","last_change":"2024-09-02T10:29:54.344897+0000","last_active":"2024-09-02T10:32:45.116294+0000","last_peered":"2024-09-02T10:32:45.116294+0000","last_clean":"2024-09-02T10:32:45.116294+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:32:45.116294+0000","last_undegraded":"2024-09-02T10:32:45.116294+0000","last_fullsized":"2024-09-02T10:32:45.116294+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:54.344678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.344678+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.e","version":"870'11887","reported_seq":12166,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:42.723563+0000","last_change":"2024-09-02T10:31:34.212685+0000","last_active":"2024-09-02T10:32:42.723563+0000","last_peered":"2024-09-02T10:32:42.723563+0000","last_clean":"2024-09-02T10:32:42.723563+0000","last_became_active":"2024-09-02T10:28:51.931936+0000","last_became_peered":"2024-09-02T10:28:51.931936+0000","last_unstale":"2024-09-02T10:32:42.723563+0000","last_undegraded":"2024-09-02T10:32:42.723563+0000","last_fullsized":"2024-09-02T10:32:42.723563+0000","mapping_epoch":792,"log_start":"806'1800","ondisk_log_start":"806'1800","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'9573","last_scrub_stamp":"2024-09-02T10:31:34.212499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:31:34.212499+0000","objects_scrubbed":3519,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:58.429729+0000","scrub_duration":849,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78053376,"num_objects":1191,"num_object_clones":0,"num_object_copies":3573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11887,"num_write_kb":418496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1026,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:31.043772+0000","last_change":"2024-09-02T10:30:14.802221+0000","last_active":"2024-09-02T10:32:31.043772+0000","last_peered":"2024-09-02T10:32:31.043772+0000","last_clean":"2024-09-02T10:32:31.043772+0000","last_became_active":"2024-09-02T10:25:02.065175+0000","last_became_peered":"2024-09-02T10:25:02.065175+0000","last_unstale":"2024-09-02T10:32:31.043772+0000","last_undegraded":"2024-09-02T10:32:31.043772+0000","last_fullsized":"2024-09-02T10:32:31.043772+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:14.802157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:14.802157+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.8","version":"871'12229","reported_seq":15995,"reported_epoch":871,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:32:44.164110+0000","last_change":"2024-09-02T10:31:40.864360+0000","last_active":"2024-09-02T10:32:44.164110+0000","last_peered":"2024-09-02T10:32:44.164110+0000","last_clean":"2024-09-02T10:31:38.550380+0000","last_became_active":"2024-09-02T10:31:40.658764+0000","last_became_peered":"2024-09-02T10:31:40.658764+0000","last_unstale":"2024-09-02T10:32:44.164110+0000","last_undegraded":"2024-09-02T10:31:40.522684+0000","last_fullsized":"2024-09-02T10:31:40.323854+0000","mapping_epoch":854,"log_start":"808'2200","ondisk_log_start":"808'2200","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7674","last_scrub_stamp":"2024-09-02T10:30:46.822238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.822238+0000","objects_scrubbed":5790,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78053376,"num_objects":1191,"num_object_clones":0,"num_object_copies":3573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1126,"num_objects_misplaced":3508,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12229,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1175,"num_bytes_recovered":74579968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,9],"acting":[3,9],"avail_no_missing":["3","5","9"],"object_location_counts":[{"shards":"3,5,9","objects":1191},{"shards":"3,9","objects":-1126}],"blocked_by":[],"up_primary":10,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":1263,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.114185+0000","last_change":"2024-09-02T10:30:48.025489+0000","last_active":"2024-09-02T10:32:45.114185+0000","last_peered":"2024-09-02T10:32:45.114185+0000","last_clean":"2024-09-02T10:32:45.114185+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:32:45.114185+0000","last_undegraded":"2024-09-02T10:32:45.114185+0000","last_fullsized":"2024-09-02T10:32:45.114185+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:48.025327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.025327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"8.b","version":"871'12058","reported_seq":12344,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.586361+0000","last_change":"2024-09-02T10:31:39.273035+0000","last_active":"2024-09-02T10:32:44.586361+0000","last_peered":"2024-09-02T10:32:44.586361+0000","last_clean":"2024-09-02T10:32:44.586361+0000","last_became_active":"2024-09-02T10:28:51.668549+0000","last_became_peered":"2024-09-02T10:28:51.668549+0000","last_unstale":"2024-09-02T10:32:44.586361+0000","last_undegraded":"2024-09-02T10:32:44.586361+0000","last_fullsized":"2024-09-02T10:32:44.586361+0000","mapping_epoch":792,"log_start":"807'2000","ondisk_log_start":"807'2000","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"853'9804","last_scrub_stamp":"2024-09-02T10:31:39.272959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:31:39.272959+0000","objects_scrubbed":3434,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:33:05.517671+0000","scrub_duration":710,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77070336,"num_objects":1176,"num_object_clones":0,"num_object_copies":3528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12058,"num_write_kb":423488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,1],"acting":[8,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1511,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.105929+0000","last_change":"2024-09-02T10:30:23.359450+0000","last_active":"2024-09-02T10:32:45.105929+0000","last_peered":"2024-09-02T10:32:45.105929+0000","last_clean":"2024-09-02T10:32:45.105929+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:32:45.105929+0000","last_undegraded":"2024-09-02T10:32:45.105929+0000","last_fullsized":"2024-09-02T10:32:45.105929+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:23.359405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:23.359405+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"871'11786","reported_seq":12056,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.629290+0000","last_change":"2024-09-02T10:31:38.628550+0000","last_active":"2024-09-02T10:32:44.629290+0000","last_peered":"2024-09-02T10:32:44.629290+0000","last_clean":"2024-09-02T10:32:44.629290+0000","last_became_active":"2024-09-02T10:28:51.932243+0000","last_became_peered":"2024-09-02T10:28:51.932243+0000","last_unstale":"2024-09-02T10:32:44.629290+0000","last_undegraded":"2024-09-02T10:32:44.629290+0000","last_fullsized":"2024-09-02T10:32:44.629290+0000","mapping_epoch":792,"log_start":"806'1700","ondisk_log_start":"806'1700","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"853'9585","last_scrub_stamp":"2024-09-02T10:31:38.628451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:31:38.628451+0000","objects_scrubbed":3368,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75235328,"num_objects":1148,"num_object_clones":0,"num_object_copies":3444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11786,"num_write_kb":413888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,9],"acting":[8,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1769,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.115793+0000","last_change":"2024-09-02T10:30:20.873940+0000","last_active":"2024-09-02T10:32:44.115793+0000","last_peered":"2024-09-02T10:32:44.115793+0000","last_clean":"2024-09-02T10:32:44.115793+0000","last_became_active":"2024-09-02T10:19:05.057978+0000","last_became_peered":"2024-09-02T10:19:05.057978+0000","last_unstale":"2024-09-02T10:32:44.115793+0000","last_undegraded":"2024-09-02T10:32:44.115793+0000","last_fullsized":"2024-09-02T10:32:44.115793+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:20.873897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:20.873897+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.9","version":"872'12142","reported_seq":12413,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:46.699161+0000","last_change":"2024-09-02T10:30:46.985280+0000","last_active":"2024-09-02T10:32:46.699161+0000","last_peered":"2024-09-02T10:32:46.699161+0000","last_clean":"2024-09-02T10:32:46.699161+0000","last_became_active":"2024-09-02T10:28:51.650885+0000","last_became_peered":"2024-09-02T10:28:51.650885+0000","last_unstale":"2024-09-02T10:32:46.699161+0000","last_undegraded":"2024-09-02T10:32:46.699161+0000","last_fullsized":"2024-09-02T10:32:46.699161+0000","mapping_epoch":792,"log_start":"807'2100","ondisk_log_start":"807'2100","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7658","last_scrub_stamp":"2024-09-02T10:30:46.985056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.985056+0000","objects_scrubbed":5619,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2021,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12142,"num_write_kb":423232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,0],"acting":[10,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1521,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.105830+0000","last_change":"2024-09-02T10:31:03.381315+0000","last_active":"2024-09-02T10:32:45.105830+0000","last_peered":"2024-09-02T10:32:45.105830+0000","last_clean":"2024-09-02T10:32:45.105830+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:32:45.105830+0000","last_undegraded":"2024-09-02T10:32:45.105830+0000","last_fullsized":"2024-09-02T10:32:45.105830+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:58.376013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:58.376013+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.6","version":"872'12122","reported_seq":20446,"reported_epoch":872,"state":"active+recovering+forced_recovery+undersized+degraded+remapped","last_fresh":"2024-09-02T10:32:46.714610+0000","last_change":"2024-09-02T10:32:22.096930+0000","last_active":"2024-09-02T10:32:46.714610+0000","last_peered":"2024-09-02T10:32:46.714610+0000","last_clean":"2024-09-02T10:31:01.786664+0000","last_became_active":"2024-09-02T10:31:04.069096+0000","last_became_peered":"2024-09-02T10:31:04.069096+0000","last_unstale":"2024-09-02T10:32:46.714610+0000","last_undegraded":"2024-09-02T10:31:03.844885+0000","last_fullsized":"2024-09-02T10:31:03.820047+0000","mapping_epoch":844,"log_start":"807'2100","ondisk_log_start":"807'2100","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"832'5937","last_scrub_stamp":"2024-09-02T10:30:17.013655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:17.013655+0000","objects_scrubbed":5883,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76283904,"num_objects":1164,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":103,"num_objects_misplaced":610,"num_objects_unfound":0,"num_objects_dirty":1164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12122,"num_write_kb":425152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4041,"num_bytes_recovered":264634368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,2],"acting":[10,7],"avail_no_missing":["10","7","11"],"object_location_counts":[{"shards":"7,10","objects":451},{"shards":"7,10,11","objects":610}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1773,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:31.043776+0000","last_change":"2024-09-02T10:30:20.714039+0000","last_active":"2024-09-02T10:32:31.043776+0000","last_peered":"2024-09-02T10:32:31.043776+0000","last_clean":"2024-09-02T10:32:31.043776+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:32:31.043776+0000","last_undegraded":"2024-09-02T10:32:31.043776+0000","last_fullsized":"2024-09-02T10:32:31.043776+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:20.713975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:20.713975+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"871'12016","reported_seq":18056,"reported_epoch":871,"state":"active+recovering+forced_recovery+undersized+degraded+remapped","last_fresh":"2024-09-02T10:32:44.328930+0000","last_change":"2024-09-02T10:32:22.108441+0000","last_active":"2024-09-02T10:32:44.328930+0000","last_peered":"2024-09-02T10:32:44.328930+0000","last_clean":"2024-09-02T10:31:01.796091+0000","last_became_active":"2024-09-02T10:31:04.054394+0000","last_became_peered":"2024-09-02T10:31:04.054394+0000","last_unstale":"2024-09-02T10:32:44.328930+0000","last_undegraded":"2024-09-02T10:31:03.853930+0000","last_fullsized":"2024-09-02T10:31:03.819998+0000","mapping_epoch":844,"log_start":"807'2000","ondisk_log_start":"807'2000","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"839'7994","last_scrub_stamp":"2024-09-02T10:30:53.579876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:53.579876+0000","objects_scrubbed":5226,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76677120,"num_objects":1170,"num_object_clones":0,"num_object_copies":3510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":628,"num_objects_misplaced":1169,"num_objects_unfound":0,"num_objects_dirty":1170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12016,"num_write_kb":421952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2904,"num_bytes_recovered":189857792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,2],"acting":[1,10],"avail_no_missing":["1","10","11"],"object_location_counts":[{"shards":"1,10","objects":-627},{"shards":"1,10,11","objects":1169}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":724,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.112129+0000","last_change":"2024-09-02T10:32:32.161493+0000","last_active":"2024-09-02T10:32:44.112129+0000","last_peered":"2024-09-02T10:32:44.112129+0000","last_clean":"2024-09-02T10:32:44.112129+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:32:44.112129+0000","last_undegraded":"2024-09-02T10:32:44.112129+0000","last_fullsized":"2024-09-02T10:32:44.112129+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:32:32.161447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:32:32.161447+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:33:41.274190+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.4","version":"870'12058","reported_seq":12322,"reported_epoch":870,"state":"active+clean","last_fresh":"2024-09-02T10:32:42.629747+0000","last_change":"2024-09-02T10:30:48.613043+0000","last_active":"2024-09-02T10:32:42.629747+0000","last_peered":"2024-09-02T10:32:42.629747+0000","last_clean":"2024-09-02T10:32:42.629747+0000","last_became_active":"2024-09-02T10:28:51.651609+0000","last_became_peered":"2024-09-02T10:28:51.651609+0000","last_unstale":"2024-09-02T10:32:42.629747+0000","last_undegraded":"2024-09-02T10:32:42.629747+0000","last_fullsized":"2024-09-02T10:32:42.629747+0000","mapping_epoch":792,"log_start":"807'2000","ondisk_log_start":"807'2000","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'7724","last_scrub_stamp":"2024-09-02T10:30:48.612984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.612984+0000","objects_scrubbed":5646,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12058,"num_write_kb":426624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,10],"acting":[5,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1549,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:45.106592+0000","last_change":"2024-09-02T10:30:46.402988+0000","last_active":"2024-09-02T10:32:45.106592+0000","last_peered":"2024-09-02T10:32:45.106592+0000","last_clean":"2024-09-02T10:32:45.106592+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:32:45.106592+0000","last_undegraded":"2024-09-02T10:32:45.106592+0000","last_fullsized":"2024-09-02T10:32:45.106592+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:46.402901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.402901+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"872'12012","reported_seq":12256,"reported_epoch":872,"state":"active+clean","last_fresh":"2024-09-02T10:32:46.685422+0000","last_change":"2024-09-02T10:30:27.169510+0000","last_active":"2024-09-02T10:32:46.685422+0000","last_peered":"2024-09-02T10:32:46.685422+0000","last_clean":"2024-09-02T10:32:46.685422+0000","last_became_active":"2024-09-02T10:28:51.668549+0000","last_became_peered":"2024-09-02T10:28:51.668549+0000","last_unstale":"2024-09-02T10:32:46.685422+0000","last_undegraded":"2024-09-02T10:32:46.685422+0000","last_fullsized":"2024-09-02T10:32:46.685422+0000","mapping_epoch":792,"log_start":"807'2000","ondisk_log_start":"807'2000","created":792,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":0,"last_scrub":"835'6417","last_scrub_stamp":"2024-09-02T10:30:27.169341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:28:50.639462+0000","last_clean_scrub_stamp":"2024-09-02T10:30:27.169341+0000","objects_scrubbed":6355,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2342,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67633152,"num_objects":1032,"num_object_clones":0,"num_object_copies":3096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12012,"num_write_kb":417408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,1],"acting":[10,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":938,"reported_epoch":871,"state":"active+clean","last_fresh":"2024-09-02T10:32:44.098043+0000","last_change":"2024-09-02T10:31:32.841160+0000","last_active":"2024-09-02T10:32:44.098043+0000","last_peered":"2024-09-02T10:32:44.098043+0000","last_clean":"2024-09-02T10:32:44.098043+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:32:44.098043+0000","last_undegraded":"2024-09-02T10:32:44.098043+0000","last_fullsized":"2024-09-02T10:32:44.098043+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:31:32.841078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:31:32.841078+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:32:47.291610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1207762968,"num_objects":18430,"num_object_clones":0,"num_object_copies":55290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2332,"num_objects_misplaced":6211,"num_objects_unfound":0,"num_objects_dirty":18430,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192128,"num_write_kb":6737793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17023,"num_bytes_recovered":1054474264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4920721408,"data_stored":4920705120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160728,"ondisk_log_size":160728,"up":48,"acting":43,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384072,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1969756,"kb_used_data":1280516,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91775396,"statfs":{"total":95995035648,"available":93978005504,"internally_reserved":0,"allocated":1311248384,"data_stored":1310971451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":705755588},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532613005,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":999060,"kb_used_data":426432,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92746092,"statfs":{"total":95995035648,"available":94971998208,"internally_reserved":0,"allocated":436666368,"data_stored":436393507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":586348992},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841937,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1258168,"kb_used_data":478696,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92486984,"statfs":{"total":95995035648,"available":94706671616,"internally_reserved":0,"allocated":490184704,"data_stored":489920285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":798161344},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070872,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":687736,"kb_used_data":227560,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":93057416,"statfs":{"total":95995035648,"available":95290793984,"internally_reserved":0,"allocated":233021440,"data_stored":232757021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299805,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846428,"kb_used_data":147468,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92898724,"statfs":{"total":95995035648,"available":95128293376,"internally_reserved":0,"allocated":151007232,"data_stored":150726747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":715717066},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528736,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":543672,"kb_used_data":16872,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":93201480,"statfs":{"total":95995035648,"available":95438315520,"internally_reserved":0,"allocated":17276928,"data_stored":17012509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":539425223},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724965,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1399364,"kb_used_data":624052,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92345788,"statfs":{"total":95995035648,"available":94562086912,"internally_reserved":0,"allocated":639029248,"data_stored":638757205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":793901509},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953898,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":761352,"kb_used_data":297012,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92983800,"statfs":{"total":95995035648,"available":95215411200,"internally_reserved":0,"allocated":304140288,"data_stored":303871648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150128,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1000328,"kb_used_data":229172,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92744824,"statfs":{"total":95995035648,"available":94970699776,"internally_reserved":0,"allocated":234672128,"data_stored":234403488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":789641672},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411763,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1034948,"kb_used_data":435892,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92710204,"statfs":{"total":95995035648,"available":94935248896,"internally_reserved":0,"allocated":446353408,"data_stored":446081365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,3,4,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575288,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1164792,"kb_used_data":457448,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":92580360,"statfs":{"total":95995035648,"available":94802288640,"internally_reserved":0,"allocated":468426752,"data_stored":468162333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":724236741},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738814,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1078840,"kb_used_data":235496,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92666312,"statfs":{"total":95995035648,"available":94890303488,"internally_reserved":0,"allocated":241147904,"data_stored":240883485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":237699072,"data_stored":237699072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":458817536,"data_stored":458817536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":442503168,"data_stored":442499096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":233111552,"data_stored":233111552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":299040768,"data_stored":299040768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":636293120,"data_stored":636289048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384000,"data_stored":16384000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":148443136,"data_stored":148439064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":228458496,"data_stored":228458496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":481165312,"data_stored":481165312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":430047232,"data_stored":430047232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":1308758016,"data_stored":1308753944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:32:57.089 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 61 2024-09-02T10:32:57.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:57.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:57.149 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:57.251 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:57.272 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:57.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:57.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:57.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:57.402 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:57 smithi151 ceph-mon[16639]: osdmap e878: 12 total, 12 up, 12 in 2024-09-02T10:32:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:57 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:32:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:57 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:32:57.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:57 smithi187 ceph-mon[20072]: osdmap e878: 12 total, 12 up, 12 in 2024-09-02T10:32:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:57 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:32:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:57 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:32:57.533 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:57.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:57 smithi162 ceph-mon[20279]: osdmap e878: 12 total, 12 up, 12 in 2024-09-02T10:32:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:57 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:32:57.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:57 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:32:57.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:32:57.665 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:57.666 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:32:57.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:32:57.799 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:57.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:57.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:32:57.901 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:57.925 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:57.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:58.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:58.049 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:58.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:58.152 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:58.174 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:58.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:58.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:58.302 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:58.303 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:58.404 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:32:58.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:58 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:32:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:58 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:32:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:58 smithi151 ceph-mon[16639]: osdmap e879: 12 total, 12 up, 12 in 2024-09-02T10:32:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:58 smithi151 ceph-mon[16639]: pgmap v2018: 47 pgs: 1 peering, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 42 active+clean; 776 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 2430/37230 objects degraded (6.527%); 4254/37230 objects misplaced (11.426%); 11 MiB/s, 297 objects/s recovering 2024-09-02T10:32:58.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:58.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:32:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:58 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:32:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:58 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:32:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:58 smithi187 ceph-mon[20072]: osdmap e879: 12 total, 12 up, 12 in 2024-09-02T10:32:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:58 smithi187 ceph-mon[20072]: pgmap v2018: 47 pgs: 1 peering, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 42 active+clean; 776 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 2430/37230 objects degraded (6.527%); 4254/37230 objects misplaced (11.426%); 11 MiB/s, 297 objects/s recovering 2024-09-02T10:32:58.540 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:32:58.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:58.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:32:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:58 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:32:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:58 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:32:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:58 smithi162 ceph-mon[20279]: osdmap e879: 12 total, 12 up, 12 in 2024-09-02T10:32:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:58 smithi162 ceph-mon[20279]: pgmap v2018: 47 pgs: 1 peering, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 42 active+clean; 776 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 2430/37230 objects degraded (6.527%); 4254/37230 objects misplaced (11.426%); 11 MiB/s, 297 objects/s recovering 2024-09-02T10:32:58.671 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:58.692 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:58.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:58.794 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:58.818 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:58.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:58.819 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:58.920 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:32:58.942 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:58.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:32:59.044 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:32:59.067 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:59.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:32:59.169 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:32:59.193 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:59.194 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:32:59.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:32:59.327 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:59.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:59.329 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:32:59.429 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:32:59.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:59.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:32:59.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:32:59.565 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:59.590 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:59.591 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:59.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:32:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2430/37230 objects degraded (6.527%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:59.692 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:32:59.716 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:32:59.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:59.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:32:59.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:32:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2430/37230 objects degraded (6.527%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:59.820 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:59.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:32:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2430/37230 objects degraded (6.527%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:32:59.850 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:59.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:59.851 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:32:59.952 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:32:59.979 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:32:59.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:32:59.980 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:00.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:00.102 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:00.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:00.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:00.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:00.234 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:00.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:00.362 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:00.363 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:00.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:00.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:00.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:00.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:00.619 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:00.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:00 smithi151 ceph-mon[16639]: 8.e scrub starts 2024-09-02T10:33:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:00 smithi151 ceph-mon[16639]: pgmap v2019: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 42 active+clean; 697 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 2494/33429 objects degraded (7.461%); 3340/33429 objects misplaced (9.991%); 16 MiB/s, 342 objects/s recovering 2024-09-02T10:33:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:00 smithi151 ceph-mon[16639]: 8.e scrub ok 2024-09-02T10:33:00.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:00.753 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:00.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:00 smithi187 ceph-mon[20072]: 8.e scrub starts 2024-09-02T10:33:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:00 smithi187 ceph-mon[20072]: pgmap v2019: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 42 active+clean; 697 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 2494/33429 objects degraded (7.461%); 3340/33429 objects misplaced (9.991%); 16 MiB/s, 342 objects/s recovering 2024-09-02T10:33:00.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:00 smithi187 ceph-mon[20072]: 8.e scrub ok 2024-09-02T10:33:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:00 smithi162 ceph-mon[20279]: 8.e scrub starts 2024-09-02T10:33:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:00 smithi162 ceph-mon[20279]: pgmap v2019: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 42 active+clean; 697 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 616 op/s; 2494/33429 objects degraded (7.461%); 3340/33429 objects misplaced (9.991%); 16 MiB/s, 342 objects/s recovering 2024-09-02T10:33:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:00 smithi162 ceph-mon[20279]: 8.e scrub ok 2024-09-02T10:33:00.854 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:00.877 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:00.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:00.980 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:01.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:01.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:01.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:01.105 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:01.127 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:01.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:01.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:01.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:01.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:01.359 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:01.390 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:01.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:01.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:01.513 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:01.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:01.615 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:01.642 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:01.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:01.643 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:01.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:01.767 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:01.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:01.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:01.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:01.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:01.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:01.891 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:01.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:02.013 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:02.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:02.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:02.115 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:02.140 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:02.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:02.141 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:02.244 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:02.268 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:02.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:02.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:02.370 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:02.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:02.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:02.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:02.545 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:02.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:02.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:02.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:02.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:02.677 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:02.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:02.779 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:02.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:02.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:02.805 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:02.906 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:02.932 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:02.932 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:02.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:02 smithi151 ceph-mon[16639]: pgmap v2020: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 42 active+clean; 678 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 2494/32502 objects degraded (7.673%); 3340/32502 objects misplaced (10.276%); 9.5 MiB/s, 237 objects/s recovering 2024-09-02T10:33:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:02 smithi187 ceph-mon[20072]: pgmap v2020: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 42 active+clean; 678 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 2494/32502 objects degraded (7.673%); 3340/32502 objects misplaced (10.276%); 9.5 MiB/s, 237 objects/s recovering 2024-09-02T10:33:03.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:03.055 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:03.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:03.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:02 smithi162 ceph-mon[20279]: pgmap v2020: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 42 active+clean; 678 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 421 op/s; 2494/32502 objects degraded (7.673%); 3340/32502 objects misplaced (10.276%); 9.5 MiB/s, 237 objects/s recovering 2024-09-02T10:33:03.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:03.179 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:03.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:03.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:03.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:03.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:03.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:03.415 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:03.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:03.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:03.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:03.573 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:03.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:03.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:03.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:03.703 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:03.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:03.805 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:03.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:33:03.830 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:03.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:03.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:03.955 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:03.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:04.030 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:04.056 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:04.058 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:04.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:04.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:04.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:04.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:04.241 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:04.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:04.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:04.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:04.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:04.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:04.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:04.537 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:04.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:04.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:04.639 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:04.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:04.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:04.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:04.724 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:04 smithi151 ceph-mon[16639]: pgmap v2021: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 586 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 2494/28086 objects degraded (8.880%); 3201/28086 objects misplaced (11.397%); 8.6 MiB/s, 243 objects/s recovering 2024-09-02T10:33:04.724 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2494/28086 objects degraded (8.880%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:04.780 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:04.804 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:04.805 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:04.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:04.940 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:04.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:04 smithi187 ceph-mon[20072]: pgmap v2021: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 586 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 2494/28086 objects degraded (8.880%); 3201/28086 objects misplaced (11.397%); 8.6 MiB/s, 243 objects/s recovering 2024-09-02T10:33:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2494/28086 objects degraded (8.880%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:05.042 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:05.071 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:05.072 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:04 smithi162 ceph-mon[20279]: pgmap v2021: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 586 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 520 op/s; 2494/28086 objects degraded (8.880%); 3201/28086 objects misplaced (11.397%); 8.6 MiB/s, 243 objects/s recovering 2024-09-02T10:33:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2494/28086 objects degraded (8.880%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:05.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:05.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:05.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:05.272 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:05.278 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:33:05.279 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:33:05.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:05.337 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:05.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:05.341 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:33:05.341 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:33:05.357 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pgp_num 61 > pg_num 31 2024-09-02T10:33:05.388 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:33:05.388 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:33:05.439 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:05.461 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:05.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:05.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:05.589 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:05.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:05.591 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:05.626 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:33:05.627 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:33:05.665 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:33:05.665 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:33:05.691 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:05.698 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:33:05.698 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:33:05.715 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:05.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:05.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:05.817 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:05.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:05.848 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:05.929 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:33:05.929 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:33:05.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:05 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:33:05.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:05 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:33:05.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2500809383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:05.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:05 smithi151 ceph-mon[16639]: osdmap e880: 12 total, 12 up, 12 in 2024-09-02T10:33:05.930 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4024464646' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:33:05.948 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:05.961 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:33:05.961 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:33:05.971 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:05.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:05.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:05.986 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:33:05.986 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:33:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:05 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:33:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:05 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:33:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2500809383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:05 smithi187 ceph-mon[20072]: osdmap e880: 12 total, 12 up, 12 in 2024-09-02T10:33:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4024464646' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:33:06.015 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:33:06.015 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:33:06.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:33:06.047 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:33:06.048 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:33:06.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:06.075 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:33:06.075 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:33:06.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:05 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:33:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:05 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:33:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2500809383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:05 smithi162 ceph-mon[20279]: osdmap e880: 12 total, 12 up, 12 in 2024-09-02T10:33:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4024464646' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:33:06.095 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:06.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:06.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:06.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:06.220 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:06.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:06.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:06.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:06.354 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:06.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:06.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:06.481 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:06.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:06.582 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:06.604 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:06.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:06.706 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:06.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:06.743 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:06.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:06.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:33:06.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:33:06.826 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:06 smithi151 ceph-mon[16639]: pgmap v2023: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 509 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 510 op/s; 2577/24405 objects degraded (10.559%); 2630/24405 objects misplaced (10.776%); 9.6 MiB/s, 214 objects/s recovering 2024-09-02T10:33:06.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:33:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:33:06.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:06 smithi162 ceph-mon[20279]: pgmap v2023: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 509 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 510 op/s; 2577/24405 objects degraded (10.559%); 2630/24405 objects misplaced (10.776%); 9.6 MiB/s, 214 objects/s recovering 2024-09-02T10:33:06.867 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:06.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:06.969 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:06.981 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:33:06.982 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:33:06.982 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:33:06.982 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:33:06.982 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:33:06.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:06.996 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:06.996 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:33:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:33:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:06 smithi187 ceph-mon[20072]: pgmap v2023: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 509 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 510 op/s; 2577/24405 objects degraded (10.559%); 2630/24405 objects misplaced (10.776%); 9.6 MiB/s, 214 objects/s recovering 2024-09-02T10:33:07.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:07.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:07.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:07.223 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:07.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:07.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:07.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:07.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:07.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:07.509 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:07.531 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:33:07.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:07.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:07.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:07.651 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:07.673 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:07.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:07.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:07.798 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:07.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:07.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:07.900 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:07.929 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:07.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:08.031 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:08.055 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:08.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:08.157 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:08.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:08.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:08.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:08.325 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:08.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:08.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:08.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:08.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:08.456 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:08.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:08.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:08.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:08.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:08.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:08.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:08.822 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:08.853 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:08.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:08 smithi151 ceph-mon[16639]: pgmap v2024: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 445 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 2577/21363 objects degraded (12.063%); 2630/21363 objects misplaced (12.311%); 8.0 MiB/s, 208 objects/s recovering 2024-09-02T10:33:08.955 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:08.978 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:08.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:08.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:08 smithi187 ceph-mon[20072]: pgmap v2024: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 445 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 2577/21363 objects degraded (12.063%); 2630/21363 objects misplaced (12.311%); 8.0 MiB/s, 208 objects/s recovering 2024-09-02T10:33:09.080 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:08 smithi162 ceph-mon[20279]: pgmap v2024: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 445 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 528 op/s; 2577/21363 objects degraded (12.063%); 2630/21363 objects misplaced (12.311%); 8.0 MiB/s, 208 objects/s recovering 2024-09-02T10:33:09.104 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:09.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:09.206 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:09.240 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:09.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:09.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:09.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:09.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:09.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:09.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:09.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:09.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:09.503 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:09.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:09.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:09.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:09.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:09.761 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:09.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:09.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:09.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:09.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:09.891 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2577/21363 objects degraded (12.063%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:09.992 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2577/21363 objects degraded (12.063%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:10.013 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:10.014 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2577/21363 objects degraded (12.063%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:10.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:10.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:10.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:10.240 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:10.268 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:10.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:10.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:33:10.370 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:10.393 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:10.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:10.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:10.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:10.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:10.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:10.518 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:10.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:10.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:10.649 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:10.649 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:10.750 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:10.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:10.789 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:10.789 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:10.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:10.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:10.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:10 smithi151 ceph-mon[16639]: pgmap v2025: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 352 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 2658/16860 objects degraded (15.765%); 2104/16860 objects misplaced (12.479%); 5.4 MiB/s, 203 objects/s recovering 2024-09-02T10:33:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:10 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:33:10.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:10 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:33:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:10 smithi187 ceph-mon[20072]: pgmap v2025: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 352 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 2658/16860 objects degraded (15.765%); 2104/16860 objects misplaced (12.479%); 5.4 MiB/s, 203 objects/s recovering 2024-09-02T10:33:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:10 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:33:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:10 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:33:11.024 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:11.046 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:11.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:10 smithi162 ceph-mon[20279]: pgmap v2025: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 352 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 552 op/s; 2658/16860 objects degraded (15.765%); 2104/16860 objects misplaced (12.479%); 5.4 MiB/s, 203 objects/s recovering 2024-09-02T10:33:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:10 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:33:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:10 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:33:11.148 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:11.172 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:11.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:11.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:11.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:11.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:11.399 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:11.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:11.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:11.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:11.525 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:11.547 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:11.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:11.548 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:11.605 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:11.649 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:11.656 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:33:11.672 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:11.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:11.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:11.797 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:11.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:11.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:11 smithi151 ceph-mon[16639]: osdmap e881: 12 total, 12 up, 12 in 2024-09-02T10:33:11.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:11.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:12.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:11 smithi187 ceph-mon[20072]: osdmap e881: 12 total, 12 up, 12 in 2024-09-02T10:33:12.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:12.083 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:12.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:12.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:11 smithi162 ceph-mon[20279]: osdmap e881: 12 total, 12 up, 12 in 2024-09-02T10:33:12.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:12.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:12.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:12.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:12.333 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:12.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:12.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:12.456 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:12.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:12.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:12.531 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:33:12.532 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:33:12.532 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-02T10:33:12.532 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.000000 2024-09-02T10:33:12.532 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 1 0 2024-09-02T10:33:12.558 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:12.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:12.583 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:12.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:12.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:12.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:12.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:12.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:12.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:12.834 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:12.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:12.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:12 smithi151 ceph-mon[16639]: pgmap v2027: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 327 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 2658/15669 objects degraded (16.963%); 2104/15669 objects misplaced (13.428%); 5.7 MiB/s, 201 objects/s recovering 2024-09-02T10:33:12.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:12.963 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:12.963 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:12 smithi187 ceph-mon[20072]: pgmap v2027: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 327 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 2658/15669 objects degraded (16.963%); 2104/15669 objects misplaced (13.428%); 5.7 MiB/s, 201 objects/s recovering 2024-09-02T10:33:13.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:13.093 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:13.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:13.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:12 smithi162 ceph-mon[20279]: pgmap v2027: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 327 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 517 op/s; 2658/15669 objects degraded (16.963%); 2104/15669 objects misplaced (13.428%); 5.7 MiB/s, 201 objects/s recovering 2024-09-02T10:33:13.194 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:13.227 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:13.227 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:13.329 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:13.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:13.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:13.459 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:13.484 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:13.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:13.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:13.607 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:13.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:13.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:13.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:13.745 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:13.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:13.847 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:13.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:13.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:13.880 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:13.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:14.020 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:14.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:14.122 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:14.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:14.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:14.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:14.256 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:14.278 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:14.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:14.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:14.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:14.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:14.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:14.548 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:14.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:14.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:14.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:14.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:14.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:14.681 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:14.782 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:14.808 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:14.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:14.809 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:14.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:14.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:14.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:14.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:14 smithi187 ceph-mon[20072]: pgmap v2028: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 221 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 559 op/s; 2658/10590 objects degraded (25.099%); 2104/10590 objects misplaced (19.868%); 2.5 MiB/s, 157 objects/s recovering 2024-09-02T10:33:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2658/10590 objects degraded (25.099%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:15.053 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:15.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:15.080 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:15.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:14 smithi162 ceph-mon[20279]: pgmap v2028: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 221 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 559 op/s; 2658/10590 objects degraded (25.099%); 2104/10590 objects misplaced (19.868%); 2.5 MiB/s, 157 objects/s recovering 2024-09-02T10:33:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2658/10590 objects degraded (25.099%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:15.158 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:14 smithi151 ceph-mon[16639]: pgmap v2028: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 221 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 559 op/s; 2658/10590 objects degraded (25.099%); 2104/10590 objects misplaced (19.868%); 2.5 MiB/s, 157 objects/s recovering 2024-09-02T10:33:15.159 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2658/10590 objects degraded (25.099%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:15.181 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:15.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:15.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:15.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:15.338 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:15.339 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:15.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:15.473 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:15.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:15.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:15.596 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:15.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:15.698 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:15.719 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:15.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:15.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:15.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:15.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:15.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:15.848 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:15.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:15.975 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:15.976 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:16.077 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:16.112 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:16.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:16.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:16.246 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:16.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:16.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:16.374 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:16.375 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:16.476 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:16.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:16.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:16.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:16.631 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:16.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:16.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:16.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:16.754 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:16.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:16.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:16.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:16.879 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:16.980 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:17.002 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:17.003 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:16 smithi187 ceph-mon[20072]: pgmap v2029: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 137 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 2736/6555 objects degraded (41.739%); 1760/6555 objects misplaced (26.850%); 3.0 MiB/s, 198 objects/s recovering 2024-09-02T10:33:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2730158996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:33:17.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:16 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:33:17.011 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:33:17.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:16 smithi151 ceph-mon[16639]: pgmap v2029: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 137 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 2736/6555 objects degraded (41.739%); 1760/6555 objects misplaced (26.850%); 3.0 MiB/s, 198 objects/s recovering 2024-09-02T10:33:17.011 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2730158996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:33:17.012 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:16 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:33:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:16 smithi162 ceph-mon[20279]: pgmap v2029: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 137 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 594 op/s; 2736/6555 objects degraded (41.739%); 1760/6555 objects misplaced (26.850%); 3.0 MiB/s, 198 objects/s recovering 2024-09-02T10:33:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2730158996' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:33:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:16 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:33:17.104 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:17.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:17.126 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:17.180 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Removed 105278 objects 2024-09-02T10:33:17.180 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Clean up completed and total clean up time :167.05 2024-09-02T10:33:17.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:17.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:17.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:17.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:17.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:17.378 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:17.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:17.500 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:17.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:17.603 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:17.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:17.625 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:17.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:17.726 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:17.748 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:17.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:17.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:17.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:17.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:17.960 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:17 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:33:17.961 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:17 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:33:17.961 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:17 smithi151 ceph-mon[16639]: osdmap e882: 12 total, 12 up, 12 in 2024-09-02T10:33:17.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:17.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:17.997 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:17.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:17 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:33:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:17 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:33:18.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:17 smithi187 ceph-mon[20072]: osdmap e882: 12 total, 12 up, 12 in 2024-09-02T10:33:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:17 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:33:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:17 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:33:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:17 smithi162 ceph-mon[20279]: osdmap e882: 12 total, 12 up, 12 in 2024-09-02T10:33:18.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:18.120 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:18.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:18.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:18.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:18.244 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:18.345 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:18.367 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:18.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:18.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:18.491 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:18.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:18.592 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:18.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:18.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:18.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:18.719 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:18.740 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:18.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:18.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:18.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:18.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:18.967 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:18.988 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:18.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:18.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:19.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:18 smithi187 ceph-mon[20072]: pgmap v2031: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 65 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 2736/3114 objects degraded (87.861%); 1760/3114 objects misplaced (56.519%); 1.2 MiB/s, 166 objects/s recovering 2024-09-02T10:33:19.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:19.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:18 smithi162 ceph-mon[20279]: pgmap v2031: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 65 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 2736/3114 objects degraded (87.861%); 1760/3114 objects misplaced (56.519%); 1.2 MiB/s, 166 objects/s recovering 2024-09-02T10:33:19.112 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:19.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:19.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:18 smithi151 ceph-mon[16639]: pgmap v2031: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 65 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 572 op/s; 2736/3114 objects degraded (87.861%); 1760/3114 objects misplaced (56.519%); 1.2 MiB/s, 166 objects/s recovering 2024-09-02T10:33:19.213 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:19.236 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:19.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:19.237 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:19.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:19.374 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:19.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:19.446 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:19.476 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:19.497 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:19.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:19.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:19.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:19.621 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:19.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:19.622 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:19.723 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:19.745 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:19.746 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:19.847 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:19.870 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:19.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2736/3114 objects degraded (87.861%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:19.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:19 smithi151 ceph-mon[16639]: osdmap e883: 12 total, 12 up, 12 in 2024-09-02T10:33:19.971 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:19.992 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:19.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2736/3114 objects degraded (87.861%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:20.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:19 smithi187 ceph-mon[20072]: osdmap e883: 12 total, 12 up, 12 in 2024-09-02T10:33:20.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2736/3114 objects degraded (87.861%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:19 smithi162 ceph-mon[20279]: osdmap e883: 12 total, 12 up, 12 in 2024-09-02T10:33:20.115 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:20.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:20.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:20.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:20.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:20.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:20.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:20.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:20.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:20.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:20.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:20.488 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:20.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:20.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:20.612 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:20.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:20.714 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:20.736 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:20.737 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:20.838 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:20.859 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:20.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:20.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:20 smithi151 ceph-mon[16639]: pgmap v2033: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 10 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 633 op/s; 2773/471 objects degraded (588.747%); 1631/471 objects misplaced (346.285%); 1.5 MiB/s, 299 objects/s recovering 2024-09-02T10:33:20.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:20.983 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:20.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:20.984 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:21.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:20 smithi187 ceph-mon[20072]: pgmap v2033: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 10 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 633 op/s; 2773/471 objects degraded (588.747%); 1631/471 objects misplaced (346.285%); 1.5 MiB/s, 299 objects/s recovering 2024-09-02T10:33:21.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:21.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:20 smithi162 ceph-mon[20279]: pgmap v2033: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 10 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 633 op/s; 2773/471 objects degraded (588.747%); 1631/471 objects misplaced (346.285%); 1.5 MiB/s, 299 objects/s recovering 2024-09-02T10:33:21.107 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:21.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:21.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:21.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:21.231 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:21.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:21.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:21.333 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:21.355 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:21.356 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:21.394 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_6 2024-09-02T10:33:21.394 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_6 unique_pool_6 --yes-i-really-really-mean-it 2024-09-02T10:33:21.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:21.481 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:21.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:21.583 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:21.604 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:21.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:21.707 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:21.728 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:21.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:21.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:21.852 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:21.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:21.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:21.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:21.976 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:21.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:22.078 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:22.104 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:22.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:22.105 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:22.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:22.229 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:22.231 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:22.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:22.355 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:22.356 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:22.457 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:22.479 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:22.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:22.581 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:22.602 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:22.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:22.603 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:22.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:22.726 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:22.727 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:22.828 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:22.850 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:22.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:22.851 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:22.952 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:22.973 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:22.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:22 smithi187 ceph-mon[20072]: pgmap v2034: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 440 op/s; 2773/9 objects degraded (30811.111%); 1631/9 objects misplaced (18122.222%); 1.5 MiB/s, 260 objects/s recovering 2024-09-02T10:33:23.075 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:23.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:22 smithi162 ceph-mon[20279]: pgmap v2034: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 440 op/s; 2773/9 objects degraded (30811.111%); 1631/9 objects misplaced (18122.222%); 1.5 MiB/s, 260 objects/s recovering 2024-09-02T10:33:23.097 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:23.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:23.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:22 smithi151 ceph-mon[16639]: pgmap v2034: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 440 op/s; 2773/9 objects degraded (30811.111%); 1631/9 objects misplaced (18122.222%); 1.5 MiB/s, 260 objects/s recovering 2024-09-02T10:33:23.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:23.221 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:23.222 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:23.323 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:23.344 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:23.345 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:23.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:23.470 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:23.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:23.573 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:23.596 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:23.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:23.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:23.720 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:23.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:23.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:23.822 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:23.843 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:23.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:23.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:23.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:23 smithi151 ceph-mon[16639]: osdmap e884: 12 total, 12 up, 12 in 2024-09-02T10:33:23.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:23.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:23.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:23.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:23 smithi187 ceph-mon[20072]: osdmap e884: 12 total, 12 up, 12 in 2024-09-02T10:33:24.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:24.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:24.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:24.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:24.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:23 smithi162 ceph-mon[20279]: osdmap e884: 12 total, 12 up, 12 in 2024-09-02T10:33:24.194 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:24.216 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:24.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:24.318 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:24.340 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:24.340 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:24.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:24.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:24.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:24.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:24.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:24.589 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:24.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:24.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:24.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:24.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:24.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:24.713 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:24.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:24.836 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:24.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:24.938 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:24.960 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:24.960 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:24 smithi187 ceph-mon[20072]: pgmap v2036: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 2773/6 objects degraded (46216.667%); 1631/6 objects misplaced (27183.333%); 428 KiB/s, 291 objects/s recovering 2024-09-02T10:33:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:24 smithi187 ceph-mon[20072]: osdmap e885: 12 total, 12 up, 12 in 2024-09-02T10:33:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:24 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:33:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:24 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:33:25.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2773/6 objects degraded (46216.667%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:25.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:24 smithi187 ceph-mon[20072]: osdmap e886: 12 total, 12 up, 12 in 2024-09-02T10:33:25.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:25.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:24 smithi151 ceph-mon[16639]: pgmap v2036: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 2773/6 objects degraded (46216.667%); 1631/6 objects misplaced (27183.333%); 428 KiB/s, 291 objects/s recovering 2024-09-02T10:33:25.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:24 smithi151 ceph-mon[16639]: osdmap e885: 12 total, 12 up, 12 in 2024-09-02T10:33:25.073 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:24 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:33:25.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:24 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:33:25.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2773/6 objects degraded (46216.667%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:25.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:24 smithi151 ceph-mon[16639]: osdmap e886: 12 total, 12 up, 12 in 2024-09-02T10:33:25.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:25.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:24 smithi162 ceph-mon[20279]: pgmap v2036: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 289 op/s; 2773/6 objects degraded (46216.667%); 1631/6 objects misplaced (27183.333%); 428 KiB/s, 291 objects/s recovering 2024-09-02T10:33:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:24 smithi162 ceph-mon[20279]: osdmap e885: 12 total, 12 up, 12 in 2024-09-02T10:33:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:24 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:33:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:24 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:33:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2773/6 objects degraded (46216.667%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:24 smithi162 ceph-mon[20279]: osdmap e886: 12 total, 12 up, 12 in 2024-09-02T10:33:25.186 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:25.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:25.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:25.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:25.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:25.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:25.439 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:25.461 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:25.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:25.462 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:25.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:25.587 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:25.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:25.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:25.714 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:25.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:25.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:25.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:25.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:25.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:25.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:25.963 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:25.964 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:26.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:26.087 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:26.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:26.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:26.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:26.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:26.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:26.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:26.339 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:26.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:26.428 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:33:26.428 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:33:26.428 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:33:26.428 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:33:26.428 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:33:26.428 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:33:26.429 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:33:26.429 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:33:26.429 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:33:26.429 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:33:26.429 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:33:26.429 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:33:26.429 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:33:26.429 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:33:26.429 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:33:26.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:33:26.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:33:26.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:33:26.430 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:33:26.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:33:26.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:33:26.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:33:26.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:33:26.430 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:33:26.441 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:26.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:26.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:26.464 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:26.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:26.588 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:26.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:26.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:26 smithi151 ceph-mon[16639]: osdmap e887: 12 total, 12 up, 12 in 2024-09-02T10:33:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:26 smithi151 ceph-mon[16639]: pgmap v2040: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 2387/6 objects degraded (39783.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 634 objects/s recovering 2024-09-02T10:33:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2318845442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:26.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:26.715 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:26.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:26.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:26 smithi187 ceph-mon[20072]: osdmap e887: 12 total, 12 up, 12 in 2024-09-02T10:33:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:26 smithi187 ceph-mon[20072]: pgmap v2040: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 2387/6 objects degraded (39783.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 634 objects/s recovering 2024-09-02T10:33:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2318845442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:26.817 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:26.840 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:26.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:26 smithi162 ceph-mon[20279]: osdmap e887: 12 total, 12 up, 12 in 2024-09-02T10:33:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:26 smithi162 ceph-mon[20279]: pgmap v2040: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+forced_recovery+undersized+degraded+remapped, 43 active+clean; 577 KiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s; 2387/6 objects degraded (39783.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 634 objects/s recovering 2024-09-02T10:33:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2318845442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:26.942 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:26.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:26.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:26.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:27.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:27.092 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:27.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:27.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:27.215 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:27.216 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:27.317 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:27.343 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:27.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:27.445 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:27.471 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:27.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:27.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:27.599 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:27.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:27.701 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:27.727 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:27.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:27.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:27.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:27.851 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:27.852 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:27.953 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:27.979 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:27.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:28.081 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:28.102 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:28.103 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:28.204 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:28.230 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:28.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:28.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:28.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:28.358 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:28.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:28.461 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:28.487 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:28.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:28.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:28.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:28.610 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:28.612 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:28.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:28.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:28.735 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:28.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:28.861 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:28.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:28 smithi151 ceph-mon[16639]: pgmap v2041: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+forced_recovery+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2284/6 objects degraded (38066.667%); 1631/6 objects misplaced (27183.333%); 0 B/s, 431 objects/s recovering 2024-09-02T10:33:28.963 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:28.987 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:28.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:29.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:28 smithi187 ceph-mon[20072]: pgmap v2041: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+forced_recovery+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2284/6 objects degraded (38066.667%); 1631/6 objects misplaced (27183.333%); 0 B/s, 431 objects/s recovering 2024-09-02T10:33:29.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:29.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:28 smithi162 ceph-mon[20279]: pgmap v2041: 47 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+forced_recovery+undersized+degraded+remapped, 44 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2284/6 objects degraded (38066.667%); 1631/6 objects misplaced (27183.333%); 0 B/s, 431 objects/s recovering 2024-09-02T10:33:29.114 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:29.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:29.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:29.217 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:29.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:29.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:29.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:29.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:29.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:29.364 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:29.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:29.486 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:29.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:29.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:29.611 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:29.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:29.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:29.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:29.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:29.735 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:29.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:29.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:29.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:29 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2284/6 objects degraded (38066.667%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:29 smithi151 ceph-mon[16639]: osdmap e888: 12 total, 12 up, 12 in 2024-09-02T10:33:29.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:29.984 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:29.984 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:29 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2284/6 objects degraded (38066.667%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:29 smithi187 ceph-mon[20072]: osdmap e888: 12 total, 12 up, 12 in 2024-09-02T10:33:30.085 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:30.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:29 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2284/6 objects degraded (38066.667%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:30.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:29 smithi162 ceph-mon[20279]: osdmap e888: 12 total, 12 up, 12 in 2024-09-02T10:33:30.112 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:30.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:30.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:30.238 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:30.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:30.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:30.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:30.364 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:30.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:30.466 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:30.490 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:30.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:30.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:30.592 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:30.614 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:30.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:30.614 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:30.715 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:30.740 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:30.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:30.841 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:30.863 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:30.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:30.864 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:30.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:30 smithi151 ceph-mon[16639]: pgmap v2043: 47 pgs: 1 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1865/6 objects degraded (31083.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 557 objects/s recovering 2024-09-02T10:33:30.964 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:30.989 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:30.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:30 smithi187 ceph-mon[20072]: pgmap v2043: 47 pgs: 1 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1865/6 objects degraded (31083.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 557 objects/s recovering 2024-09-02T10:33:31.091 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:31.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:30 smithi162 ceph-mon[20279]: pgmap v2043: 47 pgs: 1 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1865/6 objects degraded (31083.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 557 objects/s recovering 2024-09-02T10:33:31.115 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:31.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:31.217 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:31.239 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:31.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:31.340 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:31.363 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:31.363 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:31.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:31.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:31.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:31.589 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:31.610 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:31.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:31.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:31.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:31.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:31.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:31.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:31.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:31.859 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:31.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:31.992 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:31.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:32.096 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:32.118 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:32.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:32.119 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:32.220 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:32.241 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:32.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:32.343 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:32.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:32.365 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:32.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:32.489 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:32.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:32.490 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:32.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:32.617 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:32.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:32.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:32.652 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:32.653 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:32.664 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:32.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:32.741 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:32.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:32.742 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:32.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:32.869 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:32.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:32.887 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:32 smithi151 ceph-mon[16639]: pgmap v2044: 47 pgs: 1 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1865/6 objects degraded (31083.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 425 objects/s recovering 2024-09-02T10:33:32.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:32.992 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:32.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:32.994 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:32 smithi187 ceph-mon[20072]: pgmap v2044: 47 pgs: 1 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1865/6 objects degraded (31083.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 425 objects/s recovering 2024-09-02T10:33:33.095 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:32 smithi162 ceph-mon[20279]: pgmap v2044: 47 pgs: 1 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1865/6 objects degraded (31083.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 425 objects/s recovering 2024-09-02T10:33:33.121 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:33.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:33.223 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:33.244 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:33.245 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:33.346 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:33.369 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:33.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:33.471 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:33.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:33.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:33.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:33.594 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:33.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:33.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:33.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:33.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:33.744 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:33.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:33.872 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:33.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:33.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:33.972 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:33:33.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:34.001 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:34.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:34.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:34.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:34.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:34.127 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:34.228 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:34.250 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:34.251 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:34.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:34.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:34.376 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:34.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:34.504 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:34.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:34.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:34.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:34.627 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:34.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:34.652 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:33:34.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:34.750 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:34.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:34.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:34.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:34.878 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:34.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:34 smithi151 ceph-mon[16639]: pgmap v2045: 47 pgs: 1 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1865/6 objects degraded (31083.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 161 objects/s recovering 2024-09-02T10:33:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:34 smithi151 ceph-mon[16639]: osdmap e889: 12 total, 12 up, 12 in 2024-09-02T10:33:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:34 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1865/6 objects degraded (31083.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:34 smithi151 ceph-mon[16639]: osdmap e890: 12 total, 12 up, 12 in 2024-09-02T10:33:34.980 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:35.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:35.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:34 smithi187 ceph-mon[20072]: pgmap v2045: 47 pgs: 1 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1865/6 objects degraded (31083.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 161 objects/s recovering 2024-09-02T10:33:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:34 smithi187 ceph-mon[20072]: osdmap e889: 12 total, 12 up, 12 in 2024-09-02T10:33:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:34 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1865/6 objects degraded (31083.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:34 smithi187 ceph-mon[20072]: osdmap e890: 12 total, 12 up, 12 in 2024-09-02T10:33:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:34 smithi162 ceph-mon[20279]: pgmap v2045: 47 pgs: 1 active+recovering+forced_recovery+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 45 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1865/6 objects degraded (31083.333%); 1631/6 objects misplaced (27183.333%); 0 B/s, 161 objects/s recovering 2024-09-02T10:33:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:34 smithi162 ceph-mon[20279]: osdmap e889: 12 total, 12 up, 12 in 2024-09-02T10:33:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:34 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1865/6 objects degraded (31083.333%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:33:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:34 smithi162 ceph-mon[20279]: osdmap e890: 12 total, 12 up, 12 in 2024-09-02T10:33:35.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:35.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:35.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:35.227 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:35.249 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:35.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:35.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:35.351 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:35.377 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:35.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:35.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:35.504 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:35.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:35.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:35.606 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:35.631 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:35.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:35.632 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:35.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:35.759 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:35.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:35.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:35.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:35.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:35.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:35.984 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:36.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:36.011 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:36.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:36.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:36.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:36.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:36.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:36.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:36.266 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:36.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:36.368 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:36.394 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:36.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:36.496 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:36.518 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:36.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:36.519 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:36.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:36.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:36.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:36.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:36.769 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:36.770 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:36.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:36.898 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:36.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:36.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:36.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:36 smithi151 ceph-mon[16639]: pgmap v2048: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1633/6 objects degraded (27216.667%); 1631/6 objects misplaced (27183.333%); 0 B/s, 247 objects/s recovering 2024-09-02T10:33:37.000 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:36 smithi187 ceph-mon[20072]: pgmap v2048: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1633/6 objects degraded (27216.667%); 1631/6 objects misplaced (27183.333%); 0 B/s, 247 objects/s recovering 2024-09-02T10:33:37.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:37.023 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:36 smithi162 ceph-mon[20279]: pgmap v2048: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1633/6 objects degraded (27216.667%); 1631/6 objects misplaced (27183.333%); 0 B/s, 247 objects/s recovering 2024-09-02T10:33:37.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:37.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:37.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:37.248 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:37.270 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:37.270 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:37.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:37.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:37.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:37.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:37.522 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:37.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:37.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:37.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:37.649 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:37.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:37.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:37.773 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:37.774 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:37.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:37.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:37.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:37.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:37.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4107085397' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-02T10:33:38.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:38.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4107085397' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-02T10:33:38.029 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:38.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:38.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4107085397' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 0.0} : dispatch 2024-09-02T10:33:38.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:38.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:38.154 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:38.234 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.1 primary-affinity to 0 (802) 2024-09-02T10:33:38.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:38.280 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:38.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:38.382 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:38.408 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:38.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:38.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:38.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:38.532 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:38.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:38.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:38.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:38.661 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:38.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:38.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:38.763 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:38.786 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:38.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:38.888 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:38.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:38.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:38.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:38.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:38 smithi151 ceph-mon[16639]: pgmap v2049: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1633/6 objects degraded (27216.667%); 1631/6 objects misplaced (27183.333%); 0 B/s, 84 objects/s recovering 2024-09-02T10:33:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4107085397' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-02T10:33:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:38 smithi151 ceph-mon[16639]: osdmap e891: 12 total, 12 up, 12 in 2024-09-02T10:33:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:38 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:33:38.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:38 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:33:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:38 smithi187 ceph-mon[20072]: pgmap v2049: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1633/6 objects degraded (27216.667%); 1631/6 objects misplaced (27183.333%); 0 B/s, 84 objects/s recovering 2024-09-02T10:33:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4107085397' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-02T10:33:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:38 smithi187 ceph-mon[20072]: osdmap e891: 12 total, 12 up, 12 in 2024-09-02T10:33:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:38 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:33:39.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:38 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:33:39.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:39.034 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:39.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:38 smithi162 ceph-mon[20279]: pgmap v2049: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 577 KiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1633/6 objects degraded (27216.667%); 1631/6 objects misplaced (27183.333%); 0 B/s, 84 objects/s recovering 2024-09-02T10:33:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4107085397' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.0}]': finished 2024-09-02T10:33:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:38 smithi162 ceph-mon[20279]: osdmap e891: 12 total, 12 up, 12 in 2024-09-02T10:33:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:38 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:33:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:38 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:33:39.135 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:39.159 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:39.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:39.261 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:39.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:39.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:39.384 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:39.405 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:39.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:39.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:39.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:39.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:39.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:39.530 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:39.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:39.652 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:39.653 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:39.754 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:39.775 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:39.777 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:39.878 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:39.899 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:39.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:39.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:39 smithi151 ceph-mon[16639]: osdmap e892: 12 total, 12 up, 12 in 2024-09-02T10:33:39.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1633/6 objects degraded (27216.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:33:40.000 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:39 smithi187 ceph-mon[20072]: osdmap e892: 12 total, 12 up, 12 in 2024-09-02T10:33:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1633/6 objects degraded (27216.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:33:40.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:40.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:39 smithi162 ceph-mon[20279]: osdmap e892: 12 total, 12 up, 12 in 2024-09-02T10:33:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1633/6 objects degraded (27216.667%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-09-02T10:33:40.124 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:40.145 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:40.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:40.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:40.272 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:40.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:40.375 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:40.396 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:40.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:40.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:40.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:40.525 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:40.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:40.626 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:40.652 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:40.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:40.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:40.779 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:40.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:40.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:40.881 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:40.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:40.903 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:40.919 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:40 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:33:40.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:40 smithi151 ceph-mon[16639]: pgmap v2052: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 43 active+clean; 590 KiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 345 B/s rd, 8.3 KiB/s wr, 0 op/s; 1631/6 objects degraded (27183.333%); 1631/6 objects misplaced (27183.333%); 2.0 KiB/s, 0 objects/s recovering 2024-09-02T10:33:40.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:40 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:33:41.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:40 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:33:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:40 smithi187 ceph-mon[20072]: pgmap v2052: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 43 active+clean; 590 KiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 345 B/s rd, 8.3 KiB/s wr, 0 op/s; 1631/6 objects degraded (27183.333%); 1631/6 objects misplaced (27183.333%); 2.0 KiB/s, 0 objects/s recovering 2024-09-02T10:33:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:40 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:33:41.030 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:41.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:40 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:33:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:40 smithi162 ceph-mon[20279]: pgmap v2052: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 43 active+clean; 590 KiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 345 B/s rd, 8.3 KiB/s wr, 0 op/s; 1631/6 objects degraded (27183.333%); 1631/6 objects misplaced (27183.333%); 2.0 KiB/s, 0 objects/s recovering 2024-09-02T10:33:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:40 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-02T10:33:41.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:41.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:41.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:41.154 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:41.255 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:41.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:41.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:41.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:41.399 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:41.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:41.400 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:41.501 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:41.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:41.523 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:41.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:41.645 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:41.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:41.747 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:41.770 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:41.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:41.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:41.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:41.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:41.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:41.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4063877240' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:33:41.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4063877240' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:33:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:42.017 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:42.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:42.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4063877240' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:33:42.118 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:42.140 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:42.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:42.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:42.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:42.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:42.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:42.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:42.391 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:42.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:42.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:42.492 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:42.514 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:42.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:42.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:42.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:42.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:42.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:42.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:42.765 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:42.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:42.866 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:42.896 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:42.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:42.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:42 smithi151 ceph-mon[16639]: pgmap v2053: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 43 active+clean; 590 KiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 283 B/s rd, 6.8 KiB/s wr, 0 op/s; 1631/6 objects degraded (27183.333%); 1631/6 objects misplaced (27183.333%); 1.6 KiB/s, 0 objects/s recovering 2024-09-02T10:33:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4063877240' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:33:42.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:42 smithi151 ceph-mon[16639]: osdmap e893: 12 total, 12 up, 12 in 2024-09-02T10:33:42.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:42 smithi187 ceph-mon[20072]: pgmap v2053: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 43 active+clean; 590 KiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 283 B/s rd, 6.8 KiB/s wr, 0 op/s; 1631/6 objects degraded (27183.333%); 1631/6 objects misplaced (27183.333%); 1.6 KiB/s, 0 objects/s recovering 2024-09-02T10:33:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4063877240' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:33:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:42 smithi187 ceph-mon[20072]: osdmap e893: 12 total, 12 up, 12 in 2024-09-02T10:33:43.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:43.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:42 smithi162 ceph-mon[20279]: pgmap v2053: 47 pgs: 1 active+recovering+undersized+degraded+remapped, 3 peering, 43 active+clean; 590 KiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 283 B/s rd, 6.8 KiB/s wr, 0 op/s; 1631/6 objects degraded (27183.333%); 1631/6 objects misplaced (27183.333%); 1.6 KiB/s, 0 objects/s recovering 2024-09-02T10:33:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4063877240' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_6", "pool2": "unique_pool_6", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:33:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:42 smithi162 ceph-mon[20279]: osdmap e893: 12 total, 12 up, 12 in 2024-09-02T10:33:43.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:43.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:43.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:43.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:43.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:43.267 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:43.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:43.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:43.369 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:43.394 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:43.394 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:43.487 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_6' removed 2024-09-02T10:33:43.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:43.519 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:43.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:43.520 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:43.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:43.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:43.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:43.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:43.766 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:43 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:33:43.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:43 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1631/6 objects degraded (27183.333%), 1 pg degraded, 1 pg undersized) 2024-09-02T10:33:43.772 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:43.773 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:43.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:43.900 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:43.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:44.002 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:43 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:33:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:43 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1631/6 objects degraded (27183.333%), 1 pg degraded, 1 pg undersized) 2024-09-02T10:33:44.028 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:44.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:44.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:43 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-02T10:33:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:43 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1631/6 objects degraded (27183.333%), 1 pg degraded, 1 pg undersized) 2024-09-02T10:33:44.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:44.155 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:44.156 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:44.257 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:44.283 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:44.284 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:44.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:44.411 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:44.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:44.513 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:44.540 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:44.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:44.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:44.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:44.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:44.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:44.795 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:44.796 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:44.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:44.924 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:44.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:44 smithi187 ceph-mon[20072]: pgmap v2055: 31 pgs: 31 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s 2024-09-02T10:33:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:44 smithi187 ceph-mon[20072]: osdmap e894: 12 total, 12 up, 12 in 2024-09-02T10:33:45.025 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:45.051 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:45.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:44 smithi162 ceph-mon[20279]: pgmap v2055: 31 pgs: 31 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s 2024-09-02T10:33:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:44 smithi162 ceph-mon[20279]: osdmap e894: 12 total, 12 up, 12 in 2024-09-02T10:33:45.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:45.179 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:45.180 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:44 smithi151 ceph-mon[16639]: pgmap v2055: 31 pgs: 31 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s rd, 0 op/s 2024-09-02T10:33:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:44 smithi151 ceph-mon[16639]: osdmap e894: 12 total, 12 up, 12 in 2024-09-02T10:33:45.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:45.308 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:45.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:45.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:45.409 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:45.435 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:45.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:45.436 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:45.537 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:45.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:45.563 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:45.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:45.690 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:45.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:45.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:45.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:45.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:45.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:45.906 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:33:45.906 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:33:45.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:45.943 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:45.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:45.944 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:46.045 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:46.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:46.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:46.169 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:46.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:46.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:46.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:46.317 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:46.319 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:46.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:46.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:46.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:46.543 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:46.570 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:46.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:46.672 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:46.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:46.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:46.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:46.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:46.825 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:46.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:46.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:46.908 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-02T10:33:46.910 INFO:tasks.radosbench:Beginning radosbench... 2024-09-02T10:33:46.910 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_7 2024-09-02T10:33:46.910 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool create unique_pool_7 16 2024-09-02T10:33:46.927 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:46.953 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:46.954 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:47.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:46 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:33:47.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:46 smithi151 ceph-mon[16639]: pgmap v2057: 31 pgs: 31 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:46 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:33:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:46 smithi187 ceph-mon[20072]: pgmap v2057: 31 pgs: 31 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:47.055 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:47.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:47.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:46 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:33:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:46 smithi162 ceph-mon[20279]: pgmap v2057: 31 pgs: 31 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:47.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:47.205 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:47.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:47.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:47.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:47.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:47.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:47.409 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:47.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:47.443 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:47.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:33:47.458 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:47.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:47.560 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:47.586 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:47.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:47.587 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:47.688 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:47.713 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:47.714 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:47.815 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:47.840 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:47.841 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:47.942 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:47.966 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:47.967 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:48.068 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:48.093 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:48.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:48.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:48.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:48.219 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:48.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:48.322 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:48.346 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:48.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:48.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:48.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:33:48.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:33:48.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:33:48.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:33:48.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:33:48.385 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:33:48.385 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:33:48.385 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:33:48.385 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:33:48.385 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:33:48.385 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:33:48.385 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:33:48.386 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:33:48.386 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:33:48.386 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:33:48.386 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:33:48.386 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:33:48.386 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:33:48.386 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:33:48.386 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:33:48.386 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:33:48.386 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:33:48.403 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:33:48.404 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:33:48.447 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:48.472 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:48.474 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:48.575 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:48.597 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:48.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:48.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:48.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:48.725 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:48.726 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:48.741 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:33:48.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:48.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:48.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:48.949 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:48.971 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:48.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:48.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:48 smithi187 ceph-mon[20072]: pgmap v2058: 31 pgs: 31 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/187376484' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1932939569' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:33:49.072 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:49.094 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:49.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:48 smithi162 ceph-mon[20279]: pgmap v2058: 31 pgs: 31 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/187376484' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1932939569' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:33:49.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:48 smithi151 ceph-mon[16639]: pgmap v2058: 31 pgs: 31 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:49.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/187376484' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:49.156 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1932939569' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:33:49.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:49.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:49.218 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:49.319 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:49.341 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:49.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:49.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:49.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:49.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:49.567 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:49.588 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:49.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:49.589 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:49.690 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:49.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:49.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:49.785 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:33:49.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1932939569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:33:49.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: osdmap e895: 12 total, 12 up, 12 in 2024-09-02T10:33:49.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:33:49.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:33:49.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:33:49.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:33:49.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:33:49.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:33:49.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:33:49.786 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:49 smithi151 ceph-mon[16639]: osdmap e896: 12 total, 12 up, 12 in 2024-09-02T10:33:49.814 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:49.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:49.836 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:49.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:49.854 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:49.859 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:49.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:49.959 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:49.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:33:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1932939569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:33:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: osdmap e895: 12 total, 12 up, 12 in 2024-09-02T10:33:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:33:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:33:50.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:33:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:33:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:33:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:33:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:33:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:49 smithi187 ceph-mon[20072]: osdmap e896: 12 total, 12 up, 12 in 2024-09-02T10:33:50.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:50.083 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:50.083 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:33:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1932939569' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:33:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: osdmap e895: 12 total, 12 up, 12 in 2024-09-02T10:33:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:33:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:33:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:33:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:33:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:33:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:33:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:33:50.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:49 smithi162 ceph-mon[20279]: osdmap e896: 12 total, 12 up, 12 in 2024-09-02T10:33:50.183 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:50.205 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:50.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:50.307 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:50.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:50.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:50.330 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:50.431 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:50.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:50.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:50.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:50.554 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:50.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:50.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:50.578 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:50.602 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:33:50.602 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:33:50.609 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:33:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:33:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:33:50.609 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:33:50.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:50.705 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:50.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:50.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:50.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:50.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:50.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:50.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:50.952 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:50.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:50.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:50 smithi187 ceph-mon[20072]: pgmap v2061: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:51.054 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:51.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:50 smithi151 ceph-mon[16639]: pgmap v2061: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:51.081 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:51.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:50 smithi162 ceph-mon[20279]: pgmap v2061: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:51.182 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:51.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:33:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:51.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:51.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:51.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:51.326 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:51.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:51.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:51.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:51.449 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:51.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:51.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:51.551 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:51.568 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:33:51.573 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:51.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:51.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:51.633 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:51.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:51.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:51.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:51.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:51.742 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_7' created 2024-09-02T10:33:51.800 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:51.813 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:33:51.827 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:51.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:51.829 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:51.835 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1078673421' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-02T10:33:51.930 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:51.955 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:51.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:51.956 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:52.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1078673421' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-02T10:33:52.057 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:52.083 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:52.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:52.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1078673421' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16} : dispatch 2024-09-02T10:33:52.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:52.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:52.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:52.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:52.336 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:52.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:52.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:52.446 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:33:52.446 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:33:52.461 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:52.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:52.476 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:33:52.476 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:33:52.481 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:52.494 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:52.499 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:33:52.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:33:52.525 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:33:52.525 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:33:52.547 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:33:52.547 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:33:52.563 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:52.568 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:33:52.568 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:33:52.584 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:52.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:52.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:52.590 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:33:52.590 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:33:52.611 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:33:52.612 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:33:52.633 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:33:52.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:33:52.655 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:33:52.655 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:33:52.676 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:33:52.676 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:33:52.685 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:52.697 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:33:52.698 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:33:52.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:52.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:52.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:52.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:52.837 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:52.838 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:52 smithi151 ceph-mon[16639]: pgmap v2062: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1078673421' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-02T10:33:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:52 smithi151 ceph-mon[16639]: osdmap e897: 12 total, 12 up, 12 in 2024-09-02T10:33:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2700274383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:52.938 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:52.964 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:52.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:52.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:52 smithi187 ceph-mon[20072]: pgmap v2062: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1078673421' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-02T10:33:53.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:52 smithi187 ceph-mon[20072]: osdmap e897: 12 total, 12 up, 12 in 2024-09-02T10:33:53.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2700274383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:53.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:53.088 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:53.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:53.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:53.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:52 smithi162 ceph-mon[20279]: pgmap v2062: 31 pgs: 1 active+clean+scrubbing, 30 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1078673421' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_7", "pg_num": 16}]': finished 2024-09-02T10:33:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:52 smithi162 ceph-mon[20279]: osdmap e897: 12 total, 12 up, 12 in 2024-09-02T10:33:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2700274383' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:53.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:53.212 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:53.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:53.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:53.313 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:53.340 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:53.341 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:53.442 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:53.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:53.470 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:53.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:53.571 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:53.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:53.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:53.695 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:53.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:53.721 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:53.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:53.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:53.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:53.849 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:53.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool application enable unique_pool_7 rados --yes-i-really-mean-it || true 2024-09-02T10:33:53.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:53.976 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:53.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:54.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:53 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:33:54.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:53 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:33:54.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:53 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:33:54.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:53 smithi151 ceph-mon[16639]: osdmap e898: 12 total, 12 up, 12 in 2024-09-02T10:33:54.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:53 smithi151 ceph-mon[16639]: osdmap e899: 12 total, 12 up, 12 in 2024-09-02T10:33:54.077 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:53 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:33:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:53 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:33:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:53 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:33:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:53 smithi162 ceph-mon[20279]: osdmap e898: 12 total, 12 up, 12 in 2024-09-02T10:33:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:53 smithi162 ceph-mon[20279]: osdmap e899: 12 total, 12 up, 12 in 2024-09-02T10:33:54.103 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:54.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:54.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:54.205 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:54.231 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:54.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:54.232 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:53 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:33:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:53 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:33:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:53 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:33:54.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:53 smithi187 ceph-mon[20072]: osdmap e898: 12 total, 12 up, 12 in 2024-09-02T10:33:54.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:53 smithi187 ceph-mon[20072]: osdmap e899: 12 total, 12 up, 12 in 2024-09-02T10:33:54.333 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:54.353 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:33:54.360 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:54.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:54.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:54.462 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:54.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:54.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:54.489 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:54.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:54.616 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:54.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:54.718 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:54.742 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:54.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:54.822 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:33:54.823 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:33:54.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:54.870 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:54.872 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:54.973 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:54.999 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:55.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:55.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:54 smithi162 ceph-mon[20279]: pgmap v2065: 47 pgs: 7 unknown, 7 creating+peering, 33 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1155293900' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:33:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:54 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:33:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1155293900' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:33:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:54 smithi162 ceph-mon[20279]: osdmap e900: 12 total, 12 up, 12 in 2024-09-02T10:33:55.101 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:55.123 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:55.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:54 smithi151 ceph-mon[16639]: pgmap v2065: 47 pgs: 7 unknown, 7 creating+peering, 33 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1155293900' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:33:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:54 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:33:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1155293900' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:33:55.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:54 smithi151 ceph-mon[16639]: osdmap e900: 12 total, 12 up, 12 in 2024-09-02T10:33:55.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:55.253 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:55.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:54 smithi187 ceph-mon[20072]: pgmap v2065: 47 pgs: 7 unknown, 7 creating+peering, 33 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1155293900' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:33:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:54 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:33:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1155293900' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:33:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:54 smithi187 ceph-mon[20072]: osdmap e900: 12 total, 12 up, 12 in 2024-09-02T10:33:55.353 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:55.379 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:55.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:55.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:55.475 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:55.480 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:33:55.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:55.507 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:55.507 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:55.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:55.635 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:55.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:55.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:55.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:55.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:55.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:33:55.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:55.883 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:55.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:33:55.929 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:33:55.985 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:56.005 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:56.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:56.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:55 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:33:56.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:55 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:33:56.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:55 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:33:56.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:55 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:33:56.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:55 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:33:56.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:55 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:33:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:55 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:33:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:55 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:33:56.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:55 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:33:56.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:55 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:33:56.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:55 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:33:56.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:55 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:33:56.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:56.134 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:56.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:56.135 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:56.236 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:56.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:55 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:33:56.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:55 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:33:56.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:55 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:33:56.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:55 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:33:56.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:55 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:33:56.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:55 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:33:56.262 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:56.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:56.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:56.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:56.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:56.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:56.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:56.516 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:56.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:56.606 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:56.607 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:56.617 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:56.643 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:56.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:56.644 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:56.745 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:56.771 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:56.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:56.776 INFO:teuthology.orchestra.run.smithi151.stderr:enabled application 'rados' on pool 'unique_pool_7' 2024-09-02T10:33:56.872 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:56.899 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:56.900 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:57.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:57.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:57.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:33:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:33:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:33:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:33:57.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: pgmap v2068: 47 pgs: 2 unknown, 7 creating+peering, 38 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:57.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:33:57.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:33:57.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:33:57.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:33:57.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:33:57.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:33:57.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3711813571' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:33:57.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:57.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:57.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:33:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:33:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:33:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:33:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: pgmap v2068: 47 pgs: 2 unknown, 7 creating+peering, 38 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:33:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:33:57.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:33:57.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:33:57.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:33:57.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:33:57.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3711813571' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:33:57.253 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:33:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:33:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:33:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:33:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:33:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: pgmap v2068: 47 pgs: 2 unknown, 7 creating+peering, 38 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:33:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:33:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:33:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:33:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:33:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:33:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3711813571' entity='client.admin' cmd={"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true} : dispatch 2024-09-02T10:33:57.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:57.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:33:57.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:33:57.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:33:57.380 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:33:57.380 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:33:57.404 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:57.405 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:33:57.506 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:57.533 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:57.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:57.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:57.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:57.661 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:57.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:57.763 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:57.789 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:57.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:57.790 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:57.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:57.917 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:57.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:58.019 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:58.045 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:58.046 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:57 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:33:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:57 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:33:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3711813571' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:33:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:57 smithi162 ceph-mon[20279]: osdmap e901: 12 total, 12 up, 12 in 2024-09-02T10:33:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:57 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:33:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:57 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:33:58.100 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:57 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:33:58.101 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:57 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:33:58.101 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3711813571' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:33:58.101 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:57 smithi151 ceph-mon[16639]: osdmap e901: 12 total, 12 up, 12 in 2024-09-02T10:33:58.101 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:57 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:33:58.101 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:57 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:33:58.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:33:58.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:33:58.172 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:58.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:33:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:57 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:33:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:57 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:33:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3711813571' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_7", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-02T10:33:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:57 smithi187 ceph-mon[20072]: osdmap e901: 12 total, 12 up, 12 in 2024-09-02T10:33:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:57 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:33:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:57 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:33:58.274 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:58.296 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:58.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:58.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:58.423 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:58.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:58.525 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:33:58.551 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:58.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:58.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:33:58.654 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:33:58.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:58.681 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:33:58.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:58.807 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:58.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:33:58.904 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:58.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:58.932 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:58.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:59.034 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:33:59.060 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:59.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:33:59.083 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:58 smithi151 ceph-mon[16639]: pgmap v2070: 47 pgs: 4 creating+peering, 43 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:58 smithi162 ceph-mon[20279]: pgmap v2070: 47 pgs: 4 creating+peering, 43 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:59.162 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:59.189 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:59.190 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:58 smithi187 ceph-mon[20072]: pgmap v2070: 47 pgs: 4 creating+peering, 43 active+clean; 590 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:33:59.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:59.313 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:59.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:59.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:59.415 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:33:59.429 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:33:59.429 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:33:59.439 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:59.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:59.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:33:59.449 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:33:59.450 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:33:59.470 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:33:59.470 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:33:59.490 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:33:59.490 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:33:59.512 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:33:59.513 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:33:59.535 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:33:59.535 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:33:59.541 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:33:59.556 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:33:59.556 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:33:59.564 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:33:59.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:59.565 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:33:59.578 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:33:59.578 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:33:59.599 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:33:59.599 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:33:59.623 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:33:59.623 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:33:59.650 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:33:59.651 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:33:59.653 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:33:59.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:33:59.673 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:33:59.673 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:33:59.692 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:59.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:33:59.796 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:33:59.817 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:33:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:59.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:33:59.919 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:33:59.935 DEBUG:teuthology.orchestra.run.smithi187:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.2 -b 65536 --object-size 65536 -p unique_pool_7 bench 90 write' 2024-09-02T10:33:59.936 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-02T10:33:59.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:33:59.938 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-02T10:33:59.938 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-02T10:33:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:59 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:33:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:59 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:33:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:59 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:33:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:59 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:33:59.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:59 smithi151 ceph-mon[16639]: osdmap e902: 12 total, 12 up, 12 in 2024-09-02T10:33:59.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:33:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3348513230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:33:59.946 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:33:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:33:59.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:00.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:hints = 1 2024-09-02T10:34:00.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-02T10:34:00.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object prefix: benchmark_data_smithi187_154927 2024-09-02T10:34:00.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:34:00.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 0 0 0 0 0 0 - 0 2024-09-02T10:34:00.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:00.073 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:00.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:59 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:34:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:59 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:34:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:59 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:34:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:59 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:34:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:59 smithi162 ceph-mon[20279]: osdmap e902: 12 total, 12 up, 12 in 2024-09-02T10:34:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:33:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3348513230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:00.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:00.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:00.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:59 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:34:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:59 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:34:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:59 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:34:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:59 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:34:00.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:59 smithi187 ceph-mon[20072]: osdmap e902: 12 total, 12 up, 12 in 2024-09-02T10:34:00.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:33:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3348513230' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:00.298 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:00.326 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:00.327 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:00.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:00.454 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:00.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:00.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:00.582 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:00.583 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:00.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:00.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:00.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:00.707 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:00.785 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:34:00.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:00.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:00.834 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:00.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:00.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:00 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:34:00.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:00 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:34:00.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:00 smithi151 ceph-mon[16639]: pgmap v2072: 47 pgs: 47 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:34:00.918 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4016542748' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:00.935 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:00.964 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:00.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:01.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 1 16 2176 2160 134.992 135 0.00122492 0.00731777 2024-09-02T10:34:01.066 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:01.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:01.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:01.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:00 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:34:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:00 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:34:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:00 smithi162 ceph-mon[20279]: pgmap v2072: 47 pgs: 47 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:34:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4016542748' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:01.140 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:34:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:34:01.140 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 0 2024-09-02T10:34:01.140 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:34:01.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:01.214 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:01.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:00 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:34:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:00 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:34:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:00 smithi187 ceph-mon[20072]: pgmap v2072: 47 pgs: 47 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:34:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4016542748' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:01.317 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:01.339 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:01.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:01.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:01.441 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:01.463 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:01.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:01.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:01.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:01.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:01.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:01.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:01.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:01.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:01.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:01.818 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:01.847 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:01.847 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:01.948 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:01.973 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:01.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:01.974 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:02.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 2 16 3723 3707 115.835 96.6875 0.00587753 0.00859715 2024-09-02T10:34:02.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:34:02.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:34:02.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:34:02.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:34:02.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:34:02.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:34:02.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4016542748' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:02.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: osdmap e903: 12 total, 12 up, 12 in 2024-09-02T10:34:02.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:34:02.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:34:02.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:34:02.062 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:01 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:34:02.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:34:02.075 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4016542748' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: osdmap e903: 12 total, 12 up, 12 in 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:34:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:01 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:34:02.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:02.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:02.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:34:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:34:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:34:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:34:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:34:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:34:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4016542748' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: osdmap e903: 12 total, 12 up, 12 in 2024-09-02T10:34:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:34:02.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:34:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:34:02.258 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:01 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:34:02.265 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:02.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:02.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:02.368 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:02.397 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:02.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:02.398 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:02.499 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:02.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:02.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:02.552 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:02.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:02.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:02.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:02.697 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:34:02.757 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:02.778 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:02.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:02.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:02.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:02.902 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:02.904 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:03.005 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:03.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 3 16 5558 5542 115.45 114.688 0.00224192 0.0086453 2024-09-02T10:34:03.032 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:03.033 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:02 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:34:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:02 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:34:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:02 smithi162 ceph-mon[20279]: pgmap v2074: 47 pgs: 47 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:34:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:02 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:02 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:02 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:34:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:02 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:34:03.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:03.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:02 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:34:03.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:02 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:34:03.138 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:02 smithi151 ceph-mon[16639]: pgmap v2074: 47 pgs: 47 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:34:03.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:02 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:03.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:02 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:03.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:02 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:34:03.139 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:02 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:34:03.148 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:03.155 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:03.157 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:03.179 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:02 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:34:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:02 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:34:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:02 smithi187 ceph-mon[20072]: pgmap v2074: 47 pgs: 47 active+clean; 577 KiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:34:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:02 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:02 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:03.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:02 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:34:03.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:02 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:34:03.258 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:03.279 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:03.280 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:03.340 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:34:03.341 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:34:03.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:03.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:03.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:03.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:03.531 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:03.532 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:03.633 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:03.661 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:03.663 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:03.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:03.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:03.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:03.888 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:03.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:03.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:03.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:03.914 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:34:03.914 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:34:03.914 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:34:03.915 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:34:03.915 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:34:04.010 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:04.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 4 16 7029 7013 109.57 91.9375 0.00286921 0.00910786 2024-09-02T10:34:04.035 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:04.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:04.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:03 smithi162 ceph-mon[20279]: osdmap e904: 12 total, 12 up, 12 in 2024-09-02T10:34:04.137 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:04.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:04.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:04.163 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:04.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:03 smithi151 ceph-mon[16639]: osdmap e904: 12 total, 12 up, 12 in 2024-09-02T10:34:04.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:03 smithi187 ceph-mon[20072]: osdmap e904: 12 total, 12 up, 12 in 2024-09-02T10:34:04.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:04.287 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:04.289 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:04.389 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:04.415 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:04.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:04.517 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:04.546 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:04.547 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:04.565 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:34:04.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:04.668 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:04.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:04.744 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2075,"stamp":"2024-09-02T10:34:02.813254+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":154927648,"num_objects":2357,"num_object_clones":0,"num_object_copies":7071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":150720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5207,"ondisk_log_size":5207,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":8312328,"kb_used_data":341276,"kb_used_omap":18,"kb_used_meta":7970797,"kb_avail":1116629496,"statfs":{"total":1151940427776,"available":1143428603904,"internally_reserved":0,"allocated":349466624,"data_stored":344574089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":8162096443},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":154323862,"num_objects":2355,"num_object_clones":0,"num_object_copies":7065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":-2,"num_read_kb":-2,"num_write":2352,"num_write_kb":150671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":-12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.037314"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":1218,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.800377+0000","last_change":"2024-09-02T10:32:56.291546+0000","last_active":"2024-09-02T10:34:00.800377+0000","last_peered":"2024-09-02T10:34:00.800377+0000","last_clean":"2024-09-02T10:34:00.800377+0000","last_became_active":"2024-09-02T10:11:51.395698+0000","last_became_peered":"2024-09-02T10:11:51.395698+0000","last_unstale":"2024-09-02T10:34:00.800377+0000","last_undegraded":"2024-09-02T10:34:00.800377+0000","last_fullsized":"2024-09-02T10:34:00.800377+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:32:56.291498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:32:56.291498+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:17.937660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":359,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-02T10:33:54.859261+0000","last_change":"2024-09-02T10:33:38.660115+0000","last_active":"2024-09-02T10:33:54.859261+0000","last_peered":"2024-09-02T10:33:54.859261+0000","last_clean":"2024-09-02T10:33:54.859261+0000","last_became_active":"2024-09-02T10:33:38.659815+0000","last_became_peered":"2024-09-02T10:33:38.659815+0000","last_unstale":"2024-09-02T10:33:54.859261+0000","last_undegraded":"2024-09-02T10:33:54.859261+0000","last_fullsized":"2024-09-02T10:33:54.859261+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:54.883293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.883293+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:33:56.970316+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1491,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-02T10:33:59.355903+0000","last_change":"2024-09-02T10:33:59.232980+0000","last_active":"2024-09-02T10:33:59.355903+0000","last_peered":"2024-09-02T10:33:59.355903+0000","last_clean":"2024-09-02T10:33:59.355903+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:33:59.355903+0000","last_undegraded":"2024-09-02T10:33:59.355903+0000","last_fullsized":"2024-09-02T10:33:59.355903+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:59.232916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:59.232916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.466149+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1467,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-02T10:33:59.355792+0000","last_change":"2024-09-02T10:33:55.174212+0000","last_active":"2024-09-02T10:33:59.355792+0000","last_peered":"2024-09-02T10:33:59.355792+0000","last_clean":"2024-09-02T10:33:59.355792+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:33:59.355792+0000","last_undegraded":"2024-09-02T10:33:59.355792+0000","last_fullsized":"2024-09-02T10:33:59.355792+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.174147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.174147+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:22.067581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":786,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-02T10:33:59.676004+0000","last_change":"2024-09-02T10:33:55.588678+0000","last_active":"2024-09-02T10:33:59.676004+0000","last_peered":"2024-09-02T10:33:59.676004+0000","last_clean":"2024-09-02T10:33:59.676004+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:33:59.676004+0000","last_undegraded":"2024-09-02T10:33:59.676004+0000","last_fullsized":"2024-09-02T10:33:59.676004+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.588605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.588605+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:57.688518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":1331,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.796828+0000","last_change":"2024-09-02T10:33:56.473614+0000","last_active":"2024-09-02T10:34:00.796828+0000","last_peered":"2024-09-02T10:34:00.796828+0000","last_clean":"2024-09-02T10:34:00.796828+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:34:00.796828+0000","last_undegraded":"2024-09-02T10:34:00.796828+0000","last_fullsized":"2024-09-02T10:34:00.796828+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.473555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.473555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:04.434869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1850,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.787961+0000","last_change":"2024-09-02T10:34:00.430544+0000","last_active":"2024-09-02T10:34:00.787961+0000","last_peered":"2024-09-02T10:34:00.787961+0000","last_clean":"2024-09-02T10:34:00.787961+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:34:00.787961+0000","last_undegraded":"2024-09-02T10:34:00.787961+0000","last_fullsized":"2024-09-02T10:34:00.787961+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.430459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.430459+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:19.434576+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1491,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-02T10:33:59.355936+0000","last_change":"2024-09-02T10:33:56.194537+0000","last_active":"2024-09-02T10:33:59.355936+0000","last_peered":"2024-09-02T10:33:59.355936+0000","last_clean":"2024-09-02T10:33:59.355936+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:33:59.355936+0000","last_undegraded":"2024-09-02T10:33:59.355936+0000","last_fullsized":"2024-09-02T10:33:59.355936+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.194480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.194480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:02.488817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":52,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:01.244965+0000","last_change":"2024-09-02T10:33:56.167103+0000","last_active":"2024-09-02T10:34:01.244965+0000","last_peered":"2024-09-02T10:34:01.244965+0000","last_clean":"2024-09-02T10:34:01.244965+0000","last_became_active":"2024-09-02T10:33:38.657147+0000","last_became_peered":"2024-09-02T10:33:38.657147+0000","last_unstale":"2024-09-02T10:34:01.244965+0000","last_undegraded":"2024-09-02T10:34:01.244965+0000","last_fullsized":"2024-09-02T10:34:01.244965+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.167029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.167029+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.270011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1460,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.789666+0000","last_change":"2024-09-02T10:33:55.173514+0000","last_active":"2024-09-02T10:34:00.789666+0000","last_peered":"2024-09-02T10:34:00.789666+0000","last_clean":"2024-09-02T10:34:00.789666+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:34:00.789666+0000","last_undegraded":"2024-09-02T10:34:00.789666+0000","last_fullsized":"2024-09-02T10:34:00.789666+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.173450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.173450+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:59.387368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":698,"reported_epoch":900,"state":"active+clean","last_fresh":"2024-09-02T10:33:56.943878+0000","last_change":"2024-09-02T10:33:56.943878+0000","last_active":"2024-09-02T10:33:56.943878+0000","last_peered":"2024-09-02T10:33:56.943878+0000","last_clean":"2024-09-02T10:33:56.943878+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:33:56.943878+0000","last_undegraded":"2024-09-02T10:33:56.943878+0000","last_fullsized":"2024-09-02T10:33:56.943878+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.943826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.943826+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.719187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":519,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.796940+0000","last_change":"2024-09-02T10:33:38.657745+0000","last_active":"2024-09-02T10:34:00.796940+0000","last_peered":"2024-09-02T10:34:00.796940+0000","last_clean":"2024-09-02T10:34:00.796940+0000","last_became_active":"2024-09-02T10:33:38.657430+0000","last_became_peered":"2024-09-02T10:33:38.657430+0000","last_unstale":"2024-09-02T10:34:00.796940+0000","last_undegraded":"2024-09-02T10:34:00.796940+0000","last_fullsized":"2024-09-02T10:34:00.796940+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:48.014073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.014073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1741,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.788073+0000","last_change":"2024-09-02T10:30:23.108717+0000","last_active":"2024-09-02T10:34:00.788073+0000","last_peered":"2024-09-02T10:34:00.788073+0000","last_clean":"2024-09-02T10:34:00.788073+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:34:00.788073+0000","last_undegraded":"2024-09-02T10:34:00.788073+0000","last_fullsized":"2024-09-02T10:34:00.788073+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:23.108646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:23.108646+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":488,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:01.246043+0000","last_change":"2024-09-02T10:34:01.246043+0000","last_active":"2024-09-02T10:34:01.246043+0000","last_peered":"2024-09-02T10:34:01.246043+0000","last_clean":"2024-09-02T10:34:01.246043+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:34:01.246043+0000","last_undegraded":"2024-09-02T10:34:01.246043+0000","last_fullsized":"2024-09-02T10:34:01.246043+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.245989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.245989+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.756195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1711,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.788074+0000","last_change":"2024-09-02T10:30:16.984287+0000","last_active":"2024-09-02T10:34:00.788074+0000","last_peered":"2024-09-02T10:34:00.788074+0000","last_clean":"2024-09-02T10:34:00.788074+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:34:00.788074+0000","last_undegraded":"2024-09-02T10:34:00.788074+0000","last_fullsized":"2024-09-02T10:34:00.788074+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:16.984241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:16.984241+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"903'308","reported_seq":331,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:02.527388+0000","last_change":"2024-09-02T10:33:52.925516+0000","last_active":"2024-09-02T10:34:02.527388+0000","last_peered":"2024-09-02T10:34:02.527388+0000","last_clean":"2024-09-02T10:34:02.527388+0000","last_became_active":"2024-09-02T10:33:52.925200+0000","last_became_peered":"2024-09-02T10:33:52.925200+0000","last_unstale":"2024-09-02T10:34:02.527388+0000","last_undegraded":"2024-09-02T10:34:02.527388+0000","last_fullsized":"2024-09-02T10:34:02.527388+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:15.876694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20185088,"num_objects":308,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":308,"num_write_kb":19712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,8],"acting":[6,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1805,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.788204+0000","last_change":"2024-09-02T10:30:22.096061+0000","last_active":"2024-09-02T10:34:00.788204+0000","last_peered":"2024-09-02T10:34:00.788204+0000","last_clean":"2024-09-02T10:34:00.788204+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:34:00.788204+0000","last_undegraded":"2024-09-02T10:34:00.788204+0000","last_fullsized":"2024-09-02T10:34:00.788204+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:22.095998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:22.095998+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":19,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-02T10:33:57.531378+0000","last_change":"2024-09-02T10:33:52.750341+0000","last_active":"2024-09-02T10:33:57.531378+0000","last_peered":"2024-09-02T10:33:57.531378+0000","last_clean":"2024-09-02T10:33:57.531378+0000","last_became_active":"2024-09-02T10:33:52.750137+0000","last_became_peered":"2024-09-02T10:33:52.750137+0000","last_unstale":"2024-09-02T10:33:57.531378+0000","last_undegraded":"2024-09-02T10:33:57.531378+0000","last_fullsized":"2024-09-02T10:33:57.531378+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:58.157142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1931,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.789516+0000","last_change":"2024-09-02T10:30:24.407289+0000","last_active":"2024-09-02T10:34:00.789516+0000","last_peered":"2024-09-02T10:34:00.789516+0000","last_clean":"2024-09-02T10:34:00.789516+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:34:00.789516+0000","last_undegraded":"2024-09-02T10:34:00.789516+0000","last_fullsized":"2024-09-02T10:34:00.789516+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.407218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.407218+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":19,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-02T10:33:57.531387+0000","last_change":"2024-09-02T10:33:52.925398+0000","last_active":"2024-09-02T10:33:57.531387+0000","last_peered":"2024-09-02T10:33:57.531387+0000","last_clean":"2024-09-02T10:33:57.531387+0000","last_became_active":"2024-09-02T10:33:52.925083+0000","last_became_peered":"2024-09-02T10:33:52.925083+0000","last_unstale":"2024-09-02T10:33:57.531387+0000","last_undegraded":"2024-09-02T10:33:57.531387+0000","last_fullsized":"2024-09-02T10:33:57.531387+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.033120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1586,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-02T10:33:57.532412+0000","last_change":"2024-09-02T10:33:56.586452+0000","last_active":"2024-09-02T10:33:57.532412+0000","last_peered":"2024-09-02T10:33:57.532412+0000","last_clean":"2024-09-02T10:33:57.532412+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:33:57.532412+0000","last_undegraded":"2024-09-02T10:33:57.532412+0000","last_fullsized":"2024-09-02T10:33:57.532412+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.586392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.586392+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:20.658780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":19,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-02T10:33:57.532466+0000","last_change":"2024-09-02T10:33:52.751342+0000","last_active":"2024-09-02T10:33:57.532466+0000","last_peered":"2024-09-02T10:33:57.532466+0000","last_clean":"2024-09-02T10:33:57.532466+0000","last_became_active":"2024-09-02T10:33:52.750752+0000","last_became_peered":"2024-09-02T10:33:52.750752+0000","last_unstale":"2024-09-02T10:33:57.532466+0000","last_undegraded":"2024-09-02T10:33:57.532466+0000","last_fullsized":"2024-09-02T10:33:57.532466+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:16.573736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":559,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:01.244971+0000","last_change":"2024-09-02T10:32:46.502973+0000","last_active":"2024-09-02T10:34:01.244971+0000","last_peered":"2024-09-02T10:34:01.244971+0000","last_clean":"2024-09-02T10:34:01.244971+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:34:01.244971+0000","last_undegraded":"2024-09-02T10:34:01.244971+0000","last_fullsized":"2024-09-02T10:34:01.244971+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:32:46.502893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:32:46.502893+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:05.433959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":19,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-02T10:33:57.531482+0000","last_change":"2024-09-02T10:33:52.761107+0000","last_active":"2024-09-02T10:33:57.531482+0000","last_peered":"2024-09-02T10:33:57.531482+0000","last_clean":"2024-09-02T10:33:57.531482+0000","last_became_active":"2024-09-02T10:33:52.760804+0000","last_became_peered":"2024-09-02T10:33:52.760804+0000","last_unstale":"2024-09-02T10:33:57.531482+0000","last_undegraded":"2024-09-02T10:33:57.531482+0000","last_fullsized":"2024-09-02T10:33:57.531482+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:03.781816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1893,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.800376+0000","last_change":"2024-09-02T10:33:58.800754+0000","last_active":"2024-09-02T10:34:00.800376+0000","last_peered":"2024-09-02T10:34:00.800376+0000","last_clean":"2024-09-02T10:34:00.800376+0000","last_became_active":"2024-09-02T10:15:29.194898+0000","last_became_peered":"2024-09-02T10:15:29.194898+0000","last_unstale":"2024-09-02T10:34:00.800376+0000","last_undegraded":"2024-09-02T10:34:00.800376+0000","last_fullsized":"2024-09-02T10:34:00.800376+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:58.800694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:58.800694+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:20.393937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"9.d","version":"903'177","reported_seq":200,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:01.582980+0000","last_change":"2024-09-02T10:33:52.926793+0000","last_active":"2024-09-02T10:34:01.582980+0000","last_peered":"2024-09-02T10:34:01.582980+0000","last_clean":"2024-09-02T10:34:01.582980+0000","last_became_active":"2024-09-02T10:33:52.926467+0000","last_became_peered":"2024-09-02T10:33:52.926467+0000","last_unstale":"2024-09-02T10:34:01.582980+0000","last_undegraded":"2024-09-02T10:34:01.582980+0000","last_fullsized":"2024-09-02T10:34:01.582980+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.878242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599872,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":11328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":1346,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.796898+0000","last_change":"2024-09-02T10:30:55.890418+0000","last_active":"2024-09-02T10:34:00.796898+0000","last_peered":"2024-09-02T10:34:00.796898+0000","last_clean":"2024-09-02T10:34:00.796898+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:34:00.796898+0000","last_undegraded":"2024-09-02T10:34:00.796898+0000","last_fullsized":"2024-09-02T10:34:00.796898+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:55.890351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:55.890351+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"903'164","reported_seq":187,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:01.570858+0000","last_change":"2024-09-02T10:33:52.925479+0000","last_active":"2024-09-02T10:34:01.570858+0000","last_peered":"2024-09-02T10:34:01.570858+0000","last_clean":"2024-09-02T10:34:01.570858+0000","last_became_active":"2024-09-02T10:33:52.925170+0000","last_became_peered":"2024-09-02T10:33:52.925170+0000","last_unstale":"2024-09-02T10:34:01.570858+0000","last_undegraded":"2024-09-02T10:34:01.570858+0000","last_fullsized":"2024-09-02T10:34:01.570858+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:05.519875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10747904,"num_objects":164,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":164,"num_write_kb":10496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":1311,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.797131+0000","last_change":"2024-09-02T10:29:54.344897+0000","last_active":"2024-09-02T10:34:00.797131+0000","last_peered":"2024-09-02T10:34:00.797131+0000","last_clean":"2024-09-02T10:34:00.797131+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:34:00.797131+0000","last_undegraded":"2024-09-02T10:34:00.797131+0000","last_fullsized":"2024-09-02T10:34:00.797131+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:29:54.344678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:29:54.344678+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"903'327","reported_seq":350,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:02.645845+0000","last_change":"2024-09-02T10:33:52.925486+0000","last_active":"2024-09-02T10:34:02.645845+0000","last_peered":"2024-09-02T10:34:02.645845+0000","last_clean":"2024-09-02T10:34:02.645845+0000","last_became_active":"2024-09-02T10:33:52.925153+0000","last_became_peered":"2024-09-02T10:33:52.925153+0000","last_unstale":"2024-09-02T10:34:02.645845+0000","last_undegraded":"2024-09-02T10:34:02.645845+0000","last_fullsized":"2024-09-02T10:34:02.645845+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.337828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21430272,"num_objects":327,"num_object_clones":0,"num_object_copies":981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":327,"num_write_kb":20928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1101,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.788131+0000","last_change":"2024-09-02T10:34:00.268590+0000","last_active":"2024-09-02T10:34:00.788131+0000","last_peered":"2024-09-02T10:34:00.788131+0000","last_clean":"2024-09-02T10:34:00.788131+0000","last_became_active":"2024-09-02T10:25:02.065175+0000","last_became_peered":"2024-09-02T10:25:02.065175+0000","last_unstale":"2024-09-02T10:34:00.788131+0000","last_undegraded":"2024-09-02T10:34:00.788131+0000","last_fullsized":"2024-09-02T10:34:00.788131+0000","mapping_epoch":693,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":694,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.268512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.268512+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:28.803616+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,11,3],"acting":[6,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"903'203","reported_seq":225,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:01.757583+0000","last_change":"2024-09-02T10:33:52.792238+0000","last_active":"2024-09-02T10:34:01.757583+0000","last_peered":"2024-09-02T10:34:01.757583+0000","last_clean":"2024-09-02T10:34:01.757583+0000","last_became_active":"2024-09-02T10:33:52.791896+0000","last_became_peered":"2024-09-02T10:33:52.791896+0000","last_unstale":"2024-09-02T10:34:01.757583+0000","last_undegraded":"2024-09-02T10:34:01.757583+0000","last_fullsized":"2024-09-02T10:34:01.757583+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:56.037019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13303808,"num_objects":203,"num_object_clones":0,"num_object_copies":609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":203,"num_write_kb":12992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,11,5],"acting":[10,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":1326,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.797013+0000","last_change":"2024-09-02T10:30:48.025489+0000","last_active":"2024-09-02T10:34:00.797013+0000","last_peered":"2024-09-02T10:34:00.797013+0000","last_clean":"2024-09-02T10:34:00.797013+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:34:00.797013+0000","last_undegraded":"2024-09-02T10:34:00.797013+0000","last_fullsized":"2024-09-02T10:34:00.797013+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:48.025327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.025327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"903'316","reported_seq":338,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:02.647965+0000","last_change":"2024-09-02T10:33:52.925507+0000","last_active":"2024-09-02T10:34:02.647965+0000","last_peered":"2024-09-02T10:34:02.647965+0000","last_clean":"2024-09-02T10:34:02.647965+0000","last_became_active":"2024-09-02T10:33:52.925177+0000","last_became_peered":"2024-09-02T10:33:52.925177+0000","last_unstale":"2024-09-02T10:34:02.647965+0000","last_undegraded":"2024-09-02T10:34:02.647965+0000","last_fullsized":"2024-09-02T10:34:02.647965+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:10.535011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20709376,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":316,"num_write_kb":20224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1574,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.789665+0000","last_change":"2024-09-02T10:30:23.359450+0000","last_active":"2024-09-02T10:34:00.789665+0000","last_peered":"2024-09-02T10:34:00.789665+0000","last_clean":"2024-09-02T10:34:00.789665+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:34:00.789665+0000","last_undegraded":"2024-09-02T10:34:00.789665+0000","last_fullsized":"2024-09-02T10:34:00.789665+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:23.359405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:23.359405+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":21,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-02T10:33:59.355786+0000","last_change":"2024-09-02T10:33:52.750617+0000","last_active":"2024-09-02T10:33:59.355786+0000","last_peered":"2024-09-02T10:33:59.355786+0000","last_clean":"2024-09-02T10:33:59.355786+0000","last_became_active":"2024-09-02T10:33:52.750198+0000","last_became_peered":"2024-09-02T10:33:52.750198+0000","last_unstale":"2024-09-02T10:33:59.355786+0000","last_undegraded":"2024-09-02T10:33:59.355786+0000","last_fullsized":"2024-09-02T10:33:59.355786+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.904796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":874,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.787892+0000","last_change":"2024-09-02T10:33:59.302485+0000","last_active":"2024-09-02T10:34:00.787892+0000","last_peered":"2024-09-02T10:34:00.787892+0000","last_clean":"2024-09-02T10:34:00.787892+0000","last_became_active":"2024-09-02T10:33:38.664311+0000","last_became_peered":"2024-09-02T10:33:38.664311+0000","last_unstale":"2024-09-02T10:34:00.787892+0000","last_undegraded":"2024-09-02T10:34:00.787892+0000","last_fullsized":"2024-09-02T10:34:00.787892+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:59.302421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:59.302421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:05.057279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":19,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-02T10:33:57.532366+0000","last_change":"2024-09-02T10:33:52.753464+0000","last_active":"2024-09-02T10:33:57.532366+0000","last_peered":"2024-09-02T10:33:57.532366+0000","last_clean":"2024-09-02T10:33:57.532366+0000","last_became_active":"2024-09-02T10:33:52.753083+0000","last_became_peered":"2024-09-02T10:33:52.753083+0000","last_unstale":"2024-09-02T10:33:57.532366+0000","last_undegraded":"2024-09-02T10:33:57.532366+0000","last_fullsized":"2024-09-02T10:33:57.532366+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:55.293684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1592,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:01.291835+0000","last_change":"2024-09-02T10:34:01.291835+0000","last_active":"2024-09-02T10:34:01.291835+0000","last_peered":"2024-09-02T10:34:01.291835+0000","last_clean":"2024-09-02T10:34:01.291835+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:34:01.291835+0000","last_undegraded":"2024-09-02T10:34:01.291835+0000","last_fullsized":"2024-09-02T10:34:01.291835+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.291779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.291779+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.558074+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"903'194","reported_seq":216,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:01.753974+0000","last_change":"2024-09-02T10:33:52.753639+0000","last_active":"2024-09-02T10:34:01.753974+0000","last_peered":"2024-09-02T10:34:01.753974+0000","last_clean":"2024-09-02T10:34:01.753974+0000","last_became_active":"2024-09-02T10:33:52.753326+0000","last_became_peered":"2024-09-02T10:33:52.753326+0000","last_unstale":"2024-09-02T10:34:01.753974+0000","last_undegraded":"2024-09-02T10:34:01.753974+0000","last_fullsized":"2024-09-02T10:34:01.753974+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.852817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12713984,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":194,"num_write_kb":12416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1847,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.788222+0000","last_change":"2024-09-02T10:33:55.238766+0000","last_active":"2024-09-02T10:34:00.788222+0000","last_peered":"2024-09-02T10:34:00.788222+0000","last_clean":"2024-09-02T10:34:00.788222+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:34:00.788222+0000","last_undegraded":"2024-09-02T10:34:00.788222+0000","last_fullsized":"2024-09-02T10:34:00.788222+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.238683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.238683+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.350030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"903'353","reported_seq":375,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:02.639531+0000","last_change":"2024-09-02T10:33:52.926432+0000","last_active":"2024-09-02T10:34:02.639531+0000","last_peered":"2024-09-02T10:34:02.639531+0000","last_clean":"2024-09-02T10:34:02.639531+0000","last_became_active":"2024-09-02T10:33:52.926053+0000","last_became_peered":"2024-09-02T10:33:52.926053+0000","last_unstale":"2024-09-02T10:34:02.639531+0000","last_undegraded":"2024-09-02T10:34:02.639531+0000","last_fullsized":"2024-09-02T10:34:02.639531+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:04.035839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23134208,"num_objects":353,"num_object_clones":0,"num_object_copies":1059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":353,"num_write_kb":22592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":786,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-02T10:33:59.676022+0000","last_change":"2024-09-02T10:32:32.161493+0000","last_active":"2024-09-02T10:33:59.676022+0000","last_peered":"2024-09-02T10:33:59.676022+0000","last_clean":"2024-09-02T10:33:59.676022+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:33:59.676022+0000","last_undegraded":"2024-09-02T10:33:59.676022+0000","last_fullsized":"2024-09-02T10:33:59.676022+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:32:32.161447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:32:32.161447+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"903'313","reported_seq":336,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:02.624910+0000","last_change":"2024-09-02T10:33:52.927094+0000","last_active":"2024-09-02T10:34:02.624910+0000","last_peered":"2024-09-02T10:34:02.624910+0000","last_clean":"2024-09-02T10:34:02.624910+0000","last_became_active":"2024-09-02T10:33:52.926709+0000","last_became_peered":"2024-09-02T10:33:52.926709+0000","last_unstale":"2024-09-02T10:34:02.624910+0000","last_undegraded":"2024-09-02T10:34:02.624910+0000","last_fullsized":"2024-09-02T10:34:02.624910+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:16.295134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20512768,"num_objects":313,"num_object_clones":0,"num_object_copies":939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":313,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":313,"num_write_kb":20032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1611,"reported_epoch":903,"state":"active+clean","last_fresh":"2024-09-02T10:34:00.789776+0000","last_change":"2024-09-02T10:30:46.402988+0000","last_active":"2024-09-02T10:34:00.789776+0000","last_peered":"2024-09-02T10:34:00.789776+0000","last_clean":"2024-09-02T10:34:00.789776+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:34:00.789776+0000","last_undegraded":"2024-09-02T10:34:00.789776+0000","last_fullsized":"2024-09-02T10:34:00.789776+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:46.402901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.402901+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":21,"reported_epoch":902,"state":"active+clean","last_fresh":"2024-09-02T10:33:59.675989+0000","last_change":"2024-09-02T10:33:52.925410+0000","last_active":"2024-09-02T10:33:59.675989+0000","last_peered":"2024-09-02T10:33:59.675989+0000","last_clean":"2024-09-02T10:33:59.675989+0000","last_became_active":"2024-09-02T10:33:52.925040+0000","last_became_peered":"2024-09-02T10:33:52.925040+0000","last_unstale":"2024-09-02T10:33:59.675989+0000","last_undegraded":"2024-09-02T10:33:59.675989+0000","last_fullsized":"2024-09-02T10:33:59.675989+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.330509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":1007,"reported_epoch":901,"state":"active+clean","last_fresh":"2024-09-02T10:33:57.531469+0000","last_change":"2024-09-02T10:33:55.336238+0000","last_active":"2024-09-02T10:33:57.531469+0000","last_peered":"2024-09-02T10:33:57.531469+0000","last_clean":"2024-09-02T10:33:57.531469+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:33:57.531469+0000","last_undegraded":"2024-09-02T10:33:57.531469+0000","last_fullsized":"2024-09-02T10:33:57.531469+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.336182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.336182+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:16.545175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":154337280,"num_objects":2355,"num_object_clones":0,"num_object_copies":7065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2355,"num_write_kb":150720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":379584512,"data_stored":379584512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2355,"ondisk_log_size":2355,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384087,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":694664,"kb_used_data":13752,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":93050488,"statfs":{"total":95995035648,"available":95283699712,"internally_reserved":0,"allocated":14082048,"data_stored":13670429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":697235908},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532613020,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":644300,"kb_used_data":53624,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93100852,"statfs":{"total":95995035648,"available":95335272448,"internally_reserved":0,"allocated":54910976,"data_stored":54499357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":604830144},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841952,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":776372,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":775294,"kb_avail":92968780,"statfs":{"total":95995035648,"available":95200030720,"internally_reserved":0,"allocated":1081344,"data_stored":678631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":793901504},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070887,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":465396,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":93279756,"statfs":{"total":95995035648,"available":95518470144,"internally_reserved":0,"allocated":1081344,"data_stored":678631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299820,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758356,"kb_used_data":59392,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":92986796,"statfs":{"total":95995035648,"available":95218479104,"internally_reserved":0,"allocated":60817408,"data_stored":60402237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":715717066},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528752,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":558664,"kb_used_data":36024,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":93186488,"statfs":{"total":95995035648,"available":95422963712,"internally_reserved":0,"allocated":36888576,"data_stored":36476957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":535165383},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724981,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":963600,"kb_used_data":170240,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":92781552,"statfs":{"total":95995035648,"available":95008309248,"internally_reserved":0,"allocated":174325760,"data_stored":173910589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953913,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":505652,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":93239500,"statfs":{"total":95995035648,"available":95477248000,"internally_reserved":0,"allocated":1081344,"data_stored":678631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150143,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768372,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":766974,"kb_avail":92976780,"statfs":{"total":95995035648,"available":95208222720,"internally_reserved":0,"allocated":1409024,"data_stored":1006311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":785381832},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614136,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93131016,"statfs":{"total":95995035648,"available":95366160384,"internally_reserved":0,"allocated":1273856,"data_stored":871704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575303,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":704180,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":93040972,"statfs":{"total":95995035648,"available":95273955328,"internally_reserved":0,"allocated":1081344,"data_stored":678631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858636,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92886516,"statfs":{"total":95995035648,"available":95115792384,"internally_reserved":0,"allocated":1433600,"data_stored":1021981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":190119936,"data_stored":190119936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":42401792,"data_stored":42401792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":67633152,"data_stored":67633152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":64946176,"data_stored":64946176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":14483456,"data_stored":14483456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:34:04.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 61 2024-09-02T10:34:04.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:04.815 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:04.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:04.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:04.917 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:04.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:04 smithi151 ceph-mon[16639]: from='client.18819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:04 smithi151 ceph-mon[16639]: pgmap v2076: 47 pgs: 47 active+clean; 191 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 507 op/s 2024-09-02T10:34:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:04 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:34:04.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:04 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:34:04.941 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:04.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:04.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:05.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 5 16 8645 8629 107.854 101 0.00204888 0.00921126 2024-09-02T10:34:05.043 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:05.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:05.066 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:05.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:04 smithi162 ceph-mon[20279]: from='client.18819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:04 smithi162 ceph-mon[20279]: pgmap v2076: 47 pgs: 47 active+clean; 191 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 507 op/s 2024-09-02T10:34:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:04 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:34:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:04 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:34:05.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:05.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:05.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:05.239 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:34:05.239 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:34:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:04 smithi187 ceph-mon[20072]: from='client.18819 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:04 smithi187 ceph-mon[20072]: pgmap v2076: 47 pgs: 47 active+clean; 191 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 32 MiB/s wr, 507 op/s 2024-09-02T10:34:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:04 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:34:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:04 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:34:05.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:34:05.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:34:05.292 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:34:05.293 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:34:05.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:05.322 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:34:05.322 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:34:05.324 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:05.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:05.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:05.354 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:34:05.354 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:34:05.376 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:34:05.376 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:34:05.401 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:34:05.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:34:05.424 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:34:05.425 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:34:05.425 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:05.450 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:34:05.450 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:34:05.453 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:05.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:05.474 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:34:05.474 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:34:05.502 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:34:05.502 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:34:05.530 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:34:05.530 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:34:05.555 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:05.585 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:05.586 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:05.687 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:05.709 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:05.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:05.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:05.811 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:05.837 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:05.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:05.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:05.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:34:05.939 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:05.963 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:05.964 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:06.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 6 16 10225 10209 106.335 98.75 0.0337144 0.00936287 2024-09-02T10:34:06.065 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:06.087 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:06.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:06.088 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:05 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:05 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:05 smithi162 ceph-mon[20279]: osdmap e905: 12 total, 12 up, 12 in 2024-09-02T10:34:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:05 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:05 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2715461421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:05 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:34:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:05 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:34:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:05 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:05 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:05 smithi151 ceph-mon[16639]: osdmap e905: 12 total, 12 up, 12 in 2024-09-02T10:34:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:05 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:05 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2715461421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:05 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:34:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:05 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:34:06.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:06.202 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:06.202 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:06.210 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:06.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:06.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:05 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:05 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:05 smithi187 ceph-mon[20072]: osdmap e905: 12 total, 12 up, 12 in 2024-09-02T10:34:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:05 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:05 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2715461421' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:06.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:05 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:34:06.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:05 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:34:06.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:06.365 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:06.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:06.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:06.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:06.493 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:06.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:06.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:06.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:06.621 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:06.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:06.723 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:06.749 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:06.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:06.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:06.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:06.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:06.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:06 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:34:06.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:06 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:34:06.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:06 smithi151 ceph-mon[16639]: pgmap v2078: 47 pgs: 47 active+clean; 346 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 919 op/s 2024-09-02T10:34:06.913 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:06 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:06.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:06 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:06.914 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:06 smithi151 ceph-mon[16639]: osdmap e906: 12 total, 12 up, 12 in 2024-09-02T10:34:06.975 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:06.997 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:06.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:06.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:07.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 7 16 11579 11563 103.233 84.625 0.00250224 0.00966878 2024-09-02T10:34:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:06 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:34:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:06 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:34:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:06 smithi162 ceph-mon[20279]: pgmap v2078: 47 pgs: 47 active+clean; 346 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 919 op/s 2024-09-02T10:34:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:06 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:06 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:06 smithi162 ceph-mon[20279]: osdmap e906: 12 total, 12 up, 12 in 2024-09-02T10:34:07.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:07.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:07.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:07.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:07.248 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:07.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:06 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:34:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:06 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:34:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:06 smithi187 ceph-mon[20072]: pgmap v2078: 47 pgs: 47 active+clean; 346 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 57 MiB/s wr, 919 op/s 2024-09-02T10:34:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:06 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:06 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:06 smithi187 ceph-mon[20072]: osdmap e906: 12 total, 12 up, 12 in 2024-09-02T10:34:07.350 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:07.375 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:07.376 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:07.413 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pgp_num 61 > pg_num 31 2024-09-02T10:34:07.477 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:07.499 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:07.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:07.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:07.601 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:07.625 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:07.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:07.727 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:07.751 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:07.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:07.853 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:07.877 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:07.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:07.932 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:07.978 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:07.999 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:08.000 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:08.016 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 8 16 13062 13046 101.913 92.6875 0.118054 0.00964321 2024-09-02T10:34:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3468471067' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:34:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:07 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:34:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:07 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:07 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:08.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:08.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3468471067' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:34:08.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:07 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:34:08.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:07 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:08.103 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:07 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:08.132 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:08.133 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:08.235 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:08.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3468471067' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:34:08.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:07 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "61"} : dispatch 2024-09-02T10:34:08.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:07 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:08.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:07 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:08.261 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:08.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:08.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:08.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:08.391 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:08.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:08.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:08.511 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:34:08.511 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:34:08.519 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:08.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:08.520 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:08.621 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:08.650 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:08.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:34:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:08.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:08.752 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:08.782 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:08.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:08.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:08.854 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:34:08.883 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:08.906 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:08.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:08.907 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:08 smithi151 ceph-mon[16639]: pgmap v2080: 47 pgs: 47 active+clean; 498 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-02T10:34:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:08 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:34:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:08 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:34:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:08 smithi151 ceph-mon[16639]: osdmap e907: 12 total, 12 up, 12 in 2024-09-02T10:34:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3399064549' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:34:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:08 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:34:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:08 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:34:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:08 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:34:08.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:08 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:34:09.007 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:09.017 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 9 16 14863 14847 103.095 112.562 0.00164141 0.00967811 2024-09-02T10:34:09.030 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:09.031 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:08 smithi162 ceph-mon[20279]: pgmap v2080: 47 pgs: 47 active+clean; 498 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-02T10:34:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:08 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:34:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:08 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:34:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:08 smithi162 ceph-mon[20279]: osdmap e907: 12 total, 12 up, 12 in 2024-09-02T10:34:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3399064549' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:34:09.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:08 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:34:09.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:08 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:34:09.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:08 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:34:09.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:08 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:34:09.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:09.160 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:09.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:08 smithi187 ceph-mon[20072]: pgmap v2080: 47 pgs: 47 active+clean; 498 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 83 MiB/s wr, 1.33k op/s 2024-09-02T10:34:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:08 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:34:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:08 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:34:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:08 smithi187 ceph-mon[20072]: osdmap e907: 12 total, 12 up, 12 in 2024-09-02T10:34:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3399064549' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:34:09.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:08 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:34:09.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:08 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:34:09.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:08 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:34:09.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:08 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:34:09.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:09.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:09.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:09.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:09.376 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:34:09.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:09.406 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:09.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:09.508 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:09.532 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:09.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:09.634 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:09.662 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:09.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:09.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:09.764 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:09.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:09.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:09.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:09.889 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:09.913 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:09.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:09.988 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:34:10.015 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:10.017 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 10 16 15964 15948 99.6666 68.8125 0.00174199 0.00998566 2024-09-02T10:34:10.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:10.037 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:10.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:09 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:34:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:09 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:34:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:09 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3399064549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:34:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:09 smithi162 ceph-mon[20279]: osdmap e908: 12 total, 12 up, 12 in 2024-09-02T10:34:10.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:10.160 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:10.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:09 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:34:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:09 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:34:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:09 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3399064549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:34:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:09 smithi151 ceph-mon[16639]: osdmap e908: 12 total, 12 up, 12 in 2024-09-02T10:34:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:09 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:34:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:09 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:34:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:09 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3399064549' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:34:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:09 smithi187 ceph-mon[20072]: osdmap e908: 12 total, 12 up, 12 in 2024-09-02T10:34:10.262 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:10.289 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:10.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:10.391 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:10.420 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:10.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:10.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:10.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:10.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:10.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:10.679 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:10.680 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:10.781 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:10.807 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:10.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:10.808 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:10.831 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:10.838 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:10.908 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:10.937 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:10.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:10.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:11.017 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 11 16 17896 17880 101.582 120.75 0.0306406 0.00979217 2024-09-02T10:34:11.039 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:11.061 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:11.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:10 smithi162 ceph-mon[20279]: pgmap v2083: 47 pgs: 47 active+clean; 770 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-02T10:34:11.163 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:11.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:10 smithi151 ceph-mon[16639]: pgmap v2083: 47 pgs: 47 active+clean; 770 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-02T10:34:11.193 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:11.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:11.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:10 smithi187 ceph-mon[20072]: pgmap v2083: 47 pgs: 47 active+clean; 770 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 89 MiB/s wr, 1.42k op/s 2024-09-02T10:34:11.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:11.323 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:11.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:11.324 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:11.425 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:11.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:11.450 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:11.551 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:11.578 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:11.579 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:11.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:11.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:11.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:11.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:11.834 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:11.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:11.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:11.962 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:11.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:12.017 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 12 16 19525 19509 101.6 101.812 0.00248375 0.00983549 2024-09-02T10:34:12.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:12.087 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:12.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:12.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3560803266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:12.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:12.213 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:12.214 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3560803266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:12.315 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:12.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3560803266' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:12.347 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:12.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:12.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:12.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:12.471 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:12.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:12.528 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:34:12.528 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:34:12.554 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:34:12.554 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:34:12.574 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:12.582 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:34:12.582 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:34:12.598 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:12.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:12.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:12.604 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:34:12.605 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:34:12.627 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:34:12.627 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:34:12.653 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:34:12.653 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:34:12.680 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:34:12.681 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:34:12.700 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:12.723 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:12.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:12.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:34:12.802 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:34:12.803 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:34:12.823 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:20988 9.2 9:4a4eab46:::benchmark_data_smithi187_154927_object20987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:12.799797+0000", 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.021993718999999998, 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022023924, 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:12.824 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 20988 2024-09-02T10:34:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.799797+0000", 2024-09-02T10:34:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.825 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.799797+0000", 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.799798+0000", 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7000000000000003e-07 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.799846+0000", 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8548999999999999e-05 2024-09-02T10:34:12.826 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.799857+0000", 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.083e-05 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.799862+0000", 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9420000000000003e-06 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.827 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.799902+0000", 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9913000000000002e-05 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.800006+0000", 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010410499999999999 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.800094+0000", 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7445e-05 2024-09-02T10:34:12.828 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.801977+0000", 2024-09-02T10:34:12.829 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018832950000000001 2024-09-02T10:34:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.804989+0000", 2024-09-02T10:34:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003012256 2024-09-02T10:34:12.830 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:20994 9.1 9:872a6a72:::benchmark_data_smithi187_154927_object20993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:12.805473+0000", 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.016317636999999999, 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016423635999999998, 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:12.831 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 20994 2024-09-02T10:34:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.832 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.805473+0000", 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.805473+0000", 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.805474+0000", 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.296e-06 2024-09-02T10:34:12.833 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.805518+0000", 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4292999999999998e-05 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.805533+0000", 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4849e-05 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:12.834 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.805541+0000", 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7910000000000005e-06 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.805602+0000", 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0454000000000001e-05 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.805774+0000", 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000172619 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.835 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 6,9", 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.805920+0000", 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000146162 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.806875+0000", 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00095455900000000001 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.809808+0000", 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002932961 2024-09-02T10:34:12.836 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:21001 9.2 9:4716a5d8:::benchmark_data_smithi187_154927_object21000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:12.810213+0000", 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.011577639000000001, 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011754628, 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:12.837 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 21001 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.810213+0000", 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.810213+0000", 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:12.838 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.810214+0000", 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.142e-06 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.810245+0000", 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0686999999999999e-05 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.810256+0000", 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1214e-05 2024-09-02T10:34:12.839 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.810263+0000", 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3000000000000004e-06 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.810326+0000", 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2315000000000002e-05 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:12.840 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.810491+0000", 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000165706 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.810655+0000", 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000164093 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.812729+0000", 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0020732680000000001 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.841 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.813244+0000", 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000515155 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:21004 9.2 9:419f4e1d:::benchmark_data_smithi187_154927_object21003:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e908)", 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:12.811477+0000", 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.010313703, 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010561612, 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:12.842 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 21004 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.811477+0000", 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.811477+0000", 2024-09-02T10:34:12.843 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.811478+0000", 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3399999999999998e-07 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.811560+0000", 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2764999999999998e-05 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:12.844 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.811572+0000", 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1960000000000001e-05 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.811580+0000", 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2969999999999998e-06 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.811630+0000", 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0594999999999999e-05 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.845 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.811778+0000", 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014812500000000001 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.811934+0000", 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015512699999999999 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.813146+0000", 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00121207 2024-09-02T10:34:12.846 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:12.814972+0000", 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0018268620000000001 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 4 2024-09-02T10:34:12.847 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:34:12.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:12.848 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:34:12.849 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:34:12.868 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:34:12.868 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:34:12.874 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:12.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:12.895 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:34:12.895 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:34:12.921 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:34:12.922 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:34:12.976 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:13.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:13.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:13.001 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:13.017 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 13 16 21301 21285 102.323 111 0.0101799 0.00975891 2024-09-02T10:34:13.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:12 smithi151 ceph-mon[16639]: pgmap v2084: 47 pgs: 47 active+clean; 822 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-02T10:34:13.103 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:13.128 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:13.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:13.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:12 smithi187 ceph-mon[20072]: pgmap v2084: 47 pgs: 47 active+clean; 822 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-02T10:34:13.261 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:13.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:13.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:13.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:12 smithi162 ceph-mon[20279]: pgmap v2084: 47 pgs: 47 active+clean; 822 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-02T10:34:13.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:13.391 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:13.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:13.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:13.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:13.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:13.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:13.623 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:13.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:13.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:13.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:13.778 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:13.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:13.881 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:13.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:13.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:13.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:14.004 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:14.017 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 14 16 23012 22996 102.652 106.938 0.00160317 0.00970204 2024-09-02T10:34:14.027 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:14.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:14.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:13 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:14.112 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:13 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:14.129 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:14.152 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:14.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:14.254 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:13 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:14.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:13 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:14.278 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:14.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:13 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:14.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:13 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:14.380 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:14.407 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:14.408 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:14.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:14.539 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:14.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:14.602 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:34:14.641 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:14.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:14.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:14.671 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:14.729 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:14.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:14.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:14.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:14.899 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:14.924 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:14.925 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:14.990 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:34:14.990 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:34:14.990 INFO:tasks.thrashosds.thrasher:Removing osd 11, in_osds are: [0, 1, 3, 5, 6, 7, 8, 9, 11, 10, 4, 2] 2024-09-02T10:34:14.990 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd out 11 2024-09-02T10:34:15.017 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 15 16 24774 24758 103.149 110.125 0.00745864 0.00968612 2024-09-02T10:34:15.025 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:15.049 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:15.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:15.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:15.172 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:15.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:15.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:14 smithi151 ceph-mon[16639]: pgmap v2085: 47 pgs: 47 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-02T10:34:15.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:14 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:15.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:14 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:15.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:14 smithi151 ceph-mon[16639]: osdmap e909: 12 total, 12 up, 12 in 2024-09-02T10:34:15.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:14 smithi151 ceph-mon[16639]: osdmap e910: 12 total, 12 up, 12 in 2024-09-02T10:34:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:14 smithi187 ceph-mon[20072]: pgmap v2085: 47 pgs: 47 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-02T10:34:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:14 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:14 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:14 smithi187 ceph-mon[20072]: osdmap e909: 12 total, 12 up, 12 in 2024-09-02T10:34:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:14 smithi187 ceph-mon[20072]: osdmap e910: 12 total, 12 up, 12 in 2024-09-02T10:34:15.274 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:15.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:15.304 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:14 smithi162 ceph-mon[20279]: pgmap v2085: 47 pgs: 47 active+clean; 1.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-02T10:34:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:14 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:14 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:14 smithi162 ceph-mon[20279]: osdmap e909: 12 total, 12 up, 12 in 2024-09-02T10:34:15.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:14 smithi162 ceph-mon[20279]: osdmap e910: 12 total, 12 up, 12 in 2024-09-02T10:34:15.382 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:15.403 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:15.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:15.439 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:15.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:15.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:15.562 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:15.563 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:15.664 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:15.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:15.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:15.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:15.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:15.821 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:15.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:15.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:15.870 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:34:15.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:15.950 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:15.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:15.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:34:16.017 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 16 16 26046 26030 101.671 79.5 0.0214065 0.00981467 2024-09-02T10:34:16.052 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:16.077 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:16.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:16.179 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2328843823' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:16.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:15 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:16.204 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:16.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2328843823' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:15 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:16.306 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:16.332 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:16.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:16.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2328843823' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:16.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:15 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:16.434 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:16.460 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:16.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:16.461 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:16.562 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:16.588 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:16.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:16.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:16.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:16.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:16.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:16.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:16.840 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:16.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:16.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:16.868 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.11. 2024-09-02T10:34:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:16 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:34:16.679+0000 7fe3ffed4640 -1 mon.a@0(leader).osd e911 definitely_dead 0 2024-09-02T10:34:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:16 smithi151 ceph-mon[16639]: pgmap v2088: 47 pgs: 47 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-02T10:34:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:16 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:16 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:16 smithi151 ceph-mon[16639]: osdmap e911: 12 total, 12 up, 12 in 2024-09-02T10:34:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1860134753' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-09-02T10:34:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:16 smithi151 ceph-mon[16639]: Client client.admin marked osd.11 out, while it was still marked up 2024-09-02T10:34:16.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:16.964 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:16.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:16.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:17.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 17 16 27161 27145 99.7891 69.6875 0.00316946 0.0100043 2024-09-02T10:34:17.065 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:17.087 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:17.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:17.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:17.209 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:17.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:17.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:16 smithi187 ceph-mon[20072]: pgmap v2088: 47 pgs: 47 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-02T10:34:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:16 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:16 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:16 smithi187 ceph-mon[20072]: osdmap e911: 12 total, 12 up, 12 in 2024-09-02T10:34:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1860134753' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-09-02T10:34:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:16 smithi187 ceph-mon[20072]: Client client.admin marked osd.11 out, while it was still marked up 2024-09-02T10:34:17.311 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:17.332 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:17.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:16 smithi162 ceph-mon[20279]: pgmap v2088: 47 pgs: 47 active+clean; 1.3 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-02T10:34:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:16 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:16 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:16 smithi162 ceph-mon[20279]: osdmap e911: 12 total, 12 up, 12 in 2024-09-02T10:34:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1860134753' entity='client.admin' cmd={"prefix": "osd out", "ids": ["11"]} : dispatch 2024-09-02T10:34:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:16 smithi162 ceph-mon[20279]: Client client.admin marked osd.11 out, while it was still marked up 2024-09-02T10:34:17.435 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:17.460 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:17.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:17.562 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:17.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:17.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:17.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:17.711 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:17.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:17.813 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:17.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:17.839 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:17.839 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:17.901 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:34:17.901 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:34:17.901 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:34:17.901 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:34:17.901 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:34:17.939 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:17.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:17.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:17.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:18.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 18 16 27232 27216 94.4914 4.4375 0.00437868 0.0101583 2024-09-02T10:34:18.062 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:18.089 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:18.090 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1860134753' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-09-02T10:34:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:17 smithi151 ceph-mon[16639]: osdmap e912: 12 total, 12 up, 11 in 2024-09-02T10:34:18.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:18.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:18.217 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1860134753' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-09-02T10:34:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:17 smithi187 ceph-mon[20072]: osdmap e912: 12 total, 12 up, 11 in 2024-09-02T10:34:18.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:18.340 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:18.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:18.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:18.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1860134753' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["11"]}]': finished 2024-09-02T10:34:18.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:17 smithi162 ceph-mon[20279]: osdmap e912: 12 total, 12 up, 11 in 2024-09-02T10:34:18.443 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:18.465 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:18.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:18.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:18.589 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:18.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:18.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:18.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:18.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:18.717 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:18.818 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:18.843 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:18.844 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:18.945 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:18.968 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:18.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:18.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:19.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 19 16 27275 27259 89.6594 2.6875 0.00202959 0.0101857 2024-09-02T10:34:19.069 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:19.091 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:19.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:19.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:19.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:18 smithi151 ceph-mon[16639]: pgmap v2091: 47 pgs: 47 active+clean; 1.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-02T10:34:19.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:18 smithi151 ceph-mon[16639]: osdmap e913: 12 total, 12 up, 11 in 2024-09-02T10:34:19.093 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:18 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:34:19.130 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:34:19.193 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:19.223 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:19.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:18 smithi187 ceph-mon[20072]: pgmap v2091: 47 pgs: 47 active+clean; 1.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-02T10:34:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:18 smithi187 ceph-mon[20072]: osdmap e913: 12 total, 12 up, 11 in 2024-09-02T10:34:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:18 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:34:19.325 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:18 smithi162 ceph-mon[20279]: pgmap v2091: 47 pgs: 47 active+clean; 1.4 GiB data, 11 GiB used, 972 GiB / 983 GiB avail; 71 MiB/s wr, 1.14k op/s 2024-09-02T10:34:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:18 smithi162 ceph-mon[20279]: osdmap e913: 12 total, 12 up, 11 in 2024-09-02T10:34:19.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:18 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:34:19.346 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:19.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:19.448 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:19.477 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:19.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:19.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:19.607 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:19.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:19.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:19.709 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:19.733 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:19.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:19.835 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:19.858 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:19.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:19.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:19.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:19.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:19.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:20.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:34:20.017938+0000 min lat: 0.00099454 max lat: 2.15641 avg lat: 0.0111878 2024-09-02T10:34:20.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:34:20.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 20 16 28604 28588 89.329 83.0625 0.0032066 0.0111878 2024-09-02T10:34:20.084 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:20.084 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:20.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:20.109 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:20.110 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:20.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:19 smithi151 ceph-mon[16639]: osdmap e914: 12 total, 12 up, 11 in 2024-09-02T10:34:20.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:20.234 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:20.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:20.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:19 smithi187 ceph-mon[20072]: osdmap e914: 12 total, 12 up, 11 in 2024-09-02T10:34:20.336 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:20.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:19 smithi162 ceph-mon[20279]: osdmap e914: 12 total, 12 up, 11 in 2024-09-02T10:34:20.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:20.366 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:20.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:20.496 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:20.498 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:20.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:20.629 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:20.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:20.730 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:20.757 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:20.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:20.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:20.860 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:34:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:34:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:34:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:34:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:135860 7.1 e740/717)", 2024-09-02T10:34:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.726925+0000", 2024-09-02T10:34:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 459.12999055900002, 2024-09-02T10:34:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.362811098, 2024-09-02T10:34:20.860 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726925+0000", 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726925+0000", 2024-09-02T10:34:20.861 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726926+0000", 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8400000000000004e-07 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.726931+0000", 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.6339999999999998e-06 2024-09-02T10:34:20.862 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872187+0000", 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.145255784 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872192+0000", 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7990000000000001e-06 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.863 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.089719+0000", 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.217526953 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.089736+0000", 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7244e-05 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135866 7.2 7.308b4c22 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:34:20.864 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.733884+0000", 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 459.12303110900001, 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38588288599999998, 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135866 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.865 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733884+0000", 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733884+0000", 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.866 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733886+0000", 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7269999999999999e-06 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733891+0000", 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8690000000000003e-06 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733902+0000", 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1139e-05 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.867 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.733910+0000", 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5820000000000004e-06 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.119723+0000", 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38581336799999999 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.119767+0000", 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4200999999999999e-05 2024-09-02T10:34:20.868 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:135869 7.12 7:4e2fbeef:::benchmark_data_smithi187_120712_object51927:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:41.872032+0000", 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 458.98488338800001, 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.95676479699999994, 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.869 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135869 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872032+0000", 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872032+0000", 2024-09-02T10:34:20.870 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872034+0000", 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9099999999999999e-06 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872043+0000", 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6319999999999997e-06 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.871 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872054+0000", 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1352000000000001e-05 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:41.872063+0000", 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1470000000000007e-06 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.120273+0000", 2024-09-02T10:34:20.872 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24820998499999999 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.120282+0000", 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7499999999999992e-06 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.820947+0000", 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70066504299999999 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.873 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821053+0000", 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010563100000000001 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.821156+0000", 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010351199999999999 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.823544+0000", 2024-09-02T10:34:20.874 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002387672 2024-09-02T10:34:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.823920+0000", 2024-09-02T10:34:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00037607899999999997 2024-09-02T10:34:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.828702+0000", 2024-09-02T10:34:20.875 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047823479999999996 2024-09-02T10:34:20.876 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.876 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.828757+0000", 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5010999999999999e-05 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:42.828797+0000", 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9725000000000003e-05 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.877 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:136913 7.10 7.335c64d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.758150+0000", 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 457.09876545899999, 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47337745399999998, 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136913 2024-09-02T10:34:20.878 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758150+0000", 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758150+0000", 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.879 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758152+0000", 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0839999999999999e-06 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758158+0000", 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8889999999999999e-06 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758173+0000", 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4831000000000001e-05 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.880 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758181+0000", 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4120000000000001e-06 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758256+0000", 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4867e-05 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758265+0000", 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1710000000000008e-06 2024-09-02T10:34:20.881 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231504+0000", 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.47323864300000001 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231528+0000", 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3557000000000002e-05 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.882 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:136914 7.3 7:c30a193a:::benchmark_data_smithi187_120712_object52972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.758512+0000", 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 457.09840308899999, 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33439487099999998, 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136914 2024-09-02T10:34:20.883 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758512+0000", 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758512+0000", 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.884 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758513+0000", 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0299999999999998e-07 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758517+0000", 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6799999999999999e-06 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758527+0000", 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0404999999999999e-05 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.885 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758534+0000", 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3729999999999998e-06 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758589+0000", 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5315999999999997e-05 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758687+0000", 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7879000000000003e-05 2024-09-02T10:34:20.886 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,4", 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758807+0000", 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119839 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.760731+0000", 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001924413 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.771256+0000", 2024-09-02T10:34:20.887 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010524939000000001 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092834+0000", 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.321577844 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092874+0000", 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0209000000000001e-05 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092907+0000", 2024-09-02T10:34:20.888 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3271e-05 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:136923 7.10 7.4a3ead10 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.891481+0000", 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 456.96543419, 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34014060299999999, 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.889 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 136923 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891481+0000", 2024-09-02T10:34:20.890 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891481+0000", 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891483+0000", 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2899999999999999e-06 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.891 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891489+0000", 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8830000000000003e-06 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891502+0000", 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3086e-05 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891508+0000", 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9889999999999997e-06 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.892 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891569+0000", 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1042000000000003e-05 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.891577+0000", 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1289999999999996e-06 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231610+0000", 2024-09-02T10:34:20.893 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34003330300000001 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.231622+0000", 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1881e-05 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:154719 7.6 e749/741 7:65814d4b:::benchmark_data_smithi187_120712_object70777:head v 749'9238, mlcod=749'9235)", 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:34:20.894 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 422.95229286400001, 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26617824499999998, 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.895 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904623+0000", 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3999999999999997e-07 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:17.904629+0000", 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5419999999999996e-06 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.896 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.164896+0000", 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26026724899999998 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.164907+0000", 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0862999999999999e-05 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.165442+0000", 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053543600000000003 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.897 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.165465+0000", 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2442e-05 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.170745+0000", 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0052807360000000003 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:18.170801+0000", 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5637000000000003e-05 2024-09-02T10:34:20.898 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:109466 8.0 8:09d4259d:::benchmark_data_smithi187_136059_object4185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e836)", 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:30:34.224731+0000", 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 226.63218468299999, 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.26449957800000001, 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:34:20.899 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 109466 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224731+0000", 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224731+0000", 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.900 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224732+0000", 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4300000000000001e-06 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224738+0000", 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9689999999999999e-06 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.901 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224750+0000", 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1479e-05 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224756+0000", 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1999999999999999e-06 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224821+0000", 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5056999999999997e-05 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.902 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474358+0000", 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.249537286 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,9", 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474480+0000", 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012162799999999999 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.480187+0000", 2024-09-02T10:34:20.903 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.005706994 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.480196+0000", 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5079999999999994e-06 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.489130+0000", 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0089338339999999999 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.904 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.489180+0000", 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0167999999999998e-05 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.489230+0000", 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0025000000000003e-05 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.905 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27559.0:109464 8.c e836/792 8:3bd17a8d:::benchmark_data_smithi187_136059_object4183:head v 836'6827, mlcod=836'6825)", 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:30:34.224895+0000", 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 226.632020299, 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25820176, 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224895+0000", 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.906 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224895+0000", 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224896+0000", 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3600000000000002e-07 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.224903+0000", 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.708e-06 2024-09-02T10:34:20.907 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474294+0000", 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.249391523 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474313+0000", 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8998999999999999e-05 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.908 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474382+0000", 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8394999999999998e-05 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.474406+0000", 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.4085000000000001e-05 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.483074+0000", 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0086678429999999997 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.909 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:30:34.483097+0000", 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3371e-05 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:149577 8.0 8:0861d807:::benchmark_data_smithi187_136059_object44296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e851)", 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:27.296602+0000", 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 173.56031311800001, 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.23824097699999999, 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.910 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 149577 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.296602+0000", 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.911 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.296602+0000", 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.296604+0000", 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.024e-06 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.296613+0000", 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7700000000000007e-06 2024-09-02T10:34:20.912 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.296643+0000", 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0286000000000001e-05 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.296654+0000", 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0304e-05 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.913 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.296729+0000", 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4955000000000003e-05 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.296852+0000", 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000122812 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,9", 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.296961+0000", 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109518 2024-09-02T10:34:20.914 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.301670+0000", 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0047090949999999999 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.301769+0000", 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9283000000000005e-05 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.915 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.301954+0000", 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000184594 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.301989+0000", 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4684000000000002e-05 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:27.534843+0000", 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.232854652 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.916 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:183134 8.8 8:1b5d0daa:::benchmark_data_smithi187_136059_object77853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:32:29.342000+0000", 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 111.514915092, 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29184906700000002, 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 183134 2024-09-02T10:34:20.917 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.342000+0000", 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.342000+0000", 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.918 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.342002+0000", 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.581e-06 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.342007+0000", 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4480000000000002e-06 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.342018+0000", 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0725e-05 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.919 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.342025+0000", 2024-09-02T10:34:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6800000000000004e-06 2024-09-02T10:34:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.920 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.343175+0000", 2024-09-02T10:34:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011505280000000001 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for rw locks", 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.343258+0000", 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3049999999999999e-05 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.599267+0000", 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25600869900000001 2024-09-02T10:34:20.922 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.599352+0000", 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4553000000000002e-05 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 8,9,10", 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.599456+0000", 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000104446 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.625634+0000", 2024-09-02T10:34:20.923 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.026177385000000001 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.627739+0000", 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021050460000000002 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.627840+0000", 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101922 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.924 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.631884+0000", 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0040438280000000002 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.631945+0000", 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0726000000000001e-05 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.633849+0000", 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00190445 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.925 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(8.8 869/854 [PushReplyOp(8:1b5d0daa:::benchmark_data_smithi187_136059_object77853:head)])", 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:32:29.345797+0000", 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 111.51111872600001, 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25343984400000003, 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.926 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.345797+0000", 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.345797+0000", 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.345798+0000", 2024-09-02T10:34:20.927 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0900000000000001e-07 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.345802+0000", 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9019999999999998e-06 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.595644+0000", 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.24984173200000001 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.928 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.595651+0000", 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4339999999999997e-06 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.595892+0000", 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024151500000000001 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:32:29.599237+0000", 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0033445520000000002 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.929 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:25303 9.c 9:3757a8e5:::benchmark_data_smithi187_154927_object25302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e910)", 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:15.428520+0000", 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 5.4283952869999998, 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21200388000000001, 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.930 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 25303 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.428520+0000", 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.428520+0000", 2024-09-02T10:34:20.931 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.428522+0000", 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.894e-06 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.428572+0000", 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0118999999999999e-05 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.932 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.428592+0000", 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0072e-05 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.428601+0000", 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1719999999999996e-06 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.428684+0000", 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2501000000000002e-05 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.933 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.428870+0000", 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000186064 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.429043+0000", 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000172528 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.430817+0000", 2024-09-02T10:34:20.934 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00177392 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.430834+0000", 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7859e-05 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.640453+0000", 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20961904200000001 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.935 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.640492+0000", 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8705999999999998e-05 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:15.640524+0000", 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.2002999999999998e-05 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:26213 9.2 9:49d4197e:::benchmark_data_smithi187_154927_object26212:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e911)", 2024-09-02T10:34:20.936 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:16.107684+0000", 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7492313639999999, 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21261698000000001, 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 26213 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.937 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.107684+0000", 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.107684+0000", 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.107695+0000", 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0892e-05 2024-09-02T10:34:20.938 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.107756+0000", 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0924999999999997e-05 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.107769+0000", 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2815e-05 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.107776+0000", 2024-09-02T10:34:20.939 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8839999999999997e-06 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.107823+0000", 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7105999999999999e-05 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.107940+0000", 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117518 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.940 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.108080+0000", 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000139927 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.109999+0000", 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0019189400000000001 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.111623+0000", 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0016240219999999999 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.941 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.320206+0000", 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20858293 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.320248+0000", 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2179999999999999e-05 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.942 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.320301+0000", 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2840999999999998e-05 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:26217 9.2 9:4387bf95:::benchmark_data_smithi187_154927_object26216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e911)", 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:16.108741+0000", 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7481745159999997, 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21165233999999999, 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.943 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 26217 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.108741+0000", 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.944 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.108741+0000", 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.108742+0000", 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4300000000000002e-07 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.108830+0000", 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8096999999999994e-05 2024-09-02T10:34:20.945 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.108841+0000", 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0667e-05 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.108848+0000", 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9099999999999999e-06 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.946 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.109012+0000", 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016460300000000001 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.109132+0000", 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000119774 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.109264+0000", 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013224299999999999 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.947 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110864+0000", 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0015995359999999999 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.112323+0000", 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0014591230000000001 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.320332+0000", 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.208009264 2024-09-02T10:34:20.948 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.320362+0000", 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0258999999999999e-05 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.320393+0000", 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1021e-05 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.949 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:26219 9.2 9:47563893:::benchmark_data_smithi187_154927_object26218:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e911)", 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:16.109749+0000", 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7471666890000002, 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.212291747, 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 26219 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.950 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.109749+0000", 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.109749+0000", 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.951 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.109750+0000", 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5499999999999996e-07 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.109802+0000", 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1785e-05 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.109814+0000", 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2497000000000001e-05 2024-09-02T10:34:20.952 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.109821+0000", 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6850000000000004e-06 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110032+0000", 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00021145100000000001 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.953 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110161+0000", 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000128529 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110278+0000", 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000116804 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.111594+0000", 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001316308 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.954 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.113318+0000", 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001724389 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.321939+0000", 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.208620746 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.321999+0000", 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0328999999999998e-05 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.955 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.322041+0000", 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1269000000000001e-05 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:26222 9.2 9:48183ea8:::benchmark_data_smithi187_154927_object26221:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e911)", 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:16.110839+0000", 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.746076199, 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21129371899999999, 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.956 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 26222 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110839+0000", 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.957 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110839+0000", 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110840+0000", 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.052e-06 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110898+0000", 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7448000000000003e-05 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.958 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110910+0000", 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2252999999999999e-05 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110918+0000", 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4959999999999999e-06 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110970+0000", 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2070999999999997e-05 2024-09-02T10:34:20.959 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.111075+0000", 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010532200000000001 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.111148+0000", 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2884000000000002e-05 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.960 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.112336+0000", 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001188152 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.113515+0000", 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011788949999999999 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.322081+0000", 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20856635100000001 2024-09-02T10:34:20.961 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.322112+0000", 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0477e-05 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.322133+0000", 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1318000000000002e-05 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.962 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:26223 9.2 9:4ae3e6d5:::benchmark_data_smithi187_154927_object26222:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e911)", 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:16.110935+0000", 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.7459803449999995, 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.21128299, 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 26223 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.963 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110935+0000", 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110935+0000", 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110936+0000", 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.89e-07 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.964 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110970+0000", 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4285000000000001e-05 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110990+0000", 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0228999999999999e-05 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.110996+0000", 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8499999999999999e-06 2024-09-02T10:34:20.965 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.111646+0000", 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00065031900000000001 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.111740+0000", 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3425999999999996e-05 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.111818+0000", 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8193000000000002e-05 2024-09-02T10:34:20.966 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.112443+0000", 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00062468300000000001 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.115220+0000", 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0027769600000000002 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.322166+0000", 2024-09-02T10:34:20.967 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20694646999999999 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.322195+0000", 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9258e-05 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.322218+0000", 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.2827999999999999e-05 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.968 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:27062 9.c 9:324a8f0c:::benchmark_data_smithi187_154927_object27061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e911)", 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:16.768784+0000", 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 4.0881313270000001, 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.214441719, 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 27062 2024-09-02T10:34:20.969 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.768784+0000", 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.768784+0000", 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.970 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.768786+0000", 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8700000000000001e-06 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.768839+0000", 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3375999999999999e-05 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.768865+0000", 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.563e-05 2024-09-02T10:34:20.971 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.768880+0000", 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.468e-05 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.768951+0000", 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.1390000000000006e-05 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.972 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.769046+0000", 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4810999999999997e-05 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.769130+0000", 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3899000000000004e-05 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.769986+0000", 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00085634199999999995 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.973 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.774637+0000", 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0046511629999999998 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.983158+0000", 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.20852042700000001 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.983196+0000", 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8152000000000003e-05 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.974 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:16.983226+0000", 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9978999999999999e-05 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:27181 9.c 9:376430b6:::benchmark_data_smithi187_154927_object27180:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:34:17.167399+0000", 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 3.689516561, 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.297789201, 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:34:20.975 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 27181 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.167399+0000", 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.976 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.167399+0000", 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.167400+0000", 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.235e-06 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.167431+0000", 2024-09-02T10:34:20.977 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0510000000000001e-05 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.167443+0000", 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2143999999999999e-05 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.167448+0000", 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1919999999999996e-06 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.978 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.167494+0000", 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.5627000000000002e-05 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.167589+0000", 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5122000000000002e-05 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.167676+0000", 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7559999999999995e-05 2024-09-02T10:34:20.979 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.168523+0000", 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00084688899999999998 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.169054+0000", 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00053039000000000005 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.465105+0000", 2024-09-02T10:34:20.980 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29605119000000002 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.465142+0000", 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6746999999999999e-05 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:34:17.465188+0000", 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6594999999999997e-05 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:20.981 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:20.982 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:34:20.982 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:20.982 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:21.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 21 16 30079 30063 89.4647 92.1875 0.0128883 0.0111706 2024-09-02T10:34:21.084 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:21.107 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:20 smithi151 ceph-mon[16639]: pgmap v2094: 47 pgs: 1 peering, 46 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-02T10:34:21.109 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:21.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:21.110 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:21.137 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:34:21.163 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:34:21.163 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:34:21.188 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:34:21.188 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:34:21.211 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:21.215 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:34:21.216 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:34:21.239 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:21.240 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:21.242 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:34:21.242 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:34:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:20 smithi187 ceph-mon[20072]: pgmap v2094: 47 pgs: 1 peering, 46 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-02T10:34:21.270 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:34:21.270 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:34:21.297 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:34:21.298 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:34:21.326 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:34:21.327 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:34:21.341 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:21.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:20 smithi162 ceph-mon[20279]: pgmap v2094: 47 pgs: 1 peering, 46 active+clean; 1.6 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 86 MiB/s wr, 1.38k op/s 2024-09-02T10:34:21.353 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:34:21.353 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:34:21.368 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:21.370 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:21.384 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:34:21.384 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:34:21.411 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:34:21.411 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:34:21.438 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:34:21.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:34:21.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:21.501 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:21.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:21.570 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:34:21.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:21.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:21.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:21.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:21.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:21.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:21.779 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:21.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:21.909 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:21.910 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:22.011 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:22.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 22 16 32085 32069 91.0965 125.375 0.00444141 0.0109713 2024-09-02T10:34:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:22.042 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:22.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:22.142 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:22.157 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/447408998' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:22.174 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:22.175 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:22.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/447408998' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:22.275 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:22.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:22.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:22.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/447408998' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:22.393 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:34:22.405 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:22.462 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:22.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:22.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:22.564 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:22.589 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:22.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:22.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:22.705 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:22.714 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:22.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:22.816 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:22.844 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:22.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:22.845 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:22.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:22.972 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:22.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:23.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 23 16 33642 33626 91.3664 97.3125 0.00910577 0.0109356 2024-09-02T10:34:23.074 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:22 smithi162 ceph-mon[20279]: pgmap v2095: 47 pgs: 1 peering, 46 active+clean; 1.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 357 B/s rd, 66 MiB/s wr, 1.06k op/s; 2.1 KiB/s, 0 objects/s recovering 2024-09-02T10:34:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:23.099 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:23.099 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:23.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:22 smithi151 ceph-mon[16639]: pgmap v2095: 47 pgs: 1 peering, 46 active+clean; 1.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 357 B/s rd, 66 MiB/s wr, 1.06k op/s; 2.1 KiB/s, 0 objects/s recovering 2024-09-02T10:34:23.173 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:23.199 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:23.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:23.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:23.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:22 smithi187 ceph-mon[20072]: pgmap v2095: 47 pgs: 1 peering, 46 active+clean; 1.7 GiB data, 12 GiB used, 971 GiB / 983 GiB avail; 357 B/s rd, 66 MiB/s wr, 1.06k op/s; 2.1 KiB/s, 0 objects/s recovering 2024-09-02T10:34:23.327 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:23.356 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:23.357 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:23.458 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:23.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:23.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:23.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:23.587 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:23.610 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:23.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:23.619 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:34:23.619 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:34:23.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:23.735 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:23.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:23.736 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:23.837 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:23.859 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:23.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:23.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:23.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:23.990 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:23.991 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:24.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 24 16 34756 34740 90.4603 69.625 0.0163555 0.0110485 2024-09-02T10:34:24.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:24.120 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:24.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:24.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:24.255 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:24.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:24.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:34:24.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:24.385 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:24.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:24.386 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:24.487 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:24.517 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:24.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:24.618 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:24.644 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:24.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:24.655 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:24.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:24.772 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:24.773 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:24.875 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:24.895 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:34:24.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:24.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:25.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:25.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 25 16 36231 36215 90.5291 92.1875 0.00574974 0.01104 2024-09-02T10:34:25.025 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:25.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:25.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:24 smithi162 ceph-mon[20279]: pgmap v2096: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 1.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 304 B/s rd, 72 MiB/s wr, 1.15k op/s; 1449/93030 objects misplaced (1.558%); 2.4 MiB/s, 39 objects/s recovering 2024-09-02T10:34:25.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:24 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:34:25.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3145122456' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:25.064 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:24 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:25.102 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 10, 4, 2] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:34:25.103 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:34:25.103 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-02T10:34:25.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:34:25.127 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:25.148 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:25.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:25.149 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:25.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:34:25.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:24 smithi151 ceph-mon[16639]: pgmap v2096: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 1.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 304 B/s rd, 72 MiB/s wr, 1.15k op/s; 1449/93030 objects misplaced (1.558%); 2.4 MiB/s, 39 objects/s recovering 2024-09-02T10:34:25.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:24 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:34:25.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3145122456' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:25.183 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:24 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:25.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:24 smithi187 ceph-mon[20072]: pgmap v2096: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 1.9 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 304 B/s rd, 72 MiB/s wr, 1.15k op/s; 1449/93030 objects misplaced (1.558%); 2.4 MiB/s, 39 objects/s recovering 2024-09-02T10:34:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:24 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:34:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3145122456' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:25.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:24 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:25.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:25.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:25.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:25.395 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:25.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:25.497 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:25.518 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:25.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:25.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:25.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:25.651 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:25.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:25.682 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:34:25.682 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:34:25.753 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:25.782 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:25.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:25.844 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:25.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:25.914 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:25.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:25.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:25 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:25.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:25 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:34:25.976 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:25 smithi151 ceph-mon[16639]: osdmap e915: 12 total, 12 up, 11 in 2024-09-02T10:34:26.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:26.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 26 16 37338 37322 89.708 69.1875 0.00788375 0.0111413 2024-09-02T10:34:26.041 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:26.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:25 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:25 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:34:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:25 smithi162 ceph-mon[20279]: osdmap e915: 12 total, 12 up, 11 in 2024-09-02T10:34:26.142 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:26.165 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:26.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:25 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:25 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:34:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:25 smithi187 ceph-mon[20072]: osdmap e915: 12 total, 12 up, 11 in 2024-09-02T10:34:26.267 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:26.290 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:26.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:26.393 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:26.414 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:26.415 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:26.516 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:26.540 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:26.541 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:26.547 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:26.642 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:26.672 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:26.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:26.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:26.673 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:34:26.673 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:34:26.673 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 915, 2024-09-02T10:34:26.673 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:34:26.673 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:34:24.889565+0000", 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:34:16.863567+0000", 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5804032, 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "nodeep-scrub", 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:34:26.674 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 9, 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.675 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:34:26.676 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:34:26.677 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "264", 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "241", 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:34:26.678 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:34:26.679 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:34:26.680 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 1.7699999809265137, 2024-09-02T10:34:26.681 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 1.7699999809265137, 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 1.7699999809265137, 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 1.7699999809265137, 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 9, 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_7", 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:33:50.981540+0000", 2024-09-02T10:34:26.682 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:34:26.683 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 16, 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 16, 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 16, 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 16, 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 16, 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.684 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "901", 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "0", 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:34:26.685 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.686 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:34:26.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 2.7400000095367432, 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.7400000095367432, 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 0.81999999284744263, 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 2.25, 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.25, 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.688 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 891, 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.689 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.691 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.691 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.691 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.691 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.691 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.692 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:34:26.694 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.695 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0, 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 889, 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:34:26.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:34:26.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:34:26.698 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:34:26.699 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.700 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.701 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 891, 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:34:26.702 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:34:26.703 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.704 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.705 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 897, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.707 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.708 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.709 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.710 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:34:26.712 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 897, 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.713 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.714 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.715 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:34:26.717 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.718 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 897, 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:34:26.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:34:26.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:34:26.722 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.723 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:34:26.724 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 912, 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:34:26.725 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:34:26.726 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:34:26.727 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:34:26.728 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.728 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.728 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.728 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.728 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.728 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:34:26.728 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:34:26.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.729 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.729 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:34:26.729 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:34:26.729 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.729 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:34:26.730 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 897, 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.731 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.733 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.734 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:34:26.735 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.736 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 912, 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:34:26.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:34:26.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:34:26.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:34:26.740 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.741 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.742 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 897, 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:34:26.743 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:34:26.744 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.745 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:34:26.746 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 913, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.748 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.749 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.752 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:34:26.753 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 792, 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.754 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.755 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.756 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.757 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:34:26.758 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.759 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:34:07.746669+0000", 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:33:52.639268+0000", 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:34:12.976183+0000", 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.761 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:34:04.654446+0000", 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.762 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:33:10.271184+0000", 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:33:24.143936+0000", 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.763 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:34:02.032903+0000", 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.764 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:33:38.437819+0000", 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:34:05.090504+0000", 2024-09-02T10:34:26.765 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:33:04.887338+0000", 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:34:26.766 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:34:05.756700+0000", 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:34:26.767 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 13958, 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:34:13.625500+0000", 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.e", 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: 4, 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: 2 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.1e", 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: 11, 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: 9 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [ 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:34:26.769 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "9.9", 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: 5 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:34:26.770 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:34:26.771 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:34:26.772 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:34:26.773 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:34:26.773 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:34:26.774 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:26.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:26.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:26.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:34:26.902 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:26.929 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:26.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:26.930 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:26 smithi151 ceph-mon[16639]: pgmap v2098: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 2.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 265 B/s rd, 66 MiB/s wr, 1.06k op/s; 1449/100362 objects misplaced (1.444%); 2.1 MiB/s, 33 objects/s recovering 2024-09-02T10:34:26.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1611706473' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:34:27.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 27 16 38441 38425 88.9386 68.9375 0.00452037 0.0112358 2024-09-02T10:34:27.031 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:27.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:27.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:27.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:26 smithi162 ceph-mon[20279]: pgmap v2098: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 2.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 265 B/s rd, 66 MiB/s wr, 1.06k op/s; 1449/100362 objects misplaced (1.444%); 2.1 MiB/s, 33 objects/s recovering 2024-09-02T10:34:27.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1611706473' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:34:27.137 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:27.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:27.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:27.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:26 smithi187 ceph-mon[20072]: pgmap v2098: 47 pgs: 1 active+clean+scrubbing, 1 active+recovering+undersized+remapped, 45 active+clean; 2.0 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 265 B/s rd, 66 MiB/s wr, 1.06k op/s; 1449/100362 objects misplaced (1.444%); 2.1 MiB/s, 33 objects/s recovering 2024-09-02T10:34:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1611706473' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:34:27.281 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:27.303 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:27.304 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:27.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:27.428 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:27.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:27.430 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:27.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:34:27.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:34:27.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:34:27.437 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:34:27.437 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:34:27.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:34:27.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:34:27.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:34:27.453 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:34:27.453 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:34:27.476 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:34:27.477 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:34:27.513 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:34:27.513 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:34:27.531 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:27.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:34:27.539 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:34:27.554 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:27.555 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:27.566 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:34:27.566 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:34:27.589 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:34:27.590 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:34:27.613 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:34:27.613 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:34:27.637 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:34:27.637 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:34:27.651 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:34:27.651 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:34:27.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27220.0:71627 7.8 e711/661 7:158e78ca:::benchmark_data_smithi187_120712_object71626:head v 711'4451, mlcod=711'4448)", 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:27.818601+0000", 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 539.83034225500001, 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48232343500000002, 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:27.818601+0000", 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:27.818601+0000", 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:27.818602+0000", 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1939999999999999e-06 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:27.818645+0000", 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2200000000000003e-05 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.298400+0000", 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.47975562399999999 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.298408+0000", 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.43e-06 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.298467+0000", 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8974000000000002e-05 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.298483+0000", 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6563999999999999e-05 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.300875+0000", 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002391649 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.300925+0000", 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9799999999999998e-05 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.656 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(7.1 711/694 [PushReplyOp(7:8ab75d99:::benchmark_data_smithi187_120712_object29750:head)])", 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:27.919550+0000", 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 539.72939337900004, 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.244038844, 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:27.919550+0000", 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:27.919550+0000", 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:27.919551+0000", 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7700000000000004e-07 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:27.919557+0000", 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5250000000000002e-06 2024-09-02T10:34:27.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.162258+0000", 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.242700834 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.162266+0000", 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4680000000000003e-06 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.162320+0000", 2024-09-02T10:34:27.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3940000000000002e-05 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.163589+0000", 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012694 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPushReply(7.1 711/694 [PushReplyOp(7:8815082a:::benchmark_data_smithi187_120712_object29835:head)])", 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:28.381714+0000", 2024-09-02T10:34:27.660 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 539.26722940699995, 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25300843699999997, 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.381714+0000", 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.381714+0000", 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.381715+0000", 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2500000000000004e-07 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.381720+0000", 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.656e-06 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.633297+0000", 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25157748200000002 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.633305+0000", 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1259999999999998e-06 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.633354+0000", 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8977000000000002e-05 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:28.634722+0000", 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001368371 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:136921 7.1 7:85320a40:::benchmark_data_smithi187_120712_object52979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:43.767219+0000", 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 463.88172472799999, 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32672289999999998, 2024-09-02T10:34:27.664 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136921 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.767219+0000", 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.767219+0000", 2024-09-02T10:34:27.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.767221+0000", 2024-09-02T10:34:27.666 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:27.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.951e-06 2024-09-02T10:34:27.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.767235+0000", 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4205e-05 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.767253+0000", 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7740999999999998e-05 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.767261+0000", 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7469999999999994e-06 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.767327+0000", 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5909999999999997e-05 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.767419+0000", 2024-09-02T10:34:27.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1797000000000002e-05 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,10", 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.767496+0000", 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6699000000000005e-05 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.769252+0000", 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001756638 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:27.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.769520+0000", 2024-09-02T10:34:27.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00026790800000000002 2024-09-02T10:34:27.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:27.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.093868+0000", 2024-09-02T10:34:27.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32434744900000001 2024-09-02T10:34:27.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.093906+0000", 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8612999999999998e-05 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.093942+0000", 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5241999999999999e-05 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:136921 7.1 e743/741 ondisk, result = 0)", 2024-09-02T10:34:27.672 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:43.769492+0000", 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 463.87945158600002, 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32445480900000001, 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.769492+0000", 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.769492+0000", 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.769492+0000", 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4099999999999999e-07 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.769495+0000", 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7659999999999999e-06 2024-09-02T10:34:27.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.093784+0000", 2024-09-02T10:34:27.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.32428902799999998 2024-09-02T10:34:27.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.093792+0000", 2024-09-02T10:34:27.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1489999999999994e-06 2024-09-02T10:34:27.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.093854+0000", 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2024000000000001e-05 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.093862+0000", 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8320000000000006e-06 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.093947+0000", 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4468999999999994e-05 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:145346 7.1 7:81df0754:::benchmark_data_smithi187_120712_object61404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:27:01.791085+0000", 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 445.85785873200001, 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.235686331, 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.677 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 145346 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.791085+0000", 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.791085+0000", 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.791086+0000", 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.669e-06 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.791093+0000", 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7309999999999996e-06 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.791109+0000", 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6047999999999999e-05 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.791117+0000", 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5580000000000003e-06 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.791173+0000", 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6081000000000001e-05 2024-09-02T10:34:27.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.791279+0000", 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010597199999999999 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,10", 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.791407+0000", 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012847499999999999 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:27.681 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.793285+0000", 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018779370000000001 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.798117+0000", 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004831307 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:02.026681+0000", 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.228564722 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:02.026722+0000", 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0979999999999997e-05 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:02.026771+0000", 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8850999999999997e-05 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:108696 8.b e836/792 8:dc38f6ba:::benchmark_data_smithi187_136059_object3415:head v 836'6835, mlcod=836'6830)", 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:30:33.454294+0000", 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 234.19464913600001, 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14507748600000001, 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.454294+0000", 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.454294+0000", 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.454295+0000", 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2399999999999998e-07 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.454302+0000", 2024-09-02T10:34:27.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0230000000000004e-06 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.596618+0000", 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.142315786 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.596640+0000", 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2671999999999999e-05 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.596713+0000", 2024-09-02T10:34:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2206000000000006e-05 2024-09-02T10:34:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.596727+0000", 2024-09-02T10:34:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.47e-05 2024-09-02T10:34:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.599338+0000", 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026112459999999998 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.599372+0000", 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3328999999999999e-05 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.688 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:108768 8.b e836/792 8:de0ae72d:::benchmark_data_smithi187_136059_object3487:head v 836'6836, mlcod=836'6832)", 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:30:33.517455+0000", 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 234.13148852399999, 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15458195399999999, 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.517455+0000", 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.517455+0000", 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.517455+0000", 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3900000000000005e-07 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.517460+0000", 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5510000000000001e-06 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.669397+0000", 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15193670500000001 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.669403+0000", 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8200000000000002e-06 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.669479+0000", 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6138999999999997e-05 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.669493+0000", 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3869000000000001e-05 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.672006+0000", 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025134950000000001 2024-09-02T10:34:27.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:33.672037+0000", 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0836000000000001e-05 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:136228 8.e e845/792 8:7fea821d:::benchmark_data_smithi187_136059_object30947:head v 845'8471, mlcod=845'8468)", 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:05.969400+0000", 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 201.67954305800001, 2024-09-02T10:34:27.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17049145399999999, 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:05.969400+0000", 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:05.969400+0000", 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:05.969401+0000", 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1500000000000003e-07 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:05.969407+0000", 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8119999999999996e-06 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:06.136171+0000", 2024-09-02T10:34:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.166763776 2024-09-02T10:34:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:06.136178+0000", 2024-09-02T10:34:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6100000000000002e-06 2024-09-02T10:34:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:06.136226+0000", 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8971999999999998e-05 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:06.136242+0000", 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5911000000000001e-05 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:06.139837+0000", 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0035942639999999998 2024-09-02T10:34:27.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:06.139892+0000", 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5194e-05 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:161715 8.2 e861/792 8:4b59345d:::benchmark_data_smithi187_136059_object56434:head v 861'10081, mlcod=861'10077)", 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:50.839825+0000", 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 156.809118162, 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18231459899999999, 2024-09-02T10:34:27.698 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:50.839825+0000", 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:50.839825+0000", 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:50.839826+0000", 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7399999999999991e-07 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:50.839834+0000", 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7689999999999996e-06 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:51.017112+0000", 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.177278201 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:51.017121+0000", 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7679999999999997e-06 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:51.017184+0000", 2024-09-02T10:34:27.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3162000000000003e-05 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:51.017208+0000", 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3828e-05 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:51.022090+0000", 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004882076 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:51.022140+0000", 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9821000000000003e-05 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:183064 8.2 e869/792 8:4467006a:::benchmark_data_smithi187_136059_object77783:head v 869'11377, mlcod=869'11375)", 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:32:29.067716+0000", 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 118.58122746799999, 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.226587548, 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.703 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.067716+0000", 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.067716+0000", 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.067718+0000", 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.716e-06 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.067727+0000", 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8829999999999995e-06 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.290216+0000", 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22248953299999999 2024-09-02T10:34:27.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.290222+0000", 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8379999999999998e-06 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.290278+0000", 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5668000000000003e-05 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.290300+0000", 2024-09-02T10:34:27.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1934e-05 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.294270+0000", 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003970482 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.294304+0000", 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3494000000000002e-05 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:183137 8.2 e869/792 8:4378da73:::benchmark_data_smithi187_136059_object77856:head v 869'11381, mlcod=869'11377)", 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:32:29.352118+0000", 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 118.29682504500001, 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14894824700000001, 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.352118+0000", 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.352118+0000", 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.352119+0000", 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9100000000000002e-07 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.352126+0000", 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3319999999999997e-06 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.497290+0000", 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14516401200000001 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.497295+0000", 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9300000000000002e-06 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.497345+0000", 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0809000000000001e-05 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.497362+0000", 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6172e-05 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.501039+0000", 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036770520000000001 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.501067+0000", 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8048999999999999e-05 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:183777 8.2 e869/792 8:48dec00e:::benchmark_data_smithi187_136059_object78496:head v 869'11435, mlcod=869'11431)", 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:32:30.603219+0000", 2024-09-02T10:34:27.712 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 117.045724126, 2024-09-02T10:34:27.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15032983, 2024-09-02T10:34:27.713 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.713 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.714 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.714 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:30.603219+0000", 2024-09-02T10:34:27.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:30.603219+0000", 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:30.603221+0000", 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.581e-06 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:30.603229+0000", 2024-09-02T10:34:27.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0879999999999995e-06 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:30.749576+0000", 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.146347265 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:30.749591+0000", 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4562999999999999e-05 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:30.749697+0000", 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010635499999999999 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:30.749733+0000", 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5882e-05 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:30.753500+0000", 2024-09-02T10:34:27.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0037673939999999999 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:30.753549+0000", 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8702000000000002e-05 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:184005 8.e e870/792 8:7e46dd25:::benchmark_data_smithi187_136059_object78724:head v 870'11448, mlcod=869'11442)", 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:32:31.101215+0000", 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 116.547728089, 2024-09-02T10:34:27.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17040161200000001, 2024-09-02T10:34:27.719 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.719 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.719 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.719 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:31.101215+0000", 2024-09-02T10:34:27.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:31.101215+0000", 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:31.101216+0000", 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1999999999999998e-07 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:31.101221+0000", 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9699999999999998e-06 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:31.266899+0000", 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16567764200000001 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:31.266907+0000", 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8679999999999999e-06 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:31.266955+0000", 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8575999999999997e-05 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:31.266972+0000", 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6288000000000001e-05 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:31.271574+0000", 2024-09-02T10:34:27.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0046022770000000001 2024-09-02T10:34:27.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:31.271617+0000", 2024-09-02T10:34:27.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3170999999999997e-05 2024-09-02T10:34:27.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.723 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.723 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:2581 9.c e903/897 9:33a96762:::benchmark_data_smithi187_154927_object2580:head v 903'153, mlcod=903'150)", 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:01.309454+0000", 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 26.339488970000001, 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.166697183, 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:01.309454+0000", 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:01.309454+0000", 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:01.309455+0000", 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7100000000000011e-07 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:01.309545+0000", 2024-09-02T10:34:27.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9629000000000005e-05 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:01.475735+0000", 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16618962100000001 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:01.475740+0000", 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2020000000000003e-06 2024-09-02T10:34:27.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:01.475792+0000", 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1724000000000001e-05 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:01.475811+0000", 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8913e-05 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:01.476123+0000", 2024-09-02T10:34:27.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031264900000000002 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:01.476152+0000", 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8473999999999999e-05 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:5820 9.4 9:284d3d41:::benchmark_data_smithi187_154927_object5819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e904)", 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:03.173774+0000", 2024-09-02T10:34:27.728 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 24.475169893, 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.144817843, 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 5820 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.173774+0000", 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.173774+0000", 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.173775+0000", 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4500000000000001e-06 2024-09-02T10:34:27.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.173825+0000", 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9715000000000001e-05 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.173837+0000", 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2448e-05 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.173843+0000", 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5149999999999997e-06 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.173889+0000", 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6230999999999998e-05 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.173985+0000", 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5843999999999999e-05 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.174069+0000", 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4356000000000003e-05 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.174515+0000", 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00044547199999999998 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.174999+0000", 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00048439000000000002 2024-09-02T10:34:27.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.318428+0000", 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14342935200000001 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.318521+0000", 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2554999999999997e-05 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.318591+0000", 2024-09-02T10:34:27.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0514999999999998e-05 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:6705 9.4 9:25a06170:::benchmark_data_smithi187_154927_object6704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e904)", 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:03.709744+0000", 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 23.939199358, 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210540697, 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:27.735 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 6705 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.709744+0000", 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.709744+0000", 2024-09-02T10:34:27.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.709746+0000", 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4160000000000001e-06 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.709805+0000", 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9444000000000002e-05 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.737 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.709822+0000", 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7115000000000001e-05 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.709831+0000", 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4489999999999999e-06 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.709885+0000", 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4528999999999999e-05 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.710034+0000", 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014915800000000001 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.710177+0000", 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014256300000000001 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.710817+0000", 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00063999300000000005 2024-09-02T10:34:27.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.711361+0000", 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00054421799999999996 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.920214+0000", 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20885263300000001 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.920246+0000", 2024-09-02T10:34:27.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2057000000000002e-05 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.920285+0000", 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9122000000000003e-05 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:6705 9.4 e904/897 ondisk, result = 0)", 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:03.711324+0000", 2024-09-02T10:34:27.741 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 23.937619861999998, 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20897846, 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.711324+0000", 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.711324+0000", 2024-09-02T10:34:27.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.711324+0000", 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2600000000000002e-07 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.711327+0000", 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.948e-06 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.919460+0000", 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.208133338 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.919466+0000", 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3979999999999998e-06 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.920205+0000", 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000739498 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.920210+0000", 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.118e-06 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:03.920302+0000", 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1633999999999997e-05 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.745 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:27066 9.c e911/897 9:34842941:::benchmark_data_smithi187_154927_object27065:head v 911'1669, mlcod=911'1666)", 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:16.773972+0000", 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.87497157, 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.209032463, 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:16.773972+0000", 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:16.773972+0000", 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:16.773977+0000", 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1019999999999996e-06 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:16.774089+0000", 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011198500000000001 2024-09-02T10:34:27.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:16.980998+0000", 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20690929399999999 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:16.981004+0000", 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.212e-06 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:16.981062+0000", 2024-09-02T10:34:27.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7955999999999999e-05 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:16.981090+0000", 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7974e-05 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:16.982956+0000", 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018652709999999999 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:16.983004+0000", 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8668999999999998e-05 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:27181 9.c e912/897 9:376430b6:::benchmark_data_smithi187_154927_object27180:head v 912'1674, mlcod=912'1671)", 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:17.167805+0000", 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 10.481138614000001, 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.297139871, 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:27.750 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.167805+0000", 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.167805+0000", 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.167808+0000", 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.472e-06 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.167861+0000", 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2948000000000002e-05 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.463842+0000", 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29598026199999999 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.463857+0000", 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5288999999999999e-05 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.463948+0000", 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0987000000000001e-05 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.463964+0000", 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6189999999999999e-05 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.464914+0000", 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00095011100000000001 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.464945+0000", 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0611999999999998e-05 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:27.754 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:34:27.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:34:27.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:27.757 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:27.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:27.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:27.884 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:27.885 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:27.986 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:28.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 28 16 39461 39445 88.0388 63.75 0.00314854 0.0113531 2024-09-02T10:34:28.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:28.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:28.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:28.152 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:28.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:28.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:28.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1708377686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:28.254 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:28.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1708377686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:28.276 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:28.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1708377686' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:28.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:28.398 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:28.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:28.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:28.500 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:28.522 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:28.523 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:28.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:28.635 INFO:tasks.thrashosds.thrasher:j is {'epoch': 915, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:34:24.889565+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:34:16.863567+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7699999809265137, 'score_stable': 1.7699999809265137, 'optimal_score': 1, 'raw_score_acting': 1.7699999809265137, 'raw_score_stable': 1.7699999809265137, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-09-02T10:33:50.981540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '901', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.740000009536743, 'score_stable': 2.740000009536743, 'optimal_score': 0.8199999928474426, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 891, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 889, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 891, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 897, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 897, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 897, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 912, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 897, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 912, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 897, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 913, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:34:07.746669+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:33:52.639268+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:34:12.976183+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:34:04.654446+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:33:10.271184+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:33:24.143936+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:34:02.032903+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:33:38.437819+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:34:05.090504+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:33:04.887338+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:34:05.756700+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 13958, 'last_purged_snaps_scrub': '2024-09-02T10:34:13.625500+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.e', 'osds': [4, 9, 2]}, {'pgid': '1.1e', 'osds': [11, 10, 9]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.9', 'osds': [10, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:34:28.635 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:34:28.651 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:28.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:28.753 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:28.782 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:28.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:28.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:28.884 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:28.909 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:28.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:28.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:29.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:29.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 29 16 40483 40467 87.2054 63.875 0.00812671 0.0114596 2024-09-02T10:34:29.037 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:29.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:29.139 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:29.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:28 smithi151 ceph-mon[16639]: pgmap v2099: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 2.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 235 B/s rd, 68 MiB/s wr, 1.08k op/s; 861/106317 objects misplaced (0.810%); 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:34:29.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:28 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:29.148 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:28 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:29.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:29.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:29.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:29.195 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:28 smithi187 ceph-mon[20072]: pgmap v2099: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 2.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 235 B/s rd, 68 MiB/s wr, 1.08k op/s; 861/106317 objects misplaced (0.810%); 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:34:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:28 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:28 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:29.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:29.288 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:29.289 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:28 smithi162 ceph-mon[20279]: pgmap v2099: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 2.2 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 235 B/s rd, 68 MiB/s wr, 1.08k op/s; 861/106317 objects misplaced (0.810%); 6.1 MiB/s, 97 objects/s recovering 2024-09-02T10:34:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:28 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:28 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:29.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:29.412 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:29.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:29.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:29.514 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:29.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:29.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:29.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:29.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:29.665 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:29.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:29.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:29.792 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:29.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:29.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:29.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:29.916 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:30.018 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:30.018 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 30 16 41266 41250 85.9297 48.9375 0.00412253 0.0116181 2024-09-02T10:34:30.023 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:30.040 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:30.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:30.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:29 smithi151 ceph-mon[16639]: osdmap e916: 12 total, 12 up, 11 in 2024-09-02T10:34:30.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:29 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:34:30.133 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:29 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:34:30.134 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:34:30.135 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:34:30.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:30.167 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:30.167 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:29 smithi187 ceph-mon[20072]: osdmap e916: 12 total, 12 up, 11 in 2024-09-02T10:34:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:29 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:34:30.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:29 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:34:30.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:30.296 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:30.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:30.298 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:29 smithi162 ceph-mon[20279]: osdmap e916: 12 total, 12 up, 11 in 2024-09-02T10:34:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:29 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:34:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:29 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:34:30.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:30.419 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:30.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:30.521 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:30.543 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:30.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:30.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:30.671 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:30.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:30.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:30.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:30.797 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:30.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:30.922 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:30.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:30.923 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:31.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 31 16 42093 42077 84.8249 51.6875 0.0307401 0.0117648 2024-09-02T10:34:31.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:31.047 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:31.047 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:30 smithi162 ceph-mon[20279]: pgmap v2101: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 2.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s; 861/116088 objects misplaced (0.742%); 6.6 MiB/s, 106 objects/s recovering 2024-09-02T10:34:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:30 smithi162 ceph-mon[20279]: from='client.18885 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:31.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:30 smithi151 ceph-mon[16639]: pgmap v2101: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 2.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s; 861/116088 objects misplaced (0.742%); 6.6 MiB/s, 106 objects/s recovering 2024-09-02T10:34:31.126 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:30 smithi151 ceph-mon[16639]: from='client.18885 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:31.148 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:31.156 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2101,"stamp":"2024-09-02T10:34:29.589202+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2536453770,"num_objects":38696,"num_object_clones":0,"num_object_copies":116088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":861,"num_objects_unfound":0,"num_objects_dirty":38696,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":38697,"num_write_kb":2476465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":854,"num_bytes_recovered":56649045,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41546,"ondisk_log_size":41546,"up":141,"acting":140,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":14950764,"kb_used_data":7341348,"kb_used_omap":17,"kb_used_meta":7609198,"kb_avail":1016245908,"statfs":{"total":1055945392128,"available":1040635809792,"internally_reserved":0,"allocated":7517540352,"data_stored":7512424545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17545,"internal_metadata":7791819639},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":745406464,"num_objects":11374,"num_object_clones":0,"num_object_copies":34122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":861,"num_objects_unfound":0,"num_objects_dirty":11374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11374,"num_write_kb":727936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":850,"num_bytes_recovered":55705600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001738"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":20,"reported_epoch":914,"state":"active+clean+scrubbing","last_fresh":"2024-09-02T10:34:24.223790+0000","last_change":"2024-09-02T10:34:24.223690+0000","last_active":"2024-09-02T10:34:24.223790+0000","last_peered":"2024-09-02T10:34:24.223790+0000","last_clean":"2024-09-02T10:34:24.223790+0000","last_became_active":"2024-09-02T10:34:17.880876+0000","last_became_peered":"2024-09-02T10:34:17.880876+0000","last_unstale":"2024-09-02T10:34:24.223790+0000","last_undegraded":"2024-09-02T10:34:24.223790+0000","last_fullsized":"2024-09-02T10:34:24.223790+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:32:56.291498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:32:56.291498+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13418,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":12021,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":397,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.891922+0000","last_change":"2024-09-02T10:34:00.928724+0000","last_active":"2024-09-02T10:34:24.891922+0000","last_peered":"2024-09-02T10:34:24.891922+0000","last_clean":"2024-09-02T10:34:24.891922+0000","last_became_active":"2024-09-02T10:33:38.659815+0000","last_became_peered":"2024-09-02T10:33:38.659815+0000","last_unstale":"2024-09-02T10:34:24.891922+0000","last_undegraded":"2024-09-02T10:34:24.891922+0000","last_fullsized":"2024-09-02T10:34:24.891922+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.928661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.928661+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:26.870517+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1516,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-09-02T10:34:18.876117+0000","last_change":"2024-09-02T10:33:59.232980+0000","last_active":"2024-09-02T10:34:18.876117+0000","last_peered":"2024-09-02T10:34:18.876117+0000","last_clean":"2024-09-02T10:34:18.876117+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:34:18.876117+0000","last_undegraded":"2024-09-02T10:34:18.876117+0000","last_fullsized":"2024-09-02T10:34:18.876117+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:59.232916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:59.232916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.466149+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1492,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-09-02T10:34:18.876004+0000","last_change":"2024-09-02T10:33:55.174212+0000","last_active":"2024-09-02T10:34:18.876004+0000","last_peered":"2024-09-02T10:34:18.876004+0000","last_clean":"2024-09-02T10:34:18.876004+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:34:18.876004+0000","last_undegraded":"2024-09-02T10:34:18.876004+0000","last_fullsized":"2024-09-02T10:34:18.876004+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.174147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.174147+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:22.067581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":811,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-09-02T10:34:19.056657+0000","last_change":"2024-09-02T10:33:55.588678+0000","last_active":"2024-09-02T10:34:19.056657+0000","last_peered":"2024-09-02T10:34:19.056657+0000","last_clean":"2024-09-02T10:34:19.056657+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:34:19.056657+0000","last_undegraded":"2024-09-02T10:34:19.056657+0000","last_fullsized":"2024-09-02T10:34:19.056657+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.588605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.588605+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:57.688518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":1356,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.898784+0000","last_change":"2024-09-02T10:33:56.473614+0000","last_active":"2024-09-02T10:34:24.898784+0000","last_peered":"2024-09-02T10:34:24.898784+0000","last_clean":"2024-09-02T10:34:24.898784+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:34:24.898784+0000","last_undegraded":"2024-09-02T10:34:24.898784+0000","last_fullsized":"2024-09-02T10:34:24.898784+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.473555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.473555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:04.434869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1874,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.894531+0000","last_change":"2024-09-02T10:34:00.430544+0000","last_active":"2024-09-02T10:34:24.894531+0000","last_peered":"2024-09-02T10:34:24.894531+0000","last_clean":"2024-09-02T10:34:24.894531+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:34:24.894531+0000","last_undegraded":"2024-09-02T10:34:24.894531+0000","last_fullsized":"2024-09-02T10:34:24.894531+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.430459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.430459+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:19.434576+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1515,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-09-02T10:34:18.876153+0000","last_change":"2024-09-02T10:33:56.194537+0000","last_active":"2024-09-02T10:34:18.876153+0000","last_peered":"2024-09-02T10:34:18.876153+0000","last_clean":"2024-09-02T10:34:18.876153+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:34:18.876153+0000","last_undegraded":"2024-09-02T10:34:18.876153+0000","last_fullsized":"2024-09-02T10:34:18.876153+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.194480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.194480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:02.488817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":76,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:25.178366+0000","last_change":"2024-09-02T10:33:56.167103+0000","last_active":"2024-09-02T10:34:25.178366+0000","last_peered":"2024-09-02T10:34:25.178366+0000","last_clean":"2024-09-02T10:34:25.178366+0000","last_became_active":"2024-09-02T10:33:38.657147+0000","last_became_peered":"2024-09-02T10:33:38.657147+0000","last_unstale":"2024-09-02T10:34:25.178366+0000","last_undegraded":"2024-09-02T10:34:25.178366+0000","last_fullsized":"2024-09-02T10:34:25.178366+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.167029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.167029+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.270011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1484,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.896247+0000","last_change":"2024-09-02T10:33:55.173514+0000","last_active":"2024-09-02T10:34:24.896247+0000","last_peered":"2024-09-02T10:34:24.896247+0000","last_clean":"2024-09-02T10:34:24.896247+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:34:24.896247+0000","last_undegraded":"2024-09-02T10:34:24.896247+0000","last_fullsized":"2024-09-02T10:34:24.896247+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.173450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.173450+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:59.387368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":728,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.891913+0000","last_change":"2024-09-02T10:33:56.943878+0000","last_active":"2024-09-02T10:34:24.891913+0000","last_peered":"2024-09-02T10:34:24.891913+0000","last_clean":"2024-09-02T10:34:24.891913+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:34:24.891913+0000","last_undegraded":"2024-09-02T10:34:24.891913+0000","last_fullsized":"2024-09-02T10:34:24.891913+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.943826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.943826+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.719187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":543,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.898864+0000","last_change":"2024-09-02T10:33:38.657745+0000","last_active":"2024-09-02T10:34:24.898864+0000","last_peered":"2024-09-02T10:34:24.898864+0000","last_clean":"2024-09-02T10:34:24.898864+0000","last_became_active":"2024-09-02T10:33:38.657430+0000","last_became_peered":"2024-09-02T10:33:38.657430+0000","last_unstale":"2024-09-02T10:34:24.898864+0000","last_undegraded":"2024-09-02T10:34:24.898864+0000","last_fullsized":"2024-09-02T10:34:24.898864+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:48.014073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.014073+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1773,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.894672+0000","last_change":"2024-09-02T10:34:08.628331+0000","last_active":"2024-09-02T10:34:24.894672+0000","last_peered":"2024-09-02T10:34:24.894672+0000","last_clean":"2024-09-02T10:34:24.894672+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:34:24.894672+0000","last_undegraded":"2024-09-02T10:34:24.894672+0000","last_fullsized":"2024-09-02T10:34:24.894672+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.628247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.628247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:31.896890+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":513,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:25.178166+0000","last_change":"2024-09-02T10:34:01.246043+0000","last_active":"2024-09-02T10:34:25.178166+0000","last_peered":"2024-09-02T10:34:25.178166+0000","last_clean":"2024-09-02T10:34:25.178166+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:34:25.178166+0000","last_undegraded":"2024-09-02T10:34:25.178166+0000","last_fullsized":"2024-09-02T10:34:25.178166+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.245989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.245989+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.756195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1743,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.894590+0000","last_change":"2024-09-02T10:34:05.558912+0000","last_active":"2024-09-02T10:34:24.894590+0000","last_peered":"2024-09-02T10:34:24.894590+0000","last_clean":"2024-09-02T10:34:24.894590+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:34:24.894590+0000","last_undegraded":"2024-09-02T10:34:24.894590+0000","last_fullsized":"2024-09-02T10:34:24.894590+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:05.558852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:05.558852+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:06.139483+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"915'2518","reported_seq":2564,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:27.485010+0000","last_change":"2024-09-02T10:33:52.925516+0000","last_active":"2024-09-02T10:34:27.485010+0000","last_peered":"2024-09-02T10:34:27.485010+0000","last_clean":"2024-09-02T10:34:27.485010+0000","last_became_active":"2024-09-02T10:33:52.925200+0000","last_became_peered":"2024-09-02T10:33:52.925200+0000","last_unstale":"2024-09-02T10:34:27.485010+0000","last_undegraded":"2024-09-02T10:34:27.485010+0000","last_fullsized":"2024-09-02T10:34:27.485010+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2518,"log_dups_size":0,"ondisk_log_size":2518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:15.876694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165019648,"num_objects":2518,"num_object_clones":0,"num_object_copies":7554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2518,"num_write_kb":161152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,8],"acting":[6,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1838,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.894723+0000","last_change":"2024-09-02T10:34:07.590524+0000","last_active":"2024-09-02T10:34:24.894723+0000","last_peered":"2024-09-02T10:34:24.894723+0000","last_clean":"2024-09-02T10:34:24.894723+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:34:24.894723+0000","last_undegraded":"2024-09-02T10:34:24.894723+0000","last_fullsized":"2024-09-02T10:34:24.894723+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:07.590464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:07.590464+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:30.846352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"916'2527","reported_seq":2574,"reported_epoch":916,"state":"active+clean","last_fresh":"2024-09-02T10:34:29.271421+0000","last_change":"2024-09-02T10:33:52.750341+0000","last_active":"2024-09-02T10:34:29.271421+0000","last_peered":"2024-09-02T10:34:29.271421+0000","last_clean":"2024-09-02T10:34:29.271421+0000","last_became_active":"2024-09-02T10:33:52.750137+0000","last_became_peered":"2024-09-02T10:33:52.750137+0000","last_unstale":"2024-09-02T10:34:29.271421+0000","last_undegraded":"2024-09-02T10:34:29.271421+0000","last_fullsized":"2024-09-02T10:34:29.271421+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2527,"log_dups_size":0,"ondisk_log_size":2527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:58.157142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165609472,"num_objects":2527,"num_object_clones":0,"num_object_copies":7581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2527,"num_write_kb":161728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1956,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.896105+0000","last_change":"2024-09-02T10:30:24.407289+0000","last_active":"2024-09-02T10:34:24.896105+0000","last_peered":"2024-09-02T10:34:24.896105+0000","last_clean":"2024-09-02T10:34:24.896105+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:34:24.896105+0000","last_undegraded":"2024-09-02T10:34:24.896105+0000","last_fullsized":"2024-09-02T10:34:24.896105+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.407218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.407218+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"916'2566","reported_seq":2615,"reported_epoch":916,"state":"active+clean","last_fresh":"2024-09-02T10:34:29.210912+0000","last_change":"2024-09-02T10:33:52.925398+0000","last_active":"2024-09-02T10:34:29.210912+0000","last_peered":"2024-09-02T10:34:29.210912+0000","last_clean":"2024-09-02T10:34:29.210912+0000","last_became_active":"2024-09-02T10:33:52.925083+0000","last_became_peered":"2024-09-02T10:33:52.925083+0000","last_unstale":"2024-09-02T10:34:29.210912+0000","last_undegraded":"2024-09-02T10:34:29.210912+0000","last_fullsized":"2024-09-02T10:34:29.210912+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2566,"log_dups_size":0,"ondisk_log_size":2566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.033120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168165376,"num_objects":2566,"num_object_clones":0,"num_object_copies":7698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2566,"num_write_kb":164224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1614,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.894954+0000","last_change":"2024-09-02T10:33:56.586452+0000","last_active":"2024-09-02T10:34:24.894954+0000","last_peered":"2024-09-02T10:34:24.894954+0000","last_clean":"2024-09-02T10:34:24.894954+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:34:24.894954+0000","last_undegraded":"2024-09-02T10:34:24.894954+0000","last_fullsized":"2024-09-02T10:34:24.894954+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.586392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.586392+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:20.658780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"915'2571","reported_seq":2618,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:28.019625+0000","last_change":"2024-09-02T10:33:52.751342+0000","last_active":"2024-09-02T10:34:28.019625+0000","last_peered":"2024-09-02T10:34:28.019625+0000","last_clean":"2024-09-02T10:34:28.019625+0000","last_became_active":"2024-09-02T10:33:52.750752+0000","last_became_peered":"2024-09-02T10:33:52.750752+0000","last_unstale":"2024-09-02T10:34:28.019625+0000","last_undegraded":"2024-09-02T10:34:28.019625+0000","last_fullsized":"2024-09-02T10:34:28.019625+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2571,"log_dups_size":0,"ondisk_log_size":2571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:16.573736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168493056,"num_objects":2571,"num_object_clones":0,"num_object_copies":7713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2571,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2571,"num_write_kb":164544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":591,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:25.178340+0000","last_change":"2024-09-02T10:34:08.328616+0000","last_active":"2024-09-02T10:34:25.178340+0000","last_peered":"2024-09-02T10:34:25.178340+0000","last_clean":"2024-09-02T10:34:25.178340+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:34:25.178340+0000","last_undegraded":"2024-09-02T10:34:25.178340+0000","last_fullsized":"2024-09-02T10:34:25.178340+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.328570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.328570+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.052868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"916'2522","reported_seq":2571,"reported_epoch":916,"state":"active+clean","last_fresh":"2024-09-02T10:34:29.125042+0000","last_change":"2024-09-02T10:33:52.761107+0000","last_active":"2024-09-02T10:34:29.125042+0000","last_peered":"2024-09-02T10:34:29.125042+0000","last_clean":"2024-09-02T10:34:29.125042+0000","last_became_active":"2024-09-02T10:33:52.760804+0000","last_became_peered":"2024-09-02T10:33:52.760804+0000","last_unstale":"2024-09-02T10:34:29.125042+0000","last_undegraded":"2024-09-02T10:34:29.125042+0000","last_fullsized":"2024-09-02T10:34:29.125042+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2522,"log_dups_size":0,"ondisk_log_size":2522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:03.781816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165281792,"num_objects":2522,"num_object_clones":0,"num_object_copies":7566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2522,"num_write_kb":161408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":919,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-09-02T10:34:18.876161+0000","last_change":"2024-09-02T10:34:17.882133+0000","last_active":"2024-09-02T10:34:18.876161+0000","last_peered":"2024-09-02T10:34:18.876161+0000","last_clean":"2024-09-02T10:34:18.876161+0000","last_became_active":"2024-09-02T10:34:17.881745+0000","last_became_peered":"2024-09-02T10:34:17.881745+0000","last_unstale":"2024-09-02T10:34:18.876161+0000","last_undegraded":"2024-09-02T10:34:18.876161+0000","last_fullsized":"2024-09-02T10:34:18.876161+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:58.800694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:58.800694+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:25.599849+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"915'2304","reported_seq":2351,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:26.508613+0000","last_change":"2024-09-02T10:33:52.926793+0000","last_active":"2024-09-02T10:34:26.508613+0000","last_peered":"2024-09-02T10:34:26.508613+0000","last_clean":"2024-09-02T10:34:26.508613+0000","last_became_active":"2024-09-02T10:33:52.926467+0000","last_became_peered":"2024-09-02T10:33:52.926467+0000","last_unstale":"2024-09-02T10:34:26.508613+0000","last_undegraded":"2024-09-02T10:34:26.508613+0000","last_fullsized":"2024-09-02T10:34:26.508613+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2304,"log_dups_size":0,"ondisk_log_size":2304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.878242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150994944,"num_objects":2304,"num_object_clones":0,"num_object_copies":6912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2304,"num_write_kb":147456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":1378,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.898803+0000","last_change":"2024-09-02T10:34:08.280819+0000","last_active":"2024-09-02T10:34:24.898803+0000","last_peered":"2024-09-02T10:34:24.898803+0000","last_clean":"2024-09-02T10:34:24.898803+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:34:24.898803+0000","last_undegraded":"2024-09-02T10:34:24.898803+0000","last_fullsized":"2024-09-02T10:34:24.898803+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.280762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.280762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:19.946116+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"915'2300","reported_seq":2347,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:26.535078+0000","last_change":"2024-09-02T10:33:52.925479+0000","last_active":"2024-09-02T10:34:26.535078+0000","last_peered":"2024-09-02T10:34:26.535078+0000","last_clean":"2024-09-02T10:34:26.535078+0000","last_became_active":"2024-09-02T10:33:52.925170+0000","last_became_peered":"2024-09-02T10:33:52.925170+0000","last_unstale":"2024-09-02T10:34:26.535078+0000","last_undegraded":"2024-09-02T10:34:26.535078+0000","last_fullsized":"2024-09-02T10:34:26.535078+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2300,"log_dups_size":0,"ondisk_log_size":2300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:05.519875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150732800,"num_objects":2300,"num_object_clones":0,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2300,"num_write_kb":147200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":1343,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.899019+0000","last_change":"2024-09-02T10:34:05.282359+0000","last_active":"2024-09-02T10:34:24.899019+0000","last_peered":"2024-09-02T10:34:24.899019+0000","last_clean":"2024-09-02T10:34:24.899019+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:34:24.899019+0000","last_undegraded":"2024-09-02T10:34:24.899019+0000","last_fullsized":"2024-09-02T10:34:24.899019+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:05.282307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:05.282307+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:12.971380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"915'2430","reported_seq":2477,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:27.691764+0000","last_change":"2024-09-02T10:33:52.925486+0000","last_active":"2024-09-02T10:34:27.691764+0000","last_peered":"2024-09-02T10:34:27.691764+0000","last_clean":"2024-09-02T10:34:27.691764+0000","last_became_active":"2024-09-02T10:33:52.925153+0000","last_became_peered":"2024-09-02T10:33:52.925153+0000","last_unstale":"2024-09-02T10:34:27.691764+0000","last_undegraded":"2024-09-02T10:34:27.691764+0000","last_fullsized":"2024-09-02T10:34:27.691764+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2430,"log_dups_size":0,"ondisk_log_size":2430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.337828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159252480,"num_objects":2430,"num_object_clones":0,"num_object_copies":7290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2430,"num_write_kb":155520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1132,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.892482+0000","last_change":"2024-09-02T10:34:17.882691+0000","last_active":"2024-09-02T10:34:24.892482+0000","last_peered":"2024-09-02T10:34:24.892482+0000","last_clean":"2024-09-02T10:34:24.892482+0000","last_became_active":"2024-09-02T10:34:17.882376+0000","last_became_peered":"2024-09-02T10:34:17.882376+0000","last_unstale":"2024-09-02T10:34:24.892482+0000","last_undegraded":"2024-09-02T10:34:24.892482+0000","last_fullsized":"2024-09-02T10:34:24.892482+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.268512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.268512+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:22.220969+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"915'2361","reported_seq":4112,"reported_epoch":915,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:34:26.762425+0000","last_change":"2024-09-02T10:34:19.106251+0000","last_active":"2024-09-02T10:34:26.762425+0000","last_peered":"2024-09-02T10:34:26.762425+0000","last_clean":"2024-09-02T10:34:16.764686+0000","last_became_active":"2024-09-02T10:34:19.091896+0000","last_became_peered":"2024-09-02T10:34:19.091896+0000","last_unstale":"2024-09-02T10:34:26.762425+0000","last_undegraded":"2024-09-02T10:34:26.762425+0000","last_fullsized":"2024-09-02T10:34:18.878643+0000","mapping_epoch":913,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2361,"log_dups_size":0,"ondisk_log_size":2361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154730496,"num_objects":2361,"num_object_clones":0,"num_object_copies":7083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":861,"num_objects_unfound":0,"num_objects_dirty":2361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2361,"num_write_kb":151104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":850,"num_bytes_recovered":55705600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,3],"acting":[10,5],"avail_no_missing":["10","5","11"],"object_location_counts":[{"shards":"5,10","objects":1500},{"shards":"5,10,11","objects":861}],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":1350,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.898896+0000","last_change":"2024-09-02T10:30:48.025489+0000","last_active":"2024-09-02T10:34:24.898896+0000","last_peered":"2024-09-02T10:34:24.898896+0000","last_clean":"2024-09-02T10:34:24.898896+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:34:24.898896+0000","last_undegraded":"2024-09-02T10:34:24.898896+0000","last_fullsized":"2024-09-02T10:34:24.898896+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:48.025327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:48.025327+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"915'2489","reported_seq":2536,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:27.701793+0000","last_change":"2024-09-02T10:33:52.925507+0000","last_active":"2024-09-02T10:34:27.701793+0000","last_peered":"2024-09-02T10:34:27.701793+0000","last_clean":"2024-09-02T10:34:27.701793+0000","last_became_active":"2024-09-02T10:33:52.925177+0000","last_became_peered":"2024-09-02T10:33:52.925177+0000","last_unstale":"2024-09-02T10:34:27.701793+0000","last_undegraded":"2024-09-02T10:34:27.701793+0000","last_fullsized":"2024-09-02T10:34:27.701793+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2489,"log_dups_size":0,"ondisk_log_size":2489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:10.535011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163119104,"num_objects":2489,"num_object_clones":0,"num_object_copies":7467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2489,"num_write_kb":159296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1606,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.897292+0000","last_change":"2024-09-02T10:34:04.115873+0000","last_active":"2024-09-02T10:34:24.897292+0000","last_peered":"2024-09-02T10:34:24.897292+0000","last_clean":"2024-09-02T10:34:24.897292+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:34:24.897292+0000","last_undegraded":"2024-09-02T10:34:24.897292+0000","last_fullsized":"2024-09-02T10:34:24.897292+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:04.115830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:04.115830+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:04.432089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"914'2217","reported_seq":2260,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.607306+0000","last_change":"2024-09-02T10:33:52.750617+0000","last_active":"2024-09-02T10:34:24.607306+0000","last_peered":"2024-09-02T10:34:24.607306+0000","last_clean":"2024-09-02T10:34:24.607306+0000","last_became_active":"2024-09-02T10:33:52.750198+0000","last_became_peered":"2024-09-02T10:33:52.750198+0000","last_unstale":"2024-09-02T10:34:24.607306+0000","last_undegraded":"2024-09-02T10:34:24.607306+0000","last_fullsized":"2024-09-02T10:34:24.607306+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2217,"log_dups_size":0,"ondisk_log_size":2217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.904796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145293312,"num_objects":2217,"num_object_clones":0,"num_object_copies":6651,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2217,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2217,"num_write_kb":141888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":898,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.892483+0000","last_change":"2024-09-02T10:33:59.302485+0000","last_active":"2024-09-02T10:34:24.892483+0000","last_peered":"2024-09-02T10:34:24.892483+0000","last_clean":"2024-09-02T10:34:24.892483+0000","last_became_active":"2024-09-02T10:33:38.664311+0000","last_became_peered":"2024-09-02T10:33:38.664311+0000","last_unstale":"2024-09-02T10:34:24.892483+0000","last_undegraded":"2024-09-02T10:34:24.892483+0000","last_fullsized":"2024-09-02T10:34:24.892483+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:59.302421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:59.302421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:05.057279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"915'2505","reported_seq":2552,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:28.001679+0000","last_change":"2024-09-02T10:33:52.753464+0000","last_active":"2024-09-02T10:34:28.001679+0000","last_peered":"2024-09-02T10:34:28.001679+0000","last_clean":"2024-09-02T10:34:28.001679+0000","last_became_active":"2024-09-02T10:33:52.753083+0000","last_became_peered":"2024-09-02T10:33:52.753083+0000","last_unstale":"2024-09-02T10:34:28.001679+0000","last_undegraded":"2024-09-02T10:34:28.001679+0000","last_fullsized":"2024-09-02T10:34:28.001679+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2505,"log_dups_size":0,"ondisk_log_size":2505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:55.293684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164167680,"num_objects":2505,"num_object_clones":0,"num_object_copies":7515,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2505,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2505,"num_write_kb":160320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1616,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.896901+0000","last_change":"2024-09-02T10:34:01.291835+0000","last_active":"2024-09-02T10:34:24.896901+0000","last_peered":"2024-09-02T10:34:24.896901+0000","last_clean":"2024-09-02T10:34:24.896901+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:34:24.896901+0000","last_undegraded":"2024-09-02T10:34:24.896901+0000","last_fullsized":"2024-09-02T10:34:24.896901+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.291779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.291779+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.558074+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"915'2309","reported_seq":2356,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:26.732200+0000","last_change":"2024-09-02T10:33:52.753639+0000","last_active":"2024-09-02T10:34:26.732200+0000","last_peered":"2024-09-02T10:34:26.732200+0000","last_clean":"2024-09-02T10:34:26.732200+0000","last_became_active":"2024-09-02T10:33:52.753326+0000","last_became_peered":"2024-09-02T10:33:52.753326+0000","last_unstale":"2024-09-02T10:34:26.732200+0000","last_undegraded":"2024-09-02T10:34:26.732200+0000","last_fullsized":"2024-09-02T10:34:26.732200+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2309,"log_dups_size":0,"ondisk_log_size":2309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.852817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151322624,"num_objects":2309,"num_object_clones":0,"num_object_copies":6927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2309,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2309,"num_write_kb":147776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1872,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.892585+0000","last_change":"2024-09-02T10:33:55.238766+0000","last_active":"2024-09-02T10:34:24.892585+0000","last_peered":"2024-09-02T10:34:24.892585+0000","last_clean":"2024-09-02T10:34:24.892585+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:34:24.892585+0000","last_undegraded":"2024-09-02T10:34:24.892585+0000","last_fullsized":"2024-09-02T10:34:24.892585+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.238683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.238683+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.350030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"915'2439","reported_seq":2486,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:27.688113+0000","last_change":"2024-09-02T10:33:52.926432+0000","last_active":"2024-09-02T10:34:27.688113+0000","last_peered":"2024-09-02T10:34:27.688113+0000","last_clean":"2024-09-02T10:34:27.688113+0000","last_became_active":"2024-09-02T10:33:52.926053+0000","last_became_peered":"2024-09-02T10:33:52.926053+0000","last_unstale":"2024-09-02T10:34:27.688113+0000","last_undegraded":"2024-09-02T10:34:27.688113+0000","last_fullsized":"2024-09-02T10:34:27.688113+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2439,"log_dups_size":0,"ondisk_log_size":2439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:04.035839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2439,"num_write_kb":156096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":818,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-09-02T10:34:19.056654+0000","last_change":"2024-09-02T10:34:02.569694+0000","last_active":"2024-09-02T10:34:19.056654+0000","last_peered":"2024-09-02T10:34:19.056654+0000","last_clean":"2024-09-02T10:34:19.056654+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:34:19.056654+0000","last_undegraded":"2024-09-02T10:34:19.056654+0000","last_fullsized":"2024-09-02T10:34:19.056654+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:02.569628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:02.569628+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:27.473229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"915'2365","reported_seq":2412,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:27.707732+0000","last_change":"2024-09-02T10:33:52.927094+0000","last_active":"2024-09-02T10:34:27.707732+0000","last_peered":"2024-09-02T10:34:27.707732+0000","last_clean":"2024-09-02T10:34:27.707732+0000","last_became_active":"2024-09-02T10:33:52.926709+0000","last_became_peered":"2024-09-02T10:33:52.926709+0000","last_unstale":"2024-09-02T10:34:27.707732+0000","last_undegraded":"2024-09-02T10:34:27.707732+0000","last_fullsized":"2024-09-02T10:34:27.707732+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2365,"log_dups_size":0,"ondisk_log_size":2365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:16.295134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154992640,"num_objects":2365,"num_object_clones":0,"num_object_copies":7095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2365,"num_write_kb":151360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1636,"reported_epoch":915,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.896379+0000","last_change":"2024-09-02T10:30:46.402988+0000","last_active":"2024-09-02T10:34:24.896379+0000","last_peered":"2024-09-02T10:34:24.896379+0000","last_clean":"2024-09-02T10:34:24.896379+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:34:24.896379+0000","last_undegraded":"2024-09-02T10:34:24.896379+0000","last_fullsized":"2024-09-02T10:34:24.896379+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:46.402901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.402901+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:27.009032+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"914'2271","reported_seq":2316,"reported_epoch":914,"state":"active+clean","last_fresh":"2024-09-02T10:34:24.689562+0000","last_change":"2024-09-02T10:33:52.925410+0000","last_active":"2024-09-02T10:34:24.689562+0000","last_peered":"2024-09-02T10:34:24.689562+0000","last_clean":"2024-09-02T10:34:24.689562+0000","last_became_active":"2024-09-02T10:33:52.925040+0000","last_became_peered":"2024-09-02T10:33:52.925040+0000","last_unstale":"2024-09-02T10:34:24.689562+0000","last_undegraded":"2024-09-02T10:34:24.689562+0000","last_fullsized":"2024-09-02T10:34:24.689562+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2271,"log_dups_size":0,"ondisk_log_size":2271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.330509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148832256,"num_objects":2271,"num_object_clones":0,"num_object_copies":6813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2271,"num_write_kb":145344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":1037,"reported_epoch":916,"state":"active+clean","last_fresh":"2024-09-02T10:34:28.924566+0000","last_change":"2024-09-02T10:33:55.336238+0000","last_active":"2024-09-02T10:34:28.924566+0000","last_peered":"2024-09-02T10:34:28.924566+0000","last_clean":"2024-09-02T10:34:28.924566+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:34:28.924566+0000","last_undegraded":"2024-09-02T10:34:28.924566+0000","last_fullsized":"2024-09-02T10:34:28.924566+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.336182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.336182+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:16.545175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2535849984,"num_objects":38694,"num_object_clones":0,"num_object_copies":116082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":861,"num_objects_unfound":0,"num_objects_dirty":38694,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38694,"num_write_kb":2476416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":850,"num_bytes_recovered":55705600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7740784640,"data_stored":7740784640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38694,"ondisk_log_size":38694,"up":48,"acting":47,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":603786,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":3,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":943445,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":343597384092,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532613025,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1365212,"kb_used_data":742600,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":92379940,"statfs":{"total":95995035648,"available":94597058560,"internally_reserved":0,"allocated":760422400,"data_stored":759957886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":637532608},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841958,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1797532,"kb_used_data":940296,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":91947620,"statfs":{"total":95995035648,"available":94154362880,"internally_reserved":0,"allocated":962863104,"data_stored":962398590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":877787584},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070892,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":905228,"kb_used_data":428412,"kb_used_omap":1,"kb_used_meta":476798,"kb_avail":92839924,"statfs":{"total":95995035648,"available":95068082176,"internally_reserved":0,"allocated":438693888,"data_stored":438233827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":488241605},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299825,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1625892,"kb_used_data":890832,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":92119260,"statfs":{"total":95995035648,"available":94330122240,"internally_reserved":0,"allocated":912211968,"data_stored":911743902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":752679370},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528757,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":860184,"kb_used_data":315336,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92884968,"statfs":{"total":95995035648,"available":95114207232,"internally_reserved":0,"allocated":322904064,"data_stored":322439550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":557906375},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724986,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2170656,"kb_used_data":1367568,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":91574496,"statfs":{"total":95995035648,"available":93772283904,"internally_reserved":0,"allocated":1400389632,"data_stored":1399921566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":822344133},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953918,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":789836,"kb_used_data":285244,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92955316,"statfs":{"total":95995035648,"available":95186243584,"internally_reserved":0,"allocated":292089856,"data_stored":291629795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150149,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1756568,"kb_used_data":909064,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":91988584,"statfs":{"total":95995035648,"available":94196310016,"internally_reserved":0,"allocated":930881536,"data_stored":930417022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":867826120},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614308,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93130844,"statfs":{"total":95995035648,"available":95365984256,"internally_reserved":0,"allocated":1458176,"data_stored":990110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575309,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2206568,"kb_used_data":1459028,"kb_used_omap":1,"kb_used_meta":747518,"kb_avail":91538584,"statfs":{"total":95995035648,"available":93735510016,"internally_reserved":0,"allocated":1494044672,"data_stored":1493575755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":765458885},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858780,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92886372,"statfs":{"total":95995035648,"available":95115644928,"internally_reserved":0,"allocated":1581056,"data_stored":1116542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1500643328,"data_stored":1500643328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":962002944,"data_stored":962002944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":293339136,"data_stored":293339136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1438253056,"data_stored":1438253056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":323616768,"data_stored":323616768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":918224896,"data_stored":918224896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":446300160,"data_stored":446300160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":978911232,"data_stored":978911232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":767361024,"data_stored":767361024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":112132096,"data_stored":112132096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:34:31.158 INFO:tasks.thrashosds.thrasher:Setting 1.f to [11, 3, 9, 4, 1, 5] 2024-09-02T10:34:31.158 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.f', '11', '3', '9', '4', '1', '5'] 2024-09-02T10:34:31.158 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 1.f 11 3 9 4 1 5 2024-09-02T10:34:31.176 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:31.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:31.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:34:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:30 smithi187 ceph-mon[20072]: pgmap v2101: 47 pgs: 1 active+recovering+undersized+remapped, 1 active+clean+scrubbing, 45 active+clean; 2.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 89 MiB/s wr, 1.42k op/s; 861/116088 objects misplaced (0.742%); 6.6 MiB/s, 106 objects/s recovering 2024-09-02T10:34:31.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:30 smithi187 ceph-mon[20072]: from='client.18885 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:31.278 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:31.302 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:34:31.312 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:31.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:31.414 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:31.438 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:31.440 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:31.524 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:31.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:31.561 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:31.572 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:31.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:31.661 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:31.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:31.700 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:31.701 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:31.801 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:31.826 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:31.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:31.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:31.946 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:34:31.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/863165484' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:31.947 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:31 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:31.956 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:31.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:31.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:32.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 32 16 43091 43075 84.1231 62.375 0.0116037 0.011874 2024-09-02T10:34:32.058 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:32.084 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/863165484' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:32.084 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:31 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:32.085 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:32.086 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:32.187 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:32.213 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:32.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/863165484' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:31 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:32.268 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:34:32.268 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:34:32.268 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.268 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:43399 9.5 9:a2492d6c:::benchmark_data_smithi187_154927_object43398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-02T10:34:32.268 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:32.226500+0000", 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0058784529999999996, 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059073700000000003, 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 43399 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.269 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226500+0000", 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226500+0000", 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226501+0000", 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8100000000000001e-07 2024-09-02T10:34:32.270 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226558+0000", 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7388e-05 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226570+0000", 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1977999999999999e-05 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226577+0000", 2024-09-02T10:34:32.271 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.883e-06 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226633+0000", 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5470999999999998e-05 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226777+0000", 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000143795 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.272 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226923+0000", 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014622499999999999 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.228542+0000", 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001619475 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.228730+0000", 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00018726300000000001 2024-09-02T10:34:32.273 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:43400 9.a 9:57614ed1:::benchmark_data_smithi187_154927_object43399:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:32.226613+0000", 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0057660510000000003, 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0058764569999999999, 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:32.274 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 43400 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226613+0000", 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226613+0000", 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:32.275 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226613+0000", 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5499999999999998e-07 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226678+0000", 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5338000000000004e-05 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:32.276 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226686+0000", 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1300000000000001e-06 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226692+0000", 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4020000000000001e-06 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226752+0000", 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0229999999999998e-05 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.277 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226888+0000", 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013610799999999999 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.227024+0000", 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013599900000000001 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.229126+0000", 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002101538 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.278 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.229205+0000", 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.9142999999999998e-05 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:43403 9.6 e917/897 ondisk, result = 0)", 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:32.231366+0000", 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0010124610000000001, 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011832209999999999, 2024-09-02T10:34:32.279 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.231366+0000", 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.231366+0000", 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.280 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.231367+0000", 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2900000000000003e-07 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.231371+0000", 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9940000000000002e-06 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.232383+0000", 2024-09-02T10:34:32.281 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0010125799999999999 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.232388+0000", 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6750000000000001e-06 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.232434+0000", 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5633000000000002e-05 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:32.282 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.232443+0000", 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3360000000000001e-06 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.232549+0000", 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010637399999999999 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:43397 9.f 9:f9188eac:::benchmark_data_smithi187_154927_object43396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e917)", 2024-09-02T10:34:32.283 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:32.225514+0000", 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0068643050000000002, 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0071007199999999996, 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 43397 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.284 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.225514+0000", 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.225514+0000", 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.225516+0000", 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5510000000000001e-06 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.285 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.225608+0000", 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2459000000000004e-05 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.225623+0000", 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4789e-05 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.225631+0000", 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8679999999999999e-06 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.286 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.225730+0000", 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9448000000000001e-05 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.225887+0000", 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015659300000000001 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.226040+0000", 2024-09-02T10:34:32.287 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015254799999999999 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.227829+0000", 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001788932 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:32.227868+0000", 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9376999999999999e-05 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:32.288 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:32.289 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:32.289 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:34:32.289 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 4 2024-09-02T10:34:32.289 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:34:32.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:32.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:32.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:32.442 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:32.469 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:32.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:32.470 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:32.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:32.598 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:32.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:32.700 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:32.721 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:32.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:32.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:32.823 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:32.850 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:32.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:32.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:32.951 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:32.980 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:32.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:32.981 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:32.989 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:32 smithi151 ceph-mon[16639]: pgmap v2102: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 861/117945 objects misplaced (0.730%); 4.6 MiB/s, 73 objects/s recovering 2024-09-02T10:34:32.989 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:32 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:34:32.989 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:32 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:34:32.989 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:32 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:32.989 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:32 smithi151 ceph-mon[16639]: osdmap e917: 12 total, 12 up, 11 in 2024-09-02T10:34:33.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 33 16 44527 44511 84.2933 89.75 0.0120203 0.0118582 2024-09-02T10:34:33.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:34:33.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:32 smithi162 ceph-mon[20279]: pgmap v2102: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 861/117945 objects misplaced (0.730%); 4.6 MiB/s, 73 objects/s recovering 2024-09-02T10:34:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:32 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:34:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:32 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:34:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:32 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:32 smithi162 ceph-mon[20279]: osdmap e917: 12 total, 12 up, 11 in 2024-09-02T10:34:33.108 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:33.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:33.109 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:33.128 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:34:33.128 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:34:33.155 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:34:33.155 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:34:33.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:34:33.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:34:33.210 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:33.211 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:34:33.211 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:34:33.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:33.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:33.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:33.239 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:34:33.239 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:34:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:32 smithi187 ceph-mon[20072]: pgmap v2102: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.4 GiB data, 14 GiB used, 969 GiB / 983 GiB avail; 65 MiB/s wr, 1.04k op/s; 861/117945 objects misplaced (0.730%); 4.6 MiB/s, 73 objects/s recovering 2024-09-02T10:34:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:32 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:34:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:32 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:34:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:32 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:33.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:32 smithi187 ceph-mon[20072]: osdmap e917: 12 total, 12 up, 11 in 2024-09-02T10:34:33.266 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:34:33.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:34:33.291 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:34:33.291 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:34:33.317 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:34:33.317 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:34:33.339 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:33.346 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:34:33.346 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:34:33.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:33.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:33.373 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:34:33.373 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:34:33.402 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:34:33.402 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:34:33.410 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:33.426 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:34:33.426 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:34:33.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:33.491 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:33.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:33.593 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:33.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:33.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:33.719 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:33.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:33.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:33.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:33.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:33.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:33.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:33.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:33.975 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.f pg_upmap_items mapping to [11->3,9->4,1->5] 2024-09-02T10:34:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:33.993 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:33.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:33.997 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:34:33.997 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:34:34.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 34 16 45643 45627 83.8653 69.75 0.0623871 0.0118784 2024-09-02T10:34:34.097 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:34.122 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:34.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:34.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:34.225 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:34.246 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:34.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3788804931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:33 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/87345075' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [11, 3, 9, 4, 1, 5]} : dispatch 2024-09-02T10:34:34.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:33 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [11, 3, 9, 4, 1, 5]} : dispatch 2024-09-02T10:34:34.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3788804931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:34.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:33 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/87345075' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [11, 3, 9, 4, 1, 5]} : dispatch 2024-09-02T10:34:34.295 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:33 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [11, 3, 9, 4, 1, 5]} : dispatch 2024-09-02T10:34:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3788804931' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:33 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/87345075' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [11, 3, 9, 4, 1, 5]} : dispatch 2024-09-02T10:34:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:33 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [11, 3, 9, 4, 1, 5]} : dispatch 2024-09-02T10:34:34.348 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:34.369 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:34.370 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:34.471 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:34.492 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:34.493 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:34.595 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:34.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:34.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:34.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:34.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:34.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:34:34.740 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:34.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:34.843 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:34.865 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:34.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:34.906 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:34:34.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:34.988 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:34.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:34.997 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:35.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 35 16 45851 45835 81.8406 13 0.00247047 0.0119629 2024-09-02T10:34:35.090 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:34 smithi162 ceph-mon[20279]: pgmap v2104: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s; 263/128298 objects misplaced (0.205%); 9.3 MiB/s, 148 objects/s recovering 2024-09-02T10:34:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:34 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [11, 3, 9, 4, 1, 5]}]': finished 2024-09-02T10:34:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:34 smithi162 ceph-mon[20279]: osdmap e918: 12 total, 12 up, 11 in 2024-09-02T10:34:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:34 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:35.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:34 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:35.112 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:35.113 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:34 smithi151 ceph-mon[16639]: pgmap v2104: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s; 263/128298 objects misplaced (0.205%); 9.3 MiB/s, 148 objects/s recovering 2024-09-02T10:34:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:34 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [11, 3, 9, 4, 1, 5]}]': finished 2024-09-02T10:34:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:34 smithi151 ceph-mon[16639]: osdmap e918: 12 total, 12 up, 11 in 2024-09-02T10:34:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:34 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:34 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:35.214 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:35.246 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:35.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:35.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:34 smithi187 ceph-mon[20072]: pgmap v2104: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.6 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 73 MiB/s wr, 1.16k op/s; 263/128298 objects misplaced (0.205%); 9.3 MiB/s, 148 objects/s recovering 2024-09-02T10:34:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:34 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.f", "id": [11, 3, 9, 4, 1, 5]}]': finished 2024-09-02T10:34:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:34 smithi187 ceph-mon[20072]: osdmap e918: 12 total, 12 up, 11 in 2024-09-02T10:34:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:34 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:34 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:35.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:35.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:35.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:35.477 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:35.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:35.506 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:35.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:35.550 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:34:35.550 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:34:35.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:35.635 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:35.636 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:35.737 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:35.740 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:35.767 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:35.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:35.869 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:35.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:35.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:36.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:36.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 36 16 47537 47521 82.4941 105.375 0.00870866 0.0121167 2024-09-02T10:34:36.029 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:36.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:36.030 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:36.131 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:36.161 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:36.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:35 smithi151 ceph-mon[16639]: osdmap e919: 12 total, 12 up, 11 in 2024-09-02T10:34:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:35 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:34:36.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:35 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:34:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:35 smithi187 ceph-mon[20072]: osdmap e919: 12 total, 12 up, 11 in 2024-09-02T10:34:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:35 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:34:36.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:35 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:34:36.263 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:36.293 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:36.294 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:35 smithi162 ceph-mon[20279]: osdmap e919: 12 total, 12 up, 11 in 2024-09-02T10:34:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:35 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:34:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:35 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:34:36.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:36.416 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:36.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:36.417 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:36.498 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:34:36.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:36.574 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:36.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:36.677 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:36.704 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:36.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:36.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:36.806 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:36.830 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:36.831 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:36.859 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:36.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:36.954 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:34:36.960 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:36.961 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:37.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 37 16 48774 48758 82.3539 77.3125 0.00448804 0.0121375 2024-09-02T10:34:37.062 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:37.086 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:37.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:37.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:37.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:37.210 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:37.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:37.211 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:36 smithi187 ceph-mon[20072]: pgmap v2107: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 935 op/s; 263/132942 objects misplaced (0.198%); 6.2 MiB/s, 99 objects/s recovering 2024-09-02T10:34:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:36 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:34:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:36 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:34:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/19164855' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:34:37.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:36 smithi162 ceph-mon[20279]: pgmap v2107: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 935 op/s; 263/132942 objects misplaced (0.198%); 6.2 MiB/s, 99 objects/s recovering 2024-09-02T10:34:37.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:36 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:34:37.258 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:36 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:34:37.259 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/19164855' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:34:37.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:37.339 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:37.340 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:37.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:36 smithi151 ceph-mon[16639]: pgmap v2107: 47 pgs: 1 active+recovering+undersized+remapped, 46 active+clean; 2.7 GiB data, 15 GiB used, 968 GiB / 983 GiB avail; 58 MiB/s wr, 935 op/s; 263/132942 objects misplaced (0.198%); 6.2 MiB/s, 99 objects/s recovering 2024-09-02T10:34:37.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:36 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:34:37.360 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:36 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:34:37.361 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/19164855' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:34:37.414 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:34:37.415 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:34:37.440 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:37.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:37.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:37.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:37.590 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:37.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:37.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:37.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:34:37.720 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:37.721 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:37.822 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:37.849 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:37.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:37.850 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:37.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:37.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:37.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:37.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:38.015 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:37 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:38.015 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:37 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/19164855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:34:38.015 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:37 smithi162 ceph-mon[20279]: osdmap e920: 12 total, 12 up, 11 in 2024-09-02T10:34:38.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 38 16 50348 50332 82.7753 98.375 0.0078392 0.0120748 2024-09-02T10:34:38.026 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:38.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:34:38.081 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:38.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:38.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:38.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:37 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:38.150 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:37 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/19164855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:34:38.151 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:37 smithi151 ceph-mon[16639]: osdmap e920: 12 total, 12 up, 11 in 2024-09-02T10:34:38.239 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:37 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:37 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/19164855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:34:38.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:37 smithi187 ceph-mon[20072]: osdmap e920: 12 total, 12 up, 11 in 2024-09-02T10:34:38.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:38.264 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:38.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:38.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:38.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:38.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:38.400 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:38.490 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:38.517 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:38.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:38.518 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:38.619 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:38.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:38.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:38.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:38.768 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:38.769 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:38.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:38.894 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:38.896 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:38.978 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:34:38.978 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:34:38.996 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:39.019 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 39 16 51528 51512 82.5437 73.75 0.00707858 0.0121092 2024-09-02T10:34:39.021 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:39.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:39.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:38 smithi162 ceph-mon[20279]: pgmap v2109: 47 pgs: 47 active+clean; 2.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 60 MiB/s wr, 961 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-09-02T10:34:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2622102044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:39.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:39.150 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:39.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:39.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:39.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:38 smithi187 ceph-mon[20072]: pgmap v2109: 47 pgs: 47 active+clean; 2.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 60 MiB/s wr, 961 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-09-02T10:34:39.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2622102044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:39.280 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:39.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:39.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:39.382 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:39.408 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:39.409 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:39.422 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:34:39.423 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:34:39.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:38 smithi151 ceph-mon[16639]: pgmap v2109: 47 pgs: 47 active+clean; 2.8 GiB data, 16 GiB used, 968 GiB / 983 GiB avail; 60 MiB/s wr, 961 op/s; 2.9 MiB/s, 46 objects/s recovering 2024-09-02T10:34:39.423 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2622102044' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:39.450 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:34:39.450 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:34:39.477 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:34:39.477 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:34:39.501 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:34:39.501 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:34:39.509 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:39.529 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:34:39.529 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:34:39.539 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:39.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:39.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:39.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:34:39.558 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:34:39.589 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:34:39.589 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:34:39.615 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:34:39.616 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:34:39.641 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:39.642 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:34:39.642 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:34:39.665 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:34:39.666 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:34:39.668 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:39.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:39.691 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:34:39.691 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:34:39.717 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:34:39.718 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:34:39.769 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:39.785 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 10, 4, 2] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:34:39.785 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:34:39.785 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_7 2024-09-02T10:34:39.785 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:34:39.793 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:39.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:39.794 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:39.847 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:34:39.891 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:34:39.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:39.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:39.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:40.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:34:40.019677+0000 min lat: 0.00099454 max lat: 2.15641 avg lat: 0.0121131 2024-09-02T10:34:40.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:34:40.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 40 16 52817 52801 82.4941 80.5625 0.00920254 0.0121131 2024-09-02T10:34:40.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:40.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:40.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:40.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:40.150 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:40.174 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:40.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:40.237 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:39 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:39 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:40.275 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:40.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:40.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:39 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:40.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:39 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:40.404 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:39 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:40.404 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:39 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:40.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:40.430 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:40.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:40.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:40.532 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:40.564 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:40.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:40.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:40.693 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:40.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:40.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:40.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:40.821 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:40.822 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:40.923 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:40.942 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:40.955 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:40.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:40.969 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:34:40.969 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:34:40.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:65697 7.d 7:b3a5eaf4:::benchmark_data_smithi187_120712_object65696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:20.774076+0000", 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 560.19272366200005, 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48971618, 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:40.970 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 65697 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774076+0000", 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774076+0000", 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774077+0000", 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8950000000000001e-06 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774127+0000", 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9299999999999999e-05 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:40.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774144+0000", 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6858999999999999e-05 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774151+0000", 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9500000000000004e-06 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774205+0000", 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4392999999999999e-05 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774314+0000", 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010882699999999999 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.774413+0000", 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9253999999999995e-05 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776257+0000", 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001843896 2024-09-02T10:34:40.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776758+0000", 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000501294 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263673+0000", 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48691518299999997 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263745+0000", 2024-09-02T10:34:40.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1769999999999999e-05 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263792+0000", 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6558999999999999e-05 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:65697 7.d e706/661 ondisk, result = 0)", 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:20.776789+0000", 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 560.19001034200005, 2024-09-02T10:34:40.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48700850400000001, 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776789+0000", 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776789+0000", 2024-09-02T10:34:40.977 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776789+0000", 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3999999999999997e-07 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.776792+0000", 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.576e-06 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:40.978 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263583+0000", 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.486791272 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263589+0000", 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5779999999999998e-06 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263660+0000", 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1297000000000005e-05 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263667+0000", 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0879999999999997e-06 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.263797+0000", 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013035299999999999 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:65699 7.c 7:3d3e89d8:::benchmark_data_smithi187_120712_object65698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e706)", 2024-09-02T10:34:40.980 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:20.788356+0000", 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 560.17844285800004, 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27550548699999999, 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 65699 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:40.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788356+0000", 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788356+0000", 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788359+0000", 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.131e-06 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788410+0000", 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1156999999999998e-05 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788427+0000", 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7198999999999999e-05 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788437+0000", 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7680000000000003e-06 2024-09-02T10:34:40.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788502+0000", 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5049999999999996e-05 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788627+0000", 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012572000000000001 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.788741+0000", 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000113211 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.790598+0000", 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018573159999999999 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.792472+0000", 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001874393 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062706+0000", 2024-09-02T10:34:40.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27023351600000001 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062753+0000", 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6978000000000003e-05 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.063862+0000", 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0011090480000000001 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:40.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:65699 7.c e706/694 ondisk, result = 0)", 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:25:20.797961+0000", 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 560.16883786699998, 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26590793699999998, 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.797961+0000", 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.797961+0000", 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.797962+0000", 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8700000000000005e-07 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:40.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:20.797966+0000", 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.614e-06 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062630+0000", 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26466413300000002 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062638+0000", 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4689999999999997e-06 2024-09-02T10:34:40.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062692+0000", 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4214e-05 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.062701+0000", 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999992e-06 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:25:21.063869+0000", 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00116807 2024-09-02T10:34:40.991 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:100943 7.c 7:3c179c28:::benchmark_data_smithi187_120712_object17001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:01.695117+0000", 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.27168175199995, 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26333421299999998, 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:40.992 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 100943 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695117+0000", 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695117+0000", 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.993 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695120+0000", 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2139999999999999e-06 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695130+0000", 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8959999999999998e-06 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695159+0000", 2024-09-02T10:34:40.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9539e-05 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695174+0000", 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4867e-05 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695336+0000", 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016231700000000001 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695535+0000", 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00019830499999999999 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,11", 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.695643+0000", 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108521 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.698192+0000", 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025493569999999999 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.698560+0000", 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036743600000000001 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958382+0000", 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25982233300000002 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958419+0000", 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6749999999999999e-05 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958452+0000", 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2678000000000001e-05 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:100999 7.d 7:bd5ad45b:::benchmark_data_smithi187_120712_object17057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e723)", 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:01.958635+0000", 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.00816378900004, 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42676124199999999, 2024-09-02T10:34:40.998 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 100999 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958635+0000", 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:40.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958635+0000", 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958636+0000", 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1699999999999997e-07 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958640+0000", 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.286e-06 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958649+0000", 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9889999999999998e-06 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958653+0000", 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6439999999999999e-06 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958693+0000", 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0079e-05 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958790+0000", 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6644000000000005e-05 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,1", 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.958854+0000", 2024-09-02T10:34:41.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3767000000000001e-05 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.961222+0000", 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023684969999999998 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.961990+0000", 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000768178 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385337+0000", 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.423346471 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385370+0000", 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3349999999999997e-05 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385397+0000", 2024-09-02T10:34:41.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.652e-05 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:100999 7.d e723/661 ondisk, result = 0)", 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:01.962006+0000", 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 519.00479295399998, 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42339489600000002, 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:41.005 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.962006+0000", 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.962006+0000", 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.962007+0000", 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0200000000000002e-07 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:01.962010+0000", 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1810000000000001e-06 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385252+0000", 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.42324162900000001 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385257+0000", 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2959999999999998e-06 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385328+0000", 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0612999999999996e-05 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385333+0000", 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.933e-06 2024-09-02T10:34:41.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:02.385401+0000", 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8641999999999998e-05 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:135857 7.17 7:e979241f:::benchmark_data_smithi187_120712_object51915:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:41.733787+0000", 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 479.23301270899998, 2024-09-02T10:34:41.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0657094760000001, 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135857 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733787+0000", 2024-09-02T10:34:41.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733787+0000", 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733787+0000", 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2200000000000003e-07 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733791+0000", 2024-09-02T10:34:41.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.213e-06 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733801+0000", 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0173000000000001e-05 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733807+0000", 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7089999999999996e-06 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.882008+0000", 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14820015 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.882017+0000", 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9900000000000003e-06 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.799411+0000", 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91739405500000004 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.799496+0000", 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5364000000000005e-05 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:135859 7.17 7:eb549113:::benchmark_data_smithi187_120712_object51917:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e741)", 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:41.733821+0000", 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 479.23297785, 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.066307179, 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.014 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 135859 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733821+0000", 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733821+0000", 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733822+0000", 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4499999999999997e-07 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733824+0000", 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6139999999999999e-06 2024-09-02T10:34:41.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733831+0000", 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.827e-06 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.733837+0000", 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1940000000000003e-06 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.882038+0000", 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.14820045000000001 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:41.882043+0000", 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4079999999999997e-06 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.800039+0000", 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.91799532699999997 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:42.800129+0000", 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9913999999999993e-05 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:136915 7.19 7.95b520b9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:43.759447+0000", 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 477.207352453, 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.47116866499999999, 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.019 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "reached pg", 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 136915 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759447+0000", 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759447+0000", 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759448+0000", 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1069999999999999e-06 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759453+0000", 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.096e-06 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759465+0000", 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2265e-05 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759471+0000", 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.519e-06 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759524+0000", 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2970999999999998e-05 2024-09-02T10:34:41.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for peered", 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:43.759531+0000", 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3080000000000002e-06 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.230604+0000", 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.47107291099999998 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:44.230615+0000", 2024-09-02T10:34:41.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1488000000000001e-05 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:144199 7.7 7:e22c0a38:::benchmark_data_smithi187_120712_object60257:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:26:59.586729+0000", 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 461.380070472, 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25041072800000003, 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:41.024 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 144199 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.586729+0000", 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.586729+0000", 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.586730+0000", 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2109999999999999e-06 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.586735+0000", 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3489999999999999e-06 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.586748+0000", 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2278e-05 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.586753+0000", 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4020000000000001e-06 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.586807+0000", 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3720000000000001e-05 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.028 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 41 16 54359 54343 82.8324 96.375 0.0140303 0.0120645 2024-09-02T10:34:41.029 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:41.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.586890+0000", 2024-09-02T10:34:41.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3028000000000005e-05 2024-09-02T10:34:41.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.586959+0000", 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9437999999999995e-05 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.588687+0000", 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0017276800000000001 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.591277+0000", 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025901180000000002 2024-09-02T10:34:41.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.837069+0000", 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24579241299999999 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.837104+0000", 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4995e-05 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:26:59.837139+0000", 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5095999999999997e-05 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:144842 7.7 7:e641c971:::benchmark_data_smithi187_120712_object60900:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:27:00.806788+0000", 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 460.16001085200003, 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.279226382, 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.032 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 144842 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.806788+0000", 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.806788+0000", 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.806790+0000", 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.502e-06 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.806796+0000", 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.981e-06 2024-09-02T10:34:41.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.806816+0000", 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9845000000000001e-05 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.806830+0000", 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4576000000000001e-05 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.806890+0000", 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0032999999999999e-05 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.807020+0000", 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0001292 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.807136+0000", 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116706 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.809464+0000", 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002328007 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.811928+0000", 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024640579999999999 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.085911+0000", 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27398250800000001 2024-09-02T10:34:41.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.085966+0000", 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5452999999999999e-05 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.086015+0000", 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.8513000000000001e-05 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:144842 7.7 e749/741 ondisk, result = 0)", 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:27:00.809266+0000", 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 460.15753295000002, 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27675360500000001, 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.809266+0000", 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.809266+0000", 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.809267+0000", 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8700000000000004e-07 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.809274+0000", 2024-09-02T10:34:41.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0450000000000003e-06 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.085851+0000", 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27657709800000002 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.085856+0000", 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4870000000000004e-06 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.085896+0000", 2024-09-02T10:34:41.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0497000000000002e-05 2024-09-02T10:34:41.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.085904+0000", 2024-09-02T10:34:41.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5989999999999996e-06 2024-09-02T10:34:41.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.086020+0000", 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000115992 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:144883 7.7 7:e596abba:::benchmark_data_smithi187_120712_object60941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:27:00.873399+0000", 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 460.09340039900002, 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31778895099999999, 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.043 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 144883 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.873399+0000", 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.873399+0000", 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.873400+0000", 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.155e-06 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.873405+0000", 2024-09-02T10:34:41.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9779999999999996e-06 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.873415+0000", 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8490000000000001e-06 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.873419+0000", 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0319999999999997e-06 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.873472+0000", 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2695999999999998e-05 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.873566+0000", 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4171000000000004e-05 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.873664+0000", 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8690000000000005e-05 2024-09-02T10:34:41.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.876106+0000", 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024416989999999999 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.884382+0000", 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0082761520000000002 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:41.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.191087+0000", 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30670478499999998 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.191141+0000", 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3885000000000003e-05 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.191188+0000", 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6859e-05 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.049 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27220.0:144883 7.7 e749/741 ondisk, result = 0)", 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:27:00.877207+0000", 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 460.08959264800001, 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.313986494, 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.877207+0000", 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.877207+0000", 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.877207+0000", 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2500000000000004e-07 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:00.877213+0000", 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1259999999999997e-06 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.191019+0000", 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31380648300000002 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.191029+0000", 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8919999999999995e-06 2024-09-02T10:34:41.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.191074+0000", 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5574000000000001e-05 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.191082+0000", 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.063e-06 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:01.191193+0000", 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111531 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27220.0:151701 7.7 7:e10306ba:::benchmark_data_smithi187_120712_object67759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:27:12.958405+0000", 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 448.00839470099999, 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23854761999999999, 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:41.054 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27220", 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 151701 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.958405+0000", 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.958405+0000", 2024-09-02T10:34:41.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.958405+0000", 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4200000000000005e-07 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.958410+0000", 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.583e-06 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.958420+0000", 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0047000000000001e-05 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.958425+0000", 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0119999999999996e-06 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.958481+0000", 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5584000000000002e-05 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.958597+0000", 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116889 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.958707+0000", 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010907 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.960577+0000", 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001870694 2024-09-02T10:34:41.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:12.960784+0000", 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020685500000000001 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:13.196888+0000", 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23610371199999999 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:13.196921+0000", 2024-09-02T10:34:41.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3232000000000003e-05 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:27:13.196952+0000", 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1099999999999997e-05 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:183131 8.3 e869/844 8:cd9a7f61:::benchmark_data_smithi187_136059_object77850:head v 869'11438, mlcod=823'5021)", 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:32:29.334258+0000", 2024-09-02T10:34:41.060 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.63254151999999, 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19000534299999999, 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.334258+0000", 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.334258+0000", 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.334259+0000", 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1909999999999999e-06 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.334272+0000", 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3077e-05 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.517754+0000", 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.18348175 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.517767+0000", 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3674e-05 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.517824+0000", 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6906e-05 2024-09-02T10:34:41.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.517845+0000", 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0696999999999999e-05 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.524232+0000", 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0063872219999999997 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.064 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.524263+0000", 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0825999999999999e-05 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(8.3 869/844 [PushOp(8:c5892a6c:::benchmark_data_smithi187_136059_object80224:head, version: 823'5023, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:c5892a6c:::benchmark_data_smithi187_136059_object80224:head@823'5023, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:32:29.348403+0000", 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 131.618395795, 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21878360499999999, 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.065 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.348403+0000", 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.348403+0000", 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.348404+0000", 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0550000000000001e-06 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.348452+0000", 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7568000000000003e-05 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564206+0000", 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21575388700000001 2024-09-02T10:34:41.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564225+0000", 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9377000000000001e-05 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564314+0000", 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8251999999999995e-05 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564323+0000", 2024-09-02T10:34:41.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0000000000000002e-06 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.567187+0000", 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002864466 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(8.3 e883,844 8:cf663c8e:::benchmark_data_smithi187_136059_object26166:head,841'8229)", 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:33:20.314918+0000", 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 80.651880986999998, 2024-09-02T10:34:41.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.79838157499999995, 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:20.314918+0000", 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:20.314918+0000", 2024-09-02T10:34:41.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:20.314919+0000", 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7000000000000004e-07 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:20.314923+0000", 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2429999999999999e-06 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.113040+0000", 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.798116771 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.113057+0000", 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6923e-05 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.113201+0000", 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014450899999999999 2024-09-02T10:34:41.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.113213+0000", 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2075e-05 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.113300+0000", 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6384000000000005e-05 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(8.3 e883,844 8:ca2eb16a:::benchmark_data_smithi187_136059_object27945:head,842'8339)", 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:33:21.817622+0000", 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 79.149177453999997, 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35858564599999998, 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.817622+0000", 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.817622+0000", 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.817628+0000", 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3940000000000001e-06 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.817638+0000", 2024-09-02T10:34:41.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.893e-06 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:22.175898+0000", 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35825989899999999 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:22.175919+0000", 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1100000000000001e-05 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:22.176081+0000", 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000161767 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:22.176097+0000", 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6180000000000001e-05 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:22.176207+0000", 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110413 2024-09-02T10:34:41.077 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:41.078 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:41.078 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:34:41.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:40 smithi162 ceph-mon[20279]: pgmap v2110: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 2.7 MiB/s, 43 objects/s recovering 2024-09-02T10:34:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:40 smithi162 ceph-mon[20279]: osdmap e921: 12 total, 12 up, 11 in 2024-09-02T10:34:41.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:40 smithi151 ceph-mon[16639]: pgmap v2110: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 2.7 MiB/s, 43 objects/s recovering 2024-09-02T10:34:41.097 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:40 smithi151 ceph-mon[16639]: osdmap e921: 12 total, 12 up, 11 in 2024-09-02T10:34:41.104 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:41.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:41.206 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:41.235 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:41.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:40 smithi187 ceph-mon[20072]: pgmap v2110: 47 pgs: 47 active+clean; 3.0 GiB data, 16 GiB used, 967 GiB / 983 GiB avail; 68 MiB/s wr, 1.09k op/s; 2.7 MiB/s, 43 objects/s recovering 2024-09-02T10:34:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:40 smithi187 ceph-mon[20072]: osdmap e921: 12 total, 12 up, 11 in 2024-09-02T10:34:41.337 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:41.369 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:41.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:41.470 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:41.500 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:41.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:41.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:41.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:41.626 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:41.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:41.627 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:41.728 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:41.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:41.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:41.751 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:41.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:41.875 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:41.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:41.876 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:41.978 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:34:41.978 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:34:41.979 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:42.011 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:42.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:42.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 42 16 55518 55502 82.5848 72.4375 0.0178809 0.0121015 2024-09-02T10:34:42.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:42.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:34:42.143 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:42.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:42.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:42.272 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:42.273 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:42.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:42.400 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:42.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:42.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:42.522 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:42.527 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:42.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:42.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:42.659 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:42.660 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:42.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:42.786 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:42.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:42.875 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2112,"stamp":"2024-09-02T10:34:41.591722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3285058080,"num_objects":50119,"num_object_clones":0,"num_object_copies":150357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50117,"num_write_kb":3207488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1716,"num_bytes_recovered":113129056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52969,"ondisk_log_size":52969,"up":141,"acting":141,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":17350588,"kb_used_data":9605088,"kb_used_omap":17,"kb_used_meta":7745262,"kb_avail":1013846084,"statfs":{"total":1055945392128,"available":1038178390016,"internally_reserved":0,"allocated":9835610112,"data_stored":9830259070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17545,"internal_metadata":7931149175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":380436480,"num_objects":5805,"num_object_clones":0,"num_object_copies":17415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-263,"num_objects_unfound":0,"num_objects_dirty":5805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5805,"num_write_kb":371520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":264,"num_bytes_recovered":17301504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.642600"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":40,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.955450+0000","last_change":"2024-09-02T10:34:29.260151+0000","last_active":"2024-09-02T10:34:36.955450+0000","last_peered":"2024-09-02T10:34:36.955450+0000","last_clean":"2024-09-02T10:34:36.955450+0000","last_became_active":"2024-09-02T10:34:17.880876+0000","last_became_peered":"2024-09-02T10:34:17.880876+0000","last_unstale":"2024-09-02T10:34:36.955450+0000","last_undegraded":"2024-09-02T10:34:36.955450+0000","last_fullsized":"2024-09-02T10:34:36.955450+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:29.260104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:29.260104+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:31.804926+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":407,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:37.254749+0000","last_change":"2024-09-02T10:34:00.928724+0000","last_active":"2024-09-02T10:34:37.254749+0000","last_peered":"2024-09-02T10:34:37.254749+0000","last_clean":"2024-09-02T10:34:37.254749+0000","last_became_active":"2024-09-02T10:33:38.659815+0000","last_became_peered":"2024-09-02T10:33:38.659815+0000","last_unstale":"2024-09-02T10:34:37.254749+0000","last_undegraded":"2024-09-02T10:34:37.254749+0000","last_fullsized":"2024-09-02T10:34:37.254749+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.928661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.928661+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:26.870517+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1528,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.955567+0000","last_change":"2024-09-02T10:33:59.232980+0000","last_active":"2024-09-02T10:34:36.955567+0000","last_peered":"2024-09-02T10:34:36.955567+0000","last_clean":"2024-09-02T10:34:36.955567+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:34:36.955567+0000","last_undegraded":"2024-09-02T10:34:36.955567+0000","last_fullsized":"2024-09-02T10:34:36.955567+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:59.232916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:59.232916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.466149+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1504,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.955404+0000","last_change":"2024-09-02T10:33:55.174212+0000","last_active":"2024-09-02T10:34:36.955404+0000","last_peered":"2024-09-02T10:34:36.955404+0000","last_clean":"2024-09-02T10:34:36.955404+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:34:36.955404+0000","last_undegraded":"2024-09-02T10:34:36.955404+0000","last_fullsized":"2024-09-02T10:34:36.955404+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.174147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.174147+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:22.067581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":823,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.961338+0000","last_change":"2024-09-02T10:33:55.588678+0000","last_active":"2024-09-02T10:34:36.961338+0000","last_peered":"2024-09-02T10:34:36.961338+0000","last_clean":"2024-09-02T10:34:36.961338+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:34:36.961338+0000","last_undegraded":"2024-09-02T10:34:36.961338+0000","last_fullsized":"2024-09-02T10:34:36.961338+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.588605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.588605+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:57.688518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":1364,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:34.949214+0000","last_change":"2024-09-02T10:33:56.473614+0000","last_active":"2024-09-02T10:34:34.949214+0000","last_peered":"2024-09-02T10:34:34.949214+0000","last_clean":"2024-09-02T10:34:34.949214+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:34:34.949214+0000","last_undegraded":"2024-09-02T10:34:34.949214+0000","last_fullsized":"2024-09-02T10:34:34.949214+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.473555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.473555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:04.434869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1884,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.958367+0000","last_change":"2024-09-02T10:34:00.430544+0000","last_active":"2024-09-02T10:34:36.958367+0000","last_peered":"2024-09-02T10:34:36.958367+0000","last_clean":"2024-09-02T10:34:36.958367+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:34:36.958367+0000","last_undegraded":"2024-09-02T10:34:36.958367+0000","last_fullsized":"2024-09-02T10:34:36.958367+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.430459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.430459+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:19.434576+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1527,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.955621+0000","last_change":"2024-09-02T10:33:56.194537+0000","last_active":"2024-09-02T10:34:36.955621+0000","last_peered":"2024-09-02T10:34:36.955621+0000","last_clean":"2024-09-02T10:34:36.955621+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:34:36.955621+0000","last_undegraded":"2024-09-02T10:34:36.955621+0000","last_fullsized":"2024-09-02T10:34:36.955621+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.194480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.194480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:02.488817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":86,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:37.255011+0000","last_change":"2024-09-02T10:33:56.167103+0000","last_active":"2024-09-02T10:34:37.255011+0000","last_peered":"2024-09-02T10:34:37.255011+0000","last_clean":"2024-09-02T10:34:37.255011+0000","last_became_active":"2024-09-02T10:33:38.657147+0000","last_became_peered":"2024-09-02T10:33:38.657147+0000","last_unstale":"2024-09-02T10:34:37.255011+0000","last_undegraded":"2024-09-02T10:34:37.255011+0000","last_fullsized":"2024-09-02T10:34:37.255011+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.167029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.167029+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.270011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1492,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:34.948073+0000","last_change":"2024-09-02T10:33:55.173514+0000","last_active":"2024-09-02T10:34:34.948073+0000","last_peered":"2024-09-02T10:34:34.948073+0000","last_clean":"2024-09-02T10:34:34.948073+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:34:34.948073+0000","last_undegraded":"2024-09-02T10:34:34.948073+0000","last_fullsized":"2024-09-02T10:34:34.948073+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.173450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.173450+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:59.387368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":738,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:37.254739+0000","last_change":"2024-09-02T10:33:56.943878+0000","last_active":"2024-09-02T10:34:37.254739+0000","last_peered":"2024-09-02T10:34:37.254739+0000","last_clean":"2024-09-02T10:34:37.254739+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:34:37.254739+0000","last_undegraded":"2024-09-02T10:34:37.254739+0000","last_fullsized":"2024-09-02T10:34:37.254739+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.943826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.943826+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.719187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":559,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.258204+0000","last_change":"2024-09-02T10:34:36.258204+0000","last_active":"2024-09-02T10:34:36.258204+0000","last_peered":"2024-09-02T10:34:36.258204+0000","last_clean":"2024-09-02T10:34:36.258204+0000","last_became_active":"2024-09-02T10:33:38.657430+0000","last_became_peered":"2024-09-02T10:33:38.657430+0000","last_unstale":"2024-09-02T10:34:36.258204+0000","last_undegraded":"2024-09-02T10:34:36.258204+0000","last_fullsized":"2024-09-02T10:34:36.258204+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:36.258130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:36.258130+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:56.899533+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1783,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.958497+0000","last_change":"2024-09-02T10:34:08.628331+0000","last_active":"2024-09-02T10:34:36.958497+0000","last_peered":"2024-09-02T10:34:36.958497+0000","last_clean":"2024-09-02T10:34:36.958497+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:34:36.958497+0000","last_undegraded":"2024-09-02T10:34:36.958497+0000","last_fullsized":"2024-09-02T10:34:36.958497+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.628247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.628247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:31.896890+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":523,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:37.254695+0000","last_change":"2024-09-02T10:34:01.246043+0000","last_active":"2024-09-02T10:34:37.254695+0000","last_peered":"2024-09-02T10:34:37.254695+0000","last_clean":"2024-09-02T10:34:37.254695+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:34:37.254695+0000","last_undegraded":"2024-09-02T10:34:37.254695+0000","last_fullsized":"2024-09-02T10:34:37.254695+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.245989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.245989+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.756195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1753,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.958430+0000","last_change":"2024-09-02T10:34:05.558912+0000","last_active":"2024-09-02T10:34:36.958430+0000","last_peered":"2024-09-02T10:34:36.958430+0000","last_clean":"2024-09-02T10:34:36.958430+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:34:36.958430+0000","last_undegraded":"2024-09-02T10:34:36.958430+0000","last_fullsized":"2024-09-02T10:34:36.958430+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:05.558852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:05.558852+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:06.139483+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"920'3176","reported_seq":3233,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:37.525280+0000","last_change":"2024-09-02T10:33:52.925516+0000","last_active":"2024-09-02T10:34:37.525280+0000","last_peered":"2024-09-02T10:34:37.525280+0000","last_clean":"2024-09-02T10:34:37.525280+0000","last_became_active":"2024-09-02T10:33:52.925200+0000","last_became_peered":"2024-09-02T10:33:52.925200+0000","last_unstale":"2024-09-02T10:34:37.525280+0000","last_undegraded":"2024-09-02T10:34:37.525280+0000","last_fullsized":"2024-09-02T10:34:37.525280+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3176,"log_dups_size":0,"ondisk_log_size":3176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:15.876694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208142336,"num_objects":3176,"num_object_clones":0,"num_object_copies":9528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3176,"num_write_kb":203264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,8],"acting":[6,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1848,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.958466+0000","last_change":"2024-09-02T10:34:07.590524+0000","last_active":"2024-09-02T10:34:36.958466+0000","last_peered":"2024-09-02T10:34:36.958466+0000","last_clean":"2024-09-02T10:34:36.958466+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:34:36.958466+0000","last_undegraded":"2024-09-02T10:34:36.958466+0000","last_fullsized":"2024-09-02T10:34:36.958466+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:07.590464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:07.590464+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:30.846352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"920'3216","reported_seq":3272,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:39.247949+0000","last_change":"2024-09-02T10:33:52.750341+0000","last_active":"2024-09-02T10:34:39.247949+0000","last_peered":"2024-09-02T10:34:39.247949+0000","last_clean":"2024-09-02T10:34:39.247949+0000","last_became_active":"2024-09-02T10:33:52.750137+0000","last_became_peered":"2024-09-02T10:33:52.750137+0000","last_unstale":"2024-09-02T10:34:39.247949+0000","last_undegraded":"2024-09-02T10:34:39.247949+0000","last_fullsized":"2024-09-02T10:34:39.247949+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3216,"log_dups_size":0,"ondisk_log_size":3216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:58.157142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210763776,"num_objects":3216,"num_object_clones":0,"num_object_copies":9648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3216,"num_write_kb":205824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1964,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:34.947836+0000","last_change":"2024-09-02T10:30:24.407289+0000","last_active":"2024-09-02T10:34:34.947836+0000","last_peered":"2024-09-02T10:34:34.947836+0000","last_clean":"2024-09-02T10:34:34.947836+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:34:34.947836+0000","last_undegraded":"2024-09-02T10:34:34.947836+0000","last_fullsized":"2024-09-02T10:34:34.947836+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.407218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.407218+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"920'3264","reported_seq":3321,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:39.267584+0000","last_change":"2024-09-02T10:33:52.925398+0000","last_active":"2024-09-02T10:34:39.267584+0000","last_peered":"2024-09-02T10:34:39.267584+0000","last_clean":"2024-09-02T10:34:39.267584+0000","last_became_active":"2024-09-02T10:33:52.925083+0000","last_became_peered":"2024-09-02T10:33:52.925083+0000","last_unstale":"2024-09-02T10:34:39.267584+0000","last_undegraded":"2024-09-02T10:34:39.267584+0000","last_fullsized":"2024-09-02T10:34:39.267584+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3264,"log_dups_size":0,"ondisk_log_size":3264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.033120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213909504,"num_objects":3264,"num_object_clones":0,"num_object_copies":9792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3264,"num_write_kb":208896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1624,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.960228+0000","last_change":"2024-09-02T10:33:56.586452+0000","last_active":"2024-09-02T10:34:36.960228+0000","last_peered":"2024-09-02T10:34:36.960228+0000","last_clean":"2024-09-02T10:34:36.960228+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:34:36.960228+0000","last_undegraded":"2024-09-02T10:34:36.960228+0000","last_fullsized":"2024-09-02T10:34:36.960228+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.586392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.586392+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:20.658780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"920'3249","reported_seq":3306,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:38.033143+0000","last_change":"2024-09-02T10:33:52.751342+0000","last_active":"2024-09-02T10:34:38.033143+0000","last_peered":"2024-09-02T10:34:38.033143+0000","last_clean":"2024-09-02T10:34:38.033143+0000","last_became_active":"2024-09-02T10:33:52.750752+0000","last_became_peered":"2024-09-02T10:33:52.750752+0000","last_unstale":"2024-09-02T10:34:38.033143+0000","last_undegraded":"2024-09-02T10:34:38.033143+0000","last_fullsized":"2024-09-02T10:34:38.033143+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3249,"log_dups_size":0,"ondisk_log_size":3249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:16.573736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212926464,"num_objects":3249,"num_object_clones":0,"num_object_copies":9747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3249,"num_write_kb":207936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":601,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:37.255040+0000","last_change":"2024-09-02T10:34:08.328616+0000","last_active":"2024-09-02T10:34:37.255040+0000","last_peered":"2024-09-02T10:34:37.255040+0000","last_clean":"2024-09-02T10:34:37.255040+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:34:37.255040+0000","last_undegraded":"2024-09-02T10:34:37.255040+0000","last_fullsized":"2024-09-02T10:34:37.255040+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.328570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.328570+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.052868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"920'3189","reported_seq":3246,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:39.262036+0000","last_change":"2024-09-02T10:33:52.761107+0000","last_active":"2024-09-02T10:34:39.262036+0000","last_peered":"2024-09-02T10:34:39.262036+0000","last_clean":"2024-09-02T10:34:39.262036+0000","last_became_active":"2024-09-02T10:33:52.760804+0000","last_became_peered":"2024-09-02T10:33:52.760804+0000","last_unstale":"2024-09-02T10:34:39.262036+0000","last_undegraded":"2024-09-02T10:34:39.262036+0000","last_fullsized":"2024-09-02T10:34:39.262036+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:03.781816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208994304,"num_objects":3189,"num_object_clones":0,"num_object_copies":9567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3189,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3189,"num_write_kb":204096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":931,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.955451+0000","last_change":"2024-09-02T10:34:17.882133+0000","last_active":"2024-09-02T10:34:36.955451+0000","last_peered":"2024-09-02T10:34:36.955451+0000","last_clean":"2024-09-02T10:34:36.955451+0000","last_became_active":"2024-09-02T10:34:17.881745+0000","last_became_peered":"2024-09-02T10:34:17.881745+0000","last_unstale":"2024-09-02T10:34:36.955451+0000","last_undegraded":"2024-09-02T10:34:36.955451+0000","last_fullsized":"2024-09-02T10:34:36.955451+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:58.800694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:58.800694+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:25.599849+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"919'2958","reported_seq":3013,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.567875+0000","last_change":"2024-09-02T10:33:52.926793+0000","last_active":"2024-09-02T10:34:36.567875+0000","last_peered":"2024-09-02T10:34:36.567875+0000","last_clean":"2024-09-02T10:34:36.567875+0000","last_became_active":"2024-09-02T10:33:52.926467+0000","last_became_peered":"2024-09-02T10:33:52.926467+0000","last_unstale":"2024-09-02T10:34:36.567875+0000","last_undegraded":"2024-09-02T10:34:36.567875+0000","last_fullsized":"2024-09-02T10:34:36.567875+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2958,"log_dups_size":0,"ondisk_log_size":2958,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.878242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193855488,"num_objects":2958,"num_object_clones":0,"num_object_copies":8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2958,"num_write_kb":189312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":1386,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:34.949296+0000","last_change":"2024-09-02T10:34:08.280819+0000","last_active":"2024-09-02T10:34:34.949296+0000","last_peered":"2024-09-02T10:34:34.949296+0000","last_clean":"2024-09-02T10:34:34.949296+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:34:34.949296+0000","last_undegraded":"2024-09-02T10:34:34.949296+0000","last_fullsized":"2024-09-02T10:34:34.949296+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.280762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.280762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:19.946116+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"919'2935","reported_seq":2990,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.587372+0000","last_change":"2024-09-02T10:33:52.925479+0000","last_active":"2024-09-02T10:34:36.587372+0000","last_peered":"2024-09-02T10:34:36.587372+0000","last_clean":"2024-09-02T10:34:36.587372+0000","last_became_active":"2024-09-02T10:33:52.925170+0000","last_became_peered":"2024-09-02T10:33:52.925170+0000","last_unstale":"2024-09-02T10:34:36.587372+0000","last_undegraded":"2024-09-02T10:34:36.587372+0000","last_fullsized":"2024-09-02T10:34:36.587372+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2935,"log_dups_size":0,"ondisk_log_size":2935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:05.519875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192348160,"num_objects":2935,"num_object_clones":0,"num_object_copies":8805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2935,"num_write_kb":187840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":1351,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:34.949414+0000","last_change":"2024-09-02T10:34:05.282359+0000","last_active":"2024-09-02T10:34:34.949414+0000","last_peered":"2024-09-02T10:34:34.949414+0000","last_clean":"2024-09-02T10:34:34.949414+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:34:34.949414+0000","last_undegraded":"2024-09-02T10:34:34.949414+0000","last_fullsized":"2024-09-02T10:34:34.949414+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:05.282307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:05.282307+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:12.971380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"920'3118","reported_seq":3174,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:37.706194+0000","last_change":"2024-09-02T10:33:52.925486+0000","last_active":"2024-09-02T10:34:37.706194+0000","last_peered":"2024-09-02T10:34:37.706194+0000","last_clean":"2024-09-02T10:34:37.706194+0000","last_became_active":"2024-09-02T10:33:52.925153+0000","last_became_peered":"2024-09-02T10:33:52.925153+0000","last_unstale":"2024-09-02T10:34:37.706194+0000","last_undegraded":"2024-09-02T10:34:37.706194+0000","last_fullsized":"2024-09-02T10:34:37.706194+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3118,"log_dups_size":0,"ondisk_log_size":3118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.337828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204341248,"num_objects":3118,"num_object_clones":0,"num_object_copies":9354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3118,"num_write_kb":199552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1142,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.957538+0000","last_change":"2024-09-02T10:34:17.882691+0000","last_active":"2024-09-02T10:34:36.957538+0000","last_peered":"2024-09-02T10:34:36.957538+0000","last_clean":"2024-09-02T10:34:36.957538+0000","last_became_active":"2024-09-02T10:34:17.882376+0000","last_became_peered":"2024-09-02T10:34:17.882376+0000","last_unstale":"2024-09-02T10:34:36.957538+0000","last_undegraded":"2024-09-02T10:34:36.957538+0000","last_fullsized":"2024-09-02T10:34:36.957538+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.268512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.268512+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:22.220969+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"919'3020","reported_seq":6513,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.753584+0000","last_change":"2024-09-02T10:34:35.145394+0000","last_active":"2024-09-02T10:34:36.753584+0000","last_peered":"2024-09-02T10:34:36.753584+0000","last_clean":"2024-09-02T10:34:36.753584+0000","last_became_active":"2024-09-02T10:34:35.117833+0000","last_became_peered":"2024-09-02T10:34:35.117833+0000","last_unstale":"2024-09-02T10:34:36.753584+0000","last_undegraded":"2024-09-02T10:34:36.753584+0000","last_fullsized":"2024-09-02T10:34:36.753584+0000","mapping_epoch":918,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3020,"log_dups_size":0,"ondisk_log_size":3020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:03.849248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197918720,"num_objects":3020,"num_object_clones":0,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3020,"num_write_kb":193280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":112197632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,3],"acting":[10,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":1366,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:35.240578+0000","last_change":"2024-09-02T10:34:35.240578+0000","last_active":"2024-09-02T10:34:35.240578+0000","last_peered":"2024-09-02T10:34:35.240578+0000","last_clean":"2024-09-02T10:34:35.240578+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:34:35.240578+0000","last_undegraded":"2024-09-02T10:34:35.240578+0000","last_fullsized":"2024-09-02T10:34:35.240578+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:35.240504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:35.240504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:36.945732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"920'3149","reported_seq":3206,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:37.703347+0000","last_change":"2024-09-02T10:33:52.925507+0000","last_active":"2024-09-02T10:34:37.703347+0000","last_peered":"2024-09-02T10:34:37.703347+0000","last_clean":"2024-09-02T10:34:37.703347+0000","last_became_active":"2024-09-02T10:33:52.925177+0000","last_became_peered":"2024-09-02T10:33:52.925177+0000","last_unstale":"2024-09-02T10:34:37.703347+0000","last_undegraded":"2024-09-02T10:34:37.703347+0000","last_fullsized":"2024-09-02T10:34:37.703347+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3149,"log_dups_size":0,"ondisk_log_size":3149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:10.535011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206372864,"num_objects":3149,"num_object_clones":0,"num_object_copies":9447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3149,"num_write_kb":201536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1614,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:34.947958+0000","last_change":"2024-09-02T10:34:04.115873+0000","last_active":"2024-09-02T10:34:34.947958+0000","last_peered":"2024-09-02T10:34:34.947958+0000","last_clean":"2024-09-02T10:34:34.947958+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:34:34.947958+0000","last_undegraded":"2024-09-02T10:34:34.947958+0000","last_fullsized":"2024-09-02T10:34:34.947958+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:04.115830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:04.115830+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:04.432089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"920'3236","reported_seq":3291,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:39.651914+0000","last_change":"2024-09-02T10:33:52.750617+0000","last_active":"2024-09-02T10:34:39.651914+0000","last_peered":"2024-09-02T10:34:39.651914+0000","last_clean":"2024-09-02T10:34:39.651914+0000","last_became_active":"2024-09-02T10:33:52.750198+0000","last_became_peered":"2024-09-02T10:33:52.750198+0000","last_unstale":"2024-09-02T10:34:39.651914+0000","last_undegraded":"2024-09-02T10:34:39.651914+0000","last_fullsized":"2024-09-02T10:34:39.651914+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3236,"log_dups_size":0,"ondisk_log_size":3236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.904796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212074496,"num_objects":3236,"num_object_clones":0,"num_object_copies":9708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3236,"num_write_kb":207104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":908,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.957230+0000","last_change":"2024-09-02T10:33:59.302485+0000","last_active":"2024-09-02T10:34:36.957230+0000","last_peered":"2024-09-02T10:34:36.957230+0000","last_clean":"2024-09-02T10:34:36.957230+0000","last_became_active":"2024-09-02T10:33:38.664311+0000","last_became_peered":"2024-09-02T10:33:38.664311+0000","last_unstale":"2024-09-02T10:34:36.957230+0000","last_undegraded":"2024-09-02T10:34:36.957230+0000","last_fullsized":"2024-09-02T10:34:36.957230+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:59.302421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:59.302421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:05.057279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"920'3220","reported_seq":3276,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:38.014631+0000","last_change":"2024-09-02T10:33:52.753464+0000","last_active":"2024-09-02T10:34:38.014631+0000","last_peered":"2024-09-02T10:34:38.014631+0000","last_clean":"2024-09-02T10:34:38.014631+0000","last_became_active":"2024-09-02T10:33:52.753083+0000","last_became_peered":"2024-09-02T10:33:52.753083+0000","last_unstale":"2024-09-02T10:34:38.014631+0000","last_undegraded":"2024-09-02T10:34:38.014631+0000","last_fullsized":"2024-09-02T10:34:38.014631+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3220,"log_dups_size":0,"ondisk_log_size":3220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:55.293684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211025920,"num_objects":3220,"num_object_clones":0,"num_object_copies":9660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3220,"num_write_kb":206080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1624,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:34.947874+0000","last_change":"2024-09-02T10:34:01.291835+0000","last_active":"2024-09-02T10:34:34.947874+0000","last_peered":"2024-09-02T10:34:34.947874+0000","last_clean":"2024-09-02T10:34:34.947874+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:34:34.947874+0000","last_undegraded":"2024-09-02T10:34:34.947874+0000","last_fullsized":"2024-09-02T10:34:34.947874+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.291779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.291779+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.558074+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"919'2904","reported_seq":2959,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.749410+0000","last_change":"2024-09-02T10:33:52.753639+0000","last_active":"2024-09-02T10:34:36.749410+0000","last_peered":"2024-09-02T10:34:36.749410+0000","last_clean":"2024-09-02T10:34:36.749410+0000","last_became_active":"2024-09-02T10:33:52.753326+0000","last_became_peered":"2024-09-02T10:33:52.753326+0000","last_unstale":"2024-09-02T10:34:36.749410+0000","last_undegraded":"2024-09-02T10:34:36.749410+0000","last_fullsized":"2024-09-02T10:34:36.749410+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":2904,"log_dups_size":0,"ondisk_log_size":2904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.852817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190316544,"num_objects":2904,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2904,"num_write_kb":185856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1882,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.957630+0000","last_change":"2024-09-02T10:33:55.238766+0000","last_active":"2024-09-02T10:34:36.957630+0000","last_peered":"2024-09-02T10:34:36.957630+0000","last_clean":"2024-09-02T10:34:36.957630+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:34:36.957630+0000","last_undegraded":"2024-09-02T10:34:36.957630+0000","last_fullsized":"2024-09-02T10:34:36.957630+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.238683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.238683+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.350030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"920'3119","reported_seq":3176,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:37.679977+0000","last_change":"2024-09-02T10:33:52.926432+0000","last_active":"2024-09-02T10:34:37.679977+0000","last_peered":"2024-09-02T10:34:37.679977+0000","last_clean":"2024-09-02T10:34:37.679977+0000","last_became_active":"2024-09-02T10:33:52.926053+0000","last_became_peered":"2024-09-02T10:33:52.926053+0000","last_unstale":"2024-09-02T10:34:37.679977+0000","last_undegraded":"2024-09-02T10:34:37.679977+0000","last_fullsized":"2024-09-02T10:34:37.679977+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3119,"log_dups_size":0,"ondisk_log_size":3119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:04.035839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204406784,"num_objects":3119,"num_object_clones":0,"num_object_copies":9357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3119,"num_write_kb":199616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":830,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.961494+0000","last_change":"2024-09-02T10:34:02.569694+0000","last_active":"2024-09-02T10:34:36.961494+0000","last_peered":"2024-09-02T10:34:36.961494+0000","last_clean":"2024-09-02T10:34:36.961494+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:34:36.961494+0000","last_undegraded":"2024-09-02T10:34:36.961494+0000","last_fullsized":"2024-09-02T10:34:36.961494+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:02.569628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:02.569628+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:27.473229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"920'3048","reported_seq":3105,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:37.684095+0000","last_change":"2024-09-02T10:33:52.927094+0000","last_active":"2024-09-02T10:34:37.684095+0000","last_peered":"2024-09-02T10:34:37.684095+0000","last_clean":"2024-09-02T10:34:37.684095+0000","last_became_active":"2024-09-02T10:33:52.926709+0000","last_became_peered":"2024-09-02T10:33:52.926709+0000","last_unstale":"2024-09-02T10:34:37.684095+0000","last_undegraded":"2024-09-02T10:34:37.684095+0000","last_fullsized":"2024-09-02T10:34:37.684095+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3048,"log_dups_size":0,"ondisk_log_size":3048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:16.295134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199753728,"num_objects":3048,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3048,"num_write_kb":195072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1644,"reported_epoch":919,"state":"active+clean","last_fresh":"2024-09-02T10:34:34.948181+0000","last_change":"2024-09-02T10:30:46.402988+0000","last_active":"2024-09-02T10:34:34.948181+0000","last_peered":"2024-09-02T10:34:34.948181+0000","last_clean":"2024-09-02T10:34:34.948181+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:34:34.948181+0000","last_undegraded":"2024-09-02T10:34:34.948181+0000","last_fullsized":"2024-09-02T10:34:34.948181+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:46.402901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.402901+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"920'3316","reported_seq":3373,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:39.683595+0000","last_change":"2024-09-02T10:33:52.925410+0000","last_active":"2024-09-02T10:34:39.683595+0000","last_peered":"2024-09-02T10:34:39.683595+0000","last_clean":"2024-09-02T10:34:39.683595+0000","last_became_active":"2024-09-02T10:33:52.925040+0000","last_became_peered":"2024-09-02T10:33:52.925040+0000","last_unstale":"2024-09-02T10:34:39.683595+0000","last_undegraded":"2024-09-02T10:34:39.683595+0000","last_fullsized":"2024-09-02T10:34:39.683595+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3316,"log_dups_size":0,"ondisk_log_size":3316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:00.330509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217317376,"num_objects":3316,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3316,"num_write_kb":212224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":1045,"reported_epoch":920,"state":"active+clean","last_fresh":"2024-09-02T10:34:36.957151+0000","last_change":"2024-09-02T10:33:55.336238+0000","last_active":"2024-09-02T10:34:36.957151+0000","last_peered":"2024-09-02T10:34:36.957151+0000","last_clean":"2024-09-02T10:34:36.957151+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:34:36.957151+0000","last_undegraded":"2024-09-02T10:34:36.957151+0000","last_fullsized":"2024-09-02T10:34:36.957151+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.336182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.336182+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:16.545175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3284467712,"num_objects":50117,"num_object_clones":0,"num_object_copies":150351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50117,"num_write_kb":3207488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":112197632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10028056576,"data_stored":10028056576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50117,"ondisk_log_size":50117,"up":48,"acting":48,"num_store_stats":10},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":343597384094,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532613027,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1580560,"kb_used_data":945464,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92164592,"statfs":{"total":95995035648,"available":94376542208,"internally_reserved":0,"allocated":968155136,"data_stored":967672906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":650312128},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841960,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2055384,"kb_used_data":1198148,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":91689768,"statfs":{"total":95995035648,"available":93890322432,"internally_reserved":0,"allocated":1226903552,"data_stored":1226416840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":877787584},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070895,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1137172,"kb_used_data":628420,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":92607980,"statfs":{"total":95995035648,"available":94830571520,"internally_reserved":0,"allocated":643502080,"data_stored":643015368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299827,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1865816,"kb_used_data":1130752,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":91879336,"statfs":{"total":95995035648,"available":94084440064,"internally_reserved":0,"allocated":1157890048,"data_stored":1157404266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":752679370},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528759,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":928796,"kb_used_data":396420,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92816356,"statfs":{"total":95995035648,"available":95043948544,"internally_reserved":0,"allocated":405934080,"data_stored":405447368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":545126855},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724988,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2666528,"kb_used_data":1800972,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":91078624,"statfs":{"total":95995035648,"available":93264510976,"internally_reserved":0,"allocated":1844195328,"data_stored":1843705064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953921,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":914132,"kb_used_data":417860,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92831020,"statfs":{"total":95995035648,"available":95058964480,"internally_reserved":0,"allocated":427888640,"data_stored":427401928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":508164549},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150151,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2069396,"kb_used_data":1217732,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91675756,"statfs":{"total":95995035648,"available":93875974144,"internally_reserved":0,"allocated":1246957568,"data_stored":1246470856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":872085960},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":614360,"kb_used_data":1472,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93130792,"statfs":{"total":95995035648,"available":95365931008,"internally_reserved":0,"allocated":1507328,"data_stored":1021546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575311,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2659604,"kb_used_data":1866244,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":91085548,"statfs":{"total":95995035648,"available":93271601152,"internally_reserved":0,"allocated":1911033856,"data_stored":1910547144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":858840,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":857214,"kb_avail":92886312,"statfs":{"total":95995035648,"available":95115583488,"internally_reserved":0,"allocated":1642496,"data_stored":1155784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":877787589},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1920335872,"data_stored":1920335872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1271267328,"data_stored":1271267328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":428277760,"data_stored":428277760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1842937856,"data_stored":1842937856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":409862144,"data_stored":409862144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1168179200,"data_stored":1168179200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":650575872,"data_stored":650575872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1246560256,"data_stored":1246560256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":977928192,"data_stored":977928192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":112132096,"data_stored":112132096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:34:42.878 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-02T10:34:42.878 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set unique_pool_7 pg_num 26 2024-09-02T10:34:42.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:42.890 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:34:42.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:42.921 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:42.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:42.993 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:34:43.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 43 16 57031 57015 82.8632 94.5625 0.00511908 0.0120626 2024-09-02T10:34:43.022 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:43.065 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:43.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:43.086 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:42 smithi162 ceph-mon[20279]: pgmap v2112: 47 pgs: 47 active+clean; 3.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 55 MiB/s wr, 873 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-09-02T10:34:43.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:42 smithi162 ceph-mon[20279]: from='client.18921 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:43.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1274060787' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:43.087 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:42 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:43.088 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:34:43.088 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:34:43.088 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.088 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:57123 9.d 9:b4996816:::benchmark_data_smithi187_154927_object57122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-09-02T10:34:43.088 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:43.080888+0000", 2024-09-02T10:34:43.088 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.004135494, 2024-09-02T10:34:43.088 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0041648329999999997, 2024-09-02T10:34:43.088 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57123 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.080888+0000", 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.089 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.080888+0000", 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.080889+0000", 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4750000000000001e-06 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:43.090 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.080975+0000", 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5354999999999998e-05 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.080998+0000", 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2989999999999998e-05 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081033+0000", 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5743e-05 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.091 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081101+0000", 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7267999999999994e-05 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081237+0000", 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013633499999999999 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081392+0000", 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154847 2024-09-02T10:34:43.092 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:57126 9.d 9:b3fef284:::benchmark_data_smithi187_154927_object57125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:43.081082+0000", 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0039415470000000001, 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040381460000000003, 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:34:43.093 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57126 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081082+0000", 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081082+0000", 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.094 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081083+0000", 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2900000000000003e-07 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081139+0000", 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.6759999999999999e-05 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081154+0000", 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4729000000000001e-05 2024-09-02T10:34:43.095 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081161+0000", 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1840000000000002e-06 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083406+0000", 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0022451239999999998 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:43.096 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083537+0000", 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013053199999999999 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083672+0000", 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013483399999999999 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:57118 9.d e922/897)", 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:43.080823+0000", 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0042001779999999997, 2024-09-02T10:34:43.097 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0043504279999999999, 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.080823+0000", 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:43.098 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.080823+0000", 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.080824+0000", 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7000000000000003e-07 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.080831+0000", 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6080000000000001e-06 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.099 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081219+0000", 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00038783399999999999 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.081224+0000", 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.75e-06 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.100 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:57128 9.7 e922/897 9:eaaa8f5d:::benchmark_data_smithi187_154927_object57127:head v 922'3496, mlcod=922'3494)", 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:43.083363+0000", 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.001660789, 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018485610000000001, 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083363+0000", 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.101 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083363+0000", 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083363+0000", 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8800000000000002e-07 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083398+0000", 2024-09-02T10:34:43.102 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4165000000000002e-05 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083425+0000", 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6939000000000001e-05 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083430+0000", 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8270000000000003e-06 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.103 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083475+0000", 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5092000000000001e-05 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083492+0000", 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6646999999999998e-05 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:57130 9.e 9:737914ca:::benchmark_data_smithi187_154927_object57129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-09-02T10:34:43.104 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:43.083818+0000", 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0012056440000000001, 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0014437950000000001, 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57130 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.105 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083818+0000", 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083818+0000", 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083820+0000", 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.739e-06 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.106 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083857+0000", 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7425999999999997e-05 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083871+0000", 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4025e-05 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083879+0000", 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7090000000000002e-06 2024-09-02T10:34:43.107 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.083937+0000", 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8496999999999999e-05 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.084087+0000", 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015004600000000001 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,10", 2024-09-02T10:34:43.108 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.084243+0000", 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015568 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:57099 9.d e922/897)", 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:43.084467+0000", 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.00055646099999999996, 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00084625399999999995, 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:43.109 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "queued for pg", 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.084467+0000", 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.084467+0000", 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.110 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.084468+0000", 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6400000000000002e-07 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.084472+0000", 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2230000000000001e-06 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.084504+0000", 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1717999999999997e-05 2024-09-02T10:34:43.111 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.084510+0000", 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4189999999999999e-06 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:57099 9.d 9:b2076e01:::benchmark_data_smithi187_154927_object57098:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:43.067036+0000", 2024-09-02T10:34:43.112 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.017987617000000001, 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.018318542, 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57099 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:43.113 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.067036+0000", 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.067036+0000", 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.067037+0000", 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.584e-06 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.114 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.067111+0000", 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3376999999999999e-05 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.067127+0000", 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5996999999999999e-05 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.067135+0000", 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3529999999999995e-06 2024-09-02T10:34:43.115 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.067198+0000", 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2420000000000002e-05 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.067330+0000", 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013221700000000001 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-09-02T10:34:43.116 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.067456+0000", 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000126503 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.069533+0000", 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0020764239999999999 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.069986+0000", 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00045375400000000001 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.117 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:57118 9.d 9:b488246c:::benchmark_data_smithi187_154927_object57117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e922)", 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:43.077288+0000", 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.0077357349999999997, 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0081358179999999995, 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:43.118 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 57118 2024-09-02T10:34:43.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.119 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:34:43.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:34:43.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.077288+0000", 2024-09-02T10:34:43.119 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.119 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.119 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.119 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:34:43.119 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.077288+0000", 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.077289+0000", 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.778e-06 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.077331+0000", 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1427999999999997e-05 2024-09-02T10:34:43.120 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.077354+0000", 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3142e-05 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.077363+0000", 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0480000000000004e-06 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.077430+0000", 2024-09-02T10:34:43.121 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7181000000000006e-05 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.077558+0000", 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012734899999999999 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,5", 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.077682+0000", 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000123991 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:34:43.122 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:43.079815+0000", 2024-09-02T10:34:43.123 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021335780000000001 2024-09-02T10:34:43.123 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.123 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:34:43.123 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.123 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:34:43.123 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:34:43.123 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 8 2024-09-02T10:34:43.123 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:34:43.167 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:43.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:42 smithi151 ceph-mon[16639]: pgmap v2112: 47 pgs: 47 active+clean; 3.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 55 MiB/s wr, 873 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-09-02T10:34:43.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:42 smithi151 ceph-mon[16639]: from='client.18921 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:43.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1274060787' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:43.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:42 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:43.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:43.197 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:43.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:42 smithi187 ceph-mon[20072]: pgmap v2112: 47 pgs: 47 active+clean; 3.1 GiB data, 17 GiB used, 967 GiB / 983 GiB avail; 55 MiB/s wr, 873 op/s; 2.5 MiB/s, 39 objects/s recovering 2024-09-02T10:34:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:42 smithi187 ceph-mon[20072]: from='client.18921 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1274060787' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:42 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:43.298 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:43.331 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:43.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:43.432 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:43.462 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:43.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:43.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:43.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:43.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:43.689 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:43.712 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:43.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:43.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:43.814 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:43.844 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:43.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:43.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:34:43.908 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:43.947 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:43.977 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:43.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:43.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:44.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 44 15 58402 58387 82.9286 85.75 0.0087911 0.0120528 2024-09-02T10:34:44.079 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:44.103 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:44.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:44.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:44 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:44.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:44 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:44.186 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:44 smithi151 ceph-mon[16639]: osdmap e922: 12 total, 12 up, 11 in 2024-09-02T10:34:44.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:44.216 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:44.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:44.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:44.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:44 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:44 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:44 smithi187 ceph-mon[20072]: osdmap e922: 12 total, 12 up, 11 in 2024-09-02T10:34:44.331 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:44.343 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:44 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:44.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:44 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:44.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:44 smithi162 ceph-mon[20279]: osdmap e922: 12 total, 12 up, 11 in 2024-09-02T10:34:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:44.353 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:44.353 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:44.434 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:44.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:44.474 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:44.476 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:44.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:44.608 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:44.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:44.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:44.738 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:44.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:44.827 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:34:44.827 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:34:44.827 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:34:44.827 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:34:44.827 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:34:44.839 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:44.866 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:44.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:44.968 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:44.995 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:44.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:44.997 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:45.003 INFO:teuthology.orchestra.run.smithi151.stderr:set pool 9 pg_num to 26 2024-09-02T10:34:45.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 45 16 59332 59316 82.376 58.0625 0.00402182 0.0121343 2024-09-02T10:34:45.065 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:34:45.065 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:34:45.091 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:34:45.091 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:34:45.098 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:45.118 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:34:45.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:34:45.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:45.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:45.145 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:34:45.145 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:34:45.171 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:34:45.171 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:34:45.172 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:45 smithi151 ceph-mon[16639]: pgmap v2114: 47 pgs: 47 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-02T10:34:45.172 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4090122311' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:34:45.198 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:34:45.198 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:34:45.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:45.229 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:34:45.230 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:34:45.253 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:45.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:45 smithi162 ceph-mon[20279]: pgmap v2114: 47 pgs: 47 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-02T10:34:45.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4090122311' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:34:45.355 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:34:45.355 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:34:45.355 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:45.379 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:34:45.379 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:34:45.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:45.382 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:45.404 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:34:45.404 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:34:45.434 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:34:45.434 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:34:45.458 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:34:45.459 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:34:45.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:45 smithi187 ceph-mon[20072]: pgmap v2114: 47 pgs: 47 active+clean; 3.4 GiB data, 17 GiB used, 966 GiB / 983 GiB avail; 88 MiB/s wr, 1.40k op/s 2024-09-02T10:34:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4090122311' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"} : dispatch 2024-09-02T10:34:45.506 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:45.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:45.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:45.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:45.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:45.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:45.732 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:45.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:45.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:45.810 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:34:45.856 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:45.877 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:45.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:45.878 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:45.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:34:45.979 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:46.004 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:46.005 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:46.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 46 16 60424 60408 82.0688 68.25 0.0206421 0.0121777 2024-09-02T10:34:46.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4090122311' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-09-02T10:34:46.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:46 smithi162 ceph-mon[20279]: osdmap e923: 12 total, 12 up, 11 in 2024-09-02T10:34:46.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3767362483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:46.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:46 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:34:46.106 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:46.149 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:46.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:46.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4090122311' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-09-02T10:34:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:46 smithi151 ceph-mon[16639]: osdmap e923: 12 total, 12 up, 11 in 2024-09-02T10:34:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3767362483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:46.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:46 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:34:46.192 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:34:46.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:46.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:46.273 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:46.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:46.374 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:46.396 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:46.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:46.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:46.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4090122311' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num", "val": "26"}]': finished 2024-09-02T10:34:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:46 smithi187 ceph-mon[20072]: osdmap e923: 12 total, 12 up, 11 in 2024-09-02T10:34:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3767362483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:46 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"} : dispatch 2024-09-02T10:34:46.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:46.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:46.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:46.621 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:46.642 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:46.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:46.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:46.744 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:46.764 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:46.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:46.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:46.854 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:46.866 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:46.887 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:46.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:46.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:46.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:47.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:47.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:47.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 47 16 60434 60418 80.3359 0.625 0.00300901 0.0122006 2024-09-02T10:34:47.115 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:47.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:47.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:47.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:47 smithi162 ceph-mon[20279]: pgmap v2116: 47 pgs: 47 active+clean; 3.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-02T10:34:47.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:47 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-02T10:34:47.174 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:47 smithi162 ceph-mon[20279]: osdmap e924: 12 total, 12 up, 11 in 2024-09-02T10:34:47.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:47.272 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:47.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:47.334 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:34:47.334 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:34:47.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:47 smithi151 ceph-mon[16639]: pgmap v2116: 47 pgs: 47 active+clean; 3.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-02T10:34:47.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:47 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-02T10:34:47.338 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:47 smithi151 ceph-mon[16639]: osdmap e924: 12 total, 12 up, 11 in 2024-09-02T10:34:47.374 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:47.403 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:47.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:47.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:47.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:47 smithi187 ceph-mon[20072]: pgmap v2116: 47 pgs: 47 active+clean; 3.5 GiB data, 18 GiB used, 966 GiB / 983 GiB avail; 79 MiB/s wr, 1.27k op/s 2024-09-02T10:34:47.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:47 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-02T10:34:47.510 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:47 smithi187 ceph-mon[20072]: osdmap e924: 12 total, 12 up, 11 in 2024-09-02T10:34:47.543 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:47.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:47.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:47.645 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:47.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:47.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:47.770 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:47.791 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:47.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:47.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:47.894 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:47.928 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:47.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:47.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:48.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 48 16 62008 61992 80.7116 98.375 0.00313176 0.0123852 2024-09-02T10:34:48.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:48.032 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:34:48.055 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:48.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:48.099 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:34:48.156 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:48.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:48.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:48.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:48.302 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:48.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:48.341 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:34:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:48 smithi162 ceph-mon[20279]: osdmap e925: 12 total, 12 up, 11 in 2024-09-02T10:34:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:48 smithi162 ceph-mon[20279]: 9.7 scrub starts 2024-09-02T10:34:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:48 smithi162 ceph-mon[20279]: 9.4 scrub starts 2024-09-02T10:34:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:48 smithi162 ceph-mon[20279]: 9.8 scrub starts 2024-09-02T10:34:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:48 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:34:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/744065055' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:48.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:48 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:48.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:48.425 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:48.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:48 smithi151 ceph-mon[16639]: osdmap e925: 12 total, 12 up, 11 in 2024-09-02T10:34:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:48 smithi151 ceph-mon[16639]: 9.7 scrub starts 2024-09-02T10:34:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:48 smithi151 ceph-mon[16639]: 9.4 scrub starts 2024-09-02T10:34:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:48 smithi151 ceph-mon[16639]: 9.8 scrub starts 2024-09-02T10:34:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:48 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:34:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/744065055' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:48 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:48.462 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:48 smithi187 ceph-mon[20072]: osdmap e925: 12 total, 12 up, 11 in 2024-09-02T10:34:48.462 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:48 smithi187 ceph-mon[20072]: 9.7 scrub starts 2024-09-02T10:34:48.462 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:48 smithi187 ceph-mon[20072]: 9.4 scrub starts 2024-09-02T10:34:48.462 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:48 smithi187 ceph-mon[20072]: 9.8 scrub starts 2024-09-02T10:34:48.462 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:48 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"} : dispatch 2024-09-02T10:34:48.463 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/744065055' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:48.463 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:48 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:34:48.527 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:48.549 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:48.550 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:48.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:48.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:48.673 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:48.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:48.735 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:34:48.775 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:48.796 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:48.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:48.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:48.898 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:48.920 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:48.921 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:49.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 49 16 62050 62034 79.118 2.625 0.00312003 0.0124312 2024-09-02T10:34:49.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:49.048 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:49.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:49.049 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:49.106 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:49.150 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:49.172 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:49.173 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:49.272 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: 9.1 scrub starts 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: pgmap v2119: 57 pgs: 2 peering, 9 unknown, 46 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: 9.6 scrub starts 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: 9.0 scrub starts 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: osdmap e926: 12 total, 12 up, 11 in 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: 9.9 scrub starts 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: 9.4 scrub starts 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: 9.8 scrub starts 2024-09-02T10:34:49.273 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:49 smithi151 ceph-mon[16639]: osdmap e927: 12 total, 12 up, 11 in 2024-09-02T10:34:49.274 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:49.295 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:49.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:49.313 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: 9.1 scrub starts 2024-09-02T10:34:49.313 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: pgmap v2119: 57 pgs: 2 peering, 9 unknown, 46 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-02T10:34:49.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: 9.6 scrub starts 2024-09-02T10:34:49.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: 9.0 scrub starts 2024-09-02T10:34:49.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:49.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-02T10:34:49.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:34:49.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: osdmap e926: 12 total, 12 up, 11 in 2024-09-02T10:34:49.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: 9.9 scrub starts 2024-09-02T10:34:49.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: 9.4 scrub starts 2024-09-02T10:34:49.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: 9.8 scrub starts 2024-09-02T10:34:49.314 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:49 smithi187 ceph-mon[20072]: osdmap e927: 12 total, 12 up, 11 in 2024-09-02T10:34:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: 9.1 scrub starts 2024-09-02T10:34:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: pgmap v2119: 57 pgs: 2 peering, 9 unknown, 46 active+clean; 3.6 GiB data, 18 GiB used, 965 GiB / 983 GiB avail; 62 MiB/s wr, 993 op/s 2024-09-02T10:34:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: 9.6 scrub starts 2024-09-02T10:34:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: 9.0 scrub starts 2024-09-02T10:34:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:34:49.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_7", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-02T10:34:49.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:34:49.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: osdmap e926: 12 total, 12 up, 11 in 2024-09-02T10:34:49.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: 9.9 scrub starts 2024-09-02T10:34:49.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: 9.4 scrub starts 2024-09-02T10:34:49.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: 9.8 scrub starts 2024-09-02T10:34:49.349 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:49 smithi162 ceph-mon[20279]: osdmap e927: 12 total, 12 up, 11 in 2024-09-02T10:34:49.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:49.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:49.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:49.424 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:49.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:49.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:49.562 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:49.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:49.574 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:34:49.574 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:34:49.663 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:49.694 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:49.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:49.796 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:49.823 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:49.824 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:49.854 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:34:49.854 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:34:49.901 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:34:49.901 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:34:49.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:49.947 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:34:49.947 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:34:49.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:49.991 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:49.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:49.991 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:34:49.991 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:34:50.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 50 16 62292 62276 77.8381 15.125 0.0277158 0.0128405 2024-09-02T10:34:50.030 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:34:50.031 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:34:50.072 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:34:50.073 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:34:50.092 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:50.108 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:34:50.108 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:34:50.116 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:50.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:50.117 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:50.145 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:34:50.145 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:34:50.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:34:50.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:34:50.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.2 scrub starts 2024-09-02T10:34:50.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.5 scrub starts 2024-09-02T10:34:50.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.4 scrub ok 2024-09-02T10:34:50.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.2 scrub ok 2024-09-02T10:34:50.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.0 scrub starts 2024-09-02T10:34:50.184 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.7 scrub starts 2024-09-02T10:34:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.9 scrub ok 2024-09-02T10:34:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.0 scrub ok 2024-09-02T10:34:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.5 scrub ok 2024-09-02T10:34:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.8 scrub ok 2024-09-02T10:34:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.7 scrub ok 2024-09-02T10:34:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: osdmap e928: 12 total, 12 up, 11 in 2024-09-02T10:34:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:34:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: 9.3 scrub starts 2024-09-02T10:34:50.185 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/505671799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:50.215 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:34:50.215 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:34:50.218 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:50.243 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:50.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:50.244 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:50.251 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:34:50.251 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:34:50.282 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:34:50.282 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.2 scrub starts 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.5 scrub starts 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.4 scrub ok 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.2 scrub ok 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.0 scrub starts 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.7 scrub starts 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.9 scrub ok 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.0 scrub ok 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.5 scrub ok 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.8 scrub ok 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.7 scrub ok 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: osdmap e928: 12 total, 12 up, 11 in 2024-09-02T10:34:50.295 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:34:50.296 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: 9.3 scrub starts 2024-09-02T10:34:50.296 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/505671799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:50.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:34:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.2 scrub starts 2024-09-02T10:34:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.5 scrub starts 2024-09-02T10:34:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.4 scrub ok 2024-09-02T10:34:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.2 scrub ok 2024-09-02T10:34:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.0 scrub starts 2024-09-02T10:34:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.7 scrub starts 2024-09-02T10:34:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.9 scrub ok 2024-09-02T10:34:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.0 scrub ok 2024-09-02T10:34:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.5 scrub ok 2024-09-02T10:34:50.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.8 scrub ok 2024-09-02T10:34:50.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.7 scrub ok 2024-09-02T10:34:50.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: osdmap e928: 12 total, 12 up, 11 in 2024-09-02T10:34:50.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:34:50.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: 9.3 scrub starts 2024-09-02T10:34:50.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/505671799' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:34:50.346 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:50.403 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:50.404 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:50.505 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:50.528 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:50.529 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:50.624 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:34:50.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:50.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:50.654 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:50.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:50.778 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:50.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:50.779 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:50.861 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 10, 4, 2] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:34:50.861 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:34:50.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:34:50.880 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:50.901 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:50.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:50.902 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:51.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:51.020 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 51 15 62990 62975 77.1684 43.6875 0.101543 0.0129413 2024-09-02T10:34:51.031 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:51.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:51.060 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:34:51.132 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:51.159 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:51.160 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:51.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:51 smithi151 ceph-mon[16639]: 9.1 scrub starts 2024-09-02T10:34:51.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:51 smithi151 ceph-mon[16639]: pgmap v2123: 57 pgs: 1 active+clean+scrubbing, 2 peering, 4 remapped+peering, 4 unknown, 46 active+clean; 3.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 18 KiB/s, 0 objects/s recovering 2024-09-02T10:34:51.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:51 smithi151 ceph-mon[16639]: 9.6 scrub starts 2024-09-02T10:34:51.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:51 smithi151 ceph-mon[16639]: 9.3 scrub ok 2024-09-02T10:34:51.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:34:51.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:34:51.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:51 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:34:51.195 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:34:51.196 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:34:51.196 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:34:51.196 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:34:51.196 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:34:51.261 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:51.286 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:51.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:51.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:51 smithi162 ceph-mon[20279]: 9.1 scrub starts 2024-09-02T10:34:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:51 smithi162 ceph-mon[20279]: pgmap v2123: 57 pgs: 1 active+clean+scrubbing, 2 peering, 4 remapped+peering, 4 unknown, 46 active+clean; 3.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 18 KiB/s, 0 objects/s recovering 2024-09-02T10:34:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:51 smithi162 ceph-mon[20279]: 9.6 scrub starts 2024-09-02T10:34:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:51 smithi162 ceph-mon[20279]: 9.3 scrub ok 2024-09-02T10:34:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:34:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:34:51.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:51 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:34:51.389 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:51.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:51.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:51 smithi187 ceph-mon[20072]: 9.1 scrub starts 2024-09-02T10:34:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:51 smithi187 ceph-mon[20072]: pgmap v2123: 57 pgs: 1 active+clean+scrubbing, 2 peering, 4 remapped+peering, 4 unknown, 46 active+clean; 3.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 18 KiB/s, 0 objects/s recovering 2024-09-02T10:34:51.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:51 smithi187 ceph-mon[20072]: 9.6 scrub starts 2024-09-02T10:34:51.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:51 smithi187 ceph-mon[20072]: 9.3 scrub ok 2024-09-02T10:34:51.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:34:51.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:34:51.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:51 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:34:51.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:51.550 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:51.551 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:51.652 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:51.675 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:51.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:51.777 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:51.802 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:51.803 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:51.889 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:34:51.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:51.957 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:51.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:51.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:52.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 52 16 64020 64004 76.9211 64.3125 0.0134609 0.0129949 2024-09-02T10:34:52.060 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:52.088 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:52.089 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:52.190 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:52.208 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:34:52.219 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:52.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:52.221 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:52 smithi187 ceph-mon[20072]: 9.6 scrub ok 2024-09-02T10:34:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:52 smithi187 ceph-mon[20072]: 9.1 scrub ok 2024-09-02T10:34:52.322 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:52.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:52 smithi162 ceph-mon[20279]: 9.6 scrub ok 2024-09-02T10:34:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:52 smithi162 ceph-mon[20279]: 9.1 scrub ok 2024-09-02T10:34:52.361 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:52.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:52.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:52 smithi151 ceph-mon[16639]: 9.6 scrub ok 2024-09-02T10:34:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:52 smithi151 ceph-mon[16639]: 9.1 scrub ok 2024-09-02T10:34:52.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:52.489 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:52.490 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:52.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:52.620 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:52.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:52.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:52.722 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:52.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:52.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:52.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:52.787 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:34:52.787 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:34:52.787 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55464 7.f 7:f848bf24:::benchmark_data_smithi187_120712_object55463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.558211+0000", 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 585.226777868, 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.219308693, 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:34:52.788 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55464 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558211+0000", 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558211+0000", 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.789 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558213+0000", 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5579999999999999e-06 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558264+0000", 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0971999999999999e-05 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558276+0000", 2024-09-02T10:34:52.790 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2336e-05 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.558283+0000", 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8410000000000002e-06 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.564067+0000", 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0057837560000000001 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.791 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.564243+0000", 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017600800000000001 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,9,11", 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.564442+0000", 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000199469 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.566944+0000", 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002502116 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.792 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.567038+0000", 2024-09-02T10:34:52.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3987999999999995e-05 2024-09-02T10:34:52.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.793 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.793 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.593916+0000", 2024-09-02T10:34:52.793 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.026877429000000001 2024-09-02T10:34:52.793 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.793 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.777338+0000", 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.18342222399999999 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.777398+0000", 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0353000000000003e-05 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.777520+0000", 2024-09-02T10:34:52.794 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000121643 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:136911 7.16 7.a2712c76 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:43.756684+0000", 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 489.0283048, 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.473282537, 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.795 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 136911 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756684+0000", 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756684+0000", 2024-09-02T10:34:52.796 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756686+0000", 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8729999999999999e-06 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756692+0000", 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1430000000000003e-06 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.797 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756705+0000", 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2863e-05 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756712+0000", 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7680000000000003e-06 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756763+0000", 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1360000000000003e-05 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.798 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.756772+0000", 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6119999999999999e-06 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.229956+0000", 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.473184084 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.229967+0000", 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0834e-05 2024-09-02T10:34:52.799 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:144199 7.7 e749/741 7:e22c0a38:::benchmark_data_smithi187_120712_object60257:head v 749'8678, mlcod=749'8675)", 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:59.587094+0000", 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 473.19789483599999, 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.249879298, 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.800 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.587094+0000", 2024-09-02T10:34:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.587094+0000", 2024-09-02T10:34:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.801 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.587095+0000", 2024-09-02T10:34:52.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2999999999999999e-07 2024-09-02T10:34:52.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.802 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.587102+0000", 2024-09-02T10:34:52.802 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5119999999999997e-06 2024-09-02T10:34:52.802 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.802 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.802 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.834486+0000", 2024-09-02T10:34:52.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.247384403 2024-09-02T10:34:52.803 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.803 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.803 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.803 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.834490+0000", 2024-09-02T10:34:52.803 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5210000000000004e-06 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.834533+0000", 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2959e-05 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.834547+0000", 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3328e-05 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.804 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.836929+0000", 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.002382103 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:59.836973+0000", 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4641999999999997e-05 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.805 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:134975 8.a 8:5dafd561:::benchmark_data_smithi187_136059_object29694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e842)", 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:01.809897+0000", 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 230.97509155500001, 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29535587200000002, 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 134975 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.806 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.809897+0000", 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.809897+0000", 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.809899+0000", 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.176e-06 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.807 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.809904+0000", 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5770000000000001e-06 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.809918+0000", 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3565e-05 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.809924+0000", 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6089999999999997e-06 2024-09-02T10:34:52.808 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.809975+0000", 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0677000000000003e-05 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.810139+0000", 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016422600000000001 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.810242+0000", 2024-09-02T10:34:52.809 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000103152 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.813920+0000", 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0036775060000000001 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814730+0000", 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00080953999999999998 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.810 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105190+0000", 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29046051000000001 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105225+0000", 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4820999999999997e-05 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105253+0000", 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8513e-05 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.811 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:134975 8.a e842/792 ondisk, result = 0)", 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:01.814451+0000", 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 230.97053782099999, 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29080757600000001, 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.812 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814451+0000", 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814451+0000", 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814452+0000", 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6799999999999999e-07 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.813 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.814457+0000", 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1139999999999997e-06 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105138+0000", 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.29068111400000002 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105143+0000", 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7870000000000001e-06 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.814 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105182+0000", 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8844999999999998e-05 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105187+0000", 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9049999999999996e-06 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105259+0000", 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1942999999999999e-05 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.815 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:134995 8.a 8:5fe3121c:::benchmark_data_smithi187_136059_object29714:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:02.101566+0000", 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 230.68342323300001, 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22082365700000001, 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:52.816 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 134995 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.101566+0000", 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.101566+0000", 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.817 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.101567+0000", 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1999999999999999e-06 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.101571+0000", 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.4599999999999996e-06 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.101583+0000", 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1447999999999999e-05 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.818 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.101589+0000", 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5990000000000001e-06 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.101646+0000", 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7046999999999998e-05 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.101733+0000", 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6964000000000003e-05 2024-09-02T10:34:52.819 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.101803+0000", 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0914999999999994e-05 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105061+0000", 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032580700000000001 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.820 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105293+0000", 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00023157200000000001 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.322305+0000", 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.217012174 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.322342+0000", 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6616000000000003e-05 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.821 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.322389+0000", 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7592000000000001e-05 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:134995 8.a e843/792 ondisk, result = 0)", 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:02.105117+0000", 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 230.679872321, 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21727786499999999, 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.822 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:52.823 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105117+0000", 2024-09-02T10:34:52.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.823 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.823 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.823 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.823 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105117+0000", 2024-09-02T10:34:52.823 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105118+0000", 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4900000000000005e-07 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105124+0000", 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6000000000000003e-06 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.824 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.322179+0000", 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21705460500000001 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.322191+0000", 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.181e-05 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.322295+0000", 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000104478 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.825 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.322302+0000", 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7150000000000001e-06 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.322395+0000", 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2807999999999996e-05 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.826 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:180229 8.a 8:5e771414:::benchmark_data_smithi187_136059_object74948:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e868)", 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:24.042114+0000", 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 148.742874661, 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.245006855, 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 180229 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.827 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.042114+0000", 2024-09-02T10:34:52.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.828 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.042114+0000", 2024-09-02T10:34:52.828 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.828 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.828 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.828 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.042117+0000", 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1500000000000002e-06 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.042126+0000", 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1880000000000008e-06 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.042145+0000", 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9230999999999999e-05 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.829 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.042155+0000", 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0428000000000001e-05 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.042220+0000", 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4212000000000001e-05 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.042348+0000", 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012863800000000001 2024-09-02T10:34:52.830 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:34:52.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.042466+0000", 2024-09-02T10:34:52.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011765900000000001 2024-09-02T10:34:52.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.831 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.831 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.831 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.052199+0000", 2024-09-02T10:34:52.831 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0097331919999999999 2024-09-02T10:34:52.831 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.084026+0000", 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.031826884 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.287017+0000", 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.202991485 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.832 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.287077+0000", 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9916e-05 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:24.287121+0000", 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3872000000000003e-05 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:180980 8.b 8:d933f37f:::benchmark_data_smithi187_136059_object75699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e868)", 2024-09-02T10:34:52.833 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:25.352845+0000", 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 147.43214361099999, 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21205447699999999, 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 180980 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.834 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.352845+0000", 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.352845+0000", 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.352847+0000", 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.885e-06 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.835 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.352856+0000", 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2800000000000003e-06 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.352871+0000", 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5580999999999999e-05 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.352880+0000", 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5809999999999997e-06 2024-09-02T10:34:52.836 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.352943+0000", 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3004999999999997e-05 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.353080+0000", 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013768299999999999 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:34:52.837 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.353205+0000", 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012464600000000001 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.357926+0000", 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047204719999999999 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.357987+0000", 2024-09-02T10:34:52.838 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0992999999999997e-05 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.564798+0000", 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20681117399999999 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.564859+0000", 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1531000000000004e-05 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.839 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:25.564900+0000", 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0645999999999997e-05 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:183127 8.a 8:57a34520:::benchmark_data_smithi187_136059_object77846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:29.327188+0000", 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 143.45780133299999, 2024-09-02T10:34:52.840 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.230357273, 2024-09-02T10:34:52.841 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183127 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.327188+0000", 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.842 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.327188+0000", 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.327189+0000", 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7400000000000001e-06 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.327197+0000", 2024-09-02T10:34:52.843 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.486e-06 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.327210+0000", 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3514e-05 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.327218+0000", 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9470000000000003e-06 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.844 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.327288+0000", 2024-09-02T10:34:52.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9332000000000006e-05 2024-09-02T10:34:52.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.845 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.845 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.845 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.327421+0000", 2024-09-02T10:34:52.845 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013279899999999999 2024-09-02T10:34:52.845 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:34:52.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.327546+0000", 2024-09-02T10:34:52.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012543600000000001 2024-09-02T10:34:52.846 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.846 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.846 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.846 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.338856+0000", 2024-09-02T10:34:52.846 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.011310008 2024-09-02T10:34:52.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.338873+0000", 2024-09-02T10:34:52.847 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7333000000000002e-05 2024-09-02T10:34:52.847 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.847 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.847 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.847 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.557448+0000", 2024-09-02T10:34:52.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.218574817 2024-09-02T10:34:52.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.557497+0000", 2024-09-02T10:34:52.848 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8868e-05 2024-09-02T10:34:52.848 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.848 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.848 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.848 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.557545+0000", 2024-09-02T10:34:52.849 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7992999999999999e-05 2024-09-02T10:34:52.849 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.849 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.849 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.849 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.849 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.851 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:183130 8.a 8:5b1048a3:::benchmark_data_smithi187_136059_object77849:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-09-02T10:34:52.851 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:29.330618+0000", 2024-09-02T10:34:52.851 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 143.45437074, 2024-09-02T10:34:52.851 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22712231199999999, 2024-09-02T10:34:52.851 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.852 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.852 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.852 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.852 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:52.852 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:52.852 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183130 2024-09-02T10:34:52.852 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.852 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.853 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.853 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.330618+0000", 2024-09-02T10:34:52.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.853 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.853 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.853 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.853 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.330618+0000", 2024-09-02T10:34:52.853 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.330621+0000", 2024-09-02T10:34:52.854 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3489999999999999e-06 2024-09-02T10:34:52.854 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.854 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.854 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.854 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.330628+0000", 2024-09-02T10:34:52.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3590000000000002e-06 2024-09-02T10:34:52.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.330649+0000", 2024-09-02T10:34:52.855 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1077000000000002e-05 2024-09-02T10:34:52.855 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.855 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.855 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.855 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.330659+0000", 2024-09-02T10:34:52.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6549999999999999e-06 2024-09-02T10:34:52.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.856 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.333929+0000", 2024-09-02T10:34:52.856 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0032702510000000001 2024-09-02T10:34:52.856 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.856 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.856 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.334066+0000", 2024-09-02T10:34:52.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013670300000000001 2024-09-02T10:34:52.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:34:52.857 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.334193+0000", 2024-09-02T10:34:52.857 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012778 2024-09-02T10:34:52.857 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.857 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.857 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.338911+0000", 2024-09-02T10:34:52.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0047180039999999996 2024-09-02T10:34:52.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.858 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.858 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.338929+0000", 2024-09-02T10:34:52.858 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7138000000000001e-05 2024-09-02T10:34:52.858 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.858 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.557641+0000", 2024-09-02T10:34:52.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.218712552 2024-09-02T10:34:52.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.859 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.859 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.557697+0000", 2024-09-02T10:34:52.859 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6289999999999998e-05 2024-09-02T10:34:52.859 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.859 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.860 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.860 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.557741+0000", 2024-09-02T10:34:52.860 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3154000000000001e-05 2024-09-02T10:34:52.860 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.860 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.860 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.860 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.860 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.860 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:186203 8.b 8:d63983d6:::benchmark_data_smithi187_136059_object80922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e870)", 2024-09-02T10:34:52.861 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:34.709034+0000", 2024-09-02T10:34:52.861 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 138.07595464799999, 2024-09-02T10:34:52.861 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24847477800000001, 2024-09-02T10:34:52.861 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.861 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.861 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.861 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.861 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:52.861 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:52.862 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 186203 2024-09-02T10:34:52.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.862 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.862 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.709034+0000", 2024-09-02T10:34:52.862 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.862 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.862 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.862 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.709034+0000", 2024-09-02T10:34:52.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.863 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.709036+0000", 2024-09-02T10:34:52.863 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.55e-06 2024-09-02T10:34:52.863 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.863 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.863 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.709043+0000", 2024-09-02T10:34:52.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7109999999999998e-06 2024-09-02T10:34:52.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.864 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.864 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.709057+0000", 2024-09-02T10:34:52.864 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3896999999999999e-05 2024-09-02T10:34:52.864 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.864 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.865 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.865 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.709065+0000", 2024-09-02T10:34:52.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0490000000000004e-06 2024-09-02T10:34:52.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.865 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.865 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.709113+0000", 2024-09-02T10:34:52.865 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8565e-05 2024-09-02T10:34:52.865 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.865 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.709203+0000", 2024-09-02T10:34:52.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9796999999999994e-05 2024-09-02T10:34:52.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.866 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:34:52.866 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.709279+0000", 2024-09-02T10:34:52.866 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6260000000000005e-05 2024-09-02T10:34:52.866 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.866 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.713174+0000", 2024-09-02T10:34:52.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003894471 2024-09-02T10:34:52.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.867 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.867 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.867 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.713911+0000", 2024-09-02T10:34:52.867 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00073766900000000004 2024-09-02T10:34:52.867 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.957417+0000", 2024-09-02T10:34:52.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24350593500000001 2024-09-02T10:34:52.868 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.868 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.868 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.868 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.957467+0000", 2024-09-02T10:34:52.868 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9904999999999997e-05 2024-09-02T10:34:52.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.869 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.869 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:34.957509+0000", 2024-09-02T10:34:52.869 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1968999999999997e-05 2024-09-02T10:34:52.869 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.869 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.869 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.869 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.869 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.870 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:207405 8.a 8:591bfba2:::benchmark_data_smithi187_136059_object102124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:34:52.870 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:11.381725+0000", 2024-09-02T10:34:52.870 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 101.403263817, 2024-09-02T10:34:52.870 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.239265121, 2024-09-02T10:34:52.870 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.870 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.870 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.870 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.870 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:52.870 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:52.871 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207405 2024-09-02T10:34:52.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.871 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.871 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.871 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381725+0000", 2024-09-02T10:34:52.871 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.871 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.871 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381725+0000", 2024-09-02T10:34:52.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.872 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.872 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.872 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381727+0000", 2024-09-02T10:34:52.872 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.95e-06 2024-09-02T10:34:52.872 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381735+0000", 2024-09-02T10:34:52.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2150000000000001e-06 2024-09-02T10:34:52.873 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.873 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.873 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.873 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381754+0000", 2024-09-02T10:34:52.873 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8964999999999999e-05 2024-09-02T10:34:52.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381765+0000", 2024-09-02T10:34:52.874 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0531e-05 2024-09-02T10:34:52.874 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.874 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.874 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.874 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381824+0000", 2024-09-02T10:34:52.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8879999999999999e-05 2024-09-02T10:34:52.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.875 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381938+0000", 2024-09-02T10:34:52.875 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000114046 2024-09-02T10:34:52.875 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.875 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.875 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:34:52.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.382076+0000", 2024-09-02T10:34:52.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000138362 2024-09-02T10:34:52.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.876 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.876 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.387022+0000", 2024-09-02T10:34:52.876 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0049459559999999996 2024-09-02T10:34:52.876 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.876 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.387115+0000", 2024-09-02T10:34:52.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3067999999999997e-05 2024-09-02T10:34:52.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.877 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.877 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.620880+0000", 2024-09-02T10:34:52.877 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23376439099999999 2024-09-02T10:34:52.877 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.877 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.620944+0000", 2024-09-02T10:34:52.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4524999999999996e-05 2024-09-02T10:34:52.878 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.878 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.878 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.878 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.620990+0000", 2024-09-02T10:34:52.878 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6232000000000001e-05 2024-09-02T10:34:52.878 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.879 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.879 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.879 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.879 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.879 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGRecoveryDelete(8.8 e883,854 8:1f451c45:::benchmark_data_smithi187_136059_object4240:head,836'6950)", 2024-09-02T10:34:52.879 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:20.481817+0000", 2024-09-02T10:34:52.879 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 92.303172469000003, 2024-09-02T10:34:52.879 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98829783999999998, 2024-09-02T10:34:52.879 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.880 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.880 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:52.880 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.880 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.880 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.880 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:20.481817+0000", 2024-09-02T10:34:52.880 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.880 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:20.481817+0000", 2024-09-02T10:34:52.881 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.881 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.881 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.881 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.881 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:20.481818+0000", 2024-09-02T10:34:52.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.457e-06 2024-09-02T10:34:52.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.882 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.882 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:20.481828+0000", 2024-09-02T10:34:52.882 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6900000000000004e-06 2024-09-02T10:34:52.882 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.882 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.469733+0000", 2024-09-02T10:34:52.883 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.98790496500000002 2024-09-02T10:34:52.883 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.883 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.883 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.883 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.469750+0000", 2024-09-02T10:34:52.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.774e-05 2024-09-02T10:34:52.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.884 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.884 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.884 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.469943+0000", 2024-09-02T10:34:52.884 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00019263099999999999 2024-09-02T10:34:52.884 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.469972+0000", 2024-09-02T10:34:52.885 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9356000000000002e-05 2024-09-02T10:34:52.885 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.885 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.885 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.885 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.470114+0000", 2024-09-02T10:34:52.886 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000142001 2024-09-02T10:34:52.886 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.886 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.886 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.886 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.886 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.886 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27148 9.b 9:db6b1465:::benchmark_data_smithi187_154927_object27147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:34:52.887 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.006945+0000", 2024-09-02T10:34:52.887 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 35.778044403999999, 2024-09-02T10:34:52.887 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.225576742, 2024-09-02T10:34:52.887 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.887 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.888 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.888 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.888 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:34:52.888 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:52.888 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27148 2024-09-02T10:34:52.888 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.888 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.889 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.006945+0000", 2024-09-02T10:34:52.889 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.889 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.889 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.889 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.006945+0000", 2024-09-02T10:34:52.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.890 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.890 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.890 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.006946+0000", 2024-09-02T10:34:52.890 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2399999999999996e-07 2024-09-02T10:34:52.890 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.006965+0000", 2024-09-02T10:34:52.891 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9094999999999999e-05 2024-09-02T10:34:52.891 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.891 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.891 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.891 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.006975+0000", 2024-09-02T10:34:52.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0614999999999999e-05 2024-09-02T10:34:52.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.892 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.892 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.892 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.006980+0000", 2024-09-02T10:34:52.892 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0649999999999998e-06 2024-09-02T10:34:52.892 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.893 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.007020+0000", 2024-09-02T10:34:52.893 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9761000000000001e-05 2024-09-02T10:34:52.893 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.893 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.893 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.007132+0000", 2024-09-02T10:34:52.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011223800000000001 2024-09-02T10:34:52.894 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.894 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.894 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,10", 2024-09-02T10:34:52.894 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.007228+0000", 2024-09-02T10:34:52.894 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5882000000000001e-05 2024-09-02T10:34:52.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.895 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.895 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008330+0000", 2024-09-02T10:34:52.895 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001102245 2024-09-02T10:34:52.895 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.895 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008711+0000", 2024-09-02T10:34:52.896 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00038042099999999998 2024-09-02T10:34:52.896 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.896 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.896 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.896 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.232454+0000", 2024-09-02T10:34:52.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.22374355600000001 2024-09-02T10:34:52.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.897 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.897 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.232491+0000", 2024-09-02T10:34:52.897 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6847999999999997e-05 2024-09-02T10:34:52.897 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.897 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.898 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.898 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.232521+0000", 2024-09-02T10:34:52.898 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.0091999999999999e-05 2024-09-02T10:34:52.898 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.898 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.898 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.898 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.899 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.899 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27194 9.b 9:dad5b9a8:::benchmark_data_smithi187_154927_object27193:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:34:52.899 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.230754+0000", 2024-09-02T10:34:52.899 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 35.554235179000003, 2024-09-02T10:34:52.899 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.264560564, 2024-09-02T10:34:52.899 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.899 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.899 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.900 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.900 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:34:52.900 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:52.900 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27194 2024-09-02T10:34:52.900 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.900 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.900 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.230754+0000", 2024-09-02T10:34:52.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.901 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.901 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.901 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.901 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.230754+0000", 2024-09-02T10:34:52.901 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.902 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.230755+0000", 2024-09-02T10:34:52.902 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.302e-06 2024-09-02T10:34:52.902 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.902 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.902 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.230776+0000", 2024-09-02T10:34:52.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0463e-05 2024-09-02T10:34:52.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.903 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.903 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.903 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.230788+0000", 2024-09-02T10:34:52.903 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2619999999999999e-05 2024-09-02T10:34:52.903 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.904 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.230794+0000", 2024-09-02T10:34:52.904 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4280000000000004e-06 2024-09-02T10:34:52.904 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.904 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.904 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.230856+0000", 2024-09-02T10:34:52.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2445000000000003e-05 2024-09-02T10:34:52.905 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.905 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.905 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.905 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.230980+0000", 2024-09-02T10:34:52.905 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000123733 2024-09-02T10:34:52.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.906 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,10", 2024-09-02T10:34:52.906 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.231140+0000", 2024-09-02T10:34:52.906 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000160161 2024-09-02T10:34:52.906 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.906 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.232177+0000", 2024-09-02T10:34:52.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010371009999999999 2024-09-02T10:34:52.907 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.907 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.907 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.907 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.232588+0000", 2024-09-02T10:34:52.907 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00041109000000000002 2024-09-02T10:34:52.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.908 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.908 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.495229+0000", 2024-09-02T10:34:52.908 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26264104300000002 2024-09-02T10:34:52.908 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.908 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.495266+0000", 2024-09-02T10:34:52.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7168000000000001e-05 2024-09-02T10:34:52.909 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.909 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.909 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.909 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.495314+0000", 2024-09-02T10:34:52.909 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8010000000000003e-05 2024-09-02T10:34:52.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.910 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.910 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.910 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.910 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.910 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27198 9.b 9:d0c8a230:::benchmark_data_smithi187_154927_object27197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:34:52.910 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.238773+0000", 2024-09-02T10:34:52.911 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 35.546216272999999, 2024-09-02T10:34:52.911 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27268435499999999, 2024-09-02T10:34:52.911 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.911 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.911 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:34:52.911 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27198 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.238773+0000", 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.912 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.238773+0000", 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.238774+0000", 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5300000000000003e-07 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.238793+0000", 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9267999999999999e-05 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.913 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.238803+0000", 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0623e-05 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.238809+0000", 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1379999999999998e-06 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.238868+0000", 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9840999999999998e-05 2024-09-02T10:34:52.914 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.239002+0000", 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013354 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,10", 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.239157+0000", 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015504900000000001 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.239889+0000", 2024-09-02T10:34:52.915 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00073237699999999998 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.240282+0000", 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00039251899999999997 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511293+0000", 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27101079900000002 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.916 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511385+0000", 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2244999999999995e-05 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511457+0000", 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2102999999999997e-05 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.28030.0:27194 9.b e912/897 ondisk, result = 0)", 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.240130+0000", 2024-09-02T10:34:52.917 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 35.544859215999999, 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25519113599999999, 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.240130+0000", 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.918 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.240130+0000", 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.240130+0000", 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5499999999999998e-07 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.240134+0000", 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4240000000000002e-06 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.919 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.495168+0000", 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25503383400000001 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.495173+0000", 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2399999999999998e-06 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.920 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.495220+0000", 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6915000000000001e-05 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.495226+0000", 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0850000000000002e-06 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.495321+0000", 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5082999999999997e-05 2024-09-02T10:34:52.921 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.28030.0:41645 9.4 e916/897 9:238711ac:::benchmark_data_smithi187_154927_object41644:head v 916'2611, mlcod=916'2608)", 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:30.364000+0000", 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 22.420989061, 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21375677400000001, 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.922 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:30.364000+0000", 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:30.364000+0000", 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:30.364001+0000", 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6400000000000002e-07 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.923 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:30.364046+0000", 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5769999999999997e-05 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:30.576079+0000", 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21203285999999999 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:30.576086+0000", 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4679999999999997e-06 2024-09-02T10:34:52.924 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:30.576176+0000", 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0514999999999997e-05 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:30.576196+0000", 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9599999999999999e-05 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:30.577725+0000", 2024-09-02T10:34:52.925 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0015291980000000001 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:30.577757+0000", 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1699000000000003e-05 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:55144 9.b 9:da61d72e:::benchmark_data_smithi187_154927_object55143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e921)", 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:41.667543+0000", 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 11.117446162, 2024-09-02T10:34:52.926 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.14073524500000001, 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55144 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.667543+0000", 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.927 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.667543+0000", 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.667544+0000", 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.144e-06 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.667564+0000", 2024-09-02T10:34:52.928 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0017000000000002e-05 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.667576+0000", 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1800999999999999e-05 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.667581+0000", 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3079999999999998e-06 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:52.929 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.667614+0000", 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3191000000000001e-05 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.667709+0000", 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4771000000000005e-05 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,10", 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.667802+0000", 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2707000000000005e-05 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.930 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.669311+0000", 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00150906 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.669329+0000", 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7974000000000001e-05 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.931 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.808199+0000", 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.138870362 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.808235+0000", 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6142000000000001e-05 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:41.808278+0000", 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2768000000000001e-05 2024-09-02T10:34:52.932 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.933 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.933 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:52.933 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:52.933 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:34:52.933 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:52.966 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:52.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:52.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:53.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 53 16 64945 64929 76.5605 57.8125 0.00665892 0.0130545 2024-09-02T10:34:53.069 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:53.095 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:53.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:53.096 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:53.135 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:53.182 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:34:53.197 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:53.223 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:53.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:53.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:53 smithi151 ceph-mon[16639]: pgmap v2124: 57 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 peering, 4 remapped+peering, 1 unknown, 45 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 11776/178947 objects degraded (6.581%); 7895/178947 objects misplaced (4.412%); 28 KiB/s, 0 objects/s recovering 2024-09-02T10:34:53.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:53 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 11776/178947 objects degraded (6.581%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:34:53.325 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:53.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:53 smithi162 ceph-mon[20279]: pgmap v2124: 57 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 peering, 4 remapped+peering, 1 unknown, 45 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 11776/178947 objects degraded (6.581%); 7895/178947 objects misplaced (4.412%); 28 KiB/s, 0 objects/s recovering 2024-09-02T10:34:53.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:53 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 11776/178947 objects degraded (6.581%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:34:53.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:53.352 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:53.365 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:53 smithi187 ceph-mon[20072]: pgmap v2124: 57 pgs: 1 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing, 2 peering, 4 remapped+peering, 1 unknown, 45 active+clean; 3.6 GiB data, 19 GiB used, 965 GiB / 983 GiB avail; 11776/178947 objects degraded (6.581%); 7895/178947 objects misplaced (4.412%); 28 KiB/s, 0 objects/s recovering 2024-09-02T10:34:53.365 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:53 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 11776/178947 objects degraded (6.581%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:34:53.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:53.476 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:53.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:53.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:53.577 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:53.608 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:53.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:53.710 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:53.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:53.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:53.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:53.870 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:53.871 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:53.972 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:53.997 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:53.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:53.999 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:54.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 54 16 65779 65763 76.1079 52.125 0.0143597 0.0131336 2024-09-02T10:34:54.099 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:54.135 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:54.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:54.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:54.150 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:34:54.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:54.267 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:54.267 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:54.369 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:54.401 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:54.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:54.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:54.515 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:34:54.536 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:54.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:54.638 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:54.663 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:54.664 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:54.765 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:54.797 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:54.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:54.899 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:54.931 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:54.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:55.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 55 16 66759 66743 75.8377 61.25 0.0446163 0.01318 2024-09-02T10:34:55.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:55.054 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:34:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:34:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:66788 9.3 9:c1d10dbe:::benchmark_data_smithi187_154927_object66787:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e928)", 2024-09-02T10:34:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:55.050825+0000", 2024-09-02T10:34:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 0.00099574400000000001, 2024-09-02T10:34:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001028097, 2024-09-02T10:34:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:55.054 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 66788 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:55.050825+0000", 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:55.055 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:55.050825+0000", 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:55.050828+0000", 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7989999999999998e-06 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:55.050864+0000", 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6133000000000001e-05 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:55.056 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:55.050884+0000", 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.003e-05 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:55.050894+0000", 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.91e-06 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:55.050960+0000", 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5796000000000005e-05 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:55.057 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:55.051135+0000", 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017552900000000001 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,5", 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:55.051285+0000", 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000149865 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: ], 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 1 2024-09-02T10:34:55.058 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:34:55.062 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:55.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:55.063 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:55.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:34:55.164 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:55.188 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:55.189 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:55 smithi187 ceph-mon[20072]: pgmap v2125: 57 pgs: 3 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 47 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 11776/189495 objects degraded (6.214%); 13296/189495 objects misplaced (7.017%) 2024-09-02T10:34:55.291 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:55.316 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:55.317 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:55.344 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:55 smithi162 ceph-mon[20279]: pgmap v2125: 57 pgs: 3 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 47 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 11776/189495 objects degraded (6.214%); 13296/189495 objects misplaced (7.017%) 2024-09-02T10:34:55.377 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:55 smithi151 ceph-mon[16639]: pgmap v2125: 57 pgs: 3 active+recovering+undersized+remapped, 2 activating+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 remapped+peering, 47 active+clean; 3.9 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 11776/189495 objects degraded (6.214%); 13296/189495 objects misplaced (7.017%) 2024-09-02T10:34:55.418 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:55.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:55.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:55.444 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:55.545 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:55.583 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:55.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:55.602 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:34:55.602 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:34:55.684 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:55.710 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:55.711 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:55.737 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:34:55.812 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:55.838 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:55.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:55.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:55.940 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:55.971 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:55.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:56.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 56 15 67718 67703 75.5548 60 0.00710048 0.0132287 2024-09-02T10:34:56.072 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:56.077 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:34:56.097 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:56.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:56.098 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:56.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:56.225 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:56.226 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:56.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:56.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1712774773' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:56.340 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:56 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:56.349 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1712774773' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:56.349 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:56 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:56.366 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:56.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:56.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1712774773' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:56 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:34:56.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:56.502 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:56.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:56.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:56.604 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:56.605 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:34:56.615 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:34:56.615 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:34:56.615 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:34:56.615 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:34:56.615 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:34:56.640 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:56.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:56.742 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:56.781 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:56.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:56.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:56.883 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:34:56.911 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:56.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:34:57.013 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:57.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 57 16 68563 68547 75.1546 52.75 0.103195 0.0132742 2024-09-02T10:34:57.036 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:57.037 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:57.138 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:57.168 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:57.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:57.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:57 smithi187 ceph-mon[20072]: from='client.18951 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:57 smithi187 ceph-mon[20072]: pgmap v2126: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 9842/195771 objects degraded (5.027%); 26756/195771 objects misplaced (13.667%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:34:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:57 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:34:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:57 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:57 smithi187 ceph-mon[20072]: osdmap e929: 12 total, 12 up, 11 in 2024-09-02T10:34:57.270 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:57.294 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:57.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:57.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:34:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:57 smithi162 ceph-mon[20279]: from='client.18951 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:57 smithi162 ceph-mon[20279]: pgmap v2126: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 9842/195771 objects degraded (5.027%); 26756/195771 objects misplaced (13.667%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:34:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:57 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:34:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:57 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:57.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:57 smithi162 ceph-mon[20279]: osdmap e929: 12 total, 12 up, 11 in 2024-09-02T10:34:57.396 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:34:57.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:57.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:57.423 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:34:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:57 smithi151 ceph-mon[16639]: from='client.18951 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:34:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:57 smithi151 ceph-mon[16639]: pgmap v2126: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.0 GiB data, 19 GiB used, 964 GiB / 983 GiB avail; 9842/195771 objects degraded (5.027%); 26756/195771 objects misplaced (13.667%); 6.3 MiB/s, 100 objects/s recovering 2024-09-02T10:34:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:57 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:34:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:57 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:34:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:57 smithi151 ceph-mon[16639]: osdmap e929: 12 total, 12 up, 11 in 2024-09-02T10:34:57.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:57.557 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:57.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:57.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:57.616 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:34:57.662 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:57.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:57.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:57.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:57.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:57.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:57.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:57.956 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:57.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:57.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:58.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 58 16 69512 69496 74.8814 59.3125 0.0219666 0.0133489 2024-09-02T10:34:58.058 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:34:58.085 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:58.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:58.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:34:58.188 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:58.189 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:34:58.189 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:34:58.189 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55524 7.a 7:5888f7f6:::benchmark_data_smithi187_120712_object55523:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.784823+0000", 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 590.40119807799999, 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25595624, 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:58.190 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:34:58.191 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:58.191 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55524 2024-09-02T10:34:58.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.191 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.191 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.191 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784823+0000", 2024-09-02T10:34:58.191 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.191 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.191 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784823+0000", 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784825+0000", 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.289e-06 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.192 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784866+0000", 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0327000000000002e-05 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784884+0000", 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7697999999999999e-05 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784893+0000", 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5510000000000005e-06 2024-09-02T10:34:58.193 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.784968+0000", 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4975999999999995e-05 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.785119+0000", 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000151404 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.194 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:34:58.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.785270+0000", 2024-09-02T10:34:58.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00015045799999999999 2024-09-02T10:34:58.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:58.195 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.787505+0000", 2024-09-02T10:34:58.195 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0022353439999999998 2024-09-02T10:34:58.195 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.195 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.195 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.787795+0000", 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00028950500000000001 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040702+0000", 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25290737699999999 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040745+0000", 2024-09-02T10:34:58.196 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.337e-05 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040779+0000", 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3940999999999999e-05 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:55525 7.5 e696/690 7:a5f161ff:::benchmark_data_smithi187_120712_object55524:head v 696'3504, mlcod=696'3501)", 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.788514+0000", 2024-09-02T10:34:58.197 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 590.39750755199998, 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.2522201, 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788514+0000", 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.198 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788514+0000", 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788515+0000", 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9299999999999996e-07 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.199 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788550+0000", 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5129000000000002e-05 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788586+0000", 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6294000000000002e-05 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788593+0000", 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9299999999999997e-06 2024-09-02T10:34:58.200 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788652+0000", 2024-09-02T10:34:58.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9002999999999997e-05 2024-09-02T10:34:58.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.201 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.201 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.201 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788676+0000", 2024-09-02T10:34:58.201 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.4290999999999998e-05 2024-09-02T10:34:58.201 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040675+0000", 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25199830000000001 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040734+0000", 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9259999999999998e-05 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.202 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:55524 7.a e696/661 ondisk, result = 0)", 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.788585+0000", 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 590.39743656500002, 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25220463100000001, 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.203 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788585+0000", 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788585+0000", 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788586+0000", 2024-09-02T10:34:58.204 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1399999999999996e-07 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.788593+0000", 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5510000000000002e-06 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040578+0000", 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25198481900000003 2024-09-02T10:34:58.205 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.206 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040594+0000", 2024-09-02T10:34:58.206 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6563e-05 2024-09-02T10:34:58.206 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.206 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.206 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040692+0000", 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7800999999999994e-05 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040698+0000", 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8390000000000003e-06 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040789+0000", 2024-09-02T10:34:58.207 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1243999999999996e-05 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55529 7.a 7:5c0a98d2:::benchmark_data_smithi187_120712_object55528:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.789084+0000", 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 590.39693738899996, 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25619412899999999, 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.208 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55529 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789084+0000", 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.209 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789084+0000", 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789086+0000", 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3910000000000001e-06 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.210 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789109+0000", 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2734e-05 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789121+0000", 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2435e-05 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789129+0000", 2024-09-02T10:34:58.211 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.52e-06 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789235+0000", 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010628000000000001 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789361+0000", 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000125351 2024-09-02T10:34:58.212 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789484+0000", 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012378000000000001 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040657+0000", 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25117236700000001 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.213 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040831+0000", 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017463 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045180+0000", 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043483080000000004 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.214 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045232+0000", 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2074000000000003e-05 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045278+0000", 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6258999999999999e-05 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.215 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:55530 7.4 e696/690 7:2b921397:::benchmark_data_smithi187_120712_object55529:head v 696'3599, mlcod=696'3597)", 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.789951+0000", 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 590.39607074399999, 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.2507779, 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789951+0000", 2024-09-02T10:34:58.216 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789951+0000", 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.789954+0000", 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1530000000000001e-06 2024-09-02T10:34:58.217 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790032+0000", 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8168000000000001e-05 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790048+0000", 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5862000000000002e-05 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.218 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790054+0000", 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.584e-06 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790120+0000", 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5668999999999998e-05 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.219 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.790141+0000", 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0818999999999998e-05 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040685+0000", 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25054374800000001 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040728+0000", 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3896999999999997e-05 2024-09-02T10:34:58.220 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:55529 7.a e696/661 ondisk, result = 0)", 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.791795+0000", 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 590.39422605100003, 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25349172599999997, 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.221 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791795+0000", 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791795+0000", 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.222 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791796+0000", 2024-09-02T10:34:58.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9599999999999999e-07 2024-09-02T10:34:58.223 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.223 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.223 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.223 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.791803+0000", 2024-09-02T10:34:58.223 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9299999999999997e-06 2024-09-02T10:34:58.225 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.225 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.225 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040780+0000", 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.248977435 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040785+0000", 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9910000000000002e-06 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.226 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045163+0000", 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0043775949999999998 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045171+0000", 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2109999999999998e-06 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045287+0000", 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115868 2024-09-02T10:34:58.227 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:55529 7.a e696/661 ondisk, result = 0)", 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.793833+0000", 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 590.39218839900002, 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24701554000000001, 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:58.228 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793833+0000", 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793833+0000", 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.229 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793834+0000", 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2900000000000003e-07 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.793838+0000", 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8720000000000001e-06 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.230 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040586+0000", 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.246747148 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040598+0000", 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1943999999999999e-05 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040817+0000", 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00021921700000000001 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.231 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040825+0000", 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5939999999999994e-06 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040848+0000", 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3036000000000001e-05 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.232 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:55539 7.a 7:5096ce06:::benchmark_data_smithi187_120712_object55538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e696)", 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:07.802709+0000", 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 590.383312542, 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.24498555799999999, 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:34:58.233 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 55539 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802709+0000", 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802709+0000", 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.234 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802711+0000", 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5840000000000002e-06 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:07.802737+0000", 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5948e-05 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.235 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040797+0000", 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23805955500000001 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040805+0000", 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9629999999999998e-06 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.236 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.040878+0000", 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.305e-05 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.041029+0000", 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000151337 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.041152+0000", 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012326200000000001 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.237 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045348+0000", 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0041960260000000003 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.045394+0000", 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.587e-05 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:58.238 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.047524+0000", 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021297120000000002 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.047622+0000", 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8387999999999994e-05 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:08.047694+0000", 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1863e-05 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.239 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:71694 7.a 7:589b7e94:::benchmark_data_smithi187_120712_object71693:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e711)", 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:28.310151+0000", 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 569.87586994200001, 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31770000700000001, 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.240 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 71694 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310151+0000", 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.241 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310151+0000", 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310153+0000", 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5910000000000001e-06 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.242 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310173+0000", 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.052e-05 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310187+0000", 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.397e-05 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310194+0000", 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1580000000000002e-06 2024-09-02T10:34:58.243 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310240+0000", 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6564999999999999e-05 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310346+0000", 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010611999999999999 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.244 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,11", 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.310438+0000", 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2083999999999995e-05 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312967+0000", 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025282730000000002 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.245 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.313222+0000", 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025570999999999998 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627778+0000", 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31455519399999998 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.246 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627814+0000", 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6374000000000001e-05 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627851+0000", 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7447999999999998e-05 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.247 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:71694 7.a e711/661 ondisk, result = 0)", 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:28.312737+0000", 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 569.87328392100005, 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.315119227, 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.248 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312737+0000", 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312737+0000", 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312738+0000", 2024-09-02T10:34:58.249 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1500000000000003e-07 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.312742+0000", 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0389999999999998e-06 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627726+0000", 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31498364600000001 2024-09-02T10:34:58.250 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627732+0000", 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0789999999999997e-06 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627771+0000", 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8642e-05 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.251 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627774+0000", 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9060000000000004e-06 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.627857+0000", 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2100000000000003e-05 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.252 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:99751 7.e e722/661 7:70408914:::benchmark_data_smithi187_120712_object15809:head v 722'6054, mlcod=722'6050)", 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:00.293617+0000", 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 537.89240442599998, 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26051671900000001, 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.253 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293617+0000", 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293617+0000", 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293618+0000", 2024-09-02T10:34:58.254 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5099999999999999e-07 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.293623+0000", 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8810000000000001e-06 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551233+0000", 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.257609745 2024-09-02T10:34:58.255 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551238+0000", 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9609999999999996e-06 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551289+0000", 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0492999999999999e-05 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.256 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.551309+0000", 2024-09-02T10:34:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0426999999999999e-05 2024-09-02T10:34:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.554093+0000", 2024-09-02T10:34:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0027834660000000001 2024-09-02T10:34:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.257 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:00.554134+0000", 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0995000000000003e-05 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "replica_scrub(pg: 1.6,from:0'0,to:0'0,epoch:821/301,start:1:60000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:53.986902+0000", 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 304.19911903399998, 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35521325199999998, 2024-09-02T10:34:58.258 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:53.986902+0000", 2024-09-02T10:34:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.259 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:53.986902+0000", 2024-09-02T10:34:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:53.986903+0000", 2024-09-02T10:34:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5499999999999997e-07 2024-09-02T10:34:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.260 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:53.986909+0000", 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0399999999999998e-06 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:54.341139+0000", 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.35423013199999998 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:54.341149+0000", 2024-09-02T10:34:58.261 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6150000000000003e-06 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:54.341954+0000", 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00080544199999999996 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:54.342075+0000", 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012106 2024-09-02T10:34:58.262 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:54.342115+0000", 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.0108000000000003e-05 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:109464 8.c 8:3bd17a8d:::benchmark_data_smithi187_136059_object4183:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e836)", 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:30:34.224434+0000", 2024-09-02T10:34:58.263 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 263.96158728400002, 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25893373800000002, 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 109464 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.264 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.224434+0000", 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.224434+0000", 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.224436+0000", 2024-09-02T10:34:58.265 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7940000000000001e-06 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.224444+0000", 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.7719999999999994e-06 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.224461+0000", 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7462e-05 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.266 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.224471+0000", 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0101e-05 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.224546+0000", 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5074000000000007e-05 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.267 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.224652+0000", 2024-09-02T10:34:58.268 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106084 2024-09-02T10:34:58.268 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.268 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.268 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:34:58.268 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.224829+0000", 2024-09-02T10:34:58.268 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00017700300000000001 2024-09-02T10:34:58.268 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.268 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.268 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.268 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.228214+0000", 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003384551 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.228228+0000", 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4070000000000001e-05 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.483268+0000", 2024-09-02T10:34:58.269 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25504036600000002 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.483323+0000", 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4283000000000002e-05 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:30:34.483368+0000", 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5178e-05 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.270 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:134990 8.c 8:3887e519:::benchmark_data_smithi187_136059_object29709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:01.997958+0000", 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 236.18806323499999, 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.45244231600000001, 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:58.271 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 134990 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.997958+0000", 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.272 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.997958+0000", 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.997959+0000", 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.37e-06 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.997964+0000", 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5630000000000002e-06 2024-09-02T10:34:58.273 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.997976+0000", 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1556000000000001e-05 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.997981+0000", 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8130000000000001e-06 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.274 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.998029+0000", 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7327000000000003e-05 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.998113+0000", 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4724999999999999e-05 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:34:58.275 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:01.998179+0000", 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5296000000000006e-05 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.001615+0000", 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003436705 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.001765+0000", 2024-09-02T10:34:58.276 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000149252 2024-09-02T10:34:58.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.277 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.277 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.450286+0000", 2024-09-02T10:34:58.277 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.448521053 2024-09-02T10:34:58.277 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.277 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.277 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.277 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.450349+0000", 2024-09-02T10:34:58.277 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3369999999999998e-05 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.450400+0000", 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1286000000000003e-05 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:134990 8.c e843/792 ondisk, result = 0)", 2024-09-02T10:34:58.278 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:02.001522+0000", 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 236.18449879799999, 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44888478500000001, 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.001522+0000", 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.279 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.001522+0000", 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.001523+0000", 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0200000000000001e-07 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.280 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.001529+0000", 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3920000000000002e-06 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.450194+0000", 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.44866484400000001 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.281 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.450201+0000", 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9610000000000005e-06 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.450268+0000", 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6402000000000006e-05 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.450278+0000", 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0353000000000001e-05 2024-09-02T10:34:58.282 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.450407+0000", 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012903099999999999 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:134996 8.c 8:3b9a60f9:::benchmark_data_smithi187_136059_object29715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:34:58.283 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:02.105391+0000", 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 236.08063042500001, 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36654656200000002, 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 134996 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.284 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105391+0000", 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105391+0000", 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.285 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105392+0000", 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1880000000000001e-06 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105396+0000", 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.484e-06 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.286 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105406+0000", 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0314e-05 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105411+0000", 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1669999999999998e-06 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105457+0000", 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5553000000000002e-05 2024-09-02T10:34:58.287 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105536+0000", 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9893000000000002e-05 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 3,8", 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.105607+0000", 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0724999999999998e-05 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.288 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.108796+0000", 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003188597 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.109061+0000", 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00026546799999999999 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.289 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.471860+0000", 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36279869599999998 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.471894+0000", 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4399e-05 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.471937+0000", 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3078000000000003e-05 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.290 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:134996 8.c e843/792 ondisk, result = 0)", 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:02.109157+0000", 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 236.076864726, 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36278542800000002, 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.291 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.109157+0000", 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.109157+0000", 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.292 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.293 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:58 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:34:58.293 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:58 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:34:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:58.294 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.109157+0000", 2024-09-02T10:34:58.294 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3399999999999998e-07 2024-09-02T10:34:58.294 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.294 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.294 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.294 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.109161+0000", 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.67e-06 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.471783+0000", 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36262165200000002 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.471787+0000", 2024-09-02T10:34:58.295 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5059999999999997e-06 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.471850+0000", 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3225999999999994e-05 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.471856+0000", 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2730000000000002e-06 2024-09-02T10:34:58.296 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:02.471942+0000", 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6366999999999995e-05 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27192 9.8 9:1354060c:::benchmark_data_smithi187_154927_object27191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:34:58.297 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.189629+0000", 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 40.996392366999999, 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.30789178499999997, 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27192 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.298 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.189629+0000", 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.189629+0000", 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.299 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.189630+0000", 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8999999999999995e-07 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.189650+0000", 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0364e-05 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.189661+0000", 2024-09-02T10:34:58.300 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1012000000000001e-05 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.189666+0000", 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.417e-06 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.189710+0000", 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3170999999999997e-05 2024-09-02T10:34:58.301 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.189804+0000", 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.4074999999999996e-05 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.189896+0000", 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2353000000000002e-05 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.302 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.190851+0000", 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00095445199999999995 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.191234+0000", 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00038311999999999998 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.303 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.497460+0000", 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.3062259 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.497494+0000", 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4167e-05 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.497521+0000", 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6964000000000001e-05 2024-09-02T10:34:58.304 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27210 9.8 9:1812d6c0:::benchmark_data_smithi187_154927_object27209:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.511700+0000", 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 40.674321212999999, 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.38030343300000002, 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.305 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27210 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511700+0000", 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.306 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511700+0000", 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511701+0000", 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1800000000000005e-07 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.307 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511743+0000", 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2021999999999998e-05 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511754+0000", 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1117999999999999e-05 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511759+0000", 2024-09-02T10:34:58.308 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9819999999999999e-06 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511798+0000", 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9239000000000002e-05 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511884+0000", 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5356000000000001e-05 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.309 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.511962+0000", 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8276999999999996e-05 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.512551+0000", 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00058917699999999997 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.513051+0000", 2024-09-02T10:34:58.310 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00050014899999999997 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.891935+0000", 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.37888422799999999 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.891975+0000", 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9909e-05 2024-09-02T10:34:58.311 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.892003+0000", 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8158e-05 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:65348 9.8 9:15eee5c0:::benchmark_data_smithi187_154927_object65347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e928)", 2024-09-02T10:34:58.312 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:53.489056+0000", 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 4.6969655100000001, 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.082470307000000007, 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 65348 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:34:58.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.489056+0000", 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.489056+0000", 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:34:58.314 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.489058+0000", 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.798e-06 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.489079+0000", 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1704000000000001e-05 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.489095+0000", 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5466000000000001e-05 2024-09-02T10:34:58.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.489101+0000", 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2190000000000001e-06 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.489156+0000", 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4737999999999997e-05 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.489263+0000", 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106947 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 5,7", 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.489353+0000", 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0016e-05 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:34:58.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.490499+0000", 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0011461990000000001 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.491461+0000", 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00096189400000000003 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.571420+0000", 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.079958843000000002 2024-09-02T10:34:58.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.571478+0000", 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8799999999999999e-05 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:53.571526+0000", 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7682999999999997e-05 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.319 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.320 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.320 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:34:58.320 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:34:58.320 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:34:58.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:58.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:58.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:58 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:34:58.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:58 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:34:58.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:34:58.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:58 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:34:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:58 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:34:58.447 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:58.449 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:34:58.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:34:58.582 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:58.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:34:58.684 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:58.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:58.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:58.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:58.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:58.837 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:58.839 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:58.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:58.987 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:58.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:58.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:59.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 59 16 70309 70293 74.4564 49.8125 0.00533051 0.0134241 2024-09-02T10:34:59.089 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:34:59.095 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:34:59.126 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:59.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:59.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:34:59.228 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:34:59.252 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:59.253 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:34:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:59 smithi162 ceph-mon[20279]: pgmap v2128: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 9842/199191 objects degraded (4.941%); 26172/199191 objects misplaced (13.139%); 8.0 MiB/s, 127 objects/s recovering 2024-09-02T10:34:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 9842/199191 objects degraded (4.941%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:34:59.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:34:59 smithi162 ceph-mon[20279]: osdmap e930: 12 total, 12 up, 11 in 2024-09-02T10:34:59.353 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:34:59.375 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:59.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:34:59.378 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:34:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:59 smithi151 ceph-mon[16639]: pgmap v2128: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 9842/199191 objects degraded (4.941%); 26172/199191 objects misplaced (13.139%); 8.0 MiB/s, 127 objects/s recovering 2024-09-02T10:34:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 9842/199191 objects degraded (4.941%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:34:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:34:59 smithi151 ceph-mon[16639]: osdmap e930: 12 total, 12 up, 11 in 2024-09-02T10:34:59.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:34:59.500 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:59 smithi187 ceph-mon[20072]: pgmap v2128: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.1 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 9842/199191 objects degraded (4.941%); 26172/199191 objects misplaced (13.139%); 8.0 MiB/s, 127 objects/s recovering 2024-09-02T10:34:59.501 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 9842/199191 objects degraded (4.941%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:34:59.501 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:34:59 smithi187 ceph-mon[20072]: osdmap e930: 12 total, 12 up, 11 in 2024-09-02T10:34:59.503 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:34:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:59.504 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:34:59.605 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:34:59.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:59.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:59.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:34:59.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:34:59.771 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:34:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:59.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:34:59.873 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:34:59.912 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:34:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:34:59.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:34:59.935 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:34:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:34:59.936 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:34:59.936 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:35:00.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:00.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:35:00.021264+0000 min lat: 0.00099454 max lat: 2.15641 avg lat: 0.0134837 2024-09-02T10:35:00.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:35:00.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 60 16 71123 71107 74.0633 50.875 0.0828195 0.0134837 2024-09-02T10:35:00.044 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:00.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:00.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:00.146 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:00.178 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:00.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:00.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:00.306 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:00.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:00.408 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:00.438 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:00.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:00.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:00.540 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:00.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:00.564 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:00.664 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:00.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:00 smithi151 ceph-mon[16639]: pgmap v2130: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 9842/206277 objects degraded (4.771%); 25011/206277 objects misplaced (12.125%); 17 MiB/s, 276 objects/s recovering 2024-09-02T10:35:00.688 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:00.689 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:00.719 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:00 smithi187 ceph-mon[20072]: pgmap v2130: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 9842/206277 objects degraded (4.771%); 25011/206277 objects misplaced (12.125%); 17 MiB/s, 276 objects/s recovering 2024-09-02T10:35:00.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:35:00.790 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:00.823 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:00.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:00.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:00 smithi162 ceph-mon[20279]: pgmap v2130: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.2 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 9842/206277 objects degraded (4.771%); 25011/206277 objects misplaced (12.125%); 17 MiB/s, 276 objects/s recovering 2024-09-02T10:35:00.924 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:00.953 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:00.954 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:01.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 61 16 72143 72127 73.8942 63.75 0.0282944 0.0135255 2024-09-02T10:35:01.055 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:01.087 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:01.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:01.097 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:35:01.189 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:01.216 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:01.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:01.318 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:01.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:01.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:01.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:01.479 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:01.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:01.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:01.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:01.612 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:01.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:01.615 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:35:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:35:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:35:01.615 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:35:01.615 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:35:01.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:01.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:01.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:01.842 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:01.865 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:01.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:01.867 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:01.968 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:02.002 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:02.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:02.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 62 16 73061 73045 73.6277 57.375 0.00744177 0.0135765 2024-09-02T10:35:02.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:02.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:02.126 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:02.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:02.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:02.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:02.254 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:02.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:02.385 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:02.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:02.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:02.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:35:02.486 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:02.511 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:02.513 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:02.614 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:02.643 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:02.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:02.739 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:35:02.744 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:02.772 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:02.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:02.874 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:02.912 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:02.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:02.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:02 smithi151 ceph-mon[16639]: pgmap v2131: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 36 MiB/s wr, 574 op/s; 9842/209268 objects degraded (4.703%); 23587/209268 objects misplaced (11.271%); 26 MiB/s, 418 objects/s recovering 2024-09-02T10:35:03.001 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:02 smithi187 ceph-mon[20072]: pgmap v2131: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 36 MiB/s wr, 574 op/s; 9842/209268 objects degraded (4.703%); 23587/209268 objects misplaced (11.271%); 26 MiB/s, 418 objects/s recovering 2024-09-02T10:35:03.014 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:03.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 63 16 73978 73962 73.3686 57.3125 0.014908 0.0136247 2024-09-02T10:35:03.044 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:03.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:02 smithi162 ceph-mon[20279]: pgmap v2131: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.3 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 36 MiB/s wr, 574 op/s; 9842/209268 objects degraded (4.703%); 23587/209268 objects misplaced (11.271%); 26 MiB/s, 418 objects/s recovering 2024-09-02T10:35:03.146 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:03.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:03.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:03.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:03.284 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:03.312 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:03.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:03.374 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:35:03.374 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:35:03.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:35:03.374 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:35:03.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.374 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146099 8.6 8:6509e9b8:::benchmark_data_smithi187_136059_object40818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:35:03.374 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.705971+0000", 2024-09-02T10:35:03.374 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 221.55196483, 2024-09-02T10:35:03.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.37556117500000002, 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146099 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705971+0000", 2024-09-02T10:35:03.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705971+0000", 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705972+0000", 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2359999999999999e-06 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705976+0000", 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1470000000000003e-06 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705986+0000", 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0161e-05 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705993+0000", 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9330000000000004e-06 2024-09-02T10:35:03.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.719530+0000", 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013537077999999999 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.719606+0000", 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5570999999999999e-05 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.719685+0000", 2024-09-02T10:35:03.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9579000000000005e-05 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.723078+0000", 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033929590000000001 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.756155+0000", 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.033076326000000003 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:03.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.080171+0000", 2024-09-02T10:35:03.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32401667499999998 2024-09-02T10:35:03.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:03.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.080216+0000", 2024-09-02T10:35:03.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5018999999999997e-05 2024-09-02T10:35:03.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.380 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.081532+0000", 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001315491 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146131 8.6 8:6708c0fe:::benchmark_data_smithi187_136059_object40850:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.749657+0000", 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 221.508278875, 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.38950238999999998, 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.381 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146131 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749657+0000", 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749657+0000", 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749658+0000", 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1799999999999999e-06 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749662+0000", 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6519999999999997e-06 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749674+0000", 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1518000000000001e-05 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749681+0000", 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8759999999999999e-06 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.751393+0000", 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017116729999999999 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.751501+0000", 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108157 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.751635+0000", 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013469600000000001 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:03.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.756230+0000", 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045950890000000001 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.097478+0000", 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34124748199999999 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.137802+0000", 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.040324368999999999 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.137851+0000", 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8349000000000001e-05 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.139159+0000", 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013083490000000001 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:210545 8.5 8:a9a4e4c5:::benchmark_data_smithi187_136059_object105264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:35:03.387 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:16.481782+0000", 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 106.776153802, 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.47180958000000001, 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 210545 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481782+0000", 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481782+0000", 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481783+0000", 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.68e-06 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481791+0000", 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8089999999999993e-06 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481806+0000", 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5201999999999999e-05 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481814+0000", 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0469999999999994e-06 2024-09-02T10:35:03.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481887+0000", 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2628999999999999e-05 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.482010+0000", 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012311000000000001 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:35:03.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.482127+0000", 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117141 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.488686+0000", 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0065583769999999998 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.490795+0000", 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021094870000000002 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953492+0000", 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46269710800000002 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953548+0000", 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5680999999999998e-05 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953591+0000", 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3309000000000002e-05 2024-09-02T10:35:03.393 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:210545 8.5 e881/792 ondisk, result = 0)", 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:16.487082+0000", 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 106.770853382, 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46651547900000001, 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.487082+0000", 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.487082+0000", 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.487083+0000", 2024-09-02T10:35:03.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3499999999999996e-07 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.487087+0000", 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1819999999999999e-06 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953386+0000", 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46629904500000002 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953393+0000", 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5379999999999999e-06 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953475+0000", 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2639000000000006e-05 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953485+0000", 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0113e-05 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953598+0000", 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112327 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:210555 8.5 8:a562631a:::benchmark_data_smithi187_136059_object105274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:16.494622+0000", 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 106.763313631, 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.68545412900000002, 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.398 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 210555 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494622+0000", 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494622+0000", 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494623+0000", 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.37e-06 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494629+0000", 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.293e-06 2024-09-02T10:35:03.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494639+0000", 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0536999999999999e-05 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494647+0000", 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5179999999999998e-06 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494746+0000", 2024-09-02T10:35:03.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9341999999999999e-05 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494908+0000", 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016205899999999999 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.495036+0000", 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000127606 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.500063+0000", 2024-09-02T10:35:03.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050271839999999996 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.557999+0000", 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.057935954999999997 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.180008+0000", 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.62200955499999999 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.180048+0000", 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9977e-05 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.180076+0000", 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7733e-05 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27559.0:210554 8.4 e881/792 8:25e06d4f:::benchmark_data_smithi187_136059_object105273:head v 881'13332, mlcod=881'13328)", 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:16.494855+0000", 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 106.76308045099999, 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.513357536, 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494855+0000", 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494855+0000", 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494856+0000", 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3900000000000004e-07 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494862+0000", 2024-09-02T10:35:03.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8560000000000003e-06 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.004275+0000", 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.50941320800000001 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.004280+0000", 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5809999999999996e-06 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.004348+0000", 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7230999999999994e-05 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.004367+0000", 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8856e-05 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.008181+0000", 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003814797 2024-09-02T10:35:03.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.008213+0000", 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1267999999999999e-05 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:210555 8.5 e881/792 ondisk, result = 0)", 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:16.500168+0000", 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 106.757767532, 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.67991180699999998, 2024-09-02T10:35:03.409 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.500168+0000", 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.500168+0000", 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.500169+0000", 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8599999999999998e-07 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.500176+0000", 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6389999999999992e-06 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.179888+0000", 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.67971182900000005 2024-09-02T10:35:03.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.179901+0000", 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2437e-05 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.179998+0000", 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6984999999999994e-05 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.180004+0000", 2024-09-02T10:35:03.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1249999999999998e-06 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.180080+0000", 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6106000000000006e-05 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGRecoveryDeleteReply(8.6 e882,844 8:60b27707:::benchmark_data_smithi187_136059_object22977:head,839'8144)", 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:18.621005+0000", 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 104.636930603, 2024-09-02T10:35:03.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43784977800000002, 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:18.621005+0000", 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:18.621005+0000", 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:18.621007+0000", 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.59e-06 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:18.621016+0000", 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9259999999999995e-06 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:19.058550+0000", 2024-09-02T10:35:03.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43753405000000001 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:19.058557+0000", 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0730000000000001e-06 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:19.058602+0000", 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5318999999999998e-05 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:19.058855+0000", 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025282000000000001 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGRecoveryDelete(8.8 e883,854 8:1147bf7c:::benchmark_data_smithi187_136059_object4341:head,836'6957)", 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:21.592106+0000", 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 101.665829452, 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94375713100000003, 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.417 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.592106+0000", 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.592106+0000", 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.592108+0000", 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5969999999999999e-06 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.592120+0000", 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2757e-05 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:22.535701+0000", 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94358019999999998 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:22.535706+0000", 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2680000000000002e-06 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:22.535786+0000", 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0296999999999994e-05 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:22.535793+0000", 2024-09-02T10:35:03.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4609999999999996e-06 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:22.535863+0000", 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0550999999999996e-05 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27090 9.9 9.411b16f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.942484+0000", 2024-09-02T10:35:03.421 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 46.315451803999999, 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93450280299999999, 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27090 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942484+0000", 2024-09-02T10:35:03.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942484+0000", 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942485+0000", 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5900000000000005e-07 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942501+0000", 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6467e-05 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942511+0000", 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9699999999999994e-06 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942516+0000", 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2299999999999999e-06 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942564+0000", 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7747999999999998e-05 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942571+0000", 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4339999999999997e-06 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.876944+0000", 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93437385699999997 2024-09-02T10:35:03.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.876987+0000", 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2138000000000002e-05 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27095 9.9 9.a3492329 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.979259+0000", 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 46.278676396999998, 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89779759400000003, 2024-09-02T10:35:03.426 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27095 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979259+0000", 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979259+0000", 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979261+0000", 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4529999999999999e-06 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979280+0000", 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9395e-05 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979292+0000", 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.169e-05 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979297+0000", 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4149999999999998e-06 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979356+0000", 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8548000000000003e-05 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979367+0000", 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1421e-05 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877039+0000", 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89767237799999999 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.430 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877057+0000", 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7294000000000001e-05 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27114 9.9 9.7327df9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.985981+0000", 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 46.271954278000003, 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89113120999999995, 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.431 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27114 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.985981+0000", 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.985981+0000", 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.985985+0000", 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.839e-06 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.986009+0000", 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3682000000000001e-05 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.986030+0000", 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1725e-05 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.986036+0000", 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7150000000000003e-06 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.986108+0000", 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2285000000000003e-05 2024-09-02T10:35:03.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.986120+0000", 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1292e-05 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877103+0000", 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89098296300000002 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.435 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877112+0000", 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7089999999999997e-06 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27124 9.9 9.9deeea19 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.989404+0000", 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 46.268531889000002, 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88776398999999995, 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.436 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27124 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989404+0000", 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989404+0000", 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989405+0000", 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8100000000000001e-07 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989424+0000", 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9490000000000001e-05 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989435+0000", 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1039000000000001e-05 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989442+0000", 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0639999999999996e-06 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989501+0000", 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8409000000000003e-05 2024-09-02T10:35:03.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989508+0000", 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8669999999999994e-06 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877157+0000", 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88764881399999995 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877168+0000", 2024-09-02T10:35:03.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0326000000000001e-05 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27131 9.9 9.f983e449 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.991168+0000", 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 46.266767733000002, 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88604997500000005, 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.441 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27131 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991168+0000", 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991168+0000", 2024-09-02T10:35:03.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991169+0000", 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2200000000000003e-07 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991185+0000", 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6841e-05 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991196+0000", 2024-09-02T10:35:03.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0463e-05 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991202+0000", 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6720000000000001e-06 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991257+0000", 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4999e-05 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991266+0000", 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2429999999999994e-06 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877208+0000", 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.885942283 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877218+0000", 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6520000000000001e-06 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.445 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27134 9.9 9.f56117f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.993011+0000", 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 46.264924331000003, 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.884242682, 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.446 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27134 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993011+0000", 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993011+0000", 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993012+0000", 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.186e-06 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993032+0000", 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9831999999999999e-05 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993043+0000", 2024-09-02T10:35:03.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1216999999999999e-05 2024-09-02T10:35:03.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993049+0000", 2024-09-02T10:35:03.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5389999999999998e-06 2024-09-02T10:35:03.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993111+0000", 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2469000000000001e-05 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993121+0000", 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0044000000000001e-05 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877245+0000", 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88412303599999997 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877254+0000", 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3589999999999997e-06 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27144 9.9 9.d4d23b29 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.998807+0000", 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 46.259128089999997, 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.87849941300000001, 2024-09-02T10:35:03.451 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27144 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998807+0000", 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998807+0000", 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998809+0000", 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3599999999999999e-06 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998831+0000", 2024-09-02T10:35:03.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2566e-05 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998857+0000", 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5573999999999999e-05 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998866+0000", 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9080000000000001e-06 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998924+0000", 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8588999999999998e-05 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998934+0000", 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8220000000000002e-06 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877298+0000", 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.87836354100000003 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877307+0000", 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0529999999999996e-06 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27151 9.9 9.31ef2d19 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.008837+0000", 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 46.249098342000003, 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.86850359600000004, 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.456 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27151 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008837+0000", 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008837+0000", 2024-09-02T10:35:03.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008838+0000", 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2330000000000001e-06 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008867+0000", 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8999000000000001e-05 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008878+0000", 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0888e-05 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008885+0000", 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1160000000000001e-06 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008934+0000", 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8463e-05 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008940+0000", 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0020000000000003e-06 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877332+0000", 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.86839186499999999 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877341+0000", 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0299999999999999e-06 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.460 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27173 9.9 9.e697529 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.028584+0000", 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 46.229351969, 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.84879224399999997, 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.461 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27173 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028584+0000", 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028584+0000", 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028584+0000", 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0999999999999997e-07 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028603+0000", 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8598999999999999e-05 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028613+0000", 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.03e-05 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028621+0000", 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4699999999999996e-06 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028671+0000", 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0302000000000001e-05 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028678+0000", 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.255e-06 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877367+0000", 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.84868848299999999 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877376+0000", 2024-09-02T10:35:03.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9250000000000001e-06 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27202 9.9 9.cbef6c09 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.465428+0000", 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 45.792507837999999, 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41198401099999998, 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.466 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27202 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465428+0000", 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465428+0000", 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465429+0000", 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1999999999999998e-07 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465447+0000", 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8578999999999999e-05 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465458+0000", 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1276e-05 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465464+0000", 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6949999999999997e-06 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465513+0000", 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9415e-05 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465521+0000", 2024-09-02T10:35:03.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4120000000000004e-06 2024-09-02T10:35:03.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877403+0000", 2024-09-02T10:35:03.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41188169400000002 2024-09-02T10:35:03.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877412+0000", 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1200000000000008e-06 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27209 9.9 9.84fc6679 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.508005+0000", 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 45.749930843999998, 2024-09-02T10:35:03.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.36944183800000002, 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27209 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.508005+0000", 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.508005+0000", 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.508006+0000", 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.082e-06 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.508026+0000", 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9930000000000001e-05 2024-09-02T10:35:03.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.508038+0000", 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2274e-05 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.508044+0000", 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4980000000000003e-06 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.508097+0000", 2024-09-02T10:35:03.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2741999999999997e-05 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.508103+0000", 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2179999999999996e-06 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877438+0000", 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.369334581 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:03.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:03.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877447+0000", 2024-09-02T10:35:03.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5129999999999996e-06 2024-09-02T10:35:03.477 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.477 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.477 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.477 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:03.477 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:03.477 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:35:03.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:03.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:03.503 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:03.604 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:03.637 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:03.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:03.739 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:03.761 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:03.762 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:03.863 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:03.886 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:03.887 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:03.943 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:03 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 9842/216132 objects degraded (4.554%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:35:03.989 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:04.021 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:04.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 64 16 75046 75030 73.2651 66.75 0.0130454 0.0136441 2024-09-02T10:35:04.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:04.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:03 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 9842/216132 objects degraded (4.554%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:35:04.123 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:04.145 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:04.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:04.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:03 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 9842/216132 objects degraded (4.554%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:35:04.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:04.274 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:04.275 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:04.376 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:04.409 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:04.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:04.485 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:35:04.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:04.544 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:04.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:04.646 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:04.676 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:04.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:04.778 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:04.799 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:35:04.808 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:04.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:04.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:04.910 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:04.945 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:04.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:04 smithi187 ceph-mon[20072]: pgmap v2132: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 53 MiB/s wr, 848 op/s; 9842/216132 objects degraded (4.554%); 22265/216132 objects misplaced (10.302%); 24 MiB/s, 379 objects/s recovering 2024-09-02T10:35:05.021 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 65 16 76125 76109 73.1754 67.4375 0.0104031 0.0136604 2024-09-02T10:35:05.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:05.074 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:05.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:04 smithi162 ceph-mon[20279]: pgmap v2132: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 53 MiB/s wr, 848 op/s; 9842/216132 objects degraded (4.554%); 22265/216132 objects misplaced (10.302%); 24 MiB/s, 379 objects/s recovering 2024-09-02T10:35:05.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:05.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:04 smithi151 ceph-mon[16639]: pgmap v2132: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.4 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 53 MiB/s wr, 848 op/s; 9842/216132 objects degraded (4.554%); 22265/216132 objects misplaced (10.302%); 24 MiB/s, 379 objects/s recovering 2024-09-02T10:35:05.206 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:05.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:05.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:05.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:05.336 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:05.337 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:05.438 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:05.468 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:05.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:05.570 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:05.592 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:05.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:05.593 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:05.694 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:05.721 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:05.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:05.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:05.854 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:05.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:05.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:05.988 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:05.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:06.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 66 16 77262 77246 73.1433 71.0625 0.0234981 0.0136648 2024-09-02T10:35:06.082 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:35:06.082 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:35:06.082 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:35:06.082 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:35:06.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:06.125 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:06.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:06.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:06.262 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:06.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:06.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:06.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:06.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:06.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:06.498 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:06.523 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:06.524 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:06.625 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:06.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:06.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:06.755 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:06.779 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:06.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:06.780 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:06.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:35:06.881 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:06.912 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:06.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:06.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:06 smithi187 ceph-mon[20072]: pgmap v2133: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s; 9842/223254 objects degraded (4.408%); 20088/223254 objects misplaced (8.998%); 29 MiB/s, 459 objects/s recovering 2024-09-02T10:35:07.013 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:07.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 67 16 78214 78198 72.9396 59.5 0.00361254 0.0137031 2024-09-02T10:35:07.043 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:07.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:06 smithi162 ceph-mon[20279]: pgmap v2133: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s; 9842/223254 objects degraded (4.408%); 20088/223254 objects misplaced (8.998%); 29 MiB/s, 459 objects/s recovering 2024-09-02T10:35:07.141 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:35:07.145 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:07.179 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:07.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:07.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:07.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:06 smithi151 ceph-mon[16639]: pgmap v2133: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.5 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 60 MiB/s wr, 960 op/s; 9842/223254 objects degraded (4.408%); 20088/223254 objects misplaced (8.998%); 29 MiB/s, 459 objects/s recovering 2024-09-02T10:35:07.281 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:07.304 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:07.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:07.406 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:07.442 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:07.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:07.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:07.571 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:07.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:07.571 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:07.672 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:07.705 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:07.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:07.714 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:35:07.715 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:35:07.715 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:35:07.715 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:35:07.715 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:35:07.807 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:07.880 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:07.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:07.984 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:08.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:08.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:08.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 68 16 78991 78975 72.581 48.5625 0.00945048 0.0137605 2024-09-02T10:35:08.117 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:08.144 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:08.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:08.246 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:08.272 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:08.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:08.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:08.374 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:08.396 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:08.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:08.397 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:08.495 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:35:08.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:08.524 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:08.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:08.626 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:08.652 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:08.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:08.726 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:08 smithi187 ceph-mon[20072]: pgmap v2134: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 9842/227010 objects degraded (4.335%); 19568/227010 objects misplaced (8.620%); 27 MiB/s, 435 objects/s recovering 2024-09-02T10:35:08.726 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:08 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:35:08.726 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:08 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:35:08.726 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:08 smithi187 ceph-mon[20072]: osdmap e931: 12 total, 12 up, 11 in 2024-09-02T10:35:08.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:08.773 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:35:08.782 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:08.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:08.884 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:08.917 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:08.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:09.019 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:09.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 69 16 79515 79499 72.0038 32.75 0.0152321 0.0138788 2024-09-02T10:35:09.044 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:09.045 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:08 smithi162 ceph-mon[20279]: pgmap v2134: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 9842/227010 objects degraded (4.335%); 19568/227010 objects misplaced (8.620%); 27 MiB/s, 435 objects/s recovering 2024-09-02T10:35:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:08 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:35:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:08 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:35:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:08 smithi162 ceph-mon[20279]: osdmap e931: 12 total, 12 up, 11 in 2024-09-02T10:35:09.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:09.177 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:09.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:09.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:08 smithi151 ceph-mon[16639]: pgmap v2134: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.6 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 58 MiB/s wr, 926 op/s; 9842/227010 objects degraded (4.335%); 19568/227010 objects misplaced (8.620%); 27 MiB/s, 435 objects/s recovering 2024-09-02T10:35:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:08 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:35:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:08 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:35:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:08 smithi151 ceph-mon[16639]: osdmap e931: 12 total, 12 up, 11 in 2024-09-02T10:35:09.280 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:09.311 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:09.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:09.312 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:09.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:09.436 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:71617 7.b e711/661 7:d1c8b968:::benchmark_data_smithi187_120712_object71616:head v 711'4410, mlcod=711'4407)", 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:25:27.803500+0000", 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 581.52972509899996, 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33549837500000002, 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:27.803500+0000", 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:27.803500+0000", 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:27.803501+0000", 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0449999999999999e-06 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:27.803556+0000", 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5214999999999999e-05 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.136887+0000", 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33333045 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.136896+0000", 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1300000000000007e-06 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.136955+0000", 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8792999999999998e-05 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.136974+0000", 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9411000000000001e-05 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.138967+0000", 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0019933170000000001 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:25:28.138998+0000", 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1013999999999998e-05 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.441 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27220.0:100943 7.c e723/717 7:3c179c28:::benchmark_data_smithi187_120712_object17001:head v 723'6262, mlcod=723'6259)", 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:01.695823+0000", 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 547.63740223599996, 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.26241625899999999, 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.695823+0000", 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.695823+0000", 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.695824+0000", 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6599999999999995e-07 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.695830+0000", 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.9340000000000003e-06 2024-09-02T10:35:09.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.955758+0000", 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.25992827400000001 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.955762+0000", 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1749999999999998e-06 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.955831+0000", 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8554999999999997e-05 2024-09-02T10:35:09.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.955859+0000", 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8867999999999999e-05 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.958208+0000", 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0023483219999999999 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:01.958239+0000", 2024-09-02T10:35:09.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1365000000000002e-05 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:136920 7.5 7:a235c29c:::benchmark_data_smithi187_120712_object52978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:43.766046+0000", 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.56717884300002, 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46275040899999997, 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:35:09.446 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 136920 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766046+0000", 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766046+0000", 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766048+0000", 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.415e-06 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766053+0000", 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0549999999999999e-06 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766064+0000", 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1457999999999999e-05 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766070+0000", 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0700000000000003e-06 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766119+0000", 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9104999999999998e-05 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766205+0000", 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.5873999999999998e-05 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.766280+0000", 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4931999999999993e-05 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768103+0000", 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0018229419999999999 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768304+0000", 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00020122 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.450 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228691+0000", 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46038674499999999 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228748+0000", 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6935000000000003e-05 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228797+0000", 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8658000000000001e-05 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.451 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27220.0:136920 7.5 e743/741 ondisk, result = 0)", 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:43.768313+0000", 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.56491161700001, 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46048927299999998, 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768313+0000", 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768313+0000", 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768314+0000", 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8500000000000002e-07 2024-09-02T10:35:09.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:43.768317+0000", 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.6070000000000002e-06 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228601+0000", 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.460284949 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228610+0000", 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.8789999999999992e-06 2024-09-02T10:35:09.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228674+0000", 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3646999999999996e-05 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228684+0000", 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0288e-05 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.228803+0000", 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118418 2024-09-02T10:35:09.455 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:136934 7.5 7:a1526dd1:::benchmark_data_smithi187_120712_object52992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:44.230426+0000", 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 505.10279866000002, 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.212588995, 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:35:09.456 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 136934 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230426+0000", 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230426+0000", 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230428+0000", 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6190000000000001e-06 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230434+0000", 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1709999999999999e-06 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230449+0000", 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4666e-05 2024-09-02T10:35:09.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.230457+0000", 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8590000000000005e-06 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.231739+0000", 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00128243 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.231878+0000", 2024-09-02T10:35:09.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013864199999999999 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.231991+0000", 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011313499999999999 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.234150+0000", 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021594489999999999 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.460 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.234408+0000", 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025732799999999997 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.442935+0000", 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.208527407 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.442978+0000", 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2558000000000001e-05 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:44.443015+0000", 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7731000000000002e-05 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27220.0:136942 7.5 7:a20fdecd:::benchmark_data_smithi187_120712_object53000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e745)", 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:26:45.462956+0000", 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 503.87026921, 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23792718600000001, 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.462 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27220", 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 136942 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.462956+0000", 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.462956+0000", 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.462958+0000", 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.294e-06 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.462968+0000", 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6569999999999992e-06 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.462990+0000", 2024-09-02T10:35:09.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2633000000000002e-05 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.463003+0000", 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2517000000000001e-05 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.463095+0000", 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1850999999999999e-05 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op must wait for map", 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.463104+0000", 2024-09-02T10:35:09.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.7509999999999997e-06 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.465089+0000", 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001985419 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.465219+0000", 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012985199999999999 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 0,9", 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.465339+0000", 2024-09-02T10:35:09.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000120386 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.467783+0000", 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024435809999999998 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.468202+0000", 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00041971700000000001 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.700799+0000", 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.232596409 2024-09-02T10:35:09.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.700836+0000", 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6619999999999998e-05 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:26:45.700883+0000", 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7499e-05 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.468 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:21803 8.6 8:6cea7233:::benchmark_data_smithi187_136059_object21802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e804)", 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:13.620281+0000", 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 355.71294440999998, 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21402042600000001, 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 21803 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620281+0000", 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620281+0000", 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620282+0000", 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1570000000000001e-06 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620313+0000", 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1090000000000002e-05 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620324+0000", 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.115e-05 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620332+0000", 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0120000000000005e-06 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620376+0000", 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3488000000000001e-05 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620478+0000", 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000102246 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620611+0000", 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013364200000000001 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.621411+0000", 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00079942399999999997 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.623938+0000", 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025271460000000001 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834210+0000", 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21027237400000001 2024-09-02T10:35:09.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834263+0000", 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3063999999999999e-05 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834301+0000", 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7632999999999997e-05 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:21820 8.6 8:6a3536fe:::benchmark_data_smithi187_136059_object21819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e804)", 2024-09-02T10:35:09.474 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:13.629639+0000", 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 355.70358625599999, 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21678998799999999, 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 21820 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629639+0000", 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629639+0000", 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629641+0000", 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0169999999999999e-06 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629670+0000", 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9173999999999999e-05 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629689+0000", 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9045000000000001e-05 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629697+0000", 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6199999999999999e-06 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629747+0000", 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9968e-05 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629904+0000", 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000157656 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.630021+0000", 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116347 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.631099+0000", 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010783819999999999 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.631117+0000", 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7915e-05 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.846333+0000", 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21521643500000001 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.479 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.846391+0000", 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7169000000000001e-05 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.846429+0000", 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8260000000000003e-05 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28292 8.d 8:b2a9e2ae:::benchmark_data_smithi187_136059_object28291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:35:09.480 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.105898+0000", 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 351.22732694899997, 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21874021900000001, 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28292 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105898+0000", 2024-09-02T10:35:09.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105898+0000", 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105899+0000", 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.068e-06 2024-09-02T10:35:09.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105924+0000", 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5179000000000001e-05 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105938+0000", 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3713000000000001e-05 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105946+0000", 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6559999999999992e-06 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.106008+0000", 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2001999999999994e-05 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.106149+0000", 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014115800000000001 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.106281+0000", 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000132466 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.107636+0000", 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001355055 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.113043+0000", 2024-09-02T10:35:09.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0054062870000000001 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324567+0000", 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21152405399999999 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324604+0000", 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7017999999999997e-05 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324638+0000", 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4563000000000001e-05 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28300 8.f 8:f195aa31:::benchmark_data_smithi187_136059_object28299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.108959+0000", 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 351.22426633399999, 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21860515799999999, 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.487 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28300 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108959+0000", 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108959+0000", 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108960+0000", 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5209999999999999e-06 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108983+0000", 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3189000000000001e-05 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108998+0000", 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4542000000000001e-05 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109005+0000", 2024-09-02T10:35:09.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4170000000000003e-06 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109063+0000", 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7337999999999999e-05 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109178+0000", 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115203 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109310+0000", 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000132156 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.110649+0000", 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013386990000000001 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111465+0000", 2024-09-02T10:35:09.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0008164 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327469+0000", 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.216004207 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327527+0000", 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7271000000000001e-05 2024-09-02T10:35:09.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327564+0000", 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7215000000000002e-05 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28305 8.d 8:bd02d929:::benchmark_data_smithi187_136059_object28304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.111275+0000", 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 351.22194983899999, 2024-09-02T10:35:09.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21380102300000001, 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28305 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111275+0000", 2024-09-02T10:35:09.495 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111275+0000", 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111276+0000", 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4399999999999999e-07 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.496 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111295+0000", 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8816000000000001e-05 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111305+0000", 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0589e-05 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111311+0000", 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2650000000000004e-06 2024-09-02T10:35:09.497 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111387+0000", 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6338e-05 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111513+0000", 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012598000000000001 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:35:09.498 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111655+0000", 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000141726 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112644+0000", 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00098899699999999992 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.127377+0000", 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014733448 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.499 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324992+0000", 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19761451799999999 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.325035+0000", 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3201999999999997e-05 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.325076+0000", 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1300000000000001e-05 2024-09-02T10:35:09.500 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28306 8.f 8:fcc25ce2:::benchmark_data_smithi187_136059_object28305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.112056+0000", 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 351.22116904400002, 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21564313299999999, 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.501 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28306 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112056+0000", 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112056+0000", 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.502 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112057+0000", 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.144e-06 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112081+0000", 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3713000000000001e-05 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112091+0000", 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8090000000000004e-06 2024-09-02T10:35:09.503 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112098+0000", 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0990000000000001e-06 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112150+0000", 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1768999999999999e-05 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.504 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112236+0000", 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6237999999999997e-05 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112316+0000", 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0603000000000001e-05 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.113365+0000", 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010481539999999999 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.505 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.113551+0000", 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018640599999999999 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327611+0000", 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21405996499999999 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327664+0000", 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2982999999999997e-05 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.506 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327699+0000", 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5250000000000003e-05 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28313 8.f 8:fb500fbb:::benchmark_data_smithi187_136059_object28312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.117307+0000", 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 351.21591784200001, 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.210606557, 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.507 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28313 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117307+0000", 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117307+0000", 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.508 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117308+0000", 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1990000000000001e-06 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117328+0000", 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9528999999999999e-05 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117342+0000", 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4219000000000001e-05 2024-09-02T10:35:09.509 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117350+0000", 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0120000000000005e-06 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117456+0000", 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106291 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117573+0000", 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011608 2024-09-02T10:35:09.510 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117677+0000", 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010426500000000001 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.118658+0000", 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00098097000000000006 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.119074+0000", 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000416384 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.511 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327819+0000", 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20874446899999999 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327880+0000", 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1617000000000003e-05 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327914+0000", 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3522000000000002e-05 2024-09-02T10:35:09.512 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146135 8.f 8:ffa01c02:::benchmark_data_smithi187_136059_object40854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.753711+0000", 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 227.579514012, 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33445889499999998, 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.513 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146135 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753711+0000", 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753711+0000", 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.514 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753713+0000", 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8589999999999999e-06 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753720+0000", 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2579999999999998e-06 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753739+0000", 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8916999999999999e-05 2024-09-02T10:35:09.515 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753748+0000", 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1190000000000003e-06 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.755012+0000", 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012635649999999999 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.516 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.755145+0000", 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013299500000000001 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.755260+0000", 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115469 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.758833+0000", 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035724749999999999 2024-09-02T10:35:09.517 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.768082+0000", 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0092492670000000003 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.088097+0000", 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32001463499999999 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.088136+0000", 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9756999999999999e-05 2024-09-02T10:35:09.518 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.088170+0000", 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3578999999999998e-05 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146139 8.f 8:f849bf1d:::benchmark_data_smithi187_136059_object40858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.773477+0000", 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 227.55974810500001, 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31800191100000003, 2024-09-02T10:35:09.519 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146139 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773477+0000", 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.520 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773477+0000", 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773478+0000", 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1790000000000001e-06 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773483+0000", 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8280000000000002e-06 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.521 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773494+0000", 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1419e-05 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773500+0000", 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.874e-06 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773553+0000", 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2957999999999997e-05 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773667+0000", 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113812 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773777+0000", 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010969899999999999 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.777216+0000", 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034393179999999998 2024-09-02T10:35:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.777885+0000", 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00066886699999999996 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.091413+0000", 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31352819799999998 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.524 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.091446+0000", 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2802999999999997e-05 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.091479+0000", 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2956000000000001e-05 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:183126 8.f 8:fec0a0af:::benchmark_data_smithi187_136059_object77845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-09-02T10:35:09.525 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:29.326892+0000", 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 160.006333251, 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23921741799999999, 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183126 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.526 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326892+0000", 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326892+0000", 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326894+0000", 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2029999999999999e-06 2024-09-02T10:35:09.527 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326903+0000", 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2240000000000001e-06 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326925+0000", 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2033000000000001e-05 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.528 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326937+0000", 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1522000000000001e-05 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.329339+0000", 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024024099999999998 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.329467+0000", 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000127508 2024-09-02T10:35:09.529 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.329604+0000", 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013698200000000001 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.334705+0000", 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051010439999999999 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.530 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.347683+0000", 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012978701 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.566044+0000", 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.218360309 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.566080+0000", 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6727999999999998e-05 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.531 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.566109+0000", 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8753999999999999e-05 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:207403 8.f 8:f9743dd7:::benchmark_data_smithi187_136059_object102122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:11.376024+0000", 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 117.957201314, 2024-09-02T10:35:09.532 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.277723781, 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207403 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376024+0000", 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.533 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376024+0000", 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376026+0000", 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2500000000000001e-06 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.534 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376035+0000", 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2390000000000008e-06 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376057+0000", 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1421999999999999e-05 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376066+0000", 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.166e-06 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.535 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376136+0000", 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9982999999999997e-05 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376274+0000", 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013833300000000001 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376405+0000", 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013087900000000001 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.536 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381691+0000", 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0052860909999999997 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381713+0000", 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2206999999999999e-05 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653638+0000", 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27192507599999999 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.537 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653697+0000", 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8286999999999999e-05 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653748+0000", 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0847999999999998e-05 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:207412 8.f 8:ff87a35e:::benchmark_data_smithi187_136059_object102131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:11.416635+0000", 2024-09-02T10:35:09.538 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 117.916589562, 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.237262055, 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207412 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416635+0000", 2024-09-02T10:35:09.539 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416635+0000", 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416637+0000", 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9360000000000002e-06 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.540 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416645+0000", 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4739999999999999e-06 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416662+0000", 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7523000000000001e-05 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416671+0000", 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3599999999999996e-06 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.541 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416737+0000", 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6088999999999997e-05 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416865+0000", 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012849500000000001 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416984+0000", 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118202 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.542 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.421494+0000", 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045106759999999999 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.421568+0000", 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3586999999999999e-05 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653798+0000", 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23222986800000001 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.543 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653839+0000", 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.087e-05 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653898+0000", 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8974999999999997e-05 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.544 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.28030.0:6771 9.9 e904/897 9:9a514e7d:::benchmark_data_smithi187_154927_object6770:head v 904'417, mlcod=904'414)", 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:03.762823+0000", 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 65.570401654999998, 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083770215999999995, 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.762823+0000", 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.545 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.762823+0000", 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.762824+0000", 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0979999999999999e-06 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.762859+0000", 2024-09-02T10:35:09.546 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4480999999999997e-05 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846015+0000", 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083155899000000005 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846021+0000", 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.55e-06 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846077+0000", 2024-09-02T10:35:09.547 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5758999999999999e-05 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846095+0000", 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7768e-05 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846560+0000", 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00046457399999999999 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846594+0000", 2024-09-02T10:35:09.548 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4087000000000001e-05 2024-09-02T10:35:09.549 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.549 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.549 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.549 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:35:09.549 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:35:09.549 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:35:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:09.550 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:09.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:09.651 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:09.696 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:09.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:09.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:09.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:09.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:09.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:09.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:09.931 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:09.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:09 smithi151 ceph-mon[16639]: osdmap e932: 12 total, 12 up, 11 in 2024-09-02T10:35:09.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 9842/227010 objects degraded (4.335%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:35:09.961 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:09.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:09.962 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:09 smithi187 ceph-mon[20072]: osdmap e932: 12 total, 12 up, 11 in 2024-09-02T10:35:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 9842/227010 objects degraded (4.335%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:35:10.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 70 16 80580 80564 71.9259 66.5625 0.0171918 0.0138913 2024-09-02T10:35:10.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:10.093 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:10.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:10.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:09 smithi162 ceph-mon[20279]: osdmap e932: 12 total, 12 up, 11 in 2024-09-02T10:35:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 9842/227010 objects degraded (4.335%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:35:10.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:10.218 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:10.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:10.320 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:10.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:10.352 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:10.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:10.453 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:10.483 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:10.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:10.585 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:10.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:10.626 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:10.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:10.655 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2125,"stamp":"2024-09-02T10:34:53.594184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4139942432,"num_objects":63165,"num_object_clones":0,"num_object_copies":189495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11776,"num_objects_misplaced":13296,"num_objects_unfound":0,"num_objects_dirty":63165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45637,"num_write_kb":2920672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1214,"num_bytes_recovered":80229984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104049,"ondisk_log_size":104049,"up":171,"acting":161,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":20051932,"kb_used_data":12203700,"kb_used_omap":17,"kb_used_meta":7847982,"kb_avail":1011144740,"statfs":{"total":1055945392128,"available":1035412213760,"internally_reserved":0,"allocated":12496588800,"data_stored":12490869226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17545,"internal_metadata":8036334455},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,0,2,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":300187648,"num_objects":4582,"num_object_clones":0,"num_object_copies":13746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11776,"num_objects_misplaced":13296,"num_objects_unfound":0,"num_objects_dirty":4582,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-12944,"num_write_kb":-828512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-502,"num_bytes_recovered":-32899072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001327"},"pg_stats":[{"pgid":"9.17","version":"925'3755","reported_seq":3834,"reported_epoch":927,"state":"remapped+peering","last_fresh":"2024-09-02T10:34:49.030299+0000","last_change":"2024-09-02T10:34:49.030299+0000","last_active":"2024-09-02T10:34:49.030299+0000","last_peered":"2024-09-02T10:34:48.002702+0000","last_clean":"2024-09-02T10:34:48.002702+0000","last_became_active":"2024-09-02T10:34:47.047333+0000","last_became_peered":"2024-09-02T10:34:47.047333+0000","last_unstale":"2024-09-02T10:34:49.030299+0000","last_undegraded":"2024-09-02T10:34:49.030299+0000","last_fullsized":"2024-09-02T10:34:49.030299+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3755,"log_dups_size":0,"ondisk_log_size":3755,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124813312,"num_objects":1905,"num_object_clones":0,"num_object_copies":5715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1905,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1905,"num_write_kb":121888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"925'3833","reported_seq":3917,"reported_epoch":928,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-02T10:34:49.606972+0000","last_change":"2024-09-02T10:34:49.606972+0000","last_active":"2024-09-02T10:34:49.606972+0000","last_peered":"2024-09-02T10:34:49.606972+0000","last_clean":"2024-09-02T10:34:48.015158+0000","last_became_active":"2024-09-02T10:34:49.566014+0000","last_became_peered":"2024-09-02T10:34:49.566014+0000","last_unstale":"2024-09-02T10:34:49.606972+0000","last_undegraded":"2024-09-02T10:34:49.557689+0000","last_fullsized":"2024-09-02T10:34:49.368232+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127434752,"num_objects":1945,"num_object_clones":0,"num_object_copies":5835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3870,"num_objects_misplaced":1935,"num_objects_unfound":0,"num_objects_dirty":1945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1945,"num_write_kb":124448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[1,2],"avail_no_missing":["1","5","10"],"object_location_counts":[{"shards":"1,2","objects":10},{"shards":"1,5,10","objects":1935}],"blocked_by":[],"up_primary":9,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1e","version":"22'92","reported_seq":56,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.367592+0000","last_change":"2024-09-02T10:34:29.260151+0000","last_active":"2024-09-02T10:34:49.367592+0000","last_peered":"2024-09-02T10:34:49.367592+0000","last_clean":"2024-09-02T10:34:49.367592+0000","last_became_active":"2024-09-02T10:34:17.880876+0000","last_became_peered":"2024-09-02T10:34:17.880876+0000","last_unstale":"2024-09-02T10:34:49.367592+0000","last_undegraded":"2024-09-02T10:34:49.367592+0000","last_fullsized":"2024-09-02T10:34:49.367592+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:29.260104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:29.260104+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:31.804926+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.15","version":"928'3848","reported_seq":4170,"reported_epoch":928,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:34:51.591305+0000","last_change":"2024-09-02T10:34:49.604616+0000","last_active":"2024-09-02T10:34:51.591305+0000","last_peered":"2024-09-02T10:34:51.591305+0000","last_clean":"2024-09-02T10:34:48.004469+0000","last_became_active":"2024-09-02T10:34:49.578169+0000","last_became_peered":"2024-09-02T10:34:49.578169+0000","last_unstale":"2024-09-02T10:34:51.591305+0000","last_undegraded":"2024-09-02T10:34:51.591305+0000","last_fullsized":"2024-09-02T10:34:49.369803+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129695744,"num_objects":1979,"num_object_clones":0,"num_object_copies":5937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3696,"num_objects_unfound":0,"num_objects_dirty":1979,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1979,"num_write_kb":126656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":5177344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,4],"acting":[7,4],"avail_no_missing":["7","1","5"],"object_location_counts":[{"shards":"1,5,7","objects":1848},{"shards":"4,7","objects":131}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":423,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.366586+0000","last_change":"2024-09-02T10:34:00.928724+0000","last_active":"2024-09-02T10:34:49.366586+0000","last_peered":"2024-09-02T10:34:49.366586+0000","last_clean":"2024-09-02T10:34:49.366586+0000","last_became_active":"2024-09-02T10:33:38.659815+0000","last_became_peered":"2024-09-02T10:33:38.659815+0000","last_unstale":"2024-09-02T10:34:49.366586+0000","last_undegraded":"2024-09-02T10:34:49.366586+0000","last_fullsized":"2024-09-02T10:34:49.366586+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.928661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.928661+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:26.870517+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"928'3863","reported_seq":3948,"reported_epoch":928,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:34:49.624954+0000","last_change":"2024-09-02T10:34:49.624954+0000","last_active":"2024-09-02T10:34:49.624954+0000","last_peered":"2024-09-02T10:34:49.624954+0000","last_clean":"2024-09-02T10:34:47.929009+0000","last_became_active":"2024-09-02T10:34:49.545468+0000","last_became_peered":"2024-09-02T10:34:49.545468+0000","last_unstale":"2024-09-02T10:34:49.624954+0000","last_undegraded":"2024-09-02T10:34:49.624954+0000","last_fullsized":"2024-09-02T10:34:49.368158+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3863,"log_dups_size":0,"ondisk_log_size":3863,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127991808,"num_objects":1953,"num_object_clones":0,"num_object_copies":5859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3946,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1953,"num_write_kb":124992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,3],"acting":[1,3],"avail_no_missing":["1","4","8"],"object_location_counts":[{"shards":"1,3","objects":-20},{"shards":"1,4,8","objects":1973}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1544,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.367768+0000","last_change":"2024-09-02T10:33:59.232980+0000","last_active":"2024-09-02T10:34:49.367768+0000","last_peered":"2024-09-02T10:34:49.367768+0000","last_clean":"2024-09-02T10:34:49.367768+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:34:49.367768+0000","last_undegraded":"2024-09-02T10:34:49.367768+0000","last_fullsized":"2024-09-02T10:34:49.367768+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:59.232916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:59.232916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.466149+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"925'3966","reported_seq":4041,"reported_epoch":927,"state":"remapped+peering","last_fresh":"2024-09-02T10:34:49.030232+0000","last_change":"2024-09-02T10:34:49.030232+0000","last_active":"2024-09-02T10:34:49.030232+0000","last_peered":"2024-09-02T10:34:48.017248+0000","last_clean":"2024-09-02T10:34:48.017248+0000","last_became_active":"2024-09-02T10:34:47.047141+0000","last_became_peered":"2024-09-02T10:34:47.047141+0000","last_unstale":"2024-09-02T10:34:49.030232+0000","last_undegraded":"2024-09-02T10:34:49.030232+0000","last_fullsized":"2024-09-02T10:34:49.030232+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131727360,"num_objects":2010,"num_object_clones":0,"num_object_copies":6030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2010,"num_write_kb":128640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1520,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.367453+0000","last_change":"2024-09-02T10:33:55.174212+0000","last_active":"2024-09-02T10:34:49.367453+0000","last_peered":"2024-09-02T10:34:49.367453+0000","last_clean":"2024-09-02T10:34:49.367453+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:34:49.367453+0000","last_undegraded":"2024-09-02T10:34:49.367453+0000","last_fullsized":"2024-09-02T10:34:49.367453+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.174147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.174147+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:22.067581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.12","version":"925'3827","reported_seq":3906,"reported_epoch":928,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-09-02T10:34:49.544799+0000","last_change":"2024-09-02T10:34:49.544799+0000","last_active":"2024-09-02T10:34:49.032573+0000","last_peered":"2024-09-02T10:34:47.998416+0000","last_clean":"2024-09-02T10:34:47.998416+0000","last_became_active":"2024-09-02T10:34:47.045913+0000","last_became_peered":"2024-09-02T10:34:47.045913+0000","last_unstale":"2024-09-02T10:34:49.544799+0000","last_undegraded":"2024-09-02T10:34:49.543652+0000","last_fullsized":"2024-09-02T10:34:49.368156+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127533056,"num_objects":1946,"num_object_clones":0,"num_object_copies":5838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3878,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1946,"num_write_kb":124544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[1,0],"avail_no_missing":["1","3","9"],"object_location_counts":[{"shards":"0,1","objects":7},{"shards":"1,3,9","objects":1939}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":839,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.529376+0000","last_change":"2024-09-02T10:33:55.588678+0000","last_active":"2024-09-02T10:34:49.529376+0000","last_peered":"2024-09-02T10:34:49.529376+0000","last_clean":"2024-09-02T10:34:49.529376+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:34:49.529376+0000","last_undegraded":"2024-09-02T10:34:49.529376+0000","last_fullsized":"2024-09-02T10:34:49.529376+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.588605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.588605+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:57.688518+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"925'3747","reported_seq":3823,"reported_epoch":927,"state":"remapped+peering","last_fresh":"2024-09-02T10:34:49.030416+0000","last_change":"2024-09-02T10:34:49.030416+0000","last_active":"2024-09-02T10:34:49.030416+0000","last_peered":"2024-09-02T10:34:48.007602+0000","last_clean":"2024-09-02T10:34:48.007602+0000","last_became_active":"2024-09-02T10:34:47.045993+0000","last_became_peered":"2024-09-02T10:34:47.045993+0000","last_unstale":"2024-09-02T10:34:49.030416+0000","last_undegraded":"2024-09-02T10:34:49.030416+0000","last_fullsized":"2024-09-02T10:34:49.030416+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3747,"log_dups_size":0,"ondisk_log_size":3747,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123928576,"num_objects":1891,"num_object_clones":0,"num_object_copies":5673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1891,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1891,"num_write_kb":121024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":1382,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.398889+0000","last_change":"2024-09-02T10:33:56.473614+0000","last_active":"2024-09-02T10:34:49.398889+0000","last_peered":"2024-09-02T10:34:49.398889+0000","last_clean":"2024-09-02T10:34:49.398889+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:34:49.398889+0000","last_undegraded":"2024-09-02T10:34:49.398889+0000","last_fullsized":"2024-09-02T10:34:49.398889+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.473555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.473555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:04.434869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.10","version":"925'3925","reported_seq":4005,"reported_epoch":928,"state":"activating+undersized+degraded+remapped","last_fresh":"2024-09-02T10:34:49.612746+0000","last_change":"2024-09-02T10:34:49.558909+0000","last_active":"2024-09-02T10:34:49.032545+0000","last_peered":"2024-09-02T10:34:48.025225+0000","last_clean":"2024-09-02T10:34:48.025225+0000","last_became_active":"2024-09-02T10:34:47.039732+0000","last_became_peered":"2024-09-02T10:34:47.039732+0000","last_unstale":"2024-09-02T10:34:49.612746+0000","last_undegraded":"2024-09-02T10:34:49.555323+0000","last_fullsized":"2024-09-02T10:34:49.368475+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3925,"log_dups_size":0,"ondisk_log_size":3925,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130875392,"num_objects":1997,"num_object_clones":0,"num_object_copies":5991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4028,"num_objects_misplaced":2014,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1997,"num_write_kb":127808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[1,0],"avail_no_missing":["1","6","8"],"object_location_counts":[{"shards":"0,1","objects":-17},{"shards":"1,6,8","objects":2014}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1900,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.369706+0000","last_change":"2024-09-02T10:34:00.430544+0000","last_active":"2024-09-02T10:34:49.369706+0000","last_peered":"2024-09-02T10:34:49.369706+0000","last_clean":"2024-09-02T10:34:49.369706+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:34:49.369706+0000","last_undegraded":"2024-09-02T10:34:49.369706+0000","last_fullsized":"2024-09-02T10:34:49.369706+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.430459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.430459+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:19.434576+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1543,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.367709+0000","last_change":"2024-09-02T10:33:56.194537+0000","last_active":"2024-09-02T10:34:49.367709+0000","last_peered":"2024-09-02T10:34:49.367709+0000","last_clean":"2024-09-02T10:34:49.367709+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:34:49.367709+0000","last_undegraded":"2024-09-02T10:34:49.367709+0000","last_fullsized":"2024-09-02T10:34:49.367709+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.194480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.194480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:02.488817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":102,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.675074+0000","last_change":"2024-09-02T10:33:56.167103+0000","last_active":"2024-09-02T10:34:49.675074+0000","last_peered":"2024-09-02T10:34:49.675074+0000","last_clean":"2024-09-02T10:34:49.675074+0000","last_became_active":"2024-09-02T10:33:38.657147+0000","last_became_peered":"2024-09-02T10:33:38.657147+0000","last_unstale":"2024-09-02T10:34:49.675074+0000","last_undegraded":"2024-09-02T10:34:49.675074+0000","last_fullsized":"2024-09-02T10:34:49.675074+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.167029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.167029+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.270011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1510,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.424561+0000","last_change":"2024-09-02T10:33:55.173514+0000","last_active":"2024-09-02T10:34:49.424561+0000","last_peered":"2024-09-02T10:34:49.424561+0000","last_clean":"2024-09-02T10:34:49.424561+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:34:49.424561+0000","last_undegraded":"2024-09-02T10:34:49.424561+0000","last_fullsized":"2024-09-02T10:34:49.424561+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.173450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.173450+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:59.387368+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":754,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.366656+0000","last_change":"2024-09-02T10:33:56.943878+0000","last_active":"2024-09-02T10:34:49.366656+0000","last_peered":"2024-09-02T10:34:49.366656+0000","last_clean":"2024-09-02T10:34:49.366656+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:34:49.366656+0000","last_undegraded":"2024-09-02T10:34:49.366656+0000","last_fullsized":"2024-09-02T10:34:49.366656+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.943826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.943826+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.719187+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":577,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.396254+0000","last_change":"2024-09-02T10:34:36.258204+0000","last_active":"2024-09-02T10:34:49.396254+0000","last_peered":"2024-09-02T10:34:49.396254+0000","last_clean":"2024-09-02T10:34:49.396254+0000","last_became_active":"2024-09-02T10:33:38.657430+0000","last_became_peered":"2024-09-02T10:33:38.657430+0000","last_unstale":"2024-09-02T10:34:49.396254+0000","last_undegraded":"2024-09-02T10:34:49.396254+0000","last_fullsized":"2024-09-02T10:34:49.396254+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:36.258130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:36.258130+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:56.899533+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1799,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.369982+0000","last_change":"2024-09-02T10:34:08.628331+0000","last_active":"2024-09-02T10:34:49.369982+0000","last_peered":"2024-09-02T10:34:49.369982+0000","last_clean":"2024-09-02T10:34:49.369982+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:34:49.369982+0000","last_undegraded":"2024-09-02T10:34:49.369982+0000","last_fullsized":"2024-09-02T10:34:49.369982+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.628247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.628247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:31.896890+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"925'3853","reported_seq":7369,"reported_epoch":927,"state":"remapped+peering","last_fresh":"2024-09-02T10:34:49.030943+0000","last_change":"2024-09-02T10:34:49.030943+0000","last_active":"2024-09-02T10:34:48.034532+0000","last_peered":"2024-09-02T10:34:48.012916+0000","last_clean":"2024-09-02T10:34:48.012916+0000","last_became_active":"2024-09-02T10:34:47.052335+0000","last_became_peered":"2024-09-02T10:34:47.052335+0000","last_unstale":"2024-09-02T10:34:49.030943+0000","last_undegraded":"2024-09-02T10:34:49.030943+0000","last_fullsized":"2024-09-02T10:34:49.030943+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3853,"log_dups_size":0,"ondisk_log_size":3853,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127631360,"num_objects":1948,"num_object_clones":0,"num_object_copies":5844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1948,"num_write_kb":124640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":56098816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":539,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.675397+0000","last_change":"2024-09-02T10:34:01.246043+0000","last_active":"2024-09-02T10:34:49.675397+0000","last_peered":"2024-09-02T10:34:49.675397+0000","last_clean":"2024-09-02T10:34:49.675397+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:34:49.675397+0000","last_undegraded":"2024-09-02T10:34:49.675397+0000","last_fullsized":"2024-09-02T10:34:49.675397+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.245989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.245989+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.756195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"928'4017","reported_seq":4646,"reported_epoch":928,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:34:52.718273+0000","last_change":"2024-09-02T10:34:49.486439+0000","last_active":"2024-09-02T10:34:52.718273+0000","last_peered":"2024-09-02T10:34:52.718273+0000","last_clean":"2024-09-02T10:34:48.014805+0000","last_became_active":"2024-09-02T10:34:49.453529+0000","last_became_peered":"2024-09-02T10:34:49.453529+0000","last_unstale":"2024-09-02T10:34:52.718273+0000","last_undegraded":"2024-09-02T10:34:52.718273+0000","last_fullsized":"2024-09-02T10:34:49.369952+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136249344,"num_objects":2079,"num_object_clones":0,"num_object_copies":6237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1705,"num_objects_unfound":0,"num_objects_dirty":2079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2079,"num_write_kb":133056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":17891328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,5],"acting":[5,7],"avail_no_missing":["5","7","9"],"object_location_counts":[{"shards":"5,7","objects":374},{"shards":"5,7,9","objects":1705}],"blocked_by":[],"up_primary":10,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1769,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.370153+0000","last_change":"2024-09-02T10:34:05.558912+0000","last_active":"2024-09-02T10:34:49.370153+0000","last_peered":"2024-09-02T10:34:49.370153+0000","last_clean":"2024-09-02T10:34:49.370153+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:34:49.370153+0000","last_undegraded":"2024-09-02T10:34:49.370153+0000","last_fullsized":"2024-09-02T10:34:49.370153+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:05.558852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:05.558852+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:06.139483+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"928'3983","reported_seq":4095,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:52.509813+0000","last_change":"2024-09-02T10:34:49.121880+0000","last_active":"2024-09-02T10:34:52.509813+0000","last_peered":"2024-09-02T10:34:52.509813+0000","last_clean":"2024-09-02T10:34:52.509813+0000","last_became_active":"2024-09-02T10:34:47.026177+0000","last_became_peered":"2024-09-02T10:34:47.026177+0000","last_unstale":"2024-09-02T10:34:52.509813+0000","last_undegraded":"2024-09-02T10:34:52.509813+0000","last_fullsized":"2024-09-02T10:34:52.509813+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'3905","last_scrub_stamp":"2024-09-02T10:34:49.121789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.121789+0000","objects_scrubbed":1960,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:58.536597+0000","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133562368,"num_objects":2038,"num_object_clones":0,"num_object_copies":6114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78,"num_write_kb":4992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,8],"acting":[6,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1864,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.370321+0000","last_change":"2024-09-02T10:34:07.590524+0000","last_active":"2024-09-02T10:34:49.370321+0000","last_peered":"2024-09-02T10:34:49.370321+0000","last_clean":"2024-09-02T10:34:49.370321+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:34:49.370321+0000","last_undegraded":"2024-09-02T10:34:49.370321+0000","last_fullsized":"2024-09-02T10:34:49.370321+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:07.590464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:07.590464+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:30.846352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"926'3760","reported_seq":3843,"reported_epoch":927,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.030654+0000","last_change":"2024-09-02T10:34:47.425374+0000","last_active":"2024-09-02T10:34:49.030654+0000","last_peered":"2024-09-02T10:34:49.030654+0000","last_clean":"2024-09-02T10:34:49.030654+0000","last_became_active":"2024-09-02T10:34:47.038024+0000","last_became_peered":"2024-09-02T10:34:47.038024+0000","last_unstale":"2024-09-02T10:34:49.030654+0000","last_undegraded":"2024-09-02T10:34:49.030654+0000","last_fullsized":"2024-09-02T10:34:49.030654+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124780544,"num_objects":1904,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1904,"num_write_kb":121856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":1982,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.370896+0000","last_change":"2024-09-02T10:30:24.407289+0000","last_active":"2024-09-02T10:34:49.370896+0000","last_peered":"2024-09-02T10:34:49.370896+0000","last_clean":"2024-09-02T10:34:49.370896+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:34:49.370896+0000","last_undegraded":"2024-09-02T10:34:49.370896+0000","last_fullsized":"2024-09-02T10:34:49.370896+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.407218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.407218+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"925'3826","reported_seq":3934,"reported_epoch":927,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.029933+0000","last_change":"2024-09-02T10:34:48.857422+0000","last_active":"2024-09-02T10:34:49.029933+0000","last_peered":"2024-09-02T10:34:49.029933+0000","last_clean":"2024-09-02T10:34:49.029933+0000","last_became_active":"2024-09-02T10:34:47.038074+0000","last_became_peered":"2024-09-02T10:34:47.038074+0000","last_unstale":"2024-09-02T10:34:49.029933+0000","last_undegraded":"2024-09-02T10:34:49.029933+0000","last_fullsized":"2024-09-02T10:34:49.029933+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3826","last_scrub_stamp":"2024-09-02T10:34:48.857333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:48.857333+0000","objects_scrubbed":1952,"log_size":3826,"log_dups_size":0,"ondisk_log_size":3826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:55.613240+0000","scrub_duration":482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127926272,"num_objects":1952,"num_object_clones":0,"num_object_copies":5856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1640,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.390127+0000","last_change":"2024-09-02T10:33:56.586452+0000","last_active":"2024-09-02T10:34:49.390127+0000","last_peered":"2024-09-02T10:34:49.390127+0000","last_clean":"2024-09-02T10:34:49.390127+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:34:49.390127+0000","last_undegraded":"2024-09-02T10:34:49.390127+0000","last_fullsized":"2024-09-02T10:34:49.390127+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.586392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.586392+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:20.658780+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"928'4066","reported_seq":4176,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:53.018367+0000","last_change":"2024-09-02T10:34:49.856610+0000","last_active":"2024-09-02T10:34:53.018367+0000","last_peered":"2024-09-02T10:34:53.018367+0000","last_clean":"2024-09-02T10:34:53.018367+0000","last_became_active":"2024-09-02T10:34:47.042672+0000","last_became_peered":"2024-09-02T10:34:47.042672+0000","last_unstale":"2024-09-02T10:34:53.018367+0000","last_undegraded":"2024-09-02T10:34:53.018367+0000","last_fullsized":"2024-09-02T10:34:53.018367+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'3969","last_scrub_stamp":"2024-09-02T10:34:49.856383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.856383+0000","objects_scrubbed":1996,"log_size":4066,"log_dups_size":0,"ondisk_log_size":4066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:55.854139+0000","scrub_duration":330,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137363456,"num_objects":2096,"num_object_clones":0,"num_object_copies":6288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":617,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.674943+0000","last_change":"2024-09-02T10:34:08.328616+0000","last_active":"2024-09-02T10:34:49.674943+0000","last_peered":"2024-09-02T10:34:49.674943+0000","last_clean":"2024-09-02T10:34:49.674943+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:34:49.674943+0000","last_undegraded":"2024-09-02T10:34:49.674943+0000","last_fullsized":"2024-09-02T10:34:49.674943+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.328570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.328570+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.052868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"926'3790","reported_seq":3864,"reported_epoch":927,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.029853+0000","last_change":"2024-09-02T10:33:52.761107+0000","last_active":"2024-09-02T10:34:49.029853+0000","last_peered":"2024-09-02T10:34:49.029853+0000","last_clean":"2024-09-02T10:34:49.029853+0000","last_became_active":"2024-09-02T10:33:52.760804+0000","last_became_peered":"2024-09-02T10:33:52.760804+0000","last_unstale":"2024-09-02T10:34:49.029853+0000","last_undegraded":"2024-09-02T10:34:49.029853+0000","last_fullsized":"2024-09-02T10:34:49.029853+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:18.328079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":248381440,"num_objects":3790,"num_object_clones":0,"num_object_copies":11370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3790,"num_write_kb":242560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":947,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.367576+0000","last_change":"2024-09-02T10:34:17.882133+0000","last_active":"2024-09-02T10:34:49.367576+0000","last_peered":"2024-09-02T10:34:49.367576+0000","last_clean":"2024-09-02T10:34:49.367576+0000","last_became_active":"2024-09-02T10:34:17.881745+0000","last_became_peered":"2024-09-02T10:34:17.881745+0000","last_unstale":"2024-09-02T10:34:49.367576+0000","last_undegraded":"2024-09-02T10:34:49.367576+0000","last_fullsized":"2024-09-02T10:34:49.367576+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:58.800694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:58.800694+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:25.599849+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"928'3915","reported_seq":3990,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:51.593026+0000","last_change":"2024-09-02T10:33:52.926793+0000","last_active":"2024-09-02T10:34:51.593026+0000","last_peered":"2024-09-02T10:34:51.593026+0000","last_clean":"2024-09-02T10:34:51.593026+0000","last_became_active":"2024-09-02T10:33:52.926467+0000","last_became_peered":"2024-09-02T10:33:52.926467+0000","last_unstale":"2024-09-02T10:34:51.593026+0000","last_undegraded":"2024-09-02T10:34:51.593026+0000","last_fullsized":"2024-09-02T10:34:51.593026+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3915,"log_dups_size":0,"ondisk_log_size":3915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:34:52.202014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256573440,"num_objects":3915,"num_object_clones":0,"num_object_copies":11745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3915,"num_write_kb":250560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":1404,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.396320+0000","last_change":"2024-09-02T10:34:08.280819+0000","last_active":"2024-09-02T10:34:49.396320+0000","last_peered":"2024-09-02T10:34:49.396320+0000","last_clean":"2024-09-02T10:34:49.396320+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:34:49.396320+0000","last_undegraded":"2024-09-02T10:34:49.396320+0000","last_fullsized":"2024-09-02T10:34:49.396320+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.280762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.280762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:19.946116+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"928'3898","reported_seq":3974,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:51.584785+0000","last_change":"2024-09-02T10:33:52.925479+0000","last_active":"2024-09-02T10:34:51.584785+0000","last_peered":"2024-09-02T10:34:51.584785+0000","last_clean":"2024-09-02T10:34:51.584785+0000","last_became_active":"2024-09-02T10:33:52.925170+0000","last_became_peered":"2024-09-02T10:33:52.925170+0000","last_unstale":"2024-09-02T10:34:51.584785+0000","last_undegraded":"2024-09-02T10:34:51.584785+0000","last_fullsized":"2024-09-02T10:34:51.584785+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3898,"log_dups_size":0,"ondisk_log_size":3898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:21.416932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255459328,"num_objects":3898,"num_object_clones":0,"num_object_copies":11694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3898,"num_write_kb":249472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":1369,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.396546+0000","last_change":"2024-09-02T10:34:05.282359+0000","last_active":"2024-09-02T10:34:49.396546+0000","last_peered":"2024-09-02T10:34:49.396546+0000","last_clean":"2024-09-02T10:34:49.396546+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:34:49.396546+0000","last_undegraded":"2024-09-02T10:34:49.396546+0000","last_fullsized":"2024-09-02T10:34:49.396546+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:05.282307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:05.282307+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:12.971380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"928'4035","reported_seq":4111,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:52.686975+0000","last_change":"2024-09-02T10:33:52.925486+0000","last_active":"2024-09-02T10:34:52.686975+0000","last_peered":"2024-09-02T10:34:52.686975+0000","last_clean":"2024-09-02T10:34:52.686975+0000","last_became_active":"2024-09-02T10:33:52.925153+0000","last_became_peered":"2024-09-02T10:33:52.925153+0000","last_unstale":"2024-09-02T10:34:52.686975+0000","last_undegraded":"2024-09-02T10:34:52.686975+0000","last_fullsized":"2024-09-02T10:34:52.686975+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":4035,"log_dups_size":0,"ondisk_log_size":4035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:09.789775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264437760,"num_objects":4035,"num_object_clones":0,"num_object_copies":12105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4035,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4035,"num_write_kb":258240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1158,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.367245+0000","last_change":"2024-09-02T10:34:17.882691+0000","last_active":"2024-09-02T10:34:49.367245+0000","last_peered":"2024-09-02T10:34:49.367245+0000","last_clean":"2024-09-02T10:34:49.367245+0000","last_became_active":"2024-09-02T10:34:17.882376+0000","last_became_peered":"2024-09-02T10:34:17.882376+0000","last_unstale":"2024-09-02T10:34:49.367245+0000","last_undegraded":"2024-09-02T10:34:49.367245+0000","last_fullsized":"2024-09-02T10:34:49.367245+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.268512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.268512+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:22.220969+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"928'3903","reported_seq":7451,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:51.594613+0000","last_change":"2024-09-02T10:34:49.043289+0000","last_active":"2024-09-02T10:34:51.594613+0000","last_peered":"2024-09-02T10:34:51.594613+0000","last_clean":"2024-09-02T10:34:51.594613+0000","last_became_active":"2024-09-02T10:34:47.042191+0000","last_became_peered":"2024-09-02T10:34:47.042191+0000","last_unstale":"2024-09-02T10:34:51.594613+0000","last_undegraded":"2024-09-02T10:34:51.594613+0000","last_fullsized":"2024-09-02T10:34:51.594613+0000","mapping_epoch":918,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'3857","last_scrub_stamp":"2024-09-02T10:34:49.043221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.043221+0000","objects_scrubbed":1904,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:49.241029+0000","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127795200,"num_objects":1950,"num_object_clones":0,"num_object_copies":5850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,3],"acting":[10,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":1384,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.398785+0000","last_change":"2024-09-02T10:34:35.240578+0000","last_active":"2024-09-02T10:34:49.398785+0000","last_peered":"2024-09-02T10:34:49.398785+0000","last_clean":"2024-09-02T10:34:49.398785+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:34:49.398785+0000","last_undegraded":"2024-09-02T10:34:49.398785+0000","last_fullsized":"2024-09-02T10:34:49.398785+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:35.240504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:35.240504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:36.945732+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"928'4075","reported_seq":4151,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:52.706667+0000","last_change":"2024-09-02T10:33:52.925507+0000","last_active":"2024-09-02T10:34:52.706667+0000","last_peered":"2024-09-02T10:34:52.706667+0000","last_clean":"2024-09-02T10:34:52.706667+0000","last_became_active":"2024-09-02T10:33:52.925177+0000","last_became_peered":"2024-09-02T10:33:52.925177+0000","last_unstale":"2024-09-02T10:34:52.706667+0000","last_undegraded":"2024-09-02T10:34:52.706667+0000","last_fullsized":"2024-09-02T10:34:52.706667+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":4075,"log_dups_size":0,"ondisk_log_size":4075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:14.485757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267059200,"num_objects":4075,"num_object_clones":0,"num_object_copies":12225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4075,"num_write_kb":260800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1632,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.373489+0000","last_change":"2024-09-02T10:34:04.115873+0000","last_active":"2024-09-02T10:34:49.373489+0000","last_peered":"2024-09-02T10:34:49.373489+0000","last_clean":"2024-09-02T10:34:49.373489+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:34:49.373489+0000","last_undegraded":"2024-09-02T10:34:49.373489+0000","last_fullsized":"2024-09-02T10:34:49.373489+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:04.115830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:04.115830+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:04.432089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"928'3835","reported_seq":3911,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.577510+0000","last_change":"2024-09-02T10:33:52.750617+0000","last_active":"2024-09-02T10:34:49.577510+0000","last_peered":"2024-09-02T10:34:49.577510+0000","last_clean":"2024-09-02T10:34:49.577510+0000","last_became_active":"2024-09-02T10:33:52.750198+0000","last_became_peered":"2024-09-02T10:33:52.750198+0000","last_unstale":"2024-09-02T10:34:49.577510+0000","last_undegraded":"2024-09-02T10:34:49.577510+0000","last_fullsized":"2024-09-02T10:34:49.577510+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:07.980031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251330560,"num_objects":3835,"num_object_clones":0,"num_object_copies":11505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3835,"num_write_kb":245440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":924,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.367044+0000","last_change":"2024-09-02T10:33:59.302485+0000","last_active":"2024-09-02T10:34:49.367044+0000","last_peered":"2024-09-02T10:34:49.367044+0000","last_clean":"2024-09-02T10:34:49.367044+0000","last_became_active":"2024-09-02T10:33:38.664311+0000","last_became_peered":"2024-09-02T10:33:38.664311+0000","last_unstale":"2024-09-02T10:34:49.367044+0000","last_undegraded":"2024-09-02T10:34:49.367044+0000","last_fullsized":"2024-09-02T10:34:49.367044+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:59.302421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:59.302421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:05.057279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"928'4032","reported_seq":4146,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:53.054535+0000","last_change":"2024-09-02T10:34:49.255198+0000","last_active":"2024-09-02T10:34:53.054535+0000","last_peered":"2024-09-02T10:34:53.054535+0000","last_clean":"2024-09-02T10:34:53.054535+0000","last_became_active":"2024-09-02T10:34:47.044129+0000","last_became_peered":"2024-09-02T10:34:47.044129+0000","last_unstale":"2024-09-02T10:34:53.054535+0000","last_undegraded":"2024-09-02T10:34:53.054535+0000","last_fullsized":"2024-09-02T10:34:53.054535+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'3926","last_scrub_stamp":"2024-09-02T10:34:49.255099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.255099+0000","objects_scrubbed":1999,"log_size":4032,"log_dups_size":0,"ondisk_log_size":4032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:12.758346+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137953280,"num_objects":2105,"num_object_clones":0,"num_object_copies":6315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":6784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1643,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.371071+0000","last_change":"2024-09-02T10:34:01.291835+0000","last_active":"2024-09-02T10:34:49.371071+0000","last_peered":"2024-09-02T10:34:49.371071+0000","last_clean":"2024-09-02T10:34:49.371071+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:34:49.371071+0000","last_undegraded":"2024-09-02T10:34:49.371071+0000","last_fullsized":"2024-09-02T10:34:49.371071+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.291779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.291779+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:24.558074+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"928'3807","reported_seq":3918,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:51.727932+0000","last_change":"2024-09-02T10:34:49.259122+0000","last_active":"2024-09-02T10:34:51.727932+0000","last_peered":"2024-09-02T10:34:51.727932+0000","last_clean":"2024-09-02T10:34:51.727932+0000","last_became_active":"2024-09-02T10:34:47.039696+0000","last_became_peered":"2024-09-02T10:34:47.039696+0000","last_unstale":"2024-09-02T10:34:51.727932+0000","last_undegraded":"2024-09-02T10:34:51.727932+0000","last_fullsized":"2024-09-02T10:34:51.727932+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3753","last_scrub_stamp":"2024-09-02T10:34:49.259057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.259057+0000","objects_scrubbed":1910,"log_size":3807,"log_dups_size":0,"ondisk_log_size":3807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:15.936951+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128712704,"num_objects":1964,"num_object_clones":0,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1964,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1898,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.367250+0000","last_change":"2024-09-02T10:33:55.238766+0000","last_active":"2024-09-02T10:34:49.367250+0000","last_peered":"2024-09-02T10:34:49.367250+0000","last_clean":"2024-09-02T10:34:49.367250+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:34:49.367250+0000","last_undegraded":"2024-09-02T10:34:49.367250+0000","last_fullsized":"2024-09-02T10:34:49.367250+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.238683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.238683+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:13.350030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"928'3909","reported_seq":4021,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:52.702494+0000","last_change":"2024-09-02T10:34:50.216635+0000","last_active":"2024-09-02T10:34:52.702494+0000","last_peered":"2024-09-02T10:34:52.702494+0000","last_clean":"2024-09-02T10:34:52.702494+0000","last_became_active":"2024-09-02T10:34:47.033777+0000","last_became_peered":"2024-09-02T10:34:47.033777+0000","last_unstale":"2024-09-02T10:34:52.702494+0000","last_undegraded":"2024-09-02T10:34:52.702494+0000","last_fullsized":"2024-09-02T10:34:52.702494+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'3847","last_scrub_stamp":"2024-09-02T10:34:50.216498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:50.216498+0000","objects_scrubbed":1961,"log_size":3909,"log_dups_size":0,"ondisk_log_size":3909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:11.868424+0000","scrub_duration":552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133038080,"num_objects":2030,"num_object_clones":0,"num_object_copies":6090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69,"num_write_kb":4416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":847,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.529852+0000","last_change":"2024-09-02T10:34:02.569694+0000","last_active":"2024-09-02T10:34:49.529852+0000","last_peered":"2024-09-02T10:34:49.529852+0000","last_clean":"2024-09-02T10:34:49.529852+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:34:49.529852+0000","last_undegraded":"2024-09-02T10:34:49.529852+0000","last_fullsized":"2024-09-02T10:34:49.529852+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:02.569628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:02.569628+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:27.473229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"928'3865","reported_seq":3975,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:52.615575+0000","last_change":"2024-09-02T10:34:49.168775+0000","last_active":"2024-09-02T10:34:52.615575+0000","last_peered":"2024-09-02T10:34:52.615575+0000","last_clean":"2024-09-02T10:34:52.615575+0000","last_became_active":"2024-09-02T10:34:47.049681+0000","last_became_peered":"2024-09-02T10:34:47.049681+0000","last_unstale":"2024-09-02T10:34:52.615575+0000","last_undegraded":"2024-09-02T10:34:52.615575+0000","last_fullsized":"2024-09-02T10:34:52.615575+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3787","last_scrub_stamp":"2024-09-02T10:34:49.168686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.168686+0000","objects_scrubbed":1913,"log_size":3865,"log_dups_size":0,"ondisk_log_size":3865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:16.457037+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130482176,"num_objects":1991,"num_object_clones":0,"num_object_copies":5973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78,"num_write_kb":4992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1662,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.424715+0000","last_change":"2024-09-02T10:30:46.402988+0000","last_active":"2024-09-02T10:34:49.424715+0000","last_peered":"2024-09-02T10:34:49.424715+0000","last_clean":"2024-09-02T10:34:49.424715+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:34:49.424715+0000","last_undegraded":"2024-09-02T10:34:49.424715+0000","last_fullsized":"2024-09-02T10:34:49.424715+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:46.402901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.402901+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"928'3864","reported_seq":3976,"reported_epoch":928,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.530982+0000","last_change":"2024-09-02T10:34:48.713406+0000","last_active":"2024-09-02T10:34:49.530982+0000","last_peered":"2024-09-02T10:34:49.530982+0000","last_clean":"2024-09-02T10:34:49.530982+0000","last_became_active":"2024-09-02T10:34:47.029634+0000","last_became_peered":"2024-09-02T10:34:47.029634+0000","last_unstale":"2024-09-02T10:34:49.530982+0000","last_undegraded":"2024-09-02T10:34:49.530982+0000","last_fullsized":"2024-09-02T10:34:49.530982+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3863","last_scrub_stamp":"2024-09-02T10:34:48.713331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:48.713331+0000","objects_scrubbed":1931,"log_size":3864,"log_dups_size":0,"ondisk_log_size":3864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:15.136168+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126615552,"num_objects":1932,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":1059,"reported_epoch":927,"state":"active+clean","last_fresh":"2024-09-02T10:34:49.029977+0000","last_change":"2024-09-02T10:33:55.336238+0000","last_active":"2024-09-02T10:34:49.029977+0000","last_peered":"2024-09-02T10:34:49.029977+0000","last_clean":"2024-09-02T10:34:49.029977+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:34:49.029977+0000","last_undegraded":"2024-09-02T10:34:49.029977+0000","last_fullsized":"2024-09-02T10:34:49.029977+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.336182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.336182+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:35:16.545175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":4139352064,"num_objects":63163,"num_object_clones":0,"num_object_copies":189489,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11776,"num_objects_misplaced":13296,"num_objects_unfound":0,"num_objects_dirty":63163,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45637,"num_write_kb":2920672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1210,"num_bytes_recovered":79298560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12665618432,"data_stored":12665618432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101197,"ondisk_log_size":101197,"up":78,"acting":68,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":343597384097,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532613030,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1886332,"kb_used_data":1263716,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91858820,"statfs":{"total":95995035648,"available":94063431680,"internally_reserved":0,"allocated":1294045184,"data_stored":1293524708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":637532608},"hb_peers":[0,1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841963,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2422708,"kb_used_data":1543268,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":91322444,"statfs":{"total":95995035648,"available":93514182656,"internally_reserved":0,"allocated":1580306432,"data_stored":1579785956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":900528576},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070897,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1251380,"kb_used_data":750936,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92493772,"statfs":{"total":95995035648,"available":94713622528,"internally_reserved":0,"allocated":768958464,"data_stored":768442148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299830,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2264900,"kb_used_data":1521516,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":91480252,"statfs":{"total":95995035648,"available":93675778048,"internally_reserved":0,"allocated":1558032384,"data_stored":1557508356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":761199050},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528762,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1071924,"kb_used_data":531236,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92673228,"statfs":{"total":95995035648,"available":94897385472,"internally_reserved":0,"allocated":543985664,"data_stored":543465188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":553646535},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724991,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3223868,"kb_used_data":2340268,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":90521284,"statfs":{"total":95995035648,"available":92693794816,"internally_reserved":0,"allocated":2396434432,"data_stored":2395910404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953923,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":994304,"kb_used_data":493860,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":92750848,"statfs":{"total":95995035648,"available":94976868352,"internally_reserved":0,"allocated":505712640,"data_stored":505192164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150153,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2374308,"kb_used_data":1486540,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":91370844,"statfs":{"total":95995035648,"available":93563744256,"internally_reserved":0,"allocated":1522216960,"data_stored":1521704482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":909048264},"hb_peers":[0,1,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411789,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":621436,"kb_used_data":8556,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":93123716,"statfs":{"total":95995035648,"available":95358685184,"internally_reserved":0,"allocated":8761344,"data_stored":8237316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575313,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3051760,"kb_used_data":2236184,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":90693392,"statfs":{"total":95995035648,"available":92870033408,"internally_reserved":0,"allocated":2289852416,"data_stored":2289336100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":835123653},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738840,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889012,"kb_used_data":27620,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92856140,"statfs":{"total":95995035648,"available":95084687360,"internally_reserved":0,"allocated":28282880,"data_stored":27762404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":882047429},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33161216,"data_stored":33161216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2289369088,"data_stored":2289369088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":9699328,"data_stored":9699328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1520566272,"data_stored":1520566272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":505479168,"data_stored":505479168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2399010816,"data_stored":2399010816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":554696704,"data_stored":554696704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1570897920,"data_stored":1570897920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":768802816,"data_stored":768802816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1592131584,"data_stored":1592131584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1309671424,"data_stored":1309671424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":112132096,"data_stored":112132096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:35:10.687 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:35:10.727 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:10.753 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:10.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:10.754 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:10.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:10.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:10 smithi151 ceph-mon[16639]: 9.18 scrub starts 2024-09-02T10:35:10.863 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:10 smithi151 ceph-mon[16639]: pgmap v2137: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s; 9842/234132 objects degraded (4.204%); 18715/234132 objects misplaced (7.993%); 27 MiB/s, 426 objects/s recovering 2024-09-02T10:35:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:10.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:10.881 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:10.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-02T10:35:10.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:11.006 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:10 smithi187 ceph-mon[20072]: 9.18 scrub starts 2024-09-02T10:35:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:10 smithi187 ceph-mon[20072]: pgmap v2137: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s; 9842/234132 objects degraded (4.204%); 18715/234132 objects misplaced (7.993%); 27 MiB/s, 426 objects/s recovering 2024-09-02T10:35:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:11.007 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:11.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 71 16 81665 81649 71.8679 67.8125 0.0187578 0.0139066 2024-09-02T10:35:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:10 smithi162 ceph-mon[20279]: 9.18 scrub starts 2024-09-02T10:35:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:10 smithi162 ceph-mon[20279]: pgmap v2137: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 65 MiB/s wr, 1.03k op/s; 9842/234132 objects degraded (4.204%); 18715/234132 objects misplaced (7.993%); 27 MiB/s, 426 objects/s recovering 2024-09-02T10:35:11.108 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:11.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:11.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:11.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:11.270 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:11.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:11.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:11.372 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:11.402 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:11.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:11.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:11.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:11.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:11.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:11.662 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:11.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:11.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:11.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:35:11.764 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:11.789 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:11.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:11.790 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:11.891 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:11.928 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:11.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:12.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:11 smithi187 ceph-mon[20072]: 9.18 scrub ok 2024-09-02T10:35:12.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 72 16 82564 82548 71.6501 56.1875 0.00551743 0.0139516 2024-09-02T10:35:12.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:12.055 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:12.056 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:12.064 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:11 smithi151 ceph-mon[16639]: 9.18 scrub ok 2024-09-02T10:35:12.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:11 smithi162 ceph-mon[20279]: 9.18 scrub ok 2024-09-02T10:35:12.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:12.183 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:12.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:12.184 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:12.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:12.315 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:12.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:12.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:12.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:12.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:12.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:12.573 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:12.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:12.574 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:12.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:12.700 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:12.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:12.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:12.802 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:12.823 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:12.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:12.925 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:12.958 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:12.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:12.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:13.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:12 smithi187 ceph-mon[20072]: pgmap v2138: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 55 MiB/s wr, 872 op/s; 9842/237081 objects degraded (4.151%); 16766/237081 objects misplaced (7.072%); 26 MiB/s, 417 objects/s recovering 2024-09-02T10:35:13.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 73 15 83497 83482 71.4682 58.375 0.0110328 0.0139879 2024-09-02T10:35:13.060 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:13.082 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:13.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:12 smithi162 ceph-mon[20279]: pgmap v2138: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 55 MiB/s wr, 872 op/s; 9842/237081 objects degraded (4.151%); 16766/237081 objects misplaced (7.072%); 26 MiB/s, 417 objects/s recovering 2024-09-02T10:35:13.102 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:12 smithi151 ceph-mon[16639]: pgmap v2138: 57 pgs: 7 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 47 active+clean; 4.8 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 55 MiB/s wr, 872 op/s; 9842/237081 objects degraded (4.151%); 16766/237081 objects misplaced (7.072%); 26 MiB/s, 417 objects/s recovering 2024-09-02T10:35:13.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:13.209 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:13.210 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:13.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:13.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:13.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:13.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:13.474 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:13.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:13.475 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:13.576 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:13.605 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:13.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:13.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:13.707 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:13.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:13.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:13.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:13.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:13.870 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:13.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:13.994 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:13.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:13.995 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:14.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 74 16 84427 84411 71.287 58.0625 0.00677956 0.014018 2024-09-02T10:35:14.096 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:14.124 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:14.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:14.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:14.226 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:14.247 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:14.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:14.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:14.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:14.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:14.476 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:14.505 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:14.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:14.607 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:14.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:14.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:14.631 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:14.732 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:14.760 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:14.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:14.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:14.886 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:14.887 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:14.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:14 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:35:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:14 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:35:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:14 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:35:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:14 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:35:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:14 smithi187 ceph-mon[20072]: pgmap v2139: 57 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 48 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 35 MiB/s wr, 565 op/s; 9842/244401 objects degraded (4.027%); 15760/244401 objects misplaced (6.448%); 6.4 MiB/s, 102 objects/s recovering 2024-09-02T10:35:15.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:14 smithi187 ceph-mon[20072]: osdmap e933: 12 total, 12 up, 11 in 2024-09-02T10:35:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3788257061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:35:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 9842/244401 objects degraded (4.027%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:35:15.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:14 smithi187 ceph-mon[20072]: osdmap e934: 12 total, 12 up, 11 in 2024-09-02T10:35:15.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:15.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:15.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:15.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 75 16 85242 85226 71.0156 50.9375 0.0156568 0.0140767 2024-09-02T10:35:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:14 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:35:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:14 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:35:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:14 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:35:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:14 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:35:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:14 smithi162 ceph-mon[20279]: pgmap v2139: 57 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 48 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 35 MiB/s wr, 565 op/s; 9842/244401 objects degraded (4.027%); 15760/244401 objects misplaced (6.448%); 6.4 MiB/s, 102 objects/s recovering 2024-09-02T10:35:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:14 smithi162 ceph-mon[20279]: osdmap e933: 12 total, 12 up, 11 in 2024-09-02T10:35:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3788257061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:35:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 9842/244401 objects degraded (4.027%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:35:15.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:14 smithi162 ceph-mon[20279]: osdmap e934: 12 total, 12 up, 11 in 2024-09-02T10:35:15.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:14 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:35:15.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:14 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:35:15.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:14 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:35:15.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:14 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:35:15.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:14 smithi151 ceph-mon[16639]: pgmap v2139: 57 pgs: 6 active+recovering+undersized+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 48 active+clean; 5.0 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 35 MiB/s wr, 565 op/s; 9842/244401 objects degraded (4.027%); 15760/244401 objects misplaced (6.448%); 6.4 MiB/s, 102 objects/s recovering 2024-09-02T10:35:15.114 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:14 smithi151 ceph-mon[16639]: osdmap e933: 12 total, 12 up, 11 in 2024-09-02T10:35:15.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3788257061' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:35:15.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 9842/244401 objects degraded (4.027%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:35:15.115 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:14 smithi151 ceph-mon[16639]: osdmap e934: 12 total, 12 up, 11 in 2024-09-02T10:35:15.115 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:35:15.115 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:35:15.115 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:35:15.116 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:35:15.116 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:35:15.116 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:35:15.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:35:15.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:35:15.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:35:15.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:35:15.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:35:15.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:35:15.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:35:15.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:35:15.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:35:15.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:35:15.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:35:15.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:35:15.117 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:35:15.118 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:35:15.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:35:15.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:35:15.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:35:15.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:35:15.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:15.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:15.155 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:15.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:15.256 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:15.291 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:15.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:15.393 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:15.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:15.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:15.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:15.559 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:15.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:15.659 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 10, 4, 2] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:35:15.659 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:35:15.660 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-02T10:35:15.660 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:35:15.661 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:15.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:15.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:15.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:15.799 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:15.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:15.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:15.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:15.954 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:15.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:15.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:16.022 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 76 16 86042 86026 70.739 50 0.00320472 0.0140903 2024-09-02T10:35:16.055 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:16.077 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:16.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:16.078 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:16.179 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:16.200 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:16.201 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:16.302 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:16.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:16.325 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:16.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:16.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:16.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:16.457 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:16.558 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:16.588 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:16.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:16 smithi151 ceph-mon[16639]: 9.11 scrub starts 2024-09-02T10:35:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:16 smithi151 ceph-mon[16639]: osdmap e935: 12 total, 12 up, 11 in 2024-09-02T10:35:16.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:16 smithi151 ceph-mon[16639]: pgmap v2143: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 48 active+clean; 5.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 31 MiB/s wr, 496 op/s; 4026/250632 objects degraded (1.606%); 18972/250632 objects misplaced (7.570%); 12 MiB/s, 195 objects/s recovering 2024-09-02T10:35:16.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:16.718 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:16.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:16.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:16 smithi187 ceph-mon[20072]: 9.11 scrub starts 2024-09-02T10:35:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:16 smithi187 ceph-mon[20072]: osdmap e935: 12 total, 12 up, 11 in 2024-09-02T10:35:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:16 smithi187 ceph-mon[20072]: pgmap v2143: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 48 active+clean; 5.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 31 MiB/s wr, 496 op/s; 4026/250632 objects degraded (1.606%); 18972/250632 objects misplaced (7.570%); 12 MiB/s, 195 objects/s recovering 2024-09-02T10:35:16.819 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:16.844 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:16 smithi162 ceph-mon[20279]: 9.11 scrub starts 2024-09-02T10:35:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:16 smithi162 ceph-mon[20279]: osdmap e935: 12 total, 12 up, 11 in 2024-09-02T10:35:16.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:16 smithi162 ceph-mon[20279]: pgmap v2143: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 8 active+recovering+undersized+remapped, 48 active+clean; 5.1 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 31 MiB/s wr, 496 op/s; 4026/250632 objects degraded (1.606%); 18972/250632 objects misplaced (7.570%); 12 MiB/s, 195 objects/s recovering 2024-09-02T10:35:16.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:16.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:16.882 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:35:16.902 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:35:16.947 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:16.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:16.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:16.973 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:17.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 77 16 86926 86910 70.5378 55.25 0.0095516 0.0141719 2024-09-02T10:35:17.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:17.107 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:17.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:17.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:17.209 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:17.233 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:17.234 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:17.335 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:17.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:17.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:17.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:17.470 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:17.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:17.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:17.595 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:17.621 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:17.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:17.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:17 smithi151 ceph-mon[16639]: 9.11 scrub ok 2024-09-02T10:35:17.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:17 smithi151 ceph-mon[16639]: osdmap e936: 12 total, 12 up, 11 in 2024-09-02T10:35:17.633 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:17 smithi151 ceph-mon[16639]: 9.19 scrub starts 2024-09-02T10:35:17.723 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:17.748 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:17.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:17 smithi187 ceph-mon[20072]: 9.11 scrub ok 2024-09-02T10:35:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:17 smithi187 ceph-mon[20072]: osdmap e936: 12 total, 12 up, 11 in 2024-09-02T10:35:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:17 smithi187 ceph-mon[20072]: 9.19 scrub starts 2024-09-02T10:35:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:17 smithi162 ceph-mon[20279]: 9.11 scrub ok 2024-09-02T10:35:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:17 smithi162 ceph-mon[20279]: osdmap e936: 12 total, 12 up, 11 in 2024-09-02T10:35:17.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:17 smithi162 ceph-mon[20279]: 9.19 scrub starts 2024-09-02T10:35:17.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:17.882 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:17.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:17.984 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:18.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:18.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:18.013 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:18.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 78 16 87909 87893 70.4211 61.4375 0.0082623 0.0141945 2024-09-02T10:35:18.113 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:18.141 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:18.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:18.243 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:18.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:18.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:18.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:18.395 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:18.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:18.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:18.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:18.527 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:18.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:18.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:18.655 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:18.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:18.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:18.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:18 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:35:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:18 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:35:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:18 smithi151 ceph-mon[16639]: osdmap e937: 12 total, 12 up, 11 in 2024-09-02T10:35:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:18 smithi151 ceph-mon[16639]: pgmap v2146: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 49 active+clean; 5.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 14 MiB/s wr, 227 op/s; 4026/254652 objects degraded (1.581%); 18021/254652 objects misplaced (7.077%); 4.9 MiB/s, 77 objects/s recovering 2024-09-02T10:35:18.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:18 smithi151 ceph-mon[16639]: 9.19 scrub ok 2024-09-02T10:35:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:18 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:35:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:18 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:35:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:18 smithi187 ceph-mon[20072]: osdmap e937: 12 total, 12 up, 11 in 2024-09-02T10:35:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:18 smithi187 ceph-mon[20072]: pgmap v2146: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 49 active+clean; 5.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 14 MiB/s wr, 227 op/s; 4026/254652 objects degraded (1.581%); 18021/254652 objects misplaced (7.077%); 4.9 MiB/s, 77 objects/s recovering 2024-09-02T10:35:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:18 smithi187 ceph-mon[20072]: 9.19 scrub ok 2024-09-02T10:35:18.756 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:18.794 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:18.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:18 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:35:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:18 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:35:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:18 smithi162 ceph-mon[20279]: osdmap e937: 12 total, 12 up, 11 in 2024-09-02T10:35:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:18 smithi162 ceph-mon[20279]: pgmap v2146: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 7 active+recovering+undersized+remapped, 49 active+clean; 5.2 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 14 MiB/s wr, 227 op/s; 4026/254652 objects degraded (1.581%); 18021/254652 objects misplaced (7.077%); 4.9 MiB/s, 77 objects/s recovering 2024-09-02T10:35:18.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:18 smithi162 ceph-mon[20279]: 9.19 scrub ok 2024-09-02T10:35:18.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:18.920 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:18.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:19.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:19.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 79 16 89008 88992 70.3991 68.6875 0.00677648 0.0141999 2024-09-02T10:35:19.053 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:19.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:19.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:19.188 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:19.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:19.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:19.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:19.329 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:19.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:19.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:19.455 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:19.456 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:19.557 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:19.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:19.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:19.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:19.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 4026/254652 objects degraded (1.581%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:35:19.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:19 smithi151 ceph-mon[16639]: osdmap e938: 12 total, 12 up, 11 in 2024-09-02T10:35:19.693 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:19.732 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:19.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 4026/254652 objects degraded (1.581%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:35:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:19 smithi187 ceph-mon[20072]: osdmap e938: 12 total, 12 up, 11 in 2024-09-02T10:35:19.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:19.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 4026/254652 objects degraded (1.581%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:35:19.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:19 smithi162 ceph-mon[20279]: osdmap e938: 12 total, 12 up, 11 in 2024-09-02T10:35:19.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:19.860 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:19.860 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:19.961 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:19.986 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:19.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:19.987 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:20.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:2024-09-02T10:35:20.022853+0000 min lat: 0.00099454 max lat: 2.15641 avg lat: 0.0142009 2024-09-02T10:35:20.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-02T10:35:20.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 80 16 90128 90112 70.394 70 0.0200534 0.0142009 2024-09-02T10:35:20.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:20.128 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:20.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:20.230 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:20.252 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:20.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:20.354 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:20.378 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:20.379 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:20.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:20.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:20.502 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:20.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:20.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:20.639 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:20.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:20 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:35:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:20 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:35:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:20 smithi151 ceph-mon[16639]: pgmap v2148: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 50 active+clean; 5.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 4026/261270 objects degraded (1.541%); 16953/261270 objects misplaced (6.489%) 2024-09-02T10:35:20.740 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:20 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:35:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:20 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:35:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:20 smithi187 ceph-mon[20072]: pgmap v2148: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 50 active+clean; 5.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 4026/261270 objects degraded (1.541%); 16953/261270 objects misplaced (6.489%) 2024-09-02T10:35:20.762 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:20.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:20.763 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:20 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:35:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:20 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:35:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:20 smithi162 ceph-mon[20279]: pgmap v2148: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 50 active+clean; 5.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 4026/261270 objects degraded (1.541%); 16953/261270 objects misplaced (6.489%) 2024-09-02T10:35:20.864 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:20.888 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:20.889 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:20.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:21.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:21.012 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:21.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 81 16 91049 91033 70.2355 57.5625 0.028263 0.0142284 2024-09-02T10:35:21.113 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:21.143 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:21.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:21.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:21.267 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:21.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:21.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:21.396 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:21.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:21.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:21.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:21.531 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:21.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:21.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:21.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:21.667 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:21.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:21.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:21.769 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:21.795 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:21.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:21.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:21.921 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:21.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:22.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 82 16 91799 91783 69.9506 46.875 0.00508335 0.0142791 2024-09-02T10:35:22.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:22.049 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:22.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:22.151 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:22.178 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:22.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:22.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:22.301 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:22.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:22.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:22.403 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:22.424 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:22.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:22.526 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:22.555 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:22.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:22.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:22.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:22.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:22.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:22.816 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:22.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:22.918 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:22.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:22 smithi151 ceph-mon[16639]: pgmap v2149: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 50 active+clean; 5.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 4026/264417 objects degraded (1.523%); 14940/264417 objects misplaced (5.650%) 2024-09-02T10:35:22.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:22 smithi151 ceph-mon[16639]: osdmap e939: 12 total, 12 up, 11 in 2024-09-02T10:35:22.951 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:22.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:22 smithi187 ceph-mon[20072]: pgmap v2149: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 50 active+clean; 5.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 4026/264417 objects degraded (1.523%); 14940/264417 objects misplaced (5.650%) 2024-09-02T10:35:23.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:22 smithi187 ceph-mon[20072]: osdmap e939: 12 total, 12 up, 11 in 2024-09-02T10:35:23.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 83 16 92324 92308 69.5031 32.8125 0.0149795 0.0143827 2024-09-02T10:35:23.053 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:23.083 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:23.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:23.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:22 smithi162 ceph-mon[20279]: pgmap v2149: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped, 50 active+clean; 5.4 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 4026/264417 objects degraded (1.523%); 14940/264417 objects misplaced (5.650%) 2024-09-02T10:35:23.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:22 smithi162 ceph-mon[20279]: osdmap e939: 12 total, 12 up, 11 in 2024-09-02T10:35:23.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:23.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:23.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:23.212 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:23.313 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:23.340 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:23.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:23.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:23.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:23.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:23.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:23.581 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:23.610 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:23.611 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:23.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:23.738 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:23.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:23.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:23.791 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:23 smithi151 ceph-mon[16639]: osdmap e940: 12 total, 12 up, 11 in 2024-09-02T10:35:23.840 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:23.864 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:23.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:23.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:23.966 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:23.989 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:23.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:24.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:23 smithi187 ceph-mon[20072]: osdmap e940: 12 total, 12 up, 11 in 2024-09-02T10:35:24.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 84 16 93222 93206 69.3438 56.125 0.00839048 0.0143967 2024-09-02T10:35:24.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:24.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:23 smithi162 ceph-mon[20279]: osdmap e940: 12 total, 12 up, 11 in 2024-09-02T10:35:24.113 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:24.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:24.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:24.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:24.237 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:24.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:24.239 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:24.340 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:24.362 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:24.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:24.464 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:24.499 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:24.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:24.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:24.574 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:35:24.602 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:24.627 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:24.628 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:24.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:24 smithi187 ceph-mon[20072]: pgmap v2152: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 51 active+clean; 5.5 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 32 MiB/s wr, 519 op/s; 4026/272490 objects degraded (1.477%); 13914/272490 objects misplaced (5.106%) 2024-09-02T10:35:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:24 smithi187 ceph-mon[20072]: 9.12 scrub starts 2024-09-02T10:35:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:24 smithi187 ceph-mon[20072]: osdmap e941: 12 total, 12 up, 11 in 2024-09-02T10:35:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:24 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 4026/272490 objects degraded (1.477%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:35:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:24 smithi187 ceph-mon[20072]: osdmap e942: 12 total, 12 up, 11 in 2024-09-02T10:35:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:24.789 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:24.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:24.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:24.920 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:24.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:24.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:24 smithi151 ceph-mon[16639]: pgmap v2152: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 51 active+clean; 5.5 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 32 MiB/s wr, 519 op/s; 4026/272490 objects degraded (1.477%); 13914/272490 objects misplaced (5.106%) 2024-09-02T10:35:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:24 smithi151 ceph-mon[16639]: 9.12 scrub starts 2024-09-02T10:35:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:24 smithi151 ceph-mon[16639]: osdmap e941: 12 total, 12 up, 11 in 2024-09-02T10:35:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:24 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 4026/272490 objects degraded (1.477%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:35:24.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:24 smithi151 ceph-mon[16639]: osdmap e942: 12 total, 12 up, 11 in 2024-09-02T10:35:25.023 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:25.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 85 16 94008 93992 69.1059 49.125 0.0026895 0.0144658 2024-09-02T10:35:25.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:25.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:25.050 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:24 smithi162 ceph-mon[20279]: pgmap v2152: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 51 active+clean; 5.5 GiB data, 25 GiB used, 958 GiB / 983 GiB avail; 32 MiB/s wr, 519 op/s; 4026/272490 objects degraded (1.477%); 13914/272490 objects misplaced (5.106%) 2024-09-02T10:35:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:24 smithi162 ceph-mon[20279]: 9.12 scrub starts 2024-09-02T10:35:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:24 smithi162 ceph-mon[20279]: osdmap e941: 12 total, 12 up, 11 in 2024-09-02T10:35:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:24 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 4026/272490 objects degraded (1.477%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:35:25.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:24 smithi162 ceph-mon[20279]: osdmap e942: 12 total, 12 up, 11 in 2024-09-02T10:35:25.151 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:25.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:25.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:25.177 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:25.277 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:25.305 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:25.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:25.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:25.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:25.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:25.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:25.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:25.544 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:25.580 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:25.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:25.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:25.707 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:25.708 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:25.809 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:25.843 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:25.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:25.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:25.968 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:25.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:25.969 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:25 smithi187 ceph-mon[20072]: 9.12 scrub ok 2024-09-02T10:35:26.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:25 smithi187 ceph-mon[20072]: 9.15 scrub starts 2024-09-02T10:35:26.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:25 smithi151 ceph-mon[16639]: 9.12 scrub ok 2024-09-02T10:35:26.021 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:25 smithi151 ceph-mon[16639]: 9.15 scrub starts 2024-09-02T10:35:26.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 86 16 94972 94956 69.0029 60.25 0.00404255 0.0144856 2024-09-02T10:35:26.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:26.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:26.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:26.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:25 smithi162 ceph-mon[20279]: 9.12 scrub ok 2024-09-02T10:35:26.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:25 smithi162 ceph-mon[20279]: 9.15 scrub starts 2024-09-02T10:35:26.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:26.225 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:26.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:26.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:26.353 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:26.355 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:26.456 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:26.486 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:26.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:26.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:26.587 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:26.610 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:26.611 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:26.712 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:26.739 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:26.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:26.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:26.840 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:26.864 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:26.865 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:26.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:26.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:26 smithi162 ceph-mon[20279]: pgmap v2155: 57 pgs: 6 active+recovering+undersized+remapped, 51 active+clean; 5.6 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 15361/277254 objects misplaced (5.540%); 24 MiB/s, 388 objects/s recovering 2024-09-02T10:35:26.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:26 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4026/272490 objects degraded (1.477%), 1 pg degraded) 2024-09-02T10:35:26.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:26 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:35:26.996 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:26 smithi162 ceph-mon[20279]: 9.15 scrub ok 2024-09-02T10:35:26.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:26.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:27.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:26 smithi187 ceph-mon[20072]: pgmap v2155: 57 pgs: 6 active+recovering+undersized+remapped, 51 active+clean; 5.6 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 15361/277254 objects misplaced (5.540%); 24 MiB/s, 388 objects/s recovering 2024-09-02T10:35:27.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:26 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4026/272490 objects degraded (1.477%), 1 pg degraded) 2024-09-02T10:35:27.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:26 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:35:27.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:26 smithi187 ceph-mon[20072]: 9.15 scrub ok 2024-09-02T10:35:27.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 87 16 96036 96020 68.974 66.5 0.00632401 0.0144926 2024-09-02T10:35:27.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:27.125 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:27.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:27.126 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:27.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:26 smithi151 ceph-mon[16639]: pgmap v2155: 57 pgs: 6 active+recovering+undersized+remapped, 51 active+clean; 5.6 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 67 MiB/s wr, 1.07k op/s; 15361/277254 objects misplaced (5.540%); 24 MiB/s, 388 objects/s recovering 2024-09-02T10:35:27.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:26 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4026/272490 objects degraded (1.477%), 1 pg degraded) 2024-09-02T10:35:27.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:26 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:35:27.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:26 smithi151 ceph-mon[16639]: 9.15 scrub ok 2024-09-02T10:35:27.226 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:27.254 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:27.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:27.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:27.379 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:27.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:27.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:27.504 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:27.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:27.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:27.606 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:27.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:27.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:27.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:27.758 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:27.759 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:27 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:35:27.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:27 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:35:27.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:27.886 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:27.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:27.988 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:27 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:35:28.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:27 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:35:28.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:28.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:28.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 88 16 96681 96665 68.6483 40.3125 0.0712185 0.0145418 2024-09-02T10:35:28.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:28.133 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:28.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:28.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:27 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:35:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:27 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:35:28.234 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:28.257 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:28.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:28.359 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:28.380 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:28.381 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:28.482 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:28.505 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:28.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:28.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:28.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:28.636 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:28.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:28.738 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:28.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:28.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:28.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:28.902 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:28.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:28.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: pgmap v2156: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 5.7 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 18 MiB/s wr, 292 op/s; 15231/281091 objects misplaced (5.419%) 2024-09-02T10:35:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: osdmap e943: 12 total, 12 up, 11 in 2024-09-02T10:35:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=3636847619,config=2353167246,health=861809967,kv=2420544001}) 2024-09-02T10:35:28.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3710491460,logm=1031758641}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2612787033}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3943549059}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=44594616}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1538071941}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523255881}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3549035449}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259413083}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665159833}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491247785}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3891170195}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847873942}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200425152}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496888923}) 2024-09-02T10:35:28.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104273}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944778524}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605066914}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419934106}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=566523105}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3623933594}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346936626}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=198087378}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2248362138}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2653695458}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520950104}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254367257}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3449629162}) 2024-09-02T10:35:28.940 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1982726813}) 2024-09-02T10:35:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673213398}) 2024-09-02T10:35:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4097647588}) 2024-09-02T10:35:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4183953198}) 2024-09-02T10:35:28.941 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383791129}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4164848907}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878929884}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3964847026}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374452880}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38750898}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782824332}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807011271}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=121055713}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28996956}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307011806}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356291506}) 2024-09-02T10:35:28.942 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4171280873}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2011838619}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039158286}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012341232}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=429329950}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986816243}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2910651837}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3561983478}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196167028}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110008513}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912963387}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238412201}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426725054}) 2024-09-02T10:35:28.943 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2761914502}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620775143}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180482029}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=675484366}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2751051210}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128850075}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3613820095}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231878486}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197266906}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=644729684}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1388734094}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2952075181}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=896047351}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2953782940}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893593668}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867029598}) 2024-09-02T10:35:28.944 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189747897}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2750980273}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=124643453}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1252048329}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=220866110}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1484342290}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663530092}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885781708}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1720288768}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287922232}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61932315}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=4112124997,mdsmap=1083786272,mgr=3227228554}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mgr=15,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=69} crc {mgr=1225262768,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=3765095827,mon_config_key=213767840}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3948140636}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3275622721}) 2024-09-02T10:35:28.945 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=22,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=60} crc {mon_config_key=1901667993,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=1480046200,osdmap=3112810628}) 2024-09-02T10:35:28.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4127556856}) 2024-09-02T10:35:28.946 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:28 smithi151 ceph-mon[16639]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=61} crc {osdmap=1577116086}) 2024-09-02T10:35:29.004 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: pgmap v2156: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 5.7 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 18 MiB/s wr, 292 op/s; 15231/281091 objects misplaced (5.419%) 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: osdmap e943: 12 total, 12 up, 11 in 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=3636847619,config=2353167246,health=861809967,kv=2420544001}) 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3710491460,logm=1031758641}) 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2612787033}) 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3943549059}) 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=44594616}) 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1538071941}) 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523255881}) 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3549035449}) 2024-09-02T10:35:29.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259413083}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665159833}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491247785}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3891170195}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847873942}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200425152}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496888923}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104273}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944778524}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605066914}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419934106}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=566523105}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3623933594}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346936626}) 2024-09-02T10:35:29.008 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=198087378}) 2024-09-02T10:35:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2248362138}) 2024-09-02T10:35:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2653695458}) 2024-09-02T10:35:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520950104}) 2024-09-02T10:35:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254367257}) 2024-09-02T10:35:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3449629162}) 2024-09-02T10:35:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1982726813}) 2024-09-02T10:35:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673213398}) 2024-09-02T10:35:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4097647588}) 2024-09-02T10:35:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4183953198}) 2024-09-02T10:35:29.009 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383791129}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4164848907}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878929884}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3964847026}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374452880}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38750898}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782824332}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807011271}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=121055713}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28996956}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307011806}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356291506}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4171280873}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2011838619}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039158286}) 2024-09-02T10:35:29.010 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012341232}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=429329950}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986816243}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2910651837}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3561983478}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196167028}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110008513}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912963387}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238412201}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426725054}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2761914502}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620775143}) 2024-09-02T10:35:29.011 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180482029}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=675484366}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2751051210}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128850075}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3613820095}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231878486}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197266906}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=644729684}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1388734094}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2952075181}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=896047351}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2953782940}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893593668}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867029598}) 2024-09-02T10:35:29.012 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189747897}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2750980273}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=124643453}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1252048329}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=220866110}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1484342290}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663530092}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885781708}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1720288768}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287922232}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61932315}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=4112124997,mdsmap=1083786272,mgr=3227228554}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mgr=15,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=69} crc {mgr=1225262768,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=3765095827,mon_config_key=213767840}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3948140636}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3275622721}) 2024-09-02T10:35:29.013 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=22,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=60} crc {mon_config_key=1901667993,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=1480046200,osdmap=3112810628}) 2024-09-02T10:35:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4127556856}) 2024-09-02T10:35:29.014 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:28 smithi187 ceph-mon[20072]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=61} crc {osdmap=1577116086}) 2024-09-02T10:35:29.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 89 16 97066 97050 68.1473 24.0625 0.0148886 0.0146691 2024-09-02T10:35:29.028 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:29.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:29.029 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:29.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: pgmap v2156: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 5.7 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 18 MiB/s wr, 292 op/s; 15231/281091 objects misplaced (5.419%) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: osdmap e943: 12 total, 12 up, 11 in 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=3636847619,config=2353167246,health=861809967,kv=2420544001}) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {kv=2,logm=98} crc {kv=3710491460,logm=1031758641}) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2612787033}) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3943549059}) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=44594616}) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1538071941}) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1523255881}) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3549035449}) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4259413083}) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=665159833}) 2024-09-02T10:35:29.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3491247785}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3891170195}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1847873942}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4200425152}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1496888923}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3104273}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3944778524}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=605066914}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3419934106}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=566523105}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3623933594}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2346936626}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=198087378}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2248362138}) 2024-09-02T10:35:29.097 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2653695458}) 2024-09-02T10:35:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3520950104}) 2024-09-02T10:35:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1254367257}) 2024-09-02T10:35:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3449629162}) 2024-09-02T10:35:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1982726813}) 2024-09-02T10:35:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673213398}) 2024-09-02T10:35:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4097647588}) 2024-09-02T10:35:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4183953198}) 2024-09-02T10:35:29.098 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2383791129}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4164848907}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1878929884}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3964847026}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=374452880}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=38750898}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782824332}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1807011271}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=121055713}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=28996956}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3307011806}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1356291506}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4171280873}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2011838619}) 2024-09-02T10:35:29.099 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039158286}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2012341232}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=429329950}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=986816243}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2910651837}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3561983478}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1196167028}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110008513}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3912963387}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2238412201}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=426725054}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2761914502}) 2024-09-02T10:35:29.100 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1620775143}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180482029}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=675484366}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2751051210}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4128850075}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3613820095}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=231878486}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3197266906}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=644729684}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1388734094}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2952075181}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=896047351}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2953782940}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=893593668}) 2024-09-02T10:35:29.101 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=867029598}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4189747897}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2750980273}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=124643453}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1252048329}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=220866110}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1484342290}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2663530092}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2885781708}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1720288768}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3287922232}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=61932315}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=4112124997,mdsmap=1083786272,mgr=3227228554}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mgr=15,mgr_command_descs=1,mgr_metadata=2,mgrstat=13,mon_config_key=69} crc {mgr=1225262768,mgr_command_descs=899306875,mgr_metadata=2096458327,mgrstat=3765095827,mon_config_key=213767840}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3948140636}) 2024-09-02T10:35:29.102 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3275622721}) 2024-09-02T10:35:29.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=22,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=60} crc {mon_config_key=1901667993,monmap=3739207698,osd_metadata=1182194493,osd_pg_creating=1480046200,osdmap=3112810628}) 2024-09-02T10:35:29.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4127556856}) 2024-09-02T10:35:29.103 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:28 smithi162 ceph-mon[20279]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=61} crc {osdmap=1577116086}) 2024-09-02T10:35:29.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:29.153 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:29.154 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:29.255 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:29.286 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:29.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:29.388 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:29.415 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:29.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:29.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:29.545 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:29.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:29.647 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:29.690 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:29.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:29.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:29.736 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:35:29.792 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:29.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:29.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:29.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:29.930 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:29.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:29 smithi151 ceph-mon[16639]: osdmap e944: 12 total, 12 up, 11 in 2024-09-02T10:35:29.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4170436613' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:35:29.962 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:29.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:29 smithi187 ceph-mon[20072]: osdmap e944: 12 total, 12 up, 11 in 2024-09-02T10:35:30.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4170436613' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:35:30.023 INFO:tasks.radosbench.radosbench.2.smithi187.stdout: 90 11 98109 98098 68.1179 65.5 0.0344488 0.0146743 2024-09-02T10:35:30.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:30.093 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:30.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:30.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:30.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:29 smithi162 ceph-mon[20279]: osdmap e944: 12 total, 12 up, 11 in 2024-09-02T10:35:30.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4170436613' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:35:30.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:30.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:30.218 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:30.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:30.319 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:30.341 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:30.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:30.443 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:30.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:30.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:30.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:30.567 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:30.589 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:30.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:30.691 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:30.712 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:30.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:30.714 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:30.815 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:30.836 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:30.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:30.938 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:30.959 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:30.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:30.960 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:30.999 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:30 smithi151 ceph-mon[16639]: 9.14 scrub starts 2024-09-02T10:35:31.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:30 smithi151 ceph-mon[16639]: pgmap v2159: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 5.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 14269/287766 objects misplaced (4.959%) 2024-09-02T10:35:31.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:30 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:35:31.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4170436613' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:35:31.000 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:30 smithi151 ceph-mon[16639]: osdmap e945: 12 total, 12 up, 11 in 2024-09-02T10:35:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:30 smithi187 ceph-mon[20072]: 9.14 scrub starts 2024-09-02T10:35:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:30 smithi187 ceph-mon[20072]: pgmap v2159: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 5.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 14269/287766 objects misplaced (4.959%) 2024-09-02T10:35:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:30 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:35:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4170436613' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:35:31.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:30 smithi187 ceph-mon[20072]: osdmap e945: 12 total, 12 up, 11 in 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total time run: 90.8076 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Total writes made: 98109 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Write size: 65536 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Object size: 65536 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Bandwidth (MB/sec): 67.5253 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Bandwidth: 27.5326 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max bandwidth (MB/sec): 135 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min bandwidth (MB/sec): 0.625 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average IOPS: 1080 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev IOPS: 440.522 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max IOPS: 2160 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min IOPS: 10 2024-09-02T10:35:31.024 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Average Latency(s): 0.0147113 2024-09-02T10:35:31.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Stddev Latency(s): 0.0433552 2024-09-02T10:35:31.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Max latency(s): 2.15641 2024-09-02T10:35:31.025 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Min latency(s): 0.00099454 2024-09-02T10:35:31.029 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Cleaning up (deleting benchmark objects) 2024-09-02T10:35:31.061 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:31.086 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:31.087 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:30 smithi162 ceph-mon[20279]: 9.14 scrub starts 2024-09-02T10:35:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:30 smithi162 ceph-mon[20279]: pgmap v2159: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 5.9 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 14269/287766 objects misplaced (4.959%) 2024-09-02T10:35:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:30 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:35:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4170436613' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:35:31.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:30 smithi162 ceph-mon[20279]: osdmap e945: 12 total, 12 up, 11 in 2024-09-02T10:35:31.188 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:31.222 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:31.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:31.323 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:31.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:31.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:31.446 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:31.476 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:31.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:31.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:31.609 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:31.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:31.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:31.711 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:31.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:31.734 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:31.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:31.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:31.859 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:31.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:31.986 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:31.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:32.088 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:32.117 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:32.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:32.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:32.219 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:32.245 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:32.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:32.275 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:35:32.275 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:35:32.347 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:32.376 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:32.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:32.478 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:32.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:32.502 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:32.603 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:32.630 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:32.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:32.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:32.757 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:32.758 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:32.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:32.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:32.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:32.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:32 smithi187 ceph-mon[20072]: pgmap v2161: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 5.9 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 13091/289767 objects misplaced (4.518%) 2024-09-02T10:35:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:32 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:35:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:32 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:35:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:32 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:35:33.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:32 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:35:33.022 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:33.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:32 smithi162 ceph-mon[20279]: pgmap v2161: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 5.9 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 13091/289767 objects misplaced (4.518%) 2024-09-02T10:35:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:32 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:35:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:32 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:35:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:32 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:35:33.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:32 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:35:33.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:33.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:33.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:33.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:32 smithi151 ceph-mon[16639]: pgmap v2161: 57 pgs: 5 active+recovering+undersized+remapped, 52 active+clean; 5.9 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 13091/289767 objects misplaced (4.518%) 2024-09-02T10:35:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:32 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:35:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:32 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:35:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:32 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:35:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:32 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:35:33.249 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:33.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:33.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:33.276 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:33.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:33.402 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:33.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:33.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:33.504 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:33.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:33.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:33.631 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:33.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:33.654 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:33.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:33.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:33.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:33.887 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:33.914 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:33.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:34.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:33 smithi187 ceph-mon[20072]: osdmap e946: 12 total, 12 up, 11 in 2024-09-02T10:35:34.016 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:34.043 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:34.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:34.088 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:33 smithi151 ceph-mon[16639]: osdmap e946: 12 total, 12 up, 11 in 2024-09-02T10:35:34.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:33 smithi162 ceph-mon[20279]: osdmap e946: 12 total, 12 up, 11 in 2024-09-02T10:35:34.144 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:34.180 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:34.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:34.282 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:34.303 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:34.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:34.405 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:34.436 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:34.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:34.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:34.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:34.567 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:34.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:34.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:34.669 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:34.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:34.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:34.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:34.829 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:34.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:34.931 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:34.956 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:34.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:34.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:35.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:34 smithi187 ceph-mon[20072]: pgmap v2163: 57 pgs: 4 active+recovering+undersized+remapped, 53 active+clean; 5.9 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 173 B/s rd, 14 MiB/s wr, 357 op/s; 12811/291228 objects misplaced (4.399%) 2024-09-02T10:35:35.059 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:35.086 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:35.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:35.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:35.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:34 smithi162 ceph-mon[20279]: pgmap v2163: 57 pgs: 4 active+recovering+undersized+remapped, 53 active+clean; 5.9 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 173 B/s rd, 14 MiB/s wr, 357 op/s; 12811/291228 objects misplaced (4.399%) 2024-09-02T10:35:35.187 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:34 smithi151 ceph-mon[16639]: pgmap v2163: 57 pgs: 4 active+recovering+undersized+remapped, 53 active+clean; 5.9 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 173 B/s rd, 14 MiB/s wr, 357 op/s; 12811/291228 objects misplaced (4.399%) 2024-09-02T10:35:35.216 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:35.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:35.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:35.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:35.339 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:35.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:35.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:35.469 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:35.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:35.571 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:35.597 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:35.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:35.699 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:35.723 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:35.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:35.723 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:35.824 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:35.852 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:35.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:35.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:35.954 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:35.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:35.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:35.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:36.084 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:36.105 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:36.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:36.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:36.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:36.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:36.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:36.323 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:35:36.341 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:36.367 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:36.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:36.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:36.496 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:36.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:36.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:36.598 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:36.626 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:36.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:36.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:36.759 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:36.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:36.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:36.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:36.887 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:36.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:36.989 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:36 smithi187 ceph-mon[20072]: 9.14 scrub starts 2024-09-02T10:35:37.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:36 smithi187 ceph-mon[20072]: pgmap v2164: 57 pgs: 4 active+recovering+undersized+remapped, 53 active+clean; 5.8 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 148 B/s rd, 28 MiB/s wr, 775 op/s; 10317/287421 objects misplaced (3.590%); 17 MiB/s, 268 objects/s recovering 2024-09-02T10:35:37.011 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:37.012 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:36 smithi162 ceph-mon[20279]: 9.14 scrub starts 2024-09-02T10:35:37.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:36 smithi162 ceph-mon[20279]: pgmap v2164: 57 pgs: 4 active+recovering+undersized+remapped, 53 active+clean; 5.8 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 148 B/s rd, 28 MiB/s wr, 775 op/s; 10317/287421 objects misplaced (3.590%); 17 MiB/s, 268 objects/s recovering 2024-09-02T10:35:37.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:37.141 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:37.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:37.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:36 smithi151 ceph-mon[16639]: 9.14 scrub starts 2024-09-02T10:35:37.163 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:36 smithi151 ceph-mon[16639]: pgmap v2164: 57 pgs: 4 active+recovering+undersized+remapped, 53 active+clean; 5.8 GiB data, 27 GiB used, 957 GiB / 983 GiB avail; 148 B/s rd, 28 MiB/s wr, 775 op/s; 10317/287421 objects misplaced (3.590%); 17 MiB/s, 268 objects/s recovering 2024-09-02T10:35:37.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:37.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:37.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:37.373 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:37.398 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:37.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:37.399 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:37.500 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:37.528 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:37.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:37.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:37.630 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:37.653 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:37.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:37.654 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:37.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:37.803 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:37.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:37.905 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:37.927 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:37.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:37.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:38.028 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:38.050 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:38.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:38.152 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:38.173 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:38.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:38.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:38.275 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:38.297 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:38.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:38.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:38.420 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:38.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:38.522 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:38.544 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:38.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:38.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:38.669 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:38.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:38.771 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:38.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:38.793 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:38.793 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:38.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:38.922 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:38.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:38.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:38 smithi187 ceph-mon[20072]: pgmap v2165: 57 pgs: 4 active+recovering+undersized+remapped, 53 active+clean; 5.7 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 127 B/s rd, 17 MiB/s wr, 762 op/s; 10317/282390 objects misplaced (3.653%); 11 MiB/s, 181 objects/s recovering 2024-09-02T10:35:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:38 smithi187 ceph-mon[20072]: osdmap e947: 12 total, 12 up, 11 in 2024-09-02T10:35:39.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:39.053 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:39.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:39.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:38 smithi162 ceph-mon[20279]: pgmap v2165: 57 pgs: 4 active+recovering+undersized+remapped, 53 active+clean; 5.7 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 127 B/s rd, 17 MiB/s wr, 762 op/s; 10317/282390 objects misplaced (3.653%); 11 MiB/s, 181 objects/s recovering 2024-09-02T10:35:39.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:38 smithi162 ceph-mon[20279]: osdmap e947: 12 total, 12 up, 11 in 2024-09-02T10:35:39.155 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:39.185 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:39.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:39.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:38 smithi151 ceph-mon[16639]: pgmap v2165: 57 pgs: 4 active+recovering+undersized+remapped, 53 active+clean; 5.7 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 127 B/s rd, 17 MiB/s wr, 762 op/s; 10317/282390 objects misplaced (3.653%); 11 MiB/s, 181 objects/s recovering 2024-09-02T10:35:39.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:38 smithi151 ceph-mon[16639]: osdmap e947: 12 total, 12 up, 11 in 2024-09-02T10:35:39.287 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:39.315 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:39.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:39.418 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:39.441 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:39.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:39.543 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:39.569 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:39.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:39.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:39.696 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:39.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:39.697 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:39.798 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:39.826 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:39.827 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:39.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:39.953 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:39.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:40.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:39 smithi187 ceph-mon[20072]: osdmap e948: 12 total, 12 up, 11 in 2024-09-02T10:35:40.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:40.076 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:40.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:40.078 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:40.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:39 smithi162 ceph-mon[20279]: osdmap e948: 12 total, 12 up, 11 in 2024-09-02T10:35:40.178 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:40.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:39 smithi151 ceph-mon[16639]: osdmap e948: 12 total, 12 up, 11 in 2024-09-02T10:35:40.205 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:40.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:40.307 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:40.329 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:40.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:40.431 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:40.454 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:40.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:40.455 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:40.556 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:40.578 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:40.579 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:40.680 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:40.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:40.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:40.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:40.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:40.834 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:40.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:40.835 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:40.936 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:40.966 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:40.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:40 smithi187 ceph-mon[20072]: pgmap v2168: 57 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 53 active+clean; 5.6 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 5.6 MiB/s wr, 988 op/s; 8692/274845 objects misplaced (3.163%); 14 MiB/s, 228 objects/s recovering 2024-09-02T10:35:41.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:40 smithi187 ceph-mon[20072]: 9.16 scrub starts 2024-09-02T10:35:41.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:40 smithi151 ceph-mon[16639]: pgmap v2168: 57 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 53 active+clean; 5.6 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 5.6 MiB/s wr, 988 op/s; 8692/274845 objects misplaced (3.163%); 14 MiB/s, 228 objects/s recovering 2024-09-02T10:35:41.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:40 smithi151 ceph-mon[16639]: 9.16 scrub starts 2024-09-02T10:35:41.067 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:40 smithi162 ceph-mon[20279]: pgmap v2168: 57 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 53 active+clean; 5.6 GiB data, 26 GiB used, 957 GiB / 983 GiB avail; 5.6 MiB/s wr, 988 op/s; 8692/274845 objects misplaced (3.163%); 14 MiB/s, 228 objects/s recovering 2024-09-02T10:35:41.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:40 smithi162 ceph-mon[20279]: 9.16 scrub starts 2024-09-02T10:35:41.098 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:41.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:41.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:41.226 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:41.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:41.227 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:41.328 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:41.360 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:41.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:41.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:41.462 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:41.492 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:41.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:41.594 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:41.619 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:41.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:41.721 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:41.744 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:41.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:41.745 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:41.846 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:41.875 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:41.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:41.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:41.977 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:42.001 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:42.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:42.002 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:42.103 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:42.141 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:42.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:42.242 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:42.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:42.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:42.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:42.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:42.394 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:42.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:42.497 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:42.531 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:42.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:42.633 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:42.660 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:42.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:42.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:42.786 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:42.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:42.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:42.909 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:42.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:42.910 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:42 smithi187 ceph-mon[20072]: 9.14 scrub starts 2024-09-02T10:35:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:42 smithi187 ceph-mon[20072]: pgmap v2169: 57 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 53 active+clean; 5.5 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 1.4 MiB/s wr, 895 op/s; 8227/270813 objects misplaced (3.038%); 13 MiB/s, 215 objects/s recovering 2024-09-02T10:35:43.011 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:43.040 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:43.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:42 smithi162 ceph-mon[20279]: 9.14 scrub starts 2024-09-02T10:35:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:42 smithi162 ceph-mon[20279]: pgmap v2169: 57 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 53 active+clean; 5.5 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 1.4 MiB/s wr, 895 op/s; 8227/270813 objects misplaced (3.038%); 13 MiB/s, 215 objects/s recovering 2024-09-02T10:35:43.141 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:43.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:43.163 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:43.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:42 smithi151 ceph-mon[16639]: 9.14 scrub starts 2024-09-02T10:35:43.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:42 smithi151 ceph-mon[16639]: pgmap v2169: 57 pgs: 1 creating+peering, 3 active+recovering+undersized+remapped, 53 active+clean; 5.5 GiB data, 26 GiB used, 958 GiB / 983 GiB avail; 1.4 MiB/s wr, 895 op/s; 8227/270813 objects misplaced (3.038%); 13 MiB/s, 215 objects/s recovering 2024-09-02T10:35:43.263 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:43.289 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:43.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:43.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:43.391 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:43.414 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:43.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:43.441 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:35:43.516 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:43.546 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:43.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:43.648 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:43.670 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:43.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:43.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:43.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:43.797 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:43.898 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:43.919 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:43.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:44.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:44.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:44.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:44.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:44.146 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:44.170 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:44.171 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:44.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:44.295 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:44.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:44.296 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:44.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:44.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:44.421 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:44.522 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:44.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:44.549 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:44.650 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:44.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:44.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:44.782 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:44.811 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:44.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:44.913 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:44.934 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:44.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:44 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:35:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:44 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:35:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:44 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:35:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:44 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:35:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:44 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:35:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:44 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:35:45.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:44 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:35:45.036 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:45.064 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:45.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:44 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:35:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:44 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:35:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:44 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:35:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:44 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:35:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:44 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:35:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:44 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:35:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:44 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:35:45.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:44 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:35:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:44 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:35:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:44 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:35:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:44 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:35:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:44 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:35:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:44 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:35:45.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:44 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:35:45.192 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:45.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:45.294 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:45.319 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:45.320 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:45.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:45.443 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:45.444 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:45.545 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:45.579 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:45.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:45.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:45.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:45.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:45.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:45.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:45.838 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:45.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:45.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:45.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:45.967 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:45.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:46.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:45 smithi151 ceph-mon[16639]: 9.16 scrub starts 2024-09-02T10:35:46.069 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:46.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:45 smithi162 ceph-mon[20279]: 9.16 scrub starts 2024-09-02T10:35:46.095 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:46.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:46.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:46.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-02T10:35:46.108 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:35:46.196 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:46.217 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:46.218 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:46.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:45 smithi187 ceph-mon[20072]: 9.16 scrub starts 2024-09-02T10:35:46.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:46.368 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:46.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:46.469 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:46.501 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:46.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:46.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:46.603 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:46.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:46.625 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:46.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:46.751 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:46.752 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:46.853 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:46.876 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:46.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:46.878 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:46.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:47.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:47.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:47.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:46 smithi162 ceph-mon[20279]: pgmap v2171: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 5.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s; 6471/258141 objects misplaced (2.507%); 12 MiB/s, 185 objects/s recovering 2024-09-02T10:35:47.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:47.136 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:47.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:46 smithi151 ceph-mon[16639]: pgmap v2171: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 5.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s; 6471/258141 objects misplaced (2.507%); 12 MiB/s, 185 objects/s recovering 2024-09-02T10:35:47.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:46 smithi187 ceph-mon[20072]: pgmap v2171: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 5.3 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s; 6471/258141 objects misplaced (2.507%); 12 MiB/s, 185 objects/s recovering 2024-09-02T10:35:47.261 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:47.262 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:47.363 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:47.391 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:47.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:47.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:47.519 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:47.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:47.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:47.644 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:47.645 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:47.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:47.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:47.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:47.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:47.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:47.899 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:47.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:48.001 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:48.032 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:48.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:47 smithi162 ceph-mon[20279]: 9.14 scrub starts 2024-09-02T10:35:48.134 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:48.164 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:48.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:48.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:47 smithi151 ceph-mon[16639]: 9.14 scrub starts 2024-09-02T10:35:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:47 smithi187 ceph-mon[20072]: 9.14 scrub starts 2024-09-02T10:35:48.266 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:48.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:48.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:48.397 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:48.422 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:48.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:48.423 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:48.524 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:48.554 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:48.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:48.656 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:48.680 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:48.681 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:48.781 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:48.814 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:48.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:48.915 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:48.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:48.940 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:49.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:49.071 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:49.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:49.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:49.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:48 smithi162 ceph-mon[20279]: pgmap v2172: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 5.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s; 6471/252636 objects misplaced (2.561%); 9.4 MiB/s, 150 objects/s recovering 2024-09-02T10:35:49.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:49.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:48 smithi151 ceph-mon[16639]: pgmap v2172: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 5.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s; 6471/252636 objects misplaced (2.561%); 9.4 MiB/s, 150 objects/s recovering 2024-09-02T10:35:49.201 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:49.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:49.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:48 smithi187 ceph-mon[20072]: pgmap v2172: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 5.1 GiB data, 25 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.00k op/s; 6471/252636 objects misplaced (2.561%); 9.4 MiB/s, 150 objects/s recovering 2024-09-02T10:35:49.303 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:49.330 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:49.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:49.433 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:49.461 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:49.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:49.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:49.586 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:49.587 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:49.688 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:49.729 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:49.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:49.831 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:49.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:49.857 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:49.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:49.958 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:49.980 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:49.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:50.082 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:50.103 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:50.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:50.104 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:50.205 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:50.226 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:50.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:50.328 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:50.349 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:50.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:50.351 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:50.452 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:50.473 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:50.474 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:50.575 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:50.598 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:50.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:35:50.607 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:35:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 949, 2024-09-02T10:35:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:35:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:35:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:35:49.766455+0000", 2024-09-02T10:35:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:35:50.607 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:34:16.863567+0000", 2024-09-02T10:35:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:35:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5801984, 2024-09-02T10:35:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:35:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "noscrub", 2024-09-02T10:35:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:35:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:35:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:35:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:35:50.608 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 9, 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:35:50.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:35:50.610 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:35:50.611 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "264", 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "241", 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:35:50.612 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:35:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:35:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:35:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:35:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:35:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:35:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:35:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:35:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:35:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:35:50.613 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:35:50.614 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:35:50.615 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 1.7699999809265137, 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 1.7699999809265137, 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 1.7699999809265137, 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 1.7699999809265137, 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.92000001668930054 2024-09-02T10:35:50.616 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 9, 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_7", 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:33:50.981540+0000", 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:35:50.617 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 26, 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 26, 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 26, 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 26, 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 26, 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:35:50.618 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "926", 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "924", 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:35:50.619 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:35:50.620 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:35:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:35:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:35:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:35:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:35:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:35:50.621 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:35:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:35:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:35:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:35:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:35:50.622 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:35:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:35:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:35:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:35:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:35:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:35:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:35:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:35:50.623 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:35:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:35:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:35:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:35:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:35:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 5, 2024-09-02T10:35:50.624 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.119999885559082, 2024-09-02T10:35:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:35:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 5, 2024-09-02T10:35:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.119999885559082, 2024-09-02T10:35:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.91666656732559204, 2024-09-02T10:35:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 1, 2024-09-02T10:35:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.92000001668930054, 2024-09-02T10:35:50.625 INFO:teuthology.orchestra.run.smithi151.stdout: "error_message": "pool 9 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.626 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 891, 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:35:50.627 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.628 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:35:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:35:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.629 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.630 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:35:50.631 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.632 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0, 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 946, 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.633 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:35:50.634 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:35:50.635 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:35:50.636 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.637 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.638 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:35:50.639 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 891, 2024-09-02T10:35:50.639 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.639 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.639 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.639 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.639 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.640 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.641 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.642 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:35:50.643 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:35:50.644 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 948, 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.645 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.646 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.647 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.648 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.648 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.648 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.648 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:35:50.648 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:35:50.648 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.648 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.648 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.648 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:35:50.649 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:35:50.649 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.649 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.649 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.649 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.649 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.650 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.650 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:35:50.650 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:35:50.650 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.650 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:35:50.651 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.652 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.652 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.652 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.652 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.652 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.652 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:35:50.652 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:35:50.652 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.652 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 926, 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.653 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.654 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.655 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.656 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:35:50.657 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.658 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 939, 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:35:50.659 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.660 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.661 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.662 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.663 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.663 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.663 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.663 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:35:50.663 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:35:50.664 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.664 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.664 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.664 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:35:50.664 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:35:50.664 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:35:50.665 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 926, 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.666 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.667 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.668 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.669 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.670 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.671 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 941, 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:35:50.672 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.673 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.674 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.675 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.676 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.677 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 926, 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:35:50.678 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.679 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.680 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.681 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.682 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.683 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 947, 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:35:50.684 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:35:50.685 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.686 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.686 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:35:50.686 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:35:50.686 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.686 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.686 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.686 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.686 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.686 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.687 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.688 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:35:50.689 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:35:50.689 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:35:50.689 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:35:50.689 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.689 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.690 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.690 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.690 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.690 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.690 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:35:50.690 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:35:50.691 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.691 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:35:50.691 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:35:50.691 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.691 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.691 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.691 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:35:50.691 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 943, 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:35:50.692 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.694 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.695 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.696 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:35:50.697 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 792, 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.698 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.699 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:35:50.699 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:35:50.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.699 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.699 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:35:50.699 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:35:50.699 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.699 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.700 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:35:50.700 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.700 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.700 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:35:50.700 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:35:50.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:35:50.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:35:50.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:35:50.703 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.704 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:35:16.979832+0000", 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:34:57.431566+0000", 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:35:31.761070+0000", 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:35:50.706 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:35:13.541230+0000", 2024-09-02T10:35:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.708 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:35:50.708 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.708 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.708 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.708 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.708 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.708 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:35:48.838794+0000", 2024-09-02T10:35:50.708 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.709 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:35:50.709 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.709 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.709 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.709 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.709 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:34:39.090176+0000", 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:35:31.749775+0000", 2024-09-02T10:35:50.710 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:35:07.603431+0000", 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:35:50.711 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:35:19.332549+0000", 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.712 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:34:34.046347+0000", 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:35:13.511653+0000", 2024-09-02T10:35:50.713 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 13958, 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:35:26.806817+0000", 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-09-02T10:35:50.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.e", 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: 4, 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: 2 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.1e", 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: 11, 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:35:50.715 INFO:teuthology.orchestra.run.smithi151.stdout: 9 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [ 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "9.10", 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: 0 2024-09-02T10:35:50.716 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "9.17", 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: 3, 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: 0 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:35:50.717 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:35:50.718 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:35:50.719 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:35:50.720 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:35:50.720 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:35:50.720 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:35:50.720 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:35:50.720 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:35:50.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:50.721 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:35:50.721 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:35:50.721 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:50 smithi151 ceph-mon[16639]: pgmap v2173: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 5.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s; 4837/244269 objects misplaced (1.980%); 12 MiB/s, 199 objects/s recovering 2024-09-02T10:35:50.721 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:50 smithi151 ceph-mon[16639]: osdmap e949: 12 total, 12 up, 11 in 2024-09-02T10:35:50.721 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1595669822' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:35:50.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:50.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:50.743 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:35:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:35:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:50 smithi187 ceph-mon[20072]: pgmap v2173: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 5.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s; 4837/244269 objects misplaced (1.980%); 12 MiB/s, 199 objects/s recovering 2024-09-02T10:35:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:50 smithi187 ceph-mon[20072]: osdmap e949: 12 total, 12 up, 11 in 2024-09-02T10:35:50.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1595669822' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:35:50.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:35:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:35:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:50 smithi162 ceph-mon[20279]: pgmap v2173: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 5.0 GiB data, 24 GiB used, 959 GiB / 983 GiB avail; 0 B/s wr, 1.06k op/s; 4837/244269 objects misplaced (1.980%); 12 MiB/s, 199 objects/s recovering 2024-09-02T10:35:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:50 smithi162 ceph-mon[20279]: osdmap e949: 12 total, 12 up, 11 in 2024-09-02T10:35:50.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1595669822' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:35:50.870 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:50.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:50.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:50.972 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:51.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:51.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:51.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:51.132 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:51.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:51.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:51.234 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:51.269 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:51.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:51.371 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:51.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:51.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:51.494 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:51.521 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:51.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:51.523 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:51.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:51.653 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:51.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:51.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:51 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:35:51.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:51 smithi151 ceph-mon[16639]: osdmap e950: 12 total, 12 up, 11 in 2024-09-02T10:35:51.754 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:51 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:35:51.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:51 smithi187 ceph-mon[20072]: osdmap e950: 12 total, 12 up, 11 in 2024-09-02T10:35:51.778 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:51.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:51 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-09-02T10:35:51.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:51 smithi162 ceph-mon[20279]: osdmap e950: 12 total, 12 up, 11 in 2024-09-02T10:35:51.883 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:51.910 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:51.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:51.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:52.012 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:52.037 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:52.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:52.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:52.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:52.163 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:52.265 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:52.293 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:52.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:52.395 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:52.420 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:52.421 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:52.522 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:52.547 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:52.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:52.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:52.649 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:52.672 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:52.672 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:52.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:52 smithi151 ceph-mon[16639]: 9.16 scrub starts 2024-09-02T10:35:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:52 smithi151 ceph-mon[16639]: pgmap v2176: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 4.9 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s; 4249/239508 objects misplaced (1.774%); 12 MiB/s, 193 objects/s recovering 2024-09-02T10:35:52.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:52 smithi151 ceph-mon[16639]: 9.13 scrub starts 2024-09-02T10:35:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:52 smithi187 ceph-mon[20072]: 9.16 scrub starts 2024-09-02T10:35:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:52 smithi187 ceph-mon[20072]: pgmap v2176: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 4.9 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s; 4249/239508 objects misplaced (1.774%); 12 MiB/s, 193 objects/s recovering 2024-09-02T10:35:52.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:52 smithi187 ceph-mon[20072]: 9.13 scrub starts 2024-09-02T10:35:52.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:52.794 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:52.795 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:52.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:52 smithi162 ceph-mon[20279]: 9.16 scrub starts 2024-09-02T10:35:52.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:52 smithi162 ceph-mon[20279]: pgmap v2176: 57 pgs: 3 active+recovering+undersized+remapped, 54 active+clean; 4.9 GiB data, 24 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.01k op/s; 4249/239508 objects misplaced (1.774%); 12 MiB/s, 193 objects/s recovering 2024-09-02T10:35:52.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:52 smithi162 ceph-mon[20279]: 9.13 scrub starts 2024-09-02T10:35:52.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:52.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:52.920 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:53.021 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:53.051 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:53.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:53.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:53.177 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:53.178 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:53.279 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:53.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:53.314 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:53.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:53.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:53.445 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:53.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:53.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:53.547 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:53.577 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:53.578 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:53.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:53 smithi187 ceph-mon[20072]: 9.14 scrub starts 2024-09-02T10:35:53.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:53 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:35:53.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:53 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:35:53.657 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:53 smithi187 ceph-mon[20072]: osdmap e951: 12 total, 12 up, 11 in 2024-09-02T10:35:53.679 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:53.687 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:53 smithi151 ceph-mon[16639]: 9.14 scrub starts 2024-09-02T10:35:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:53 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:35:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:53 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:35:53.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:53 smithi151 ceph-mon[16639]: osdmap e951: 12 total, 12 up, 11 in 2024-09-02T10:35:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:53.709 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:53.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:53.810 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:53.841 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:53.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:53.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:53 smithi162 ceph-mon[20279]: 9.14 scrub starts 2024-09-02T10:35:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:53 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:35:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:53 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:35:53.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:53 smithi162 ceph-mon[20279]: osdmap e951: 12 total, 12 up, 11 in 2024-09-02T10:35:53.943 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:53.963 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:53.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:53.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:54.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:54.098 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:54.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:54.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:54.200 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:54.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:54.234 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:54.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:54.335 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:54.364 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:54.365 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:54.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:54.487 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:54.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:54.488 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:54.589 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:54.607 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:54 smithi187 ceph-mon[20072]: pgmap v2178: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s; 4211/232365 objects misplaced (1.812%); 9.3 MiB/s, 148 objects/s recovering 2024-09-02T10:35:54.610 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:54.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:54.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:54.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:54 smithi151 ceph-mon[16639]: pgmap v2178: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s; 4211/232365 objects misplaced (1.812%); 9.3 MiB/s, 148 objects/s recovering 2024-09-02T10:35:54.713 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:54.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:54.742 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:54.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:54.844 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:54 smithi162 ceph-mon[20279]: pgmap v2178: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.7 GiB data, 23 GiB used, 960 GiB / 983 GiB avail; 0 B/s wr, 1.12k op/s; 4211/232365 objects misplaced (1.812%); 9.3 MiB/s, 148 objects/s recovering 2024-09-02T10:35:54.869 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:54.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:54.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:54.971 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:55.000 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:55.002 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:55.102 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:55.130 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:55.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:55.131 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:55.232 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:55.263 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:55.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:55.365 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:55.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:55.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:55.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:55.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:55.526 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:55.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:55.628 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:35:55.655 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:55.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:55.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:35:55.758 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:55.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:55.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:55.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:55.910 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:55.911 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:56.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:56.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:56.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:56.141 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:56.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:56.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:56.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:56.294 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:56.295 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:56.396 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:56.422 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:56.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:56.524 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:56.548 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:56.549 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:56.650 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:56.673 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:56.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:56.674 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:56.775 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:35:56.796 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:56.797 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:35:56.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:56.921 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:56.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:56.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:56 smithi151 ceph-mon[16639]: pgmap v2179: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.6 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 916 op/s; 2645/227763 objects misplaced (1.161%); 7.7 MiB/s, 122 objects/s recovering 2024-09-02T10:35:57.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:56 smithi187 ceph-mon[20072]: pgmap v2179: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.6 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 916 op/s; 2645/227763 objects misplaced (1.161%); 7.7 MiB/s, 122 objects/s recovering 2024-09-02T10:35:57.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:57.043 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:57.044 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:56 smithi162 ceph-mon[20279]: pgmap v2179: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.6 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 916 op/s; 2645/227763 objects misplaced (1.161%); 7.7 MiB/s, 122 objects/s recovering 2024-09-02T10:35:57.145 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:57.167 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:57.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:57.268 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:57.290 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:57.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:57.291 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:57.392 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:35:57.417 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:57.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:57.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:35:57.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:57.541 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:57.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:57.643 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:35:57.665 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:57.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:35:57.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:35:57.788 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:57.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:57.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:35:57.890 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:57.923 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:57.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:57.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:57 smithi187 ceph-mon[20072]: 9.16 scrub starts 2024-09-02T10:35:58.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:57 smithi187 ceph-mon[20072]: osdmap e952: 12 total, 12 up, 11 in 2024-09-02T10:35:58.024 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:58.045 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:57 smithi151 ceph-mon[16639]: 9.16 scrub starts 2024-09-02T10:35:58.046 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:57 smithi151 ceph-mon[16639]: osdmap e952: 12 total, 12 up, 11 in 2024-09-02T10:35:58.049 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:58.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:58.050 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:57 smithi162 ceph-mon[20279]: 9.16 scrub starts 2024-09-02T10:35:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:57 smithi162 ceph-mon[20279]: osdmap e952: 12 total, 12 up, 11 in 2024-09-02T10:35:58.150 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:58.184 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:58.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:58.286 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:58.312 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:58.313 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:58.414 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:58.436 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:58.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:58.538 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:35:58.569 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:58.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:58.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:35:58.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:58.692 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:58.693 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:58.794 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:58.827 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:58.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:58.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:58.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:58.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:58 smithi187 ceph-mon[20072]: 9.13 scrub starts 2024-09-02T10:35:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:58 smithi187 ceph-mon[20072]: pgmap v2181: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 986 op/s; 2645/221904 objects misplaced (1.192%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:35:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:58 smithi187 ceph-mon[20072]: osdmap e953: 12 total, 12 up, 11 in 2024-09-02T10:35:59.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:58 smithi187 ceph-mon[20072]: 9.17 scrub starts 2024-09-02T10:35:59.062 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:35:59.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:58 smithi151 ceph-mon[16639]: 9.13 scrub starts 2024-09-02T10:35:59.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:58 smithi151 ceph-mon[16639]: pgmap v2181: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 986 op/s; 2645/221904 objects misplaced (1.192%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:35:59.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:58 smithi151 ceph-mon[16639]: osdmap e953: 12 total, 12 up, 11 in 2024-09-02T10:35:59.079 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:58 smithi151 ceph-mon[16639]: 9.17 scrub starts 2024-09-02T10:35:59.084 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:59.085 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:35:59.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:58 smithi162 ceph-mon[20279]: 9.13 scrub starts 2024-09-02T10:35:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:58 smithi162 ceph-mon[20279]: pgmap v2181: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.5 GiB data, 23 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 986 op/s; 2645/221904 objects misplaced (1.192%); 4.9 MiB/s, 79 objects/s recovering 2024-09-02T10:35:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:58 smithi162 ceph-mon[20279]: osdmap e953: 12 total, 12 up, 11 in 2024-09-02T10:35:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:58 smithi162 ceph-mon[20279]: 9.17 scrub starts 2024-09-02T10:35:59.185 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:59.207 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:59.208 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:59.243 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:35:59.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:35:59.265 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:35:59.266 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:35:59.287 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:35:59.287 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:35:59.308 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:35:59.309 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:35:59.309 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:35:59.332 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:59.333 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:35:59.334 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:35:59.334 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:35:59.358 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:35:59.358 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:35:59.381 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:35:59.382 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:35:59.407 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:35:59.407 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:35:59.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:35:59.435 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:35:59.435 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:35:59.463 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:59.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:35:59.465 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:35:59.465 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:35:59.498 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:35:59.499 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:35:59.506 INFO:tasks.thrashosds.thrasher:j is {'epoch': 949, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:35:49.766455+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:34:16.863567+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7699999809265137, 'score_stable': 1.7699999809265137, 'optimal_score': 1, 'raw_score_acting': 1.7699999809265137, 'raw_score_stable': 1.7699999809265137, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-09-02T10:33:50.981540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '926', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '924', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5, 'score_stable': 2.119999885559082, 'optimal_score': 1, 'raw_score_acting': 5, 'raw_score_stable': 2.119999885559082, 'primary_affinity_weighted': 0.916666567325592, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.9200000166893005, 'error_message': 'pool 9 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 891, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 946, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 891, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 948, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 926, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 939, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 926, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 941, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 926, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 947, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 943, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:35:16.979832+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:34:57.431566+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:35:31.761070+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:35:13.541230+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:35:48.838794+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:34:39.090176+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:35:31.749775+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:35:07.603431+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:35:19.332549+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:34:34.046347+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:35:13.511653+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 13958, 'last_purged_snaps_scrub': '2024-09-02T10:35:26.806817+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.e', 'osds': [4, 9, 2]}, {'pgid': '1.1e', 'osds': [11, 10, 9]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.10', 'osds': [1, 0]}, {'pgid': '9.17', 'osds': [3, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:35:59.506 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:35:59.513 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:35:59.523 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:35:59.535 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:35:59.535 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:35:59.565 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:59.592 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:59.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:59.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:35:59.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:35:59.723 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:35:59.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:59.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:35:59.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:35:59.848 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:35:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:59.849 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:35:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:59 smithi151 ceph-mon[16639]: 9.14 scrub starts 2024-09-02T10:35:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2241213124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:35:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:35:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:35:59.950 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:35:59.981 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:35:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:35:59.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:59 smithi187 ceph-mon[20072]: 9.14 scrub starts 2024-09-02T10:36:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2241213124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:00.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:35:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:36:00.083 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:00.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:59 smithi162 ceph-mon[20279]: 9.14 scrub starts 2024-09-02T10:36:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2241213124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:35:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-09-02T10:36:00.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:00.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:00.211 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:00.237 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:00.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:00.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:00.339 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:00.366 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:00.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:00.467 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:00.490 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:00.491 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:00.591 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:00.619 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:00.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:00.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:00.721 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:00.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:00.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:00.850 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:00.874 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:00.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:00 smithi151 ceph-mon[16639]: pgmap v2183: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 2645/215355 objects misplaced (1.228%); 4.7 MiB/s, 74 objects/s recovering 2024-09-02T10:36:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3411739163' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:00.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:00.998 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:00.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:00 smithi187 ceph-mon[20072]: pgmap v2183: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 2645/215355 objects misplaced (1.228%); 4.7 MiB/s, 74 objects/s recovering 2024-09-02T10:36:01.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3411739163' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:00 smithi162 ceph-mon[20279]: pgmap v2183: 57 pgs: 2 active+recovering+undersized+remapped, 55 active+clean; 4.4 GiB data, 22 GiB used, 961 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 2645/215355 objects misplaced (1.228%); 4.7 MiB/s, 74 objects/s recovering 2024-09-02T10:36:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3411739163' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:01.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:01.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:01.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:01.123 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:01.224 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:01.249 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:01.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:01.250 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:01.351 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:01.377 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:01.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:01.388 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:36:01.479 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:01.501 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:01.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:01.502 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:01.513 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:36:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:36:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:36:01.513 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:36:01.513 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:36:01.603 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:01.636 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:01.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:01.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:01.738 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:01.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:01 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:01.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3411739163' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:01.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:01 smithi151 ceph-mon[16639]: osdmap e954: 12 total, 12 up, 11 in 2024-09-02T10:36:01.767 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:01.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:01.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:01.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:01.895 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:01.896 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:01.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:01 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3411739163' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:02.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:01 smithi187 ceph-mon[20072]: osdmap e954: 12 total, 12 up, 11 in 2024-09-02T10:36:02.026 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:02.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:02.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:02.038 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:02.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:01 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3411739163' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:01 smithi162 ceph-mon[20279]: osdmap e954: 12 total, 12 up, 11 in 2024-09-02T10:36:02.128 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:02.151 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:02.152 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:02.253 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:02.278 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:02.279 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:02.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:02.408 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:02.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:02.510 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:02.539 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:02.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:02.641 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:02.665 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:02.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:02.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:02.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:02.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:02.798 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-02T10:36:02.897 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:02.927 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:02.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:02 smithi187 ceph-mon[20072]: 9.16 scrub starts 2024-09-02T10:36:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:02 smithi187 ceph-mon[20072]: pgmap v2185: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 4.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 964 op/s; 1082/210405 objects misplaced (0.514%); 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:36:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:02 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:02 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:03.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:02 smithi187 ceph-mon[20072]: osdmap e955: 12 total, 12 up, 11 in 2024-09-02T10:36:03.029 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:03.053 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:03.054 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:02 smithi162 ceph-mon[20279]: 9.16 scrub starts 2024-09-02T10:36:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:02 smithi162 ceph-mon[20279]: pgmap v2185: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 4.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 964 op/s; 1082/210405 objects misplaced (0.514%); 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:36:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:02 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:02 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:02 smithi162 ceph-mon[20279]: osdmap e955: 12 total, 12 up, 11 in 2024-09-02T10:36:03.155 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:03.181 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:03.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:03.182 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:03.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:02 smithi151 ceph-mon[16639]: 9.16 scrub starts 2024-09-02T10:36:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:02 smithi151 ceph-mon[16639]: pgmap v2185: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 4.3 GiB data, 22 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 964 op/s; 1082/210405 objects misplaced (0.514%); 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:36:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:02 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:02 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:02 smithi151 ceph-mon[16639]: osdmap e955: 12 total, 12 up, 11 in 2024-09-02T10:36:03.246 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:36:03.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:03.320 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:03.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:03.422 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:03.446 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:03.447 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:03.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:03.615 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:03.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:03.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:03.716 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:03.741 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:03.742 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:03.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:03 smithi151 ceph-mon[16639]: 9.13 scrub starts 2024-09-02T10:36:03.824 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:03 smithi151 ceph-mon[16639]: 9.17 scrub starts 2024-09-02T10:36:03.842 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:03.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:03.870 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:03.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:03.958 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:03.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:03 smithi187 ceph-mon[20072]: 9.13 scrub starts 2024-09-02T10:36:04.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:03 smithi187 ceph-mon[20072]: 9.17 scrub starts 2024-09-02T10:36:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:04.019 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:04.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:04.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:03 smithi162 ceph-mon[20279]: 9.13 scrub starts 2024-09-02T10:36:04.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:03 smithi162 ceph-mon[20279]: 9.17 scrub starts 2024-09-02T10:36:04.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:04.145 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:04.146 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:04.178 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:36:04.179 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:36:04.247 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:04.275 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:04.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:04.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:04.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:04.405 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:04.406 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:04.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:04.535 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:04.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:04.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:04.637 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:04.663 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:04.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:04.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:04.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:04.794 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:04.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:04.896 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:04.928 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:04.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:04 smithi187 ceph-mon[20072]: 9.14 scrub starts 2024-09-02T10:36:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:04 smithi187 ceph-mon[20072]: pgmap v2187: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 4.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 1082/202446 objects misplaced (0.534%); 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:36:05.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-02T10:36:05.031 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:05.055 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:05.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:05.056 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:05.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:04 smithi151 ceph-mon[16639]: 9.14 scrub starts 2024-09-02T10:36:05.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:04 smithi151 ceph-mon[16639]: pgmap v2187: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 4.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 1082/202446 objects misplaced (0.534%); 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:36:05.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-02T10:36:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:04 smithi162 ceph-mon[20279]: 9.14 scrub starts 2024-09-02T10:36:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:04 smithi162 ceph-mon[20279]: pgmap v2187: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 4.1 GiB data, 21 GiB used, 962 GiB / 983 GiB avail; 0 B/s wr, 1.08k op/s; 1082/202446 objects misplaced (0.534%); 5.4 MiB/s, 86 objects/s recovering 2024-09-02T10:36:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-09-02T10:36:05.157 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:05.186 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:05.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:05.287 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:05.311 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:05.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:05.413 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:05.446 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:05.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:05.548 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:05.575 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:05.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:05.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:05.583 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:36:05.677 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:05.701 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:05.703 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:05.804 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:05.828 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:05.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:05.896 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:05 smithi151 ceph-mon[16639]: from='client.18990 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:36:05.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:05.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:05.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:05.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:06.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:05 smithi187 ceph-mon[20072]: from='client.18990 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:36:06.065 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:06.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:06.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:06.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:06.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:05 smithi162 ceph-mon[20279]: from='client.18990 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:36:06.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:06.218 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:06.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:06.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:06.348 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:06.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:06.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:06.418 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:36:06.418 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:36:06.450 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:36:06.450 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:36:06.450 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:06.481 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:36:06.481 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:36:06.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:06.485 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:06.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:06.515 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:36:06.516 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:36:06.585 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:06.608 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:06.609 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:06.710 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:06.734 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:06.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:06.735 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:06.746 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:36:06.746 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:36:06.774 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:36:06.774 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:36:06.803 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:36:06.803 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:36:06.833 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:36:06.833 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:36:06.836 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:06.863 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:36:06.863 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:36:06.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:06.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:06.890 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:36:06.890 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:36:06.918 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:36:06.919 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:36:06.950 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:36:06.950 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:36:06.966 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:06.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:06.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:06 smithi151 ceph-mon[16639]: pgmap v2188: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 4.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 279/197154 objects misplaced (0.142%); 6.2 MiB/s, 99 objects/s recovering 2024-09-02T10:36:06.970 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1597491211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:06.973 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2187,"stamp":"2024-09-02T10:36:03.612467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4422828600,"num_objects":67482,"num_object_clones":0,"num_object_copies":202446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1082,"num_objects_unfound":0,"num_objects_dirty":67482,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95439,"num_write_kb":4149889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9489,"num_bytes_recovered":622474848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":169522,"ondisk_log_size":169522,"up":171,"acting":170,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":180,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":22296584,"kb_used_data":13879156,"kb_used_omap":17,"kb_used_meta":8417198,"kb_avail":1008900088,"statfs":{"total":1055945392128,"available":1033113690112,"internally_reserved":0,"allocated":14212255744,"data_stored":14208963280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17545,"internal_metadata":8619211639},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,4,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-425066496,"num_objects":-6486,"num_object_clones":0,"num_object_copies":-19458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1563,"num_objects_unfound":0,"num_objects_dirty":-6486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6486,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":521,"num_bytes_recovered":34144256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001346"},"pg_stats":[{"pgid":"9.17","version":"953'5806","reported_seq":13544,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:59.657183+0000","last_change":"2024-09-02T10:35:58.106218+0000","last_active":"2024-09-02T10:35:59.657183+0000","last_peered":"2024-09-02T10:35:59.657183+0000","last_clean":"2024-09-02T10:35:59.657183+0000","last_became_active":"2024-09-02T10:35:57.707720+0000","last_became_peered":"2024-09-02T10:35:57.707720+0000","last_unstale":"2024-09-02T10:35:59.657183+0000","last_undegraded":"2024-09-02T10:35:59.657183+0000","last_fullsized":"2024-09-02T10:35:59.657183+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":5806,"log_dups_size":0,"ondisk_log_size":5806,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-02T10:36:03.098709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144408576,"num_objects":2204,"num_object_clones":0,"num_object_copies":6612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3956,"num_write_kb":197088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1898,"num_bytes_recovered":124387328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.16","version":"955'6012","reported_seq":13919,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:03.046181+0000","last_change":"2024-09-02T10:36:00.953605+0000","last_active":"2024-09-02T10:36:03.046181+0000","last_peered":"2024-09-02T10:36:03.046181+0000","last_clean":"2024-09-02T10:36:03.046181+0000","last_became_active":"2024-09-02T10:35:38.829158+0000","last_became_peered":"2024-09-02T10:35:38.829158+0000","last_unstale":"2024-09-02T10:36:03.046181+0000","last_undegraded":"2024-09-02T10:36:03.046181+0000","last_fullsized":"2024-09-02T10:36:03.046181+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":6012,"log_dups_size":0,"ondisk_log_size":6012,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-02T10:36:05.948238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136151040,"num_objects":2078,"num_object_clones":0,"num_object_copies":6234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4124,"num_write_kb":198432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1936,"num_bytes_recovered":126812160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1e","version":"22'92","reported_seq":106,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.705457+0000","last_change":"2024-09-02T10:34:29.260151+0000","last_active":"2024-09-02T10:35:57.705457+0000","last_peered":"2024-09-02T10:35:57.705457+0000","last_clean":"2024-09-02T10:35:57.705457+0000","last_became_active":"2024-09-02T10:34:17.880876+0000","last_became_peered":"2024-09-02T10:34:17.880876+0000","last_unstale":"2024-09-02T10:35:57.705457+0000","last_undegraded":"2024-09-02T10:35:57.705457+0000","last_fullsized":"2024-09-02T10:35:57.705457+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:29.260104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:29.260104+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.15","version":"953'5803","reported_seq":11764,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.278800+0000","last_change":"2024-09-02T10:35:25.713754+0000","last_active":"2024-09-02T10:36:01.278800+0000","last_peered":"2024-09-02T10:36:01.278800+0000","last_clean":"2024-09-02T10:36:01.278800+0000","last_became_active":"2024-09-02T10:35:24.380270+0000","last_became_peered":"2024-09-02T10:35:24.380270+0000","last_unstale":"2024-09-02T10:36:01.278800+0000","last_undegraded":"2024-09-02T10:36:01.278800+0000","last_fullsized":"2024-09-02T10:36:01.278800+0000","mapping_epoch":941,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":942,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'4781","last_scrub_stamp":"2024-09-02T10:35:25.713545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:25.713545+0000","objects_scrubbed":2905,"log_size":5803,"log_dups_size":0,"ondisk_log_size":5803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:36.926480+0000","scrub_duration":554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137691136,"num_objects":2101,"num_object_clones":0,"num_object_copies":6303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1034,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,4],"acting":[7,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":475,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.390500+0000","last_change":"2024-09-02T10:34:00.928724+0000","last_active":"2024-09-02T10:36:01.390500+0000","last_peered":"2024-09-02T10:36:01.390500+0000","last_clean":"2024-09-02T10:36:01.390500+0000","last_became_active":"2024-09-02T10:33:38.659815+0000","last_became_peered":"2024-09-02T10:33:38.659815+0000","last_unstale":"2024-09-02T10:36:01.390500+0000","last_undegraded":"2024-09-02T10:36:01.390500+0000","last_fullsized":"2024-09-02T10:36:01.390500+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.928661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.928661+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:03.722813+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"954'5997","reported_seq":12092,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.734618+0000","last_change":"2024-09-02T10:35:57.985571+0000","last_active":"2024-09-02T10:36:01.734618+0000","last_peered":"2024-09-02T10:36:01.734618+0000","last_clean":"2024-09-02T10:36:01.734618+0000","last_became_active":"2024-09-02T10:35:28.717046+0000","last_became_peered":"2024-09-02T10:35:28.717046+0000","last_unstale":"2024-09-02T10:36:01.734618+0000","last_undegraded":"2024-09-02T10:36:01.734618+0000","last_fullsized":"2024-09-02T10:36:01.734618+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":5997,"log_dups_size":0,"ondisk_log_size":5997,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-02T10:36:02.982657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135593984,"num_objects":2069,"num_object_clones":0,"num_object_copies":6207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4087,"num_write_kb":196992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1974,"num_bytes_recovered":129368064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,3],"acting":[10,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1594,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.705473+0000","last_change":"2024-09-02T10:33:59.232980+0000","last_active":"2024-09-02T10:35:57.705473+0000","last_peered":"2024-09-02T10:35:57.705473+0000","last_clean":"2024-09-02T10:35:57.705473+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:35:57.705473+0000","last_undegraded":"2024-09-02T10:35:57.705473+0000","last_fullsized":"2024-09-02T10:35:57.705473+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:59.232916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:59.232916+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"955'6177","reported_seq":12405,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.544379+0000","last_change":"2024-09-02T10:36:01.903494+0000","last_active":"2024-09-02T10:36:02.544379+0000","last_peered":"2024-09-02T10:36:02.544379+0000","last_clean":"2024-09-02T10:36:02.544379+0000","last_became_active":"2024-09-02T10:35:50.773879+0000","last_became_peered":"2024-09-02T10:35:50.773879+0000","last_unstale":"2024-09-02T10:36:02.544379+0000","last_undegraded":"2024-09-02T10:36:02.544379+0000","last_fullsized":"2024-09-02T10:36:02.544379+0000","mapping_epoch":949,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":950,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":6177,"log_dups_size":0,"ondisk_log_size":6177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"user requested scrub scheduled @ 2024-09-02T10:36:06.903178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139395072,"num_objects":2127,"num_object_clones":0,"num_object_copies":6381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4221,"num_write_kb":203136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2024,"num_bytes_recovered":132644864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1570,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.705346+0000","last_change":"2024-09-02T10:33:55.174212+0000","last_active":"2024-09-02T10:35:57.705346+0000","last_peered":"2024-09-02T10:35:57.705346+0000","last_clean":"2024-09-02T10:35:57.705346+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:35:57.705346+0000","last_undegraded":"2024-09-02T10:35:57.705346+0000","last_fullsized":"2024-09-02T10:35:57.705346+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.174147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.174147+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.12","version":"955'5940","reported_seq":11942,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.726111+0000","last_change":"2024-09-02T10:35:24.913387+0000","last_active":"2024-09-02T10:36:02.726111+0000","last_peered":"2024-09-02T10:36:02.726111+0000","last_clean":"2024-09-02T10:36:02.726111+0000","last_became_active":"2024-09-02T10:35:22.670593+0000","last_became_peered":"2024-09-02T10:35:22.670593+0000","last_unstale":"2024-09-02T10:36:02.726111+0000","last_undegraded":"2024-09-02T10:36:02.726111+0000","last_fullsized":"2024-09-02T10:36:02.726111+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'4822","last_scrub_stamp":"2024-09-02T10:35:24.913280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:24.913280+0000","objects_scrubbed":2914,"log_size":5940,"log_dups_size":0,"ondisk_log_size":5940,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:36.920495+0000","scrub_duration":1293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134873088,"num_objects":2058,"num_object_clones":0,"num_object_copies":6174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1138,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":889,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.700331+0000","last_change":"2024-09-02T10:33:55.588678+0000","last_active":"2024-09-02T10:35:57.700331+0000","last_peered":"2024-09-02T10:35:57.700331+0000","last_clean":"2024-09-02T10:35:57.700331+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:35:57.700331+0000","last_undegraded":"2024-09-02T10:35:57.700331+0000","last_fullsized":"2024-09-02T10:35:57.700331+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.588605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.588605+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"953'5760","reported_seq":9546,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.320508+0000","last_change":"2024-09-02T10:35:15.638534+0000","last_active":"2024-09-02T10:36:01.320508+0000","last_peered":"2024-09-02T10:36:01.320508+0000","last_clean":"2024-09-02T10:36:01.320508+0000","last_became_active":"2024-09-02T10:35:14.386404+0000","last_became_peered":"2024-09-02T10:35:14.386404+0000","last_unstale":"2024-09-02T10:36:01.320508+0000","last_undegraded":"2024-09-02T10:36:01.320508+0000","last_fullsized":"2024-09-02T10:36:01.320508+0000","mapping_epoch":933,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"935'4488","last_scrub_stamp":"2024-09-02T10:35:15.638360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:15.638360+0000","objects_scrubbed":2543,"log_size":5760,"log_dups_size":0,"ondisk_log_size":5760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:42.403420+0000","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130285568,"num_objects":1988,"num_object_clones":0,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1988,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1275,"num_write_kb":23040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":1434,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.391797+0000","last_change":"2024-09-02T10:33:56.473614+0000","last_active":"2024-09-02T10:36:01.391797+0000","last_peered":"2024-09-02T10:36:01.391797+0000","last_clean":"2024-09-02T10:36:01.391797+0000","last_became_active":"2024-09-02T10:24:54.196721+0000","last_became_peered":"2024-09-02T10:24:54.196721+0000","last_unstale":"2024-09-02T10:36:01.391797+0000","last_undegraded":"2024-09-02T10:36:01.391797+0000","last_fullsized":"2024-09-02T10:36:01.391797+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.473555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.473555+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:06.030545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.10","version":"953'6056","reported_seq":12805,"reported_epoch":953,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:35:59.711696+0000","last_change":"2024-09-02T10:35:21.626094+0000","last_active":"2024-09-02T10:35:59.711696+0000","last_peered":"2024-09-02T10:35:59.711696+0000","last_clean":"2024-09-02T10:34:48.025225+0000","last_became_active":"2024-09-02T10:34:49.797386+0000","last_became_peered":"2024-09-02T10:34:49.797386+0000","last_unstale":"2024-09-02T10:35:59.711696+0000","last_undegraded":"2024-09-02T10:35:59.711696+0000","last_fullsized":"2024-09-02T10:34:49.368475+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":6056,"log_dups_size":0,"ondisk_log_size":6056,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144703488,"num_objects":2208,"num_object_clones":0,"num_object_copies":6624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1082,"num_objects_unfound":0,"num_objects_dirty":2208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4128,"num_write_kb":202752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1653,"num_bytes_recovered":108331008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[1,0],"avail_no_missing":["1","6","8"],"object_location_counts":[{"shards":"0,1","objects":1847},{"shards":"1,6,7,8","objects":1},{"shards":"1,6,8","objects":360}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1955,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.399264+0000","last_change":"2024-09-02T10:34:00.430544+0000","last_active":"2024-09-02T10:36:02.399264+0000","last_peered":"2024-09-02T10:36:02.399264+0000","last_clean":"2024-09-02T10:36:02.399264+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:36:02.399264+0000","last_undegraded":"2024-09-02T10:36:02.399264+0000","last_fullsized":"2024-09-02T10:36:02.399264+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.430459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.430459+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1594,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.705533+0000","last_change":"2024-09-02T10:33:56.194537+0000","last_active":"2024-09-02T10:35:57.705533+0000","last_peered":"2024-09-02T10:35:57.705533+0000","last_clean":"2024-09-02T10:35:57.705533+0000","last_became_active":"2024-09-02T10:09:58.903726+0000","last_became_peered":"2024-09-02T10:09:58.903726+0000","last_unstale":"2024-09-02T10:35:57.705533+0000","last_undegraded":"2024-09-02T10:35:57.705533+0000","last_fullsized":"2024-09-02T10:35:57.705533+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.194480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.194480+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":157,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.396061+0000","last_change":"2024-09-02T10:33:56.167103+0000","last_active":"2024-09-02T10:36:02.396061+0000","last_peered":"2024-09-02T10:36:02.396061+0000","last_clean":"2024-09-02T10:36:02.396061+0000","last_became_active":"2024-09-02T10:33:38.657147+0000","last_became_peered":"2024-09-02T10:33:38.657147+0000","last_unstale":"2024-09-02T10:36:02.396061+0000","last_undegraded":"2024-09-02T10:36:02.396061+0000","last_fullsized":"2024-09-02T10:36:02.396061+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.167029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.167029+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1561,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.717534+0000","last_change":"2024-09-02T10:33:55.173514+0000","last_active":"2024-09-02T10:35:57.717534+0000","last_peered":"2024-09-02T10:35:57.717534+0000","last_clean":"2024-09-02T10:35:57.717534+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:35:57.717534+0000","last_undegraded":"2024-09-02T10:35:57.717534+0000","last_fullsized":"2024-09-02T10:35:57.717534+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.173450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.173450+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":807,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.390383+0000","last_change":"2024-09-02T10:33:56.943878+0000","last_active":"2024-09-02T10:36:01.390383+0000","last_peered":"2024-09-02T10:36:01.390383+0000","last_clean":"2024-09-02T10:36:01.390383+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:36:01.390383+0000","last_undegraded":"2024-09-02T10:36:01.390383+0000","last_fullsized":"2024-09-02T10:36:01.390383+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.943826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.943826+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:02.711831+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":630,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.394769+0000","last_change":"2024-09-02T10:34:36.258204+0000","last_active":"2024-09-02T10:36:01.394769+0000","last_peered":"2024-09-02T10:36:01.394769+0000","last_clean":"2024-09-02T10:36:01.394769+0000","last_became_active":"2024-09-02T10:33:38.657430+0000","last_became_peered":"2024-09-02T10:33:38.657430+0000","last_unstale":"2024-09-02T10:36:01.394769+0000","last_undegraded":"2024-09-02T10:36:01.394769+0000","last_fullsized":"2024-09-02T10:36:01.394769+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:36.258130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:36.258130+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,0],"acting":[10,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1854,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.402610+0000","last_change":"2024-09-02T10:34:08.628331+0000","last_active":"2024-09-02T10:36:02.402610+0000","last_peered":"2024-09-02T10:36:02.402610+0000","last_clean":"2024-09-02T10:36:02.402610+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:36:02.402610+0000","last_undegraded":"2024-09-02T10:36:02.402610+0000","last_fullsized":"2024-09-02T10:36:02.402610+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.628247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.628247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"953'5965","reported_seq":13572,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:59.126482+0000","last_change":"2024-09-02T10:35:17.922435+0000","last_active":"2024-09-02T10:35:59.126482+0000","last_peered":"2024-09-02T10:35:59.126482+0000","last_clean":"2024-09-02T10:35:59.126482+0000","last_became_active":"2024-09-02T10:35:16.385649+0000","last_became_peered":"2024-09-02T10:35:16.385649+0000","last_unstale":"2024-09-02T10:35:59.126482+0000","last_undegraded":"2024-09-02T10:35:59.126482+0000","last_fullsized":"2024-09-02T10:35:59.126482+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":5,"last_scrub":"937'4711","last_scrub_stamp":"2024-09-02T10:35:17.922197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:17.922197+0000","objects_scrubbed":2830,"log_size":5965,"log_dups_size":0,"ondisk_log_size":5965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:17.941708+0000","scrub_duration":1077,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148570112,"num_objects":2267,"num_object_clones":0,"num_object_copies":6801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1277,"num_write_kb":22848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":593,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.395726+0000","last_change":"2024-09-02T10:34:01.246043+0000","last_active":"2024-09-02T10:36:02.395726+0000","last_peered":"2024-09-02T10:36:02.395726+0000","last_clean":"2024-09-02T10:36:02.395726+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:36:02.395726+0000","last_undegraded":"2024-09-02T10:36:02.395726+0000","last_fullsized":"2024-09-02T10:36:02.395726+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.245989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.245989+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"954'5992","reported_seq":10126,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.752063+0000","last_change":"2024-09-02T10:35:09.830530+0000","last_active":"2024-09-02T10:36:01.752063+0000","last_peered":"2024-09-02T10:36:01.752063+0000","last_clean":"2024-09-02T10:36:01.752063+0000","last_became_active":"2024-09-02T10:35:08.727384+0000","last_became_peered":"2024-09-02T10:35:08.727384+0000","last_unstale":"2024-09-02T10:36:01.752063+0000","last_undegraded":"2024-09-02T10:36:01.752063+0000","last_fullsized":"2024-09-02T10:36:01.752063+0000","mapping_epoch":931,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":932,"parent":"0.0","parent_split_bits":5,"last_scrub":"932'4480","last_scrub_stamp":"2024-09-02T10:35:09.830443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:09.830443+0000","objects_scrubbed":2522,"log_size":5992,"log_dups_size":0,"ondisk_log_size":5992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:22.109704+0000","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137560064,"num_objects":2099,"num_object_clones":0,"num_object_copies":6297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1521,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,5],"acting":[10,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1824,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.399355+0000","last_change":"2024-09-02T10:34:05.558912+0000","last_active":"2024-09-02T10:36:02.399355+0000","last_peered":"2024-09-02T10:36:02.399355+0000","last_clean":"2024-09-02T10:36:02.399355+0000","last_became_active":"2024-09-02T10:09:58.909791+0000","last_became_peered":"2024-09-02T10:09:58.909791+0000","last_unstale":"2024-09-02T10:36:02.399355+0000","last_undegraded":"2024-09-02T10:36:02.399355+0000","last_fullsized":"2024-09-02T10:36:02.399355+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:05.558852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:05.558852+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:02.575766+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"955'6039","reported_seq":6206,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.548496+0000","last_change":"2024-09-02T10:34:49.121880+0000","last_active":"2024-09-02T10:36:02.548496+0000","last_peered":"2024-09-02T10:36:02.548496+0000","last_clean":"2024-09-02T10:36:02.548496+0000","last_became_active":"2024-09-02T10:34:47.026177+0000","last_became_peered":"2024-09-02T10:34:47.026177+0000","last_unstale":"2024-09-02T10:36:02.548496+0000","last_undegraded":"2024-09-02T10:36:02.548496+0000","last_fullsized":"2024-09-02T10:36:02.548496+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'3905","last_scrub_stamp":"2024-09-02T10:34:49.121789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.121789+0000","objects_scrubbed":1960,"log_size":6039,"log_dups_size":0,"ondisk_log_size":6039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":225,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133169152,"num_objects":2032,"num_object_clones":0,"num_object_copies":6096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2134,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,8],"acting":[6,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1918,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.399388+0000","last_change":"2024-09-02T10:34:07.590524+0000","last_active":"2024-09-02T10:36:02.399388+0000","last_peered":"2024-09-02T10:36:02.399388+0000","last_clean":"2024-09-02T10:36:02.399388+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:36:02.399388+0000","last_undegraded":"2024-09-02T10:36:02.399388+0000","last_fullsized":"2024-09-02T10:36:02.399388+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:07.590464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:07.590464+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"953'5811","reported_seq":5973,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:59.352531+0000","last_change":"2024-09-02T10:34:50.303405+0000","last_active":"2024-09-02T10:35:59.352531+0000","last_peered":"2024-09-02T10:35:59.352531+0000","last_clean":"2024-09-02T10:35:59.352531+0000","last_became_active":"2024-09-02T10:34:47.038024+0000","last_became_peered":"2024-09-02T10:34:47.038024+0000","last_unstale":"2024-09-02T10:35:59.352531+0000","last_undegraded":"2024-09-02T10:35:59.352531+0000","last_fullsized":"2024-09-02T10:35:59.352531+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'3772","last_scrub_stamp":"2024-09-02T10:34:50.303087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:50.303087+0000","objects_scrubbed":1991,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:13.243929+0000","scrub_duration":777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142016512,"num_objects":2167,"num_object_clones":0,"num_object_copies":6501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2050,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":2032,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.717226+0000","last_change":"2024-09-02T10:30:24.407289+0000","last_active":"2024-09-02T10:35:57.717226+0000","last_peered":"2024-09-02T10:35:57.717226+0000","last_clean":"2024-09-02T10:35:57.717226+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:35:57.717226+0000","last_undegraded":"2024-09-02T10:35:57.717226+0000","last_fullsized":"2024-09-02T10:35:57.717226+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:24.407218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:24.407218+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"953'5861","reported_seq":6022,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:59.198648+0000","last_change":"2024-09-02T10:34:48.857422+0000","last_active":"2024-09-02T10:35:59.198648+0000","last_peered":"2024-09-02T10:35:59.198648+0000","last_clean":"2024-09-02T10:35:59.198648+0000","last_became_active":"2024-09-02T10:34:47.038074+0000","last_became_peered":"2024-09-02T10:34:47.038074+0000","last_unstale":"2024-09-02T10:35:59.198648+0000","last_undegraded":"2024-09-02T10:35:59.198648+0000","last_fullsized":"2024-09-02T10:35:59.198648+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3826","last_scrub_stamp":"2024-09-02T10:34:48.857333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:48.857333+0000","objects_scrubbed":1952,"log_size":5861,"log_dups_size":0,"ondisk_log_size":5861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:03.160949+0000","scrub_duration":482,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143196160,"num_objects":2185,"num_object_clones":0,"num_object_copies":6555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2035,"num_write_kb":72576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1695,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.404135+0000","last_change":"2024-09-02T10:33:56.586452+0000","last_active":"2024-09-02T10:36:02.404135+0000","last_peered":"2024-09-02T10:36:02.404135+0000","last_clean":"2024-09-02T10:36:02.404135+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:36:02.404135+0000","last_undegraded":"2024-09-02T10:36:02.404135+0000","last_fullsized":"2024-09-02T10:36:02.404135+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:56.586392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:56.586392+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"955'6069","reported_seq":6234,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:03.071126+0000","last_change":"2024-09-02T10:34:49.856610+0000","last_active":"2024-09-02T10:36:03.071126+0000","last_peered":"2024-09-02T10:36:03.071126+0000","last_clean":"2024-09-02T10:36:03.071126+0000","last_became_active":"2024-09-02T10:34:47.042672+0000","last_became_peered":"2024-09-02T10:34:47.042672+0000","last_unstale":"2024-09-02T10:36:03.071126+0000","last_undegraded":"2024-09-02T10:36:03.071126+0000","last_fullsized":"2024-09-02T10:36:03.071126+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'3969","last_scrub_stamp":"2024-09-02T10:34:49.856383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.856383+0000","objects_scrubbed":1996,"log_size":6069,"log_dups_size":0,"ondisk_log_size":6069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:03.921535+0000","scrub_duration":330,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135331840,"num_objects":2065,"num_object_clones":0,"num_object_copies":6195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":69504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":672,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.396465+0000","last_change":"2024-09-02T10:34:08.328616+0000","last_active":"2024-09-02T10:36:02.396465+0000","last_peered":"2024-09-02T10:36:02.396465+0000","last_clean":"2024-09-02T10:36:02.396465+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:36:02.396465+0000","last_undegraded":"2024-09-02T10:36:02.396465+0000","last_fullsized":"2024-09-02T10:36:02.396465+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.328570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.328570+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"953'7764","reported_seq":7890,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:59.332294+0000","last_change":"2024-09-02T10:33:52.761107+0000","last_active":"2024-09-02T10:35:59.332294+0000","last_peered":"2024-09-02T10:35:59.332294+0000","last_clean":"2024-09-02T10:35:59.332294+0000","last_became_active":"2024-09-02T10:33:52.760804+0000","last_became_peered":"2024-09-02T10:33:52.760804+0000","last_unstale":"2024-09-02T10:35:59.332294+0000","last_undegraded":"2024-09-02T10:35:59.332294+0000","last_fullsized":"2024-09-02T10:35:59.332294+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":7764,"log_dups_size":0,"ondisk_log_size":7764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:02.152790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279183360,"num_objects":4260,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7764,"num_write_kb":384768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":997,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.705405+0000","last_change":"2024-09-02T10:34:17.882133+0000","last_active":"2024-09-02T10:35:57.705405+0000","last_peered":"2024-09-02T10:35:57.705405+0000","last_clean":"2024-09-02T10:35:57.705405+0000","last_became_active":"2024-09-02T10:34:17.881745+0000","last_became_peered":"2024-09-02T10:34:17.881745+0000","last_unstale":"2024-09-02T10:35:57.705405+0000","last_undegraded":"2024-09-02T10:35:57.705405+0000","last_fullsized":"2024-09-02T10:35:57.705405+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:58.800694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:58.800694+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"953'7865","reported_seq":7992,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.315984+0000","last_change":"2024-09-02T10:33:52.926793+0000","last_active":"2024-09-02T10:36:01.315984+0000","last_peered":"2024-09-02T10:36:01.315984+0000","last_clean":"2024-09-02T10:36:01.315984+0000","last_became_active":"2024-09-02T10:33:52.926467+0000","last_became_peered":"2024-09-02T10:33:52.926467+0000","last_unstale":"2024-09-02T10:36:01.315984+0000","last_undegraded":"2024-09-02T10:36:01.315984+0000","last_fullsized":"2024-09-02T10:36:01.315984+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":7865,"log_dups_size":0,"ondisk_log_size":7865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271319064,"num_objects":4141,"num_object_clones":0,"num_object_copies":12423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4141,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7865,"num_write_kb":384129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":1457,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.391894+0000","last_change":"2024-09-02T10:34:08.280819+0000","last_active":"2024-09-02T10:36:01.391894+0000","last_peered":"2024-09-02T10:36:01.391894+0000","last_clean":"2024-09-02T10:36:01.391894+0000","last_became_active":"2024-09-02T10:20:52.103682+0000","last_became_peered":"2024-09-02T10:20:52.103682+0000","last_unstale":"2024-09-02T10:36:01.391894+0000","last_undegraded":"2024-09-02T10:36:01.391894+0000","last_fullsized":"2024-09-02T10:36:01.391894+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.280762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.280762+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"953'7931","reported_seq":8057,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.306447+0000","last_change":"2024-09-02T10:33:52.925479+0000","last_active":"2024-09-02T10:36:01.306447+0000","last_peered":"2024-09-02T10:36:01.306447+0000","last_clean":"2024-09-02T10:36:01.306447+0000","last_became_active":"2024-09-02T10:33:52.925170+0000","last_became_peered":"2024-09-02T10:33:52.925170+0000","last_unstale":"2024-09-02T10:36:01.306447+0000","last_undegraded":"2024-09-02T10:36:01.306447+0000","last_fullsized":"2024-09-02T10:36:01.306447+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":7931,"log_dups_size":0,"ondisk_log_size":7931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276627456,"num_objects":4221,"num_object_clones":0,"num_object_copies":12663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7931,"num_write_kb":388864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":1422,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.393009+0000","last_change":"2024-09-02T10:34:05.282359+0000","last_active":"2024-09-02T10:36:01.393009+0000","last_peered":"2024-09-02T10:36:01.393009+0000","last_clean":"2024-09-02T10:36:01.393009+0000","last_became_active":"2024-09-02T10:09:58.908246+0000","last_became_peered":"2024-09-02T10:09:58.908246+0000","last_unstale":"2024-09-02T10:36:01.393009+0000","last_undegraded":"2024-09-02T10:36:01.393009+0000","last_fullsized":"2024-09-02T10:36:01.393009+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:05.282307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:05.282307+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:03.999088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"955'8146","reported_seq":8276,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.743315+0000","last_change":"2024-09-02T10:33:52.925486+0000","last_active":"2024-09-02T10:36:02.743315+0000","last_peered":"2024-09-02T10:36:02.743315+0000","last_clean":"2024-09-02T10:36:02.743315+0000","last_became_active":"2024-09-02T10:33:52.925153+0000","last_became_peered":"2024-09-02T10:33:52.925153+0000","last_unstale":"2024-09-02T10:36:02.743315+0000","last_undegraded":"2024-09-02T10:36:02.743315+0000","last_fullsized":"2024-09-02T10:36:02.743315+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":8146,"log_dups_size":0,"ondisk_log_size":8146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272498688,"num_objects":4158,"num_object_clones":0,"num_object_copies":12474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8146,"num_write_kb":393728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1213,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.396861+0000","last_change":"2024-09-02T10:34:17.882691+0000","last_active":"2024-09-02T10:36:02.396861+0000","last_peered":"2024-09-02T10:36:02.396861+0000","last_clean":"2024-09-02T10:36:02.396861+0000","last_became_active":"2024-09-02T10:34:17.882376+0000","last_became_peered":"2024-09-02T10:34:17.882376+0000","last_unstale":"2024-09-02T10:36:02.396861+0000","last_undegraded":"2024-09-02T10:36:02.396861+0000","last_fullsized":"2024-09-02T10:36:02.396861+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:00.268512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:00.268512+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"954'5868","reported_seq":9468,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.785382+0000","last_change":"2024-09-02T10:34:49.043289+0000","last_active":"2024-09-02T10:36:01.785382+0000","last_peered":"2024-09-02T10:36:01.785382+0000","last_clean":"2024-09-02T10:36:01.785382+0000","last_became_active":"2024-09-02T10:34:47.042191+0000","last_became_peered":"2024-09-02T10:34:47.042191+0000","last_unstale":"2024-09-02T10:36:01.785382+0000","last_undegraded":"2024-09-02T10:36:01.785382+0000","last_fullsized":"2024-09-02T10:36:01.785382+0000","mapping_epoch":918,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'3857","last_scrub_stamp":"2024-09-02T10:34:49.043221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.043221+0000","objects_scrubbed":1904,"log_size":5868,"log_dups_size":0,"ondisk_log_size":5868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134021120,"num_objects":2045,"num_object_clones":0,"num_object_copies":6135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2011,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,3],"acting":[10,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":1437,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.392010+0000","last_change":"2024-09-02T10:34:35.240578+0000","last_active":"2024-09-02T10:36:01.392010+0000","last_peered":"2024-09-02T10:36:01.392010+0000","last_clean":"2024-09-02T10:36:01.392010+0000","last_became_active":"2024-09-02T10:25:24.606055+0000","last_became_peered":"2024-09-02T10:25:24.606055+0000","last_unstale":"2024-09-02T10:36:01.392010+0000","last_undegraded":"2024-09-02T10:36:01.392010+0000","last_fullsized":"2024-09-02T10:36:01.392010+0000","mapping_epoch":708,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":709,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:35.240504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:35.240504+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"955'8240","reported_seq":8370,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.748305+0000","last_change":"2024-09-02T10:33:52.925507+0000","last_active":"2024-09-02T10:36:02.748305+0000","last_peered":"2024-09-02T10:36:02.748305+0000","last_clean":"2024-09-02T10:36:02.748305+0000","last_became_active":"2024-09-02T10:33:52.925177+0000","last_became_peered":"2024-09-02T10:33:52.925177+0000","last_unstale":"2024-09-02T10:36:02.748305+0000","last_undegraded":"2024-09-02T10:36:02.748305+0000","last_fullsized":"2024-09-02T10:36:02.748305+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":8240,"log_dups_size":0,"ondisk_log_size":8240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269746176,"num_objects":4116,"num_object_clones":0,"num_object_copies":12348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8240,"num_write_kb":395392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1682,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.717320+0000","last_change":"2024-09-02T10:34:04.115873+0000","last_active":"2024-09-02T10:35:57.717320+0000","last_peered":"2024-09-02T10:35:57.717320+0000","last_clean":"2024-09-02T10:35:57.717320+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:35:57.717320+0000","last_undegraded":"2024-09-02T10:35:57.717320+0000","last_fullsized":"2024-09-02T10:35:57.717320+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:04.115830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:04.115830+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"953'7857","reported_seq":7981,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:59.698640+0000","last_change":"2024-09-02T10:33:52.750617+0000","last_active":"2024-09-02T10:35:59.698640+0000","last_peered":"2024-09-02T10:35:59.698640+0000","last_clean":"2024-09-02T10:35:59.698640+0000","last_became_active":"2024-09-02T10:33:52.750198+0000","last_became_peered":"2024-09-02T10:33:52.750198+0000","last_unstale":"2024-09-02T10:35:59.698640+0000","last_undegraded":"2024-09-02T10:35:59.698640+0000","last_fullsized":"2024-09-02T10:35:59.698640+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":7857,"log_dups_size":0,"ondisk_log_size":7857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285147136,"num_objects":4351,"num_object_clones":0,"num_object_copies":13053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":979,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.396344+0000","last_change":"2024-09-02T10:33:59.302485+0000","last_active":"2024-09-02T10:36:02.396344+0000","last_peered":"2024-09-02T10:36:02.396344+0000","last_clean":"2024-09-02T10:36:02.396344+0000","last_became_active":"2024-09-02T10:33:38.664311+0000","last_became_peered":"2024-09-02T10:33:38.664311+0000","last_unstale":"2024-09-02T10:36:02.396344+0000","last_undegraded":"2024-09-02T10:36:02.396344+0000","last_fullsized":"2024-09-02T10:36:02.396344+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:59.302421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:59.302421+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:02.898486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"955'6138","reported_seq":6307,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:03.080679+0000","last_change":"2024-09-02T10:34:49.255198+0000","last_active":"2024-09-02T10:36:03.080679+0000","last_peered":"2024-09-02T10:36:03.080679+0000","last_clean":"2024-09-02T10:36:03.080679+0000","last_became_active":"2024-09-02T10:34:47.044129+0000","last_became_peered":"2024-09-02T10:34:47.044129+0000","last_unstale":"2024-09-02T10:36:03.080679+0000","last_undegraded":"2024-09-02T10:36:03.080679+0000","last_fullsized":"2024-09-02T10:36:03.080679+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'3926","last_scrub_stamp":"2024-09-02T10:34:49.255099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.255099+0000","objects_scrubbed":1999,"log_size":6138,"log_dups_size":0,"ondisk_log_size":6138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:12.758346+0000","scrub_duration":734,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137035776,"num_objects":2091,"num_object_clones":0,"num_object_copies":6273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2212,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1693,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.717374+0000","last_change":"2024-09-02T10:34:01.291835+0000","last_active":"2024-09-02T10:35:57.717374+0000","last_peered":"2024-09-02T10:35:57.717374+0000","last_clean":"2024-09-02T10:35:57.717374+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:35:57.717374+0000","last_undegraded":"2024-09-02T10:35:57.717374+0000","last_fullsized":"2024-09-02T10:35:57.717374+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.291779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.291779+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"954'5777","reported_seq":5941,"reported_epoch":954,"state":"active+clean","last_fresh":"2024-09-02T10:36:01.747136+0000","last_change":"2024-09-02T10:34:49.259122+0000","last_active":"2024-09-02T10:36:01.747136+0000","last_peered":"2024-09-02T10:36:01.747136+0000","last_clean":"2024-09-02T10:36:01.747136+0000","last_became_active":"2024-09-02T10:34:47.039696+0000","last_became_peered":"2024-09-02T10:34:47.039696+0000","last_unstale":"2024-09-02T10:36:01.747136+0000","last_undegraded":"2024-09-02T10:36:01.747136+0000","last_fullsized":"2024-09-02T10:36:01.747136+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3753","last_scrub_stamp":"2024-09-02T10:34:49.259057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.259057+0000","objects_scrubbed":1910,"log_size":5777,"log_dups_size":0,"ondisk_log_size":5777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:15.936951+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138412032,"num_objects":2112,"num_object_clones":0,"num_object_copies":6336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2024,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1952,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.396966+0000","last_change":"2024-09-02T10:33:55.238766+0000","last_active":"2024-09-02T10:36:02.396966+0000","last_peered":"2024-09-02T10:36:02.396966+0000","last_clean":"2024-09-02T10:36:02.396966+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:36:02.396966+0000","last_undegraded":"2024-09-02T10:36:02.396966+0000","last_fullsized":"2024-09-02T10:36:02.396966+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.238683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.238683+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:03.898238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"955'5933","reported_seq":6101,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.721940+0000","last_change":"2024-09-02T10:34:50.216635+0000","last_active":"2024-09-02T10:36:02.721940+0000","last_peered":"2024-09-02T10:36:02.721940+0000","last_clean":"2024-09-02T10:36:02.721940+0000","last_became_active":"2024-09-02T10:34:47.033777+0000","last_became_peered":"2024-09-02T10:34:47.033777+0000","last_unstale":"2024-09-02T10:36:02.721940+0000","last_undegraded":"2024-09-02T10:36:02.721940+0000","last_fullsized":"2024-09-02T10:36:02.721940+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'3847","last_scrub_stamp":"2024-09-02T10:34:50.216498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:50.216498+0000","objects_scrubbed":1961,"log_size":5933,"log_dups_size":0,"ondisk_log_size":5933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:11.868424+0000","scrub_duration":552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133038080,"num_objects":2030,"num_object_clones":0,"num_object_copies":6090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2093,"num_write_kb":69184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":897,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.700385+0000","last_change":"2024-09-02T10:34:02.569694+0000","last_active":"2024-09-02T10:35:57.700385+0000","last_peered":"2024-09-02T10:35:57.700385+0000","last_clean":"2024-09-02T10:35:57.700385+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:35:57.700385+0000","last_undegraded":"2024-09-02T10:35:57.700385+0000","last_fullsized":"2024-09-02T10:35:57.700385+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:02.569628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:02.569628+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"955'5951","reported_seq":6116,"reported_epoch":955,"state":"active+clean","last_fresh":"2024-09-02T10:36:02.733721+0000","last_change":"2024-09-02T10:34:49.168775+0000","last_active":"2024-09-02T10:36:02.733721+0000","last_peered":"2024-09-02T10:36:02.733721+0000","last_clean":"2024-09-02T10:36:02.733721+0000","last_became_active":"2024-09-02T10:34:47.049681+0000","last_became_peered":"2024-09-02T10:34:47.049681+0000","last_unstale":"2024-09-02T10:36:02.733721+0000","last_undegraded":"2024-09-02T10:36:02.733721+0000","last_fullsized":"2024-09-02T10:36:02.733721+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3787","last_scrub_stamp":"2024-09-02T10:34:49.168686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.168686+0000","objects_scrubbed":1913,"log_size":5951,"log_dups_size":0,"ondisk_log_size":5951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:16.457037+0000","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139526144,"num_objects":2129,"num_object_clones":0,"num_object_copies":6387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2164,"num_write_kb":76160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1712,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.717635+0000","last_change":"2024-09-02T10:30:46.402988+0000","last_active":"2024-09-02T10:35:57.717635+0000","last_peered":"2024-09-02T10:35:57.717635+0000","last_clean":"2024-09-02T10:35:57.717635+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:35:57.717635+0000","last_undegraded":"2024-09-02T10:35:57.717635+0000","last_fullsized":"2024-09-02T10:35:57.717635+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:30:46.402901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:30:46.402901+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"953'5912","reported_seq":6074,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:59.700467+0000","last_change":"2024-09-02T10:34:48.713406+0000","last_active":"2024-09-02T10:35:59.700467+0000","last_peered":"2024-09-02T10:35:59.700467+0000","last_clean":"2024-09-02T10:35:59.700467+0000","last_became_active":"2024-09-02T10:34:47.029634+0000","last_became_peered":"2024-09-02T10:34:47.029634+0000","last_unstale":"2024-09-02T10:35:59.700467+0000","last_undegraded":"2024-09-02T10:35:59.700467+0000","last_fullsized":"2024-09-02T10:35:59.700467+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3863","last_scrub_stamp":"2024-09-02T10:34:48.713331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:48.713331+0000","objects_scrubbed":1931,"log_size":5912,"log_dups_size":0,"ondisk_log_size":5912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:15.136168+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142737408,"num_objects":2178,"num_object_clones":0,"num_object_copies":6534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2049,"num_write_kb":73472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":1112,"reported_epoch":953,"state":"active+clean","last_fresh":"2024-09-02T10:35:57.714464+0000","last_change":"2024-09-02T10:33:55.336238+0000","last_active":"2024-09-02T10:35:57.714464+0000","last_peered":"2024-09-02T10:35:57.714464+0000","last_clean":"2024-09-02T10:35:57.714464+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:35:57.714464+0000","last_undegraded":"2024-09-02T10:35:57.714464+0000","last_fullsized":"2024-09-02T10:35:57.714464+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.336182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.336182+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:01.136936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":4422238232,"num_objects":67480,"num_object_clones":0,"num_object_copies":202440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1082,"num_objects_unfound":0,"num_objects_dirty":67480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":95439,"num_write_kb":4149889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9485,"num_bytes_recovered":621543424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14219882496,"data_stored":14219870280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166670,"ondisk_log_size":166670,"up":78,"acting":77,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":343597384111,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532613044,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1883844,"kb_used_data":1220980,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":91861308,"statfs":{"total":95995035648,"available":94065979392,"internally_reserved":0,"allocated":1250283520,"data_stored":1249989109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":678754752},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841977,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2105636,"kb_used_data":1185932,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":91639516,"statfs":{"total":95995035648,"available":93838864384,"internally_reserved":0,"allocated":1214394368,"data_stored":1214091101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":941750720},"hb_peers":[1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070912,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1567876,"kb_used_data":1035508,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92177276,"statfs":{"total":95995035648,"available":94389530624,"internally_reserved":0,"allocated":1060360192,"data_stored":1060065781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299844,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2690960,"kb_used_data":1875392,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":91054192,"statfs":{"total":95995035648,"available":93239492608,"internally_reserved":0,"allocated":1920401408,"data_stored":1920099373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528776,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1501396,"kb_used_data":932928,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92243756,"statfs":{"total":95995035648,"available":94457606144,"internally_reserved":0,"allocated":955318272,"data_stored":955019408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":582089159},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503725005,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3213160,"kb_used_data":2261528,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":90531992,"statfs":{"total":95995035648,"available":92704759808,"internally_reserved":0,"allocated":2315804672,"data_stored":2315493781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953937,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1336648,"kb_used_data":737652,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92408504,"statfs":{"total":95995035648,"available":94626308096,"internally_reserved":0,"allocated":755355648,"data_stored":755061237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150167,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2310156,"kb_used_data":1386292,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":91434996,"statfs":{"total":95995035648,"available":93629435904,"internally_reserved":0,"allocated":1419563008,"data_stored":1419268597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":946010568},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411803,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":749020,"kb_used_data":136136,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92996132,"statfs":{"total":95995035648,"available":95228039168,"internally_reserved":0,"allocated":139403264,"data_stored":139100848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575327,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3459532,"kb_used_data":2575928,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":90285620,"statfs":{"total":95995035648,"available":92452474880,"internally_reserved":0,"allocated":2637750272,"data_stored":2637451789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":904788421},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738854,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1478356,"kb_used_data":530880,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92266796,"statfs":{"total":95995035648,"available":94481199104,"internally_reserved":0,"allocated":543621120,"data_stored":543322256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":970193349},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":537919488,"data_stored":537919488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2615480320,"data_stored":2615476248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":137166848,"data_stored":137166848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1416822784,"data_stored":1416822784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":752680960,"data_stored":752680960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2307395584,"data_stored":2307391512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":946012160,"data_stored":946012160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1898582016,"data_stored":1898577944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":1053491200,"data_stored":1053491200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":1210187776,"data_stored":1210187776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":1232011264,"data_stored":1232011264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":112132096,"data_stored":112132096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:36:06.976 INFO:tasks.thrashosds.thrasher:Setting 1.13 to [4, 1, 2] 2024-09-02T10:36:06.976 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.13', '4', '1', '2'] 2024-09-02T10:36:06.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap 1.13 4 1 2 2024-09-02T10:36:06.996 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:06.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:06 smithi187 ceph-mon[20072]: pgmap v2188: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 4.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 279/197154 objects misplaced (0.142%); 6.2 MiB/s, 99 objects/s recovering 2024-09-02T10:36:07.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1597491211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:06 smithi162 ceph-mon[20279]: pgmap v2188: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 4.0 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.04k op/s; 279/197154 objects misplaced (0.142%); 6.2 MiB/s, 99 objects/s recovering 2024-09-02T10:36:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1597491211' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:07.098 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:07.128 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:07.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:07.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:07.230 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:07.249 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:07.253 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:07.254 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:07.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:07.382 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:07.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:07.484 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:07.509 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:07.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:07.510 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:07.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:07.635 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:07.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:07.737 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:07.769 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:07.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:07.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:07.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:07.897 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:07.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:07 smithi151 ceph-mon[16639]: 9.16 scrub starts 2024-09-02T10:36:07.999 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:08.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:07 smithi187 ceph-mon[20072]: 9.16 scrub starts 2024-09-02T10:36:08.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:08.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:08.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:07 smithi162 ceph-mon[20279]: 9.16 scrub starts 2024-09-02T10:36:08.125 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:08.147 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:08.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:08.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:08.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:08.270 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:08.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:08.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:08.394 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:08.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:08.496 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:08.522 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:08.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:08.624 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:08.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:08.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:08.725 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:36:08.748 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:08.771 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:08.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:08.772 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:08.873 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:08.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:08.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:09.003 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:08 smithi187 ceph-mon[20072]: 9.13 scrub starts 2024-09-02T10:36:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:08 smithi187 ceph-mon[20072]: pgmap v2189: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 3.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 279/190638 objects misplaced (0.146%); 6.2 MiB/s, 98 objects/s recovering 2024-09-02T10:36:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:08 smithi187 ceph-mon[20072]: osdmap e956: 12 total, 12 up, 11 in 2024-09-02T10:36:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:08 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:09.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:08 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:09.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:08 smithi187 ceph-mon[20072]: 9.17 scrub starts 2024-09-02T10:36:09.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2442260789' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:36:09.032 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:09.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:08 smithi162 ceph-mon[20279]: 9.13 scrub starts 2024-09-02T10:36:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:08 smithi162 ceph-mon[20279]: pgmap v2189: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 3.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 279/190638 objects misplaced (0.146%); 6.2 MiB/s, 98 objects/s recovering 2024-09-02T10:36:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:08 smithi162 ceph-mon[20279]: osdmap e956: 12 total, 12 up, 11 in 2024-09-02T10:36:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:08 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:08 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:08 smithi162 ceph-mon[20279]: 9.17 scrub starts 2024-09-02T10:36:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2442260789' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:36:09.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:09.162 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:09.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:08 smithi151 ceph-mon[16639]: 9.13 scrub starts 2024-09-02T10:36:09.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:08 smithi151 ceph-mon[16639]: pgmap v2189: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 3.9 GiB data, 21 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.03k op/s; 279/190638 objects misplaced (0.146%); 6.2 MiB/s, 98 objects/s recovering 2024-09-02T10:36:09.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:08 smithi151 ceph-mon[16639]: osdmap e956: 12 total, 12 up, 11 in 2024-09-02T10:36:09.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:08 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:09.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:08 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:09.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:08 smithi151 ceph-mon[16639]: 9.17 scrub starts 2024-09-02T10:36:09.195 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2442260789' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:36:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:09.196 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:09.297 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:09.303 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:36:09.303 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:36:09.322 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:09.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:09.424 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:09.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:09.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:09.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:09.580 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:09.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:09.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:09.682 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:09.714 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:09.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:09.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:09.817 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:09.838 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:09.838 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:09.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:09.978 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:09.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:09 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2442260789' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:36:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:09 smithi187 ceph-mon[20072]: osdmap e957: 12 total, 12 up, 11 in 2024-09-02T10:36:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:09 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:36:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:09 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:36:10.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:09 smithi187 ceph-mon[20072]: 9.10 scrub starts 2024-09-02T10:36:10.080 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:09 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2442260789' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:36:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:09 smithi162 ceph-mon[20279]: osdmap e957: 12 total, 12 up, 11 in 2024-09-02T10:36:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:09 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:36:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:09 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:36:10.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:09 smithi162 ceph-mon[20279]: 9.10 scrub starts 2024-09-02T10:36:10.113 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:10.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:10.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:09 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2442260789' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:36:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:09 smithi151 ceph-mon[16639]: osdmap e957: 12 total, 12 up, 11 in 2024-09-02T10:36:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:09 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:36:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:09 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:36:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:09 smithi151 ceph-mon[16639]: 9.10 scrub starts 2024-09-02T10:36:10.215 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:10.238 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:10.238 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:10.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:10.352 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-02T10:36:10.373 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:10.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:10.395 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:10.475 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:10.502 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:10.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:10.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:10.604 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:10.628 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:10.629 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:10.731 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:10.755 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:10.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:10.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:10.857 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:10.880 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:10.881 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:10.912 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:36:10.982 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: 9.14 scrub starts 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: 9.14 scrub ok 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: 9.a scrub starts 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: pgmap v2192: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 3.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s; 279/182826 objects misplaced (0.153%); 2.3 MiB/s, 37 objects/s recovering 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: 9.10 scrub ok 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:36:10.996 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:10 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:36:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: 9.14 scrub starts 2024-09-02T10:36:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: 9.14 scrub ok 2024-09-02T10:36:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: 9.a scrub starts 2024-09-02T10:36:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: pgmap v2192: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 3.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s; 279/182826 objects misplaced (0.153%); 2.3 MiB/s, 37 objects/s recovering 2024-09-02T10:36:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:36:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:36:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: 9.10 scrub ok 2024-09-02T10:36:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:36:11.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:36:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:36:11.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:10 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:36:11.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:11.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: 9.14 scrub starts 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: 9.14 scrub ok 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: 9.a scrub starts 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: pgmap v2192: 57 pgs: 1 active+recovering+undersized+remapped, 56 active+clean; 3.7 GiB data, 20 GiB used, 963 GiB / 983 GiB avail; 0 B/s wr, 1.19k op/s; 279/182826 objects misplaced (0.153%); 2.3 MiB/s, 37 objects/s recovering 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: 9.10 scrub ok 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:36:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:10 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:36:11.111 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:11.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:11.147 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:11.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:11.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:11.277 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:11.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:11.379 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:11.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:11.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:11.468 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:11.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:11.519 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:11.534 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:11.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:11.635 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:11.664 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:11.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:11.750 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.13 pg_upmap mapping to [4,1,2] 2024-09-02T10:36:11.766 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:11.793 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:11.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:11.895 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:11.920 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:11.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:11.921 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:11.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:11 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:36:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:11 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:36:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:11 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:36:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:11 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:36:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:11 smithi151 ceph-mon[16639]: 9.a scrub ok 2024-09-02T10:36:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:11 smithi151 ceph-mon[16639]: 9.16 scrub starts 2024-09-02T10:36:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:11 smithi151 ceph-mon[16639]: 9.16 scrub ok 2024-09-02T10:36:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:11 smithi151 ceph-mon[16639]: 9.d scrub starts 2024-09-02T10:36:11.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2616362132' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.13", "id": [4, 1, 2]} : dispatch 2024-09-02T10:36:12.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:11 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:36:12.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:11 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:36:12.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:11 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:36:12.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:11 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:36:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:11 smithi187 ceph-mon[20072]: 9.a scrub ok 2024-09-02T10:36:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:11 smithi187 ceph-mon[20072]: 9.16 scrub starts 2024-09-02T10:36:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:11 smithi187 ceph-mon[20072]: 9.16 scrub ok 2024-09-02T10:36:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:11 smithi187 ceph-mon[20072]: 9.d scrub starts 2024-09-02T10:36:12.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2616362132' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.13", "id": [4, 1, 2]} : dispatch 2024-09-02T10:36:12.022 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:12.054 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:12.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:12.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:11 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:36:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:11 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:36:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:11 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:36:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:11 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:36:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:11 smithi162 ceph-mon[20279]: 9.a scrub ok 2024-09-02T10:36:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:11 smithi162 ceph-mon[20279]: 9.16 scrub starts 2024-09-02T10:36:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:11 smithi162 ceph-mon[20279]: 9.16 scrub ok 2024-09-02T10:36:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:11 smithi162 ceph-mon[20279]: 9.d scrub starts 2024-09-02T10:36:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2616362132' entity='client.admin' cmd={"prefix": "osd pg-upmap", "pgid": "1.13", "id": [4, 1, 2]} : dispatch 2024-09-02T10:36:12.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:12.190 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:12.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:12.263 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:36:12.263 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:36:12.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:12.313 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:12.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:12.314 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:12.415 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:12.446 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:12.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:12.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:12.548 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:12.571 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:12.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:12.572 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:12.673 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:12.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:12.696 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:12.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:12.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:12.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:12.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:36:12.928 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:12.960 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:12.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:12.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:13.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:13.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:12 smithi151 ceph-mon[16639]: pgmap v2193: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 2.8 MiB/s, 45 objects/s recovering 2024-09-02T10:36:13.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:12 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-02T10:36:13.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2616362132' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [4, 1, 2]}]': finished 2024-09-02T10:36:13.065 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:12 smithi151 ceph-mon[16639]: osdmap e958: 12 total, 12 up, 11 in 2024-09-02T10:36:13.066 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:12 smithi151 ceph-mon[16639]: 9.c scrub starts 2024-09-02T10:36:13.092 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:13.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:12 smithi162 ceph-mon[20279]: pgmap v2193: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 2.8 MiB/s, 45 objects/s recovering 2024-09-02T10:36:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:12 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-02T10:36:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2616362132' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [4, 1, 2]}]': finished 2024-09-02T10:36:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:12 smithi162 ceph-mon[20279]: osdmap e958: 12 total, 12 up, 11 in 2024-09-02T10:36:13.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:12 smithi162 ceph-mon[20279]: 9.c scrub starts 2024-09-02T10:36:13.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:13.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:13.223 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:13.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:12 smithi187 ceph-mon[20072]: pgmap v2193: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.6 GiB data, 20 GiB used, 964 GiB / 983 GiB avail; 0 B/s wr, 1.02k op/s; 2.8 MiB/s, 45 objects/s recovering 2024-09-02T10:36:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:12 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-09-02T10:36:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2616362132' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.13", "id": [4, 1, 2]}]': finished 2024-09-02T10:36:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:12 smithi187 ceph-mon[20072]: osdmap e958: 12 total, 12 up, 11 in 2024-09-02T10:36:13.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:12 smithi187 ceph-mon[20072]: 9.c scrub starts 2024-09-02T10:36:13.324 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:13.351 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:13.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:13.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:13.476 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:13.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:13.477 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:13.578 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:13.601 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:13.602 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:13.704 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:13.726 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:13.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:13.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:13.828 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:13.856 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:13.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:13.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:13 smithi151 ceph-mon[16639]: 9.13 scrub starts 2024-09-02T10:36:13.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:13 smithi151 ceph-mon[16639]: 9.13 scrub ok 2024-09-02T10:36:13.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:13 smithi151 ceph-mon[16639]: 9.d scrub ok 2024-09-02T10:36:13.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:13 smithi151 ceph-mon[16639]: osdmap e959: 12 total, 12 up, 11 in 2024-09-02T10:36:13.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:13 smithi151 ceph-mon[16639]: 9.c scrub ok 2024-09-02T10:36:13.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:13 smithi151 ceph-mon[16639]: 9.2 scrub starts 2024-09-02T10:36:13.957 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:13.983 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:13.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:13.984 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:14.034 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-02T10:36:14.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:14.094 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:13 smithi162 ceph-mon[20279]: 9.13 scrub starts 2024-09-02T10:36:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:13 smithi162 ceph-mon[20279]: 9.13 scrub ok 2024-09-02T10:36:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:13 smithi162 ceph-mon[20279]: 9.d scrub ok 2024-09-02T10:36:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:13 smithi162 ceph-mon[20279]: osdmap e959: 12 total, 12 up, 11 in 2024-09-02T10:36:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:13 smithi162 ceph-mon[20279]: 9.c scrub ok 2024-09-02T10:36:14.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:13 smithi162 ceph-mon[20279]: 9.2 scrub starts 2024-09-02T10:36:14.110 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:14.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:14.111 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:14.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:14.236 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:14.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:13 smithi187 ceph-mon[20072]: 9.13 scrub starts 2024-09-02T10:36:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:13 smithi187 ceph-mon[20072]: 9.13 scrub ok 2024-09-02T10:36:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:13 smithi187 ceph-mon[20072]: 9.d scrub ok 2024-09-02T10:36:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:13 smithi187 ceph-mon[20072]: osdmap e959: 12 total, 12 up, 11 in 2024-09-02T10:36:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:13 smithi187 ceph-mon[20072]: 9.c scrub ok 2024-09-02T10:36:14.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:13 smithi187 ceph-mon[20072]: 9.2 scrub starts 2024-09-02T10:36:14.338 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:14.378 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:14.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:14.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:14.480 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:14.511 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:14.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:14.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:14.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:14.640 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:14.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:14.685 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:14.704 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:14.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:14.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:14.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:14.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:14.899 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:14.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:14 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:36:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:14 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:36:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:14 smithi151 ceph-mon[16639]: pgmap v2196: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail 2024-09-02T10:36:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:14 smithi151 ceph-mon[16639]: 9.2 scrub ok 2024-09-02T10:36:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:14 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:36:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:14 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:36:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:14 smithi151 ceph-mon[16639]: 9.17 scrub starts 2024-09-02T10:36:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:14 smithi151 ceph-mon[16639]: 9.17 scrub ok 2024-09-02T10:36:14.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3057524100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:15.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:15.028 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:15.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:14 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:36:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:14 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:36:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:14 smithi162 ceph-mon[20279]: pgmap v2196: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail 2024-09-02T10:36:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:14 smithi162 ceph-mon[20279]: 9.2 scrub ok 2024-09-02T10:36:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:14 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:36:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:14 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:36:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:14 smithi162 ceph-mon[20279]: 9.17 scrub starts 2024-09-02T10:36:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:14 smithi162 ceph-mon[20279]: 9.17 scrub ok 2024-09-02T10:36:15.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3057524100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:15.130 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:15.159 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:15.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:15.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:14 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:36:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:14 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:36:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:14 smithi187 ceph-mon[20072]: pgmap v2196: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.5 GiB data, 19 GiB used, 964 GiB / 983 GiB avail 2024-09-02T10:36:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:14 smithi187 ceph-mon[20072]: 9.2 scrub ok 2024-09-02T10:36:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:14 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:36:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:14 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:36:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:14 smithi187 ceph-mon[20072]: 9.17 scrub starts 2024-09-02T10:36:15.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:14 smithi187 ceph-mon[20072]: 9.17 scrub ok 2024-09-02T10:36:15.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3057524100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:15.259 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:36:15.259 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:36:15.261 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:15.285 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:15.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:15.287 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:15.387 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:15.416 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:15.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:15.417 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:15.497 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:36:15.497 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:36:15.517 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:15.530 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:36:15.530 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:36:15.543 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:15.544 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:15.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:36:15.558 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:36:15.592 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:36:15.592 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:36:15.622 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:36:15.622 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:36:15.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:15.649 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:36:15.650 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:36:15.673 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:15.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:15.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:15.680 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:36:15.680 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:36:15.708 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:36:15.708 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:36:15.738 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:36:15.738 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:36:15.762 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:36:15.762 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:36:15.776 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:15.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:15 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:36:15.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:15 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:36:15.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:15 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:36:15.784 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:15 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:36:15.791 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:36:15.791 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:36:15.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:15.802 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:15.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:15.817 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:36:15.818 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:36:15.903 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:15.930 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:15.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:15.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:16.032 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:16.057 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:16.058 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:15 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:36:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:15 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:36:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:15 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:36:16.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:15 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:36:16.159 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:16.189 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:16.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:16.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:15 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:36:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:15 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:36:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:15 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:36:16.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:15 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:36:16.291 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:16.318 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:16.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:16.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:16.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:16.447 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:16.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:16.549 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:16.572 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:16.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:16.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:16.674 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:16.680 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-02T10:36:16.697 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:16.698 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:16.786 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:36:16.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:16.823 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:16.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:16.925 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:36:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:36:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:36:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:36:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: pgmap v2197: 57 pgs: 57 active+clean; 3.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail 2024-09-02T10:36:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:36:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:36:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 9.3 scrub starts 2024-09-02T10:36:16.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1068904855' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:16.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:16.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 9.3 scrub ok 2024-09-02T10:36:16.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:36:16.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:36:16.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:36:16.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:16 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:36:16.953 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:16.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:17.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:17.080 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:17.081 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:36:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:36:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:36:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:36:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: pgmap v2197: 57 pgs: 57 active+clean; 3.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail 2024-09-02T10:36:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:36:17.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:36:17.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 9.3 scrub starts 2024-09-02T10:36:17.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1068904855' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:17.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:17.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 9.3 scrub ok 2024-09-02T10:36:17.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:36:17.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:36:17.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:36:17.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:16 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:36:17.182 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:17.211 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:17.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:36:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:36:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: pgmap v2197: 57 pgs: 57 active+clean; 3.4 GiB data, 19 GiB used, 965 GiB / 983 GiB avail 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 9.3 scrub starts 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1068904855' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 9.3 scrub ok 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:36:17.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:16 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:36:17.314 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:17.344 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:17.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:17.444 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:17.471 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:17.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:17.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:17.573 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:17.602 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:17.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:17.704 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:17.728 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:17.730 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:17.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:17.855 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:17.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:17.957 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:17.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:17.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:18.085 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:17 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:36:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:17 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:36:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:17 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:17 smithi162 ceph-mon[20279]: osdmap e960: 12 total, 12 up, 11 in 2024-09-02T10:36:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:17 smithi162 ceph-mon[20279]: 9.1 scrub starts 2024-09-02T10:36:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:17 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:36:18.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:17 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:36:18.110 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:18.112 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:17 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:36:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:17 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:36:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:17 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:17 smithi151 ceph-mon[16639]: osdmap e960: 12 total, 12 up, 11 in 2024-09-02T10:36:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:17 smithi151 ceph-mon[16639]: 9.1 scrub starts 2024-09-02T10:36:18.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:17 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:36:18.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:17 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:36:18.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:18.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:18.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:17 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:36:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:17 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:36:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:17 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:17 smithi187 ceph-mon[20072]: osdmap e960: 12 total, 12 up, 11 in 2024-09-02T10:36:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:17 smithi187 ceph-mon[20072]: 9.1 scrub starts 2024-09-02T10:36:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:17 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:36:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:17 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:36:18.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:18.365 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:18.366 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:18.449 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 10, 4, 2] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:36:18.449 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:36:18.449 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 3, 5, 6, 7, 8, 9, 10, 4, 2] 2024-09-02T10:36:18.449 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd out 10 2024-09-02T10:36:18.467 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:18.504 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:18.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:18.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:18.605 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:18.637 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:18.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:18.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:18.739 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:18.766 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:18.767 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:18.868 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:18.894 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:18.895 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:18.996 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:19.027 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:19.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:18 smithi162 ceph-mon[20279]: pgmap v2199: 57 pgs: 57 active+clean; 3.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail 2024-09-02T10:36:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:18 smithi162 ceph-mon[20279]: 9.1 scrub ok 2024-09-02T10:36:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:18 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:36:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:18 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:36:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:18 smithi162 ceph-mon[20279]: 9.8 scrub starts 2024-09-02T10:36:19.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:18 smithi162 ceph-mon[20279]: 9.8 scrub ok 2024-09-02T10:36:19.129 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:19.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:19.151 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:18 smithi151 ceph-mon[16639]: pgmap v2199: 57 pgs: 57 active+clean; 3.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail 2024-09-02T10:36:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:18 smithi151 ceph-mon[16639]: 9.1 scrub ok 2024-09-02T10:36:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:18 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:36:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:18 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:36:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:18 smithi151 ceph-mon[16639]: 9.8 scrub starts 2024-09-02T10:36:19.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:18 smithi151 ceph-mon[16639]: 9.8 scrub ok 2024-09-02T10:36:19.252 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:18 smithi187 ceph-mon[20072]: pgmap v2199: 57 pgs: 57 active+clean; 3.2 GiB data, 18 GiB used, 965 GiB / 983 GiB avail 2024-09-02T10:36:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:18 smithi187 ceph-mon[20072]: 9.1 scrub ok 2024-09-02T10:36:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:18 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:36:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:18 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:36:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:18 smithi187 ceph-mon[20072]: 9.8 scrub starts 2024-09-02T10:36:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:18 smithi187 ceph-mon[20072]: 9.8 scrub ok 2024-09-02T10:36:19.278 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:19.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:19.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:19.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:19.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:19.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:19.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:19.445 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:36:19.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:19.531 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:19.533 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:19.633 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:19.659 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:19.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:19.760 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:19.787 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:19.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:19.889 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:19.915 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:19.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:19.916 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:20.017 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:20.039 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:20.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:20.040 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:20.041 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:20.140 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:20.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:20.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:20.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:20.272 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:20.309 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:20.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:20.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:20.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:20.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:20.533 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:20 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:36:20.533 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:20 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:36:20.533 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:20 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:36:20.533 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:20 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:36:20.533 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:20 smithi151 ceph-mon[16639]: pgmap v2200: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail 2024-09-02T10:36:20.541 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:20.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:20.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:20.663 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:20.664 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:20.681 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:20.694 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:36:20.694 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:36:20.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:36:20.694 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:36:20.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.694 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:134987 8.7 8.f7ae077 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:01.902047+0000", 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 318.78753192200003, 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91672499900000004, 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 134987 2024-09-02T10:36:20.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.696 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.902047+0000", 2024-09-02T10:36:20.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.902047+0000", 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.902049+0000", 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.579e-06 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.902055+0000", 2024-09-02T10:36:20.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3400000000000003e-06 2024-09-02T10:36:20.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.698 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.698 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.698 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.902072+0000", 2024-09-02T10:36:20.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6475999999999999e-05 2024-09-02T10:36:20.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.902078+0000", 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.883e-06 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.902138+0000", 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9766999999999999e-05 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:36:20.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:01.902146+0000", 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3529999999999997e-06 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.818644+0000", 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.91649861200000005 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.818772+0000", 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012798900000000001 2024-09-02T10:36:20.700 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:134992 8.3 8.7c2b2683 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:02.005792+0000", 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 318.68378728499999, 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.81356366300000005, 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.701 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 134992 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.005792+0000", 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.005792+0000", 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.005793+0000", 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3880000000000001e-06 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.005801+0000", 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0099999999999995e-06 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.005817+0000", 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5704000000000001e-05 2024-09-02T10:36:20.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.704 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.005825+0000", 2024-09-02T10:36:20.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6690000000000006e-06 2024-09-02T10:36:20.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.005880+0000", 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5479999999999997e-05 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.005889+0000", 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.5450000000000003e-06 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.819329+0000", 2024-09-02T10:36:20.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.81344062800000005 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.819355+0000", 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.6239000000000001e-05 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:134993 8.3 8.1faecc33 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:02.072071+0000", 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 318.61750859300003, 2024-09-02T10:36:20.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.74732150399999997, 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 134993 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.072071+0000", 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.072071+0000", 2024-09-02T10:36:20.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.708 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.072072+0000", 2024-09-02T10:36:20.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.61e-06 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.072081+0000", 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3639999999999999e-06 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.072097+0000", 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6711e-05 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.072106+0000", 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7460000000000006e-06 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.072176+0000", 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9759999999999996e-05 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.072188+0000", 2024-09-02T10:36:20.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2319e-05 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.819385+0000", 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.74719712999999999 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.819392+0000", 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8639999999999998e-06 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.711 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.712 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.712 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:134997 8.3 8.b5778683 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:36:20.712 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:02.113062+0000", 2024-09-02T10:36:20.712 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 318.57651700500003, 2024-09-02T10:36:20.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70637061000000001, 2024-09-02T10:36:20.712 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.712 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.712 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.713 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.713 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.713 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.713 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 134997 2024-09-02T10:36:20.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.713 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.113062+0000", 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.113062+0000", 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.113065+0000", 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5170000000000002e-06 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.113076+0000", 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1049e-05 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.113105+0000", 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9754e-05 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.113119+0000", 2024-09-02T10:36:20.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.402e-05 2024-09-02T10:36:20.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.113267+0000", 2024-09-02T10:36:20.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014711100000000001 2024-09-02T10:36:20.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.113283+0000", 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.6667000000000002e-05 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.819414+0000", 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.70613097199999997 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.819433+0000", 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8519999999999999e-05 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:135015 8.7 8.5bc05427 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:02.497716+0000", 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 318.19186281600003, 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.321208677, 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.718 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135015 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.497716+0000", 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.497716+0000", 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.497718+0000", 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.502e-06 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.497724+0000", 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.207e-06 2024-09-02T10:36:20.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.497739+0000", 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4562000000000001e-05 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.497746+0000", 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4939999999999997e-06 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.497812+0000", 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6060000000000001e-05 2024-09-02T10:36:20.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.497825+0000", 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2716e-05 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.818917+0000", 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32109199500000002 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.818925+0000", 2024-09-02T10:36:20.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1410000000000005e-06 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:135016 8.7 8.928e9797 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:02.499797+0000", 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 318.18978179599998, 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31915444500000001, 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.723 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 135016 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.499797+0000", 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.499797+0000", 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.499798+0000", 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3799999999999997e-07 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.499801+0000", 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1149999999999998e-06 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.499814+0000", 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.253e-05 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.499819+0000", 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.2870000000000004e-06 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.499861+0000", 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2150999999999997e-05 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.499868+0000", 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.922e-06 2024-09-02T10:36:20.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.818946+0000", 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.319078323 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:02.818952+0000", 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4790000000000004e-06 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:134992 8.3 8:c164d43e:::benchmark_data_smithi187_136059_object29711:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e845)", 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:03.854035+0000", 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 316.83554374599998, 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34168040100000002, 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 134992 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854035+0000", 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854035+0000", 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854037+0000", 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.804e-06 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854045+0000", 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.6380000000000004e-06 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854059+0000", 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4351e-05 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854071+0000", 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1719000000000001e-05 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854960+0000", 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00088894799999999997 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854980+0000", 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0245999999999999e-05 2024-09-02T10:36:20.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.179668+0000", 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32468806500000003 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.182768+0000", 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0031000569999999998 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:36:20.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.182873+0000", 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00010526699999999999 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.192840+0000", 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0099666579999999998 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.192859+0000", 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9074000000000001e-05 2024-09-02T10:36:20.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.192983+0000", 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012363399999999999 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.193056+0000", 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2756999999999995e-05 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.195716+0000", 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002660183 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:134993 8.3 8:cc3375f8:::benchmark_data_smithi187_136059_object29712:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e845)", 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:03.854092+0000", 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 316.835486624, 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36511791300000002, 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:20.736 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 134993 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854092+0000", 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854092+0000", 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854093+0000", 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4999999999999998e-07 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854095+0000", 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.691e-06 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854098+0000", 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.8279999999999996e-06 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854102+0000", 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.3830000000000001e-06 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.855020+0000", 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00091828099999999996 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:36:20.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.855030+0000", 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8069999999999994e-06 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.188768+0000", 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33373845400000002 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.188944+0000", 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000175632 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.189109+0000", 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000165064 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.199758+0000", 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.010649466999999999 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.199897+0000", 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013837699999999999 2024-09-02T10:36:20.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.216566+0000", 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016669395 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.216628+0000", 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.1926000000000003e-05 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.219210+0000", 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.002582258 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:134997 8.3 8:c161eead:::benchmark_data_smithi187_136059_object29716:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e845)", 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:03.854106+0000", 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 316.83547299200001, 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.36769817300000002, 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.743 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 134997 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854106+0000", 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854106+0000", 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854106+0000", 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.35e-07 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854108+0000", 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3820000000000001e-06 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854111+0000", 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1580000000000001e-06 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.854114+0000", 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9629999999999998e-06 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.855059+0000", 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00094503399999999998 2024-09-02T10:36:20.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for peered", 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:03.855066+0000", 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3590000000000002e-06 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.195768+0000", 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34070223999999999 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.195932+0000", 2024-09-02T10:36:20.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016331599999999999 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 5,6", 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.196091+0000", 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000159476 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.207978+0000", 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011886659000000001 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.748 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.212129+0000", 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004150862 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.219270+0000", 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0071407490000000001 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.219310+0000", 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0297999999999999e-05 2024-09-02T10:36:20.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:04.221804+0000", 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0024944419999999999 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:146120 8.7 8:ef3a04bb:::benchmark_data_smithi187_136059_object40839:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:21.736155+0000", 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 298.95342375899997, 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.363407125, 2024-09-02T10:36:20.750 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 146120 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.736155+0000", 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.736155+0000", 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.736157+0000", 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3030000000000001e-06 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.736161+0000", 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3059999999999999e-06 2024-09-02T10:36:20.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.736181+0000", 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9539e-05 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.736188+0000", 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.4540000000000001e-06 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.739815+0000", 2024-09-02T10:36:20.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.003626593 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.739924+0000", 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000109197 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.740071+0000", 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014690600000000001 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.759118+0000", 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.019047452999999999 2024-09-02T10:36:20.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079848+0000", 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32072952399999999 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.097159+0000", 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.017311106999999999 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.097199+0000", 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9882999999999997e-05 2024-09-02T10:36:20.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:20.756 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:20.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.099562+0000", 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0023638600000000002 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27559.0:146120 8.7 e848/844 ondisk, result = 0)", 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:21.749523+0000", 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 298.94005570500002, 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33033565399999998, 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.749523+0000", 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.749523+0000", 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.749532+0000", 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6060000000000003e-06 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.749537+0000", 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4779999999999999e-06 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079721+0000", 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33018389199999998 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079728+0000", 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9330000000000004e-06 2024-09-02T10:36:20.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079825+0000", 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7015000000000005e-05 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079838+0000", 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2649999999999999e-05 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079859+0000", 2024-09-02T10:36:20.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1080000000000001e-05 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:146134 8.1 8:8f786eb3:::benchmark_data_smithi187_136059_object40853:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:21.752946+0000", 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 298.93663329200001, 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33163602199999997, 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.762 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 146134 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.752946+0000", 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.752946+0000", 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.752948+0000", 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.888e-06 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.752958+0000", 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8600000000000005e-06 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.752976+0000", 2024-09-02T10:36:20.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7989e-05 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.752984+0000", 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.9199999999999993e-06 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.753053+0000", 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.8151000000000005e-05 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:20.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.753186+0000", 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013371600000000001 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,9", 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.753332+0000", 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000146131 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.757764+0000", 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0044320130000000003 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.757834+0000", 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9475999999999997e-05 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.084474+0000", 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32664032999999998 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.084530+0000", 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5732000000000001e-05 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.084582+0000", 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1816e-05 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27559.0:207393 8.7 8:e7dff5f2:::benchmark_data_smithi187_136059_object102112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:11.361416+0000", 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 189.328163354, 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28753896800000001, 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.768 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27559", 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 207393 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.361416+0000", 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.361416+0000", 2024-09-02T10:36:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.361417+0000", 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3149999999999999e-06 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.361422+0000", 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1290000000000004e-06 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.361433+0000", 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1096e-05 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.361440+0000", 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0910000000000004e-06 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.364500+0000", 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0030593130000000001 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:20.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.364659+0000", 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000158918 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 2,10", 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.364819+0000", 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00016017700000000001 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.369746+0000", 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.004927196 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.392407+0000", 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.022661191000000001 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.647718+0000", 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.25531097400000002 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:20.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.647819+0000", 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000101263 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:11.648955+0000", 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0011353050000000001 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(8.3 e882,844 8:c04512e7:::benchmark_data_smithi187_136059_object19286:head,838'7781)", 2024-09-02T10:36:20.776 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:17.112877+0000", 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 183.57670197100001, 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97456486799999997, 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:17.112877+0000", 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:17.112877+0000", 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:17.112878+0000", 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.4200000000000005e-07 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:17.112883+0000", 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6999999999999999e-06 2024-09-02T10:36:20.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.086964+0000", 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.97408130500000001 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.086970+0000", 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.0530000000000003e-06 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.087016+0000", 2024-09-02T10:36:20.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.6023000000000003e-05 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.087442+0000", 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000425945 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(8.3 e882,844 8:c8ab4c52:::benchmark_data_smithi187_136059_object21144:head,839'7904)", 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:18.722190+0000", 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 181.96738872500001, 2024-09-02T10:36:20.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.63038991300000002, 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.722190+0000", 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.722190+0000", 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.722191+0000", 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5099999999999999e-07 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.722195+0000", 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1690000000000002e-06 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.782 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.351878+0000", 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.62968315100000005 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.351888+0000", 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.8949999999999993e-06 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.352006+0000", 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000117432 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.783 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.352580+0000", 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00057451500000000005 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(8.7 e882,844 8:ee253e12:::benchmark_data_smithi187_136059_object9565:head,837'7191)", 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:18.850410+0000", 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 181.839168973, 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32497067499999999, 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.784 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.850410+0000", 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.850410+0000", 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.785 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.850412+0000", 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5149999999999999e-06 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.850418+0000", 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.64e-06 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.174696+0000", 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32427790200000001 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.786 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.174709+0000", 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.2877999999999999e-05 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.174769+0000", 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9791999999999999e-05 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.175381+0000", 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00061194799999999996 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.787 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(8.7 e883,844 8:ebdb0402:::benchmark_data_smithi187_136059_object10842:head,837'7267)", 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:19.695341+0000", 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 180.994237658, 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29206983199999997, 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.695341+0000", 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.788 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.695341+0000", 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.695342+0000", 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6499999999999999e-07 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.695346+0000", 2024-09-02T10:36:20.789 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0040000000000001e-06 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.986720+0000", 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29137421099999999 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.986724+0000", 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.523e-06 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.986766+0000", 2024-09-02T10:36:20.790 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.1817999999999998e-05 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.987411+0000", 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00064561100000000004 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(8.3 e883,844 8:c06a250d:::benchmark_data_smithi187_136059_object26388:head,841'8244)", 2024-09-02T10:36:20.791 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:21.168838+0000", 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 179.52074133900001, 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31984945100000001, 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.168838+0000", 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.792 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.168838+0000", 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.168838+0000", 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.9299999999999997e-07 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.793 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.168843+0000", 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2309999999999998e-06 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.488273+0000", 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.319430626 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.488282+0000", 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.2060000000000007e-06 2024-09-02T10:36:20.794 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.488332+0000", 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0043000000000002e-05 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.488687+0000", 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035565199999999998 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.795 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDeleteReply(8.7 e886,844 8:eed84c3b:::benchmark_data_smithi187_136059_object24726:head,839'8153)", 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:24.929116+0000", 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 175.760463571, 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28751482499999997, 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.796 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:24.929116+0000", 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:24.929116+0000", 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.797 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:24.929117+0000", 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5900000000000005e-07 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:24.929122+0000", 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.5219999999999998e-06 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:25.216073+0000", 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.28695056699999999 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.798 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:25.216080+0000", 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.3069999999999997e-06 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:25.216138+0000", 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8014999999999999e-05 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:25.216630+0000", 2024-09-02T10:36:20.799 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00049245500000000004 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.28030.0:148384 9.6 e960/924 9:63c4bb7b:::benchmark_data_smithi187_154927_object50272:head v 960'6505, mlcod=960'6501)", 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:36:19.086210+0000", 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 1.603369169, 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.18547538499999999, 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:20.800 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.086210+0000", 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.086210+0000", 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.086212+0000", 2024-09-02T10:36:20.801 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.345e-06 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.086217+0000", 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7899999999999999e-06 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.259857+0000", 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.173639564 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.802 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.259861+0000", 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.3789999999999999e-06 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.263772+0000", 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0039111409999999999 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.263783+0000", 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.0975000000000001e-05 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.803 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.271658+0000", 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0078743690000000009 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.271685+0000", 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.7821999999999999e-05 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:20.804 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:36:20.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:20 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:36:20.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:20 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:36:20.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:20 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:36:20.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:20 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:36:20.805 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:20 smithi187 ceph-mon[20072]: pgmap v2200: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail 2024-09-02T10:36:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:20 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:36:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:20 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:36:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:20 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:36:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:20 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:36:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:20 smithi162 ceph-mon[20279]: pgmap v2200: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 3.1 GiB data, 18 GiB used, 965 GiB / 983 GiB avail 2024-09-02T10:36:20.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:20.885 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:20.886 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:20.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:21.013 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:21.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:21.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:21.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:21.146 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:21.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:21.248 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:21.277 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:21.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:21.379 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:21.404 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:21.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:21.506 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:21.529 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:21.530 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:21.631 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:21.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:21 smithi151 ceph-mon[16639]: 9.b scrub starts 2024-09-02T10:36:21.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:21 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:36:21.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:21 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:36:21.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:21 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:36:21.653 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:21 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:36:21.660 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:21.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:21 smithi187 ceph-mon[20072]: 9.b scrub starts 2024-09-02T10:36:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:21 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:36:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:21 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:36:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:21 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:36:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:21 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:36:21.762 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:21.789 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:21.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:21.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:21 smithi162 ceph-mon[20279]: 9.b scrub starts 2024-09-02T10:36:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:21 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:36:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:21 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:36:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:21 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:36:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:21 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:36:21.893 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:21.914 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:21.915 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:22.002 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:36:22.016 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:22.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:22.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:22.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:22.127 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:36:22.127 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:36:22.172 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:36:22.172 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:36:22.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:22.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:36:22.209 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:36:22.218 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:22.219 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:22.244 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:36:22.244 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:36:22.281 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:36:22.281 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:36:22.318 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:36:22.318 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:36:22.319 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:22.351 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:22.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:22.353 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:36:22.354 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:36:22.391 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:36:22.392 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:36:22.392 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:22 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:36:22.237+0000 7fe3ffed4640 -1 mon.a@0(leader).osd e960 definitely_dead 0 2024-09-02T10:36:22.411 INFO:teuthology.orchestra.run.smithi151.stderr:marked out osd.10. 2024-09-02T10:36:22.428 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:36:22.428 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:36:22.453 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:22.464 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:36:22.464 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:36:22.478 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:22.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:22.501 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:36:22.501 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:36:22.524 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:36:22.524 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:36:22.579 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:22.601 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:22.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:22 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:36:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:22 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:36:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:22 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:36:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:22 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:36:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:22 smithi151 ceph-mon[16639]: pgmap v2201: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail 2024-09-02T10:36:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2363961407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1768648277' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-02T10:36:22.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:22 smithi151 ceph-mon[16639]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-02T10:36:22.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:22 smithi151 ceph-mon[16639]: 9.19 scrub starts 2024-09-02T10:36:22.702 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:22.724 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:22.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:22 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:36:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:22 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:36:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:22 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:36:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:22 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:36:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:22 smithi187 ceph-mon[20072]: pgmap v2201: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail 2024-09-02T10:36:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2363961407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:22.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1768648277' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-02T10:36:22.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:22 smithi187 ceph-mon[20072]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-02T10:36:22.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:22 smithi187 ceph-mon[20072]: 9.19 scrub starts 2024-09-02T10:36:22.824 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:22 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:36:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:22 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:36:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:22 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:36:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:22 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:36:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:22 smithi162 ceph-mon[20279]: pgmap v2201: 57 pgs: 1 active+clean+scrubbing, 56 active+clean; 2.9 GiB data, 17 GiB used, 966 GiB / 983 GiB avail 2024-09-02T10:36:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2363961407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1768648277' entity='client.admin' cmd={"prefix": "osd out", "ids": ["10"]} : dispatch 2024-09-02T10:36:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:22 smithi162 ceph-mon[20279]: Client client.admin marked osd.10 out, while it was still marked up 2024-09-02T10:36:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:22 smithi162 ceph-mon[20279]: 9.19 scrub starts 2024-09-02T10:36:22.846 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:22.847 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:22.948 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:22.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:22.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:23.071 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:23.092 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:23.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:23.194 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:23.216 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:23.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:23.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:23.339 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:23.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:23.442 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:23.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:23.464 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:23.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:23.566 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:23.587 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:23.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:23.654 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-02T10:36:23.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:23 smithi151 ceph-mon[16639]: 9.b scrub ok 2024-09-02T10:36:23.677 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1768648277' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-02T10:36:23.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:23 smithi151 ceph-mon[16639]: osdmap e961: 12 total, 12 up, 10 in 2024-09-02T10:36:23.678 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:23 smithi151 ceph-mon[16639]: 9.19 scrub ok 2024-09-02T10:36:23.689 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:23.714 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:23.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:23.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:23 smithi187 ceph-mon[20072]: 9.b scrub ok 2024-09-02T10:36:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1768648277' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-02T10:36:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:23 smithi187 ceph-mon[20072]: osdmap e961: 12 total, 12 up, 10 in 2024-09-02T10:36:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:23 smithi187 ceph-mon[20072]: 9.19 scrub ok 2024-09-02T10:36:23.816 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:23.838 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:23.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:23.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:23 smithi162 ceph-mon[20279]: 9.b scrub ok 2024-09-02T10:36:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1768648277' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["10"]}]': finished 2024-09-02T10:36:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:23 smithi162 ceph-mon[20279]: osdmap e961: 12 total, 12 up, 10 in 2024-09-02T10:36:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:23 smithi162 ceph-mon[20279]: 9.19 scrub ok 2024-09-02T10:36:23.940 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:23.961 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:23.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:23.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:24.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:24.089 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:24.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:24.191 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:24.216 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:24.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:24.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:24.318 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:24.343 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:24.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:24.445 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:24.479 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:24.480 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:24.581 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:24.617 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:24.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:24 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:24 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:24 smithi151 ceph-mon[16639]: 9.0 scrub starts 2024-09-02T10:36:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:24 smithi151 ceph-mon[16639]: 9.0 scrub ok 2024-09-02T10:36:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:24 smithi151 ceph-mon[16639]: osdmap e962: 12 total, 12 up, 10 in 2024-09-02T10:36:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:24 smithi151 ceph-mon[16639]: pgmap v2204: 57 pgs: 1 unknown, 6 peering, 50 active+clean; 2.8 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-02T10:36:24.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:24 smithi151 ceph-mon[16639]: osdmap e963: 12 total, 12 up, 10 in 2024-09-02T10:36:24.720 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:24.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:24.747 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:24.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:24 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:24 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:24 smithi187 ceph-mon[20072]: 9.0 scrub starts 2024-09-02T10:36:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:24 smithi187 ceph-mon[20072]: 9.0 scrub ok 2024-09-02T10:36:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:24 smithi187 ceph-mon[20072]: osdmap e962: 12 total, 12 up, 10 in 2024-09-02T10:36:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:24 smithi187 ceph-mon[20072]: pgmap v2204: 57 pgs: 1 unknown, 6 peering, 50 active+clean; 2.8 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-02T10:36:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:24 smithi187 ceph-mon[20072]: osdmap e963: 12 total, 12 up, 10 in 2024-09-02T10:36:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:24 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:24 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:24 smithi162 ceph-mon[20279]: 9.0 scrub starts 2024-09-02T10:36:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:24 smithi162 ceph-mon[20279]: 9.0 scrub ok 2024-09-02T10:36:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:24 smithi162 ceph-mon[20279]: osdmap e962: 12 total, 12 up, 10 in 2024-09-02T10:36:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:24 smithi162 ceph-mon[20279]: pgmap v2204: 57 pgs: 1 unknown, 6 peering, 50 active+clean; 2.8 GiB data, 16 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 1.22k op/s 2024-09-02T10:36:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:24 smithi162 ceph-mon[20279]: osdmap e963: 12 total, 12 up, 10 in 2024-09-02T10:36:24.848 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:24.876 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:24.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:24.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:24.978 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:25.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:25.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:25.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:25.138 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:25.139 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:25.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:25.275 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:25.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:25.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:25.376 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:25.401 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:25.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:25.403 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:25.504 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:25.542 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:25.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:25.643 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:25.665 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:25.666 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:25.667 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:25.668 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:25.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:25 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:36:25.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:25 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:36:25.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:25 smithi151 ceph-mon[16639]: 9.f scrub starts 2024-09-02T10:36:25.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:25 smithi151 ceph-mon[16639]: 9.14 starting backfill to osd.8 from (0'0,925'3862] MIN to 960'6707 2024-09-02T10:36:25.691 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:25 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-02T10:36:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:25 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:36:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:25 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:36:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:25 smithi187 ceph-mon[20072]: 9.f scrub starts 2024-09-02T10:36:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:25 smithi187 ceph-mon[20072]: 9.14 starting backfill to osd.8 from (0'0,925'3862] MIN to 960'6707 2024-09-02T10:36:25.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:25 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-02T10:36:25.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:25.795 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:25.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:25.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:25 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:36:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:25 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:36:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:25 smithi162 ceph-mon[20279]: 9.f scrub starts 2024-09-02T10:36:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:25 smithi162 ceph-mon[20279]: 9.14 starting backfill to osd.8 from (0'0,925'3862] MIN to 960'6707 2024-09-02T10:36:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:25 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 2 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-02T10:36:25.896 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:25.919 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:25.920 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:26.021 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:26.050 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:26.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:26.152 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:26.175 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:26.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:26.176 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:26.277 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:26.301 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:26.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:26.303 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:26.404 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:26.438 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:26.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:26.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:26.539 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:26.563 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:26.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:26.565 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:26.665 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:26 smithi151 ceph-mon[16639]: 9.f scrub ok 2024-09-02T10:36:26.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:26 smithi151 ceph-mon[16639]: pgmap v2206: 57 pgs: 1 active+undersized+degraded+remapped, 1 remapped+peering, 1 unknown, 6 peering, 48 active+clean; 2.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 923 op/s; 6104/133965 objects degraded (4.556%) 2024-09-02T10:36:26.695 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:26.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:26 smithi187 ceph-mon[20072]: 9.f scrub ok 2024-09-02T10:36:26.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:26 smithi187 ceph-mon[20072]: pgmap v2206: 57 pgs: 1 active+undersized+degraded+remapped, 1 remapped+peering, 1 unknown, 6 peering, 48 active+clean; 2.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 923 op/s; 6104/133965 objects degraded (4.556%) 2024-09-02T10:36:26.797 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:26.826 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:26.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:26.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:26 smithi162 ceph-mon[20279]: 9.f scrub ok 2024-09-02T10:36:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:26 smithi162 ceph-mon[20279]: pgmap v2206: 57 pgs: 1 active+undersized+degraded+remapped, 1 remapped+peering, 1 unknown, 6 peering, 48 active+clean; 2.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 923 op/s; 6104/133965 objects degraded (4.556%) 2024-09-02T10:36:26.897 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:36:26.897 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:36:26.928 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:26.963 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:26.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:26.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:27.064 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:27.090 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:27.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:27.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:27.192 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:27.234 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:27.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:27.335 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:27.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:27.374 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:27.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:27.445 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:36:27.475 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:27.500 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:27.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:27.501 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:27.540 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:27 smithi151 ceph-mon[16639]: Health check failed: Degraded data redundancy: 6104/133965 objects degraded (4.556%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:36:27.540 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/350467482' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:36:27.582 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-02T10:36:27.596 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:36:27.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:27.625 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:27.627 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:27.728 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:27.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:27 smithi162 ceph-mon[20279]: Health check failed: Degraded data redundancy: 6104/133965 objects degraded (4.556%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:36:27.746 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/350467482' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:36:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:27 smithi187 ceph-mon[20072]: Health check failed: Degraded data redundancy: 6104/133965 objects degraded (4.556%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:36:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/350467482' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:36:27.760 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:27.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:27.862 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:27.896 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:27.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:27.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:27.998 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:28.020 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:28.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:28.021 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:28.122 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:28.154 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:28.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:28.256 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:28.278 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:28.279 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:28.380 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:28.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:28.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:28.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:28.542 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:28.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:28.573 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:28.617 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:28.644 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:28.668 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:28.669 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:28.701 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:28 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:28.701 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/350467482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:36:28.701 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:28 smithi151 ceph-mon[16639]: osdmap e964: 12 total, 12 up, 10 in 2024-09-02T10:36:28.701 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:28 smithi151 ceph-mon[16639]: pgmap v2208: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 remapped+peering, 1 unknown, 5 peering, 46 active+clean; 2.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 674 op/s; 6142/131925 objects degraded (4.656%); 5288/131925 objects misplaced (4.008%); 2.8 MiB/s, 45 objects/s recovering 2024-09-02T10:36:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:28 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/350467482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:36:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:28 smithi187 ceph-mon[20072]: osdmap e964: 12 total, 12 up, 10 in 2024-09-02T10:36:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:28 smithi187 ceph-mon[20072]: pgmap v2208: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 remapped+peering, 1 unknown, 5 peering, 46 active+clean; 2.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 674 op/s; 6142/131925 objects degraded (4.656%); 5288/131925 objects misplaced (4.008%); 2.8 MiB/s, 45 objects/s recovering 2024-09-02T10:36:28.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:28.800 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:28.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:28 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/350467482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:36:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:28 smithi162 ceph-mon[20279]: osdmap e964: 12 total, 12 up, 10 in 2024-09-02T10:36:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:28 smithi162 ceph-mon[20279]: pgmap v2208: 57 pgs: 3 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 1 remapped+peering, 1 unknown, 5 peering, 46 active+clean; 2.7 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 674 op/s; 6142/131925 objects degraded (4.656%); 5288/131925 objects misplaced (4.008%); 2.8 MiB/s, 45 objects/s recovering 2024-09-02T10:36:28.902 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:28.945 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:28.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:29.047 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:29.076 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:29.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:29.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:29.180 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:29.203 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:29.205 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:29.301 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 4, 2] out_osds: [11, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:36:29.302 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:36:29.302 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-02T10:36:29.302 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:36:29.305 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:29.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:29.385 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:29.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:29.486 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:29.526 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:36:29.526 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:36:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:29.528 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:29.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:29.558 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:36:29.558 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:36:29.597 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:36:29.597 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:36:29.629 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:29.633 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:36:29.633 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:36:29.658 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:29.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:29.662 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:36:29.662 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:36:29.697 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:36:29.697 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:36:29.730 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:36:29.730 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:36:29.731 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:36:29.731 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [], 2024-09-02T10:36:29.731 INFO:teuthology.orchestra.run.smithi151.stdout: "complaint_time": 30, 2024-09-02T10:36:29.731 INFO:teuthology.orchestra.run.smithi151.stdout: "num_blocked_ops": 0 2024-09-02T10:36:29.731 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:36:29.760 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:29.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:36:29.767 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:36:29.796 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:36:29.796 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:36:29.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:29.797 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:29.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:29.823 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:36:29.823 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:36:29.852 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:36:29.852 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:36:29.880 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:36:29.880 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:36:29.898 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:29.925 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:29.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:29.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:30.027 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:30.051 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:30.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:30.153 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:30.189 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:30.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:30.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:30.290 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:30.324 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:30.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:30.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:30.453 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:30.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:30.487 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:36:30.555 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:30.599 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:30.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:30.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-02T10:36:30.668 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3218096009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:30.668 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:30 smithi151 ceph-mon[16639]: pgmap v2209: 57 pgs: 1 active+remapped+backfilling, 5 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 50 active+clean; 2.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 499 op/s; 6145/127512 objects degraded (4.819%); 8531/127512 objects misplaced (6.690%); 9.6 MiB/s, 154 objects/s recovering 2024-09-02T10:36:30.701 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:30.739 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:30.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3218096009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:30 smithi187 ceph-mon[20072]: pgmap v2209: 57 pgs: 1 active+remapped+backfilling, 5 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 50 active+clean; 2.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 499 op/s; 6145/127512 objects degraded (4.819%); 8531/127512 objects misplaced (6.690%); 9.6 MiB/s, 154 objects/s recovering 2024-09-02T10:36:30.841 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3218096009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:30 smithi162 ceph-mon[20279]: pgmap v2209: 57 pgs: 1 active+remapped+backfilling, 5 active+recovering+undersized+degraded+remapped, 1 active+undersized+degraded+remapped, 50 active+clean; 2.6 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 499 op/s; 6145/127512 objects degraded (4.819%); 8531/127512 objects misplaced (6.690%); 9.6 MiB/s, 154 objects/s recovering 2024-09-02T10:36:30.845 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:30.878 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:30.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:30.981 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:31.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:31.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:31.112 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:31.138 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:31.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:31.240 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:31.265 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:31.266 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:31.368 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:31.393 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:31.394 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:31.495 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:31.520 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:31.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:31.570 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:31 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-02T10:36:31.622 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:31.647 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:31.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:31.680 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:31.749 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:31 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-02T10:36:31.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:31.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:31 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 3 pgs peering) 2024-09-02T10:36:31.871 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:31.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:31.904 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:31.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:32.005 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:32.045 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:32.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:32.147 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:32.171 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:32.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:32.172 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:32.200 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:32.268 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:36:32.270 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:36:32.273 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:32.296 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:32.297 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:32.398 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:32.434 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:32.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:32.536 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:32.562 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:32.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:32.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:32.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:32 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:32.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:32 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:32.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:32 smithi151 ceph-mon[16639]: osdmap e965: 12 total, 12 up, 10 in 2024-09-02T10:36:32.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:32 smithi151 ceph-mon[16639]: pgmap v2211: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 561 op/s; 124/123126 objects degraded (0.101%); 10288/123126 objects misplaced (8.356%); 17 MiB/s, 264 objects/s recovering 2024-09-02T10:36:32.692 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:36:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 20, 2024-09-02T10:36:32.692 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 600, 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27220.0:136914 7.3 e743/741 7:c30a193a:::benchmark_data_smithi187_120712_object52972:head v 743'8327, mlcod=742'8324)", 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:26:43.758995+0000", 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 588.67134503499994, 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33365049099999999, 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.693 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758995+0000", 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758995+0000", 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.758996+0000", 2024-09-02T10:36:32.694 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.1100000000000002e-07 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:43.759001+0000", 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9309999999999999e-06 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090734+0000", 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33173325799999998 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.695 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090757+0000", 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3033000000000001e-05 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090882+0000", 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012556700000000001 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.090902+0000", 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9519999999999999e-05 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.696 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092615+0000", 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001713387 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:26:44.092646+0000", 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0484000000000002e-05 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.697 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:145939 7.12 7:4b7e5994:::benchmark_data_smithi187_120712_object61997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.921348+0000", 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 569.50899266500005, 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38239470399999997, 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 145939 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.698 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921348+0000", 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921348+0000", 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921349+0000", 2024-09-02T10:36:32.699 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5349999999999999e-06 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921355+0000", 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3940000000000004e-06 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921368+0000", 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3499000000000001e-05 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.700 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.921375+0000", 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4729999999999997e-06 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.928654+0000", 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0072797640000000002 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.928971+0000", 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000317127 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:36:32.701 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.929306+0000", 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00033418199999999998 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.939106+0000", 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0098002920000000004 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.951104+0000", 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.011997733 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.702 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302905+0000", 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.351801429 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.303056+0000", 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00015083300000000001 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.303742+0000", 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00068644300000000001 2024-09-02T10:36:32.703 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:145939 7.12 e749/744 ondisk, result = 0)", 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.932006+0000", 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 569.49833442900001, 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37174302799999998, 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.704 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932006+0000", 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932006+0000", 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932007+0000", 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1039999999999999e-06 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.705 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.932012+0000", 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8289999999999998e-06 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302586+0000", 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37057435599999999 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.706 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302593+0000", 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4300000000000003e-06 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302888+0000", 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000295378 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.302898+0000", 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.6469999999999993e-06 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.707 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.303749+0000", 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000851284 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPushReply(7.12 749/744 [PushReplyOp(7:481a1f6c:::benchmark_data_smithi187_120712_object69540:head)])", 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.953782+0000", 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 569.47655835399996, 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.375520674, 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.708 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "reached pg", 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953782+0000", 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953782+0000", 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.709 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953783+0000", 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.6099999999999999e-07 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.953789+0000", 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6279999999999999e-06 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.328024+0000", 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37423416799999998 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.710 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.328033+0000", 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1700000000000003e-06 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.328279+0000", 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000246519 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.329303+0000", 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001023328 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.711 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.27220.0:145959 7.12 7:4e913d9f:::benchmark_data_smithi187_120712_object62017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e749)", 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.957341+0000", 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 569.47299962900001, 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.39245286600000001, 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.27220", 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/350728921", 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 145959 2024-09-02T10:36:32.712 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957341+0000", 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957341+0000", 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.713 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957342+0000", 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5549999999999999e-06 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957350+0000", 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5889999999999997e-06 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957361+0000", 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1075e-05 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.714 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957369+0000", 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8010000000000004e-06 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957569+0000", 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00020020500000000001 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957715+0000", 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00014592899999999999 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.715 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 0,6", 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.957956+0000", 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00024078399999999999 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.972919+0000", 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.014963476999999999 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.974010+0000", 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0010906589999999999 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.716 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349010+0000", 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.37499981799999998 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349077+0000", 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.7599000000000001e-05 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349794+0000", 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000716375 2024-09-02T10:36:32.717 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop_reply(client.27220.0:145959 7.12 e749/744 ondisk, result = 0)", 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:27:02.961683+0000", 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 569.46865753600002, 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38811612099999998, 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.718 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961683+0000", 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961683+0000", 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961684+0000", 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5600000000000005e-07 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.719 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:02.961688+0000", 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.831e-06 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.348905+0000", 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38721641600000001 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.348912+0000", 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.2559999999999996e-06 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.720 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.348993+0000", 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.1050999999999994e-05 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349002+0000", 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.3309999999999993e-06 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:27:03.349799+0000", 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00079648000000000002 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.721 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27559.0:146099 8.6 e848/844 8:6509e9b8:::benchmark_data_smithi187_136059_object40818:head v 848'9190, mlcod=810'2569)", 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:21.719743+0000", 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 310.71059754800001, 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35979581900000002, 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.722 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.719743+0000", 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.719743+0000", 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.719744+0000", 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5000000000000002e-07 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.723 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.719750+0000", 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.6030000000000001e-06 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.747355+0000", 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.027604733999999999 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.747364+0000", 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.5980000000000003e-06 2024-09-02T10:36:32.724 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.747429+0000", 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4233000000000007e-05 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.747454+0000", 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.5265e-05 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.725 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079487+0000", 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.332033407 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079539+0000", 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1229e-05 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27559.0:146125 8.6 e848/844 8:68b9d70e:::benchmark_data_smithi187_136059_object40844:head v 848'9191, mlcod=810'2570)", 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:21.747429+0000", 2024-09-02T10:36:32.726 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 310.68291172300002, 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.35615970200000002, 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.747429+0000", 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.747429+0000", 2024-09-02T10:36:32.727 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.747429+0000", 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.8800000000000002e-07 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.747438+0000", 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.7700000000000007e-06 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.728 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.751718+0000", 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0042799889999999997 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.751723+0000", 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.288e-06 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079588+0000", 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32786517900000001 2024-09-02T10:36:32.729 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079612+0000", 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3163000000000002e-05 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.103548+0000", 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.023935951 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.103588+0000", 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0774e-05 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.730 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27559.0:146131 8.6 e848/844 8:6708c0fe:::benchmark_data_smithi187_136059_object40850:head v 848'9192, mlcod=810'2570)", 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:21.751694+0000", 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 310.67864595100002, 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.38592803799999997, 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.751694+0000", 2024-09-02T10:36:32.731 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.751694+0000", 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.751695+0000", 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.7299999999999996e-07 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.732 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.751700+0000", 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.3369999999999999e-06 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.083578+0000", 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.3318777 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.083586+0000", 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3049999999999993e-06 2024-09-02T10:36:32.733 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.088432+0000", 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0048460869999999998 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.088453+0000", 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.0397000000000001e-05 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.137574+0000", 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.049121608999999997 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.734 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.137622+0000", 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8029999999999999e-05 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27559.0:146135 8.f e848/792 8:ffa01c02:::benchmark_data_smithi187_136059_object40854:head v 848'9021, mlcod=848'9018)", 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:21.755320+0000", 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 310.67502010300001, 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33252133499999997, 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.735 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755320+0000", 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755320+0000", 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.736 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755321+0000", 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.3900000000000004e-07 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755326+0000", 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7260000000000002e-06 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755337+0000", 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1209999999999999e-05 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.737 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755342+0000", 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7129999999999996e-06 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755384+0000", 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2299999999999998e-05 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755397+0000", 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3072e-05 2024-09-02T10:36:32.738 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.087805+0000", 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.33240810500000001 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.087842+0000", 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6369999999999999e-05 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.739 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27559.0:146127 8.d e848/844 8:bbf1b620:::benchmark_data_smithi187_136059_object40846:head v 848'9118, mlcod=810'2550)", 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:31:21.755628+0000", 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 310.67471253799999, 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.34018841700000002, 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755628+0000", 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.740 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755628+0000", 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755628+0000", 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5799999999999999e-07 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:21.755633+0000", 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.9880000000000004e-06 2024-09-02T10:36:32.741 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079464+0000", 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.32383039400000002 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079479+0000", 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.4786999999999999e-05 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079556+0000", 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7439000000000002e-05 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.742 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.079587+0000", 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.0800999999999999e-05 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.095772+0000", 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.016184737000000001 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:31:22.095816+0000", 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.4613000000000001e-05 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.743 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.27559.0:210553 8.d e881/878 8:bd49576e:::benchmark_data_smithi187_136059_object105272:head v 881'13133, mlcod=881'13130)", 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:16.489135+0000", 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 195.941204954, 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51567623100000004, 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:16.489135+0000", 2024-09-02T10:36:32.744 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:16.489135+0000", 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:16.489136+0000", 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.0200000000000001e-07 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.745 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:16.489143+0000", 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.3339999999999998e-06 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:16.999720+0000", 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.51057704299999995 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:16.999725+0000", 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.135e-06 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.746 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:16.999802+0000", 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7486999999999999e-05 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:16.999825+0000", 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.3054999999999999e-05 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:17.004765+0000", 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0049393709999999997 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.747 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:17.004812+0000", 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.7003999999999999e-05 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(8.6 e882,844 8:68c5b3c8:::benchmark_data_smithi187_136059_object22986:head,839'8145)", 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:18.621860+0000", 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 193.808480532, 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43800437800000003, 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:36:32.748 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.621860+0000", 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.621860+0000", 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.621860+0000", 2024-09-02T10:36:32.749 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.5199999999999996e-07 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:18.621864+0000", 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.4199999999999999e-06 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.059713+0000", 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.43784931999999999 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.750 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.059721+0000", 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.242e-06 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.059786+0000", 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.4766999999999998e-05 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.059794+0000", 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.8509999999999999e-06 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.751 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:19.059864+0000", 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0125999999999999e-05 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(8.6 e883,844 8:6c0bd50b:::benchmark_data_smithi187_136059_object28548:head,842'8465)", 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:20.737883+0000", 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 191.692457171, 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29862281000000002, 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:36:32.752 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:20.737883+0000", 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:20.737883+0000", 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.753 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:20.737884+0000", 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4499999999999997e-07 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:20.737887+0000", 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.7179999999999998e-06 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.036416+0000", 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.29852836100000002 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.754 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.036421+0000", 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0760000000000002e-06 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.036461+0000", 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0005000000000001e-05 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.036466+0000", 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0150000000000003e-06 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.755 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.036506+0000", 2024-09-02T10:36:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.0089999999999997e-05 2024-09-02T10:36:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.756 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGRecoveryDelete(8.6 e883,844 8:6ba9feb3:::benchmark_data_smithi187_136059_object28608:head,842'8473)", 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:33:21.068083+0000", 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 191.36225752999999, 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75875921599999996, 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.068083+0000", 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.757 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.068083+0000", 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.068083+0000", 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.1699999999999998e-07 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.758 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.068086+0000", 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.061e-06 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.826729+0000", 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.75864276100000005 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.826735+0000", 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.609e-06 2024-09-02T10:36:32.759 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.826785+0000", 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.0172e-05 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.826794+0000", 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 8.8210000000000008e-06 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.760 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:33:21.826842+0000", 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.8275000000000001e-05 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(9.16 933/927 [PushOp(9:690b48c7:::benchmark_data_smithi187_154927_object27430:head, version: 914'1730, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:690b48c7:::benchmark_data_smithi187_154927_object27430:head@914'1730, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:35:14.031832+0000", 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 78.398507933999994, 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31402407300000001, 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.761 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:35:14.031832+0000", 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:35:14.031832+0000", 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.762 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:35:14.031834+0000", 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9099999999999999e-06 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:35:14.031877+0000", 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 4.2902999999999999e-05 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:35:14.344403+0000", 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.31252614499999998 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.763 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:35:14.344422+0000", 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8805999999999999e-05 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:35:14.344537+0000", 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000114529 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:35:14.344555+0000", 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.8729e-05 2024-09-02T10:36:32.764 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:35:14.345856+0000", 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0013010509999999999 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.28030.0:144441 9.16 e959/947 9:6fbe757d:::benchmark_data_smithi187_154927_object46329:head v 959'6446, mlcod=959'6443)", 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:36:15.725018+0000", 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 16.705322829, 2024-09-02T10:36:32.765 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.10654385299999999, 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:15.725018+0000", 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:15.725018+0000", 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.766 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:15.725019+0000", 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.1230000000000001e-06 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:15.725026+0000", 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.7409999999999992e-06 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:15.829468+0000", 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.104441141 2024-09-02T10:36:32.767 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:15.829477+0000", 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.1290000000000002e-06 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:15.829539+0000", 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 6.2069999999999994e-05 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:15.829560+0000", 2024-09-02T10:36:32.768 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.1158999999999999e-05 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:15.831525+0000", 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001964922 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:15.831561+0000", 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.6568e-05 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.769 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_repop(client.28030.0:148371 9.16 e960/947 9:697b5315:::benchmark_data_smithi187_154927_object50259:head v 960'6557, mlcod=960'6554)", 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:36:19.075076+0000", 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 13.355264045, 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.19000695400000001, 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.075076+0000", 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.770 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.075076+0000", 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.075078+0000", 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3519999999999999e-06 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.075085+0000", 2024-09-02T10:36:32.771 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0190000000000001e-06 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.263119+0000", 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.188033851 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.263126+0000", 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.0129999999999996e-06 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.772 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.263180+0000", 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.4177e-05 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.263195+0000", 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5631e-05 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "commit_sent", 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.265054+0000", 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001858279 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.773 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:19.265083+0000", 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 2.9632e-05 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(9.7 964/962 [PushOp(9:e780f089:::benchmark_data_smithi187_154927_object63042:head, version: 928'3788, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:e780f089:::benchmark_data_smithi187_154927_object63042:head@928'3788, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:36:29.561565+0000", 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 2.868775479, 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.108268683, 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.774 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:29.561565+0000", 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:29.561565+0000", 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:29.561565+0000", 2024-09-02T10:36:32.775 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.79e-07 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:29.561604+0000", 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 3.9001000000000001e-05 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:29.667550+0000", 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.105945914 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.776 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:29.667568+0000", 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.7584e-05 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:29.667682+0000", 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00011453199999999999 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:29.667702+0000", 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9876000000000001e-05 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.777 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:29.669834+0000", 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0021315969999999998 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "MOSDPGPush(9.7 964/962 [PushOp(9:e6b64ceb:::benchmark_data_smithi187_154927_object63312:head, version: 928'3797, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:e6b64ceb:::benchmark_data_smithi187_154927_object63312:head@928'3797, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:36:30.017627+0000", 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 2.4127129489999999, 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.114481108, 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "started", 2024-09-02T10:36:32.778 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:30.017627+0000", 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:30.017627+0000", 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:30.017628+0000", 2024-09-02T10:36:32.779 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.7100000000000011e-07 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:30.017761+0000", 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00013304699999999999 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:30.130337+0000", 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.112575231 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:32.780 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:30.130356+0000", 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.9397000000000001e-05 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:30.130450+0000", 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.4352999999999996e-05 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:30.130460+0000", 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 9.9080000000000007e-06 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "done", 2024-09-02T10:36:32.781 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:30.132109+0000", 2024-09-02T10:36:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.001648201 2024-09-02T10:36:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:32.782 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:32.782 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:36:32.783 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:32 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:32.783 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:32 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:32.783 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:32 smithi187 ceph-mon[20072]: osdmap e965: 12 total, 12 up, 10 in 2024-09-02T10:36:32.783 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:32 smithi187 ceph-mon[20072]: pgmap v2211: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 561 op/s; 124/123126 objects degraded (0.101%); 10288/123126 objects misplaced (8.356%); 17 MiB/s, 264 objects/s recovering 2024-09-02T10:36:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:32.784 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:32.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:32 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:32 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:32 smithi162 ceph-mon[20279]: osdmap e965: 12 total, 12 up, 10 in 2024-09-02T10:36:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:32 smithi162 ceph-mon[20279]: pgmap v2211: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.5 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 561 op/s; 124/123126 objects degraded (0.101%); 10288/123126 objects misplaced (8.356%); 17 MiB/s, 264 objects/s recovering 2024-09-02T10:36:32.885 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:32.921 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:32.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:32.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:33.023 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:33.056 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:33.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:33.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:33.159 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:33.183 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:33.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:33.285 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:33.314 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:33.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:33.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:33.365 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:36:33.416 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:33.445 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:33.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:33.547 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:33.581 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:33.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:33.638 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-02T10:36:33.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:33 smithi151 ceph-mon[16639]: from='client.28304 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:36:33.666 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:33 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 124/123126 objects degraded (0.101%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:33.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:33.703 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:33.704 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:33 smithi187 ceph-mon[20072]: from='client.28304 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:36:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:33 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 124/123126 objects degraded (0.101%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:33.805 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:33.827 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:33.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:33.828 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:33 smithi162 ceph-mon[20279]: from='client.28304 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:36:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:33 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 124/123126 objects degraded (0.101%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:33.929 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:33.960 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:33.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:33.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:34.062 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:34.095 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:34.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:34.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:34.197 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:34.224 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:34.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:34.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:34.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:34.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:34.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:34.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:34 smithi151 ceph-mon[16639]: pgmap v2212: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 629 op/s; 124/118860 objects degraded (0.104%); 9741/118860 objects misplaced (8.195%); 19 MiB/s, 307 objects/s recovering 2024-09-02T10:36:34.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:34.484 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:34.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:34.586 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:34.613 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:34.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:34.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:34.691 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2211,"stamp":"2024-09-02T10:36:31.621752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2690122296,"num_objects":41042,"num_object_clones":0,"num_object_copies":123126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":124,"num_objects_misplaced":10288,"num_objects_unfound":0,"num_objects_dirty":41042,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99716,"num_write_kb":3151489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1915,"num_bytes_recovered":125711968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195997,"ondisk_log_size":195997,"up":171,"acting":165,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":10,"num_per_pool_osds":10,"num_per_pool_omap_osds":10,"kb":937451520,"kb_used":15458732,"kb_used_data":7645692,"kb_used_omap":15,"kb_used_meta":7812656,"kb_avail":921992788,"statfs":{"total":959950356480,"available":944120614912,"internally_reserved":0,"allocated":7829188608,"data_stored":7826253184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15945,"internal_metadata":8000160183},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-265486336,"num_objects":-4051,"num_object_clones":0,"num_object_copies":-12153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":124,"num_objects_misplaced":10288,"num_objects_unfound":0,"num_objects_dirty":-4051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4051,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1911,"num_bytes_recovered":124780544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.211765"},"pg_stats":[{"pgid":"9.17","version":"964'6671","reported_seq":14462,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:29.645499+0000","last_change":"2024-09-02T10:36:14.428237+0000","last_active":"2024-09-02T10:36:29.645499+0000","last_peered":"2024-09-02T10:36:29.645499+0000","last_clean":"2024-09-02T10:36:29.645499+0000","last_became_active":"2024-09-02T10:35:57.707720+0000","last_became_peered":"2024-09-02T10:35:57.707720+0000","last_unstale":"2024-09-02T10:36:29.645499+0000","last_undegraded":"2024-09-02T10:36:29.645499+0000","last_fullsized":"2024-09-02T10:36:29.645499+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":5,"last_scrub":"959'6285","last_scrub_stamp":"2024-09-02T10:36:14.428021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:14.428021+0000","objects_scrubbed":1721,"log_size":6671,"log_dups_size":0,"ondisk_log_size":6671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:38.286207+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87228416,"num_objects":1331,"num_object_clones":0,"num_object_copies":3993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":390,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.16","version":"964'6739","reported_seq":14692,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:28.066602+0000","last_change":"2024-09-02T10:36:11.419544+0000","last_active":"2024-09-02T10:36:28.066602+0000","last_peered":"2024-09-02T10:36:28.066602+0000","last_clean":"2024-09-02T10:36:28.066602+0000","last_became_active":"2024-09-02T10:35:38.829158+0000","last_became_peered":"2024-09-02T10:35:38.829158+0000","last_unstale":"2024-09-02T10:36:28.066602+0000","last_undegraded":"2024-09-02T10:36:28.066602+0000","last_fullsized":"2024-09-02T10:36:28.066602+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'6289","last_scrub_stamp":"2024-09-02T10:36:11.419498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:11.419498+0000","objects_scrubbed":1805,"log_size":6739,"log_dups_size":0,"ondisk_log_size":6739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:29.611986+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87883776,"num_objects":1341,"num_object_clones":0,"num_object_copies":4023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":464,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1e","version":"22'92","reported_seq":136,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.450694+0000","last_change":"2024-09-02T10:36:23.524334+0000","last_active":"2024-09-02T10:36:27.450694+0000","last_peered":"2024-09-02T10:36:27.450694+0000","last_clean":"2024-09-02T10:36:27.450694+0000","last_became_active":"2024-09-02T10:36:23.523952+0000","last_became_peered":"2024-09-02T10:36:23.523952+0000","last_unstale":"2024-09-02T10:36:27.450694+0000","last_undegraded":"2024-09-02T10:36:27.450694+0000","last_fullsized":"2024-09-02T10:36:27.450694+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:29.260104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:29.260104+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,4],"acting":[8,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.15","version":"964'6707","reported_seq":12689,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:31.363048+0000","last_change":"2024-09-02T10:35:25.713754+0000","last_active":"2024-09-02T10:36:31.363048+0000","last_peered":"2024-09-02T10:36:31.363048+0000","last_clean":"2024-09-02T10:36:31.363048+0000","last_became_active":"2024-09-02T10:35:24.380270+0000","last_became_peered":"2024-09-02T10:35:24.380270+0000","last_unstale":"2024-09-02T10:36:31.363048+0000","last_undegraded":"2024-09-02T10:36:31.363048+0000","last_fullsized":"2024-09-02T10:36:31.363048+0000","mapping_epoch":941,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":942,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'4781","last_scrub_stamp":"2024-09-02T10:35:25.713545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:25.713545+0000","objects_scrubbed":2905,"log_size":6707,"log_dups_size":0,"ondisk_log_size":6707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:36.926480+0000","scrub_duration":554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78446592,"num_objects":1197,"num_object_clones":0,"num_object_copies":3591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1938,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,4],"acting":[7,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":502,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-02T10:36:24.589400+0000","last_change":"2024-09-02T10:36:09.745084+0000","last_active":"2024-09-02T10:36:24.589400+0000","last_peered":"2024-09-02T10:36:24.589400+0000","last_clean":"2024-09-02T10:36:24.589400+0000","last_became_active":"2024-09-02T10:33:38.659815+0000","last_became_peered":"2024-09-02T10:33:38.659815+0000","last_unstale":"2024-09-02T10:36:24.589400+0000","last_undegraded":"2024-09-02T10:36:24.589400+0000","last_fullsized":"2024-09-02T10:36:24.589400+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:09.745020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.745020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:31.345354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"964'6824","reported_seq":13713,"reported_epoch":964,"state":"active+remapped+backfilling","last_fresh":"2024-09-02T10:36:29.348741+0000","last_change":"2024-09-02T10:36:24.524027+0000","last_active":"2024-09-02T10:36:29.348741+0000","last_peered":"2024-09-02T10:36:29.348741+0000","last_clean":"2024-09-02T10:36:22.325557+0000","last_became_active":"2024-09-02T10:36:24.499594+0000","last_became_peered":"2024-09-02T10:36:24.499594+0000","last_unstale":"2024-09-02T10:36:29.348741+0000","last_undegraded":"2024-09-02T10:36:29.348741+0000","last_fullsized":"2024-09-02T10:36:29.348741+0000","mapping_epoch":962,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'6265","last_scrub_stamp":"2024-09-02T10:36:09.284280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.284280+0000","objects_scrubbed":1834,"log_size":6824,"log_dups_size":0,"ondisk_log_size":6824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82837504,"num_objects":1264,"num_object_clones":0,"num_object_copies":3792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":917,"num_objects_unfound":0,"num_objects_dirty":1264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":570,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":369,"num_bytes_recovered":24182784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,8],"acting":[3,1,10],"avail_no_missing":["3","1","4","10"],"object_location_counts":[{"shards":"1,3,10","objects":1264}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1624,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.450867+0000","last_change":"2024-09-02T10:36:21.119287+0000","last_active":"2024-09-02T10:36:27.450867+0000","last_peered":"2024-09-02T10:36:27.450867+0000","last_clean":"2024-09-02T10:36:27.450867+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:36:27.450867+0000","last_undegraded":"2024-09-02T10:36:27.450867+0000","last_fullsized":"2024-09-02T10:36:27.450867+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:21.119207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:21.119207+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:49.420928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"964'6956","reported_seq":13232,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.524145+0000","last_change":"2024-09-02T10:36:12.267636+0000","last_active":"2024-09-02T10:36:27.524145+0000","last_peered":"2024-09-02T10:36:27.524145+0000","last_clean":"2024-09-02T10:36:27.524145+0000","last_became_active":"2024-09-02T10:35:50.773879+0000","last_became_peered":"2024-09-02T10:35:50.773879+0000","last_unstale":"2024-09-02T10:36:27.524145+0000","last_undegraded":"2024-09-02T10:36:27.524145+0000","last_fullsized":"2024-09-02T10:36:27.524145+0000","mapping_epoch":949,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":950,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'6522","last_scrub_stamp":"2024-09-02T10:36:12.267481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:12.267481+0000","objects_scrubbed":1798,"log_size":6956,"log_dups_size":0,"ondisk_log_size":6956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:26.103198+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89260032,"num_objects":1362,"num_object_clones":0,"num_object_copies":4086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":436,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1600,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.461396+0000","last_change":"2024-09-02T10:36:09.169541+0000","last_active":"2024-09-02T10:36:27.461396+0000","last_peered":"2024-09-02T10:36:27.461396+0000","last_clean":"2024-09-02T10:36:27.461396+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:36:27.461396+0000","last_undegraded":"2024-09-02T10:36:27.461396+0000","last_fullsized":"2024-09-02T10:36:27.461396+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:09.169469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.169469+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:36.117296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.12","version":"964'6678","reported_seq":12698,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.682894+0000","last_change":"2024-09-02T10:35:24.913387+0000","last_active":"2024-09-02T10:36:27.682894+0000","last_peered":"2024-09-02T10:36:27.682894+0000","last_clean":"2024-09-02T10:36:27.682894+0000","last_became_active":"2024-09-02T10:35:22.670593+0000","last_became_peered":"2024-09-02T10:35:22.670593+0000","last_unstale":"2024-09-02T10:36:27.682894+0000","last_undegraded":"2024-09-02T10:36:27.682894+0000","last_fullsized":"2024-09-02T10:36:27.682894+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'4822","last_scrub_stamp":"2024-09-02T10:35:24.913280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:24.913280+0000","objects_scrubbed":2914,"log_size":6678,"log_dups_size":0,"ondisk_log_size":6678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:36.920495+0000","scrub_duration":1293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86507520,"num_objects":1320,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1876,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":919,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.450568+0000","last_change":"2024-09-02T10:36:16.494287+0000","last_active":"2024-09-02T10:36:27.450568+0000","last_peered":"2024-09-02T10:36:27.450568+0000","last_clean":"2024-09-02T10:36:27.450568+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:36:27.450568+0000","last_undegraded":"2024-09-02T10:36:27.450568+0000","last_fullsized":"2024-09-02T10:36:27.450568+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:16.494209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:16.494209+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:17.512138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"964'6565","reported_seq":10374,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:31.229715+0000","last_change":"2024-09-02T10:35:15.638534+0000","last_active":"2024-09-02T10:36:31.229715+0000","last_peered":"2024-09-02T10:36:31.229715+0000","last_clean":"2024-09-02T10:36:31.229715+0000","last_became_active":"2024-09-02T10:35:14.386404+0000","last_became_peered":"2024-09-02T10:35:14.386404+0000","last_unstale":"2024-09-02T10:36:31.229715+0000","last_undegraded":"2024-09-02T10:36:31.229715+0000","last_fullsized":"2024-09-02T10:36:31.229715+0000","mapping_epoch":933,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"935'4488","last_scrub_stamp":"2024-09-02T10:35:15.638360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:15.638360+0000","objects_scrubbed":2543,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:42.403420+0000","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77529088,"num_objects":1183,"num_object_clones":0,"num_object_copies":3549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2080,"num_write_kb":23040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":877,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.449394+0000","last_change":"2024-09-02T10:36:23.415443+0000","last_active":"2024-09-02T10:36:27.449394+0000","last_peered":"2024-09-02T10:36:27.449394+0000","last_clean":"2024-09-02T10:36:27.449394+0000","last_became_active":"2024-09-02T10:36:23.415088+0000","last_became_peered":"2024-09-02T10:36:23.415088+0000","last_unstale":"2024-09-02T10:36:27.449394+0000","last_undegraded":"2024-09-02T10:36:27.449394+0000","last_fullsized":"2024-09-02T10:36:27.449394+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.939026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.939026+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:32.736704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.10","version":"964'6941","reported_seq":15191,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:29.572218+0000","last_change":"2024-09-02T10:36:09.887048+0000","last_active":"2024-09-02T10:36:29.572218+0000","last_peered":"2024-09-02T10:36:29.572218+0000","last_clean":"2024-09-02T10:36:29.572218+0000","last_became_active":"2024-09-02T10:36:08.733746+0000","last_became_peered":"2024-09-02T10:36:08.733746+0000","last_unstale":"2024-09-02T10:36:29.572218+0000","last_undegraded":"2024-09-02T10:36:29.572218+0000","last_fullsized":"2024-09-02T10:36:29.572218+0000","mapping_epoch":956,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":957,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'6404","last_scrub_stamp":"2024-09-02T10:36:09.886760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.886760+0000","objects_scrubbed":1881,"log_size":6941,"log_dups_size":0,"ondisk_log_size":6941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:10.899840+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87818240,"num_objects":1340,"num_object_clones":0,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":541,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":1981,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.456303+0000","last_change":"2024-09-02T10:36:10.478222+0000","last_active":"2024-09-02T10:36:27.456303+0000","last_peered":"2024-09-02T10:36:27.456303+0000","last_clean":"2024-09-02T10:36:27.456303+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:36:27.456303+0000","last_undegraded":"2024-09-02T10:36:27.456303+0000","last_fullsized":"2024-09-02T10:36:27.456303+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:10.478148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.478148+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:30.586610+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1631,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.450786+0000","last_change":"2024-09-02T10:36:23.416629+0000","last_active":"2024-09-02T10:36:27.450786+0000","last_peered":"2024-09-02T10:36:27.450786+0000","last_clean":"2024-09-02T10:36:27.450786+0000","last_became_active":"2024-09-02T10:36:23.416310+0000","last_became_peered":"2024-09-02T10:36:23.416310+0000","last_unstale":"2024-09-02T10:36:27.450786+0000","last_undegraded":"2024-09-02T10:36:27.450786+0000","last_fullsized":"2024-09-02T10:36:27.450786+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:19.291427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:19.291427+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:32.660749+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,0],"acting":[8,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":183,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.449609+0000","last_change":"2024-09-02T10:36:15.323804+0000","last_active":"2024-09-02T10:36:27.449609+0000","last_peered":"2024-09-02T10:36:27.449609+0000","last_clean":"2024-09-02T10:36:27.449609+0000","last_became_active":"2024-09-02T10:33:38.657147+0000","last_became_peered":"2024-09-02T10:33:38.657147+0000","last_unstale":"2024-09-02T10:36:27.449609+0000","last_undegraded":"2024-09-02T10:36:27.449609+0000","last_fullsized":"2024-09-02T10:36:27.449609+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.323738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.323738+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:28.492236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1583,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.447544+0000","last_change":"2024-09-02T10:33:55.173514+0000","last_active":"2024-09-02T10:36:27.447544+0000","last_peered":"2024-09-02T10:36:27.447544+0000","last_clean":"2024-09-02T10:36:27.447544+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:36:27.447544+0000","last_undegraded":"2024-09-02T10:36:27.447544+0000","last_fullsized":"2024-09-02T10:36:27.447544+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:55.173450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:55.173450+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":833,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-02T10:36:24.586424+0000","last_change":"2024-09-02T10:36:13.719520+0000","last_active":"2024-09-02T10:36:24.586424+0000","last_peered":"2024-09-02T10:36:24.586424+0000","last_clean":"2024-09-02T10:36:24.586424+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:36:24.586424+0000","last_undegraded":"2024-09-02T10:36:24.586424+0000","last_fullsized":"2024-09-02T10:36:24.586424+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:13.719461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.719461+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:25.562912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":30,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.450131+0000","last_change":"2024-09-02T10:36:20.573861+0000","last_active":"2024-09-02T10:36:27.450131+0000","last_peered":"2024-09-02T10:36:27.450131+0000","last_clean":"2024-09-02T10:36:27.450131+0000","last_became_active":"2024-09-02T10:36:12.767657+0000","last_became_peered":"2024-09-02T10:36:12.767657+0000","last_unstale":"2024-09-02T10:36:27.450131+0000","last_undegraded":"2024-09-02T10:36:27.450131+0000","last_fullsized":"2024-09-02T10:36:27.450131+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:20.573798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:20.573798+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:48.205451+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1872,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.457891+0000","last_change":"2024-09-02T10:34:08.628331+0000","last_active":"2024-09-02T10:36:27.457891+0000","last_peered":"2024-09-02T10:36:27.457891+0000","last_clean":"2024-09-02T10:36:27.457891+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:36:27.457891+0000","last_undegraded":"2024-09-02T10:36:27.457891+0000","last_fullsized":"2024-09-02T10:36:27.457891+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.628247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.628247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"964'6895","reported_seq":14547,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:29.239585+0000","last_change":"2024-09-02T10:36:22.570216+0000","last_active":"2024-09-02T10:36:29.239585+0000","last_peered":"2024-09-02T10:36:29.239585+0000","last_clean":"2024-09-02T10:36:29.239585+0000","last_became_active":"2024-09-02T10:35:16.385649+0000","last_became_peered":"2024-09-02T10:35:16.385649+0000","last_unstale":"2024-09-02T10:36:29.239585+0000","last_undegraded":"2024-09-02T10:36:29.239585+0000","last_fullsized":"2024-09-02T10:36:29.239585+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":5,"last_scrub":"961'6769","last_scrub_stamp":"2024-09-02T10:36:22.570101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:22.570101+0000","objects_scrubbed":1466,"log_size":6895,"log_dups_size":0,"ondisk_log_size":6895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:29.453651+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87621632,"num_objects":1337,"num_object_clones":0,"num_object_copies":4011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2207,"num_write_kb":22848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":619,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.449507+0000","last_change":"2024-09-02T10:36:20.316217+0000","last_active":"2024-09-02T10:36:27.449507+0000","last_peered":"2024-09-02T10:36:27.449507+0000","last_clean":"2024-09-02T10:36:27.449507+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:36:27.449507+0000","last_undegraded":"2024-09-02T10:36:27.449507+0000","last_fullsized":"2024-09-02T10:36:27.449507+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:20.316139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:20.316139+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:21.453571+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"964'6877","reported_seq":11791,"reported_epoch":964,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:36:31.402254+0000","last_change":"2024-09-02T10:36:24.702513+0000","last_active":"2024-09-02T10:36:31.402254+0000","last_peered":"2024-09-02T10:36:31.402254+0000","last_clean":"2024-09-02T10:36:22.363425+0000","last_became_active":"2024-09-02T10:36:24.565256+0000","last_became_peered":"2024-09-02T10:36:24.565256+0000","last_unstale":"2024-09-02T10:36:31.402254+0000","last_undegraded":"2024-09-02T10:36:24.448539+0000","last_fullsized":"2024-09-02T10:36:24.413844+0000","mapping_epoch":962,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":932,"parent":"0.0","parent_split_bits":5,"last_scrub":"932'4480","last_scrub_stamp":"2024-09-02T10:35:09.830443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:09.830443+0000","objects_scrubbed":2522,"log_size":6877,"log_dups_size":0,"ondisk_log_size":6877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79560704,"num_objects":1214,"num_object_clones":0,"num_object_copies":3642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":995,"num_objects_unfound":0,"num_objects_dirty":1214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2406,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":24772608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5],"avail_no_missing":["7","5","10"],"object_location_counts":[{"shards":"5,7","objects":217},{"shards":"5,7,10","objects":995}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1857,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.456514+0000","last_change":"2024-09-02T10:36:23.417669+0000","last_active":"2024-09-02T10:36:27.456514+0000","last_peered":"2024-09-02T10:36:27.456514+0000","last_clean":"2024-09-02T10:36:27.456514+0000","last_became_active":"2024-09-02T10:36:23.417301+0000","last_became_peered":"2024-09-02T10:36:23.417301+0000","last_unstale":"2024-09-02T10:36:27.456514+0000","last_undegraded":"2024-09-02T10:36:27.456514+0000","last_fullsized":"2024-09-02T10:36:27.456514+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.409382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.409382+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:44.400602+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,8],"acting":[5,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"964'6753","reported_seq":6958,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.542285+0000","last_change":"2024-09-02T10:36:23.228909+0000","last_active":"2024-09-02T10:36:27.542285+0000","last_peered":"2024-09-02T10:36:27.542285+0000","last_clean":"2024-09-02T10:36:27.542285+0000","last_became_active":"2024-09-02T10:34:47.026177+0000","last_became_peered":"2024-09-02T10:34:47.026177+0000","last_unstale":"2024-09-02T10:36:27.542285+0000","last_undegraded":"2024-09-02T10:36:27.542285+0000","last_fullsized":"2024-09-02T10:36:27.542285+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"961'6703","last_scrub_stamp":"2024-09-02T10:36:23.228789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:23.228789+0000","objects_scrubbed":1368,"log_size":6753,"log_dups_size":0,"ondisk_log_size":6753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:35.431706+0000","scrub_duration":376,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86376448,"num_objects":1318,"num_object_clones":0,"num_object_copies":3954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1318,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2848,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,8],"acting":[6,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1936,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.456612+0000","last_change":"2024-09-02T10:34:07.590524+0000","last_active":"2024-09-02T10:36:27.456612+0000","last_peered":"2024-09-02T10:36:27.456612+0000","last_clean":"2024-09-02T10:36:27.456612+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:36:27.456612+0000","last_undegraded":"2024-09-02T10:36:27.456612+0000","last_fullsized":"2024-09-02T10:36:27.456612+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:07.590464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:07.590464+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"964'6724","reported_seq":6932,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:29.336821+0000","last_change":"2024-09-02T10:36:17.751426+0000","last_active":"2024-09-02T10:36:29.336821+0000","last_peered":"2024-09-02T10:36:29.336821+0000","last_clean":"2024-09-02T10:36:29.336821+0000","last_became_active":"2024-09-02T10:34:47.038024+0000","last_became_peered":"2024-09-02T10:34:47.038024+0000","last_unstale":"2024-09-02T10:36:29.336821+0000","last_undegraded":"2024-09-02T10:36:29.336821+0000","last_fullsized":"2024-09-02T10:36:29.336821+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'6457","last_scrub_stamp":"2024-09-02T10:36:17.751343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:17.751343+0000","objects_scrubbed":1534,"log_size":6724,"log_dups_size":0,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:27.232651+0000","scrub_duration":530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82182144,"num_objects":1254,"num_object_clones":0,"num_object_copies":3762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2963,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":2063,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.447197+0000","last_change":"2024-09-02T10:36:15.555892+0000","last_active":"2024-09-02T10:36:27.447197+0000","last_peered":"2024-09-02T10:36:27.447197+0000","last_clean":"2024-09-02T10:36:27.447197+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:36:27.447197+0000","last_undegraded":"2024-09-02T10:36:27.447197+0000","last_fullsized":"2024-09-02T10:36:27.447197+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.555822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.555822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:33.398682+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"964'6760","reported_seq":6967,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:29.340359+0000","last_change":"2024-09-02T10:36:13.627438+0000","last_active":"2024-09-02T10:36:29.340359+0000","last_peered":"2024-09-02T10:36:29.340359+0000","last_clean":"2024-09-02T10:36:29.340359+0000","last_became_active":"2024-09-02T10:34:47.038074+0000","last_became_peered":"2024-09-02T10:34:47.038074+0000","last_unstale":"2024-09-02T10:36:29.340359+0000","last_undegraded":"2024-09-02T10:36:29.340359+0000","last_fullsized":"2024-09-02T10:36:29.340359+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'6355","last_scrub_stamp":"2024-09-02T10:36:13.627233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.627233+0000","objects_scrubbed":1695,"log_size":6760,"log_dups_size":0,"ondisk_log_size":6760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:22.911073+0000","scrub_duration":452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84279296,"num_objects":1286,"num_object_clones":0,"num_object_copies":3858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2934,"num_write_kb":72576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1721,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.452645+0000","last_change":"2024-09-02T10:36:10.022054+0000","last_active":"2024-09-02T10:36:27.452645+0000","last_peered":"2024-09-02T10:36:27.452645+0000","last_clean":"2024-09-02T10:36:27.452645+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:36:27.452645+0000","last_undegraded":"2024-09-02T10:36:27.452645+0000","last_fullsized":"2024-09-02T10:36:27.452645+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:10.021994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.021994+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:36.323969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"964'6874","reported_seq":7079,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:28.007319+0000","last_change":"2024-09-02T10:36:16.376436+0000","last_active":"2024-09-02T10:36:28.007319+0000","last_peered":"2024-09-02T10:36:28.007319+0000","last_clean":"2024-09-02T10:36:28.007319+0000","last_became_active":"2024-09-02T10:34:47.042672+0000","last_became_peered":"2024-09-02T10:34:47.042672+0000","last_unstale":"2024-09-02T10:36:28.007319+0000","last_undegraded":"2024-09-02T10:36:28.007319+0000","last_fullsized":"2024-09-02T10:36:28.007319+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'6561","last_scrub_stamp":"2024-09-02T10:36:16.376239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:16.376239+0000","objects_scrubbed":1576,"log_size":6874,"log_dups_size":0,"ondisk_log_size":6874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:24.146392+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82575360,"num_objects":1260,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":69504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":698,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.449521+0000","last_change":"2024-09-02T10:36:13.308543+0000","last_active":"2024-09-02T10:36:27.449521+0000","last_peered":"2024-09-02T10:36:27.449521+0000","last_clean":"2024-09-02T10:36:27.449521+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:36:27.449521+0000","last_undegraded":"2024-09-02T10:36:27.449521+0000","last_fullsized":"2024-09-02T10:36:27.449521+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:13.308459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.308459+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:42.491914+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"964'9505","reported_seq":9696,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:29.345873+0000","last_change":"2024-09-02T10:36:13.014102+0000","last_active":"2024-09-02T10:36:29.345873+0000","last_peered":"2024-09-02T10:36:29.345873+0000","last_clean":"2024-09-02T10:36:29.345873+0000","last_became_active":"2024-09-02T10:33:52.760804+0000","last_became_peered":"2024-09-02T10:33:52.760804+0000","last_unstale":"2024-09-02T10:36:29.345873+0000","last_undegraded":"2024-09-02T10:36:29.345873+0000","last_fullsized":"2024-09-02T10:36:29.345873+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'8682","last_scrub_stamp":"2024-09-02T10:36:13.014038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.014038+0000","objects_scrubbed":3370,"log_size":9505,"log_dups_size":0,"ondisk_log_size":9505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:31.690235+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165085184,"num_objects":2519,"num_object_clones":0,"num_object_copies":7557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9505,"num_write_kb":384768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1020,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.450758+0000","last_change":"2024-09-02T10:34:17.882133+0000","last_active":"2024-09-02T10:36:27.450758+0000","last_peered":"2024-09-02T10:36:27.450758+0000","last_clean":"2024-09-02T10:36:27.450758+0000","last_became_active":"2024-09-02T10:34:17.881745+0000","last_became_peered":"2024-09-02T10:34:17.881745+0000","last_unstale":"2024-09-02T10:36:27.450758+0000","last_undegraded":"2024-09-02T10:36:27.450758+0000","last_fullsized":"2024-09-02T10:36:27.450758+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:33:58.800694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:33:58.800694+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"964'9629","reported_seq":9823,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:31.353849+0000","last_change":"2024-09-02T10:36:12.744199+0000","last_active":"2024-09-02T10:36:31.353849+0000","last_peered":"2024-09-02T10:36:31.353849+0000","last_clean":"2024-09-02T10:36:31.353849+0000","last_became_active":"2024-09-02T10:33:52.926467+0000","last_became_peered":"2024-09-02T10:33:52.926467+0000","last_unstale":"2024-09-02T10:36:31.353849+0000","last_undegraded":"2024-09-02T10:36:31.353849+0000","last_fullsized":"2024-09-02T10:36:31.353849+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'8646","last_scrub_stamp":"2024-09-02T10:36:12.744115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:12.744115+0000","objects_scrubbed":3403,"log_size":9629,"log_dups_size":0,"ondisk_log_size":9629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:13.209545+0000","scrub_duration":1221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155713560,"num_objects":2377,"num_object_clones":0,"num_object_copies":7131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2377,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9629,"num_write_kb":384129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":672,"reported_epoch":963,"state":"active+clean","last_fresh":"2024-09-02T10:36:24.589558+0000","last_change":"2024-09-02T10:36:23.414090+0000","last_active":"2024-09-02T10:36:24.589558+0000","last_peered":"2024-09-02T10:36:24.589558+0000","last_clean":"2024-09-02T10:36:24.589558+0000","last_became_active":"2024-09-02T10:36:23.413619+0000","last_became_peered":"2024-09-02T10:36:23.413619+0000","last_unstale":"2024-09-02T10:36:24.589558+0000","last_undegraded":"2024-09-02T10:36:24.589558+0000","last_fullsized":"2024-09-02T10:36:24.589558+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:20.991722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:20.991722+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:46.019731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.e","version":"964'9651","reported_seq":10227,"reported_epoch":964,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:36:31.410669+0000","last_change":"2024-09-02T10:36:24.866151+0000","last_active":"2024-09-02T10:36:31.410669+0000","last_peered":"2024-09-02T10:36:31.410669+0000","last_clean":"2024-09-02T10:36:22.396656+0000","last_became_active":"2024-09-02T10:36:24.746283+0000","last_became_peered":"2024-09-02T10:36:24.746283+0000","last_unstale":"2024-09-02T10:36:31.410669+0000","last_undegraded":"2024-09-02T10:36:24.464398+0000","last_fullsized":"2024-09-02T10:36:24.413716+0000","mapping_epoch":962,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":9651,"log_dups_size":0,"ondisk_log_size":9651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163905536,"num_objects":2501,"num_object_clones":0,"num_object_copies":7503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":96,"num_objects_misplaced":2498,"num_objects_unfound":0,"num_objects_dirty":2501,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9651,"num_write_kb":388864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":212,"num_bytes_recovered":13500416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,9],"acting":[7,1],"avail_no_missing":["7","1","10"],"object_location_counts":[{"shards":"1,7","objects":-93},{"shards":"1,7,10","objects":2498}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":264,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.458496+0000","last_change":"2024-09-02T10:36:23.524403+0000","last_active":"2024-09-02T10:36:27.458496+0000","last_peered":"2024-09-02T10:36:27.458496+0000","last_clean":"2024-09-02T10:36:27.458496+0000","last_became_active":"2024-09-02T10:36:23.524017+0000","last_became_peered":"2024-09-02T10:36:23.524017+0000","last_unstale":"2024-09-02T10:36:27.458496+0000","last_undegraded":"2024-09-02T10:36:27.458496+0000","last_fullsized":"2024-09-02T10:36:27.458496+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:17.904033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:17.904033+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:25.121539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,4],"acting":[9,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.f","version":"964'9653","reported_seq":9838,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.716629+0000","last_change":"2024-09-02T10:36:25.554143+0000","last_active":"2024-09-02T10:36:27.716629+0000","last_peered":"2024-09-02T10:36:27.716629+0000","last_clean":"2024-09-02T10:36:27.716629+0000","last_became_active":"2024-09-02T10:33:52.925153+0000","last_became_peered":"2024-09-02T10:33:52.925153+0000","last_unstale":"2024-09-02T10:36:27.716629+0000","last_undegraded":"2024-09-02T10:36:27.716629+0000","last_fullsized":"2024-09-02T10:36:27.716629+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"963'9547","last_scrub_stamp":"2024-09-02T10:36:25.553889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:25.553889+0000","objects_scrubbed":2782,"log_size":9653,"log_dups_size":0,"ondisk_log_size":9653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:45.328325+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173735936,"num_objects":2651,"num_object_clones":0,"num_object_copies":7953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2651,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9653,"num_write_kb":393728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1246,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.452361+0000","last_change":"2024-09-02T10:36:23.415668+0000","last_active":"2024-09-02T10:36:27.452361+0000","last_peered":"2024-09-02T10:36:27.452361+0000","last_clean":"2024-09-02T10:36:27.452361+0000","last_became_active":"2024-09-02T10:36:23.415265+0000","last_became_peered":"2024-09-02T10:36:23.415265+0000","last_unstale":"2024-09-02T10:36:27.452361+0000","last_undegraded":"2024-09-02T10:36:27.452361+0000","last_fullsized":"2024-09-02T10:36:27.452361+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:18.936502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:18.936502+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:37.944518+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"964'6647","reported_seq":10672,"reported_epoch":964,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:36:27.759462+0000","last_change":"2024-09-02T10:36:24.681132+0000","last_active":"2024-09-02T10:36:27.759462+0000","last_peered":"2024-09-02T10:36:27.759462+0000","last_clean":"2024-09-02T10:36:22.353657+0000","last_became_active":"2024-09-02T10:36:24.666634+0000","last_became_peered":"2024-09-02T10:36:24.666634+0000","last_unstale":"2024-09-02T10:36:27.759462+0000","last_undegraded":"2024-09-02T10:36:24.441472+0000","last_fullsized":"2024-09-02T10:36:24.414977+0000","mapping_epoch":962,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'3857","last_scrub_stamp":"2024-09-02T10:34:49.043221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.043221+0000","objects_scrubbed":1904,"log_size":6647,"log_dups_size":0,"ondisk_log_size":6647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82968576,"num_objects":1266,"num_object_clones":0,"num_object_copies":3798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1141,"num_objects_unfound":0,"num_objects_dirty":1266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2790,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":13107200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","3","10"],"object_location_counts":[{"shards":"3,5","objects":124},{"shards":"3,5,10","objects":1141}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":925,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.452529+0000","last_change":"2024-09-02T10:36:24.088105+0000","last_active":"2024-09-02T10:36:27.452529+0000","last_peered":"2024-09-02T10:36:27.452529+0000","last_clean":"2024-09-02T10:36:27.452529+0000","last_became_active":"2024-09-02T10:36:23.418656+0000","last_became_peered":"2024-09-02T10:36:23.418656+0000","last_unstale":"2024-09-02T10:36:27.452529+0000","last_undegraded":"2024-09-02T10:36:27.452529+0000","last_fullsized":"2024-09-02T10:36:27.452529+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:24.088035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:24.088035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:31.649797+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2],"acting":[9,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.a","version":"964'9706","reported_seq":9900,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.753748+0000","last_change":"2024-09-02T10:36:10.561335+0000","last_active":"2024-09-02T10:36:27.753748+0000","last_peered":"2024-09-02T10:36:27.753748+0000","last_clean":"2024-09-02T10:36:27.753748+0000","last_became_active":"2024-09-02T10:33:52.925177+0000","last_became_peered":"2024-09-02T10:33:52.925177+0000","last_unstale":"2024-09-02T10:36:27.753748+0000","last_undegraded":"2024-09-02T10:36:27.753748+0000","last_fullsized":"2024-09-02T10:36:27.753748+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"957'8761","last_scrub_stamp":"2024-09-02T10:36:10.561133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.561133+0000","objects_scrubbed":3627,"log_size":9706,"log_dups_size":0,"ondisk_log_size":9706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:21.163961+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173670400,"num_objects":2650,"num_object_clones":0,"num_object_copies":7950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9706,"num_write_kb":395392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1705,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.448920+0000","last_change":"2024-09-02T10:34:04.115873+0000","last_active":"2024-09-02T10:36:27.448920+0000","last_peered":"2024-09-02T10:36:27.448920+0000","last_clean":"2024-09-02T10:36:27.448920+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:36:27.448920+0000","last_undegraded":"2024-09-02T10:36:27.448920+0000","last_fullsized":"2024-09-02T10:36:27.448920+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:04.115830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:04.115830+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"964'9644","reported_seq":10267,"reported_epoch":964,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:36:29.730046+0000","last_change":"2024-09-02T10:36:24.790188+0000","last_active":"2024-09-02T10:36:29.730046+0000","last_peered":"2024-09-02T10:36:29.730046+0000","last_clean":"2024-09-02T10:36:22.360171+0000","last_became_active":"2024-09-02T10:36:24.713808+0000","last_became_peered":"2024-09-02T10:36:24.713808+0000","last_unstale":"2024-09-02T10:36:29.730046+0000","last_undegraded":"2024-09-02T10:36:24.458271+0000","last_fullsized":"2024-09-02T10:36:24.413775+0000","mapping_epoch":962,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'9386","last_scrub_stamp":"2024-09-02T10:36:21.798044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:21.798044+0000","objects_scrubbed":2871,"log_size":9644,"log_dups_size":0,"ondisk_log_size":9644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168034304,"num_objects":2564,"num_object_clones":0,"num_object_copies":7692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19,"num_objects_misplaced":2556,"num_objects_unfound":0,"num_objects_dirty":2564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9644,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":215,"num_bytes_recovered":14024704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,1],"acting":[8,5],"avail_no_missing":["8","5","10"],"object_location_counts":[{"shards":"5,8","objects":-11},{"shards":"5,8,10","objects":2556}],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1005,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.452313+0000","last_change":"2024-09-02T10:36:13.900889+0000","last_active":"2024-09-02T10:36:27.452313+0000","last_peered":"2024-09-02T10:36:27.452313+0000","last_clean":"2024-09-02T10:36:27.452313+0000","last_became_active":"2024-09-02T10:33:38.664311+0000","last_became_peered":"2024-09-02T10:33:38.664311+0000","last_unstale":"2024-09-02T10:36:27.452313+0000","last_undegraded":"2024-09-02T10:36:27.452313+0000","last_fullsized":"2024-09-02T10:36:27.452313+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:13.900825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.900825+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:40.109001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"964'6898","reported_seq":7109,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:28.101115+0000","last_change":"2024-09-02T10:36:18.281248+0000","last_active":"2024-09-02T10:36:28.101115+0000","last_peered":"2024-09-02T10:36:28.101115+0000","last_clean":"2024-09-02T10:36:28.101115+0000","last_became_active":"2024-09-02T10:34:47.044129+0000","last_became_peered":"2024-09-02T10:34:47.044129+0000","last_unstale":"2024-09-02T10:36:28.101115+0000","last_undegraded":"2024-09-02T10:36:28.101115+0000","last_fullsized":"2024-09-02T10:36:28.101115+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'6666","last_scrub_stamp":"2024-09-02T10:36:18.281061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:18.281061+0000","objects_scrubbed":1568,"log_size":6898,"log_dups_size":0,"ondisk_log_size":6898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:34.149765+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87228416,"num_objects":1331,"num_object_clones":0,"num_object_copies":3993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2972,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1715,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.448102+0000","last_change":"2024-09-02T10:34:01.291835+0000","last_active":"2024-09-02T10:36:27.448102+0000","last_peered":"2024-09-02T10:36:27.448102+0000","last_clean":"2024-09-02T10:36:27.448102+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:36:27.448102+0000","last_undegraded":"2024-09-02T10:36:27.448102+0000","last_fullsized":"2024-09-02T10:36:27.448102+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.291779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.291779+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"964'6677","reported_seq":7633,"reported_epoch":964,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:36:31.402252+0000","last_change":"2024-09-02T10:36:24.737575+0000","last_active":"2024-09-02T10:36:31.402252+0000","last_peered":"2024-09-02T10:36:31.402252+0000","last_clean":"2024-09-02T10:36:22.388595+0000","last_became_active":"2024-09-02T10:36:24.702299+0000","last_became_peered":"2024-09-02T10:36:24.702299+0000","last_unstale":"2024-09-02T10:36:31.402252+0000","last_undegraded":"2024-09-02T10:36:24.445893+0000","last_fullsized":"2024-09-02T10:36:24.413355+0000","mapping_epoch":962,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3753","last_scrub_stamp":"2024-09-02T10:34:49.259057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.259057+0000","objects_scrubbed":1910,"log_size":6677,"log_dups_size":0,"ondisk_log_size":6677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79429632,"num_objects":1212,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":996,"num_objects_unfound":0,"num_objects_dirty":1212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2924,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":384,"num_bytes_recovered":25165824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3],"avail_no_missing":["7","3","10"],"object_location_counts":[{"shards":"3,7","objects":212},{"shards":"3,7,10","objects":996}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":1979,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.452490+0000","last_change":"2024-09-02T10:36:09.993500+0000","last_active":"2024-09-02T10:36:27.452490+0000","last_peered":"2024-09-02T10:36:27.452490+0000","last_clean":"2024-09-02T10:36:27.452490+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:36:27.452490+0000","last_undegraded":"2024-09-02T10:36:27.452490+0000","last_fullsized":"2024-09-02T10:36:27.452490+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:09.993425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.993425+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:29.323758+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"964'6687","reported_seq":7183,"reported_epoch":964,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:36:27.752241+0000","last_change":"2024-09-02T10:36:24.739977+0000","last_active":"2024-09-02T10:36:27.752241+0000","last_peered":"2024-09-02T10:36:27.752241+0000","last_clean":"2024-09-02T10:36:22.372231+0000","last_became_active":"2024-09-02T10:36:24.565316+0000","last_became_peered":"2024-09-02T10:36:24.565316+0000","last_unstale":"2024-09-02T10:36:27.752241+0000","last_undegraded":"2024-09-02T10:36:24.442627+0000","last_fullsized":"2024-09-02T10:36:24.413905+0000","mapping_epoch":962,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'3847","last_scrub_stamp":"2024-09-02T10:34:50.216498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:50.216498+0000","objects_scrubbed":1961,"log_size":6687,"log_dups_size":0,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83623936,"num_objects":1276,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1185,"num_objects_unfound":0,"num_objects_dirty":1276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2847,"num_write_kb":69184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":153,"num_bytes_recovered":10027008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1],"avail_no_missing":["5","1","10"],"object_location_counts":[{"shards":"1,5","objects":89},{"shards":"1,5,10","objects":1185}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":927,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.450403+0000","last_change":"2024-09-02T10:36:17.472218+0000","last_active":"2024-09-02T10:36:27.450403+0000","last_peered":"2024-09-02T10:36:27.450403+0000","last_clean":"2024-09-02T10:36:27.450403+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:36:27.450403+0000","last_undegraded":"2024-09-02T10:36:27.450403+0000","last_fullsized":"2024-09-02T10:36:27.450403+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:17.472148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:17.472148+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:17.539247+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"964'6692","reported_seq":6875,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.633659+0000","last_change":"2024-09-02T10:34:49.168775+0000","last_active":"2024-09-02T10:36:27.633659+0000","last_peered":"2024-09-02T10:36:27.633659+0000","last_clean":"2024-09-02T10:36:27.633659+0000","last_became_active":"2024-09-02T10:34:47.049681+0000","last_became_peered":"2024-09-02T10:34:47.049681+0000","last_unstale":"2024-09-02T10:36:27.633659+0000","last_undegraded":"2024-09-02T10:36:27.633659+0000","last_fullsized":"2024-09-02T10:36:27.633659+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3787","last_scrub_stamp":"2024-09-02T10:34:49.168686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.168686+0000","objects_scrubbed":1913,"log_size":6692,"log_dups_size":0,"ondisk_log_size":6692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90963968,"num_objects":1388,"num_object_clones":0,"num_object_copies":4164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1388,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2905,"num_write_kb":76160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1743,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.447672+0000","last_change":"2024-09-02T10:36:16.543124+0000","last_active":"2024-09-02T10:36:27.447672+0000","last_peered":"2024-09-02T10:36:27.447672+0000","last_clean":"2024-09-02T10:36:27.447672+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:36:27.447672+0000","last_undegraded":"2024-09-02T10:36:27.447672+0000","last_fullsized":"2024-09-02T10:36:27.447672+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:16.543059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:16.543059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:17.166860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"964'6792","reported_seq":6977,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:29.651234+0000","last_change":"2024-09-02T10:34:48.713406+0000","last_active":"2024-09-02T10:36:29.651234+0000","last_peered":"2024-09-02T10:36:29.651234+0000","last_clean":"2024-09-02T10:36:29.651234+0000","last_became_active":"2024-09-02T10:34:47.029634+0000","last_became_peered":"2024-09-02T10:34:47.029634+0000","last_unstale":"2024-09-02T10:36:29.651234+0000","last_undegraded":"2024-09-02T10:36:29.651234+0000","last_fullsized":"2024-09-02T10:36:29.651234+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3863","last_scrub_stamp":"2024-09-02T10:34:48.713331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:48.713331+0000","objects_scrubbed":1931,"log_size":6792,"log_dups_size":0,"ondisk_log_size":6792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:36:34.473070+0000","scrub_duration":204,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85065728,"num_objects":1298,"num_object_clones":0,"num_object_copies":3894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2929,"num_write_kb":73472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":1142,"reported_epoch":964,"state":"active+clean","last_fresh":"2024-09-02T10:36:27.452685+0000","last_change":"2024-09-02T10:36:10.301898+0000","last_active":"2024-09-02T10:36:27.452685+0000","last_peered":"2024-09-02T10:36:27.452685+0000","last_clean":"2024-09-02T10:36:27.452685+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:36:27.452685+0000","last_undegraded":"2024-09-02T10:36:27.452685+0000","last_fullsized":"2024-09-02T10:36:27.452685+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:10.301844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.301844+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:18.968378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":2689531928,"num_objects":41040,"num_object_clones":0,"num_object_copies":123120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":124,"num_objects_misplaced":10288,"num_objects_unfound":0,"num_objects_dirty":41040,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":99716,"num_write_kb":3151489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1911,"num_bytes_recovered":124780544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8687136768,"data_stored":8687124552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":193145,"ondisk_log_size":193145,"up":78,"acting":72,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":343597384116,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":0,"seq":313532613049,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841982,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1692676,"kb_used_data":759092,"kb_used_omap":1,"kb_used_meta":933566,"kb_avail":92052476,"statfs":{"total":95995035648,"available":94261735424,"internally_reserved":0,"allocated":777310208,"data_stored":777017256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":955972032},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070918,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1175684,"kb_used_data":634996,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92569468,"statfs":{"total":95995035648,"available":94791135232,"internally_reserved":0,"allocated":650235904,"data_stored":649942952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299850,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1912848,"kb_used_data":1101440,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":91832304,"statfs":{"total":95995035648,"available":94036279296,"internally_reserved":0,"allocated":1127874560,"data_stored":1127573984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":830863818},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,6,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528782,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1110392,"kb_used_data":523880,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":92634760,"statfs":{"total":95995035648,"available":94857994240,"internally_reserved":0,"allocated":536453120,"data_stored":536164410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":600570311},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503725010,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2391876,"kb_used_data":1440180,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":91353276,"statfs":{"total":95995035648,"available":93545754624,"internally_reserved":0,"allocated":1474744320,"data_stored":1474447986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953943,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1032456,"kb_used_data":433460,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92712696,"statfs":{"total":95995035648,"available":94937800704,"internally_reserved":0,"allocated":443863040,"data_stored":443570088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150173,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1735688,"kb_used_data":820084,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92009464,"statfs":{"total":95995035648,"available":94217691136,"internally_reserved":0,"allocated":839766016,"data_stored":839473064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":937490888},"hb_peers":[0,1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411808,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":710144,"kb_used_data":93104,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93035008,"statfs":{"total":95995035648,"available":95267848192,"internally_reserved":0,"allocated":95338496,"data_stored":95046234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575334,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2385420,"kb_used_data":1487864,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":91359732,"statfs":{"total":95995035648,"available":93552365568,"internally_reserved":0,"allocated":1523572736,"data_stored":1523275712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738859,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1311548,"kb_used_data":351592,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92433604,"statfs":{"total":95995035648,"available":94652010496,"internally_reserved":0,"allocated":360030208,"data_stored":359741498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":982972869},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":355860480,"data_stored":355860480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1508380672,"data_stored":1508376600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94568448,"data_stored":94568448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":838402048,"data_stored":838402048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":441843712,"data_stored":441843712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1466372096,"data_stored":1466368024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":529137664,"data_stored":529137664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1108480000,"data_stored":1108475928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":646316032,"data_stored":646316032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":775159808,"data_stored":775159808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":810483712,"data_stored":810483712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":112132096,"data_stored":112132096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:36:34.694 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-02T10:36:34.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pg_num 71 2024-09-02T10:36:34.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:34.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:34.738 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:34.739 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:34 smithi187 ceph-mon[20072]: pgmap v2212: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 629 op/s; 124/118860 objects degraded (0.104%); 9741/118860 objects misplaced (8.195%); 19 MiB/s, 307 objects/s recovering 2024-09-02T10:36:34.757 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:34.757 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:34.840 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:34 smithi162 ceph-mon[20279]: pgmap v2212: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.4 GiB data, 15 GiB used, 879 GiB / 894 GiB avail; 0 B/s wr, 629 op/s; 124/118860 objects degraded (0.104%); 9741/118860 objects misplaced (8.195%); 19 MiB/s, 307 objects/s recovering 2024-09-02T10:36:34.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:34.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:34.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:35.000 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:35.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:35.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:35.127 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:35.128 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:35.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:35.257 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:35.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:35.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:35.359 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:35.383 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:35.384 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:35.430 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:36:35.430 INFO:teuthology.orchestra.run.smithi151.stdout: "ops": [ 2024-09-02T10:36:35.430 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.430 INFO:teuthology.orchestra.run.smithi151.stdout: "description": "osd_op(client.28030.0:159357 9.c 9:3114d6de:::benchmark_data_smithi187_154927_object61245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-09-02T10:36:35.430 INFO:teuthology.orchestra.run.smithi151.stdout: "initiated_at": "2024-09-02T10:36:35.421092+0000", 2024-09-02T10:36:35.430 INFO:teuthology.orchestra.run.smithi151.stdout: "age": 0.007410926, 2024-09-02T10:36:35.430 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0074425790000000004, 2024-09-02T10:36:35.430 INFO:teuthology.orchestra.run.smithi151.stdout: "continuous": false, 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: "type_data": { 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client_info": { 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client": "client.28030", 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: "tid": 159357 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: "events": [ 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "initiated", 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.421092+0000", 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.431 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "header_read", 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.421092+0000", 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "throttled", 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.421093+0000", 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.3340000000000001e-06 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "all_read", 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.421099+0000", 2024-09-02T10:36:35.432 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 5.9479999999999996e-06 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "dispatched", 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.421114+0000", 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 1.5537999999999999e-05 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "queued_for_pg", 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.421122+0000", 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.357e-06 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "reached_pg", 2024-09-02T10:36:35.433 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.421198+0000", 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 7.5956000000000003e-05 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "started", 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.421346+0000", 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.000148197 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "waiting for subops from 4,9", 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.421473+0000", 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00012735700000000001 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.434 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "op_commit", 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.426830+0000", 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.0053568009999999996 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: "time": "2024-09-02T10:36:35.427186+0000", 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: "duration": 0.00035578900000000002 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:36:35.435 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:36:35.436 INFO:teuthology.orchestra.run.smithi151.stdout: "num_ops": 1 2024-09-02T10:36:35.436 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:36:35.446 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:36:35.485 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:35.518 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:35.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:35.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:35.646 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4006241350' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:35.660 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:35.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4006241350' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:35.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:35.797 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:35.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:35.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4006241350' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:35.899 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:35.926 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:35.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:35.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:36.027 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:36.054 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:36.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:36.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:36.156 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:36.190 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:36.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:36.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:36.315 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:36.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:36.417 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:36.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:36.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:36.497 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:36.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:36.568 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:36.569 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:36.670 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:36.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:36 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:36.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4006241350' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:36.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:36 smithi151 ceph-mon[16639]: osdmap e966: 12 total, 12 up, 10 in 2024-09-02T10:36:36.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:36 smithi151 ceph-mon[16639]: pgmap v2214: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 689 op/s; 132/115380 objects degraded (0.114%); 8865/115380 objects misplaced (7.683%); 24 MiB/s, 380 objects/s recovering 2024-09-02T10:36:36.704 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:36.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:36 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4006241350' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:36 smithi187 ceph-mon[20072]: osdmap e966: 12 total, 12 up, 10 in 2024-09-02T10:36:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:36 smithi187 ceph-mon[20072]: pgmap v2214: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 689 op/s; 132/115380 objects degraded (0.114%); 8865/115380 objects misplaced (7.683%); 24 MiB/s, 380 objects/s recovering 2024-09-02T10:36:36.806 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:36.839 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:36.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:36.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:36 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4006241350' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:36 smithi162 ceph-mon[20279]: osdmap e966: 12 total, 12 up, 10 in 2024-09-02T10:36:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:36 smithi162 ceph-mon[20279]: pgmap v2214: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 689 op/s; 132/115380 objects degraded (0.114%); 8865/115380 objects misplaced (7.683%); 24 MiB/s, 380 objects/s recovering 2024-09-02T10:36:36.941 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:36.965 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:36.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:36.966 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:37.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-02T10:36:37.067 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:37.091 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:37.092 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:37.192 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:37.222 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:37.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:37.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:37.347 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:37.349 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:37.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:37.476 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:37.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:37.578 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:37.615 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:37.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:37.642 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:36:37.642 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:36:37.674 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:36:37.698 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:36:37.698 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:36:37.716 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:37.725 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:36:37.725 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:36:37.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:37.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:37.752 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:36:37.752 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:36:37.789 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:36:37.789 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:36:37.819 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:36:37.819 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:36:37.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:37.852 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:36:37.853 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:36:37.869 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:37.870 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:37.880 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:36:37.880 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:36:37.909 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:36:37.909 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:36:37.939 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:36:37.939 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:36:37.968 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:36:37.968 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:36:37.971 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:38.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:38.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:38.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:38.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:38.155 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:38.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:38.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:38.257 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:38.285 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:38.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:38.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:38.387 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:38.414 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pg_num 71 < pg_num_max 32 2024-09-02T10:36:38.418 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:38.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:38.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:38.498 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:36:38.519 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:38.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:38.567 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:38.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:38.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:38 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:38 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:38 smithi151 ceph-mon[16639]: osdmap e967: 12 total, 12 up, 10 in 2024-09-02T10:36:38.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1095771039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:38.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:38 smithi151 ceph-mon[16639]: pgmap v2216: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 727 op/s; 188/112299 objects degraded (0.167%); 8058/112299 objects misplaced (7.175%); 29 MiB/s, 460 objects/s recovering 2024-09-02T10:36:38.689 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:38 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/658963193' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "71"} : dispatch 2024-09-02T10:36:38.695 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:38.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:38.696 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:38.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:38 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:38.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:38 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:38.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:38 smithi187 ceph-mon[20072]: osdmap e967: 12 total, 12 up, 10 in 2024-09-02T10:36:38.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1095771039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:38.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:38 smithi187 ceph-mon[20072]: pgmap v2216: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 727 op/s; 188/112299 objects degraded (0.167%); 8058/112299 objects misplaced (7.175%); 29 MiB/s, 460 objects/s recovering 2024-09-02T10:36:38.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:38 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/658963193' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "71"} : dispatch 2024-09-02T10:36:38.797 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:38.819 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:38.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:38.820 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:38 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:38 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:38 smithi162 ceph-mon[20279]: osdmap e967: 12 total, 12 up, 10 in 2024-09-02T10:36:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1095771039' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:38 smithi162 ceph-mon[20279]: pgmap v2216: 57 pgs: 1 active+remapped+backfilling, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.3 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 727 op/s; 188/112299 objects degraded (0.167%); 8058/112299 objects misplaced (7.175%); 29 MiB/s, 460 objects/s recovering 2024-09-02T10:36:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:38 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/658963193' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "71"} : dispatch 2024-09-02T10:36:38.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:38.944 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:38.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:38.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:39.046 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:39.073 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:39.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:39.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:39.175 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:39.204 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:39.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:39.306 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:39.317 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:39.329 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:39.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:39.431 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:39.459 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:39.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:39.561 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:39.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:39.589 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:39.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:39.690 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:39.735 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:39.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:39.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:39 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 188/112299 objects degraded (0.167%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:39.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:39 smithi187 ceph-mon[20072]: osdmap e968: 12 total, 12 up, 10 in 2024-09-02T10:36:39.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:36:39.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:36:39.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:36:39.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:36:39.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:39 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:39 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 188/112299 objects degraded (0.167%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:39.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:39 smithi151 ceph-mon[16639]: osdmap e968: 12 total, 12 up, 10 in 2024-09-02T10:36:39.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:36:39.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:36:39.798 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:36:39.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:36:39.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.799 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:39 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.836 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:39.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:39 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 188/112299 objects degraded (0.167%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:39.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:39 smithi162 ceph-mon[20279]: osdmap e968: 12 total, 12 up, 10 in 2024-09-02T10:36:39.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:36:39.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:36:39.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:36:39.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:36:39.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:39 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:39.873 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:39.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:39.951 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:36:39.952 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:36:40.023 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:40.047 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:40.048 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:40.149 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:40.190 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:40.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:40.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:40.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:40.324 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:40.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:40.428 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:40.452 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:40.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:40.555 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:40.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:40.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:40.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:40.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:40.710 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:40.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:40.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:40 smithi187 ceph-mon[20072]: osdmap e969: 12 total, 12 up, 10 in 2024-09-02T10:36:40.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:40 smithi187 ceph-mon[20072]: pgmap v2219: 57 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 686 op/s; 188/106785 objects degraded (0.176%); 7225/106785 objects misplaced (6.766%); 24 MiB/s, 377 objects/s recovering 2024-09-02T10:36:40.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:40.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:40.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:40 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:40.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:40 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:40.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:36:40.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:36:40.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:40 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:40.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:40 smithi151 ceph-mon[16639]: osdmap e969: 12 total, 12 up, 10 in 2024-09-02T10:36:40.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:40 smithi151 ceph-mon[16639]: pgmap v2219: 57 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 686 op/s; 188/106785 objects degraded (0.176%); 7225/106785 objects misplaced (6.766%); 24 MiB/s, 377 objects/s recovering 2024-09-02T10:36:40.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:40.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:40.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:40 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:40.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:40 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:40.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:36:40.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:36:40.768 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:40 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:40.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:40.827 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-02T10:36:40.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:40.837 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:40 smithi162 ceph-mon[20279]: osdmap e969: 12 total, 12 up, 10 in 2024-09-02T10:36:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:40 smithi162 ceph-mon[20279]: pgmap v2219: 57 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.2 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 686 op/s; 188/106785 objects degraded (0.176%); 7225/106785 objects misplaced (6.766%); 24 MiB/s, 377 objects/s recovering 2024-09-02T10:36:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:40 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:40 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:36:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:36:40.846 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:40 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:36:40.939 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:40.975 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:40.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:41.061 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:36:41.076 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:41.110 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:41.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:41.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:41.212 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:41.242 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:41.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:41.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:41.366 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:41.368 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:41.469 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:41.494 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:41.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:41.596 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:41.629 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:41.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:41.731 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:41.756 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:41.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:41.757 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:41.860 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:41.895 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:41.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:41.997 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:42.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:41 smithi187 ceph-mon[20072]: osdmap e970: 12 total, 12 up, 10 in 2024-09-02T10:36:42.022 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:42.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:42.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:42.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:41 smithi162 ceph-mon[20279]: osdmap e970: 12 total, 12 up, 10 in 2024-09-02T10:36:42.124 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:42.150 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:42.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:42.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:41 smithi151 ceph-mon[16639]: osdmap e970: 12 total, 12 up, 10 in 2024-09-02T10:36:42.252 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:42.275 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:42.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:42.276 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:42.377 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:42.400 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:42.401 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:42.502 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:42.535 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:42.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:42.637 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:42.659 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:42.660 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:42.761 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:42.793 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:42.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:42.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:42.895 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:42.930 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:42.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:42.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:43.033 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:43.063 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:43.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:43.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:42 smithi162 ceph-mon[20279]: pgmap v2221: 57 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 282/102651 objects degraded (0.275%); 6141/102651 objects misplaced (5.982%); 27 MiB/s, 434 objects/s recovering 2024-09-02T10:36:43.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:43.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:42 smithi151 ceph-mon[16639]: pgmap v2221: 57 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 282/102651 objects degraded (0.275%); 6141/102651 objects misplaced (5.982%); 27 MiB/s, 434 objects/s recovering 2024-09-02T10:36:43.190 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:43.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:43.236 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:36:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:42 smithi187 ceph-mon[20072]: pgmap v2221: 57 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.1 GiB data, 14 GiB used, 880 GiB / 894 GiB avail; 0 B/s wr, 824 op/s; 282/102651 objects degraded (0.275%); 6141/102651 objects misplaced (5.982%); 27 MiB/s, 434 objects/s recovering 2024-09-02T10:36:43.274 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:43.291 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:43.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:43.328 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:43.333 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:43.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:43.435 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:43.464 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:43.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:43.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:43.594 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:43.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:43.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:43.696 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:43.723 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:43.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:43.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:43.826 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:43.854 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:43.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:43.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:43.956 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:43.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:43 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:43.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:43 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:43.966 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:43 smithi151 ceph-mon[16639]: osdmap e971: 12 total, 12 up, 10 in 2024-09-02T10:36:43.979 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:43.980 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:44.081 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:43 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:43 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:43 smithi162 ceph-mon[20279]: osdmap e971: 12 total, 12 up, 10 in 2024-09-02T10:36:44.109 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:44.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:44.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:44.211 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:44.238 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:44.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:44.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:43 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:43 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:44.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:43 smithi187 ceph-mon[20072]: osdmap e971: 12 total, 12 up, 10 in 2024-09-02T10:36:44.318 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:36:44.318 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:36:44.337 INFO:teuthology.orchestra.run.smithi151.stdout:ERROR: (22) Invalid argument 2024-09-02T10:36:44.338 INFO:teuthology.orchestra.run.smithi151.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:36:44.340 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:44.349 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:36:44.349 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:36:44.369 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:44.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:44.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:44.377 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:36:44.377 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:36:44.407 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:36:44.408 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:36:44.438 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:36:44.438 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:36:44.471 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:44.472 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:36:44.497 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:36:44.497 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:36:44.498 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:44.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:44.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:44.526 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:36:44.526 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:36:44.554 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:36:44.554 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:36:44.578 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:36:44.579 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:36:44.600 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:44.603 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:36:44.603 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:36:44.629 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:44.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:44.634 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:36:44.635 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:36:44.731 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:44.755 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:44.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:44.857 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:44.880 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:44.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:44.882 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:44 smithi151 ceph-mon[16639]: pgmap v2223: 57 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 647 op/s; 282/98493 objects degraded (0.286%); 5340/98493 objects misplaced (5.422%); 25 MiB/s, 397 objects/s recovering 2024-09-02T10:36:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:44 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 282/98493 objects degraded (0.286%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:44.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3279526956' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:44.982 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:45.006 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:45.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:44 smithi162 ceph-mon[20279]: pgmap v2223: 57 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 647 op/s; 282/98493 objects degraded (0.286%); 5340/98493 objects misplaced (5.422%); 25 MiB/s, 397 objects/s recovering 2024-09-02T10:36:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:44 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 282/98493 objects degraded (0.286%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3279526956' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:45.108 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:45.127 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-02T10:36:45.140 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:45.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:45.241 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:44 smithi187 ceph-mon[20072]: pgmap v2223: 57 pgs: 1 peering, 6 active+recovering+undersized+degraded+remapped, 50 active+clean; 2.0 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 647 op/s; 282/98493 objects degraded (0.286%); 5340/98493 objects misplaced (5.422%); 25 MiB/s, 397 objects/s recovering 2024-09-02T10:36:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:44 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 282/98493 objects degraded (0.286%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:45.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3279526956' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:45.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:45.265 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:45.366 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:45.388 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:45.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:45.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:45.490 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:45.544 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:45.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:45.553 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:36:45.647 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:45.675 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:45.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:45.777 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:45.811 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:45.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:45.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:45.913 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:45.935 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:45.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:45.936 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:46.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:46.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:46.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:46.165 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:46.192 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:46.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:46.295 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:46.318 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:46.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:46.319 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:46.420 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:46.449 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:46.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:46.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:46.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:46.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:46.580 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:46.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:46.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:46.707 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:46.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:46.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:46.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:46.833 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:46.834 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:46.853 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:46 smithi151 ceph-mon[16639]: pgmap v2224: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 655 op/s; 282/94665 objects degraded (0.298%); 5090/94665 objects misplaced (5.377%); 21 MiB/s, 331 objects/s recovering 2024-09-02T10:36:46.936 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:46.969 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:46.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:47.071 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:47.094 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:47.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:47.095 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:46 smithi162 ceph-mon[20279]: pgmap v2224: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 655 op/s; 282/94665 objects degraded (0.298%); 5090/94665 objects misplaced (5.377%); 21 MiB/s, 331 objects/s recovering 2024-09-02T10:36:47.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:47.230 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:47.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:47.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:46 smithi187 ceph-mon[20072]: pgmap v2224: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 655 op/s; 282/94665 objects degraded (0.298%); 5090/94665 objects misplaced (5.377%); 21 MiB/s, 331 objects/s recovering 2024-09-02T10:36:47.332 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:47.353 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:47.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:47.354 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:47.455 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:47.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:47.492 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:47.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:47.594 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:47.618 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:47.619 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:47.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:47.744 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:47.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:47.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:47.788 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:36:47.846 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:47.872 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:47.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:47.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:47.999 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:48.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:48.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:47 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:47 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:47 smithi162 ceph-mon[20279]: osdmap e972: 12 total, 12 up, 10 in 2024-09-02T10:36:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2552347321' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:36:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:47 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:36:48.102 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:48.126 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:48.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:48.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:47 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:48.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:47 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:48.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:47 smithi151 ceph-mon[16639]: osdmap e972: 12 total, 12 up, 10 in 2024-09-02T10:36:48.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2552347321' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:36:48.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:47 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:36:48.229 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:48.238 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 4, 2] out_osds: [11, 10] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:36:48.238 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:36:48.238 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-09-02T10:36:48.238 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd in 10 2024-09-02T10:36:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:48.254 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:48.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:47 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:47 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:47 smithi187 ceph-mon[20072]: osdmap e972: 12 total, 12 up, 10 in 2024-09-02T10:36:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2552347321' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:36:48.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:47 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:36:48.355 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:48.386 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:48.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:48.488 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:48.521 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:48.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:48.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:48.618 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:36:48.624 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:48.691 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:48.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:48.793 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:48.815 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:48.816 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:48.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:48.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:48 smithi151 ceph-mon[16639]: pgmap v2226: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 660 op/s; 333/91575 objects degraded (0.364%); 4125/91575 objects misplaced (4.505%); 24 MiB/s, 392 objects/s recovering 2024-09-02T10:36:48.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:48 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:48.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:48 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:36:48.934 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:48 smithi151 ceph-mon[16639]: osdmap e973: 12 total, 12 up, 10 in 2024-09-02T10:36:48.939 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:48.940 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:49.041 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:49.048 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:49.059 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:49.070 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:49.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:48 smithi162 ceph-mon[20279]: pgmap v2226: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 660 op/s; 333/91575 objects degraded (0.364%); 4125/91575 objects misplaced (4.505%); 24 MiB/s, 392 objects/s recovering 2024-09-02T10:36:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:48 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:48 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:36:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:48 smithi162 ceph-mon[20279]: osdmap e973: 12 total, 12 up, 10 in 2024-09-02T10:36:49.172 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:49.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:49.208 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:49.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:48 smithi187 ceph-mon[20072]: pgmap v2226: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.9 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 660 op/s; 333/91575 objects degraded (0.364%); 4125/91575 objects misplaced (4.505%); 24 MiB/s, 392 objects/s recovering 2024-09-02T10:36:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:48 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:36:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:48 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:36:49.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:48 smithi187 ceph-mon[20072]: osdmap e973: 12 total, 12 up, 10 in 2024-09-02T10:36:49.309 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:49.332 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:49.333 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:49.434 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:49.462 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:49.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:49.564 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:49.591 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:49.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:49.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:49.727 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:49.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:49.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:49.829 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:49.857 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:49.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:49.959 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:49.991 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:49.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:50.094 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:49 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 333/91575 objects degraded (0.364%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:50.120 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:50.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:50.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:49 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 333/91575 objects degraded (0.364%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:50.152 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:49 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:36:49.936+0000 7fe3ffed4640 -1 mon.a@0(leader).osd e974 definitely_dead 0 2024-09-02T10:36:50.224 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:50.254 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:50.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:50.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:49 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 333/91575 objects degraded (0.364%), 6 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:50.283 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:36:50.284 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:36:50.311 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:36:50.311 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:36:50.334 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:168823 9.4 9:2694957a:::benchmark_data_smithi187_154927_object70711:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e973)", 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:49.348768+0000", 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.051456204999999998, 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.051488506000000003, 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:36:50.335 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168823 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.348768+0000", 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.348768+0000", 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.336 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.348770+0000", 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6589999999999999e-06 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.348777+0000", 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3960000000000003e-06 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:36:50.337 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.348794+0000", 2024-09-02T10:36:50.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7764000000000001e-05 2024-09-02T10:36:50.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:36:50.338 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.348803+0000", 2024-09-02T10:36:50.338 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6559999999999998e-06 2024-09-02T10:36:50.338 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.338 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.338 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.348864+0000", 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1498e-05 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.349043+0000", 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000178147 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-09-02T10:36:50.339 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.349171+0000", 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128047 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.352217+0000", 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030465959999999999 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:49.354949+0000", 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027318160000000002 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:36:50.340 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:36:50.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:36:50.341 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:36:50.341 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:36:50.341 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-02T10:36:50.341 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:36:50.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:50.389 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:50.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:50.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:50.516 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:50.517 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:50.536 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:36:50.537 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:36:50.568 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:36:50.568 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:36:50.596 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:36:50.597 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:36:50.618 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:50.629 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:36:50.629 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:36:50.642 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:50.643 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:50.659 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:36:50.660 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:36:50.744 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:50.765 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:50.766 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:50.826 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:36:50.827 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:36:50.846 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.10. 2024-09-02T10:36:50.856 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:36:50.856 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:36:50.867 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:50.882 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:36:50.883 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:36:50.892 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:50.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:50.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:50.904 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:36:50.904 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:36:50.925 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:36:50.925 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:36:50.993 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:36:50.994 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:51.015 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:51.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:50 smithi162 ceph-mon[20279]: pgmap v2228: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 760 op/s; 333/86214 objects degraded (0.386%); 3709/86214 objects misplaced (4.302%); 22 MiB/s, 399 objects/s recovering 2024-09-02T10:36:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:50 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:50 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:50 smithi162 ceph-mon[20279]: osdmap e974: 12 total, 12 up, 10 in 2024-09-02T10:36:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/809629233' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:36:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:50 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:36:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1429619932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:51.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:51.138 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:51.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:51.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:51.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:50 smithi151 ceph-mon[16639]: pgmap v2228: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 760 op/s; 333/86214 objects degraded (0.386%); 3709/86214 objects misplaced (4.302%); 22 MiB/s, 399 objects/s recovering 2024-09-02T10:36:51.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:50 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:51.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:50 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:51.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:50 smithi151 ceph-mon[16639]: osdmap e974: 12 total, 12 up, 10 in 2024-09-02T10:36:51.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/809629233' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:36:51.165 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:50 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:36:51.165 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1429619932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:51.240 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:50 smithi187 ceph-mon[20072]: pgmap v2228: 57 pgs: 6 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.8 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 760 op/s; 333/86214 objects degraded (0.386%); 3709/86214 objects misplaced (4.302%); 22 MiB/s, 399 objects/s recovering 2024-09-02T10:36:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:50 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:50 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:50 smithi187 ceph-mon[20072]: osdmap e974: 12 total, 12 up, 10 in 2024-09-02T10:36:51.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/809629233' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:36:51.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:50 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd in", "ids": ["10"]} : dispatch 2024-09-02T10:36:51.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1429619932' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:51.261 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:51.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:51.363 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:51.384 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:51.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:51.486 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:51.509 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:51.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:51.510 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:51.611 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:51.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:51.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:51.736 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:51.759 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:51.760 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:51.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:51.885 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:51.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:51.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:51.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:52.008 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:52.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:52.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:52.136 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:52.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:52.137 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:51 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-02T10:36:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:51 smithi151 ceph-mon[16639]: osdmap e975: 12 total, 12 up, 11 in 2024-09-02T10:36:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:51 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:36:52.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:51 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:36:52.238 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:51 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-02T10:36:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:51 smithi187 ceph-mon[20072]: osdmap e975: 12 total, 12 up, 11 in 2024-09-02T10:36:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:51 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:36:52.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:51 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:36:52.265 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:52.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:52.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:51 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["10"]}]': finished 2024-09-02T10:36:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:51 smithi162 ceph-mon[20279]: osdmap e975: 12 total, 12 up, 11 in 2024-09-02T10:36:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:51 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:36:52.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:51 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:36:52.367 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:52.389 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:52.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:52.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:52.491 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:52.521 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:52.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:52.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:52.623 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:52.644 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:52.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:52.746 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:52.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:52.768 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:52.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:52.870 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:52.893 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:52.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:52.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:52.996 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:53.018 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:53.019 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:53.102 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-09-02T10:36:53.120 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:53.123 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:53.156 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:53.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:53.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:52 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:36:53.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:52 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:36:53.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:52 smithi187 ceph-mon[20072]: pgmap v2231: 57 pgs: 3 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 667 op/s; 30/82503 objects degraded (0.036%); 1559/82503 objects misplaced (1.890%); 11 MiB/s, 359 objects/s recovering 2024-09-02T10:36:53.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:52 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:36:53.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:52 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:36:53.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:52 smithi187 ceph-mon[20072]: osdmap e976: 12 total, 12 up, 11 in 2024-09-02T10:36:53.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:52 smithi187 ceph-mon[20072]: 9.4 scrub starts 2024-09-02T10:36:53.158 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:52 smithi187 ceph-mon[20072]: 9.4 scrub ok 2024-09-02T10:36:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:52 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:36:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:52 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:36:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:52 smithi151 ceph-mon[16639]: pgmap v2231: 57 pgs: 3 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 667 op/s; 30/82503 objects degraded (0.036%); 1559/82503 objects misplaced (1.890%); 11 MiB/s, 359 objects/s recovering 2024-09-02T10:36:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:52 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:36:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:52 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:36:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:52 smithi151 ceph-mon[16639]: osdmap e976: 12 total, 12 up, 11 in 2024-09-02T10:36:53.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:52 smithi151 ceph-mon[16639]: 9.4 scrub starts 2024-09-02T10:36:53.189 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:52 smithi151 ceph-mon[16639]: 9.4 scrub ok 2024-09-02T10:36:53.216 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:52 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:36:53.217 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:52 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:36:53.217 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:52 smithi162 ceph-mon[20279]: pgmap v2231: 57 pgs: 3 remapped+peering, 3 active+recovering+undersized+degraded+remapped, 51 active+clean; 1.7 GiB data, 13 GiB used, 881 GiB / 894 GiB avail; 0 B/s wr, 667 op/s; 30/82503 objects degraded (0.036%); 1559/82503 objects misplaced (1.890%); 11 MiB/s, 359 objects/s recovering 2024-09-02T10:36:53.217 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:52 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:36:53.217 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:52 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:36:53.217 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:52 smithi162 ceph-mon[20279]: osdmap e976: 12 total, 12 up, 11 in 2024-09-02T10:36:53.217 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:52 smithi162 ceph-mon[20279]: 9.4 scrub starts 2024-09-02T10:36:53.217 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:52 smithi162 ceph-mon[20279]: 9.4 scrub ok 2024-09-02T10:36:53.258 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:53.284 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:53.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:53.385 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:53.416 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:53.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:53.518 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:53.550 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:53.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:53.653 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:53.680 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:53.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:53.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:53.782 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:53.812 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:53.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:53.914 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:53 smithi151 ceph-mon[16639]: osdmap e977: 12 total, 12 up, 11 in 2024-09-02T10:36:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:53 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:53.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:53 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:53.942 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:53.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:53.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:54.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:54.075 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:54.076 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:54.177 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:54.200 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:54.202 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:53 smithi187 ceph-mon[20072]: osdmap e977: 12 total, 12 up, 11 in 2024-09-02T10:36:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:53 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:54.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:53 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:54.303 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:54.329 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:54.330 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:53 smithi162 ceph-mon[20279]: osdmap e977: 12 total, 12 up, 11 in 2024-09-02T10:36:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:53 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:54.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:53 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:54.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:54.457 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:54.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:54.458 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:54.559 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:54.582 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:54.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:54.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:36:54.684 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:54.707 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:54.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:54.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:54.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:54.836 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:54.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:54.837 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:54.864 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:36:54.937 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:36:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:54 smithi151 ceph-mon[16639]: pgmap v2234: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 peering, 1 unknown, 3 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 279 op/s; 590/77355 objects degraded (0.763%); 1559/77355 objects misplaced (2.015%); 0 B/s, 57 objects/s recovering 2024-09-02T10:36:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:54 smithi151 ceph-mon[16639]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-02T10:36:54.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:54 smithi151 ceph-mon[16639]: osdmap e978: 12 total, 12 up, 11 in 2024-09-02T10:36:54.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3136269275' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:54.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:54 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 590/77355 objects degraded (0.763%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:54.963 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:54.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:54.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:36:55.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:55.092 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:55.093 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:55.195 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:55.226 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:55.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:55.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:55.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:54 smithi187 ceph-mon[20072]: pgmap v2234: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 peering, 1 unknown, 3 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 279 op/s; 590/77355 objects degraded (0.763%); 1559/77355 objects misplaced (2.015%); 0 B/s, 57 objects/s recovering 2024-09-02T10:36:55.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:54 smithi187 ceph-mon[20072]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-02T10:36:55.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:54 smithi187 ceph-mon[20072]: osdmap e978: 12 total, 12 up, 11 in 2024-09-02T10:36:55.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3136269275' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:55.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:54 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 590/77355 objects degraded (0.763%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:55.327 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:54 smithi162 ceph-mon[20279]: pgmap v2234: 57 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 peering, 1 unknown, 3 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 46 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 279 op/s; 590/77355 objects degraded (0.763%); 1559/77355 objects misplaced (2.015%); 0 B/s, 57 objects/s recovering 2024-09-02T10:36:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:54 smithi162 ceph-mon[20279]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-02T10:36:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:54 smithi162 ceph-mon[20279]: osdmap e978: 12 total, 12 up, 11 in 2024-09-02T10:36:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3136269275' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:36:55.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:54 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 590/77355 objects degraded (0.763%), 2 pgs degraded (PG_DEGRADED) 2024-09-02T10:36:55.358 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:55.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:55.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:55.460 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:55.486 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:55.487 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:55.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:55.613 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:55.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:55.615 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:55.716 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:55.747 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:55.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:55.789 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:55.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:55.879 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:55.880 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:55.981 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:56.007 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:56.008 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:56.019 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-02T10:36:56.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:55 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:36:56.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:55 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3136269275' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:56.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:55 smithi151 ceph-mon[16639]: osdmap e979: 12 total, 12 up, 11 in 2024-09-02T10:36:56.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:55 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:36:56.074 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:55 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:36:56.109 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:56.131 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:56.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:56.233 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:56.255 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:56.256 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:56.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:55 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:36:56.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:55 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3136269275' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:56.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:55 smithi187 ceph-mon[20072]: osdmap e979: 12 total, 12 up, 11 in 2024-09-02T10:36:56.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:55 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:36:56.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:55 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:36:56.312 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:36:56.332 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:36:56.332 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:36:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:55 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:36:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:55 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3136269275' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:36:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:55 smithi162 ceph-mon[20279]: osdmap e979: 12 total, 12 up, 11 in 2024-09-02T10:36:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:55 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:36:56.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:55 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:36:56.356 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:56.357 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:36:56.357 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:36:56.381 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:36:56.381 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:36:56.383 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:56.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:56.403 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:36:56.403 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:36:56.425 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:36:56.425 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:36:56.448 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:36:56.449 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:36:56.469 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:36:56.470 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:36:56.484 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:56.494 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:36:56.494 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:36:56.506 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:56.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:56.517 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:36:56.517 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:36:56.539 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:36:56.540 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:36:56.560 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:36:56.561 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:36:56.584 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:36:56.584 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:36:56.608 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:56.634 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:56.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:56.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:56.736 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:56.757 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:56.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:56.859 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:56.881 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:56.882 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:56.983 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:57.009 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:57.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:57.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:57.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:57.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:36:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:56 smithi151 ceph-mon[16639]: pgmap v2237: 57 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 peering, 3 remapped+peering, 46 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 4796/76908 objects degraded (6.236%); 960 KiB/s, 139 objects/s recovering 2024-09-02T10:36:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:56 smithi151 ceph-mon[16639]: osdmap e980: 12 total, 12 up, 11 in 2024-09-02T10:36:57.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3103024870' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:57.242 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:56 smithi187 ceph-mon[20072]: pgmap v2237: 57 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 peering, 3 remapped+peering, 46 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 4796/76908 objects degraded (6.236%); 960 KiB/s, 139 objects/s recovering 2024-09-02T10:36:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:56 smithi187 ceph-mon[20072]: osdmap e980: 12 total, 12 up, 11 in 2024-09-02T10:36:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3103024870' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:57.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:57.275 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:57.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:57.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:56 smithi162 ceph-mon[20279]: pgmap v2237: 57 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 4 peering, 3 remapped+peering, 46 active+clean; 1.6 GiB data, 14 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 466 op/s; 4796/76908 objects degraded (6.236%); 960 KiB/s, 139 objects/s recovering 2024-09-02T10:36:57.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:56 smithi162 ceph-mon[20279]: osdmap e980: 12 total, 12 up, 11 in 2024-09-02T10:36:57.338 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3103024870' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:36:57.377 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:36:57.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:57.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:36:57.507 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:57.536 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:57.537 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:57.638 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:36:57.665 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:57.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:36:57.755 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:36:57.767 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:36:57.794 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:57.795 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:36:57.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:57.920 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:57.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:58.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:58.047 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:58.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:58.048 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:58.052 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:58.103 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 4, 2, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:36:58.104 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:36:58.104 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-02T10:36:58.104 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.604787 2024-09-02T10:36:58.104 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 2 0.604787004298673 2024-09-02T10:36:58.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:57 smithi151 ceph-mon[16639]: osdmap e981: 12 total, 12 up, 11 in 2024-09-02T10:36:58.149 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:58.177 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:58.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:58.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:57 smithi187 ceph-mon[20072]: osdmap e981: 12 total, 12 up, 11 in 2024-09-02T10:36:58.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:58.304 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:58.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:58.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:57 smithi162 ceph-mon[20279]: osdmap e981: 12 total, 12 up, 11 in 2024-09-02T10:36:58.407 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:58.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:58.435 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:58.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:58.536 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:58.561 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:58.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:58.664 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:36:58.694 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:58.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:36:58.727 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:58.795 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:58.819 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:58.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:58.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:58.921 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:58.937 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:58 smithi151 ceph-mon[16639]: pgmap v2240: 57 pgs: 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 50 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 4796/73665 objects degraded (6.511%); 20 MiB/s, 629 objects/s recovering 2024-09-02T10:36:58.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:58 smithi151 ceph-mon[16639]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-09-02T10:36:58.949 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:58.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:58.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:59.051 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:36:59.072 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:59.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:59.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:36:59.175 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:36:59.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:58 smithi162 ceph-mon[20279]: pgmap v2240: 57 pgs: 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 50 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 4796/73665 objects degraded (6.511%); 20 MiB/s, 629 objects/s recovering 2024-09-02T10:36:59.176 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:58 smithi162 ceph-mon[20279]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-09-02T10:36:59.199 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:59.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:59.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:36:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:58 smithi187 ceph-mon[20072]: pgmap v2240: 57 pgs: 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 50 active+clean; 1.5 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 765 op/s; 4796/73665 objects degraded (6.511%); 20 MiB/s, 629 objects/s recovering 2024-09-02T10:36:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:58 smithi187 ceph-mon[20072]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-09-02T10:36:59.301 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:36:59.324 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:59.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:59.325 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:36:59.326 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:36:59.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:36:59.449 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:59.450 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:36:59.551 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:59.574 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:59.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:59.604 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:36:59.676 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:36:59.698 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:36:59.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:59.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:36:59.776 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:36:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:36:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:36:59.776 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:36:59.776 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:36:59.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:36:59.821 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:36:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:59.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:36:59.892 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:36:59.892 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.2 primary-affinity to 0.604787 (8396352) 2024-09-02T10:36:59.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:36:59.952 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:36:59.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:36:59.953 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:00.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:59 smithi151 ceph-mon[16639]: osdmap e982: 12 total, 12 up, 11 in 2024-09-02T10:37:00.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/42952596' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:37:00.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:59 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 4796/73665 objects degraded (6.511%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:37:00.042 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:36:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2900688855' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.604787004298673} : dispatch 2024-09-02T10:37:00.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:00.085 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:00.085 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:59 smithi162 ceph-mon[20279]: osdmap e982: 12 total, 12 up, 11 in 2024-09-02T10:37:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/42952596' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:37:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:59 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 4796/73665 objects degraded (6.511%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:37:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:36:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2900688855' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.604787004298673} : dispatch 2024-09-02T10:37:00.186 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:00.211 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:00.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:00.212 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:59 smithi187 ceph-mon[20072]: osdmap e982: 12 total, 12 up, 11 in 2024-09-02T10:37:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/42952596' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:37:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:59 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 4796/73665 objects degraded (6.511%), 3 pgs degraded (PG_DEGRADED) 2024-09-02T10:37:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:36:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2900688855' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 0.604787004298673} : dispatch 2024-09-02T10:37:00.313 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:00.340 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:00.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:00.442 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:00.464 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:00.465 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:00.566 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:00.597 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:00.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:00.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:00.699 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:00.724 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:00.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:00.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:00.826 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:00.857 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:00.858 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:00.959 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:00.985 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:00.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:00.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:01.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:00 smithi151 ceph-mon[16639]: pgmap v2242: 57 pgs: 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 50 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 874 op/s; 4796/67737 objects degraded (7.080%); 16 MiB/s, 607 objects/s recovering 2024-09-02T10:37:01.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:00 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:01.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/42952596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:37:01.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2900688855' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.604787004298673}]': finished 2024-09-02T10:37:01.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:00 smithi151 ceph-mon[16639]: osdmap e983: 12 total, 12 up, 11 in 2024-09-02T10:37:01.041 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3837077834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:01.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:01.114 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:01.114 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:01.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:37:01.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:37:01.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:37:01.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:37:01.183 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:37:01.183 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:37:01.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:37:01.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:37:01.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:37:01.184 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:37:01.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:37:01.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:37:01.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:37:01.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:37:01.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:37:01.184 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:37:01.185 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:37:01.185 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:37:01.185 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:37:01.185 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:37:01.185 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:37:01.185 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:37:01.185 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:37:01.186 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:37:01.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:01.239 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:01.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:01.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:00 smithi187 ceph-mon[20072]: pgmap v2242: 57 pgs: 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 50 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 874 op/s; 4796/67737 objects degraded (7.080%); 16 MiB/s, 607 objects/s recovering 2024-09-02T10:37:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:00 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/42952596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:37:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2900688855' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.604787004298673}]': finished 2024-09-02T10:37:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:00 smithi187 ceph-mon[20072]: osdmap e983: 12 total, 12 up, 11 in 2024-09-02T10:37:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3837077834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:01.341 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:01.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:00 smithi162 ceph-mon[20279]: pgmap v2242: 57 pgs: 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 50 active+clean; 1.4 GiB data, 13 GiB used, 970 GiB / 983 GiB avail; 0 B/s wr, 874 op/s; 4796/67737 objects degraded (7.080%); 16 MiB/s, 607 objects/s recovering 2024-09-02T10:37:01.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:00 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:01.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/42952596' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:37:01.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2900688855' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.604787004298673}]': finished 2024-09-02T10:37:01.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:00 smithi162 ceph-mon[20279]: osdmap e983: 12 total, 12 up, 11 in 2024-09-02T10:37:01.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3837077834' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:01.366 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:01.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:01.468 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:01.495 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:01.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:01.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:01.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:01.624 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:01.625 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:01.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:01.750 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:01.751 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:01.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:01.883 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:01.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:01.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:01.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:02.016 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:02.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:02.116 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:02.149 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:02.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:02.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:02.279 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:02.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:02.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:02.381 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:02.406 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:02.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:02.508 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:02.533 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:02.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:02.635 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:02.660 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:02.661 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:02.693 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-02T10:37:02.762 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:02.785 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:02.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:02.887 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:02.913 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:02.914 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:03.015 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:03.020 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:37:03.039 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:03.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:02 smithi162 ceph-mon[20279]: pgmap v2244: 57 pgs: 2 peering, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 50 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 3662/63669 objects degraded (5.752%); 0 B/s, 454 objects/s recovering 2024-09-02T10:37:03.142 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:03.170 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:03.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:03.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:03.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:02 smithi151 ceph-mon[16639]: pgmap v2244: 57 pgs: 2 peering, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 50 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 3662/63669 objects degraded (5.752%); 0 B/s, 454 objects/s recovering 2024-09-02T10:37:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:02 smithi187 ceph-mon[20072]: pgmap v2244: 57 pgs: 2 peering, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 50 active+clean; 1.3 GiB data, 13 GiB used, 971 GiB / 983 GiB avail; 0 B/s wr, 766 op/s; 3662/63669 objects degraded (5.752%); 0 B/s, 454 objects/s recovering 2024-09-02T10:37:03.272 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:03.295 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:03.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:03.396 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:03.423 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:03.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:03.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:03.525 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:03.548 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:03.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:03.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:03.595 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:37:03.595 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:37:03.651 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:03.675 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:03.676 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:03.777 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:03.807 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:03.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:03.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:37:03.909 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:03.938 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:03.939 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:04.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:04.064 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:04.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:04.166 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:04.184 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:04.192 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:04.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:04.286 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-02T10:37:04.293 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:04.319 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:04.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:04.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:04.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-02T10:37:04.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:04.446 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:04.448 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:04.549 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:04.572 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:04.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:04.573 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:04.674 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:04.697 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:04.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:04.719 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:37:04.799 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:04.822 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:04.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:04.924 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:04.947 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:04.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:05.049 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:05.071 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:05.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:05.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:04 smithi151 ceph-mon[16639]: osdmap e984: 12 total, 12 up, 11 in 2024-09-02T10:37:05.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:37:05.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:37:05.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:37:05.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:37:05.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:37:05.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:04 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:37:05.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:04 smithi151 ceph-mon[16639]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:37:05.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:04 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 3099/58068 objects degraded (5.337%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:05.164 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:04 smithi151 ceph-mon[16639]: osdmap e985: 12 total, 12 up, 11 in 2024-09-02T10:37:05.173 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:05.186 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:04 smithi162 ceph-mon[20279]: osdmap e984: 12 total, 12 up, 11 in 2024-09-02T10:37:05.186 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:04 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:37:05.186 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:04 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:37:05.187 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:04 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:37:05.187 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:04 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:37:05.187 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:04 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:37:05.187 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:04 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:37:05.187 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:04 smithi162 ceph-mon[20279]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:37:05.187 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:04 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 3099/58068 objects degraded (5.337%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:05.187 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:04 smithi162 ceph-mon[20279]: osdmap e985: 12 total, 12 up, 11 in 2024-09-02T10:37:05.194 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:05.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:05.195 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:04 smithi187 ceph-mon[20072]: osdmap e984: 12 total, 12 up, 11 in 2024-09-02T10:37:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:04 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:37:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:04 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:37:05.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:04 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:37:05.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:04 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:37:05.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:04 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:37:05.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:04 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:37:05.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:04 smithi187 ceph-mon[20072]: from='mgr.14151 ' entity='mgr.y' 2024-09-02T10:37:05.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:04 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 3099/58068 objects degraded (5.337%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:05.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:04 smithi187 ceph-mon[20072]: osdmap e985: 12 total, 12 up, 11 in 2024-09-02T10:37:05.285 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:37:05.285 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:37:05.296 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:05.313 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:05.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:05.319 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:05.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:05.420 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:05.443 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:05.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:05.545 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:05.572 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:05.574 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:05.675 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:05.703 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:05.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:05.805 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:05.843 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:05.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:05.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:05.951 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-02T10:37:05.971 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:05.972 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:06.073 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:06.078 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 4, 2, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:37:06.078 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:37:06.079 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-02T10:37:06.079 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:37:06.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3683958018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:06.109 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:05 smithi151 ceph-mon[16639]: osdmap e986: 12 total, 12 up, 11 in 2024-09-02T10:37:06.113 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:06.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:06.116 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:37:06.117 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:37:06.156 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:37:06.156 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:37:06.192 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:37:06.193 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:37:06.214 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:06.225 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:37:06.225 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:37:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:06.252 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:06.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3683958018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:05 smithi187 ceph-mon[20072]: osdmap e986: 12 total, 12 up, 11 in 2024-09-02T10:37:06.259 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:37:06.259 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:37:06.291 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:37:06.291 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:37:06.314 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:37:06.320 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:37:06.320 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:37:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3683958018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:06.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:05 smithi162 ceph-mon[20279]: osdmap e986: 12 total, 12 up, 11 in 2024-09-02T10:37:06.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:06.356 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:37:06.356 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:37:06.381 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:06.382 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:06.392 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:37:06.392 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:37:06.431 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:37:06.431 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:37:06.465 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:37:06.466 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:37:06.470 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:06.483 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:06.500 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:37:06.500 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:37:06.510 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:06.511 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:06.612 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:06.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:06.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:06.748 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:06.780 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:06.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:06.880 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:37:06.880 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:37:06.880 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:37:06.880 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:37:06.880 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:37:06.882 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:06.906 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:06.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:06.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:06.911 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is set 2024-09-02T10:37:07.008 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:07.031 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:07.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:06 smithi162 ceph-mon[20279]: pgmap v2249: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovering+undersized+remapped, 53 active+clean; 1.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 248/53748 objects degraded (0.461%); 468/53748 objects misplaced (0.871%); 859 KiB/s, 268 objects/s recovering 2024-09-02T10:37:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4211687584' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:07.133 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:07.157 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:07.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:07.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:06 smithi151 ceph-mon[16639]: pgmap v2249: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovering+undersized+remapped, 53 active+clean; 1.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 248/53748 objects degraded (0.461%); 468/53748 objects misplaced (0.871%); 859 KiB/s, 268 objects/s recovering 2024-09-02T10:37:07.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4211687584' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:06 smithi187 ceph-mon[20072]: pgmap v2249: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 active+recovering+undersized+remapped, 53 active+clean; 1.1 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 712 op/s; 248/53748 objects degraded (0.461%); 468/53748 objects misplaced (0.871%); 859 KiB/s, 268 objects/s recovering 2024-09-02T10:37:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4211687584' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:07.259 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:07.287 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:07.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:07.390 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:07.418 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:07.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:07.520 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:07.546 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:07.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:07.547 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:07.648 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:07.683 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:07.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:07.735 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:37:07.737 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:37:07.785 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:07.814 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:07.814 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:07.915 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:07.953 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:07.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:07.961 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-02T10:37:08.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:07 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:37:08.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:07 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:37:08.048 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:07 smithi151 ceph-mon[16639]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:08.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4211687584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:37:08.049 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:07 smithi151 ceph-mon[16639]: osdmap e987: 12 total, 12 up, 11 in 2024-09-02T10:37:08.055 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:08.082 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:08.083 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:08.184 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:08.209 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:08.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:08.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:07 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:37:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:07 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:37:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:07 smithi187 ceph-mon[20072]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4211687584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:37:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:07 smithi187 ceph-mon[20072]: osdmap e987: 12 total, 12 up, 11 in 2024-09-02T10:37:08.311 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:08.337 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:08.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:08.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:07 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:37:08.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:07 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:37:08.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:07 smithi162 ceph-mon[20279]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:08.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4211687584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-02T10:37:08.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:07 smithi162 ceph-mon[20279]: osdmap e987: 12 total, 12 up, 11 in 2024-09-02T10:37:08.381 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:37:08.439 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:08.465 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:08.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:08.567 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:08.592 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:08.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:08.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:08.694 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:08.716 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:08.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:08.815 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2251,"stamp":"2024-09-02T10:37:07.630414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1091043896,"num_objects":16642,"num_object_clones":0,"num_object_copies":49926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":468,"num_objects_unfound":0,"num_objects_dirty":16642,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124116,"num_write_kb":3151489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14409,"num_bytes_recovered":577517152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":213997,"ondisk_log_size":213997,"up":171,"acting":169,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":178,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":12075204,"kb_used_data":3396712,"kb_used_omap":17,"kb_used_meta":8678190,"kb_avail":1019121468,"statfs":{"total":1055945392128,"available":1043580383232,"internally_reserved":0,"allocated":3478233088,"data_stored":3474996917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17545,"internal_metadata":8886467447},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-177864704,"num_objects":-2714,"num_object_clones":0,"num_object_copies":-8142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2851,"num_objects_misplaced":468,"num_objects_unfound":0,"num_objects_dirty":-2714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2714,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1100,"num_bytes_recovered":5046272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001233"},"pg_stats":[{"pgid":"9.17","version":"985'7441","reported_seq":15274,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.668723+0000","last_change":"2024-09-02T10:36:14.428237+0000","last_active":"2024-09-02T10:37:04.668723+0000","last_peered":"2024-09-02T10:37:04.668723+0000","last_clean":"2024-09-02T10:37:04.668723+0000","last_became_active":"2024-09-02T10:35:57.707720+0000","last_became_peered":"2024-09-02T10:35:57.707720+0000","last_unstale":"2024-09-02T10:37:04.668723+0000","last_undegraded":"2024-09-02T10:37:04.668723+0000","last_fullsized":"2024-09-02T10:37:04.668723+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":5,"last_scrub":"959'6285","last_scrub_stamp":"2024-09-02T10:36:14.428021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:14.428021+0000","objects_scrubbed":1721,"log_size":7441,"log_dups_size":0,"ondisk_log_size":7441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:38.286207+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36765696,"num_objects":561,"num_object_clones":0,"num_object_copies":1683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":561,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1160,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.16","version":"983'7520","reported_seq":15512,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:37:03.022870+0000","last_change":"2024-09-02T10:36:11.419544+0000","last_active":"2024-09-02T10:37:03.022870+0000","last_peered":"2024-09-02T10:37:03.022870+0000","last_clean":"2024-09-02T10:37:03.022870+0000","last_became_active":"2024-09-02T10:35:38.829158+0000","last_became_peered":"2024-09-02T10:35:38.829158+0000","last_unstale":"2024-09-02T10:37:03.022870+0000","last_undegraded":"2024-09-02T10:37:03.022870+0000","last_fullsized":"2024-09-02T10:37:03.022870+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'6289","last_scrub_stamp":"2024-09-02T10:36:11.419498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:11.419498+0000","objects_scrubbed":1805,"log_size":7520,"log_dups_size":0,"ondisk_log_size":7520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:29.611986+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36700160,"num_objects":560,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1245,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1e","version":"22'92","reported_seq":185,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.409672+0000","last_change":"2024-09-02T10:36:51.864368+0000","last_active":"2024-09-02T10:37:04.409672+0000","last_peered":"2024-09-02T10:37:04.409672+0000","last_clean":"2024-09-02T10:37:04.409672+0000","last_became_active":"2024-09-02T10:36:51.864030+0000","last_became_peered":"2024-09-02T10:36:51.864030+0000","last_unstale":"2024-09-02T10:37:04.409672+0000","last_undegraded":"2024-09-02T10:37:04.409672+0000","last_fullsized":"2024-09-02T10:37:04.409672+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:29.260104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:29.260104+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.15","version":"986'7419","reported_seq":13446,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.375876+0000","last_change":"2024-09-02T10:35:25.713754+0000","last_active":"2024-09-02T10:37:06.375876+0000","last_peered":"2024-09-02T10:37:06.375876+0000","last_clean":"2024-09-02T10:37:06.375876+0000","last_became_active":"2024-09-02T10:35:24.380270+0000","last_became_peered":"2024-09-02T10:35:24.380270+0000","last_unstale":"2024-09-02T10:37:06.375876+0000","last_undegraded":"2024-09-02T10:37:06.375876+0000","last_fullsized":"2024-09-02T10:37:06.375876+0000","mapping_epoch":941,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":942,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'4781","last_scrub_stamp":"2024-09-02T10:35:25.713545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:25.713545+0000","objects_scrubbed":2905,"log_size":7419,"log_dups_size":0,"ondisk_log_size":7419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31784960,"num_objects":485,"num_object_clones":0,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2650,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,4],"acting":[7,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":550,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.989361+0000","last_change":"2024-09-02T10:36:09.745084+0000","last_active":"2024-09-02T10:37:06.989361+0000","last_peered":"2024-09-02T10:37:06.989361+0000","last_clean":"2024-09-02T10:37:06.989361+0000","last_became_active":"2024-09-02T10:33:38.659815+0000","last_became_peered":"2024-09-02T10:33:38.659815+0000","last_unstale":"2024-09-02T10:37:06.989361+0000","last_undegraded":"2024-09-02T10:37:06.989361+0000","last_fullsized":"2024-09-02T10:37:06.989361+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:09.745020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.745020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:31.345354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"985'7580","reported_seq":16412,"reported_epoch":985,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:37:04.916652+0000","last_change":"2024-09-02T10:37:03.055733+0000","last_active":"2024-09-02T10:37:04.916652+0000","last_peered":"2024-09-02T10:37:04.916652+0000","last_clean":"2024-09-02T10:36:50.538501+0000","last_became_active":"2024-09-02T10:36:52.998041+0000","last_became_peered":"2024-09-02T10:36:52.998041+0000","last_unstale":"2024-09-02T10:37:04.916652+0000","last_undegraded":"2024-09-02T10:36:52.867136+0000","last_fullsized":"2024-09-02T10:36:52.850550+0000","mapping_epoch":976,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":969,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'6265","last_scrub_stamp":"2024-09-02T10:36:09.284280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.284280+0000","objects_scrubbed":1834,"log_size":7580,"log_dups_size":0,"ondisk_log_size":7580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33292288,"num_objects":508,"num_object_clones":0,"num_object_copies":1524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":468,"num_objects_unfound":0,"num_objects_dirty":508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1326,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":85917696,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,3],"acting":[1,3],"avail_no_missing":["1","3","8"],"object_location_counts":[{"shards":"1,3","objects":-208},{"shards":"1,3,8","objects":468}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1667,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.410099+0000","last_change":"2024-09-02T10:36:21.119287+0000","last_active":"2024-09-02T10:37:04.410099+0000","last_peered":"2024-09-02T10:37:04.410099+0000","last_clean":"2024-09-02T10:37:04.410099+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:37:04.410099+0000","last_undegraded":"2024-09-02T10:37:04.410099+0000","last_fullsized":"2024-09-02T10:37:04.410099+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:21.119207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:21.119207+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:49.420928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"987'7858","reported_seq":14178,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-09-02T10:37:07.500687+0000","last_change":"2024-09-02T10:36:12.267636+0000","last_active":"2024-09-02T10:37:07.500687+0000","last_peered":"2024-09-02T10:37:07.500687+0000","last_clean":"2024-09-02T10:37:07.500687+0000","last_became_active":"2024-09-02T10:35:50.773879+0000","last_became_peered":"2024-09-02T10:35:50.773879+0000","last_unstale":"2024-09-02T10:37:07.500687+0000","last_undegraded":"2024-09-02T10:37:07.500687+0000","last_fullsized":"2024-09-02T10:37:07.500687+0000","mapping_epoch":949,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":950,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'6522","last_scrub_stamp":"2024-09-02T10:36:12.267481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:12.267481+0000","objects_scrubbed":1798,"log_size":7858,"log_dups_size":0,"ondisk_log_size":7858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:26.103198+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":460,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1338,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1643,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.412783+0000","last_change":"2024-09-02T10:36:09.169541+0000","last_active":"2024-09-02T10:37:04.412783+0000","last_peered":"2024-09-02T10:37:04.412783+0000","last_clean":"2024-09-02T10:37:04.412783+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:37:04.412783+0000","last_undegraded":"2024-09-02T10:37:04.412783+0000","last_fullsized":"2024-09-02T10:37:04.412783+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:09.169469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.169469+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:36.117296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.12","version":"983'7458","reported_seq":13515,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:37:02.742820+0000","last_change":"2024-09-02T10:35:24.913387+0000","last_active":"2024-09-02T10:37:02.742820+0000","last_peered":"2024-09-02T10:37:02.742820+0000","last_clean":"2024-09-02T10:37:02.742820+0000","last_became_active":"2024-09-02T10:35:22.670593+0000","last_became_peered":"2024-09-02T10:35:22.670593+0000","last_unstale":"2024-09-02T10:37:02.742820+0000","last_undegraded":"2024-09-02T10:37:02.742820+0000","last_fullsized":"2024-09-02T10:37:02.742820+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'4822","last_scrub_stamp":"2024-09-02T10:35:24.913280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:24.913280+0000","objects_scrubbed":2914,"log_size":7458,"log_dups_size":0,"ondisk_log_size":7458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35389440,"num_objects":540,"num_object_clones":0,"num_object_copies":1620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":540,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2656,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":962,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.405505+0000","last_change":"2024-09-02T10:36:16.494287+0000","last_active":"2024-09-02T10:37:04.405505+0000","last_peered":"2024-09-02T10:37:04.405505+0000","last_clean":"2024-09-02T10:37:04.405505+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:37:04.405505+0000","last_undegraded":"2024-09-02T10:37:04.405505+0000","last_fullsized":"2024-09-02T10:37:04.405505+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:16.494209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:16.494209+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:17.512138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"986'7302","reported_seq":11155,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.495109+0000","last_change":"2024-09-02T10:35:15.638534+0000","last_active":"2024-09-02T10:37:06.495109+0000","last_peered":"2024-09-02T10:37:06.495109+0000","last_clean":"2024-09-02T10:37:06.495109+0000","last_became_active":"2024-09-02T10:35:14.386404+0000","last_became_peered":"2024-09-02T10:35:14.386404+0000","last_unstale":"2024-09-02T10:37:06.495109+0000","last_undegraded":"2024-09-02T10:37:06.495109+0000","last_fullsized":"2024-09-02T10:37:06.495109+0000","mapping_epoch":933,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"935'4488","last_scrub_stamp":"2024-09-02T10:35:15.638360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:15.638360+0000","objects_scrubbed":2543,"log_size":7302,"log_dups_size":0,"ondisk_log_size":7302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29229056,"num_objects":446,"num_object_clones":0,"num_object_copies":1338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":23040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":32,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:05.410032+0000","last_change":"2024-09-02T10:36:51.854047+0000","last_active":"2024-09-02T10:37:05.410032+0000","last_peered":"2024-09-02T10:37:05.410032+0000","last_clean":"2024-09-02T10:37:05.410032+0000","last_became_active":"2024-09-02T10:36:51.853478+0000","last_became_peered":"2024-09-02T10:36:51.853478+0000","last_unstale":"2024-09-02T10:37:05.410032+0000","last_undegraded":"2024-09-02T10:37:05.410032+0000","last_fullsized":"2024-09-02T10:37:05.410032+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.939026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.939026+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:22.416375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.10","version":"985'7775","reported_seq":16067,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.502945+0000","last_change":"2024-09-02T10:36:09.887048+0000","last_active":"2024-09-02T10:37:04.502945+0000","last_peered":"2024-09-02T10:37:04.502945+0000","last_clean":"2024-09-02T10:37:04.502945+0000","last_became_active":"2024-09-02T10:36:08.733746+0000","last_became_peered":"2024-09-02T10:36:08.733746+0000","last_unstale":"2024-09-02T10:37:04.502945+0000","last_undegraded":"2024-09-02T10:37:04.502945+0000","last_fullsized":"2024-09-02T10:37:04.502945+0000","mapping_epoch":956,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":957,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'6404","last_scrub_stamp":"2024-09-02T10:36:09.886760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.886760+0000","objects_scrubbed":1881,"log_size":7775,"log_dups_size":0,"ondisk_log_size":7775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:10.899840+0000","scrub_duration":325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33161216,"num_objects":506,"num_object_clones":0,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1375,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":2019,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:36:59.895827+0000","last_change":"2024-09-02T10:36:10.478222+0000","last_active":"2024-09-02T10:36:59.895827+0000","last_peered":"2024-09-02T10:36:59.895827+0000","last_clean":"2024-09-02T10:36:59.895827+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:36:59.895827+0000","last_undegraded":"2024-09-02T10:36:59.895827+0000","last_fullsized":"2024-09-02T10:36:59.895827+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:10.478148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.478148+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:30.586610+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1680,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.410220+0000","last_change":"2024-09-02T10:36:51.855832+0000","last_active":"2024-09-02T10:37:04.410220+0000","last_peered":"2024-09-02T10:37:04.410220+0000","last_clean":"2024-09-02T10:37:04.410220+0000","last_became_active":"2024-09-02T10:36:51.855455+0000","last_became_peered":"2024-09-02T10:36:51.855455+0000","last_unstale":"2024-09-02T10:37:04.410220+0000","last_undegraded":"2024-09-02T10:37:04.410220+0000","last_fullsized":"2024-09-02T10:37:04.410220+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:19.291427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:19.291427+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:45.792830+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":221,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:36:59.898277+0000","last_change":"2024-09-02T10:36:15.323804+0000","last_active":"2024-09-02T10:36:59.898277+0000","last_peered":"2024-09-02T10:36:59.898277+0000","last_clean":"2024-09-02T10:36:59.898277+0000","last_became_active":"2024-09-02T10:33:38.657147+0000","last_became_peered":"2024-09-02T10:33:38.657147+0000","last_unstale":"2024-09-02T10:36:59.898277+0000","last_undegraded":"2024-09-02T10:36:59.898277+0000","last_fullsized":"2024-09-02T10:36:59.898277+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.323738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.323738+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:28.492236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1635,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:05.407729+0000","last_change":"2024-09-02T10:36:51.589123+0000","last_active":"2024-09-02T10:37:05.407729+0000","last_peered":"2024-09-02T10:37:05.407729+0000","last_clean":"2024-09-02T10:37:05.407729+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:37:05.407729+0000","last_undegraded":"2024-09-02T10:37:05.407729+0000","last_fullsized":"2024-09-02T10:37:05.407729+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:51.589079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:51.589079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:07.978045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":881,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.989321+0000","last_change":"2024-09-02T10:36:13.719520+0000","last_active":"2024-09-02T10:37:06.989321+0000","last_peered":"2024-09-02T10:37:06.989321+0000","last_clean":"2024-09-02T10:37:06.989321+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:37:06.989321+0000","last_undegraded":"2024-09-02T10:37:06.989321+0000","last_fullsized":"2024-09-02T10:37:06.989321+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:13.719461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.719461+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:25.562912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":72,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.405621+0000","last_change":"2024-09-02T10:36:20.573861+0000","last_active":"2024-09-02T10:37:04.405621+0000","last_peered":"2024-09-02T10:37:04.405621+0000","last_clean":"2024-09-02T10:37:04.405621+0000","last_became_active":"2024-09-02T10:36:12.767657+0000","last_became_peered":"2024-09-02T10:36:12.767657+0000","last_unstale":"2024-09-02T10:37:04.405621+0000","last_undegraded":"2024-09-02T10:37:04.405621+0000","last_fullsized":"2024-09-02T10:37:04.405621+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:20.573798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:20.573798+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:48.205451+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1910,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:36:59.896040+0000","last_change":"2024-09-02T10:34:08.628331+0000","last_active":"2024-09-02T10:36:59.896040+0000","last_peered":"2024-09-02T10:36:59.896040+0000","last_clean":"2024-09-02T10:36:59.896040+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:36:59.896040+0000","last_undegraded":"2024-09-02T10:36:59.896040+0000","last_fullsized":"2024-09-02T10:36:59.896040+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.628247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.628247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"984'7681","reported_seq":15372,"reported_epoch":984,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.229427+0000","last_change":"2024-09-02T10:36:22.570216+0000","last_active":"2024-09-02T10:37:04.229427+0000","last_peered":"2024-09-02T10:37:04.229427+0000","last_clean":"2024-09-02T10:37:04.229427+0000","last_became_active":"2024-09-02T10:35:16.385649+0000","last_became_peered":"2024-09-02T10:35:16.385649+0000","last_unstale":"2024-09-02T10:37:04.229427+0000","last_undegraded":"2024-09-02T10:37:04.229427+0000","last_fullsized":"2024-09-02T10:37:04.229427+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":5,"last_scrub":"961'6769","last_scrub_stamp":"2024-09-02T10:36:22.570101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:22.570101+0000","objects_scrubbed":1466,"log_size":7681,"log_dups_size":0,"ondisk_log_size":7681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:29.453651+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36110336,"num_objects":551,"num_object_clones":0,"num_object_copies":1653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2993,"num_write_kb":22848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":658,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:36:59.897965+0000","last_change":"2024-09-02T10:36:20.316217+0000","last_active":"2024-09-02T10:36:59.897965+0000","last_peered":"2024-09-02T10:36:59.897965+0000","last_clean":"2024-09-02T10:36:59.897965+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:36:59.897965+0000","last_undegraded":"2024-09-02T10:36:59.897965+0000","last_fullsized":"2024-09-02T10:36:59.897965+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:20.316139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:20.316139+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:21.453571+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"986'7631","reported_seq":16106,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.746728+0000","last_change":"2024-09-02T10:36:56.911246+0000","last_active":"2024-09-02T10:37:06.746728+0000","last_peered":"2024-09-02T10:37:06.746728+0000","last_clean":"2024-09-02T10:37:06.746728+0000","last_became_active":"2024-09-02T10:36:56.889585+0000","last_became_peered":"2024-09-02T10:36:56.889585+0000","last_unstale":"2024-09-02T10:37:06.746728+0000","last_undegraded":"2024-09-02T10:37:06.746728+0000","last_fullsized":"2024-09-02T10:37:06.746728+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":5,"last_scrub":"932'4480","last_scrub_stamp":"2024-09-02T10:35:09.830443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:09.830443+0000","objects_scrubbed":2522,"log_size":7631,"log_dups_size":0,"ondisk_log_size":7631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":460,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3160,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":89980928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,5],"acting":[10,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1902,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:36:59.895925+0000","last_change":"2024-09-02T10:36:51.864364+0000","last_active":"2024-09-02T10:36:59.895925+0000","last_peered":"2024-09-02T10:36:59.895925+0000","last_clean":"2024-09-02T10:36:59.895925+0000","last_became_active":"2024-09-02T10:36:51.863957+0000","last_became_peered":"2024-09-02T10:36:51.863957+0000","last_unstale":"2024-09-02T10:36:59.895925+0000","last_undegraded":"2024-09-02T10:36:59.895925+0000","last_fullsized":"2024-09-02T10:36:59.895925+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.409382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.409382+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:32.575551+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"987'7594","reported_seq":7846,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-09-02T10:37:07.533851+0000","last_change":"2024-09-02T10:36:23.228909+0000","last_active":"2024-09-02T10:37:07.533851+0000","last_peered":"2024-09-02T10:37:07.533851+0000","last_clean":"2024-09-02T10:37:07.533851+0000","last_became_active":"2024-09-02T10:34:47.026177+0000","last_became_peered":"2024-09-02T10:34:47.026177+0000","last_unstale":"2024-09-02T10:37:07.533851+0000","last_undegraded":"2024-09-02T10:37:07.533851+0000","last_fullsized":"2024-09-02T10:37:07.533851+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"961'6703","last_scrub_stamp":"2024-09-02T10:36:23.228789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:23.228789+0000","objects_scrubbed":1368,"log_size":7594,"log_dups_size":0,"ondisk_log_size":7594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:35.431706+0000","scrub_duration":376,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31260672,"num_objects":477,"num_object_clones":0,"num_object_copies":1431,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":477,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3689,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,8],"acting":[6,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1983,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:36:59.895938+0000","last_change":"2024-09-02T10:36:51.423663+0000","last_active":"2024-09-02T10:36:59.895938+0000","last_peered":"2024-09-02T10:36:59.895938+0000","last_clean":"2024-09-02T10:36:59.895938+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:36:59.895938+0000","last_undegraded":"2024-09-02T10:36:59.895938+0000","last_fullsized":"2024-09-02T10:36:59.895938+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:51.423616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:51.423616+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:52.566389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"984'7480","reported_seq":7728,"reported_epoch":984,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.310614+0000","last_change":"2024-09-02T10:36:17.751426+0000","last_active":"2024-09-02T10:37:04.310614+0000","last_peered":"2024-09-02T10:37:04.310614+0000","last_clean":"2024-09-02T10:37:04.310614+0000","last_became_active":"2024-09-02T10:34:47.038024+0000","last_became_peered":"2024-09-02T10:34:47.038024+0000","last_unstale":"2024-09-02T10:37:04.310614+0000","last_undegraded":"2024-09-02T10:37:04.310614+0000","last_fullsized":"2024-09-02T10:37:04.310614+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'6457","last_scrub_stamp":"2024-09-02T10:36:17.751343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:17.751343+0000","objects_scrubbed":1534,"log_size":7480,"log_dups_size":0,"ondisk_log_size":7480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:27.232651+0000","scrub_duration":530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32636928,"num_objects":498,"num_object_clones":0,"num_object_copies":1494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3719,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":2107,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:05.405994+0000","last_change":"2024-09-02T10:36:15.555892+0000","last_active":"2024-09-02T10:37:05.405994+0000","last_peered":"2024-09-02T10:37:05.405994+0000","last_clean":"2024-09-02T10:37:05.405994+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:37:05.405994+0000","last_undegraded":"2024-09-02T10:37:05.405994+0000","last_fullsized":"2024-09-02T10:37:05.405994+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.555822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.555822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:33.398682+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"984'7519","reported_seq":7766,"reported_epoch":984,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.297030+0000","last_change":"2024-09-02T10:36:13.627438+0000","last_active":"2024-09-02T10:37:04.297030+0000","last_peered":"2024-09-02T10:37:04.297030+0000","last_clean":"2024-09-02T10:37:04.297030+0000","last_became_active":"2024-09-02T10:34:47.038074+0000","last_became_peered":"2024-09-02T10:34:47.038074+0000","last_unstale":"2024-09-02T10:37:04.297030+0000","last_undegraded":"2024-09-02T10:37:04.297030+0000","last_fullsized":"2024-09-02T10:37:04.297030+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'6355","last_scrub_stamp":"2024-09-02T10:36:13.627233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.627233+0000","objects_scrubbed":1695,"log_size":7519,"log_dups_size":0,"ondisk_log_size":7519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:22.911073+0000","scrub_duration":452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34537472,"num_objects":527,"num_object_clones":0,"num_object_copies":1581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3693,"num_write_kb":72576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1759,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:36:59.926558+0000","last_change":"2024-09-02T10:36:10.022054+0000","last_active":"2024-09-02T10:36:59.926558+0000","last_peered":"2024-09-02T10:36:59.926558+0000","last_clean":"2024-09-02T10:36:59.926558+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:36:59.926558+0000","last_undegraded":"2024-09-02T10:36:59.926558+0000","last_fullsized":"2024-09-02T10:36:59.926558+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:10.021994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.021994+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:36.323969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"983'7602","reported_seq":7846,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:37:03.097275+0000","last_change":"2024-09-02T10:36:16.376436+0000","last_active":"2024-09-02T10:37:03.097275+0000","last_peered":"2024-09-02T10:37:03.097275+0000","last_clean":"2024-09-02T10:37:03.097275+0000","last_became_active":"2024-09-02T10:34:47.042672+0000","last_became_peered":"2024-09-02T10:34:47.042672+0000","last_unstale":"2024-09-02T10:37:03.097275+0000","last_undegraded":"2024-09-02T10:37:03.097275+0000","last_fullsized":"2024-09-02T10:37:03.097275+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'6561","last_scrub_stamp":"2024-09-02T10:36:16.376239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:16.376239+0000","objects_scrubbed":1576,"log_size":7602,"log_dups_size":0,"ondisk_log_size":7602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:24.146392+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":532,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3636,"num_write_kb":69504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":736,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:36:59.898388+0000","last_change":"2024-09-02T10:36:13.308543+0000","last_active":"2024-09-02T10:36:59.898388+0000","last_peered":"2024-09-02T10:36:59.898388+0000","last_clean":"2024-09-02T10:36:59.898388+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:36:59.898388+0000","last_undegraded":"2024-09-02T10:36:59.898388+0000","last_fullsized":"2024-09-02T10:36:59.898388+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:13.308459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.308459+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:42.491914+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"984'10963","reported_seq":11194,"reported_epoch":984,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.320905+0000","last_change":"2024-09-02T10:36:13.014102+0000","last_active":"2024-09-02T10:37:04.320905+0000","last_peered":"2024-09-02T10:37:04.320905+0000","last_clean":"2024-09-02T10:37:04.320905+0000","last_became_active":"2024-09-02T10:33:52.760804+0000","last_became_peered":"2024-09-02T10:33:52.760804+0000","last_unstale":"2024-09-02T10:37:04.320905+0000","last_undegraded":"2024-09-02T10:37:04.320905+0000","last_fullsized":"2024-09-02T10:37:04.320905+0000","mapping_epoch":897,"log_start":"907'900","ondisk_log_start":"907'900","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'8682","last_scrub_stamp":"2024-09-02T10:36:13.014038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.014038+0000","objects_scrubbed":3370,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:31.690235+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69533696,"num_objects":1061,"num_object_clones":0,"num_object_copies":3183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10963,"num_write_kb":384768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1070,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.409966+0000","last_change":"2024-09-02T10:36:51.845852+0000","last_active":"2024-09-02T10:37:04.409966+0000","last_peered":"2024-09-02T10:37:04.409966+0000","last_clean":"2024-09-02T10:37:04.409966+0000","last_became_active":"2024-09-02T10:34:17.881745+0000","last_became_peered":"2024-09-02T10:34:17.881745+0000","last_unstale":"2024-09-02T10:37:04.409966+0000","last_undegraded":"2024-09-02T10:37:04.409966+0000","last_fullsized":"2024-09-02T10:37:04.409966+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:51.845782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:51.845782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:05.174827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"986'11030","reported_seq":11268,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.454110+0000","last_change":"2024-09-02T10:36:12.744199+0000","last_active":"2024-09-02T10:37:06.454110+0000","last_peered":"2024-09-02T10:37:06.454110+0000","last_clean":"2024-09-02T10:37:06.454110+0000","last_became_active":"2024-09-02T10:33:52.926467+0000","last_became_peered":"2024-09-02T10:33:52.926467+0000","last_unstale":"2024-09-02T10:37:06.454110+0000","last_undegraded":"2024-09-02T10:37:06.454110+0000","last_fullsized":"2024-09-02T10:37:06.454110+0000","mapping_epoch":897,"log_start":"908'1000","ondisk_log_start":"908'1000","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'8646","last_scrub_stamp":"2024-09-02T10:36:12.744115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:12.744115+0000","objects_scrubbed":3403,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:13.209545+0000","scrub_duration":1221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63897624,"num_objects":976,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":976,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11030,"num_write_kb":384129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":32,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:05.410136+0000","last_change":"2024-09-02T10:36:51.860373+0000","last_active":"2024-09-02T10:37:05.410136+0000","last_peered":"2024-09-02T10:37:05.410136+0000","last_clean":"2024-09-02T10:37:05.410136+0000","last_became_active":"2024-09-02T10:36:51.859972+0000","last_became_peered":"2024-09-02T10:36:51.859972+0000","last_unstale":"2024-09-02T10:37:05.410136+0000","last_undegraded":"2024-09-02T10:37:05.410136+0000","last_fullsized":"2024-09-02T10:37:05.410136+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:20.991722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:20.991722+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:22.066390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"986'11203","reported_seq":14806,"reported_epoch":986,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-02T10:37:06.544819+0000","last_change":"2024-09-02T10:36:55.390676+0000","last_active":"2024-09-02T10:37:06.544819+0000","last_peered":"2024-09-02T10:37:06.544819+0000","last_clean":"2024-09-02T10:36:22.396656+0000","last_became_active":"2024-09-02T10:36:51.931556+0000","last_became_peered":"2024-09-02T10:36:51.931556+0000","last_unstale":"2024-09-02T10:37:06.544819+0000","last_undegraded":"2024-09-02T10:37:06.544819+0000","last_fullsized":"2024-09-02T10:36:51.848466+0000","mapping_epoch":975,"log_start":"908'1200","ondisk_log_start":"908'1200","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:33:51.735815+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62193664,"num_objects":949,"num_object_clones":0,"num_object_copies":2847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11203,"num_write_kb":388864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1701,"num_bytes_recovered":51904512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":949}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":32,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:05.410610+0000","last_change":"2024-09-02T10:36:51.854752+0000","last_active":"2024-09-02T10:37:05.410610+0000","last_peered":"2024-09-02T10:37:05.410610+0000","last_clean":"2024-09-02T10:37:05.410610+0000","last_became_active":"2024-09-02T10:36:51.854167+0000","last_became_peered":"2024-09-02T10:36:51.854167+0000","last_unstale":"2024-09-02T10:37:05.410610+0000","last_undegraded":"2024-09-02T10:37:05.410610+0000","last_fullsized":"2024-09-02T10:37:05.410610+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:17.904033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:17.904033+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:21.221795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"983'11189","reported_seq":11412,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:37:02.799217+0000","last_change":"2024-09-02T10:36:25.554143+0000","last_active":"2024-09-02T10:37:02.799217+0000","last_peered":"2024-09-02T10:37:02.799217+0000","last_clean":"2024-09-02T10:37:02.799217+0000","last_became_active":"2024-09-02T10:33:52.925153+0000","last_became_peered":"2024-09-02T10:33:52.925153+0000","last_unstale":"2024-09-02T10:37:02.799217+0000","last_undegraded":"2024-09-02T10:37:02.799217+0000","last_fullsized":"2024-09-02T10:37:02.799217+0000","mapping_epoch":897,"log_start":"908'1100","ondisk_log_start":"908'1100","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"963'9547","last_scrub_stamp":"2024-09-02T10:36:25.553889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:25.553889+0000","objects_scrubbed":2782,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:45.328325+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73072640,"num_objects":1115,"num_object_clones":0,"num_object_copies":3345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11189,"num_write_kb":393728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1299,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.918076+0000","last_change":"2024-09-02T10:36:51.855098+0000","last_active":"2024-09-02T10:37:06.918076+0000","last_peered":"2024-09-02T10:37:06.918076+0000","last_clean":"2024-09-02T10:37:06.918076+0000","last_became_active":"2024-09-02T10:36:51.854769+0000","last_became_peered":"2024-09-02T10:36:51.854769+0000","last_unstale":"2024-09-02T10:37:06.918076+0000","last_undegraded":"2024-09-02T10:37:06.918076+0000","last_fullsized":"2024-09-02T10:37:06.918076+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:18.936502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:18.936502+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:23.889891+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"986'7471","reported_seq":15980,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.778785+0000","last_change":"2024-09-02T10:37:04.428889+0000","last_active":"2024-09-02T10:37:06.778785+0000","last_peered":"2024-09-02T10:37:06.778785+0000","last_clean":"2024-09-02T10:37:06.778785+0000","last_became_active":"2024-09-02T10:37:04.411494+0000","last_became_peered":"2024-09-02T10:37:04.411494+0000","last_unstale":"2024-09-02T10:37:06.778785+0000","last_undegraded":"2024-09-02T10:37:06.778785+0000","last_fullsized":"2024-09-02T10:37:06.778785+0000","mapping_epoch":984,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":985,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'3857","last_scrub_stamp":"2024-09-02T10:34:49.043221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.043221+0000","objects_scrubbed":1904,"log_size":7471,"log_dups_size":0,"ondisk_log_size":7471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28966912,"num_objects":442,"num_object_clones":0,"num_object_copies":1326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3614,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2413,"num_bytes_recovered":87883776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,3],"acting":[10,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":32,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:05.410148+0000","last_change":"2024-09-02T10:36:51.853818+0000","last_active":"2024-09-02T10:37:05.410148+0000","last_peered":"2024-09-02T10:37:05.410148+0000","last_clean":"2024-09-02T10:37:05.410148+0000","last_became_active":"2024-09-02T10:36:51.853340+0000","last_became_peered":"2024-09-02T10:36:51.853340+0000","last_unstale":"2024-09-02T10:37:05.410148+0000","last_undegraded":"2024-09-02T10:37:05.410148+0000","last_fullsized":"2024-09-02T10:37:05.410148+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:24.088035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:24.088035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:34.852082+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"983'11172","reported_seq":11404,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:37:02.747092+0000","last_change":"2024-09-02T10:36:10.561335+0000","last_active":"2024-09-02T10:37:02.747092+0000","last_peered":"2024-09-02T10:37:02.747092+0000","last_clean":"2024-09-02T10:37:02.747092+0000","last_became_active":"2024-09-02T10:33:52.925177+0000","last_became_peered":"2024-09-02T10:33:52.925177+0000","last_unstale":"2024-09-02T10:37:02.747092+0000","last_undegraded":"2024-09-02T10:37:02.747092+0000","last_fullsized":"2024-09-02T10:37:02.747092+0000","mapping_epoch":897,"log_start":"908'1100","ondisk_log_start":"908'1100","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"957'8761","last_scrub_stamp":"2024-09-02T10:36:10.561133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.561133+0000","objects_scrubbed":3627,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:21.163961+0000","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77594624,"num_objects":1184,"num_object_clones":0,"num_object_copies":3552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":395392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1749,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:05.406079+0000","last_change":"2024-09-02T10:34:04.115873+0000","last_active":"2024-09-02T10:37:05.406079+0000","last_peered":"2024-09-02T10:37:05.406079+0000","last_clean":"2024-09-02T10:37:05.406079+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:37:05.406079+0000","last_undegraded":"2024-09-02T10:37:05.406079+0000","last_fullsized":"2024-09-02T10:37:05.406079+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:04.115830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:04.115830+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"984'11168","reported_seq":16205,"reported_epoch":984,"state":"peering","last_fresh":"2024-09-02T10:37:04.390259+0000","last_change":"2024-09-02T10:37:04.412377+0000","last_active":"2024-09-02T10:37:04.412377+0000","last_peered":"2024-09-02T10:37:04.390259+0000","last_clean":"2024-09-02T10:36:22.360171+0000","last_became_active":"2024-09-02T10:36:51.918416+0000","last_became_peered":"2024-09-02T10:36:51.918416+0000","last_unstale":"2024-09-02T10:37:04.390259+0000","last_undegraded":"2024-09-02T10:37:04.390259+0000","last_fullsized":"2024-09-02T10:36:51.849117+0000","mapping_epoch":985,"log_start":"908'1100","ondisk_log_start":"908'1100","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'9386","last_scrub_stamp":"2024-09-02T10:36:21.798044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:21.798044+0000","objects_scrubbed":2871,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11168,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2398,"num_bytes_recovered":82771968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5,10],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1051,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.917371+0000","last_change":"2024-09-02T10:36:13.900889+0000","last_active":"2024-09-02T10:37:06.917371+0000","last_peered":"2024-09-02T10:37:06.917371+0000","last_clean":"2024-09-02T10:37:06.917371+0000","last_became_active":"2024-09-02T10:33:38.664311+0000","last_became_peered":"2024-09-02T10:33:38.664311+0000","last_unstale":"2024-09-02T10:37:06.917371+0000","last_undegraded":"2024-09-02T10:37:06.917371+0000","last_fullsized":"2024-09-02T10:37:06.917371+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:13.900825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.900825+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:40.109001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"983'7646","reported_seq":7895,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:37:03.068402+0000","last_change":"2024-09-02T10:36:18.281248+0000","last_active":"2024-09-02T10:37:03.068402+0000","last_peered":"2024-09-02T10:37:03.068402+0000","last_clean":"2024-09-02T10:37:03.068402+0000","last_became_active":"2024-09-02T10:34:47.044129+0000","last_became_peered":"2024-09-02T10:34:47.044129+0000","last_unstale":"2024-09-02T10:37:03.068402+0000","last_undegraded":"2024-09-02T10:37:03.068402+0000","last_fullsized":"2024-09-02T10:37:03.068402+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'6666","last_scrub_stamp":"2024-09-02T10:36:18.281061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:18.281061+0000","objects_scrubbed":1568,"log_size":7646,"log_dups_size":0,"ondisk_log_size":7646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:34.149765+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207488,"num_objects":583,"num_object_clones":0,"num_object_copies":1749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3720,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1760,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:05.405896+0000","last_change":"2024-09-02T10:34:01.291835+0000","last_active":"2024-09-02T10:37:05.405896+0000","last_peered":"2024-09-02T10:37:05.405896+0000","last_clean":"2024-09-02T10:37:05.405896+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:37:05.405896+0000","last_undegraded":"2024-09-02T10:37:05.405896+0000","last_fullsized":"2024-09-02T10:37:05.405896+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:01.291779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:01.291779+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"986'7431","reported_seq":12240,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.768521+0000","last_change":"2024-09-02T10:36:56.903650+0000","last_active":"2024-09-02T10:37:06.768521+0000","last_peered":"2024-09-02T10:37:06.768521+0000","last_clean":"2024-09-02T10:37:06.768521+0000","last_became_active":"2024-09-02T10:36:56.889540+0000","last_became_peered":"2024-09-02T10:36:56.889540+0000","last_unstale":"2024-09-02T10:37:06.768521+0000","last_undegraded":"2024-09-02T10:37:06.768521+0000","last_fullsized":"2024-09-02T10:37:06.768521+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3753","last_scrub_stamp":"2024-09-02T10:34:49.259057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.259057+0000","objects_scrubbed":1910,"log_size":7431,"log_dups_size":0,"ondisk_log_size":7431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30015488,"num_objects":458,"num_object_clones":0,"num_object_copies":1374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3678,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2281,"num_bytes_recovered":90439680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":2025,"reported_epoch":987,"state":"active+clean","last_fresh":"2024-09-02T10:37:06.918168+0000","last_change":"2024-09-02T10:36:09.993500+0000","last_active":"2024-09-02T10:37:06.918168+0000","last_peered":"2024-09-02T10:37:06.918168+0000","last_clean":"2024-09-02T10:37:06.918168+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:37:06.918168+0000","last_undegraded":"2024-09-02T10:37:06.918168+0000","last_fullsized":"2024-09-02T10:37:06.918168+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:09.993425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.993425+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:29.323758+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"983'7399","reported_seq":11981,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:37:02.757129+0000","last_change":"2024-09-02T10:36:59.916449+0000","last_active":"2024-09-02T10:37:02.757129+0000","last_peered":"2024-09-02T10:37:02.757129+0000","last_clean":"2024-09-02T10:37:02.757129+0000","last_became_active":"2024-09-02T10:36:59.901856+0000","last_became_peered":"2024-09-02T10:36:59.901856+0000","last_unstale":"2024-09-02T10:37:02.757129+0000","last_undegraded":"2024-09-02T10:37:02.757129+0000","last_fullsized":"2024-09-02T10:37:02.757129+0000","mapping_epoch":982,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":983,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'3847","last_scrub_stamp":"2024-09-02T10:34:50.216498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:50.216498+0000","objects_scrubbed":1961,"log_size":7399,"log_dups_size":0,"ondisk_log_size":7399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":564,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":564,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3559,"num_write_kb":69184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2170,"num_bytes_recovered":87687168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":970,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.405927+0000","last_change":"2024-09-02T10:36:17.472218+0000","last_active":"2024-09-02T10:37:04.405927+0000","last_peered":"2024-09-02T10:37:04.405927+0000","last_clean":"2024-09-02T10:37:04.405927+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:37:04.405927+0000","last_undegraded":"2024-09-02T10:37:04.405927+0000","last_fullsized":"2024-09-02T10:37:04.405927+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:17.472148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:17.472148+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:17.539247+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"983'7447","reported_seq":7668,"reported_epoch":983,"state":"active+clean","last_fresh":"2024-09-02T10:37:02.711179+0000","last_change":"2024-09-02T10:34:49.168775+0000","last_active":"2024-09-02T10:37:02.711179+0000","last_peered":"2024-09-02T10:37:02.711179+0000","last_clean":"2024-09-02T10:37:02.711179+0000","last_became_active":"2024-09-02T10:34:47.049681+0000","last_became_peered":"2024-09-02T10:34:47.049681+0000","last_unstale":"2024-09-02T10:37:02.711179+0000","last_undegraded":"2024-09-02T10:37:02.711179+0000","last_fullsized":"2024-09-02T10:37:02.711179+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3787","last_scrub_stamp":"2024-09-02T10:34:49.168686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.168686+0000","objects_scrubbed":1913,"log_size":7447,"log_dups_size":0,"ondisk_log_size":7447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41484288,"num_objects":633,"num_object_clones":0,"num_object_copies":1899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3660,"num_write_kb":76160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1787,"reported_epoch":986,"state":"active+clean","last_fresh":"2024-09-02T10:37:05.407833+0000","last_change":"2024-09-02T10:36:16.543124+0000","last_active":"2024-09-02T10:37:05.407833+0000","last_peered":"2024-09-02T10:37:05.407833+0000","last_clean":"2024-09-02T10:37:05.407833+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:37:05.407833+0000","last_undegraded":"2024-09-02T10:37:05.407833+0000","last_fullsized":"2024-09-02T10:37:05.407833+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:16.543059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:16.543059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:17.166860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"985'7566","reported_seq":7809,"reported_epoch":985,"state":"active+clean","last_fresh":"2024-09-02T10:37:04.519945+0000","last_change":"2024-09-02T10:36:52.528377+0000","last_active":"2024-09-02T10:37:04.519945+0000","last_peered":"2024-09-02T10:37:04.519945+0000","last_clean":"2024-09-02T10:37:04.519945+0000","last_became_active":"2024-09-02T10:34:47.029634+0000","last_became_peered":"2024-09-02T10:34:47.029634+0000","last_unstale":"2024-09-02T10:37:04.519945+0000","last_undegraded":"2024-09-02T10:37:04.519945+0000","last_fullsized":"2024-09-02T10:37:04.519945+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"976'7251","last_scrub_stamp":"2024-09-02T10:36:52.528264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:52.528264+0000","objects_scrubbed":839,"log_size":7566,"log_dups_size":0,"ondisk_log_size":7566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:10.591294+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34340864,"num_objects":524,"num_object_clones":0,"num_object_copies":1572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":524,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3703,"num_write_kb":73472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":1182,"reported_epoch":984,"state":"active+clean","last_fresh":"2024-09-02T10:37:03.912420+0000","last_change":"2024-09-02T10:36:10.301898+0000","last_active":"2024-09-02T10:37:03.912420+0000","last_peered":"2024-09-02T10:37:03.912420+0000","last_clean":"2024-09-02T10:37:03.912420+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:37:03.912420+0000","last_undegraded":"2024-09-02T10:37:03.912420+0000","last_fullsized":"2024-09-02T10:37:03.912420+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:10.301844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.301844+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:18.968378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":1090453528,"num_objects":16640,"num_object_clones":0,"num_object_copies":49920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":468,"num_objects_unfound":0,"num_objects_dirty":16640,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":124116,"num_write_kb":3151489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14405,"num_bytes_recovered":576585728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3505860608,"data_stored":3505848392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":211145,"ondisk_log_size":211145,"up":78,"acting":76,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":343597384124,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532613057,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":968296,"kb_used_data":273484,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":92776856,"statfs":{"total":95995035648,"available":95003500544,"internally_reserved":0,"allocated":280047616,"data_stored":279749154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":711457216},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841989,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1285496,"kb_used_data":325544,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92459656,"statfs":{"total":95995035648,"available":94678687744,"internally_reserved":0,"allocated":333357056,"data_stored":333071664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":982972864},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070925,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":854712,"kb_used_data":282088,"kb_used_omap":1,"kb_used_meta":572606,"kb_avail":92890440,"statfs":{"total":95995035648,"available":95119810560,"internally_reserved":0,"allocated":288858112,"data_stored":288572720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":586348997},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1245424,"kb_used_data":429848,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":92499728,"statfs":{"total":95995035648,"available":94719721472,"internally_reserved":0,"allocated":440164352,"data_stored":439858266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":835123658},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528791,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":788896,"kb_used_data":184332,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92956256,"statfs":{"total":95995035648,"available":95187206144,"internally_reserved":0,"allocated":188755968,"data_stored":188457506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":619051463},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503725017,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1617668,"kb_used_data":629876,"kb_used_omap":1,"kb_used_meta":987710,"kb_avail":92127484,"statfs":{"total":95995035648,"available":94338543616,"internally_reserved":0,"allocated":644993024,"data_stored":644700008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1011415493},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953950,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":766872,"kb_used_data":167872,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":92978280,"statfs":{"total":95995035648,"available":95209758720,"internally_reserved":0,"allocated":171900928,"data_stored":171606849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150180,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1289736,"kb_used_data":333940,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92455416,"statfs":{"total":95995035648,"available":94674345984,"internally_reserved":0,"allocated":341954560,"data_stored":341664835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":978713032},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411816,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663652,"kb_used_data":32724,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93081500,"statfs":{"total":95995035648,"available":95315456000,"internally_reserved":0,"allocated":33509376,"data_stored":33207362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575341,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1489372,"kb_used_data":587716,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92255780,"statfs":{"total":95995035648,"available":94469918720,"internally_reserved":0,"allocated":601821184,"data_stored":601523033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738866,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1105080,"kb_used_data":149288,"kb_used_omap":1,"kb_used_meta":955774,"kb_avail":92640072,"statfs":{"total":95995035648,"available":94863433728,"internally_reserved":0,"allocated":152870912,"data_stored":152585520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":978713029},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":149291008,"data_stored":149291008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593825792,"data_stored":593821720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31260672,"data_stored":31260672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":338493440,"data_stored":338493440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":169738240,"data_stored":169738240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":635637760,"data_stored":635633688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":178454528,"data_stored":178454528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":414126080,"data_stored":414122008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":282001408,"data_stored":282001408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":327417856,"data_stored":327417856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":273481728,"data_stored":273481728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":112132096,"data_stored":112132096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:37:08.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool set .mgr pgp_num 71 2024-09-02T10:37:08.819 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:08.842 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:08.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:08.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:08.944 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:08.950 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:37:08.950 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:37:08.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:37:08.950 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:37:08.950 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.950 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:109455 8.8 8:14dc587f:::benchmark_data_smithi187_136059_object4174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e836)", 2024-09-02T10:37:08.950 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:30:34.214622+0000", 2024-09-02T10:37:08.950 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 394.73201027900001, 2024-09-02T10:37:08.950 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26031262100000002, 2024-09-02T10:37:08.951 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:08.951 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:08.951 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:08.951 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:08.951 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:37:08.951 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:08.951 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 109455 2024-09-02T10:37:08.951 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.951 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214622+0000", 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214622+0000", 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.952 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214624+0000", 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.345e-06 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214629+0000", 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2159999999999997e-06 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214641+0000", 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2097e-05 2024-09-02T10:37:08.953 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214648+0000", 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1969999999999999e-06 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214704+0000", 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5408000000000002e-05 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.954 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214841+0000", 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013739399999999999 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.214978+0000", 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013713199999999999 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.955 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:08.956 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218166+0000", 2024-09-02T10:37:08.956 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003187722 2024-09-02T10:37:08.956 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.956 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218948+0000", 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00078261400000000005 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474854+0000", 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25590502999999998 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474907+0000", 2024-09-02T10:37:08.957 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3647999999999999e-05 2024-09-02T10:37:08.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.958 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.958 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:08.958 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474935+0000", 2024-09-02T10:37:08.958 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7818e-05 2024-09-02T10:37:08.958 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.958 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:08.958 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.958 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27559.0:109455 8.8 e836/792 ondisk, result = 0)", 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:30:34.218001+0000", 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 394.72863146399999, 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25693914899999998, 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218001+0000", 2024-09-02T10:37:08.959 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.960 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.960 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:08.960 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218001+0000", 2024-09-02T10:37:08.960 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.960 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:08.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218002+0000", 2024-09-02T10:37:08.961 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5099999999999998e-07 2024-09-02T10:37:08.961 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.961 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.961 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:08.961 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.218006+0000", 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3680000000000004e-06 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474642+0000", 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25663574099999997 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474658+0000", 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5990000000000001e-05 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.962 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474828+0000", 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00017020300000000001 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474844+0000", 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5889999999999999e-05 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:30:34.474940+0000", 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6106000000000004e-05 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:08.963 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:134989 8.8 8:15a31c2e:::benchmark_data_smithi187_136059_object29708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e843)", 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:01.997645+0000", 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 366.94898754600001, 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48778658600000002, 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 134989 2024-09-02T10:37:08.964 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997645+0000", 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997645+0000", 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.965 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997646+0000", 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1480000000000001e-06 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997650+0000", 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0799999999999999e-06 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997662+0000", 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1946999999999999e-05 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.966 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997668+0000", 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4060000000000004e-06 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997739+0000", 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1211999999999995e-05 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997836+0000", 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6628999999999999e-05 2024-09-02T10:37:08.967 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:01.997908+0000", 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2268000000000007e-05 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001365+0000", 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034576619999999998 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:08.968 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001726+0000", 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000360997 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485369+0000", 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48364271800000003 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485401+0000", 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2274000000000003e-05 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.969 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485432+0000", 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0244999999999999e-05 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27559.0:134989 8.8 e843/792 ondisk, result = 0)", 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:02.001592+0000", 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 366.94504090100003, 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48384515900000002, 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:08.970 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001592+0000", 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001592+0000", 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.971 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.972 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:08.972 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001592+0000", 2024-09-02T10:37:08.972 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5700000000000002e-07 2024-09-02T10:37:08.972 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.972 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:08.973 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.001599+0000", 2024-09-02T10:37:08.973 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.669e-06 2024-09-02T10:37:08.973 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.973 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.973 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:08.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485331+0000", 2024-09-02T10:37:08.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.48373196800000001 2024-09-02T10:37:08.974 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.974 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.974 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:08.974 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485335+0000", 2024-09-02T10:37:08.974 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3059999999999999e-06 2024-09-02T10:37:08.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:08.975 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485362+0000", 2024-09-02T10:37:08.975 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6495e-05 2024-09-02T10:37:08.975 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.975 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.975 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:08.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485366+0000", 2024-09-02T10:37:08.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3889999999999998e-06 2024-09-02T10:37:08.976 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.976 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.976 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:08.976 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:02.485437+0000", 2024-09-02T10:37:08.976 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0674999999999997e-05 2024-09-02T10:37:08.977 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.977 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:08.977 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.977 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.977 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.977 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:149583 8.8 8:1df6255f:::benchmark_data_smithi187_136059_object44302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e851)", 2024-09-02T10:37:08.978 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:27.306490+0000", 2024-09-02T10:37:08.978 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 341.64014284400002, 2024-09-02T10:37:08.978 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.23605482999999999, 2024-09-02T10:37:08.978 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:08.978 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:08.978 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:08.978 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:08.978 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:37:08.979 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:08.979 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 149583 2024-09-02T10:37:08.979 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.979 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:08.979 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.979 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:08.979 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306490+0000", 2024-09-02T10:37:08.979 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.980 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:08.980 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306490+0000", 2024-09-02T10:37:08.980 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.980 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.980 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:08.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306490+0000", 2024-09-02T10:37:08.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4700000000000002e-07 2024-09-02T10:37:08.981 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.981 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.981 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:08.981 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306493+0000", 2024-09-02T10:37:08.981 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3329999999999999e-06 2024-09-02T10:37:08.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.982 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:08.982 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306500+0000", 2024-09-02T10:37:08.982 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.5700000000000004e-06 2024-09-02T10:37:08.982 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.982 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:08.983 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306504+0000", 2024-09-02T10:37:08.983 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5710000000000002e-06 2024-09-02T10:37:08.983 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.983 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.983 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:08.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306550+0000", 2024-09-02T10:37:08.984 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5964999999999998e-05 2024-09-02T10:37:08.984 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.984 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.984 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:08.984 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306648+0000", 2024-09-02T10:37:08.985 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8740000000000007e-05 2024-09-02T10:37:08.985 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.985 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.985 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,9", 2024-09-02T10:37:08.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.306746+0000", 2024-09-02T10:37:08.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7916000000000002e-05 2024-09-02T10:37:08.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.986 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.986 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:08.986 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.311648+0000", 2024-09-02T10:37:08.986 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.004901954 2024-09-02T10:37:08.986 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:08.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.336249+0000", 2024-09-02T10:37:08.987 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.02460066 2024-09-02T10:37:08.987 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.987 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.987 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:08.987 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.542476+0000", 2024-09-02T10:37:08.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20622670900000001 2024-09-02T10:37:08.988 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.988 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.988 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:08.988 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.542514+0000", 2024-09-02T10:37:08.988 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8563000000000003e-05 2024-09-02T10:37:08.989 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.989 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.989 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:08.989 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:27.542545+0000", 2024-09-02T10:37:08.989 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0402000000000001e-05 2024-09-02T10:37:08.989 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.990 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:08.990 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.990 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.990 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(8.d 851/844 [PushOp(8:bc6811a2:::benchmark_data_smithi187_136059_object50100:head, version: 814'3129, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:bc6811a2:::benchmark_data_smithi187_136059_object50100:head@814'3129, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:35.080864+0000", 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 333.86576829699999, 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21813774, 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.080864+0000", 2024-09-02T10:37:08.993 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.080864+0000", 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.080865+0000", 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5000000000000002e-07 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:08.994 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.080900+0000", 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5438999999999997e-05 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.295229+0000", 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.214328291 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.295246+0000", 2024-09-02T10:37:08.995 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7334999999999999e-05 2024-09-02T10:37:08.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:08.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.295363+0000", 2024-09-02T10:37:08.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116778 2024-09-02T10:37:08.996 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.996 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.996 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:08.996 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.295375+0000", 2024-09-02T10:37:08.996 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.26e-05 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:35.299002+0000", 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036266470000000002 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:183131 8.3 e869/844 8:cd9a7f61:::benchmark_data_smithi187_136059_object77850:head v 869'11438, mlcod=823'5021)", 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:32:29.334219+0000", 2024-09-02T10:37:08.997 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 279.61241378800003, 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.234334403, 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.334219+0000", 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.334219+0000", 2024-09-02T10:37:08.998 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.334219+0000", 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8699999999999995e-07 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.334225+0000", 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0610000000000001e-06 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:08.999 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564145+0000", 2024-09-02T10:37:09.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22991979400000001 2024-09-02T10:37:09.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564151+0000", 2024-09-02T10:37:09.000 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.071e-06 2024-09-02T10:37:09.000 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.000 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.000 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.000 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564203+0000", 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1966999999999999e-05 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564230+0000", 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6441999999999999e-05 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.568517+0000", 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042867599999999997 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.001 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.002 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.002 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.568553+0000", 2024-09-02T10:37:09.002 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6721000000000003e-05 2024-09-02T10:37:09.002 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.002 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.002 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:210554 8.4 8:25e06d4f:::benchmark_data_smithi187_136059_object105273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:33:16.494512+0000", 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 232.45212038299999, 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51392917599999999, 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:09.003 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 210554 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.494512+0000", 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.494512+0000", 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.004 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.494514+0000", 2024-09-02T10:37:09.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4479999999999999e-06 2024-09-02T10:37:09.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.005 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.005 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.494518+0000", 2024-09-02T10:37:09.005 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5820000000000003e-06 2024-09-02T10:37:09.005 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.005 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.494530+0000", 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1868000000000001e-05 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.494536+0000", 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0750000000000003e-06 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.494578+0000", 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2246000000000002e-05 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.006 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.494666+0000", 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7268000000000005e-05 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 9,10", 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.494740+0000", 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3798000000000001e-05 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.499679+0000", 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0049392459999999996 2024-09-02T10:37:09.007 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.500087+0000", 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00040827499999999999 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:17.008374+0000", 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.50828675099999998 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.008 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:17.008414+0000", 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9752000000000002e-05 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:17.008441+0000", 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.7866999999999999e-05 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:27192 9.8 e912/897 9:1354060c:::benchmark_data_smithi187_154927_object27191:head v 912'1739, mlcod=912'1737)", 2024-09-02T10:37:09.009 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:17.189989+0000", 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 171.75664330699999, 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30730787199999998, 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.189989+0000", 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.010 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.189989+0000", 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.189990+0000", 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6899999999999997e-07 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.190073+0000", 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3265999999999999e-05 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.011 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.496091+0000", 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30601773700000001 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.496108+0000", 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7465999999999998e-05 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.496217+0000", 2024-09-02T10:37:09.012 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108649 2024-09-02T10:37:09.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.013 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.013 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.496241+0000", 2024-09-02T10:37:09.013 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3572999999999999e-05 2024-09-02T10:37:09.013 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.013 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.497268+0000", 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001026931 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.497297+0000", 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9581e-05 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.014 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:27210 9.8 e912/897 9:1812d6c0:::benchmark_data_smithi187_154927_object27209:head v 912'1742, mlcod=912'1739)", 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:17.512126+0000", 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 171.43450647399999, 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37966033700000001, 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.512126+0000", 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.015 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.512126+0000", 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.512127+0000", 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5600000000000004e-07 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.512175+0000", 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7997000000000001e-05 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.016 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.890438+0000", 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.37826263500000001 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.890452+0000", 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4119e-05 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.890604+0000", 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000152621 2024-09-02T10:37:09.017 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.890655+0000", 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0735000000000001e-05 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.891747+0000", 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001092104 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:17.891786+0000", 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9169999999999999e-05 2024-09-02T10:37:09.018 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:70418 9.f 9:ff8a4b33:::benchmark_data_smithi187_154927_object70417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e930)", 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:59.109436+0000", 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 129.83719639899999, 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22075229499999999, 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:09.019 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 70418 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.109436+0000", 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.109436+0000", 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.020 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.109437+0000", 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7499999999999999e-07 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.109522+0000", 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4581999999999998e-05 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.109533+0000", 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1219e-05 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.021 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.109541+0000", 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8220000000000007e-06 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.111988+0000", 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024469520000000001 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.112142+0000", 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000154439 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.022 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.112277+0000", 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013494399999999999 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.119337+0000", 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0070595579999999996 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.119655+0000", 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00031826599999999999 2024-09-02T10:37:09.023 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.330098+0000", 2024-09-02T10:37:09.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21044270000000001 2024-09-02T10:37:09.024 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.024 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.024 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.024 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.330147+0000", 2024-09-02T10:37:09.024 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9710999999999999e-05 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.330189+0000", 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1226999999999997e-05 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:70423 9.f 9:fbb1e966:::benchmark_data_smithi187_154927_object70422:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e930)", 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:59.114306+0000", 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 129.832326962, 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22337052600000001, 2024-09-02T10:37:09.025 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 70423 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.114306+0000", 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.026 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.114306+0000", 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.114307+0000", 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.7010000000000001e-06 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.114385+0000", 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7645000000000006e-05 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.027 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.114398+0000", 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2975e-05 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.114405+0000", 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3350000000000001e-06 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.115954+0000", 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001548235 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.028 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.116054+0000", 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000100048 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.116174+0000", 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000120858 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.119342+0000", 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031672470000000002 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.029 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.125626+0000", 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0062845940000000001 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.337597+0000", 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21197076400000001 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.337650+0000", 2024-09-02T10:37:09.030 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2675e-05 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.337676+0000", 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.6449e-05 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:70433 9.f 9:f59530f8:::benchmark_data_smithi187_154927_object70432:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e930)", 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:59.121789+0000", 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 129.82484319299999, 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21693120299999999, 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.031 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 70433 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.121789+0000", 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.032 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.121789+0000", 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.121793+0000", 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6789999999999998e-06 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.121830+0000", 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6449999999999998e-05 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.033 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.121841+0000", 2024-09-02T10:37:09.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1871e-05 2024-09-02T10:37:09.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.121850+0000", 2024-09-02T10:37:09.034 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3780000000000001e-06 2024-09-02T10:37:09.034 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.034 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.034 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.034 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.121921+0000", 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0687999999999998e-05 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.122059+0000", 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013868100000000001 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.122187+0000", 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012752999999999999 2024-09-02T10:37:09.035 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.125657+0000", 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0034701950000000001 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.126912+0000", 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012549729999999999 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.036 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.338594+0000", 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21168162300000001 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.338655+0000", 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1203000000000003e-05 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.338721+0000", 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5932000000000005e-05 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.037 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:151224 9.f 9:f232989b:::benchmark_data_smithi187_154927_object53112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:21.767813+0000", 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.17881929, 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22285450100000001, 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:09.038 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 151224 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.767813+0000", 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.767813+0000", 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.767815+0000", 2024-09-02T10:37:09.039 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1599999999999999e-06 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.767820+0000", 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7690000000000002e-06 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.767829+0000", 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1039999999999996e-06 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.040 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.767836+0000", 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0589999999999997e-06 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.767990+0000", 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00015343900000000001 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768109+0000", 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119466 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.041 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768213+0000", 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010319600000000001 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.774197+0000", 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0059844879999999996 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.787776+0000", 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.013579206 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.042 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.990567+0000", 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.202790307 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.990624+0000", 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7961000000000003e-05 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.990668+0000", 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3346000000000001e-05 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.043 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:151225 9.5 9:a1d5e0a9:::benchmark_data_smithi187_154927_object53113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:21.768001+0000", 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.178632120000003, 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.217821814, 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 151225 2024-09-02T10:37:09.044 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.045 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768001+0000", 2024-09-02T10:37:09.045 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.045 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.045 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.045 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.045 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768001+0000", 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768001+0000", 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0800000000000004e-07 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768005+0000", 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8879999999999999e-06 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.046 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768015+0000", 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0141e-05 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768021+0000", 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2889999999999997e-06 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768066+0000", 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5133000000000003e-05 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.047 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768135+0000", 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9023999999999995e-05 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.768195+0000", 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0640000000000002e-05 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.770852+0000", 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002656304 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.048 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.776965+0000", 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0061137520000000001 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.985719+0000", 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20875395399999999 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.985782+0000", 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2780999999999994e-05 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.049 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.985822+0000", 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0200000000000001e-05 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:158192 9.a 9:5648f215:::benchmark_data_smithi187_154927_object60080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:33.616457+0000", 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 35.330175642, 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21677021099999999, 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.050 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 158192 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.616457+0000", 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.051 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.616457+0000", 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.616458+0000", 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3680000000000001e-06 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.616464+0000", 2024-09-02T10:37:09.052 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2889999999999997e-06 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.616475+0000", 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1119e-05 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.616482+0000", 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9330000000000004e-06 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.053 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.616540+0000", 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8245000000000002e-05 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.616642+0000", 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102528 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,7", 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.616737+0000", 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4370999999999995e-05 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.054 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.622087+0000", 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0053504679999999997 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.623407+0000", 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013199830000000001 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.833133+0000", 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20972557999999999 2024-09-02T10:37:09.055 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.833191+0000", 2024-09-02T10:37:09.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8196000000000003e-05 2024-09-02T10:37:09.056 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.056 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.056 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.056 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:33.833227+0000", 2024-09-02T10:37:09.056 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.6130999999999997e-05 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:158424 9.9 9:97db4827:::benchmark_data_smithi187_154927_object60312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:34.083690+0000", 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 34.862942302, 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22546139800000001, 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:09.057 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 158424 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.083690+0000", 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.083690+0000", 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.058 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.083692+0000", 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5600000000000001e-06 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.083697+0000", 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1100000000000002e-06 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.083721+0000", 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4261999999999999e-05 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.059 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.083729+0000", 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4660000000000002e-06 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.083821+0000", 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1860999999999993e-05 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.083962+0000", 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014134799999999999 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.060 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.084104+0000", 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014165599999999999 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.088394+0000", 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0042907470000000001 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.089995+0000", 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0016002519999999999 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.061 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.307960+0000", 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.217965462 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.308051+0000", 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1376000000000001e-05 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:34.309152+0000", 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001100298 2024-09-02T10:37:09.062 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:168474 9.6 9:6398c559:::benchmark_data_smithi187_154927_object70362:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e973)", 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:48.711768+0000", 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.234864318, 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22353018399999999, 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.063 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:09.063 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:09.064 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.064 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:09.064 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 168474 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.711768+0000", 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.711768+0000", 2024-09-02T10:37:09.065 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.711770+0000", 2024-09-02T10:37:09.066 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3740000000000001e-06 2024-09-02T10:37:09.066 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.066 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.066 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.066 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.711776+0000", 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3169999999999998e-06 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.711790+0000", 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3579e-05 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.711797+0000", 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1999999999999997e-06 2024-09-02T10:37:09.067 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.711857+0000", 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0674000000000002e-05 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.711966+0000", 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000108726 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,9", 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.712056+0000", 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9370000000000007e-05 2024-09-02T10:37:09.068 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.714741+0000", 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00268506 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.717438+0000", 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026976220000000002 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.934969+0000", 2024-09-02T10:37:09.069 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21753030200000001 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.935010+0000", 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1418999999999998e-05 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.935298+0000", 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00028854100000000001 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.070 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:168474 9.6 e973/962 ondisk, result = 0)", 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:48.716507+0000", 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 20.230125180000002, 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21880232599999999, 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.716507+0000", 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.071 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.716507+0000", 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.716508+0000", 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0440000000000001e-06 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.716514+0000", 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3879999999999999e-06 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.072 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.926711+0000", 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.210197411 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.926717+0000", 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9869999999999996e-06 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.934953+0000", 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.008235994 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.073 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.934962+0000", 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.2979999999999992e-06 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:48.935310+0000", 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00034820400000000002 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:181029 9.19 e986/935 9:9ae43988:::benchmark_data_smithi187_154927_object82917:head v 986'7740, mlcod=986'7736)", 2024-09-02T10:37:09.074 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:37:06.849364+0000", 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 2.0972690169999999, 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22288205999999999, 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:06.849364+0000", 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:06.849364+0000", 2024-09-02T10:37:09.075 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:06.849365+0000", 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.249e-06 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:06.849380+0000", 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5330999999999999e-05 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:07.069441+0000", 2024-09-02T10:37:09.076 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22006042200000001 2024-09-02T10:37:09.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:09.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:07.069445+0000", 2024-09-02T10:37:09.077 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4100000000000001e-06 2024-09-02T10:37:09.077 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.077 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.077 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:09.077 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:07.069530+0000", 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5259999999999993e-05 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:07.069550+0000", 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9639e-05 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:07.072206+0000", 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026555989999999998 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:09.078 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:09.079 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:07.072246+0000", 2024-09-02T10:37:09.079 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.015e-05 2024-09-02T10:37:09.079 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.079 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.079 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.079 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:09.079 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:09.079 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:37:09.079 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:08 smithi162 ceph-mon[20279]: pgmap v2251: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 54 active+clean; 1.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 248/49926 objects degraded (0.497%); 468/49926 objects misplaced (0.937%); 1.2 MiB/s, 274 objects/s recovering 2024-09-02T10:37:09.079 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:08 smithi162 ceph-mon[20279]: from='client.28342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:37:09.164 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:08 smithi151 ceph-mon[16639]: pgmap v2251: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 54 active+clean; 1.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 248/49926 objects degraded (0.497%); 468/49926 objects misplaced (0.937%); 1.2 MiB/s, 274 objects/s recovering 2024-09-02T10:37:09.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:08 smithi151 ceph-mon[16639]: from='client.28342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:37:09.193 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:09.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:09.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:08 smithi187 ceph-mon[20072]: pgmap v2251: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 54 active+clean; 1.0 GiB data, 12 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 678 op/s; 248/49926 objects degraded (0.497%); 468/49926 objects misplaced (0.937%); 1.2 MiB/s, 274 objects/s recovering 2024-09-02T10:37:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:08 smithi187 ceph-mon[20072]: from='client.28342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:37:09.294 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:09.323 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:09.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:09.425 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:09.457 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:09.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:09.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:09.470 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:09.559 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:09.588 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:09.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:09.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:09.690 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:09.716 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:09.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:09.717 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:09.818 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:09.849 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:09.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:09.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:09.951 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:09.982 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:09.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:09.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:10.082 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-02T10:37:10.084 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:10.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:37:10.105 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:10.106 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:10.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:09 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 248/49926 objects degraded (0.497%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:10.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:10.229 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:10.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:10.252 INFO:teuthology.orchestra.run.smithi151.stderr:Error EINVAL: specified pgp_num 71 > pg_num 31 2024-09-02T10:37:10.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:09 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 248/49926 objects degraded (0.497%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:10.331 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:10.337 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:37:10.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:09 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 248/49926 objects degraded (0.497%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:10.355 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:10.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:10.457 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:10.479 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:10.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:10.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:10.581 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:10.602 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:10.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:10.752 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:10.773 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:10.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:10.875 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:10.896 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:10.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:10.897 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:10.998 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:11.023 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:11.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:11.103 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:11.125 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:11.130 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:10 smithi162 ceph-mon[20279]: pgmap v2252: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 54 active+clean; 926 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 793 op/s; 248/44451 objects degraded (0.558%); 468/44451 objects misplaced (1.053%); 861 KiB/s, 192 objects/s recovering 2024-09-02T10:37:11.130 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:10 smithi162 ceph-mon[20279]: osdmap e988: 12 total, 12 up, 11 in 2024-09-02T10:37:11.130 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1808670878' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"} : dispatch 2024-09-02T10:37:11.130 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:10 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"} : dispatch 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:184316 9.13 9:cc0e3951:::benchmark_data_smithi187_154927_object86204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e989)", 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:37:11.117883+0000", 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.011105459, 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011136959, 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "waiting for sub ops", 2024-09-02T10:37:11.131 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 184316 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.117883+0000", 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:11.132 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.117883+0000", 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.117885+0000", 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3209999999999999e-06 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.117890+0000", 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3340000000000001e-06 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.133 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.117902+0000", 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1858000000000001e-05 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.117910+0000", 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.588e-06 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.117972+0000", 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2650999999999994e-05 2024-09-02T10:37:11.134 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.118083+0000", 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011049599999999999 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.118187+0000", 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000103899 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.120980+0000", 2024-09-02T10:37:11.135 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027929769999999999 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:11.123150+0000", 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021706099999999999 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: ], 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout: "num_ops": 1 2024-09-02T10:37:11.136 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:37:11.150 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:11.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:11.151 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:11.252 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:10 smithi187 ceph-mon[20072]: pgmap v2252: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 54 active+clean; 926 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 793 op/s; 248/44451 objects degraded (0.558%); 468/44451 objects misplaced (1.053%); 861 KiB/s, 192 objects/s recovering 2024-09-02T10:37:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:10 smithi187 ceph-mon[20072]: osdmap e988: 12 total, 12 up, 11 in 2024-09-02T10:37:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1808670878' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"} : dispatch 2024-09-02T10:37:11.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:10 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"} : dispatch 2024-09-02T10:37:11.281 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:11.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:11.343 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:37:11.375 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:10 smithi151 ceph-mon[16639]: pgmap v2252: 57 pgs: 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 54 active+clean; 926 MiB data, 11 GiB used, 972 GiB / 983 GiB avail; 0 B/s wr, 793 op/s; 248/44451 objects degraded (0.558%); 468/44451 objects misplaced (1.053%); 861 KiB/s, 192 objects/s recovering 2024-09-02T10:37:11.375 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:10 smithi151 ceph-mon[16639]: osdmap e988: 12 total, 12 up, 11 in 2024-09-02T10:37:11.375 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1808670878' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"} : dispatch 2024-09-02T10:37:11.375 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:10 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "71"} : dispatch 2024-09-02T10:37:11.383 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:11.418 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:11.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:11.520 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:11.555 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:11.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:11.657 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:11.678 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:11.679 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:11.780 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:11.806 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:11.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:11.858 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-02T10:37:11.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:11.932 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:11.933 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:12.034 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:12.058 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:12.059 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:11 smithi162 ceph-mon[20279]: osdmap e989: 12 total, 12 up, 11 in 2024-09-02T10:37:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:37:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:37:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:37:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:37:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:37:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:37:12.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:37:12.160 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:12.187 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:12.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:12.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:12.209 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:11 smithi151 ceph-mon[16639]: osdmap e989: 12 total, 12 up, 11 in 2024-09-02T10:37:12.209 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:37:12.209 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:37:12.209 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:37:12.210 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:37:12.210 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:37:12.210 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:37:12.210 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:37:12.237 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:37:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:11 smithi187 ceph-mon[20072]: osdmap e989: 12 total, 12 up, 11 in 2024-09-02T10:37:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:37:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:37:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:37:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:37:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:37:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:37:12.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:37:12.290 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:12.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:12.317 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:12.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:12.419 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:12.444 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:12.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:12.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:12.546 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:12.605 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:12.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:12.707 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:12.733 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:12.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:12.734 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:12.806 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:37:12.806 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [], 2024-09-02T10:37:12.806 INFO:teuthology.orchestra.run.smithi162.stdout: "complaint_time": 30, 2024-09-02T10:37:12.806 INFO:teuthology.orchestra.run.smithi162.stdout: "num_blocked_ops": 0 2024-09-02T10:37:12.806 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:37:12.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:12.858 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:12.859 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:12.960 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:12.983 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:12.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:13.085 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:13.110 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:13.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:13.212 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:13.242 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:13.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:13.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:12 smithi187 ceph-mon[20072]: pgmap v2255: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 831 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 248/39876 objects degraded (0.622%); 90/39876 objects misplaced (0.226%); 3.9 MiB/s, 184 objects/s recovering 2024-09-02T10:37:13.280 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:12 smithi162 ceph-mon[20279]: pgmap v2255: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 831 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 248/39876 objects degraded (0.622%); 90/39876 objects misplaced (0.226%); 3.9 MiB/s, 184 objects/s recovering 2024-09-02T10:37:13.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:13.366 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:13.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:13.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:12 smithi151 ceph-mon[16639]: pgmap v2255: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 831 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 770 op/s; 248/39876 objects degraded (0.622%); 90/39876 objects misplaced (0.226%); 3.9 MiB/s, 184 objects/s recovering 2024-09-02T10:37:13.467 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:13.489 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:13.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:13.591 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:13.617 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:13.618 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:13.650 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-02T10:37:13.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:13.746 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:13.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:13.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:13.848 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:13.886 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:13.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:13.988 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:14.015 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:14.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:14.045 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:37:14.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:14.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:14.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:14.142 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:14.243 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:14.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:14.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:14.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:14.403 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:14.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:14.505 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:14.539 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:14.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:14.640 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:14.659 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(8.3 869/844 [PushOp(8:c5892a6c:::benchmark_data_smithi187_136059_object80224:head, version: 823'5023, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:c5892a6c:::benchmark_data_smithi187_136059_object80224:head@823'5023, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:32:29.348403+0000", 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 285.30787611199997, 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21878360499999999, 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:14.660 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.348403+0000", 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.348403+0000", 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.348404+0000", 2024-09-02T10:37:14.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0550000000000001e-06 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.348452+0000", 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7568000000000003e-05 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564206+0000", 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21575388700000001 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564225+0000", 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9377000000000001e-05 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564314+0000", 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.8251999999999995e-05 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.564323+0000", 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.0000000000000002e-06 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:32:29.567187+0000", 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002864466 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(8.3 e883,844 8:cf663c8e:::benchmark_data_smithi187_136059_object26166:head,841'8229)", 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:33:20.314918+0000", 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 234.341361304, 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.79838157499999995, 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.664 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:20.314918+0000", 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:20.314918+0000", 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:20.314919+0000", 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7000000000000004e-07 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:20.314923+0000", 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2429999999999999e-06 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.113040+0000", 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.798116771 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.113057+0000", 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6923e-05 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.113201+0000", 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014450899999999999 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.113213+0000", 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2075e-05 2024-09-02T10:37:14.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.113300+0000", 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6384000000000005e-05 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGRecoveryDelete(8.3 e883,844 8:ca2eb16a:::benchmark_data_smithi187_136059_object27945:head,842'8339)", 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:33:21.817622+0000", 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 232.83865777099999, 2024-09-02T10:37:14.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35858564599999998, 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.817622+0000", 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.817622+0000", 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.817628+0000", 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3940000000000001e-06 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:21.817638+0000", 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.893e-06 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:22.175898+0000", 2024-09-02T10:37:14.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35825989899999999 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:22.175919+0000", 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1100000000000001e-05 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:22.176081+0000", 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000161767 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:22.176097+0000", 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6180000000000001e-05 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:22.176207+0000", 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110413 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:80854 9.0 9:04a09027:::benchmark_data_smithi187_154927_object80853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e932)", 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:35:10.277362+0000", 2024-09-02T10:37:14.672 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 124.378917906, 2024-09-02T10:37:14.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.22093997900000001, 2024-09-02T10:37:14.673 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.673 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:14.674 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.674 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:14.674 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:14.674 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 80854 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.277362+0000", 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.277362+0000", 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.277363+0000", 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2720000000000001e-06 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.277408+0000", 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5192999999999999e-05 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.277419+0000", 2024-09-02T10:37:14.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0967e-05 2024-09-02T10:37:14.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.277425+0000", 2024-09-02T10:37:14.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8089999999999998e-06 2024-09-02T10:37:14.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.277487+0000", 2024-09-02T10:37:14.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.2383000000000002e-05 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.277637+0000", 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014966099999999999 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.277800+0000", 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000163318 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.678 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:14.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.279120+0000", 2024-09-02T10:37:14.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0013197269999999999 2024-09-02T10:37:14.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.279843+0000", 2024-09-02T10:37:14.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00072298600000000001 2024-09-02T10:37:14.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.498199+0000", 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21835603000000001 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.498261+0000", 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1526999999999996e-05 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:10.498302+0000", 2024-09-02T10:37:14.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1106000000000003e-05 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:127423 9.0 9:0755a8e0:::benchmark_data_smithi187_154927_object29311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e953)", 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:00.189000+0000", 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.467279511000001, 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20421281899999999, 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.681 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 127423 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.189000+0000", 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.189000+0000", 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.189002+0000", 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0420000000000001e-06 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.189008+0000", 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4770000000000002e-06 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.189021+0000", 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3839000000000001e-05 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.189028+0000", 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0520000000000004e-06 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.189090+0000", 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1666000000000002e-05 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.189209+0000", 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011917999999999999 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.189314+0000", 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000104566 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.191440+0000", 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021256510000000001 2024-09-02T10:37:14.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.191460+0000", 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0707999999999999e-05 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.393101+0000", 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20164121099999999 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.393165+0000", 2024-09-02T10:37:14.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3676999999999994e-05 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.393213+0000", 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.7750000000000002e-05 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:127423 9.0 e953/924 ondisk, result = 0)", 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:00.191724+0000", 2024-09-02T10:37:14.687 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 74.464555378, 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20149668200000001, 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.191724+0000", 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.191724+0000", 2024-09-02T10:37:14.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.191725+0000", 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0699999999999996e-07 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.191729+0000", 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1019999999999999e-06 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.393021+0000", 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.201292151 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.393037+0000", 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.56e-05 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.393088+0000", 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0813999999999998e-05 2024-09-02T10:37:14.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.393096+0000", 2024-09-02T10:37:14.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3149999999999992e-06 2024-09-02T10:37:14.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.691 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:00.393221+0000", 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000124993 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:157101 9.16 e965/947 9:69843baf:::benchmark_data_smithi187_154927_object58989:head v 965'6822, mlcod=965'6819)", 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:32.176714+0000", 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 42.479565467, 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20651987599999999, 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.692 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.693 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.693 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:32.176714+0000", 2024-09-02T10:37:14.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:32.176714+0000", 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:32.176715+0000", 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6000000000000002e-07 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:32.176722+0000", 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2649999999999999e-06 2024-09-02T10:37:14.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:32.380805+0000", 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20408285000000001 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:32.380812+0000", 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.294e-06 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:32.380862+0000", 2024-09-02T10:37:14.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.9190999999999997e-05 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:32.380874+0000", 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2813000000000001e-05 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:32.383189+0000", 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023150549999999999 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:32.383234+0000", 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4548000000000001e-05 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:159048 9.11 e965/933 9:8f5b66f7:::benchmark_data_smithi187_154927_object60936:head v 965'6627, mlcod=965'6624)", 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:35.029697+0000", 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.626582229999997, 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.288991944, 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.697 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029697+0000", 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029697+0000", 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029698+0000", 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5000000000000001e-07 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029707+0000", 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.3499999999999997e-06 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315983+0000", 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28627594699999998 2024-09-02T10:37:14.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315989+0000", 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3999999999999997e-06 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.316197+0000", 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00020794100000000001 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.316225+0000", 2024-09-02T10:37:14.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8286999999999999e-05 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.318596+0000", 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023710649999999999 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.318689+0000", 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3004000000000006e-05 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:159264 9.11 e965/933 9:8a51152d:::benchmark_data_smithi187_154927_object61152:head v 965'6628, mlcod=965'6625)", 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:35.315957+0000", 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 39.340322702000002, 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31677973500000001, 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315957+0000", 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315957+0000", 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315958+0000", 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7199999999999998e-07 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315965+0000", 2024-09-02T10:37:14.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9979999999999997e-06 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.622049+0000", 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30608423600000001 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.622055+0000", 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6989999999999997e-06 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.629807+0000", 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0077514790000000004 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.629907+0000", 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9636000000000006e-05 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.632703+0000", 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027960900000000002 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.632737+0000", 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3924999999999998e-05 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:159874 9.0 9:0512ef68:::benchmark_data_smithi187_154927_object61762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e966)", 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:36.149008+0000", 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.507271121000002, 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31198891000000001, 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.706 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 159874 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.149008+0000", 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.149008+0000", 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.149010+0000", 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9240000000000001e-06 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.149018+0000", 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7600000000000002e-06 2024-09-02T10:37:14.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.149036+0000", 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.8244e-05 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.149046+0000", 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9000000000000001e-06 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.149113+0000", 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7080000000000001e-05 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.149261+0000", 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000148154 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.149389+0000", 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012716100000000001 2024-09-02T10:37:14.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.151799+0000", 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002410092 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.152389+0000", 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059073100000000005 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.460886+0000", 2024-09-02T10:37:14.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30849674500000002 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.460946+0000", 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.9488999999999999e-05 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.460997+0000", 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1629999999999999e-05 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:159874 9.0 e966/924 ondisk, result = 0)", 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:36.151904+0000", 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.504375101999997, 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30910514300000003, 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.151904+0000", 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.151904+0000", 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.151906+0000", 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1850000000000001e-06 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.151913+0000", 2024-09-02T10:37:14.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3100000000000003e-06 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.460792+0000", 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30887890299999998 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.460801+0000", 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7879999999999995e-06 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.460869+0000", 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8596999999999994e-05 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.460879+0000", 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.4010000000000003e-06 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:36.461010+0000", 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013095899999999999 2024-09-02T10:37:14.716 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:165685 9.13 9:cdfa7346:::benchmark_data_smithi187_154927_object67573:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e971)", 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:44.716673+0000", 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.939606447999999, 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.204669306, 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:14.717 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 165685 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.716673+0000", 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.716673+0000", 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.719 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:14.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:14.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.716675+0000", 2024-09-02T10:37:14.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.782e-06 2024-09-02T10:37:14.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.716681+0000", 2024-09-02T10:37:14.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0780000000000001e-06 2024-09-02T10:37:14.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.716695+0000", 2024-09-02T10:37:14.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3988000000000001e-05 2024-09-02T10:37:14.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.716703+0000", 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7370000000000006e-06 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.716770+0000", 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7484999999999995e-05 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.716904+0000", 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013399500000000001 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.717001+0000", 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7107000000000004e-05 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.719761+0000", 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027595810000000001 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.720479+0000", 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00071799799999999997 2024-09-02T10:37:14.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.921235+0000", 2024-09-02T10:37:14.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20075620599999999 2024-09-02T10:37:14.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.921296+0000", 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1049000000000005e-05 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.921342+0000", 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6300000000000001e-05 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:165797 9.13 9:cfd68e90:::benchmark_data_smithi187_154927_object67685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e971)", 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:44.916041+0000", 2024-09-02T10:37:14.725 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 29.740238585, 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.196220845, 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 165797 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.916041+0000", 2024-09-02T10:37:14.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.916041+0000", 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.916043+0000", 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.04e-06 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.916049+0000", 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.1789999999999996e-06 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.916065+0000", 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6141e-05 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.916073+0000", 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8730000000000007e-06 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.916141+0000", 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7959000000000004e-05 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.916250+0000", 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010846299999999999 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.916346+0000", 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6251999999999999e-05 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.919084+0000", 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027385999999999999 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.921581+0000", 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002496421 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:45.112159+0000", 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.19057829700000001 2024-09-02T10:37:14.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:45.112216+0000", 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.7052000000000002e-05 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:45.112262+0000", 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5568000000000001e-05 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:173228 9.16 e980/947 9:6e47ec70:::benchmark_data_smithi187_154927_object75116:head v 980'7339, mlcod=980'7337)", 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:56.712369+0000", 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 17.943910893999998, 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21034608599999999, 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:56.712369+0000", 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:56.712369+0000", 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:56.712371+0000", 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.95e-06 2024-09-02T10:37:14.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:56.712385+0000", 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4306e-05 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:56.920061+0000", 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.20767608200000001 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:56.920069+0000", 2024-09-02T10:37:14.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6520000000000006e-06 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:56.920156+0000", 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.7108999999999996e-05 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:56.920179+0000", 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2886000000000001e-05 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:56.922680+0000", 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025008859999999999 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:56.922715+0000", 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5215000000000001e-05 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.736 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:184049 9.13 9:caa43e75:::benchmark_data_smithi187_154927_object85937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e988)", 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:37:10.031889+0000", 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.624390236, 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25151836500000002, 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 184049 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.737 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.031889+0000", 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.031889+0000", 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.738 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.031891+0000", 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0870000000000001e-06 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.031899+0000", 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0199999999999994e-06 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.031921+0000", 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1104999999999999e-05 2024-09-02T10:37:14.739 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.031931+0000", 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0282000000000001e-05 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.031995+0000", 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4335e-05 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.740 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.032086+0000", 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.1182999999999998e-05 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.032154+0000", 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7834000000000001e-05 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.034390+0000", 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002236123 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.741 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.034969+0000", 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00057912999999999997 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.283350+0000", 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24838091200000001 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.283383+0000", 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2780999999999997e-05 2024-09-02T10:37:14.742 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.283408+0000", 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.4573e-05 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:184049 9.13 e988/949 ondisk, result = 0)", 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:37:10.035191+0000", 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.621088361, 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24821964599999999, 2024-09-02T10:37:14.743 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.035191+0000", 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.035191+0000", 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.744 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.035192+0000", 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7299999999999995e-07 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.035196+0000", 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0720000000000001e-06 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.283300+0000", 2024-09-02T10:37:14.745 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.24810429000000001 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.283304+0000", 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.738e-06 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.283343+0000", 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8699999999999999e-05 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.746 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.283347+0000", 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3800000000000004e-06 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.283411+0000", 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3793000000000004e-05 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:184071 9.0 9:0231537d:::benchmark_data_smithi187_154927_object85959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e988)", 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:37:10.074639+0000", 2024-09-02T10:37:14.747 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.5816407330000004, 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.17216965200000001, 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 184071 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.748 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.074639+0000", 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.074639+0000", 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.074641+0000", 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.2069999999999998e-06 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.749 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.074650+0000", 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.49e-06 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.074671+0000", 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1429000000000001e-05 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.074681+0000", 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6770000000000007e-06 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.750 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.074754+0000", 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3671999999999995e-05 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.074905+0000", 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000151061 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.075027+0000", 2024-09-02T10:37:14.751 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000121449 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.078091+0000", 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0030637379999999999 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.078130+0000", 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9014000000000003e-05 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.752 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.246737+0000", 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.16860736000000001 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.246778+0000", 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.0742999999999999e-05 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.246808+0000", 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.0812000000000002e-05 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.753 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:184142 9.13 9:cbdae675:::benchmark_data_smithi187_154927_object86030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e988)", 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:37:10.394697+0000", 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.2615821330000001, 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.15236309100000001, 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:14.754 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 184142 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.394697+0000", 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.394697+0000", 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.755 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.394699+0000", 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.344e-06 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.394704+0000", 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1549999999999998e-06 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.394715+0000", 2024-09-02T10:37:14.756 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1511e-05 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.394722+0000", 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.251e-06 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.394775+0000", 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3052e-05 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.757 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.394854+0000", 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8977e-05 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 0,5", 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.394920+0000", 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6445000000000005e-05 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.397061+0000", 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021409139999999998 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.758 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.397652+0000", 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00059077299999999995 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.546946+0000", 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.149294537 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.547032+0000", 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.5158e-05 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.759 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.547061+0000", 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.8974000000000001e-05 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:184142 9.13 e988/949 ondisk, result = 0)", 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:37:10.397690+0000", 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 4.258589196, 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.149374743, 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.760 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.397690+0000", 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.397690+0000", 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.761 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.397691+0000", 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3900000000000004e-07 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.397695+0000", 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.5999999999999998e-06 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.546799+0000", 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.149104662 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.762 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.546812+0000", 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3213999999999999e-05 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.546920+0000", 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107478 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.546935+0000", 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4882000000000001e-05 2024-09-02T10:37:14.763 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:10.547065+0000", 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00013026800000000001 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:186715 9.0 9:07249da8:::benchmark_data_smithi187_154927_object88603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e989)", 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:37:13.710801+0000", 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 0.94547864999999998, 2024-09-02T10:37:14.764 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.062620720000000005, 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 186715 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.710801+0000", 2024-09-02T10:37:14.765 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.710801+0000", 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.710803+0000", 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.787e-06 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:14.766 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.710810+0000", 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6850000000000001e-06 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.710831+0000", 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0322e-05 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.710841+0000", 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0241000000000001e-05 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.767 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.710920+0000", 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8974000000000007e-05 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.711060+0000", 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000139622 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 1,8", 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.711168+0000", 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010825800000000001 2024-09-02T10:37:14.768 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.714558+0000", 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.003390536 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.716517+0000", 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.001958619 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.773274+0000", 2024-09-02T10:37:14.769 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.056757223000000002 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.773371+0000", 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6645999999999996e-05 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.773422+0000", 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0807000000000003e-05 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:14.770 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:14.771 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:37:14.820 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:14.847 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:14.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:14.949 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:14.976 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:14.977 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:15.028 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-02T10:37:15.078 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:15.105 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:15.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:15.207 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:15.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:14 smithi162 ceph-mon[20279]: pgmap v2256: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 710 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 248/34080 objects degraded (0.728%); 90/34080 objects misplaced (0.264%); 3.5 MiB/s, 92 objects/s recovering 2024-09-02T10:37:15.228 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:14 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 248/34080 objects degraded (0.728%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:15.235 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:15.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:14 smithi187 ceph-mon[20072]: pgmap v2256: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 710 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 248/34080 objects degraded (0.728%); 90/34080 objects misplaced (0.264%); 3.5 MiB/s, 92 objects/s recovering 2024-09-02T10:37:15.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:14 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 248/34080 objects degraded (0.728%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:15.336 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:15.363 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:15.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:14 smithi151 ceph-mon[16639]: pgmap v2256: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 710 MiB data, 11 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 819 op/s; 248/34080 objects degraded (0.728%); 90/34080 objects misplaced (0.264%); 3.5 MiB/s, 92 objects/s recovering 2024-09-02T10:37:15.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:14 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 248/34080 objects degraded (0.728%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:15.465 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:15.488 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:15.489 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:15.590 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:15.615 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:15.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:15.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:15.717 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:15.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:15.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:15.750 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-02T10:37:15.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:15.877 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:15.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:15.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:15.979 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:16.012 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:16.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:16.050 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:16.088 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:37:16.088 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:37:16.114 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:16.118 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:37:16.118 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:37:16.137 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:37:16.146 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:37:16.147 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:37:16.149 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:16.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:16.175 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:37:16.175 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:37:16.209 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:37:16.210 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:37:16.246 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:37:16.247 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:37:16.251 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:16.279 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:37:16.280 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:37:16.282 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:16.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:16.312 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:37:16.312 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:37:16.338 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:37:16.338 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:37:16.370 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:37:16.370 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:37:16.383 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:16.401 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:37:16.401 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:37:16.409 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:16.410 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:16.479 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:37:16.479 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:37:16.510 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:16.543 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:16.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:16.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:16.646 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:16.678 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:16.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:16.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:16.781 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:16.812 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:16.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:16.914 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:16.937 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:16.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:16.938 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:16.979 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:37:17.039 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:17.062 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:17.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:17.164 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:17.191 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:17.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:16 smithi187 ceph-mon[20072]: pgmap v2257: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 623 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 835 op/s; 248/29865 objects degraded (0.830%); 3.7 MiB/s, 126 objects/s recovering 2024-09-02T10:37:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3772398898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/306678429' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:37:17.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:16 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:37:17.292 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:17.320 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:17.321 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:17.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:16 smithi151 ceph-mon[16639]: pgmap v2257: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 623 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 835 op/s; 248/29865 objects degraded (0.830%); 3.7 MiB/s, 126 objects/s recovering 2024-09-02T10:37:17.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3772398898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:17.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/306678429' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:37:17.322 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:16 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:37:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:16 smithi162 ceph-mon[20279]: pgmap v2257: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 623 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 835 op/s; 248/29865 objects degraded (0.830%); 3.7 MiB/s, 126 objects/s recovering 2024-09-02T10:37:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3772398898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/306678429' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:37:17.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:16 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:37:17.422 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:17.448 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:17.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:17.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:17.550 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:17.580 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:17.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:17.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:17.682 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:17.706 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:17.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:17.707 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:17.808 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:17.833 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:17.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:17.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:17.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:17.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:17.963 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:17.963 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:18.064 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:18.100 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:18.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:18.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:18.201 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:18.224 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:18.225 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:17 smithi187 ceph-mon[20072]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:17 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:37:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:17 smithi187 ceph-mon[20072]: osdmap e990: 12 total, 12 up, 11 in 2024-09-02T10:37:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:17 smithi187 ceph-mon[20072]: 9.10 scrub starts 2024-09-02T10:37:18.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:17 smithi187 ceph-mon[20072]: 9.10 scrub ok 2024-09-02T10:37:18.269 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:17 smithi162 ceph-mon[20279]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:18.269 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:17 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:37:18.269 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:17 smithi162 ceph-mon[20279]: osdmap e990: 12 total, 12 up, 11 in 2024-09-02T10:37:18.269 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:17 smithi162 ceph-mon[20279]: 9.10 scrub starts 2024-09-02T10:37:18.269 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:17 smithi162 ceph-mon[20279]: 9.10 scrub ok 2024-09-02T10:37:18.326 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:18.351 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:18.352 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:18.418 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:37:18.418 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:37:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:17 smithi151 ceph-mon[16639]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:17 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:37:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:17 smithi151 ceph-mon[16639]: osdmap e990: 12 total, 12 up, 11 in 2024-09-02T10:37:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:17 smithi151 ceph-mon[16639]: 9.10 scrub starts 2024-09-02T10:37:18.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:17 smithi151 ceph-mon[16639]: 9.10 scrub ok 2024-09-02T10:37:18.453 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:18.476 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:18.477 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:18.578 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:18.603 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:18.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:18.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:18.730 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:18.731 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:18.832 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:18.861 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:18.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:18.962 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:18.984 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:18.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:19.086 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:19.107 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:19.108 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:19.209 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:19.232 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:19.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:18 smithi187 ceph-mon[20072]: pgmap v2259: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 523 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 248/25101 objects degraded (0.988%); 748 KiB/s, 82 objects/s recovering 2024-09-02T10:37:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:18 smithi187 ceph-mon[20072]: 9.e scrub starts 2024-09-02T10:37:19.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:18 smithi187 ceph-mon[20072]: 9.e scrub ok 2024-09-02T10:37:19.268 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:18 smithi162 ceph-mon[20279]: pgmap v2259: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 523 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 248/25101 objects degraded (0.988%); 748 KiB/s, 82 objects/s recovering 2024-09-02T10:37:19.268 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:19 smithi162 ceph-mon[20279]: 9.e scrub starts 2024-09-02T10:37:19.269 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:19 smithi162 ceph-mon[20279]: 9.e scrub ok 2024-09-02T10:37:19.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:19.357 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:19.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:18 smithi151 ceph-mon[16639]: pgmap v2259: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 523 MiB data, 10 GiB used, 973 GiB / 983 GiB avail; 0 B/s wr, 737 op/s; 248/25101 objects degraded (0.988%); 748 KiB/s, 82 objects/s recovering 2024-09-02T10:37:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:18 smithi151 ceph-mon[16639]: 9.e scrub starts 2024-09-02T10:37:19.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:18 smithi151 ceph-mon[16639]: 9.e scrub ok 2024-09-02T10:37:19.459 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:19.486 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:19.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:19.588 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:19.612 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:19.613 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:19.678 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:37:19.714 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:19.749 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:19.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:19.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:19.852 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:19.885 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:19.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:19.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:19.886 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-02T10:37:19.987 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:20.009 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:20.016 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:20.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:20.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:20.029 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 3, 5, 6, 7, 8, 9, 4, 2, 10] out_osds: [11] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-09-02T10:37:20.029 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-02T10:37:20.030 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-02T10:37:20.030 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-02T10:37:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:19 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:37:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:19 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:37:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:19 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 248/25101 objects degraded (0.988%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:20.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:19 smithi162 ceph-mon[20279]: osdmap e991: 12 total, 12 up, 11 in 2024-09-02T10:37:20.118 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:20.141 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:20.142 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:20.180 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:37:20.242 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:19 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:37:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:19 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:37:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:19 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 248/25101 objects degraded (0.988%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:20.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:19 smithi187 ceph-mon[20072]: osdmap e991: 12 total, 12 up, 11 in 2024-09-02T10:37:20.275 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:20.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:20.334 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:37:20.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:19 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:37:20.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:19 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:37:20.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:19 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 248/25101 objects degraded (0.988%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:20.345 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:19 smithi151 ceph-mon[16639]: osdmap e991: 12 total, 12 up, 11 in 2024-09-02T10:37:20.377 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:20.428 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:20.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:20.530 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:20.560 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:20.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:20.656 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:37:20.656 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:37:20.660 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:20.662 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:20.685 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:37:20.685 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:37:20.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:20.697 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:20.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:20.717 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:37:20.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:37:20.745 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:37:20.745 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:37:20.775 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:37:20.776 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:37:20.798 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:20.807 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:37:20.807 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:37:20.831 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:20.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:20.835 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:37:20.835 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:37:20.934 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:20.962 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:20.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:20.970 INFO:teuthology.orchestra.run.smithi162.stdout:ERROR: (22) Invalid argument 2024-09-02T10:37:20.970 INFO:teuthology.orchestra.run.smithi162.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:37:21.063 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:21.089 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:21.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:21.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:20 smithi162 ceph-mon[20279]: pgmap v2261: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 388 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 885 op/s; 248/18603 objects degraded (1.333%); 720 KiB/s, 49 objects/s recovering 2024-09-02T10:37:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:20 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:37:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:20 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:37:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:20 smithi162 ceph-mon[20279]: 9.15 scrub starts 2024-09-02T10:37:21.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2060486945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:21.191 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:21.197 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:21.232 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:21.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:20 smithi187 ceph-mon[20072]: pgmap v2261: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 388 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 885 op/s; 248/18603 objects degraded (1.333%); 720 KiB/s, 49 objects/s recovering 2024-09-02T10:37:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:20 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:37:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:20 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:37:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:20 smithi187 ceph-mon[20072]: 9.15 scrub starts 2024-09-02T10:37:21.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2060486945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:21.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:21.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:20 smithi151 ceph-mon[16639]: pgmap v2261: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 388 MiB data, 9.7 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 885 op/s; 248/18603 objects degraded (1.333%); 720 KiB/s, 49 objects/s recovering 2024-09-02T10:37:21.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:20 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:37:21.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:20 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:37:21.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:20 smithi151 ceph-mon[16639]: 9.15 scrub starts 2024-09-02T10:37:21.335 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2060486945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:21.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:21.365 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:21.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:21.465 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:21.501 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:21.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:21.522 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout:{ 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "epoch": 991, 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "fsid": "1530c7be-6911-11ef-bcd4-c7b262605968", 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "created": "2024-09-02T09:53:53.308187+0000", 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "modified": "2024-09-02T10:37:19.405550+0000", 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "last_up_change": "2024-09-02T10:00:25.816021+0000", 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "last_in_change": "2024-09-02T10:36:50.796061+0000", 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_num": 5804032, 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_set": [ 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "nodeep-scrub", 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "pglog_hardlimit", 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "purged_snapdirs", 2024-09-02T10:37:21.523 INFO:teuthology.orchestra.run.smithi151.stdout: "recovery_deletes", 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "sortbitwise" 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_version": 26, 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "full_ratio": 0.94999998807907104, 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_snapshot": "", 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_max": 9, 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "max_osd": 12, 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "require_min_compat_client": "luminous", 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "min_compat_client": "luminous", 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "require_osd_release": "squid", 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "allow_crimson": false, 2024-09-02T10:37:21.524 INFO:teuthology.orchestra.run.smithi151.stdout: "pools": [ 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 1, 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": ".mgr", 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T09:56:53.284814+0000", 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:37:21.525 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 31, 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 31, 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 31, 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 31, 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 31, 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:37:21.526 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "264", 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "241", 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:37:21.527 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:37:21.528 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "options": { 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_max": 32, 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_min": 1 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.529 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "mgr": {} 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 1.7699999809265137, 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 1.7699999809265137, 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 1.7699999809265137, 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 1.7699999809265137, 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.88373178243637085, 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.530 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.531 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.531 INFO:teuthology.orchestra.run.smithi151.stdout: "pool": 9, 2024-09-02T10:37:21.531 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_name": "unique_pool_7", 2024-09-02T10:37:21.531 INFO:teuthology.orchestra.run.smithi151.stdout: "create_time": "2024-09-02T10:33:50.981540+0000", 2024-09-02T10:37:21.531 INFO:teuthology.orchestra.run.smithi151.stdout: "flags": 1, 2024-09-02T10:37:21.531 INFO:teuthology.orchestra.run.smithi151.stdout: "flags_names": "hashpspool", 2024-09-02T10:37:21.531 INFO:teuthology.orchestra.run.smithi151.stdout: "type": 1, 2024-09-02T10:37:21.531 INFO:teuthology.orchestra.run.smithi151.stdout: "size": 3, 2024-09-02T10:37:21.531 INFO:teuthology.orchestra.run.smithi151.stdout: "min_size": 2, 2024-09-02T10:37:21.531 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_rule": 0, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_count": 0, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_target": 0, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_barrier": 0, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "is_stretch_pool": false, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "object_hash": 2, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_autoscale_mode": "off", 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num": 26, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num": 26, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_placement_num_target": 26, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_target": 26, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_num_pending": 26, 2024-09-02T10:37:21.532 INFO:teuthology.orchestra.run.smithi151.stdout: "last_pg_merge_meta": { 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "source_pgid": "0.0", 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "ready_epoch": 0, 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_started": 0, 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "last_epoch_clean": 0, 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "source_version": "0'0", 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "target_version": "0'0" 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "last_change": "926", 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend": "0", 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "last_force_op_resend_preluminous": "924", 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "auid": 0, 2024-09-02T10:37:21.533 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_mode": "selfmanaged", 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_seq": 0, 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "snap_epoch": 0, 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "pool_snaps": [], 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps": "[]", 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_bytes": 0, 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "quota_max_objects": 0, 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "tiers": [], 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "tier_of": -1, 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "read_tier": -1, 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "write_tier": -1, 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_mode": "none", 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_bytes": 0, 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "target_max_objects": 0, 2024-09-02T10:37:21.534 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_flush_age": 0, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "cache_min_evict_age": 0, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profile": "", 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_params": { 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "none" 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_period": 0, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_count": 0, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "use_gmt_hitset": true, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "min_read_recency_for_promote": 0, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "min_write_recency_for_promote": 0, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_grade_decay_rate": 0, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "hit_set_search_last_n": 0, 2024-09-02T10:37:21.535 INFO:teuthology.orchestra.run.smithi151.stdout: "grade_table": [], 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "stripe_width": 0, 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "expected_num_objects": 0, 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "fast_read": false, 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "options": {}, 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "application_metadata": { 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "rados": {} 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "read_balance": { 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "score_type": "Fair distribution", 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "score_acting": 4.820000171661377, 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "score_stable": 2.119999885559082, 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "optimal_score": 1, 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_acting": 4.820000171661377, 2024-09-02T10:37:21.536 INFO:teuthology.orchestra.run.smithi151.stdout: "raw_score_stable": 2.119999885559082, 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity_weighted": 0.88373178243637085, 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity": 0.95999997854232788, 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: "average_primary_affinity_weighted": 0.87999999523162842, 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: "error_message": "pool 9 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate" 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e76499ad-5f7e-4788-aa85-0bad0d843e29", 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.537 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.538 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.538 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:37:21.538 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.538 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.538 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 8, 2024-09-02T10:37:21.538 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 961, 2024-09-02T10:37:21.538 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.538 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.538 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.538 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6802", 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6803", 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.539 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6804", 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6805", 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.540 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6808", 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6809", 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.541 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6806", 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6807", 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1059446772 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.542 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6803/1059446772", 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6805/1059446772", 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6809/1059446772", 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6807/1059446772", 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:37:21.543 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "8949ba9c-4a71-4327-a1dc-e7d25286d8f1", 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0, 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 13, 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 976, 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.544 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6810", 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6811", 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.545 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6812", 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6813", 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.546 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6816", 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6817", 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.547 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6814", 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6815", 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 457777866 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6811/457777866", 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6813/457777866", 2024-09-02T10:37:21.548 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6817/457777866", 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6815/457777866", 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2dead774-0f59-4e90-8503-9a3f0fb68b48", 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 0.6047821044921875, 2024-09-02T10:37:21.549 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 18, 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 961, 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6818", 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.550 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6819", 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6820", 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.551 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6821", 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6824", 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:37:21.552 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6825", 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6822", 2024-09-02T10:37:21.553 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6823", 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 654044340 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6819/654044340", 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6821/654044340", 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6825/654044340", 2024-09-02T10:37:21.554 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6823/654044340", 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "524f32ed-95aa-4459-9585-72577db12107", 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:37:21.555 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 26, 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 968, 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6826", 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:37:21.556 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6827", 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6828", 2024-09-02T10:37:21.557 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:37:21.558 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.558 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.558 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.558 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6829", 2024-09-02T10:37:21.558 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:37:21.558 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6832", 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6833", 2024-09-02T10:37:21.559 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6830", 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.560 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.151:6831", 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 1912547696 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.151:6827/1912547696", 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.151:6829/1912547696", 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.151:6833/1912547696", 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.151:6831/1912547696", 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.561 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.562 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.562 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.562 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:37:21.562 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "2c43a79f-8ee6-4088-af19-6144fc8bdbe4", 2024-09-02T10:37:21.562 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.562 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.562 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.562 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:37:21.562 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.562 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 32, 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 958, 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6800", 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.563 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6801", 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6802", 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.564 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6803", 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6806", 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:37:21.565 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6807", 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6804", 2024-09-02T10:37:21.566 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6805", 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 295415389 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6801/295415389", 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6803/295415389", 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6807/295415389", 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6805/295415389", 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.567 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "71810aab-c54f-4048-9fe4-d1272302d53a", 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.568 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 39, 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 983, 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6808", 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6809", 2024-09-02T10:37:21.569 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:37:21.570 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.570 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.570 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.570 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.570 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.570 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.571 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6810", 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6811", 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.572 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.573 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.573 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6814", 2024-09-02T10:37:21.573 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:37:21.573 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.573 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.573 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.573 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6815", 2024-09-02T10:37:21.573 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:37:21.573 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.573 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6812", 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6813", 2024-09-02T10:37:21.574 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3782691981 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6809/3782691981", 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6811/3782691981", 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6815/3782691981", 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6813/3782691981", 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:37:21.575 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "45f93bd7-8e30-43ee-b57a-c007aec6d308", 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 45, 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 975, 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.576 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6816", 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6817", 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.577 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6818", 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6819", 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.578 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6822", 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6823", 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.579 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6820", 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6821", 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 759599342 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.580 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6817/759599342", 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6819/759599342", 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6823/759599342", 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6821/759599342", 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "e9ab0b99-0451-4ebd-8758-cbde6350c764", 2024-09-02T10:37:21.581 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 52, 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 988, 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.582 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6824", 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6825", 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.583 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6826", 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6827", 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.584 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6830", 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6831", 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.585 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6828", 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.162:6829", 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 2389923814 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.162:6825/2389923814", 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.162:6827/2389923814", 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.162:6831/2389923814", 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.162:6829/2389923814", 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.586 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "b0cf652b-4d1b-441a-85a8-fd5c16380634", 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.587 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 59, 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 985, 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6800", 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6801", 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:37:21.588 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6802", 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6803", 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:37:21.589 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6806", 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6807", 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.590 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6804", 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6805", 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3014427013 2024-09-02T10:37:21.591 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6801/3014427013", 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6803/3014427013", 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6807/3014427013", 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6805/3014427013", 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.592 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "1752a57c-aab9-4f0d-b2fd-629489170a0b", 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 66, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 961, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.593 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6808", 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6809", 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.594 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6810", 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6811", 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.595 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6814", 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6815", 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.596 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6812", 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6813", 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 108453727 2024-09-02T10:37:21.597 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.598 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.598 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.598 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6809/108453727", 2024-09-02T10:37:21.598 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6811/108453727", 2024-09-02T10:37:21.598 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6815/108453727", 2024-09-02T10:37:21.598 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6813/108453727", 2024-09-02T10:37:21.598 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "6f48d49d-6636-4142-8cf1-04f371f6317a", 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 1, 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 1, 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.599 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 73, 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 984, 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6816", 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.600 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6817", 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6818", 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:37:21.601 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6819", 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6822", 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:37:21.602 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6823", 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6820", 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:37:21.603 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6821", 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 401612597 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6817/401612597", 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6819/401612597", 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6823/401612597", 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6821/401612597", 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.604 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "uuid": "f545e120-27c3-4fc6-883f-7c588d0eb8d2", 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "up": 1, 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "in": 0, 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "weight": 0, 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_affinity": 1, 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_begin": 0, 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "last_clean_end": 0, 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "up_from": 80, 2024-09-02T10:37:21.605 INFO:teuthology.orchestra.run.smithi151.stdout: "up_thru": 792, 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: "down_at": 0, 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: "lost_at": 0, 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addrs": { 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6824", 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6825", 2024-09-02T10:37:21.606 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addrs": { 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6826", 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6827", 2024-09-02T10:37:21.607 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addrs": { 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6830", 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6831", 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:37:21.608 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addrs": { 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: "addrvec": [ 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v2", 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6828", 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: "type": "v1", 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: "addr": "172.21.15.187:6829", 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: "nonce": 3639916212 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.609 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "public_addr": "172.21.15.187:6825/3639916212", 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "cluster_addr": "172.21.15.187:6827/3639916212", 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_back_addr": "172.21.15.187:6831/3639916212", 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "heartbeat_front_addr": "172.21.15.187:6829/3639916212", 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "state": [ 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "exists", 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "up" 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "osd_xinfo": [ 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 0, 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.610 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:46.058353+0000", 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 1, 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.611 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:07.857026+0000", 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 2, 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:40.067579+0000", 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.612 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 3, 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:42.667539+0000", 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 4, 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.613 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:37:06.908211+0000", 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 5, 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:01.630550+0000", 2024-09-02T10:37:21.614 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 6, 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:55.058256+0000", 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.615 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 7, 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:36.877416+0000", 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 8, 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.616 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:31.202078+0000", 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 9, 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.617 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:53.201763+0000", 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 10, 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 0, 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:22.826971+0000", 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.618 INFO:teuthology.orchestra.run.smithi151.stdout: "osd": 11, 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: "down_stamp": "0.000000", 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_probability": 0, 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: "laggy_interval": 0, 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: "features": 4540719139924082687, 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: "old_weight": 13958, 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: "last_purged_snaps_scrub": "2024-09-02T10:36:49.672350+0000", 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: "dead_epoch": 0 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap": [ 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.e", 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:37:21.619 INFO:teuthology.orchestra.run.smithi151.stdout: 4, 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: 9, 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: 2 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.13", 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: 4, 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: 2 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.620 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "1.1e", 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: 11, 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: 10, 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: 9 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_items": [], 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_upmap_primaries": [], 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: "pg_temp": [ 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: { 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: "pgid": "9.14", 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: "osds": [ 2024-09-02T10:37:21.621 INFO:teuthology.orchestra.run.smithi151.stdout: 1, 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: 3 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: ] 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: ], 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: "primary_temp": [], 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: "blocklist": { 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1566053508": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/1701887065": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2339434463": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/2251539148": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/394946850": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/609062416": "2024-09-03T09:54:38.970754+0000", 2024-09-02T10:37:21.622 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6801/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:6800/3972807807": "2024-09-03T09:54:15.686961+0000", 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: "172.21.15.151:0/3380605706": "2024-09-03T09:54:15.686961+0000" 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: "range_blocklist": {}, 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: "erasure_code_profiles": { 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: "default": { 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: "crush-failure-domain": "osd", 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: "k": "2", 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: "m": "1", 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: "plugin": "jerasure", 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: "technique": "reed_sol_van" 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.623 INFO:teuthology.orchestra.run.smithi151.stdout: }, 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "removed_snaps_queue": [], 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "new_removed_snaps": [], 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "new_purged_snaps": [], 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "crush_node_flags": {}, 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "device_class_flags": {}, 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode": { 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_enabled": false, 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_bucket_count": 0, 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "degraded_stretch_mode": 0, 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "recovering_stretch_mode": 0, 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: "stretch_mode_bucket": 0 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout: } 2024-09-02T10:37:21.624 INFO:teuthology.orchestra.run.smithi151.stdout:} 2024-09-02T10:37:21.625 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:21.644 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-02T10:37:21.653 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:21.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:21.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:21.671 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:37:21.671 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:37:21.696 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:37:21.696 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:37:21.726 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:37:21.726 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:37:21.755 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:21.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:37:21.761 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:37:21.789 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:37:21.789 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:37:21.795 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:21.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:21.897 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:21.927 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:21.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:22.026 INFO:teuthology.orchestra.run.smithi162.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.b/config 2024-09-02T10:37:22.028 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:22.051 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:22.052 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:22.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:22.178 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:22.179 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:21 smithi151 ceph-mon[16639]: 9.15 scrub ok 2024-09-02T10:37:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:21 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:37:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:21 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:37:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:21 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:37:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:21 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:37:22.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3206160665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:37:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:21 smithi187 ceph-mon[20072]: 9.15 scrub ok 2024-09-02T10:37:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:21 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:37:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:21 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:37:22.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:21 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:37:22.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:21 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:37:22.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3206160665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:37:22.280 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:22.309 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:22.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:22.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:22.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:21 smithi162 ceph-mon[20279]: 9.15 scrub ok 2024-09-02T10:37:22.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:21 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:37:22.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:21 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:37:22.314 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:21 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:37:22.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:21 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:37:22.315 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3206160665' entity='client.admin' cmd={"prefix": "osd dump", "format": "json-pretty"} : dispatch 2024-09-02T10:37:22.411 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:22.438 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:22.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:22.540 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:22.568 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:22.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:22.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:22.620 INFO:teuthology.orchestra.run.smithi162.stdout:{ 2024-09-02T10:37:22.620 INFO:teuthology.orchestra.run.smithi162.stdout: "size": 20, 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 600, 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "ops": [ 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:146122 8.d 8:b6be1d7e:::benchmark_data_smithi187_136059_object40841:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:21.741003+0000", 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 360.87587741700003, 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.36294522699999998, 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:37:22.621 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 146122 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.741003+0000", 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.741003+0000", 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.622 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.741005+0000", 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1289999999999999e-06 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.741011+0000", 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6819999999999997e-06 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.741023+0000", 2024-09-02T10:37:22.623 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1250000000000001e-05 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.741030+0000", 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4889999999999998e-06 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.755240+0000", 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.014209375999999999 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.624 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for rw locks", 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.755316+0000", 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6915000000000005e-05 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.766880+0000", 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.011563571 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.766955+0000", 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.4609000000000003e-05 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.625 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,5", 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.767054+0000", 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9434999999999999e-05 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.783106+0000", 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.016052358999999999 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.783162+0000", 2024-09-02T10:37:22.626 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5182999999999997e-05 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.101149+0000", 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.317987723 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.101292+0000", 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014295900000000001 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.627 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.103948+0000", 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026555469999999999 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:146127 8.d 8:bbf1b620:::benchmark_data_smithi187_136059_object40846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:21.744642+0000", 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 360.87223783600001, 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.353809441, 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.628 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 146127 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.744642+0000", 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.629 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.744642+0000", 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.744644+0000", 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4729999999999999e-06 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.744654+0000", 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0057999999999999e-05 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.630 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.744668+0000", 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4747e-05 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.744675+0000", 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9040000000000003e-06 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.755346+0000", 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.010671073 2024-09-02T10:37:22.631 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.755440+0000", 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.3096000000000004e-05 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,5", 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.755535+0000", 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5933999999999993e-05 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.782956+0000", 2024-09-02T10:37:22.632 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.027420184 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.783022+0000", 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.6221000000000002e-05 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.095967+0000", 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31294509599999998 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.633 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.096010+0000", 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2904000000000001e-05 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.098452+0000", 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0024417509999999998 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.634 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:146128 8.d 8:b2d42e18:::benchmark_data_smithi187_136059_object40847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:37:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:21.746449+0000", 2024-09-02T10:37:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 360.87043089500003, 2024-09-02T10:37:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.35462012599999998, 2024-09-02T10:37:22.635 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 146128 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.746449+0000", 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.636 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.746449+0000", 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.746450+0000", 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.6400000000000001e-07 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.637 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.746454+0000", 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1389999999999997e-06 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.746466+0000", 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.1941e-05 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.746471+0000", 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3349999999999997e-06 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.638 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.758993+0000", 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.012521674999999999 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.759108+0000", 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00011491 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,5", 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.759223+0000", 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000114616 2024-09-02T10:37:22.639 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.782965+0000", 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.023742248000000001 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.783088+0000", 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012306200000000001 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.640 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.098509+0000", 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.315421222 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.098570+0000", 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0989000000000002e-05 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.101069+0000", 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002499125 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.641 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.27559.0:146131 8.6 e848/844 8:6708c0fe:::benchmark_data_smithi187_136059_object40850:head v 848'9192, mlcod=810'2570)", 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:31:21.751731+0000", 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 360.86514875699999, 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.33561356599999997, 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.642 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.751731+0000", 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.751731+0000", 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.751732+0000", 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.102e-06 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.643 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:21.751742+0000", 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9059999999999997e-06 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.083603+0000", 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.331860824 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.083609+0000", 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.784e-06 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.644 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.083659+0000", 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.0164999999999998e-05 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.083679+0000", 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0018999999999999e-05 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.087291+0000", 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0036119440000000002 2024-09-02T10:37:22.645 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:31:22.087345+0000", 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3822000000000001e-05 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:207390 8.d 8:b804cc8f:::benchmark_data_smithi187_136059_object102109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:33:11.352678+0000", 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 251.264202299, 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.300767165, 2024-09-02T10:37:22.646 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 207390 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.352678+0000", 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.647 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.352678+0000", 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.352679+0000", 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.2399999999999996e-07 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.352685+0000", 2024-09-02T10:37:22.648 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.241e-06 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.352699+0000", 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4161999999999999e-05 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.352706+0000", 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9990000000000002e-06 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.649 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.355264+0000", 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.002557616 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.355368+0000", 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010426599999999999 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,5", 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.355449+0000", 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0589999999999999e-05 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.650 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.359393+0000", 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0039440569999999999 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.361287+0000", 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0018942309999999999 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.653361+0000", 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29207421099999997 2024-09-02T10:37:22.651 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.653398+0000", 2024-09-02T10:37:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7141999999999998e-05 2024-09-02T10:37:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.653445+0000", 2024-09-02T10:37:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6726e-05 2024-09-02T10:37:22.652 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.27559.0:207390 8.d e881/878 ondisk, result = 0)", 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:33:11.359623+0000", 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 251.25725724, 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29382914700000001, 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.653 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.359623+0000", 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.359623+0000", 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.359623+0000", 2024-09-02T10:37:22.654 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8599999999999999e-07 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.359626+0000", 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.452e-06 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.652978+0000", 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29335190700000002 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.655 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.652985+0000", 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.2250000000000003e-06 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.653350+0000", 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00036504899999999999 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.653357+0000", 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0119999999999999e-06 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.656 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:11.653452+0000", 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.5116000000000002e-05 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.27559.0:210553 8.d 8:bd49576e:::benchmark_data_smithi187_136059_object105272:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:33:16.488671+0000", 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 246.12820866999999, 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.516418559, 2024-09-02T10:37:22.657 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.27559", 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 210553 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.488671+0000", 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.658 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.488671+0000", 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.488673+0000", 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.767e-06 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.488687+0000", 2024-09-02T10:37:22.659 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3974e-05 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.488703+0000", 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.6123000000000001e-05 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.488710+0000", 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0870000000000001e-06 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.660 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.488780+0000", 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.9782000000000004e-05 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.488902+0000", 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012219499999999999 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 2,5", 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.489025+0000", 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000122731 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.661 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.493125+0000", 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040999110000000004 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:16.493206+0000", 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1108999999999999e-05 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:17.005017+0000", 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.51181085699999995 2024-09-02T10:37:22.662 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:17.005057+0000", 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.9749000000000002e-05 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:33:17.005090+0000", 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.3274e-05 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.663 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:70470 9.8 e930/924 9:17bda43b:::benchmark_data_smithi187_154927_object70469:head v 930'4183, mlcod=930'4180)", 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:34:59.156852+0000", 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 143.46002840099999, 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.314151177, 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.156852+0000", 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.664 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.156852+0000", 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.156852+0000", 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.6600000000000002e-07 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.156885+0000", 2024-09-02T10:37:22.665 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2722000000000003e-05 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.469619+0000", 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31273467999999999 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.469633+0000", 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.3671e-05 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.666 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.469678+0000", 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5256000000000002e-05 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.469699+0000", 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.0602e-05 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.470958+0000", 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0012586050000000001 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.667 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:34:59.471003+0000", 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5175e-05 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "MOSDPGPush(9.11 931/927 [PushOp(9:8a260e2b:::benchmark_data_smithi187_154927_object49335:head, version: 920'3061, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:8a260e2b:::benchmark_data_smithi187_154927_object49335:head@920'3061, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:35:08.459100+0000", 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 134.15777992700001, 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.26356417999999998, 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.668 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:08.459100+0000", 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:08.459100+0000", 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.669 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:08.459101+0000", 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8400000000000004e-07 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:08.459132+0000", 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.1378999999999999e-05 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:08.721669+0000", 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.262537352 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.670 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:08.721682+0000", 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2235000000000001e-05 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:08.721762+0000", 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.0605000000000005e-05 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:08.721772+0000", 2024-09-02T10:37:22.671 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8980000000000008e-06 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:35:08.722664+0000", 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00089212700000000005 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:148314 9.15 9:a9e37b25:::benchmark_data_smithi187_154927_object50202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:19.023740+0000", 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 63.593139960000002, 2024-09-02T10:37:22.672 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25567458599999998, 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 148314 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.023740+0000", 2024-09-02T10:37:22.673 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.023740+0000", 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.023741+0000", 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.359e-06 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.674 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.023747+0000", 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5470000000000004e-06 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.023770+0000", 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3218999999999999e-05 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.023778+0000", 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.481e-06 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.675 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.024227+0000", 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00044899399999999998 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.024355+0000", 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000128245 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,8", 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.024495+0000", 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00014034799999999999 2024-09-02T10:37:22.676 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.027223+0000", 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027277769999999998 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.027258+0000", 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.4598000000000003e-05 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.677 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.279281+0000", 2024-09-02T10:37:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.25202328099999999 2024-09-02T10:37:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.679 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.279370+0000", 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.9030000000000006e-05 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.279415+0000", 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.4706999999999997e-05 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.680 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:148341 9.11 9:8d168ece:::benchmark_data_smithi187_154927_object50229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:19.046047+0000", 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 63.570833100000002, 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.300008573, 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 148341 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.681 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.046047+0000", 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.046047+0000", 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.046048+0000", 2024-09-02T10:37:22.682 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.02e-06 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.046052+0000", 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.2259999999999999e-06 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.046063+0000", 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0776e-05 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.683 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.046068+0000", 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2490000000000001e-06 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.046112+0000", 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3443999999999999e-05 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.046190+0000", 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8243000000000003e-05 2024-09-02T10:37:22.684 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.046258+0000", 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8462999999999998e-05 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.048425+0000", 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0021666049999999998 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.048592+0000", 2024-09-02T10:37:22.685 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00016734399999999999 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.345956+0000", 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.297363936 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.346010+0000", 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.3690999999999998e-05 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.686 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.346055+0000", 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5575999999999999e-05 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:148341 9.11 e960/933 ondisk, result = 0)", 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:19.048766+0000", 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 63.568113658999998, 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29729380999999999, 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:22.687 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.048766+0000", 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.048766+0000", 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.048767+0000", 2024-09-02T10:37:22.688 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.92e-07 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.048771+0000", 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1670000000000001e-06 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.345829+0000", 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29705773000000002 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.689 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.345835+0000", 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.0959999999999997e-06 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.345943+0000", 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000107667 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.345950+0000", 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.097e-06 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.690 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:19.346060+0000", 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000110361 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:151241 9.11 9:8dbc236f:::benchmark_data_smithi187_154927_object53129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e960)", 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:21.782769+0000", 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.834110637000002, 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28745173600000001, 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.691 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 151241 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.782769+0000", 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.782769+0000", 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.692 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.782771+0000", 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.598e-06 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.782870+0000", 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.9172999999999994e-05 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.693 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.782886+0000", 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.5996999999999999e-05 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.782893+0000", 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.3520000000000001e-06 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.782946+0000", 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.2879e-05 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.694 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.783052+0000", 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105238 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.783148+0000", 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.6212999999999995e-05 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.785791+0000", 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0026430899999999998 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.695 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.785913+0000", 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012204799999999999 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:22.070121+0000", 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28420795700000001 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:22.070176+0000", 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5010999999999999e-05 2024-09-02T10:37:22.696 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:22.070221+0000", 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.5179999999999998e-05 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:151241 9.11 e960/933 ondisk, result = 0)", 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:21.785600+0000", 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 60.831280272000001, 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28462640099999997, 2024-09-02T10:37:22.697 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.785600+0000", 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.785600+0000", 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.698 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.785600+0000", 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1299999999999999e-07 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:21.785604+0000", 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.602e-06 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:22.067791+0000", 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28218683 2024-09-02T10:37:22.699 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:22.067797+0000", 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.8799999999999996e-06 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:22.070107+0000", 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0023104879999999999 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:22.070115+0000", 2024-09-02T10:37:22.700 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.7589999999999997e-06 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:22.070226+0000", 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111129 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:159048 9.11 9:8f5b66f7:::benchmark_data_smithi187_154927_object60936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:35.029127+0000", 2024-09-02T10:37:22.701 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.587753435000003, 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28980116900000003, 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 159048 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029127+0000", 2024-09-02T10:37:22.702 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029127+0000", 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029128+0000", 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.652e-06 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.703 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029135+0000", 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.703e-06 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029149+0000", 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.4079e-05 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029156+0000", 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.0570000000000004e-06 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.704 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029220+0000", 2024-09-02T10:37:22.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.3909000000000001e-05 2024-09-02T10:37:22.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.705 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029336+0000", 2024-09-02T10:37:22.705 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000116257 2024-09-02T10:37:22.705 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.705 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.705 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.029448+0000", 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000111712 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.033535+0000", 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0040866619999999996 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.041217+0000", 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0076827969999999999 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.706 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.318800+0000", 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.27758260699999998 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.318843+0000", 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3226999999999998e-05 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.318928+0000", 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.4506999999999996e-05 2024-09-02T10:37:22.707 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:159264 9.11 9:8a51152d:::benchmark_data_smithi187_154927_object61152:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:35.315474+0000", 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 47.301405715999998, 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.31794165499999999, 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.708 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 159264 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315474+0000", 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315474+0000", 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.709 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315475+0000", 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.8500000000000001e-07 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315478+0000", 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.9050000000000001e-06 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.710 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315575+0000", 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.7122999999999998e-05 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315585+0000", 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.0169e-05 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315651+0000", 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5531000000000007e-05 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.711 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315770+0000", 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000119431 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.315873+0000", 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000102642 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.318445+0000", 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0025720750000000001 2024-09-02T10:37:22.712 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.327903+0000", 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0094584430000000004 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.633102+0000", 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30519865000000002 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.633361+0000", 2024-09-02T10:37:22.713 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00025882599999999998 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:35.633416+0000", 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.5174999999999999e-05 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:165620 9.11 9:8beb64d9:::benchmark_data_smithi187_154927_object67508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e971)", 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:44.566231+0000", 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.050648545000001, 2024-09-02T10:37:22.714 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29320566599999998, 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 165620 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.566231+0000", 2024-09-02T10:37:22.715 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.566231+0000", 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.566234+0000", 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1469999999999999e-06 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.716 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.566240+0000", 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.404e-06 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.566253+0000", 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.2711000000000001e-05 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.566261+0000", 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 8.1640000000000001e-06 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.717 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.566326+0000", 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.5356000000000002e-05 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.566451+0000", 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00012455699999999999 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 3,6", 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.566557+0000", 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.000105808 2024-09-02T10:37:22.718 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.569309+0000", 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0027527279999999999 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.577581+0000", 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0082718270000000007 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.859347+0000", 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28176623299999998 2024-09-02T10:37:22.719 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.859399+0000", 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.1270000000000002e-05 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.859437+0000", 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.8461000000000003e-05 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.720 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop_reply(client.28030.0:165620 9.11 e971/933 ondisk, result = 0)", 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:44.569443+0000", 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 38.047437467000002, 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29000025800000001, 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "started", 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.569443+0000", 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.721 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.569443+0000", 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.569443+0000", 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.13e-07 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.569448+0000", 2024-09-02T10:37:22.722 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.259e-06 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.859284+0000", 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.28983608599999999 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.859291+0000", 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.387e-06 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.723 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.859334+0000", 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3574e-05 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.859341+0000", 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.1289999999999998e-06 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:44.859443+0000", 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010131 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.724 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_repop(client.28030.0:175376 9.8 e982/924 9:1432c434:::benchmark_data_smithi187_154927_object77264:head v 982'7549, mlcod=982'7546)", 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:36:59.635248+0000", 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 22.981631720999999, 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.30025453299999999, 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.725 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:59.635248+0000", 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:59.635248+0000", 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:59.635249+0000", 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.4199999999999996e-07 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.726 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:59.635253+0000", 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.3869999999999997e-06 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:59.932453+0000", 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.29720027500000001 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:59.932460+0000", 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.4049999999999997e-06 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.727 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:59.932538+0000", 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.8269999999999994e-05 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:59.932559+0000", 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.1117000000000001e-05 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:59.935470+0000", 2024-09-02T10:37:22.728 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0029105810000000002 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:36:59.935503+0000", 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.2956000000000001e-05 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: "description": "osd_op(client.28030.0:186480 9.15 9:afd1101a:::benchmark_data_smithi187_154927_object88368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e989)", 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: "initiated_at": "2024-09-02T10:37:13.309963+0000", 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: "age": 9.3069173470000006, 2024-09-02T10:37:22.729 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.222713839, 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "continuous": false, 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "type_data": { 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client_info": { 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client": "client.28030", 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "tid": 186480 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "events": [ 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "initiated", 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.309963+0000", 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.730 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "header_read", 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.309963+0000", 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "throttled", 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.309963+0000", 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 5.37e-07 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "all_read", 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.309966+0000", 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 2.3929999999999998e-06 2024-09-02T10:37:22.731 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "dispatched", 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.309972+0000", 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 6.7139999999999996e-06 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "queued_for_pg", 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.309980+0000", 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 7.6290000000000001e-06 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "reached_pg", 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.310018+0000", 2024-09-02T10:37:22.732 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 3.7948000000000003e-05 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "started", 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.310127+0000", 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010893899999999999 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "waiting for subops from 4,8", 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.310225+0000", 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 9.8307000000000006e-05 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.733 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.313373+0000", 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.0031476569999999999 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "op_commit", 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.313392+0000", 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 1.9669000000000001e-05 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.532530+0000", 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.21913753699999999 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "commit_sent", 2024-09-02T10:37:22.734 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.532635+0000", 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 0.00010463099999999999 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: }, 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: { 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: "event": "done", 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: "time": "2024-09-02T10:37:13.532676+0000", 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: "duration": 4.1878000000000001e-05 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: } 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout: ] 2024-09-02T10:37:22.735 INFO:teuthology.orchestra.run.smithi162.stdout:} 2024-09-02T10:37:22.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:22.737 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:22.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:22.838 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:22.866 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:22.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:22.968 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:22.998 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:22.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:23.040 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:37:23.100 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:23.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:23.127 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:23.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:23.228 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:23.251 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:23.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:23.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:23.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:22 smithi187 ceph-mon[20072]: pgmap v2262: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 287 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 845 op/s; 248/13776 objects degraded (1.800%); 720 KiB/s, 110 objects/s recovering 2024-09-02T10:37:23.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:22 smithi187 ceph-mon[20072]: 9.11 scrub starts 2024-09-02T10:37:23.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:22 smithi187 ceph-mon[20072]: 9.11 scrub ok 2024-09-02T10:37:23.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2716785666' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:23.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:22 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:23.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:22 smithi162 ceph-mon[20279]: pgmap v2262: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 287 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 845 op/s; 248/13776 objects degraded (1.800%); 720 KiB/s, 110 objects/s recovering 2024-09-02T10:37:23.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:22 smithi162 ceph-mon[20279]: 9.11 scrub starts 2024-09-02T10:37:23.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:22 smithi162 ceph-mon[20279]: 9.11 scrub ok 2024-09-02T10:37:23.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2716785666' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:23.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:22 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:23.352 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:23.377 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:23.378 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:23.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:22 smithi151 ceph-mon[16639]: pgmap v2262: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 287 MiB data, 9.3 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 845 op/s; 248/13776 objects degraded (1.800%); 720 KiB/s, 110 objects/s recovering 2024-09-02T10:37:23.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:23 smithi151 ceph-mon[16639]: 9.11 scrub starts 2024-09-02T10:37:23.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:23 smithi151 ceph-mon[16639]: 9.11 scrub ok 2024-09-02T10:37:23.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2716785666' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:23.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:23 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:23.479 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:23.506 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:23.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:23.608 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:23.630 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:23.632 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:23.733 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:23.762 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:23.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:23.864 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:23.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:23.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:23.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:23.990 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:24.012 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:24.013 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:24.114 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:24.135 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:24.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:24.137 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:24.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:23 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:37:24.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:23 smithi151 ceph-mon[16639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:37:24.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:23 smithi151 ceph-mon[16639]: osdmap e992: 12 total, 12 up, 11 in 2024-09-02T10:37:24.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:23 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:37:24.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:23 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:37:24.237 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:24.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:23 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:37:24.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:23 smithi187 ceph-mon[20072]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:37:24.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:23 smithi187 ceph-mon[20072]: osdmap e992: 12 total, 12 up, 11 in 2024-09-02T10:37:24.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:23 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:37:24.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:23 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:37:24.259 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:24.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:24.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:24.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:23 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-02T10:37:24.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:23 smithi162 ceph-mon[20279]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:37:24.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:23 smithi162 ceph-mon[20279]: osdmap e992: 12 total, 12 up, 11 in 2024-09-02T10:37:24.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:23 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:37:24.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:23 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:37:24.362 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:24.383 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:24.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:24.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:24.485 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:24.506 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:24.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:24.507 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:24.608 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:24.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:24.633 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:24.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:24.661 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Removed 98109 objects 2024-09-02T10:37:24.661 INFO:tasks.radosbench.radosbench.2.smithi187.stdout:Clean up completed and total clean up time :113.632 2024-09-02T10:37:24.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:24.743 INFO:tasks.thrashosds.thrasher:j is {'epoch': 991, 'fsid': '1530c7be-6911-11ef-bcd4-c7b262605968', 'created': '2024-09-02T09:53:53.308187+0000', 'modified': '2024-09-02T10:37:19.405550+0000', 'last_up_change': '2024-09-02T10:00:25.816021+0000', 'last_in_change': '2024-09-02T10:36:50.796061+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 26, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 12, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-02T09:56:53.284814+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 31, 'pg_placement_num_target': 31, 'pg_num_target': 31, 'pg_num_pending': 31, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '264', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '241', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7699999809265137, 'score_stable': 1.7699999809265137, 'optimal_score': 1, 'raw_score_acting': 1.7699999809265137, 'raw_score_stable': 1.7699999809265137, 'primary_affinity_weighted': 0.8837317824363708, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 9, 'pool_name': 'unique_pool_7', 'create_time': '2024-09-02T10:33:50.981540+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '926', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '924', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.820000171661377, 'score_stable': 2.119999885559082, 'optimal_score': 1, 'raw_score_acting': 4.820000171661377, 'raw_score_stable': 2.119999885559082, 'primary_affinity_weighted': 0.8837317824363708, 'average_primary_affinity': 0.9599999785423279, 'average_primary_affinity_weighted': 0.8799999952316284, 'error_message': 'pool 9 has acting primaries on OSD(s) with primary affinity 0, read balance score is not accurate'}}], 'osds': [{'osd': 0, 'uuid': 'e76499ad-5f7e-4788-aa85-0bad0d843e29', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 961, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6802', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6803', 'nonce': 1059446772}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6804', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6805', 'nonce': 1059446772}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6808', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6809', 'nonce': 1059446772}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6806', 'nonce': 1059446772}, {'type': 'v1', 'addr': '172.21.15.151:6807', 'nonce': 1059446772}]}, 'public_addr': '172.21.15.151:6803/1059446772', 'cluster_addr': '172.21.15.151:6805/1059446772', 'heartbeat_back_addr': '172.21.15.151:6809/1059446772', 'heartbeat_front_addr': '172.21.15.151:6807/1059446772', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8949ba9c-4a71-4327-a1dc-e7d25286d8f1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 976, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6810', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6811', 'nonce': 457777866}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6812', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6813', 'nonce': 457777866}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6816', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6817', 'nonce': 457777866}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6814', 'nonce': 457777866}, {'type': 'v1', 'addr': '172.21.15.151:6815', 'nonce': 457777866}]}, 'public_addr': '172.21.15.151:6811/457777866', 'cluster_addr': '172.21.15.151:6813/457777866', 'heartbeat_back_addr': '172.21.15.151:6817/457777866', 'heartbeat_front_addr': '172.21.15.151:6815/457777866', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2dead774-0f59-4e90-8503-9a3f0fb68b48', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6047821044921875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 961, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6818', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6819', 'nonce': 654044340}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6820', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6821', 'nonce': 654044340}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6824', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6825', 'nonce': 654044340}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6822', 'nonce': 654044340}, {'type': 'v1', 'addr': '172.21.15.151:6823', 'nonce': 654044340}]}, 'public_addr': '172.21.15.151:6819/654044340', 'cluster_addr': '172.21.15.151:6821/654044340', 'heartbeat_back_addr': '172.21.15.151:6825/654044340', 'heartbeat_front_addr': '172.21.15.151:6823/654044340', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '524f32ed-95aa-4459-9585-72577db12107', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 968, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6826', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6827', 'nonce': 1912547696}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6828', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6829', 'nonce': 1912547696}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6832', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6833', 'nonce': 1912547696}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.151:6830', 'nonce': 1912547696}, {'type': 'v1', 'addr': '172.21.15.151:6831', 'nonce': 1912547696}]}, 'public_addr': '172.21.15.151:6827/1912547696', 'cluster_addr': '172.21.15.151:6829/1912547696', 'heartbeat_back_addr': '172.21.15.151:6833/1912547696', 'heartbeat_front_addr': '172.21.15.151:6831/1912547696', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2c43a79f-8ee6-4088-af19-6144fc8bdbe4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 958, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6800', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6801', 'nonce': 295415389}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6802', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6803', 'nonce': 295415389}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6806', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6807', 'nonce': 295415389}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6804', 'nonce': 295415389}, {'type': 'v1', 'addr': '172.21.15.162:6805', 'nonce': 295415389}]}, 'public_addr': '172.21.15.162:6801/295415389', 'cluster_addr': '172.21.15.162:6803/295415389', 'heartbeat_back_addr': '172.21.15.162:6807/295415389', 'heartbeat_front_addr': '172.21.15.162:6805/295415389', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '71810aab-c54f-4048-9fe4-d1272302d53a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 983, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6808', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6809', 'nonce': 3782691981}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6810', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6811', 'nonce': 3782691981}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6814', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6815', 'nonce': 3782691981}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6812', 'nonce': 3782691981}, {'type': 'v1', 'addr': '172.21.15.162:6813', 'nonce': 3782691981}]}, 'public_addr': '172.21.15.162:6809/3782691981', 'cluster_addr': '172.21.15.162:6811/3782691981', 'heartbeat_back_addr': '172.21.15.162:6815/3782691981', 'heartbeat_front_addr': '172.21.15.162:6813/3782691981', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '45f93bd7-8e30-43ee-b57a-c007aec6d308', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 975, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6816', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6817', 'nonce': 759599342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6818', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6819', 'nonce': 759599342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6822', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6823', 'nonce': 759599342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6820', 'nonce': 759599342}, {'type': 'v1', 'addr': '172.21.15.162:6821', 'nonce': 759599342}]}, 'public_addr': '172.21.15.162:6817/759599342', 'cluster_addr': '172.21.15.162:6819/759599342', 'heartbeat_back_addr': '172.21.15.162:6823/759599342', 'heartbeat_front_addr': '172.21.15.162:6821/759599342', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'e9ab0b99-0451-4ebd-8758-cbde6350c764', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 988, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6824', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6825', 'nonce': 2389923814}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6826', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6827', 'nonce': 2389923814}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6830', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6831', 'nonce': 2389923814}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.162:6828', 'nonce': 2389923814}, {'type': 'v1', 'addr': '172.21.15.162:6829', 'nonce': 2389923814}]}, 'public_addr': '172.21.15.162:6825/2389923814', 'cluster_addr': '172.21.15.162:6827/2389923814', 'heartbeat_back_addr': '172.21.15.162:6831/2389923814', 'heartbeat_front_addr': '172.21.15.162:6829/2389923814', 'state': ['exists', 'up']}, {'osd': 8, 'uuid': 'b0cf652b-4d1b-441a-85a8-fd5c16380634', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 985, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6800', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6801', 'nonce': 3014427013}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6802', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6803', 'nonce': 3014427013}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6806', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6807', 'nonce': 3014427013}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6804', 'nonce': 3014427013}, {'type': 'v1', 'addr': '172.21.15.187:6805', 'nonce': 3014427013}]}, 'public_addr': '172.21.15.187:6801/3014427013', 'cluster_addr': '172.21.15.187:6803/3014427013', 'heartbeat_back_addr': '172.21.15.187:6807/3014427013', 'heartbeat_front_addr': '172.21.15.187:6805/3014427013', 'state': ['exists', 'up']}, {'osd': 9, 'uuid': '1752a57c-aab9-4f0d-b2fd-629489170a0b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 66, 'up_thru': 961, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6808', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6809', 'nonce': 108453727}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6810', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6811', 'nonce': 108453727}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6814', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6815', 'nonce': 108453727}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6812', 'nonce': 108453727}, {'type': 'v1', 'addr': '172.21.15.187:6813', 'nonce': 108453727}]}, 'public_addr': '172.21.15.187:6809/108453727', 'cluster_addr': '172.21.15.187:6811/108453727', 'heartbeat_back_addr': '172.21.15.187:6815/108453727', 'heartbeat_front_addr': '172.21.15.187:6813/108453727', 'state': ['exists', 'up']}, {'osd': 10, 'uuid': '6f48d49d-6636-4142-8cf1-04f371f6317a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 73, 'up_thru': 984, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6816', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6817', 'nonce': 401612597}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6818', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6819', 'nonce': 401612597}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6822', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6823', 'nonce': 401612597}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6820', 'nonce': 401612597}, {'type': 'v1', 'addr': '172.21.15.187:6821', 'nonce': 401612597}]}, 'public_addr': '172.21.15.187:6817/401612597', 'cluster_addr': '172.21.15.187:6819/401612597', 'heartbeat_back_addr': '172.21.15.187:6823/401612597', 'heartbeat_front_addr': '172.21.15.187:6821/401612597', 'state': ['exists', 'up']}, {'osd': 11, 'uuid': 'f545e120-27c3-4fc6-883f-7c588d0eb8d2', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 80, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6824', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6825', 'nonce': 3639916212}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6826', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6827', 'nonce': 3639916212}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6830', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6831', 'nonce': 3639916212}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6828', 'nonce': 3639916212}, {'type': 'v1', 'addr': '172.21.15.187:6829', 'nonce': 3639916212}]}, 'public_addr': '172.21.15.187:6825/3639916212', 'cluster_addr': '172.21.15.187:6827/3639916212', 'heartbeat_back_addr': '172.21.15.187:6831/3639916212', 'heartbeat_front_addr': '172.21.15.187:6829/3639916212', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:36:46.058353+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:36:07.857026+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:36:40.067579+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:36:42.667539+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:37:06.908211+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:36:01.630550+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:36:55.058256+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:36:36.877416+0000', 'dead_epoch': 0}, {'osd': 8, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:36:31.202078+0000', 'dead_epoch': 0}, {'osd': 9, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:36:53.201763+0000', 'dead_epoch': 0}, {'osd': 10, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-02T10:36:22.826971+0000', 'dead_epoch': 0}, {'osd': 11, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540719139924082687, 'old_weight': 13958, 'last_purged_snaps_scrub': '2024-09-02T10:36:49.672350+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.e', 'osds': [4, 9, 2]}, {'pgid': '1.13', 'osds': [4, 1, 2]}, {'pgid': '1.1e', 'osds': [11, 10, 9]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '9.14', 'osds': [1, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.151:0/1566053508': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/1701887065': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2339434463': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6801/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:0/2251539148': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/394946850': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6800/609062416': '2024-09-03T09:54:38.970754+0000', '172.21.15.151:6801/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:6800/3972807807': '2024-09-03T09:54:15.686961+0000', '172.21.15.151:0/3380605706': '2024-09-03T09:54:15.686961+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-02T10:37:24.743 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:37:24.755 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:24.756 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:24.857 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:24.879 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:24.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:24.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:24.981 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:25.003 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:25.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:25.105 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:25.126 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:25.127 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:25.227 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:25.248 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:25.249 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:25.256 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:25 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:37:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:25 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:37:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:25 smithi162 ceph-mon[20279]: pgmap v2264: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 157 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 922 op/s; 248/7521 objects degraded (3.297%); 0 B/s, 73 objects/s recovering 2024-09-02T10:37:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:25 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:37:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:25 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:37:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:25 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 248/7521 objects degraded (3.297%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:25.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:25 smithi162 ceph-mon[20279]: osdmap e993: 12 total, 12 up, 11 in 2024-09-02T10:37:25.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:25.372 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:25.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:25.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:25 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:37:25.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:25 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:37:25.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:25 smithi151 ceph-mon[16639]: pgmap v2264: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 157 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 922 op/s; 248/7521 objects degraded (3.297%); 0 B/s, 73 objects/s recovering 2024-09-02T10:37:25.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:25 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:37:25.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:25 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:37:25.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:25 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 248/7521 objects degraded (3.297%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:25.379 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:25 smithi151 ceph-mon[16639]: osdmap e993: 12 total, 12 up, 11 in 2024-09-02T10:37:25.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:25.495 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:25.496 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:25 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:37:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:25 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:37:25.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:25 smithi187 ceph-mon[20072]: pgmap v2264: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 157 MiB data, 8.9 GiB used, 974 GiB / 983 GiB avail; 0 B/s wr, 922 op/s; 248/7521 objects degraded (3.297%); 0 B/s, 73 objects/s recovering 2024-09-02T10:37:25.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:25 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:37:25.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:25 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:37:25.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:25 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 248/7521 objects degraded (3.297%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:25.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:25 smithi187 ceph-mon[20072]: osdmap e993: 12 total, 12 up, 11 in 2024-09-02T10:37:25.597 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:25.618 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:25.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:25.720 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:25.742 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:25.743 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:25.817 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-02T10:37:25.844 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:25.872 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:25.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:25.955 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-09-02T10:37:25.970 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:37:25.971 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:37:25.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:25.996 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:25.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:26.097 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:26.119 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:26.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:26.197 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:26.221 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:26.242 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:26.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:26.244 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:26 smithi187 ceph-mon[20072]: 9.d scrub starts 2024-09-02T10:37:26.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:26 smithi187 ceph-mon[20072]: 9.d scrub ok 2024-09-02T10:37:26.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:26.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:26 smithi162 ceph-mon[20279]: 9.d scrub starts 2024-09-02T10:37:26.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:26 smithi162 ceph-mon[20279]: 9.d scrub ok 2024-09-02T10:37:26.366 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:26.367 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:26.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:26 smithi151 ceph-mon[16639]: 9.d scrub starts 2024-09-02T10:37:26.418 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:26 smithi151 ceph-mon[16639]: 9.d scrub ok 2024-09-02T10:37:26.468 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:26.489 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:26.490 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:26.591 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:26.613 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:26.614 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:26.715 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:26.737 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2266,"stamp":"2024-09-02T10:37:25.636967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":68092472,"num_objects":1033,"num_object_clones":0,"num_object_copies":3099,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1033,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139725,"num_write_kb":3151489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16434,"num_bytes_recovered":608188000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223706,"ondisk_log_size":223706,"up":171,"acting":170,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":11,"num_per_pool_osds":11,"num_per_pool_omap_osds":11,"kb":1031196672,"kb_used":9076960,"kb_used_data":312464,"kb_used_omap":17,"kb_used_meta":8764270,"kb_avail":1022119712,"statfs":{"total":1055945392128,"available":1046650585088,"internally_reserved":0,"allocated":319963136,"data_stored":316384418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17545,"internal_metadata":8974613367},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,2,1,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-338690048,"num_objects":-5168,"num_object_clones":0,"num_object_copies":-15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5168,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1009,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.229608"},"pg_stats":[{"pgid":"9.17","version":"992'8002","reported_seq":15851,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410462+0000","last_change":"2024-09-02T10:36:14.428237+0000","last_active":"2024-09-02T10:37:24.410462+0000","last_peered":"2024-09-02T10:37:24.410462+0000","last_clean":"2024-09-02T10:37:24.410462+0000","last_became_active":"2024-09-02T10:35:57.707720+0000","last_became_peered":"2024-09-02T10:35:57.707720+0000","last_unstale":"2024-09-02T10:37:24.410462+0000","last_undegraded":"2024-09-02T10:37:24.410462+0000","last_fullsized":"2024-09-02T10:37:24.410462+0000","mapping_epoch":952,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":953,"parent":"0.0","parent_split_bits":5,"last_scrub":"959'6285","last_scrub_stamp":"2024-09-02T10:36:14.428021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:14.428021+0000","objects_scrubbed":1721,"log_size":8002,"log_dups_size":0,"ondisk_log_size":8002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:38.286207+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1721,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,0],"acting":[8,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.16","version":"992'8055","reported_seq":16065,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-02T10:37:23.047578+0000","last_change":"2024-09-02T10:36:11.419544+0000","last_active":"2024-09-02T10:37:23.047578+0000","last_peered":"2024-09-02T10:37:23.047578+0000","last_clean":"2024-09-02T10:37:23.047578+0000","last_became_active":"2024-09-02T10:35:38.829158+0000","last_became_peered":"2024-09-02T10:35:38.829158+0000","last_unstale":"2024-09-02T10:37:23.047578+0000","last_undegraded":"2024-09-02T10:37:23.047578+0000","last_fullsized":"2024-09-02T10:37:23.047578+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'6289","last_scrub_stamp":"2024-09-02T10:36:11.419498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:11.419498+0000","objects_scrubbed":1805,"log_size":8055,"log_dups_size":0,"ondisk_log_size":8055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:29.611986+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,6],"acting":[9,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.1e","version":"22'92","reported_seq":209,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410286+0000","last_change":"2024-09-02T10:37:20.829739+0000","last_active":"2024-09-02T10:37:24.410286+0000","last_peered":"2024-09-02T10:37:24.410286+0000","last_clean":"2024-09-02T10:37:24.410286+0000","last_became_active":"2024-09-02T10:36:51.864030+0000","last_became_peered":"2024-09-02T10:36:51.864030+0000","last_unstale":"2024-09-02T10:37:24.410286+0000","last_undegraded":"2024-09-02T10:37:24.410286+0000","last_fullsized":"2024-09-02T10:37:24.410286+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:20.829656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:20.829656+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:25.344868+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,10],"acting":[8,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.15","version":"991'7836","reported_seq":13882,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:21.475987+0000","last_change":"2024-09-02T10:37:20.637644+0000","last_active":"2024-09-02T10:37:21.475987+0000","last_peered":"2024-09-02T10:37:21.475987+0000","last_clean":"2024-09-02T10:37:21.475987+0000","last_became_active":"2024-09-02T10:35:24.380270+0000","last_became_peered":"2024-09-02T10:35:24.380270+0000","last_unstale":"2024-09-02T10:37:21.475987+0000","last_undegraded":"2024-09-02T10:37:21.475987+0000","last_fullsized":"2024-09-02T10:37:21.475987+0000","mapping_epoch":941,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":942,"parent":"0.0","parent_split_bits":5,"last_scrub":"991'7810","last_scrub_stamp":"2024-09-02T10:37:20.637353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:37:20.637353+0000","objects_scrubbed":94,"log_size":7836,"log_dups_size":0,"ondisk_log_size":7836,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:30.428875+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3067,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,4],"acting":[7,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":558,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.481061+0000","last_change":"2024-09-02T10:36:09.745084+0000","last_active":"2024-09-02T10:37:19.481061+0000","last_peered":"2024-09-02T10:37:19.481061+0000","last_clean":"2024-09-02T10:37:19.481061+0000","last_became_active":"2024-09-02T10:33:38.659815+0000","last_became_peered":"2024-09-02T10:33:38.659815+0000","last_unstale":"2024-09-02T10:37:19.481061+0000","last_undegraded":"2024-09-02T10:37:19.481061+0000","last_fullsized":"2024-09-02T10:37:19.481061+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:09.745020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.745020+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:31.345354+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.14","version":"992'8088","reported_seq":20501,"reported_epoch":993,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-02T10:37:24.942799+0000","last_change":"2024-09-02T10:37:03.055733+0000","last_active":"2024-09-02T10:37:24.942799+0000","last_peered":"2024-09-02T10:37:24.942799+0000","last_clean":"2024-09-02T10:36:50.538501+0000","last_became_active":"2024-09-02T10:36:52.998041+0000","last_became_peered":"2024-09-02T10:36:52.998041+0000","last_unstale":"2024-09-02T10:37:24.942799+0000","last_undegraded":"2024-09-02T10:36:52.867136+0000","last_fullsized":"2024-09-02T10:36:52.850550+0000","mapping_epoch":976,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":969,"parent":"0.0","parent_split_bits":5,"last_scrub":"957'6265","last_scrub_stamp":"2024-09-02T10:36:09.284280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.284280+0000","objects_scrubbed":1834,"log_size":8088,"log_dups_size":0,"ondisk_log_size":8088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1834,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3095,"num_bytes_recovered":116588544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,3],"acting":[1,3],"avail_no_missing":["1","3","8"],"object_location_counts":[{"shards":"1,3","objects":-248}],"blocked_by":[],"up_primary":10,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1683,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410344+0000","last_change":"2024-09-02T10:36:21.119287+0000","last_active":"2024-09-02T10:37:24.410344+0000","last_peered":"2024-09-02T10:37:24.410344+0000","last_clean":"2024-09-02T10:37:24.410344+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:37:24.410344+0000","last_undegraded":"2024-09-02T10:37:24.410344+0000","last_fullsized":"2024-09-02T10:37:24.410344+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:21.119207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:21.119207+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:49.420928+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.13","version":"991'8281","reported_seq":14611,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:22.488490+0000","last_change":"2024-09-02T10:36:12.267636+0000","last_active":"2024-09-02T10:37:22.488490+0000","last_peered":"2024-09-02T10:37:22.488490+0000","last_clean":"2024-09-02T10:37:22.488490+0000","last_became_active":"2024-09-02T10:35:50.773879+0000","last_became_peered":"2024-09-02T10:35:50.773879+0000","last_unstale":"2024-09-02T10:37:22.488490+0000","last_undegraded":"2024-09-02T10:37:22.488490+0000","last_fullsized":"2024-09-02T10:37:22.488490+0000","mapping_epoch":949,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":950,"parent":"0.0","parent_split_bits":5,"last_scrub":"958'6522","last_scrub_stamp":"2024-09-02T10:36:12.267481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:12.267481+0000","objects_scrubbed":1798,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:26.103198+0000","scrub_duration":315,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2424832,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1761,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1659,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410226+0000","last_change":"2024-09-02T10:36:09.169541+0000","last_active":"2024-09-02T10:37:24.410226+0000","last_peered":"2024-09-02T10:37:24.410226+0000","last_clean":"2024-09-02T10:37:24.410226+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:37:24.410226+0000","last_undegraded":"2024-09-02T10:37:24.410226+0000","last_fullsized":"2024-09-02T10:37:24.410226+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:09.169469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.169469+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:36.117296+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.12","version":"991'7972","reported_seq":14046,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:22.864884+0000","last_change":"2024-09-02T10:35:24.913387+0000","last_active":"2024-09-02T10:37:22.864884+0000","last_peered":"2024-09-02T10:37:22.864884+0000","last_clean":"2024-09-02T10:37:22.864884+0000","last_became_active":"2024-09-02T10:35:22.670593+0000","last_became_peered":"2024-09-02T10:35:22.670593+0000","last_unstale":"2024-09-02T10:37:22.864884+0000","last_undegraded":"2024-09-02T10:37:22.864884+0000","last_fullsized":"2024-09-02T10:37:22.864884+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'4822","last_scrub_stamp":"2024-09-02T10:35:24.913280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:24.913280+0000","objects_scrubbed":2914,"log_size":7972,"log_dups_size":0,"ondisk_log_size":7972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1293,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1703936,"num_objects":26,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3170,"num_write_kb":9024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":986,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410670+0000","last_change":"2024-09-02T10:37:21.431724+0000","last_active":"2024-09-02T10:37:24.410670+0000","last_peered":"2024-09-02T10:37:24.410670+0000","last_clean":"2024-09-02T10:37:24.410670+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:37:24.410670+0000","last_undegraded":"2024-09-02T10:37:24.410670+0000","last_fullsized":"2024-09-02T10:37:24.410670+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:21.431651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:21.431651+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:35.327234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"991'7676","reported_seq":11538,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:21.495495+0000","last_change":"2024-09-02T10:35:15.638534+0000","last_active":"2024-09-02T10:37:21.495495+0000","last_peered":"2024-09-02T10:37:21.495495+0000","last_clean":"2024-09-02T10:37:21.495495+0000","last_became_active":"2024-09-02T10:35:14.386404+0000","last_became_peered":"2024-09-02T10:35:14.386404+0000","last_unstale":"2024-09-02T10:37:21.495495+0000","last_undegraded":"2024-09-02T10:37:21.495495+0000","last_fullsized":"2024-09-02T10:37:21.495495+0000","mapping_epoch":933,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":934,"parent":"0.0","parent_split_bits":5,"last_scrub":"935'4488","last_scrub_stamp":"2024-09-02T10:35:15.638360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:15.638360+0000","objects_scrubbed":2543,"log_size":7676,"log_dups_size":0,"ondisk_log_size":7676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4718592,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3191,"num_write_kb":23040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":42,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.413323+0000","last_change":"2024-09-02T10:36:51.854047+0000","last_active":"2024-09-02T10:37:19.413323+0000","last_peered":"2024-09-02T10:37:19.413323+0000","last_clean":"2024-09-02T10:37:19.413323+0000","last_became_active":"2024-09-02T10:36:51.853478+0000","last_became_peered":"2024-09-02T10:36:51.853478+0000","last_unstale":"2024-09-02T10:37:19.413323+0000","last_undegraded":"2024-09-02T10:37:19.413323+0000","last_fullsized":"2024-09-02T10:37:19.413323+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.939026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.939026+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:22.416375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.10","version":"992'8281","reported_seq":16600,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410581+0000","last_change":"2024-09-02T10:37:17.474434+0000","last_active":"2024-09-02T10:37:24.410581+0000","last_peered":"2024-09-02T10:37:24.410581+0000","last_clean":"2024-09-02T10:37:24.410581+0000","last_became_active":"2024-09-02T10:36:08.733746+0000","last_became_peered":"2024-09-02T10:36:08.733746+0000","last_unstale":"2024-09-02T10:37:24.410581+0000","last_undegraded":"2024-09-02T10:37:24.410581+0000","last_fullsized":"2024-09-02T10:37:24.410581+0000","mapping_epoch":956,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":957,"parent":"0.0","parent_split_bits":5,"last_scrub":"990'8094","last_scrub_stamp":"2024-09-02T10:37:17.474302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:37:17.474302+0000","objects_scrubbed":187,"log_size":8281,"log_dups_size":0,"ondisk_log_size":8281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:45.316833+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1881,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":2035,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.413840+0000","last_change":"2024-09-02T10:36:10.478222+0000","last_active":"2024-09-02T10:37:19.413840+0000","last_peered":"2024-09-02T10:37:19.413840+0000","last_clean":"2024-09-02T10:37:19.413840+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:37:19.413840+0000","last_undegraded":"2024-09-02T10:37:19.413840+0000","last_fullsized":"2024-09-02T10:37:19.413840+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:10.478148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.478148+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:30.586610+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1696,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410465+0000","last_change":"2024-09-02T10:36:51.855832+0000","last_active":"2024-09-02T10:37:24.410465+0000","last_peered":"2024-09-02T10:37:24.410465+0000","last_clean":"2024-09-02T10:37:24.410465+0000","last_became_active":"2024-09-02T10:36:51.855455+0000","last_became_peered":"2024-09-02T10:36:51.855455+0000","last_unstale":"2024-09-02T10:37:24.410465+0000","last_undegraded":"2024-09-02T10:37:24.410465+0000","last_fullsized":"2024-09-02T10:37:24.410465+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:19.291427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:19.291427+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:45.792830+0000","scrub_duration":160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":237,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.416722+0000","last_change":"2024-09-02T10:36:15.323804+0000","last_active":"2024-09-02T10:37:19.416722+0000","last_peered":"2024-09-02T10:37:19.416722+0000","last_clean":"2024-09-02T10:37:19.416722+0000","last_became_active":"2024-09-02T10:33:38.657147+0000","last_became_peered":"2024-09-02T10:33:38.657147+0000","last_unstale":"2024-09-02T10:37:19.416722+0000","last_undegraded":"2024-09-02T10:37:19.416722+0000","last_fullsized":"2024-09-02T10:37:19.416722+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.323738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.323738+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:28.492236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1645,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.423424+0000","last_change":"2024-09-02T10:36:51.589123+0000","last_active":"2024-09-02T10:37:19.423424+0000","last_peered":"2024-09-02T10:37:19.423424+0000","last_clean":"2024-09-02T10:37:19.423424+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:37:19.423424+0000","last_undegraded":"2024-09-02T10:37:19.423424+0000","last_fullsized":"2024-09-02T10:37:19.423424+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:51.589079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:51.589079+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:07.978045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":890,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.480952+0000","last_change":"2024-09-02T10:36:13.719520+0000","last_active":"2024-09-02T10:37:19.480952+0000","last_peered":"2024-09-02T10:37:19.480952+0000","last_clean":"2024-09-02T10:37:19.480952+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:37:19.480952+0000","last_undegraded":"2024-09-02T10:37:19.480952+0000","last_fullsized":"2024-09-02T10:37:19.480952+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:13.719461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.719461+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:25.562912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":88,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410635+0000","last_change":"2024-09-02T10:36:20.573861+0000","last_active":"2024-09-02T10:37:24.410635+0000","last_peered":"2024-09-02T10:37:24.410635+0000","last_clean":"2024-09-02T10:37:24.410635+0000","last_became_active":"2024-09-02T10:36:12.767657+0000","last_became_peered":"2024-09-02T10:36:12.767657+0000","last_unstale":"2024-09-02T10:37:24.410635+0000","last_undegraded":"2024-09-02T10:37:24.410635+0000","last_fullsized":"2024-09-02T10:37:24.410635+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:20.573798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:20.573798+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:48.205451+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":1926,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.414270+0000","last_change":"2024-09-02T10:34:08.628331+0000","last_active":"2024-09-02T10:37:19.414270+0000","last_peered":"2024-09-02T10:37:19.414270+0000","last_clean":"2024-09-02T10:37:19.414270+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:37:19.414270+0000","last_undegraded":"2024-09-02T10:37:19.414270+0000","last_fullsized":"2024-09-02T10:37:19.414270+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:34:08.628247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:34:08.628247+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"992'8232","reported_seq":15941,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-02T10:37:23.964424+0000","last_change":"2024-09-02T10:36:22.570216+0000","last_active":"2024-09-02T10:37:23.964424+0000","last_peered":"2024-09-02T10:37:23.964424+0000","last_clean":"2024-09-02T10:37:23.964424+0000","last_became_active":"2024-09-02T10:35:16.385649+0000","last_became_peered":"2024-09-02T10:35:16.385649+0000","last_unstale":"2024-09-02T10:37:23.964424+0000","last_undegraded":"2024-09-02T10:37:23.964424+0000","last_fullsized":"2024-09-02T10:37:23.964424+0000","mapping_epoch":935,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":936,"parent":"0.0","parent_split_bits":5,"last_scrub":"961'6769","last_scrub_stamp":"2024-09-02T10:36:22.570101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:22.570101+0000","objects_scrubbed":1466,"log_size":8232,"log_dups_size":0,"ondisk_log_size":8232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:29.453651+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3544,"num_write_kb":22848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":674,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.416513+0000","last_change":"2024-09-02T10:36:20.316217+0000","last_active":"2024-09-02T10:37:19.416513+0000","last_peered":"2024-09-02T10:37:19.416513+0000","last_clean":"2024-09-02T10:37:19.416513+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:37:19.416513+0000","last_undegraded":"2024-09-02T10:37:19.416513+0000","last_fullsized":"2024-09-02T10:37:19.416513+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:20.316139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:20.316139+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:21.453571+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"991'8031","reported_seq":16515,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:21.775374+0000","last_change":"2024-09-02T10:36:56.911246+0000","last_active":"2024-09-02T10:37:21.775374+0000","last_peered":"2024-09-02T10:37:21.775374+0000","last_clean":"2024-09-02T10:37:21.775374+0000","last_became_active":"2024-09-02T10:36:56.889585+0000","last_became_peered":"2024-09-02T10:36:56.889585+0000","last_unstale":"2024-09-02T10:37:21.775374+0000","last_undegraded":"2024-09-02T10:37:21.775374+0000","last_fullsized":"2024-09-02T10:37:21.775374+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":924,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":5,"last_scrub":"932'4480","last_scrub_stamp":"2024-09-02T10:35:09.830443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:35:09.830443+0000","objects_scrubbed":2522,"log_size":8031,"log_dups_size":0,"ondisk_log_size":8031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3932160,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3560,"num_write_kb":35136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2129,"num_bytes_recovered":89980928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,5],"acting":[10,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":1918,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.414002+0000","last_change":"2024-09-02T10:36:51.864364+0000","last_active":"2024-09-02T10:37:19.414002+0000","last_peered":"2024-09-02T10:37:19.414002+0000","last_clean":"2024-09-02T10:37:19.414002+0000","last_became_active":"2024-09-02T10:36:51.863957+0000","last_became_peered":"2024-09-02T10:36:51.863957+0000","last_unstale":"2024-09-02T10:37:19.414002+0000","last_undegraded":"2024-09-02T10:37:19.414002+0000","last_fullsized":"2024-09-02T10:37:19.414002+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.409382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.409382+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:32.575551+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"991'8031","reported_seq":8291,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:22.507893+0000","last_change":"2024-09-02T10:36:23.228909+0000","last_active":"2024-09-02T10:37:22.507893+0000","last_peered":"2024-09-02T10:37:22.507893+0000","last_clean":"2024-09-02T10:37:22.507893+0000","last_became_active":"2024-09-02T10:34:47.026177+0000","last_became_peered":"2024-09-02T10:34:47.026177+0000","last_unstale":"2024-09-02T10:37:22.507893+0000","last_undegraded":"2024-09-02T10:37:22.507893+0000","last_fullsized":"2024-09-02T10:37:22.507893+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"961'6703","last_scrub_stamp":"2024-09-02T10:36:23.228789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:23.228789+0000","objects_scrubbed":1368,"log_size":8031,"log_dups_size":0,"ondisk_log_size":8031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:35.431706+0000","scrub_duration":376,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4126,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,8],"acting":[6,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":1999,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.414175+0000","last_change":"2024-09-02T10:36:51.423663+0000","last_active":"2024-09-02T10:37:19.414175+0000","last_peered":"2024-09-02T10:37:19.414175+0000","last_clean":"2024-09-02T10:37:19.414175+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:37:19.414175+0000","last_undegraded":"2024-09-02T10:37:19.414175+0000","last_fullsized":"2024-09-02T10:37:19.414175+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:51.423616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:51.423616+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:52.566389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"992'7978","reported_seq":8242,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-02T10:37:23.945206+0000","last_change":"2024-09-02T10:36:17.751426+0000","last_active":"2024-09-02T10:37:23.945206+0000","last_peered":"2024-09-02T10:37:23.945206+0000","last_clean":"2024-09-02T10:37:23.945206+0000","last_became_active":"2024-09-02T10:34:47.038024+0000","last_became_peered":"2024-09-02T10:34:47.038024+0000","last_unstale":"2024-09-02T10:37:23.945206+0000","last_undegraded":"2024-09-02T10:37:23.945206+0000","last_fullsized":"2024-09-02T10:37:23.945206+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'6457","last_scrub_stamp":"2024-09-02T10:36:17.751343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:17.751343+0000","objects_scrubbed":1534,"log_size":7978,"log_dups_size":0,"ondisk_log_size":7978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:27.232651+0000","scrub_duration":530,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4217,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,6],"acting":[3,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":2117,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.414314+0000","last_change":"2024-09-02T10:36:15.555892+0000","last_active":"2024-09-02T10:37:19.414314+0000","last_peered":"2024-09-02T10:37:19.414314+0000","last_clean":"2024-09-02T10:37:19.414314+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:37:19.414314+0000","last_undegraded":"2024-09-02T10:37:19.414314+0000","last_fullsized":"2024-09-02T10:37:19.414314+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:15.555822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:15.555822+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:33.398682+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2","version":"992'8046","reported_seq":8309,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-02T10:37:23.858497+0000","last_change":"2024-09-02T10:36:13.627438+0000","last_active":"2024-09-02T10:37:23.858497+0000","last_peered":"2024-09-02T10:37:23.858497+0000","last_clean":"2024-09-02T10:37:23.858497+0000","last_became_active":"2024-09-02T10:34:47.038074+0000","last_became_peered":"2024-09-02T10:34:47.038074+0000","last_unstale":"2024-09-02T10:37:23.858497+0000","last_undegraded":"2024-09-02T10:37:23.858497+0000","last_fullsized":"2024-09-02T10:37:23.858497+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'6355","last_scrub_stamp":"2024-09-02T10:36:13.627233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.627233+0000","objects_scrubbed":1695,"log_size":8046,"log_dups_size":0,"ondisk_log_size":8046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4220,"num_write_kb":72576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,9],"acting":[3,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1777,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-02T10:37:23.037910+0000","last_change":"2024-09-02T10:36:10.022054+0000","last_active":"2024-09-02T10:37:23.037910+0000","last_peered":"2024-09-02T10:37:23.037910+0000","last_clean":"2024-09-02T10:37:23.037910+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:37:23.037910+0000","last_undegraded":"2024-09-02T10:37:23.037910+0000","last_fullsized":"2024-09-02T10:37:23.037910+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:10.021994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.021994+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:36.323969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"9.3","version":"992'8103","reported_seq":8365,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-02T10:37:23.107357+0000","last_change":"2024-09-02T10:36:16.376436+0000","last_active":"2024-09-02T10:37:23.107357+0000","last_peered":"2024-09-02T10:37:23.107357+0000","last_clean":"2024-09-02T10:37:23.107357+0000","last_became_active":"2024-09-02T10:34:47.042672+0000","last_became_peered":"2024-09-02T10:34:47.042672+0000","last_unstale":"2024-09-02T10:37:23.107357+0000","last_undegraded":"2024-09-02T10:37:23.107357+0000","last_fullsized":"2024-09-02T10:37:23.107357+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'6561","last_scrub_stamp":"2024-09-02T10:36:16.376239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:16.376239+0000","objects_scrubbed":1576,"log_size":8103,"log_dups_size":0,"ondisk_log_size":8103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:24.146392+0000","scrub_duration":358,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2031616,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4137,"num_write_kb":69504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,5],"acting":[9,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":752,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.419041+0000","last_change":"2024-09-02T10:36:13.308543+0000","last_active":"2024-09-02T10:37:19.419041+0000","last_peered":"2024-09-02T10:37:19.419041+0000","last_clean":"2024-09-02T10:37:19.419041+0000","last_became_active":"2024-09-02T10:31:02.819629+0000","last_became_peered":"2024-09-02T10:31:02.819629+0000","last_unstale":"2024-09-02T10:37:19.419041+0000","last_undegraded":"2024-09-02T10:37:19.419041+0000","last_fullsized":"2024-09-02T10:37:19.419041+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:13.308459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.308459+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:42.491914+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"992'12024","reported_seq":12271,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-02T10:37:23.944963+0000","last_change":"2024-09-02T10:36:13.014102+0000","last_active":"2024-09-02T10:37:23.944963+0000","last_peered":"2024-09-02T10:37:23.944963+0000","last_clean":"2024-09-02T10:37:23.944963+0000","last_became_active":"2024-09-02T10:33:52.760804+0000","last_became_peered":"2024-09-02T10:33:52.760804+0000","last_unstale":"2024-09-02T10:37:23.944963+0000","last_undegraded":"2024-09-02T10:37:23.944963+0000","last_fullsized":"2024-09-02T10:37:23.944963+0000","mapping_epoch":897,"log_start":"914'2000","ondisk_log_start":"914'2000","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'8682","last_scrub_stamp":"2024-09-02T10:36:13.014038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.014038+0000","objects_scrubbed":3370,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:31.690235+0000","scrub_duration":797,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12024,"num_write_kb":384768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,9],"acting":[3,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":1086,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410286+0000","last_change":"2024-09-02T10:36:51.845852+0000","last_active":"2024-09-02T10:37:24.410286+0000","last_peered":"2024-09-02T10:37:24.410286+0000","last_clean":"2024-09-02T10:37:24.410286+0000","last_became_active":"2024-09-02T10:34:17.881745+0000","last_became_peered":"2024-09-02T10:34:17.881745+0000","last_unstale":"2024-09-02T10:37:24.410286+0000","last_undegraded":"2024-09-02T10:37:24.410286+0000","last_fullsized":"2024-09-02T10:37:24.410286+0000","mapping_epoch":912,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":913,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:51.845782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:51.845782+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:05.174827+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,6],"acting":[8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"9.d","version":"991'11854","reported_seq":12102,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:21.523034+0000","last_change":"2024-09-02T10:36:12.744199+0000","last_active":"2024-09-02T10:37:21.523034+0000","last_peered":"2024-09-02T10:37:21.523034+0000","last_clean":"2024-09-02T10:37:21.523034+0000","last_became_active":"2024-09-02T10:33:52.926467+0000","last_became_peered":"2024-09-02T10:33:52.926467+0000","last_unstale":"2024-09-02T10:37:21.523034+0000","last_undegraded":"2024-09-02T10:37:21.523034+0000","last_fullsized":"2024-09-02T10:37:21.523034+0000","mapping_epoch":897,"log_start":"914'1800","ondisk_log_start":"914'1800","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"958'8646","last_scrub_stamp":"2024-09-02T10:36:12.744115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:12.744115+0000","objects_scrubbed":3403,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1221,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9895960,"num_objects":152,"num_object_clones":0,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11854,"num_write_kb":384129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":42,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.413397+0000","last_change":"2024-09-02T10:36:51.860373+0000","last_active":"2024-09-02T10:37:19.413397+0000","last_peered":"2024-09-02T10:37:19.413397+0000","last_clean":"2024-09-02T10:37:19.413397+0000","last_became_active":"2024-09-02T10:36:51.859972+0000","last_became_peered":"2024-09-02T10:36:51.859972+0000","last_unstale":"2024-09-02T10:37:19.413397+0000","last_undegraded":"2024-09-02T10:37:19.413397+0000","last_fullsized":"2024-09-02T10:37:19.413397+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:20.991722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:20.991722+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:22.066390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.e","version":"991'11984","reported_seq":16104,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:21.533600+0000","last_change":"2024-09-02T10:37:17.846772+0000","last_active":"2024-09-02T10:37:21.533600+0000","last_peered":"2024-09-02T10:37:21.533600+0000","last_clean":"2024-09-02T10:37:21.533600+0000","last_became_active":"2024-09-02T10:37:10.948920+0000","last_became_peered":"2024-09-02T10:37:10.948920+0000","last_unstale":"2024-09-02T10:37:21.533600+0000","last_undegraded":"2024-09-02T10:37:21.533600+0000","last_fullsized":"2024-09-02T10:37:21.533600+0000","mapping_epoch":988,"log_start":"914'1900","ondisk_log_start":"914'1900","created":897,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"990'11808","last_scrub_stamp":"2024-09-02T10:37:17.846641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:37:17.846641+0000","objects_scrubbed":345,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:44.247928+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11010048,"num_objects":168,"num_object_clones":0,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11984,"num_write_kb":388864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1944,"num_bytes_recovered":51904512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,10,1],"acting":[7,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":42,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.413544+0000","last_change":"2024-09-02T10:36:51.854752+0000","last_active":"2024-09-02T10:37:19.413544+0000","last_peered":"2024-09-02T10:37:19.413544+0000","last_clean":"2024-09-02T10:37:19.413544+0000","last_became_active":"2024-09-02T10:36:51.854167+0000","last_became_peered":"2024-09-02T10:36:51.854167+0000","last_unstale":"2024-09-02T10:37:19.413544+0000","last_undegraded":"2024-09-02T10:37:19.413544+0000","last_fullsized":"2024-09-02T10:37:19.413544+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:17.904033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:17.904033+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:21.221795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.f","version":"991'12247","reported_seq":12486,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:22.895800+0000","last_change":"2024-09-02T10:36:25.554143+0000","last_active":"2024-09-02T10:37:22.895800+0000","last_peered":"2024-09-02T10:37:22.895800+0000","last_clean":"2024-09-02T10:37:22.895800+0000","last_became_active":"2024-09-02T10:33:52.925153+0000","last_became_peered":"2024-09-02T10:33:52.925153+0000","last_unstale":"2024-09-02T10:37:22.895800+0000","last_undegraded":"2024-09-02T10:37:22.895800+0000","last_fullsized":"2024-09-02T10:37:22.895800+0000","mapping_epoch":897,"log_start":"914'2200","ondisk_log_start":"914'2200","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"963'9547","last_scrub_stamp":"2024-09-02T10:36:25.553889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:25.553889+0000","objects_scrubbed":2782,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:45.328325+0000","scrub_duration":1147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12247,"num_write_kb":393728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,9],"acting":[5,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1307,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.415411+0000","last_change":"2024-09-02T10:36:51.855098+0000","last_active":"2024-09-02T10:37:19.415411+0000","last_peered":"2024-09-02T10:37:19.415411+0000","last_clean":"2024-09-02T10:37:19.415411+0000","last_became_active":"2024-09-02T10:36:51.854769+0000","last_became_peered":"2024-09-02T10:36:51.854769+0000","last_unstale":"2024-09-02T10:37:19.415411+0000","last_undegraded":"2024-09-02T10:37:19.415411+0000","last_fullsized":"2024-09-02T10:37:19.415411+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:18.936502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:18.936502+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:23.889891+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"991'7851","reported_seq":16370,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:21.775611+0000","last_change":"2024-09-02T10:37:04.428889+0000","last_active":"2024-09-02T10:37:21.775611+0000","last_peered":"2024-09-02T10:37:21.775611+0000","last_clean":"2024-09-02T10:37:21.775611+0000","last_became_active":"2024-09-02T10:37:04.411494+0000","last_became_peered":"2024-09-02T10:37:04.411494+0000","last_unstale":"2024-09-02T10:37:21.775611+0000","last_undegraded":"2024-09-02T10:37:21.775611+0000","last_fullsized":"2024-09-02T10:37:21.775611+0000","mapping_epoch":984,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":985,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'3857","last_scrub_stamp":"2024-09-02T10:34:49.043221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.043221+0000","objects_scrubbed":1904,"log_size":7851,"log_dups_size":0,"ondisk_log_size":7851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":979,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4063232,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3994,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2413,"num_bytes_recovered":87883776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,3],"acting":[10,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":42,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.413380+0000","last_change":"2024-09-02T10:36:51.853818+0000","last_active":"2024-09-02T10:37:19.413380+0000","last_peered":"2024-09-02T10:37:19.413380+0000","last_clean":"2024-09-02T10:37:19.413380+0000","last_became_active":"2024-09-02T10:36:51.853340+0000","last_became_peered":"2024-09-02T10:36:51.853340+0000","last_unstale":"2024-09-02T10:37:19.413380+0000","last_undegraded":"2024-09-02T10:37:19.413380+0000","last_fullsized":"2024-09-02T10:37:19.413380+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:24.088035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:24.088035+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:34.852082+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"9.a","version":"991'12285","reported_seq":12533,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:22.880548+0000","last_change":"2024-09-02T10:36:10.561335+0000","last_active":"2024-09-02T10:37:22.880548+0000","last_peered":"2024-09-02T10:37:22.880548+0000","last_clean":"2024-09-02T10:37:22.880548+0000","last_became_active":"2024-09-02T10:33:52.925177+0000","last_became_peered":"2024-09-02T10:33:52.925177+0000","last_unstale":"2024-09-02T10:37:22.880548+0000","last_undegraded":"2024-09-02T10:37:22.880548+0000","last_fullsized":"2024-09-02T10:37:22.880548+0000","mapping_epoch":897,"log_start":"914'2200","ondisk_log_start":"914'2200","created":897,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"957'8761","last_scrub_stamp":"2024-09-02T10:36:10.561133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.561133+0000","objects_scrubbed":3627,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1082,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12285,"num_write_kb":395392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1767,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.418545+0000","last_change":"2024-09-02T10:37:18.684291+0000","last_active":"2024-09-02T10:37:19.418545+0000","last_peered":"2024-09-02T10:37:19.418545+0000","last_clean":"2024-09-02T10:37:19.418545+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:37:19.418545+0000","last_undegraded":"2024-09-02T10:37:19.418545+0000","last_fullsized":"2024-09-02T10:37:19.418545+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:18.683668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:18.683668+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:41.355271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"992'12208","reported_seq":17269,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410205+0000","last_change":"2024-09-02T10:37:05.457342+0000","last_active":"2024-09-02T10:37:24.410205+0000","last_peered":"2024-09-02T10:37:24.410205+0000","last_clean":"2024-09-02T10:37:24.410205+0000","last_became_active":"2024-09-02T10:37:05.411373+0000","last_became_peered":"2024-09-02T10:37:05.411373+0000","last_unstale":"2024-09-02T10:37:24.410205+0000","last_undegraded":"2024-09-02T10:37:24.410205+0000","last_fullsized":"2024-09-02T10:37:24.410205+0000","mapping_epoch":985,"log_start":"914'2200","ondisk_log_start":"914'2200","created":897,"last_epoch_clean":986,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'9386","last_scrub_stamp":"2024-09-02T10:36:21.798044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:21.798044+0000","objects_scrubbed":2871,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:36.003470+0000","scrub_duration":1634,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12208,"num_write_kb":390656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2398,"num_bytes_recovered":82771968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1059,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.415085+0000","last_change":"2024-09-02T10:36:13.900889+0000","last_active":"2024-09-02T10:37:19.415085+0000","last_peered":"2024-09-02T10:37:19.415085+0000","last_clean":"2024-09-02T10:37:19.415085+0000","last_became_active":"2024-09-02T10:33:38.664311+0000","last_became_peered":"2024-09-02T10:33:38.664311+0000","last_unstale":"2024-09-02T10:37:19.415085+0000","last_undegraded":"2024-09-02T10:37:19.415085+0000","last_fullsized":"2024-09-02T10:37:19.415085+0000","mapping_epoch":891,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":892,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:13.900825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:13.900825+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:40.109001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"992'8199","reported_seq":8466,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-02T10:37:23.104363+0000","last_change":"2024-09-02T10:36:18.281248+0000","last_active":"2024-09-02T10:37:23.104363+0000","last_peered":"2024-09-02T10:37:23.104363+0000","last_clean":"2024-09-02T10:37:23.104363+0000","last_became_active":"2024-09-02T10:34:47.044129+0000","last_became_peered":"2024-09-02T10:34:47.044129+0000","last_unstale":"2024-09-02T10:37:23.104363+0000","last_undegraded":"2024-09-02T10:37:23.104363+0000","last_fullsized":"2024-09-02T10:37:23.104363+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"960'6666","last_scrub_stamp":"2024-09-02T10:36:18.281061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:18.281061+0000","objects_scrubbed":1568,"log_size":8199,"log_dups_size":0,"ondisk_log_size":8199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:34.149765+0000","scrub_duration":241,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4273,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,7],"acting":[9,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1778,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.661688+0000","last_change":"2024-09-02T10:37:19.661688+0000","last_active":"2024-09-02T10:37:19.661688+0000","last_peered":"2024-09-02T10:37:19.661688+0000","last_clean":"2024-09-02T10:37:19.661688+0000","last_became_active":"2024-09-02T10:31:03.380903+0000","last_became_peered":"2024-09-02T10:31:03.380903+0000","last_unstale":"2024-09-02T10:37:19.661688+0000","last_undegraded":"2024-09-02T10:37:19.661688+0000","last_fullsized":"2024-09-02T10:37:19.661688+0000","mapping_epoch":843,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":844,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:19.661616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:19.661616+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:44.865190+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,9],"acting":[7,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.7","version":"991'7822","reported_seq":12641,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:21.781426+0000","last_change":"2024-09-02T10:36:56.903650+0000","last_active":"2024-09-02T10:37:21.781426+0000","last_peered":"2024-09-02T10:37:21.781426+0000","last_clean":"2024-09-02T10:37:21.781426+0000","last_became_active":"2024-09-02T10:36:56.889540+0000","last_became_peered":"2024-09-02T10:36:56.889540+0000","last_unstale":"2024-09-02T10:37:21.781426+0000","last_undegraded":"2024-09-02T10:37:21.781426+0000","last_fullsized":"2024-09-02T10:37:21.781426+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3753","last_scrub_stamp":"2024-09-02T10:34:49.259057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.259057+0000","objects_scrubbed":1910,"log_size":7822,"log_dups_size":0,"ondisk_log_size":7822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4390912,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4069,"num_write_kb":71232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2281,"num_bytes_recovered":90439680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,7,3],"acting":[10,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":2033,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.415506+0000","last_change":"2024-09-02T10:36:09.993500+0000","last_active":"2024-09-02T10:37:19.415506+0000","last_peered":"2024-09-02T10:37:19.415506+0000","last_clean":"2024-09-02T10:37:19.415506+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:37:19.415506+0000","last_undegraded":"2024-09-02T10:37:19.415506+0000","last_fullsized":"2024-09-02T10:37:19.415506+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:09.993425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:09.993425+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:37:29.323758+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"991'7934","reported_seq":12532,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:22.852550+0000","last_change":"2024-09-02T10:36:59.916449+0000","last_active":"2024-09-02T10:37:22.852550+0000","last_peered":"2024-09-02T10:37:22.852550+0000","last_clean":"2024-09-02T10:37:22.852550+0000","last_became_active":"2024-09-02T10:36:59.901856+0000","last_became_peered":"2024-09-02T10:36:59.901856+0000","last_unstale":"2024-09-02T10:37:22.852550+0000","last_undegraded":"2024-09-02T10:37:22.852550+0000","last_fullsized":"2024-09-02T10:37:22.852550+0000","mapping_epoch":982,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":983,"parent":"0.0","parent_split_bits":0,"last_scrub":"928'3847","last_scrub_stamp":"2024-09-02T10:34:50.216498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:50.216498+0000","objects_scrubbed":1961,"log_size":7934,"log_dups_size":0,"ondisk_log_size":7934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":552,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4094,"num_write_kb":69184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2170,"num_bytes_recovered":87687168,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,1],"acting":[5,10,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":994,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410803+0000","last_change":"2024-09-02T10:37:23.475624+0000","last_active":"2024-09-02T10:37:24.410803+0000","last_peered":"2024-09-02T10:37:24.410803+0000","last_clean":"2024-09-02T10:37:24.410803+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:37:24.410803+0000","last_undegraded":"2024-09-02T10:37:24.410803+0000","last_fullsized":"2024-09-02T10:37:24.410803+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:23.475557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:23.475557+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:43.424724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.5","version":"991'8044","reported_seq":8281,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:22.901871+0000","last_change":"2024-09-02T10:34:49.168775+0000","last_active":"2024-09-02T10:37:22.901871+0000","last_peered":"2024-09-02T10:37:22.901871+0000","last_clean":"2024-09-02T10:37:22.901871+0000","last_became_active":"2024-09-02T10:34:47.049681+0000","last_became_peered":"2024-09-02T10:34:47.049681+0000","last_unstale":"2024-09-02T10:37:22.901871+0000","last_undegraded":"2024-09-02T10:37:22.901871+0000","last_fullsized":"2024-09-02T10:37:22.901871+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"925'3787","last_scrub_stamp":"2024-09-02T10:34:49.168686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:34:49.168686+0000","objects_scrubbed":1913,"log_size":8044,"log_dups_size":0,"ondisk_log_size":8044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":539,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2359296,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":76160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1797,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-02T10:37:19.423511+0000","last_change":"2024-09-02T10:36:16.543124+0000","last_active":"2024-09-02T10:37:19.423511+0000","last_peered":"2024-09-02T10:37:19.423511+0000","last_clean":"2024-09-02T10:37:19.423511+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:37:19.423511+0000","last_undegraded":"2024-09-02T10:37:19.423511+0000","last_fullsized":"2024-09-02T10:37:19.423511+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:16.543059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:16.543059+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"992'8090","reported_seq":8349,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-02T10:37:24.410780+0000","last_change":"2024-09-02T10:36:52.528377+0000","last_active":"2024-09-02T10:37:24.410780+0000","last_peered":"2024-09-02T10:37:24.410780+0000","last_clean":"2024-09-02T10:37:24.410780+0000","last_became_active":"2024-09-02T10:34:47.029634+0000","last_became_peered":"2024-09-02T10:34:47.029634+0000","last_unstale":"2024-09-02T10:37:24.410780+0000","last_undegraded":"2024-09-02T10:37:24.410780+0000","last_fullsized":"2024-09-02T10:37:24.410780+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":897,"last_epoch_clean":925,"parent":"0.0","parent_split_bits":0,"last_scrub":"976'7251","last_scrub_stamp":"2024-09-02T10:36:52.528264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T10:33:51.735815+0000","last_clean_scrub_stamp":"2024-09-02T10:36:52.528264+0000","objects_scrubbed":839,"log_size":8090,"log_dups_size":0,"ondisk_log_size":8090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:38:10.591294+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4227,"num_write_kb":73472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,8],"acting":[4,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":1198,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-02T10:37:23.047149+0000","last_change":"2024-09-02T10:36:10.301898+0000","last_active":"2024-09-02T10:37:23.047149+0000","last_peered":"2024-09-02T10:37:23.047149+0000","last_clean":"2024-09-02T10:37:23.047149+0000","last_became_active":"2024-09-02T10:15:12.085585+0000","last_became_peered":"2024-09-02T10:15:12.085585+0000","last_unstale":"2024-09-02T10:37:23.047149+0000","last_undegraded":"2024-09-02T10:37:23.047149+0000","last_fullsized":"2024-09-02T10:37:23.047149+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:36:10.301844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:36:10.301844+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":26,"stat_sum":{"num_bytes":67502104,"num_objects":1031,"num_object_clones":0,"num_object_copies":3093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":139725,"num_write_kb":3151489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16430,"num_bytes_recovered":607256576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":349446144,"data_stored":349438000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":220854,"ondisk_log_size":220854,"up":78,"acting":77,"num_store_stats":12},{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":931424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":0,"seq":343597384127,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532613060,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":766364,"kb_used_data":45192,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":92978788,"statfs":{"total":95995035648,"available":95210278912,"internally_reserved":0,"allocated":46276608,"data_stored":45956162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":738458048},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467841993,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":967704,"kb_used_data":20232,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92777448,"statfs":{"total":95995035648,"available":95004106752,"internally_reserved":0,"allocated":20717568,"data_stored":20397122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":970193344},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070929,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":638264,"kb_used_data":51744,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":93106888,"statfs":{"total":95995035648,"available":95341453312,"internally_reserved":0,"allocated":52985856,"data_stored":52656570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299860,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":928804,"kb_used_data":89620,"kb_used_omap":1,"kb_used_meta":839166,"kb_avail":92816348,"statfs":{"total":95995035648,"available":95043940352,"internally_reserved":0,"allocated":91770880,"data_stored":91442810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":859306442},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528794,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":643352,"kb_used_data":26312,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93101800,"statfs":{"total":95995035648,"available":95336243200,"internally_reserved":0,"allocated":26943488,"data_stored":26623042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":631830983},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503725021,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1042404,"kb_used_data":50516,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":92702748,"statfs":{"total":95995035648,"available":94927613952,"internally_reserved":0,"allocated":51728384,"data_stored":51400314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953954,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595704,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93149448,"statfs":{"total":95995035648,"available":95385034752,"internally_reserved":0,"allocated":884736,"data_stored":555450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150184,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":968744,"kb_used_data":8788,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92776408,"statfs":{"total":95995035648,"available":95003041792,"internally_reserved":0,"allocated":8998912,"data_stored":8674046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":982972872},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411819,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636512,"kb_used_data":5584,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":93108640,"statfs":{"total":95995035648,"available":95343247360,"internally_reserved":0,"allocated":5718016,"data_stored":5394018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575345,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924732,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92820420,"statfs":{"total":95995035648,"available":95048110080,"internally_reserved":0,"allocated":888832,"data_stored":555474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738870,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":964376,"kb_used_data":12744,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92780776,"statfs":{"total":95995035648,"available":95007514624,"internally_reserved":0,"allocated":13049856,"data_stored":12729410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8978432,"data_stored":8978432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3276800,"data_stored":3276800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":35655680,"data_stored":35651608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":15990784,"data_stored":15990784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":66981888,"data_stored":66977816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":52101120,"data_stored":52101120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":14614528,"data_stored":14614528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":39714816,"data_stored":39714816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":112132096,"data_stored":112132096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:37:26.740 INFO:tasks.thrashosds.thrasher:Setting 1.c to [10, 2, 6, 7, 4, 1] 2024-09-02T10:37:26.740 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.c', '10', '2', '6', '7', '4', '1'] 2024-09-02T10:37:26.740 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pg-upmap-items 1.c 10 2 6 7 4 1 2024-09-02T10:37:26.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:26.741 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:26.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:26.832 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:26.843 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:26.865 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:26.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:26.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:26.953 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:37:26.953 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:37:26.953 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:37:26.953 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:37:26.967 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:26.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:26.991 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:26.992 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:27.092 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:27.116 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:27.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:27.196 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd set noscrub 2024-09-02T10:37:27.218 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:27.244 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:27.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:27 smithi187 ceph-mon[20072]: 9.6 scrub starts 2024-09-02T10:37:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:27 smithi187 ceph-mon[20072]: 9.6 scrub ok 2024-09-02T10:37:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:27 smithi187 ceph-mon[20072]: pgmap v2266: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 65 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 829 op/s; 248/3099 objects degraded (8.003%); 0 B/s, 161 objects/s recovering 2024-09-02T10:37:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:27 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:37:27.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:27 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:37:27.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:27 smithi187 ceph-mon[20072]: from='client.19176 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:37:27.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:27 smithi151 ceph-mon[16639]: 9.6 scrub starts 2024-09-02T10:37:27.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:27 smithi151 ceph-mon[16639]: 9.6 scrub ok 2024-09-02T10:37:27.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:27 smithi151 ceph-mon[16639]: pgmap v2266: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 65 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 829 op/s; 248/3099 objects degraded (8.003%); 0 B/s, 161 objects/s recovering 2024-09-02T10:37:27.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:27 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:37:27.310 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:27 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:37:27.311 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:27 smithi151 ceph-mon[16639]: from='client.19176 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:37:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:27 smithi162 ceph-mon[20279]: 9.6 scrub starts 2024-09-02T10:37:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:27 smithi162 ceph-mon[20279]: 9.6 scrub ok 2024-09-02T10:37:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:27 smithi162 ceph-mon[20279]: pgmap v2266: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 65 MiB data, 8.7 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 829 op/s; 248/3099 objects degraded (8.003%); 0 B/s, 161 objects/s recovering 2024-09-02T10:37:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:27 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:37:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:27 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:37:27.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:27 smithi162 ceph-mon[20279]: from='client.19176 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:37:27.346 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:27.346 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:27.350 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:37:27.350 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:37:27.371 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:27.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:27.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:27.385 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_7 2024-09-02T10:37:27.385 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd pool rm unique_pool_7 unique_pool_7 --yes-i-really-really-mean-it 2024-09-02T10:37:27.473 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:27.495 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:27.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:27.573 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:37:27.573 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:37:27.593 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:37:27.594 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:37:27.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-09-02T10:37:27.597 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:27.613 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:37:27.613 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:37:27.618 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:27.619 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:27.638 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:37:27.639 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:37:27.672 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:37:27.673 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:37:27.694 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:37:27.694 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:37:27.716 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:37:27.717 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:37:27.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:27.737 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:37:27.737 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:37:27.742 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:27.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:27.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:27.760 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:37:27.760 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:37:27.782 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:37:27.783 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:37:27.806 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:37:27.806 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:37:27.845 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:27.855 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:27.872 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:27.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:27.974 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:28.000 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:28.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:28.102 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:28.124 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:28.125 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:28.182 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:28.187 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:28 smithi151 ceph-mon[16639]: 9.3 scrub starts 2024-09-02T10:37:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:28 smithi151 ceph-mon[16639]: 9.3 scrub ok 2024-09-02T10:37:28.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/335919204' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:28.225 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:28.247 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:28.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:28.248 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:28.248 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:28 smithi187 ceph-mon[20072]: 9.3 scrub starts 2024-09-02T10:37:28.249 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:28 smithi187 ceph-mon[20072]: 9.3 scrub ok 2024-09-02T10:37:28.249 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/335919204' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:28 smithi162 ceph-mon[20279]: 9.3 scrub starts 2024-09-02T10:37:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:28 smithi162 ceph-mon[20279]: 9.3 scrub ok 2024-09-02T10:37:28.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/335919204' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:28.348 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:28.349 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:37:28.349 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:37:28.370 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:28.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:28.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:28.472 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:28.499 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:28.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:28.601 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:28.622 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:28.623 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:28.724 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:28.748 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:28.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:28.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:28.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:28.873 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:28.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:28.964 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-09-02T10:37:28.974 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:28.997 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:28.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:28.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:29.072 INFO:teuthology.orchestra.run.smithi151.stderr:set 1.c pg_upmap_items mapping to [10->2,6->7,4->1] 2024-09-02T10:37:29.100 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:29.122 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:29.122 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:29.211 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:29.223 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:29.249 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:29.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:29 smithi187 ceph-mon[20072]: 9.5 scrub starts 2024-09-02T10:37:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:29 smithi187 ceph-mon[20072]: 9.5 scrub ok 2024-09-02T10:37:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:29 smithi187 ceph-mon[20072]: pgmap v2267: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 20 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 736 op/s; 248/921 objects degraded (26.927%); 0 B/s, 126 objects/s recovering 2024-09-02T10:37:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2984632649' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [10, 2, 6, 7, 4, 1]} : dispatch 2024-09-02T10:37:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:29 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:37:29.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:29 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:37:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:29 smithi162 ceph-mon[20279]: 9.5 scrub starts 2024-09-02T10:37:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:29 smithi162 ceph-mon[20279]: 9.5 scrub ok 2024-09-02T10:37:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:29 smithi162 ceph-mon[20279]: pgmap v2267: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 20 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 736 op/s; 248/921 objects degraded (26.927%); 0 B/s, 126 objects/s recovering 2024-09-02T10:37:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2984632649' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [10, 2, 6, 7, 4, 1]} : dispatch 2024-09-02T10:37:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:29 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:37:29.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:29 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:37:29.351 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:29.372 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:29.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:29.373 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:29 smithi151 ceph-mon[16639]: 9.5 scrub starts 2024-09-02T10:37:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:29 smithi151 ceph-mon[16639]: 9.5 scrub ok 2024-09-02T10:37:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:29 smithi151 ceph-mon[16639]: pgmap v2267: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 56 active+clean; 20 MiB data, 8.5 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 736 op/s; 248/921 objects degraded (26.927%); 0 B/s, 126 objects/s recovering 2024-09-02T10:37:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2984632649' entity='client.admin' cmd={"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [10, 2, 6, 7, 4, 1]} : dispatch 2024-09-02T10:37:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:29 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:37:29.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:29 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:37:29.474 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:29.496 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:29.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:29.497 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:29.599 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:29.625 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:29.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:29.626 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:29.727 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:29.749 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:37:29.749 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:37:29.754 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:29.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:29.856 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:29.882 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:29.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:29.985 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:30.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:30.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:30.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:30.133 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:30.134 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:30.235 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:30.256 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:30.257 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:30 smithi162 ceph-mon[20279]: 9.12 scrub starts 2024-09-02T10:37:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:30 smithi162 ceph-mon[20279]: 9.12 scrub ok 2024-09-02T10:37:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2984632649' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [10, 2, 6, 7, 4, 1]}]': finished 2024-09-02T10:37:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:30 smithi162 ceph-mon[20279]: osdmap e994: 12 total, 12 up, 11 in 2024-09-02T10:37:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:30 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 248/921 objects degraded (26.927%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:30.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:30 smithi162 ceph-mon[20279]: osdmap e995: 12 total, 12 up, 11 in 2024-09-02T10:37:30.358 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:30.380 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:30.380 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:30.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:30 smithi151 ceph-mon[16639]: 9.12 scrub starts 2024-09-02T10:37:30.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:30 smithi151 ceph-mon[16639]: 9.12 scrub ok 2024-09-02T10:37:30.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2984632649' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [10, 2, 6, 7, 4, 1]}]': finished 2024-09-02T10:37:30.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:30 smithi151 ceph-mon[16639]: osdmap e994: 12 total, 12 up, 11 in 2024-09-02T10:37:30.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:30 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 248/921 objects degraded (26.927%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:30.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:30 smithi151 ceph-mon[16639]: osdmap e995: 12 total, 12 up, 11 in 2024-09-02T10:37:30.471 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:30 smithi187 ceph-mon[20072]: 9.12 scrub starts 2024-09-02T10:37:30.471 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:30 smithi187 ceph-mon[20072]: 9.12 scrub ok 2024-09-02T10:37:30.471 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2984632649' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.c", "id": [10, 2, 6, 7, 4, 1]}]': finished 2024-09-02T10:37:30.472 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:30 smithi187 ceph-mon[20072]: osdmap e994: 12 total, 12 up, 11 in 2024-09-02T10:37:30.472 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:30 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 248/921 objects degraded (26.927%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:30.472 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:30 smithi187 ceph-mon[20072]: osdmap e995: 12 total, 12 up, 11 in 2024-09-02T10:37:30.481 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:30.503 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-09-02T10:37:30.507 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:30.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:30.609 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:30.632 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:30.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:30.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:30.745 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:30.758 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:30.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:30.859 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:30.881 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:30.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:30.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:30.983 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:31.010 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:31.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:31.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:31.112 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:31.134 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:31 smithi187 ceph-mon[20072]: pgmap v2270: 57 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 55 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 417 op/s; 248/6 objects degraded (4133.333%); 0 B/s, 86 objects/s recovering 2024-09-02T10:37:31.134 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:31 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:37:31.134 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:31 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:37:31.134 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:31 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:37:31.134 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:31 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:37:31.134 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:31 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:37:31.134 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:31 smithi187 ceph-mon[20072]: osdmap e996: 12 total, 12 up, 11 in 2024-09-02T10:37:31.134 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:31 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:37:31.134 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:31 smithi187 ceph-mon[20072]: 9.13 scrub starts 2024-09-02T10:37:31.135 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:31 smithi187 ceph-mon[20072]: 9.13 scrub ok 2024-09-02T10:37:31.137 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:31.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:31.239 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:31.253 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:37:31.254 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:37:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:31.263 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:31.263 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:31 smithi162 ceph-mon[20279]: pgmap v2270: 57 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 55 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 417 op/s; 248/6 objects degraded (4133.333%); 0 B/s, 86 objects/s recovering 2024-09-02T10:37:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:31 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:37:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:31 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:37:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:31 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:37:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:31 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:37:31.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:31 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:37:31.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:31 smithi162 ceph-mon[20279]: osdmap e996: 12 total, 12 up, 11 in 2024-09-02T10:37:31.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:31 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:37:31.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:31 smithi162 ceph-mon[20279]: 9.13 scrub starts 2024-09-02T10:37:31.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:31 smithi162 ceph-mon[20279]: 9.13 scrub ok 2024-09-02T10:37:31.364 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:31.389 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:31.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:31.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:31 smithi151 ceph-mon[16639]: pgmap v2270: 57 pgs: 1 peering, 1 active+recovering+undersized+degraded+remapped, 55 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 417 op/s; 248/6 objects degraded (4133.333%); 0 B/s, 86 objects/s recovering 2024-09-02T10:37:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:31 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:37:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:31 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:37:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:31 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:37:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:31 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:37:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:31 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:37:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:31 smithi151 ceph-mon[16639]: osdmap e996: 12 total, 12 up, 11 in 2024-09-02T10:37:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:31 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:37:31.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:31 smithi151 ceph-mon[16639]: 9.13 scrub starts 2024-09-02T10:37:31.439 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:31 smithi151 ceph-mon[16639]: 9.13 scrub ok 2024-09-02T10:37:31.490 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3 2024-09-02T10:37:31.515 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:31.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:31.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.3' 2024-09-02T10:37:31.617 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:31.643 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:31.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:31.746 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:31.767 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:31.768 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:31.869 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10 2024-09-02T10:37:31.890 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:31.891 INFO:tasks.cephadm.osd.10:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.10' 2024-09-02T10:37:31.992 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5 2024-09-02T10:37:31.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-09-02T10:37:32.018 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:32.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.5' 2024-09-02T10:37:32.120 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:32.142 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:32.143 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:32.244 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:32 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:37:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:32 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:37:32.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:32 smithi187 ceph-mon[20072]: osdmap e997: 12 total, 12 up, 11 in 2024-09-02T10:37:32.257 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:32.267 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:32.268 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:32 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:37:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:32 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:37:32.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:32 smithi162 ceph-mon[20279]: osdmap e997: 12 total, 12 up, 11 in 2024-09-02T10:37:32.369 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:32.392 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:32.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:32.417 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:32.437 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:32 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:37:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:32 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:37:32.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:32 smithi151 ceph-mon[16639]: osdmap e997: 12 total, 12 up, 11 in 2024-09-02T10:37:32.493 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1 2024-09-02T10:37:32.514 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:32.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.1' 2024-09-02T10:37:32.616 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:32.641 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:32.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:32.728 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:37:32.728 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:37:32.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0 2024-09-02T10:37:32.763 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:32.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:32.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.0' 2024-09-02T10:37:32.866 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:32.888 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:32.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:32.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:32.990 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:33.016 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:33.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:33.119 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:33.144 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:33.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:33.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:33 smithi151 ceph-mon[16639]: pgmap v2273: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 248/6 objects degraded (4133.333%); 0 B/s, 141 objects/s recovering 2024-09-02T10:37:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:33 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:37:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:33 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:37:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:33 smithi151 ceph-mon[16639]: 9.16 scrub starts 2024-09-02T10:37:33.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:33 smithi151 ceph-mon[16639]: 9.16 scrub ok 2024-09-02T10:37:33.245 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:33.270 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:33.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:33 smithi162 ceph-mon[20279]: pgmap v2273: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 248/6 objects degraded (4133.333%); 0 B/s, 141 objects/s recovering 2024-09-02T10:37:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:33 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:37:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:33 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:37:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:33 smithi162 ceph-mon[20279]: 9.16 scrub starts 2024-09-02T10:37:33.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:33 smithi162 ceph-mon[20279]: 9.16 scrub ok 2024-09-02T10:37:33.373 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:33.395 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:33.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:33.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:33.414 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:33 smithi187 ceph-mon[20072]: pgmap v2273: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 0 B/s wr, 76 op/s; 248/6 objects degraded (4133.333%); 0 B/s, 141 objects/s recovering 2024-09-02T10:37:33.414 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:33 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:37:33.414 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:33 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:37:33.414 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:33 smithi187 ceph-mon[20072]: 9.16 scrub starts 2024-09-02T10:37:33.414 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:33 smithi187 ceph-mon[20072]: 9.16 scrub ok 2024-09-02T10:37:33.442 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-09-02T10:37:33.497 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:33.503 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-02T10:37:33.518 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:33.519 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:33.620 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:33.646 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:33.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:33.689 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:33.749 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11 2024-09-02T10:37:33.770 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:33.771 INFO:tasks.cephadm.osd.11:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.11' 2024-09-02T10:37:33.872 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:33.898 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:33.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:34.000 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7 2024-09-02T10:37:34.007 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:34.026 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:34.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.7' 2024-09-02T10:37:34.086 INFO:teuthology.orchestra.run.smithi151.stderr:pool 'unique_pool_7' removed 2024-09-02T10:37:34.127 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:34.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:34 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:37:34.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:34 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:37:34.129 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/798852677' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:37:34.148 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:34.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:34.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:34.226 INFO:teuthology.orchestra.run.smithi187.stdout:ERROR: (22) Invalid argument 2024-09-02T10:37:34.226 INFO:teuthology.orchestra.run.smithi187.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-02T10:37:34.250 DEBUG:teuthology.orchestra.run.smithi151:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2 2024-09-02T10:37:34.271 INFO:teuthology.orchestra.run.smithi151.stderr:sudo: docker: command not found 2024-09-02T10:37:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:34.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi151 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.2' 2024-09-02T10:37:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:34 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:37:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:34 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:37:34.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/798852677' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:37:34.373 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:34.405 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:34.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:34 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:37:34.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:34 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:37:34.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/798852677' entity='client.admin' cmd={"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true} : dispatch 2024-09-02T10:37:34.507 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:34.528 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:34.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:34.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:34.630 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:34.651 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:34.652 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:34.753 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9 2024-09-02T10:37:34.766 INFO:teuthology.orchestra.run.smithi151.stdout:osd.0: {} 2024-09-02T10:37:34.766 INFO:teuthology.orchestra.run.smithi151.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-02T10:37:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:34.776 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:34.776 INFO:tasks.cephadm.osd.9:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.9' 2024-09-02T10:37:34.787 INFO:teuthology.orchestra.run.smithi151.stdout:osd.1: {} 2024-09-02T10:37:34.787 INFO:teuthology.orchestra.run.smithi151.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-02T10:37:34.807 INFO:teuthology.orchestra.run.smithi151.stdout:osd.2: {} 2024-09-02T10:37:34.807 INFO:teuthology.orchestra.run.smithi151.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-02T10:37:34.827 INFO:teuthology.orchestra.run.smithi151.stdout:osd.3: {} 2024-09-02T10:37:34.827 INFO:teuthology.orchestra.run.smithi151.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-02T10:37:34.848 INFO:teuthology.orchestra.run.smithi151.stdout:osd.4: {} 2024-09-02T10:37:34.849 INFO:teuthology.orchestra.run.smithi151.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-02T10:37:34.872 INFO:teuthology.orchestra.run.smithi151.stdout:osd.5: {} 2024-09-02T10:37:34.872 INFO:teuthology.orchestra.run.smithi151.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-02T10:37:34.877 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:34.893 INFO:teuthology.orchestra.run.smithi151.stdout:osd.6: {} 2024-09-02T10:37:34.893 INFO:teuthology.orchestra.run.smithi151.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-02T10:37:34.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-09-02T10:37:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:34.901 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:34.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:34.913 INFO:teuthology.orchestra.run.smithi151.stdout:osd.7: {} 2024-09-02T10:37:34.913 INFO:teuthology.orchestra.run.smithi151.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-02T10:37:34.935 INFO:teuthology.orchestra.run.smithi151.stdout:osd.8: {} 2024-09-02T10:37:34.935 INFO:teuthology.orchestra.run.smithi151.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-02T10:37:34.957 INFO:teuthology.orchestra.run.smithi151.stdout:osd.9: {} 2024-09-02T10:37:34.957 INFO:teuthology.orchestra.run.smithi151.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-02T10:37:34.978 INFO:teuthology.orchestra.run.smithi151.stdout:osd.10: {} 2024-09-02T10:37:34.978 INFO:teuthology.orchestra.run.smithi151.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-02T10:37:34.998 INFO:teuthology.orchestra.run.smithi151.stdout:osd.11: {} 2024-09-02T10:37:34.998 INFO:teuthology.orchestra.run.smithi151.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-02T10:37:35.002 DEBUG:teuthology.orchestra.run.smithi187:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8 2024-09-02T10:37:35.022 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: docker: command not found 2024-09-02T10:37:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:35.024 INFO:tasks.cephadm.osd.8:Ignoring exception while sending signal: Command failed on smithi187 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.8' 2024-09-02T10:37:35.095 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is set 2024-09-02T10:37:35.124 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6 2024-09-02T10:37:35.145 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:35.147 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:35.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.6' 2024-09-02T10:37:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:35 smithi151 ceph-mon[16639]: 9.a scrub starts 2024-09-02T10:37:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:35 smithi151 ceph-mon[16639]: 9.a scrub ok 2024-09-02T10:37:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:35 smithi151 ceph-mon[16639]: pgmap v2274: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 248/6 objects degraded (4133.333%); 0 B/s, 124 objects/s recovering 2024-09-02T10:37:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:35 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:37:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:35 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:37:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/798852677' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:37:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:35 smithi151 ceph-mon[16639]: osdmap e998: 12 total, 12 up, 11 in 2024-09-02T10:37:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1739943062' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:37:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:35 smithi151 ceph-mon[16639]: Health check update: Degraded data redundancy: 248/6 objects degraded (4133.333%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:35.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:35 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1224706697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:35.249 DEBUG:teuthology.orchestra.run.smithi162:> sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4 2024-09-02T10:37:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:35 smithi187 ceph-mon[20072]: 9.a scrub starts 2024-09-02T10:37:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:35 smithi187 ceph-mon[20072]: 9.a scrub ok 2024-09-02T10:37:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:35 smithi187 ceph-mon[20072]: pgmap v2274: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 248/6 objects degraded (4133.333%); 0 B/s, 124 objects/s recovering 2024-09-02T10:37:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:35 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:37:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:35 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:37:35.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/798852677' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:37:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:35 smithi187 ceph-mon[20072]: osdmap e998: 12 total, 12 up, 11 in 2024-09-02T10:37:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1739943062' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:37:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:35 smithi187 ceph-mon[20072]: Health check update: Degraded data redundancy: 248/6 objects degraded (4133.333%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:35.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:35 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1224706697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:35.271 INFO:teuthology.orchestra.run.smithi162.stderr:sudo: docker: command not found 2024-09-02T10:37:35.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T10:37:35.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi162 with status 1: 'sudo docker kill -s 1 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd.4' 2024-09-02T10:37:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:35 smithi162 ceph-mon[20279]: 9.a scrub starts 2024-09-02T10:37:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:35 smithi162 ceph-mon[20279]: 9.a scrub ok 2024-09-02T10:37:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:35 smithi162 ceph-mon[20279]: pgmap v2274: 57 pgs: 1 active+recovering+undersized+degraded+remapped, 1 peering, 55 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail; 248/6 objects degraded (4133.333%); 0 B/s, 124 objects/s recovering 2024-09-02T10:37:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:35 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:37:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:35 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:37:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/798852677' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_7", "pool2": "unique_pool_7", "yes_i_really_really_mean_it": true}]': finished 2024-09-02T10:37:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:35 smithi162 ceph-mon[20279]: osdmap e998: 12 total, 12 up, 11 in 2024-09-02T10:37:35.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1739943062' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-09-02T10:37:35.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:35 smithi162 ceph-mon[20279]: Health check update: Degraded data redundancy: 248/6 objects degraded (4133.333%), 1 pg degraded (PG_DEGRADED) 2024-09-02T10:37:35.346 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:35 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1224706697' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-09-02T10:37:35.349 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-02T10:37:35.361 INFO:tasks.thrashosds:joining thrashosds 2024-09-02T10:37:35.608 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:37:35.608 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:37:35.608 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:37:35.608 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:37:36.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:36 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:37:36.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:36 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:37:36.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:36 smithi151 ceph-mon[16639]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:36.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:36 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1739943062' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:37:36.253 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:36 smithi151 ceph-mon[16639]: osdmap e999: 12 total, 12 up, 11 in 2024-09-02T10:37:36.300 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-09-02T10:37:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:36 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:37:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:36 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:37:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:36 smithi162 ceph-mon[20279]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:36 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1739943062' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:37:36.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:36 smithi162 ceph-mon[20279]: osdmap e999: 12 total, 12 up, 11 in 2024-09-02T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:36 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:36 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:36 smithi187 ceph-mon[20072]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-02T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:36 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1739943062' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-02T10:37:36.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:36 smithi187 ceph-mon[20072]: osdmap e999: 12 total, 12 up, 11 in 2024-09-02T10:37:36.541 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:36.975 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:37:36.976 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:37:36.976 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:37:36.976 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:37:36.976 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:37:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:37 smithi187 ceph-mon[20072]: pgmap v2277: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:37 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:37:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:37 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:37:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:37 smithi187 ceph-mon[20072]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 248/6 objects degraded (4133.333%), 1 pg degraded) 2024-09-02T10:37:37.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:37 smithi187 ceph-mon[20072]: osdmap e1000: 12 total, 12 up, 11 in 2024-09-02T10:37:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:37 smithi162 ceph-mon[20279]: pgmap v2277: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:37 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:37:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:37 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:37:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:37 smithi162 ceph-mon[20279]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 248/6 objects degraded (4133.333%), 1 pg degraded) 2024-09-02T10:37:37.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:37 smithi162 ceph-mon[20279]: osdmap e1000: 12 total, 12 up, 11 in 2024-09-02T10:37:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:37 smithi151 ceph-mon[16639]: pgmap v2277: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:37 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:37:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:37 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:37:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:37 smithi151 ceph-mon[16639]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 248/6 objects degraded (4133.333%), 1 pg degraded) 2024-09-02T10:37:37.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:37 smithi151 ceph-mon[16639]: osdmap e1000: 12 total, 12 up, 11 in 2024-09-02T10:37:37.621 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-09-02T10:37:37.867 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:37.945 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-02T10:37:37.945 INFO:tasks.thrashosds.thrasher:Adding osd 11 2024-09-02T10:37:37.946 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd in 11 2024-09-02T10:37:38.204 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146099 8.6 8:6509e9b8:::benchmark_data_smithi187_136059_object40818:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.705971+0000", 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 376.67471340100002, 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.37556117500000002, 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.383 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146099 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705971+0000", 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705971+0000", 2024-09-02T10:37:38.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705972+0000", 2024-09-02T10:37:38.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2359999999999999e-06 2024-09-02T10:37:38.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705976+0000", 2024-09-02T10:37:38.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1470000000000003e-06 2024-09-02T10:37:38.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705986+0000", 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0161e-05 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.705993+0000", 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9330000000000004e-06 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.719530+0000", 2024-09-02T10:37:38.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.013537077999999999 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.719606+0000", 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5570999999999999e-05 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.719685+0000", 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.9579000000000005e-05 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:38.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.723078+0000", 2024-09-02T10:37:38.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0033929590000000001 2024-09-02T10:37:38.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:38.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.756155+0000", 2024-09-02T10:37:38.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.033076326000000003 2024-09-02T10:37:38.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.080171+0000", 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32401667499999998 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.080216+0000", 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5018999999999997e-05 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.081532+0000", 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001315491 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146131 8.6 8:6708c0fe:::benchmark_data_smithi187_136059_object40850:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.749657+0000", 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 376.63102744600002, 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.38950238999999998, 2024-09-02T10:37:38.393 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.394 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.394 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:38.394 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.394 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:38.394 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:38.394 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146131 2024-09-02T10:37:38.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.394 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749657+0000", 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749657+0000", 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749658+0000", 2024-09-02T10:37:38.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1799999999999999e-06 2024-09-02T10:37:38.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749662+0000", 2024-09-02T10:37:38.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.6519999999999997e-06 2024-09-02T10:37:38.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749674+0000", 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1518000000000001e-05 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.749681+0000", 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8759999999999999e-06 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.751393+0000", 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0017116729999999999 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.751501+0000", 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000108157 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.751635+0000", 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013469600000000001 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.756230+0000", 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045950890000000001 2024-09-02T10:37:38.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.097478+0000", 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.34124748199999999 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.137802+0000", 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.040324368999999999 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.137851+0000", 2024-09-02T10:37:38.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8349000000000001e-05 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.139159+0000", 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013083490000000001 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:210545 8.5 8:a9a4e4c5:::benchmark_data_smithi187_136059_object105264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:16.481782+0000", 2024-09-02T10:37:38.400 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 261.898902373, 2024-09-02T10:37:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.47180958000000001, 2024-09-02T10:37:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:38.401 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:38.402 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 210545 2024-09-02T10:37:38.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.402 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481782+0000", 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481782+0000", 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481783+0000", 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.68e-06 2024-09-02T10:37:38.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481791+0000", 2024-09-02T10:37:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8089999999999993e-06 2024-09-02T10:37:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481806+0000", 2024-09-02T10:37:38.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5201999999999999e-05 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481814+0000", 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0469999999999994e-06 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.481887+0000", 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2628999999999999e-05 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.482010+0000", 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012311000000000001 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.482127+0000", 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000117141 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:38.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.488686+0000", 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0065583769999999998 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.490795+0000", 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0021094870000000002 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953492+0000", 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46269710800000002 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953548+0000", 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5680999999999998e-05 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953591+0000", 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3309000000000002e-05 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.408 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:210545 8.5 e881/792 ondisk, result = 0)", 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:16.487082+0000", 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 261.89360195299997, 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46651547900000001, 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.487082+0000", 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.409 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.487082+0000", 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.487083+0000", 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3499999999999996e-07 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.487087+0000", 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1819999999999999e-06 2024-09-02T10:37:38.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953386+0000", 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.46629904500000002 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953393+0000", 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.5379999999999999e-06 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953475+0000", 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2639000000000006e-05 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953485+0000", 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0113e-05 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.953598+0000", 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000112327 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.412 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:210555 8.5 8:a562631a:::benchmark_data_smithi187_136059_object105274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:16.494622+0000", 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 261.88606220200001, 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.68545412900000002, 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 210555 2024-09-02T10:37:38.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494622+0000", 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494622+0000", 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494623+0000", 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.37e-06 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494629+0000", 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.293e-06 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494639+0000", 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0536999999999999e-05 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494647+0000", 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.5179999999999998e-06 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494746+0000", 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9341999999999999e-05 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494908+0000", 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00016205899999999999 2024-09-02T10:37:38.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,4", 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.495036+0000", 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000127606 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.500063+0000", 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0050271839999999996 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.557999+0000", 2024-09-02T10:37:38.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.057935954999999997 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.180008+0000", 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.62200955499999999 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.180048+0000", 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9977e-05 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.180076+0000", 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.7733e-05 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.27559.0:210554 8.4 e881/792 8:25e06d4f:::benchmark_data_smithi187_136059_object105273:head v 881'13332, mlcod=881'13328)", 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:16.494855+0000", 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 261.885829022, 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.513357536, 2024-09-02T10:37:38.419 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494855+0000", 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494855+0000", 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494856+0000", 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3900000000000004e-07 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.494862+0000", 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8560000000000003e-06 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.004275+0000", 2024-09-02T10:37:38.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.50941320800000001 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.004280+0000", 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5809999999999996e-06 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.004348+0000", 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.7230999999999994e-05 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.004367+0000", 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8856e-05 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.008181+0000", 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.003814797 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.008213+0000", 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1267999999999999e-05 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop_reply(client.27559.0:210555 8.5 e881/792 ondisk, result = 0)", 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:16.500168+0000", 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 261.88051610299999, 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.67991180699999998, 2024-09-02T10:37:38.425 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.500168+0000", 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.500168+0000", 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.426 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.500169+0000", 2024-09-02T10:37:38.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.8599999999999998e-07 2024-09-02T10:37:38.427 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.427 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.427 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.427 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:16.500176+0000", 2024-09-02T10:37:38.427 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6389999999999992e-06 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.179888+0000", 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.67971182900000005 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.179901+0000", 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2437e-05 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.428 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.179998+0000", 2024-09-02T10:37:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6984999999999994e-05 2024-09-02T10:37:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.180004+0000", 2024-09-02T10:37:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1249999999999998e-06 2024-09-02T10:37:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.429 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:17.180080+0000", 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6106000000000006e-05 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGRecoveryDeleteReply(8.6 e882,844 8:60b27707:::benchmark_data_smithi187_136059_object22977:head,839'8144)", 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:18.621005+0000", 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 259.75967917399998, 2024-09-02T10:37:38.430 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43784977800000002, 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:18.621005+0000", 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.431 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:18.621005+0000", 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:18.621007+0000", 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.59e-06 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.432 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:18.621016+0000", 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9259999999999995e-06 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:19.058550+0000", 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.43753405000000001 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:19.058557+0000", 2024-09-02T10:37:38.433 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0730000000000001e-06 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:19.058602+0000", 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.5318999999999998e-05 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:19.058855+0000", 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00025282000000000001 2024-09-02T10:37:38.434 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGRecoveryDelete(8.8 e883,854 8:1147bf7c:::benchmark_data_smithi187_136059_object4341:head,836'6957)", 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:21.592106+0000", 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 256.78857802300001, 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94375713100000003, 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.435 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.592106+0000", 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.592106+0000", 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.436 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.592108+0000", 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5969999999999999e-06 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:21.592120+0000", 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2757e-05 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.437 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:22.535701+0000", 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.94358019999999998 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:22.535706+0000", 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2680000000000002e-06 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:22.535786+0000", 2024-09-02T10:37:38.438 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0296999999999994e-05 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:22.535793+0000", 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4609999999999996e-06 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:22.535863+0000", 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0550999999999996e-05 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.439 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27090 9.9 9.411b16f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.942484+0000", 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 201.43820037500001, 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93450280299999999, 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:38.440 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27090 2024-09-02T10:37:38.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.441 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942484+0000", 2024-09-02T10:37:38.441 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.441 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.441 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.441 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.441 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942484+0000", 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942485+0000", 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.5900000000000005e-07 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942501+0000", 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6467e-05 2024-09-02T10:37:38.442 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942511+0000", 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.9699999999999994e-06 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942516+0000", 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2299999999999999e-06 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942564+0000", 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.7747999999999998e-05 2024-09-02T10:37:38.443 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.942571+0000", 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.4339999999999997e-06 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.876944+0000", 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.93437385699999997 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.876987+0000", 2024-09-02T10:37:38.444 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.2138000000000002e-05 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27095 9.9 9.a3492329 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.979259+0000", 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 201.40142496799999, 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89779759400000003, 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.445 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27095 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979259+0000", 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979259+0000", 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.446 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979261+0000", 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4529999999999999e-06 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979280+0000", 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9395e-05 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.447 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979292+0000", 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.169e-05 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979297+0000", 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4149999999999998e-06 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979356+0000", 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8548000000000003e-05 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.448 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.979367+0000", 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1421e-05 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877039+0000", 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89767237799999999 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877057+0000", 2024-09-02T10:37:38.449 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7294000000000001e-05 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27114 9.9 9.7327df9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.985981+0000", 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 201.394702849, 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89113120999999995, 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.450 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27114 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.985981+0000", 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.451 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.985981+0000", 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.985985+0000", 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.839e-06 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.986009+0000", 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3682000000000001e-05 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.452 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.986030+0000", 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1725e-05 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.986036+0000", 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7150000000000003e-06 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.986108+0000", 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2285000000000003e-05 2024-09-02T10:37:38.453 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.986120+0000", 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1292e-05 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877103+0000", 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.89098296300000002 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877112+0000", 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.7089999999999997e-06 2024-09-02T10:37:38.454 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27124 9.9 9.9deeea19 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.989404+0000", 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 201.39128045999999, 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88776398999999995, 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:37:38.455 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27124 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989404+0000", 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989404+0000", 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.456 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989405+0000", 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8100000000000001e-07 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989424+0000", 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9490000000000001e-05 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989435+0000", 2024-09-02T10:37:38.457 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1039000000000001e-05 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989442+0000", 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0639999999999996e-06 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989501+0000", 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8409000000000003e-05 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:37:38.458 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.989508+0000", 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.8669999999999994e-06 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877157+0000", 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88764881399999995 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877168+0000", 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0326000000000001e-05 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.459 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27131 9.9 9.f983e449 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.991168+0000", 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 201.38951630400001, 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88604997500000005, 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27131 2024-09-02T10:37:38.460 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991168+0000", 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991168+0000", 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991169+0000", 2024-09-02T10:37:38.461 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.2200000000000003e-07 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991185+0000", 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.6841e-05 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991196+0000", 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0463e-05 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.462 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991202+0000", 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6720000000000001e-06 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991257+0000", 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4999e-05 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.991266+0000", 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2429999999999994e-06 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.463 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877208+0000", 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.885942283 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877218+0000", 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.6520000000000001e-06 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27134 9.9 9.f56117f9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.993011+0000", 2024-09-02T10:37:38.464 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 201.38767290199999, 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.884242682, 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27134 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.465 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993011+0000", 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993011+0000", 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993012+0000", 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.186e-06 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.466 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993032+0000", 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9831999999999999e-05 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993043+0000", 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1216999999999999e-05 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993049+0000", 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5389999999999998e-06 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.467 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993111+0000", 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2469000000000001e-05 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.993121+0000", 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0044000000000001e-05 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877245+0000", 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.88412303599999997 2024-09-02T10:37:38.468 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877254+0000", 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.3589999999999997e-06 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27144 9.9 9.d4d23b29 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:16.998807+0000", 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 201.38187666100001, 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.87849941300000001, 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.469 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27144 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998807+0000", 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.470 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998807+0000", 2024-09-02T10:37:38.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.471 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998809+0000", 2024-09-02T10:37:38.471 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3599999999999999e-06 2024-09-02T10:37:38.471 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.471 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.471 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998831+0000", 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2566e-05 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998857+0000", 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5573999999999999e-05 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998866+0000", 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9080000000000001e-06 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.472 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998924+0000", 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8588999999999998e-05 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:16.998934+0000", 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8220000000000002e-06 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877298+0000", 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.87836354100000003 2024-09-02T10:37:38.473 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877307+0000", 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0529999999999996e-06 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27151 9.9 9.31ef2d19 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.008837+0000", 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 201.37184691300001, 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.86850359600000004, 2024-09-02T10:37:38.474 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27151 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008837+0000", 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.475 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008837+0000", 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008838+0000", 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.2330000000000001e-06 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008867+0000", 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8999000000000001e-05 2024-09-02T10:37:38.476 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008878+0000", 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0888e-05 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008885+0000", 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1160000000000001e-06 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008934+0000", 2024-09-02T10:37:38.477 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8463e-05 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.008940+0000", 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.0020000000000003e-06 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877332+0000", 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.86839186499999999 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877341+0000", 2024-09-02T10:37:38.478 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0299999999999999e-06 2024-09-02T10:37:38.479 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.479 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.479 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.479 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.479 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.479 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27173 9.9 9.e697529 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.028584+0000", 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 201.35210054000001, 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.84879224399999997, 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27173 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.480 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028584+0000", 2024-09-02T10:37:38.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.481 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.481 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.481 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.481 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028584+0000", 2024-09-02T10:37:38.481 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028584+0000", 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.0999999999999997e-07 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028603+0000", 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8598999999999999e-05 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.482 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028613+0000", 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.03e-05 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028621+0000", 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4699999999999996e-06 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028671+0000", 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0302000000000001e-05 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.483 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.028678+0000", 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.255e-06 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877367+0000", 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.84868848299999999 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.484 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877376+0000", 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.9250000000000001e-06 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.28030.0:27202 9.9 9.cbef6c09 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e912)", 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:17.465428+0000", 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 200.91525640899999, 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41198401099999998, 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.485 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "reached pg", 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.28030", 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/3317059155", 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 27202 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465428+0000", 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.486 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465428+0000", 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465429+0000", 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.1999999999999998e-07 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465447+0000", 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8578999999999999e-05 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.487 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465458+0000", 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1276e-05 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465464+0000", 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.6949999999999997e-06 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465513+0000", 2024-09-02T10:37:38.488 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9415e-05 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for peered", 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.465521+0000", 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4120000000000004e-06 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877403+0000", 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41188169400000002 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.489 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:17.877412+0000", 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1200000000000008e-06 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "MOSDPGRecoveryDelete(9.14 e992,976 9:2c6fadb7:::benchmark_data_smithi187_154927_object33875:head,955'6091)", 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:37:23.997590+0000", 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 14.383093611, 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41465239500000001, 2024-09-02T10:37:38.490 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "started", 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:37:23.997590+0000", 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:37:23.997590+0000", 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:38.491 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:37:23.997591+0000", 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.3e-07 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:37:23.997597+0000", 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.4249999999999997e-06 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:37:24.412108+0000", 2024-09-02T10:37:38.492 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.41451161399999997 2024-09-02T10:37:38.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:38.493 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:37:24.412115+0000", 2024-09-02T10:37:38.493 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.1189999999999999e-06 2024-09-02T10:37:38.493 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.493 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.493 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:37:24.412174+0000", 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8554999999999998e-05 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:37:24.412185+0000", 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0822000000000001e-05 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:37:24.412243+0000", 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8230000000000003e-05 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.494 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.495 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.495 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:38.495 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:38.495 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:37:38.734 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset noscrub 2024-09-02T10:37:38.986 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:39.097 INFO:teuthology.orchestra.run.smithi151.stderr:marked in osd.11. 2024-09-02T10:37:39.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:38 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:37:38.883+0000 7fe3ffed4640 -1 mon.a@0(leader).osd e1000 definitely_dead 0 2024-09-02T10:37:39.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:39 smithi151 ceph-mon[16639]: pgmap v2279: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:39.144 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:39 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1445033189' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-09-02T10:37:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:39 smithi162 ceph-mon[20279]: pgmap v2279: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:39.345 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:39 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1445033189' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-09-02T10:37:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:39 smithi187 ceph-mon[20072]: pgmap v2279: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:39.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:39 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1445033189' entity='client.admin' cmd={"prefix": "osd in", "ids": ["11"]} : dispatch 2024-09-02T10:37:39.629 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-09-02T10:37:39.873 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1445033189' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-09-02T10:37:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:40 smithi187 ceph-mon[20072]: osdmap e1001: 12 total, 12 up, 12 in 2024-09-02T10:37:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:40 smithi187 ceph-mon[20072]: osdmap e1002: 12 total, 12 up, 12 in 2024-09-02T10:37:40.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:40 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3513024076' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:37:40.333 INFO:tasks.thrashosds.thrasher:Added osd 11 2024-09-02T10:37:40.333 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-02T10:37:40.353 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:37:40.353 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:37:40.354 INFO:teuthology.orchestra.run.smithi187.stdout: "num_ops": 0 2024-09-02T10:37:40.354 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:37:40.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1445033189' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-09-02T10:37:40.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:40 smithi151 ceph-mon[16639]: osdmap e1001: 12 total, 12 up, 12 in 2024-09-02T10:37:40.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:40 smithi151 ceph-mon[16639]: osdmap e1002: 12 total, 12 up, 12 in 2024-09-02T10:37:40.355 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:40 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3513024076' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:37:40.425 INFO:teuthology.orchestra.run.smithi151.stderr:noscrub is unset 2024-09-02T10:37:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1445033189' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["11"]}]': finished 2024-09-02T10:37:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:40 smithi162 ceph-mon[20279]: osdmap e1001: 12 total, 12 up, 12 in 2024-09-02T10:37:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:40 smithi162 ceph-mon[20279]: osdmap e1002: 12 total, 12 up, 12 in 2024-09-02T10:37:40.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:40 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3513024076' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-09-02T10:37:40.953 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-09-02T10:37:41.037 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-02T10:37:41.127 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:41.195 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:41 smithi187 ceph-mon[20072]: pgmap v2282: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:41.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:41 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:37:41.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:41 smithi187 ceph-mon[20072]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-02T10:37:41.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:41 smithi187 ceph-mon[20072]: Cluster is now healthy 2024-09-02T10:37:41.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:41 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3513024076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:37:41.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:41 smithi187 ceph-mon[20072]: osdmap e1003: 12 total, 12 up, 12 in 2024-09-02T10:37:41.277 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:41 smithi151 ceph-mon[16639]: pgmap v2282: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:41.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:41 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:37:41.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:41 smithi151 ceph-mon[16639]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-02T10:37:41.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:41 smithi151 ceph-mon[16639]: Cluster is now healthy 2024-09-02T10:37:41.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:41 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3513024076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:37:41.278 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:41 smithi151 ceph-mon[16639]: osdmap e1003: 12 total, 12 up, 12 in 2024-09-02T10:37:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:41 smithi162 ceph-mon[20279]: pgmap v2282: 31 pgs: 31 active+clean; 577 KiB data, 8.5 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:41 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:37:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:41 smithi162 ceph-mon[20279]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-02T10:37:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:41 smithi162 ceph-mon[20279]: Cluster is now healthy 2024-09-02T10:37:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:41 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3513024076' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-02T10:37:41.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:41 smithi162 ceph-mon[20279]: osdmap e1003: 12 total, 12 up, 12 in 2024-09-02T10:37:41.646 INFO:teuthology.orchestra.run.smithi151.stdout:mon.a: {} 2024-09-02T10:37:41.646 INFO:teuthology.orchestra.run.smithi151.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-02T10:37:41.658 INFO:teuthology.orchestra.run.smithi151.stdout:mon.c: {} 2024-09-02T10:37:41.658 INFO:teuthology.orchestra.run.smithi151.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-02T10:37:41.662 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:37:41.662 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [], 2024-09-02T10:37:41.662 INFO:teuthology.orchestra.run.smithi187.stdout: "complaint_time": 30, 2024-09-02T10:37:41.662 INFO:teuthology.orchestra.run.smithi187.stdout: "num_blocked_ops": 0 2024-09-02T10:37:41.662 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:37:41.673 INFO:teuthology.orchestra.run.smithi151.stdout:mon.b: {} 2024-09-02T10:37:41.673 INFO:teuthology.orchestra.run.smithi151.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-02T10:37:41.681 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:42.282 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:42 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2851648593' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T10:37:42.282 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:42 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:37:42.282 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:42 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:37:42.293 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:42 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2851648593' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T10:37:42.293 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:42 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:37:42.293 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:42 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:37:42.312 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-09-02T10:37:42.320 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 0 1 2024-09-02T10:37:42.554 INFO:teuthology.orchestra.run.smithi187.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.c/config 2024-09-02T10:37:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:42 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2851648593' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-09-02T10:37:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:42 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:37:42.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:42 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:37:42.852 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:43.107 INFO:teuthology.orchestra.run.smithi151.stderr:nodeep-scrub is unset 2024-09-02T10:37:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:43 smithi187 ceph-mon[20072]: pgmap v2284: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:43 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:37:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:43 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:37:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:43 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:37:43.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:43 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:37:43.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:43 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:37:43.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:43 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:37:43.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:43 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2175362096' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:43.312 INFO:teuthology.orchestra.run.smithi187.stdout:{ 2024-09-02T10:37:43.312 INFO:teuthology.orchestra.run.smithi187.stdout: "size": 20, 2024-09-02T10:37:43.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 600, 2024-09-02T10:37:43.313 INFO:teuthology.orchestra.run.smithi187.stdout: "ops": [ 2024-09-02T10:37:43.313 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.313 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:21803 8.6 8:6cea7233:::benchmark_data_smithi187_136059_object21802:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e804)", 2024-09-02T10:37:43.313 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:13.620281+0000", 2024-09-02T10:37:43.313 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 509.40495162899998, 2024-09-02T10:37:43.313 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21402042600000001, 2024-09-02T10:37:43.313 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.313 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.314 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.314 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.314 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.314 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.314 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 21803 2024-09-02T10:37:43.314 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.314 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.314 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.314 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620281+0000", 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620281+0000", 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620282+0000", 2024-09-02T10:37:43.315 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1570000000000001e-06 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620313+0000", 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.1090000000000002e-05 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620324+0000", 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.115e-05 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.316 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620332+0000", 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0120000000000005e-06 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620376+0000", 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3488000000000001e-05 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620478+0000", 2024-09-02T10:37:43.317 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000102246 2024-09-02T10:37:43.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.318 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:37:43.318 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.620611+0000", 2024-09-02T10:37:43.318 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013364200000000001 2024-09-02T10:37:43.318 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.318 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.319 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.621411+0000", 2024-09-02T10:37:43.319 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00079942399999999997 2024-09-02T10:37:43.319 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.319 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.319 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.623938+0000", 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0025271460000000001 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834210+0000", 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21027237400000001 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.320 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834263+0000", 2024-09-02T10:37:43.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.3063999999999999e-05 2024-09-02T10:37:43.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.321 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.321 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.321 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.834301+0000", 2024-09-02T10:37:43.321 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7632999999999997e-05 2024-09-02T10:37:43.321 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.321 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.321 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.321 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.322 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.322 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:21820 8.6 8:6a3536fe:::benchmark_data_smithi187_136059_object21819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e804)", 2024-09-02T10:37:43.322 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:13.629639+0000", 2024-09-02T10:37:43.322 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 509.395593475, 2024-09-02T10:37:43.322 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21678998799999999, 2024-09-02T10:37:43.323 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.323 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.323 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.323 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.323 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.324 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.324 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 21820 2024-09-02T10:37:43.324 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.324 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.324 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.324 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.325 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629639+0000", 2024-09-02T10:37:43.325 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.325 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.325 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.325 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.326 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629639+0000", 2024-09-02T10:37:43.326 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.326 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.326 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.326 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629641+0000", 2024-09-02T10:37:43.327 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.0169999999999999e-06 2024-09-02T10:37:43.327 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.327 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.327 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.327 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629670+0000", 2024-09-02T10:37:43.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.9173999999999999e-05 2024-09-02T10:37:43.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.328 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.328 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.328 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629689+0000", 2024-09-02T10:37:43.328 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9045000000000001e-05 2024-09-02T10:37:43.328 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.329 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.329 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629697+0000", 2024-09-02T10:37:43.329 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6199999999999999e-06 2024-09-02T10:37:43.329 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.329 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629747+0000", 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.9968e-05 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.629904+0000", 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000157656 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.330 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 7,10", 2024-09-02T10:37:43.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.630021+0000", 2024-09-02T10:37:43.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000116347 2024-09-02T10:37:43.331 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.331 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.331 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.331 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.631099+0000", 2024-09-02T10:37:43.331 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010783819999999999 2024-09-02T10:37:43.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.332 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.631117+0000", 2024-09-02T10:37:43.332 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7915e-05 2024-09-02T10:37:43.332 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.332 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.332 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.846333+0000", 2024-09-02T10:37:43.333 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21521643500000001 2024-09-02T10:37:43.333 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.333 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.333 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.333 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.846391+0000", 2024-09-02T10:37:43.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7169000000000001e-05 2024-09-02T10:37:43.334 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.334 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.334 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.334 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:13.846429+0000", 2024-09-02T10:37:43.334 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.8260000000000003e-05 2024-09-02T10:37:43.334 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.335 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.335 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.335 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.335 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.335 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28292 8.d 8:b2a9e2ae:::benchmark_data_smithi187_136059_object28291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:37:43.335 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.105898+0000", 2024-09-02T10:37:43.336 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 504.91933416799998, 2024-09-02T10:37:43.336 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21874021900000001, 2024-09-02T10:37:43.336 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.336 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.336 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.336 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.337 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.337 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.337 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28292 2024-09-02T10:37:43.337 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.337 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.337 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.337 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.337 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105898+0000", 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105898+0000", 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105899+0000", 2024-09-02T10:37:43.338 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.068e-06 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105924+0000", 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.5179000000000001e-05 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105938+0000", 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.3713000000000001e-05 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.339 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.105946+0000", 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6559999999999992e-06 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.106008+0000", 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.2001999999999994e-05 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.340 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.106149+0000", 2024-09-02T10:37:43.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00014115800000000001 2024-09-02T10:37:43.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:37:43.341 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.106281+0000", 2024-09-02T10:37:43.341 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000132466 2024-09-02T10:37:43.341 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.341 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.341 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.107636+0000", 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.001355055 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.113043+0000", 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0054062870000000001 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324567+0000", 2024-09-02T10:37:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21152405399999999 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324604+0000", 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7017999999999997e-05 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324638+0000", 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4563000000000001e-05 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.343 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28300 8.f 8:f195aa31:::benchmark_data_smithi187_136059_object28299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.108959+0000", 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 504.916273553, 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21860515799999999, 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.344 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.345 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28300 2024-09-02T10:37:43.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.345 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.345 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108959+0000", 2024-09-02T10:37:43.345 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.345 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.345 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.345 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108959+0000", 2024-09-02T10:37:43.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.346 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.346 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108960+0000", 2024-09-02T10:37:43.346 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.5209999999999999e-06 2024-09-02T10:37:43.346 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.346 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108983+0000", 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3189000000000001e-05 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.108998+0000", 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4542000000000001e-05 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.347 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109005+0000", 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4170000000000003e-06 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109063+0000", 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7337999999999999e-05 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.348 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109178+0000", 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115203 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.109310+0000", 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000132156 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.110649+0000", 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0013386990000000001 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.349 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111465+0000", 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0008164 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327469+0000", 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.216004207 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327527+0000", 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.7271000000000001e-05 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.350 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327564+0000", 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.7215000000000002e-05 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28305 8.d 8:bd02d929:::benchmark_data_smithi187_136059_object28304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.111275+0000", 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 504.91395705799999, 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21380102300000001, 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.351 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28305 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111275+0000", 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.352 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111275+0000", 2024-09-02T10:37:43.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.353 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.353 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111276+0000", 2024-09-02T10:37:43.353 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.4399999999999999e-07 2024-09-02T10:37:43.353 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.353 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111295+0000", 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8816000000000001e-05 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111305+0000", 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0589e-05 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111311+0000", 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2650000000000004e-06 2024-09-02T10:37:43.354 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111387+0000", 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.6338e-05 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111513+0000", 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012598000000000001 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 2,7", 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.111655+0000", 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000141726 2024-09-02T10:37:43.355 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112644+0000", 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00098899699999999992 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.127377+0000", 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.014733448 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.324992+0000", 2024-09-02T10:37:43.356 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.19761451799999999 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.325035+0000", 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.3201999999999997e-05 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.325076+0000", 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.1300000000000001e-05 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.357 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28306 8.f 8:fcc25ce2:::benchmark_data_smithi187_136059_object28305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.112056+0000", 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 504.91317626300003, 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21564313299999999, 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28306 2024-09-02T10:37:43.358 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112056+0000", 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112056+0000", 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.359 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112057+0000", 2024-09-02T10:37:43.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.144e-06 2024-09-02T10:37:43.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.360 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.360 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.360 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112081+0000", 2024-09-02T10:37:43.360 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.3713000000000001e-05 2024-09-02T10:37:43.360 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112091+0000", 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.8090000000000004e-06 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112098+0000", 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.0990000000000001e-06 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.361 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112150+0000", 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.1768999999999999e-05 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112236+0000", 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.6237999999999997e-05 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.112316+0000", 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0603000000000001e-05 2024-09-02T10:37:43.362 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.113365+0000", 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0010481539999999999 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.113551+0000", 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00018640599999999999 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327611+0000", 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.21405996499999999 2024-09-02T10:37:43.363 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327664+0000", 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2982999999999997e-05 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327699+0000", 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.5250000000000003e-05 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.364 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:28313 8.f 8:fb500fbb:::benchmark_data_smithi187_136059_object28312:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e806)", 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:29:18.117307+0000", 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 504.90792506100001, 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.210606557, 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.365 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 28313 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117307+0000", 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117307+0000", 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.366 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117308+0000", 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1990000000000001e-06 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117328+0000", 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9528999999999999e-05 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117342+0000", 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.4219000000000001e-05 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.367 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117350+0000", 2024-09-02T10:37:43.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.0120000000000005e-06 2024-09-02T10:37:43.368 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.368 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.368 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.368 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117456+0000", 2024-09-02T10:37:43.368 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000106291 2024-09-02T10:37:43.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.369 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.369 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.369 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117573+0000", 2024-09-02T10:37:43.369 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00011608 2024-09-02T10:37:43.369 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.370 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.370 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:37:43.370 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.117677+0000", 2024-09-02T10:37:43.370 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010426500000000001 2024-09-02T10:37:43.370 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.118658+0000", 2024-09-02T10:37:43.371 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00098097000000000006 2024-09-02T10:37:43.371 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.371 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.371 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.371 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.119074+0000", 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000416384 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327819+0000", 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.20874446899999999 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327880+0000", 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.1617000000000003e-05 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.372 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:29:18.327914+0000", 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3522000000000002e-05 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146135 8.f 8:ffa01c02:::benchmark_data_smithi187_136059_object40854:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.753711+0000", 2024-09-02T10:37:43.373 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.27152123100001, 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.33445889499999998, 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146135 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.374 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753711+0000", 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753711+0000", 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753713+0000", 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8589999999999999e-06 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.375 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753720+0000", 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.2579999999999998e-06 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753739+0000", 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.8916999999999999e-05 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.753748+0000", 2024-09-02T10:37:43.376 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.1190000000000003e-06 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.755012+0000", 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0012635649999999999 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.755145+0000", 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013299500000000001 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:37:43.377 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.755260+0000", 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000115469 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.758833+0000", 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0035724749999999999 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.768082+0000", 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0092492670000000003 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.378 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.088097+0000", 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.32001463499999999 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.088136+0000", 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.9756999999999999e-05 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.379 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.380 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.088170+0000", 2024-09-02T10:37:43.380 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.3578999999999998e-05 2024-09-02T10:37:43.380 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.380 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.380 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.380 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.380 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.380 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:146139 8.f 8:f849bf1d:::benchmark_data_smithi187_136059_object40858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e848)", 2024-09-02T10:37:43.380 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:31:21.773477+0000", 2024-09-02T10:37:43.381 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 381.25175532399999, 2024-09-02T10:37:43.381 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31800191100000003, 2024-09-02T10:37:43.381 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.381 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.381 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.381 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.381 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 146139 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773477+0000", 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.382 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773477+0000", 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773478+0000", 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1790000000000001e-06 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773483+0000", 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.8280000000000002e-06 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.383 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773494+0000", 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1419e-05 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773500+0000", 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.874e-06 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.384 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773553+0000", 2024-09-02T10:37:43.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.2957999999999997e-05 2024-09-02T10:37:43.385 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.385 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.385 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.385 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773667+0000", 2024-09-02T10:37:43.385 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000113812 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.773777+0000", 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00010969899999999999 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.777216+0000", 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0034393179999999998 2024-09-02T10:37:43.386 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:21.777885+0000", 2024-09-02T10:37:43.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00066886699999999996 2024-09-02T10:37:43.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.387 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.387 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.387 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.091413+0000", 2024-09-02T10:37:43.387 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.31352819799999998 2024-09-02T10:37:43.387 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.388 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.091446+0000", 2024-09-02T10:37:43.388 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2802999999999997e-05 2024-09-02T10:37:43.388 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.388 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.388 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:31:22.091479+0000", 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.2956000000000001e-05 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:183126 8.f 8:fec0a0af:::benchmark_data_smithi187_136059_object77845:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e869)", 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:32:29.326892+0000", 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 313.69834047000001, 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23921741799999999, 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.389 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 183126 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326892+0000", 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.390 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326892+0000", 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326894+0000", 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2029999999999999e-06 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326903+0000", 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2240000000000001e-06 2024-09-02T10:37:43.391 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326925+0000", 2024-09-02T10:37:43.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2033000000000001e-05 2024-09-02T10:37:43.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.392 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.392 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.392 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.326937+0000", 2024-09-02T10:37:43.392 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.1522000000000001e-05 2024-09-02T10:37:43.392 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.329339+0000", 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0024024099999999998 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.329467+0000", 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000127508 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.393 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.329604+0000", 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013698200000000001 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.334705+0000", 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0051010439999999999 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.347683+0000", 2024-09-02T10:37:43.394 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.012978701 2024-09-02T10:37:43.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.566044+0000", 2024-09-02T10:37:43.395 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.218360309 2024-09-02T10:37:43.395 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.395 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.395 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.395 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.566080+0000", 2024-09-02T10:37:43.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.6727999999999998e-05 2024-09-02T10:37:43.396 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.396 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.396 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.396 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:32:29.566109+0000", 2024-09-02T10:37:43.396 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.8753999999999999e-05 2024-09-02T10:37:43.396 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.396 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.397 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.397 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.397 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.397 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:207403 8.f 8:f9743dd7:::benchmark_data_smithi187_136059_object102122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:37:43.397 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:11.376024+0000", 2024-09-02T10:37:43.397 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 271.64920853299998, 2024-09-02T10:37:43.397 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.277723781, 2024-09-02T10:37:43.397 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.398 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.398 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.398 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.398 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.398 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.398 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207403 2024-09-02T10:37:43.398 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.398 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.399 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.399 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.399 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376024+0000", 2024-09-02T10:37:43.399 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.399 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376024+0000", 2024-09-02T10:37:43.400 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.400 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.400 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.400 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.400 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376026+0000", 2024-09-02T10:37:43.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2500000000000001e-06 2024-09-02T10:37:43.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.401 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.401 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376035+0000", 2024-09-02T10:37:43.401 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.2390000000000008e-06 2024-09-02T10:37:43.401 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.401 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376057+0000", 2024-09-02T10:37:43.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.1421999999999999e-05 2024-09-02T10:37:43.402 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.402 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.402 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.402 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376066+0000", 2024-09-02T10:37:43.402 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 9.166e-06 2024-09-02T10:37:43.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376136+0000", 2024-09-02T10:37:43.403 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.9982999999999997e-05 2024-09-02T10:37:43.403 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.403 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.403 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.403 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376274+0000", 2024-09-02T10:37:43.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013833300000000001 2024-09-02T10:37:43.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.404 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.404 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:37:43.404 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.376405+0000", 2024-09-02T10:37:43.404 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00013087900000000001 2024-09-02T10:37:43.404 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381691+0000", 2024-09-02T10:37:43.405 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0052860909999999997 2024-09-02T10:37:43.405 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.405 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.405 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.405 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.381713+0000", 2024-09-02T10:37:43.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 2.2206999999999999e-05 2024-09-02T10:37:43.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.406 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.406 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.406 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653638+0000", 2024-09-02T10:37:43.406 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.27192507599999999 2024-09-02T10:37:43.406 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653697+0000", 2024-09-02T10:37:43.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8286999999999999e-05 2024-09-02T10:37:43.407 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.407 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.407 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.407 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653748+0000", 2024-09-02T10:37:43.407 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.0847999999999998e-05 2024-09-02T10:37:43.408 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.408 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.408 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.408 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.408 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.408 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_op(client.27559.0:207412 8.f 8:ff87a35e:::benchmark_data_smithi187_136059_object102131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e881)", 2024-09-02T10:37:43.408 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:33:11.416635+0000", 2024-09-02T10:37:43.408 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 271.60859678100002, 2024-09-02T10:37:43.409 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.237262055, 2024-09-02T10:37:43.409 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.409 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.409 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.409 INFO:teuthology.orchestra.run.smithi187.stdout: "client_info": { 2024-09-02T10:37:43.409 INFO:teuthology.orchestra.run.smithi187.stdout: "client": "client.27559", 2024-09-02T10:37:43.409 INFO:teuthology.orchestra.run.smithi187.stdout: "client_addr": "172.21.15.187:0/471386761", 2024-09-02T10:37:43.409 INFO:teuthology.orchestra.run.smithi187.stdout: "tid": 207412 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416635+0000", 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416635+0000", 2024-09-02T10:37:43.410 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416637+0000", 2024-09-02T10:37:43.411 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.9360000000000002e-06 2024-09-02T10:37:43.411 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.411 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.411 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.411 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416645+0000", 2024-09-02T10:37:43.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.4739999999999999e-06 2024-09-02T10:37:43.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.412 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416662+0000", 2024-09-02T10:37:43.412 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7523000000000001e-05 2024-09-02T10:37:43.412 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.412 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.412 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416671+0000", 2024-09-02T10:37:43.413 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 8.3599999999999996e-06 2024-09-02T10:37:43.413 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.413 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.413 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.413 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416737+0000", 2024-09-02T10:37:43.414 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.6088999999999997e-05 2024-09-02T10:37:43.414 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.414 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.414 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416865+0000", 2024-09-02T10:37:43.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00012849500000000001 2024-09-02T10:37:43.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.415 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "waiting for subops from 1,2", 2024-09-02T10:37:43.415 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.416984+0000", 2024-09-02T10:37:43.415 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.000118202 2024-09-02T10:37:43.415 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.415 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.416 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.416 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.421494+0000", 2024-09-02T10:37:43.416 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.0045106759999999999 2024-09-02T10:37:43.416 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.416 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "op_commit", 2024-09-02T10:37:43.417 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.421568+0000", 2024-09-02T10:37:43.417 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 7.3586999999999999e-05 2024-09-02T10:37:43.417 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.417 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.417 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "sub_op_commit_rec", 2024-09-02T10:37:43.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653798+0000", 2024-09-02T10:37:43.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.23222986800000001 2024-09-02T10:37:43.418 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.418 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.418 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.418 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653839+0000", 2024-09-02T10:37:43.418 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 4.087e-05 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:33:11.653898+0000", 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.8974999999999997e-05 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: "description": "osd_repop(client.28030.0:6771 9.9 e904/897 9:9a514e7d:::benchmark_data_smithi187_154927_object6770:head v 904'417, mlcod=904'414)", 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: "initiated_at": "2024-09-02T10:34:03.762823+0000", 2024-09-02T10:37:43.419 INFO:teuthology.orchestra.run.smithi187.stdout: "age": 219.26240887399999, 2024-09-02T10:37:43.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083770215999999995, 2024-09-02T10:37:43.420 INFO:teuthology.orchestra.run.smithi187.stdout: "continuous": false, 2024-09-02T10:37:43.420 INFO:teuthology.orchestra.run.smithi187.stdout: "type_data": { 2024-09-02T10:37:43.420 INFO:teuthology.orchestra.run.smithi187.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-02T10:37:43.420 INFO:teuthology.orchestra.run.smithi187.stdout: "events": [ 2024-09-02T10:37:43.420 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.420 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "initiated", 2024-09-02T10:37:43.420 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.762823+0000", 2024-09-02T10:37:43.420 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.420 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "header_read", 2024-09-02T10:37:43.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.762823+0000", 2024-09-02T10:37:43.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0 2024-09-02T10:37:43.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.421 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.421 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "throttled", 2024-09-02T10:37:43.421 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.762824+0000", 2024-09-02T10:37:43.421 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.0979999999999999e-06 2024-09-02T10:37:43.421 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "all_read", 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.762859+0000", 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4480999999999997e-05 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "dispatched", 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846015+0000", 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.083155899000000005 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.422 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "queued_for_pg", 2024-09-02T10:37:43.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846021+0000", 2024-09-02T10:37:43.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 6.55e-06 2024-09-02T10:37:43.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.423 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.423 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "reached_pg", 2024-09-02T10:37:43.423 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846077+0000", 2024-09-02T10:37:43.423 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 5.5758999999999999e-05 2024-09-02T10:37:43.423 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "started", 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846095+0000", 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 1.7768e-05 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "commit_sent", 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846560+0000", 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 0.00046457399999999999 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: }, 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: { 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: "event": "done", 2024-09-02T10:37:43.424 INFO:teuthology.orchestra.run.smithi187.stdout: "time": "2024-09-02T10:34:03.846594+0000", 2024-09-02T10:37:43.425 INFO:teuthology.orchestra.run.smithi187.stdout: "duration": 3.4087000000000001e-05 2024-09-02T10:37:43.425 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.425 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.425 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.425 INFO:teuthology.orchestra.run.smithi187.stdout: } 2024-09-02T10:37:43.425 INFO:teuthology.orchestra.run.smithi187.stdout: ] 2024-09-02T10:37:43.425 INFO:teuthology.orchestra.run.smithi187.stdout:} 2024-09-02T10:37:43.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:43 smithi151 ceph-mon[16639]: pgmap v2284: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:43.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:43 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:37:43.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:43 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:37:43.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:43 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:37:43.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:43 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:37:43.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:43 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:37:43.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:43 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:37:43.426 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:43 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2175362096' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:43 smithi162 ceph-mon[20279]: pgmap v2284: 31 pgs: 31 active+clean; 577 KiB data, 8.4 GiB used, 975 GiB / 983 GiB avail 2024-09-02T10:37:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:43 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:37:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:43 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:37:43.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:43 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:37:43.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:43 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:37:43.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:43 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:37:43.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:43 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:37:43.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:43 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2175362096' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-09-02T10:37:44.113 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.0 to 1 (10000) 2024-09-02T10:37:44.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:44 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:37:44.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:44 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:37:44.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2175362096' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:37:44.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:44 smithi151 ceph-mon[16639]: osdmap e1004: 12 total, 12 up, 12 in 2024-09-02T10:37:44.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:44 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/942598012' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:37:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:44 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:37:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:44 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:37:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2175362096' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:37:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:44 smithi187 ceph-mon[20072]: osdmap e1004: 12 total, 12 up, 12 in 2024-09-02T10:37:44.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:44 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/942598012' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:37:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:44 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:37:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:44 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:37:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2175362096' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-02T10:37:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:44 smithi162 ceph-mon[20279]: osdmap e1004: 12 total, 12 up, 12 in 2024-09-02T10:37:44.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:44 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/942598012' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:37:44.772 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-02T10:37:45.014 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:45.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:45 smithi151 ceph-mon[16639]: pgmap v2286: 31 pgs: 31 active+clean; 590 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 450 B/s rd, 11 KiB/s wr, 1 op/s; 2.6 KiB/s, 0 objects/s recovering 2024-09-02T10:37:45.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:45 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/942598012' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:37:45.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:45 smithi151 ceph-mon[16639]: osdmap e1005: 12 total, 12 up, 12 in 2024-09-02T10:37:45.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:45 smithi151 ceph-mon[16639]: 1.b scrub starts 2024-09-02T10:37:45.389 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:45 smithi151 ceph-mon[16639]: 1.b scrub ok 2024-09-02T10:37:45.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:45 smithi151 ceph-mon[16639]: osdmap e1006: 12 total, 12 up, 12 in 2024-09-02T10:37:45.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:45 smithi151 ceph-mon[16639]: 1.1 scrub starts 2024-09-02T10:37:45.390 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:45 smithi151 ceph-mon[16639]: 1.1 scrub ok 2024-09-02T10:37:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:45 smithi187 ceph-mon[20072]: pgmap v2286: 31 pgs: 31 active+clean; 590 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 450 B/s rd, 11 KiB/s wr, 1 op/s; 2.6 KiB/s, 0 objects/s recovering 2024-09-02T10:37:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:45 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/942598012' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:37:45.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:45 smithi187 ceph-mon[20072]: osdmap e1005: 12 total, 12 up, 12 in 2024-09-02T10:37:45.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:45 smithi187 ceph-mon[20072]: 1.b scrub starts 2024-09-02T10:37:45.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:45 smithi187 ceph-mon[20072]: 1.b scrub ok 2024-09-02T10:37:45.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:45 smithi187 ceph-mon[20072]: osdmap e1006: 12 total, 12 up, 12 in 2024-09-02T10:37:45.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:45 smithi187 ceph-mon[20072]: 1.1 scrub starts 2024-09-02T10:37:45.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:45 smithi187 ceph-mon[20072]: 1.1 scrub ok 2024-09-02T10:37:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:45 smithi162 ceph-mon[20279]: pgmap v2286: 31 pgs: 31 active+clean; 590 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 450 B/s rd, 11 KiB/s wr, 1 op/s; 2.6 KiB/s, 0 objects/s recovering 2024-09-02T10:37:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:45 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/942598012' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:37:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:45 smithi162 ceph-mon[20279]: osdmap e1005: 12 total, 12 up, 12 in 2024-09-02T10:37:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:45 smithi162 ceph-mon[20279]: 1.b scrub starts 2024-09-02T10:37:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:45 smithi162 ceph-mon[20279]: 1.b scrub ok 2024-09-02T10:37:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:45 smithi162 ceph-mon[20279]: osdmap e1006: 12 total, 12 up, 12 in 2024-09-02T10:37:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:45 smithi162 ceph-mon[20279]: 1.1 scrub starts 2024-09-02T10:37:45.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:45 smithi162 ceph-mon[20279]: 1.1 scrub ok 2024-09-02T10:37:46.126 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-02T10:37:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:46 smithi151 ceph-mon[16639]: 1.1e scrub starts 2024-09-02T10:37:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:46 smithi151 ceph-mon[16639]: 1.1e scrub ok 2024-09-02T10:37:46.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:46 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2858678648' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:37:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:46 smithi187 ceph-mon[20072]: 1.1e scrub starts 2024-09-02T10:37:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:46 smithi187 ceph-mon[20072]: 1.1e scrub ok 2024-09-02T10:37:46.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:46 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2858678648' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:37:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:46 smithi162 ceph-mon[20279]: 1.1e scrub starts 2024-09-02T10:37:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:46 smithi162 ceph-mon[20279]: 1.1e scrub ok 2024-09-02T10:37:46.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:46 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2858678648' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 0, "weight": 1.0} : dispatch 2024-09-02T10:37:46.892 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 1 1 2024-09-02T10:37:47.138 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:47.422 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:47 smithi151 ceph-mon[16639]: pgmap v2289: 31 pgs: 31 active+clean; 590 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 392 B/s rd, 9.4 KiB/s wr, 0 op/s; 2.2 KiB/s, 0 objects/s recovering 2024-09-02T10:37:47.422 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:47 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2858678648' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:37:47.422 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:47 smithi151 ceph-mon[16639]: osdmap e1007: 12 total, 12 up, 12 in 2024-09-02T10:37:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:47 smithi187 ceph-mon[20072]: pgmap v2289: 31 pgs: 31 active+clean; 590 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 392 B/s rd, 9.4 KiB/s wr, 0 op/s; 2.2 KiB/s, 0 objects/s recovering 2024-09-02T10:37:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:47 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2858678648' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:37:47.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:47 smithi187 ceph-mon[20072]: osdmap e1007: 12 total, 12 up, 12 in 2024-09-02T10:37:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:47 smithi162 ceph-mon[20279]: pgmap v2289: 31 pgs: 31 active+clean; 590 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 392 B/s rd, 9.4 KiB/s wr, 0 op/s; 2.2 KiB/s, 0 objects/s recovering 2024-09-02T10:37:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:47 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2858678648' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-02T10:37:47.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:47 smithi162 ceph-mon[20279]: osdmap e1007: 12 total, 12 up, 12 in 2024-09-02T10:37:48.133 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.1 to 1 (10000) 2024-09-02T10:37:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:48 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:37:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:48 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:37:48.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4102961260' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:37:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:48 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:37:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:48 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:37:48.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4102961260' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:37:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:48 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:37:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:48 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:37:48.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4102961260' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:37:48.830 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-02T10:37:49.077 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:49 smithi151 ceph-mon[16639]: pgmap v2291: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:49 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4102961260' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:37:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:49 smithi151 ceph-mon[16639]: osdmap e1008: 12 total, 12 up, 12 in 2024-09-02T10:37:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:49 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:37:49.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:49 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:37:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:49 smithi187 ceph-mon[20072]: pgmap v2291: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:49 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4102961260' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:37:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:49 smithi187 ceph-mon[20072]: osdmap e1008: 12 total, 12 up, 12 in 2024-09-02T10:37:49.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:49 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:37:49.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:49 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:37:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:49 smithi162 ceph-mon[20279]: pgmap v2291: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:49 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4102961260' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:37:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:49 smithi162 ceph-mon[20279]: osdmap e1008: 12 total, 12 up, 12 in 2024-09-02T10:37:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:49 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:37:49.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:49 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:37:50.137 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-02T10:37:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:50 smithi151 ceph-mon[16639]: 1.17 scrub starts 2024-09-02T10:37:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:50 smithi151 ceph-mon[16639]: 1.17 scrub ok 2024-09-02T10:37:50.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3736020956' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:37:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:50 smithi187 ceph-mon[20072]: 1.17 scrub starts 2024-09-02T10:37:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:50 smithi187 ceph-mon[20072]: 1.17 scrub ok 2024-09-02T10:37:50.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3736020956' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:37:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:50 smithi162 ceph-mon[20279]: 1.17 scrub starts 2024-09-02T10:37:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:50 smithi162 ceph-mon[20279]: 1.17 scrub ok 2024-09-02T10:37:50.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3736020956' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 1, "weight": 1.0} : dispatch 2024-09-02T10:37:50.850 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 2 1 2024-09-02T10:37:51.096 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:51.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:51 smithi151 ceph-mon[16639]: pgmap v2293: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:51.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:51 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3736020956' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:37:51.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:51 smithi151 ceph-mon[16639]: osdmap e1009: 12 total, 12 up, 12 in 2024-09-02T10:37:51.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:51 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:37:51.414 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:51 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:37:51.415 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:51 smithi151 ceph-mon[16639]: osdmap e1010: 12 total, 12 up, 12 in 2024-09-02T10:37:51.508 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:51 smithi187 ceph-mon[20072]: pgmap v2293: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:51.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:51 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3736020956' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:37:51.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:51 smithi187 ceph-mon[20072]: osdmap e1009: 12 total, 12 up, 12 in 2024-09-02T10:37:51.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:51 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:37:51.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:51 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:37:51.509 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:51 smithi187 ceph-mon[20072]: osdmap e1010: 12 total, 12 up, 12 in 2024-09-02T10:37:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:51 smithi162 ceph-mon[20279]: pgmap v2293: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:51 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3736020956' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-02T10:37:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:51 smithi162 ceph-mon[20279]: osdmap e1009: 12 total, 12 up, 12 in 2024-09-02T10:37:51.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:51 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:37:51.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:51 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:37:51.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:51 smithi162 ceph-mon[20279]: osdmap e1010: 12 total, 12 up, 12 in 2024-09-02T10:37:52.154 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.2 to 1 (10000) 2024-09-02T10:37:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:52 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:37:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:52 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:37:52.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1760045418' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:37:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:52 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:37:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:52 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:37:52.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1760045418' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:37:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:52 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:37:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:52 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:37:52.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1760045418' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:37:52.871 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-02T10:37:53.124 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:53.422 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:53 smithi187 ceph-mon[20072]: pgmap v2296: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:53.422 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1760045418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:37:53.422 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:53 smithi187 ceph-mon[20072]: osdmap e1011: 12 total, 12 up, 12 in 2024-09-02T10:37:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:53 smithi151 ceph-mon[16639]: pgmap v2296: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1760045418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:37:53.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:53 smithi151 ceph-mon[16639]: osdmap e1011: 12 total, 12 up, 12 in 2024-09-02T10:37:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:53 smithi162 ceph-mon[20279]: pgmap v2296: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1760045418' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:37:53.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:53 smithi162 ceph-mon[20279]: osdmap e1011: 12 total, 12 up, 12 in 2024-09-02T10:37:54.173 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-02T10:37:54.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:54 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4173844073' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:37:54.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:54 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4173844073' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:37:54.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:54 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4173844073' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 2, "weight": 1.0} : dispatch 2024-09-02T10:37:54.932 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 3 1 2024-09-02T10:37:55.178 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:55.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:55 smithi151 ceph-mon[16639]: pgmap v2298: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:55.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:55 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4173844073' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:37:55.416 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:55 smithi151 ceph-mon[16639]: osdmap e1012: 12 total, 12 up, 12 in 2024-09-02T10:37:55.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:55 smithi187 ceph-mon[20072]: pgmap v2298: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:55.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:55 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4173844073' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:37:55.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:55 smithi187 ceph-mon[20072]: osdmap e1012: 12 total, 12 up, 12 in 2024-09-02T10:37:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:55 smithi162 ceph-mon[20279]: pgmap v2298: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:55 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4173844073' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-02T10:37:55.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:55 smithi162 ceph-mon[20279]: osdmap e1012: 12 total, 12 up, 12 in 2024-09-02T10:37:56.167 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.3 to 1 (10000) 2024-09-02T10:37:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:56 smithi151 ceph-mon[16639]: 1.18 scrub starts 2024-09-02T10:37:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:56 smithi151 ceph-mon[16639]: 1.18 scrub ok 2024-09-02T10:37:56.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2386789182' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:37:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:56 smithi187 ceph-mon[20072]: 1.18 scrub starts 2024-09-02T10:37:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:56 smithi187 ceph-mon[20072]: 1.18 scrub ok 2024-09-02T10:37:56.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2386789182' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:37:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:56 smithi162 ceph-mon[20279]: 1.18 scrub starts 2024-09-02T10:37:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:56 smithi162 ceph-mon[20279]: 1.18 scrub ok 2024-09-02T10:37:56.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2386789182' entity='client.admin' cmd={"prefix": "osd reweight", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:37:56.862 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-02T10:37:57.105 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:57 smithi151 ceph-mon[16639]: pgmap v2300: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2386789182' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:37:57.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:57 smithi151 ceph-mon[16639]: osdmap e1013: 12 total, 12 up, 12 in 2024-09-02T10:37:57.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:57 smithi162 ceph-mon[20279]: pgmap v2300: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:57.496 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2386789182' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:37:57.497 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:57 smithi162 ceph-mon[20279]: osdmap e1013: 12 total, 12 up, 12 in 2024-09-02T10:37:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:57 smithi187 ceph-mon[20072]: pgmap v2300: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2386789182' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:37:57.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:57 smithi187 ceph-mon[20072]: osdmap e1013: 12 total, 12 up, 12 in 2024-09-02T10:37:58.177 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-02T10:37:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:58 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:37:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:58 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:37:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:58 smithi151 ceph-mon[16639]: pgmap v2302: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:58.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:58 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/391532363' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:37:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:58 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:37:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:58 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:37:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:58 smithi187 ceph-mon[20072]: pgmap v2302: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:58.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:58 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/391532363' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:37:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:58 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:37:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:58 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:37:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:58 smithi162 ceph-mon[20279]: pgmap v2302: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:37:58.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:58 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/391532363' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 3, "weight": 1.0} : dispatch 2024-09-02T10:37:58.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 4 1 2024-09-02T10:37:59.158 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:37:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/391532363' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:37:59.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:37:59 smithi151 ceph-mon[16639]: osdmap e1014: 12 total, 12 up, 12 in 2024-09-02T10:37:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/391532363' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:37:59.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:37:59 smithi187 ceph-mon[20072]: osdmap e1014: 12 total, 12 up, 12 in 2024-09-02T10:37:59.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/391532363' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-02T10:37:59.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:37:59 smithi162 ceph-mon[20279]: osdmap e1014: 12 total, 12 up, 12 in 2024-09-02T10:38:00.428 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.4 to 1 (10000) 2024-09-02T10:38:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:00 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:00 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:00 smithi151 ceph-mon[16639]: osdmap e1015: 12 total, 12 up, 12 in 2024-09-02T10:38:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:00 smithi151 ceph-mon[16639]: osdmap e1016: 12 total, 12 up, 12 in 2024-09-02T10:38:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:00 smithi151 ceph-mon[16639]: pgmap v2306: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:00.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:00 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4201532032' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:38:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:00 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:00 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:00 smithi187 ceph-mon[20072]: osdmap e1015: 12 total, 12 up, 12 in 2024-09-02T10:38:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:00 smithi187 ceph-mon[20072]: osdmap e1016: 12 total, 12 up, 12 in 2024-09-02T10:38:00.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:00 smithi187 ceph-mon[20072]: pgmap v2306: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:00.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:00 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4201532032' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:38:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:00 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:00.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:00 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:00.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:00 smithi162 ceph-mon[20279]: osdmap e1015: 12 total, 12 up, 12 in 2024-09-02T10:38:00.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:00 smithi162 ceph-mon[20279]: osdmap e1016: 12 total, 12 up, 12 in 2024-09-02T10:38:00.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:00 smithi162 ceph-mon[20279]: pgmap v2306: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:00.596 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:00 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4201532032' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:38:01.067 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-02T10:38:01.317 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:01.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:01 smithi151 ceph-mon[16639]: 1.8 scrub starts 2024-09-02T10:38:01.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:01 smithi151 ceph-mon[16639]: 1.8 scrub ok 2024-09-02T10:38:01.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:01 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:01.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:01 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:01.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4201532032' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:38:01.352 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:01 smithi151 ceph-mon[16639]: osdmap e1017: 12 total, 12 up, 12 in 2024-09-02T10:38:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:01 smithi187 ceph-mon[20072]: 1.8 scrub starts 2024-09-02T10:38:01.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:01 smithi187 ceph-mon[20072]: 1.8 scrub ok 2024-09-02T10:38:01.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:01 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:01.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:01 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:01.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4201532032' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:38:01.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:01 smithi187 ceph-mon[20072]: osdmap e1017: 12 total, 12 up, 12 in 2024-09-02T10:38:01.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:01 smithi162 ceph-mon[20279]: 1.8 scrub starts 2024-09-02T10:38:01.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:01 smithi162 ceph-mon[20279]: 1.8 scrub ok 2024-09-02T10:38:01.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:01 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:01.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:01 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:01.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4201532032' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:38:01.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:01 smithi162 ceph-mon[20279]: osdmap e1017: 12 total, 12 up, 12 in 2024-09-02T10:38:02.194 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-02T10:38:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:02 smithi187 ceph-mon[20072]: pgmap v2308: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:02.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2708934875' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:38:02.566 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:02 smithi151 ceph-mon[16639]: pgmap v2308: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:02.566 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2708934875' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:38:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:02 smithi162 ceph-mon[20279]: pgmap v2308: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:02.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2708934875' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 4, "weight": 1.0} : dispatch 2024-09-02T10:38:02.896 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 5 1 2024-09-02T10:38:03.142 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:03 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2708934875' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:38:03.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:03 smithi187 ceph-mon[20072]: osdmap e1018: 12 total, 12 up, 12 in 2024-09-02T10:38:03.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:03 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2708934875' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:38:03.553 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:03 smithi151 ceph-mon[16639]: osdmap e1018: 12 total, 12 up, 12 in 2024-09-02T10:38:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:03 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2708934875' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-02T10:38:03.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:03 smithi162 ceph-mon[20279]: osdmap e1018: 12 total, 12 up, 12 in 2024-09-02T10:38:04.204 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.5 to 1 (10000) 2024-09-02T10:38:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:04 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:04 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:04 smithi187 ceph-mon[20072]: osdmap e1019: 12 total, 12 up, 12 in 2024-09-02T10:38:04.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:04 smithi187 ceph-mon[20072]: pgmap v2311: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:04.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:04 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1269120927' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:38:04.531 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:04 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:04.532 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:04 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:04.532 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:04 smithi151 ceph-mon[16639]: osdmap e1019: 12 total, 12 up, 12 in 2024-09-02T10:38:04.532 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:04 smithi151 ceph-mon[16639]: pgmap v2311: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:04.532 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:04 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1269120927' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:38:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:04 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:04 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:04 smithi162 ceph-mon[20279]: osdmap e1019: 12 total, 12 up, 12 in 2024-09-02T10:38:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:04 smithi162 ceph-mon[20279]: pgmap v2311: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:04.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:04 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1269120927' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:38:04.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-02T10:38:05.109 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:05.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:05 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1269120927' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:38:05.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:05 smithi187 ceph-mon[20072]: osdmap e1020: 12 total, 12 up, 12 in 2024-09-02T10:38:05.519 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:05 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1269120927' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:38:05.519 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:05 smithi151 ceph-mon[16639]: osdmap e1020: 12 total, 12 up, 12 in 2024-09-02T10:38:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:05 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1269120927' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:38:05.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:05 smithi162 ceph-mon[20279]: osdmap e1020: 12 total, 12 up, 12 in 2024-09-02T10:38:06.221 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-02T10:38:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:06 smithi187 ceph-mon[20072]: pgmap v2313: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:06.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:06 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1651531417' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:38:06.531 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:06 smithi151 ceph-mon[16639]: pgmap v2313: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:06.531 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:06 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1651531417' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:38:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:06 smithi162 ceph-mon[20279]: pgmap v2313: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:06.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:06 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1651531417' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 5, "weight": 1.0} : dispatch 2024-09-02T10:38:06.975 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 6 1 2024-09-02T10:38:07.228 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:07.492 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:07 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1651531417' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:38:07.492 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:07 smithi151 ceph-mon[16639]: osdmap e1021: 12 total, 12 up, 12 in 2024-09-02T10:38:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:07 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1651531417' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:38:07.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:07 smithi187 ceph-mon[20072]: osdmap e1021: 12 total, 12 up, 12 in 2024-09-02T10:38:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:07 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1651531417' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-02T10:38:07.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:07 smithi162 ceph-mon[20279]: osdmap e1021: 12 total, 12 up, 12 in 2024-09-02T10:38:08.229 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.6 to 1 (10000) 2024-09-02T10:38:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:08 smithi187 ceph-mon[20072]: pgmap v2315: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:08.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/893631158' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:38:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:08 smithi162 ceph-mon[20279]: pgmap v2315: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:08.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/893631158' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:38:08.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:08 smithi151 ceph-mon[16639]: pgmap v2315: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:08.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/893631158' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:38:09.063 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-02T10:38:09.314 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:09.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:09 smithi151 ceph-mon[16639]: 1.4 scrub starts 2024-09-02T10:38:09.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:09 smithi151 ceph-mon[16639]: 1.4 scrub ok 2024-09-02T10:38:09.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:09 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/893631158' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:38:09.436 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:09 smithi151 ceph-mon[16639]: osdmap e1022: 12 total, 12 up, 12 in 2024-09-02T10:38:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:09 smithi187 ceph-mon[20072]: 1.4 scrub starts 2024-09-02T10:38:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:09 smithi187 ceph-mon[20072]: 1.4 scrub ok 2024-09-02T10:38:09.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:09 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/893631158' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:38:09.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:09 smithi187 ceph-mon[20072]: osdmap e1022: 12 total, 12 up, 12 in 2024-09-02T10:38:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:09 smithi162 ceph-mon[20279]: 1.4 scrub starts 2024-09-02T10:38:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:09 smithi162 ceph-mon[20279]: 1.4 scrub ok 2024-09-02T10:38:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:09 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/893631158' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:38:09.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:09 smithi162 ceph-mon[20279]: osdmap e1022: 12 total, 12 up, 12 in 2024-09-02T10:38:10.242 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-02T10:38:10.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:10 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:10.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:10 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:10.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:10 smithi187 ceph-mon[20072]: pgmap v2317: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:10.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:10 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3199211065' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:38:10.538 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:10 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:10.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:10 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:10.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:10 smithi151 ceph-mon[16639]: pgmap v2317: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:10.539 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:10 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3199211065' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:38:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:10 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:10 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:10 smithi162 ceph-mon[20279]: pgmap v2317: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:10.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:10 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3199211065' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 6, "weight": 1.0} : dispatch 2024-09-02T10:38:10.874 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 7 1 2024-09-02T10:38:11.129 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:11 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:11 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:11 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3199211065' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:38:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:11 smithi187 ceph-mon[20072]: osdmap e1023: 12 total, 12 up, 12 in 2024-09-02T10:38:11.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:11 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:38:11.542 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:11 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:11.542 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:11 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:11.542 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:11 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3199211065' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:38:11.543 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:11 smithi151 ceph-mon[16639]: osdmap e1023: 12 total, 12 up, 12 in 2024-09-02T10:38:11.543 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:11 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:38:11.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:11 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:11.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:11 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:11.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:11 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3199211065' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-02T10:38:11.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:11 smithi162 ceph-mon[20279]: osdmap e1023: 12 total, 12 up, 12 in 2024-09-02T10:38:11.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:11 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:38:12.245 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.7 to 1 (10000) 2024-09-02T10:38:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:12 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:12 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:38:12.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:38:12.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:12 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:38:12.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:12 smithi187 ceph-mon[20072]: pgmap v2319: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:12.507 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:12 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4008218061' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:38:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:12 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:12 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:38:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:38:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:12 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:38:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:12 smithi162 ceph-mon[20279]: pgmap v2319: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:12.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:12 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4008218061' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:38:12.596 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:12 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:12.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:12 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:12.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:38:12.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:38:12.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:12 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:38:12.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:12 smithi151 ceph-mon[16639]: pgmap v2319: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:12.597 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:12 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4008218061' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:38:12.926 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-02T10:38:13.177 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:13 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4008218061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:38:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:13 smithi187 ceph-mon[20072]: osdmap e1024: 12 total, 12 up, 12 in 2024-09-02T10:38:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:13 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:13.506 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:13 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:13.542 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:13 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4008218061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:38:13.542 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:13 smithi151 ceph-mon[16639]: osdmap e1024: 12 total, 12 up, 12 in 2024-09-02T10:38:13.542 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:13 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:13.542 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:13 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:13.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:13 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4008218061' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:38:13.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:13 smithi162 ceph-mon[20279]: osdmap e1024: 12 total, 12 up, 12 in 2024-09-02T10:38:13.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:13 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:13.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:13 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:14.266 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-02T10:38:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:14 smithi162 ceph-mon[20279]: osdmap e1025: 12 total, 12 up, 12 in 2024-09-02T10:38:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:14 smithi162 ceph-mon[20279]: pgmap v2322: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:14.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:14 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/265936952' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:38:14.654 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:14 smithi151 ceph-mon[16639]: osdmap e1025: 12 total, 12 up, 12 in 2024-09-02T10:38:14.654 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:14 smithi151 ceph-mon[16639]: pgmap v2322: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:14.654 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:14 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/265936952' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:38:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:14 smithi187 ceph-mon[20072]: osdmap e1025: 12 total, 12 up, 12 in 2024-09-02T10:38:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:14 smithi187 ceph-mon[20072]: pgmap v2322: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:14.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:14 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/265936952' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 7, "weight": 1.0} : dispatch 2024-09-02T10:38:14.966 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 8 1 2024-09-02T10:38:15.211 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:15.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:15 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/265936952' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:38:15.525 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:15 smithi151 ceph-mon[16639]: osdmap e1026: 12 total, 12 up, 12 in 2024-09-02T10:38:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:15 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/265936952' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:38:15.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:15 smithi162 ceph-mon[20279]: osdmap e1026: 12 total, 12 up, 12 in 2024-09-02T10:38:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:15 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/265936952' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-02T10:38:15.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:15 smithi187 ceph-mon[20072]: osdmap e1026: 12 total, 12 up, 12 in 2024-09-02T10:38:16.272 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.8 to 1 (10000) 2024-09-02T10:38:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:16 smithi162 ceph-mon[20279]: 1.15 scrub starts 2024-09-02T10:38:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:16 smithi162 ceph-mon[20279]: 1.15 scrub ok 2024-09-02T10:38:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:16 smithi162 ceph-mon[20279]: pgmap v2324: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:16.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:16 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/674381969' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:38:16.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:16 smithi151 ceph-mon[16639]: 1.15 scrub starts 2024-09-02T10:38:16.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:16 smithi151 ceph-mon[16639]: 1.15 scrub ok 2024-09-02T10:38:16.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:16 smithi151 ceph-mon[16639]: pgmap v2324: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:16.645 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:16 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/674381969' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:38:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:16 smithi187 ceph-mon[20072]: 1.15 scrub starts 2024-09-02T10:38:16.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:16 smithi187 ceph-mon[20072]: 1.15 scrub ok 2024-09-02T10:38:16.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:16 smithi187 ceph-mon[20072]: pgmap v2324: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:16.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:16 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/674381969' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:38:17.023 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 8 1 2024-09-02T10:38:17.271 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:17.529 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:17 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/674381969' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:38:17.529 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:17 smithi151 ceph-mon[16639]: osdmap e1027: 12 total, 12 up, 12 in 2024-09-02T10:38:17.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:17 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/674381969' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:38:17.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:17 smithi162 ceph-mon[20279]: osdmap e1027: 12 total, 12 up, 12 in 2024-09-02T10:38:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:17 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/674381969' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:38:17.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:17 smithi187 ceph-mon[20072]: osdmap e1027: 12 total, 12 up, 12 in 2024-09-02T10:38:18.283 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-02T10:38:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:18 smithi162 ceph-mon[20279]: pgmap v2326: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:18.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:18 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/613500839' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:38:18.610 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:18 smithi151 ceph-mon[16639]: pgmap v2326: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:18.611 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:18 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/613500839' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:38:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:18 smithi187 ceph-mon[20072]: pgmap v2326: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:18.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:18 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/613500839' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 8, "weight": 1.0} : dispatch 2024-09-02T10:38:19.001 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 9 1 2024-09-02T10:38:19.245 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:19.536 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:19 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/613500839' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:38:19.536 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:19 smithi151 ceph-mon[16639]: osdmap e1028: 12 total, 12 up, 12 in 2024-09-02T10:38:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:19 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/613500839' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:38:19.595 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:19 smithi162 ceph-mon[20279]: osdmap e1028: 12 total, 12 up, 12 in 2024-09-02T10:38:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:19 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/613500839' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 8, "weight": 1.0}]': finished 2024-09-02T10:38:19.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:19 smithi187 ceph-mon[20072]: osdmap e1028: 12 total, 12 up, 12 in 2024-09-02T10:38:20.432 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.9 to 1 (10000) 2024-09-02T10:38:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:20 smithi151 ceph-mon[16639]: pgmap v2328: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:20.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:20 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2736913960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:38:20.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:20 smithi187 ceph-mon[20072]: pgmap v2328: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:20.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:20 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2736913960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:38:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:20 smithi162 ceph-mon[20279]: pgmap v2328: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:20.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:20 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2736913960' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:38:21.189 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 9 1 2024-09-02T10:38:21.440 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:21 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2736913960' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:38:21.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:21 smithi151 ceph-mon[16639]: osdmap e1029: 12 total, 12 up, 12 in 2024-09-02T10:38:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:21 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2736913960' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:38:21.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:21 smithi187 ceph-mon[20072]: osdmap e1029: 12 total, 12 up, 12 in 2024-09-02T10:38:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:21 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2736913960' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:38:21.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:21 smithi162 ceph-mon[20279]: osdmap e1029: 12 total, 12 up, 12 in 2024-09-02T10:38:22.443 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-02T10:38:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:22 smithi151 ceph-mon[16639]: pgmap v2330: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:22.688 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:22 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2677314937' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:38:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:22 smithi187 ceph-mon[20072]: pgmap v2330: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:22.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:22 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2677314937' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:38:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:22 smithi162 ceph-mon[20279]: pgmap v2330: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:22.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:22 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2677314937' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 9, "weight": 1.0} : dispatch 2024-09-02T10:38:23.163 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 10 1 2024-09-02T10:38:23.406 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:23.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:23 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2677314937' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:38:23.695 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:23 smithi151 ceph-mon[16639]: osdmap e1030: 12 total, 12 up, 12 in 2024-09-02T10:38:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:23 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2677314937' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:38:23.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:23 smithi187 ceph-mon[20072]: osdmap e1030: 12 total, 12 up, 12 in 2024-09-02T10:38:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:23 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2677314937' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 9, "weight": 1.0}]': finished 2024-09-02T10:38:23.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:23 smithi162 ceph-mon[20279]: osdmap e1030: 12 total, 12 up, 12 in 2024-09-02T10:38:24.450 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.10 to 1 (10000) 2024-09-02T10:38:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:24 smithi187 ceph-mon[20072]: pgmap v2332: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:24.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:24 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2534651273' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:38:24.777 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:24 smithi151 ceph-mon[16639]: pgmap v2332: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:24.777 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:24 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2534651273' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:38:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:24 smithi162 ceph-mon[20279]: pgmap v2332: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:24.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:24 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2534651273' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:38:25.116 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 10 1 2024-09-02T10:38:25.361 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:25.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:25 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2534651273' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:38:25.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:25 smithi187 ceph-mon[20072]: osdmap e1031: 12 total, 12 up, 12 in 2024-09-02T10:38:25.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:25 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2534651273' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:38:25.805 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:25 smithi151 ceph-mon[16639]: osdmap e1031: 12 total, 12 up, 12 in 2024-09-02T10:38:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:25 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2534651273' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:38:25.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:25 smithi162 ceph-mon[20279]: osdmap e1031: 12 total, 12 up, 12 in 2024-09-02T10:38:26.459 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-02T10:38:26.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:26 smithi187 ceph-mon[20072]: pgmap v2334: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:26.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:26 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:26.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:26 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:26.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:26 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/198104639' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:38:26.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:26 smithi151 ceph-mon[16639]: pgmap v2334: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:26.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:26 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:26.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:26 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:26.776 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:26 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/198104639' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:38:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:26 smithi162 ceph-mon[20279]: pgmap v2334: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:26 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:26 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:26.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:26 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/198104639' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 10, "weight": 1.0} : dispatch 2024-09-02T10:38:27.202 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd reweight 11 1 2024-09-02T10:38:27.448 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:27.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:27 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/198104639' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:38:27.748 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:27 smithi151 ceph-mon[16639]: osdmap e1032: 12 total, 12 up, 12 in 2024-09-02T10:38:27.749 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:27 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/198104639' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:38:27.749 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:27 smithi162 ceph-mon[20279]: osdmap e1032: 12 total, 12 up, 12 in 2024-09-02T10:38:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:27 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/198104639' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 10, "weight": 1.0}]': finished 2024-09-02T10:38:27.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:27 smithi187 ceph-mon[20072]: osdmap e1032: 12 total, 12 up, 12 in 2024-09-02T10:38:28.462 INFO:teuthology.orchestra.run.smithi151.stderr:reweighted osd.11 to 1 (10000) 2024-09-02T10:38:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:28 smithi187 ceph-mon[20072]: pgmap v2336: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:28.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:28 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1165536568' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:38:28.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:28 smithi151 ceph-mon[16639]: pgmap v2336: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:28.767 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:28 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1165536568' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:38:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:28 smithi162 ceph-mon[20279]: pgmap v2336: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:28.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:28 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1165536568' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:38:29.090 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd primary-affinity 11 1 2024-09-02T10:38:29.334 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:29.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:29 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1165536568' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:38:29.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:29 smithi151 ceph-mon[16639]: osdmap e1033: 12 total, 12 up, 12 in 2024-09-02T10:38:29.755 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:29 smithi151 ceph-mon[16639]: osdmap e1034: 12 total, 12 up, 12 in 2024-09-02T10:38:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:29 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1165536568' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:38:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:29 smithi187 ceph-mon[20072]: osdmap e1033: 12 total, 12 up, 12 in 2024-09-02T10:38:29.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:29 smithi187 ceph-mon[20072]: osdmap e1034: 12 total, 12 up, 12 in 2024-09-02T10:38:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:29 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1165536568' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:38:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:29 smithi162 ceph-mon[20279]: osdmap e1033: 12 total, 12 up, 12 in 2024-09-02T10:38:29.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:29 smithi162 ceph-mon[20279]: osdmap e1034: 12 total, 12 up, 12 in 2024-09-02T10:38:30.474 INFO:teuthology.orchestra.run.smithi151.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-02T10:38:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:30 smithi187 ceph-mon[20072]: pgmap v2339: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:30.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:30 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3011421744' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:38:30.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:30 smithi151 ceph-mon[16639]: pgmap v2339: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:30.775 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:30 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3011421744' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:38:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:30 smithi162 ceph-mon[20279]: pgmap v2339: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:30.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:30 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3011421744' entity='client.admin' cmd={"prefix": "osd primary-affinity", "id": 11, "weight": 1.0} : dispatch 2024-09-02T10:38:31.109 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-02T10:38:31.109 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-02T10:38:31.109 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-02T10:38:31.109 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-02T10:38:31.109 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-02T10:38:31.110 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-09-02T10:38:31.355 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:31 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3011421744' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:38:31.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:31 smithi187 ceph-mon[20072]: osdmap e1035: 12 total, 12 up, 12 in 2024-09-02T10:38:31.813 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:31 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3011421744' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:38:31.813 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:31 smithi151 ceph-mon[16639]: osdmap e1035: 12 total, 12 up, 12 in 2024-09-02T10:38:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:31 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3011421744' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 11, "weight": 1.0}]': finished 2024-09-02T10:38:31.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:31 smithi162 ceph-mon[20279]: osdmap e1035: 12 total, 12 up, 12 in 2024-09-02T10:38:32.083 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:38:32.083 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1035,"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","created":"2024-09-02T09:53:53.308187+0000","modified":"2024-09-02T10:38:30.467741+0000","last_up_change":"2024-09-02T10:00:25.816021+0000","last_in_change":"2024-09-02T10:37:39.090092+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-02T09:56:53.284814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"241","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e76499ad-5f7e-4788-aa85-0bad0d843e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":961,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6803","nonce":1059446772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6805","nonce":1059446772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6809","nonce":1059446772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6807","nonce":1059446772}]},"public_addr":"172.21.15.151:6803/1059446772","cluster_addr":"172.21.15.151:6805/1059446772","heartbeat_back_addr":"172.21.15.151:6809/1059446772","heartbeat_front_addr":"172.21.15.151:6807/1059446772","state":["exists","up"]},{"osd":1,"uuid":"8949ba9c-4a71-4327-a1dc-e7d25286d8f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1033,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6811","nonce":457777866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6813","nonce":457777866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6817","nonce":457777866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6815","nonce":457777866}]},"public_addr":"172.21.15.151:6811/457777866","cluster_addr":"172.21.15.151:6813/457777866","heartbeat_back_addr":"172.21.15.151:6817/457777866","heartbeat_front_addr":"172.21.15.151:6815/457777866","state":["exists","up"]},{"osd":2,"uuid":"2dead774-0f59-4e90-8503-9a3f0fb68b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":961,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6819","nonce":654044340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6821","nonce":654044340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6825","nonce":654044340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6823","nonce":654044340}]},"public_addr":"172.21.15.151:6819/654044340","cluster_addr":"172.21.15.151:6821/654044340","heartbeat_back_addr":"172.21.15.151:6825/654044340","heartbeat_front_addr":"172.21.15.151:6823/654044340","state":["exists","up"]},{"osd":3,"uuid":"524f32ed-95aa-4459-9585-72577db12107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":994,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6827","nonce":1912547696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6829","nonce":1912547696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6833","nonce":1912547696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6831","nonce":1912547696}]},"public_addr":"172.21.15.151:6827/1912547696","cluster_addr":"172.21.15.151:6829/1912547696","heartbeat_back_addr":"172.21.15.151:6833/1912547696","heartbeat_front_addr":"172.21.15.151:6831/1912547696","state":["exists","up"]},{"osd":4,"uuid":"2c43a79f-8ee6-4088-af19-6144fc8bdbe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":958,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6801","nonce":295415389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6803","nonce":295415389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6807","nonce":295415389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6805","nonce":295415389}]},"public_addr":"172.21.15.162:6801/295415389","cluster_addr":"172.21.15.162:6803/295415389","heartbeat_back_addr":"172.21.15.162:6807/295415389","heartbeat_front_addr":"172.21.15.162:6805/295415389","state":["exists","up"]},{"osd":5,"uuid":"71810aab-c54f-4048-9fe4-d1272302d53a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":983,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6809","nonce":3782691981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6811","nonce":3782691981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6815","nonce":3782691981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6813","nonce":3782691981}]},"public_addr":"172.21.15.162:6809/3782691981","cluster_addr":"172.21.15.162:6811/3782691981","heartbeat_back_addr":"172.21.15.162:6815/3782691981","heartbeat_front_addr":"172.21.15.162:6813/3782691981","state":["exists","up"]},{"osd":6,"uuid":"45f93bd7-8e30-43ee-b57a-c007aec6d308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":975,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6817","nonce":759599342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6819","nonce":759599342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6823","nonce":759599342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6821","nonce":759599342}]},"public_addr":"172.21.15.162:6817/759599342","cluster_addr":"172.21.15.162:6819/759599342","heartbeat_back_addr":"172.21.15.162:6823/759599342","heartbeat_front_addr":"172.21.15.162:6821/759599342","state":["exists","up"]},{"osd":7,"uuid":"e9ab0b99-0451-4ebd-8758-cbde6350c764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":1033,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6825","nonce":2389923814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6827","nonce":2389923814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6831","nonce":2389923814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6829","nonce":2389923814}]},"public_addr":"172.21.15.162:6825/2389923814","cluster_addr":"172.21.15.162:6827/2389923814","heartbeat_back_addr":"172.21.15.162:6831/2389923814","heartbeat_front_addr":"172.21.15.162:6829/2389923814","state":["exists","up"]},{"osd":8,"uuid":"b0cf652b-4d1b-441a-85a8-fd5c16380634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":985,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6801","nonce":3014427013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6803","nonce":3014427013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6807","nonce":3014427013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6805","nonce":3014427013}]},"public_addr":"172.21.15.187:6801/3014427013","cluster_addr":"172.21.15.187:6803/3014427013","heartbeat_back_addr":"172.21.15.187:6807/3014427013","heartbeat_front_addr":"172.21.15.187:6805/3014427013","state":["exists","up"]},{"osd":9,"uuid":"1752a57c-aab9-4f0d-b2fd-629489170a0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":961,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6809","nonce":108453727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6811","nonce":108453727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6815","nonce":108453727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6813","nonce":108453727}]},"public_addr":"172.21.15.187:6809/108453727","cluster_addr":"172.21.15.187:6811/108453727","heartbeat_back_addr":"172.21.15.187:6815/108453727","heartbeat_front_addr":"172.21.15.187:6813/108453727","state":["exists","up"]},{"osd":10,"uuid":"6f48d49d-6636-4142-8cf1-04f371f6317a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":984,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6817","nonce":401612597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6819","nonce":401612597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6823","nonce":401612597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6821","nonce":401612597}]},"public_addr":"172.21.15.187:6817/401612597","cluster_addr":"172.21.15.187:6819/401612597","heartbeat_back_addr":"172.21.15.187:6823/401612597","heartbeat_front_addr":"172.21.15.187:6821/401612597","state":["exists","up"]},{"osd":11,"uuid":"f545e120-27c3-4fc6-883f-7c588d0eb8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":1033,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6825","nonce":3639916212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6827","nonce":3639916212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6831","nonce":3639916212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6829","nonce":3639916212}]},"public_addr":"172.21.15.187:6825/3639916212","cluster_addr":"172.21.15.187:6827/3639916212","heartbeat_back_addr":"172.21.15.187:6831/3639916212","heartbeat_front_addr":"172.21.15.187:6829/3639916212","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:12.336258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:37:36.012041+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:37:59.070522+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:02.507777+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:25.685911+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:37:30.435519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:10.117665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:00.244886+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:37:50.292876+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:09.295070+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:37:29.803129+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:11.510847+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.e","osds":[4,9,2]},{"pgid":"1.13","osds":[4,1,2]},{"pgid":"1.1e","osds":[11,10,9]}],"pg_upmap_items":[{"pgid":"1.c","mappings":[{"from":6,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/1566053508":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/1701887065":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/2339434463":"2024-09-03T09:54:15.686961+0000","172.21.15.151:6801/609062416":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/2251539148":"2024-09-03T09:54:15.686961+0000","172.21.15.151:0/394946850":"2024-09-03T09:54:38.970754+0000","172.21.15.151:6800/609062416":"2024-09-03T09:54:38.970754+0000","172.21.15.151:6801/3972807807":"2024-09-03T09:54:15.686961+0000","172.21.15.151:6800/3972807807":"2024-09-03T09:54:15.686961+0000","172.21.15.151:0/3380605706":"2024-09-03T09:54:15.686961+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-02T10:38:32.717 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:32 smithi151 ceph-mon[16639]: 1.0 scrub starts 2024-09-02T10:38:32.717 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:32 smithi151 ceph-mon[16639]: 1.0 scrub ok 2024-09-02T10:38:32.717 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:32 smithi151 ceph-mon[16639]: pgmap v2341: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:32.718 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:32 smithi151 ceph-mon[16639]: 1.7 scrub starts 2024-09-02T10:38:32.718 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:32 smithi151 ceph-mon[16639]: 1.7 scrub ok 2024-09-02T10:38:32.718 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:32 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3959190279' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:38:32.745 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-02T10:38:32.745 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd dump --format=json 2024-09-02T10:38:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:32 smithi187 ceph-mon[20072]: 1.0 scrub starts 2024-09-02T10:38:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:32 smithi187 ceph-mon[20072]: 1.0 scrub ok 2024-09-02T10:38:32.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:32 smithi187 ceph-mon[20072]: pgmap v2341: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:32.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:32 smithi187 ceph-mon[20072]: 1.7 scrub starts 2024-09-02T10:38:32.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:32 smithi187 ceph-mon[20072]: 1.7 scrub ok 2024-09-02T10:38:32.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:32 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3959190279' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:38:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:32 smithi162 ceph-mon[20279]: 1.0 scrub starts 2024-09-02T10:38:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:32 smithi162 ceph-mon[20279]: 1.0 scrub ok 2024-09-02T10:38:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:32 smithi162 ceph-mon[20279]: pgmap v2341: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:32 smithi162 ceph-mon[20279]: 1.7 scrub starts 2024-09-02T10:38:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:32 smithi162 ceph-mon[20279]: 1.7 scrub ok 2024-09-02T10:38:32.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:32 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3959190279' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:38:32.991 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:33.676 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:38:33.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:33 smithi151 ceph-mon[16639]: 1.14 scrub starts 2024-09-02T10:38:33.681 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:33 smithi151 ceph-mon[16639]: 1.14 scrub ok 2024-09-02T10:38:33.681 INFO:teuthology.orchestra.run.smithi151.stdout:{"epoch":1035,"fsid":"1530c7be-6911-11ef-bcd4-c7b262605968","created":"2024-09-02T09:53:53.308187+0000","modified":"2024-09-02T10:38:30.467741+0000","last_up_change":"2024-09-02T10:00:25.816021+0000","last_in_change":"2024-09-02T10:37:39.090092+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-02T09:56:53.284814+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":31,"pg_placement_num":31,"pg_placement_num_target":31,"pg_num_target":31,"pg_num_pending":31,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"264","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"241","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.940000057220459,"score_stable":1.940000057220459,"optimal_score":1,"raw_score_acting":1.940000057220459,"raw_score_stable":1.940000057220459,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e76499ad-5f7e-4788-aa85-0bad0d843e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":961,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6802","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6803","nonce":1059446772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6804","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6805","nonce":1059446772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6808","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6809","nonce":1059446772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6806","nonce":1059446772},{"type":"v1","addr":"172.21.15.151:6807","nonce":1059446772}]},"public_addr":"172.21.15.151:6803/1059446772","cluster_addr":"172.21.15.151:6805/1059446772","heartbeat_back_addr":"172.21.15.151:6809/1059446772","heartbeat_front_addr":"172.21.15.151:6807/1059446772","state":["exists","up"]},{"osd":1,"uuid":"8949ba9c-4a71-4327-a1dc-e7d25286d8f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1033,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6810","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6811","nonce":457777866}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6812","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6813","nonce":457777866}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6816","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6817","nonce":457777866}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6814","nonce":457777866},{"type":"v1","addr":"172.21.15.151:6815","nonce":457777866}]},"public_addr":"172.21.15.151:6811/457777866","cluster_addr":"172.21.15.151:6813/457777866","heartbeat_back_addr":"172.21.15.151:6817/457777866","heartbeat_front_addr":"172.21.15.151:6815/457777866","state":["exists","up"]},{"osd":2,"uuid":"2dead774-0f59-4e90-8503-9a3f0fb68b48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":961,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6818","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6819","nonce":654044340}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6820","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6821","nonce":654044340}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6824","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6825","nonce":654044340}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6822","nonce":654044340},{"type":"v1","addr":"172.21.15.151:6823","nonce":654044340}]},"public_addr":"172.21.15.151:6819/654044340","cluster_addr":"172.21.15.151:6821/654044340","heartbeat_back_addr":"172.21.15.151:6825/654044340","heartbeat_front_addr":"172.21.15.151:6823/654044340","state":["exists","up"]},{"osd":3,"uuid":"524f32ed-95aa-4459-9585-72577db12107","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":994,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6826","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6827","nonce":1912547696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6828","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6829","nonce":1912547696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6832","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6833","nonce":1912547696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.151:6830","nonce":1912547696},{"type":"v1","addr":"172.21.15.151:6831","nonce":1912547696}]},"public_addr":"172.21.15.151:6827/1912547696","cluster_addr":"172.21.15.151:6829/1912547696","heartbeat_back_addr":"172.21.15.151:6833/1912547696","heartbeat_front_addr":"172.21.15.151:6831/1912547696","state":["exists","up"]},{"osd":4,"uuid":"2c43a79f-8ee6-4088-af19-6144fc8bdbe4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":958,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6800","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6801","nonce":295415389}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6802","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6803","nonce":295415389}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6806","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6807","nonce":295415389}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6804","nonce":295415389},{"type":"v1","addr":"172.21.15.162:6805","nonce":295415389}]},"public_addr":"172.21.15.162:6801/295415389","cluster_addr":"172.21.15.162:6803/295415389","heartbeat_back_addr":"172.21.15.162:6807/295415389","heartbeat_front_addr":"172.21.15.162:6805/295415389","state":["exists","up"]},{"osd":5,"uuid":"71810aab-c54f-4048-9fe4-d1272302d53a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":983,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6808","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6809","nonce":3782691981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6810","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6811","nonce":3782691981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6814","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6815","nonce":3782691981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6812","nonce":3782691981},{"type":"v1","addr":"172.21.15.162:6813","nonce":3782691981}]},"public_addr":"172.21.15.162:6809/3782691981","cluster_addr":"172.21.15.162:6811/3782691981","heartbeat_back_addr":"172.21.15.162:6815/3782691981","heartbeat_front_addr":"172.21.15.162:6813/3782691981","state":["exists","up"]},{"osd":6,"uuid":"45f93bd7-8e30-43ee-b57a-c007aec6d308","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":975,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6816","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6817","nonce":759599342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6818","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6819","nonce":759599342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6822","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6823","nonce":759599342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6820","nonce":759599342},{"type":"v1","addr":"172.21.15.162:6821","nonce":759599342}]},"public_addr":"172.21.15.162:6817/759599342","cluster_addr":"172.21.15.162:6819/759599342","heartbeat_back_addr":"172.21.15.162:6823/759599342","heartbeat_front_addr":"172.21.15.162:6821/759599342","state":["exists","up"]},{"osd":7,"uuid":"e9ab0b99-0451-4ebd-8758-cbde6350c764","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":1033,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6824","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6825","nonce":2389923814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6826","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6827","nonce":2389923814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6830","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6831","nonce":2389923814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.162:6828","nonce":2389923814},{"type":"v1","addr":"172.21.15.162:6829","nonce":2389923814}]},"public_addr":"172.21.15.162:6825/2389923814","cluster_addr":"172.21.15.162:6827/2389923814","heartbeat_back_addr":"172.21.15.162:6831/2389923814","heartbeat_front_addr":"172.21.15.162:6829/2389923814","state":["exists","up"]},{"osd":8,"uuid":"b0cf652b-4d1b-441a-85a8-fd5c16380634","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":985,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6801","nonce":3014427013}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6803","nonce":3014427013}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6807","nonce":3014427013}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3014427013},{"type":"v1","addr":"172.21.15.187:6805","nonce":3014427013}]},"public_addr":"172.21.15.187:6801/3014427013","cluster_addr":"172.21.15.187:6803/3014427013","heartbeat_back_addr":"172.21.15.187:6807/3014427013","heartbeat_front_addr":"172.21.15.187:6805/3014427013","state":["exists","up"]},{"osd":9,"uuid":"1752a57c-aab9-4f0d-b2fd-629489170a0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":66,"up_thru":961,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6809","nonce":108453727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6811","nonce":108453727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6815","nonce":108453727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":108453727},{"type":"v1","addr":"172.21.15.187:6813","nonce":108453727}]},"public_addr":"172.21.15.187:6809/108453727","cluster_addr":"172.21.15.187:6811/108453727","heartbeat_back_addr":"172.21.15.187:6815/108453727","heartbeat_front_addr":"172.21.15.187:6813/108453727","state":["exists","up"]},{"osd":10,"uuid":"6f48d49d-6636-4142-8cf1-04f371f6317a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":984,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6817","nonce":401612597}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6819","nonce":401612597}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6823","nonce":401612597}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":401612597},{"type":"v1","addr":"172.21.15.187:6821","nonce":401612597}]},"public_addr":"172.21.15.187:6817/401612597","cluster_addr":"172.21.15.187:6819/401612597","heartbeat_back_addr":"172.21.15.187:6823/401612597","heartbeat_front_addr":"172.21.15.187:6821/401612597","state":["exists","up"]},{"osd":11,"uuid":"f545e120-27c3-4fc6-883f-7c588d0eb8d2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":1033,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6825","nonce":3639916212}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6827","nonce":3639916212}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6831","nonce":3639916212}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":3639916212},{"type":"v1","addr":"172.21.15.187:6829","nonce":3639916212}]},"public_addr":"172.21.15.187:6825/3639916212","cluster_addr":"172.21.15.187:6827/3639916212","heartbeat_back_addr":"172.21.15.187:6831/3639916212","heartbeat_front_addr":"172.21.15.187:6829/3639916212","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:12.336258+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:37:36.012041+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:37:59.070522+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:02.507777+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:25.685911+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:37:30.435519+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:10.117665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:00.244886+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:37:50.292876+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:09.295070+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:37:29.803129+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-02T10:38:11.510847+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.e","osds":[4,9,2]},{"pgid":"1.13","osds":[4,1,2]},{"pgid":"1.1e","osds":[11,10,9]}],"pg_upmap_items":[{"pgid":"1.c","mappings":[{"from":6,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.151:0/1566053508":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/1701887065":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/2339434463":"2024-09-03T09:54:15.686961+0000","172.21.15.151:6801/609062416":"2024-09-03T09:54:38.970754+0000","172.21.15.151:0/2251539148":"2024-09-03T09:54:15.686961+0000","172.21.15.151:0/394946850":"2024-09-03T09:54:38.970754+0000","172.21.15.151:6800/609062416":"2024-09-03T09:54:38.970754+0000","172.21.15.151:6801/3972807807":"2024-09-03T09:54:15.686961+0000","172.21.15.151:6800/3972807807":"2024-09-03T09:54:15.686961+0000","172.21.15.151:0/3380605706":"2024-09-03T09:54:15.686961+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-02T10:38:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:33 smithi187 ceph-mon[20072]: 1.14 scrub starts 2024-09-02T10:38:33.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:33 smithi187 ceph-mon[20072]: 1.14 scrub ok 2024-09-02T10:38:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:33 smithi162 ceph-mon[20279]: 1.14 scrub starts 2024-09-02T10:38:33.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:33 smithi162 ceph-mon[20279]: 1.14 scrub ok 2024-09-02T10:38:34.279 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-02T10:38:34.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-02T10:38:34.280 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-02T10:38:34.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-02T10:38:34.281 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-02T10:38:34.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-02T10:38:34.282 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-02T10:38:34.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-02T10:38:34.283 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-09-02T10:38:34.284 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-09-02T10:38:34.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-09-02T10:38:34.285 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-09-02T10:38:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:34 smithi187 ceph-mon[20072]: pgmap v2342: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:34 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2472814157' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:38:34.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:34 smithi187 ceph-mon[20072]: osdmap e1036: 12 total, 12 up, 12 in 2024-09-02T10:38:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:34 smithi162 ceph-mon[20279]: pgmap v2342: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:34 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2472814157' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:38:34.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:34 smithi162 ceph-mon[20279]: osdmap e1036: 12 total, 12 up, 12 in 2024-09-02T10:38:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:34 smithi151 ceph-mon[16639]: pgmap v2342: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:34 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2472814157' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-09-02T10:38:34.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:34 smithi151 ceph-mon[16639]: osdmap e1036: 12 total, 12 up, 12 in 2024-09-02T10:38:35.038 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.047 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.051 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.051 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.051 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.052 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.052 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.055 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.056 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.066 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.073 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.076 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:35.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:35 smithi151 ceph-mon[16639]: 1.1d scrub starts 2024-09-02T10:38:35.752 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:35 smithi151 ceph-mon[16639]: 1.1d scrub ok 2024-09-02T10:38:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:35 smithi187 ceph-mon[20072]: 1.1d scrub starts 2024-09-02T10:38:35.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:35 smithi187 ceph-mon[20072]: 1.1d scrub ok 2024-09-02T10:38:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:35 smithi162 ceph-mon[20279]: 1.1d scrub starts 2024-09-02T10:38:35.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:35 smithi162 ceph-mon[20279]: 1.1d scrub ok 2024-09-02T10:38:36.707 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:36 smithi151 ceph-mon[16639]: 1.1a scrub starts 2024-09-02T10:38:36.707 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:36 smithi151 ceph-mon[16639]: 1.1a scrub ok 2024-09-02T10:38:36.707 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:36 smithi151 ceph-mon[16639]: pgmap v2344: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:36 smithi187 ceph-mon[20072]: 1.1a scrub starts 2024-09-02T10:38:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:36 smithi187 ceph-mon[20072]: 1.1a scrub ok 2024-09-02T10:38:36.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:36 smithi187 ceph-mon[20072]: pgmap v2344: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:36 smithi162 ceph-mon[20279]: 1.1a scrub starts 2024-09-02T10:38:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:36 smithi162 ceph-mon[20279]: 1.1a scrub ok 2024-09-02T10:38:36.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:36 smithi162 ceph-mon[20279]: pgmap v2344: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:38.729 INFO:teuthology.orchestra.run.smithi151.stdout:77309411835 2024-09-02T10:38:38.729 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-02T10:38:38.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:38 smithi162 ceph-mon[20279]: pgmap v2345: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:39.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:38 smithi187 ceph-mon[20072]: pgmap v2345: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:39.040 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:38 smithi151 ceph-mon[16639]: pgmap v2345: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:39.909 INFO:teuthology.orchestra.run.smithi151.stdout:343597384142 2024-09-02T10:38:39.909 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-09-02T10:38:40.663 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:40 smithi151 ceph-mon[16639]: pgmap v2346: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:40.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:40 smithi187 ceph-mon[20072]: pgmap v2346: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:40.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:40 smithi162 ceph-mon[20279]: pgmap v2346: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:40.976 INFO:teuthology.orchestra.run.smithi151.stdout:223338299876 2024-09-02T10:38:40.976 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-02T10:38:42.843 INFO:teuthology.orchestra.run.smithi151.stdout:34359738885 2024-09-02T10:38:42.844 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-02T10:38:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:42 smithi187 ceph-mon[20072]: pgmap v2347: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:42 smithi187 ceph-mon[20072]: 1.d scrub starts 2024-09-02T10:38:43.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:42 smithi187 ceph-mon[20072]: 1.d scrub ok 2024-09-02T10:38:43.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:42 smithi187 ceph-mon[20072]: 1.16 scrub starts 2024-09-02T10:38:43.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:42 smithi187 ceph-mon[20072]: 1.16 scrub ok 2024-09-02T10:38:43.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:42 smithi151 ceph-mon[16639]: pgmap v2347: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:43.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:42 smithi151 ceph-mon[16639]: 1.d scrub starts 2024-09-02T10:38:43.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:42 smithi151 ceph-mon[16639]: 1.d scrub ok 2024-09-02T10:38:43.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:42 smithi151 ceph-mon[16639]: 1.16 scrub starts 2024-09-02T10:38:43.067 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:42 smithi151 ceph-mon[16639]: 1.16 scrub ok 2024-09-02T10:38:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:42 smithi162 ceph-mon[20279]: pgmap v2347: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:42 smithi162 ceph-mon[20279]: 1.d scrub starts 2024-09-02T10:38:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:42 smithi162 ceph-mon[20279]: 1.d scrub ok 2024-09-02T10:38:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:42 smithi162 ceph-mon[20279]: 1.16 scrub starts 2024-09-02T10:38:43.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:42 smithi162 ceph-mon[20279]: 1.16 scrub ok 2024-09-02T10:38:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:43 smithi187 ceph-mon[20072]: 1.2 scrub starts 2024-09-02T10:38:44.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:43 smithi187 ceph-mon[20072]: 1.2 scrub ok 2024-09-02T10:38:44.086 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:43 smithi151 ceph-mon[16639]: 1.2 scrub starts 2024-09-02T10:38:44.086 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:43 smithi151 ceph-mon[16639]: 1.2 scrub ok 2024-09-02T10:38:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:43 smithi162 ceph-mon[20279]: 1.2 scrub starts 2024-09-02T10:38:44.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:43 smithi162 ceph-mon[20279]: 1.2 scrub ok 2024-09-02T10:38:44.118 INFO:teuthology.orchestra.run.smithi151.stdout:55834575361 2024-09-02T10:38:44.118 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-02T10:38:44.866 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:44 smithi151 ceph-mon[16639]: pgmap v2348: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:44.895 INFO:teuthology.orchestra.run.smithi151.stdout:283467842011 2024-09-02T10:38:44.895 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-09-02T10:38:45.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:44 smithi187 ceph-mon[20072]: pgmap v2348: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:45.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:44 smithi162 ceph-mon[20279]: pgmap v2348: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:45.611 INFO:teuthology.orchestra.run.smithi151.stdout:313532613077 2024-09-02T10:38:45.612 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-09-02T10:38:46.046 INFO:teuthology.orchestra.run.smithi151.stdout:193273528810 2024-09-02T10:38:46.046 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-02T10:38:46.807 INFO:teuthology.orchestra.run.smithi151.stdout:167503725038 2024-09-02T10:38:46.807 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-02T10:38:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:46 smithi187 ceph-mon[20072]: pgmap v2349: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:46 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:46 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:46 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:47.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:46 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:46 smithi162 ceph-mon[20279]: pgmap v2349: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:46 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:46 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:46 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:47.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:46 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:46 smithi151 ceph-mon[16639]: pgmap v2349: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:46 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:46 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:46 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:47.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:46 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:47.245 INFO:teuthology.orchestra.run.smithi151.stdout:137438953969 2024-09-02T10:38:47.245 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-02T10:38:47.741 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:47.767 INFO:teuthology.orchestra.run.smithi151.stdout:253403070945 2024-09-02T10:38:47.767 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-09-02T10:38:48.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:47 smithi187 ceph-mon[20072]: osdmap e1037: 12 total, 12 up, 12 in 2024-09-02T10:38:48.024 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:47 smithi151 ceph-mon[16639]: osdmap e1037: 12 total, 12 up, 12 in 2024-09-02T10:38:48.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:47 smithi162 ceph-mon[20279]: osdmap e1037: 12 total, 12 up, 12 in 2024-09-02T10:38:48.287 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:48.289 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:48.291 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:48.291 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:48.473 INFO:teuthology.orchestra.run.smithi151.stdout:193273528812 2024-09-02T10:38:48.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:48 smithi151 ceph-mon[16639]: pgmap v2351: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:48.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:48 smithi151 ceph-mon[16639]: 1.e scrub starts 2024-09-02T10:38:48.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:48 smithi151 ceph-mon[16639]: 1.e scrub ok 2024-09-02T10:38:48.910 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:48 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1067237122' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-02T10:38:49.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:48 smithi187 ceph-mon[20072]: pgmap v2351: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:48 smithi187 ceph-mon[20072]: 1.e scrub starts 2024-09-02T10:38:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:48 smithi187 ceph-mon[20072]: 1.e scrub ok 2024-09-02T10:38:49.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:48 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1067237122' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-02T10:38:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:48 smithi162 ceph-mon[20279]: pgmap v2351: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:48 smithi162 ceph-mon[20279]: 1.e scrub starts 2024-09-02T10:38:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:48 smithi162 ceph-mon[20279]: 1.e scrub ok 2024-09-02T10:38:49.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:48 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1067237122' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-09-02T10:38:49.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:49 smithi151 ceph-mon[16639]: 1.c scrub starts 2024-09-02T10:38:49.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:49 smithi151 ceph-mon[16639]: 1.c scrub ok 2024-09-02T10:38:49.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:49 smithi151 ceph-mon[16639]: osdmap e1038: 12 total, 12 up, 12 in 2024-09-02T10:38:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:49 smithi187 ceph-mon[20072]: 1.c scrub starts 2024-09-02T10:38:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:49 smithi187 ceph-mon[20072]: 1.c scrub ok 2024-09-02T10:38:50.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:49 smithi187 ceph-mon[20072]: osdmap e1038: 12 total, 12 up, 12 in 2024-09-02T10:38:50.092 INFO:teuthology.orchestra.run.smithi151.stdout:111669150201 2024-09-02T10:38:50.093 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-02T10:38:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:49 smithi162 ceph-mon[20279]: 1.c scrub starts 2024-09-02T10:38:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:49 smithi162 ceph-mon[20279]: 1.c scrub ok 2024-09-02T10:38:50.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:49 smithi162 ceph-mon[20279]: osdmap e1038: 12 total, 12 up, 12 in 2024-09-02T10:38:50.462 INFO:teuthology.orchestra.run.smithi151.stdout:55834575362 2024-09-02T10:38:50.991 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:50 smithi151 ceph-mon[16639]: pgmap v2353: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:50.991 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:38:50.991 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:38:50.991 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:38:50.991 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:38:50.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:38:50.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:38:50.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:50 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:38:50.992 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:50 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3947151040' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-02T10:38:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:50 smithi187 ceph-mon[20072]: pgmap v2353: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:38:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:38:51.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:38:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:38:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:38:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:38:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:50 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:38:51.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:50 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3947151040' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-02T10:38:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:50 smithi162 ceph-mon[20279]: pgmap v2353: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-09-02T10:38:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-09-02T10:38:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-09-02T10:38:51.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:38:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-09-02T10:38:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-09-02T10:38:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:50 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' 2024-09-02T10:38:51.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:50 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3947151040' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-09-02T10:38:51.556 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528810 got 193273528812 for osd.6 2024-09-02T10:38:51.556 DEBUG:teuthology.parallel:result is None 2024-09-02T10:38:51.923 INFO:teuthology.orchestra.run.smithi151.stdout:77309411837 2024-09-02T10:38:52.428 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575361 got 55834575362 for osd.1 2024-09-02T10:38:52.428 DEBUG:teuthology.parallel:result is None 2024-09-02T10:38:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:52 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:52 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:52 smithi151 ceph-mon[16639]: pgmap v2354: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:52 smithi151 ceph-mon[16639]: osdmap e1039: 12 total, 12 up, 12 in 2024-09-02T10:38:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:52 smithi151 ceph-mon[16639]: 1.1c scrub starts 2024-09-02T10:38:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:52 smithi151 ceph-mon[16639]: 1.1c scrub ok 2024-09-02T10:38:52.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:52 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1223358015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-02T10:38:52.950 INFO:teuthology.orchestra.run.smithi151.stdout:223338299879 2024-09-02T10:38:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:52 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:52 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:53.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:52 smithi187 ceph-mon[20072]: pgmap v2354: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:53.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:52 smithi187 ceph-mon[20072]: osdmap e1039: 12 total, 12 up, 12 in 2024-09-02T10:38:53.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:52 smithi187 ceph-mon[20072]: 1.1c scrub starts 2024-09-02T10:38:53.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:52 smithi187 ceph-mon[20072]: 1.1c scrub ok 2024-09-02T10:38:53.007 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:52 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1223358015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-02T10:38:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:52 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:52 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:52 smithi162 ceph-mon[20279]: pgmap v2354: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:52 smithi162 ceph-mon[20279]: osdmap e1039: 12 total, 12 up, 12 in 2024-09-02T10:38:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:52 smithi162 ceph-mon[20279]: 1.1c scrub starts 2024-09-02T10:38:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:52 smithi162 ceph-mon[20279]: 1.1c scrub ok 2024-09-02T10:38:53.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:52 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1223358015' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-09-02T10:38:53.619 INFO:teuthology.orchestra.run.smithi151.stdout:34359738888 2024-09-02T10:38:53.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/1975958469' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-02T10:38:53.920 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:53 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/4211191164' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-02T10:38:53.948 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411835 got 77309411837 for osd.2 2024-09-02T10:38:53.949 DEBUG:teuthology.parallel:result is None 2024-09-02T10:38:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/1975958469' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-02T10:38:54.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:53 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/4211191164' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-02T10:38:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/1975958469' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-09-02T10:38:54.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:53 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/4211191164' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-09-02T10:38:54.172 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:54.173 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:54.174 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:54.699 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738885 got 34359738888 for osd.0 2024-09-02T10:38:54.699 DEBUG:teuthology.parallel:result is None 2024-09-02T10:38:54.717 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:54 smithi151 ceph-mon[16639]: pgmap v2356: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:54.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:54 smithi151 ceph-mon[16639]: osdmap e1040: 12 total, 12 up, 12 in 2024-09-02T10:38:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:54 smithi187 ceph-mon[20072]: pgmap v2356: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:55.006 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:54 smithi187 ceph-mon[20072]: osdmap e1040: 12 total, 12 up, 12 in 2024-09-02T10:38:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:54 smithi162 ceph-mon[20279]: pgmap v2356: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:55.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:54 smithi162 ceph-mon[20279]: osdmap e1040: 12 total, 12 up, 12 in 2024-09-02T10:38:55.153 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:55.162 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299876 got 223338299879 for osd.7 2024-09-02T10:38:55.163 DEBUG:teuthology.parallel:result is None 2024-09-02T10:38:56.037 INFO:teuthology.orchestra.run.smithi151.stdout:137438953973 2024-09-02T10:38:56.679 INFO:teuthology.orchestra.run.smithi151.stdout:253403070948 2024-09-02T10:38:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:56 smithi151 ceph-mon[16639]: 1.f scrub starts 2024-09-02T10:38:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:56 smithi151 ceph-mon[16639]: 1.f scrub ok 2024-09-02T10:38:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:56 smithi151 ceph-mon[16639]: pgmap v2358: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:56 smithi151 ceph-mon[16639]: 1.3 scrub starts 2024-09-02T10:38:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:56 smithi151 ceph-mon[16639]: 1.3 scrub ok 2024-09-02T10:38:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3738282046' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-02T10:38:56.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:56 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2989584885' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-02T10:38:56.946 INFO:teuthology.orchestra.run.smithi151.stdout:111669150203 2024-09-02T10:38:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:56 smithi162 ceph-mon[20279]: 1.f scrub starts 2024-09-02T10:38:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:56 smithi162 ceph-mon[20279]: 1.f scrub ok 2024-09-02T10:38:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:56 smithi162 ceph-mon[20279]: pgmap v2358: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:56 smithi162 ceph-mon[20279]: 1.3 scrub starts 2024-09-02T10:38:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:56 smithi162 ceph-mon[20279]: 1.3 scrub ok 2024-09-02T10:38:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3738282046' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-02T10:38:57.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:56 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2989584885' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-02T10:38:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:56 smithi187 ceph-mon[20072]: 1.f scrub starts 2024-09-02T10:38:57.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:56 smithi187 ceph-mon[20072]: 1.f scrub ok 2024-09-02T10:38:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:56 smithi187 ceph-mon[20072]: pgmap v2358: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:56 smithi187 ceph-mon[20072]: 1.3 scrub starts 2024-09-02T10:38:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:56 smithi187 ceph-mon[20072]: 1.3 scrub ok 2024-09-02T10:38:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3738282046' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-09-02T10:38:57.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:56 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2989584885' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-09-02T10:38:57.702 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953969 got 137438953973 for osd.4 2024-09-02T10:38:57.703 DEBUG:teuthology.parallel:result is None 2024-09-02T10:38:57.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:57 smithi151 ceph-mon[16639]: 1.11 scrub starts 2024-09-02T10:38:57.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:57 smithi151 ceph-mon[16639]: 1.11 scrub ok 2024-09-02T10:38:57.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:57 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/189664169' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-02T10:38:57.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:57 smithi151 ceph-mon[16639]: purged_snaps scrub starts 2024-09-02T10:38:57.927 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:57 smithi151 ceph-mon[16639]: purged_snaps scrub ok 2024-09-02T10:38:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:57 smithi162 ceph-mon[20279]: 1.11 scrub starts 2024-09-02T10:38:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:57 smithi162 ceph-mon[20279]: 1.11 scrub ok 2024-09-02T10:38:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:57 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/189664169' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-02T10:38:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:57 smithi162 ceph-mon[20279]: purged_snaps scrub starts 2024-09-02T10:38:58.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:57 smithi162 ceph-mon[20279]: purged_snaps scrub ok 2024-09-02T10:38:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:57 smithi187 ceph-mon[20072]: 1.11 scrub starts 2024-09-02T10:38:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:57 smithi187 ceph-mon[20072]: 1.11 scrub ok 2024-09-02T10:38:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:57 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/189664169' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-09-02T10:38:58.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:57 smithi187 ceph-mon[20072]: purged_snaps scrub starts 2024-09-02T10:38:58.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:57 smithi187 ceph-mon[20072]: purged_snaps scrub ok 2024-09-02T10:38:58.601 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669150201 got 111669150203 for osd.3 2024-09-02T10:38:58.601 DEBUG:teuthology.parallel:result is None 2024-09-02T10:38:58.892 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:58 smithi151 ceph-mon[16639]: pgmap v2359: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:58.892 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:58 smithi151 ceph-mon[16639]: osdmap e1041: 12 total, 12 up, 12 in 2024-09-02T10:38:58.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:58 smithi151 ceph-mon[16639]: 1.1b scrub starts 2024-09-02T10:38:58.893 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:58 smithi151 ceph-mon[16639]: 1.1b scrub ok 2024-09-02T10:38:58.958 INFO:teuthology.orchestra.run.smithi151.stdout:343597384147 2024-09-02T10:38:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:58 smithi162 ceph-mon[20279]: pgmap v2359: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:58 smithi162 ceph-mon[20279]: osdmap e1041: 12 total, 12 up, 12 in 2024-09-02T10:38:59.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:58 smithi162 ceph-mon[20279]: 1.1b scrub starts 2024-09-02T10:38:59.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:58 smithi162 ceph-mon[20279]: 1.1b scrub ok 2024-09-02T10:38:59.174 INFO:teuthology.orchestra.run.smithi151.stdout:167503725040 2024-09-02T10:38:59.175 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:38:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:58 smithi187 ceph-mon[20072]: pgmap v2359: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:38:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:58 smithi187 ceph-mon[20072]: osdmap e1041: 12 total, 12 up, 12 in 2024-09-02T10:38:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:58 smithi187 ceph-mon[20072]: 1.1b scrub starts 2024-09-02T10:38:59.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:58 smithi187 ceph-mon[20072]: 1.1b scrub ok 2024-09-02T10:38:59.692 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070945 got 253403070948 for osd.8 2024-09-02T10:38:59.692 DEBUG:teuthology.parallel:result is None 2024-09-02T10:38:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:59 smithi151 ceph-mon[16639]: 1.9 scrub starts 2024-09-02T10:38:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:59 smithi151 ceph-mon[16639]: 1.9 scrub ok 2024-09-02T10:38:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2282519267' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-02T10:38:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:59 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3152979555' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-02T10:38:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:59 smithi151 ceph-mon[16639]: osdmap e1042: 12 total, 12 up, 12 in 2024-09-02T10:38:59.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:59 smithi151 ceph-mon[16639]: 1.13 scrub starts 2024-09-02T10:38:59.939 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:38:59 smithi151 ceph-mon[16639]: 1.13 scrub ok 2024-09-02T10:39:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:59 smithi162 ceph-mon[20279]: 1.9 scrub starts 2024-09-02T10:39:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:59 smithi162 ceph-mon[20279]: 1.9 scrub ok 2024-09-02T10:39:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2282519267' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-02T10:39:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:59 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3152979555' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-02T10:39:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:59 smithi162 ceph-mon[20279]: osdmap e1042: 12 total, 12 up, 12 in 2024-09-02T10:39:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:59 smithi162 ceph-mon[20279]: 1.13 scrub starts 2024-09-02T10:39:00.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:38:59 smithi162 ceph-mon[20279]: 1.13 scrub ok 2024-09-02T10:39:00.136 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597384142 got 343597384147 for osd.11 2024-09-02T10:39:00.136 DEBUG:teuthology.parallel:result is None 2024-09-02T10:39:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:59 smithi187 ceph-mon[20072]: 1.9 scrub starts 2024-09-02T10:39:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:59 smithi187 ceph-mon[20072]: 1.9 scrub ok 2024-09-02T10:39:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2282519267' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-09-02T10:39:00.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:59 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3152979555' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-09-02T10:39:00.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:59 smithi187 ceph-mon[20072]: osdmap e1042: 12 total, 12 up, 12 in 2024-09-02T10:39:00.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:59 smithi187 ceph-mon[20072]: 1.13 scrub starts 2024-09-02T10:39:00.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:38:59 smithi187 ceph-mon[20072]: 1.13 scrub ok 2024-09-02T10:39:00.331 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:39:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:00 smithi151 ceph-mon[16639]: 1.12 scrub starts 2024-09-02T10:39:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:00 smithi151 ceph-mon[16639]: 1.12 scrub ok 2024-09-02T10:39:00.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:00 smithi151 ceph-mon[16639]: pgmap v2362: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:00 smithi162 ceph-mon[20279]: 1.12 scrub starts 2024-09-02T10:39:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:00 smithi162 ceph-mon[20279]: 1.12 scrub ok 2024-09-02T10:39:01.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:00 smithi162 ceph-mon[20279]: pgmap v2362: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:01.109 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503725038 got 167503725040 for osd.5 2024-09-02T10:39:01.110 DEBUG:teuthology.parallel:result is None 2024-09-02T10:39:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:00 smithi187 ceph-mon[20072]: 1.12 scrub starts 2024-09-02T10:39:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:00 smithi187 ceph-mon[20072]: 1.12 scrub ok 2024-09-02T10:39:01.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:00 smithi187 ceph-mon[20072]: pgmap v2362: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:01.775 INFO:teuthology.orchestra.run.smithi151.stdout:283467842014 2024-09-02T10:39:01.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:01 smithi151 ceph-mon[16639]: 1.10 scrub starts 2024-09-02T10:39:01.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:01 smithi151 ceph-mon[16639]: 1.10 scrub ok 2024-09-02T10:39:01.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:01 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3038418628' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-02T10:39:01.960 INFO:teuthology.orchestra.run.smithi151.stderr:2024-09-02T10:39:01.959+0000 7fe83affd640 0 --2- 172.21.15.151:0/489052793 >> [v2:172.21.15.151:3300/0,v1:172.21.15.151:6789/0] conn(0x7fe83c1027e0 0x7fe83c102bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-02T10:39:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:01 smithi162 ceph-mon[20279]: 1.10 scrub starts 2024-09-02T10:39:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:01 smithi162 ceph-mon[20279]: 1.10 scrub ok 2024-09-02T10:39:02.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:01 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3038418628' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-02T10:39:02.156 INFO:teuthology.orchestra.run.smithi151.stdout:313532613080 2024-09-02T10:39:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:01 smithi187 ceph-mon[20072]: 1.10 scrub starts 2024-09-02T10:39:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:01 smithi187 ceph-mon[20072]: 1.10 scrub ok 2024-09-02T10:39:02.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:01 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3038418628' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-09-02T10:39:02.576 INFO:tasks.cephadm.ceph_manager.ceph:need seq 283467842011 got 283467842014 for osd.9 2024-09-02T10:39:02.577 DEBUG:teuthology.parallel:result is None 2024-09-02T10:39:03.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:02 smithi162 ceph-mon[20279]: 1.19 scrub starts 2024-09-02T10:39:03.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:02 smithi162 ceph-mon[20279]: 1.19 scrub ok 2024-09-02T10:39:03.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:02 smithi162 ceph-mon[20279]: pgmap v2363: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:03.096 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:02 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/3059515643' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-02T10:39:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:02 smithi151 ceph-mon[16639]: 1.19 scrub starts 2024-09-02T10:39:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:02 smithi151 ceph-mon[16639]: 1.19 scrub ok 2024-09-02T10:39:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:02 smithi151 ceph-mon[16639]: pgmap v2363: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:03.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:02 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/3059515643' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-02T10:39:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:02 smithi187 ceph-mon[20072]: 1.19 scrub starts 2024-09-02T10:39:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:02 smithi187 ceph-mon[20072]: 1.19 scrub ok 2024-09-02T10:39:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:02 smithi187 ceph-mon[20072]: pgmap v2363: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:03.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:02 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/3059515643' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-09-02T10:39:03.263 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532613077 got 313532613080 for osd.10 2024-09-02T10:39:03.263 DEBUG:teuthology.parallel:result is None 2024-09-02T10:39:03.263 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-02T10:39:03.263 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:39:03.513 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:39:04.162 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:39:04.162 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:39:05.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:04 smithi151 ceph-mon[16639]: pgmap v2364: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:05.060 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:04 smithi151 ceph-mon[16639]: from='client.19539 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:39:05.089 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2364,"stamp":"2024-09-02T10:39:03.656229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1259104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9505712,"kb_used_data":13104,"kb_used_omap":18,"kb_used_meta":9492333,"kb_avail":1115436112,"statfs":{"total":1151940427776,"available":1142206578688,"internally_reserved":0,"allocated":13418496,"data_stored":8748090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":9720149307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001516"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":101,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:39:00.000212+0000","last_change":"2024-09-02T10:37:45.085566+0000","last_active":"2024-09-02T10:39:00.000212+0000","last_peered":"2024-09-02T10:39:00.000212+0000","last_clean":"2024-09-02T10:39:00.000212+0000","last_became_active":"2024-09-02T10:37:39.768292+0000","last_became_peered":"2024-09-02T10:37:39.768292+0000","last_unstale":"2024-09-02T10:39:00.000212+0000","last_undegraded":"2024-09-02T10:39:00.000212+0000","last_fullsized":"2024-09-02T10:39:00.000212+0000","mapping_epoch":1001,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":1002,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:45.085477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:45.085477+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:08.172195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":1082,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437260+0000","last_change":"2024-09-02T10:38:33.663241+0000","last_active":"2024-09-02T10:38:59.437260+0000","last_peered":"2024-09-02T10:38:59.437260+0000","last_clean":"2024-09-02T10:38:59.437260+0000","last_became_active":"2024-09-02T10:38:29.431342+0000","last_became_peered":"2024-09-02T10:38:29.431342+0000","last_unstale":"2024-09-02T10:38:59.437260+0000","last_undegraded":"2024-09-02T10:38:59.437260+0000","last_fullsized":"2024-09-02T10:38:59.437260+0000","mapping_epoch":1033,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":1034,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:33.663192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:33.663192+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:46.609763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1796,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437647+0000","last_change":"2024-09-02T10:38:51.776035+0000","last_active":"2024-09-02T10:38:59.437647+0000","last_peered":"2024-09-02T10:38:59.437647+0000","last_clean":"2024-09-02T10:38:59.437647+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:38:59.437647+0000","last_undegraded":"2024-09-02T10:38:59.437647+0000","last_fullsized":"2024-09-02T10:38:59.437647+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:51.775977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:51.775977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:20.531641+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1772,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437624+0000","last_change":"2024-09-02T10:38:57.854177+0000","last_active":"2024-09-02T10:38:59.437624+0000","last_peered":"2024-09-02T10:38:59.437624+0000","last_clean":"2024-09-02T10:38:59.437624+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:38:59.437624+0000","last_undegraded":"2024-09-02T10:38:59.437624+0000","last_fullsized":"2024-09-02T10:38:59.437624+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:57.854100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:57.854100+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:10.367337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":1092,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437961+0000","last_change":"2024-09-02T10:38:35.363540+0000","last_active":"2024-09-02T10:38:59.437961+0000","last_peered":"2024-09-02T10:38:59.437961+0000","last_clean":"2024-09-02T10:38:59.437961+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:38:59.437961+0000","last_undegraded":"2024-09-02T10:38:59.437961+0000","last_fullsized":"2024-09-02T10:38:59.437961+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:35.363478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:35.363478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:02.348692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":157,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:39:00.846457+0000","last_change":"2024-09-02T10:39:00.846457+0000","last_active":"2024-09-02T10:39:00.846457+0000","last_peered":"2024-09-02T10:39:00.846457+0000","last_clean":"2024-09-02T10:39:00.846457+0000","last_became_active":"2024-09-02T10:36:51.853478+0000","last_became_peered":"2024-09-02T10:36:51.853478+0000","last_unstale":"2024-09-02T10:39:00.846457+0000","last_undegraded":"2024-09-02T10:39:00.846457+0000","last_fullsized":"2024-09-02T10:39:00.846457+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:39:00.846394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:39:00.846394+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:22.581830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":2145,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.456033+0000","last_change":"2024-09-02T10:37:54.680908+0000","last_active":"2024-09-02T10:38:59.456033+0000","last_peered":"2024-09-02T10:38:59.456033+0000","last_clean":"2024-09-02T10:38:59.456033+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:38:59.456033+0000","last_undegraded":"2024-09-02T10:38:59.456033+0000","last_fullsized":"2024-09-02T10:38:59.456033+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:54.680824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:54.680824+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:14.294311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1802,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437733+0000","last_change":"2024-09-02T10:37:48.715208+0000","last_active":"2024-09-02T10:38:59.437733+0000","last_peered":"2024-09-02T10:38:59.437733+0000","last_clean":"2024-09-02T10:38:59.437733+0000","last_became_active":"2024-09-02T10:36:51.855455+0000","last_became_peered":"2024-09-02T10:36:51.855455+0000","last_unstale":"2024-09-02T10:38:59.437733+0000","last_undegraded":"2024-09-02T10:38:59.437733+0000","last_fullsized":"2024-09-02T10:38:59.437733+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:48.715132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:48.715132+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:07.751474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":1547,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437193+0000","last_change":"2024-09-02T10:38:42.345548+0000","last_active":"2024-09-02T10:38:59.437193+0000","last_peered":"2024-09-02T10:38:59.437193+0000","last_clean":"2024-09-02T10:38:59.437193+0000","last_became_active":"2024-09-02T10:37:51.143616+0000","last_became_peered":"2024-09-02T10:37:51.143616+0000","last_unstale":"2024-09-02T10:38:59.437193+0000","last_undegraded":"2024-09-02T10:38:59.437193+0000","last_fullsized":"2024-09-02T10:38:59.437193+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:42.345496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:42.345496+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:59.275859+0000","scrub_duration":698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1754,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999775+0000","last_change":"2024-09-02T10:38:14.911474+0000","last_active":"2024-09-02T10:38:59.999775+0000","last_peered":"2024-09-02T10:38:59.999775+0000","last_clean":"2024-09-02T10:38:59.999775+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:38:59.999775+0000","last_undegraded":"2024-09-02T10:38:59.999775+0000","last_fullsized":"2024-09-02T10:38:59.999775+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:14.911414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:14.911414+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:15.621873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":1007,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.554008+0000","last_change":"2024-09-02T10:38:32.755904+0000","last_active":"2024-09-02T10:38:59.554008+0000","last_peered":"2024-09-02T10:38:59.554008+0000","last_clean":"2024-09-02T10:38:59.554008+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:38:59.554008+0000","last_undegraded":"2024-09-02T10:38:59.554008+0000","last_fullsized":"2024-09-02T10:38:59.554008+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:32.755851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:32.755851+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:33.040326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":203,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.555858+0000","last_change":"2024-09-02T10:38:59.555858+0000","last_active":"2024-09-02T10:38:59.555858+0000","last_peered":"2024-09-02T10:38:59.555858+0000","last_clean":"2024-09-02T10:38:59.555858+0000","last_became_active":"2024-09-02T10:36:12.767657+0000","last_became_peered":"2024-09-02T10:36:12.767657+0000","last_unstale":"2024-09-02T10:38:59.555858+0000","last_undegraded":"2024-09-02T10:38:59.555858+0000","last_fullsized":"2024-09-02T10:38:59.555858+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:59.555800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:59.555800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:08.396231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":2044,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.457449+0000","last_change":"2024-09-02T10:38:59.457449+0000","last_active":"2024-09-02T10:38:59.457449+0000","last_peered":"2024-09-02T10:38:59.457449+0000","last_clean":"2024-09-02T10:38:59.457449+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:38:59.457449+0000","last_undegraded":"2024-09-02T10:38:59.457449+0000","last_fullsized":"2024-09-02T10:38:59.457449+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:59.457358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:59.457358+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:03.152837+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":790,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999547+0000","last_change":"2024-09-02T10:38:56.282726+0000","last_active":"2024-09-02T10:38:59.999547+0000","last_peered":"2024-09-02T10:38:59.999547+0000","last_clean":"2024-09-02T10:38:59.999547+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:38:59.999547+0000","last_undegraded":"2024-09-02T10:38:59.999547+0000","last_fullsized":"2024-09-02T10:38:59.999547+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:56.282656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:56.282656+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:21.787743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":2036,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:39:00.423048+0000","last_change":"2024-09-02T10:39:00.423048+0000","last_active":"2024-09-02T10:39:00.423048+0000","last_peered":"2024-09-02T10:39:00.423048+0000","last_clean":"2024-09-02T10:39:00.423048+0000","last_became_active":"2024-09-02T10:36:51.863957+0000","last_became_peered":"2024-09-02T10:36:51.863957+0000","last_unstale":"2024-09-02T10:39:00.423048+0000","last_undegraded":"2024-09-02T10:39:00.423048+0000","last_fullsized":"2024-09-02T10:39:00.423048+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:39:00.422988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:39:00.422988+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:19.862397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":2109,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.456266+0000","last_change":"2024-09-02T10:37:59.593108+0000","last_active":"2024-09-02T10:38:59.456266+0000","last_peered":"2024-09-02T10:38:59.456266+0000","last_clean":"2024-09-02T10:38:59.456266+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:38:59.456266+0000","last_undegraded":"2024-09-02T10:38:59.456266+0000","last_fullsized":"2024-09-02T10:38:59.456266+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:59.593045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:59.593045+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:21.698679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":2234,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999600+0000","last_change":"2024-09-02T10:38:58.848609+0000","last_active":"2024-09-02T10:38:59.999600+0000","last_peered":"2024-09-02T10:38:59.999600+0000","last_clean":"2024-09-02T10:38:59.999600+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:38:59.999600+0000","last_undegraded":"2024-09-02T10:38:59.999600+0000","last_fullsized":"2024-09-02T10:38:59.999600+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:58.848532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:58.848532+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:25.023379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1885,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999829+0000","last_change":"2024-09-02T10:37:42.695272+0000","last_active":"2024-09-02T10:38:59.999829+0000","last_peered":"2024-09-02T10:38:59.999829+0000","last_clean":"2024-09-02T10:38:59.999829+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:38:59.999829+0000","last_undegraded":"2024-09-02T10:38:59.999829+0000","last_fullsized":"2024-09-02T10:38:59.999829+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:42.695226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:42.695226+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":28,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999767+0000","last_change":"2024-09-02T10:38:29.430324+0000","last_active":"2024-09-02T10:38:59.999767+0000","last_peered":"2024-09-02T10:38:59.999767+0000","last_clean":"2024-09-02T10:38:59.999767+0000","last_became_active":"2024-09-02T10:38:29.429895+0000","last_became_peered":"2024-09-02T10:38:29.429895+0000","last_unstale":"2024-09-02T10:38:59.999767+0000","last_undegraded":"2024-09-02T10:38:59.999767+0000","last_fullsized":"2024-09-02T10:38:59.999767+0000","mapping_epoch":1033,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":1034,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:44.366042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:44.366042+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:05.291908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":98,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999819+0000","last_change":"2024-09-02T10:38:08.016867+0000","last_active":"2024-09-02T10:38:59.999819+0000","last_peered":"2024-09-02T10:38:59.999819+0000","last_clean":"2024-09-02T10:38:59.999819+0000","last_became_active":"2024-09-02T10:37:39.417330+0000","last_became_peered":"2024-09-02T10:37:39.417330+0000","last_unstale":"2024-09-02T10:38:59.999819+0000","last_undegraded":"2024-09-02T10:38:59.999819+0000","last_fullsized":"2024-09-02T10:38:59.999819+0000","mapping_epoch":1001,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":1002,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:08.016799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:08.016799+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:29.290205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":150,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999859+0000","last_change":"2024-09-02T10:37:47.634469+0000","last_active":"2024-09-02T10:38:59.999859+0000","last_peered":"2024-09-02T10:38:59.999859+0000","last_clean":"2024-09-02T10:38:59.999859+0000","last_became_active":"2024-09-02T10:36:51.859972+0000","last_became_peered":"2024-09-02T10:36:51.859972+0000","last_unstale":"2024-09-02T10:38:59.999859+0000","last_undegraded":"2024-09-02T10:38:59.999859+0000","last_fullsized":"2024-09-02T10:38:59.999859+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:47.634393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:47.634393+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:04.033773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":150,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:39:00.000123+0000","last_change":"2024-09-02T10:37:41.726908+0000","last_active":"2024-09-02T10:39:00.000123+0000","last_peered":"2024-09-02T10:39:00.000123+0000","last_clean":"2024-09-02T10:39:00.000123+0000","last_became_active":"2024-09-02T10:36:51.854167+0000","last_became_peered":"2024-09-02T10:36:51.854167+0000","last_unstale":"2024-09-02T10:39:00.000123+0000","last_undegraded":"2024-09-02T10:39:00.000123+0000","last_fullsized":"2024-09-02T10:39:00.000123+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:41.726856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:41.726856+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:04.930045+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1425,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.457569+0000","last_change":"2024-09-02T10:38:31.790306+0000","last_active":"2024-09-02T10:38:59.457569+0000","last_peered":"2024-09-02T10:38:59.457569+0000","last_clean":"2024-09-02T10:38:59.457569+0000","last_became_active":"2024-09-02T10:36:51.854769+0000","last_became_peered":"2024-09-02T10:36:51.854769+0000","last_unstale":"2024-09-02T10:38:59.457569+0000","last_undegraded":"2024-09-02T10:38:59.457569+0000","last_fullsized":"2024-09-02T10:38:59.457569+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:31.790221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:31.790221+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:59.150430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":150,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:39:00.000147+0000","last_change":"2024-09-02T10:37:44.608737+0000","last_active":"2024-09-02T10:39:00.000147+0000","last_peered":"2024-09-02T10:39:00.000147+0000","last_clean":"2024-09-02T10:39:00.000147+0000","last_became_active":"2024-09-02T10:36:51.853340+0000","last_became_peered":"2024-09-02T10:36:51.853340+0000","last_unstale":"2024-09-02T10:39:00.000147+0000","last_undegraded":"2024-09-02T10:39:00.000147+0000","last_fullsized":"2024-09-02T10:39:00.000147+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:44.608672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:44.608672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:11.747621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1876,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999648+0000","last_change":"2024-09-02T10:38:42.950585+0000","last_active":"2024-09-02T10:38:59.999648+0000","last_peered":"2024-09-02T10:38:59.999648+0000","last_clean":"2024-09-02T10:38:59.999648+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:38:59.999648+0000","last_undegraded":"2024-09-02T10:38:59.999648+0000","last_fullsized":"2024-09-02T10:38:59.999648+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:42.950523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:42.950523+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:00.417957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1891,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437174+0000","last_change":"2024-09-02T10:38:55.678692+0000","last_active":"2024-09-02T10:38:59.437174+0000","last_peered":"2024-09-02T10:38:59.437174+0000","last_clean":"2024-09-02T10:38:59.437174+0000","last_became_active":"2024-09-02T10:37:51.211903+0000","last_became_peered":"2024-09-02T10:37:51.211903+0000","last_unstale":"2024-09-02T10:38:59.437174+0000","last_undegraded":"2024-09-02T10:38:59.437174+0000","last_fullsized":"2024-09-02T10:38:59.437174+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:55.678614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:55.678614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:11.885624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1894,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999873+0000","last_change":"2024-09-02T10:38:31.016593+0000","last_active":"2024-09-02T10:38:59.999873+0000","last_peered":"2024-09-02T10:38:59.999873+0000","last_clean":"2024-09-02T10:38:59.999873+0000","last_became_active":"2024-09-02T10:38:29.432324+0000","last_became_peered":"2024-09-02T10:38:29.432324+0000","last_unstale":"2024-09-02T10:38:59.999873+0000","last_undegraded":"2024-09-02T10:38:59.999873+0000","last_fullsized":"2024-09-02T10:38:59.999873+0000","mapping_epoch":1033,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1034,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:31.016526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:31.016526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:56.736346+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":2151,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.457575+0000","last_change":"2024-09-02T10:38:55.591855+0000","last_active":"2024-09-02T10:38:59.457575+0000","last_peered":"2024-09-02T10:38:59.457575+0000","last_clean":"2024-09-02T10:38:59.457575+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:38:59.457575+0000","last_undegraded":"2024-09-02T10:38:59.457575+0000","last_fullsized":"2024-09-02T10:38:59.457575+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:55.591791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:55.591791+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:02.111114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":1100,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437930+0000","last_change":"2024-09-02T10:38:48.429396+0000","last_active":"2024-09-02T10:38:59.437930+0000","last_peered":"2024-09-02T10:38:59.437930+0000","last_clean":"2024-09-02T10:38:59.437930+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:38:59.437930+0000","last_undegraded":"2024-09-02T10:38:59.437930+0000","last_fullsized":"2024-09-02T10:38:59.437930+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:48.429329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:48.429329+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:13.419453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1914,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999844+0000","last_change":"2024-09-02T10:38:41.950832+0000","last_active":"2024-09-02T10:38:59.999844+0000","last_peered":"2024-09-02T10:38:59.999844+0000","last_clean":"2024-09-02T10:38:59.999844+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:38:59.999844+0000","last_undegraded":"2024-09-02T10:38:59.999844+0000","last_fullsized":"2024-09-02T10:38:59.999844+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:41.950766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:41.950766+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:55.467321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":1322,"reported_epoch":1041,"state":"active+clean","last_fresh":"2024-09-02T10:38:57.923463+0000","last_change":"2024-09-02T10:38:49.193181+0000","last_active":"2024-09-02T10:38:57.923463+0000","last_peered":"2024-09-02T10:38:57.923463+0000","last_clean":"2024-09-02T10:38:57.923463+0000","last_became_active":"2024-09-02T10:37:29.684767+0000","last_became_peered":"2024-09-02T10:37:29.684767+0000","last_unstale":"2024-09-02T10:38:57.923463+0000","last_undegraded":"2024-09-02T10:38:57.923463+0000","last_fullsized":"2024-09-02T10:38:57.923463+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:49.193117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:49.193117+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:15.342792+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1259104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384148,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686040,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93059112,"statfs":{"total":95995035648,"available":95292530688,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":701495748},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532613081,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722136,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93023016,"statfs":{"total":95995035648,"available":95255568384,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":738458048},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467842015,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952600,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792552,"statfs":{"total":95995035648,"available":95019573248,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":974453184},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070949,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":591628,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93153524,"statfs":{"total":95995035648,"available":95389208576,"internally_reserved":0,"allocated":974848,"data_stored":590523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299881,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848992,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92896160,"statfs":{"total":95995035648,"available":95125667840,"internally_reserved":0,"allocated":1519616,"data_stored":1126954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":867826122},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618012,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93127140,"statfs":{"total":95995035648,"available":95362191360,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":631830983},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503725042,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":997220,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":996030,"kb_avail":92747932,"statfs":{"total":95995035648,"available":94973882368,"internally_reserved":0,"allocated":1191936,"data_stored":799274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1019935173},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953974,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595804,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93149348,"statfs":{"total":95995035648,"available":95384932352,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150204,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":961228,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92783924,"statfs":{"total":95995035648,"available":95010738176,"internally_reserved":0,"allocated":1302528,"data_stored":918203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":982972872},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636260,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93108892,"statfs":{"total":95995035648,"available":95343505408,"internally_reserved":0,"allocated":1191936,"data_stored":799274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575365,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924824,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92820328,"statfs":{"total":95995035648,"available":95048015872,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":970968,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92774184,"statfs":{"total":95995035648,"available":95000764416,"internally_reserved":0,"allocated":1314816,"data_stored":925706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:39:05.091 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph pg dump --format=json 2024-09-02T10:39:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:04 smithi162 ceph-mon[20279]: pgmap v2364: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:05.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:04 smithi162 ceph-mon[20279]: from='client.19539 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:39:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:04 smithi187 ceph-mon[20072]: pgmap v2364: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:05.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:04 smithi187 ceph-mon[20072]: from='client.19539 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:39:05.348 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:39:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:05 smithi162 ceph-mon[20279]: 1.5 scrub starts 2024-09-02T10:39:06.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:05 smithi162 ceph-mon[20279]: 1.5 scrub ok 2024-09-02T10:39:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:05 smithi151 ceph-mon[16639]: 1.5 scrub starts 2024-09-02T10:39:06.188 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:05 smithi151 ceph-mon[16639]: 1.5 scrub ok 2024-09-02T10:39:06.219 INFO:teuthology.orchestra.run.smithi151.stdout: 2024-09-02T10:39:06.220 INFO:teuthology.orchestra.run.smithi151.stderr:dumped all 2024-09-02T10:39:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:05 smithi187 ceph-mon[20072]: 1.5 scrub starts 2024-09-02T10:39:06.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:05 smithi187 ceph-mon[20072]: 1.5 scrub ok 2024-09-02T10:39:07.030 INFO:teuthology.orchestra.run.smithi151.stdout:{"pg_ready":true,"pg_map":{"version":2365,"stamp":"2024-09-02T10:39:05.656790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1259104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":93,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":9505740,"kb_used_data":13128,"kb_used_omap":18,"kb_used_meta":9492333,"kb_avail":1115436084,"statfs":{"total":1151940427776,"available":1142206550016,"internally_reserved":0,"allocated":13443072,"data_stored":8763096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":9720149307},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001657"},"pg_stats":[{"pgid":"1.1e","version":"22'92","reported_seq":101,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:39:00.000212+0000","last_change":"2024-09-02T10:37:45.085566+0000","last_active":"2024-09-02T10:39:00.000212+0000","last_peered":"2024-09-02T10:39:00.000212+0000","last_clean":"2024-09-02T10:39:00.000212+0000","last_became_active":"2024-09-02T10:37:39.768292+0000","last_became_peered":"2024-09-02T10:37:39.768292+0000","last_unstale":"2024-09-02T10:39:00.000212+0000","last_undegraded":"2024-09-02T10:39:00.000212+0000","last_fullsized":"2024-09-02T10:39:00.000212+0000","mapping_epoch":1001,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":1002,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:45.085477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:45.085477+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:08.172195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,9],"acting":[11,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.1d","version":"22'92","reported_seq":1082,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437260+0000","last_change":"2024-09-02T10:38:33.663241+0000","last_active":"2024-09-02T10:38:59.437260+0000","last_peered":"2024-09-02T10:38:59.437260+0000","last_clean":"2024-09-02T10:38:59.437260+0000","last_became_active":"2024-09-02T10:38:29.431342+0000","last_became_peered":"2024-09-02T10:38:29.431342+0000","last_unstale":"2024-09-02T10:38:59.437260+0000","last_undegraded":"2024-09-02T10:38:59.437260+0000","last_fullsized":"2024-09-02T10:38:59.437260+0000","mapping_epoch":1033,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":1034,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:33.663192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:33.663192+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:46.609763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,2],"acting":[1,11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1c","version":"22'92","reported_seq":1797,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437701+0000","last_change":"2024-09-02T10:38:51.776035+0000","last_active":"2024-09-02T10:38:59.437701+0000","last_peered":"2024-09-02T10:38:59.437701+0000","last_clean":"2024-09-02T10:38:59.437701+0000","last_became_active":"2024-09-02T10:06:34.851643+0000","last_became_peered":"2024-09-02T10:06:34.851643+0000","last_unstale":"2024-09-02T10:38:59.437701+0000","last_undegraded":"2024-09-02T10:38:59.437701+0000","last_fullsized":"2024-09-02T10:38:59.437701+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:51.775977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:51.775977+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:20.531641+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,6,1],"acting":[8,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1b","version":"22'92","reported_seq":1773,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437698+0000","last_change":"2024-09-02T10:38:57.854177+0000","last_active":"2024-09-02T10:38:59.437698+0000","last_peered":"2024-09-02T10:38:59.437698+0000","last_clean":"2024-09-02T10:38:59.437698+0000","last_became_active":"2024-09-02T10:06:34.849721+0000","last_became_peered":"2024-09-02T10:06:34.849721+0000","last_unstale":"2024-09-02T10:38:59.437698+0000","last_undegraded":"2024-09-02T10:38:59.437698+0000","last_fullsized":"2024-09-02T10:38:59.437698+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:57.854100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:57.854100+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:10.367337+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,0],"acting":[8,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.1a","version":"22'92","reported_seq":1093,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437993+0000","last_change":"2024-09-02T10:38:35.363540+0000","last_active":"2024-09-02T10:38:59.437993+0000","last_peered":"2024-09-02T10:38:59.437993+0000","last_clean":"2024-09-02T10:38:59.437993+0000","last_became_active":"2024-09-02T10:19:05.058537+0000","last_became_peered":"2024-09-02T10:19:05.058537+0000","last_unstale":"2024-09-02T10:38:59.437993+0000","last_undegraded":"2024-09-02T10:38:59.437993+0000","last_fullsized":"2024-09-02T10:38:59.437993+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:35.363478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:35.363478+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:02.348692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.19","version":"22'92","reported_seq":157,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:39:00.846457+0000","last_change":"2024-09-02T10:39:00.846457+0000","last_active":"2024-09-02T10:39:00.846457+0000","last_peered":"2024-09-02T10:39:00.846457+0000","last_clean":"2024-09-02T10:39:00.846457+0000","last_became_active":"2024-09-02T10:36:51.853478+0000","last_became_peered":"2024-09-02T10:36:51.853478+0000","last_unstale":"2024-09-02T10:39:00.846457+0000","last_undegraded":"2024-09-02T10:39:00.846457+0000","last_fullsized":"2024-09-02T10:39:00.846457+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:39:00.846394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:39:00.846394+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:22.581830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,0,6],"acting":[10,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.18","version":"22'92","reported_seq":2145,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.456033+0000","last_change":"2024-09-02T10:37:54.680908+0000","last_active":"2024-09-02T10:38:59.456033+0000","last_peered":"2024-09-02T10:38:59.456033+0000","last_clean":"2024-09-02T10:38:59.456033+0000","last_became_active":"2024-09-02T10:20:52.104428+0000","last_became_peered":"2024-09-02T10:20:52.104428+0000","last_unstale":"2024-09-02T10:38:59.456033+0000","last_undegraded":"2024-09-02T10:38:59.456033+0000","last_fullsized":"2024-09-02T10:38:59.456033+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:54.680824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:54.680824+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:14.294311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":603744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.17","version":"22'92","reported_seq":1802,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437733+0000","last_change":"2024-09-02T10:37:48.715208+0000","last_active":"2024-09-02T10:38:59.437733+0000","last_peered":"2024-09-02T10:38:59.437733+0000","last_clean":"2024-09-02T10:38:59.437733+0000","last_became_active":"2024-09-02T10:36:51.855455+0000","last_became_peered":"2024-09-02T10:36:51.855455+0000","last_unstale":"2024-09-02T10:38:59.437733+0000","last_undegraded":"2024-09-02T10:38:59.437733+0000","last_fullsized":"2024-09-02T10:38:59.437733+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:48.715132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:48.715132+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:07.751474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,5],"acting":[8,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"1.16","version":"22'92","reported_seq":1547,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437193+0000","last_change":"2024-09-02T10:38:42.345548+0000","last_active":"2024-09-02T10:38:59.437193+0000","last_peered":"2024-09-02T10:38:59.437193+0000","last_clean":"2024-09-02T10:38:59.437193+0000","last_became_active":"2024-09-02T10:37:51.143616+0000","last_became_peered":"2024-09-02T10:37:51.143616+0000","last_unstale":"2024-09-02T10:38:59.437193+0000","last_undegraded":"2024-09-02T10:38:59.437193+0000","last_fullsized":"2024-09-02T10:38:59.437193+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:42.345496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:42.345496+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:59.275859+0000","scrub_duration":698,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.15","version":"22'92","reported_seq":1754,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999775+0000","last_change":"2024-09-02T10:38:14.911474+0000","last_active":"2024-09-02T10:38:59.999775+0000","last_peered":"2024-09-02T10:38:59.999775+0000","last_clean":"2024-09-02T10:38:59.999775+0000","last_became_active":"2024-09-02T10:06:34.854446+0000","last_became_peered":"2024-09-02T10:06:34.854446+0000","last_unstale":"2024-09-02T10:38:59.999775+0000","last_undegraded":"2024-09-02T10:38:59.999775+0000","last_fullsized":"2024-09-02T10:38:59.999775+0000","mapping_epoch":243,"log_start":"0'0","ondisk_log_start":"0'0","created":241,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:14.911414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:14.911414+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:15.621873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.14","version":"22'92","reported_seq":1007,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.554008+0000","last_change":"2024-09-02T10:38:32.755904+0000","last_active":"2024-09-02T10:38:59.554008+0000","last_peered":"2024-09-02T10:38:59.554008+0000","last_clean":"2024-09-02T10:38:59.554008+0000","last_became_active":"2024-09-02T10:20:52.104176+0000","last_became_peered":"2024-09-02T10:20:52.104176+0000","last_unstale":"2024-09-02T10:38:59.554008+0000","last_undegraded":"2024-09-02T10:38:59.554008+0000","last_fullsized":"2024-09-02T10:38:59.554008+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:32.755851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:32.755851+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:33.040326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,9],"acting":[2,7,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.13","version":"22'92","reported_seq":203,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.555858+0000","last_change":"2024-09-02T10:38:59.555858+0000","last_active":"2024-09-02T10:38:59.555858+0000","last_peered":"2024-09-02T10:38:59.555858+0000","last_clean":"2024-09-02T10:38:59.555858+0000","last_became_active":"2024-09-02T10:36:12.767657+0000","last_became_peered":"2024-09-02T10:36:12.767657+0000","last_unstale":"2024-09-02T10:38:59.555858+0000","last_undegraded":"2024-09-02T10:38:59.555858+0000","last_fullsized":"2024-09-02T10:38:59.555858+0000","mapping_epoch":958,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":959,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:59.555800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:59.555800+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:08.396231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.12","version":"22'92","reported_seq":2044,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.457449+0000","last_change":"2024-09-02T10:38:59.457449+0000","last_active":"2024-09-02T10:38:59.457449+0000","last_peered":"2024-09-02T10:38:59.457449+0000","last_clean":"2024-09-02T10:38:59.457449+0000","last_became_active":"2024-09-02T10:15:07.058958+0000","last_became_peered":"2024-09-02T10:15:07.058958+0000","last_unstale":"2024-09-02T10:38:59.457449+0000","last_undegraded":"2024-09-02T10:38:59.457449+0000","last_fullsized":"2024-09-02T10:38:59.457449+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":446,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:59.457358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:59.457358+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:03.152837+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.11","version":"22'92","reported_seq":790,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999547+0000","last_change":"2024-09-02T10:38:56.282726+0000","last_active":"2024-09-02T10:38:59.999547+0000","last_peered":"2024-09-02T10:38:59.999547+0000","last_clean":"2024-09-02T10:38:59.999547+0000","last_became_active":"2024-09-02T10:24:54.194400+0000","last_became_peered":"2024-09-02T10:24:54.194400+0000","last_unstale":"2024-09-02T10:38:59.999547+0000","last_undegraded":"2024-09-02T10:38:59.999547+0000","last_fullsized":"2024-09-02T10:38:59.999547+0000","mapping_epoch":689,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":690,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:56.282656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:56.282656+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:21.787743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,9],"acting":[0,8,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.10","version":"22'92","reported_seq":2036,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:39:00.423048+0000","last_change":"2024-09-02T10:39:00.423048+0000","last_active":"2024-09-02T10:39:00.423048+0000","last_peered":"2024-09-02T10:39:00.423048+0000","last_clean":"2024-09-02T10:39:00.423048+0000","last_became_active":"2024-09-02T10:36:51.863957+0000","last_became_peered":"2024-09-02T10:36:51.863957+0000","last_unstale":"2024-09-02T10:39:00.423048+0000","last_undegraded":"2024-09-02T10:39:00.423048+0000","last_fullsized":"2024-09-02T10:39:00.423048+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":5,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:39:00.422988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:39:00.422988+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:19.862397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,10],"acting":[5,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"22'92","reported_seq":2109,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.456266+0000","last_change":"2024-09-02T10:37:59.593108+0000","last_active":"2024-09-02T10:38:59.456266+0000","last_peered":"2024-09-02T10:38:59.456266+0000","last_clean":"2024-09-02T10:38:59.456266+0000","last_became_active":"2024-09-02T10:20:52.111263+0000","last_became_peered":"2024-09-02T10:20:52.111263+0000","last_unstale":"2024-09-02T10:38:59.456266+0000","last_undegraded":"2024-09-02T10:38:59.456266+0000","last_fullsized":"2024-09-02T10:38:59.456266+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:59.593045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:59.593045+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:21.698679+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.9","version":"22'92","reported_seq":2234,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999600+0000","last_change":"2024-09-02T10:38:58.848609+0000","last_active":"2024-09-02T10:38:59.999600+0000","last_peered":"2024-09-02T10:38:59.999600+0000","last_clean":"2024-09-02T10:38:59.999600+0000","last_became_active":"2024-09-02T10:20:52.105187+0000","last_became_peered":"2024-09-02T10:20:52.105187+0000","last_unstale":"2024-09-02T10:38:59.999600+0000","last_undegraded":"2024-09-02T10:38:59.999600+0000","last_fullsized":"2024-09-02T10:38:59.999600+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:58.848532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:58.848532+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:25.023379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,2],"acting":[7,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"22'92","reported_seq":1885,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999829+0000","last_change":"2024-09-02T10:37:42.695272+0000","last_active":"2024-09-02T10:38:59.999829+0000","last_peered":"2024-09-02T10:38:59.999829+0000","last_clean":"2024-09-02T10:38:59.999829+0000","last_became_active":"2024-09-02T10:19:05.057702+0000","last_became_peered":"2024-09-02T10:19:05.057702+0000","last_unstale":"2024-09-02T10:38:59.999829+0000","last_undegraded":"2024-09-02T10:38:59.999829+0000","last_fullsized":"2024-09-02T10:38:59.999829+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:42.695226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:42.695226+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,6,4],"acting":[9,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"1.b","version":"22'92","reported_seq":28,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999767+0000","last_change":"2024-09-02T10:38:29.430324+0000","last_active":"2024-09-02T10:38:59.999767+0000","last_peered":"2024-09-02T10:38:59.999767+0000","last_clean":"2024-09-02T10:38:59.999767+0000","last_became_active":"2024-09-02T10:38:29.429895+0000","last_became_peered":"2024-09-02T10:38:29.429895+0000","last_unstale":"2024-09-02T10:38:59.999767+0000","last_undegraded":"2024-09-02T10:38:59.999767+0000","last_fullsized":"2024-09-02T10:38:59.999767+0000","mapping_epoch":1033,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":1034,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:44.366042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:44.366042+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:05.291908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,5],"acting":[11,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.4","version":"22'92","reported_seq":98,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999819+0000","last_change":"2024-09-02T10:38:08.016867+0000","last_active":"2024-09-02T10:38:59.999819+0000","last_peered":"2024-09-02T10:38:59.999819+0000","last_clean":"2024-09-02T10:38:59.999819+0000","last_became_active":"2024-09-02T10:37:39.417330+0000","last_became_peered":"2024-09-02T10:37:39.417330+0000","last_unstale":"2024-09-02T10:38:59.999819+0000","last_undegraded":"2024-09-02T10:38:59.999819+0000","last_fullsized":"2024-09-02T10:38:59.999819+0000","mapping_epoch":1001,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":1002,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:08.016799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:08.016799+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:29.290205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7],"acting":[11,8,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.5","version":"22'92","reported_seq":150,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999859+0000","last_change":"2024-09-02T10:37:47.634469+0000","last_active":"2024-09-02T10:38:59.999859+0000","last_peered":"2024-09-02T10:38:59.999859+0000","last_clean":"2024-09-02T10:38:59.999859+0000","last_became_active":"2024-09-02T10:36:51.859972+0000","last_became_peered":"2024-09-02T10:36:51.859972+0000","last_unstale":"2024-09-02T10:38:59.999859+0000","last_undegraded":"2024-09-02T10:38:59.999859+0000","last_fullsized":"2024-09-02T10:38:59.999859+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:47.634393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:47.634393+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:04.033773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,6],"acting":[10,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.6","version":"22'92","reported_seq":150,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:39:00.000123+0000","last_change":"2024-09-02T10:37:41.726908+0000","last_active":"2024-09-02T10:39:00.000123+0000","last_peered":"2024-09-02T10:39:00.000123+0000","last_clean":"2024-09-02T10:39:00.000123+0000","last_became_active":"2024-09-02T10:36:51.854167+0000","last_became_peered":"2024-09-02T10:36:51.854167+0000","last_unstale":"2024-09-02T10:39:00.000123+0000","last_undegraded":"2024-09-02T10:39:00.000123+0000","last_fullsized":"2024-09-02T10:39:00.000123+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:41.726856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:41.726856+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:04.930045+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.7","version":"22'92","reported_seq":1425,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.457569+0000","last_change":"2024-09-02T10:38:31.790306+0000","last_active":"2024-09-02T10:38:59.457569+0000","last_peered":"2024-09-02T10:38:59.457569+0000","last_clean":"2024-09-02T10:38:59.457569+0000","last_became_active":"2024-09-02T10:36:51.854769+0000","last_became_peered":"2024-09-02T10:36:51.854769+0000","last_unstale":"2024-09-02T10:38:59.457569+0000","last_undegraded":"2024-09-02T10:38:59.457569+0000","last_fullsized":"2024-09-02T10:38:59.457569+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:31.790221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:31.790221+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:59.150430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,3],"acting":[6,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"22'92","reported_seq":150,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:39:00.000147+0000","last_change":"2024-09-02T10:37:44.608737+0000","last_active":"2024-09-02T10:39:00.000147+0000","last_peered":"2024-09-02T10:39:00.000147+0000","last_clean":"2024-09-02T10:39:00.000147+0000","last_became_active":"2024-09-02T10:36:51.853340+0000","last_became_peered":"2024-09-02T10:36:51.853340+0000","last_unstale":"2024-09-02T10:39:00.000147+0000","last_undegraded":"2024-09-02T10:39:00.000147+0000","last_fullsized":"2024-09-02T10:39:00.000147+0000","mapping_epoch":975,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":976,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:37:44.608672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:37:44.608672+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:11.747621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"1.2","version":"22'92","reported_seq":1876,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999648+0000","last_change":"2024-09-02T10:38:42.950585+0000","last_active":"2024-09-02T10:38:59.999648+0000","last_peered":"2024-09-02T10:38:59.999648+0000","last_clean":"2024-09-02T10:38:59.999648+0000","last_became_active":"2024-09-02T10:19:05.058493+0000","last_became_peered":"2024-09-02T10:19:05.058493+0000","last_unstale":"2024-09-02T10:38:59.999648+0000","last_undegraded":"2024-09-02T10:38:59.999648+0000","last_fullsized":"2024-09-02T10:38:59.999648+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:42.950523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:42.950523+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:00.417957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"22'92","reported_seq":1891,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437174+0000","last_change":"2024-09-02T10:38:55.678692+0000","last_active":"2024-09-02T10:38:59.437174+0000","last_peered":"2024-09-02T10:38:59.437174+0000","last_clean":"2024-09-02T10:38:59.437174+0000","last_became_active":"2024-09-02T10:37:51.211903+0000","last_became_peered":"2024-09-02T10:37:51.211903+0000","last_unstale":"2024-09-02T10:38:59.437174+0000","last_undegraded":"2024-09-02T10:38:59.437174+0000","last_fullsized":"2024-09-02T10:38:59.437174+0000","mapping_epoch":1009,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":1010,"parent":"0.0","parent_split_bits":3,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:55.678614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:55.678614+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:11.885624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"22'92","reported_seq":1894,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999873+0000","last_change":"2024-09-02T10:38:31.016593+0000","last_active":"2024-09-02T10:38:59.999873+0000","last_peered":"2024-09-02T10:38:59.999873+0000","last_clean":"2024-09-02T10:38:59.999873+0000","last_became_active":"2024-09-02T10:38:29.432324+0000","last_became_peered":"2024-09-02T10:38:29.432324+0000","last_unstale":"2024-09-02T10:38:59.999873+0000","last_undegraded":"2024-09-02T10:38:59.999873+0000","last_fullsized":"2024-09-02T10:38:59.999873+0000","mapping_epoch":1033,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":1034,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:31.016526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:31.016526+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:56.736346+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,6],"acting":[7,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.f","version":"22'92","reported_seq":2151,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.457575+0000","last_change":"2024-09-02T10:38:55.591855+0000","last_active":"2024-09-02T10:38:59.457575+0000","last_peered":"2024-09-02T10:38:59.457575+0000","last_clean":"2024-09-02T10:38:59.457575+0000","last_became_active":"2024-09-02T10:19:05.065898+0000","last_became_peered":"2024-09-02T10:19:05.065898+0000","last_unstale":"2024-09-02T10:38:59.457575+0000","last_undegraded":"2024-09-02T10:38:59.457575+0000","last_fullsized":"2024-09-02T10:38:59.457575+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:55.591791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:55.591791+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:02.111114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.e","version":"22'92","reported_seq":1100,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.437930+0000","last_change":"2024-09-02T10:38:48.429396+0000","last_active":"2024-09-02T10:38:59.437930+0000","last_peered":"2024-09-02T10:38:59.437930+0000","last_clean":"2024-09-02T10:38:59.437930+0000","last_became_active":"2024-09-02T10:20:52.104697+0000","last_became_peered":"2024-09-02T10:20:52.104697+0000","last_unstale":"2024-09-02T10:38:59.437930+0000","last_undegraded":"2024-09-02T10:38:59.437930+0000","last_fullsized":"2024-09-02T10:38:59.437930+0000","mapping_epoch":585,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":586,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:48.429329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:48.429329+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:13.419453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,2],"acting":[4,9,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.d","version":"22'92","reported_seq":1914,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999844+0000","last_change":"2024-09-02T10:38:41.950832+0000","last_active":"2024-09-02T10:38:59.999844+0000","last_peered":"2024-09-02T10:38:59.999844+0000","last_clean":"2024-09-02T10:38:59.999844+0000","last_became_active":"2024-09-02T10:19:05.059080+0000","last_became_peered":"2024-09-02T10:19:05.059080+0000","last_unstale":"2024-09-02T10:38:59.999844+0000","last_undegraded":"2024-09-02T10:38:59.999844+0000","last_fullsized":"2024-09-02T10:38:59.999844+0000","mapping_epoch":548,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":549,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:41.950766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:41.950766+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:39:55.467321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"22'92","reported_seq":1324,"reported_epoch":1042,"state":"active+clean","last_fresh":"2024-09-02T10:38:59.999408+0000","last_change":"2024-09-02T10:38:49.193181+0000","last_active":"2024-09-02T10:38:59.999408+0000","last_peered":"2024-09-02T10:38:59.999408+0000","last_clean":"2024-09-02T10:38:59.999408+0000","last_became_active":"2024-09-02T10:37:29.684767+0000","last_became_peered":"2024-09-02T10:37:29.684767+0000","last_unstale":"2024-09-02T10:38:59.999408+0000","last_undegraded":"2024-09-02T10:38:59.999408+0000","last_fullsized":"2024-09-02T10:38:59.999408+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":4,"last_scrub":"22'92","last_scrub_stamp":"2024-09-02T10:38:49.193117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-02T09:56:54.060967+0000","last_clean_scrub_stamp":"2024-09-02T10:38:49.193117+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-02T10:40:15.342792+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":31,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1259104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2852,"ondisk_log_size":2852,"up":93,"acting":93,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597384148,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":686040,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":685054,"kb_avail":93059112,"statfs":{"total":95995035648,"available":95292530688,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":701495748},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532613081,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":722136,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":721150,"kb_avail":93023016,"statfs":{"total":95995035648,"available":95255568384,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":738458048},"hb_peers":[0,1,2,3,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":66,"seq":283467842015,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952600,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792552,"statfs":{"total":95995035648,"available":95019573248,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":974453184},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070950,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":591640,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":93153512,"statfs":{"total":95995035648,"available":95389196288,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,1,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299881,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":848992,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92896160,"statfs":{"total":95995035648,"available":95125667840,"internally_reserved":0,"allocated":1519616,"data_stored":1126954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":867826122},"hb_peers":[0,1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":618012,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":93127140,"statfs":{"total":95995035648,"available":95362191360,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":631830983},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503725042,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":997220,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":996030,"kb_avail":92747932,"statfs":{"total":95995035648,"available":94973882368,"internally_reserved":0,"allocated":1191936,"data_stored":799274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1019935173},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953975,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":595804,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":93149348,"statfs":{"total":95995035648,"available":95384932352,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669150205,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":961244,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":959934,"kb_avail":92783908,"statfs":{"total":95995035648,"available":95010721792,"internally_reserved":0,"allocated":1314816,"data_stored":925706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":982972872},"hb_peers":[0,1,2,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":636260,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":93108892,"statfs":{"total":95995035648,"available":95343505408,"internally_reserved":0,"allocated":1191936,"data_stored":799274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,1,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575366,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924824,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92820328,"statfs":{"total":95995035648,"available":95048015872,"internally_reserved":0,"allocated":987136,"data_stored":598026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[0,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":970968,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92774184,"statfs":{"total":95995035648,"available":95000764416,"internally_reserved":0,"allocated":1314816,"data_stored":925706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":201248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-02T10:39:07.033 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-02T10:39:07.033 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07 shell --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-02T10:39:07.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:06 smithi151 ceph-mon[16639]: pgmap v2365: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:07.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:06 smithi151 ceph-mon[16639]: 1.6 scrub starts 2024-09-02T10:39:07.057 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:06 smithi151 ceph-mon[16639]: 1.6 scrub ok 2024-09-02T10:39:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:06 smithi162 ceph-mon[20279]: pgmap v2365: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:06 smithi162 ceph-mon[20279]: 1.6 scrub starts 2024-09-02T10:39:07.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:06 smithi162 ceph-mon[20279]: 1.6 scrub ok 2024-09-02T10:39:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:06 smithi187 ceph-mon[20072]: pgmap v2365: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:07.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:06 smithi187 ceph-mon[20072]: 1.6 scrub starts 2024-09-02T10:39:07.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:06 smithi187 ceph-mon[20072]: 1.6 scrub ok 2024-09-02T10:39:07.284 INFO:teuthology.orchestra.run.smithi151.stderr:Inferring config /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/mon.a/config 2024-09-02T10:39:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:07 smithi151 ceph-mon[16639]: from='client.19545 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:39:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:07 smithi151 ceph-mon[16639]: 1.a scrub starts 2024-09-02T10:39:07.938 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:07 smithi151 ceph-mon[16639]: 1.a scrub ok 2024-09-02T10:39:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:07 smithi162 ceph-mon[20279]: from='client.19545 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:39:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:07 smithi162 ceph-mon[20279]: 1.a scrub starts 2024-09-02T10:39:08.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:07 smithi162 ceph-mon[20279]: 1.a scrub ok 2024-09-02T10:39:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:07 smithi187 ceph-mon[20072]: from='client.19545 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-02T10:39:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:07 smithi187 ceph-mon[20072]: 1.a scrub starts 2024-09-02T10:39:08.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:07 smithi187 ceph-mon[20072]: 1.a scrub ok 2024-09-02T10:39:08.731 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-02T10:39:08.741 INFO:tasks.cephadm:Teardown begin 2024-09-02T10:39:08.741 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T10:39:08.825 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T10:39:08.858 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T10:39:08.891 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-02T10:39:08.892 DEBUG:teuthology.orchestra.run.smithi151:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-02T10:39:08.935 DEBUG:teuthology.orchestra.run.smithi162:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-02T10:39:09.010 DEBUG:teuthology.orchestra.run.smithi187:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-02T10:39:09.039 INFO:tasks.cephadm:Stopping all daemons... 2024-09-02T10:39:09.039 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-02T10:39:09.040 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.a 2024-09-02T10:39:09.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:08 smithi151 ceph-mon[16639]: pgmap v2366: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:09.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2904131522' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-02T10:39:09.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:08 smithi151 ceph-mon[16639]: from='client.? 172.21.15.151:0/2904131522' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-02T10:39:09.061 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:08 smithi151 ceph-mon[16639]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:39:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:08 smithi162 ceph-mon[20279]: pgmap v2366: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2904131522' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-02T10:39:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:08 smithi162 ceph-mon[20279]: from='client.? 172.21.15.151:0/2904131522' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-02T10:39:09.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:08 smithi162 ceph-mon[20279]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:39:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:08 smithi187 ceph-mon[20072]: pgmap v2366: 31 pgs: 31 active+clean; 577 KiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail 2024-09-02T10:39:09.256 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2904131522' entity='client.admin' cmd={"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"} : dispatch 2024-09-02T10:39:09.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:08 smithi187 ceph-mon[20072]: from='client.? 172.21.15.151:0/2904131522' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-02T10:39:09.257 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:08 smithi187 ceph-mon[20072]: from='mgr.14151 172.21.15.151:0/2312538260' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-09-02T10:39:09.347 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:09 smithi151 systemd[1]: Stopping Ceph mon.a for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:10.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:09 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:39:09.699+0000 7fe4056df640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:39:10.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:09 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a[16635]: 2024-09-02T10:39:09.699+0000 7fe4056df640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-02T10:39:10.052 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:09 smithi151 podman[274580]: 2024-09-02 10:39:09.763288758 +0000 UTC m=+0.184321684 container died 51c3f66d7ab549ac4fd81bf0c90868257642b481b93cc6a839b50fd5ef4e297e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-02T10:39:10.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:10 smithi151 podman[274580]: 2024-09-02 10:39:10.26746819 +0000 UTC m=+0.688501147 container cleanup 51c3f66d7ab549ac4fd81bf0c90868257642b481b93cc6a839b50fd5ef4e297e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-02T10:39:10.438 INFO:journalctl@ceph.mon.a.smithi151.stdout:Sep 02 10:39:10 smithi151 bash[274580]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-a 2024-09-02T10:39:10.682 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.a.service' 2024-09-02T10:39:10.773 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:39:10.773 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-02T10:39:10.773 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-09-02T10:39:10.773 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.b 2024-09-02T10:39:11.095 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:10 smithi162 systemd[1]: Stopping Ceph mon.b for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:11.571 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:11 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-b[20275]: 2024-09-02T10:39:11.201+0000 7f2b75a0b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:39:11.571 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:11 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-b[20275]: 2024-09-02T10:39:11.201+0000 7f2b75a0b640 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-02T10:39:11.571 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:11 smithi162 podman[170865]: 2024-09-02 10:39:11.252964066 +0000 UTC m=+0.146012712 container died aec4e088fc3b9f4847502f7018b50f60bb451321b0deebe722ca4ee61a2cb15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-09-02T10:39:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:11 smithi162 podman[170865]: 2024-09-02 10:39:11.660088581 +0000 UTC m=+0.553137224 container cleanup aec4e088fc3b9f4847502f7018b50f60bb451321b0deebe722ca4ee61a2cb15f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-02T10:39:11.845 INFO:journalctl@ceph.mon.b.smithi162.stdout:Sep 02 10:39:11 smithi162 bash[170865]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-b 2024-09-02T10:39:12.040 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.b.service' 2024-09-02T10:39:12.086 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:39:12.086 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-09-02T10:39:12.087 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-09-02T10:39:12.087 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.c 2024-09-02T10:39:12.434 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:12 smithi187 systemd[1]: Stopping Ceph mon.c for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-c[20068]: 2024-09-02T10:39:12.432+0000 7f0a74343640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:39:12.756 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-c[20068]: 2024-09-02T10:39:12.432+0000 7f0a74343640 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-02T10:39:12.757 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:12 smithi187 podman[166844]: 2024-09-02 10:39:12.524188928 +0000 UTC m=+0.200309555 container died b9415a1035d30d14c4ea0b57e4707050a6160536546f7cbeabe852acd81355af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-02T10:39:13.093 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:12 smithi187 podman[166844]: 2024-09-02 10:39:12.866060282 +0000 UTC m=+0.542180952 container cleanup b9415a1035d30d14c4ea0b57e4707050a6160536546f7cbeabe852acd81355af (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_BRANCH=HEAD) 2024-09-02T10:39:13.093 INFO:journalctl@ceph.mon.c.smithi187.stdout:Sep 02 10:39:12 smithi187 bash[166844]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mon-c 2024-09-02T10:39:13.303 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mon.c.service' 2024-09-02T10:39:13.349 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:39:13.349 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-09-02T10:39:13.350 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-02T10:39:13.350 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mgr.y 2024-09-02T10:39:13.688 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 10:39:13 smithi151 systemd[1]: Stopping Ceph mgr.y for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:14.014 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 10:39:13 smithi151 podman[274780]: 2024-09-02 10:39:13.702802528 +0000 UTC m=+0.159257479 container died af37bedd602803253f07228280565f8fa40ee1b0781feac509fd2a9e874c976c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-02T10:39:14.425 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 10:39:14 smithi151 podman[274780]: 2024-09-02 10:39:14.214456098 +0000 UTC m=+0.670911040 container cleanup af37bedd602803253f07228280565f8fa40ee1b0781feac509fd2a9e874c976c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:82daa6ef0815320144b25b212c6b8c25a1d59c07, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-02T10:39:14.425 INFO:journalctl@ceph.mgr.y.smithi151.stdout:Sep 02 10:39:14 smithi151 bash[274780]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-y 2024-09-02T10:39:14.552 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mgr.y.service' 2024-09-02T10:39:14.587 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:39:14.587 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-02T10:39:14.588 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-02T10:39:14.588 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mgr.x 2024-09-02T10:39:14.903 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 10:39:14 smithi162 systemd[1]: Stopping Ceph mgr.x for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:15.183 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 10:39:14 smithi162 podman[170988]: 2024-09-02 10:39:14.902470925 +0000 UTC m=+0.126646795 container died 4c2ca83c89da3997c37a50376989e34c9dd89a0352060f75947cbedcbafdfd33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T10:39:15.525 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 10:39:15 smithi162 podman[170988]: 2024-09-02 10:39:15.257011359 +0000 UTC m=+0.481187228 container cleanup 4c2ca83c89da3997c37a50376989e34c9dd89a0352060f75947cbedcbafdfd33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-x, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T10:39:15.526 INFO:journalctl@ceph.mgr.x.smithi162.stdout:Sep 02 10:39:15 smithi162 bash[170988]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-mgr-x 2024-09-02T10:39:15.678 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@mgr.x.service' 2024-09-02T10:39:15.712 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:39:15.712 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-02T10:39:15.712 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-02T10:39:15.712 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.0 2024-09-02T10:39:16.012 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:15 smithi151 systemd[1]: Stopping Ceph osd.0 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:16.438 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:16 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0[30637]: 2024-09-02T10:39:16.009+0000 7f07ded1d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:39:16.438 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:16 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0[30637]: 2024-09-02T10:39:16.009+0000 7f07ded1d640 -1 osd.0 1042 *** Got signal Terminated *** 2024-09-02T10:39:16.438 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:16 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0[30637]: 2024-09-02T10:39:16.009+0000 7f07ded1d640 -1 osd.0 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:39:21.688 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:21 smithi151 podman[274901]: 2024-09-02 10:39:21.433338135 +0000 UTC m=+5.528685296 container died d5de177c8a7d63456f633eb5769a1c97702af316570ef57e5fe0cb1e71384df3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-02T10:39:22.129 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:21 smithi151 podman[274901]: 2024-09-02 10:39:21.952343873 +0000 UTC m=+6.047691021 container cleanup d5de177c8a7d63456f633eb5769a1c97702af316570ef57e5fe0cb1e71384df3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T10:39:22.129 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:21 smithi151 bash[274901]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0 2024-09-02T10:39:22.438 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:22 smithi151 podman[274915]: 2024-09-02 10:39:22.128828674 +0000 UTC m=+0.693026338 container remove d5de177c8a7d63456f633eb5769a1c97702af316570ef57e5fe0cb1e71384df3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-02T10:39:22.875 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:22 smithi151 podman[275006]: 2024-09-02 10:39:22.468325741 +0000 UTC m=+0.015792557 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:39:22.875 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:22 smithi151 podman[275006]: 2024-09-02 10:39:22.58801662 +0000 UTC m=+0.135483436 container create 1b28c216a3c78713cf10bf460df114a3d398dece86bb3c4510e5bb65ddd8da2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-02T10:39:22.875 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:22 smithi151 podman[275006]: 2024-09-02 10:39:22.790313212 +0000 UTC m=+0.337780029 container init 1b28c216a3c78713cf10bf460df114a3d398dece86bb3c4510e5bb65ddd8da2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T10:39:22.875 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:22 smithi151 podman[275006]: 2024-09-02 10:39:22.793652113 +0000 UTC m=+0.341118939 container start 1b28c216a3c78713cf10bf460df114a3d398dece86bb3c4510e5bb65ddd8da2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-02T10:39:22.875 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:22 smithi151 podman[275006]: 2024-09-02 10:39:22.873660659 +0000 UTC m=+0.421127480 container attach 1b28c216a3c78713cf10bf460df114a3d398dece86bb3c4510e5bb65ddd8da2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True) 2024-09-02T10:39:23.188 INFO:journalctl@ceph.osd.0.smithi151.stdout:Sep 02 10:39:23 smithi151 podman[275006]: 2024-09-02 10:39:23.037018395 +0000 UTC m=+0.584485219 container died 1b28c216a3c78713cf10bf460df114a3d398dece86bb3c4510e5bb65ddd8da2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-09-02T10:39:23.705 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.0.service' 2024-09-02T10:39:23.746 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:39:23.746 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-02T10:39:23.746 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-02T10:39:23.747 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.1 2024-09-02T10:39:23.938 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:23 smithi151 systemd[1]: Stopping Ceph osd.1 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:24.438 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:24 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1[36292]: 2024-09-02T10:39:24.087+0000 7f52b944a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:39:24.438 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:24 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1[36292]: 2024-09-02T10:39:24.087+0000 7f52b944a640 -1 osd.1 1042 *** Got signal Terminated *** 2024-09-02T10:39:24.438 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:24 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1[36292]: 2024-09-02T10:39:24.087+0000 7f52b944a640 -1 osd.1 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:39:30.023 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:29 smithi151 podman[275113]: 2024-09-02 10:39:29.760973924 +0000 UTC m=+5.786036804 container died 16fac4dc87253ec3d9eeb9c8f990c28803f019b25d039439457b2c44be67500c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-02T10:39:30.407 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:30 smithi151 podman[275113]: 2024-09-02 10:39:30.196903542 +0000 UTC m=+6.221966456 container cleanup 16fac4dc87253ec3d9eeb9c8f990c28803f019b25d039439457b2c44be67500c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-02T10:39:30.407 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:30 smithi151 bash[275113]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1 2024-09-02T10:39:30.688 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:30 smithi151 podman[275128]: 2024-09-02 10:39:30.406606963 +0000 UTC m=+0.646109739 container remove 16fac4dc87253ec3d9eeb9c8f990c28803f019b25d039439457b2c44be67500c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-02T10:39:31.060 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:30 smithi151 podman[275215]: 2024-09-02 10:39:30.701072748 +0000 UTC m=+0.015567080 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:39:31.061 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:30 smithi151 podman[275215]: 2024-09-02 10:39:30.824143489 +0000 UTC m=+0.138637820 container create f53478c4302800f291d17e308e753c13becfbafc86bc813b6816536bacbba2e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-02T10:39:31.061 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:31 smithi151 podman[275215]: 2024-09-02 10:39:31.060006629 +0000 UTC m=+0.374500960 container init f53478c4302800f291d17e308e753c13becfbafc86bc813b6816536bacbba2e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-09-02T10:39:31.438 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:31 smithi151 podman[275215]: 2024-09-02 10:39:31.063522981 +0000 UTC m=+0.378017313 container start f53478c4302800f291d17e308e753c13becfbafc86bc813b6816536bacbba2e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1-deactivate, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T10:39:31.438 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:31 smithi151 podman[275215]: 2024-09-02 10:39:31.143074577 +0000 UTC m=+0.457568908 container attach f53478c4302800f291d17e308e753c13becfbafc86bc813b6816536bacbba2e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-02T10:39:31.438 INFO:journalctl@ceph.osd.1.smithi151.stdout:Sep 02 10:39:31 smithi151 podman[275215]: 2024-09-02 10:39:31.258939784 +0000 UTC m=+0.573434119 container died f53478c4302800f291d17e308e753c13becfbafc86bc813b6816536bacbba2e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-02T10:39:31.931 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.1.service' 2024-09-02T10:39:31.969 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:39:31.970 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-02T10:39:31.970 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-02T10:39:31.970 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.2 2024-09-02T10:39:32.188 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:32 smithi151 systemd[1]: Stopping Ceph osd.2 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:32.688 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:32 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2[42108]: 2024-09-02T10:39:32.298+0000 7fb381e2d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:39:32.688 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:32 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2[42108]: 2024-09-02T10:39:32.298+0000 7fb381e2d640 -1 osd.2 1042 *** Got signal Terminated *** 2024-09-02T10:39:32.688 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:32 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2[42108]: 2024-09-02T10:39:32.298+0000 7fb381e2d640 -1 osd.2 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:39:38.168 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:37 smithi151 podman[275316]: 2024-09-02 10:39:37.883775127 +0000 UTC m=+5.685426630 container died 0622e247602eadb69c4c664cf3ecad5dd7c2c7f5da15a529dd7e970ee61d3350 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-02T10:39:38.438 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:38 smithi151 podman[275316]: 2024-09-02 10:39:38.374854185 +0000 UTC m=+6.176505673 container cleanup 0622e247602eadb69c4c664cf3ecad5dd7c2c7f5da15a529dd7e970ee61d3350 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-02T10:39:38.438 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:38 smithi151 bash[275316]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2 2024-09-02T10:39:38.926 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:38 smithi151 podman[275334]: 2024-09-02 10:39:38.617869781 +0000 UTC m=+0.728471373 container remove 0622e247602eadb69c4c664cf3ecad5dd7c2c7f5da15a529dd7e970ee61d3350 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-02T10:39:39.188 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:38 smithi151 podman[275420]: 2024-09-02 10:39:38.828486865 +0000 UTC m=+0.015951102 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:39:39.188 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:38 smithi151 podman[275420]: 2024-09-02 10:39:38.951998394 +0000 UTC m=+0.139462631 container create c2c1439470200e8ba339f40ec35a967a233b7363e88df638d66df8b2fde3995b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-09-02T10:39:39.188 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:39 smithi151 podman[275420]: 2024-09-02 10:39:39.162896188 +0000 UTC m=+0.350360428 container init c2c1439470200e8ba339f40ec35a967a233b7363e88df638d66df8b2fde3995b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-02T10:39:39.188 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:39 smithi151 podman[275420]: 2024-09-02 10:39:39.166246441 +0000 UTC m=+0.353710684 container start c2c1439470200e8ba339f40ec35a967a233b7363e88df638d66df8b2fde3995b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T10:39:39.688 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:39 smithi151 podman[275420]: 2024-09-02 10:39:39.296298161 +0000 UTC m=+0.483762408 container attach c2c1439470200e8ba339f40ec35a967a233b7363e88df638d66df8b2fde3995b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T10:39:39.688 INFO:journalctl@ceph.osd.2.smithi151.stdout:Sep 02 10:39:39 smithi151 podman[275420]: 2024-09-02 10:39:39.357281989 +0000 UTC m=+0.544746227 container died c2c1439470200e8ba339f40ec35a967a233b7363e88df638d66df8b2fde3995b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T10:39:40.132 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.2.service' 2024-09-02T10:39:40.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:39:40.173 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-02T10:39:40.173 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-02T10:39:40.173 DEBUG:teuthology.orchestra.run.smithi151:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.3 2024-09-02T10:39:40.383 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:40 smithi151 systemd[1]: Stopping Ceph osd.3 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:40.688 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3[47996]: 2024-09-02T10:39:40.502+0000 7fdb28ce3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:39:40.688 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3[47996]: 2024-09-02T10:39:40.502+0000 7fdb28ce3640 -1 osd.3 1042 *** Got signal Terminated *** 2024-09-02T10:39:40.688 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:40 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3[47996]: 2024-09-02T10:39:40.502+0000 7fdb28ce3640 -1 osd.3 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:39:42.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:42 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:42.491+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:42.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:42 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:42.421+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:42.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:42 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:42.404+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:43.649 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:43 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:43.535+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:43.650 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:43 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:43.389+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:43.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:43 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:43.357+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:44.006 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:43 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:43.648+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:44.595 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:44 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:44.309+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:44.595 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:44 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:44.332+0000 7f5fdf34c640 -1 osd.4 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.227008+0000 front 2024-09-02T10:39:19.227164+0000 (oldest deadline 2024-09-02T10:39:43.926913+0000) 2024-09-02T10:39:44.596 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:44 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:44.520+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:44.698 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:44.567+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:44.698 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:44.396+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:45.006 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:44.696+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:45.006 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:44.817+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:45.438 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:45 smithi151 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3[47996]: 2024-09-02T10:39:45.072+0000 7fdb212f0640 -1 osd.3 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.039834+0000 front 2024-09-02T10:39:20.039916+0000 (oldest deadline 2024-09-02T10:39:44.139943+0000) 2024-09-02T10:39:45.595 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:45 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:45.326+0000 7f5fdf34c640 -1 osd.4 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.227008+0000 front 2024-09-02T10:39:19.227164+0000 (oldest deadline 2024-09-02T10:39:43.926913+0000) 2024-09-02T10:39:45.595 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:45 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:45.305+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:45.595 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:45 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:45.484+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:45.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:45.663+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:45.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:45.611+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:45.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:45.446+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:46.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:45.831+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:46.412 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:46 smithi151 podman[275521]: 2024-09-02 10:39:46.114799405 +0000 UTC m=+5.719847888 container died 6961140c15c825e9b96826be793d3a662fb5d7e73b5a85f7e1d250efd790b81a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-02T10:39:46.595 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:46 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:46.291+0000 7f5fdf34c640 -1 osd.4 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.227008+0000 front 2024-09-02T10:39:19.227164+0000 (oldest deadline 2024-09-02T10:39:43.926913+0000) 2024-09-02T10:39:46.595 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:46 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:46.476+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:46.596 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:46 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:46.340+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:46.687 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:46 smithi151 podman[275521]: 2024-09-02 10:39:46.58575749 +0000 UTC m=+6.190805972 container cleanup 6961140c15c825e9b96826be793d3a662fb5d7e73b5a85f7e1d250efd790b81a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-02T10:39:46.688 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:46 smithi151 bash[275521]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3 2024-09-02T10:39:46.698 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:46.605+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:46.698 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:46.433+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:47.006 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:46.864+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:47.006 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:46.697+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:47.188 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:46 smithi151 podman[275550]: 2024-09-02 10:39:46.845581681 +0000 UTC m=+0.734123873 container remove 6961140c15c825e9b96826be793d3a662fb5d7e73b5a85f7e1d250efd790b81a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-02T10:39:47.457 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:47 smithi151 podman[275636]: 2024-09-02 10:39:47.094596009 +0000 UTC m=+0.015910212 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:39:47.457 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:47 smithi151 podman[275636]: 2024-09-02 10:39:47.238025485 +0000 UTC m=+0.159339729 container create 6f504b74fb9c10ddae1626043be892bc480eda186f9683c2b72a680fdfec690a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-02T10:39:47.595 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:47 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:47.274+0000 7f5fdf34c640 -1 osd.4 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.227008+0000 front 2024-09-02T10:39:19.227164+0000 (oldest deadline 2024-09-02T10:39:43.926913+0000) 2024-09-02T10:39:47.595 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:47 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:47.364+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:47.595 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:47 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:47.508+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:47.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:47.654+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:47.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:47.637+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:47.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:47.446+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:47.935 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:47 smithi151 podman[275636]: 2024-09-02 10:39:47.456988123 +0000 UTC m=+0.378302322 container init 6f504b74fb9c10ddae1626043be892bc480eda186f9683c2b72a680fdfec690a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-02T10:39:47.935 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:47 smithi151 podman[275636]: 2024-09-02 10:39:47.460841322 +0000 UTC m=+0.382155530 container start 6f504b74fb9c10ddae1626043be892bc480eda186f9683c2b72a680fdfec690a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-02T10:39:47.935 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:47 smithi151 podman[275636]: 2024-09-02 10:39:47.540234772 +0000 UTC m=+0.461548978 container attach 6f504b74fb9c10ddae1626043be892bc480eda186f9683c2b72a680fdfec690a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T10:39:47.935 INFO:journalctl@ceph.osd.3.smithi151.stdout:Sep 02 10:39:47 smithi151 podman[275636]: 2024-09-02 10:39:47.646434766 +0000 UTC m=+0.567748964 container died 6f504b74fb9c10ddae1626043be892bc480eda186f9683c2b72a680fdfec690a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-3-deactivate, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-02T10:39:48.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:47.863+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:48.296 DEBUG:teuthology.orchestra.run.smithi151:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.3.service' 2024-09-02T10:39:48.324 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:48 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:48.230+0000 7f5fdf34c640 -1 osd.4 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.227008+0000 front 2024-09-02T10:39:19.227164+0000 (oldest deadline 2024-09-02T10:39:43.926913+0000) 2024-09-02T10:39:48.324 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:48 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:48.040+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:48.336 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:39:48.336 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-02T10:39:48.336 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-02T10:39:48.336 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.4 2024-09-02T10:39:48.595 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:48 smithi162 systemd[1]: Stopping Ceph osd.4 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:48.595 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:48 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:48.321+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:48.595 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:48 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:48.550+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:48.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:48.666+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:48.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:48.608+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:48.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:48.485+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:49.050 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:48 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:48.647+0000 7f5fe2d37640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:39:49.050 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:48 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:48.647+0000 7f5fe2d37640 -1 osd.4 1042 *** Got signal Terminated *** 2024-09-02T10:39:49.050 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:48 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:48.647+0000 7f5fe2d37640 -1 osd.4 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:39:49.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:48.903+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:49.313 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:49 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:49.274+0000 7f5fdf34c640 -1 osd.4 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.227008+0000 front 2024-09-02T10:39:19.227164+0000 (oldest deadline 2024-09-02T10:39:43.926913+0000) 2024-09-02T10:39:49.313 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:49 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:49.048+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:49.595 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:49 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:49.311+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:49.595 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:49 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:49.560+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:49.876 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:49.587+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:49.877 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:49.689+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:49.877 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:49.514+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:50.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:49.875+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:50.345 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:50 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:50.256+0000 7f5fdf34c640 -1 osd.4 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.227008+0000 front 2024-09-02T10:39:19.227164+0000 (oldest deadline 2024-09-02T10:39:43.926913+0000) 2024-09-02T10:39:50.345 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:50 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:50.319+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:50.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:50 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:50.090+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:50.888 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:50 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:50.658+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:50.888 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:50 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:50.568+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:50.888 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:50 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:50.525+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:51.063 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:50 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:50.610+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:51.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:50 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:50.887+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:51.324 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:51 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:51.217+0000 7f5fdf34c640 -1 osd.4 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.227008+0000 front 2024-09-02T10:39:19.227164+0000 (oldest deadline 2024-09-02T10:39:43.926913+0000) 2024-09-02T10:39:51.325 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:51 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:51.062+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:51.595 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:51 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:51.323+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:51.937 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:51 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:51.572+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:51.938 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:51 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:51.519+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:51.938 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:51 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:51.639+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:51.938 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:51 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:51.639+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:39:52.081 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:51 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:51.653+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:52.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:51 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:51.936+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:52.345 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:52 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:52.176+0000 7f5fdf34c640 -1 osd.4 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.227008+0000 front 2024-09-02T10:39:19.227164+0000 (oldest deadline 2024-09-02T10:39:43.926913+0000) 2024-09-02T10:39:52.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:52 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:52.079+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:52.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:52 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:52.079+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:39:52.655 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:52 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:52.359+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:52.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:52 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:52.486+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:52.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:52 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:52.486+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:39:52.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:52 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:52.620+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:52.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:52 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:52.620+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:39:52.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:52 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:52.532+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:52.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:52 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:52.532+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:39:53.059 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:52 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:52.654+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:53.060 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:52 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:52.654+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:39:53.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:52 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:52.904+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:53.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:52 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:52.904+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:39:53.345 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:53 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4[24600]: 2024-09-02T10:39:53.148+0000 7f5fdf34c640 -1 osd.4 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.227008+0000 front 2024-09-02T10:39:19.227164+0000 (oldest deadline 2024-09-02T10:39:43.926913+0000) 2024-09-02T10:39:53.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:53 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:53.058+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:53.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:53 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:53.058+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:39:53.634 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:53 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:53.380+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:53.635 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:53 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:53.380+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.457294+0000 front 2024-09-02T10:39:27.457369+0000 (oldest deadline 2024-09-02T10:39:53.357313+0000) 2024-09-02T10:39:53.936 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:53 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:53.622+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:53.936 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:53 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:53.622+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:39:53.937 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:53 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:53.533+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:53.937 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:53 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:53.533+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:39:53.937 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:53 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:53.510+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:53.937 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:53 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:53.510+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:39:54.051 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:53 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:53.633+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:54.051 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:53 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:53.633+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:39:54.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:53 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:53.935+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:54.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:53 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:53.935+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:39:54.345 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:54 smithi162 podman[171124]: 2024-09-02 10:39:54.204105492 +0000 UTC m=+5.670734730 container died 430b9025429b9f5784d9d8e78f8f84eeb52d386441d92676eeefdfa3b03bf486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-02T10:39:54.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:54 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:54.049+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:54.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:54 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:54.049+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:39:54.632 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:54 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:54.622+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:54.632 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:54 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:54.622+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:39:54.632 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:54 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:54.380+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:54.633 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:54 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:54.380+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.457294+0000 front 2024-09-02T10:39:27.457369+0000 (oldest deadline 2024-09-02T10:39:53.357313+0000) 2024-09-02T10:39:54.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:54 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:54.480+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:54.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:54 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:54.480+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:39:54.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:54 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:54.596+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:54.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:54 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:54.596+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:39:54.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:54 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:54.524+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:54.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:54 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:54.524+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:39:55.003 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:54 smithi162 podman[171124]: 2024-09-02 10:39:54.631775164 +0000 UTC m=+6.098404404 container cleanup 430b9025429b9f5784d9d8e78f8f84eeb52d386441d92676eeefdfa3b03bf486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-02T10:39:55.003 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:54 smithi162 bash[171124]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4 2024-09-02T10:39:55.003 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:54 smithi162 podman[171140]: 2024-09-02 10:39:54.842053392 +0000 UTC m=+0.630078283 container remove 430b9025429b9f5784d9d8e78f8f84eeb52d386441d92676eeefdfa3b03bf486 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-02T10:39:55.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:54 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:54.960+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:55.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:54 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:54.960+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:39:55.268 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:55 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:55.001+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:55.268 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:55 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:55.001+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:39:55.269 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:55 smithi162 podman[171232]: 2024-09-02 10:39:55.144060426 +0000 UTC m=+0.016217059 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:39:55.269 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:55 smithi162 podman[171232]: 2024-09-02 10:39:55.268113021 +0000 UTC m=+0.140269686 container create d15004b5ba24aca957a812306c945d8a209416e01f10bc6343727a10615bc9fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-02T10:39:55.538 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:55 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:55.427+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:55.538 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:55 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:55.427+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.457294+0000 front 2024-09-02T10:39:27.457369+0000 (oldest deadline 2024-09-02T10:39:53.357313+0000) 2024-09-02T10:39:55.538 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:55 smithi162 podman[171232]: 2024-09-02 10:39:55.462163412 +0000 UTC m=+0.334320030 container init d15004b5ba24aca957a812306c945d8a209416e01f10bc6343727a10615bc9fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-02T10:39:55.538 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:55 smithi162 podman[171232]: 2024-09-02 10:39:55.465472213 +0000 UTC m=+0.337628835 container start d15004b5ba24aca957a812306c945d8a209416e01f10bc6343727a10615bc9fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-02T10:39:55.538 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:55 smithi162 podman[171232]: 2024-09-02 10:39:55.537156879 +0000 UTC m=+0.409313511 container attach d15004b5ba24aca957a812306c945d8a209416e01f10bc6343727a10615bc9fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True) 2024-09-02T10:39:55.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:55 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:55.452+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:55.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:55 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:55.452+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:39:55.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:55 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:55.592+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:55.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:55 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:55.592+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:39:55.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:55 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:55.479+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:55.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:55 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:55.479+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:39:55.845 INFO:journalctl@ceph.osd.4.smithi162.stdout:Sep 02 10:39:55 smithi162 podman[171232]: 2024-09-02 10:39:55.701216127 +0000 UTC m=+0.573372747 container died d15004b5ba24aca957a812306c945d8a209416e01f10bc6343727a10615bc9fd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-02T10:39:55.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:55 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:55.645+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:55.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:55 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:55.645+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:39:56.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:56 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:56.003+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:56.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:56 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:56.003+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:39:56.283 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:56.022+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:56.283 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:56.022+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:39:56.304 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.4.service' 2024-09-02T10:39:56.344 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:39:56.344 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-02T10:39:56.344 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-02T10:39:56.344 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.5 2024-09-02T10:39:56.555 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:56 smithi162 systemd[1]: Stopping Ceph osd.5 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:39:56.556 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:56.464+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:56.556 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:56.464+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.457294+0000 front 2024-09-02T10:39:27.457369+0000 (oldest deadline 2024-09-02T10:39:53.357313+0000) 2024-09-02T10:39:56.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:56 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:56.484+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:56.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:56 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:56.484+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:39:56.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:56 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:56.596+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:56.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:56 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:56.596+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:39:56.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:56 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:56.509+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:56.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:56 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:56.509+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:39:56.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:56.651+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:56.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:56.651+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:39:56.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:56.686+0000 7ff3555b6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:39:56.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:56.686+0000 7ff3555b6640 -1 osd.5 1042 *** Got signal Terminated *** 2024-09-02T10:39:56.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:56.686+0000 7ff3555b6640 -1 osd.5 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:39:57.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:56.993+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:57.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:56 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:56.993+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:39:57.487 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:57 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:57.030+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:57.487 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:57 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:57.030+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:39:57.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:57 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:57.592+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:57.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:57 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:57.592+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:39:57.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:57 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:57.486+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:57.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:57 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:57.486+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:39:57.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:57 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:57.552+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:57.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:57 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:57.552+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:39:57.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:57 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:57.502+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:57.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:57 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:57.502+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.457294+0000 front 2024-09-02T10:39:27.457369+0000 (oldest deadline 2024-09-02T10:39:53.357313+0000) 2024-09-02T10:39:57.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:57 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:57.690+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:57.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:57 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:57.690+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:39:58.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:57 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:57.996+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:58.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:57 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:57.996+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:39:58.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:58 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:58.047+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:58.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:58 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:58.047+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:39:58.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:58 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:58.525+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:58.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:58 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:58.525+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.457294+0000 front 2024-09-02T10:39:27.457369+0000 (oldest deadline 2024-09-02T10:39:53.357313+0000) 2024-09-02T10:39:58.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:58 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:58.730+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:58.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:58 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:58.730+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:39:59.005 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:58 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:58.566+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:59.005 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:58 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:58.566+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:39:59.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:58 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:58.515+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:59.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:58 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:58.515+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:39:59.006 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:58 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:58.579+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:59.006 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:58 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:58.579+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:39:59.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:59.004+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:39:59.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:39:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:39:59.004+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:39:59.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:59.012+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:39:59.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:59.012+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:39:59.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:59.012+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:39:59.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:59.522+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:39:59.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:59.522+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:39:59.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:39:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:39:59.522+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:39:59.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:59.483+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:39:59.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:59.483+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:39:59.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:39:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:39:59.483+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:39:59.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:59.581+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:39:59.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:39:59 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:39:59.581+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:39:59.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:59.564+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:39:59.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:59.564+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.457294+0000 front 2024-09-02T10:39:27.457369+0000 (oldest deadline 2024-09-02T10:39:53.357313+0000) 2024-09-02T10:39:59.845 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:39:59.564+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.357917+0000 front 2024-09-02T10:39:33.357867+0000 (oldest deadline 2024-09-02T10:39:58.658029+0000) 2024-09-02T10:39:59.846 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:59.730+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:39:59.846 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:59.730+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:39:59.846 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:39:59.730+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:00.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:59.976+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:00.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:59.976+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:00.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:39:59 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:39:59.976+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:00.481 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:00.040+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:00.481 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:00.040+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:00.751 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:00 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:40:00.549+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:40:00.751 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:00 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:40:00.549+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.457294+0000 front 2024-09-02T10:39:27.457369+0000 (oldest deadline 2024-09-02T10:39:53.357313+0000) 2024-09-02T10:40:00.751 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:00 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:40:00.549+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.357917+0000 front 2024-09-02T10:39:33.357867+0000 (oldest deadline 2024-09-02T10:39:58.658029+0000) 2024-09-02T10:40:00.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:00.505+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:00.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:00.505+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:00.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:00.505+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:00.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:00.564+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:00.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:00.564+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:00.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:00.564+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:00.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:00.606+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:00.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:00.606+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:00.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:00 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:00.606+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:01.095 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:00 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:00.980+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:01.095 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:00 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:00.980+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:01.095 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:00 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:00.980+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:01.095 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:00 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:00.749+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:40:01.096 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:00 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:00.749+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:40:01.096 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:00 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:00.749+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:01.467 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:01.080+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:01.467 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:01.080+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:01.595 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:40:01.551+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.256264+0000 front 2024-09-02T10:39:19.256034+0000 (oldest deadline 2024-09-02T10:39:42.156125+0000) 2024-09-02T10:40:01.595 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:40:01.551+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.457294+0000 front 2024-09-02T10:39:27.457369+0000 (oldest deadline 2024-09-02T10:39:53.357313+0000) 2024-09-02T10:40:01.596 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5[30145]: 2024-09-02T10:40:01.551+0000 7ff34dbc3640 -1 osd.5 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.357917+0000 front 2024-09-02T10:39:33.357867+0000 (oldest deadline 2024-09-02T10:39:58.658029+0000) 2024-09-02T10:40:01.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:01.542+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:01.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:01.542+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:01.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:01.542+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:01.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:01.466+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:01.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:01.466+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:01.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:01.466+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:01.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:01.581+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:01.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:01.581+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:01.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:01 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:01.581+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:01.986 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:01.702+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:40:01.986 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:01.702+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:40:01.986 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:01.702+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:02.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:01.984+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:02.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:01.984+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:02.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:01 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:01.984+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:02.504 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:02.038+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:02.504 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:02.038+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:02.680 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:02 smithi162 podman[171336]: 2024-09-02 10:40:02.411499822 +0000 UTC m=+5.838862886 container died 8b8c0b9200dc3267e2a58cd1e63d0794fbc83bc22c2d8b005368443ea5e55cad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-02T10:40:02.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:02.503+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:02.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:02.503+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:02.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:02.503+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:02.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:02.572+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:02.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:02.572+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:02.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:02.572+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:02.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:02.533+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:02.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:02.533+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:02.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:02 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:02.533+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:02.951 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:02 smithi162 podman[171336]: 2024-09-02 10:40:02.810705748 +0000 UTC m=+6.238068811 container cleanup 8b8c0b9200dc3267e2a58cd1e63d0794fbc83bc22c2d8b005368443ea5e55cad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-02T10:40:02.951 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:02 smithi162 bash[171336]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5 2024-09-02T10:40:02.951 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:02 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:02.678+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:40:02.951 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:02 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:02.678+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:40:02.951 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:02 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:02.678+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:03.345 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:03 smithi162 podman[171365]: 2024-09-02 10:40:03.053993649 +0000 UTC m=+0.635959836 container remove 8b8c0b9200dc3267e2a58cd1e63d0794fbc83bc22c2d8b005368443ea5e55cad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-02T10:40:03.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:02 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:02.949+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:03.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:02 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:02.949+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:03.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:02 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:02.949+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:03.468 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:03.024+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:03.468 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:03.024+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:03.700 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:03 smithi162 podman[171452]: 2024-09-02 10:40:03.277922881 +0000 UTC m=+0.016128402 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:40:03.700 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:03 smithi162 podman[171452]: 2024-09-02 10:40:03.388223879 +0000 UTC m=+0.126429399 container create 314bf94cd30907d75d04766f5a3e3e11b4da07aedb589b8773f1c0dd0592c8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS) 2024-09-02T10:40:03.700 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:03 smithi162 podman[171452]: 2024-09-02 10:40:03.616021383 +0000 UTC m=+0.354226904 container init 314bf94cd30907d75d04766f5a3e3e11b4da07aedb589b8773f1c0dd0592c8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-09-02T10:40:03.700 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:03 smithi162 podman[171452]: 2024-09-02 10:40:03.619447167 +0000 UTC m=+0.357652695 container start 314bf94cd30907d75d04766f5a3e3e11b4da07aedb589b8773f1c0dd0592c8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS) 2024-09-02T10:40:03.700 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:03 smithi162 podman[171452]: 2024-09-02 10:40:03.699346145 +0000 UTC m=+0.437551674 container attach 314bf94cd30907d75d04766f5a3e3e11b4da07aedb589b8773f1c0dd0592c8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T10:40:03.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:03.466+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:03.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:03.466+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:03.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:03.466+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:03.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:03.585+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:03.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:03.585+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:03.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:03.585+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:03.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:03.561+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:03.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:03.561+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:03.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:03.561+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:03.967 INFO:journalctl@ceph.osd.5.smithi162.stdout:Sep 02 10:40:03 smithi162 podman[171452]: 2024-09-02 10:40:03.809255753 +0000 UTC m=+0.547461275 container died 314bf94cd30907d75d04766f5a3e3e11b4da07aedb589b8773f1c0dd0592c8ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-5-deactivate, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T10:40:03.968 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:03 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:03.706+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:40:03.968 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:03 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:03.706+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:40:03.968 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:03 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:03.706+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:04.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:03.989+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:04.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:03 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:03.989+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:04.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:03 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:03.966+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:04.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:03 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:03.966+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:04.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:03 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:03.966+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:04.410 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.5.service' 2024-09-02T10:40:04.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:40:04.449 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-02T10:40:04.449 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-02T10:40:04.449 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.6 2024-09-02T10:40:04.666 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:04 smithi162 systemd[1]: Stopping Ceph osd.6 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:40:04.667 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:04 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:04.665+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:40:04.667 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:04 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:04.665+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:40:04.667 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:04 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:04.665+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:04.949 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:04 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:04.764+0000 7f533c7b6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:40:04.949 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:04 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:04.764+0000 7f533c7b6640 -1 osd.6 1042 *** Got signal Terminated *** 2024-09-02T10:40:04.949 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:04 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:04.764+0000 7f533c7b6640 -1 osd.6 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:40:05.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:04 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:04.511+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:05.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:04 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:04.511+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:05.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:04 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:04.511+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:05.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:04 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:04.545+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:05.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:04 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:04.545+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:05.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:04 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:04.545+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:05.007 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:04 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:04.527+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:05.007 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:04 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:04.527+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:05.007 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:04 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:04.527+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:05.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:04 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:04.947+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:05.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:04 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:04.947+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:05.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:04 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:04.947+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:05.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:05.031+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:05.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:05.031+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:05.982 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:05 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:05.673+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:40:05.982 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:05 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:05.673+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:40:05.982 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:05 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:05.673+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:06.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:05.516+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:06.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:05.516+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:06.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:05.516+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:06.006 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:05.510+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:06.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:05.510+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:06.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:05.510+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:06.007 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:05.515+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:06.007 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:05.515+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:06.007 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:05 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:05.515+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:06.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:05 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:05.980+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:06.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:05 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:05.980+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:06.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:05 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:05.980+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:06.485 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:06.009+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:06.485 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:06.009+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:06.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:06.484+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:06.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:06.484+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:06.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:06.484+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:06.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:06.544+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:06.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:06.544+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:06.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:06.544+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:06.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:06.544+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:06.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:06.490+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:06.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:06.490+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:06.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:06 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:06.490+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:06.979 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:06 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:06.672+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:40:06.979 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:06 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:06.672+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:40:06.979 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:06 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:06.672+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:07.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:06 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:06.977+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:07.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:06 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:06.977+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:07.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:06 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:06.977+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:07.482 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:07.012+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:07.482 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:07.012+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:07.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:07.497+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:07.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:07.497+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:07.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:07.497+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:07.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:07.509+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:07.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:07.509+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:07.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:07.509+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:07.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:07.509+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:07.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:07.481+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:07.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:07.481+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:07.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:07 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:07.481+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:07.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:07 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:07.693+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:40:07.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:07 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:07.693+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:40:07.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:07 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:07.693+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:07.845 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:07 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:07.693+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:41.608661+0000 front 2024-09-02T10:39:41.608821+0000 (oldest deadline 2024-09-02T10:40:06.908700+0000) 2024-09-02T10:40:08.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:07 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:07.985+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:08.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:07 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:07.985+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:08.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:07 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:07.985+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:08.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:07 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:07.985+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.229176+0000 front 2024-09-02T10:39:44.229190+0000 (oldest deadline 2024-09-02T10:40:07.129195+0000) 2024-09-02T10:40:08.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:08.019+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:08.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:08.019+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:08.998 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:08.512+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:08.998 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:08.512+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:08.998 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:08.512+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:08.998 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:08.529+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:08.998 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:08.529+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:08.998 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:08.529+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:08.998 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:08.529+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:08.999 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:08.511+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:08.999 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:08.511+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:08.999 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:08.511+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:09.025 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:08 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:08.699+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:40:09.025 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:08 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:08.699+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:40:09.025 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:08 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:08.699+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:09.025 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:08 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:08.699+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:41.608661+0000 front 2024-09-02T10:39:41.608821+0000 (oldest deadline 2024-09-02T10:40:06.908700+0000) 2024-09-02T10:40:09.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:08.996+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:09.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:08 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:08.996+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:09.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:09.024+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:09.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:09.024+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:09.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:09.024+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:09.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:09.024+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.229176+0000 front 2024-09-02T10:39:44.229190+0000 (oldest deadline 2024-09-02T10:40:07.129195+0000) 2024-09-02T10:40:09.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:09.466+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:09.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:09.466+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:09.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:09.466+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:09.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:09.466+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:09.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:09.512+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:09.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:09.512+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:09.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:09.512+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:09.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:09.512+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:09.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:09.474+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:09.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:09.474+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:09.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:09.474+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:09.976 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:09.676+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:17.903983+0000 front 2024-09-02T10:39:17.904076+0000 (oldest deadline 2024-09-02T10:39:44.304511+0000) 2024-09-02T10:40:09.976 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:09.676+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.605775+0000 front 2024-09-02T10:39:26.605710+0000 (oldest deadline 2024-09-02T10:39:51.806069+0000) 2024-09-02T10:40:09.976 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:09.676+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.506967+0000 front 2024-09-02T10:39:33.507073+0000 (oldest deadline 2024-09-02T10:39:59.307580+0000) 2024-09-02T10:40:09.976 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6[35795]: 2024-09-02T10:40:09.676+0000 7f5334dc3640 -1 osd.6 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:41.608661+0000 front 2024-09-02T10:39:41.608821+0000 (oldest deadline 2024-09-02T10:40:06.908700+0000) 2024-09-02T10:40:10.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:09.961+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:10.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:09.961+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:10.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:09 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:09.961+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:10.345 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:10 smithi162 podman[171554]: 2024-09-02 10:40:10.173163966 +0000 UTC m=+5.505447574 container died 8f66cd61af64a8c309f05783b7fc5a62f7e356e5547ec40ae5e3999909f77a5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-02T10:40:10.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:09.975+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:10.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:09.975+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:10.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:09.975+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:10.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:09 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:09.975+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.229176+0000 front 2024-09-02T10:39:44.229190+0000 (oldest deadline 2024-09-02T10:40:07.129195+0000) 2024-09-02T10:40:10.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:10.481+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:10.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:10.481+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:10.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:10.481+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:10.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:10.481+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:10.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:10.491+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:10.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:10.491+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:10.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:10.491+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:10.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:10.491+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:10.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:10.491+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:10.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:10.491+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:10.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:10.491+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:10.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:10 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:10.491+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:10.808 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:10 smithi162 podman[171554]: 2024-09-02 10:40:10.589738096 +0000 UTC m=+5.922021702 container cleanup 8f66cd61af64a8c309f05783b7fc5a62f7e356e5547ec40ae5e3999909f77a5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-02T10:40:10.808 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:10 smithi162 bash[171554]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6 2024-09-02T10:40:11.095 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:10 smithi162 podman[171571]: 2024-09-02 10:40:10.807704324 +0000 UTC m=+0.635150163 container remove 8f66cd61af64a8c309f05783b7fc5a62f7e356e5547ec40ae5e3999909f77a5d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-02T10:40:11.095 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:10 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:10.992+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:11.095 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:10 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:10.992+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:11.095 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:10 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:10.992+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:11.095 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:10 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:10.992+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.229176+0000 front 2024-09-02T10:39:44.229190+0000 (oldest deadline 2024-09-02T10:40:07.129195+0000) 2024-09-02T10:40:11.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:11.003+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:11.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:11.003+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:11.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:11.003+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:11.520 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:11 smithi162 podman[171657]: 2024-09-02 10:40:11.113885037 +0000 UTC m=+0.016010650 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:40:11.520 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:11 smithi162 podman[171657]: 2024-09-02 10:40:11.216876538 +0000 UTC m=+0.119002145 container create 5171016fd1c35be355db33dc5e0bcd8af0d20da37c4a5650a426d0a3906a3edd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-02T10:40:11.520 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:11 smithi162 podman[171657]: 2024-09-02 10:40:11.410914176 +0000 UTC m=+0.313039784 container init 5171016fd1c35be355db33dc5e0bcd8af0d20da37c4a5650a426d0a3906a3edd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-02T10:40:11.520 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:11 smithi162 podman[171657]: 2024-09-02 10:40:11.414938008 +0000 UTC m=+0.317063618 container start 5171016fd1c35be355db33dc5e0bcd8af0d20da37c4a5650a426d0a3906a3edd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-02T10:40:11.520 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:11 smithi162 podman[171657]: 2024-09-02 10:40:11.519126956 +0000 UTC m=+0.421252566 container attach 5171016fd1c35be355db33dc5e0bcd8af0d20da37c4a5650a426d0a3906a3edd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-02T10:40:11.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:11.498+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:11.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:11.498+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:11.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:11.498+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:11.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:11.498+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:11.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:11.480+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:11.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:11.480+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:11.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:11.480+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:11.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:11.480+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:11.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:11.542+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:11.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:11.542+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:11.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:11.542+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:11.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:11 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:11.542+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:11.846 INFO:journalctl@ceph.osd.6.smithi162.stdout:Sep 02 10:40:11 smithi162 podman[171657]: 2024-09-02 10:40:11.610414169 +0000 UTC m=+0.512539778 container died 5171016fd1c35be355db33dc5e0bcd8af0d20da37c4a5650a426d0a3906a3edd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-02T10:40:12.173 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:11 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:11.993+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:12.174 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:11 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:11.993+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:12.174 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:11 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:11.993+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:12.174 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:11 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:11.993+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.229176+0000 front 2024-09-02T10:39:44.229190+0000 (oldest deadline 2024-09-02T10:40:07.129195+0000) 2024-09-02T10:40:12.190 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.6.service' 2024-09-02T10:40:12.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:40:12.228 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-02T10:40:12.229 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-02T10:40:12.229 DEBUG:teuthology.orchestra.run.smithi162:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.7 2024-09-02T10:40:12.500 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:12.005+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:12.501 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:12.005+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:12.501 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:12.005+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:12.552 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:12 smithi162 systemd[1]: Stopping Ceph osd.7 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:40:12.552 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:12 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:12.550+0000 7f09de168640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:40:12.552 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:12 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:12.550+0000 7f09de168640 -1 osd.7 1042 *** Got signal Terminated *** 2024-09-02T10:40:12.552 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:12 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:12.550+0000 7f09de168640 -1 osd.7 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:40:12.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:12.499+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:12.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:12.499+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:12.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:12.499+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:12.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:12.499+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:12.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:12.504+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:12.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:12.504+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:12.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:12.504+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:12.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:12.504+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:12.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:12.590+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:12.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:12.590+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:12.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:12.590+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:12.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:12.590+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:13.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:12.960+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:13.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:12.960+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:13.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:12 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:12.960+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:13.307 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:13 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:13.036+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:13.308 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:13 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:13.036+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:13.308 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:13 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:13.036+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:13.308 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:13 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:13.036+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.229176+0000 front 2024-09-02T10:39:44.229190+0000 (oldest deadline 2024-09-02T10:40:07.129195+0000) 2024-09-02T10:40:13.986 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:13.510+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:13.986 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:13.510+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:13.987 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:13.510+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:13.987 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:13.510+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:13.987 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:13.548+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:13.987 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:13.548+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:13.987 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:13.548+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:13.987 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:13.548+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:13.988 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:13.514+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:13.988 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:13.514+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:13.988 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:13.514+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:13.988 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:13.514+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:14.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:13.984+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:14.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:13.984+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:14.256 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:13 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:13.984+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:14.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:14 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:14.030+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:14.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:14 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:14.030+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:14.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:14 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:14.030+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:14.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:14 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:14.030+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.229176+0000 front 2024-09-02T10:39:44.229190+0000 (oldest deadline 2024-09-02T10:40:07.129195+0000) 2024-09-02T10:40:14.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:14.487+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:14.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:14.487+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:14.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:14.487+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:14.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:14.487+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:14.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:14.509+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:14.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:14.509+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:14.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:14.509+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:14.759 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:14.509+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:14.760 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:14.509+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:14.760 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:14.508+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:14.760 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:14.508+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:14.760 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:14.508+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:14.761 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:14 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:14.508+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:15.266 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:15.003+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:15.267 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:15.003+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:15.267 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:15.003+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:15.267 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:15.003+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:15.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:15 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:15.026+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:15.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:15 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:15.026+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:15.346 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:15 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:15.026+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:15.346 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:15 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:15.026+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.229176+0000 front 2024-09-02T10:39:44.229190+0000 (oldest deadline 2024-09-02T10:40:07.129195+0000) 2024-09-02T10:40:15.346 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:15 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:15.026+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.230345+0000 front 2024-09-02T10:39:51.230364+0000 (oldest deadline 2024-09-02T10:40:14.630925+0000) 2024-09-02T10:40:15.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:15.493+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:15.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:15.493+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:15.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:15.493+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:15.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:15.493+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:15.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:15.536+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:15.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:15.536+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:15.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:15.536+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:15.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:15.536+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:15.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:15.536+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:15.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:15.497+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:15.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:15.497+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:15.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:15.497+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:15.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:15 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:15.497+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:16.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:16 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:16.041+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:16.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:16 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:16.041+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:16.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:16 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:16.041+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:16.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:16 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:16.041+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.229176+0000 front 2024-09-02T10:39:44.229190+0000 (oldest deadline 2024-09-02T10:40:07.129195+0000) 2024-09-02T10:40:16.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:16 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:16.041+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.230345+0000 front 2024-09-02T10:39:51.230364+0000 (oldest deadline 2024-09-02T10:40:14.630925+0000) 2024-09-02T10:40:16.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:16.050+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:16.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:16.050+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:16.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:16.050+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:16.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:16.050+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:17.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:16.531+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:17.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:16.531+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:17.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:16.531+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:17.006 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:16.531+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:17.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:16.528+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:17.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:16.528+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:17.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:16.528+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:17.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:16.528+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:17.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:16.528+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:17.007 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:16.536+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:17.007 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:16.536+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:17.008 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:16.536+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:17.008 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:16 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:16.536+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:17.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:17 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:17.005+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.325185+0000 front 2024-09-02T10:39:20.325246+0000 (oldest deadline 2024-09-02T10:39:47.325694+0000) 2024-09-02T10:40:17.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:17 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:17.005+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.326198+0000 front 2024-09-02T10:39:27.326387+0000 (oldest deadline 2024-09-02T10:39:51.426285+0000) 2024-09-02T10:40:17.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:17 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:17.005+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:31.426963+0000 front 2024-09-02T10:39:31.426859+0000 (oldest deadline 2024-09-02T10:39:58.427458+0000) 2024-09-02T10:40:17.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:17 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:17.005+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.229176+0000 front 2024-09-02T10:39:44.229190+0000 (oldest deadline 2024-09-02T10:40:07.129195+0000) 2024-09-02T10:40:17.345 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:17 smithi162 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7[41451]: 2024-09-02T10:40:17.005+0000 7f09da77d640 -1 osd.7 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.230345+0000 front 2024-09-02T10:39:51.230364+0000 (oldest deadline 2024-09-02T10:40:14.630925+0000) 2024-09-02T10:40:17.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:17.084+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:17.506 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:17.084+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:17.507 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:17.084+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:17.507 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:17.084+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:18.006 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:17.510+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:18.006 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:17.510+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:18.006 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:17.510+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:18.006 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:17.510+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:18.007 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:17.525+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:18.007 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:17.525+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:18.007 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:17.525+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:18.007 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:17.525+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:18.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:17.528+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:18.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:17.528+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:18.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:17.528+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:18.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:17.528+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:18.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:17 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:17.528+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:18.474 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:18.085+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:18.474 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:18.085+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:18.474 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:18.085+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:18.474 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:18.085+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:18.518 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:18 smithi162 podman[171761]: 2024-09-02 10:40:18.262731669 +0000 UTC m=+5.803169494 container died b6a86de9480d4de7cf5dd0c52d2f79af011fc71e3cc2aded8ac2207559f8e3ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, io.buildah.version=1.37.2) 2024-09-02T10:40:18.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:18.563+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:18.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:18.563+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:18.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:18.563+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:18.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:18.563+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:18.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:18.511+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:18.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:18.511+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:18.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:18.511+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:18.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:18.511+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:18.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:18.511+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:18.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:18.472+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:18.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:18.472+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:18.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:18.472+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:18.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:18 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:18.472+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:18.845 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:18 smithi162 podman[171761]: 2024-09-02 10:40:18.608263932 +0000 UTC m=+6.148701757 container cleanup b6a86de9480d4de7cf5dd0c52d2f79af011fc71e3cc2aded8ac2207559f8e3ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-02T10:40:18.845 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:18 smithi162 bash[171761]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7 2024-09-02T10:40:19.169 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:18 smithi162 podman[171790]: 2024-09-02 10:40:18.867661667 +0000 UTC m=+0.603804575 container remove b6a86de9480d4de7cf5dd0c52d2f79af011fc71e3cc2aded8ac2207559f8e3ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-02T10:40:19.438 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:19 smithi162 podman[171876]: 2024-09-02 10:40:19.168526828 +0000 UTC m=+0.108444133 container create 4ce06dfc5e38b29c1006347c0ed15a47ebf1b7f4633be68b4e81791b2423cbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-02T10:40:19.439 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:19 smithi162 podman[171876]: 2024-09-02 10:40:19.076339918 +0000 UTC m=+0.016257227 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:40:19.439 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:19 smithi162 podman[171876]: 2024-09-02 10:40:19.371414316 +0000 UTC m=+0.311331615 container init 4ce06dfc5e38b29c1006347c0ed15a47ebf1b7f4633be68b4e81791b2423cbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7-deactivate, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-02T10:40:19.439 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:19 smithi162 podman[171876]: 2024-09-02 10:40:19.374628247 +0000 UTC m=+0.314545554 container start 4ce06dfc5e38b29c1006347c0ed15a47ebf1b7f4633be68b4e81791b2423cbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T10:40:19.439 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:19 smithi162 podman[171876]: 2024-09-02 10:40:19.438050298 +0000 UTC m=+0.377967597 container attach 4ce06dfc5e38b29c1006347c0ed15a47ebf1b7f4633be68b4e81791b2423cbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T10:40:19.470 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:19.074+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:19.470 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:19.074+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:19.470 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:19.074+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:19.470 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:19.074+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:19.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:19.529+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:19.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:19.529+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:19.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:19.529+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:19.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:19.529+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:19.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:19.502+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:19.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:19.502+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:19.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:19.502+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:19.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:19.502+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:19.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:19.502+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:19.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:19.469+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:19.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:19.469+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:19.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:19.469+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:19.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:19 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:19.469+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:19.845 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:19 smithi162 podman[171876]: 2024-09-02 10:40:19.577152509 +0000 UTC m=+0.517069810 container died 4ce06dfc5e38b29c1006347c0ed15a47ebf1b7f4633be68b4e81791b2423cbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T10:40:20.117 DEBUG:teuthology.orchestra.run.smithi162:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.7.service' 2024-09-02T10:40:20.141 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:20 smithi162 podman[171899]: 2024-09-02 10:40:20.091514399 +0000 UTC m=+0.508041087 container remove 4ce06dfc5e38b29c1006347c0ed15a47ebf1b7f4633be68b4e81791b2423cbce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-7-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T10:40:20.141 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:20 smithi162 systemd[1]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.7.service: Deactivated successfully. 2024-09-02T10:40:20.142 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:20 smithi162 systemd[1]: Stopped Ceph osd.7 for 1530c7be-6911-11ef-bcd4-c7b262605968. 2024-09-02T10:40:20.142 INFO:journalctl@ceph.osd.7.smithi162.stdout:Sep 02 10:40:20 smithi162 systemd[1]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.7.service: Consumed 28min 18.250s CPU time. 2024-09-02T10:40:20.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:40:20.156 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-02T10:40:20.156 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-09-02T10:40:20.156 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.8 2024-09-02T10:40:20.466 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:20.100+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:20.466 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:20.100+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:20.466 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:20.100+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:20.466 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:20.100+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:20.466 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:20 smithi187 systemd[1]: Stopping Ceph osd.8 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:40:20.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:20.530+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:20.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:20.530+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:20.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:20.530+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:20.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:20.530+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:20.757 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:20.464+0000 7f4ba3b7c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:40:20.757 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:20.464+0000 7f4ba3b7c640 -1 osd.8 1042 *** Got signal Terminated *** 2024-09-02T10:40:20.757 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:20.464+0000 7f4ba3b7c640 -1 osd.8 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:40:20.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:20.480+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:20.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:20.480+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:20.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:20.480+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:20.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:20.480+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:20.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:20.480+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:20.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:20.464+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:20.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:20.464+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:20.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:20.464+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:20.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:20 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:20.464+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:21.432 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:21.145+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:21.432 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:21.145+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:21.432 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:21.145+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:21.432 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:21.145+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:21.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:21.512+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:21.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:21.512+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:21.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:21.512+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:21.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:21.512+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:21.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:21.430+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:21.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:21.430+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:21.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:21.430+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:21.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:21.430+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:21.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:21.447+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:21.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:21.447+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:21.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:21.447+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:21.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:21.447+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:21.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:21 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:21.447+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:22.396 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:22.108+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:22.397 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:22.108+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:22.397 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:22.108+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:22.397 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:22.108+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:22.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:22.395+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:22.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:22.395+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:22.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:22.395+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:22.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:22.395+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:22.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:22.534+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:22.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:22.534+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:22.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:22.534+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:22.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:22.534+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:22.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:22.400+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:22.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:22.400+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:22.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:22.400+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:22.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:22.400+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:22.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:22 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:22.400+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:23.351 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:23.071+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:23.351 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:23.071+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:23.351 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:23.071+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:23.351 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:23.071+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:23.351 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:23.071+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.107822+0000 front 2024-09-02T10:40:01.107836+0000 (oldest deadline 2024-09-02T10:40:22.807710+0000) 2024-09-02T10:40:23.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:23.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:23.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:23.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:23.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:23.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:23.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:23.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:23.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:23.409+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:23.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:23.349+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:23.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:23.349+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:23.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:23.349+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:23.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:23.349+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:23.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:23.487+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:23.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:23.487+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:23.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:23.487+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:23.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:23 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:23.487+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:24.309 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:24.055+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:24.309 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:24.055+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:24.310 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:24.055+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:24.310 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:24.055+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:24.310 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:24.055+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.107822+0000 front 2024-09-02T10:40:01.107836+0000 (oldest deadline 2024-09-02T10:40:22.807710+0000) 2024-09-02T10:40:24.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:24.418+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:24.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:24.418+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:24.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:24.418+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:24.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:24.418+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:24.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:24.418+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:24.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:24.308+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:24.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:24.308+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:24.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:24.308+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:24.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:24.308+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:24.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:24.308+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:24.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:24.503+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:24.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:24.503+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:24.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:24.503+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:24.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:24 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:24.503+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:25.359 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:25.019+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.901111+0000 front 2024-09-02T10:39:19.901093+0000 (oldest deadline 2024-09-02T10:39:44.000949+0000) 2024-09-02T10:40:25.360 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:25.019+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:28.701878+0000 front 2024-09-02T10:39:28.701853+0000 (oldest deadline 2024-09-02T10:39:52.202004+0000) 2024-09-02T10:40:25.360 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:25.019+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:44.904828+0000 front 2024-09-02T10:39:44.904814+0000 (oldest deadline 2024-09-02T10:40:09.604614+0000) 2024-09-02T10:40:25.360 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:25.019+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.905832+0000 front 2024-09-02T10:39:51.905649+0000 (oldest deadline 2024-09-02T10:40:14.805517+0000) 2024-09-02T10:40:25.360 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8[24745]: 2024-09-02T10:40:25.019+0000 7f4ba0191640 -1 osd.8 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.107822+0000 front 2024-09-02T10:40:01.107836+0000 (oldest deadline 2024-09-02T10:40:22.807710+0000) 2024-09-02T10:40:25.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:25.358+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:25.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:25.358+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:25.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:25.358+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:25.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:25.358+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:25.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:25.358+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:25.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:25.473+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:25.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:25.473+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:25.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:25.473+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:25.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:25.473+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:25.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:25.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:25.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:25.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:25.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:25.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:25.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:25.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:25.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:25 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:25.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:26.373 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:26 smithi187 podman[166982]: 2024-09-02 10:40:26.112109963 +0000 UTC m=+5.758939411 container died 4166efe56d7b781bdc468f1390bbc9d5a5efaf2e4a9bf49289269e28ea799bae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-02T10:40:26.627 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:26 smithi187 podman[166982]: 2024-09-02 10:40:26.606995399 +0000 UTC m=+6.253824879 container cleanup 4166efe56d7b781bdc468f1390bbc9d5a5efaf2e4a9bf49289269e28ea799bae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-02T10:40:26.627 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:26 smithi187 bash[166982]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8 2024-09-02T10:40:26.628 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:26.446+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:26.628 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:26.446+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:26.628 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:26.446+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:26.628 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:26.446+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:26.628 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:26.371+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:26.628 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:26.371+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:26.629 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:26.371+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:26.629 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:26.371+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:26.629 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:26.371+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:26.629 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:26.414+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:26.629 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:26.414+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:26.629 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:26.414+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:26.629 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:26.414+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:26.629 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:26.414+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:26.629 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:26 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:26.414+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.386281+0000 front 2024-09-02T10:40:01.386275+0000 (oldest deadline 2024-09-02T10:40:26.086158+0000) 2024-09-02T10:40:27.168 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:26 smithi187 podman[166997]: 2024-09-02 10:40:26.850131325 +0000 UTC m=+0.725355199 container remove 4166efe56d7b781bdc468f1390bbc9d5a5efaf2e4a9bf49289269e28ea799bae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T10:40:27.437 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:27.383+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:27.437 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:27.383+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:27.437 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:27.383+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:27.437 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:27.383+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:27.437 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:27.383+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:27.438 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:27 smithi187 podman[167087]: 2024-09-02 10:40:27.167621794 +0000 UTC m=+0.113067638 container create 815d21ea5b661a2c4d4b91207881afa3d555ffdf603949bc59b2ecf713420fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-02T10:40:27.438 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:27 smithi187 podman[167087]: 2024-09-02 10:40:27.070391496 +0000 UTC m=+0.015837352 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:40:27.438 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:27 smithi187 podman[167087]: 2024-09-02 10:40:27.362225343 +0000 UTC m=+0.307671187 container init 815d21ea5b661a2c4d4b91207881afa3d555ffdf603949bc59b2ecf713420fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-02T10:40:27.438 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:27 smithi187 podman[167087]: 2024-09-02 10:40:27.365435941 +0000 UTC m=+0.310881788 container start 815d21ea5b661a2c4d4b91207881afa3d555ffdf603949bc59b2ecf713420fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8-deactivate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-02T10:40:27.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:27.401+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:27.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:27.401+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:27.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:27.401+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:27.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:27.401+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:27.438 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:27.401+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:27.756 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:27 smithi187 podman[167087]: 2024-09-02 10:40:27.472397233 +0000 UTC m=+0.417843095 container attach 815d21ea5b661a2c4d4b91207881afa3d555ffdf603949bc59b2ecf713420fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-02T10:40:27.757 INFO:journalctl@ceph.osd.8.smithi187.stdout:Sep 02 10:40:27 smithi187 podman[167087]: 2024-09-02 10:40:27.588469176 +0000 UTC m=+0.533915030 container died 815d21ea5b661a2c4d4b91207881afa3d555ffdf603949bc59b2ecf713420fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-02T10:40:27.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:27.435+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:27.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:27.435+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:27.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:27.435+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:27.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:27.435+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:27.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:27.435+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:27.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:27 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:27.435+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.386281+0000 front 2024-09-02T10:40:01.386275+0000 (oldest deadline 2024-09-02T10:40:26.086158+0000) 2024-09-02T10:40:28.220 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.8.service' 2024-09-02T10:40:28.262 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:40:28.262 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-09-02T10:40:28.262 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-09-02T10:40:28.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.9 2024-09-02T10:40:28.483 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:28.425+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:28.483 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:28.425+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:28.483 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:28.425+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:28.483 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:28.425+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:28.483 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:28.425+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:28.484 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:28 smithi187 systemd[1]: Stopping Ceph osd.9 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:40:28.484 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:28.412+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:28.484 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:28.413+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:28.484 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:28.413+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:28.484 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:28.413+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:28.484 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:28.413+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:28.484 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:28.413+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.386281+0000 front 2024-09-02T10:40:01.386275+0000 (oldest deadline 2024-09-02T10:40:26.086158+0000) 2024-09-02T10:40:28.484 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:28.380+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:28.485 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:28.380+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:28.485 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:28.380+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:28.485 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:28.380+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:28.485 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:28.380+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:28.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:28.610+0000 7f0e15dcf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:40:28.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:28.610+0000 7f0e15dcf640 -1 osd.9 1042 *** Got signal Terminated *** 2024-09-02T10:40:28.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:28 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:28.610+0000 7f0e15dcf640 -1 osd.9 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:40:29.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:29.417+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:29.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:29.417+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:29.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:29.417+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:29.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:29.417+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:29.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:29.417+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:29.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:29.417+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.386281+0000 front 2024-09-02T10:40:01.386275+0000 (oldest deadline 2024-09-02T10:40:26.086158+0000) 2024-09-02T10:40:29.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:29.394+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:29.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:29.394+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:29.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:29.394+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:29.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:29.394+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:29.758 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:29.394+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:29.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:29.397+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:29.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:29.397+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:29.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:29.397+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:29.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:29.397+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:29.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:29 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:29.397+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:30.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:30.396+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:30.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:30.396+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:30.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:30.396+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:30.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:30.396+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:30.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:30.396+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:30.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:30.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:30.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:30.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:30.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:30.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:30.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:30.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:30.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:30.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:30.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:30.428+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:30.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:30.428+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:30.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:30.428+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:30.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:30.428+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:30.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:30.428+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:30.759 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:30 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:30.428+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.386281+0000 front 2024-09-02T10:40:01.386275+0000 (oldest deadline 2024-09-02T10:40:26.086158+0000) 2024-09-02T10:40:31.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:31.416+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:31.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:31.416+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:31.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:31.416+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:31.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:31.416+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:31.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:31.416+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:31.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:31.375+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:31.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:31.375+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:31.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:31.375+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:31.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:31.375+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:31.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:31.375+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:31.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:31.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:31.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:31.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:31.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:31.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:31.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:31.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:31.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:31.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:31.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:31.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.386281+0000 front 2024-09-02T10:40:01.386275+0000 (oldest deadline 2024-09-02T10:40:26.086158+0000) 2024-09-02T10:40:31.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:31 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:31.408+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.086799+0000 front 2024-09-02T10:40:06.086702+0000 (oldest deadline 2024-09-02T10:40:30.786624+0000) 2024-09-02T10:40:32.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:32.368+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:32.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:32.368+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:32.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:32.368+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:32.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:32.368+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:32.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:32.368+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:32.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:32.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:32.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:32.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:32.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:32.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:32.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:32.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:32.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:32.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:32.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:32.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:32.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:32.423+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:32.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:32.423+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:32.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:32.423+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:32.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:32.423+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:32.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:32.423+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:32.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:32.423+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.386281+0000 front 2024-09-02T10:40:01.386275+0000 (oldest deadline 2024-09-02T10:40:26.086158+0000) 2024-09-02T10:40:32.758 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:32 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:32.423+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.086799+0000 front 2024-09-02T10:40:06.086702+0000 (oldest deadline 2024-09-02T10:40:30.786624+0000) 2024-09-02T10:40:33.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:33.402+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:33.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:33.402+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:33.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:33.402+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:33.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:33.402+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:33.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:33.402+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:33.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:33.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:19.880767+0000 front 2024-09-02T10:39:19.880906+0000 (oldest deadline 2024-09-02T10:39:42.180762+0000) 2024-09-02T10:40:33.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:33.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:26.881668+0000 front 2024-09-02T10:39:26.881737+0000 (oldest deadline 2024-09-02T10:39:50.981822+0000) 2024-09-02T10:40:33.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:33.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:33.283018+0000 front 2024-09-02T10:39:33.283006+0000 (oldest deadline 2024-09-02T10:39:59.182770+0000) 2024-09-02T10:40:33.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:33.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.083904+0000 front 2024-09-02T10:39:45.083885+0000 (oldest deadline 2024-09-02T10:40:09.183883+0000) 2024-09-02T10:40:33.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:33.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6804 osd.4 since back 2024-09-02T10:39:51.484891+0000 front 2024-09-02T10:39:51.484806+0000 (oldest deadline 2024-09-02T10:40:14.384664+0000) 2024-09-02T10:40:33.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:33.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.386281+0000 front 2024-09-02T10:40:01.386275+0000 (oldest deadline 2024-09-02T10:40:26.086158+0000) 2024-09-02T10:40:33.757 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9[30298]: 2024-09-02T10:40:33.456+0000 7f0e0e3dc640 -1 osd.9 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.086799+0000 front 2024-09-02T10:40:06.086702+0000 (oldest deadline 2024-09-02T10:40:30.786624+0000) 2024-09-02T10:40:33.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:33.317+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:33.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:33.317+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:33.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:33.317+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:33.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:33.317+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:33.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:33.317+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:33.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:33 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:33.317+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:34.504 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:34 smithi187 podman[167190]: 2024-09-02 10:40:34.198823179 +0000 UTC m=+5.696666590 container died f50f4a134e98f8f5168e3030709fa6e7478dadfd3a4824708b8a89338cdaf127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-02T10:40:34.504 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:34.400+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:34.504 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:34.400+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:34.504 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:34.400+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:34.504 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:34.400+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:34.504 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:34.400+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:34.505 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:34.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:34.505 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:34.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:34.505 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:34.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:34.505 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:34.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:34.505 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:34.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:34.505 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:34 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:34.361+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:34.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:34 smithi187 podman[167190]: 2024-09-02 10:40:34.620220116 +0000 UTC m=+6.118063527 container cleanup f50f4a134e98f8f5168e3030709fa6e7478dadfd3a4824708b8a89338cdaf127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T10:40:34.756 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:34 smithi187 bash[167190]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9 2024-09-02T10:40:35.256 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:34 smithi187 podman[167215]: 2024-09-02 10:40:34.888172478 +0000 UTC m=+0.685627140 container remove f50f4a134e98f8f5168e3030709fa6e7478dadfd3a4824708b8a89338cdaf127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-02T10:40:35.617 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:35 smithi187 podman[167302]: 2024-09-02 10:40:35.272697694 +0000 UTC m=+0.015887608 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:40:35.617 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:35 smithi187 podman[167302]: 2024-09-02 10:40:35.397735813 +0000 UTC m=+0.140925718 container create 59ad5c718002f3cae66ef62d545a988e35a00d61af331feac659a35c081f808b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-09-02T10:40:35.618 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:35 smithi187 podman[167302]: 2024-09-02 10:40:35.616931487 +0000 UTC m=+0.360121406 container init 59ad5c718002f3cae66ef62d545a988e35a00d61af331feac659a35c081f808b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-02T10:40:35.618 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:35.365+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:35.618 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:35.365+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:35.618 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:35.365+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:35.618 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:35.365+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:35.618 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:35.365+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:35.618 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:35.373+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:35.618 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:35.373+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:35.619 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:35.373+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:35.619 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:35.373+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:35.619 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:35.373+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:35.619 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:35 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:35.373+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:36.006 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:35 smithi187 podman[167302]: 2024-09-02 10:40:35.62024985 +0000 UTC m=+0.363439756 container start 59ad5c718002f3cae66ef62d545a988e35a00d61af331feac659a35c081f808b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-02T10:40:36.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:35 smithi187 podman[167302]: 2024-09-02 10:40:35.683630309 +0000 UTC m=+0.426820215 container attach 59ad5c718002f3cae66ef62d545a988e35a00d61af331feac659a35c081f808b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True) 2024-09-02T10:40:36.007 INFO:journalctl@ceph.osd.9.smithi187.stdout:Sep 02 10:40:35 smithi187 podman[167302]: 2024-09-02 10:40:35.805580661 +0000 UTC m=+0.548770575 container died 59ad5c718002f3cae66ef62d545a988e35a00d61af331feac659a35c081f808b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-02T10:40:36.362 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:36.327+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:36.362 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:36.327+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:36.362 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:36.327+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:36.362 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:36.327+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:36.362 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:36.327+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:36.362 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:36.327+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:09.483331+0000 front 2024-09-02T10:40:09.483254+0000 (oldest deadline 2024-09-02T10:40:35.383213+0000) 2024-09-02T10:40:36.363 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:36.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:36.363 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:36.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:36.363 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:36.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:36.363 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:36.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:36.363 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:36.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:36.363 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:36.360+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:36.426 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.9.service' 2024-09-02T10:40:36.464 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:40:36.464 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-09-02T10:40:36.464 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-09-02T10:40:36.464 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.10 2024-09-02T10:40:36.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:36 smithi187 systemd[1]: Stopping Ceph osd.10 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:40:37.256 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:36.790+0000 7f7ed2249640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:40:37.256 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:36.790+0000 7f7ed2249640 -1 osd.10 1042 *** Got signal Terminated *** 2024-09-02T10:40:37.256 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:36 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:36.790+0000 7f7ed2249640 -1 osd.10 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:40:37.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:37.332+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:37.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:37.332+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:37.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:37.332+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:37.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:37.332+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:37.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:37.332+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:37.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:37.332+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:09.483331+0000 front 2024-09-02T10:40:09.483254+0000 (oldest deadline 2024-09-02T10:40:35.383213+0000) 2024-09-02T10:40:37.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:37.396+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:37.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:37.396+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:37.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:37.396+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:37.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:37.396+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:37.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:37.396+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:37.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:37 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:37.396+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:38.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:38.363+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:38.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:38.363+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:38.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:38.363+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:38.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:38.363+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:38.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:38.363+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:38.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:38.363+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:09.483331+0000 front 2024-09-02T10:40:09.483254+0000 (oldest deadline 2024-09-02T10:40:35.383213+0000) 2024-09-02T10:40:38.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:38.378+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:38.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:38.378+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:38.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:38.378+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:38.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:38.378+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:38.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:38.378+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:38.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:38 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:38.378+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:39.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:39.357+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:39.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:39.357+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:39.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:39.357+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:39.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:39.357+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:39.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:39.357+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:39.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:39.357+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:09.483331+0000 front 2024-09-02T10:40:09.483254+0000 (oldest deadline 2024-09-02T10:40:35.383213+0000) 2024-09-02T10:40:39.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:39.393+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:39.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:39.393+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:39.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:39.393+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:39.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:39.393+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:39.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:39.393+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:39.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:39 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:39.393+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:40.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:40.379+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:40.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:40.379+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:40.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:40.379+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:40.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:40.379+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:40.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:40.379+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:40.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:40.379+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:09.483331+0000 front 2024-09-02T10:40:09.483254+0000 (oldest deadline 2024-09-02T10:40:35.383213+0000) 2024-09-02T10:40:40.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:40.387+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:40.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:40.387+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:40.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:40.387+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:40.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:40.387+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:40.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:40.387+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:40.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:40.387+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:40.758 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:40 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:40.387+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-09-02T10:40:11.819930+0000 front 2024-09-02T10:40:11.819947+0000 (oldest deadline 2024-09-02T10:40:40.020274+0000) 2024-09-02T10:40:41.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:41.388+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:20.376313+0000 front 2024-09-02T10:39:20.376571+0000 (oldest deadline 2024-09-02T10:39:42.676183+0000) 2024-09-02T10:40:41.756 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:41.388+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.377584+0000 front 2024-09-02T10:39:27.377795+0000 (oldest deadline 2024-09-02T10:39:52.077541+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:41.388+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:34.978622+0000 front 2024-09-02T10:39:34.978447+0000 (oldest deadline 2024-09-02T10:39:59.678570+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:41.388+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:45.480189+0000 front 2024-09-02T10:39:45.480130+0000 (oldest deadline 2024-09-02T10:40:09.579953+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:41.388+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:39:59.482321+0000 front 2024-09-02T10:39:59.482333+0000 (oldest deadline 2024-09-02T10:40:24.182191+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10[36062]: 2024-09-02T10:40:41.388+0000 7f7eca856640 -1 osd.10 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:09.483331+0000 front 2024-09-02T10:40:09.483254+0000 (oldest deadline 2024-09-02T10:40:35.383213+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:41.366+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:41.366+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:41.366+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:41.366+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:41.366+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:41.366+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:41.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:41 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:41.366+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-09-02T10:40:11.819930+0000 front 2024-09-02T10:40:11.819947+0000 (oldest deadline 2024-09-02T10:40:40.020274+0000) 2024-09-02T10:40:42.752 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:42 smithi187 podman[167406]: 2024-09-02 10:40:42.466142087 +0000 UTC m=+5.788531542 container died f7de0cc3400af69da37912c240e127e9091cf6d23249f6300df5082301d703bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-02T10:40:42.753 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:42 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:42.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:42.753 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:42 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:42.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:42.753 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:42 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:42.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:42.753 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:42 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:42.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:42.753 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:42 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:42.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:42.753 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:42 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:42.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:42.753 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:42 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:42.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-09-02T10:40:11.819930+0000 front 2024-09-02T10:40:11.819947+0000 (oldest deadline 2024-09-02T10:40:40.020274+0000) 2024-09-02T10:40:43.008 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:42 smithi187 podman[167406]: 2024-09-02 10:40:42.823873453 +0000 UTC m=+6.146262911 container cleanup f7de0cc3400af69da37912c240e127e9091cf6d23249f6300df5082301d703bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-02T10:40:43.008 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:42 smithi187 bash[167406]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10 2024-09-02T10:40:43.393 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:43 smithi187 podman[167422]: 2024-09-02 10:40:43.100345749 +0000 UTC m=+0.630885414 container remove f7de0cc3400af69da37912c240e127e9091cf6d23249f6300df5082301d703bf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-02T10:40:43.393 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:43 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:43.281+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:43.394 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:43 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:43.281+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:43.394 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:43 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:43.281+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:43.394 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:43 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:43.281+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:43.394 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:43 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:43.281+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:43.394 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:43 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:43.281+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:43.394 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:43 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:43.281+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-09-02T10:40:11.819930+0000 front 2024-09-02T10:40:11.819947+0000 (oldest deadline 2024-09-02T10:40:40.020274+0000) 2024-09-02T10:40:43.713 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:43 smithi187 podman[167514]: 2024-09-02 10:40:43.392774735 +0000 UTC m=+0.098895417 container create 77d41ee13dc54c2718ff74dc4062ca9f624f8c82bbe7334532289f32103187aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-02T10:40:43.713 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:43 smithi187 podman[167514]: 2024-09-02 10:40:43.309542284 +0000 UTC m=+0.015662938 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:40:43.713 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:43 smithi187 podman[167514]: 2024-09-02 10:40:43.629054629 +0000 UTC m=+0.335175278 container init 77d41ee13dc54c2718ff74dc4062ca9f624f8c82bbe7334532289f32103187aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-02T10:40:43.713 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:43 smithi187 podman[167514]: 2024-09-02 10:40:43.632430584 +0000 UTC m=+0.338551224 container start 77d41ee13dc54c2718ff74dc4062ca9f624f8c82bbe7334532289f32103187aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-09-02T10:40:43.713 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:43 smithi187 podman[167514]: 2024-09-02 10:40:43.712040009 +0000 UTC m=+0.418160649 container attach 77d41ee13dc54c2718ff74dc4062ca9f624f8c82bbe7334532289f32103187aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-02T10:40:44.006 INFO:journalctl@ceph.osd.10.smithi187.stdout:Sep 02 10:40:43 smithi187 podman[167514]: 2024-09-02 10:40:43.81752549 +0000 UTC m=+0.523646144 container died 77d41ee13dc54c2718ff74dc4062ca9f624f8c82bbe7334532289f32103187aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T10:40:44.409 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:44.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:44.409 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:44.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:44.409 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:44.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:44.409 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:44.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:44.409 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:44.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:44.409 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:44.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:44.409 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:44.319+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-09-02T10:40:11.819930+0000 front 2024-09-02T10:40:11.819947+0000 (oldest deadline 2024-09-02T10:40:40.020274+0000) 2024-09-02T10:40:44.427 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.10.service' 2024-09-02T10:40:44.465 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:40:44.466 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-09-02T10:40:44.466 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-09-02T10:40:44.466 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.11 2024-09-02T10:40:44.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 systemd[1]: Stopping Ceph osd.11 for 1530c7be-6911-11ef-bcd4-c7b262605968... 2024-09-02T10:40:45.256 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:44.794+0000 7f0e856af640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-02T10:40:45.256 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:44.794+0000 7f0e856af640 -1 osd.11 1042 *** Got signal Terminated *** 2024-09-02T10:40:45.256 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:44 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:44.794+0000 7f0e856af640 -1 osd.11 1042 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-02T10:40:45.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:45.365+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:45.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:45.365+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:45.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:45.365+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:45.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:45.365+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:45.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:45.365+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:45.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:45.365+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:45.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:45 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:45.365+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-09-02T10:40:11.819930+0000 front 2024-09-02T10:40:11.819947+0000 (oldest deadline 2024-09-02T10:40:40.020274+0000) 2024-09-02T10:40:46.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:46.384+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:46.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:46.384+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:46.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:46.384+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:46.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:46.384+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:46.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:46.384+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:46.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:46.384+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:46.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:46 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:46.384+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-09-02T10:40:11.819930+0000 front 2024-09-02T10:40:11.819947+0000 (oldest deadline 2024-09-02T10:40:40.020274+0000) 2024-09-02T10:40:47.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:47.364+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:47.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:47.364+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:47.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:47.364+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:47.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:47.364+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:47.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:47.364+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:47.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:47.364+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:47.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:47 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:47.364+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-09-02T10:40:11.819930+0000 front 2024-09-02T10:40:11.819947+0000 (oldest deadline 2024-09-02T10:40:40.020274+0000) 2024-09-02T10:40:48.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:48.367+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:48.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:48.367+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:48.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:48.367+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:48.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:48.367+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:48.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:48.367+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:48.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:48.367+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:48.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:48.367+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-09-02T10:40:11.819930+0000 front 2024-09-02T10:40:11.819947+0000 (oldest deadline 2024-09-02T10:40:40.020274+0000) 2024-09-02T10:40:48.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:48 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:48.367+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.187:6804 osd.8 since back 2024-09-02T10:40:24.121033+0000 front 2024-09-02T10:40:24.121098+0000 (oldest deadline 2024-09-02T10:40:47.621044+0000) 2024-09-02T10:40:49.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:49.336+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6806 osd.0 since back 2024-09-02T10:39:16.411120+0000 front 2024-09-02T10:39:16.411432+0000 (oldest deadline 2024-09-02T10:39:41.611914+0000) 2024-09-02T10:40:49.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:49.336+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6814 osd.1 since back 2024-09-02T10:39:27.512958+0000 front 2024-09-02T10:39:27.513088+0000 (oldest deadline 2024-09-02T10:39:51.613102+0000) 2024-09-02T10:40:49.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:49.336+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6822 osd.2 since back 2024-09-02T10:39:35.614805+0000 front 2024-09-02T10:39:35.614828+0000 (oldest deadline 2024-09-02T10:39:59.114945+0000) 2024-09-02T10:40:49.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:49.336+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.151:6830 osd.3 since back 2024-09-02T10:39:43.215884+0000 front 2024-09-02T10:39:43.216031+0000 (oldest deadline 2024-09-02T10:40:06.115951+0000) 2024-09-02T10:40:49.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:49.336+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6812 osd.5 since back 2024-09-02T10:40:01.218971+0000 front 2024-09-02T10:40:01.218881+0000 (oldest deadline 2024-09-02T10:40:26.518784+0000) 2024-09-02T10:40:49.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:49.336+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6820 osd.6 since back 2024-09-02T10:40:06.519660+0000 front 2024-09-02T10:40:06.519642+0000 (oldest deadline 2024-09-02T10:40:31.819463+0000) 2024-09-02T10:40:49.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:49.336+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.162:6828 osd.7 since back 2024-09-02T10:40:11.819930+0000 front 2024-09-02T10:40:11.819947+0000 (oldest deadline 2024-09-02T10:40:40.020274+0000) 2024-09-02T10:40:49.757 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:49 smithi187 ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11[41823]: 2024-09-02T10:40:49.336+0000 7f0e7dcbc640 -1 osd.11 1042 heartbeat_check: no reply from 172.21.15.187:6804 osd.8 since back 2024-09-02T10:40:24.121033+0000 front 2024-09-02T10:40:24.121098+0000 (oldest deadline 2024-09-02T10:40:47.621044+0000) 2024-09-02T10:40:50.506 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:50 smithi187 podman[167618]: 2024-09-02 10:40:50.232165649 +0000 UTC m=+5.547525029 container died 4081a6d6eb753963cf59a368512f68db1db9f58a053b329bf4c0c79473bdb124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-09-02T10:40:50.854 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:50 smithi187 podman[167618]: 2024-09-02 10:40:50.585834962 +0000 UTC m=+5.901194343 container cleanup 4081a6d6eb753963cf59a368512f68db1db9f58a053b329bf4c0c79473bdb124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-02T10:40:50.855 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:50 smithi187 bash[167618]: ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11 2024-09-02T10:40:51.157 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:50 smithi187 podman[167634]: 2024-09-02 10:40:50.853477462 +0000 UTC m=+0.613069679 container remove 4081a6d6eb753963cf59a368512f68db1db9f58a053b329bf4c0c79473bdb124 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-02T10:40:51.449 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:51 smithi187 podman[167721]: 2024-09-02 10:40:51.059996107 +0000 UTC m=+0.015868924 image pull 27cca6ea3cd60cd93d5374c7fc1ce8fb4083464e5e85ce2229dc383f6ad710b0 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4 2024-09-02T10:40:51.450 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:51 smithi187 podman[167721]: 2024-09-02 10:40:51.171401945 +0000 UTC m=+0.127274802 container create 1308e742e8005d1536acb91ff0d56c4e3cfa364d8143ebe48d6c604bd3f416bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-02T10:40:51.450 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:51 smithi187 podman[167721]: 2024-09-02 10:40:51.382081306 +0000 UTC m=+0.337954121 container init 1308e742e8005d1536acb91ff0d56c4e3cfa364d8143ebe48d6c604bd3f416bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-02T10:40:51.450 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:51 smithi187 podman[167721]: 2024-09-02 10:40:51.38548619 +0000 UTC m=+0.341359014 container start 1308e742e8005d1536acb91ff0d56c4e3cfa364d8143ebe48d6c604bd3f416bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e) 2024-09-02T10:40:51.450 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:51 smithi187 podman[167721]: 2024-09-02 10:40:51.448720971 +0000 UTC m=+0.404593804 container attach 1308e742e8005d1536acb91ff0d56c4e3cfa364d8143ebe48d6c604bd3f416bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-02T10:40:51.756 INFO:journalctl@ceph.osd.11.smithi187.stdout:Sep 02 10:40:51 smithi187 podman[167721]: 2024-09-02 10:40:51.573271738 +0000 UTC m=+0.529144553 container died 1308e742e8005d1536acb91ff0d56c4e3cfa364d8143ebe48d6c604bd3f416bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0283f2b78054a50bee3be60b665600a80c9421b88d845265f63b86fe8a20c2d4, name=ceph-1530c7be-6911-11ef-bcd4-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-bharath3-testing-2024-09-01-1328-82daa6e, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-02T10:40:52.193 DEBUG:teuthology.orchestra.run.smithi187:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1530c7be-6911-11ef-bcd4-c7b262605968@osd.11.service' 2024-09-02T10:40:52.230 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-02T10:40:52.230 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-09-02T10:40:52.230 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 --force --keep-logs 2024-09-02T10:40:52.436 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T10:40:54.222 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 --force --keep-logs 2024-09-02T10:40:54.426 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T10:40:56.386 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 --force --keep-logs 2024-09-02T10:40:56.589 INFO:teuthology.orchestra.run.smithi187.stdout:Deleting cluster with fsid: 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T10:40:58.193 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T10:40:58.227 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T10:40:58.261 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-02T10:40:58.289 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-02T10:40:58.291 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453/remote/smithi151/crash 2024-09-02T10:40:58.292 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/crash -- . 2024-09-02T10:40:58.345 INFO:teuthology.orchestra.run.smithi151.stderr:tar: /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-09-02T10:40:58.346 INFO:teuthology.orchestra.run.smithi151.stderr:tar: Error is not recoverable: exiting now 2024-09-02T10:40:58.348 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453/remote/smithi162/crash 2024-09-02T10:40:58.349 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/crash -- . 2024-09-02T10:40:58.467 INFO:teuthology.orchestra.run.smithi162.stderr:tar: /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-09-02T10:40:58.467 INFO:teuthology.orchestra.run.smithi162.stderr:tar: Error is not recoverable: exiting now 2024-09-02T10:40:58.469 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/crash to /home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453/remote/smithi187/crash 2024-09-02T10:40:58.470 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/crash -- . 2024-09-02T10:40:58.505 INFO:teuthology.orchestra.run.smithi187.stderr:tar: /var/lib/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/crash: Cannot open: No such file or directory 2024-09-02T10:40:58.505 INFO:teuthology.orchestra.run.smithi187.stderr:tar: Error is not recoverable: exiting now 2024-09-02T10:40:58.506 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-02T10:40:58.507 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-02T10:40:58.606 INFO:teuthology.orchestra.run.smithi151.stdout:2024-09-02T10:10:00.000156+0000 mon.a (mon.0) 1594 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 279/37245 objects degraded (0.749%), 2 pgs degraded, 3 pgs undersized 2024-09-02T10:40:58.607 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-09-02T10:40:58.607 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[SEC\]' /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-02T10:40:58.644 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[ERR\]' /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-02T10:40:58.720 DEBUG:teuthology.orchestra.run.smithi151:> sudo egrep '\[WRN\]' /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-02T10:40:58.794 INFO:teuthology.orchestra.run.smithi151.stdout:2024-09-02T10:10:00.000156+0000 mon.a (mon.0) 1594 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 279/37245 objects degraded (0.749%), 2 pgs degraded, 3 pgs undersized 2024-09-02T10:40:58.795 INFO:tasks.cephadm:Compressing logs... 2024-09-02T10:40:58.795 DEBUG:teuthology.orchestra.run.smithi151:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-02T10:40:58.838 DEBUG:teuthology.orchestra.run.smithi162:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-02T10:40:58.842 DEBUG:teuthology.orchestra.run.smithi187:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-02T10:40:58.880 INFO:teuthology.orchestra.run.smithi151.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-02T10:40:58.881 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.0.log 2024-09-02T10:40:58.881 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.1.log 2024-09-02T10:40:58.881 INFO:teuthology.orchestra.run.smithi151.stderr:gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-volume.log 2024-09-02T10:40:58.881 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.0.log: /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.audit.log 2024-09-02T10:40:58.881 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-09-02T10:40:58.881 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mon.a.log 2024-09-02T10:40:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.2.log 2024-09-02T10:40:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mgr.y.log 2024-09-02T10:40:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log 2024-09-02T10:40:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.3.log 2024-09-02T10:40:58.882 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-02T10:40:58.889 INFO:teuthology.orchestra.run.smithi151.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 85.7% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log.gz 2024-09-02T10:40:58.890 INFO:teuthology.orchestra.run.smithi187.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory 2024-09-02T10:40:58.890 INFO:teuthology.orchestra.run.smithi187.stderr: /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.10.log 2024-09-02T10:40:58.890 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.9.log 2024-09-02T10:40:58.890 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-volume.log 2024-09-02T10:40:58.891 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.audit.log 2024-09-02T10:40:58.892 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mon.c.log 2024-09-02T10:40:58.892 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-09-02T10:40:58.892 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.11.log 2024-09-02T10:40:58.893 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log 2024-09-02T10:40:58.893 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.8.log 2024-09-02T10:40:58.893 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-02T10:40:58.895 INFO:teuthology.orchestra.run.smithi162.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.7.log 2024-09-02T10:40:58.895 INFO:teuthology.orchestra.run.smithi162.stderr:: No such file or directory 2024-09-02T10:40:58.896 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-volume.log 2024-09-02T10:40:58.896 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.4.log 2024-09-02T10:40:58.896 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.7.log: /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.audit.log 2024-09-02T10:40:58.898 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.cephadm.log 2024-09-02T10:40:58.898 INFO:teuthology.orchestra.run.smithi162.stderr:gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.6.log 2024-09-02T10:40:58.898 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.audit.log: /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.5.log 2024-09-02T10:40:58.898 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log 2024-09-02T10:40:58.898 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mgr.x.log 2024-09-02T10:40:58.899 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mon.b.log 2024-09-02T10:40:58.899 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-02T10:40:58.938 INFO:teuthology.orchestra.run.smithi151.stderr: 82.6% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-09-02T10:40:58.947 INFO:teuthology.orchestra.run.smithi162.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 95.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-02T10:40:58.951 INFO:teuthology.orchestra.run.smithi162.stderr: 82.5% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-09-02T10:40:58.982 INFO:teuthology.orchestra.run.smithi162.stderr: 89.8% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-09-02T10:40:58.996 INFO:teuthology.orchestra.run.smithi151.stderr: 91.2% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-09-02T10:40:59.018 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.8.log: /var/log/ceph/cephadm.log: 95.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-02T10:40:59.042 INFO:teuthology.orchestra.run.smithi162.stderr: 93.0% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mgr.x.log.gz 2024-09-02T10:40:59.173 INFO:teuthology.orchestra.run.smithi162.stderr: 85.7% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log.gz 2024-09-02T10:40:59.193 INFO:teuthology.orchestra.run.smithi162.stderr: 91.2% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-09-02T10:40:59.271 INFO:teuthology.orchestra.run.smithi151.stderr: 95.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-02T10:40:59.304 INFO:teuthology.orchestra.run.smithi187.stderr: 82.5% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.cephadm.log.gz 2024-09-02T10:40:59.310 INFO:teuthology.orchestra.run.smithi151.stderr: 89.7% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-09-02T10:40:59.335 INFO:teuthology.orchestra.run.smithi187.stderr: 85.7% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.log.gz 2024-09-02T10:40:59.362 INFO:teuthology.orchestra.run.smithi187.stderr: 89.8% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph.audit.log.gz 2024-09-02T10:40:59.529 INFO:teuthology.orchestra.run.smithi187.stderr: 91.0% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-volume.log.gz 2024-09-02T10:41:05.014 INFO:teuthology.orchestra.run.smithi162.stderr: 90.9% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mon.b.log.gz 2024-09-02T10:41:11.907 INFO:teuthology.orchestra.run.smithi151.stderr: 91.5% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mgr.y.log.gz 2024-09-02T10:41:15.179 INFO:teuthology.orchestra.run.smithi187.stderr: 91.0% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mon.c.log.gz 2024-09-02T10:41:37.703 INFO:teuthology.orchestra.run.smithi151.stderr: 90.0% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-mon.a.log.gz 2024-09-02T10:44:48.451 INFO:teuthology.orchestra.run.smithi187.stderr: 92.1% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.8.log.gz 2024-09-02T10:48:58.652 INFO:teuthology.orchestra.run.smithi187.stderr: 93.1% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.11.log.gz 2024-09-02T10:50:23.296 INFO:teuthology.orchestra.run.smithi187.stderr: 93.1% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.9.log.gz 2024-09-02T10:50:45.792 INFO:teuthology.orchestra.run.smithi187.stderr: 92.2% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.10.log.gz 2024-09-02T10:50:45.794 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-09-02T10:50:45.794 INFO:teuthology.orchestra.run.smithi187.stderr:real 9m46.937s 2024-09-02T10:50:45.794 INFO:teuthology.orchestra.run.smithi187.stderr:user 5m51.534s 2024-09-02T10:50:45.794 INFO:teuthology.orchestra.run.smithi187.stderr:sys 0m15.923s 2024-09-02T10:56:36.500 INFO:teuthology.orchestra.run.smithi162.stderr: 92.7% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.6.log.gz 2024-09-02T11:03:39.594 INFO:teuthology.orchestra.run.smithi151.stderr: 91.6% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.2.log.gz 2024-09-02T11:04:09.917 INFO:teuthology.orchestra.run.smithi162.stderr: 91.4% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.4.log.gz 2024-09-02T11:06:09.185 INFO:teuthology.orchestra.run.smithi162.stderr: 92.6% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.5.log.gz 2024-09-02T11:07:44.783 INFO:teuthology.orchestra.run.smithi162.stderr: 91.3% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.7.log.gz 2024-09-02T11:07:44.786 INFO:teuthology.orchestra.run.smithi162.stderr: 2024-09-02T11:07:44.786 INFO:teuthology.orchestra.run.smithi162.stderr:real 26m45.929s 2024-09-02T11:07:44.786 INFO:teuthology.orchestra.run.smithi162.stderr:user 9m21.392s 2024-09-02T11:07:44.786 INFO:teuthology.orchestra.run.smithi162.stderr:sys 0m28.511s 2024-09-02T11:22:11.605 INFO:teuthology.orchestra.run.smithi151.stderr: 91.1% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.1.log.gz 2024-09-02T11:23:17.128 INFO:teuthology.orchestra.run.smithi151.stderr: 91.4% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.3.log.gz 2024-09-02T11:24:23.709 INFO:teuthology.orchestra.run.smithi151.stderr: 91.1% -- replaced with /var/log/ceph/1530c7be-6911-11ef-bcd4-c7b262605968/ceph-osd.0.log.gz 2024-09-02T11:24:23.766 INFO:teuthology.orchestra.run.smithi151.stderr: 2024-09-02T11:24:23.766 INFO:teuthology.orchestra.run.smithi151.stderr:real 43m24.916s 2024-09-02T11:24:23.766 INFO:teuthology.orchestra.run.smithi151.stderr:user 14m14.002s 2024-09-02T11:24:23.766 INFO:teuthology.orchestra.run.smithi151.stderr:sys 0m40.098s 2024-09-02T11:24:23.767 INFO:tasks.cephadm:Archiving logs... 2024-09-02T11:24:23.768 DEBUG:teuthology.misc:Transferring archived files from smithi151:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453/remote/smithi151/log 2024-09-02T11:24:23.769 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-02T11:26:58.115 DEBUG:teuthology.misc:Transferring archived files from smithi162:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453/remote/smithi162/log 2024-09-02T11:26:58.139 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-02T11:28:35.318 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453/remote/smithi187/log 2024-09-02T11:28:35.319 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-02T11:29:31.894 INFO:tasks.cephadm:Removing cluster... 2024-09-02T11:29:31.894 DEBUG:teuthology.orchestra.run.smithi151:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 --force 2024-09-02T11:29:32.678 INFO:teuthology.orchestra.run.smithi151.stdout:Deleting cluster with fsid: 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T11:29:33.800 DEBUG:teuthology.orchestra.run.smithi162:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 --force 2024-09-02T11:29:34.043 INFO:teuthology.orchestra.run.smithi162.stdout:Deleting cluster with fsid: 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T11:29:34.936 DEBUG:teuthology.orchestra.run.smithi187:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1530c7be-6911-11ef-bcd4-c7b262605968 --force 2024-09-02T11:29:35.239 INFO:teuthology.orchestra.run.smithi187.stdout:Deleting cluster with fsid: 1530c7be-6911-11ef-bcd4-c7b262605968 2024-09-02T11:29:35.944 INFO:tasks.cephadm:Removing cephadm ... 2024-09-02T11:29:35.945 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-02T11:29:35.964 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-02T11:29:35.984 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-02T11:29:36.000 INFO:tasks.cephadm:Teardown complete 2024-09-02T11:29:36.000 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-02T11:29:36.011 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-02T11:29:36.012 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-02T11:29:36.017 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-02T11:29:36.027 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-02T11:29:36.204 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-02T11:29:36.212 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-02T11:29:36.213 DEBUG:teuthology.orchestra.run.smithi151:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T11:29:36.215 DEBUG:teuthology.orchestra.run.smithi162:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T11:29:36.221 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-02T11:29:36.229 INFO:teuthology.orchestra.run.smithi151.stderr:bash: line 1: ntpq: command not found 2024-09-02T11:29:36.233 INFO:teuthology.orchestra.run.smithi162.stderr:bash: line 1: ntpq: command not found 2024-09-02T11:29:36.235 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-09-02T11:29:36.262 INFO:teuthology.orchestra.run.smithi162.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T11:29:36.262 INFO:teuthology.orchestra.run.smithi162.stdout:=============================================================================== 2024-09-02T11:29:36.262 INFO:teuthology.orchestra.run.smithi162.stdout:^* hv01.front.sepia.ceph.com 3 6 377 41 -67us[ -66us] +/- 47ms 2024-09-02T11:29:36.262 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 104 -451us[ -450us] +/- 58ms 2024-09-02T11:29:36.262 INFO:teuthology.orchestra.run.smithi162.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 104 +660us[ +662us] +/- 74ms 2024-09-02T11:29:36.263 INFO:teuthology.orchestra.run.smithi162.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:29:36.264 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T11:29:36.264 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-09-02T11:29:36.264 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv01.front.sepia.ceph.com 3 6 377 37 -19us[ -21us] +/- 47ms 2024-09-02T11:29:36.264 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 -450us[ -452us] +/- 59ms 2024-09-02T11:29:36.264 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 +638us[ +636us] +/- 74ms 2024-09-02T11:29:36.264 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:29:36.297 INFO:teuthology.orchestra.run.smithi151.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-02T11:29:36.297 INFO:teuthology.orchestra.run.smithi151.stdout:=============================================================================== 2024-09-02T11:29:36.297 INFO:teuthology.orchestra.run.smithi151.stdout:^* hv01.front.sepia.ceph.com 3 6 377 40 -38us[ -70us] +/- 47ms 2024-09-02T11:29:36.297 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 109 -453us[ -500us] +/- 58ms 2024-09-02T11:29:36.297 INFO:teuthology.orchestra.run.smithi151.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +633us[ +600us] +/- 74ms 2024-09-02T11:29:36.297 INFO:teuthology.orchestra.run.smithi151.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-02T11:29:36.303 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-02T11:29:36.338 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-02T11:29:36.338 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-02T11:29:36.425 DEBUG:teuthology.orchestra.run.smithi151:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-02T11:29:36.574 DEBUG:teuthology.orchestra.run.smithi162:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-02T11:29:36.645 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-02T11:29:36.719 DEBUG:teuthology.orchestra.run.smithi151:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-02T11:29:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:29:36.751 DEBUG:teuthology.orchestra.run.smithi162:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-02T11:29:36.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:29:36.781 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-02T11:29:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:29:36.808 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-02T11:29:36.816 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-02T11:29:36.922 INFO:teuthology.task.internal:Duration was 6038.281130 seconds 2024-09-02T11:29:36.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-02T11:29:37.129 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-02T11:29:37.129 DEBUG:teuthology.orchestra.run.smithi151:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-02T11:29:37.132 DEBUG:teuthology.orchestra.run.smithi162:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-02T11:29:37.135 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-02T11:29:37.197 INFO:teuthology.orchestra.run.smithi151.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T11:29:37.205 INFO:teuthology.orchestra.run.smithi162.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T11:29:37.226 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-02T11:29:37.649 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-02T11:29:37.650 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi151.front.sepia.ceph.com 2024-09-02T11:29:37.650 DEBUG:teuthology.orchestra.run.smithi151:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-02T11:29:37.675 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi162.front.sepia.ceph.com 2024-09-02T11:29:37.676 DEBUG:teuthology.orchestra.run.smithi162:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-02T11:29:37.713 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2024-09-02T11:29:37.713 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-02T11:29:37.748 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-02T11:29:37.749 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-02T11:29:37.751 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-02T11:29:37.755 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-02T11:29:37.818 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-02T11:29:37.818 DEBUG:teuthology.orchestra.run.smithi151:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-02T11:29:37.823 DEBUG:teuthology.orchestra.run.smithi162:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-02T11:29:37.825 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-02T11:29:40.195 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-02T11:29:40.501 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-02T11:29:40.502 DEBUG:teuthology.orchestra.run.smithi151:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-02T11:29:40.529 DEBUG:teuthology.orchestra.run.smithi162:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-02T11:29:40.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-02T11:29:40.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-02T11:29:40.691 DEBUG:teuthology.orchestra.run.smithi151:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-02T11:29:40.694 DEBUG:teuthology.orchestra.run.smithi162:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-02T11:29:40.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-02T11:29:40.729 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2024-09-02T11:29:40.735 INFO:teuthology.orchestra.run.smithi151.stdout:kernel.core_pattern = core 2024-09-02T11:29:40.736 INFO:teuthology.orchestra.run.smithi162.stdout:kernel.core_pattern = core 2024-09-02T11:29:40.783 DEBUG:teuthology.orchestra.run.smithi151:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-02T11:29:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:29:40.798 DEBUG:teuthology.orchestra.run.smithi162:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-02T11:29:40.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:29:40.833 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-02T11:29:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-02T11:29:40.850 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-02T11:29:40.859 INFO:teuthology.task.internal:Transferring archived files... 2024-09-02T11:29:40.861 DEBUG:teuthology.misc:Transferring archived files from smithi151:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453/remote/smithi151 2024-09-02T11:29:40.861 DEBUG:teuthology.orchestra.run.smithi151:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-02T11:29:40.944 DEBUG:teuthology.misc:Transferring archived files from smithi162:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453/remote/smithi162 2024-09-02T11:29:40.944 DEBUG:teuthology.orchestra.run.smithi162:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-02T11:29:40.986 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-02_08:22:44-rados-wip-bharath3-testing-2024-09-01-1328-distro-default-smithi/7886453/remote/smithi187 2024-09-02T11:29:40.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-02T11:29:41.029 INFO:teuthology.task.internal:Removing archive directory... 2024-09-02T11:29:41.029 DEBUG:teuthology.orchestra.run.smithi151:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-02T11:29:41.032 DEBUG:teuthology.orchestra.run.smithi162:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-02T11:29:41.034 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-02T11:29:41.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-02T11:29:41.090 INFO:teuthology.task.internal:Not uploading archives. 2024-09-02T11:29:41.090 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-02T11:29:41.287 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-02T11:29:41.287 DEBUG:teuthology.orchestra.run.smithi151:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-02T11:29:41.289 DEBUG:teuthology.orchestra.run.smithi162:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-02T11:29:41.292 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-02T11:29:41.302 INFO:teuthology.orchestra.run.smithi151.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 2 11:29 /home/ubuntu/cephtest 2024-09-02T11:29:41.307 INFO:teuthology.orchestra.run.smithi162.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 2 11:29 /home/ubuntu/cephtest 2024-09-02T11:29:41.309 INFO:teuthology.orchestra.run.smithi187.stdout: 262217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 2 11:29 /home/ubuntu/cephtest 2024-09-02T11:29:41.332 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-02T11:29:41.390 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-02T11:29:41.685 INFO:teuthology.run:Summary data: description: rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} duration: 6038.281130313873 failure_reason: '"2024-09-02T10:10:00.000156+0000 mon.a (mon.0) 1594 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 279/37245 objects degraded (0.749%), 2 pgs degraded, 3 pgs undersized" in cluster log' flavor: default owner: scheduled_skanta@teuthology success: false 2024-09-02T11:29:41.686 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-02T11:29:41.792 INFO:teuthology.run:FAIL